At work we are migrating towards a DevOps culture within the IT department, and that comes with some security nuances. Moving forward, our applications will be more centered around container development and Kubernetes deployment. This presents the security team with an opportunity to easily gain more control over the application/execution layer of our apps. Docker and the OCI (Open Container Initiative) provide SecOps with various ways to secure containers. My experience with Docker has made me the person on our team to tackle our security baseline policy and DevSecOps workflow as we move into the Kubernetes domain.
After doing some research, I’ve decided that the first task I’ll try to tackle is to develop a good workflow for developing seccomp profiles. Seccomp is a Linux kernel security module that allows you to limit which syscalls an application can make. This greatly reduces the capabilities of an exploited application. Docker has simplified the implementation of seccomp profiles by allowing you to configure a container’s profile with just a single json file, so that’s what I’ll be implementing in our deployments.
The only drawback to this plan is that I will probably never know anything about the code base and which syscalls the container needs. Instead of scanning the code base, I’m thinking about developing some kind of runtime monitor for identifying the syscalls a container needs. I’ll be doing lots of initial testing in my homelab environment which also runs on Docker. That way I have at least some idea about the containers as I’m just getting my feet wet. My goal is to identify a repeatable workflow and automate as much of it as possible so I still have time for my other job responsibilities. I’ll be using this page to track my progress so others can learn with me.
Initial Findings:
One of my co-workers sent me an article from Red Hat written by Valentin Rothberg about container security with seccomp. Lo-and-behold, the guy that wrote the article helped build a program that does exactly what I’m trying to do. His program is actually an OCI (Open Container Initiative) hook that monitors syscalls during the execution of the container using eBPF (extended Berkeley Packet Filter). Once the container finishes running, you can export the syscalls to a json file in the format needed for a seccomp filter.
I’m very grateful to my coworker for sending me this article, because this program is exactly what I needed. It uses Podman’s runtime instead of the Docker daemon, but the two are theoretically backward compatible. I tested the seccomp eBPF hook following the instructions in Valentin’s article, and initially it appeared to be not working. It wasn’t creating an output file with the seccomp profile. I reached out to Valentin on Twitter, and he was very helpful! He said that he suspected that the hook wasn’t running at all if it wasn’t giving me any kind of error nor outputting the json profile.
After doing some log searching, it turned out that Podman was looking in the directory ‘/usr/share/containers/oci/hooks.d’ for OCI hooks, and that directory was empty for me (in fact, it was non-existent) because I had installed the hook from source via the Github repo. Since I ran the manual installer, it installed the hook to ‘/usr/local/share/containers/oci/hooks.d’.
To solve this, I could either copy the contents to the correct directory or I could install the package with dnf since I was using Fedora 32 Server. I ended up installing the package (oci-seccomp-bpf-hook is the name — it is incorrect in the article). Once I had installed it in the way that Podman expects, it was working flawlessly for Valentin’s examples in the article. Next I will be testing it on more complex containers than just an ‘ls’ command.
Testing on my foxception container:
I wrote an earlier blog post about a container image that I made that runs a full-screen browser on a web server. I’ve published the code on my Github page if you want to check it out quick before reading any more. I wanted to test the seccomp eBPF hook on this container for a couple of reasons. First of all, it has a couple of different larger components that are working together to make it function the way it does, and I thought that would be good to see how the hook handles that. Second, it was made for Docker which will help me test the processes and potential quirks of having to run Docker images with Podman instead of the Docker daemon.
So far, all of my tests have failed. I am getting some kind of OCI runtime error when the hook tries to execute. I’ve tried both pulling the Docker-built image from my Dockerhub repo, as well as actually building it with Podman via my Github repo files. Both situations resulted in the same error. I think I will have to do some more reading on OCI hooks and the low-level differences between Podman and the Docker daemon.
I ran journalctl to check logs after I found out that the seccomp eBPF hook generates logs there. I got the following fatal error message:
paths must be absolute: \"\": invalid annotation: please refer to the syslog (e.g., journalctl(1)) for more details.
I did some more digging, but I also opened up an issue on their Github page with the error message and steps to reproduce. It turned out the issue was that I was using a relative path for the output file. The error message hinted at that for me, but it was a little unclear exactly where the error was because of the incorrect “\” variable for my output file path. I looked into the Github repo, and it turns out this was a mistake in the code. When the outputFile variable held a relative path, it was printing the error message with the inputFile variable (which I didn’t specify when I generated the error). I forked the repo, clarified the error messages, fixed the variable, and submitted a pull request (my first one on Github!).
Once I put the absolute path for the seccomp profile, it worked perfectly! I ran the hook on my foxception container without doing any browsing and then again doing some normal web browsing. It turns out that the container needed a few more syscalls just to do normal browsing! Since my testing, I’ve updated my foxception repo on Github, and it now contains the generated seccomp profile so you can check it out yourself!
Next, I want to figure out how to set this OCI hook up in a CI/CD pipeline so that it automatically generates the seccomp profile during unit testing. That way I can be sure that all of the intended functionality has been whitelisted.
CI/CD Pipeline Integration:
To be continued…