Self Hosted Docker Forensics Security

Building on our previous article we will be starting off this series with a bang! Docker Forensics. We often download docker images and run them on our server without a thought, but vulnerabilities could be lurking. There are tons of vulnerabilities on tons of software, and this article isn't going to outline them all. We will be focusing mainly on the detection and response side of the house today. Don't worry, later we will cover things like vulnerability scanning and detection.

May I introduce Baddock

I have built a little PHP application for our security article here. This application is pretty simple, it has an upload file that will take pretty much any file, and can be used to execute code on the servers side. I figure we should look at a semi-real world example instead of something crazy and improbable of happening. Finding someone with a docker image that exposes ssh with root logins enabled doesn't happen every day and this is a little more realistic. Should you decide to play with the code, please remember, it's vulnerable as all get out, and not a good idea to expose to the internet.

Let's say this code is running on a server that is exposed to the internet and someone uploads something bad to the server, how would we know? Well, there are lots of ways of telling as you are about to see and some important forensic data you can collect. Let's start with the simple stuff first. Docker Top.

Docker Top

Docker top will show us what's going on inside our container. I'm sure most of us have used this before to troubleshoot if something is running correctly in our containers, but I am covering it here for completeness. You can run Docker Top by simply running the command docker top CONTAINER_NAME obviously you want to change container_name to your container name, or even the image ID.

The Plot Thickens!

Well, I'm no genius, but I think I see a problem. So for those who don't know, typically the php:apache docker image doesn't normally run python3 cryptominer.py. I wonder how that got there? I see something saying /var/www/html/uploads as well. I bet that was the issue. But let's see what else might have been written to the container.

What Else Happened?

Did you know Docker has the ability to see what changed inside an image while it's running? Let me introduce you to Docker Diff. As the with most docker commands, all you need to type is docker diff CONTAINER_ID and you will see a list of all files written, changed and deleted inside the container since it started running. Neat isn't it? Let's look at the docker diff from my hacked PHP application.

So, the C there indicates a file was changed, and the A shows what was added. We can see 2 files were added under /var/www/html/uploads. the cryptominer.py and something called rome-shell.php. I'm no expert, but I bet that rome-shell.php is a webshell for exploiting servers. Let's google and confirm.

So, we know have some great information here. Let's outline what we know and what we need to do:

What we know

• a web php shell was uploaded to our application
• a cryptominer.py script was uploaded to our application
• someone used the webshell to execute the cryptominer.py

Next Steps

• Save the logs
• Export this container as is for evidence
• Hash the file so we can prove the file wasn't modified

We Need All The Logs!

It's common for people to freak out and shutdown everything when they find something being exploited. Please resist the urge. Let's collect some logs. This can be done by simply running docker logs CONTAINER_ID and then redirecting the output to a file. I also normally cat that file to ensure it worked.

Saving the Evidence

Did you know Docker has a way for you to save a container after it's modified, even if they are not using a volume? Let me introduce you to Docker Commit. This will save the image as is with all the modifications and logs. Apache saves it's logs at /var/log/apache normally so we want to save that in the image as well. Super simple as with most docker commands. docker commit CONTAINER_ID NEW_IMAGE_NAME. Let's commit this as a name we will remember.

Packaging Up The Evidence

Now we need to export this image so we can send it into a forensic expert or a firm. Docker again has another easy command for us docker export IMAGE > filename.tar This is basically a tar export. I normally also generate a sha256hash of the file so we can ensure the data isn't corrupted in transit. This is important in forensics.

And there we have it. Docker Forensics. This was a slightly longer article, but we covered a lot of ground and I hope you learned something along the way. Later we will dive into even more forensic data including tearing apart a program and reverse engineering malware.