Situation

So I had a production server that was setup ad-hoc that I never setup proper log rotation in the docker container and so the logs continued to grow and eat up disk usage. It was getting to the point that I was at about 90% disk usage, so I really needed to do something about it.

Step 1

The first step was to just clear out the 20GB worth of logs. After some searching, I found that Docker stores all of the container logs in a specific place on the host that you can deal with in whatever way seems best.

Apparently a common approach to wiping logs is to use the “truncate” command.

sudo truncate -s 0 /var/lib/docker/containers/$CONTAINER_ID/$CONTAINER_ID-json.log

I have never used the truncate command before but my basic understanding of it is that you can set the size of a file, similar to how you can set the length of an array in a programming language. And if you set the length of an array to 0 then all of the data is wiped. Similarly with the truncate command if you set the size to 0 then you effectively remove all of the data from the file without deleting the file itself.

Step 2

Now I needed to setup some automation around log rotation so I didn’t need to come back and ever manually clean up logs again. Fortunately, docker already has a mechanism for this I just need to set it up.

One option is to configure it on the container when you first run it. I chose this time around to instead configure it on the docker daemon itself, that way future developers won’t accidentally forget to set it.

I created this file: /etc/docker/daemon.json And added this:

{
  "log-driver": "json-file",
  "log-opts": {
    "max-size": "1g",
    "max-file": "7"
  }
}

This is overkill for what’s necessary, but the thought is leaving room for a single day to generate 1 gigabyte worth of logs and to store a week’s worth of logs. In reality we’re probably generating a little over half a gig of logs a day on average so we’re probably storing about 2 weeks worth of logs at this point. But in reality I only really need a day or two worth of logs but I can tolerate using 7 gigs on the machine at this point. That may change at some point but it’s better than before.