Cloud & Infrastructure

Docker in Production: What Tutorials Don't Tell You

If your container loses data after a restart, or images have filled up the disk, this guide points to exactly the places where introductory tutorials stop short.

Cloud & Infrastructure

The server comes up, the container comes up, but after a docker restart everything is gone. Or worse: the server disk is full and docker system df shows 47 gigabytes of unused images just sitting there. These aren't Docker bugs; they're bugs in how Docker is used in production. The difference between "it works on my laptop" and "it runs for three months untouched" comes down to four specific decisions: image size, persistent data, log management, and restart policy.

How Docker image size gets out of control in production

Every time you run docker build and overwrite the :latest tag, the previous layers remain as dangling images. On a server that deploys twice a week, this number reaches tens of gigabytes after two months. I've seen an 80-gigabyte server where 52 gigabytes were images and nobody knew.

The immediate fix:

docker image prune -a --filter "until=168h"

The --filter "until=168h" flag means only images older than seven days are removed. If you run -a without a filter, images that stopped containers depend on also get deleted, and next time you'll have to pull them from the registry again. This is a cost many don't pay upfront and then pay at the worst possible time.

Why your image is 1.2 gigabytes and should be 180 megabytes

A simple FROM node:20 is about 1.1 gigabytes. The same application with FROM node:20-alpine comes down to about 180 megabytes. The difference is in the base image, not your code. But Alpine uses musl libc, and some native packages like certain versions of sharp or database drivers don't have prebuilt binaries for it and must be compiled. If your team doesn't have time to debug the Error loading shared library ld-linux-x86-64.so.2 error, node:20-slim is a more sensible choice: about 240 megabytes, but fully compatible with glibc.

The second point that doubles the size is layer ordering. If you put COPY . . before RUN npm ci, every change to a CSS file invalidates the package installation layer's cache. The correct order: first COPY package*.json ./, then RUN npm ci --omit=dev, and finally copy the rest of the code.

Persistent data: where the most data is lost

The filesystem inside a container is temporary. Everyone has heard this sentence, but it's usually taken seriously only after they've lost a database once. If you run MySQL inside a container and haven't mounted /var/lib/mysql on a volume, the first docker compose down takes everything away.

services:
  db:
    image: mysql:8.4
    volumes:
      - db_data:/var/lib/mysql
    restart: unless-stopped
volumes:
  db_data:

Two points that are rarely mentioned in tutorials. First, a bind mount with a host path like ./data:/var/lib/mysql gives a Permission denied error on SELinux or when the user inside the container has a different UID; a named volume doesn't have this problem because Docker manages ownership itself. Second, a named volume lives in /var/lib/docker/volumes, and if one day you want to move servers, you need to know which volume you're copying where.

For user-uploaded files, a local volume is a bad solution. When you have two servers behind a load balancer, a file is uploaded to server A and the next request hits server B and gets a 404. This is where object storage is the right answer: both servers write to a shared resource and containers stay stateless.

This is where they go wrong

The common mistake is someone running docker compose down -v to "clean everything up." The -v flag deletes exactly the named volumes that hold the database data. The sign is this: after coming back up, the application works, login works, but no users exist and the tables are empty. If you don't have a backup, it's over. If you do, you have several hours of downtime. Before any down, get in the habit of running docker compose config --volumes to see what's at risk.

Container logs: the silent disk killer

Docker uses the json-file driver by default and puts no limit on log size. An application with verbose logging can produce several gigabytes of files in /var/lib/docker/containers/ in two weeks. The symptoms are clear: df -h shows the disk is 94% full, but du -sh /var/www shows only 2 gigabytes.

The solution is to limit it at the daemon level so every new container that comes up is automatically covered. In /etc/docker/daemon.json:

{
  "log-driver": "json-file",
  "log-opts": {
    "max-size": "50m",
    "max-file": "5"
  }
}

After the change, systemctl restart docker is required, and this restart brings down all containers. Containers configured with restart: always come back up on their own; the rest manually. This limit only applies to new containers, so running containers must be recreated once.

If you send logs somewhere for later analysis, the json-file driver is the wrong choice. For high volume, local or sending directly to a collector is more sensible. But for most small teams, the same 50-megabyte rotation is enough and doesn't require extra complexity.

Restart policy and service startup order

restart: always looks correct at first glance, but it has a trap: if the container keeps crashing due to a configuration error, Docker tries infinitely and fills up the logs. restart: unless-stopped behaves better: it comes up automatically after a server restart, but if you've stopped it manually, it stays untouched.

The second problem is order. depends_on only guarantees startup order, not that the service is ready. That means your application may come up before MySQL is ready and die with Connection refused. The real solution is a healthcheck:

healthcheck:
  test: ["CMD", "mysqladmin", "ping", "-h", "localhost"]
  interval: 10s
  timeout: 5s
  retries: 5
  start_period: 30s

And in the dependent service, write depends_on with a condition: condition: service_healthy. Without this, every server restart is a game of chance.

When Docker is the wrong choice

If your workload is a single application with steady traffic, Docker is an extra layer you have to maintain. systemd with a simple unit file gives you the same automatic restart and log management, without network overhead and without image management. For a single-file service, I'd choose systemd. Docker wins when you have multiple services with different dependency versions, or you want dev and prod environments to be identical, or you see a migration path to orchestration.

And if the number of services passes ten, manually managing compose across multiple servers quickly becomes unmaintainable. That's where managed Kubernetes proves its worth, because it handles rollout, health checks, and load distribution itself.

Checklist before you leave the server to itself

  1. Every stateful service has a named volume and you know its path.
  2. You've set a log size limit in daemon.json and recreated the containers once.
  3. The restart policy is unless-stopped, not always.
  4. Dependencies are configured with healthcheck and service_healthy.
  5. A weekly cron runs docker image prune -a --filter "until=168h".
  6. Volume backups go to a separate destination, not the same disk.

Take the sixth item seriously. A backup on the same disk the database is on isn't a backup; it's a copy. For heavy workloads, combining a cloud server with an NVMe disk and a separate storage destination is the least hassle. If you're still choosing infrastructure, cloud infrastructure (IaaS) gives the flexibility needed for this setup, but make the Docker versus systemd decision independently of it.

The next step is clear: right now, run docker system df -v and see what's eaten your disk. The number you see is probably bigger than you think.

Frequently asked questions

Why does my data disappear after a container restart?

Because the data was written inside the container's filesystem, and that filesystem is destroyed when the container is removed or recreated. You need to mount the data path on a named volume or bind mount. If you've mounted the path correctly but data is still lost, you've probably run docker compose down -v, which deletes the volume itself.

How do I find out what's filling up Docker's disk space?

The docker system df -v command shows a detailed breakdown of images, containers, volumes, and build cache. If the volume is in the logs section, look for *-json.log files in /var/lib/docker/containers/. To clean up unused images, run docker image prune -a with a time filter so active images aren't deleted.

What's the difference between restart always and unless-stopped in Docker?

Both bring the container up after a server restart. The difference is in manual behavior: with always, if you stop the container yourself, Docker brings it back up; with unless-stopped, it stays stopped until you start it again yourself. For production, unless-stopped is the safer choice because it lets you deliberately take the service down for maintenance.

Should I run the database inside Docker?

For development, yes, because it simplifies setup. For production, if your team doesn't have enough experience managing volumes, backups, and recovery, running the database directly on the server or using a managed service is lower risk. If you choose Docker, you should at least have automatic volume backups and periodic recovery testing.

ServerNet Support

ServerNet engineering & editorial team — specialists in infrastructure, networking and web hosting.

Cloud Infrastructure (IaaS)
Share:

Comments 0

No comments yet — be the first!

Leave a comment

Related service

Cloud Infrastructure (IaaS)

Servers, private networks, firewalls and storage — all API-driven and billed hourly. Infrastructure as code that scales with you.