notesarchiveworkabout

restart: always turns a broken container into a silent one

A container crash-looping under restart:always reports as "Up" between attempts, so uptime checks pass while the service is unavailable. A healthcheck plus restart:unless-stopped surfaces the failure instead of hiding it.

restart: always is the first thing most people add to a compose file, and it does what it says: Docker restarts the container whenever it exits. The problem is what that looks like from outside.

A container that starts, fails after four seconds, and gets restarted spends most of its life in running. docker ps shows Up 2 seconds every time you look. An uptime check that asks “is the container running?” answers yes, continuously, while the service has not served a single request all day.

Make the state honest

A healthcheck moves the truth into the status field, where monitoring can see it:

services:
  webapp:
    image: ghcr.io/example/webapp:1.4.2
    container_name: webapp
    restart: unless-stopped
    ports:
      - "127.0.0.1:8080:8080"
    volumes:
      - ./config:/config
      - ./cache:/cache
      - type: bind
        source: /srv/data
        target: /data
        read_only: true
    healthcheck:
      test: ["CMD-SHELL", "curl -fsS http://localhost:8080/health || exit 1"]
      interval: 30s
      timeout: 5s
      retries: 3
      start_period: 60s
    logging:
      driver: json-file
      options:
        max-size: "10m"
        max-file: "3"

Three deliberate choices in there:

  • restart: unless-stopped rather than always. The difference only shows up after a reboot: always restarts a container you deliberately stopped, unless-stopped respects the stop. If you stopped it to take something out of service, you want it to stay out.
  • start_period: 60s. Without it, a slow first boot counts as failed healthchecks and the container is marked unhealthy while it is legitimately still starting.
  • Log rotation. The json-file driver is unbounded by default. A container in a crash loop writes its startup error a few thousand times a day, and the first symptom is a full disk on an unrelated service.

Finding the loop after the fact

Restart count is the giveaway. It is cheap to check and does not lie:

# Anything with a non-zero restart count is worth a look.
docker inspect --format '{{.Name}} restarts={{.RestartCount}} status={{.State.Health.Status}}' $(docker ps -q)

# The logs from the previous run, not the current one.
docker logs --tail 50 --timestamps webapp

# Why the last exit happened.
docker inspect --format '{{.State.ExitCode}} {{.State.Error}}' webapp

.State.Health.Status is only populated when a healthcheck is defined; on containers without one the template renders <no value>, which is itself a useful signal about which services you still have no visibility into.

Docker’s own backoff makes the loop slower over time, starting at 100ms and doubling up to a minute, so a container that has been failing since morning may only be restarting once a minute by the time you notice. Do not read a low restart rate as recovery.