Health Checks and a Fixed Release Order for a Docker Stack

A returned command is not a healthy system

A deploy script that exits with zero has only proved that its commands ran. The containers may be restarting, the database may still be starting, or the application may be up and unable to reach its queue. Health has to be asked, not assumed.

Ask at every layer

One check at the front door hides which layer is failing. Give each service its own check, close to what it does: the application process answers a ping, the web server serves a health path, the database answers a ping, the cache answers a ping. When something is red, the check already says where.

Wait with a timeout

Services take time to become ready, so a script should poll instead of sleeping for a guessed number of seconds. Poll every couple of seconds, stop at a clear limit, and fail with a message naming the service. A deploy that waits forever is as bad as one that does not wait.

Keep the release steps in one order

The order matters. Migrate the schema first, because the new code may need it. Rebuild caches next, so they describe the new code. Reload the workers so they stop running the old code. Verify the health endpoint last, because it is the proof that everything above worked. Write the order down once and let the script own it.

Separate "alive" from "which version"

A health check says the system is alive. It does not say what is running. A small version endpoint, separate from the health check, answers the question you ask right after a release: did the new build go live?

In our own products

The Rveta stack gives the application, the web server, MySQL and Redis each their own container check, and its bootstrap script waits for services to report healthy within a timeout before continuing. RumuzePMO's deploy script follows the order above and ends by checking its health route. Rumuze Core exposes its running version on an internal endpoint that is separate from its health check.