Step 14 — Rolling back, and a deliberate disaster¶
About 25 minutes. Branch: step-14 (and step-14-broken).
The one new idea: a failed deploy is not an outage — but a successful one can be.
This is the most useful step in the tutorial
Steps 3 and 6 failed and the platform caught them. This one succeeds and is broken anyway. Nothing catches it. That is the failure that reaches your users, and the only defence is being able to undo it quickly.
First, note the good version¶
Every successful deploy is there with its image tag, and the live one is marked. Write the current tag down. In a real incident you will be doing this under pressure.
Now break it in the way nothing can catch¶
In backend/app/main.py, sort the board by a column that does not exist:
Push and deploy.
The deploy goes green. The build compiled, the container started, the health
check passed — because /api/health does not touch the database. Every part of
the machinery reported success.
Open the board. It is empty, and the browser console has a 500.
Why no gate could have stopped it¶
A health check proves the process is up. It cannot prove the app is right. You could point it at a path that queries the database and catch this particular bug — and the next one would be in a code path the health check does not exercise, because a health check that exercised everything would be the application.
So the answer is not a better gate. It is speed of reversal.
Roll back¶
This is the one place you use deploy, not redeploy
Everywhere else the tutorial ships changes with mecca app redeploy, which
rebuilds from your repository. Here you want the opposite: deploy
ships an image that already exists, so it cannot be broken by whatever
broke the build you are rolling back from. That is the property that makes
it a reliable recovery command.
The board comes back in about the time a deploy takes.
--no-build is doing real work there. It redeploys an image that already
exists, so nothing compiles — which means nothing that broke the last build can
break this one, and you are not waiting on a build while the site is down. In an
incident, always --no-build.
Rollback is manual
Nothing here notices a green deploy that serves errors and reverts it for you. The two commands above, known before you need them, are the recovery plan.
Then fix it properly — tagg to tag — and deploy forwards.
The neighbouring tools¶
mecca app retry tally # a deploy that failed for a passing reason
mecca app restart tally -b tutorial # the code is fine, the process is not
mecca app stop tally -b tutorial
mecca app start tally -b tutorial
mecca app restart tally worker -b tutorial # one service
retry is for the transient case — it re-runs the failing phase rather than
starting over. restart is for when nothing is wrong with the version and the
process has got itself stuck.
What you should see¶
- A deploy that succeeds and a board that 500s.
/api/healthreturning 200 the entire time.- A rollback that restores service without a build.
If it didn't work¶
| Symptom | Try | Usually |
|---|---|---|
build list shows one entry |
— | Only one successful build so far; deploy the fix first, then break it |
| Rollback rebuilt anyway | check the flag | --no-build missing |
| The tag is rejected | mecca build list |
Tags are per branch |
| Rolled back, still broken | mecca logs --type container |
The fault is in data or a variable, not the image |