Designing Backend Architectures That Survive

Start with the states, not the screens

Most backend problems I have inherited were not performance problems. They were disagreements about what a record is allowed to be.

Before writing a feature I now write down the states a record can pass through and the transitions between them. In Touchstone a test task moves through offered, claimed, submitted, reviewed, paid. Writing that list first made the API obvious, and it made the hard cases visible: what happens when a tester abandons a claimed task, when two testers submit the same task, when a payment fails after approval.

Screens are a consequence of that list. Meanwhile a schema designed from screens ends up with nullable columns that encode the missing decisions.

Make writes repeatable

Ingestion jobs fail and get retried. If a retry duplicates data, every number downstream is wrong and nobody notices for weeks.

The pattern I use is an idempotent write keyed on the natural identity of the row: an upsert on the pair that should be unique, so running the same import twice produces the same table. In Politrack, posts and comments are keyed by their platform identifiers; a re-run updates the text and moves on.

It costs one unique index and a few lines. It saves the class of bug that takes a day to find and destroys trust in the data.

Move the bytes out of the API

Evidence files are the other recurring mistake. A report with a screen recording should not be uploaded through the application server, and it should not be stored in the database.

Touchstone has agents request a presigned upload, put the file directly into object storage, then send the reference. The API never touches the body, the worker reads it from storage when it needs to analyse it, and the recording is served through a signed URL that expires.

The same shape applies to any product with user files: the API handles permissions and metadata, storage handles bytes.

Assume one machine

My own platform runs on a single VPS: Next.js, NestJS, Postgres, MinIO, an nginx cache, and a local model for fallbacks. That constraint is useful. When memory is finite you find the query that materialises a million rows, and when disk is finite you notice that nobody ever pruned the logs.

Two habits came out of it. Every expensive path has a limit, and every night produces a backup that someone has actually restored. I keep a written restore procedure and rehearse it, because a backup that has never been restored is a file, not a backup.

Deployments should be boring

The goal is a release where nothing surprising happens: a reproducible build, a migration that can be run twice, a way back, and a health check that fails loudly. Excitement in a deployment is a bug report waiting to be written.

That is less impressive than a distributed architecture diagram, and it is what keeps a product online for a year without a team watching it.