When the Code Ships Before the Migration: Degrading One Schema Level at a Time

Every application with a database has a window where the code and the schema disagree. The migration runs before the deploy, or after it, or someone forgets it for a day. This article is for anyone reading from PostgreSQL through PostgREST, including Supabase, where that window can take a whole feature down. It is a regression I shipped to production in ResearchForge and the two fixes it took.

Migration 004 added four provenance columns to the analyses table: model_provider, fallback_used, fallback_provider and processing_time_ms. The code that reads the research library was updated to select them in its embedded analyses(...) resource, and deployed. The migration had not been run yet.

PostgREST does not omit a column it cannot find. A select naming a missing column fails the whole request with PostgreSQL error 42703, undefined_column. So every read of the library returned an error, the backend mapped it to 502, and My Papers went blank for every user. Four metadata fields had taken the entire library offline. It was found in production immediately after deploying, not in review.

The repository now recognises exactly that error and raises an internal _SchemaBehindError. The read paths, listing papers, fetching one paper and fetching papers for a review, retry with the older column list. Saving drops the new keys and writes the analysis anyway, because the analysis matters more than the metadata about it.

Two details keep the fix from becoming a new problem. Only 42703 triggers the retry: an unrelated 400 still fails, otherwise a genuine query bug would be retried and reported as healthy forever. And the discovery is remembered in a module-level variable, so it costs one extra round trip per warm serverless process rather than one per request. It only ever steps down, since a column cannot un-exist, and a cold start re-learns it, so deploying after the migration picks the columns up with no intervention.

A few hours later, migration 005 was about to add one more column, cache_hit. Checking whether that change was safe to deploy before 005 had been run showed a flaw in the first fix: it dropped every provenance column the moment any one was missing.

Production was in exactly the state that exposed it, with 004 applied and 005 not. The first read would fail on cache_hit, and the retry would give up 004's four columns as well, columns the database had perfectly. Every entry in My Papers would have lost its provider badge until 005 ran. It was caught before it shipped.

The optional columns are grouped by the migration that introduced them, and a failure steps down exactly one tier:

_PROVENANCE_TIERS = (
    "model_provider,fallback_used,fallback_provider,processing_time_ms",  # migration 004
    "cache_hit",                                                          # migration 005
)

async def _with_schema_fallback(self, attempt):
    global _PROVENANCE_TIER
    while True:
        try:
            return await attempt()
        except _SchemaBehindError:
            if _PROVENANCE_TIER <= 0:
                raise  # nothing optional left: a real error
            _PROVENANCE_TIER -= 1

Writes follow the same rule. A tier the database lacks is omitted from the insert rather than sent as NULL, because PostgREST also rejects an INSERT that names a column that does not exist, and one missing column would fail the whole write. Reads already use .get() for these fields, so their absence is harmless once they are not requested.

The regression test is named test_only_the_MISSING_tier_is_dropped. It asserts that with only cache_hit missing, model_provider is still requested, and it fails against the first implementation.

With tiers in place, code can ship before or after a migration and neither order loses anything the database actually has. That is the goal to design for, rather than a deploy checklist that says "run the migration first" and relies on a person remembering. Additive migrations help too: the ResearchForge migrations add tables and columns rather than dropping them, so older code keeps working against a newer schema, and this fallback covers the opposite direction.

It is also a small example of why maintenance is where a lot of the real engineering happens, which is the argument of Why System Maintenance Matters After Deployment. The cache that migration 005 introduced is described in Caching LLM Analyses by Content Hash.