Engineering Practices That Survive the Stack

If the only place your checks run is the contributor's machine, you have documentation, not verification.

✓ Do this

A pull-request workflow running typecheck, lint, format check, unit tests and build as separate named jobs on a hosted runner.

✗ Not this

Keeping every check in a local hook and calling it verification — no pipeline at all, so one bypass flag skips the whole project.

A pre-commit hook is skipped by a single flag, by a machine missing part of the toolchain, or by any push that did not originate from a normal working tree. More importantly, a purely local check produces no artifact — there is no status a branch protection rule can require, so nothing can ever prove a branch was green before it merged. Run the same commands on a runner and the hook becomes a fast local preview of a gate that exists somewhere else. Four details in the workflow are worth copying verbatim: read-only permissions at the workflow level, a concurrency group keyed on the pull request that cancels in progress so a new push kills the stale run, third-party actions pinned to a commit SHA with the version in a trailing comment, and a frozen-lockfile install.

A hook slow enough to be annoying will be bypassed, so put only formatting-of-staged-files in it and move the expensive work to CI.

✓ Do this

One line in the hook — format the staged files and re-stage them. About a second, regardless of how large the repo gets.

✗ Not this

Running the full suite, plus a second language's dependency sync, on every commit — until people quietly start bypassing the hook.

This is the pairing that makes the previous rule safe. Typechecking the whole project, running every test, and syncing a second language's dependencies on every commit costs minutes per commit regardless of what changed. That is exactly the cost profile that trains people to reach for the bypass flag — and a bypassed hook with no CI behind it means zero verification. Shrink the hook until nobody wants to skip it; let the runner absorb the minutes. The mechanism is small: a hook manager installs the hook, and a staged-files runner narrows it to the paths in the index and re-stages whatever it rewrote.

An auto-fixing linter is a convenience command, never a verification command.

✓ Do this

Two scripts: a checking one wired into the hook and CI, and a fixing one that only a human ever invokes.

✗ Not this

Wiring the auto-fix variant into a hook or a runner, so violations get repaired, discarded, and never reported.

Two distinct failures come from wiring --fix into a gate. In a pre-commit hook, the fixes land on disk but are not re-staged, so the snapshot that gets committed is not the snapshot that passed. On a runner, auto-fix is worse still: it repairs the violation, reports success, and throws the repair away — the gate now hides the exact class of problem it exists to surface. Keep the fixing variant as a manual convenience and point every automated path at the read-only one.

Building CI and not marking the checks required leaves you with a dashboard, not a gate.

✓ Do this

Branch protection naming each job explicitly, so a red or missing run blocks the merge button rather than merely annotating it.

✗ Not this

Shipping the workflows and stopping there, leaving runs that look authoritative, are required by nothing, and merge red.

The failure is quiet, because everything looks correct: workflows exist, runs appear on the pull request, badges are green most of the time. But turning on branch protection is usually a repository setting rather than a file in the repo, so it is the one step that cannot be done by the same change that adds the workflow — which makes it exactly the step that gets forgotten. Write the exact list of context names into the issue, because the setting has to be applied by a human in a web form.

If a module must never reach the browser, make importing it from client code a build failure rather than a rule in a contributing guide.

✓ Do this

A server-only marker as the first line of every module that reads a credential or opens a database connection, declared as a real dependency.

✗ Not this

Leaving the boundary as a sentence in the contributing guide, where nothing objects until a key is already in a shipped bundle.

Most stacks give you some marker — a server-only import, a module boundary, a lint rule with a path restriction — that converts an accidental client import into a compile error. This is the highest-leverage single change in most codebases, because it takes an invariant that was being upheld by discipline across hundreds of files and delegates it to a tool that never gets tired. It also composes with policies you may not want to change: secrets can stay wherever your project keeps them, as long as the compiler guarantees they cannot be bundled for a browser. Declare the package explicitly even if your framework aliases it for you — Next resolves a bare server-only import to its own vendored copy, which works right up until something other than Next resolves it.

Committed configuration for a tool that is not in your dependencies produces the appearance of governance and none of the effect.

✓ Do this

Each config file maps to a declared dependency and a CI job — plus a one-off mechanical reformat landed as its own commit and recorded in the blame-ignore file.

✗ Not this

Committing a formatter config without installing the formatter, so the file reads as evidence the concern is handled while the codebase drifts.

Nobody audits a setting that already exists. The file gets read as evidence that the concern is handled, so drift accumulates unobserved and the eventual cleanup is enormous. Either install the tool and run it in CI, or delete the config so the gap is visible. When you do finally run it, do the mechanical reformat as its own commit and add that commit to your blame-ignore file, so a decade of authorship history is not attributed to the reformat. Two formatter plugins earn their place while you are in there: deterministic import ordering removes a whole category of merge conflict, and canonical class ordering does the same for utility-class markup.

A guide that describes an intent the codebase never adopted does not read as out-of-date; it reads as a target to build toward.

✓ Do this

Contributor and agent guides naming the data layer, test runner and directory layout actually present, with abandoned directions marked as abandoned.

✗ Not this

Leaving a guide that mandates a library which is not a dependency, so new work — including agent work — aims at an architecture you already rejected.

Missing documentation makes people go look at the code. Wrong documentation makes them confidently do the wrong thing — and it aims future work, including work done by agents reading that file as authoritative, in a direction the codebase deliberately walked away from. When a stated intent and the implementation diverge, the fix is to amend the document to describe what is actually there, and to note the decision if there was one. It also helps to split the audiences: human process — branch naming, commit titles, when to open an issue — in one file, agent guardrails in another, and a one-line pull-request template that costs nothing.

Runtime version, type definitions, and the declared engine range must be one fact expressed once and referenced everywhere else.

✓ Do this

A version file, an engines range and runtime type definitions all naming the same major, with dependency automation told not to bump past it.

✗ Not this

Restating the runtime in three places — version file, type definitions, engines field — and pinning none of them.

When these drift, the type checker describes a runtime nobody uses, and nothing prevents a contributor or a future runner from picking a different major. The symptom is the worst kind: code that typechecks locally and fails in production, or vice versa, with no error message pointing at the version mismatch. Pin the engine range, keep the version file in sync, and have CI read the version from that file rather than restating it.

Pipeline definitions should reference the version files the project already keeps, so upgrading a runtime is one edit rather than a search.

✓ Do this

A single shared setup action doing checkout, runtime, package manager and install — runtime read from the version file, package manager from the manifest.

✗ Not this

Pasting version literals into pipeline YAML and copy-pasting the install steps into each job, where they drift job by job.

The same applies to shared setup steps. Install, cache, and toolchain configuration copy-pasted into each job drifts job by job, and the drift is invisible until one job starts failing for a reason none of the others do. Factor it into a single composite step and call it from each job. Caching belongs in there too, with one caveat worth writing down: a cached browser binary restores without the system libraries it needs, so the cache-hit path still has to run the dependency install step.

When the same categorization appears in two layers — a query and its labels, a schema and its validator — generate one from the other so they cannot disagree.

✓ Do this

The query layer building its bucket expression from the same exported constant the UI renders labels from, so the two are provably identical.

✗ Not this

Hand-maintaining the same boundaries in application code and again in SQL, then finding out via a chart that is subtly wrong rather than broken.

This is the same principle as version pinning, applied inside the application. Bucket boundaries written once in application code and again by hand in SQL will diverge at the worst possible moment, and the divergence shows up as a subtly wrong chart rather than an error. Building the SQL fragment from the shared constant makes the two provably identical.

Pinning versions creates a maintenance debt, so the bot that pays it down has to understand which pins are deliberate.

✓ Do this

Grouped weekly updates for minors and patches, majors split into their own reviewable pull requests, and an explicit hold on anything pinned to match the runtime.

✗ Not this

No automation at all, so SHA-pinned actions and a deliberately-held type package quietly become the oldest thing in the repo.

SHA-pinned actions are a real supply-chain control and a real liability: nothing floats them forward, so without automation they rot in place. Grouping minor and patch bumps into one weekly pull request keeps the noise low enough to actually get reviewed, while splitting majors means a breaking bump arrives with its own diff and its own discussion. The one configuration that matters most is the exclusion list — a type package held at the runtime major, or a compiler held where the rest of the toolchain supports it, must be marked as intentional, or the bot will helpfully undo the decision every week.

If a check reads generated files, generate them in the same command — otherwise the result depends on what a previous build happened to leave behind.

✓ Do this

A typecheck script that regenerates route and schema types first, so a clean checkout and a warm working tree give the same answer.

✗ Not this

Typechecking against whatever generated types the last build happened to leave on disk.

Typecheckers that consume generated route types, schema types, or client stubs will silently consume a stale copy. The result is a check that passes on a developer's machine and fails on a clean checkout, or the reverse, with no signal that the inputs differed. Usually this is a one-word fix, and it is worth making before wiring anything into CI, so the pipeline is not debugging your build cache.

A check is not trustworthy until you have watched it fail on the exact condition it was written to detect.

✓ Do this

Each hand-written validator shipping with a fixture that makes it fail, run in CI beside it — so green means it looked and found nothing.

✗ Not this

Shipping a bespoke checker without ever watching it fail, so it can enumerate the wrong set of files and report success forever.

Verification code is code, and it has bugs — but its bugs are unusually well camouflaged, because the failure mode is a green result. Two patterns account for most of them: enumerating the wrong universe of inputs, and asserting a condition looser than the one that matters. Both produce a check that runs, reports success, and inspects nothing.

Smoke-test your real entry points, because module resolution differs between the environments your gates run in and the ones your code runs in.

✓ Do this

A CI script that imports the module graph of every CLI, daemon and scheduled job the way that process actually starts it.

✗ Not this

Treating a green build and a green test suite as proof the program starts.

Build systems, test runners, and standalone scripts resolve modules under different conditions, and a guard that is inert in one can throw in another. That means a production build, a full test suite, and a local run of the whole check suite can all be green while an entire class of entry point is dead on import. The cheap insurance is a script that imports each real entry point's module graph and runs it in CI — and it has to run them the way production does, not through the test runner that stubs the very module in question.

Any optimization that skips work should treat every uncertainty — an unrecognized path, a failed query, an unexpected event type — as a reason to run everything.

✓ Do this

A gate that reports source-changed on an unknown path, a failed diff or a non-pull-request event, and skips only on an explicit docs-only allowlist.

✗ Not this

Skip logic that defaults to skipping, where a failed diff quietly produces a run that did nothing and looks exactly like one that passed.

Skipping CI when only documentation changed is a real saving, especially in repos with large research or notes directories. But the logic that decides is itself untested code running outside the gate, so its default has to be the expensive-and-correct branch. A gate that can skip by accident is not a gate.

Use a job-level condition to skip work, because a workflow filtered out entirely never reports its status and leaves required checks pending forever.

✓ Do this

Every job starting and then deciding immediately to do nothing, so the required context still reports a result.

✗ Not this

Filtering the workflow itself by path, which leaves the required check pending forever and the pull request permanently unmergeable.

This is a specific trap with a general shape: the mechanism that prevents a workflow from running also prevents it from reporting that it did not need to run. The required check is then permanently pending and the pull request can never merge — a failure that looks like a platform outage rather than a configuration mistake. Let the job start, then have it decide immediately to do nothing.

Create-if-not-exists describes the current shape and can never express a change to an existing one.

✓ Do this

An ordered migrations directory, an applied-migrations table, today's schema captured as migration 0001, and a CI job applying all of it to an empty database.

✗ Not this

Building the schema from idempotent DDL at startup, then applying the first column addition by hand and out of band.

It is an appealing starting point: simple, safe to re-run, no ordering to maintain. The wall arrives the first time you need to add a column, widen a type, or add a constraint, because none of those are expressible. Those changes then get applied by hand, out of band — and at that moment the definition committed to your repository stops describing production, with nothing to detect the gap and nothing to roll back.

If more than one language issues DDL against the same database, the migration mechanism has to be legible to all of them — which usually means plain SQL files and a shared applied-migrations table.

✓ Do this

Plain SQL files plus a small runner per runtime, all reading and writing one shared applied-migrations table.

✗ Not this

Pointing a model-diffing generator at a database another runtime also writes to, where every table it cannot see reads as one to drop.

Multi-language schema ownership is common and easy to miss, because each side looks self-contained from the inside. Any tool that treats one language's model as the single source of truth will not merely be unhelpful; it will actively propose destroying the tables it does not know about. A directory of numbered SQL files with a small runner is language-neutral by construction, and both runtimes read the same history.

A pile of add-column-if-not-exists statements re-executed at startup is already a migration history — just unordered, unversioned, and re-run forever.

✓ Do this

The existing conditional ALTER block split into ordered files, in the order it was written — a mechanical move, not a redesign.

✗ Not this

Scoping the conversion as a schema redesign, when the block of conditional ALTERs running at every startup already is the history.

Finding it is good news, because it means the conversion is mechanical rather than a redesign: the statements exist and are known correct, they simply need to be split into ordered files. It is also the clearest possible evidence that the idempotent approach has already hit its ceiling, since those statements only exist because create-if-not-exists could not express the change.

Introducing a new mechanism without removing the one it replaces leaves two writers and no way to know which one produced the current state.

✓ Do this

The change that adds the migration runner also deleting the old setup entry point and its script, leaving exactly one writer.

✗ Not this

Landing the new runner and leaving the old path in place for now, since it still works — which is how the drift gets created.

The transitional period where both exist is the one where drift is created, and it tends to last much longer than planned because the old path still works. Retiring it in the same change is what makes the new mechanism the single source of truth rather than an additional opinion.

If daemons or scheduled jobs can restart concurrently with a deploy, the migration runner needs an advisory lock rather than a convention about who runs it.

✓ Do this

The runner taking a database-level advisory lock before its first statement, so a concurrent restart waits instead of interleaving.

✗ Not this

Relying on a rule about who may apply migrations — enforced by nobody, at three in the morning, mid-deploy.

Convention works right up until an unattended restart lands during a deploy, and the resulting half-applied schema is discovered by whatever queries next. The lock is a few lines. The alternative is a rule that migrations may only be applied from one place, enforced by nobody at three in the morning.

When importing a practice from another project, name the invariant it provides, then choose your own mechanism for it.

✓ Do this

Each adopted item written as the invariant — versioned migrations, tests that assert access-control rules — with the tool picked in a separate step.

✗ Not this

Rejecting a sound practice wholesale because the hosted product that implemented it elsewhere does not fit this stack.

Versioned migrations is the technique; a particular hosted database's migration folder is one implementation. Tests that assert schema and access-control invariants is the technique; a particular in-database test framework is one implementation. Stating the invariant first is what lets you keep the value when the vendor does not transfer — and it makes the eventual write-up useful to the next project too.

Ask what this tool does when it is wrong, and whether you would notice — not just what it does when it is right.

✓ Do this

An evaluation that records each candidate's destructive default and whether it is detectable, alongside the capability comparison.

✗ Not this

Choosing on the feature table, and meeting the destructive default for the first time in production.

Feature comparisons are symmetric and cheap to produce; failure-mode comparisons are neither, which is why they get skipped. A generator that computes changes by diffing its own model against reality has a specific and severe failure mode: anything outside its model reads as something to remove. That may be perfectly acceptable in a project where its model is complete, and disqualifying in one where it is not — a distinction no feature table will show you.

Tools sell as a bundle; score each part separately against what your codebase will really do, and be honest about the parts you will bypass.

✓ Do this

A per-benefit table for each integrated tool, marking which of its linked features this codebase would use and which it would route around.

✗ Not this

Paying the full migration, learning and failure-mode cost of five linked benefits in order to get one of them.

An integrated tool typically offers four or five linked benefits. If your data layer is hand-written and good, and you have no intention of rewriting it, you may be buying one of those benefits while paying the full migration, learning, and failure-mode cost of all of them. That is not an argument against integrated tools — it is an argument for doing the count before, rather than discovering it during.

An audit's rejected items are worth as much as its adopted ones, because otherwise the same suggestion returns every six months.

✓ Do this

Declined recommendations living in the same document as the adopted ones, each with its reasoning and the context in which it would be right.

✗ Not this

Recording only what you adopted, so the same already-considered proposal returns every six months with its reasoning gone.

Write the rejection next to the recommendation, in the same document, with the reasoning that produced it. This is also the honest place to note that a practice was excellent in its original context and simply does not apply here — which is a much more useful record than silence, and it survives the departure of everyone who was in the room.

A required check that goes red because of network flakiness or shared infrastructure teaches the team to ignore red, which costs you every other check too.

✓ Do this

The infra-dependent suite running nightly and on manual dispatch with its report kept as an artifact; only jobs that are hermetic on a fresh runner are required.

✗ Not this

Making a suite that needs live shared infrastructure a required check, and teaching the team that red means try again.

This is the most expensive mistake in the list, because the damage is not confined to the flaky job. Suites that need live external services, real data volumes, or long timeouts belong on a schedule and on manual trigger, where a failure is investigated rather than retried. Promote them to required status only once they can run against disposable infrastructure that the pipeline itself controls. Keep the HTML report as an artifact with a not-cancelled condition, so the overnight failure is still debuggable in the morning.

When your documentation assigns each developer or worktree a distinct port, number, or directory by hand, that is an unautomated lock.

✓ Do this

Sessions claiming a port from a lock directory at startup and releasing it on exit, so no document has to assign numbers by hand.

✗ Not this

Assigning ports to worktrees in a table in the guide, then adding a standing rule never to kill a running server.

Conventions like this are a reliable signal that a structural fix is missing, and they fail exactly when the project is busiest — the moment two sessions run at once. A lock file, a port registry, or a workspace that allocates its own resources removes the class of problem instead of documenting it. The full hermetic-environment version is a large project, but the coordination primitive underneath it usually is not.

Check required configuration at startup and exit with the exact list of what is missing, plus where to get it.

✓ Do this

One assertion helper called at the top of each script, daemon and scheduled unit, naming the missing variables and the doc that explains them.

✗ Not this

Letting a missing variable surface as a stack trace deep in a library, naming everything except the setting that is absent.

Systems with many entry points — scripts, daemons, scheduled units — fail deep inside a call stack when a variable is absent, and the stack trace names a library rather than the missing setting. A shared assertion helper called at the top of each entry point converts that into a one-line failure that says what to do next. It is cheap, and it matters most for the unattended processes nobody is watching.

Dated, append-only notes are true as of their date; you also need one living document per subsystem that is true now.

✓ Do this

Dated notes staying append-only, plus a small set of current-state documents per subsystem, validated by a link-and-anchor checker in CI.

✗ Not this

Relying on the dated archive alone, so answering how something works today means replaying it in order and guessing what was superseded.

Both forms are valuable and they are not substitutes. The journal preserves why a decision was made and what was known at the time; the living document answers what the pipeline does today, without requiring a reader to replay the archive in order and guess which parts were superseded. Once the living documents exist, a link-and-anchor checker is a small script that keeps them from rotting quietly — roughly a hundred lines that parses each markdown file and resolves every relative link and heading anchor.

Order remediation so the one-line fixes land before the ones that need a design decision, and attach a time estimate to each.

✓ Do this

The audit shipping as an ordered list: several-minute fixes first with minutes attached, half-day items next, decision-shaped items last and named as such.

✗ Not this

Delivering the audit as one undifferentiated list, which reads as a project and gets deferred as one.

An audit that arrives as an undifferentiated list gets deferred as a project. The same audit ordered by effort — with the several-minute items first, the half-day items next, and the ones requiring a decision last and named as such — gets started, and the early wins make the rest easier to justify. The estimate is also the honest way to surface that most of the value is usually in the first afternoon.