The test that refuses to lie

A technician opens the app mid-repair, asks for a diagnosis, and gets a confident, well-formatted, completely wrong answer. Nothing crashed. No error banner. The screen looked exactly like it does when the answer is right. The only thing that failed was the truth, and the interface had no way to say so.

That's the failure mode that should worry a team building an AI-assisted tool more than any outage: not the crash, which announces itself, but the plausible-looking wrong answer, which doesn't.

The gap most teams accept

The standard defense against that failure is code review: a human reads the diff, judges it reasonable, approves the merge. It's fast, it's familiar, and it catches a huge amount of real trouble. It also has a specific blind spot -- a reviewer can confirm that code looks correct without ever confirming that the product still gives a correct answer to the actual question a real user asks. A change can pass every review comment and still ship a diagnosis that quietly gets worse.

Most teams accept that gap, because closing it means building and maintaining something code review doesn't require: a way to actually run the real question through the real system and check the real answer, every time, before anything reaches a customer. The gap was worth treating as an opportunity here specifically because the product's core promise -- a technician can trust what it tells them mid-repair -- is exactly the promise code-review-only discipline cannot verify.

Teaching the concept: a promotion gate, not a review comment

A promotion gate is a hard rule enforced by the pipeline itself, not a suggestion enforced by a person's attention span: a change cannot be merged until an automated check has run something real through the system and confirmed the answer holds up. In this case, that means running an actual end-to-end customer journey -- ask a real diagnostic question, get a real AI response, check that the response is still a real analysis and not a degraded or missing one -- as a required status check. If that check hasn't passed, the merge button doesn't work. Not "shouldn't." Doesn't.

That's a meaningfully different guarantee than "someone reviewed this and it looked fine." It's "the system itself refused to let this ship until it proved, against a real customer journey, that it still works."

Why it matters

Without a gate like that, the failure a customer eventually hits isn't a crash -- it's a silent, gradual erosion of answer quality that nobody noticed because nothing ever forced a real check. That's a much harder problem to catch after the fact, because there's no error log pointing at it. It shows up as a technician who quietly stops trusting the tool, or worse, trusts a wrong answer at the worst possible moment. Silent failure in front of a paying customer costs more than a blocked merge ever will -- it just costs it later, and less visibly.

How the approach was built -- and the tradeoff accepted

The product's CI pipeline requires a passing real-analysis, end-to-end check before any non-docs change can reach the staging or production branch, with every customer-journey test explicitly inventoried so a missing or blocked test fails the build loudly instead of quietly falling out of coverage. The obvious, easier alternative was on the table and rejected: keep the gate as a manual "please run the E2E suite before you merge" reminder, which is faster to build and doesn't slow anyone down on a rushed day. That path was rejected because a gate a person can skip under deadline pressure is not actually a gate -- it's a hope. The tradeoff accepted instead: slower merges, a real maintenance cost for keeping the customer-journey inventory current, in exchange for a promotion path where "unverified" and "shipped" cannot both be true of the same change.

What to take to your own work

Ask, honestly, whether your own release process could let a plausible-looking wrong answer through on a busy day -- and if the honest answer is yes, that's not a process nitpick, it's a gap worth closing before a customer finds it for you.

Evidence: verified via AIFieldSupport-App and AIFieldSupport-API (private repos) CI workflow definitions and CRITICAL-PATHS.md customer-journey inventory -- shipped product, internal architecture record corroborated by repository history and live CI configuration, evidence dated 2026-08-24. Full record: docs/resume-hub/proof-pages/ai-field-support-multi-tenant-voice-first-platform.md and docs/resume-hub/claims-ledger.md (row C-23); public proof page pointer via danstolts.com/proof.