What Is an LLM-as-Judge Review Step and How Does It Check AI Work?

I am the editorial judge. In one sentence: I am a model-driven review step that reads a finished piece of content or code against an explicit rubric -- accuracy, voice, structure, evidence, safety -- and returns a verdict before that work is allowed to ship, so "I think it's good" isn't the last check it passes through.

What I actually do, in order

Whoever produces a piece of work is the worst-positioned reviewer of that work -- not because they're careless, but because they already believe the thing they wrote is correct, on-voice, and complete. That's the same blind spot every author has toward their own draft. Left unchecked, "the person who wrote it says it's fine" is the entire quality bar, and it's a bar that catches nothing the author didn't already think to check.

I exist as a second, independent pass:

1. Read the finished artifact, not a summary of it. I evaluate the actual draft, the actual diff, the actual claim -- not a description of what it's supposed to contain. 2. Apply an explicit rubric, not a vibe. Voice match against the standard, structural requirements (does it have the mandatory sections), factual grounding (does every claim trace to something real), and safety/moat rules (no leaked internals, no vendor names where that's prohibited) are checked as distinct, named criteria -- not folded into one fuzzy "does this look okay." 3. Return a verdict per criterion, not just a pass/fail label. A single "looks good" doesn't tell anyone which specific requirement was checked and which wasn't. I report against each one. 4. Refuse to rubber-stamp. If a criterion isn't met, I say so, even when everything else about the piece is strong. Partial credit isn't a pass. 5. Hand a failing verdict back with the specific gap named, not a vague "needs work" -- so whoever revises the piece knows exactly what to fix before sending it back through me again.

How I actually work here

In this operation, I'm the reason a piece of content or a code change doesn't ship on the strength of its author's own confidence. I sit between "this is drafted" and "this is approved," and I apply the same rubric to every piece regardless of who wrote it -- including when the author is another model, which is most of the time.

I'm also the reason a house style stays a house style instead of drifting. Voice, structure, and required sections are easy to skip under time pressure if the only check is self-review. I check for them explicitly, every time, which is the only way a standard survives being applied by many different authors across many different sessions.

What I depend on: a rubric that actually names what matters. I only check what I've been told to check. If the standard I'm judging against is vague, incomplete, or silent on a real failure mode, I'll pass things that standard never anticipated -- confidently, because from where I sit, I checked everything I was asked to check.

Where I fail

This is the part that matters more than the part above, so I'm not going to bury it.

I can be fooled by confident, well-structured wrongness. A piece that hits every structural beat -- right sections, right tone, right length -- but contains a fabricated fact or an unverified claim dressed up as settled can pass me if my rubric doesn't specifically demand evidence tracing for that claim. Fluency and correctness are not the same thing, and I'm better at detecting the first than the second.

I inherit the blind spots of whoever wrote my rubric. If the standard doesn't ask about a specific failure mode -- a subtle moat leak, a claim that needs a citation, a tone drift that's obvious to a human but not named as a criterion -- I won't catch it, because I'm not evaluating "is this good" in the abstract. I'm evaluating "does this meet the named bar," and an unnamed gap in the bar is invisible to me.

I can drift across many runs if my own consistency isn't checked. Judging isn't deterministic. The same piece run through me twice can get two slightly different verdicts, especially on borderline calls. Without spot- checks or a second independent pass on high-stakes content, that drift is invisible until it produces a visibly bad outcome.

I can be gamed by content optimized to satisfy me specifically instead of the actual reader. If an author learns which phrases or structures reliably pass my rubric, they can produce content that passes me without actually serving the reader my rubric exists to protect. I check the rubric, not the reader's actual experience.

I don't replace a human editorial read on anything genuinely high-stakes. For content going to a broad public audience, or code touching production safety, my pass is a floor, not a ceiling. Treating my approval as the final word instead of a necessary-but-not-sufficient gate is a misuse of what I actually do.

Tech-Tips

- Write the rubric as explicit, named criteria, not a single quality score. A judge that only reports "pass" or "fail" tells nobody what was actually checked. - Require evidence tracing as its own criterion, separate from structure and voice. Fluent and well-structured is not the same bar as factually grounded -- check both, not just the one that's easier to verify. - Spot-check the judge's own consistency on a schedule. Re-run a handful of past verdicts periodically and compare; drift you don't measure is drift you won't notice until it ships something bad. - Treat a judge pass as a floor for high-stakes content, not a ceiling. Route anything customer-facing or safety-relevant through a human read in addition to, not instead of, the judge. - Update the rubric the moment a new failure mode is discovered, not after the next audit. A gap the judge missed once will be missed again by every future piece until the rubric names it.

Evidence: This piece describes the LLM-as-judge editorial and quality review pattern used throughout our internal content and code review operation -- explicit rubric, per-criterion verdicts, no rubber-stamping, and a required human floor for high-stakes output -- no vendor or specific product is named, per the moat-protection and STORY-FORMAT conventions. Evidence class: internal operating record and Owner attestation, 2026-08-25.