You've built a loop: it writes, scores its own work, and rewrites the weak attempts. But there's a trap hiding in the middle. If the judge — the thing doing the scoring — is soft, the whole loop is theatre. It dutifully rewrites and re-scores while nothing actually improves, because every draft comes back an 8 out of 10.
A rubber-stamp judge is the single most common reason loops disappoint. Here's how to build one with teeth.
1. Make every check yes/no about one draft
"Is it engaging?" is not a check — it's a vibe, and the model will rate everything 7. A real check is something you could answer yes or no about a single specific draft:
- ❌ Is the writing good?
- ✅ Does the first sentence make a claim the reader will want to argue with?
- ✅ Is there exactly one core idea, not three half-ideas?
- ✅ Could a knowledgeable reader still learn something they didn't know?
Specific, observable checks force the judge to actually look. Vague ones let it wave a hand.
2. Use a different model to judge than to write
A model grading its own output flatters itself — it already "decided" the draft was good when it wrote it. Hand the draft to a separate model (or at least a fresh, judge-only prompt that never saw the writing happen) and the scores get honest fast. This site does exactly that: one model writes, a different one scores, and they don't share a conversation.
3. Tell it to hunt for what's wrong
Ask "rate this draft" and you get a polite number. Ask "find the weakest thing about this draft and say why it falls short of the bar" and you get a real critique you can feed back into the rewrite. Frame the judge as adversarial — its job is to withhold the high score until the draft earns it, not to award participation points.
4. Reserve the top of the scale
Add the instruction explicitly: "Most drafts land 5–8. Reserve 9–10 for genuinely excellent work; be willing to give low scores." Without it, models compress everything into 7–9 and your loop can't tell a good draft from a great one — the score stops carrying information.
Where it breaks: the judge gets gamed
Here's the subtle failure. Once the loop optimizes against your checks, it learns to satisfy the letter of them, not the spirit. Tell it "the first line must be a hook" and you'll start getting clickbait first lines that hook and then disappoint. The metric becomes the target, and — as the old law goes — a measure that becomes a target stops being a good measure.
The defenses: keep a check or two about the whole piece, not just its parts ("does the payoff match the promise of the opening?"); rotate or refresh your checks occasionally; and read a sample of the winners yourself now and then. A loop that judges its own work still needs a human spot-check on the judge — that's the part that doesn't automate, and it's where you stay in the loop.
A good judge is harder to write than a good prompt. It's also where most of the quality comes from. Spend your time here.