The scenario below is an illustration, not a client account.

A copywriter opens her AI’s draft on a Monday and feels it is off. Not wrong, off. She feels it again on Friday, and the week after, and can never say by how much. The unease has no units, so there is nothing to compare and nothing to act on.

How do I measure the quality of my AI’s output?

Score the output against a standard you have written down, on a regular cadence, using the same rater and the same standard each time. You do not need a perfect metric. You need a repeatable one, so that “this week feels worse” becomes a claim you can check.

“But my work is judgment work. You can’t measure that.”

I have heard this often and the reason is fair: most people have only ever experienced AI output without a standard, a review, or a number. Scoring it sounds like pretending.

I spent 36 years building mission-critical bank systems, where the job was putting numbers on things people insisted could only be felt: exposure, risk, the probability that a figure was wrong. Those measurements were never exact. They were consistent, which was enough to see when something moved. (The Real Enemy Is Drift)

The method: a standard, a score, a cadence

To measure AI output quality, write down what good looks like for one task, score each output against it from 1 to 5, and record the score weekly. The absolute number carries little meaning on its own. What it gives you is a baseline and a series, so a change is visible instead of suspected.

Three parts, none of them hard.

  1. A standard. Two sentences describing what good output looks like for one task. “A good client email is warm, under 120 words, no bullet lists, ends with one clear next step.” (AI That Learns From Your Corrections)
  2. A score. Rate each output against that standard, 1 to 5. Same rater, same standard, every week. Consistency is doing the work here, so changing the standard mid-series resets the series.
  3. A cadence. Weekly, one task. Now you have a record instead of an impression.

Consistent beats exact

People reject measurement because they picture it having to be exact. A bank does not measure risk to four decimal places of truth. It measures consistently, so that a movement means something. A rough 1–5 you actually record beats a sophisticated metric you never build. (The Month-Six Test)

What measurement gives you, and what it does not

What it gives you:

  • A baseline. You know where the output sits today against a written standard, which is the thing you could not say before.
  • Visibility of change. When the score moves, you can see it in the week it moves and go looking for the cause.
  • Something to attribute against. When you change a rule or a prompt, you can ask whether the score responded.

What it does not give you: any promise about direction. Measuring is an instrument, and instruments do not improve the thing they point at. Your series may rise, sit flat, or fall. A flat series across six weeks is a genuine finding, and the one that tells you most, because it says the changes you made never reached the output. (Stateful vs. Stateless AI)

Treat any claim that measurement itself lifts quality with suspicion, including from me. The number reports. You act.

Try this now (5 minutes)

  1. Pick the single task you hand to AI most often.
  2. Write two sentences describing what good output looks like for it.
  3. Score this week’s output against those two sentences, 1 to 5.
  4. Put the number and the date somewhere you will see it next week.
  5. Next week, score again against the same two sentences.

Stop—this counts. You now have a baseline, which is the first thing you have ever been able to compare against.

Frequently asked questions

Isn’t a 1–5 score too crude to be meaningful? Crude and consistent beats precise and absent. What you are looking for is whether this week reads differently from last week against the same written standard, and a consistent 1–5 shows that. Precision can come later if you ever need it.

Who does the scoring, me or the AI? Start with you, because you own the standard and you are the one who can tell whether a draft met it. A workflow can later run mechanical parts of the check for you, such as length or a banned phrase. The standard stays yours to set and edit.

How is this different from just “reviewing” the output like I already do? Reviewing is one-off and unrecorded, so it leaves nothing to compare against. Scoring against a written standard is recorded, so you can look at six weeks side by side. The difference is whether there is anything to look back at.

Will my score go up if I do this? That is not what measuring does. Measuring tells you what is happening, including the possibility that nothing is. A flat score over six weeks is a real result and a useful one, because it says the changes you made did not reach the output. Any method that promises the number rises is selling you something other than measurement.