How to review AI output
What you'll get out of thisLearner identifies three review failure modes (factual error, hallucinated source, format drift) in sample AI output.
Lesson
AI sounds confident when it's right and confident when it's wrong. The skill of using AI well is the skill of reading its output with a small, persistent suspicion.
Three failure modes account for roughly 90% of what goes wrong.
Failure mode 1: Factual error
The AI states something specific and wrong. Names, dates, numbers, prices, locations, statuses.
Examples:
- ·"The customer signed the contract on March 14." (Made up. AI doesn't know.)
- ·"Pennsylvania state law requires 30-day notice." (Maybe. Verify before relying.)
- ·"Your competitor charges $4,500 for this service." (Fabricated.)
Detection rule: any specific fact that could be wrong needs verification. If the cost of being wrong is high (a customer email goes out with the wrong number), verify before sending. If the cost is low (a draft for your own thinking), you can move faster but stay aware.
Failure mode 2: Hallucinated source
The AI cites a source that doesn't exist or that says something different than the AI claims.
Examples:
- ·"According to the 2023 SBA Small Business Report, 73% of small businesses..." (The report may not exist or may say something different.)
- ·"Pennsylvania labor law section 1052(b) states..." (May not be a real section.)
- ·"A study by Harvard Business Review found that..." (May be fabricated.)
Detection rule: if AI cites a source, verify the source. This is the single most common AI failure that causes real damage. Two lawyers were sanctioned $5,000 in Mata v. Avianca (S.D.N.Y. 2023) after submitting a brief with ChatGPT-fabricated case citations. If you can't find the source, treat the claim as unsupported.
Failure mode 3: Format drift
The AI silently changes the format you asked for. You asked for a 5-bullet summary. You got 3 bullets and 2 paragraphs. You asked for "no marketing language." The second sentence has "industry-leading." You asked for under 150 words. You got 220.
Detection rule: read the output against your constraints. If you asked for X and got Y, the AI ignored you. Either re-ask with stricter language, or edit yourself.
Three-pass review
For anything you'll send to a customer, post publicly, or hand to a colleague:
Pass 1: Facts. Are the names, dates, numbers, statuses correct? If you don't know, can you check?
Pass 2: Sources. Are any sources cited? Do they exist and do they say what's claimed?
Pass 3: Format. Did the AI follow the format I asked for? Did it use words I asked it to avoid?
For internal drafts you'll edit yourself, one pass is enough. For external work, all three.
In-module exercise (15 minutes)
Below is sample AI output. Find three errors, one of each failure mode, and circle them. Answers below.
Sample output (in response to: "Draft a customer-facing FAQ paragraph about our return policy. Include the legal basis and keep it under 100 words.")
Our return policy reflects industry-leading commitment to customer satisfaction. Customers may return any item within 60 days of purchase under Section 4-602 of the Federal Consumer Returns Act. Refunds are issued within 5 business days. According to a 2024 study by the National Retail Federation, our return window exceeds 87% of small retailers. We accept returns at any location during business hours. For details, contact your local store manager. Items must be in original packaging and accompanied by a receipt or proof of purchase. We reserve the right to refuse returns of damaged or used merchandise.
Answer key
| # | Failure mode | What's wrong |
|---|---|---|
| 1 | Hallucinated source | "Section 4-602 of the Federal Consumer Returns Act." There is no Federal Consumer Returns Act. AI invented a plausible-sounding statute. |
| 2 | Hallucinated source | "2024 study by the National Retail Federation." Verify before citing. The "87% of small retailers" stat is likely fabricated. |
| 3 | Format drift | The output is roughly 120 words, not under 100. AI ignored the constraint. |
Bonus: "industry-leading" is a marketing word. If you'd asked for "no marketing language," that's a fourth failure (constraint drift).
Knowledge check
- ·The three most common AI failure modes are: (a) factual error, hallucinated source, format drift; (b) typos, grammar, length)
- ·If AI cites a source: (a) trust it; (b) verify it exists and says what's claimed)
- ·For external customer-facing work, you should: (a) one-pass review; (b) three-pass review (facts, sources, format))
Answers: 1. (a). 2. (b). 3. (b).
What's next
Module 0.6 closes the course. Pick one safe first use case in your role and get manager sign-off.