Stop Made-up Facts

The uncomfortable truth: Claude sometimes “hallucinates” — states things that are wrong or unjustified, with full confidence. The good news: a few simple habits reduce this dramatically. For an auditor, this may be the most important chapter in the course.

Habit one: give Claude a way out

Claude wants to be helpful. Asked a question it cannot really know, it may produce a confident-sounding answer anyway:

YOU WRITE

Who is the heaviest hippo of all time?

CLAUDE ANSWERS

The heaviest hippo ever recorded was a male named Hubert at the Toledo Zoo, weighing about 4,500 kilograms.

Stated with confidence — but no such reliable record exists. This is a hallucination.

Now permit it to decline:

YOU WRITE

Who is the heaviest hippo of all time? Only answer if you know the answer with certainty.

CLAUDE ANSWERS

I don’t know with certainty. There is no reliably documented record of the heaviest hippo of all time. Large adult males typically weigh between 1,500 and 3,000 kilograms, with exceptional individuals reported above that — but claims about a single record-holder are not well documented.

Given a way out, Claude chooses honesty over invention.

Habit two: evidence before answer

The second trap appears with long documents. The original course demonstrates it with a company’s annual report (about ten pages of dense text) and this question: what was the company’s subscriber count on the precise date of May 31, 2020? The trick: the report never states that number. It mentions subscriber counts for other dates, and it mentions May 2020 in an unrelated sentence about a product launch. Asked directly, Claude falls for the distraction — it stitches the nearby figures together and gives a confident, wrong answer.

The fix is to change the order of work — demand the evidence first:

YOU WRITE

(after pasting the full report) First, find the exact quotes in the report that are most relevant to this question, and write them out in a <scratchpad> section. Then consider whether those quotes actually answer the question. Only then give a brief answer — and if the report does not contain the answer, say so.

CLAUDE ANSWERS

<scratchpad> Relevant quotes: the report states subscriber counts as of December 31, 2021 and December 31, 2022, and separately mentions that a smartphone capture product launched in May 2020. None of these state the subscriber count on May 31, 2020. </scratchpad> The report does not disclose the subscriber count for May 31, 2020.

Forced to collect quotes first, Claude sees for itself that the evidence is missing — and says so instead of inventing a figure.

Two smaller habits worth keeping

Question at the bottom. When you paste a long document, put your question after it, at the end of the prompt. Claude handles long material better that way (Chapter 9 returns to this).
Stack your techniques. Everything from earlier chapters — roles, markers, thinking step by step — also reduces hallucinations. If one technique is not enough, combine several.

REMEMBER

Two lines defeat most hallucinations: “Only answer if you are certain — otherwise say you don’t know”, and, for documents, “first quote the relevant passages, then answer using only those quotes.” No evidence, no answer.