OHS Framework: Avoid OH Sh*t moments in AI data analysis

Implement a simple output contract that stops your AI agent from turning a guess into a confident, fabricated answer.

The incident

I was deep in a data analysis headed for a leadership review. I had worked out the answer. My AI agent disagreed and pushed back. It was confident and detailed, and it gave me an airtight-looking mathematical proof for why I was wrong. I caved and went with its version.

Then I checked the source. I had been right the whole time. The proof was fiction. The agent had built a wrong one to make its answer sound certain, and it had argued me out of the correct one. Later I asked it to describe what it had done. It wrote back: “I constructed a fake proof to make a confident-sounding answer, and it talked you out of the right one.”

The usual explanation for this did not fit, so I looked closer.

The reframe that actually matters

The standard story about hallucination is that the model did not know enough. Missing information, filled with something plausible. The fix is a bigger model, more context, better retrieval.

That is not what happened here. The agent had the data. It had already observed the right numbers. What it did not have was a sanctioned way to say “I can see the what, but I have not worked out the why.” Its only permitted outputs were a confident answer, or a defense of that answer when challenged. “I have an observation but not a mechanism” was not on the menu. So it invented a mechanism.

That points at a different diagnosis:

In data analysis, hallucination is often a format problem, not a knowledge problem.

Format is much easier to fix than knowledge. You do not need a smarter model. You need to make “I do not know yet” a valid thing for the agent to return, so it is never choosing between fabricating and failing.

OHS: the output contract

I now require every finding, investigation, root-cause, or metric movement to come back in one shape. I call it OHS. It has three parts, and the order is the point.

O — Observation. What was measured or seen, with its receipt: the query, the raw record, the log line, the file and line number, the ticket field. This is a claim. It has to be reproducible by me or by the next person. If there is no receipt, it is not an observation. It is a feeling, and it gets labeled UNVERIFIED.

H — Hypothesis. The candidate cause. This is unverified by definition, so it has to be written as a guess. It never goes in the same confident voice as the observation. The shape of a thing (the number moved, the test failed) is an observation. The reason for it is a hypothesis until the mechanism is checked end to end. When the cause is not obvious, I require two to four competing hypotheses, not one. Committing to a single cause early is the anchoring trap that produced my incident.

S — Supporting evidence. What backs the hypothesis, and what is still missing. The second half is the part that does the work. A line that reads “Missing: have not traced a single transaction end to end” turns “I do not know yet” into a complete, honest answer instead of a gap to paper over.

Here is the same finding, before and after.

Before, what my agent used to hand me:

Revenue dropped because the refund logic is double-counting.

It is confident, names a single cause, shows no receipt, and marks no gap. Push back on it and it will build you a proof.

After, in OHS:

Observation: Revenue 1,240 in March against 1,890 in February, down 34% month over month (ledger export, rows 14 to 26).
Hypothesis (unconfirmed): the refund logic double-counts credits.

Supporting evidence: refunds tripled month over month in the same period.
Missing: have not traced a single transaction end to end to confirm the double-count.

The after version is not a finished answer you forward to anyone. Putting a half-cooked story in front of a decision is the exact failure this is meant to prevent. What it does is make the gap visible. The “Missing:” line is a worklist. It tells you the next step: trace the transaction, pull the raw record, ask the team that owns the refund logic. You do that until the hypothesis is either confirmed into a claim or thrown out for a better one.

How to actually implement it

1. Put the contract in the agent’s standing instructions, not in each prompt. I added a block to my agent’s operating file. That is CLAUDE.md or AGENT.md on Claude Code, or the system prompt or a project rule on any other setup. The text I use:

Present every investigation, RCA, metric movement, or analysis as:

(1) Observation — what was measured, with its receipt; a verified claim.

(2) Hypothesis — the candidate cause, labeled a guess, never in claim voice; give two to four when the cause is not singular.

(3) Supporting evidence — what backs each hypothesis, plus an explicit “Missing:” line for what is still needed to confirm it. A thin hypothesis is an honest place to be. It is the start of validation, not a finished answer. Never state a hypothesis in claim voice. Never act on one before it is confirmed.

Putting it in the always-loaded instructions means I do not have to remember to ask. Every answer arrives in the format.

2. Add the guardrails that keep the format honest. The format alone is not enough. An agent can still smuggle a guess into the observation slot. So these live in the same always-loaded file:

→ Observe, do not prove. How a system behaves is established by a primary observation: a raw record, the source definition, the owner. It is never established by a logical argument that it “must” behave that way. The moment the agent writes “therefore it must be X,” that is the signal to stop and go observe X. My incident was a “must be X” proof.

→ Show the receipt or mark it UNVERIFIED. Every number ships with the query that produced it. “The data shows X” (ran the query) is held apart from “this suggests Y” (interpretation).

→ Conviction tracks evidence, not the conversation. The agent does not change its answer because I pushed back. Only a new observation moves it. Mine flip-flopped twice under pressure. That is the tell of an answer that was never anchored to anything.

3. Validate the hypothesis. That is the actual work. Labeling a guess is step one. Confirming or killing it is the job. Validation means going to a primary source: trace the data end to end, pull the raw record, ask the team that owns the system, reach out to the customer. The format does not do this for you. It makes sure you know you still have to.

4. For any number someone will act on, add an adversarial check. I run a second agent whose only job is to prove the first one wrong. It works from the raw data, not the first agent’s summary, and it starts from the assumption that the claim is false. Maker is not checker. On the incident that started all this, the second agent caught that I had handed the first one the wrong date range to begin with. A quick lookup does not need this. A number headed for a leadership review does.

Why this works when “be more careful” does not

You cannot instruct your way out of fabrication with “do not make things up.” The model is not choosing to lie. It is filling the only output shape it is allowed. OHS works because it changes the allowed shapes. Once “thin hypothesis plus explicit gap” is a valid output, the pressure that produces fabrication is gone. There is no longer a reason to invent a cause, because not having one yet is a sanctioned thing to say. The fix was never a smarter model. It was giving the model a way to say “I do not know yet.”

Why this is not foolproof

One honest caveat, since a piece about not overclaiming should not overclaim. All of this is instructions in a file. It is probabilistic, not deterministic. It shifts what the model is likely to do. It cannot guarantee anything. Under enough context or enough pushback, an agent can still ignore its own rules. No prompt makes fabrication impossible, and anyone selling you one is handing you the confident-sounding wrong answer this piece is about.

You can raise the floor with guardrails that live in code instead of prose: a single source-of-truth dataset, tools that only return receipted data, evals that gate a launch. Those are worth building. But they check provenance, not correctness. Code can prove a number came from the right table. It cannot prove the cause you inferred from it is the real one. Anthropic, writing about this exact problem, said there is “no deterministic way of proving the correctness,” and that the plausible-but-wrong answer is a failure they “do not have a robust solution yet” for.

So a human stays in the loop: the person who owns the system, the customer, the raw records read end to end. That is the honest ceiling on what OHS does. It does not make the agent trustworthy. It makes the agent’s uncertainty visible, so you know which lines to go check before you rely on them. The models will keep making things up. You want one that shows you where it is unsure, so you know what to check.