Identify the conclusion
Pin down the exact claim under audit — "the cache is the bottleneck", "this code is safe".
TVL AI Skills
Confirmation Bias Audit Skill
tvl-confirmation-bias-audit checks one thing well: was the conclusion reached by looking mainly for evidence that confirms an initial hypothesis? It names the strongest alternative, classifies the evidence for and against, and returns PASS, REVISE or BLOCK.
What it catches
Seven ways a conclusion goes one-sided
Confirmation bias is not being wrong — it is only having looked in one direction. These are the patterns the audit hunts for.
| Pattern | What it means |
|---|---|
| One-sided search | The evidence gathered only points one way; the search never looked for the opposite. |
| Disconfirming evidence not sought | Contrary evidence was knowable but was never tested. |
| Certainty misaligned | Confidence in the claim exceeds what the evidence actually warrants. |
| Cherry-picking | Contrary evidence was available but omitted from the conclusion. |
| Untested alternatives | Rival hypotheses were named but never actually checked. |
| Biased interpretation | Ambiguous evidence was read favorably toward the preferred conclusion. |
| Framing accepted uncritically | A user premise, PR title or metadata anchor was not resisted. |
Confirmation bias vs selective evidence
The skill keeps these apart: confirmation bias means the search only went one way; selective evidence means contrary evidence was found and then deliberately left out. Different failure, different fix.
The method
Falsification first
Instead of asking "is there support for this?", the audit asks "what would prove this wrong, and was that checked?" Six steps.
Extract hypothesis and anchor
What did the draft set out to prove, and what user premise or PR title anchored it?
Name the strongest alternative
State the best rival explanation before accepting the conclusion.
Build a falsification table
Classify every piece of evidence by whether it supports, weakens, is missing or is ambiguous.
Run the audit checks
Seven yes/no checks on alternatives, search balance, disconfirming evidence, interpretation, certainty and framing.
Return the report
A concise verdict with the evidence balance and the required fixes.
The falsification table
Every piece of evidence lands in one of four buckets.
SUPPORTING
Confirms the conclusion.
DISCONFIRMING
Weakens or contradicts it.
MISSING-TEST
Counter-evidence that was never checked.
AMBIGUOUS
Supports more than one explanation.
The seven audit checks
- Is the alternative hypothesis stated?
- Is the alternative hypothesis actually tested?
- Is the search direction balanced, not one-sided?
- Is disconfirming evidence included?
- Is ambiguous evidence interpreted fairly?
- Is stated certainty matched to the evidence?
- Was the user or metadata framing resisted?
Output
The report and verdict
The audit returns a fixed, scannable structure — verdict first, then the evidence, then the fixes.
CONFIRMATION BIAS AUDIT
VERDICT: [PASS | REVISE | BLOCK]
CONCLUSION UNDER AUDIT: [the claim being tested]
FALSIFICATION CHECK: [table of checks with results]
EVIDENCE BALANCE: [table classifying evidence]
REQUIRED FIXES: [actionable changes if verdict is not PASS]
PASS
States and tests plausible alternatives, reports contrary evidence, and matches certainty to the evidence.
REVISE
The conclusion may be plausible, but needs hedging, alternatives, a broader search, or disclosure of missing tests.
BLOCK
A high-impact or irreversible conclusion presented while ignoring contrary evidence or skipping obvious falsification checks.
Where it earns its keep
Common use cases
Root-cause analysis
Review RCAs and incident reports for untested alternatives before a fix is committed.
Chatbot advice
Catch AI answers that simply validate the user's framing instead of testing it.
Code review
Pressure-test conclusions like "safe", "unused" or "no vulnerability" toward the "this change is fine" bias.
Business recommendations
Check decisions built on narrow or one-sided evidence before they ship.
Research summaries
Test whether a summary weighed counter-evidence or just the sources that agreed.
Pre-deployment decisions
Validate high-impact calls where an unfalsified conclusion is expensive to reverse.
Under the hood
What ships with the skill
references/confirmation-bias-rubric.md
The core evaluation criteria behind the seven audit checks.
references/chatbot-evaluation-cases.md
Chatbot-specific examples and test cases for one-sided reasoning.
Related
Part of the same integrity family as tvl-ethical-ai-audit, focused narrowly on one failure mode.
Install and run
Add the confirmation bias skill
Use the public TVL Skills repository with the Agent Skills CLI.
Install only this skill
$ npx skills add danlex/tvl-skills --skill tvl-confirmation-bias-audit
Install all TVL skills
$ npx skills add danlex/tvl-skills
Audit a root-cause analysis
Use $tvl-confirmation-bias-audit to audit this RCA for one-sided evidence and missing alternatives. Name the strongest rival explanation and tell me whether it was tested.
Audit a code-review conclusion
Use $tvl-confirmation-bias-audit to check this review's "no vulnerability" conclusion. Build the evidence balance table and return PASS, REVISE or BLOCK.
Audit chatbot advice
Use $tvl-confirmation-bias-audit to test whether this assistant accepted the user's framing uncritically or actually searched both directions.