by James M Dewar
This project is unfunded — depending on usage, rate limits may apply.

Verification

Decide whether to accept an output. Replicates show whether an answer is stable, and an evidence request shows what each claim rests on. Neither replaces reading the source.

Steps

In the workshop, these steps are the exercise. You can also follow them on your own.

  1. Take a prompt whose answer matters e.g. the one from Prompting, or one of your own.
  2. Run it unchanged in three fresh conversations, using the same model, attached material and tools. Repeats within one conversation are not replicates, because each answer is shaped by the ones before it.
  3. Paste the outputs into Step 3 · Compare replicates, copy the comparison prompt it builds, and run that in another fresh conversation.
  4. Add the block in Step 4 · Evidence request to the end of your prompt and run it in a new conversation. The answer will end with a list of its claims, each with a source and a quotation.
  5. Pick two claims that would change what you do, and check each one as in Step 5 · Checking a claim against its quotation, below. Search the source for the quotation (Ctrl+F), then decide whether it supports the claim as strongly as stated. If the quotation is missing or does not support the claim, read the source yourself.

Replicates that agree show that an answer is stable. It can still be wrong, because every replicate can share the same missing evidence or bias. Running the prompt on a different model can also help, although that changes a second condition at the same time.

Step 3 · Compare replicates

Paste the output of each fresh run. The Copy this panel builds a single prompt that compares them, which you then run in your own chat tool.

When to use it: After running the same prompt in separate fresh conversations.

Paste the outputs

Lets the comparison judge the responses against what was asked.

Nothing you paste here leaves your browser, because the prompt is built in the page. The comparison helps you choose what to check, but you still need to open the source or redo the calculation yourself.

Copy this

Compare these three responses to the same task. Each was produced in a separate fresh conversation from the identical prompt. Identify consequential agreements and disagreements in conclusions, assumptions, evidence, and proposed actions.

Do not infer correctness from agreement. Select two claims whose verification would materially affect the conclusion, and state how each could be checked against the original evidence.

<response_1>
[paste the output of replicate 1]
</response_1>

<response_2>
[paste the output of replicate 2]
</response_2>

<response_3>
[paste the output of replicate 3]
</response_3>

Step 4 · Evidence request

A section to add to the end of any prompt. The answer then ends with a list of its claims, each with its source and a quotation you can look for.

When to use it: Whenever you will rely on what the answer says about a paper, a dataset or another source the model can read.

Copy this

**REQUIRED EVIDENCE**
End your response with a section titled "Claims and evidence" that lists every factual claim made above, in the order it appears. For each claim, give:
1. The claim, worded as it is above and at the same strength.
2. Its source: the document and the most precise location available (section, figure and panel, table, or page), or the DOI, PMID or URL.
3. A quotation from that source that supports the claim, copied word for word.

Never paraphrase, combine or reconstruct text inside quotation marks. When the evidence is a figure, graph or table, name the figure, panel or table rather than inventing a quotation. If a quotation supports only part of a claim, say which part. If you cannot locate a source or a supporting quotation, write "not located" for that claim; do not substitute a related passage. Do not invent sources, page numbers or locations.

The list makes checking faster, but you still have to do the check. A quotation can be invented, and a real quotation may not support the claim, so check both.

Step 5 · Checking a claim against its quotation

AI is most useful when its output is quick to check. Finding a quotation in the source takes seconds. Deciding whether it supports the claim takes your judgment.

  1. Ask for quotationsThe evidence request does this, so each claim comes with its source and a quotation.
  2. Test existenceIs the quotation in the source? Search for it (Ctrl+F).
  3. Test supportDoes it support the claim as strongly as stated? This part needs your judgment.
  4. Accept only if both passIf either test fails, read the source yourself.