Method

For each case we ask a fixed list of questions on several days, before and after a change. We ask three times per AI product each day. A person reads every answer. Each wrong claim gets one finding: source error, unsupported statement or source unknown.

What we record

Field	What we record
Date and time	UTC, to the minute
AI product	The product and mode, with the model name if shown
Route	API, or a person using the app by hand (“manual”)
Settings	Signed out, fresh session, chosen country and language
Question	The exact text, from the case’s fixed list
Answer	The full text: raw response for API, screenshot for manual
Cited sources	Every link shown, in order

API answers can differ from what the app shows. We collect answers only in ways each product’s terms allow, so a case names any product we couldn’t check.

How we count

Answers change between runs, so one observation shows only that a claim appeared once. Four rules decide how we count:

1. A failed run, or one the product declined to answer, never counts as the claim being absent.
2. Runs on the day a change went live count as neither before nor after.
3. We report API and manual answers separately and never add them together.
4. An answer nobody has read yet counts as “not yet checked”.

Classifying a claim

Each finding comes with the searches and checks behind it, so a reader can repeat them.

Source error

A page or dataset states the wrong information, and the answer cites it or repeats its wording closely. We name the page and the date we read it.

Unsupported statement

The answer cites sources, but none says what the answer says, and we found no other page that does.

Source unknown

We couldn’t find where the claim came from. That’s a valid finding: a model can produce a claim with no source, and a page we never found may still exist.

Changes and what they can show

An intervention is a change made to correct an answer, such as fixing a directory listing or updating the business’s own site. We record what changed, where, and the date it went live.

A claim that disappears after a change hasn’t been shown to disappear because of it. Models are updated, search indexes refresh, and answers vary from run to run.

Cases and permission

We document a business only with its permission, or our own brand. A case critical of a named business also needs the publisher’s review.

Evidence we keep

For every case we keep the questions, answers, cited links and the pages we checked, each with its capture date. If a source page changes later, our record keeps what it said when we read it. Each case also records the method version it used.

Changes to this method

• October 14, 2026: First version.