Benchmark · a new test every week
Can AI answer questions on last week's filings?
Every week we take the annual reports companies filed with the SEC, ask an AI questions only those reports can answer, and check each answer against the filing. This week: 93 questions from 31 companies.
Same question, asked twiceReceipt
AutoZone, Inc.: revenue for its fiscal year, from the 10-K it filed on
- AI on its own
- "not reported"
- AI with canli-mcp
- $20,338,555,000
- Filed with the SEC
- $20,338,555,000
Matches the filing
Open the 10-K on SEC.gov- 0 of 91answered right by gpt-6.1-terra on its own; it said it did not know rather than guess
- 89 of 91answered right by the same model with canli-mcp
- 0wrong numbers given with canli-mcp; 2 questions ran out of turns
- 45annual reports filed from to
Week 2026-W44
This week's results
91 of the week's 93 questions, up to 30 of each kind, went to the same model twice, with at most six turns each.
| Who answered | Right | Wrong numbers | No number | Tokens a question |
|---|---|---|---|---|
| gpt-6.1-terra with canli-mcp 0.20.0 | 89 of 91 | 0 | 2 | 6,769 |
| gpt-6.1-terra on its own | 0 of 91 | 0 | 91 | 456 |
Both misses with canli-mcp were the same company, Here Group Limited, which had changed its name; the server's company search knew only the old one, and the model found the company another way just as it ran out of turns. A fix is merged for the next release.
How it works
Four steps, the same every week
Take the week's annual reports
The SEC's daily index lists every 10-K, 20-F and 40-F filed that week; amendments are left out.
Ask what only they can answer
Each question needs a figure the company reported for the first time that week: a lookup, a change from the year before, a ratio or assets minus liabilities.
Ask the same model twice
Once on its own, once with canli-mcp. Same prompt, same limits, run on the open-source eval.mjs.
Check every answer against the filing
An independent checker recomputes each answer from the SEC's data; nothing is tuned on the week's questions.
Everything is public
Download the questions and both runs
The questions
93 questions with their answers and the filing each one cites.
items.jsonlHow they were built
The filings read, every company used or skipped, and checksums.
manifest.jsongpt-6.1-terra with canli-mcp 0.20.0
Every answer as given, with its tool calls and tokens.
Download the rungpt-6.1-terra on its own
Every answer as given, with its tool calls and tokens.
Download the run
Questions
About the test
Why use filings from last week?
A model learns from text written before its training ended. Figures a company first reported last week cannot be in that text, so the model either reads the filing or does not know. That separates an assistant that looks things up from one that recalls or guesses.
Who writes the questions?
A program, the same every week: weekly.mjs reads the SEC's list of annual reports filed that week and asks about figures each report gives for the first time. Nobody picks questions a tool is good at.
How is an answer marked right?
It has to match the figure in the filing: a dollar amount within a tenth of a percent, a percentage within five hundredths of a point, a ratio within five ten-thousandths. Saying "not reported" is never right here, because every question has an answer in the filing.
Is this a fair test of other tools?
This page compares one model alone and with canli-mcp, which we build, so weigh that. Every question, answer and run is downloadable below, so anyone can run another tool on the same questions.
Limits: one model and one run a week; questions come from a fixed set of templates over figures companies tag in their filings; a company's earnings release can give a figure some days before its annual report. Not investment advice.