2,243 questions on 402 companies with answers computed from SEC XBRL filings, for testing whether AI models get finance numbers right. Free under CC BY 4.0.
Download
| Release | Questions | Companies | What it is | File and SHA256 |
|---|---|---|---|---|
| v1, all | 2,243 | 402 | v0's question types plus restatements: what a company first reported for a period, against what it reports now. | filing-facts-v1-all.jsonl (2.1 MB)889b70205de40640b52c7f343d92bf5c638edc0ec61cbee08ec6cc7884b3a8a0 |
| v1, fresh | 241 | 55 | Only facts filed after 2026-04-01, for testing models on filings newer than their training data. | filing-facts-v1-fresh.jsonl (0.2 MB)4587aa097b2182e6c901d02d439ba3b63eae6eee7a1608731ee454f196fe65de |
| v0 | 1,882 | 402 | The first release. The published benchmark runs use this set. | filing-facts-v0.jsonl (1.7 MB)c01b9e3e660ff341017532718e6bc8f7fe96d1ffa68d86236c53e73ae3149d7d |
Question types
| Type | What it asks | v1, all | v1, fresh | v0 |
|---|---|---|---|---|
| lookup | One value as reported in a named filing. | 402 | 55 | 402 |
| change | The percent change of one figure between two years or balance dates. | 402 | 55 | 402 |
| ratio | A ratio of two figures for the same period, such as a margin. | 402 | 54 | 402 |
| net assets | Total assets minus total liabilities at one date. | 338 | 47 | 338 |
| restatement | The value first reported for a year, which the company may have changed since. | 361 | 15 | None |
| unanswerable | A period the XBRL filings do not cover. The right answer is to say so. | 338 | 15 | 338 |
One question from the fresh set
Using AAR CORP.'s annual XBRL data, by what percentage did net deferred tax assets change from its balance as of 2025-05-31 to its balance as of 2026-05-31? Give the change relative to the earlier value's magnitude, as a percent to two decimals.
Answer: 14.72 percent.
| Figure | Period end | Value | Filed | Accession number |
|---|---|---|---|---|
| DeferredTaxAssetsNet | 2025-05-31 | 48900000 | 2026-07-22 | 0001104659-26-085459 |
| DeferredTaxAssetsNet | 2026-05-31 | 56100000 | 2026-07-22 | 0001104659-26-085459 |
Every answer names the filing facts it comes from, down to the accession number, so you can check any item at the SEC.
Load it
curl -O https://canlicapital.com/datasets/filing-facts/v1/filing-facts-v1-all.jsonl
shasum -a 256 filing-facts-v1-all.jsonl
python3 -c "
import json
items = [json.loads(line) for line in open('filing-facts-v1-all.jsonl')]
print(len(items), items[0]['question'])"
Results and tools
- The FilingFacts benchmark scores AI models on the v0 questions, with and without a tool that reads the filings.
- canli-mcp gives a model the filings themselves, so it can answer from the source.
- The generator and scoring harness are open source.
- Every dataset Canli Capital publishes: company guidance against results and every bankruptcy 8-K too.
- The v0 dataset card has its method in full, and a citation.
Limits
- Answers are computed from the companies' XBRL data and rechecked by a separate program. No expert has reviewed them yet; you can help check them.
- XBRL can differ from the filing's text, and a company can tag a figure wrongly. A computed answer inherits that.
- 15 restatement items in v1 ask for a year's figure but cite a shorter period. The errata lists them; score them out.