canli-mcp / Overfit backtests

How do I tell whether a backtest is overfit?

Count how many settings were tried and correct the Sharpe ratio for it. canli-mcp's validate_deflated_sharpe gives the probability that the true Sharpe is above zero after that many trials, and validate_overfitting, validate_reality_check and placebo_test go further, all on your machine.

Add to CursorAdd to VS Code

Free and MIT-licensed. No account and no API key. Or run claude mcp add canli -- npx -y canli-mcp in Claude Code.

Ask it

My strategy has a Sharpe of 1.4 over five years, and I tried about 40 versions. Is it real?

The agent finds the tool through canli-mcp's find_tool and runs it through run_tool. This is the call:

{
  "name": "validate_deflated_sharpe",
  "arguments": {
    "observed_sharpe_annualized": 1.4,
    "observations": 1260,
    "periods_per_year": 252,
    "skew": -0.3,
    "non_excess_kurtosis": 5,
    "effective_independent_trials": 40,
    "cross_trial_sharpe_sd_annualized": 0.5
  }
}

What came back

Recorded from canli-mcp 0.11.1 on 2026-10-25, with no API key set. The model read 1,520 characters.

The result validate_deflated_sharpe sent back
{
  "ref": "r1",
  "computed": "locally",
  "note": "Computed on this machine in local mode. Nothing was sent to canlicapital.com and no receipt was stored.",
  "data": {
    "input_mode": "contract_inputs",
    "derived_inputs": {
      "observed_sharpe_annualized": 1.4,
      "observations": 1260,
      "periods_per_year": 252,
      "skew": -0.3,
      "non_excess_kurtosis": 5,
      "effective_independent_trials": 40,
      "cross_trial_sharpe_sd_annualized": 0.5
    },
    "result": {
      "observed_sharpe_per_period": 0.0881917,
      "cross_trial_sharpe_variance_per_period": 0.000992063,
      "expected_max_sharpe_per_period": 0.0689621,
      "expected_max_sharpe_annualized": 1.09474,
      "psr_against_zero": 0.998955,
      "deflated_sharpe_ratio": 0.748865,
      "non_normality_variance_term": 1.03424,
      "search_haircut_annualized": 0.30526,
      "psr_z_score": 3.07702,
      "dsr_z_score": 0.670923
    },
    "plain_reading": "Counting only sample uncertainty, the probability this Sharpe is above zero is 99.9 percent. Deflated for the best-by-luck Sharpe the declar…"
  },
  "limits": [
    "This verdict is about the series exactly as submitted. The service never saw the data source, its costs, survivorship, or any lookahead in h…",
    "A deflated Sharpe or overfitting probability above or below any threshold is not admission to anything and is not a forecast.",
    "The receipt is content-hashed, reproducible from the open-source core it names, and signed with Ed25519 by a key published at https://canlic…"
  ],
  "receipt": null
}

Prices, filings and statistics are as the sources published them on that day; run the call again for today's. Nothing here is investment advice.

The tools behind the answer

From canli-mcp's tool index
ToolPackWhat it does
validate_deflated_sharpevalidationDeflated Sharpe ratio: the probability (0 to 1) that the selected strategy's Sharpe beats the best that luck gives across the variants tried, with the probabilistic Sharpe and that luck benchmark. Send the seven statistics or a return series. With every variant's returns use validate_overfitting; luck as a trial count, validate_luck_trials; a multiple-testing haircut, validate_haircut_sharpe. A deflated Sharpe or overfitting probability above or below any threshold is not admission to anything and is not a forecast.
validate_overfittingvalidationProbability of backtest overfitting (0 to 1) by CSCV: how often the in-sample best variant falls below the out-of-sample median. Needs every variant's returns (periods by variants); with summary statistics only, use validate_deflated_sharpe. A deflated Sharpe or overfitting probability above or below any threshold is not admission to anything and is not a forecast.
validate_reality_checkvalidationData-snooping tests on every variant a search tried: Hansen's SPA p-value that the best beat the benchmark only by luck, White's Reality Check, and the variants Romano-Wolf StepM finds better. Send all variants tried, not only the winners. A deflated Sharpe or overfitting probability above or below any threshold is not admission to anything and is not a forecast.
placebo_testvalidationDoes your pipeline find edges in noise? plan writes placebo files (your data's returns in random orders, so nothing is predictable); run the whole pipeline on real.csv and on each, then compare for a p-value that counts every choice it makes. Beating the placebos shows the pipeline finds more in the real order of the returns than in random orders of the same returns; it does not show the edge survives costs, capacity or the future.

Add canli-mcp

Claude Code
claude mcp add canli -- npx -y canli-mcp
Cursor, Claude Desktop or any MCP client
{ "mcpServers": { "canli": { "command": "npx", "args": ["-y", "canli-mcp"] } } }
claude.ai, ChatGPT or any client that takes a URL
https://canlicapital.com/mcp/canli

Everything canli-mcp does · How it compares with other finance MCP servers

Related questions