Test a rule you wrote against history, without fooling yourself

How it works

1. Write the rule precisely, for example how pullbacks to a rising moving average after a breakout from a base behaved, with exact definitions. The written rule must define entry, exit, and how one R is measured (stop price or fixed risk).
2. The agent writes Python that uses only data available at each point in time.
3. You read the generated code and confirm fill timing and the R definition before accepting any numbers.
4. It includes trading costs and slippage assumptions, and splits the data into in-sample and out-of-sample periods.
5. It reports the distribution of outcomes in R, not a single headline figure.
6. It lists the biases it checked for and anything it could not check (for example, delisted companies missing from your data).

Copy-paste prompt

Write and run a backtest for the rule in {{rule_file}} on {{daily_data_path}}.
The rule file must define entry, exit, and the stop or risk unit for R. If any is missing, stop and ask; do not invent them. Data schema is {{data_schema}}. If the file does not match, stop and report the mismatch. Long-only only.
1. Signals use only completed bars. Default fill is the next bar open. Print that fill rule. Do not fill at the signal bar close. State in code how look-ahead was prevented.
2. Include costs of {{cost_per_trade}} and slippage of {{slippage}}.
3. Split into in-sample {{in_sample_period}} and out-of-sample {{out_sample_period}}; report both.
4. Report the distribution of outcomes in R and the number of trades. No single headline return.
5. List the limitations, including survivorship bias if delisted names are missing.
6. State clearly that these results are for a single pre-defined rule only. Warn that testing many variants on this dataset would constitute p-hacking and invalidate the out-of-sample test. This is research for my own learning only; do not suggest the rule is viable for trading.
This is research for my own learning. Do not present results as a recommendation.
Print the script and its raw stdout for trade count and the in-sample and out-of-sample R distributions. Do not replace those figures with a prose summary.

Guardrails

• Watch for look-ahead and survivorship bias.
• Costs and out-of-sample checks are required.
• Results stay private; no published performance tables.
• Test only a single pre-specified rule; do not iterate or optimize parameters on the same dataset.
• Require the agent to output the full runnable script; user must execute it locally and spot-check entry/exit dates against a spreadsheet.
• Explicitly warn that even with in/out-of-sample and costs, live results will differ due to regime change and unmodeled slippage.
• If R, fill price, or column names are undefined, the agent must ask rather than assume. Quoted stats must be copied from script stdout.

What can go wrong

The most common errors are look-ahead from incorrect bar alignment, p-hacking via unlogged rule iterations on the same dataset, and survivorship bias from retail data that omits delisted tickers. Require the agent to output the exact code and have the user run it locally in a clean environment; never accept results without independent verification of the first 20 trades against manual calculation.

Source and references