Backtest validation · Quantitative research
CopyTradeAudit
A backtesting engine built to break its own results — and the honest negative it produced.
- Python
- FastAPI
- pytest
- SQLite
- uv
The problem
Plenty of products sell the same idea: prediction markets publish every trade, some traders are consistently profitable, so follow them and profit. The idea is easy to backtest and easy to backtest wrong — a leak of future information, or fifty correlated markets counted as fifty independent observations, and the result looks excellent while meaning nothing.
Who it's for
Anyone whose decision rests on a backtest — a strategy about to go live, a vendor's track record, a number that looks too good to argue with.
What it does
A deterministic research core in Python — roughly 3,500 lines behind 103 tests — measured against 370 markets, 134,000 real trades and 46 independent events. The verdict was no measurable edge: copy-selected trades returned +0.0200 per share above the 0.95 price band, while buying that same band blindly, with no traders involved at all, returned +0.0211.
Key decisions
- Leakage-safe by construction: at any signal, a trader's skill is computed only from markets that had already resolved at that moment — never from what happened afterwards.
- Event-clustered bootstrap intervals, so fifty correlated markets inside one event count as one observation rather than fifty.
- Execution simulated against a real order book with fees, slippage and partial fills, instead of assuming every order filled at the midpoint.
- Published the negative result rather than the flattering one, and marked one strategy INCONCLUSIVE where the measurement — not the model — turned out to be broken.
What I learned
I found four bugs in my own methodology, each of which had already manufactured a confident false positive — including a strategy with a 100% win rate over 36 trades whose real edge over blind entry was 0.8%. Ranking traders by win rate doesn't find skill; it finds people buying near-certainties at 98 cents. A confident wrong answer costs more than no answer.
Where it goes next
Turn the leakage and clustering checks into a harness that can be pointed at someone else's backtest, so the same scrutiny runs against a strategy I didn't write.
Paper only — no live trading and no real money, by design.