Does Backtesting Actually Work? What 9 Years of Crypto Data Tells Us

Backtesting has a reputation problem. Traders who have been burned by strategies that looked perfect in hindsight but failed in live markets often dismiss backtesting entirely. Quants and developers, on the other hand, treat it as the foundation of systematic trading. Both sides have valid points. The question is not whether backtesting works but whether you are doing it correctly.

Nine years of Bitcoin price data, analyzed across hundreds of strategy variants, gives a clear answer: when done with proper methodology, backtesting produces results that persist into live trading at a rate that is far above chance. The problem is that most retail traders violate the methodology in ways that guarantee failure.

The Skeptic's Case

The skepticism around backtesting is not irrational. It is based on real patterns that real traders have experienced.

The Overfitting Problem

Overfitting occurs when a strategy is tuned so precisely to historical data that it captures noise rather than genuine market patterns. A trader who tests 200 parameter combinations of RSI settings on 2021 Bitcoin data will almost certainly find a combination that returned 400%. That combination has zero predictive value because it was optimized to fit one specific data sequence.

This is the most common form of bad backtesting. The trader does not realize they have effectively memorized the test rather than identified a repeatable edge. When markets shift even slightly, the strategy collapses.

The Cherry-Picking Problem

Related to overfitting is selective reporting. Traders naturally remember the backtest results that impressed them and forget the ones that did not. They share the strategy that returned 300% in 2021 without mentioning that the same strategy lost 60% in 2022. Academic researchers call this publication bias. In trading, it is simply self-deception.

The Unrealistic Assumptions Problem

Many backtests ignore trading fees, slippage, and liquidity constraints. A strategy that trades 50 times per month with 0.1% fees per trade faces a 5% monthly drag before capturing any market edge. A strategy that requires entering at exact candle close prices may be impossible to execute in practice due to slippage. When these costs are excluded, backtest results look dramatically better than live performance will ever be.

The goal of backtesting is not to find a strategy that looks good. It is to find a strategy that will still look good after you have tried as hard as possible to break it.

What 9 Years of Bitcoin Data Actually Shows

CoinQuant analyzed Bitcoin trading data from 2016 through 2024 using tick-level data sourced via Kaiko from major exchanges including Binance. This timeframe spans multiple complete market cycles: two major bull runs, two prolonged bear markets, and several significant drawdown-and-recovery sequences. The dataset provides enough variation to distinguish strategies with genuine edge from those that are artifacts of a specific market period.

The Key Finding: Sharpe Ratio as a Persistence Predictor

Across hundreds of strategy configurations tested on this nine-year dataset, one metric emerged as the strongest predictor of forward performance: the Sharpe ratio from the backtest period.

Strategies with a Sharpe ratio above 1.2 during the backtest period showed 73% persistence into the following year. That means nearly three out of four strategies meeting this threshold continued to outperform a simple buy-and-hold approach in the subsequent 12 months. Strategies with Sharpe ratios below 0.8 showed only 31% persistence, barely above what random chance would predict.

This is meaningful data. A 73% persistence rate does not mean the strategy will work forever. It means the underlying logic is capturing something real rather than something random.

The Market Regime Factor

The data also revealed that strategy persistence is heavily influenced by market regime. A momentum strategy backtested primarily on 2020 and 2021 data showed only 41% persistence when tested against the 2022 bear market. The same strategy, when tested across the full nine years, showed 68% persistence going forward.

This points to a fundamental rule: a strategy must be tested across multiple market regimes to have credible forward-looking validity. Testing only on favorable data is a form of cherry-picking even when it is done unintentionally.

Fee Impact on Real Returns

Across the analyzed strategies, including realistic trading fees reduced annual returns by an average of 8.3 percentage points compared to fee-free backtests. For strategies that traded frequently (more than 20 times per month), the fee drag often eliminated the entire edge. For strategies with average holding periods above 10 days, fee impact was minimal. This is one reason longer-duration swing strategies tend to backtest more reliably than short-term scalping approaches in crypto.

Why Most Backtests Fail

Understanding the specific failure modes helps traders avoid them:

• In-sample optimization: Testing and optimizing on the same data produces results that look better than they are. A separate out-of-sample period must be reserved and never touched until final validation.

• Insufficient trade count: A strategy with 12 total trades over five years has no statistical validity. Random variation across 12 trades is enormous. A minimum of 30 to 50 trades is needed before results mean anything.

• Single timeframe testing: A strategy that works on the daily chart may fail on the 4-hour or weekly. Testing across multiple timeframes reveals whether the logic is robust or timeframe-specific.

• Ignoring drawdown duration: Maximum drawdown as a percentage tells only part of the story. A 30% drawdown that lasted 6 months is very different from a 30% drawdown that lasted 2 years. The second is psychologically almost impossible to hold through.

• No walk-forward testing: Static backtests use fixed parameters across all time. Walk-forward testing re-optimizes parameters at regular intervals using only past data, simulating how a trader would actually manage the strategy over time.

How to Do It Right

Effective backtesting follows a specific sequence:

1. Define the strategy rules completely before looking at results. Not after. If you adjust rules based on what you see in the backtest, you are overfitting.

2. Split your data. Use 70% for development and 30% as a holdout test set that you do not touch until your strategy is finalized.

3. Include all costs. Trading fees, slippage estimates (0.05 to 0.2% per trade depending on exchange and liquidity), and withdrawal costs if relevant.

4. Require multi-regime coverage. Your backtest period must include at least one bear market and one bull market. For crypto, this means 2018-2019 or 2022 must be in your dataset.

5. Check the Quality Score. CoinQuant calculates a Quality Score for every backtest that penalizes strategies showing signs of overfitting, low trade counts, or regime-specific performance spikes. A high Quality Score is not a guarantee but a strong filter.

6. Validate out-of-sample. After finalizing your strategy on the development set, run it on your holdout period. If performance degrades dramatically, you have overfitted.

7. Paper trade before going live. Run the strategy in a simulated environment with real-time data for at least 30 trades before committing real capital.

The bottom line is that backtesting works when it is approached as a methodology for disconfirmation rather than confirmation. You are trying to find reasons why the strategy should not work. If it survives that scrutiny across multiple years, multiple market regimes, and realistic cost assumptions, you have something worth testing in live markets.

Test your strategy the right way on CoinQuant. Free. No code required.