Quantitative finance has a statistics problem. Not too little statistics; too much confidence in it. Every factor, every strategy and every signal arrives dressed in t statistics, p values and Sharpe ratios. Most of them mean far less than they appear to.
What follows is about healthy scepticism, starting with my own research. The factor premia are real and the evidence for them is substantial. The methods used to find and validate them are routinely misused, and knowing those misuses is the first step to research that survives contact with a live market.
- 1 strategy tested5.0%
- 10 tested40.1%
- 20 tested64.2%
- 100 tested99.4%
Chance that at least one strategy with no edge passes a p < 0.05 test: 1 − 0.95ⁿ, for n independent tests.
The problem with normal distributions
Standard portfolio theory and most back test statistics assume returns follow a normal distribution, the bell curve. The assumption is mathematically convenient and empirically wrong.
Real returns have fat tails: extreme days happen far more often than a bell curve predicts. You do not need decades of data to see it. I ran the numbers on the Nifty 50's daily closes from 7 Oct 2024 to 9 Oct 2026, 498 daily returns including the two Budget day sessions.
| Statistic, Nifty 50 daily log returns | Actual | If returns were normal |
|---|---|---|
| Annualised volatility | 13.1% | |
| Kurtosis | 5.3 | 3.0 |
| Days beyond 3 standard deviations | 6 | about 1.3 |
| Days beyond 2 standard deviations | 22 | about 22.6 |
| Largest one day fall | −3.3% |
Computed by RupeeCase from NSE daily index closes, 7 Oct 2024 to 9 Oct 2026. Price index, dividends excluded.
Look at the pattern. The moderate moves, beyond 2 standard deviations, arrive almost exactly as often as a bell curve says. The extreme ones, beyond 3, arrive about four and a half times as often. That is what fat tails mean in practice: the model is fine on ordinary days and wrong on the days that matter.
The practical consequence: any risk number that assumes normality, such as a confidence interval built from standard deviation or a parametric value at risk, understates tail risk. Maximum drawdown is a more honest measure precisely because it assumes nothing. It just reads the worst fall that actually happened.
P values and what they actually mean
A back test shows a t statistic of 2.3 and a p value of 0.02. "Significant at the 5% level", so the strategy works? Not necessarily. The p value answers a narrow question: if this strategy had no edge at all, how likely would results this good be by chance? Here, 2%. It sounds convincing. Two problems:
- The p value is not the probability that the strategy works. It is the probability of these results if it does not work. Those are very different statements, and confusing them is the most common error in research.
- Statistical significance is not economic significance. A strategy with a t statistic of 2.1 and 0.05% a year of excess return is significant and worthless after STT and charges.
≈ Sharpe × √years
That second formula is worth memorising. It tells you that a five year back test, which is all most Indian strategies can show with clean data, can only confirm a high Sharpe ratio. A modest edge cannot be proven in five years at all. It has to be supported by economic reasoning and by out of sample evidence instead.
Harvey, Liu and Zhu (2016) counted hundreds of published factors and argued that, given how many have been tested, a new factor should clear a t statistic of about 3.0, not the traditional 2.0, before anyone takes it seriously.
MythA p value of 0.05 means a 95% chance the strategy works.
FactIt means a 5% chance of results this good if the strategy had no edge. With many strategies tested, some will pass that bar by luck alone.
The multiple testing problem
This is the most important statistical idea in quantitative research and the most ignored. Test 100 completely random signals on the Nifty 500 and about 5 will look significant at the 5% level by pure chance. Publish the best three as discovered factors and you have done what researchers call p hacking, whether you meant to or not.
Most research, published work included, does not say how many signals were tried before the winner was chosen. The back test shows the best of many trials, and the statistics are computed as if it were the only one.
Worked example: twenty moving averages
Say you test a moving average crossover on the Nifty 50 with 20 different look back lengths, 10 days to 200 days, and the best one shows a p value of 0.01. Alone that looks strong. Across 20 tries, Bonferroni says you needed 0.0025. The best of twenty random rules would pass 0.01 about 18% of the time if the tests were independent (1 − 0.99²⁰), and look back lengths are related, so treat that as a rough guide. Your winner is not evidence yet. It is a candidate, and it needs data it has never seen.
The honest check: for any factor or strategy, ask how many signals were tested to find this one. If the answer is many, and no correction was applied, treat the result with real scepticism, including your own research.
Stationarity and regime change
A signal that worked from 2005 to 2015 may not work from 2015 to 2025. Financial time series are non stationary: their statistical behaviour changes as market structure, participants and information change. That creates a tension. You want long histories for statistical power, but old data may describe a market that no longer exists.
The Nifty 50 makes the point inside just two years. Its 21 day volatility ranged from 5.2% in early September 2026 to 28.3% in early April 2026 (computed from NSE daily closes). A risk model fitted on either month alone would describe a different market.
- Write the reason firstAn economic mechanism, before any back test.
- Log every testEach variant, parameter and data set you try.
- Correct for the triesRaise the bar with the number of tests, for example Bonferroni.
- Check sub periodsLook for structural breaks, not just the whole period average.
- Hold out dataJudge the final rule on data it never saw. Module 5.5.
Autocorrelation and inflated t statistics
Standard tests assume independent observations. Strategy returns often are not. A momentum strategy holds similar positions from one month to the next, so its monthly returns are related. That overstates the effective sample size and makes t statistics look stronger than they are.
Overlapping windows make it worse. A five year back test reported as 49 rolling one year returns has 49 numbers but only about five independent years. The fix: use Newey West standard errors for time series t statistics, or judge performance on non overlapping periods such as calendar years. Treat any chart of rolling returns as a picture of consistency, not as 49 separate pieces of evidence.
Common mistake: trusting the Sharpe of a short back test
The common mistake is reading a back tested Sharpe ratio as a fact about the future. A Sharpe ratio estimated over five years has a wide error band. Using the rule of thumb above, the standard error of an annual Sharpe estimate is roughly 1 / √years, about 0.45 over five years. A measured 1.6 is consistent with a true Sharpe anywhere from below 1 to above 2.
Add selection: the strategy you are shown is usually the best of several variants, so its measured Sharpe is biased upwards. Expect live results to come in lower than the back test. The good strategies are the ones that still work after that haircut.
How this shows up in RupeeCase
Machine learning is used in RupeeCase research to search and optimise the rules behind each strategy, tested out of sample; the rule that ships is then frozen and published, so the live strategy is a fixed algorithm, not one that keeps refitting itself. The factsheets show calendar year returns, which do not overlap, next to rolling 1 year and 3 year windows, which do; read the rolling ones as consistency, not as extra evidence. Every return is labelled model, back tested, with its window. See an example on the Allcap factsheet, and the full record on the track record page.
Questions readers ask
What t statistic should I look for in a back test?
For a new idea, at least 3 rather than 2, because so many ideas have already been tested. Remember that t is roughly the Sharpe ratio times the square root of the years tested, so short back tests rarely reach it unless the Sharpe is very high.
Are Indian market returns normally distributed?
No. Daily Nifty 50 returns over the two years to October 2026 had a kurtosis of 5.3 against 3 for a normal distribution, and about four and a half times as many 3 standard deviation days as a bell curve predicts.
How long a back test do I need?
Long enough to include at least one serious fall and, ideally, more than one market regime. Five years is a minimum; for a modest edge the statistics alone will not be conclusive, so out of sample evidence and an economic reason matter more.