The factor zoo

A friend forwarded me a model stock basket last week. Six-factor edge, the pitch said. Beats momentum. Look at the backtest.

The chart was beautiful. Smooth line, up and to the right, crushing the index. And that is exactly when I get nervous.

Because a backtest is not a result. It is a hypothesis dressed up as one.

Years ago, three researchers (Hou, Xue and Zhang) took 452 published "market-beating" factors and did the boring thing nobody does. They rebuilt every one on clean data and asked how many actually survived a real statistical test. Sixty-five percent could not clear even a basic bar. Raise it to an honest bar, the kind you should use when hundreds of people are mining the same history, and eighty-two percent were gone.

Most of the famous factors were never there.

Here is why this happens, and it is not fraud. Test 100 useless signals against the past and about five will look "significant" by pure chance. Now picture an entire industry fishing in the same dataset. Someone always lands a winner. It just doesn't predict the next year, because it never described anything real in the first one.

The few that hold up (size, value, momentum, quality, low volatility) share two things the zoo doesn't. There is an economic reason they should pay you, a risk you carry or a behaviour you exploit. And they keep working on data they were never built on.

So the next time a backtest takes your breath away, don't ask how good the line looks. Ask what makes the factor real, and whether it survives a bar high enough to kill a coincidence.

Anyone can find a pattern in yesterday. The only factor worth your money is the one that still shows up in a tomorrow it never got to see.

Why five in a hundred is guaranteed

The "about five" in the opening comes straight from how a significance test is set up. The usual bar asks whether a result would happen by luck less than 5 percent of the time. So if a signal has no real edge, it still has a 5 percent chance of passing. Test 100 such signals and you expect 5 to pass. Not because anyone cheated. Because that is what the 5 percent means.

The chance that at least one of them passes is much higher than 5 percent. With 20 useless signals it is 1 minus 0.95 to the power 20, about 64 percent. With 100, it is over 99 percent. If you test enough ideas, a beautiful backtest is close to certain.

Now lower the pass rate to 1 percent, a harder bar. With 100 signals you expect one false winner and the chance of at least one falls to about 63 percent. A higher bar is the only defence when many people test many ideas on the same history. That is what the "honest bar" in the opening means.

NSE's factor indices show the same gap

India has its own small version of this test. NSE launched each of its factor indices on a date, and published history before that date as a back calculation: the rule applied to past data. The back calculated years are a model. The years after launch are live.

These are real factors with sound reasons behind them, not data mined ones. Even so, the live five years came in at about half of the long run figure. Part of that is a hard market: the Nifty 200 itself did 8.30 percent over the same five years. Part of it is that a rule chosen by people who could already see the past will look better on that past than on the years that follow.

The quality index trailed its parent over those five live years. The momentum index stayed only about one point ahead. A real factor survives out of sample. It rarely survives at the size of its back test.

Four questions before you trust a factor

The opening names the two tests that matter. Here they are as a checklist, with two more that follow from them.

1. Is there a reason?A risk the holder carries or a behaviour the rule exploits. No story, no premium.
2. Does it clear a high bar?How many versions were tried before this one? The more tries, the higher the bar must be.
3. Does it work on new data?Years after the rule was fixed, and other markets it was not built on.
4. Does it survive costs?Turnover, spreads and tax take a share of every premium before it reaches an account.

The last step is its own post: the turnover tax shows how much of a premium trading costs can take. And once a factor is famous, money crowds in; factor crowding and decay covers what that does to the edge. The free Learn module on introduction to factors covers the five classic factors and the economic reason behind each one.