Every second fund manager now says AI or machine learning. Most of them mean a gradient boosting model on financial ratios, relabelled as ML driven alpha. A few mean something more. The problem is that equity return prediction is one of the hardest places to apply ML, for reasons that have nothing to do with how clever the model is.

Machine learning is part of how RupeeCase research is done, so this is not a sceptic's essay. It is an honest map of where ML helps in systematic equity investing, where it does not, and why that line matters more than any headline.

  1. Features with a reasonEconomic priors first, not every column you have.
  2. Split by timeTrain on the past, test on the future, never shuffle.
  3. Walk forwardRefit on a rolling window, test on the next block.
  4. Regularise hardFewer parameters, shallow trees, penalties.
  5. Untouched hold outA final period nobody looked at during research.

Most ML projects that fail in production fail at step 2. Shuffle financial data at random and the model trains on information from the future.

The core problem: signal to noise ratio

In image recognition a good model can be right more than 99% of the time. In equity return prediction, a model that calls the direction of monthly stock returns correctly 55% of the time is genuinely strong. The reason is noise. The predictable part of a stock's return is a small fraction of its total variance.

0.05
An information coefficient (correlation between forecast and next period return) many quant researchers would be happy with
−0.02
Lag 1 autocorrelation of Nifty 50 daily returns, 7 Oct 2024 to 9 Oct 2026, computed from NSE closes: yesterday says almost nothing about today

That has a sharp consequence. Models built for high signal problems, deep networks with millions of parameters, are hugely over specified for equity returns. With so little signal, extra capacity goes into memorising noise.

Worked example: why a small edge is enough

The fundamental law of active management, from Grinold, links skill and breadth: information ratio ≈ IC × √(independent bets per year). Suppose your model has an IC of 0.05, barely visible on any single stock. Rank 200 stocks a month and treat about 200 of the 2,400 rankings a year as independent. IR ≈ 0.05 × √200 ≈ 0.71. That is a strong information ratio from a forecast that is wrong almost half the time.

Now the trap. Double the breadth by trading weekly and, on paper, IR rises to about 1.0. In practice weekly trading quadruples turnover, and at 0.1% STT on each side of a delivery trade plus charges, the extra costs can easily exceed the extra edge. Breadth is only free on a spreadsheet.

Small skill, wide breadth: the fundamental law

Information ratio, before costs 0.71

Grinold's fundamental law of active management: IR ≈ IC × √breadth. Costs reduce it, more so at high turnover.

The size of moves repeats; their direction does not
  1. Returns, lag 1 day−0.02
  2. Absolute returns, lag 1 day0.21
  3. Squared returns, lag 1 day0.14
  4. Absolute returns, lag 5 days0.13
  5. Absolute returns, lag 20 days0.12

Autocorrelation of Nifty 50 daily log returns, 7 Oct 2024 to 9 Oct 2026, computed by RupeeCase from NSE closes.

The sample size problem

Take the Nifty 500 and 20 years of monthly returns: 500 × 12 × 20 = 120,000 observations. It sounds like plenty. It is not, because the observations are not independent. Stocks move together on the same day, positions persist from month to month, and the market changes regime. The effective sample size is a small fraction of the raw count. In Indian data the problem is sharper, because clean, survivorship free histories for the broader market are shorter than in the US.

So complex models are almost always over fitted on financial data. The right answer is not to avoid ML. It is to use regularised, parsimonious models and validate them on data they never touched.

What actually works in practice

Searching rule designs
Works well
Using ML and large scale search to choose which signals rank a universe, how many names to hold and how often to rebalance, then testing each choice out of sample. The output is a simple, fixed rule a human can read, not a model that changes itself every day.
Feature engineering
Works well
Combining raw inputs into robust composites, detecting interactions between factors, extracting structured information from filings. ML as a feature builder, not as the return predictor.
Tree ensembles
Works with care
Random forests and gradient boosted trees handle non linear effects and give feature importance. Gu, Kelly and Xiu (2020) found tree models and shallow neural networks beat linear models on a large US sample. Keep them shallow and regularised on Indian data.
Text from filings
Conditional
Under SEBI's listing rules, listed companies publish earnings call transcripts on their websites within five working days of the call. Tone and guidance consistency can be extracted. The data is messy, often scanned PDFs, and edges decay as more people use them.
Regime classification
Conditional
Clustering or hidden Markov models label markets as calm, trending or stressed. Regimes are clear in hindsight and noisy in real time, and a false switch is expensive. A simple trailing average rule often does as well.
Deep sequence models on prices
Mostly hype
Networks that read raw price sequences need large data and strong signal; monthly equity returns have neither. Where deep learning does help is very high frequency order book data, not monthly factor strategies.
Reinforcement learning
Not practical yet
An active research area for portfolio decisions. Training needs simulations that rarely match real markets, and policies are fragile when conditions shift. Worth following, not deploying.

The honest bottom line: a well built rule with an economic reason, careful costs and disciplined rebalancing beats a complex model most of the time. ML earns its place in search, feature construction and text processing. It never replaces the question of why a signal should keep working.

A minimal honest experiment

If you want to try ML on Indian stocks yourself, start small and strict. Take NSE daily prices for a point in time universe, say the Nifty 200 as it stood on each date. Build five features with a reason behind them: 12 month return skipping the last month, 1 month return, 6 month volatility, distance from the 52 week high, and average traded value. Label each stock by whether its next month return beat the universe median.

Train a simple logistic regression and a shallow tree model on 2015 to 2020, tune on 2021 and 2022, and look at 2023 onwards exactly once. Rank the stocks each month, hold the top 20 equal weighted, and deduct STT, stamp duty, exchange charges and GST on every trade. If the shallow tree does not beat the logistic regression after costs on the final period, keep the simpler model. Most of the time, that is the answer.

Leakage: the silent killer

Leakage means information from the future sneaking into training. It produces spectacular back tests and live disasters. The common Indian forms:

  • Overlapping labels. Predict 21 day forward returns every day, and neighbouring labels share 20 days of the same future. A random train test split puts near copies of test rows in training. Purge the overlap and add a gap, an embargo, between training and test periods.
  • Restated fundamentals. Using today's version of a company's past results instead of what was filed on the day. Use data as of the filing date, which for quarterly results is within 45 days of quarter end under SEBI's listing rules, or 60 days for the year end.
  • Survivorship. Training only on stocks that are listed today drops every company that was delisted or went bust. The model learns that falling stocks recover.
  • Index membership. Using today's Nifty 500 list for a 2015 back test. Membership must be point in time; NSE reviews its indices twice a year.

MythA more complex model finds more alpha.

FactWith weak signals and short, overlapping data, extra complexity mostly fits noise. Simple, regularised models usually hold up better on data they have not seen.

Common mistake: reporting the best of many models

The common mistake is not a bad model. It is a good looking one picked from hundreds. Every hyper parameter tried, every feature set tested, every random seed is another draw. Report the best and you have a multiple testing problem wearing an ML costume. Module 5.1 has the maths.

The defence is bookkeeping. Write down every configuration you try. Judge the final choice on a hold out period you have never looked at. And discount the result for the number of tries, because the market will.

How this shows up in RupeeCase

Machine learning and RupeeCase

Machine learning is used in RupeeCase research to search and optimise the model behind each strategy: which signals rank the universe, how many names to hold and how often to rebalance, tested out of sample. The rule that ships is then frozen and published, so the live strategy is a fixed, readable algorithm, not a black box that changes on its own. Every rebalance is published, and every return figure is labelled model, back tested. Read the methodology on any strategy page, for example Large Midcap, and the rest of the record on the track record page.

Questions readers ask

Can AI predict the stock market?

Not the way headlines suggest. Models can find small, persistent tilts across many stocks, which is valuable at scale. No model reliably predicts the direction of a single stock or the index tomorrow, and anyone showing very high accuracy on Indian returns is almost certainly showing an over fitted back test.

Is a machine learning fund better than a factor fund?

Not by label. Judge any strategy on its rules, its out of sample evidence, its costs and its drawdowns. ML is a research method; it does not make a strategy safer or better by itself.

Why freeze a model instead of letting it learn every day?

A model that refits itself constantly is hard to audit and easy to over fit to recent noise. A frozen rule behaves the same way in the live market as in its tests, and every decision it makes can be checked against the published rule.