The Harsh Truth About AI and Markets
Every year, thousands of data scientists try to build AI models that predict stock prices. Most fail. Some succeed temporarily. Very few sustain profits. Here is why AI struggles with financial markets - and why the few that work are different.
What is Tabular Data?
Tabular data is structured data in rows and columns - like spreadsheets. Each row is an observation (a day, a trade, a stock). Each column is a feature (price, volume, RSI, MACD). Financial data is tabular: columns of price, volume, indicators; rows of daily observations.
XGBoost, LightGBM, and CatBoost are designed for tabular data. They capture non-linear relationships between features and targets. This is why they work well for credit scoring, fraud detection, and - sometimes - trading.
Why Financial Data is Different
1. Non-Stationarity
Financial data distributions change over time. A model trained on 2020 data may fail in 2024 because market dynamics shifted. This is called concept drift. In medical imaging, a cat is always a cat. In markets, the rules change constantly.
2. Low Signal-to-Noise Ratio
Financial data is mostly noise. The signal (predictable pattern) is tiny. In image recognition, the signal is clear - a cat looks like a cat. In markets, the signal is buried in random price movements.
3. Survivorship Bias
Backtests on current stock lists ignore companies that went bankrupt. This makes models look better than reality.
4. Look-Ahead Bias
Using future information in training (even accidentally) creates inflated backtest results. This is the most common mistake in financial ML.
5. Regime Changes
Markets go through different regimes: bull, bear, sideways, high volatility, low volatility. A model trained in one regime often fails in another.
Why XGBoost Sometimes Works
XGBoost works for trading because:
- Feature interactions: It captures non-linear relationships between features
- Regularization: It prevents overfitting better than deep learning
- Feature importance: It tells you which features matter
- Speed: Fast training and prediction
But it only works when:
- You have genuine alpha (predictive edge) in your features
- You validate properly with time-series cross-validation
- You retrain regularly to adapt to regime changes
- You keep transaction costs realistic
The Kaggle Evidence
From Optiver and Jane Street competitions:
- 1st place solutions achieve MAE improvements of only 0.2% over baselines
- Feature engineering contributes 80% of performance; model choice contributes 20%
- Ensembles of different models beat any single model
- Online learning (retraining) is essential for adapting to market changes
Why Most AI Trading Fails
- Overfitting: Model memorizes noise instead of signal
- Transaction costs: Erase small edges
- Slippage: Execution price differs from expected
- Regime change: Model fails when market dynamics shift
- Survivorship bias: Backtest looks better than reality
What Actually Works
- Simple models: XGBoost with 50-100 features beats complex deep learning
- Feature engineering: 80% of the edge comes from features, not models
- Time-series validation: Never use random splits for financial data
- Regular retraining: Adapt to changing market conditions
- Risk management: Position sizing and stop losses matter more than model accuracy
SEBI Disclaimer
This article is for educational purposes only. Algorithmic trading involves substantial risk of loss.
The Sequence-to-Sequence Illusion of Pattern Recency
Machine learning excels when the past repeats; markets execute the opposite covenant. The model's core assumption - that the distribution generating tomorrow's rows resembles today's - fails as soon as regime rotates, and the failure is compounded by the recency illusion: a model trained on the last two years of low-volatility drift will confidently mark the first genuinely turbulent quarter as noise and trade straight through it. This is not an engineering bug; it is the mathematical price of assuming the market has a stable generative process. The skill is not eliminating the assumption but bounding it: never let the model extrapolate beyond the distribution it saw, and treat every regime boundary as a full stop for the model's authority.
Information Decay: Why Old Edges Fade
The recurring engine of decay is adaptation: when a signal becomes measurable and traded, participation compresses the edge until it no longer covers costs. Momentum features that paid in 2015 were alkalised by 2020 as crowding piled in; bid-ask imbalances that read clearly become two-sided as everyone scripts them. The honest forecast is a decay curve, not a plateau: assume each detected signal erodes at some half-life, retest it on rolling windows, and retire the signal when its marginal contribution stops covering the fee stack. The market is a machine for pricing out discovered edges, and every model is a participant in that machine.
The Cost Asymmetry That Kills 55 Percent Accuracy
A system that triggers on a 55 percent probability pays 100 percent of the costs whenever the trade is placed, wins or loses. With STT, exchange charges, brokerage, and spread on an option trade routinely consuming a meaningful slice of per-unit premium, a 55 percent caller must gross a multiple of the cost stack just to break even - and most such models gross less. The arithmetic explains the graveyard of "52 percent accurate" paper systems: accuracy is per-trade, profitability is per-rupee-after-cost, and the two diverge exactly where the retail trader refuses to look. Report strategies in net-after-cost PnL, and watch how mysteriously the "edge" evaporates.
What Hedge Funds Actually Do With ML
Institutions run machine learning not to predict "what does price say next", but within three narrower, better-posed jobs: pricing the risk surface (volatility, correlation), executing with minimised market impact, and ranking assets for portfolio construction. Each of those problems has clean labels, stationary-enough features, and a natural cost term - precisely the properties retail price prediction lacks. The lesson is not that ML is useless but that its edge concentrates where the problem is well-posed; price-direction guessing from tabular EOD data is the least well-posed of all financial problems, and the funds treat it accordingly.
A Healthy Expectation Curve for Your Project
Set the expectation honestly before the project: a modest, cost-sanitised edge that holds out-of-time across regimes is the realistic prize; a breathtakingly accurate price forecaster is the marketing prize sold to the naive. Design the pipeline for the modest edge - clear labels, honest validation, a cost model that bites, drawdown governance - and let the small real positive expectancy compound into the actual return. Projects that chase the fantasy either never ship or ship a curve that evaporates at the first regime. The truth is the product; the fantasy is the fee.
- Bound the model to its training distribution; halt at regime lines.
- Retire signals on a decay curve, never a faith statement.
- Report net-after-cost PnL, and let accuracy sit in the appendix.
- Prove the edge in well-posed problems before price forecasting.
- Trade the honest expectancy; refuse the marketing curve.
The truth about tabular market data is financial, not technical: the same rows that feed a retail forecast are the rows the institutions already trade, and the edge the learner finds in a backtest is often the edge the fees and spreads remove before the fill. AI is a powerful feature extractor, but the rupee reality of execution - spread, slippage, STT - turns many an academic accuracy figure into a calendar from which the trader learns more.
