Feature Engineering Basics
Good features are crucial for ML model performance. In finance, domain knowledge is key.
Technical Indicators
- Moving averages (SMA, EMA)
- Momentum indicators (RSI, MACD)
- Volatility measures (Bollinger Bands, ATR)
- Volume indicators (OBV, VWAP)
Statistical Features
- Returns (log, simple)
- Rolling statistics (mean, std, skew)
- Correlation features
- Principal components
Domain-Specific Features
- Market microstructure
- Order flow
- Options-derived features
- Sentiment scores
Why Features Decide Model Success Before the Model Does
Machine learning in finance rarely fails because the algorithm is weak; it fails because the inputs are poorly designed. Features are the numeric representations of market state that the model learns from, and the difference between a profitable signal and a random one is usually measured in the quality of these transformations rather than the choice between XGBoost and a neural network. In Indian markets, where intraday noise and fat-tailed moves coexist, robust feature engineering separates a strategy that generalises from one that memorises a specific bull run.
The guiding principle is that a feature must carry predictive information independent of the ones already present. Raw price alone offers little, but price relative to a rolling average, volatility-normalised returns, and distance from an option strike all encode information that raw values hide. Each feature should be computed across a full look-ahead-free window to avoid the leakage that silently inflates backtest scores.
Categories of High-Value Financial Features
- Price relatives: close relative to moving average, rate of change, momentum over several horizons.
- Volatility: realised vol over 5, 20 and 60 days, plus the ratio of implied to realised vol.
- Volume: dollar turnover, volume z-scores, volume anomaly versus the 20-day baseline.
- Calendar: day of week, week of month, distance to expiry, quarter end effects.
Transforming Financial Data Correctly
Financial series are non-stationary, so features must be built from returns or ratios rather than absolute prices. A stock that rises from 100 to 200 shares the same percentage momentum as one rising from 500 to 1,000, and a model fed raw prices learns the wrong lesson. Log returns stabilise variance, while rolling z-scores normalise each feature to its own recent distribution, letting the model compare regimes regardless of the absolute level of the index.
Handling Leakage and Look-Ahead Biases
The most dangerous mistake is using future information to build today's feature. A rolling volatility computed with the current close at the edge of a window is safe; a volatility that uses the next day's close is leakage. Shift all target series backward by at least one bar and clip features at each timestamp so the model never sees the answer it is being asked to predict. In option spreads and returns across Indian index constituents, a single leaked feature can lift a backtest Sharpe from 1 to 6 and guarantee failure live.
Categorical and Cross-Sectional Features
Beyond time-series features, cross-sectional ranks tell the model how one stock behaves relative to its peers. Ranking Nifty constituents by momentum percentile each day produces features far more stable than raw momentum, because relative strength survives market-wide moves. Sector membership, expiry week and market regime labels add categorical context that tree-based models exploit naturally. Encode category columns sparingly and never let a high-cardinality feature dominate the gradient through one-hot explosion.
A Practical Feature Pipeline
- Collect raw daily data for Nifty and its constituents for at least five years.
- Compute returns, volatility, momentum and volume features over 5, 20 and 60-day windows.
- Standardise each feature with a rolling mean and standard deviation.
- Rank stocks cross-sectionally to build relative-strength inputs.
- Hold out the most recent 20 percent strictly for validation.
Feature Selection and Monitoring
Not every engineered column adds signal. Use permutation importance and drop features that fail to move the metric while adding noise. Monitor feature distributions over time, because a drift in the volatility regime changes what the model correctly weights. Keep the feature set lean, interpretable and aligned with economic logic, and the model will reward you with robustness when the market shifts from one regime to the next.
Defensive Engineering Habits
Derive features only from data available at signal time, exclude look-ahead transformations, and version every feature column with a date stamp. Keep a minimal set of robust features, because a wide feature set quietly inflates variance when the market regime rotates. Re-fit feature scalers on each rolling window so the model never leaks future statistics into a past decision.
Feature Storage and Versioning Tips
Write each engineered feature set to its own dated artefact so a model can be replayed on the exact data it learned from. Tag features with their transform version because renaming a column silently breaks walk-forward comparisons. Keep an offline feature store even if your live feed is perfect; the ability to reproduce a research result is what separates credible signals from happy accidents.
