Random Forest for Stock Prediction: Complete Python Tutorial

Random Forest for Stock Prediction: A Complete Python Tutorial

Random Forest is the gentle giant of machine learning: an ensemble of decision trees that is easy to understand, hard to overfit, and surprisingly competitive on tabular financial data. This tutorial builds a complete stock-prediction pipeline in Python - data, features, model, validation, evaluation - and explains what random forest can and cannot do for markets.

Why a Forest Instead of a Tree

A single decision tree memorises training data and goes brittle out of sample. A random forest grows hundreds of trees on bootstrapped samples, each considering a random subset of features at every split, then averages their votes. Averaging reduces variance dramatically while barely raising bias. For noisy financial data, that trade-off is precisely the edge.

The Data Pipeline

import yfinance as yf, pandas as pd, numpy as np
df = yf.download("RELIANCE.NS", start="2018-01-01", end="2026-01-01")
df = df[["Open", "High", "Low", "Close", "Volume"]]
df["ret"] = df["Close"].pct_change()

Feature Engineering for Markets

Random forests need features that encode market structure. Classic, leakage-free families:

• Momentum: 5/10/20-day returns, moving-average ratios
• Volatility: 20-day realised vol, ATR, EWMA vol
• Volume: volume ratio to its 20-day mean, OBV slope
• Calendar: day-of-week, month, days-to-quarter-end
• Cross-asset: index return, sector return, VIX level

df["ret5"] = df["Close"].pct_change(5)
df["vol20"] = df["ret"].rolling(20).std()*np.sqrt(252)
df["vr"] = df["Volume"]/df["Volume"].rolling(20).mean()

Label Design Matters More Than the Model

Binary up/down labels are noisy; risk-adjusted forward returns are better. A robust choice: sign of the 5-day forward return adjusted by realised vol:

df["fwd5"] = df["Close"].shift(-5)/df["Close"]-1
df["label"] = (df["fwd5"]/df["vol20"]).gt(0).astype(int)

Training a Random Forest

from sklearn.ensemble import RandomForestClassifier
X = df.dropna().drop(["fwd5","label"], axis=1)
y = df.dropna()["label"]
model = RandomForestClassifier(n_estimators=400, max_depth=6,
    min_samples_leaf=10, random_state=42)
model.fit(X.iloc[:-200], y.iloc[:-200])

Key parameters: n_estimators 300-1000, moderate depth (4-8) to avoid overfit, min_samples_leaf to smooth predictions, and random_state for reproducibility.

Honest Validation: No Shuffling in Finance

from sklearn.model_selection import TimeSeriesSplit
tscv = TimeSeriesSplit(n_splits=5, gap=10)
for tr, va in tscv.split(X):
    m = RandomForestClassifier(max_depth=6, min_samples_leaf=10)
    m.fit(X.iloc[tr], y.iloc[tr])
    print("acc:", m.score(X.iloc[va], y.iloc[va]), "| AUC:", roc_auc_score(y.iloc[va], m.predict_proba(X.iloc[va])[:,1]))

TimeSeriesSplit trains only on the past and validates on the future - the only honest way to evaluate a financial model. Random shuffling leaks future information and inflates results.

Feature Importance: What the Forest Learned

imp = pd.Series(model.feature_importances_, index=X.columns).sort_values()
imp.tail(12).plot.barh()

Expect volatility and momentum features to dominate. If calendar features dominate while price features contribute nothing, your label or horizon is likely wrong.

What Random Forest Really Gives You

• Interpretability (feature importances, SHAP values)
• Native handling of missing values and categoricals
• Low risk of catastrophic overfitting when parameters are sane
• Fast training on CPU; a weekend on daily data is trivial

Where It Falls Short

Random forests cannot extrapolate trends the way linear models or boosting with linear leaves can, they ignore order unless you engineer time features, and their predictions are step functions - poor for fine-grained volatility targets. For directional classification with moderate data, they remain a disciplined baseline that beats most fancier attempts.

From Backtest to Live: The Hard Part

• Include 15-25 bps of cost per side; a forest's backtest must clear it
• Retrain on a rolling window; markets drift
• Paper-trade the identical code path for 60 days
• Track live feature distribution vs training; pause on drift

SEBI Disclaimer

Algorithmic trading involves substantial risk. This tutorial is educational and is not investment advice. Validate rigorously and paper-trade before risking capital.

Hyperparameters: n_estimators and max_features on Small Data

A forest's two most load-bearing dials on a 2,500-row financial table are the number of trees and the feature-per-split count. More trees smooth the variance without memorising, and the practical plateau arrives around 300 to 500 estimators for daily data - beyond that the curve flattens while the training time grows. max_features controls the variety of each tree's view: the default of the square root of the feature count produces robust, diverse trees, while a value near the full feature set collapses the forest toward a single overfit opinion. Freeze the deeper hyperparameters - depth, min_samples_leaf - at modest values and let the ensemble size do the robustness work, because a forest is a variance machine first.

Out-of-Bag Error: The Free Validation Signal

Each forest tree trains on a bootstrap sample, leaving a third of rows untouched; the oob_score is the model's average correct rate on the rows each tree never saw, computed without any extra split. That makes OOB the cheapest honest validation on the table - no held-out fold needed to see whether the model is attaching the signal. The trap is that OOB is still within-sample: it measures the forest's generalisation across the same distribution, not across time. Use OOB for quick parameter trips and the calendar split for the final verdict, and never mistake the two jobs.

Monotonic Constraints for Financial Common Sense

Some causal directions are known before training: volatility features should not decrease the probability of a large-move label, and a 20-day return, holding all else equal, should not flip sign on the same direction call. Random forest implementations that expose monotonic constraints lock those priors in, making the model behave sensibly in the regions the data did not visit. The constraint buys robustness at the cost of a small fitting ceiling; on financial noise, robustness is the cheaper asset.

An Ensemble Bridge: Forest + Boost

The forest's variance-smoothing and boosting's bias-correction pair better than a single method. The workable bridge trains a random forest for its stable probability and an XGBoost for its sharp edge, then blends them once with a weight found on the validation window - typically leaning on whichever learned the regime first. The blend rarely wins the best single model by a mile, and rarely loses by a mile either; its productivity is the variance reduction across regime shifts. Keep the blend weight fixed after validation and resist re-optimising it every season, because the moment the weight chases the last regime, the blend becomes a laggard.

Churning: When the Forest Flips Its Mind

A forest that nearly balances its votes on every row is a forest announcing uncertainty, yet many strategies trade it anyway. Calculate the churn - the share of rows whose top-two probability gap falls below a threshold - and gate the strategy to trade only the decisive rows. A model that says 52 percent on sixty percent of days and 61 percent on the rest should trade the rest, because the 52-percent zone is exactly where the costs decide the outcome. Churn is not a flaw of the forest; it is the most honest information the forest gives you.

1. Set estimators near 300 to 500 and max_features at the square-root default.
2. Use OOB for quick tuning, the calendar split for the final word.
3. Lock monotone directions that match known market logic.
4. Blend forest and boost once, fixing the weight after validation.
5. Trade only the decisive rows; journal the churn gate.

Feature-to-Tree Budget and the Seed Ensemble

Random forest's practical knob is the split budget: a forest with shallow trees - 8 to 12 levels - and a few hundred trees trains in seconds and resists the noise that deep trees memorise on daily returns, and a sensible total parameter cap keeps the model's true complexity matched to the data volume. The seed ensemble is the reproducibility habit: train the same forest across several random seeds and trade the median prediction, because the variance across seeds is the honest measure of the model's instability and a single lucky seed is a backtest flattering itself. The permutation-based feature importance - measured by shuffling each column and watching the score drop - is more trustworthy than the impurity-based ranking on daily data, and the comparison between them names the redundant features the model tolerates. The forest you can sweep, stabilise, and read is the one that earns its keep.