Machine Learning for Stock Prediction: The Python Guide

Predicting stock prices with machine learning is the most overhyped and most attempted project in retail quant. The truth: a model can genuinely learn useful relationships in financial data, but the phrase "predict the price" misleads - what works is predicting probabilities, regime, and volatility while controlling costs and discipline. This guide builds a realistic Python workflow for stock prediction: data, features, models, evaluation, and the brutal reality check.

Reframe the Goal: Not "Will It Go Up?" but "Is It Likely?"

Markets are near-efficient; no model reliably knows tomorrow's exact price. What ML can do is estimate probabilities - the chance a stock rises over a horizon, the chance volatility stays low, or the chance a regime persists. Model outputs should therefore be probabilities, ranked against a baseline (like "always predict the majority class"), and validated on future data the model never saw. Your prediction "edge" is the small probability boot that survives honest out-of-sample testing.

The Data Pipeline

import yfinance as yf
import pandas as pd, numpy as np
df = yf.download("NIFTY.BO", start="2015-01-01", end="2026-01-01")
df = df[["Open","High","Low","Close","Volume"]].copy()
df["ret"] = df["Close"].pct_change()

For Indian markets, alternatives include broker NSE data files, and paid vendors for clean point-in-time data. Whatever the source: adjust for splits/dividends, keep point-in-time histories, and never let the future leak into training features.

Feature Engineering

Start with families: momentum (returns over 5/10/20/60 days), volatility (rolling std, ATR, EWMA), volume (ratios), calendar (weekday, month, days-to-expiry), and cross-asset (index returns, VIX). Add market-structure features like distance to moving averages and OI concentrations if you have them. Then select by validation importance - XGBoost feature_importance on a TimeSeriesSplit gives honest ranking.

Label Design (the Part Everyone Skips)

df["fwd"] = df["Close"].shift(-5)/df["Close"] - 1
df["label"] = (df["fwd"] > 0).astype(int)  # or risk-adjusted variants

A binary up/down label over 5 days is noisy; risk-adjusted labels (forward return divided by expected volatility) are more robust. In any case the horizon must match how you will actually trade, and there must be no overlap leaks between consecutive windows during validation.

Models Cheat-Sheet

  • Logistic Regression / linear: transparent baseline; quick sanity reference
  • Random Forest: robust, handles nonlinearity; can't extrapolate trends
  • XGBoost/LightGBM: current standard for tabular finance - strongest default
  • LSTM/Transformers: only when you have dense sequential data (ticks, intraday); don't flex them on 2,000 daily rows
  • Ensembles of the above: averaging honest models usually beats picking the best single one

Validation: The Only Number That Matters

from sklearn.model_selection import TimeSeriesSplit
tscv = TimeSeriesSplit(5, gap=10)
for tr, te in tscv.split(X):
    # train on tr, evaluate on te — NEVER shuffle time
    m.fit(X_train, y_train); print(metrics on test slices)

Never shuffle rows; walk-forward is the only honest setup for finance. Report AUC, and - for trading - cost-adjusted P&L, not raw accuracy. If the model's edge dissolves after costs, it has no tradable signal.

The Reality Check: Baseline, Costs, Drift

  1. Beat a naive baseline (buy-and-hold or constant-mean) after costs - if not, you have correlation, not edge
  2. Model slippage/brokerage/STT on every trade; a "profitable" backtest that doesn't clear 15-25 bps per side is noise
  3. Watch feature drift: distributions change; retrain on rolling windows and pause alerts when live features leave their training range
  4. Paper-trade the identical code path 60 days before risking capital

Why Most "97% Accurate" Stock Models Are Fake

Unscrupulous tutorials show 97% accuracy by predicting yesterday (look-ahead), symmetrically duplicating rows, or shuffling time. Any of those instantly contaminates. If a model claims near-perfect accuracy on financial data, it has leaked - it is an advertisement, not a result.

Bottom Line

Stock prediction with ML is a disciplined probability-learner: clean point-in-time data, honest features, well-formed labels, walk-forward validation including costs, and live drift monitoring. Built that way it can add a modest, real, compounding edge. Caught in hype, it burns time and capital. Build the honest pipeline, and let the market's noise be the boss you respect.

SEBI Disclaimer

Algorithmic trading involves substantial risk. This article is educational and is not investment advice; backtests do not guarantee future results.

A 200-Line Baseline Script You Should Run First

Before any sophistication, run the honest baseline: a script that pulls the daily series, builds a small feature set, trains a plain tree, and reports the walk-forward hit rate with costs. The baseline will land near the coin-flip line, and that landing is the project's most useful output - it sets the bar every louder model must beat and kills the fantasy that "adding a better algorithm" rescues the marginal edge. The 200 lines, committed to version control, become the yardstick every subsequent experiment is measured against. If the baseline already beats costs cleanly, the project is done sooner than the curriculum claimed; if it does not, the extra models earn the burden of proof.

Two Feature Sets: Close-Based and Tick-Conscious

Design the research across two practical worlds. The daily-close feature set - returns, volatility, moving-average distances, open-interest changes - is cheap, well-understood, and perfectly matched to a daily strategy's holds. The tick-conscious set adds intraday granularity once a live feed exists: the opening gap, the first hour's range relative to the day, the volume-weighted average price location. The two sets answer different questions - regime and direction on the daily, timing and momentum on the intraday - and the strategy that merges them conflates the two clocks exactly where the confusion begins. Keep the two sets separate, validate each on its own cadence, and touch the merge only when both carry independent evidence.

The Probability-to-Position Map

Convert the model's calibrated probability into size with a written table: below the no-trade threshold, nothing; the neutral band, quarter size; the confident band, full size subject to the account's overall risk cap. The table turns the model into a risk engine rather than an oracle - the position grows and shrinks with the model's conviction, and the account's survival stops depending on any single call. Write the thresholds from the calibration curve and re-derive them at each retrain, because a table that outlives its calibration is a sizing system quoting yesterday's confidence.

The Brokerage Reality Test

Project the strategy's predicted trade count per month and run it against the actual fee schedule: the brokerage per execution, the STT on the sells, the exchange charges on the turnover, and the spread cost at the size the strategy trades. A rule producing forty-one trades a month needs the arithmetic to close the loop on its own edge - the cost column is not an estimate to add later, it is the first column to fill. The strategies that survive the vetting are the ones whose equity curve stays positive after the full stack is paid; the ones that fail vanish from the chart entirely, which is the most valuable absence in research.

Recording Your Assumptions: The README of Reproducibility

Open the project with a README that declares every assumption: the data source, the surviving-universe bias or its absence, the label horizon, the cost model, the split calendar, and the version of every library. A year later, that README is the only honest sibling of the notebook's results - it explains why a production figure differs from the chart, and it is the difference between a research artifact and a decision document. The README that is painful to write at the start is the pagination that saves the audit six months later.

  1. Commit the 200-line baseline and let it set the burden of proof.
  2. Keep the close-based and tick-conscious features in separate clocks.
  3. Write the probability-to-position map from the calibration curve.
  4. Run the brokerage reality test before any size.
  5. Open the project with a README declaring every assumption.

Python's ecosystem is the guide's backbone: pandas and NumPy build the feature table, scikit-learn and XGBoost train the models, and backtesting.lib or backtrader wraps the walk-forward evaluation. The discipline that keeps the guide honest is the feature-leakage audit - every feature computable at the bar it predicts - and the walk-forward plot that separates a model that saw the future from one that did not.