Why plain k-fold silently overfits your trading model — and the 4-line fix that stops it.
The Problem With k-Fold in Time Series
Financial data is sequential. k-fold shuffles rows, so a training row from 2 PM Tuesday sits
next to a test row from 10 AM Monday. Worse: **triple-barrier labels overlap**. A label at
bar *t* looks 6 bars into the future; a training row at *t+2* "knows" part of that future.
The model leaks.
V1's history is full of "HIGH overfit" verdicts — train AUC high, test AUC flat. Plain
`TimeSeriesSplit` is only marginally better; it still lets adjacent windows bleed into each
other.
Purged + Embargoed CV
For each test window `[t0, t1]`:
1. **Purge** any train row whose label window overlaps the test window.
2. **Embargo** `max_training_horizon` bars after the test window — drop those too.
Overlapping labels are not i.i.d. Purging + embargoing makes the split honest.
def purged_embargo_split(n, n_splits=5, embargo_frac=0.02):
idx = np.arange(n)
fold = np.array_split(idx, n_splits)
splits = []
for i in range(n_splits):
test = fold[i]
emb = int(len(test) * embargo_frac)
lo, hi = max(0, test[0]-emb), min(n, test[-1]+emb+1)
train_mask = np.ones(n, bool); train_mask[lo:hi] = False
splits.append((idx[train_mask], test))
return splits
Tune Only When You Have Enough
Optuna once "won" a validation set with only **4 decisive rows** — statistically meaningless.
Rule: never tune when the decisive (non-abstained) validation rows are below ~30–50. Widen the
date range or symbol basket first; don't trust the trial.
Three-Way Split, Always
train (fit) → validation (early stop + HP select) → **disjoint calibration set** (sigmoid/
isotonic) → test (untouched, final score only). V1 sometimes conflated validation and
calibration. Keep them separate.
The Promotion Gate
Log every trial's train/val/test gap, not just the winner's test score. Promote only if
replay AND shadow (≥1 live session) both beat baseline on **buyer metrics**: 1.5x/2.0x hit
rate, MAE-before-hit, time-to-hit, wrong-side rate.
*Research only. Not investment advice.*