Preparing data for backtests: How we have designed DataKitchen

Most backtests fail before the strategy does. They fail in the series.

Unadjusted prices jump on split mornings and look like crashes. Sparse sessions invent silence — or continuity — that never traded. You fill at the close of a coarse bar and call it execution, while the path inside the minute would have wrecked you. One vendor’s feed shows a spike another never saw. Corporate-action files disagree. The Walk-Forward smiles anyway, because the engine believed every bar it was handed.

I have lost count of how many “edges” evaporated the moment I asked a simpler question: was this market behavior, or was it the data?

We pull from three vendors on purpose. None of them is gospel. Cross-checking for anomalies — jumps that only one feed believes, dividend or split mismatches, odd gaps, bars that fail basic OHLC sanity — is not optional hygiene. When sources disagree, that disagreement is a finding. Papering over it is how you ship fiction.

That is the mess we designed around.

AlgoLab is our research stack — strategies, indicators, walk-forwards, Monte Carlo, the lab bench where we ask questions once the numbers are trustworthy.

DataKitchen is the part of AlgoLab that sits in front of that bench. It lands vendor dumps, normalizes corporate actions, resamples into research products, validates them, and publishes stores a backtest can pin. Not glamorous. Just the kitchen between a raw download and anything I am willing to call research.

Here is how we built and run it.

The pipeline

1. Ingest — park the raw files somewhere sane

Everything lands under one shared data home, split by provider. Each download becomes a job folder: the market files plus an authoritative metadata.json. Folder names are for humans. Sampling logic never parses ticker, schema, or date range from the name alone.

I have been bitten by pretty folder names that did not match the query window inside. We stopped trusting names. Metadata wins.

2. Discover and select — decide what this run is about

Before we resample anything, we lock the universe: which jobs, which symbols, which UTC window. Sometimes that is an explicit input path. Sometimes discovery. Sometimes a selection file.

If you skip this and “just resample whatever is on disk,” you will mix histories and wonder why the Walk-Forward will not reproduce. We treat selection as part of the research claim.

3. Adjust — splits and dividends are data, not footnotes

We build adjustment books from vendor corporate-action feeds and anything we have curated ourselves. Normalize to our JSON. Merge across sources when they disagree, and look at the discrepancies — that is often where the anomaly is.

Without this step, momentum and vol fire on corporate actions. You get morning “crashes” that never traded. We pin an adjustment mode on the output (raw, split_adjusted, dividend_adjusted, split_dividend_adjusted) so a result always answers: which prices did you use?

Most of our research sits on split_adjusted. Raw is for when we want to see the scars.

4. Resample — this is the kitchen

Native bars (often fine-grained OHLCV) get aggregated to the intervals we actually trade and study — minutes, hours, days. Factors apply here when the mode asks for them.

Where a feed simply omits zero-volume intraday bars, we gap-fill on purpose: forward-filled OHLC, volume zero. I would rather see an explicit dead minute than let the backtester invent continuity by skipping holes.

Two products leave the kitchen:

I care about that split. Computing signals off closes and pretending those closes were fills is how you invent edge. Indicators stay out of the kitchen on purpose — AlgoLab’s engine computes them later — so the same cleaned store can feed more than one strategy without another resample.

5. Publish — the output folder tree

Cleaned products land under a DataKitchen output root. The path is the claim. Read it left to right:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14

DataKitchenOut/
  {adjustment}/
    {assetClass}/
      {market}/
        {TICKER}/
          signal/ohlcv/{interval}/
            manifest.json
            bars/year=YYYY/month=MM/...
          execution/quotes/{variant}/
            manifest.json
            data/year=YYYY/month=MM/...
          options/                    # only if we have derivatives on this underlying
            signal/ohlcv/{interval}/
            execution/quotes/{variant}/

What each segment means:

Vendor identity is not a folder. It lives in the manifest. Concurrent research jobs pin one adjustment mode and prune what they do not need.

Fixed-point prices, UTC timestamps. If I cannot say which job and which mode produced a bar, I do not trust the bar.

Quotes and slippage — how we actually fill

Signal tells the strategy when. Execution tells the engine at what price. We refuse to fill at the signal close and call it done.

How we source quotes

Proper quote history — real bid/ask tapes — is often expensive, and we do not need it for every name. So we match the fill surface to the instrument:

Highly liquid stocks and ETFs — where OHLCV is dense and trustworthy — we lean on the kitchen’s default synthesized quotes under execution/quotes/synthesized: a bid/ask surface derived from the bar data so every liquid symbol still has something fillable without buying a full quote tape for the whole universe. That is the path behind the 1-minute slippage stance below.

Illiquid names, or stretches with no (or useless) OHLCV — thin books, barely traded symbols, gaps where a bar series simply is not there — synthesized-from-OHLCV is the wrong tool. For those we use richer quote constructions when we have them. CBBO (consolidated best bid/offer) is the main one: normalized bid/ask under execution/quotes/cbbo, including periods that did not trade and would never show up as a clean OHLCV bar. Same row schema as synthesized; AlgoLab just points at a different variant folder. Market-by-order style variants can sit beside CBBO the same way when we need them.

Slippage models are not baked into the kitchen files — the engine applies them on top of whichever quote variant we pinned.

How we account for slippage with 1-minute bars

On the liquid path, we keep 1-minute Signal bars (signal/ohlcv/1m) for a reason. A daily or hourly close hides the path price took inside the bar. The 1-minute high, low, and range are where fills actually live when you are late, aggressive, or just unlucky.

In those backtests we:

1. Take the decision from Signal (prior bar — no peeking into the bar you are filling on).
2. Join the Execution quotes for that time (usually synthesized on liquid names).
3. Stress the fill using what 1-minute bars show about intra-bar movement — the range inside that minute is the honest envelope for “I got filled worse than mid.” We also run explicit slippage scenarios (tight / base / stress, in basis points per side) so a result has to survive more than one cost assumption.
4. Charge commissions the same way across studies so comparisons stay fair.

On the illiquid / CBBO path, the quote tape is the fill surface — including untraded moments OHLCV never recorded. We still do not fill at a fantasy close; we just stop pretending a sparse bar series was a liquid book.

If a strategy only “works” when you fill at the close of a coarse bar and ignore the 1-minute path (or ignore a thin book entirely), it does not work. The kitchen gives us both surfaces so AlgoLab can be strict about that.

6. Validate — fail here, not in the equity curve

summarize for a sanity pass. validate for the mean parts: manifest vs rows, bad OHLC, unordered bars, intra-session gaps. Overnight is not a “missing bar” in our book — sessions have edges. We lean hard on the 1m (and 5m / 1h) series here: that is where silent holes show up.

This is also where vendor cross-checks bite: a series that only one of the three feeds believes, or a corporate-action event the others never saw, gets flagged before it becomes “alpha.”

If validate complains, we fix the kitchen run. We do not “nudge” the strategy until the store is quiet.

7. Backtest — only after the kitchen is done

AlgoLab loads a pinned mode root, computes indicators in-process, joins quotes for fills (with the 1-minute-aware slippage stance above), then full sample, walk-forward, Monte Carlo, optimize — whatever the question is.

DataKitchen’s job ends when the store is trustworthy. The lab’s job starts there. Mixing those jobs is how cleaning decisions hide inside “alpha.”

What I actually get from cleaning

1. Prices that do not jump on splits unless I asked for raw.
2. Decisions and fills on different surfaces — with 1-minute bars keeping slippage honest.
3. Gaps that are policy, not accidents.
4. Three-vendor cross-checks so one bad feed does not own the narrative.
5. A folder tree that states the claim: mode, asset class, market, ticker, signal vs execution.
6. A paper trail — metadata.json and manifests — so a pretty result can still be challenged.
7. An honest label on the adjustment mode, because raw and split_adjusted are different research claims.

If a backtest looks too good and we skipped the kitchen, my first guess is not genius. It is data.