ML demand forecasting in production

Context

A grocery chain: 12+ stores, tens of thousands of active items, one replenishment decision per combination every week. That decision came from the ERP's forecasting module, a black box that produced a number but never a reason.

Two constraints shaped the replacement. No cloud budget and no GPU, so it runs on a shared Windows server. No data team, just me. The second one matters more: it rules out any design whose failure mode is "someone notices". Whatever I built had to fail loudly on its own.

Architecture

Medallion layout on Delta Lake, local filesystem. Bronze is raw and append-only, so a bad transform is always recoverable. Silver holds cleaned series with calendar and promotional features. Gold holds training frames. The boundaries exist because that is where the assertions go.

A tournament, not a model. Series in a grocery catalogue do not behave alike. K-Means segments them, LightGBM handles the bulk per cluster, and intermittent demand goes to Croston and SBA. A tree model on a mostly-zero series learns to predict zero: technically right, operationally useless.

The switching rule is code. A champion/challenger setup where challengers get promoted because they "looked better" becomes the engineer's taste within two quarters. The criterion is defined in the pipeline, measured on held-out horizons, applied by the job. MLflow keeps the evidence.

The hard part

Months in, I audited the pipeline. Nothing was on fire. That was the point: "everything reports success" is exactly the state where a silent failure hides.

The audit measured usable horizon, how many weeks ahead the system could still forecast from the data it actually had. That number had been shrinking for weeks.

The cause was structural. The daily ingest had stopped running. The weekly forecast declared a dependency on it but did not enforce one, so Dagster happily materialised a forecast from whatever was already in the lake. The job ran, succeeded, and produced a forecast from data that got staler every week.

Every green checkmark was accurate. The system was still wrong.

The immediate fix made the dependency real. The durable fix was the audit itself: horizon health is now measured independently, never inferred from job status. A pipeline's own success reporting cannot be the evidence that it works. The failure modes that survive longest are precisely the ones the reporting does not cover.

Trade-offs

Reliability before sophistication. Hierarchical reconciliation is not implemented. It is a real gap and the wrong next investment while the base pipeline's failure modes are not fully instrumented.

Local filesystem over cloud storage. Zero infrastructure cost and no vendor lock-in, against horizontal scale and an easy disaster-recovery story. Right at this volume, and the first decision I would revisit.

Weekly, not daily. Replenishment operates weekly. Matching the model's cadence to the decision's cadence also keeps compute inside the window a shared server can give it.

Results

Double-digit percentage WMAPE reduction against the ERP baseline across several categories, largest in fast-moving lines. Thousands of series forecast weekly across 12+ stores, feeding merchandise reordering directly. Over 350 tests, green before deploy.

Business figures are published as ranges. The company is named, the exact numbers are not mine to publish.