Fidelity Benchmark

Synthetic personas that stay close to reality.

Blind out-of-time replay across three public datasets. Real outcomes hashed before we predicted. Every number below reproducible from the public audit repo.

Churn Purchase-quantity Vision / shelf — in progress

~2×

Closer to reality than the naive baseline (Wasserstein 11.5 vs 23.7)

6 / 6

Purchase-quantity segments won on distributional distance

6 / 6

Driver directions correct — no sign-flip on churn

100%

Outcomes SHA-256 sealed before any persona was built

Why this matters

Distributional fidelity is the test the field has been failing. Ours holds up.

The recent literature is blunt: synthetic personas can recover direction but collapse the distribution. On a real shopping survey no LLM beat a naive baseline that just predicts the average — Wasserstein 0.641 vs 1.467 for the best model, more than 2× worse.

Independent peer-reviewed work finds the same failure mode from a different angle: too little variance, coefficients that drift, direction unstable over time.

The fix is elicitation and grounding — free-text-then-map recovers realistic distributions at ~90% of human test-retest, KS > 0.85. That's the same technique we run under the hood, and this page is the receipt.

How we ran it

Four integrity constraints. One seal per test.

01 · Blind replay Persona from past only Built from a customer's data up to a cutoff. Outcome window came after. No leakage.

02 · Sealed outcome SHA-256 before prediction Real outcomes hashed into a locked file before any persona existed. Verification is a hash check.

03 · Saved per-row One prediction per customer Every persona's prediction written to CSV before the outcomes were opened. The audit surface.

04 · Shipped as-is No threshold tuning Weak numbers printed weak. Recalibration disclosed separately, not swapped in silently.

Seal fingerprints · 16-char prefixes

7f43d1ae7a13fc4c… e8fa6f1ddb4ac8a5… 87ad70df131311a8… fff689b7cf6147ba…

Tier A · Churn

Binary classification across three public datasets.

Can a persona built from a customer's past predict whether they'll churn? Direction is right in every dataset. Calibration and raw ranking vary — we print both.

Dataset	n	Churn	AUC · Us / ML	Beats naive baseline?	Calibration	Driver direction
Grocery (Dunnhumby)	300	18%	0.70 / 0.74	✓ Brier 0.143 vs 0.148	good	✓
Telecom (IBM Telco)	300	27%	0.75 / 0.85	✓ 0.190 vs 0.196	moderate	✓
Subscription (KKBox)	500	6.8%	0.59 / 0.84	✗ raw → ✓ recalibrated (0.281 → 0.061)	poor raw	✓

The reads: driver direction correct 6 out of 6 — directly counters the variance-collapse critique. Predictions realistically spread. Doesn't out-rank a purpose-built ML ranker — that's fine, the edge is the why and the recalibration path, not the last basis point.

Churn is binary → KS / Wasserstein don't apply here. We don't fake one.

Tier B · Purchase quantity

Distributional distance — the exact test the field's baseline crushed everyone on.

~2×closer

Mean Wasserstein-1: 11.5 (Simulatte vs real) vs 23.7 (naive baseline vs real). Wins in all 6 segments. This is the test Moon et al.'s naive baseline said "nothing clears."

Simulatte · synthetic Naive baseline Real (target = 0)

Frequent · Heavy

13.4

Simulatte

39.1

Baseline

Frequent · Light

19.5

Simulatte

21.7

Baseline

Occasional · Heavy

8.3

Simulatte

23.9

Baseline

Occasional · Light

6.0

Simulatte

29.5

Baseline

Regular · Heavy

10.2

Simulatte

14.8

Baseline

Regular · Light

11.8

Simulatte

13.4

Baseline

Segments = frequency band (occasional / regular / frequent) × spend band (light / heavy). Decisive on Wasserstein · marginal on KS (0.27 vs 0.28) — we show both.

Known limits.

We print what we haven't fixed

Rare-event churn Raw predictions run over-confident on very low-rate churn. Standard recalibration closes the gap — we disclose raw and recalibrated side by side.

Ranker parity Purpose-built ML rankers still edge us on raw AUC. Our value sits in the why — driver direction, persona narrative, and the recalibration path.

Scale · vision/shelf Ran at 300–500 per dataset; larger n is on the roadmap. Shelf and creative fidelity: not yet run, no claim.

Verify

Every figure recomputes from the sealed outcomes and saved per-row files.

Verification is self-serve. Clone the audit repo, run the reproduce commands in the README, hash-check against the seal files. No gatekeeping.

Audit repo · clone to reproduce

https://github.com/Iqbalahmed7/simulatte-fidelity-audit

Contains: predictions/*.csv · predictions/backtest_dist_dunnhumby.json · seals/*.json · dist_metrics.py · README.md

Book a call Request access