Skip to content
In private rollout: onboarded personally, usually live within 24 hours.

Does an AI shopping assistant actually work? Measure it with a holdout

Attila Szocz, founderJuly 3, 2026

Every AI shopping assistant claims a lift. Most of them count any order the widget so much as brushed against and call it “assisted revenue.” The dashboard looks great and means almost nothing. If you want to know whether an assistant actually works, you have to measure it the way a clinical trial measures a treatment: against a control group. A percentage of your visitors never see the widget, their order rate is the control, and the difference between them is the only number that survives scrutiny.

We have moved the full version of this argument onto one page rather than keeping two copies of it. The holdout method is the canonical explainer: a worked example with the arithmetic shown, what a confidence interval should do to a renewal decision, why we put all five attribution models on screen instead of the flattering one, and a plain list of the questions a holdout cannot answer. It is also where the two live demo stores are linked, so you can watch the product refuse to invent an answer before you take our word for any of it.

The product side of it is on attribution and proof, and the thing worth tying it back to is what you changed: a campaign that leaned Sarah into a promoted collection, a coaching note that changed how she answers. Causal measurement is what turns a metric into a lever.

Your next hire

Ready to put a real associate on your storefront?

Book a 30-minute demo with the founder over Google Meet and watch Sarah work your own catalog. We onboard you personally, usually live within 24 hours. Not ready for a call? Request an invite.

Private rollout · one-click uninstall if it doesn’t earn its keep