Three days building an AI decision engine that couldn't beat a simple baseline

A working integration proves very little

The question was narrow. A shopping app has an eligible offer and a banner slot on the current screen. Should it show the offer now, or do nothing?

I wanted to know whether an AI decision layer could answer that better than the simplest possible policy. The model was Jev, TypeSafe's choice model, which returns a choice with probabilities.

A valid answer is easy to mistake for progress. The request succeeds, the JSON parses, and the decisions can still be no better than a constant.

Integration correctness and decision quality are different questions, and only the second one justifies the integration.

Most of what I built exists to answer the second question.

What I built

The repository has four packages:

• engine: the decision contract, validation, a rules baseline and a policy gate that every engine's output passes.
• jev: the adapter. Neither the engine core nor the simulator knows Jev exists.
• simulator: a seeded scenario generator, a synthetic ground truth and an evaluator. It never imports the adapter, and its labelling code never imports the rules baseline.
• runner: runs every engine on identical scenarios and writes one result file.

Two references, not one

rules-v1 is the kind of rule a team writes first: if promotions are on, the screen has a slot, the cart is not empty and an offer is eligible, show the top-priority offer.

always-show-v1 is dumber: it shows the offer whenever the hard checks allow. Any engine that claims to judge timing has to beat it.

The adapter keeps Jev's job small

If promotions are off, no offer is eligible, or the screen has no slot, the adapter returns no_action without calling Jev. Otherwise it sends eight session and shopping fields and one choice question. Jev never sees an offer ID: a show_offer answer always targets the app's top-priority offer, the same one the rules pick. Only the show-or-don't-show judgement is being compared.

Every failure becomes a typed error, is never retried, and counts as no_action. A Jev failure can never show a promotion.

Guards around paid calls

Any run that includes Jev needs an explicit --live flag, and the held-out split also needs --held-out. Both are checked before credentials are read.

The --live guard came from a mistake. While I was testing the missing-key path by hand, a run read the key from .env and made five real calls. I didn't use that output, and added the guard.

Labels from a designed model, not from users

There was no real traffic, so the scenarios are synthetic. A seeded generator draws four hidden traits per shopper, such as shopping intent and interruption sensitivity, and those shape the observable context an engine receives.

Hard-constraint scenarios are labelled no_action. The rest get show_offer when a hand-weighted utility over the hidden traits and a few observable fields, plus noise, is positive.

The labels measure agreement with a utility model I designed. They are not user behaviour, conversion or revenue.

Hard constraints decide about 40% of the full development split, which every engine gets right for free, so comparisons use the feasible subset, where a judgement is needed. Of the first 200 development scenarios, 131 are feasible: 87 labelled show_offer and 44 no_action.

Showing every offer therefore scores 66.4% without a single judgement, so I also report specificity (how many no_action cases an engine correctly declines) and the AUC of Jev's p(show_offer) (whether its probabilities rank the cases).

Stage A: the first prompt showed everything

The first prompt, offer-v1, was written before any live run:

{
  "type": "choice",
  "instructions": "A shopper is using a shopping app. One eligible promotional offer could be shown in a banner on the current screen right now. Should the app show it?",
  "criteria": {
    "show_offer": "Show the offer now: it is likely to be relevant and welcome at this moment.",
    "no_action": "Do not show it now: it is likely to be irrelevant, unwelcome, or an interruption."
  }
}

TypeSafe documents that this model version leans towards the first-listed option, so Stage A ran offer-v1 in both orders on the same 200 scenarios.

With show_offer first, it showed the offer on all 131 feasible scenarios, the same decision as always-show every time. With no_action first, it declined twice, both on scenarios labelled show_offer. Specificity was 0% in both orders.

The AUC of p(show_offer) was 0.442 and 0.472, slightly below chance: the mean probability was a little higher on no_action labels than on show_offer labels.

The two orders agreed on 129 of 131 decisions, so position was not the explanation. The wording was the likelier suspect: it says the offer "could be shown", then asks "Should the app show it?"

Stage B: a less leading prompt

offer-v2 changed only the question text. It says neither answer is the default, names the trade-off between a helpful banner and an interruption, describes each field neutrally, and makes the two criteria mirror each other. It has no numbers or directional hints, which would only teach it my synthetic weights.

Stage B was a new run of six engines on the same 200 scenarios, interleaved per scenario: both references, plus both prompts in both orders. That made 524 Jev calls, with no errors.

offer-v1 showed the offer on all 131 feasible scenarios in both orders. offer-v2 showed it on 130 of 131: it declined once in each order, on a scenario labelled show_offer. Neither prompt correctly declined anything, and offer-v2 finished one scenario behind always-show.

The rules baseline came last at 51.9%. It does decline offers, just not the right ones.

The offer-v2 AUC rose to about 0.57, which looks like progress. With only 44 negatives, the bootstrap 95% intervals are 0.474–0.668 and 0.467–0.676. Both include 0.5, so on this sample I can't distinguish that ranking from chance.

Jev also isn't deterministic: offer-v1 with no_action first declined twice in Stage A and never in Stage B, on the same scenarios. At this sample size, a couple of flipped answers are as large as the differences being measured.

Ranking is not a decision

Could a different cut-off on p(show_offer) rescue offer-v2? Thresholds from 0.50 to 0.85 on the same 131 scenarios traded recall for specificity, but none beat always-show on accuracy. I selected none, because the best row would only fit these scenarios.

The probabilities explain why the default cut-off changes nothing. In Stage B, every offer-v2 probability fell between 0.46 and 0.92, and the average offer-v1 p(show_offer) was 0.86 while only 66% of labels were show_offer. Whatever those numbers mean, they aren't calibrated probabilities of agreeing with this target, and AUC only measures ranking.

The signal was there

An always-yes model leaves two explanations: its fields carry no signal, or it isn't using the signal. To tell them apart, I ran an offline analysis with no API calls.

I built 19 features from the same eight fields Jev receives and cross-validated a regularised logistic regression on a separate, larger dataset: the 602 feasible scenarios of the 1,000-scenario development split. It used stratified 5-fold cross-validation, repeated 10 times, tuning only inside training folds.

Out of fold, it reached 73.3% accuracy against 66.3% for always-show on the same rows, with 44.1% specificity and an AUC of 0.773. It beat always-show in 48 of 50 folds and tied in the other two. The current screen carried most of the signal.

That is a ceiling estimate, not a competitor: cross-validation on development data, not a held-out test, against labels from my own utility model. It shows the signal exists in this synthetic design, not that it generalises or that Jev could use it. It only rules out "the information wasn't there" as the whole explanation.

What I learned

• Integration correctness and decision quality are separate results. Hundreds of error-free calls said nothing about the decisions.
• Accuracy needs a baseline next to it. Here, 66.4% is what you get for not deciding. Specificity exposed that at once.
• A ranking metric is not a decision rule. An AUC near 0.57, with probabilities bunched above 0.5, still produces always-show at the default cut-off.
• Prompt changes need a measured comparison. offer-v2 reads better and moved the AUC, but the decisions barely changed.
• The complex path has to pay for itself. A Stage B Jev call took roughly a third of a second at the median, and the run cost an estimated $0.012. Cheap, but always-show matched it without a network call.

What is still open

Everything above is development data, and the prompts were inspected on these same scenarios, so even small differences are optimistic.

The held-out evaluation has not been run. I wrote a plan for it before generating any held-out scenario, and it is not signed off. What exists is preparation: prompts pinned by hash, the logistic model frozen as a diagnostic reference, and a driver script, tested only on development data, that would run the evaluation once and show error counts before any metric. The bar is fixed in advance, including at least two accuracy points over always-show and a significant paired test.

The plan also states its expectation: given the development results, offer-v2 is unlikely to clear that bar. If it is ever run, it would give an unbiased estimate on unseen scenarios, still from the same synthetic design.

No real user has seen a decision from this system, and nothing here measures a conversion.

Where it ended up

The integration works: validated input, no offer IDs sent to the model, fail-closed errors, guarded paid runs.

The two prompts I tested, in both option orders, did not make better decisions than always showing the offer. That is a result about these prompts, this model version and this synthetic target, not about AI decision-making in general.

Without the always-show reference, 66.4% accuracy, zero errors and a rising AUC would have been easy to read as a working AI feature. The baseline is what kept the result honest.