1,202 Milliseconds, Zero Network

Five days after Jev answered the trolley problem over the network in 271 milliseconds, I sent the identical state and the identical eleven questions to an open-weight model on my M4 MacBook Air, with nothing sent to a server. Warmup excluded, the honest number was 1,202 milliseconds to get every answer back. The network won this round, by four times.

A different model, same category #

Laya-CoreML ports Laya, an open-weight model from Convai Innovations, to Apple’s Core ML and its Neural Engine. It is a separate model from Jev, with its own training, that happens to answer the same kind of question: a state plus typed questions, back as a choice, a score, or a noul probability, with zero generated tokens. Laya’s own maintainers benchmark it against Jev and publish accuracy and calibration numbers for the comparison. Apache-2.0, free to download, free to run.

The setup #

Laya-CoreML needs Python 3.11 through 3.13; it does not yet support 3.14. I tested two checkpoints: laya-typed-decisions-coreml, a 421-million-parameter model upstream describes as its domain-specialized checkpoint, and the smaller 322-million-parameter laya-multilingual-coreml. Both load from a local directory with local_files_only=True. After the one-time download, no network call happens at inference time.

Laya-CoreML also ships a third bundle for the Neural Engine alone: laya-multilingual-coreml-ane, capped at 96 tokens for the question, its options, and the state combined. The real trolley state alone tokenizes to 258 tokens against that model’s tokenizer. It cannot run there. For the Neural Engine sidebar below, I compressed the scenario to its bare arithmetic: five people on one track, one on the other, a lever. That is a different, smaller prompt than the rest of this comparison.

The numbers #

Jev’s 271 milliseconds covers eleven questions in a single hosted request. Laya’s exports run one question per call, so I timed the same eleven questions as one sequential local pass against the same state, hash-verified byte-identical to what Jev answered:

Per question, typed-decisions averaged 102-108 ms and multilingual 42 ms. Both figures track Laya-CoreML’s own published M3 Max figures of 13.70 ms and 11.28 ms P50 for the same two checkpoints on the same compute path, just several times slower. The M3 Max carries a 40-core GPU. The Air I’m writing this on carries ten, a quarter as many. Core ML’s cpu_gpu plan reflects that gap, and then some: a compute-unit sweep on the action question alone showed no benefit from asking for the Neural Engine or CPU instead. 105 ms on cpu_gpu, against 758-775 ms on every other setting. That is the same lopsided ratio upstream measured on the Max.

The dedicated Neural Engine bundle tells a different story. With the scenario compressed to fit its 96-token budget, one decision came back in 5.36 milliseconds, just short of the M3 Max Neural Engine figure upstream published, 4.98 ms. typed-decisions on cpu_gpu ran roughly 7x slower on the Air than the same checkpoint ran on the Max, in line with the fourfold GPU-core gap and then some. The Neural Engine result carries no such gap: it holds its ground against a much bigger chip in a way the GPU core count never could. That consistency, not raw GPU throughput, is the argument for running a typed-decision model at the edge on Apple Silicon, provided the question fits in the Neural Engine’s small pocket.

Reading the differences #

Both checkpoints land on the same headline call Jev made: pull the lever, at 81% probability from typed-decisions and 61% from the smaller multilingual. Their disagreement with Jev is instructive. typed-decisions picks duty_to_intervene over Jev’s minimize_total_harm as the dominant moral consideration, but at 1.5% confidence, a number that reads less like a wrong answer and more like a model refusing to commit. multilingual characterizes pulling the lever as cause_death rather than redirect_harm, the literal framing the trolley problem is built to test past. The switch case redirects an existing threat; it does not create one.

Across all eleven questions, Laya’s confidence sits an order of magnitude below Jev’s: single digits and low tens of a percent, against Jev’s mix of decisive high-90s and the teens, where Jev itself flags doubt. That gap may reflect how the two models define confidence, not which one is better calibrated. Laya-CoreML’s own documentation says its validation checks conversion fidelity against the original weights, not real-world accuracy, so I’d rather report the gap than paper over it with a claim I can’t back.

The full comparison, harness, and setup script reproduce every number above.

Where this fits #

A pre-trade risk check that must run inside a network-isolated enclave doesn’t have the option of a hosted API, no matter how fast. For that constraint, a typed-decision model running on Core ML is a real answer, and the Neural Engine result says Apple’s silicon can make it a fast one, once the state fits the budget the fastest path allows. For everything else, the four-times gap I measured says the network is still the better bet, and TypeSafe’s economics make it hard to argue otherwise.

Open weights got a MacBook Air to the same typed answer a hosted model gave over the network. They didn’t get there faster. Both things are true, and a production system should be built knowing which one it needs.