Jev Attaches a Probability to Every Decision. Backtest It Yourself.

You have at least one AI step that decides something. Which intake email is a real lead. Which draft is close enough to send. Which support message goes to the top of the pile.

That step answers every time, in the same confident register, whether the case is obvious or genuinely borderline. You cannot tell the two apart by reading the output. So you check all of them, which is what you were doing before you added the step.

On 15 September 2026, TypeSafe AI came out of stealth with a $40 million seed round led by DCVC and shipped a model built for exactly that step. It is called Jev, and it will not talk to you. It takes a schema you declare in advance and returns typed values, each one carrying a probability.

The speed numbers got the attention. TypeSafe’s own workflow evaluation reports 193.6 times faster and 444.6 times cheaper than frontier language models on narrow decision tasks, at $0.042 per million input tokens with output billed at nothing, and end-to-end responses in 70 to 500 milliseconds against 3 to 329 seconds. Vercel called it the fastest-adopted model in AI Gateway history: nearly 13 percent of paid teams inside 24 hours, twice the first-day share of the GPT-5.6 family and more than six times Fable 5.1’s.

Those are the numbers that will get quoted. They are not the interesting part. The interesting part is the probability, and whether it means anything.

Most of what you automate is a decision wearing an essay’s clothes

Start with what Jev is not, because that is where a week goes.

Jev does not replace the model you write with. It cannot write. It has no string output. TypeSafe’s framing is a frontier-intelligence function call: unstructured state in, typed probabilistic decisions out. The name System One points where you think it does, at fast automatic judgment rather than slow deliberation.

There are three answer shapes. Choice, Score, and Boolean. Pick one, declare the options, get back a value and a probability.

What it replaces is the if statement you have been asking a conversational model to make.

That table is the whole product thesis. If your workflow sorts, routes, scores, extracts or classifies, you are running a decision, and you have probably been paying conversational prices to make it, on every item, forever. Sunday’s digest made the cost half of that argument. This post is about the other half.

Two limits worth knowing before you design against it. A choice can carry up to 255 options; above that, TypeSafe uses a slower two-stage score-then-select. And early access is waitlisted, so most readers cannot run their own test yet.

The person who co-invented RLHF is now naming its failure mode

Here is the part of the launch that does not appear in any benchmark table.

TypeSafe’s own documentation states that RLHF “was used to train InstructGPT and ChatGPT and was co-invented by Diogo Almeida, cofounder of TypeSafe.” Almeida is TypeSafe’s chief executive. The company is not quietly moving past the objective he helped invent. It is naming what that objective costs.

The docs name two costs. The first is sycophancy, which everyone has felt. The second is the one worth your attention:

Preference optimization also causes mode dropping: the model learns to favor a particular style, such as instruction following, while reducing the probability of other possible outputs.

Mode dropping is the milder cousin of mode collapse. Train on what raters preferred, and probability mass drains away from the valid answers that were merely less popular. The model does not become wrong so much as narrow, and it stays fluent the whole way down. You will not see it in any single output. You see it as a model that keeps giving you the same three answers to a question that has six.

Then the sentence that does the real work:

Human preference and machine trustworthiness are different optimization targets.

That is a correct and unfashionable thing for a model lab to write. A rater picking the response they liked better is not measuring whether it was right. Optimise long enough against the first and you get a system tuned to be approved of, which is not the same system as one tuned to be relied on.

RLCD, Reinforcement Learning for Calibrated Decisions, is TypeSafe’s answer. Three objectives, three different things being rewarded:

I want to be precise about what is and is not published here. As of today, TypeSafe has released no paper for RLCD, no reward function, and no calibration curves. The benchmark format is one they invented, designed and graded themselves, and their launch post acknowledges possible design bias. The zero percent hallucination figure is a schema-conformance claim, and TypeSafe says in its own post that the number “is not empirical.”

So the objective is stated. The evidence for it is, so far, the vendor’s.

Calibration is a denominator claim

TypeSafe’s definition of calibration is the textbook one, and it contains its own warning label:

Outcomes assigned a probability of 0.2 should occur about 20% of the time. These rates describe groups of predictions, not a guarantee about any single answer.

Read the second sentence twice, because it is the sentence people will skip.

Calibration is a property of a population, measured on a distribution. It says nothing about the case currently in front of you. A model can be flawlessly calibrated and still be wrong on this email. Calibration does not tell you the answer is right. It tells you how often answers that looked like this one turned out to be right, across the set it was measured on.

Which raises the question that decides whether any of this is useful to you: measured on which set?

This is the same problem as the denominator nobody reports. A number that is reconciled, validated and confidently stated is still unusable until somebody says what it was computed over. A 90 percent confidence figure carries an implied population. If that population is not your work, the figure is not about your work.

Your business is, almost by definition, not the benchmark. The whole reason you have a business is that your cases are not the public distribution.

Two people have already gone and measured it

This is where the story gets genuinely useful, and where it stops being about TypeSafe.

Within four days of launch, two unaffiliated developers published reproducible calibration audits on GitHub. Neither is peer reviewed. Both state their own limits carefully. Both are worth more than the launch page.

The first, an out-of-distribution audit published 19 September, is well designed: three public benchmarks the model has probably seen, against 900 rule-generated support tickets it cannot have seen, scored by expected calibration error against a simulated noise floor.

On distributions it knows, the calibration is good. Essentially at the noise floor on one of them. That is a real result and it should be said plainly, because the vendor could not have manufactured it.

Off distribution, it degrades by roughly a factor of four. And the shape of the degradation is the finding that matters:

The sign flips between output types. Ask the same model the same case as a multiple choice and it runs hot; ask it as a yes or no and it runs cold. The author’s conclusion is the operational one: whether a fixed threshold is safe depends on the question type, not just on the task.

If you wire confidence > 0.9 to an auto-approve, you have chosen one number. That number is doing three different jobs across three output types, and it is wrong in two directions at once.

The second audit, a Russian against English comparison published 20 September against model version jev-1.13.0, tests the other axis. Same 600 paired items, same task, one language apart.

An 11.0 percentage point accuracy drop, with a 95 percent confidence interval of 7.8 to 14.2 points. Calibration error triples. And look at the last two rows, because that is the bucket your threshold actually reads. In English, 70.5 percent of items clear 0.9 confidence and 97.2 percent of those are right. In Russian, only 51.8 percent clear it, and 88.7 percent of those are right.

Same threshold. Same task. It admits fewer cases and it is wrong more often inside the ones it admits.

On a second dataset the same author found no detectable difference at that sample size, and says so rather than burying it. Both authors name their limits without being asked: one prompt wording, single synthetic task family, benchmark contamination unknowable, a black-box API, results dated to a specific day and a specific model version.

That is how you report a measurement. It is also, precisely, the discipline the vendor’s own benchmark does not have.

Banks have a rule about who grades the model

There is a reason the self-graded benchmark bothers me more than the missing paper.

I spent decades building market risk engines and regulatory reporting infrastructure at two of North America’s largest banks. A risk model that outputs a probability is the most ordinary object in that world, and nobody there accepts one because the people who built it produced a chart.

You backtest it. You take the model’s stated probabilities, line them up against what actually happened, and count. You do it per portfolio, because a model calibrated on one book is not calibrated on another. You do it again after a regime change, because the distribution moved underneath you. And the review is done by people who did not build the model, because six supervisors on three continents all require exactly that separation and one of them notes it is the requirement most often missing.

TypeSafe designed the benchmark, ran it, and graded it. That is not an accusation of bad faith. It is a structural fact about the evidence, and it is the specific structure that supervisory guidance exists to rule out. The two GitHub audits are worth more than the launch page for one reason, and it is not that the authors are smarter. It is that they did not build the model.

Which makes the operational rule simple, and it is the same rule whether or not you ever touch Jev: an unvalidated confidence number is a claim, not a feature. Somebody has to backtest it against your cases, on your distribution. If nobody does, you have not added a control. You have added a number that makes you feel like you have one.

What the probability is actually for

None of this makes Jev a bad bet. I think the direction is right, and I think the objection is smaller than it looks.

Here is the thing a calibrated probability genuinely buys you, and it is not automation.

Generation and evaluation scale differently. AI can produce drafts and decisions far faster than one person can responsibly review them, and the gap between the two rates is where the approval pile comes from. Every additional AI step makes that pile bigger. More output is not leverage once you are the bottleneck.

A trustworthy confidence number does not empty the pile. It sorts it. It tells you which twelve of the ninety cases are the ones worth your attention this morning, and lets the rest sit under a lower level of scrutiny. That is a review queue with an order, which is a materially different object from a review queue without one.

It does not remove the gate. It makes the gate routable. A gate that fires on everything is the same number as a gate that fires on nothing, and both of them get ignored by week three.

That is why the calibration question is not academic. If the number is honest on your distribution, you get your mornings back. If it is not, you have automated the wrong twelve cases and you will not find out for a month.

So the question is not whether Jev is good. It is whether you have measured the thing you are about to trust.

Try this now (10 minutes)

You do not need Jev access for this, and that is the point. Any model you currently ask to decide something can be checked the same way.

1. Pick one AI step in your work that produces a decision rather than a draft. Lead or not a lead. Urgent or not urgent. Ready to send or not ready.
2. Take the last 20 cases it handled where you already know the true answer.
3. Re-run each one, and ask for two things: the decision, and a number from 0 to 1 for how likely that decision is to be correct.
4. Put them in two buckets. The cases it scored above 0.9, and everything else.
5. Count how often it was actually right in the above-0.9 bucket.

If it said 0.95 and it was right 19 times out of 20, you have a usable number and you can start routing on it. If it said 0.95 and it was right 14 times out of 20, you have an overconfident step, and any threshold you set on it is decorating a guess.

Twenty cases will not give you a publishable calibration curve. It will tell you which of those two situations you are in, which is the decision you actually have to make this week.

Stop there. That counts.

Where to go next

The measurement discipline behind step 5 is the whole of the Agent Audit: a baseline taken from your own evidence before anything is claimed about it, not a vendor’s chart.

If you build agents, the point about the two GitHub audits generalises. Reading an artifact is not verifying it, and a check that only inspects what the producer chose to show is not a check. Running the thing on inputs the producer never picked is the control that works. That is the software engineer track version of this week.

If the ten-minute test above gave you an overconfident step, the first fix is not a better model. It is a written criterion the output can fail against, which is where The Three Fixes starts. A model change does not repair a missing standard, for the reason the model is not the operating system.

Frequently asked questions

What is Jev and what are System One models? Jev is the first model from TypeSafe AI, released in waitlisted early access on 15 September 2026. TypeSafe calls it a System One model: instead of generating text, it takes unstructured input plus a schema you declare in advance and returns typed values, each carrying a probability. Every field is produced in one parallel pass rather than one token at a time. It is built for the decisions inside software, such as classification, routing, scoring and extraction, and it cannot produce free-form text at all.

What is RLCD and how does it differ from RLHF? RLCD stands for Reinforcement Learning for Calibrated Decisions. RLHF optimises for responses human raters prefer. RLVR optimises for outputs a program can verify. RLCD optimises for calibration, meaning decisions whose stated probabilities match how often those decisions turn out to be right. TypeSafe’s documentation names two RLHF failure modes it is reacting to: sycophancy, and mode dropping, which it describes as the model learning to favour a particular style while reducing the probability of other possible outputs. As of 21 September 2026, TypeSafe has published no paper, no reward function and no calibration curves for RLCD.

Does Jev really have a zero percent hallucination rate? It has a zero percent rate of a specific failure, and TypeSafe says so plainly. The claim is schema conformance: the model fills a declared schema, so it cannot invent a field or emit malformed output. TypeSafe’s own launch post states that the number “is not empirical” and that schema matching is guaranteed by construction. Jev can still select the wrong allowed value and attach an unjustified probability to it. Shape is guaranteed. Truth is not.

Can I trust the confidence score enough to auto-approve above a threshold? Not until you have measured it on your own cases. Calibration is a property of a population of predictions on a particular distribution, not a promise about any single answer. Two independent audits published in the days after launch found calibration close to the noise floor on public benchmarks and roughly four times worse on a task the model could not have seen, with the direction of the error flipping between output types. A threshold you validated on one kind of question is not transferable to another. Measure your own before you wire it to anything that ships.