Jev 1.13: Decision Model Benchmarks and Accuracy

The short version

Jev 1.13 is a decision model from Typesafe. You give it a situation and multiple-choice questions, and it returns a probability for every option.
In our tests it's #3 of the 15 decision models we've tested: right on 92% of calls, 79% of the hard cases, calibration 94.
It's closed: you can only use it through an API.
25 regular models you can run at home did better, led by Qwen3.5-27B (100%).

How it did

Our decision test has 56 situations: routing support tickets, spotting phishing, picking the right tool, applying a refund or expense policy, triaging incidents. 19 of them are hard cases that need a little date, time-zone or money math. The best decision model, Solar Decide, got 96%.

Measure	Jev 1.13	Solar Decide
Accuracy	92	96
Hard cases	79	95
Calibration	94	96
Public questions	89	93
Secret questions	93	96
API cost per 1,000 calls	$0.02	–

Calibration shows how honest its confidence is: 100 means always right and always sure. We called it through OpenRouter's Decisions API.

How to try it

Through OpenRouter: typesafe/jev-1.13 (Decisions API).

Its actual answers

What Jev 1.13 picked on our public questions, next to the right answer.

Refund window (hard) 50% right

Question	It picked	Right answer
What does the policy give this customer?	Refund minus the restocking fee	Store credit only
Does the item count as defective under the policy?	No	No

Incident matrix (hard) 100% right

Question	It picked	Right answer
Which severity applies under the matrix?	SEV3	SEV3
Should on-call be paged under the policy?	No	No

Tool followup (hard) 100% right

Question	It picked	Right answer
Which single tool should the assistant use for the user's actual remaining request?	Create, move or look up calendar events	Create, move or look up calendar events
Would handling the request change something on the user's behalf, so it should be confirmed first?	Yes	Yes

Legit security alert (hard) 100% right

Question	It picked	Right answer
Is this email a phishing attempt?	No	No
Given the context, does the recipient need to do anything about this email?	No	No

Meeting slot (hard) 0% right

Question	It picked	Right answer
Which candidate start time works for everyone?	11:45 UTC	12:30 UTC
Is Raj's working day the one that ends earliest (in UTC)?	No	Yes

Review mixed (hard) 100% right

Question	It picked	Right answer
Should this go to the hardware team?	Yes	Yes
Should this go to support leadership?	Yes	Yes

Support checkout down 100% right

Question	It picked	Right answer
Which team should handle this message?	Bugs, errors, outages, integrations	Bugs, errors, outages, integrations
How urgently does this need a response?	Immediately: business is blocked	Immediately: business is blocked
Is a production service down or unusable for this customer?	Yes	Yes

Refund wrong plan 100% right

Question	It picked	Right answer
Which team should handle this message?	Charges, invoices, refunds, plans and payments	Charges, invoices, refunds, plans and payments
Is the customer asking for money back?	Yes	Yes
What is the writer's tone?	Polite, no frustration	Polite, no frustration

Moderation doxxing 100% right

Question	It picked	Right answer
Which content policy, if any, does this message violate?	Targets a person with abuse, threats or exposing their private information	Targets a person with abuse, threats or exposing their private information
Does the message reveal someone's private personal information?	Yes	Yes

Route calendar 100% right

Question	It picked	Right answer
Which single tool should the assistant use to handle the user's latest request?	Create, move or look up calendar events	Create, move or look up calendar events
Would handling the request change something on the user's behalf (send, create, delete or book something), so it should be confirmed first?	Yes	Yes

Doc invoice missing due 100% right

Question	It picked	Right answer
What kind of document is this?	A bill asking for payment	A bill asking for payment
Is the payment due date missing from the document?	Yes	Yes

Phishing paypal 100% right

Question	It picked	Right answer
Is this email a phishing attempt?	Yes	Yes
How likely is this email to be phishing?	High: clearly phishing	High: clearly phishing

Pii ssn email 100% right

Question	It picked	Right answer
What is the most sensitive kind of personal data in this message?	SSN, passport or national ID numbers	SSN, passport or national ID numbers
Does the message contain government IDs, financial account numbers or health information?	Yes	Yes

Review mixed 100% right

Question	It picked	Right answer
What is the writer's overall sentiment about the product?	Mostly unhappy	Mostly unhappy
Does the review report a product defect or breakage?	Yes	Yes
Would the writer recommend the product?	No	No

Decision models Rankings

Decision models that scored about the same

Solar Decide · 96%Mercury Decide · 94%Drex v1.5 · 92%Decider V1.1 27B · 91%

Jev 1.13 review: #3 of the 15 decision models we've tested

How it did

How to try it

Its actual answers

Decision models that scored about the same

Comments