Home / Decision models Decision models · updated OCT 18
Span-02 review: the most accurate decision model we've tested It got 96% of our calls right and 95% of the hard ones. Available only through an API.
The short version Span-02 is a decision model from Respan. You give it a situation and multiple-choice questions, and it returns a probability for every option. In our tests it's the most accurate decision model we've tested: right on 96% of calls, 95% of the hard cases, calibration 96. It's closed: you can only use it through an API. 13 regular models you can run at home did better, led by Qwen3.5-27B (100%). How it did
Our decision test has 56 situations:
routing support tickets, spotting phishing, picking the right tool, applying a refund or expense policy, triaging incidents.
19 of them are hard cases that need a little date, time-zone or money math.
Measure Span-02 Accuracy 96 Hard cases 95 Calibration 96 Public questions 96 Secret questions 96
Calibration shows how honest its confidence is: 100 means always right and always sure. We called it through OpenRouter's Decisions API.
Its actual answers What Span-02 picked on our public questions, next to the right answer.
Refund window (hard) 50% right Question It picked Right answer What does the policy give this customer? Refund minus the restocking fee Store credit only Does the item count as defective under the policy? No No
Incident matrix (hard) 100% right Question It picked Right answer Which severity applies under the matrix? SEV3 SEV3 Should on-call be paged under the policy? No No
Tool followup (hard) 100% right Question It picked Right answer Which single tool should the assistant use for the user's actual remaining request? Create, move or look up calendar events Create, move or look up calendar events Would handling the request change something on the user's behalf, so it should be confirmed first? Yes Yes
Legit security alert (hard) 100% right Question It picked Right answer Is this email a phishing attempt? No No Given the context, does the recipient need to do anything about this email? No No
Meeting slot (hard) 100% right Question It picked Right answer Which candidate start time works for everyone? 12:30 UTC 12:30 UTC Is Raj's working day the one that ends earliest (in UTC)? Yes Yes
Review mixed (hard) 100% right Question It picked Right answer Should this go to the hardware team? Yes Yes Should this go to support leadership? Yes Yes
Support checkout down 100% right Question It picked Right answer Which team should handle this message? Bugs, errors, outages, integrations Bugs, errors, outages, integrations How urgently does this need a response? Immediately: business is blocked Immediately: business is blocked Is a production service down or unusable for this customer? Yes Yes
Refund wrong plan 100% right Question It picked Right answer Which team should handle this message? Charges, invoices, refunds, plans and payments Charges, invoices, refunds, plans and payments Is the customer asking for money back? Yes Yes What is the writer's tone? Polite, no frustration Polite, no frustration
Moderation doxxing 100% right Question It picked Right answer Which content policy, if any, does this message violate? Targets a person with abuse, threats or exposing their private information Targets a person with abuse, threats or exposing their private information Does the message reveal someone's private personal information? Yes Yes
Route calendar 100% right Question It picked Right answer Which single tool should the assistant use to handle the user's latest request? Create, move or look up calendar events Create, move or look up calendar events Would handling the request change something on the user's behalf (send, create, delete or book something), so it should be confirmed first? Yes Yes
Doc invoice missing due 100% right Question It picked Right answer What kind of document is this? A bill asking for payment A bill asking for payment Is the payment due date missing from the document? Yes Yes
Phishing paypal 100% right Question It picked Right answer Is this email a phishing attempt? Yes Yes How likely is this email to be phishing? High: clearly phishing High: clearly phishing
Pii ssn email 100% right Question It picked Right answer What is the most sensitive kind of personal data in this message? SSN, passport or national ID numbers SSN, passport or national ID numbers Does the message contain government IDs, financial account numbers or health information? Yes Yes
Review mixed 100% right Question It picked Right answer What is the writer's overall sentiment about the product? Mostly unhappy Mostly unhappy Does the review report a product defect or breakage? Yes Yes Would the writer recommend the product? No No
Decision models that scored about the same
Comments
Sign in with GitHub to comment. Spam and abuse are hidden automatically.