The short version
- Clef Omni is a decision model from Cloudflare, about 35B parameters. You give it a situation and multiple-choice questions, and it returns a probability for every option.
- In our tests it's #14 of the 15 decision models we've tested: right on 83% of calls, 68% of the hard cases, calibration 86.
- It has open weights (Apache 2.0), so you can download and run it yourself.
- 40 regular models you can run at home did better, led by Qwen3.5-27B (100%).
How it did
Our decision test has 56 situations: routing support tickets, spotting phishing, picking the right tool, applying a refund or expense policy, triaging incidents. 19 of them are hard cases that need a little date, time-zone or money math. The best decision model, Solar Decide, got 96%.
| Measure | Clef Omni | Solar Decide |
|---|---|---|
| Accuracy | 83 | 96 |
| Hard cases | 68 | 95 |
| Calibration | 86 | 96 |
| Public questions | 89 | 93 |
| Secret questions | 81 | 96 |
| API cost per 1,000 calls | $0.06 | – |
Calibration shows how honest its confidence is: 100 means always right and always sure. We called it through OpenRouter's Decisions API.
How to try it
- Through OpenRouter:
cloudflare/clef-omni(Decisions API). - Download the weights from Hugging Face: Cloudflare/clef-omni.
Its actual answers
What Clef Omni picked on our public questions, next to the right answer.
Refund window (hard) 50% right
| Question | It picked | Right answer |
|---|---|---|
| What does the policy give this customer? | Refund minus the restocking fee | Store credit only |
| Does the item count as defective under the policy? | No | No |
Incident matrix (hard) 50% right
| Question | It picked | Right answer |
|---|---|---|
| Which severity applies under the matrix? | SEV2 | SEV3 |
| Should on-call be paged under the policy? | No | No |
Tool followup (hard) 100% right
| Question | It picked | Right answer |
|---|---|---|
| Which single tool should the assistant use for the user's actual remaining request? | Create, move or look up calendar events | Create, move or look up calendar events |
| Would handling the request change something on the user's behalf, so it should be confirmed first? | Yes | Yes |
Legit security alert (hard) 100% right
| Question | It picked | Right answer |
|---|---|---|
| Is this email a phishing attempt? | No | No |
| Given the context, does the recipient need to do anything about this email? | No | No |
Meeting slot (hard) 50% right
| Question | It picked | Right answer |
|---|---|---|
| Which candidate start time works for everyone? | 12:30 UTC | 12:30 UTC |
| Is Raj's working day the one that ends earliest (in UTC)? | No | Yes |
Review mixed (hard) 100% right
| Question | It picked | Right answer |
|---|---|---|
| Should this go to the hardware team? | Yes | Yes |
| Should this go to support leadership? | Yes | Yes |
Support checkout down 100% right
| Question | It picked | Right answer |
|---|---|---|
| Which team should handle this message? | Bugs, errors, outages, integrations | Bugs, errors, outages, integrations |
| How urgently does this need a response? | Immediately: business is blocked | Immediately: business is blocked |
| Is a production service down or unusable for this customer? | Yes | Yes |
Refund wrong plan 100% right
| Question | It picked | Right answer |
|---|---|---|
| Which team should handle this message? | Charges, invoices, refunds, plans and payments | Charges, invoices, refunds, plans and payments |
| Is the customer asking for money back? | Yes | Yes |
| What is the writer's tone? | Polite, no frustration | Polite, no frustration |
Moderation doxxing 100% right
| Question | It picked | Right answer |
|---|---|---|
| Which content policy, if any, does this message violate? | Targets a person with abuse, threats or exposing their private information | Targets a person with abuse, threats or exposing their private information |
| Does the message reveal someone's private personal information? | Yes | Yes |
Route calendar 100% right
| Question | It picked | Right answer |
|---|---|---|
| Which single tool should the assistant use to handle the user's latest request? | Create, move or look up calendar events | Create, move or look up calendar events |
| Would handling the request change something on the user's behalf (send, create, delete or book something), so it should be confirmed first? | Yes | Yes |
Doc invoice missing due 100% right
| Question | It picked | Right answer |
|---|---|---|
| What kind of document is this? | A bill asking for payment | A bill asking for payment |
| Is the payment due date missing from the document? | Yes | Yes |
Phishing paypal 100% right
| Question | It picked | Right answer |
|---|---|---|
| Is this email a phishing attempt? | Yes | Yes |
| How likely is this email to be phishing? | High: clearly phishing | High: clearly phishing |
Pii ssn email 100% right
| Question | It picked | Right answer |
|---|---|---|
| What is the most sensitive kind of personal data in this message? | SSN, passport or national ID numbers | SSN, passport or national ID numbers |
| Does the message contain government IDs, financial account numbers or health information? | Yes | Yes |
Review mixed 100% right
| Question | It picked | Right answer |
|---|---|---|
| What is the writer's overall sentiment about the product? | Mostly unhappy | Mostly unhappy |
| Does the review report a product defect or breakage? | Yes | Yes |
| Would the writer recommend the product? | No | No |
Comments
Sign in with GitHub to comment. Spam and abuse are hidden automatically.