The short version
- CLM v0.1 8B is a decision model from Contrastive-LM, about 8.2B parameters. You give it a situation and multiple-choice questions, and it returns a probability for every option.
- In our tests it's #15 of the 15 decision models we've tested: right on 57% of calls, 55% of the hard cases, calibration 70.
- It has open weights (Apache 2.0), so you can download and run it yourself.
- 59 regular models you can run at home did better, led by Qwen3.5-27B (100%).
How it did
Our decision test has 56 situations: routing support tickets, spotting phishing, picking the right tool, applying a refund or expense policy, triaging incidents. 19 of them are hard cases that need a little date, time-zone or money math. The best decision model, Solar Decide, got 96%.
| Measure | CLM v0.1 8B | Solar Decide |
|---|---|---|
| Accuracy | 57 | 96 |
| Hard cases | 55 | 95 |
| Calibration | 70 | 96 |
| Public questions | 69 | 93 |
| Secret questions | 52 | 96 |
| API cost per 1,000 calls | local | – |
Calibration shows how honest its confidence is: 100 means always right and always sure. We ran it ourselves on a Mac mini (M4, 16 GB), with the encoder in llama.cpp instead of the reference GPU setup, so scores may differ slightly from the official setup.
How to try it
- Download the weights from Hugging Face: Contrastive-LM/CLM-v0.1-8B.
Its actual answers
What CLM v0.1 8B picked on our public questions, next to the right answer.
Refund window (hard) 50% right
| Question | It picked | Right answer |
|---|---|---|
| What does the policy give this customer? | No refund or credit | Store credit only |
| Does the item count as defective under the policy? | No | No |
Incident matrix (hard) 0% right
| Question | It picked | Right answer |
|---|---|---|
| Which severity applies under the matrix? | SEV1 | SEV3 |
| Should on-call be paged under the policy? | Yes | No |
Tool followup (hard) 100% right
| Question | It picked | Right answer |
|---|---|---|
| Which single tool should the assistant use for the user's actual remaining request? | Create, move or look up calendar events | Create, move or look up calendar events |
| Would handling the request change something on the user's behalf, so it should be confirmed first? | Yes | Yes |
Legit security alert (hard) 50% right
| Question | It picked | Right answer |
|---|---|---|
| Is this email a phishing attempt? | Yes | No |
| Given the context, does the recipient need to do anything about this email? | No | No |
Meeting slot (hard) 50% right
| Question | It picked | Right answer |
|---|---|---|
| Which candidate start time works for everyone? | 12:30 UTC | 12:30 UTC |
| Is Raj's working day the one that ends earliest (in UTC)? | No | Yes |
Review mixed (hard) 50% right
| Question | It picked | Right answer |
|---|---|---|
| Should this go to the hardware team? | Yes | Yes |
| Should this go to support leadership? | No | Yes |
Support checkout down 67% right
| Question | It picked | Right answer |
|---|---|---|
| Which team should handle this message? | Charges, invoices, refunds, plans and payments | Bugs, errors, outages, integrations |
| How urgently does this need a response? | Immediately: business is blocked | Immediately: business is blocked |
| Is a production service down or unusable for this customer? | Yes | Yes |
Refund wrong plan 100% right
| Question | It picked | Right answer |
|---|---|---|
| Which team should handle this message? | Charges, invoices, refunds, plans and payments | Charges, invoices, refunds, plans and payments |
| Is the customer asking for money back? | Yes | Yes |
| What is the writer's tone? | Polite, no frustration | Polite, no frustration |
Moderation doxxing 100% right
| Question | It picked | Right answer |
|---|---|---|
| Which content policy, if any, does this message violate? | Targets a person with abuse, threats or exposing their private information | Targets a person with abuse, threats or exposing their private information |
| Does the message reveal someone's private personal information? | Yes | Yes |
Route calendar 100% right
| Question | It picked | Right answer |
|---|---|---|
| Which single tool should the assistant use to handle the user's latest request? | Create, move or look up calendar events | Create, move or look up calendar events |
| Would handling the request change something on the user's behalf (send, create, delete or book something), so it should be confirmed first? | Yes | Yes |
Doc invoice missing due 100% right
| Question | It picked | Right answer |
|---|---|---|
| What kind of document is this? | A bill asking for payment | A bill asking for payment |
| Is the payment due date missing from the document? | Yes | Yes |
Phishing paypal 100% right
| Question | It picked | Right answer |
|---|---|---|
| Is this email a phishing attempt? | Yes | Yes |
| How likely is this email to be phishing? | High: clearly phishing | High: clearly phishing |
Pii ssn email 0% right
| Question | It picked | Right answer |
|---|---|---|
| What is the most sensitive kind of personal data in this message? | Diagnoses, prescriptions or other medical details | SSN, passport or national ID numbers |
| Does the message contain government IDs, financial account numbers or health information? | No | Yes |
Review mixed 100% right
| Question | It picked | Right answer |
|---|---|---|
| What is the writer's overall sentiment about the product? | Mostly unhappy | Mostly unhappy |
| Does the review report a product defect or breakage? | Yes | Yes |
| Would the writer recommend the product? | No | No |
Comments
Sign in with GitHub to comment. Spam and abuse are hidden automatically.