Review · updated OCT 14

Ornith 1.5 9B (Q4, Mac) review: decent, with trade-offs

It scored 64 out of 100, #29 of 67. It solved 17 of 30 coding jobs and scored 71 on reading documents. Runs on an 8 GB graphics card or a Mac with 16 GB.

The short version
  • Ornith 1.5 9B (Q4, Mac) is a free model from Ornith AI that you can run on your own computer. In our tests it's decent, with trade-offs: 64 out of 100, #29 of 67.
  • It solved 17 of 30 coding jobs and scored 71 on reading documents. On our hardest tasks it scored 44.
  • Runs on an 8 GB graphics card or a Mac with 16 GB.

Coding

Our coding test is 30 programming jobs, from small ones like reading time durations or cleaning up messy data to harder ones like a config-file parser or a double-entry ledger. We run each answer against tests the model never sees, and a job only counts if everything passes. Ornith 1.5 9B (Q4, Mac) got 17 of 30 right. The best local coders solved 29 of 30.

Reading documents

The second test hands the model things like an expense claim thread, a pay stub or an insurance statement, and asks for specific numbers and dates. Many questions need a bit of math, or noticing a correction further down the email. Ornith 1.5 9B (Q4, Mac) scored 71; the best model scored 100.

TestScorePublic questionsSecret questions
Coding574361
Reading documents717470
Decisions9610094
It often thinks too long. On 11 of 54 coding and document tasks, Ornith 1.5 9B (Q4, Mac) kept thinking until it hit our limit (about 25,000 words) and never gave an answer. Those count as wrong, which pulls its score down, and through an API you'd still pay for all that thinking. On your own computer it means waiting a long time for nothing.

On the 18 hardest tasks (included in the scores above) it scored 44. This number separates the top models.

We tested the Q4 download, the version most people run, with Ollama on our Mac mini (M4, 16 GB).

Can your computer run it?

Ornith 1.5 9B (Q4, Mac) comes in one standard size, which needs about 7.5 GB of memory.

Your computerRuns it?Version to download
Graphics card, 8 GBYesQ4
Graphics card, 12 GBYesQ4
Graphics card, 16 GBYesQ4
Graphics card, 24 GBYesQ4
Graphics card, 32 GBYesQ4
Mac, 16 GBYesQ4
Mac, 24 GBYesQ4
Mac, 32 GBYesQ4
Mac, 48 GBYesQ4
Mac, 64 GBYesQ4
Mac, 96 GBYesQ4
Mac, 128 GBYesQ4

For normal-length chats. Long documents need more memory. Q4 or Q8? · How we worked this out

How to try it

The easiest way is Ollama. Install it, then run:

ollama run hf.co/ornith-ai/Ornith-1.5-9B-GGUF:Q4_K_M

Want the strongest all-rounder instead? Qwen3.6 27B scored 97.

Its actual answers

What Ornith 1.5 9B (Q4, Mac) wrote for our public questions, word for word.

Coding: INI parser (hard) Didn't work
Coding: TTL LRU (hard) Didn't work
## Implementation

```python
from collections import OrderedDict
from typing import Any, Callable, Dict, Tuple


class TTLCache:
    """LRU cache with per-entry time-to-live (TTL) expiry.

    Recency is tracked with an ``OrderedDict`` whose last element is the most
    recently used item; the least recently used item is the first element.
    Each entry stores an absolute expiry time (``now + ttl``) so that replacing
    a key simply refreshes its lifetime, and expiry is evaluated against the
    current ``clock()`` value at access/insert time.
    """

    def __init__(self, capacity: int, ttl: float, clock: Callable[[], float]) -> None:
        self.capacity = capacity
        self.ttl = ttl
        self.clock = clock
        # key -> (expiry, value); iteration order is recency order.
        self._data: "OrderedDict[Any, Tuple[float, Any]]" = OrderedDict()
        self._stats: Dict[str, int] = {
            "hits": 0,
            "misses": 0,
            "evictions": 0,
            "expirations": 0,
        }

    def put(self, key: Any, value: Any) -> None:
        if self.capacity <= 0:
            return
        now = self.clock()

        if key in self._data:
            # Replace: refresh expiry and mark as most recently used.
            self._data[key] = (now + self.ttl, value)
            self._data.move_to_end(key)
            return

        # New key: if it would overflow capacity, reclaim space first.
        if len(self._data) >= self.capacity:
            expired_keys = [
                k for k, (expiry, _) in self._data.items() if expiry <= now
            ]
            for k in expired_keys:
                del self._data[k]
                self._stats["expirations"] += 1
            if len(self._data) >= self.capacity:
                self._data.popitem(last=False)  # evict least recently used
                self._stats["evictions"] += 1

        self._data[key] = (now + self.ttl, value)

    def get(self, key: Any, default: Any = None) -> Any:
        now = self.clock()
        if key not in self._data:
            self._stats["misses"] += 1
            return default
        expiry, value = self._data[key]
        if expiry > now:
            # Expired: remove and count as both an expiration and a miss.
            del self._data[key]
            self._stats["expirations"] += 1
            self._stats["misses"] += 1
            return default
        # Present and not expired: refresh recency and count a hit.
        self._data.move_to_end(key)
        self._stats["hits"] += 1
        return value

    def __len__(self) -> int:
        now = self.clock()
        return sum(1 for (_expiry, _value) in self._data.values() if _expiry > now)

    def stats(self) -> Dict[str, int]:
        return dict(self._stats)
```

## Notes on the design

- **Clock isolation** — only `self.clock()` is ever invoked; real time is never read.
- **Recency** — `OrderedDict` keeps the LRU entry first; `move_to_end` marks the MRU entry. Replacing a key moves it to the end (assigning to an existing key keeps position, so the explicit move is required).
- **Eviction vs. expiration** — when a new key would overflow capacity, *all* expired entries are purged (counted as expirations) *before* checking whether an LRU eviction is still needed, so an expired-but-LRU entry is never miscounted as an eviction.
- **`__len__`** — pure read that counts only non-expired entries (`expiry > now`), never touching stats.
- **`capacity == 0`** — `put` returns immediately (nothing stored, nothing evicted); `get` still reports misses.
Coding: Parse duration Didn't work
## Solution

```python
import re


def parse_duration(s: str) -> int:
    """Convert a duration string like '1h30m' into total seconds.

    Units: h (hours), m (minutes), s (seconds); case-insensitive.
    Parts must follow the order h, m, s and each appear at most once.
    Whitespace is allowed around and between parts.

    Raises ValueError for empty/blank strings, numbers without a unit,
    unknown units, decimals, repeated units, or units out of order.
    """
    if not isinstance(s, str):
        raise ValueError("duration must be a string")
    if not s.strip():
        raise ValueError("empty or blank string")

    # Whole-string structure: optional surrounding whitespace, then one or
    # more parts of the form  <digits><unit>, with optional whitespace between
    # parts.  This rejects numbers without a unit, decimals, stray symbols, etc.
    pattern = r'^\s*(?:\d+[a-zA-Z]+\s*)+\s*$'
    if re.fullmatch(pattern, s) is None:
        raise ValueError(f"invalid duration string: {s!r}")

    # unit -> (seconds per unit, position in the required h, m, s sequence)
    info = {'h': (3600, 0), 'm': (60, 1), 's': (1, 2)}

    parts = re.findall(r'(\d+)([a-zA-Z]+)', s)
    seen = set()
    prev = -1
    total = 0
    for _num, unit in parts:
        unit = unit.lower()
        if unit not in info:
            raise ValueError(f"unknown or malformed unit {unit!r}")
        seconds, pos = info[unit]
        if pos in seen:
            raise ValueError(f"repeated unit {unit!r}")
        if pos <= prev:
            raise ValueError(f"units out of order")
        seen.add(pos)
        prev = pos
        total += seconds
    return total
```
Coding: Summarize ranges Didn't work
```python
def summarize_ranges(nums: list[int]) -> str:
    if not nums:
        return ""

    sorted_nums = sorted(set(nums))

    result: list[str] = []
    start = prev = sorted_nums[0]
    count = 1

    for n in sorted_nums[1:]:
        if n == prev + 1:
            count += 1
        else:
            if count >= 3:
                result.append(f"{start}..{prev}")
            else:
                result.append(str(start))
            start = prev = n
            count = 1

    if count >= 3:
        result.append(f"{start}..{prev}")
    else:
        result.append(str(start))

    return ",".join(result)
```
Coding: Top customers Worked
```python
import re

# Matches an amount after stripping the "$" and thousands separators, e.g.
# "1234.50", "1,234.50", "-5.00", "1000". Uses ASCII digits only.
_AMOUNT_RE = re.compile(r"[+-]?[0-9]+(?:,[0-9]{3})*(?:\.[0-9]+)?")


def _parse_amount(amount) -> float | None:
    """Parse an amount string like "$1,234.50", "1234.5", "-$5.00" or "$1,000".

    Returns the numeric value as a float, or ``None`` when the amount cannot be
    parsed (empty, non-numeric, or containing other characters).
    """
    if not isinstance(amount, str):
        return None
    cleaned = amount.strip().replace("$", "").replace(",", "")
    if not cleaned or not _AMOUNT_RE.fullmatch(cleaned):
        return None
    try:
        return float(cleaned)
    except ValueError:
        return None


def top_customers(rows: list[dict], n: int) -> list[tuple[str, float]]:
    """Return the top ``n`` customers by total amount.

    - Amounts are summed per customer (case-insensitive; the reported name is
      the trimmed spelling of that customer's first valid row).
    - Rows with an empty customer name or an unparseable amount are skipped.
    - Totals are rounded to 2 decimals.
    - Results are sorted by total descending, then by name ascending.
    """
    totals: dict[str, dict[str, float]] = {}

    for row in rows:
        if not isinstance(row, dict):
            continue
        name = row.get("customer")
        amount = row.get("amount")
        if not isinstance(name, str) or not isinstance(amount, str):
            continue

        trimmed = name.strip()
        if not trimmed:
            continue

        value = _parse_amount(amount)
        if value is None:
            continue

        key = trimmed.lower()
        if key not in totals:
            totals[key] = {"name": trimmed, "total": 0.0}
        totals[key]["total"] += value

    results = [(info["name"], round(info["total"], 2)) for info in totals.values()]
    results.sort(key=lambda item: (-item[1], item[0]))
    return results[:n]
```
Coding: Slugify Worked
```python
import re
import unicodedata


def slugify(title: str, max_len: int = 50) -> str:
    """Build a URL slug from a title."""
    # 1. Transliterate accented characters to ASCII (NFKD splits é -> e + combining mark);
    #    drop any remaining non-ASCII characters.
    slug = unicodedata.normalize('NFKD', title).encode('ascii', 'ignore').decode('ascii')

    # 2. Lowercase.
    slug = slug.lower()

    # 3. Replace every run of non-alphanumeric characters with a single "-", strip ends.
    slug = re.sub(r'[^a-z0-9]+', '-', slug).strip('-')

    # 4. If longer than max_len, shorten without cutting a whole word.
    if len(slug) > max_len:
        result = ''
        for word in slug.split('-'):
            if result:
                candidate = result + '-' + word
                if len(candidate) > max_len:
                    break
                result = candidate
            elif len(word) > max_len:
                # Even the first word alone is too long: hard-cut it.
                return word[:max_len]
            else:
                result = word
        return result

    # 5. By construction the result never ends with "-".
    return slug
```
Coding: Token bucket Worked
```python
from __future__ import annotations

from collections.abc import Callable


class TokenBucket:
    """Rate limiter implementing the token bucket algorithm.

    Args:
        capacity: Maximum number of tokens the bucket can hold.
        refill_per_sec: Tokens added per second of elapsed clock time.
        clock: Zero-argument callable returning the current time in seconds.
    """

    def __init__(
        self,
        capacity: float,
        refill_per_sec: float,
        clock: Callable[[], float],
    ) -> None:
        self.capacity = capacity
        self.refill_per_sec = refill_per_sec
        self.clock = clock
        self._tokens = capacity
        self._last = clock()

    def _refill(self, now: float) -> None:
        elapsed = now - self._last
        if elapsed > 0:
            self._tokens = min(self.capacity, self._tokens + elapsed * self.refill_per_sec)
            self._last = now

    @property
    def tokens(self) -> float:
        self._refill(self.clock())
        return self._tokens

    def allow(self, cost: float = 1) -> bool:
        self._refill(self.clock())
        if self._tokens >= cost:
            self._tokens -= cost
            return True
        return False
```
Decisions: Refund window (hard) 100% right
{
  "outcome": {
    "full_refund": 0.0,
    "partial_refund": 0.0,
    "store_credit": 1.0,
    "nothing": 0.0
  },
  "defective": {
    "true": 0.0,
    "false": 1.0
  }
}
Decisions: Incident matrix (hard) 100% right
{
  "severity": {
    "0": 0.01,
    "1": 0.97,
    "2": 0.01,
    "3": 0.01
  },
  "page": {
    "true": 0.03,
    "false": 0.97
  }
}
Decisions: Tool followup (hard) 100% right
{
  "tool": {
    "web_search": 0.01,
    "calculator": 0.02,
    "calendar": 0.97,
    "email": 0.0,
    "none": 0.0
  },
  "confirm": {
    "true": 0.95,
    "false": 0.05
  }
}
Decisions: Legit security alert (hard) 100% right
{
  "phishing": {
    "true": 0.01,
    "false": 0.99
  },
  "action_needed": {
    "true": 0.03,
    "false": 0.97
  }
}
Decisions: Meeting slot (hard) 100% right
{
  "slot": {"A": 0.0, "B": 1.0, "C": 0.0, "D": 0.0},
  "raj_last": {"true": 1.0, "false": 0.0}
}
Decisions: Review mixed (hard) 100% right
{
  "hardware": {
    "true": 0.95,
    "false": 0.05
  },
  "support": {
    "true": 0.95,
    "false": 0.05
  }
}
Decisions: Support checkout down 100% right
{
  "department": {
    "billing": 0.04,
    "technical": 0.93,
    "account": 0.01,
    "shipping": 0.005,
    "sales": 0.015
  },
  "urgency": {
    "0": 0.003,
    "1": 0.007,
    "2": 0.03,
    "3": 0.96
  },
  "outage": {
    "true": 0.97,
    "false": 0.03
  }
}
Decisions: Refund wrong plan 100% right
{
  "department": {
    "billing": 1.0,
    "technical": 0.0,
    "account": 0.0,
    "shipping": 0.0,
    "sales": 0.0
  },
  "refund": {
    "true": 1.0,
    "false": 0.0
  },
  "tone": {
    "frustrated": 0.0,
    "calm": 1.0
  }
}
Decisions: Moderation doxxing 100% right
{
  "policy": {
    "none": 0.005,
    "harassment": 0.99,
    "hate": 0.003,
    "spam": 0.0,
    "self_harm": 0.002
  },
  "personal_info": {
    "true": 0.99,
    "false": 0.01
  }
}
Decisions: Route calendar 100% right
{
  "tool": {
    "web_search": 0.03,
    "calculator": 0.02,
    "calendar": 0.85,
    "email": 0.05,
    "none": 0.05
  },
  "confirm": {
    "true": 0.9,
    "false": 0.1
  }
}
Decisions: Doc invoice missing due 100% right
{
  "doc_type": {
    "invoice": 0.99,
    "resume": 0.0,
    "contract": 0.0,
    "bank_statement": 0.0,
    "other": 0.01
  },
  "missing_due_date": {
    "true": 0.97,
    "false": 0.03
  }
}
Decisions: Phishing paypal 100% right
{
  "phishing": {
    "true": 0.99,
    "false": 0.01
  },
  "risk": {
    "0": 0.005,
    "1": 0.005,
    "2": 0.01,
    "3": 0.98
  }
}
Decisions: Pii ssn email 100% right
{
  "data_kind": {
    "none": 0.0,
    "contact": 0.0,
    "financial": 0.02,
    "government_id": 0.98,
    "health": 0.0
  },
  "sensitive": {
    "true": 0.98,
    "false": 0.02
  }
}
Decisions: Review mixed 100% right
{
  "sentiment": {
    "positive": 0.03,
    "neutral": 0.05,
    "negative": 0.92
  },
  "defect": {
    "true": 0.97,
    "false": 0.03
  },
  "recommend": {
    "true": 0.02,
    "false": 0.98
  }
}
Documents: Saas escalator (hard) 0% right
Documents: Expense thread 100% right
{
  "employee_id": "EMP-20417",
  "destination_city": "Lisbon",
  "trip_start": "2025-02-24",
  "trip_end": "2025-02-27",
  "approved_items": [
    { "date": "2025-02-24", "category": "airfare", "amount_usd": 1184.60 },
    { "date": "2025-02-24", "category": "ground_transport", "amount_usd": 38.88 },
    { "date": "2025-02-25", "category": "meals", "amount_usd": 229.39 },
    { "date": "2025-02-26", "category": "lodging", "amount_usd": 466.56 },
    { "date": "2025-02-27", "category": "ground_transport", "amount_usd": 44.82 }
  ],
  "rejected_item_count": 1,
  "per_diem_days": 3,
  "per_diem_usd": 195.00,
  "total_reimbursable_usd": 2159.25,
  "approver_email": "priya.raman@corvane.com"
}
Documents: Lease amendment 77% right
{
  "tenants": "Marcus Lin; Sofia Lin",
  "landlord": "Ridgeline Property Group LLC",
  "zip": "97205",
  "lease_end": "2025-11-30",
  "original_monthly_rent": 2150.00,
  "monthly_rent_from_2025_06_01": 2236.00,
  "late_fee_from_2025_06_01": 111.80,
  "security_deposit": 2150.00,
  "total_pet_deposits": 800.00,
  "total_monthly_payment_july_2025": 2306.00,
  "move_in_payment": 4700.00
}
Documents: Ticket SLA 92% right
{
  "ticket_id": "48213",
  "account_id": "ACC-7731",
  "open_issue": "inventory_sync",
  "resolved_issues": ["billing_address", "invoice_pdf"],
  "affected_orders": ["SO-99812", "SO-99820", "SO-99827"],
  "priority": "P2",
  "sla_due_local": "2025-09-15T15:30",
  "sla_due_utc": "2025-09-15T20:30:00Z",
  "reissued_invoice": "INV-2025-0812"
}
Documents: Sales footnotes 100% right
{
  "q3_total_usd": 15346000,
  "q2_total_usd": 14464000,
  "q2_central_originally_reported_usd": 3047000,
  "q2_to_q3_change_pct": 6.1,
  "top_region_q3": "East",
  "fastest_growing_region_q1_to_q3": "International",
  "regions_declining_q2_to_q3": ["East"],
  "international_q3_organic_usd": 1731000,
  "west_excluding_mountain_q3_usd": 4201000
}

Size: 9.7B parameters. First tested OCT 13.

Models that scored about the same

Comments

Sign in with GitHub to comment. Spam and abuse are hidden automatically.