Review · updated OCT 14

LFM2.5-24B-A2B review: decent, with trade-offs

It scored 62 out of 100, #30 of 67. It solved 13 of 30 coding jobs and scored 80 on reading documents. Runs on a 24 GB graphics card or a Mac with 24 GB.

The short version
  • LFM2.5-24B-A2B is a free model from Liquid AI that you can run on your own computer. In our tests it's decent, with trade-offs: 62 out of 100, #30 of 67.
  • It solved 13 of 30 coding jobs and scored 80 on reading documents. On our hardest tasks it scored 33.
  • Runs on a 24 GB graphics card or a Mac with 24 GB.

Coding

Our coding test is 30 programming jobs, from small ones like reading time durations or cleaning up messy data to harder ones like a config-file parser or a double-entry ledger. We run each answer against tests the model never sees, and a job only counts if everything passes. LFM2.5-24B-A2B got 13 of 30 right. The best local coders solved 29 of 30.

Reading documents

The second test hands the model things like an expense claim thread, a pay stub or an insurance statement, and asks for specific numbers and dates. Many questions need a bit of math, or noticing a correction further down the email. LFM2.5-24B-A2B scored 80; the best model scored 100.

TestScorePublic questionsSecret questions
Coding437135
Reading documents809776
Decisions929691
Its scores are held back by the API. The provider we test LFM2.5-24B-A2B through stops every answer at 8,192 tokens, counting its thinking. On 20 of 54 coding and document tasks it was still thinking when it was cut off, so it never gave an answer and those count as wrong. Other models get up to 32,768 tokens. Run it yourself with a higher limit and it should do better than its score here.

On the 18 hardest tasks (included in the scores above) it scored 33. This number separates the top models.

This model did much better on our public questions than on the secret ones. Sometimes that's luck; it can also mean the model has seen the public questions before.

We tested the full-size version online. The smaller versions you download usually score a little lower.

Can your computer run it?

You can download LFM2.5-24B-A2B in several sizes. Smaller versions fit on more computers and lose a little quality, so pick the biggest one that fits.

Your computerRuns it?Version to download
Graphics card, 8 GBNo–
Graphics card, 12 GBNo–
Graphics card, 16 GBNo–
Graphics card, 24 GBYesQ6
Graphics card, 32 GBYesQ8
Mac, 16 GBNo–
Mac, 24 GBYesQ4
Mac, 32 GBYesQ6
Mac, 48 GBYesQ8
Mac, 64 GBYesQ8
Mac, 96 GBYesQ8
Mac, 128 GBYesQ8

For normal-length chats. Long documents need more memory. Q4 or Q8? · How we worked this out

How to try it

Search for "LFM2.5-24B-A2B GGUF" in LM Studio or on Hugging Face and pick the version that fits your computer. New to this? Start with our Ollama guide.

Want the strongest all-rounder instead? Qwen3.6 27B scored 97.

Its actual answers

What LFM2.5-24B-A2B wrote for our public questions, word for word.

Coding: INI parser (hard) Didn't work
Coding: TTL LRU (hard) Didn't work
Coding: Parse duration Worked
```python
import re

def parse_duration(s: str) -> int:
    # Validate input is not empty or blank
    if s is None:
        raise ValueError("Empty or blank string")
    
    # Remove all whitespace characters
    cleaned = re.sub(r'\s+', '', s)
    if not cleaned:
        raise ValueError("Empty or blank string")

    # Match number followed by a unit (case‑insensitive)
    pattern = re.compile(r'(\d+)([hHmsS])')
    matches = pattern.findall(cleaned)

    # Ensure the entire string was consumed by the pattern
    reconstructed = ''.join(num + unit for num, unit in matches)
    if reconstructed != cleaned:
        raise ValueError("Invalid format")

    total_seconds = 0
    seen_units = set()
    last_order = -1
    order_map = {'h': 0, 'm': 1, 's': 2}

    for num_str, unit in matches:
        unit_lower = unit.lower()

        # Unknown unit
        if unit_lower not in order_map:
            raise ValueError(f"Unknown unit: {unit}")

        # Repeated unit
        if unit_lower in seen_units:
            raise ValueError(f"Repeated unit: {unit}")

        # Order check (h < m < s)
        idx = order_map[unit_lower]
        if idx < last_order:
            raise ValueError("Units out of order")

        seen_units.add(unit_lower)
        last_order = idx

        num = int(num_str)
        if unit_lower == 'h':
            total_seconds += num * 3600
        elif unit_lower == 'm':
            total_seconds += num * 60
        else:  # 's'
            total_seconds += num

    return total_seconds
```
Coding: Summarize ranges Worked
```python
def summarize_ranges(nums: list[int]) -> str:
    # Remove duplicates and sort the numbers
    uniq = sorted(set(nums))
    if not uniq:
        return ""

    parts = []
    i = 0
    n = len(uniq)

    while i < n:
        start = uniq[i]
        # Extend the run while numbers are consecutive
        j = i + 1
        while j < n and uniq[j] == uniq[j - 1] + 1:
            j += 1

        run_len = j - i
        if run_len >= 3:
            # Collapse a long run into a..b format
            parts.append(f"{start}..{uniq[j - 1]}")
        else:
            # List each number individually for short runs
            for num in range(start, uniq[j - 1] + 1):
                parts.append(str(num))

        i = j

    return ",".join(parts)
```
Coding: Top customers Worked
```python
from typing import List, Dict, Tuple

def top_customers(rows: List[dict], n: int) -> List[Tuple[str, float]]:
    """
    Aggregate customer spending from a list of transaction rows.

    Parameters
    ----------
    rows : list[dict]
        Each dict must contain "customer" (str) and "amount" (str).
    n : int
        Number of top customers to return.

    Returns
    -------
    list[tuple[str, float]]
        Top ``n`` customers ordered by total spending descending,
        then by name ascending. Totals are rounded to two decimal places.
    """

    def parse_amount(amount_str: str) -> float | None:
        """Convert a monetary string like '$1,234.50' or '-$5.00' to a float."""
        if not isinstance(amount_str, str):
            return None
        s = amount_str.strip()
        if not s:
            return None

        # Detect sign
        negative = False
        if s.startswith('-'):
            negative = True
            s = s[1:]

        # Remove currency symbol if present
        if s.startswith('$'):
            s = s[1:]

        # Remove thousands separators
        s = s.replace(',', '')

        try:
            value = float(s)
        except ValueError:
            return None

        if negative:
            value = -value
        return value

    # Map lower‑cased name -> {original_name, total}
    customers: Dict[str, Dict[str, object]] = {}

    for row in rows:
        # Validate customer name
        raw_customer = row.get("customer")
        if raw_customer is None:
            continue
        cust = raw_customer.strip()
        if not cust:
            continue

        # Validate and parse amount
        amount_str = row.get("amount")
        if amount_str is None:
            continue
        amount = parse_amount(amount_str)
        if amount is None:
            continue

        # Group by case‑insensitive name, preserving the first trimmed spelling
        key = cust.lower()
        if key not in customers:
            customers[key] = {"original": cust, "total": amount}
        else:
            customers[key]["total"] = customers[key]["total"] + amount

    # Prepare result list with rounded totals
    result: List[Tuple[str, float]] = []
    for info in customers.values():
        total = round(info["total"], 2)
        result.append((info["original"], total))

    # Sort: total descending, then name ascending (case‑insensitive)
    result.sort(key=lambda x: (-x[1], x[0].lower()))

    # Return top n entries
    return result[:n]
```
Coding: Slugify Worked
```python
import re
import unicodedata

def slugify(title: str, max_len: int = 50) -> str:
    """
    Convert a title into a URL-friendly slug.

    1. Transliterate accented characters to ASCII and drop other non‑ASCII characters.
    2. Lowercase.
    3. Replace runs of characters other than a‑z or 0‑9 with a single hyphen,
       then strip leading/trailing hyphens.
    4. If the slug exceeds *max_len*, truncate without breaking a word:
       keep the longest prefix of whole words whose total length is ≤ *max_len*.
       If the first word itself is longer than *max_len*, hard‑cut it to *max_len*
       characters.
    5. The result never ends with a hyphen.
    """
    # 1. Transliterate and drop non‑ASCII
    normalized = unicodedata.normalize('NFKD', title)
    ascii_str = normalized.encode('ascii', 'ignore').decode('ascii')
    # 2. Lowercase
    ascii_str = ascii_str.lower()

    # 3. Replace non‑alphanumeric runs with a hyphen
    slug = re.sub(r'[^a-z0-9]+', '-', ascii_str)
    slug = slug.strip('-')

    if not slug:
        return ''

    # 4. Truncate if necessary
    if len(slug) <= max_len:
        final_slug = slug
    else:
        words = slug.split('-')
        # If the first word is longer than max_len, hard‑cut it
        if len(words[0]) > max_len:
            final_slug = words[0][:max_len]
        else:
            cur_words = [words[0]]
            cur_len = len(words[0])
            for w in words[1:]:
                # Length including the hyphen that will separate the words
                if cur_len + 1 + len(w) <= max_len:
                    cur_words.append(w)
                    cur_len += 1 + len(w)
                else:
                    break
            final_slug = '-'.join(cur_words)

    # 5. Ensure no leading/trailing hyphen
    final_slug = final_slug.strip('-')
    return final_slug
```
Coding: Token bucket Worked
```python
from typing import Callable


class TokenBucket:
    """
    A token bucket rate limiter.

    The bucket starts full with `capacity` tokens. Tokens are refilled at a
    constant rate of `refill_per_sec` tokens per second, never exceeding the
    capacity.
    """

    def __init__(
        self,
        capacity: float,
        refill_per_sec: float,
        clock: Callable[[], float],
    ) -> None:
        self.capacity = capacity
        self.refill_per_sec = refill_per_sec
        self._clock = clock
        # Bucket is initially full.
        self._tokens = capacity
        # Record the time of the last refill (initialized to the current time).
        self._last_time = self._clock()

    def _refill(self) -> None:
        """Refresh tokens based on elapsed time since the last refill."""
        now = self._clock()
        elapsed = now - self._last_time
        if elapsed > 0:
            # Tokens added during the elapsed interval.
            added = elapsed * self.refill_per_sec
            self._tokens = min(self._tokens + added, self.capacity)
        # Update the timestamp for future refills.
        self._last_time = now

    def allow(self, cost: float = 1) -> bool:
        """
        Attempt to consume `cost` tokens.

        Returns True if enough tokens were available and were deducted,
        otherwise False and no tokens are removed.
        """
        self._refill()
        if self._tokens >= cost:
            self._tokens -= cost
            return True
        return False

    @property
    def tokens(self) -> float:
        """
        Refill and return the current number of tokens.
        """
        self._refill()
        return self._tokens
```
Decisions: Refund window (hard) 100% right
{
  "outcome": {
    "full_refund": 0,
    "partial_refund": 0,
    "store_credit": 1,
    "nothing": 0
  },
  "defective": {
    "true": 0,
    "false": 1
  }
}
Decisions: Incident matrix (hard) 100% right
{"severity": {"0": 0, "1": 1, "2": 0, "3": 0}, "page": {"true": 0, "false": 1}}
Decisions: Tool followup (hard) 50% right
{
  "tool": {
    "web_search": 0.0,
    "calculator": 0.0,
    "calendar": 1.0,
    "email": 0.0,
    "none": 0.0
  },
  "confirm": {
    "true": 0.0,
    "false": 1.0
  }
}
Decisions: Legit security alert (hard) 100% right
{
  "phishing": {"true": 0.01, "false": 0.99},
  "action_needed": {"true": 0.02, "false": 0.98}
}
Decisions: Meeting slot (hard) 100% right
{
  "slot": {
    "A": 0,
    "B": 1,
    "C": 0,
    "D": 0
  },
  "raj_last": {
    "true": 1,
    "false": 0
  }
}
Decisions: Review mixed (hard) 100% right
{
  "hardware": {"true": 0.98, "false": 0.02},
  "support": {"true": 0.97, "false": 0.03}
}
Decisions: Support checkout down 100% right
{
  "department": {
    "billing": 0.1,
    "technical": 0.9,
    "account": 0.0,
    "shipping": 0.0,
    "sales": 0.0
  },
  "urgency": {
    "0": 0.0,
    "1": 0.0,
    "2": 0.0,
    "3": 1.0
  },
  "outage": {
    "true": 1.0,
    "false": 0.0
  }
}
Decisions: Refund wrong plan 100% right
{
  "department": {
    "billing": 1.0,
    "technical": 0.0,
    "account": 0.0,
    "shipping": 0.0,
    "sales": 0.0
  },
  "refund": {
    "true": 1.0,
    "false": 0.0
  },
  "tone": {
    "frustrated": 0.0,
    "calm": 1.0
  }
}
Decisions: Moderation doxxing 100% right
{
  "policy": {
    "none": 0.0,
    "harassment": 1.0,
    "hate": 0.0,
    "spam": 0.0,
    "self_harm": 0.0
  },
  "personal_info": {
    "true": 1.0,
    "false": 0.0
  }
}
Decisions: Route calendar 100% right
{
  "tool": {
    "web_search": 0.02,
    "calculator": 0.01,
    "calendar": 0.95,
    "email": 0.01,
    "none": 0.01
  },
  "confirm": {
    "true": 1.0,
    "false": 0.0
  }
}
Decisions: Doc invoice missing due 100% right
{
  "doc_type": {
    "invoice": 1.0,
    "resume": 0.0,
    "contract": 0.0,
    "bank_statement": 0.0,
    "other": 0.0
  },
  "missing_due_date": {
    "true": 1.0,
    "false": 0.0
  }
}
Decisions: Phishing paypal 100% right
{
  "phishing": {
    "true": 0.99,
    "false": 0.01
  },
  "risk": {
    "0": 0.0,
    "1": 0.0,
    "2": 0.0,
    "3": 1.0
  }
}
Decisions: Pii ssn email 100% right
{
  "data_kind": {
    "none": 0.0,
    "contact": 0.0,
    "financial": 0.0,
    "government_id": 1.0,
    "health": 0.0
  },
  "sensitive": {
    "true": 1.0,
    "false": 0.0
  }
}
Decisions: Review mixed 100% right
{
  "sentiment": {
    "positive": 0.1,
    "neutral": 0.05,
    "negative": 0.85
  },
  "defect": {
    "true": 1.0,
    "false": 0.0
  },
  "recommend": {
    "true": 0.0,
    "false": 1.0
  }
}
Documents: Saas escalator (hard) 100% right
{
  "year2_price_per_seat_month": 47.25,
  "year3_price_per_seat_month": 47.25,
  "year1_invoice": 58320.00,
  "year2_invoice": 61236.00,
  "addon_months_billed": 6,
  "addon_invoice": 38556.00,
  "year3_invoice": 134946.00,
  "year3_discount_percent": 15.0,
  "total_contract_value": 293058.00,
  "contract_end_date": "2027-02-28"
}
Documents: Expense thread 92% right
{
  "employee_id": "EMP-20417",
  "destination_city": "Lisbon",
  "trip_start": "2025-02-24",
  "trip_end": "2025-02-27",
  "approved_items": [
    {
      "date": "2025-02-24",
      "category": "airfare",
      "amount_usd": 1184.60
    },
    {
      "date": "2025-02-24",
      "category": "ground_transport",
      "amount_usd": 38.88
    },
    {
      "date": "2025-02-25",
      "category": "meals",
      "amount_usd": 229.31
    },
    {
      "date": "2025-02-26",
      "category": "lodging",
      "amount_usd": 466.56
    },
    {
      "date": "2025-02-27",
      "category": "ground_transport",
      "amount_usd": 44.82
    }
  ],
  "rejected_item_count": 1,
  "per_diem_days": 3,
  "per_diem_usd": 195.0,
  "total_reimbursable_usd": 2159.17,
  "approver_email": "priya.raman@corvane.com"
}
Documents: Lease amendment 100% right
{
  "tenants": ["Marcus Lin", "Sofia Lin"],
  "landlord": "Ridgeline Property Group LLC",
  "zip": "97205",
  "lease_end": "2025-11-30",
  "original_monthly_rent": 2150,
  "monthly_rent_from_2025_06_01": 2236,
  "late_fee_from_2025_06_01": 111.80,
  "security_deposit": 2150,
  "total_pet_deposits": 800,
  "total_monthly_payment_july_2025": 2306,
  "move_in_payment": 4700
}
Documents: Ticket SLA 92% right
{
  "ticket_id": "48213",
  "account_id": "ACC-7731",
  "open_issue": "inventory_sync",
  "resolved_issues": ["billing_address", "invoice_pdf"],
  "affected_orders": ["SO-99812", "SO-99820", "SO-99827"],
  "priority": "P2",
  "sla_due_local": "2025-09-15T15:30",
  "sla_due_utc": "2025-09-15T20:30:00Z",
  "reissued_invoice": "INV-2025-0812"
}
Documents: Sales footnotes 100% right
{
  "q3_total_usd": 15346000,
  "q2_total_usd": 14464000,
  "q2_central_originally_reported_usd": 3047000,
  "q2_to_q3_change_pct": 6.1,
  "top_region_q3": "East",
  "fastest_growing_region_q1_to_q3": "International",
  "regions_declining_q2_to_q3": ["East"],
  "international_q3_organic_usd": 1731000,
  "west_excluding_mountain_q3_usd": 4201000
}

Size: 24B parameters. First tested OCT 14.

Models that scored about the same

Comments

Sign in with GitHub to comment. Spam and abuse are hidden automatically.