Review · updated OCT 11

Nemotron 3.5 Lightning review: not one we'd recommend right now

It scored 57 out of 100, #26 of 56. It solved 13 of 30 coding jobs and scored 72 on reading documents. Runs on a 24 GB graphics card or a Mac with 32 GB.

The short version
  • Nemotron 3.5 Lightning is a free model from NVIDIA that you can run on your own computer. In our tests it's not one we'd recommend right now: 57 out of 100, #26 of 56.
  • It solved 13 of 30 coding jobs and scored 72 on reading documents. On our hardest tasks it scored 43.
  • Runs on a 24 GB graphics card or a Mac with 32 GB.

Coding

Our coding test is 30 programming jobs, from small ones like reading time durations or cleaning up messy data to harder ones like a config-file parser or a double-entry ledger. We run each answer against tests the model never sees, and a job only counts if everything passes. Nemotron 3.5 Lightning got 13 of 30 right. The best local coders solved 29 of 30.

Reading documents

The second test hands the model things like an expense claim thread, a pay stub or an insurance statement, and asks for specific numbers and dates. Many questions need a bit of math, or noticing a correction further down the email. Nemotron 3.5 Lightning scored 72; the best model scored 100.

TestScorePublic questionsSecret questions
Coding434344
Reading documents727670
Decisions808977

On the 18 hardest tasks (included in the scores above) it scored 43. This number separates the top models.

We tested the full-size version online. The smaller versions you download usually score a little lower.

Can your computer run it?

You can download Nemotron 3.5 Lightning in several sizes. Smaller versions fit on more computers and lose a little quality, so pick the biggest one that fits.

Your computerRuns it?Version to download
Graphics card, 8 GBNo–
Graphics card, 12 GBNo–
Graphics card, 16 GBNo–
Graphics card, 24 GBYesQ5
Graphics card, 32 GBYesQ6
Mac, 16 GBNo–
Mac, 24 GBNo–
Mac, 32 GBYesQ4
Mac, 48 GBYesQ8
Mac, 64 GBYesQ8
Mac, 96 GBYesQ8
Mac, 128 GBYesQ8

For normal-length chats. Long documents need more memory. Q4 or Q8? · How we worked this out

How to try it

Search for "Nemotron 3.5 Lightning GGUF" in LM Studio or on Hugging Face and pick the version that fits your computer. New to this? Start with our Ollama guide.

Want the strongest all-rounder instead? Qwen3.6 27B scored 97.

Its actual answers

What Nemotron 3.5 Lightning wrote for our public questions, word for word.

Coding: INI parser (hard) Didn't work
Coding: TTL LRU (hard) Didn't work
```python
from __future__ import annotations
from typing import Any


class TTLCache:
    """LRU cache with per-entry TTL expiration.

    Args:
        capacity: Maximum number of entries. If 0 the cache never stores anything.
        ttl: Time-to-live in seconds; an entry expires when ``now >= expiry``.
        clock: Zero-argument callable returning the current time in seconds.
    """

    def __init__(self, capacity: int, ttl: float, clock: Any) -> None:
        if capacity < 0:
            raise ValueError("capacity must be non-negative")
        if ttl < 0:
            raise ValueError("ttl must be non-negative")
        self._capacity = capacity
        self._ttl = ttl
        self._clock = clock
        # Ordered dict preserves insertion/access order for LRU.
        # Each value is a tuple (stored_value, expiry_time).
        self._data: dict[str, tuple[Any, float]] = {}
        # Stats counters.
        self._hits = 0
        self._misses = 0
        self._evictions = 0
        self._expirations = 0

    # ------------------------------------------------------------------
    # Helper: purge all entries that have expired as of the current clock.
    # ------------------------------------------------------------------
    def _expire_stale(self) -> None:
        now = self._clock()
        # Remove every entry whose expiry time has been reached.
        expired_keys = [k for k, (_, exp) in self._data.items() if exp <= now]
        for k in expired_keys:
            self._expirations += 1
            del self._data[k]

    # ------------------------------------------------------------------
    # put
    # ------------------------------------------------------------------
    def put(self, key: str, value: Any) -> None:
        # capacity 0: nothing is stored.
        if self._capacity == 0:
            return

        now = self._clock()

        # If the key already exists, refresh it and move it to the front
        # (most-recently used).  We treat it as an update rather than an
        # insertion for capacity purposes.
        if key in self._data:
            # Bump the expiry to now + ttl.
            self._data[key] = (value, now + self._ttl)
            # Move the key to the end of the ordered dict (most recent).
            # OrderedDict.move_to_end is O(1).
            self._data.move_to_end(key)
            return

        # Key is new – first purge expired entries to free space.
        self._expire_stale()

        # If we are still at capacity, evict the least recently used entry.
        if len(self._data) >= self._capacity:
            # popitem(last=False) removes the LRU (first) item.
            lru_key, _ = self._data.popitem(last=False)
            self._evictions += 1

        # Insert the new entry with expiry now + ttl.
        self._data[key] = (value, now + self._ttl)
        # It is already at the end (most recent) because we inserted after
        # evicting, but we explicitly move it to be safe.
        self._data.move_to_end(key)

    # ------------------------------------------------------------------
    # get
    # ------------------------------------------------------------------
    def get(self, key: str, default: Any = None) -> Any:
        if self._capacity == 0:
            self._misses += 1
            return default

        now = self._clock()

        if key not in self._data:
            self._misses += 1
            return default

        value, expiry = self._data[key]

        # Entry expired?
        if now >= expiry:
            # Remove it, count as an expiration AND a miss.
            self._expirations += 1
            del self._data[key]
            self._misses += 1
            return default

        # Entry is valid – promote to most-recently used.
        self._data.move_to_end(key)
        self._hits += 1
        return value

    # ------------------------------------------------------------------
    # __len__
    # ------------------------------------------------------------------
    def __len__(self) -> int:
        # Count only non‑expired entries.
        now = self._clock()
        count = 0
        for _key, (_, exp) in self._data.items():
            if exp > now:
                count += 1
        return count

    # ------------------------------------------------------------------
    # stats
    # ------------------------------------------------------------------
    def stats(self) -> dict[str, int]:
        return {
            "hits": self._hits,
            "misses": self._misses,
            "evictions": self._evictions,
            "expirations": self._expirations,
        }
```
Coding: Parse duration Didn't work
Coding: Summarize ranges Worked
```python
def summarize_ranges(nums: list[int]) -> str:
    if not nums:
        return ""

    # Sort and remove duplicates
    sorted_nums = sorted(set(nums))

    # Group into runs of consecutive integers
    groups: list[list[int]] = []
    current: list[int] = [sorted_nums[0]]

    for n in sorted_nums[1:]:
        if n == current[-1] + 1:
            current.append(n)
        else:
            groups.append(current)
            current = [n]
    groups.append(current)

    # Build the result string
    parts: list[str] = []
    for g in groups:
        if len(g) >= 3:
            parts.append(f"{g[0]}..{g[-1]}")
        else:
            parts.append(",".join(str(x) for x in g))

    return ",".join(parts)
```
Coding: Top customers Didn't work
```python

import re
from collections import OrderedDict
from typing import Any

def top_customers(rows: list[dict[str, Any]], n: int) -> list[tuple[str, float]]:
    """
    Return top n customers as (name, total) tuples sorted by total descending,
    then name ascending.

    - Customer names are trimmed and matched case-insensitively; the trimmed
      spelling of the first valid row for each customer is used.
    - Rows with an empty customer name or an unparsable amount are skipped.
    - Amounts are strings such as "$1,234.50", "1234.5", "-$5.00" (negative = refund).
    - Totals are rounded to 2 decimal places.
    """
    # Normalise: strip whitespace from name/amount and keep first occurrence per normalized name
    customers: dict[str, tuple[str, float]] = {}  # key -> (display_name, total)

    for row in rows:
        # Guard against missing keys
        name_raw = row.get("customer", "")
        amount_raw = row.get("amount", "")

        # Trim the name
        name = name_raw.strip()
        if not name:
            continue  # skip empty customer name

        # Normalise name for matching: lower + stripped
        name_key = name.lower().strip()

        # Parse amount
        amount_str = amount_raw.strip()
        if not amount_str:
            continue  # skip unparsable amount

        # Remove surrounding whitespace, then strip leading $ if present
        # The regex below expects optional leading minus, optional $, digits, optional comma, optional decimal
        # We'll just attempt a conversion after cleaning.
        # Remove commas for float parsing
        cleaned = amount_str.replace(",", "")

        # Attempt float conversion; if it fails, skip row
        try:
            value = float(cleaned)
        except ValueError:
            continue

        # Ensure we actually have a numeric result (e.g., string was just "$")
        # float() on "$12.3" would raise, but we guard above.

        # Determine if this is the first time we see this customer
        if name_key not in customers:
            # Store the trimmed spelling of the first valid row
            customers[name_key] = (name, value)
        else:
            # Add to existing total
            existing_name, existing_total = customers[name_key]
            new_total = round(existing_total + value, 2)
            customers[name_key] = (existing_name, new_total)

    # Build list of (name, total) tuples
    result: list[tuple[str, float]] = [
        (display_name, total) for display_name, total in customers.values()
    ]

    # Sort by total descending, then name ascending (case‑insensitive but using stored name)
    result.sort(key=lambda x: (-x[1], x[0].lower()))

    # Return top n (or fewer if not enough customers)
    return result[:n]

```
Coding: Slugify Worked
```python
import unicodedata
import re


def slugify(title: str, max_len: int = 50) -> str:
    # Step 1: Transliterate accented characters to ASCII and drop other non-ASCII.
    normalized = unicodedata.normalize("NFKD", title)
    ascii_only = normalized.encode("ascii", "ignore").decode("ascii")

    # Step 2: Lowercase.
    lowered = ascii_only.lower()

    # Step 3: Replace every run of non-alphanumeric characters with a single "-"
    # and strip leading/trailing "-".
    replaced = re.sub(r'[^a-z0-9]+', '-', lowered)
    stripped = replaced.strip('-')

    # If the result is empty, return an empty string.
    if not stripped:
        return ''

    # Step 4: If longer than max_len, shorten without cutting a word.
    if len(stripped) > max_len:
        words = stripped.split('-')
        # Collect whole words from the start whose total length (including separators)
        # does not exceed max_len.
        selected = []
        current_len = 0
        for word in words:
            # Adding a word: if we already have words, we need a '-' separator.
            added_len = len(word) + (1 if selected else 0)
            if current_len + added_len <= max_len:
                selected.append(word)
                current_len += added_len
            else:
                break

        if selected:
            slug = '-'.join(selected)
        else:
            # Even the first word is too long; hard-cut it to max_len characters.
            slug = stripped[:max_len]
    else:
        slug = stripped

    # Step 5: The result never ends with "-".
    slug = slug.rstrip('-')

    return slug
```
Coding: Token bucket Worked
```python
class TokenBucket:
    def __init__(self, capacity: float, refill_per_sec: float, clock) -> None:
        self._capacity = capacity
        self._refill_per_sec = refill_per_sec
        self._clock = clock
        self._tokens = capacity
        self._last_refill = clock()

    def _refill(self) -> None:
        now = self._clock()
        elapsed = now - self._last_refill
        if elapsed > 0:
            self._tokens = min(self._capacity, self._tokens + elapsed * self._refill_per_sec)
            self._last_refill = now

    def allow(self, cost: float = 1) -> bool:
        self._refill()
        if self._tokens >= cost:
            self._tokens -= cost
            return True
        return False

    @property
    def tokens(self) -> float:
        self._refill()
        return self._tokens
```
Decisions: Refund window (hard) 100% right
{
  "outcome": {
    "full_refund": 0,
    "partial_refund": 0,
    "store_credit": 1,
    "nothing": 0
  },
  "defective": {
    "true": 0,
    "false": 1
  }
}
Decisions: Incident matrix (hard) 50% right
{
  "severity": {
    "0": 0,
    "1": 0,
    "2": 0.1,
    "3": 0.9
  },
  "page": {
    "true": 0,
    "false": 1
  }
}
Decisions: Tool followup (hard) 0% right
{
  "tool": "calendar",
  "confirm": false
}
Decisions: Legit security alert (hard) 100% right
{
  "phishing": {"true": 0.001, "false": 0.999},
  "action_needed": {"true": 0.001, "false": 0.999}
}
Decisions: Meeting slot (hard) 100% right
{
  "slot": {"A": 0, "B": 1, "C": 0, "D": 0},
  "raj_last": {"true": 1, "false": 0}
}
Decisions: Review mixed (hard) 100% right
{
  "hardware": {"true": 0.78, "false": 0.22},
  "support": {"true": 0.92, "false": 0.08}
}
Decisions: Support checkout down 100% right
{
  "department": {
    "technical": 0.8,
    "billing": 0.15,
    "account": 0.02,
    "shipping": 0.02,
    "sales": 0.01
  },
  "urgency": {
    "3": 0.95,
    "2": 0.05,
    "1": 0,
    "0": 0
  },
  "outage": {
    "true": 1.0,
    "false": 0.0
  }
}
Decisions: Refund wrong plan 100% right
{
  "department": {"billing": 1, "technical": 0, "account": 0, "shipping": 0, "sales": 0},
  "refund": {"true": 1, "false": 0},
  "tone": {"frustrated": 0, "calm": 1}
}
Decisions: Moderation doxxing 100% right
{"policy":{"none":0.0,"harassment":1.0,"hate":0.0,"spam":0.0,"self_harm":0.0},"personal_info":{"true":1.0,"false":0.0}}
Decisions: Route calendar 100% right
{"tool":{"web_search":0,"calculator":0,"calendar":1,"email":0,"none":0},"confirm":{"true":1,"false":0}}
Decisions: Doc invoice missing due 100% right
{
  "doc_type": {
    "invoice": 1.0,
    "resume": 0.0,
    "contract": 0.0,
    "bank_statement": 0.0,
    "other": 0.0
  },
  "missing_due_date": {
    "true": 1.0,
    "false": 0.0
  }
}
Decisions: Phishing paypal 100% right
{
  "phishing": {
    "false": 0.05,
    "true": 0.95
  },
  "risk": {
    "0": 0.02,
    "1": 0.03,
    "2": 0.15,
    "3": 0.8
  }
}
Decisions: Pii ssn email 100% right
{
  "data_kind": {
    "government_id": 1.0
  },
  "sensitive": {
    "true": 1.0
  }
}
Decisions: Review mixed 100% right
{
  "sentiment": {
    "positive": 0.1,
    "neutral": 0.2,
    "negative": 0.7
  },
  "defect": {
    "true": 0.95,
    "false": 0.05
  },
  "recommend": {
    "true": 0.1,
    "false": 0.9
  }
}
Documents: Saas escalator (hard) 18% right
{
  "year1_price_per_seat_month": 45.00,
  "year2_price_per_seat_month": 46.53,
  "year3_price_per_seat_month": 47.41,
  "year1_invoice": 5400.00,
  "year2_invoice": 5425.48,
  "addon_months_billed": 6,
  "addon_invoice": 7308.84,
  "year3_invoice": 5945.85,
  "year3_discount_percent": 10.00,
  "total_contract_value": 22515.14,
  "contract_end_date": "2027-02-28"
}
Documents: Expense thread 79% right
{
  "employee_id": "EMP-20417",
  "destination_city": "Lisbon",
  "trip_start": "2025-02-24",
  "trip_end": "2025-02-27",
  "approved_items": [
    {
      "date": "2025-02-24",
      "category": "airfare",
      "amount_usd": 1184.6
    },
    {
      "date": "2025-02-24",
      "category": "ground_transport",
      "amount_usd": 38.89
    },
    {
      "date": "2025-02-25",
      "category": "meals",
      "amount_usd": 228.93
    },
    {
      "date": "2025-02-26",
      "category": "lodging",
      "amount_usd": 461.76
    },
    {
      "date": "2025-02-27",
      "category": "ground_transport",
      "amount_usd": 44.67
    }
  ],
  "rejected_item_count": 1,
  "per_diem_days": 3,
  "per_diem_usd": 195,
  "total_reimbursable_usd": 2583.84,
  "approver_email": "priya.raman@corvane.com"
}
Documents: Lease amendment 83% right
{
  "tenants": ["Marcus Lin", "Sofia Lin"],
  "landlord": "Ridgeline Property Group LLC",
  "zip": "97205",
  "lease_end": "2025-11-30",
  "original_monthly_rent": 2150.00,
  "monthly_rent_from_2025_06_01": 2236.00,
  "late_fee_from_2025_06_01": 111.80,
  "security_deposit": 2150.00,
  "total_pet_deposits": 800.00,
  "total_monthly_payment_july_2025": 2607.00,
  "move_in_payment": 3500.00
}
Documents: Ticket SLA 100% right
{
  "ticket_id": "48213",
  "account_id": "ACC-7731",
  "open_issue": "inventory_sync",
  "resolved_issues": ["billing_address"],
  "affected_orders": ["SO-99812", "SO-99820", "SO-99827"],
  "priority": "P2",
  "sla_due_local": "2025-09-15T15:30",
  "sla_due_utc": "2025-09-15T20:30:00Z",
  "reissued_invoice": "INV-2025-0812"
}
Documents: Sales footnotes 100% right
{
  "q3_total_usd": 15346000,
  "q2_total_usd": 14464000,
  "q2_central_originally_reported_usd": 3047000,
  "q2_to_q3_change_pct": 6.1,
  "top_region_q3": "East",
  "fastest_growing_region_q1_to_q3": "International",
  "regions_declining_q2_to_q3": ["East"],
  "international_q3_organic_usd": 1731000,
  "west_excluding_mountain_q3_usd": 4201000
}

Size: 32B parameters. First tested OCT 10.

Models that scored about the same

Comments

Sign in with GitHub to comment. Spam and abuse are hidden automatically.