Review · updated OCT 12

Llama 4 Scout review: not one we'd recommend right now

It scored 34 out of 100, #49 of 63. It solved 9 of 30 coding jobs and scored 38 on reading documents. Runs on a Mac with 96 GB.

The short version
  • Llama 4 Scout is a free model from Meta that you can run on your own computer. In our tests it's not one we'd recommend right now: 34 out of 100, #49 of 63.
  • It solved 9 of 30 coding jobs and scored 38 on reading documents. On our hardest tasks it scored 12.
  • Runs on a Mac with 96 GB.

Coding

Our coding test is 30 programming jobs, from small ones like reading time durations or cleaning up messy data to harder ones like a config-file parser or a double-entry ledger. We run each answer against tests the model never sees, and a job only counts if everything passes. Llama 4 Scout got 9 of 30 right. The best local coders solved 29 of 30.

Reading documents

The second test hands the model things like an expense claim thread, a pay stub or an insurance statement, and asks for specific numbers and dates. Many questions need a bit of math, or noticing a correction further down the email. Llama 4 Scout scored 38; the best model scored 100.

TestScorePublic questionsSecret questions
Coding302930
Reading documents383938
Decisions818380

On the 18 hardest tasks (included in the scores above) it scored 12. This number separates the top models.

We tested the full-size version online. The smaller versions you download usually score a little lower.

Can your computer run it?

You can download Llama 4 Scout in several sizes. Smaller versions fit on more computers and lose a little quality, so pick the biggest one that fits.

Your computerRuns it?Version to download
Graphics card, 8 GBNo–
Graphics card, 12 GBNo–
Graphics card, 16 GBNo–
Graphics card, 24 GBNo–
Graphics card, 32 GBNo–
Mac, 16 GBNo–
Mac, 24 GBNo–
Mac, 32 GBNo–
Mac, 48 GBNo–
Mac, 64 GBNo–
Mac, 96 GBYesQ4
Mac, 128 GBYesQ6

For normal-length chats. Long documents need more memory. Q4 or Q8? · How we worked this out

How to try it

Search for "Llama 4 Scout GGUF" in LM Studio or on Hugging Face and pick the version that fits your computer. New to this? Start with our Ollama guide.

Want the strongest all-rounder instead? Qwen3.6 27B scored 97.

Its actual answers

What Llama 4 Scout wrote for our public questions, word for word.

Coding: INI parser (hard) Didn't work
```python
import re
from typing import Dict

def parse_ini(text: str) -> Dict[str, Dict[str, str]]:
    """
    Parse an INI file and return a dictionary of sections with their keys and values.

    :param text: The content of the INI file.
    :return: A dictionary where each key is a section name and each value is another dictionary.
             The inner dictionary maps keys to values.
    """

    config: Dict[str, Dict[str, str]] = {}
    current_section: str | None = None
    stack: list[tuple[str, str | None]] = []

    for line in text.splitlines():
        line = line.rstrip()
        if not line or line.lstrip().startswith(('#', ';')):
            continue

        # Section header
        if line.startswith('[') and line.endswith(']'):
            name = line[1:-1].strip()
            if name in config:
                raise ValueError(f"Section {name!r} already exists")
            config[name] = {}
            current_section = name
            continue

        # Key line
        match = re.match(r'\s*([^\s=:]*)(\s*[:=])\s*(.*)', line)
        if match:
            key, _, value = match.groups()
            key = key.strip().lower()
            if current_section is None:
                raise ValueError("Key before section header")

            # Continuation line
            if key == '':
                if current_section is None:
                    raise ValueError("Continuation line before section header")
                if not config[current_section]:
                    raise ValueError("Continuation line without preceding key")
                config[current_section][list(config[current_section].keys())[-1]] += '\n' + value.strip()
                continue

            # Inline comment
            comment_match = re.search(r'\s*[;#](.*)', value)
            if comment_match:
                value = value[:comment_match.start()].strip()

            # Quoted value
            if value.startswith('"') and value.endswith('"'):
                value = value[1:-1].replace('\\"', '"')

            # Interpolation
            value = interpolate(config, current_section, key, value)

            config[current_section][key] = value
            continue

        raise ValueError(f"Malformed line: {line!r}")

    return config


def interpolate(config: Dict[str, Dict[str, str]], current_section: str | None, key: str, value: str) -> str:
    """
    Perform interpolation on a value.

    :param config: The configuration dictionary.
    :param current_section: The current section.
    :param key: The key.
    :param value: The value.
    :return: The interpolated value.
    """

    def replace(match: re.Match[str]) -> str:
        full_match = match.group(0)
        section, key = (match.group(1), match.group(2)) if match.group(1) else (current_section, match.group(2))
        if section is None:
            raise KeyError(f"Missing section in interpolation: {full_match!r}")
        if key not in config[section]:
            if section == 'DEFAULT':
                raise KeyError(f"Missing key {key!r} in DEFAULT section")
            if section not in config:
                raise KeyError(f"Missing section {section!r}")
            if key not in config[section]:
                raise KeyError(f"Missing key {key!r} in section {section!r}")
        return config[section][key]

    # Detect reference cycle
    stack.append((current_section, key))
    try:
        value = re.sub(r'\${(.*?)}', lambda match: replace(match), value)
    finally:
        stack.pop()

    if len(stack) > 1 and (stack[-1][0], stack[-1][1]) == stack[-2]:
        raise ValueError("Interpolation reference cycle detected")

    return value
```
Coding: TTL LRU (hard) Didn't work
```python
from collections import OrderedDict
from typing import Callable, Dict, Optional

class TTLCache:
    def __init__(self, capacity: int, ttl: float, clock: Callable[[], float]):
        self.capacity = capacity
        self.ttl = ttl
        self.clock = clock
        self.cache: OrderedDict[str, (float, object)] = OrderedDict()
        self.stats = {"hits": 0, "misses": 0, "evictions": 0, "expirations": 0}

    def _remove_expired(self) -> None:
        now = self.clock()
        while self.cache and self.cache[next(iter(self.cache))] [0] <= now:
            key, _ = self.cache.popitem(last=False)
            self.stats["expirations"] += 1

    def _evict(self) -> None:
        key, _ = self.cache.popitem(last=False)
        self.stats["evictions"] += 1

    def put(self, key: str, value: object) -> None:
        if self.capacity == 0:
            return
        now = self.clock()
        self._remove_expired()
        if len(self.cache) >= self.capacity:
            self._evict()
        self.cache[key] = (now + self.ttl, value)

    def get(self, key: str, default: Optional[object] = None) -> Optional[object]:
        if self.capacity == 0:
            self.stats["misses"] += 1
            return default
        now = self.clock()
        self._remove_expired()
        if key in self.cache:
            expiry, value = self.cache.pop(key)
            if expiry > now:
                self.cache[key] = (expiry, value)
                self.stats["hits"] += 1
                return value
            else:
                self.stats["expirations"] += 1
        self.stats["misses"] += 1
        return default

    def __len__(self) -> int:
        now = self.clock()
        return sum(1 for expiry, _ in self.cache.values() if expiry > now)

    def stats(self) -> Dict[str, int]:
        return self.stats.copy()
```
Coding: Parse duration Didn't work
```python
import re
from typing import Dict

def parse_duration(s: str) -> int:
    """
    Converts a duration string into a total number of seconds.

    Args:
    s (str): A duration string.

    Returns:
    int: The total number of seconds.

    Raises:
    ValueError: If the input string is empty/blank, or if it contains invalid parts.
    """

    # Remove leading and trailing whitespace
    s = s.strip()

    # Check for empty/blank strings
    if not s:
        raise ValueError("Input string is empty")

    # Regular expression pattern to match a duration part
    pattern = r"(\d+)([hms])"
    parts = re.findall(pattern, s)

    # Check for invalid parts
    if len(parts) != len(set(part[1] for part in parts)):
        raise ValueError("Repeated units are not allowed")

    # Mapping of units to their conversion factors
    units: Dict[str, int] = {"h": 3600, "m": 60, "s": 1}

    # Check for unknown units and units out of order
    previous_unit = None
    total_seconds = 0
    for num, unit in parts:
        num = int(num)
        if unit.lower() not in units:
            raise ValueError(f"Unknown unit: {unit}")
        if previous_unit and units[unit.lower()] <= units[previous_unit]:
            raise ValueError("Units must appear in the order h, m, s")
        total_seconds += num * units[unit.lower()]
        previous_unit = unit.lower()

    return total_seconds
```
Coding: Summarize ranges Worked
```python
def summarize_ranges(nums: list[int]) -> str:
    """
    This function takes a list of integers, removes duplicates, sorts them, 
    and then collapses runs of 3 or more consecutive integers into "a..b". 
    Numbers in shorter runs (1 or 2 numbers) are listed individually.

    Args:
        nums (list[int]): A list of integers.

    Returns:
        str: A string representing the summarized ranges of integers.
    """

    # Remove duplicates and sort the list
    nums = sorted(set(nums))

    # Initialize variables to store the result and the start of the current range
    result = []
    range_start = None

    # Iterate over the sorted list of numbers
    for i, num in enumerate(nums):
        # If this is the first number or it's not consecutive to the previous number,
        # it starts a new range
        if i == 0 or num != nums[i - 1] + 1:
            # If we've been building a range, add it to the result
            if range_start is not None:
                if nums[i - 1] == range_start:
                    result.append(str(range_start))
                elif nums[i - 1] == range_start + 1:
                    result.extend([str(range_start), str(range_start + 1)])
                else:
                    result.append(f"{range_start}..{nums[i - 1]}")
                range_start = None
            # Start a new range
            range_start = num

    # Add the last range to the result
    if range_start is not None:
        if nums[-1] == range_start:
            result.append(str(range_start))
        elif nums[-1] == range_start + 1:
            result.extend([str(range_start), str(range_start + 1)])
        else:
            result.append(f"{range_start}..{nums[-1]}")

    # Join the result with commas and return it
    return ",".join(result)
```
Coding: Top customers Didn't work
```python
from typing import List, Dict, Tuple
from collections import defaultdict
import re

def top_customers(rows: List[Dict], n: int) -> List[Tuple[str, float]]:
    """
    Returns the top n customers by total amount.

    :param rows: A list of dictionaries, each containing a customer name and an amount.
    :param n: The number of top customers to return.
    :return: A list of tuples, each containing a customer name and their total amount.
    """

    # Create a dictionary to store the total amount for each customer
    customer_amounts = defaultdict(float)

    # Regular expression to match and extract the amount from a string
    amount_regex = re.compile(r'^-?\$?(\d{1,3}(?:,\d{3})*)(?:\.\d+)?$')

    # Iterate over each row
    for row in rows:
        # Get the customer name and amount from the row
        customer = row.get("customer", "").strip()
        amount_str = row.get("amount", "")

        # Skip rows with empty customer name or amount
        if not customer or not amount_str:
            continue

        # Try to parse the amount
        match = amount_regex.match(amount_str)
        if match:
            # Remove commas and convert to float
            amount = float(match.group(1).replace(",", ""))
            # Update the customer's total amount
            customer_amounts[customer] += amount

    # Round totals to 2 decimals and store in a list of tuples
    customer_amounts = [(customer, round(amount, 2)) for customer, amount in customer_amounts.items()]

    # Sort the customers by total amount descending, then name ascending
    customer_amounts.sort(key=lambda x: (-x[1], x[0]))

    # Return the top n customers
    return customer_amounts[:n]
```
Coding: Slugify Didn't work
```python
import unicodedata
import re

def slugify(title: str, max_len: int = 50) -> str:
    """
    Builds a URL slug.

    1. Transliterate accented characters to ASCII (é -> e, ü -> u) and drop any other non-ASCII characters.
    2. Lowercase.
    3. Replace every run of characters other than a-z and 0-9 with a single "-", and strip leading/trailing "-".
    4. If the slug is longer than `max_len`, shorten it without cutting a word: keep the longest prefix of whole words 
       (words are separated by "-") whose length is <= max_len. If even the first word is longer than max_len, 
       hard-cut it to max_len characters.
    5. The result never ends with "-".

    Args:
        title (str): The title to slugify.
        max_len (int, optional): The maximum length of the slug. Defaults to 50.

    Returns:
        str: The slugified title.
    """

    # Transliterate accented characters to ASCII and drop any other non-ASCII characters
    title = ''.join(c for c in unicodedata.normalize('NFD', title)
                    if unicodedata.category(c) != 'Mn' and c.isascii())

    # Lowercase
    title = title.lower()

    # Replace every run of characters other than a-z and 0-9 with a single "-", and strip leading/trailing "-"
    title = re.sub(r'[^\w]+', '-', title).strip('-')

    # If the slug is longer than `max_len`, shorten it without cutting a word
    if len(title) > max_len:
        words = title.split('-')
        prefix_len = 0
        for word in words:
            if prefix_len + len(word) + 1 > max_len:
                break
            prefix_len += len(word) + 1
        title = '-'.join(words[:len(words) - 1] + [words[-1][:max_len - prefix_len]]) if prefix_len < max_len else title[:max_len]

    return title
```
Coding: Token bucket Worked
```python
from typing import Callable

class TokenBucket:
    def __init__(self, capacity: float, refill_per_sec: float, clock: Callable[[], float]):
        self.capacity = capacity
        self.refill_per_sec = refill_per_sec
        self.clock = clock
        self.tokens = capacity
        self.last_refill = clock()

    @property
    def tokens(self) -> float:
        self.refill()
        return self._tokens

    @tokens.setter
    def tokens(self, value: float):
        self._tokens = value

    def refill(self) -> None:
        current_time = self.clock()
        elapsed_time = current_time - self.last_refill
        self.last_refill = current_time
        refill_amount = elapsed_time * self.refill_per_sec
        self._tokens = min(self.capacity, self._tokens + refill_amount)

    def allow(self, cost: float = 1) -> bool:
        self.refill()
        if self._tokens >= cost:
            self._tokens -= cost
            return True
        return False
```
Decisions: Refund window (hard) 50% right
```json
{
  "outcome": {
    "full_refund": 0,
    "partial_refund": 0.75,
    "store_credit": 0.25,
    "nothing": 0
  },
  "defective": {
    "true": 0,
    "false": 1
  }
}
```
Decisions: Incident matrix (hard) 100% right
```
{
  "severity": {
    "0": 0,
    "1": 0.7,
    "2": 0.3,
    "3": 0
  },
  "page": {
    "true": 0.3,
    "false": 0.7
  }
}
```
Decisions: Tool followup (hard) 100% right
```
{
  "tool": {
    "web_search": 0,
    "calculator": 0,
    "calendar": 1,
    "email": 0,
    "none": 0
  },
  "confirm": {
    "true": 0.5,
    "false": 0.5
  }
}
```
Decisions: Legit security alert (hard) 100% right
```
{
  "phishing": {
    "true": 0.05,
    "false": 0.95
  },
  "action_needed": {
    "true": 0.05,
    "false": 0.95
  }
}
```
Decisions: Meeting slot (hard) 0% right
```json
{
  "slot": {
    "A": 0.0,
    "B": 0.25,
    "C": 0.75,
    "D": 0.0
  },
  "raj_last": {
    "true": 0.0,
    "false": 1.0
  }
}
```
Decisions: Review mixed (hard) 100% right
```json
{
  "hardware": {
    "true": 0.8,
    "false": 0.2
  },
  "support": {
    "true": 0.9,
    "false": 0.1
  }
}
```
Decisions: Support checkout down 67% right
```
{
  "department": {
    "billing": 0.7,
    "technical": 0.3,
    "account": 0,
    "shipping": 0,
    "sales": 0
  },
  "urgency": {
    "0": 0,
    "1": 0,
    "2": 0.2,
    "3": 0.8
  },
  "outage": {
    "true": 0.9,
    "false": 0.1
  }
}
```
Decisions: Refund wrong plan 100% right
```
{
  "department": {
    "billing": 0.9,
    "technical": 0.05,
    "account": 0.05,
    "shipping": 0,
    "sales": 0
  },
  "refund": {
    "true": 0.8,
    "false": 0.2
  },
  "tone": {
    "frustrated": 0.1,
    "calm": 0.9
  }
}
```
Decisions: Moderation doxxing 100% right
```json
{
  "policy": {
    "none": 0,
    "harassment": 0.9,
    "hate": 0,
    "spam": 0,
    "self_harm": 0.1
  },
  "personal_info": {
    "true": 0.9,
    "false": 0.1
  }
}
```
Decisions: Route calendar 100% right
```
{
  "tool": {
    "web_search": 0,
    "calculator": 0,
    "calendar": 0.9,
    "email": 0,
    "none": 0.1
  },
  "confirm": {
    "true": 0.8,
    "false": 0.2
  }
}
```
Decisions: Doc invoice missing due 100% right
```json
{
  "doc_type": {
    "invoice": 0.99,
    "resume": 0.005,
    "contract": 0.0,
    "bank_statement": 0.0,
    "other": 0.005
  },
  "missing_due_date": {
    "true": 0.8,
    "false": 0.2
  }
}
```
Decisions: Phishing paypal 100% right
```
{
  "phishing": {
    "true": 0.9,
    "false": 0.1
  },
  "risk": {
    "0": 0.05,
    "1": 0.05,
    "2": 0.3,
    "3": 0.6
  }
}
```
Decisions: Pii ssn email 50% right
```json
{
  "data_kind": {
    "none": 0,
    "contact": 0.33,
    "financial": 0.33,
    "government_id": 0.33,
    "health": 0
  },
  "sensitive": {
    "true": 1,
    "false": 0
  }
}
```
Decisions: Review mixed 100% right
```json
{
  "sentiment": {
    "positive": 0.05,
    "neutral": 0.05,
    "negative": 0.9
  },
  "defect": {
    "true": 0.9,
    "false": 0.1
  },
  "recommend": {
    "true": 0.05,
    "false": 0.95
  }
}
```
Documents: Saas escalator (hard) 20% right
```
{
  "year2_price_per_seat_month": 47.7,
  "year3_price_per_seat_month": 47.66,
  "year1_invoice": 43200.00,
  "year2_invoice": 45816.00,
  "addon_months_billed": 6,
  "addon_invoice": 28596.00,
  "year3_invoice": 17153.92,
  "year3_discount_percent": 15,
  "total_contract_value": 134765.92,
  "contract_end_date": "2027-02-29"
}
```
Documents: Expense thread 0% right
```
{
  "employee_id": "EMP-20417",
  "destination_city": "Lisbon",
  "trip_start": "2025-02-24",
  "trip_end": "2025-02-27",
  "approved_items": [
    {
      "date": "2025-02-24",
      "category": "airfare",
      "amount_usd": 1184.60
    },
    {
      "date": "2025-02-24",
      "category": "ground_transport",
      "amount_usd": 36.00 * 1.08
    },
    {
      "date": "2025-02-25",
      "category": "meals",
      "amount_usd": 212.40 * 1.08
    },
    {
      "date": "2025-02-26",
      "category": "lodging",
      "amount_usd": 432.00 * 1.08
    },
    {
      "date": "2025-02-27",
      "category": "ground_transport",
      "amount_usd": 41.50 * 1.08
    }
  ],
  "rejected_item_count": 2,
  "per_diem_days": 2,
  "per_diem_usd": 65.00 * 2,
  "total_reimbursable_usd": null,
  "approver_email": "priya.raman@corvane.com"
}
```
Documents: Lease amendment 83% right
```json
{
  "tenants": ["Marcus Lin", "Sofia Lin"],
  "landlord": "Ridgeline Property Group LLC",
  "zip": "97205",
  "lease_end": "2025-11-30",
  "original_monthly_rent": 2150.00,
  "monthly_rent_from_2025_06_01": 2236.00,
  "late_fee_from_2025_06_01": 111.80,
  "security_deposit": 2150.00,
  "total_pet_deposits": 800.00,
  "total_monthly_payment_july_2025": 2287.00,
  "move_in_payment": 4300.00
}
```
Documents: Ticket SLA 75% right
```
{
  "ticket_id": "48213",
  "account_id": "ACC-7731",
  "open_issue": "inventory_sync",
  "resolved_issues": ["billing_address", "invoice_pdf"],
  "affected_orders": ["SO-99812", "SO-99820", "SO-99827"],
  "priority": "P2",
  "sla_due_local": "2025-09-12T17:00",
  "sla_due_utc": "2025-09-13T00:00:00Z",
  "reissued_invoice": "INV-2025-0812"
}
```
Documents: Sales footnotes 18% right
```
{
  "q3_total_usd": 15458,
  "q2_total_usd": 14264,
  "q2_central_originally_reported_usd": 3047,
  "q2_to_q3_change_pct": 8.4,
  "top_region_q3": "East",
  "fastest_growing_region_q1_to_q3": "International",
  "regions_declining_q2_to_q3": ["Central", "East", "West"],
  "international_q3_organic_usd": 1731,
  "west_excluding_mountain_q3_usd": 4201
}
```

Size: 109B parameters. First tested OCT 11.

Models that scored about the same

Comments

Sign in with GitHub to comment. Spam and abuse are hidden automatically.