Review · updated OCT 11

Llama 3.2 1B Instruct (Q4, Mac) review: not one we'd recommend right now

It scored 5 out of 100, #56 of 56. It solved 0 of 30 coding jobs and scored 9 on reading documents. Runs on an 8 GB graphics card or a Mac with 16 GB.

The short version
  • Llama 3.2 1B Instruct (Q4, Mac) is a free model from Meta that you can run on your own computer. In our tests it's not one we'd recommend right now: 5 out of 100, #56 of 56.
  • It solved 0 of 30 coding jobs and scored 9 on reading documents. On our hardest tasks it scored 3.
  • Runs on an 8 GB graphics card or a Mac with 16 GB.

Coding

Our coding test is 30 programming jobs, from small ones like reading time durations or cleaning up messy data to harder ones like a config-file parser or a double-entry ledger. We run each answer against tests the model never sees, and a job only counts if everything passes. Llama 3.2 1B Instruct (Q4, Mac) got 0 of 30 right. The best local coders solved 29 of 30.

Reading documents

The second test hands the model things like an expense claim thread, a pay stub or an insurance statement, and asks for specific numbers and dates. Many questions need a bit of math, or noticing a correction further down the email. Llama 3.2 1B Instruct (Q4, Mac) scored 9; the best model scored 100.

TestScorePublic questionsSecret questions
Coding000
Reading documents9177
Decisions???

On the 18 hardest tasks (included in the scores above) it scored 3. This number separates the top models.

We tested the Q4 download, the version most people run, with Ollama on our Mac mini (M4, 16 GB).

Can your computer run it?

Llama 3.2 1B Instruct (Q4, Mac) comes in one standard size, which needs about 1.9 GB of memory.

Your computerRuns it?Version to download
Graphics card, 8 GBYesQ4
Graphics card, 12 GBYesQ4
Graphics card, 16 GBYesQ4
Graphics card, 24 GBYesQ4
Graphics card, 32 GBYesQ4
Mac, 16 GBYesQ4
Mac, 24 GBYesQ4
Mac, 32 GBYesQ4
Mac, 48 GBYesQ4
Mac, 64 GBYesQ4
Mac, 96 GBYesQ4
Mac, 128 GBYesQ4

For normal-length chats. Long documents need more memory. Q4 or Q8? · How we worked this out

How to try it

The easiest way is Ollama. Install it, then run:

ollama run llama3.2:1b-instruct-q4_K_M

Want the strongest all-rounder instead? Qwen3.6 27B scored 97.

Its actual answers

What Llama 3.2 1B Instruct (Q4, Mac) wrote for our public questions, word for word.

Coding: INI parser (hard) Didn't work
```python
import re
from typing import Dict, List, Any

def parse_ini(text: str) -> Dict[str, Dict[str, str]]:
    """
    Parse an INI dialect text into a dictionary of sections and values.

    Args:
    text (str): The INI dialect text to parse.

    Returns:
    dict[str, dict[str, str]]: A dictionary of sections and values, where each key is a section name
    and the corresponding value is another dictionary of section keys and values.
    """

    lines = text.split('\n')
    sections: Dict[str, Dict[str, str]] = {}
    in_section = False

    for line in lines:
        # Skip empty lines
        if not line.strip():
            continue

        # Check for section headers
        if line.lstrip().startswith('[') and line.lstrip().endswith(']'):
            # Extract section name
            section_name = line[1:-1].strip()
            if section_name == 'DEFAULT':
                # Default section
                sections[section_name] = {}
            else:
                # Other sections
                sections[section_name] = {}
            continue

        # Check for key and value
        match = re.match(r'^(.*)=(.*)$', line, re.I)
        if match:
            key, value = match.groups()
            # Remove leading and trailing whitespace
            key = key.strip()
            value = value.strip()
            # Remove double quotes if the value is a string
            if value.startswith('"') and value.endswith('"'):
                value = value[1:-1]
            # Replace interpolation with key expansion
            if '${' in value:
                value = '${' + key + '}'
            if value.startswith('${') and value.endswith('}'):
                value = value[2:-1]
            # Strip empty values
            value = value.strip()

            # Add key-value pair
            sections[section_name][key] = value
            # Reset section
            if section_name == 'DEFAULT':
                section_name = 'DEFAULT'
            elif section_name not in sections:
                # Inherited DEFAULT section
                sections[section_name] = {}
            in_section = True
        # Check for continuation
        elif in_section and line.lstrip().startswith(';') or line.lstrip().startswith('#'):
            # Continuation of a value
            continue
        # Check for key line before section header or a malformed line
        elif re.match(r'^\s*$', line) or not re.match(r'^\W+', line):
            raise ValueError("Key line before section header or a malformed line")

    # Handle special section DEFAULT
    if 'DEFAULT' in sections and not in_section:
        # Inherited DEFAULT section
        in_section = False

    # Return values for inherited sections
    if in_section:
        # Inherited sections are not returned as separate sections
        # Instead, they are inherited by every other section
        for section_name, values in sections.items():
            if section_name != 'DEFAULT':
                sections[section_name] = values

    return sections
```
Coding: TTL LRU (hard) Didn't work
```python
import time
from collections import OrderedDict

class LRUCache:
    def __init__(self, capacity: int, ttl: float, clock):
        """
        Initialize the LRU cache with given capacity, time to live (TTL), and clock.
        
        :param capacity: The maximum number of items in the cache.
        :param ttl: The time to live (TTL) for each item, in seconds.
        :param clock: A zero-argument callable returning the current time in seconds.
        """
        self.capacity = capacity
        self.ttl = ttl
        self.clock = clock
        self.cache = OrderedDict()
        self.expired_time = OrderedDict()

    def put(self, key: str, value: any):
        """
        Insert or replace an item in the cache.
        
        :param key: The key of the item to insert.
        :param value: The value of the item to insert.
        """
        current_time = time.time()
        if key not in self.cache:
            if len(self.cache) == self.capacity:
                # evict the least recently used item
                self.cache.popitem(last=False)
            self.cache[key] = (current_time, value)
        else:
            current_time, value = self.cache[key]
            if current_time + self.ttl < time.time():
                # item has expired, remove and count as an eviction
                if key not in self.expired_time:
                    self.expired_time[key] = current_time
                self.cache.pop(key)
                self.cache[key] = (current_time, value)
            elif current_time > self.expired_time[key]:
                # item is most recently used, update its expiry and recency
                self.cache[key] = (current_time, value)
                if key not in self.expired_time:
                    self.expired_time[key] = current_time
            else:
                # item is not most recently used, update its recency
                self.cache[key] = (current_time, value)

    def get(self, key: str, default=None):
        """
        Get the value for a given key.
        
        :param key: The key of the item to get.
        :param default: The value to return if the key is not found.
        """
        current_time = time.time()
        if key in self.cache and current_time < self.cache[key][0]:
            # item is most recently used, update its expiry
            if key not in self.expired_time:
                self.expired_time[key] = current_time
            value, _ = self.cache[key]
            if default is None:
                return value
            return default
        elif key in self.expired_time and current_time >= self.expired_time[key]:
            # item has expired, remove and count as an eviction
            if key not in self.expired_time:
                self.expired_time[key] = current_time
            value = self.cache.pop(key)
            if default is None:
                return default
            return default
        else:
            # item is not most recently used, return the default value
            if default is None:
                return None
            return default

    def __len__(self):
        """
        Return the number of items in the cache that are not expired.
        """
        return len(self.cache)

    def stats(self):
        """
        Return a dictionary with counts for hits, misses, evictions, and expirations.
        """
        stats = {
            "hits": 0,
            "misses": 0,
            "evictions": 0,
            "expirations": 0,
        }
        for key, (_, value) in self.cache.items():
            if current_time < self.cache[key][0]:
                stats["hits"] += 1
            elif key not in self.expired_time:
                stats["misses"] += 1
            else:
                stats["evictions"] += 1
            stats["expirations"] += 1
        for key, time in self.expired_time.items():
            stats["misses"] += 1
        return stats

    def __getitem__(self, key):
        """
        Return the value for a given key.
        
        :param key: The key of the item to get.
        :return: The value of the item.
        """
        return self.get(key)

    def __setitem__(self, key, value):
        """
        Set the value for a given key.
        
        :param key: The key of the item to set.
        :param value: The value of the item to set.
        """
        self.put(key, value)

    def __str__(self):
        """
        Return a string representation of the cache.
        """
        return str({key: value for key, value in self.cache.items()})

    def clock(self, current_time):
        """
        Update the current time and evict the least recently used item.
        
        :param current_time: The current time in seconds.
        """
        current_time = time.time()
        if current_time > self.clock:
            self.cache.clear()
            self.cache = OrderedDict()
            self.expired_time.clear()
        self.clock = current_time
        for key, (_, value) in self.cache.items():
            if current_time < self.cache[key][0]:
                self.cache.pop(key)
            elif key not in self.expired_time:
                self.expired_time[key] = current_time
```

This implementation provides a basic LRU cache with the specified methods and attributes. It uses an ordered dictionary ( OrderedDict ) to store the cache items, which ensures that the least recently used item is always at the beginning of the dictionary. The cache items are updated whenever the TTL is reached, and the least recently used item is evicted when the capacity is exceeded. The clock is used to update the current time, and the expired items are removed. The cache items are also marked as evicted when they are no longer in the cache. The stats method returns a dictionary with counts for hits, misses, evictions, and expirations. The __getitem__ method provides indexing for the cache items, and the __setitem__ method provides the ability to set new cache items.
Coding: Parse duration Didn't work
```python
import re

def parse_duration(s: str) -> int:
    """
    Converts a duration string into a total number of seconds.
    
    Args:
    s (str): The duration string.
    
    Returns:
    int: The total number of seconds.
    
    Raises:
    ValueError: If the string is empty, blank, or contains invalid characters.
    ValueError: If the string does not contain exactly three parts (hour, minute, second).
    ValueError: If the units are out of order.
    ValueError: If the string contains a number without a unit.
    ValueError: If the string contains a decimal number.
    ValueError: If the string contains repeated units.
    """

    # Remove whitespace from the string
    s = s.strip()
    
    # Check for empty string
    if not s:
        raise ValueError("Input string is empty")
    
    # Check for blank string
    if " " in s:
        raise ValueError("Input string is blank")
    
    # Split the string into parts
    parts = re.split(r'(\d+)(h|m|s)', s)
    
    # Check for invalid characters
    if not all(char.isdigit() or char in ['h', 'm', 's'] for char in parts[0]):
        raise ValueError("Input string contains invalid characters")
    
    # Check for repeated units
    if len(set(parts[0])) != 1:
        raise ValueError("Input string contains repeated units")
    
    # Check for units out of order
    if not all(parts[1] == 'h' or parts[1] == 'm' or parts[1] == 's' for parts in parts[2:]):
        raise ValueError("Units are out of order")
    
    # Convert hours to seconds
    if len(parts) == 3 and parts[1] == 'h':
        # Convert hours to seconds
        hours = int(parts[0])
        minutes = 0
        seconds = 0
        for i in range(2, len(parts)):
            if parts[i] == 'm':
                minutes += int(parts[i-1]) * 30
            elif parts[i] == 's':
                seconds += int(parts[i-1])
        
        # Calculate total seconds
        total_seconds = hours * 3600 + minutes * 60 + seconds
    
    # Convert minutes to seconds
    elif len(parts) == 3 and parts[1] == 'm':
        minutes = int(parts[0])
        seconds = 0
        for i in range(2, len(parts)):
            if parts[i] == 's':
                seconds += int(parts[i-1])
        
        # Calculate total seconds
        total_seconds = minutes * 60 + seconds
    
    # Convert seconds to seconds
    elif len(parts) == 3 and parts[1] == 's':
        seconds = int(parts[0])
        # Calculate total seconds
        total_seconds = seconds
    
    return total_seconds
```
Coding: Summarize ranges Didn't work
```python
def summarize_ranges(nums):
    """
    This function takes a list of integers, sorts them, removes duplicates, 
    and collapses runs of 3 or more consecutive integers into "a..b". 
    Numbers in shorter runs (1 or 2 numbers) are listed individually. 
    Join everything with "," (no spaces). An empty list returns "".

    Args:
    nums (list[int]): A list of integers.

    Returns:
    str: The summarized list of integers.
    """

    # Sort the list of numbers and remove duplicates
    sorted_nums = sorted(set(nums))

    # Initialize the result list
    result = []

    # Initialize the start and end of the current run
    start = end = None

    # Iterate over the sorted list of numbers
    for i in range(len(sorted_nums)):
        # If this is the start of a new run or the current number is different from the previous one
        if i == 0 or sorted_nums[i] != sorted_nums[i - 1]:
            # If we have found a start for the current run
            if start is None:
                # Add the current number to the result list
                result.append(str(sorted_nums[i]))
            # If we have found an end for the current run
            else:
                # If the current run is long enough to be collapsed into "a..b"
                if i - start > 2:
                    # Collapse the run and add it to the result list
                    result.append(str(sorted_nums[start:end]))
                # If the current run is short enough to be collapsed
                else:
                    # Add the current run to the result list
                    result.append(str(sorted_nums[start:end]))
                # Update the start and end of the current run
                start = i
                end = i

    # Collapse any remaining run
    if start is not None and len(result) > 1:
        result.append(str(sorted_nums[start:end]))
    # If the list is empty, return an empty string
    elif result:
        return ",".join(result)
    # If the list is not empty, return the result
    else:
        return ",".join(result)
```
Coding: Top customers Didn't work
```python
import re
from collections import defaultdict
from typing import List, Dict, Tuple

def top_customers(rows: List[Dict[str, str]], n: int) -> List[Tuple[str, float]]:
    """
    Returns the top customers as (name, total) tuples, sorted by total descending, then name ascending.

    :param rows: A list of dictionaries where each dictionary has a "customer" name and an "amount" string.
    :type rows: List[Dict[str, str]]
    :param n: The number of top customers to return.
    :type n: int
    :return: A list of tuples containing the top customers as (name, total).
    :rtype: List[Tuple[str, float]]
    """

    # Create a dictionary to store the sum of amounts for each customer
    customer_sums = defaultdict(lambda: 0.0)

    # Iterate over each row
    for row in rows:
        # Trim and match case-insensitively to get the customer name
        customer_name = re.sub(r'[^\w\s]', '', row['customer'].lower()).strip()

        # Try to parse the amount as a float
        try:
            amount = float(row['amount'])
        except ValueError:
            # If the amount cannot be parsed, skip this row
            continue

        # If the customer name is empty, skip this row
        if not customer_name:
            continue

        # Add the amount to the total sum for the customer
        customer_sums[customer_name] += amount

    # Sort the customers by total sum and then by name
    sorted_customers = sorted(customer_sums.items(), key=lambda x: (-x[1], x[0]))

    # Return the top customers as (name, total) tuples
    return [(name, round(total, 2)) for name, total in sorted_customers[:n]]
```
Coding: Slugify Didn't work
```python
import unicodedata
import re

def slugify(title: str, max_len: int = 50) -> str:
    """
    Builds a URL slug by performing the following transformations on a given title:
    1. Transliterate accented characters to ASCII.
    2. Drop any other non-ASCII characters.
    3. Lowercase.
    4. Replace every run of characters other than a-z and 0-9 with a single "-".
    5. If the slug is longer than `max_len`, shorten it without cutting a word.
    6. If the first word is longer than `max_len`, hard-cut it to `max_len` characters.
    
    :param title: The title to slugify
    :param max_len: The maximum length of the slug (default: 50)
    :return: The slugified title
    """

    # Transliterate accented characters to ASCII
    title = ''.join(c if ord(c) < 128 else unicodedata.normalize('NFKD', c) for c in title)

    # Lowercase
    title = title.lower()

    # Replace every run of characters other than a-z and 0-9 with a single "-"
    title = re.sub(r'(\W)(\d)(\W)', r'\1-\2\3', title)

    # Strip leading/trailing "-"
    title = title.strip('-')

    # Shorten the slug if it's longer than max_len
    if len(title) > max_len:
        title = title[:max_len]

    return title
```
Coding: Token bucket Didn't work
```python
import time
from typing import Callable

class TokenBucket:
    def __init__(self, capacity: float, refill_per_sec: float, clock: Callable):
        self.capacity = capacity
        self.refill_per_sec = refill_per_sec
        self.clock = clock
        self.tokens = capacity

    def allow(self, cost: float = 1) -> bool:
        elapsed_time = time.time() - self.clock()
        tokens_refilled = int(elapsed_time // self.refill_per_sec)
        if tokens_refilled < cost:
            self.tokens -= tokens_refilled
            return True
        else:
            self.tokens -= cost
            return False

    @property
    def tokens(self) -> float:
        return self.capacity - self.tokens

def clock():
    return time.time()
```
Decisions: Support checkout down 0% right
{
  "department": {
    "billing": 0.25,
    "technical": 0.5,
    "account": 0.25,
    "shipping": 0.25,
    "sales": 0.5
  },
  "urgency": {
    "0": 0.3,
    "1": 0.4,
    "2": 0.1,
    "3": 0.5
  },
  "outage": {
    "true": 0.8,
    "false": 0.2
  }
Decisions: Refund wrong plan 0% right
{
  "department": {
    "billing": 0.6,
    "technical": 0.1,
    "account": 0.1,
    "shipping": 0.1,
    "sales": 0.1
  },
  "refund": {
    "true": 0.8,
    "false": 0.2
  },
  "tone": {
    "frustrated": 0.4,
    "calm": 0.6
}
Decisions: Moderation doxxing 0% right
{
  "1": {"policy": "none", "personal_info": "false"},
  "2": {"policy": "none", "personal_info": "false"},
  "3": {"policy": "none", "personal_info": "false"},
  "4": {"policy": "none", "personal_info": "false"},
  "5": {"policy": "none", "personal_info": "false"},
  "6": {"policy": "none", "personal_info": "false"},
  "7": {"policy": "none", "personal_info": "false"},
  "8": {"policy": "none", "personal_info": "false"},
  "9": {"policy": "none", "personal_info": "false"},
  "10": {"policy": "none", "personal_info": "false"},
  "11": {"policy": "none", "personal_info": "false"},
  "12": {"policy": "none", "personal_info": "false"},
  "13": {"policy": "none", "personal_info": "false"},
  "14": {"policy": "none", "personal_info": "false"},
  "15": {"policy": "none", "personal_info": "false"},
  "16": {"policy": "none", "personal_info": "false"},
  "17": {"policy": "none", "personal_info": "false"},
  "18": {"policy": "none", "personal_info": "false"},
  "19": {"policy": "none", "personal_info": "false"},
  "20": {"policy": "none", "personal_info": "false"},
  "21": {"policy": "none", "personal_info": "false"},
  "22": {"policy": "none", "personal_info": "false"},
  "23": {"policy": "none", "personal_info": "false"},
  "24": {"policy": "none", "personal_info": "false"},
  "25": {"policy": "none", "personal_info": "false"},
  "26": {"policy": "none", "personal_info": "false"},
  "27": {"policy": "none", "personal_info": "false"},
  "28": {"policy": "none", "personal_info": "false"},
  "29": {"policy": "none", "personal_info": "false"},
  "30": {"policy": "none", "personal_info": "false"},
  "31": {"policy": "none", "personal_info": "false"},
  "32": {"policy": "none", "personal_info": "false"},
  "33": {"policy": "none", "personal_info": "false"},
  "34": {"policy": "none", "personal_info": "false"},
  "35": {"policy": "none", "personal_info": "false"},
  "36": {"policy": "none", "personal_info": "false"},
  "37": {"policy": "none", "personal_info": "false"},
  "38": {"policy": "none", "personal_info": "false"},
  "39": {"policy": "none", "personal_info": "false"},
  "40": {"policy": "none", "personal_info": "false"},
  "41": {"policy": "none", "personal_info": "false"},
  "42": {"policy": "none", "personal_info": "false"},
  "43": {"policy": "none", "personal_info": "false"},
  "44": {"policy": "none", "personal_info": "false"},
  "45": {"policy": "none", "personal_info": "false"},
  "46": {"policy": "none", "personal_info": "false"},
  "47": {"policy": "none", "personal_info": "false"},
  "48": {"policy": "none", "personal_info": "false"},
  "49": {"policy": "none", "personal_info": "false"},
  "50": {"policy": "none", "personal_info": "false"},
  "51": {"policy": "none", "personal_info": "false"},
  "52": {"policy": "none", "personal_info": "false"},
  "53": {"policy": "none", "personal_info": "false"},
  "54": {"policy": "none", "personal_info": "false"},
  "55": {"policy": "none", "personal_info": "false"},
  "56": {"policy": "none", "personal_info": "false"},
  "57": {"policy": "none", "personal_info": "false"},
  "58": {"policy": "none", "personal_info": "false"},
  "59": {"policy": "none", "personal_info": "false"},
  "60": {"policy": "none", "personal_info": "false"},
  "61": {"policy": "none", "personal_info": "false"},
  "62": {"policy": "none", "personal_info": "false"},
  "63": {"policy": "none", "personal_info": "false"},
  "64": {"policy": "none", "personal_info": "false"},
  "65": {"policy": "none", "personal_info": "false"},
  "66": {"policy": "none", "personal_info": "false"},
  "67": {"policy": "none", "personal_info": "false"},
  "68": {"policy": "none", "personal_info": "false"},
  "69": {"policy": "none", "personal_info": "false"},
  "70": {"policy": "none", "personal_info": "false"},
  "71": {"policy": "none", "personal_info": "false"},
  "72": {"policy": "none", "personal_info": "false"},
  "73": {"policy": "none", "personal_info": "false"},
  "74": {"policy": "none", "personal_info": "false"},
  "75": {"policy": "none", "personal_info": "false"},
  "76": {"policy": "none", "personal_info": "false"},
  "77": {"policy": "none", "personal_info": "false"},
  "78": {"policy": "none", "personal_info": "false"},
  "79": {"policy": "none", "personal_info": "false"},
  "80": {"policy": "none", "personal_info": "false"},
  "81": {"policy": "none", "personal_info": "false"},
  "82": {"policy": "none", "personal_info": "false"},
  "83": {"policy": "none", "personal_info": "false"},
  "84": {"policy": "none", "personal_info": "false"},
  "85": {"policy": "none", "personal_info": "false"},
  "86": {"policy": "none", "personal_info": "false"},
  "87": {"policy": "none", "personal_info": "false"},
  "88": {"policy": "none", "personal_info": "false"},
  "89": {"policy": "none", "personal_info": "false"},
  "90": {"policy": "none", "personal_info": "false"},
  "91": {"policy": "none", "personal_info": "false"},
  "92": {"policy": "none", "personal_info": "false"},
  "93": {"policy": "none", "personal_info": "false"},
  "94": {"policy": "none", "personal_info": "false"},
  "95": {"policy": "none", "personal_info": "false"},
  "96": {"policy": "none", "personal_info": "false"},
  "97": {"policy": "none", "personal_info": "false"},
  "98": {"policy": "none", "personal_info": "false"},
  "99": {"policy": "none", "personal_info": "false"},
  "100": {"policy": "none", "personal_info": "false"},
  "101": {"policy": "none", "personal_info": "false"},
  "102": {"policy": "none", "personal_info": "false"},
  "103": {"policy": "none", "personal_info": "false"},
  "104": {"policy": "none", "personal_info": "false"},
  "105": {"policy": "none", "personal_info": "false"},
  "106": {"policy": "none", "personal_info": "false"},
  "107": {"policy": "none", "personal_info": "false"},
  "108": {"policy": "none", "personal_info": "false"},
  "109": {"policy": "none", "personal_info": "false"},
  "110": {"policy": "none", "personal_info": "false"},
  "111": {"policy": "none", "personal_info": "false"},
  "112": {"policy": "none", "personal_info": "false"},
  "113": {"policy": "none", "personal_info": "false"},
  "114": {"policy": "none", "personal_info": "false"},
  "115": {"policy": "none", "personal_info": "false"},
  "116": {"policy": "none", "personal_info": "false"},
  "117": {"policy": "none", "personal_info": "false"},
  "118": {"policy": "none", "personal_info": "false"},
  "119": {"policy": "none", "personal_info": "false"},
  "120": {"policy": "none", "personal_info": "false"},
  "121": {"policy": "none", "personal_info": "false"},
  "122": {"policy": "none", "personal_info": "false"},
  "123": {"policy": "none", "personal_info": "false"},
  "124": {"policy": "none", "personal_info": "false"},
  "125": {"policy": "none", "personal_info": "false"},
  "126": {"policy": "none", "personal_info": "false"},
  "127": {"policy": "none", "personal_info": "false"},
  "128": {"policy": "none", "personal_info": "false"},
  "129": {"policy": "none", "personal_info": "false"},
  "130": {"policy": "none", "personal_info": "false"},
  "131": {"policy": "none", "personal_info": "false"},
  "132": {"policy": "none", "personal_info": "false"},
  "133": {"policy": "none", "personal_info": "false"},
  "134": {"policy": "none", "personal_info": "false"},
  "135": {"policy": "none", "personal_info": "false"},
  "136": {"policy": "none", "personal_info": "false"},
  "137": {"policy": "none", "personal_info": "false"},
  "138": {"policy": "none", "personal_info": "false"},
  "139": {"policy": "none", "personal_info": "false"},
  "140": {"policy": "none", "personal_info": "false"},
  "141": {"policy": "none", "personal_info": "false"},
  "142": {"policy": "none", "personal_info": "false"},
  "143": {"policy": "none", "personal_info": "false"},
  "144": {"policy": "none", "personal_info": "false"},
  "145": {"policy": "none", "personal_info": "false"},
  "146": {"policy": "none", "personal_info": "false"},
  "147": {"policy": "none", "personal_info": "false"},
  "148": {"policy": "none", "personal_info": "false"},
  "149": {"policy": "none", "personal_info": "false"},
  "150": {"policy": "none", "personal_info": "false"},
  "151": {"policy": "none", "personal_info": "false"},
  "152": {"policy": "none", "personal_info": "false"},
  "153": {"policy": "none", "personal_info": "false"},
  "154": {"policy": "none", "personal_info": "false"},
  "155": {"policy": "none", "personal_info": "false"},
  "156": {"policy": "none", "personal_info": "false"},
  "157": {"policy": "none", "personal_info": "false"},
  "158": {"policy": "none", "personal_info": "false"},
  "159": {"policy": "none", "personal_info": "false"},
  "160": {"policy": "none", "personal_info": "false"},
  "161": {"policy": "none", "personal_info": "false"},
  "162": {"policy": "none", "personal_info": "false"},
  "163": {"policy": "none", "personal_info": "false"},
  "164": {"policy": "none", "personal_info": "false"},
  "165": {"policy": "none", "personal_info": "false"},
  "166": {"policy": "none", "personal_info": "false"},
  "167": {"policy": "none", "personal_info": "false"},
  "168": {"policy": "none", "personal_info": "false"},
  "169": {"policy": "none", "personal_info": "false"},
  "170": {"policy": "none", "personal_info": "false"},
  "171": {"policy": "none", "personal_info": "false"},
  "172": {"policy": "none", "personal_info": "false"},
  "173": {"policy": "none", "personal_info": "false"},
  "174": {"policy": "none", "personal_info": "false"},
  "175": {"policy": "none", "personal_info": "false"},
  "176": {"policy": "none", "personal_info": "false"},
  "177": {"policy": "none", "personal_info": "false"},
  "178": {"policy": "none", "personal_info": "false"},
  "179": {"policy": "none", "personal_info": "false"},
  "180": {"policy": "none", "personal_info": "false"},
  "181": {"policy": "none", "personal_info": "false"},
  "182": {"policy": "none", "personal_info": "false"},
  "183": {"policy": "none", "personal_info": "false"},
  "184": {"policy": "none", "personal_info": "false"},
  "185": {"policy": "none", "personal_info": "false"},
  "186": {"policy": "none", "personal_info": "false"},
  "187": {"policy": "none", "personal_info": "false"},
  "188": {"policy": "none", "personal_info": "false"},
  "189": {"policy": "none", "personal_info": "false"},
  "190": {"policy": "none", "personal_info": "false"},
  "191": {"policy": "none", "personal_info": "false"},
  "192": {"policy": "none", "personal_info": "false"},
  "193": {"policy": "none", "personal_info": "false"},
  "194": {"policy": "none", "personal_info": "false"},
  "195": {"policy": "none", "personal_info": "false"},
  "196": {"policy": "none", "personal_info": "false"},
  "197": {"policy": "none", "personal_info": "false"},
  "198": {"policy": "none", "personal_info": "false"},
  "199": {"policy": "none", "personal_info": "false"},
  "200": {"policy": "none", "personal_info": "false"},
  "201": {"policy": "none", "personal_info": "false"},
  "202": {"policy": "none", "personal_info": "false"},
  "203": {"policy": "none", "personal_info": "false"},
  "204": {"policy": "none", "personal_info": "false"},
  "205": {"policy": "none", "personal_info": "false"},
  "206": {"policy": "none", "personal_info": "false"},
  "207": {"policy": "none", "personal_info": "false"},
  "208": {"policy": "none", "personal_info": "false"},
  "209": {"policy": "none", "personal_info": "false"},
  "210": {"policy": "none", "personal_info": "false"},
  "211": {"policy": "none", "personal_info": "false"},
  "212": {"policy": "none", "personal_info": "false"},
  "213": {"policy": "none", "personal_info": "false"},
  "214": {"policy": "none", "personal_info": "false"},
  "215": {"policy": "none", "personal_info": "false"},
  "216": {"policy": "none", "personal_info": "false"},
  "217": {"policy": "none", "personal_info": "false"},
  "218": {"policy": "none", "personal_info": "false"},
  "219": {"policy": "none", "personal_info": "false"},
  "220": {"policy": "none", "personal_info": "false"},
  "221": {"policy": "none", "personal_info": "false"},
  "222": {"policy": "none", "personal_info": "false"},
  "223": {"policy": "none", "personal_info": "false"},
  "224": {"policy": "none", "personal_info": "false"},
  "225": {"policy": "none", "personal_info": "false"},
  "226": {"policy": "none", "personal_info": "false"},
  "227": {"policy": "none", "personal_info": "false"},
  "228": {"policy": "none", "personal_info": "false"},
  "229": {"policy": "none", "personal_info": "false"},
  "230": {"policy": "none", "personal_info": "false"},
  "231": {"policy": "none", "personal_info": "false"},
  "232": {"policy": "none", "personal_info": "false"},
  "233": {"policy": "none", "personal_info": "false"},
  "234": {"policy": "none", "personal_info": "false"},
  "235": {"policy": "none", "personal_info": "false"},
  "236": {"policy": "none", "personal_info": "false"},
  "237": {"policy": "none", "personal_info": "false"},
  "238": {"policy": "none", "personal_info": "false"},
  "239": {"policy": "none", "personal_info": "false"},
  "240": {"policy": "none", "personal_info": "false"},
  "241": {"policy": "none", "personal_info": "false"},
  "242": {"policy": "none", "personal_info": "false"},
  "243": {"policy": "none", "personal_info": "false"},
  "244": {"policy": "none", "personal_info": "false"},
  "245": {"policy": "none", "personal_info": "false"},
  "246": {"policy": "none", "personal_info": "false"},
  "247": {"policy": "none", "personal_info": "false"},
  "248": {"policy": "none", "personal_info": "false"},
  "249": {"policy": "none", "personal_info": "false"},
  "250": {"policy": "none", "personal_info": "false"},
  "251": {"policy": "none", "personal_info": "false"},
  "252": {"policy": "none", "personal_info": "false"},
  "253": {"policy": "none", "personal_info": "false"},
  "254": {"policy": "none", "personal_info": "false"},
  "255": {"policy": "none", "personal_info": "false"},
  "256": {"policy": "none", "personal_info": "false"},
  "257": {"policy": "none", "personal_info": "false"},
  "258": {"policy": "none", "personal_info": "false"},
  "259": {"policy": "none", "personal_info": "false"},
  "260": {"policy": "none", "personal_info": "false"},
  "261": {"policy": "none", "personal_info": "false"},
  "262": {"policy": "none", "personal_info": "false"},
  "263": {"policy": "none", "personal_info": "false"},
  "264": {"policy": "none", "personal_info": "false"},
  "265": {"policy": "none", "personal_info": "false"},
  "266": {"policy": "none", "personal_info": "false"},
  "267": {"policy": "none", "personal_info": "false"},
  "268": {"policy": "none", "personal_info": "false"},
  "269": {"policy": "none", "personal_info": "false"},
  "270": {"policy": "none", "personal_info": "false"},
  "271": {"policy": "none", "personal_info": "false"},
  "272": {"policy": "none", "personal_info": "false"},
  "273": {"policy": "none", "personal_info": "false"},
  "274": {"policy": "none", "personal_info": "false"},
  "275": {"policy": "none", "personal_info": "false"},
  "276": {"policy": "none", "personal_info": "false"},
  "277": {"policy": "none", "personal_info": "false"},
  "278": {"policy": "none", "personal_info": "false"},
  "279": {"policy": "none", "personal_info": "false"},
  "280": {"policy": "none", "personal_info": "false"},
  "281": {"policy": "none", "personal_info": "false"},
  "282": {"policy": "none", "personal_info": "false"},
  "283": {"policy": "none", "personal_info": "false"},
  "284": {"policy": "none", "personal_info": "false"},
  "285": {"policy": "none", "personal_info": "false"},
  "286": {"policy": "none", "personal_info": "false"},
  "287": {"policy": "none", "personal_info": "false"},
  "288": {"policy": "none", "personal_info": "false"},
  "289": {"policy": "none", "personal_info": "false"},
  "290": {"policy": "none", "personal_info": "false"},
  "291": {"policy": "none", "personal_info": "false"},
  "292": {"policy": "none", "personal_info": "false"},
  "293": {"policy": "none", "personal_info": "false"},
  "294": {"policy": "none", "personal_info": "false"},
  "295": {"policy": "none", "personal_info": "false"},
  "296": {"policy": "none", "personal_info": "false"},
  "297": {"policy": "none", "personal_info": "false"},
  "298": {"policy": "none", "personal_info": "false"},
  "299": {"policy": "none", "personal_info": "false"},
  "300": {"policy": "none", "personal_info": "false"},
  "301": {"policy": "none", "personal_info": "false"},
  "302": {"policy": "none", "personal_info": "false"},
  "303": {"policy": "none", "personal_info": "false"},
  "304": {"policy": "none", "personal_info": "false"},
  "305": {"policy": "none", "personal_info": "false"},
  "306": {"policy": "none", "personal_info": "false"},
  "307": {"policy": "none", "personal_info": "false"},
  "308": {"policy": "none", "personal_info": "false"},
  "309": {"policy": "none", "personal_info": "false"},
  "310": {"policy": "none", "personal_info": "false"},
  "311": {"policy": "none", "personal_info": "false"},
  "312": {"policy": "none", "personal_info": "false"},
  "313": {"policy": "none", "personal_info": "false"},
  "314": {"policy": "none", "personal_info": "false"},
  "315": {"policy": "none", "personal_info": "false"},
  "316": {"policy": "none", "personal_info": "false"},
  "317": {"policy": "none", "personal_info": "false"},
  "318": {"policy": "none", "personal_info": "false"},
  "319": {"policy": "none", "personal_info": "false"},
  "320": {"policy": "none", "personal_info": "false"},
  "321": {"policy": "none", "personal_info": "false"},
  "322": {"policy": "none", "personal_info": "false"},
  "323": {"policy": "none", "personal_info": "false"},
  "324": {"policy": "none", "personal_info": "false"},
  "325": {"policy": "none", "personal_info": "false"},
  "326": {"policy": "none", "personal_info": "false"},
  "327": {"policy": "none", "personal_info": "false"},
  "328": {"policy": "none", "personal_info": "false"},
  "329": {"policy": "none", "personal_info": "false"},
  "330": {"policy": "none", "personal_info": "false"},
  "331": {"policy": "none", "personal_info": "false"},
  "332": {"policy": "none", "personal_info": "false"},
  "333": {"policy": "none", "personal_info": "false"},
  "334": {"policy": "none", "personal_info": "false"},
  "335": {"policy": "none", "personal_info": "false"},
  "336": {"policy": "none", "personal_info": "false"},
  "337": {"policy": "none", "personal_info": "false"},
  "338": {"policy": "none", "personal_info": "false"},
  "339": {"policy": "none", "personal_info": "false"},
  "340": {"policy": "none", "personal_info": "false"},
  "341": {"policy": "none", "personal_info": "false"},
  "342": {"policy": "none", "personal_info": "false"},
  "343": {"policy": "none", "personal_info": "false"},
  "344": {"policy": "none", "personal_info": "false"},
  "345": {"policy": "none", "personal_info": "false"},
  "346": {"policy": "none", "personal_info": "false"},
  "347": {"policy": "none", "personal_info": "false"},
  "348": {"policy": "none", "personal_info": "false"},
  "349": {"policy": "none", "personal_info": "false"},
  "350": {"policy": "none", "personal_info": "false"},
  "351": {"policy": "none", "personal_info": "false"},
  "352": {"policy": "none", "personal_info": "false"},
  "353": {"policy": "none", "personal_info": "false"},
  "354": {"policy": "none", "personal_info": "false"},
  "355": {"policy": "none", "personal_info": "false"},
  "356": {"policy": "none", "personal_info": "false"},
  "357": {"policy": "none", "personal_info": "false"},
  "358": {"policy": "none", "personal_info": "false"},
  "359": {"policy": "none", "personal_info": "false"},
  "360": {"policy": "none", "personal_info": "false"},
  "361": {"policy": "none", "personal_info": "false"},
  "362": {"policy": "none", "personal_info": "false"},
  "363": {"policy": "none", "personal_info": "false"},
  "364": {"policy": "none", "personal_info": "false"},
  "365": {"policy": "none", "personal_info": "false"},
  "366": {"policy": "none", "pe
Decisions: Route calendar 0% right
{
  "0": {"tool": "web_search", "probability": 0.7},
  "1": {"tool": "calculator", "probability": 0.3},
  "2": {"tool": "calendar", "probability": 0.1},
  "3": {"tool": "email", "probability": 0.05},
  "4": {"tool": "none", "probability": 0.05},
  "5": {"tool": "confirm", "probability": 0.05},
  "6": {"tool": "none", "probability": 0.1}
}
Decisions: Doc invoice missing due 0% right
{
  "2291": {
    "doc_type": {
      "invoice": 0.5,
      "resume": 0,
      "contract": 0,
      "bank_statement": 0,
      "other": 0
    },
    "missing_due_date": {
      "true": 0.3,
      "false": 0.7
    }
  }
Decisions: Phishing paypal 0% right
{
  "1": {" phishing": 0.5, "risk": 0.2 },
  "2": {" phishing": 0.3, "risk": 0.3 },
  "3": {" phishing": 0.1, "risk": 0.9 }
}
Decisions: Pii ssn email 0% right
{
  "1": {"data_kind": "financial", "sensitive": "false", "probability": 0.25},
  "2": {"data_kind": "government_id", "sensitive": "false", "probability": 0.1},
  "3": {"data_kind": "health", "sensitive": "false", "probability": 0.1},
  "4": {"data_kind": "contact", "sensitive": "true", "probability": 0.6},
  "5": {"data_kind": "financial", "sensitive": "true", "probability": 0.3},
  "6": {"data_kind": "government_id", "sensitive": "true", "probability": 0.2},
  "7": {"data_kind": "health", "sensitive": "true", "probability": 0.4},
  "8": {"data_kind": "contact", "sensitive": "false", "probability": 0.1},
  "9": {"data_kind": "financial", "sensitive": "false", "probability": 0.4},
  "10": {"data_kind": "government_id", "sensitive": "false", "probability": 0.2},
  "11": {"data_kind": "health", "sensitive": "false", "probability": 0.3},
  "12": {"data_kind": "contact", "sensitive": "false", "probability": 0.1},
  "13": {"data_kind": "financial", "sensitive": "true", "probability": 0.3},
  "14": {"data_kind": "government_id", "sensitive": "true", "probability": 0.2},
  "15": {"data_kind": "health", "sensitive": "true", "probability": 0.4}
Decisions: Review mixed 0% right
{
  "1": {"sentiment": "positive", "defect": "false", "recommend": "false"},
  "2": {"sentiment": "positive", "defect": "false", "recommend": "true"},
  "3": {"sentiment": "positive", "defect": "true", "recommend": "false"},
  "4": {"sentiment": "positive", "defect": "false", "recommend": "false"},
  "5": {"sentiment": "positive", "defect": "false", "recommend": "true"},
  "6": {"sentiment": "neutral", "defect": "false", "recommend": "false"},
  "7": {"sentiment": "negative", "defect": "false", "recommend": "false"}
}
Documents: Saas escalator (hard) 0% right
{
  "year2_price_per_seat_month": 45.00,
  "year3_price_per_seat_month": 45.00,
  "year1_invoice": 0,
  "year2_invoice": 0,
  "addon_months_billed": 0,
  "addon_invoice": 0,
  "year3_invoice": 0,
  "year3_discount_percent": 0,
  "total_contract_value": 0,
  "contract_end_date": "2024-03-01"
}
Documents: Expense thread 0% right
{
  "employee_id": "",
  "destination_city": "Lisbon",
  "trip_start": "2025-02-24",
  "trip_end": "2025-02-27",
  "approved_items": [
    {
      "date": "2025-02-24",
      "category": "flight",
      "amount_usd": 1184.60,
      "amount_cents": 118400
    },
    {
      "date": "2025-02-24",
      "category": "taxi",
      "amount_usd": 3800.00,
      "amount_cents": 380000
    },
    {
      "date": "2025-02-25",
      "category": "client_dinner",
      "amount_usd": 21240.00,
      "amount_cents": 212400
    },
    {
      "date": "2025-02-26",
      "category": "hotel",
      "amount_usd": 4470.00,
      "amount_cents": 447000
    },
    {
      "date": "2025-02-27",
      "category": "taxi",
      "amount_usd": 4150.00,
      "amount_cents": 415000
    },
    {
      "date": "2025-02-27",
      "category": "airport_lounge_pass",
      "amount_usd": 5900.00,
      "amount_cents": 590000
    }
  ],
  "rejected_item_count": 0,
  "per_diem_days": 2,
  "per_diem_usd": 0,
  "total_reimbursable_usd": 100100.00,
  "approver_email": "dana.whitfield@corvane.com"
Documents: Lease amendment 50% right
{
  "tenants": ["Marcus Lin", "Sofia Lin"],
  "landlord": "Ridgeline Property Group LLC",
  "zip": "97205",
  "lease_end": "2025-11-30",
  "original_monthly_rent": 21500.00,
  "monthly_rent_from_2025_06_01": 21500.00,
  "late_fee_from_2025_06_01": 2150.00,
  "security_deposit": 21500.00,
  "total_pet_deposits": 800,
  "total_monthly_payment_july_2025": 23850.00,
  "move_in_payment": 21500.00
}
Documents: Ticket SLA 33% right
{
  "ticket_id": "48213",
  "account_id": "7731",
  "open_issue": "inventory_sync",
  "resolved_issues": [],
  "affected_orders": ["SO-99812", "SO-99815", "SO-99820"],
  "priority": "P2",
  "sla_due_local": "2025-09-12T15:30",
  "sla_due_utc": "2025-09-12T15:30:00",
  "reissued_invoice": null
}
Documents: Sales footnotes 0% right
{
  "q3_total_usd": null,
  "q2_total_usd": null,
  "q2_central_originally_reported_usd": null,
  "q2_to_q3_change_pct": null,
  "top_region_q3": "Central",
  "fastest_growing_region_q1_to_q3": "East",
  "regions_declining_q2_to_q3": null,
  "international_q3_organic_usd": null,
  "west_excluding_mountain_q3_usd": null
}

Size: 1.2B parameters. First tested OCT 11.

Models that scored about the same

Comments

Sign in with GitHub to comment. Spam and abuse are hidden automatically.