Best local LLM for coding: we ran the code
56 models, 30 programming jobs, each checked against tests the model never saw. Here's who writes code that works.
8 posts, newest first.
56 models, 30 programming jobs, each checked against tests the model never saw. Here's who writes code that works.
Every program it finished was correct, and it beat every local model at reading documents. But on 12 of 30 coding jobs it never gave an answer at all.
It scored 96 out of 100, #2 of 56. It solved 28 of 30 coding jobs and scored 99 on reading documents. Runs on a Mac with 96 GB.
It scored 94 out of 100, #3 of 56. It solved 27 of 30 coding jobs and scored 98 on reading documents. Runs on a 24 GB graphics card or a Mac with 32 GB.
It scored 94 out of 100, #4 of 56. It solved 28 of 30 coding jobs and scored 95 on reading documents. Runs on a Mac with 128 GB.
It scored 97 out of 100, #1 of 56. It solved 29 of 30 coding jobs and scored 97 on reading documents. Runs on a 24 GB graphics card or a Mac with 32 GB.
It scored 88 out of 100, #7 of 56. It solved 27 of 30 coding jobs and scored 87 on reading documents. Runs on a 16 GB graphics card or a Mac with 24 GB.
It scored 79 out of 100, #14 of 56. It solved 28 of 30 coding jobs and scored 65 on reading documents. Runs on a 32 GB graphics card or a Mac with 48 GB.