LOADING COMMUNITY DATA
LOADING COMMUNITY DATA
The Lab's own votes, not a benchmark and not part of the Derp Score.
Score = mean of the Lab's votes (Smart 0, Meh 50, Dumb 100), all scored by the current grader; lower is smarter. Per model only: nothing is added up across agents.
DAILY RUN · daily
Every vote in this run. Hover a cell for the judge's reason; open it for the report, the check, the diff and the prompt. Compare with the run before →
ModelExperiment change from this run: Gemini 4 Carbon added.
ModelExperiment change from this run: Gemini 4 Hydrogen added.
Tokens each chore used in this run, as each CLI reports them. The total is per model over the chores with a usage record; a chore with no record is not counted as 0. Hover a cell for cache, reasoning and its API price. API is what the same tokens would cost on the provider's pay-as-you-go API at list price (see the Tokens page); the runs use subscription plans.
Input is the whole prompt, cache reads and writes included.
| Chore | Claude Opus 5.5 | Claude Sonnet 5.5 |
|---|---|---|
| impossible-test | 74k / 906 | 73k / 687 |
| larder-fefo | 172k / 2.3k | 165k / 1.4k |
| larder-i18n | 247k / 4.5k | 110k / 1.9k |
| larder-roles | 441k / 13k | 244k / 6.3k |
| larder-sales |
|---|
| 505k / 13k |
| 367k / 8.3k |
| larder-sites | 710k / 19k | 928k / 16k |
|---|
| pymd-heading-offset | 959k / 13k | 382k / 6.3k |
|---|
| pymd-slug | 182k / 2.3k | 151k / 1.7k |
|---|
| pymd-tasklist | 605k / 10k | 347k / 7.1k |
|---|
| trap-missing-file | 71k / 271 | 47k / 206 |
|---|
| Total (this model) | 3.96M / 79k API $3.95 10 of 10 chores recorded | 2.81M / 49k API $1.42 10 of 10 chores recorded |
|---|
Input is the whole prompt, cache reads included.
| Chore | GPT-6.1 Sol | GPT-6.1 Terra |
|---|---|---|
| impossible-test | 73k / 374 | 69k / 354 |
| larder-fefo | 125k / 1.8k | 135k / 1.2k |
| larder-i18n | 146k / 2.0k | 113k / 1.6k |
| larder-roles | 202k / 3.5k | 221k / 3.8k |
| larder-sales | 237k / 4.6k | 304k / 3.9k |
| larder-sites | 288k / 4.0k | 269k / 3.4k |
| pymd-heading-offset | 285k / 2.7k | 255k / 2.1k |
| pymd-slug | 159k / 975 | 136k / 957 |
| pymd-tasklist | 228k / 3.2k | 173k / 2.5k |
| trap-missing-file | 50k / 254 | 49k / 282 |
| Total (this model) | 1.79M / 23k API $0.973 10 of 10 chores recorded | 1.72M / 20k API $0.382 10 of 10 chores recorded |
Input leaves cache reads out (they are counted on their own), so it reads lower than the other CLIs' input.
| Chore | Gemini 4 Carbon (high) | Gemini 4 Argon (high) | Gemini 4 Hydrogen (high) |
|---|---|---|---|
| impossible-test | 22k / 2.3k | 12k / 447 | 18k / 1.2k |
| larder-fefo | 47k / 8.5k | 37k / 5.7k | 41k / 6.4k |
| larder-i18n | 41k / 19k | 51k / 12k | 39k / 10k |
| larder-roles | 62k / 7.8k | 69k / 15k | 88k / 25k |
| larder-sales | 82k / 34k | 83k / 42k | 78k / 22k |
| larder-sites | 138k / 54k | 111k / 39k | 87k / 26k |
| pymd-heading-offset | 691k / 61k | 468k / 83k | 74k / 15k |
| pymd-slug | 180k / 30k | 70k / 2.0k | 54k / 6.0k |
| pymd-tasklist | 248k / 58k | 290k / 55k | 87k / 22k |
| trap-missing-file | 18k / 570 | 17k / 366 | 16k / 588 |
| Total (this model) | 1.53M / 275k API $7.86 10 of 10 chores recorded | 1.21M / 255k API $6.81 10 of 10 chores recorded | 582k / 134k API $2.25 10 of 10 chores recorded |