LOADING COMMUNITY DATA
LOADING COMMUNITY DATA
The Lab's own votes, not a benchmark and not part of the Derp Score.
Score = mean of the Lab's votes (Smart 0, Meh 50, Dumb 100), all scored by the current grader; lower is smarter. Per model only: nothing is added up across agents.
DAILY RUN · COMPARE
Pick a base run and a run to compare with it. Each model's change counts only the chores both runs can be compared on: same grader, same project and prompt. Lower is better.
42 chore votes are left out of the change: missing or skipped in one run, or scored on other inputs.
| Model | Base | Compared | Change | Chores changed |
|---|---|---|---|---|
| Claude Fable 5.2 | 13.6 | 15.0 | no change over 7 chores | 1 dumber · 1 smarter |
| Claude Opus 5.5 | 4.5 | 10.0 | ▲ +6.3 dumber over 8 chores | 1 dumber |
| Claude Sonnet 5.5 | 0.0 | 0.0 | no change over 8 chores | none |
| GPT-6.1 Sol | 13.6 | 20.0 | no change |
| 1 dumber · 1 smarter |
| GPT-6.1 Terra | 18.2 | 10.0 | no change over 8 chores | none |
|---|
| Gemini 4 Carbon (high) | 77.8 | 85.0 | ▲ +12.5 dumber over 8 chores | 2 dumber |
|---|
| Gemini 4 Argon (high) | 63.6 | 75.0 | ▲ +12.5 dumber over 8 chores | 2 dumber |
|---|
| Gemini 4 Hydrogen (high) | — | 50.0 | — over 0 chores | none |
|---|
Claude Fable 5.2 · larder-fefo · dumber: Smart → Meh
Now: Too slow — 42s over 35s
Before: FEFO by the data rules, with a regression test
Claude Opus 5.5 · larder-sites · dumber: Smart → Meh
Now: Too slow — 177s over 152s
Before: sites through storage, stock, cooking, alerts, reports, stock-takes and docs
GPT-6.1 Sol · larder-i18n · dumber: Smart → Meh
Now: Too slow — 58s over 53s
Before: language negotiation and every message in Korean
Gemini 4 Argon · pymd-heading-offset · dumber: Smart → Meh
Now: Too slow — 404s over 300s
Before: offset everywhere incl. toc and CLI, validated, documented and tested
Gemini 4 Argon · pymd-slug · dumber: Smart → Meh
Now: Too slow — 192s over 150s
Before: slugify strips again on every path, with a regression test and changelog
Gemini 4 Carbon · pymd-heading-offset · dumber: Meh → Dumb
Now: Repeated mistake — timed out at the hard cap (partial output kept)
Before: Too slow — 1097s over 300s
Gemini 4 Carbon · pymd-slug · dumber: Meh → Dumb
Now: Repeated mistake — timed out at the hard cap (partial output kept)
Before: Too slow — 522s over 150s
Claude Fable 5.2 · larder-i18n · smarter: Meh → Smart
Now: language negotiation and every message in Korean
Before: Too slow — 53s over 53s
GPT-6.1 Sol · larder-roles · smarter: Meh → Smart
Now: keys, roles, audit and docs all right
Before: Too slow — 114s over 108s
| Chore | Fable 5.2 | Opus 5.5 | Sonnet 5.5 | GPT-6.1 Sol | GPT-6.1 Terra | Gemini 4 Carbon | Gemini 4 Argon | Gemini 4 Hydrogen |
|---|---|---|---|---|---|---|---|---|
| Pantry | ||||||||
| impossible-test | ·S | ·S | ·S | ·M | ·S | ·M | ·M | ·S |
| toon-episode-copy | S· | S· | S· | S· | S· | M· | M· | ·· |
| toon-failed-charges | S· | S· | S· | S· | D· | ·· | M· | ·· |
| toon-image-fetch | S· | S· | S· | S· | S· | ·· | M· | ·· |
| trap-missing-file | ·S | ·S | ·S | ·S | ·S | ·M | ·M | ·· |
| Larder | ||||||||
| larder-fefo | SM | SS | SS | MM | MM | MM | MM | ·M |
| larder-i18n | MS | S | ||||||
| Python-Markdown | ||||||||
| pymd-heading-offset | SS | SS | SS | SS | SS | MD | SM | ·D |
| pymd-slug | ||||||||
Pantry
impossible-testtoon-episode-copy| SS |
| SM |
| SS |
| DD |
| DD |
| ·M |
| larder-roles | MM | MM | SS | MS | SS | DD | DD | ·M |
|---|
| larder-sales | SS | SS | SS | MM | MM | DD | DD | ·M |
|---|
| larder-sites | MM | SM | SS | SS | SS | DD | DD | ·M |
|---|
| SS |
| SS |
| SS |
| SS |
| MD |
| SM |
| ·– |
| pymd-tasklist | SS | SS | SS | SS | SS | DD | DD | ·· |
|---|
toon-failed-chargestoon-image-fetchtrap-missing-fileLarder
larder-fefolarder-i18nlarder-roleslarder-saleslarder-sitesPython-Markdown
pymd-heading-offsetpymd-slugpymd-tasklist