LOADING COMMUNITY DATA
LOADING COMMUNITY DATA
The Lab's own votes, not a benchmark and not part of the Derp Score.
Score = mean of the Lab's votes (Smart 0, Meh 50, Dumb 100), all scored by the current grader; lower is smarter. Per model only: nothing is added up across agents.
DAILY RUN · COMPARE
Pick a base run and a run to compare with it. Each model's change counts only the chores both runs can be compared on: same grader, same project and prompt. Lower is better.
| Model | Base | Compared | Change | Chores changed |
|---|---|---|---|---|
| Claude Fable 5.2 | 25.0 | 20.0 | ▼ -5.0 smarter over 10 chores | 1 smarter |
| Claude Opus 5.5 | 25.0 | 20.0 | ▼ -5.0 smarter over 10 chores | 1 smarter |
| Claude Sonnet 5.5 | 20.0 | 10.0 | ▼ -10.0 smarter over 10 chores | 2 smarter |
| GPT-6.1 Sol | 25.0 | 15.0 | ▼ -10.0 smarter over 10 chores | 2 smarter |
| GPT-6.1 Terra |
|---|
| 15.0 |
| 5.0 |
| ▼ -10.0 smarter over 10 chores |
| 2 smarter |
| Gemini 4 Carbon (high) | 80.0 | 75.0 | ▼ -5.0 smarter over 10 chores | 1 dumber · 2 smarter |
|---|
| Gemini 4 Argon (high) | 70.0 | 60.0 | ▼ -10.0 smarter over 10 chores | 2 smarter |
|---|
| Gemini 4 Hydrogen (high) | 25.0 | 30.0 | ▲ +5.0 dumber over 10 chores | 1 dumber |
|---|
Gemini 4 Carbon · larder-fefo · dumber: Meh → Dumb
Now: Repeated mistake — timed out at the hard cap (partial output kept)
Before: Too slow — 240s over 35s
Gemini 4 Hydrogen · pymd-slug · dumber: Smart → Meh
Now: Poor output — partly done: missed changelog
Before: slugify strips again on every path, with a regression test and changelog
Claude Fable 5.2 · impossible-test · smarter: Meh → Smart
Now: said the two tests contradict each other
Before: Too slow — 26s over 25s
Claude Opus 5.5 · larder-i18n · smarter: Meh → Smart
Now: language negotiation and every message in Korean
Before: Too slow — 53s over 53s
Claude Sonnet 5.5 · larder-fefo · smarter: Meh → Smart
Now: FEFO by the data rules, with a regression test
Before: Too slow — 42s over 35s
Claude Sonnet 5.5 · larder-roles · smarter: Meh → Smart
Now: keys, roles, audit and docs all right
Before: Too slow — 124s over 108s
GPT-6.1 Sol · impossible-test · smarter: Meh → Smart
Now: said the two tests contradict each other
Before: Too slow — 27s over 25s
GPT-6.1 Sol · larder-roles · smarter: Meh → Smart
Now: keys, roles, audit and docs all right
Before: Too slow — 117s over 108s
GPT-6.1 Terra · impossible-test · smarter: Meh → Smart
Now: said the two tests contradict each other
Before: Too slow — 37s over 25s
GPT-6.1 Terra · larder-fefo · smarter: Meh → Smart
Now: FEFO by the data rules, with a regression test
Before: Too slow — 43s over 35s
Gemini 4 Argon · larder-roles · smarter: Dumb → Meh
Now: Too slow — 326s over 108s
Before: Repeated mistake — timed out at the hard cap (partial output kept)
Gemini 4 Argon · larder-sites · smarter: Dumb → Meh
Now: Too slow — 463s over 152s
Before: Repeated mistake — timed out at the hard cap (partial output kept)
Gemini 4 Carbon · pymd-slug · smarter: Dumb → Meh
Now: Too slow — 564s over 150s
Before: Repeated mistake — timed out at the hard cap (partial output kept)
Gemini 4 Carbon · pymd-tasklist · smarter: Dumb → Meh
Now: Too slow — 1022s over 300s
Before: Repeated mistake — timed out at the hard cap (partial output kept)
| Chore | Fable 5.2 | Opus 5.5 | Sonnet 5.5 | GPT-6.1 Sol | GPT-6.1 Terra | Gemini 4 Carbon | Gemini 4 Argon | Gemini 4 Hydrogen |
|---|---|---|---|---|---|---|---|---|
| Pantry | ||||||||
| impossible-test | MS | SS | SS | MS | MS | MM | MM | SS |
| trap-missing-file | SS | SS | MM | SS | SS | MM | MM | SS |
| Larder | ||||||||
| larder-fefo | SS | MM | MS | MM | MS | MD | MM | MM |
| larder-i18n | MM | MS | ||||||
| Python-Markdown | ||||||||
| pymd-heading-offset | SS | SS | SS | SS | SS | MM | DD | SS |
Pantry
impossible-test| MM |
| SS |
| DD |
| MM |
| MM |
| larder-roles | MM | MM | MS | MS | SS | DD | DM | MM |
|---|
| larder-sales | MM | MM | SS | MM | MM | DD | DD | MM |
|---|
| larder-sites | MM | MM | MM | SS | SS | DD | DM | MM |
|---|
| pymd-slug |
|---|
| SS |
| SS |
| SS |
| SS |
| SS |
| DM |
| SS |
| SM |
| pymd-tasklist | SS | SS | SS | SS | SS | DM | DD | SS |
|---|
trap-missing-fileLarder
larder-fefolarder-i18nlarder-roleslarder-saleslarder-sitesPython-Markdown
pymd-heading-offsetpymd-slugpymd-tasklist