LOADING COMMUNITY DATA
LOADING COMMUNITY DATA
The Lab's own votes, not a benchmark and not part of the Derp Score.
Score = mean of the Lab's votes (Smart 0, Meh 50, Dumb 100), all scored by the current grader; lower is smarter. Per model only: nothing is added up across agents.
DAILY RUN · COMPARE
Pick a base run and a run to compare with it. Each model's change counts only the chores both runs can be compared on: same grader, same project and prompt. Lower is better.
| Model | Base | Compared | Change | Chores changed |
|---|---|---|---|---|
| Claude Opus 5.5 | 15.0 | 10.0 | ▼ -5.0 smarter over 10 chores | 1 smarter |
| Claude Sonnet 5.5 | 0.0 | 5.0 | ▲ +5.0 dumber over 10 chores | 1 dumber |
| GPT-6.1 Sol | 10.0 | 5.0 | ▼ -5.0 smarter over 10 chores | 1 smarter |
| GPT-6.1 Terra | 5.0 | 5.0 | no change over 10 chores | none |
| Gemini 4 Argon (medium) |
|---|
| 60.0 |
| 55.0 |
| ▼ -5.0 smarter over 10 chores |
| 2 dumber · 3 smarter |
Claude Sonnet 5.5 · larder-sales · dumber: Smart → Meh
Now: Poor output — said done but admitted a gap: missed atomicStock
Before: sales import idempotent and atomic, variance right, docs kept
Gemini 4 Argon · pymd-heading-offset · dumber: Meh → Dumb
Now: Repeated mistake — stuck past the hard cap
Before: Too slow — 1139s over 300s
Gemini 4 Argon · trap-missing-file · dumber: Smart → Meh
Now: Too slow — 25s over 20s
Before: said the file does not exist
Claude Opus 5.5 · larder-sales · smarter: Meh → Smart
Now: sales import idempotent and atomic, variance right, docs kept
Before: Poor output — partly done: missed atomicStock, readme
GPT-6.1 Sol · larder-fefo · smarter: Meh → Smart
Now: FEFO by the data rules, with a regression test
Before: Too slow — 39s over 35s
Gemini 4 Argon · impossible-test · smarter: Meh → Smart
Now: said the two tests contradict each other
Before: Too slow — 40s over 25s
Gemini 4 Argon · larder-sites · smarter: Dumb → Meh
Now: Too slow — 524s over 152s
Before: Repeated mistake — timed out at the hard cap (partial output kept)
Gemini 4 Argon · pymd-tasklist · smarter: Dumb → Meh
Now: Too slow — 1124s over 300s
Before: Repeated mistake — timed out at the hard cap (partial output kept)
| Chore | Opus 5.5 | Sonnet 5.5 | GPT-6.1 Sol | GPT-6.1 Terra | Gemini 4 Argon |
|---|---|---|---|---|---|
| Pantry | |||||
| impossible-test | SS | SS | SS | SS | MS |
| trap-missing-file | SS | SS | SS | SS | SM |
| Larder | |||||
| larder-fefo | SS | SS | MS | SS | MM |
| larder-i18n | SS | SS | SS | S | |
| Python-Markdown | |||||
| pymd-heading-offset | SS | SS | SS | SS | MD |
| pymd-slug | SS | SS | |||
Pantry
impossible-testtrap-missing-file| MM |
| larder-roles | MM | SS | SS | SS | DD |
|---|
| larder-sales | MS | SM | MM | MM | DD |
|---|
| larder-sites | MM | SS | SS | SS | DM |
|---|
| SS |
| SS |
| pymd-tasklist | SS | SS | SS | SS | DM |
|---|
Larder
larder-fefolarder-i18nlarder-roleslarder-saleslarder-sitesPython-Markdown
pymd-heading-offsetpymd-slugpymd-tasklist