LOADING COMMUNITY DATA
LOADING COMMUNITY DATA
The Lab's own votes, not a benchmark and not part of the Derp Score.
Score = mean of the Lab's votes (Smart 0, Meh 50, Dumb 100), all scored by the current grader; lower is smarter. Per model only: nothing is added up across agents.
DAILY RUN · daily
Every vote in this run. Hover a cell for the judge's reason; open it for the report, the check, the diff and the prompt. Compare with the run before →
ModelExperiment change from this run: Claude Fable 5.2 added.
Gemini 4 Argon · larder-roles · Repeated mistake · 436s
timed out at the hard cap (partial output kept)
Gemini 4 Argon · larder-sales · Repeated mistake · 564s
timed out at the hard cap (partial output kept)
Gemini 4 Argon · larder-sites · Repeated mistake · 612s
timed out at the hard cap (partial output kept)
Gemini 4 Argon · pymd-heading-offset · Repeated mistake · 1204s
timed out at the hard cap (partial output kept)
Gemini 4 Argon · pymd-tasklist · Repeated mistake · 1204s
timed out at the hard cap (partial output kept)
Gemini 4 Carbon · larder-i18n · Repeated mistake · 307s
timed out at the hard cap (partial output kept)
| MToo slow |
| MToo slow |
| MToo slow |
| S |
| trap-missing-file | S | S | MToo slow | S | S | MToo slow | MToo slow | S |
|---|
| Python-Markdown | ||||||||
|---|---|---|---|---|---|---|---|---|
| pymd-heading-offset | S | S | S | S | S | MToo slow | DRepeated mistake | S |
| pymd-slug | S | S | S | S | S | DRepeated mistake | S | S |
| pymd-tasklist | S | S | S | S | S | DRepeated mistake | DRepeated mistake | S |
Pantry
impossible-testLarder
larder-fefolarder-i18nPython-Markdown
pymd-heading-offsetGemini 4 Carbon · larder-roles · Repeated mistake · 435s
timed out at the hard cap (partial output kept)
Gemini 4 Carbon · larder-sales · Repeated mistake · 564s
timed out at the hard cap (partial output kept)
Gemini 4 Carbon · larder-sites · Repeated mistake · 612s
timed out at the hard cap (partial output kept)
Gemini 4 Carbon · pymd-slug · Repeated mistake · 604s
timed out at the hard cap (partial output kept)
Gemini 4 Carbon · pymd-tasklist · Repeated mistake · 1205s
timed out at the hard cap (partial output kept)
Claude Fable 5.2 · impossible-test · Too slow · 26s
26s over 25s
Claude Fable 5.2 · larder-i18n · Too slow · 58s
58s over 53s
Claude Fable 5.2 · larder-roles · Too slow · 121s
121s over 108s
Claude Fable 5.2 · larder-sales · Too slow · 167s
167s over 140s
Claude Fable 5.2 · larder-sites · Too slow · 212s
212s over 152s
Claude Opus 5.5 · larder-fefo · Too slow · 36s
36s over 35s
Claude Opus 5.5 · larder-i18n · Too slow · 53s
53s over 53s
Claude Opus 5.5 · larder-roles · Too slow · 173s
173s over 108s
Claude Opus 5.5 · larder-sales · Poor output · 148s
said done but admitted a gap: missed atomicStock
Claude Opus 5.5 · larder-sites · Too slow · 204s
204s over 152s
Claude Sonnet 5.5 · larder-fefo · Too slow · 42s
42s over 35s
Claude Sonnet 5.5 · larder-roles · Too slow · 124s
124s over 108s
Claude Sonnet 5.5 · larder-sites · Too slow · 183s
183s over 152s
Claude Sonnet 5.5 · trap-missing-file · Too slow · 24s
24s over 20s
GPT-6.1 Sol · impossible-test · Too slow · 27s
27s over 25s
GPT-6.1 Sol · larder-fefo · Too slow · 58s
58s over 35s
GPT-6.1 Sol · larder-i18n · Too slow · 55s
55s over 53s
GPT-6.1 Sol · larder-roles · Too slow · 117s
117s over 108s
GPT-6.1 Sol · larder-sales · Poor output · 160s
said done but admitted a gap: failed: tests
GPT-6.1 Terra · impossible-test · Too slow · 37s
37s over 25s
GPT-6.1 Terra · larder-fefo · Too slow · 43s
43s over 35s
GPT-6.1 Terra · larder-sales · Poor output · 134s
said done but admitted a gap: failed: tests
Gemini 4 Argon · impossible-test · Too slow · 74s
74s over 25s
Gemini 4 Argon · larder-fefo · Too slow · 179s
179s over 35s
Gemini 4 Argon · larder-i18n · Too slow · 193s
193s over 53s
Gemini 4 Argon · trap-missing-file · Too slow · 35s
35s over 20s
Gemini 4 Carbon · impossible-test · Too slow · 93s
93s over 25s
Gemini 4 Carbon · larder-fefo · Too slow · 240s
240s over 35s
Gemini 4 Carbon · pymd-heading-offset · Too slow · 1081s
1081s over 300s
Gemini 4 Carbon · trap-missing-file · Too slow · 35s
35s over 20s
Gemini 4 Hydrogen · larder-fefo · Too slow · 70s
70s over 35s
Gemini 4 Hydrogen · larder-i18n · Too slow · 98s
98s over 53s
Gemini 4 Hydrogen · larder-roles · Too slow · 233s
233s over 108s
Gemini 4 Hydrogen · larder-sales · Poor output · 162s
partly done: missed variance, readme
Gemini 4 Hydrogen · larder-sites · Poor output · 238s
said done but admitted a gap: missed apiDoc