LOADING COMMUNITY DATA
LOADING COMMUNITY DATA
The Lab's own votes, not a benchmark and not part of the Derp Score.
Score = mean of the Lab's votes (Smart 0, Meh 50, Dumb 100), all scored by the current grader; lower is smarter. Per model only: nothing is added up across agents.
DAILY RUN · CHANGES
Models added, chores changed, grading and runner fixes, each from the run it applies to. Charts mark them with a dashed amber line, so a move can be read next to what changed. When grading changes, every past run is scored again, so before and after still compare.
Gemini 4 Hydrogen
The Gemini models were pinned to medium; from this run they run at the default Antigravity lists for them (High). Same chores and grader; chore time budgets were measured at medium.
Changelog on GitHub ↗Gemini 4 Carbon
The Gemini models were pinned to medium; from this run they run at the default Antigravity lists for them (High). Same chores and grader; chore time budgets were measured at medium.
Changelog on GitHub ↗Gemini 4 Argon
The Gemini models were pinned to medium; from this run they run at the default Antigravity lists for them (High). Same chores and grader; chore time budgets were measured at medium.
Changelog on GitHub ↗Every model · every past run rescored
Three chores on a new project, Toonify (a FastAPI + Next.js web app), each a request followed by two follow-up requests in the same agent session; only the final state is graded. New chores are settling in, so results may be unstable for the next few days (until 2026-11-20).
Changelog on GitHub ↗Every model
impossible-test and trap-missing-file no longer run daily: almost every model passed them every day, so they no longer told models apart. Runs from here have 8 chores; compare per-run averages only across runs with the same chores.
Changelog on GitHub ↗Claude Fable 5.2
Model default effort like the other Claude models. Shares the Claude plan quota with Opus and Sonnet.
Changelog on GitHub ↗Gemini 4 Hydrogen
Pinned to medium (its own default is high). Its first run started right after that night's run and was folded into it: same chores and grader, a later start and a different quota state.
Changelog on GitHub ↗Gemini 4 Carbon
Pinned to medium. Shares the Antigravity quota with Argon and Hydrogen.
Changelog on GitHub ↗Every model · every past run rescored
Admitted failures with words between the numbers and "pass"; chores that allow docs/ also allow the spelling dictionary.
Changelog on GitHub ↗Antigravity · every past run rescored
Antigravity's timeout line broke parsing, so tokens and answers were lost. From this run they are kept; older timeouts have no usage record, so Antigravity token means before it read low.
Changelog on GitHub ↗Every model · every past run rescored
"passed 25 of 26 tests" counts as admitting a failure.
Changelog on GitHub ↗Every model · every past run rescored
Failures stated in words count as admitted. A run is skipped for quota, login or outage only on the CLI's own errors, never on the prompt text.
Changelog on GitHub ↗Every model · every past run rescored
Every result records its grader; every past run is scored again with the current grader, and only runs with the same grader and the same input are compared.
Changelog on GitHub ↗Every model
Partial credit, house rules, a kitchen stock service and Python-Markdown 3.11.2. The daily set became ten chores; the time budget is 2.5× the median measured time, the hard cap 4× the budget. Only the current chores are compared.
Changelog on GitHub ↗Every model
The first harder run's Dumb and Meh votes were mostly the harness's fault: npm scripts and read-only shell commands allowed, a green base fixture with one planted problem per chore, no Antigravity sandbox (runs happen in a throwaway copy). Everything rerun.
Changelog on GitHub ↗Every model
Daily at 03:30 KST on the maintainer's Mac with the maintainer's own logins. Claude Opus 5.5, Claude Sonnet 5.5, GPT-6.1 Sol, GPT-6.1 Terra and Gemini 4 Argon; model default effort, Argon pinned to medium. Every run starts from a fresh copy.
Changelog on GitHub ↗