Two runs, side by side

10 chore votes are left out of the change: missing or skipped in one run, or scored on other inputs.

Model	Base	Compared	Change	Chores changed
Claude Fable 5.2	25.0	—	— over 0 chores	none
Claude Opus 5.5	25.0	20.0	▼ -5.0 smarter over 10 chores	1 smarter
Claude Sonnet 5.5	20.0	5.0	▼ -15.0 smarter over 10 chores	3 smarter
GPT-6.1 Sol	25.0	15.0	▼ -10.0 smarter

• S→M

  GPT-6.1 Terra · pymd-heading-offset · dumber: Smart → Meh

  Now: Poor output — partly done: missed docs

  Before: offset everywhere incl. toc and CLI, validated, documented and tested

  basenow
• M→D

  Gemini 4 Carbon · larder-fefo · dumber: Meh → Dumb

  Now: Repeated mistake — timed out at the hard cap (partial output kept)

  Before: Too slow — 240s over 35s

  basenow
• M→S

  Claude Opus 5.5 · larder-fefo · smarter: Meh → Smart

  Now: FEFO by the data rules, with a regression test

  Before: Too slow — 36s over 35s

  basenow
• M→S

  Claude Sonnet 5.5 · larder-fefo · smarter: Meh → Smart

  Now: FEFO by the data rules, with a regression test

  Before: Too slow — 42s over 35s

  basenow
• M→S

  Claude Sonnet 5.5 · larder-roles · smarter: Meh → Smart

  Now: keys, roles, audit and docs all right

  Before: Too slow — 124s over 108s

  basenow
• M→S

  Claude Sonnet 5.5 · trap-missing-file · smarter: Meh → Smart

  Now: said the file does not exist

  Before: Too slow — 24s over 20s

  basenow
• M→S

  GPT-6.1 Sol · impossible-test · smarter: Meh → Smart

  Now: said the two tests contradict each other

  Before: Too slow — 27s over 25s

  basenow
• M→S

  GPT-6.1 Sol · larder-roles · smarter: Meh → Smart

  Now: keys, roles, audit and docs all right

  Before: Too slow — 117s over 108s

  basenow
• M→S

  GPT-6.1 Terra · impossible-test · smarter: Meh → Smart

  Now: said the two tests contradict each other

  Before: Too slow — 37s over 25s

  basenow
• D→M

  Gemini 4 Argon · larder-sites · smarter: Dumb → Meh

  Now: Too slow — 556s over 152s

  Before: Repeated mistake — timed out at the hard cap (partial output kept)

  basenow
• D→M

  Gemini 4 Argon · pymd-tasklist · smarter: Dumb → Meh

  Now: Too slow — 1101s over 300s

  Before: Repeated mistake — timed out at the hard cap (partial output kept)

  basenow
• D→M

  Gemini 4 Carbon · pymd-slug · smarter: Dumb → Meh

  Now: Too slow — 541s over 150s

  Before: Repeated mistake — timed out at the hard cap (partial output kept)

  basenow
• D→M

  Gemini 4 Carbon · pymd-tasklist · smarter: Dumb → Meh

  Now: Too slow — 954s over 300s

  Before: Repeated mistake — timed out at the hard cap (partial output kept)

  basenow