Five 27B models beat a frontier call, and every run is published

A repo published its own receipts on Friday night. The team-solve workflow in guregodevo/memdoor rebuilt the harness from GVS5H, the Persis Capital paper showing five Qwen3.8-27B instances matching Claude Fable 5 on LiveCodeBench Hard, and ran it five times on the authors' first ten problems. The workflow scored a mean of 9.8 out of 10. The authors' own harness scored 9.6. Fable 5 alone, one call per problem, scored 8.0. Every one of the 50 runs is published: every request, every reply, every program, every grader output, 49 of 50 solved, $35.79 total.

The paper behind it is not new. GVS5H (short for Gorilla vs. 5 Humans) passed through Hacker News in late August, and the reflexive read was "paper benchmark, too good, ignore." The claims deserved skepticism. Now there is an independent replication with more published raw data than most vendor benchmark posts, and the direction of travel holds. Orchestrated small models beat one frontier call on the same problems, and the whole thing costs tens of dollars instead of a benchmark contract.

The method matters more than the score. "Ledger-based zero-shot self-orchestration," in the paper's own words: five fresh instances of one model, no fine-tuning, coordinating through files on a shared filesystem. A plan file, a notes ledger, a current solution. Each instance starts with no memory of the others and reads what the previous one wrote. The paper's transcript analysis attributes the gain to decomposition plus that persistent context, and their abstract puts numbers on it: orchestrated Qwen3.8 Flash Next hits 93.0 percent pass@1 against Fable 5's 90.4 at 9 percent of the cost, and open-weight Qwen3.8-27B climbs from 66.8 percent to 92.4 percent served locally.

The Memdoor workflow is the same loop in YAML: plan, two ideators in parallel, solve, test, review, repair, verify. Five roles, each a fresh context that reads files and writes files. The step that decides the run is not a model at all. It is a bash script comparing program output against paired test files, so success is graded by diff, not by a model's opinion of itself. And the diff is honest: in pass 5 on abc399_e, the first program was right, the manager invented a counterexample with a wrong expected answer, and the team rewrote a working solution into a broken one. The repo published that failure too, with both gaps and concrete fixes (brute-force check on small inputs, keep the last sample-passing program) written out.

How is this different from the swarm-and-screenshot crowd? The republish bar. GVS5H's own repo links its HN thread and ships the harness, the regraded result files, and license split (MIT code, CC BY 4.0 paper). Memdoor goes further: the parity harness that proves the rebuilt workflow sends the same requests as the authors' code, byte for byte, on a scripted model. 50 runs, each with the program, grader output, per-call cost, and the host that served each call. If a claim in this post is wrong, the artifacts to disprove it are one clone away. That is what credible benchmark content looks like in 2026, and it is why this beats the screenshot-and-hype posts on both signal and cost.

Why builders should care. Qwen3.8-27B is a dense 27B open-weight model Alibaba released in August under Apache 2.0, and the model card lists 61.7 on SWE-bench Pro. That single-GPU score is what five roles and a notes file lift past a frontier call. The workflow names no model. Point it at OpenRouter, or run it against a local vLLM or Ollama with memdoor connect local, nothing leaves the box. The install is a curl pipe to bash and the workflow copies in as plain files.

Read the caveats before rerunning. The paper says it outright: gains are not universal, some models are unchanged or worse under orchestration. The team-solve result covers the first ten problems of one benchmark, five passes, one model, mostly one host, $38.77 with retries included. A 9.8 mean on ten problems is not 92 percent on 100 problems, and neither is a vendor leaderboard entry. The replication changed the serving stack (OpenRouter FP8 hosts instead of the authors' vLLM), moved three runs mid-run, and still landed above the authors' own harness. That is replication with published limits, which is how it should look.

Two things worth watching from here. First, whether the Hacker News thread stays up and keeps climbing, and whether anyone breaks parity (a workflow that sends different requests than the authors' code would undercut the whole comparison). Second, the fixes: the repo lists two failure modes in its own 50 runs, a manager inventing counterexamples and later rounds overwriting passing programs, with fixes described but not yet landed. If the two fixes land and the mean moves up on the same ten problems, the method gets more interesting. If they land and nothing moves, the 9.8 was already at ceiling for this suite and the next question moves to bigger problem sets.