We gave a plain Claude Code session a research brief it could fail at honestly. It reproduced our results, then corrected two of our conclusions and reported a hole in our own benchmark. This is what we learned about scoring agents on real work.
Most harness comparisons measure the setup rather than the work.
The usual method is to give two systems a task, look at both outputs, and prefer the one that reads better. That measures prose. It does not measure whether the analysis is right, whether the parameters were chosen honestly, or whether the system noticed that one part of the task was impossible.
We wanted a test where a weak run and a strong run look different on the evidence rather than on style. That means three things have to be true before any agent starts. The input has to be frozen, so two systems are not quietly working from different numbers. The task has to contain something that cannot be done, so we can see how a system handles a dead end. And there has to be a control, so a confident answer can be checked against the answer you would get with no method at all.
The ShurIQ weekly stack-rank report gave us a real problem to build that test on.
The Structural Brand Power Index scores a company on five dimensions and adds them into a composite out of 100. The weekly report ranks on the composite. That collapse throws something away.
Every company already sits at a position in five dimensions, and the ranking only ever shows where that position falls on a single line. Two companies can hold the same score and be built in opposite ways. ByteDance reaches its 67.55 on distribution and money infrastructure while scoring 45 on content and 38 on narrative. A pure-play studio reaches a similar total from the other direction. The ranking calls them neighbours. In the five dimensions they are far apart, and that distance is a competitive fact the weekly has no way to state.
Geometric graph generators are the family of algorithms that turn positions in a space into connections between things. There are eight of them in the NetworkX library, each drawing those connections under different rules. That gave us eight tasks with one shared input, which is exactly the shape a benchmark wants.
One script extracts the board into a five-dimensional point cloud and records the SHA-256 of the state file it came from. Every agent reads that file and no agent re-derives coordinates. If a later run disagrees with an earlier one, the disagreement is in the analysis rather than in the data.
One of the eight is a threshold graph, which connects companies by score alone and ignores position entirely. It reproduces the ranking we already publish. Its job is to make the geometric results prove they add something. If a geometric answer tells the same story as the control, the geometry contributed nothing and the work should not ship.
Another of the eight is a Waxman graph, which draws connections from distance alone with no reference to the real board. It is the baseline that turns an impression into a claim. Without it, "these companies group together" is something a person saw in a picture.
The sixth algorithm misbehaves on this data. At a low threshold it connects everything, and at a high one the most connected companies come out as the weakest ones, which is backwards. We documented the failure in the brief and asked each agent to either fix it with a stated reason or reject it with a stated reason. Quietly tuning around it fails. How a system handles the task it cannot complete is worth more than the seven it can.
The six scoring criteria are published to the agent in machine-readable frontmatter. Only one thing is sealed: a preliminary answer we already had to one of the three questions. The benchmark tests whether a system can do the work, not whether it can guess what the work is.
A plain Claude Code session, given the brief and nothing else, built all eight demonstrations plus the archive test, a typed graph bundle and a rendered map. It reproduced both required baselines exactly: 42 pairings in 8 groups at radius 0.30, and 94 connections passing the threshold-graph check.
Then it disagreed with us twice and reported a hole in our setup.
Our framework analysis presented the platform giants clustering together as the result that justified the whole method. It holds across a window of two hundredths in one parameter. What actually carries the finding is the archive: Amazon and Netflix formed a pair of their own in all 20 weeks before the other two joined the board, and when those two arrived they joined that pair rather than forming a new one. The radius sweep is a sensitivity check. We had been reading it as the evidence.
We expected the soft random geometric graph to earn the permanent report slot, because it is the only one of the eight whose output carries its own confidence figure. The run disproved it on the data. Because the connection probability falls steadily as distance grows, the frequency ordering is the distance ordering, so on its own the algorithm restates a result we already had with extra arithmetic. It recommended a different algorithm and attached the confidence figures to that one instead.
The null model produced the sharpest result of the eight, and it is the one a person eyeballing a diagram would never reach. The observed number of connections sits 0.45 standard deviations below what distance alone predicts, which means the density of the board is entirely ordinary. Across 200 seeds at three settings, the null model recovered the platform-giant group zero times. The density is unremarkable and the arrangement is not, and only running the baseline separates those two statements.
We asked for a note on what the run could not do. It returned the most useful document of the set.
The brief asks each agent to close with a self-report covering what it built, what it could not build, and where it is least certain. We expected a summary. What came back leads with its own failures and lists five judgement calls a reviewer might want to overturn, each with the reasoning attached so the reviewer can disagree on the data rather than on authority.
Four things in it we would not have found by reading the results:
The leak was real and the ambiguity was ours. We resolved it in the opposite direction from the obvious one: the rubric is now explicitly open and agents are told to read it, while only the sealed answer stays sealed. An agent that has to guess the criteria is being tested on mind-reading.
The lab exists because we are building toward an agency that runs continuously, and continuous work cannot be reviewed line by line. Deciding which system to trust with unattended work needs a scorecard, and a scorecard needs tasks where being wrong is visible.
Three properties of this brief turn out to matter more than the subject:
| Property | What it catches |
|---|---|
| A control with no method in it | An impressive answer that a coin flip would also have produced |
| A deliberately broken sub-task | A system that tunes parameters until a picture looks acceptable |
| A self-report that leads with failure | Confident coverage of work that was never done |
The subject is replaceable. The structure is not. Any recurring analysis with a frozen input, a published rubric and a control can be turned into this shape, which is the reason to write the brief carefully once rather than compare systems on whatever task is in front of us.
There is also a portability argument. A brief that produces good work on several model families is a brief that survives us changing models. A brief that only works on one is a brief coupled to a vendor. DeepSeek and Kimi run the same file next, unedited, and the comparison holds because the input is frozen and the rubric is public.
The run recommends one algorithm for the weekly report: the thresholded construction, producing twelve pairings with a confidence figure attached to each and a stated reason for every company left off. It recommends the broken sixth algorithm be dropped, and the null model be run every issue and never printed, because it is the check that keeps the printed result honest.
| Algorithm | Verdict | Reason |
|---|---|---|
| Thresholded geometric | Ship | Twelve pairings is a report section. Forty-two is not. |
| Soft geometric | Annotate | Its confidence figures belong on the shipped table, not in a section of their own. |
| Waxman null model | Run, never print | The check behind the printed claim. |
| Geographical threshold | Drop | On this board a high score and a distinctive position arrive together, so strength never overcomes difference. |
None of that ships yet. One system has run the brief, no reader has been shown the output, and the recommendation rests on the properties of the results rather than on anyone having read them. Two more harnesses run the same file before any of this reaches a report.