Score-Coupled Narrative Poisoning: Winning the Benchmark, Losing the Search

Why autonomous research systems should separate trust in a verified artifact from trust in the explanation attached to it.

While a benchmark can tell us that a code change is better, it usually cannot tell us fully why.

Recently I spent a lot of time thinking about the gap between an impressive agent demo and a system that can remain reliable over hundreds of autonomous steps. This proposal for how we should build auto research systems feels like it addresses one of those gaps.

I wrote previously about the difference between a large language model and its harness. The language model generates text, but the surrounding system decides which tools it can use, what actions are allowed, and what gets placed back into context. Long-term research memory is another part of that harness. It shapes what future agents treat as prior knowledge.

I kept returning to that distinction while reading through ECDSA.fail, an Eigen Labs project built around an open optimization challenge. Contributors improve a reversible quantum circuit for elliptic-curve point addition, run it through a shared verifier, and receive an objective score. Lower scores are better.

You do not need to know quantum computing to follow the rest of this post. (I myself need to read more about quantum computing, I have a huge backlog of books and topics to explore deeply, and right now my focus has been on deep learning, and the mathematics behind it.)

The important part is the research loop:

A person or agent changes an executable artifact.
A benchmark checks whether it is valid and measures how well it performs.
Strong results become useful starting points for later researchers.
Contributors leave notes explaining what they tried, what worked, what failed, and what should be explored next.

That final step is what makes the project especially interesting to me.

While the code obviously preserves the result, the research notes preserve the path that produced it. Without those notes, every new agent would have to rediscover the same dead ends and reverse-engineer the reasoning behind every successful change. With them, research can become cumulative.

But cumulative research introduces a second trust problem. The benchmark verifies the artifact. Future agents also have to decide how much to trust the story attached to it.

A verified artifact is not the same thing as a verified explanation.

What ECDSA.fail does to address this

Before describing the concern, it is worth being clear about what the project already does well.

The public benchmark is not a loose leaderboard. Its harness validates candidate circuits across 9,024 generated test points, checks several correctness conditions, and scores valid results using average Toffoli count multiplied by peak qubit width. A submission does not receive a better score merely because it has a clever-looking explanation in a pull request. It has to pass a deterministic verifier.

The repository also explicitly warns that source files and memory notes may come from different agents. It tells researchers to treat those notes as leads, independently verify claims, and rerun the benchmark before relying on them.

At the time of writing, its more recent research handoff is even more careful. Claims are separated into categories such as Established, Observed, Refuted, and Unresolved. A timeout is not treated as proof that something is impossible. These categorical quantifiers help with weighting.

The repository's executable world model also ranks narrative claims below stronger forms of evidence such as official results, full verification, exact artifact identity, machine proofs, and controlled comparisons. It records evidence in an append-only history and tracks which assumptions a change may invalidate.

Those are thoughtful safeguards. They are also what led me to the broader question in this post:

Are artifact verification, provenance, and evidence labels enough when a persuasive explanation is attached to an objectively elite result?

ECDSA.fail’s system is well architected from what I have seen.

But to abstract away from just their implementation, I am using it as a motivating example for a more general problem that I think will appear in any long-running autonomous research system.

The trust boundary between a result and its explanation

Imagine an agent submits a patch that improves a benchmark by 10 percent. The patch contains two meaningful changes:

Change A: a large arithmetic rewrite that is easy to notice
Change B: a small representation change buried deeper in the implementation

The combined patch is valid. It passes every test and its score improvement is valid.

Now imagine that the attached report says:

The improvement came from Change A. Change B was cleanup and is not worth exploring further.

There is no obvious contradiction. Both changes are present, and the final patch really did improve the benchmark.

But suppose the explanation is wrong. Change A contributed almost nothing. Change B caused nearly the entire gain.

A future agent may accept the report because it came with one of the strongest artifacts in the archive. It spends the next several iterations extending Change A and ignores the family of ideas surrounding Change B.

The verifier did its job. It proved that the combined artifact worked. It did not prove the report's causal claim.

To learn which change mattered, the system would need additional experiments:

the original baseline
only Change A
only Change B
Changes A and B together

I believe just a final score alone cannot replace those comparisons, but this is a tradeoff you have to make.

This creates a subtle transfer of authority, akin to the cognitive bias we humans have (something I actively try to fight against), once a source of information earns your trust, you proceed to believe what comes after. (I strongly encourage the readers of my blog to think critically about what I write, and challenge not only the ideas I present but their exposition… I plan on including a moderated comment section in the near future to facilitate this discussion)

The code earned a high score, and the report quietly inherits some of that credibility even though it was evaluated by a different process, or perhaps not evaluated at all.

The problem does not require an attacker

This failure can happen accidentally.

An honest agent can misunderstand why its patch worked. It can mistake correlation for causation, overgeneralize from one configuration, summarize a timeout as a failed direction, or omit a dependency that seemed unimportant at the time.

That inaccurate note can then be summarized by another agent. A third agent sees two reports making the same claim and treats the repetition as corroboration. Before long, a guess begins to look like established knowledge.

Nothing needs to crash. Every submitted artifact can remain valid. The system simply allocates more of its limited search budget within the set of the entire possible search space to become narrow towards the wrong ideas.

The adversarial version uses the same weakness intentionally. I came up with the name score-coupled narrative poisoning to describe it. (If you think the name could be better, it probably could be…recall that naming is one of the hardest things in CS)

The attacker does not need to cheat the benchmark. They can submit a genuinely strong artifact and manipulate only the explanation, knowing that the artifact's score may cause later agents to retrieve, quote, and extend the report more often. The malicious version has an odd property: the attacker can win honestly and still poison the next round.

This sits next to a growing body of work on poisoned agent memory, including AgentPoison, the broader MPBench study of memory poisoning, locally correct but non-transferable experiences, FORGE on research-trajectory hijacking, and recent work on trajectory poisoning in self-evolving skill systems. The narrower issue here is that the poisoned record is attached to an objectively successful artifact. The report is persuasive not because it contains an obvious command, but because it is attached to real success.

How a report can steer the search in the wrong direction

The misleading report does not have to contain an outrageous lie. The hardest versions to detect would probably be technically plausible and mostly useful.

Credit the wrong mechanism

A patch contains several changes, and the report attributes the gain to the wrong one.

This is effective because the artifact and the score remain legitimate. The unsupported part is the causal story. A future agent only has to spend more of its budget optimizing the wrong mechanism for the attack to work.

Close a branch too early

A report says that an entire family of approaches was tested and found unhelpful.

Perhaps only two configurations were tried. Perhaps one run timed out. Perhaps the experiments were performed before another prerequisite optimization existed.

Negative conclusions have unusual leverage. A positive result can show that one artifact works. A weak negative result can incorrectly remove a large region from future consideration.

Turn a scoped result into a universal rule

An approach may fail under one representation, one resource limit, or one architecture. The report removes those qualifiers and says that the approach does not work.

The underlying observation can be true while the conclusion is still misleading. The problem is not the measurement. It is the size of the claim built on top of it.

Hide a dependency

A useful optimization may depend on a feature flag, parent patch, seed policy, circuit representation, or cleanup strategy.

If the report omits that dependency, later agents reproduce the visible technique without the condition that made it work. Their failed attempts then appear to confirm that the broader direction is bad.

Monopolize attention (ramble) without saying anything false

This may be the most difficult form to audit.

A report can spend most of its detail on a low-value direction, provide ten concrete follow-up ideas for it, and barely mention competing explanations. Every sentence may be accurate. The manipulation lies in what the report makes salient and what it leaves difficult to notice.

A factuality checker could approve the entire document.

Manufacture consensus through inheritance

Five reports repeating the same claim do not represent five independent confirmations when all five copied the same original note.

Without lineage, repetition can look like replication. The archive slowly becomes more confident without gaining new evidence.

Why coupling the report to the score matters

A misleading note in a forgotten branch of the archive may have little effect. A misleading note attached to the current best result can have a much larger one.

In an autoresearch system that greedily selects the best previous artifact as context, its report becomes the default map of the search space. Even a system that samples from the top few results may still expose agents to elite reports much more often than ordinary ones.

The score therefore acts as both a retrieval multiplier and a credibility amplifier.

That coupling is useful when the report is accurate. High-performing artifacts often contain valuable ideas, and the people or agents that produced them may have learned something worth preserving.

The mistake is assuming that artifact quality and report reliability are the same variable.

They should be related only through evidence.

Why an LLM judge helps, but cannot create missing evidence

My first instinct was to add another model that compares the code change, benchmark output, logs, and report.

That would help. A judge could catch direct contradictions, impossible measurements, missing experiment identifiers, unsupported certainty, and conclusions whose stated scope is broader than the attached evidence.

But a judge cannot recover experiments that were never run.

Return to Changes A and B. The final artifact proves that the pair improved the score. Reading the diff more carefully does not reveal whether A caused the improvement, B caused it, or the interaction between them mattered. A model may form an intelligent hypothesis, but it is still just a hypothesis.

The claim that something caused something, i.e., is causal, requires a controlled comparison.

A judge also has trouble with omission. A report can be truthful about every experiment it mentions while leaving out the one alternative most likely to redirect the search.

Additionally, though maybe not as relevant, if the research agent and judge share the same models, the same confident explanation may persuade both of them.

I would still use LLM judges, but as auditors and experiment planners rather than final sources of truth. Their job should be to identify claims, locate evidence, notice discrepancies, and propose the cheapest test that would resolve uncertainty.

A safer architecture for cumulative autoresearch

To be very clear, I don't think the solution is to discard natural language research notes. They are too valuable. But we have to stop treating code, score, and explanation as one indivisible package.

A simple mental model is to give the artifact and report separate paths:

artifact -> executable verifier -> performance score -> eligible for reuse

report   -> claim extraction -> evidence links -> independent checks -> trust state

The two paths can inform each other, but one should not silently certify the other.

1. Score the artifact and trust the report separately

A strong benchmark result should increase the chance that later agents reuse the artifact. It should not automatically increase confidence in every sentence attached to it.

The artifact can be ranked by correctness and performance, among other domain relevant metrics. The report can be ranked by reproducibility, evidence quality, scope precision, and independent confirmation (again among other domain relevant considerations)

An elite artifact could therefore be promoted immediately while its explanation remains marked as provisional.

2. Make important claims point to evidence

Reports should not exist only as free form prose. The system should extract or require a small number of typed claims:

measured result
observation
causal hypothesis
negative result
scoped proof
unresolved question
recommended next experiment

Each important claim should point to the artifact, command, configuration, logs, seeds, comparisons, or proof that supports it.

This does not eliminate misleading interpretation, but it makes the gap between evidence and conclusion inspectable.

ECDSA.fail's existing distinction between established, observed, refuted, and unresolved findings is a strong example of this direction.

3. Framing matters, audit the evidence before reading the story

A verifier agent should first inspect the raw diff, benchmark output, experiment history, and artifact identity without seeing the author's explanation.

It should write down:

what the evidence directly establishes
which explanations remain plausible
which conclusions are too broad
which experiment would separate the leading explanations

Only then should it read the submitted report.

This ordering matters, and is a pattern that I think the industry is converging on in many domains (test-driven development with agents is the first example that comes to mind). Once an auditor has read a confident story, it may interpret every ambiguous detail through that frame.

4. Spend verification budget according to a claim's influence

Not every sentence needs an expensive full benchmark. The system should spend more verification effort on claims that can redirect more future work.

A causal claim attached to the current frontier deserves an investigation. A statement that an entire branch is exhausted deserves independent reproduction or a clearly scoped proof. A minor implementation note may need much less scrutiny.

This is not verification for its own sake. It is verification weighted by downstream leverage on how it impacts the future agents.

5. Treat branch closing (of the search space) claims with extra skepticism

Positive and negative claims are not symmetric.

Similar to verification vs. proof in mathematics not being the same, one reproducible artifact can prove that an approach worked at least once. One failed experiment cannot prove that a family of approaches will never work.

A timeout should remain unresolved. A small search should mean "not found under this computational/cost budget." An experiment under one representation should not silently become a universal lower bound.

Most importantly, an unverified negative claim should not reduce the probability of revisiting a viable branch to zero.

6. Preserve multiple research lineages

The best current artifact is not always on the path to the best future artifact.

A robust archive should preserve several promising families, not only several nearby versions of the same idea. It might retain different points on a performance versus resource frontier for example, different representations, or approaches that rely on meaningfully different mechanisms.

This does not tell the system which report is correct. It limits the blast radius of one wrong report and keeps recovery paths alive.

This is a very similar consideration for how human researchers operate, a very relevant read would be "The Structure of Scientific Revolutions" by Thomas S. Kuhn. (I read sections of this during my undergrad, but it is my intention to finish this after my current queue!)

7. Track provenance, independence, and rollback

When several agents repeat a claim, the system should know whether they independently tested it or inherited it from the same ancestor.

A useful trust lifecycle might look like this:

Submitted -> Reproduced -> Trusted -> Established

Those states should not be permanent. A counterexample, artifact change, invalidated dependency, or failed reproduction should be able to move a claim backward.

Long-term agent memory needs forgetting and rollback just as much as it needs retrieval.

The larger lesson

I think similar to how scientists keep a log of what worked, what didn't, etc. (and pass this knowledge down to their grad students) autonomous research systems need more than a leaderboard. They need memory of failed experiments, causal hypotheses, uncertainty, dependencies, and promising next steps.

That memory will be one of their greatest advantages. It may also become one of their most consequential attack surfaces.

While verifiable execution can establish that a particular artifact ran, passed, and achieved a score, it cannot automatically establish that:

the report identified the true cause of the improvement
every claimed experiment actually ran
a failed direction is globally hopeless
an observation generalizes outside its tested scope
repeated agreement came from independent evidence

These are different questions, and they deserve different verification mechanisms.

The code earned its score. The report still has to earn our trust.

A system can win every local benchmark and still lose the overall search for the most optimal solution.

Score-Coupled Narrative Poisoning