acceptodds
Under review as a conference paper at ICLR 2027

Echoes Are Not Votes: Provenance-Aware Contexts for Tool-Using Agents

Abstract

Tool-using language-model agents routinely encounter several artifacts that descend from one source: generated clients, copied documentation, cached outputs, and summaries. Current contexts present these artifacts as separate evidence, allowing one origin to acquire many apparent votes. We identify this failure as lineage amplification and isolate it with EchoBench, a controlled benchmark that adds claim-preserving derivatives while holding provenance roots fixed. Across 120 stratified tasks and four models, Raw accuracy falls from 100.0% with no derivatives to 55.2% with seven, although no root changes. A symmetric-replication control remains at 97.5%. When derivatives are rewritten with the same templates as roots, a surface-only heuristic falls from 100.0% to 50.0% and Raw agents reach 50.8%, exposing a sharp load-dependent preference for the amplified answer. We introduce duplication invariance as a paired measure of this behavior and present EchoLedger, a training-free reference context layer that records the derivation DAG, exposes one representative per root-set signature, and retains an expandable receipt. On the shared policy matrix, a calibrated metadata-free MinHash baseline reaches 97.5%, while full-context tags and compact EchoLedger reach 100%; compact EchoLedger uses 33.5% fewer characters. Across the four-model main study, EchoLedger reaches 99.8% versus 81.0% Raw (+18.75 points; 95% CI [12.29, 25.42]). On 36 overlapping-root tasks evaluated with Claude, Codex, and OpenCode runners, it reaches 100.0% versus 41.7% Raw (+58.33 points; [42.59, 73.15]) and recovers the correct unique-root-union decision in every task. On replay-validated public code with wording and path cues removed, Raw/EchoLedger accuracy is 1.3/100.0%. The analysis establishes derivation lineage as a distinct context relation and makes lineage amplification measurable and controllable.

Primary area:
foundation or frontier models, including LLMs
Keywords:
language agents, tool use, provenance, context management, evidence lineage, duplication invariance, agent memory

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_UHYuOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper explores lineage amplification in tool-using agents, where multiple artifacts derived from the same source may be incorrectly treated as independent evidence. The authors introduce EchoBench to evaluate this issue and propose EchoLedger, a training-free context management method that tracks artifact provenance and groups derived artifacts by their original sources. Experiments across multiple LLMs and controlled benchmarks show that EchoLedger improves answer accuracy and reduces context size compared with raw contexts and existing methods.

Critical strengths
  1. The paper proposes an interesting problem in tool-using agents, where multiple artifacts derived from the same source can mislead LLMs into treating duplicated information as independent evidence.

  2. The proposed EchoLedger is a simple and training-free method that improves answer accuracy across multiple models without requiring training the underlying LLMs.

  3. The paper provides many controlled experiments, including different duplication levels, multi-parent derivations, overlapping provenance roots, and missing metadata.

Critical weaknesses
  1. The paper lacks a comprehensive evaluation of potential utility loss introduced by the proposed method. The proposed method collapses multiple artifacts sharing the same provenance roots into a single representative. However, these artifacts may contain complementary or task-relevant information even when they share the same origin. Although the paper evaluates multi-parent synthesis, it remains unclear whether such compression affects task completion in realistic agent workflows, especially when important information is only available in collapsed artifacts.

  2. The practical benefits of EchoLedger remain unclear in real-world agent workflows. The experiments primarily rely on controlled settings with synthetic data where multiple artifacts are deliberately derived from the same source, but the paper does not establish how frequently such lineage amplification occurs in practice. While the paper reports latency improvements under high duplication, it remains unclear whether these benefits outweigh the additional provenance tracking overhead in realistic settings with limited or no duplicated artifacts.

  3. The proposed method heavily relies on the availability of accurate provenance information. EchoLedger assumes that the parent-child relationships between artifacts can be recorded or reconstructed. However, in realistic tool-using agent systems, information may come from external websites, third-party tools, or independently generated documents without reliable provenance metadata. It remains unclear how effectively the method can obtain and maintain accurate provenance information in real-world deployments.

  4. The evaluation setting may bring some bias to the proposed method. EchoBench defines the correct answer based on the number of independent sources, while EchoLedger is specifically designed to group information by these sources. This may make the benchmark easier for EchoLedger. Also, the tags-only baseline achieves the same accuracy as EchoLedger, so it is unclear whether the proposed method offers much benefit beyond simply providing source information to the model.

Questions for authors

Please refer to the weaknesses.

Reviewer_XxrrOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper identifies a failure mode in tool-using LLM agents it names "lineage amplification": when several surface-distinct artifacts (generated clients, cached outputs, summaries, documentation copies) descend from one provenance root, standard context assembly serializes them as a list and the agent counts them as independent votes. Section 2 formalises evidence as a derivation DAG G=(V,E) where root sets R(v) partition artifacts by independent origin, defines a root-balanced count S_root (adding another descendant of an existing root cannot change S_root), and introduces duplication invariance DI as a paired metric: f(T_k(x)) = f(T_0(x)) requires that replication alone not move the decision. ECHOBENCH generates evidence DAGs whose latent answer is the strict plurality of provenance roots and varies only the number of descendants per root (k in {0,1,2,4,7}), their wording (exact, paraphrased, mixed), and graph shape (star, chain, mixed), across 12 task domains for 480 base tasks (120 primary subset). ECHOLEDGER is a training-free context layer that registers parent IDs at the tool boundary, propagates root sets topologically, and emits one representative per lineage with an expandable receipt. Across four models (Claude Haiku 4.5, Sonnet 5, GPT-5.6 Luna, Terra), Raw accuracy falls from 100.0% at k=0 to 55.2% at k=7 with no independent evidence change, while a symmetric-replication control (replicate both answers) stays at 97.5%. Root-worded hardening (derivatives rewritten with the same six templates as roots) drops a surface-only heuristic from 100% to chance and Raw accuracy to 50.8% (below chance at k>=2, systematic preference for the amplified answer). ECHOLEDGER reaches 99.8% versus 81.0% Raw across 480 model-task decisions (+18.75 points, 95% CI [12.29, 25.42]). On a 48-task multi-parent synthesis extension where candidates appear only after combining operand roots, Raw collapses to 4.2% at load 3 while ECHOLEDGER holds 100%. On a 36-task overlapping-root stress across three runners (Claude CLI, Codex, OpenCode with Sonnet 5), the gain is +58.33 points. A 39-commit replay-validated public-code suite from six fresh Python repositories shows Raw 1.3% vs ECHOLEDGER 100.0% after wording and path cues are removed. A calibrated 128-permutation MinHash metadata-free baseline reaches 97.5% on the shared policy matrix (strong but below exact lineage at 100%); missing-edge curves show graceful degradation (Luna 98.3% at 10% edge miss, 90% at 20%). The method reduces presented artifacts by 46.9% and characters by 24.7% while matching tags-only accuracy, and batch wall-clock drops 34-49%.

Critical strengths

1. The causal design is unusually disciplined and stress-tests the authors' own hypothesis across every obvious counter-explanation. Section 4 does not stop at the main Raw-vs-ECHOLEDGER contrast. It adds (a) a symmetric replication control that replicates both answer sides and shows 97.5%, which localises the effect to asymmetric amplification rather than context length, (b) a root-worded hardening that rewrites all derivatives using the same templates as roots, where a deterministic surface-only heuristic drops from 100% to 50% (chance) and Raw agents drop to 50.8% with systematic below-chance behaviour at k>=2 (16.7% at k=4 and k=7, which rules out random guessing and shows active selection of the amplified competitor), (c) a multi-parent synthesis extension where candidates appear only after combining operand roots so "just delete derived nodes" is a demonstrably wrong policy (roots-only reaches only 51.0%), (d) an overlapping-root stress where signature count overcounts and only the unique-root union is correct (+58.33 points gain, confirmed across three runners including OpenCode), (e) a missing-edge monotone degradation curve that makes the metadata-coverage requirement operational rather than binary, and (f) a public-code transfer suite with cue hardening that preserves the effect direction. The MinHash calibration is also done cleanly: the 0.43 threshold is selected on a disjoint 360-task development split with ties broken by precision, then frozen before test evaluation. For a measurement paper this is close to the gold standard of showing that the finding is not an artefact of any one experimental choice.

2. The paper is honest about the ceiling of its own benchmark and locates the real contribution precisely. Section 4.1 states explicitly that "a policy that faithfully exposes the graph can saturate this objective, as tags-only does", and the Related Work and Limitations repeat that ECHOBENCH measures graph use, not source reliability. This means the authors do not oversell the 99.8% number as beating tags-only on accuracy (both reach 100% on the shared matrix); the real difference between ECHOLEDGER and tags-only is 46.9% fewer presented artifacts, 24.7% fewer characters, and 34-49% lower batch wall-clock, which is a practical not an accuracy claim. Naming the failure mode (lineage amplification) and introducing paired duplication invariance (DI) as a complementary measurement to accuracy are the real conceptual contributions, and these transfer to any future system beyond ECHOLEDGER. The paper does the uncommon thing of pre-emptively narrowing its own scope so a reviewer cannot catch it overclaiming.

Critical weaknesses

1. The benchmark saturates by construction, which caps what the headline 99.8% vs 81.0% can establish about the real-world problem. The ground truth in ECHOBENCH is defined as the strict plurality of supplied provenance roots (Section 4.1), and the authors acknowledge "a policy that faithfully exposes the graph can saturate this objective". That means the paper has shown two things cleanly: Raw agents fail to recover a hidden derivation relation from unstructured text (expected), and models can execute a root-counting policy when the graph is handed to them (also expected, confirmed by tags-only also reaching 100%). What the paper has not shown is that root plurality is the correct target in realistic agent tool pipelines where sources differ in reliability, where multiple roots may themselves be derived from a common upstream source outside the recorded graph, or where some roots are known to be more authoritative. In such settings the correct decision is not "count roots" but "weight roots by credibility", and the paper's Limitations explicitly scope this out ("measures graph use, not source reliability"). This leaves the lineage DAG as a lower bound on what a correct provenance-aware context layer needs. For ICLR this scoping is defensible, but it limits the paper's claim to "if root counting is your goal, we solve it exactly", not "we solve the amplification problem in the wild".

2. The method's deployment hinges on parent-metadata availability that is often absent in real agent pipelines, and the paper's missing-edge experiment shows this dependency is not negligible. Table 1 operationalises four parent-metadata sources: tool/build trace (ideal), canonical ID (good), inferred from content hash (candidate edge), and unknown ancestry (apparent root). In practice, mixed-provenance tool outputs from web search snippets, cached user content, screenshots, and generated artifacts frequently lack any recorded parent edge. The missing-edge Appendix result shows Luna retains 98.3% at 10% miss but drops to 75.0% at 40%, and Haiku drops to 65.0% at 40%. The paper's engineering requirement is strong: tool adapters must be modified at the artifact boundary to record parent IDs. For organisations building agents from heterogeneous tool stacks this is nontrivial to retrofit, and the paper's MinHash metadata-free fallback (97.5%) is close enough to ECHOLEDGER (100%) that a reviewer could reasonably argue the practical delta is small when parent metadata is partially unavailable. The paper would benefit from a quantitative claim about what fraction of real agent-tool interactions today supply recoverable parent metadata, or an argument that the right way forward is to standardise this at the protocol level (akin to MCP's resource references).

Questions for authors
  1. Real-world amplification prevalence. You replay-validated 39 commits for the transfer suite but the "controlled decision layer" is still synthetically constructed on top of real path observations. Do you have any direct measurement of how often lineage amplification actually occurs in production agent pipelines (e.g., naturally-collected traces from Claude Code or Codex sessions)? Even one estimated frequency number would strengthen the case for the problem being practically consequential.

  2. Root plurality vs source reliability. Your Limitations section separates graph use from source reliability. Have you considered a reliability-weighted variant of S_root where each root carries a credibility weight, and if so does ECHOLEDGER's representational substrate (root-set signatures plus receipts) extend naturally to it, or would it need redesign? This is the natural next question for any production deployment.

  3. MinHash vs ECHOLEDGER at saturated endpoints. Calibrated MinHash reaches 97.5% without parent metadata, while ECHOLEDGER needs recorded edges to hit 100%. In deployments where parent metadata is partially available, is there a hybrid policy that uses recorded edges where present and falls back to MinHash clustering elsewhere, and if so what accuracy does it attain under realistic edge-miss profiles (say 20-40%)?

  4. Below-chance behaviour at k>=2 under hardening. Root-worded Raw accuracy falls to 16.7% at k=4 and k=7, which you note is systematic selection of the amplified competitor. Can you say anything about which internal computation produces this preference — e.g., is this a pure positional-frequency effect (more mentions = more weight), an attention-sink effect, or something subtler? This would move the paper from "we measured the effect" to "we partly explain it".

  5. Overlapping-root signature trap failure mode. The signature-count control reaches only 66.7% because competitor-overlapping syntheses produce three signatures. Have you observed any Raw agent output that explicitly reasons about "three signatures" vs "four roots" (i.e., gets the right primitive but applies the wrong aggregation)? If so, this would suggest that a lighter intervention (just include a union-count hint) might recover most of the gain.

Reviewer_jHqzOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates "lineage amplification" in tool-augmented LLM agents: the phenomenon where multiple artifacts derived from a single historical root (such as generated code, cached outputs, summaries, or copied documentation) are treated as independent corroborating evidence, causing a single origin to exert outsized influence on decision-making. The authors introduce EchoBench, a benchmark comprising 480 controlled evidence DAGs (with 120 frozen evaluation tasks) that hold the plurality of independent provenance roots fixed while adding claim-preserving derivative artifacts. Across four frontier models, raw context accuracy drops precipitously from 100.0% with zero derivatives to 55.2% with seven derivatives, and further drops to 16.7% at load seven under root-worded template controls. To address this, the authors propose EchoLedger, a training-free context layer that records derivation DAGs at the tool boundary, collapses single-root lineages into one representative per root-set signature, and provides an expandable receipt. In evaluations across four model families, multi-parent synthesis tasks, missing-edge regimes, and 39 replay-validated public repository commits, EchoLedger achieves 99.8% accuracy (vs. 81.0% Raw) while reducing character context by 24.7%.

Critical strengths

The paper isolates derivation independence from related concepts such as semantic diversity, deduplication, and source credibility. The causal experimental design is thorough: the symmetric-replication control rules out context-length confounds (maintaining 97.5% accuracy), the root-worded control eliminates template-based surface heuristics (dropping the heuristic from 100% to 50%), and the paired duplication invariance metric directly quantifies load-dependent stability.

EchoLedger requires no model training, fine-tuning, or label supervision, functioning as a plug-and-play middleware that collapses redundant lineages while preserving verifiable audit receipts. The empirical evaluation is extensive (4,836 model-task decisions), proving robust generalization across multi-parent syntheses, overlapping-root topologies across multiple runners (Claude, Codex, OpenCode), and natural code changes from six public repositories.

Critical weaknesses

EchoLedger assumes that execution environments and tool adapters faithfully log parent artifact identifiers. While reasonable in controlled build pipelines, open web environments and heterogeneous multi-agent systems frequently lack native derivation metadata. Under a 40% missing edge rate, model accuracy drops to 65.0%–75.0%, underscoring the system's sensitivity to incomplete derivation tracking.

Correctness in EchoBench is strictly operationalized as the plurality of independent provenance roots. In real-world environments, a single authoritative or updated origin often supersedes multiple stale or unverified independent roots; the paper’s framework currently treats all independent roots as equipotent votes.

Questions for authors

In scenarios where tool outputs cross uninstrumented third-party APIs (preventing native parent ID propagation), how effectively can content-similarity heuristics or neural provenance miners supply edges to EchoLedger without introducing false merges?

If a derived artifact performs a multi-step semantic synthesis that introduces novel deductive conclusions not explicitly present in any parent root, does collapsing it into a parent root representative risk discarding critical task-relevant information?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.