acceptodds
Under review as a conference paper at ICLR 2027

EvidenceCard: Plan Evidence for Token-Efficient Coding Agents

Abstract

Tool-using coding agents can consume substantial input tokens while searching repositories, reading source files, and executing tests across long interaction trajectories. Existing token compression methods primarily reduce the size of acquired context or optimize individual context decisions. They do not explicitly track whether the observations collected across a trajectory have satisfied the requirements of the current coding plan. We formulate this limitation as a plan-level evidence management problem and introduce EvidenceCard, a lightweight framework that controls context acquisition through plan-conditioned evidence accounting. EvidenceCard exposes acquisition history and heuristic feedback, prompting the coding agent to reassess whether further context is needed. Its prompt architecture combines a stable, cache-friendly evidence-policy prefix with a small dynamic runtime tail. These signals guide the original agent to acquire, reuse, compress, or stop collecting evidence without additional training and auxiliary LLM calls. In contrast to retrospective compression, EvidenceCard acts prospectively by making evidence sufficiency visible before the next tool action. Empirical results show that EvidenceCard substantially reduces file-read tokens and requests, revealing reductions of up to an order-of-magnitude in read tokens relative to baselines, while maintaining robust task-solving performance.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM Agents, Coding Agents, Token Efficiency, Tool-augmented Language Models

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_cHwGOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes EvidenceCard, a lightweight method for reducing coding-agent context costs. It combines a stable prompt policy with a rule-based ledger and a compact runtime card to discourage unnecessary reads and exploration. Experiments across several models and benchmarks show substantial reductions in input tokens, tool-observation tokens, and API requests, with comparable single-run task success rates.

Critical strengths
  1. The problem is practically important, and the method requires neither additional training nor auxiliary LLM calls.

  2. The resource savings are substantial across multiple models and benchmarks, supported by ablations and per-task analyses.

Critical weaknesses
  1. Novelty and positioning. Prior work has explored plan-aware context engineering and subgoal-driven working memory management, including PAACE and HiAgent. Although EvidenceCard differs in its lightweight, prompt-based design for reducing future information acquisition, the paper does not sufficiently establish why its evidence-specific formulation is necessary beyond existing context-management strategies. A clearer comparison of the underlying mechanisms, ideally supported by matched prompting baselines, would better establish the methodological novelty.

  2. Insufficient controls to isolate the gains. The ablations do not compare EvidenceCard with a generic efficiency-oriented prompt paired with the same runtime statistics. It therefore remains unclear whether the evidence-specific formulation is necessary for the reported improvements.

  3. Heuristic signals are not validated. The scoring and repeat-detection rules are simple proxies for evidence quality and redundancy. Although their limitations are described, the paper does not validate whether these signals reliably reflect semantic relevance or evidence sufficiency.

  4. Limited evidence on task-success robustness. Each method–task pair is evaluated once. The authors acknowledge this and avoid statistical claims about accuracy improvements, but repeated runs would still provide stronger evidence that the resource savings do not compromise solution quality.

Questions for authors

Refer to the weaknesses.

5 comments

Response to Concern 1: Novelty and Positioning

Thank you for highlighting the relationship to PAACE and HiAgent. We agree that plan-aware context engineering and subgoal-based working memory are established research directions. EvidenceCard's contribution concerns its specific acquisition-feedback mechanism and its empirical value in coding-agent trajectories.

Mechanism-level comparison. PAACE compresses the current context conditioned on upcoming plan steps. HiAgent organizes working memory around explicit subgoals, replacing completed subgoal trajectories with summaries and retrieving their details when needed. EvidenceCard uses deterministic bookkeeping to expose acquisition history and heuristic cues to the original coding agent before its next action. Table 1 summarizes these differences.

Table 1. Mechanism-level comparison with related context-management approaches.

DimensionPAACEHiAgentEvidenceCard
Main mechanismPlan-conditioned context compressionSubgoal-based working memory managementAcquisition-history feedback through a deterministic ledger
State used by the mechanismCurrent context and upcoming plan stepsCurrent subgoal, previous subgoal summaries, and detailed trajectoriesAcquisition targets, observation types, phases, token costs, scores, and repeated-target cues
Main operationCompress and reshape the context supplied to the agentSummarize completed subgoal trajectories and retrieve details on demandPresent recent acquisition records and aggregate statistics before the next action
Role of planningUpcoming plan steps condition compressionExplicit subgoals organize actions and memoryThe original agent interprets ledger feedback relative to its current plan
Additional model processingContext-compressor inferenceObservation summarizationNo additional model inference for ledger construction or rendering

Our distinction concerns the primary intervention. EvidenceCard provides explicit acquisition-history feedback without introducing an auxiliary compressor or summarizer. Plan interpretation and evidence-sufficiency judgments remain with the original agent; the ledger does not maintain an explicit requirement-to-observation mapping or automatically certify requirement closure.

Response to Concern 2: Insufficient Controls

We agree that removing individual components does not adequately distinguish evidence-specific guidance from a generic instruction to be efficient. We therefore added a generic efficiency prompt both without feedback and with the same full EvidenceCard ledger.

The generic efficiency system prompt is:

# Runtime Efficiency
- Complete the coding task correctly while minimizing unnecessary
  tool calls and context tokens.
- Reuse information already available instead of repeating work.
- Keep capacity for implementation and verification.
- Prefer targeted inspection; expand only when needed.
- Use the runtime feedback to decide whether another action
  is worth its cost.

This control includes substantive efficiency guidance: reusing available information, preferring targeted inspection, and preserving capacity for implementation and verification.

MethodStatic policyDynamic feedbackInput tokens ↓Tool tokens ↓Requests ↓Success rate (%) ↑
Base agentOriginal base policyNone89.70M54.63M5.2K80.6
Generic efficiency promptGeneral efficiency guidanceNone46.50M26.72M4.2K80.6
Evidence policy onlyEvidenceCard policyNone51.40M25.93M4.4K80.6
Generic efficiency prompt + same ledgerGeneral efficiency guidanceFull EvidenceCard ledger25.43M11.38M2.7K80.6
Full EvidenceCardEvidenceCard policyFull EvidenceCard ledger12.45M3.44M1.7K81.7

The results show that: First, generic efficiency prompting already substantially reduces resource consumption relative to the base agent. Without dynamic feedback, the evidence policy uses more input tokens and requests than the generic prompt, although slightly fewer tool tokens, with the same observed success rate. These results negate that evidence-specific wording alone explains the gains.

Second, ledger feedback benefits both policies. This supports the value of dynamic acquisition feedback beyond static instructions in the evaluated setting.

Third, the decisive matched comparison is between Generic efficiency prompt + same ledger and Full EvidenceCard. With the same ledger mechanism, full EvidenceCard uses approximately 51.0% fewer input tokens, 69.8% fewer tool tokens, and 37.0% fewer requests. Its observed success rate is 81.7%, compared with 80.6% for the matched control.

Response to Concern 3: Interpretation and Role of the Heuristic Signals

Thank you for raising this concern. We would like to clarify the intended role of these signals and distinguish the ledger's bookkeeping from the original agent's semantic judgments.

EvidenceCard records previous tool interactions and presents acquisition-history statistics together with heuristic observation cues. The score is derived from deterministic content rules, and the repeated-target flag records whether an extracted target has appeared previously. These signals are advisory: they do not block tool execution, automatically skip observations, or determine when information acquisition must stop.

The original coding agent interprets this feedback together with the task, its current plan, and the actual observations. It remains responsible for judging semantic relevance, identifying outstanding information needs, and deciding whether the available evidence is sufficient to patch, verify, or finish. The runtime does not maintain an explicit requirement-closure state or automatically certify evidence sufficiency.

This distinction is already stated in the method: the ledger “does not itself establish the semantic relevance or sufficiency of an observation,” and its decision labels “are feedback for the next turn and do not block tool execution.” Nevertheless, we recognize that phrases such as “making evidence sufficiency visible” and “irrelevant-read aware” can suggest a stronger capability. A more precise description of our intended claim is:

EvidenceCard exposes acquisition history and heuristic observation cues to the original coding agent, which uses them when assessing whether further information is needed for its current plan.

In particular, repeated targets should not be equated with semantic redundancy. Reading a modified file or rerunning a test after a code change may provide new information despite revisiting the same target. Similarly, a low heuristic score does not establish that an observation is irrelevant. The feedback makes these acquisition patterns available for the agent to interpret rather than resolving their semantic meaning automatically.

To assess the operational contribution of the cues, we additionally evaluated their joint removal alongside the individual ablations.

Table 3. Operational contribution of the heuristic feedback signals on GLM-5 / SWE-Bench-93.

VariantInput tokens ↓Tool tokens ↓Requests ↓Success rate (%) ↑
Full EvidenceCard12.45M3.44M1.7K81.7
Without weak-observation signals15.16M4.50M1.9K81.7
Without repeat/reuse signals16.44M4.83M2.0K81.7
Without both signal groups17.84M4.92M2.1K81.7

These results support the operational usefulness of the heuristic feedback in the evaluated setting. They are not presented as ground-truth validation of semantic relevance or evidence sufficiency. The claim supported here is that inexpensive acquisition-history cues can help the original agent manage its information-gathering behavior, while semantic interpretation and sufficiency judgments remain with that agent.

Response to Concern 4: Repeated-Run Task-Success Robustness

Thank you for this suggestion. We agree that the original single-run evaluation does not characterize run-to-run variability. The existing task bootstrap measures sensitivity to task-set composition and does not substitute for repeated agent episodes.

To provide additional evidence, we evaluated the base agent and full EvidenceCard over three runs on the same 93 tasks in GLM-5 / SWE-Bench-93. Table 4. Repeated-run evaluation on GLM-5 / SWE-Bench-93.

MethodRunInput tokens ↓Tool tokens ↓Requests ↓Success rate (%) ↑
Base agent189.70M54.63M5.2K80.6
Base agent285.32M50.19M5.1K80.6
Base agent393.84M59.31M5.3K80.6
Full EvidenceCard112.45M3.44M1.7K81.7
Full EvidenceCard211.96M3.41M1.7K80.6
Full EvidenceCard313.99M3.45M1.7K81.7

Observed task success remains comparable. The base agent achieves 80.6% success in all three runs, while EvidenceCard achieves 81.7%, 80.6%, and 81.7%. Thus, none of the additional runs shows a lower aggregate success rate for EvidenceCard, and the substantial resource reductions persist alongside comparable observed success.

These repeated-run results strengthen the evidence that the efficiency gains are reproducible in this setting. However, three runs on 93 tasks do not establish statistical equivalence or rule out small changes in success probability. We therefore interpret the results as substantial resource savings with comparable observed task success, rather than as proof of an accuracy improvement or a guarantee of no degradation.

Dear Reviewer,

As the discussion period is drawing to a close, we would like to briefly follow up on our rebuttal. Your feedback has been very valuable to us, and we are eager to address any remaining concerns and further improve the quality of our work. Please let us know if anything remains unresolved; we would be happy to provide further clarification. If our responses have addressed your concerns, we would be grateful if you could reconsider your assessment in light of them.

Thank you for your time and effort in reviewing our paper. Best regards, Authors

Reviewer_AbnZOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes EvidenceCard, a training-free method that reduces token consumption in coding agents by adding a fixed system prompt and appending a rule-generated log of recent tool calls (phase, evidence kind, target, token cost, repeat flags) before each model request. Across SWE-Bench, SWE-QA, and Terminal-Bench 2.0 with four LLMs, it reports large reductions in requests and input tokens while maintaining task accuracy.

Critical strengths

The paper is well motivated: it identifies a real gap in existing token-efficiency methods, which compress or gate context locally but don't control how much evidence an agent acquires over its whole trajectory, and illustrates it clearly with concrete trajectory comparisons. The evaluation is thorough, covering four LLMs, three benchmarks, and four representative baselines. The paper is also easy to follow.

Critical weaknesses
  1. The paper presents "plan-level evidence management" as a general problem formulation, but it focuses exclusively on coding and terminal tasks, where shell commands are semantically legible. A key component of the method is a set of rules that map command names (cat, sed -n, rg, Get-Content, pytest, ...) and output keywords to evidence kinds and phases (localize/inspect/patch/verify). This works largely because shell commands carry rich semantics. It is unclear how the approach would carry over to agents using structured tools (read_file(path, range), str_replace, MCP tools), or to domains such as writing or research, where a tool call reveals little about task progress. This limits the generality of the proposed approach.
  2. The rule-based parser appears fragile. Today's agents frequently chain several operations in one tool call. Under the stated rules (first match wins, in Table 6 order), git diff && pytest would be recorded as a patch-phase edit rather than as test evidence or a verify-phase step. ls src/ && cat src/models.py would be recorded as a repo map (or as an error trace, if the file text contains error), hiding a full-file read. And cat a.py; head -20 b.py would likely count as a bounded read. Target extraction keeps a single path, so rereads of other files in the chain escape repeat detection. Even unchained reads can be mislabeled: the "error indicator in other output" rule precedes the read rules, so a plain cat of any source file mentioning error or exception would be labeled an error trace in the localize phase. Finally, the evaluation's Repeated metric also relies on normalized command signatures, so chained commands could distort the reported numbers for all methods, not just the Card's labels.
Questions for authors
  1. Which part of the Card drives the gains? The "w/o dynamic ledger tail" ablation removes the header counters and the per-call entries together. The existing ablations remove individual header signals (repeat/reuse, W_t), but none isolates the per-call entries. Could you report a K=0 variant that shows only the header counters?
  2. Does the rule-based evidence taxonomy itself matter? Can you report a variant where phase and evidence-kind labels are randomized or shuffled while everything else stays fixed? If efficiency is largely unchanged, the gains would come from the prompt and a generic activity log rather than from evidence accounting.
  3. What fraction of tool calls in your trajectories are compound (chained with &&, ;, or pipes)? On a manually labeled sample, how accurate is the parser's assignment of phase, evidence kind, and target?
3 comments

Response to Weakness 1: Scope and Adaptation to Structured Tools

Thank you for raising this concern. We agree that the current empirical evidence is limited to coding and terminal agents. Extending the approach to writing, research, and other domains requires further development and evaluation.

We would like to clarify that the acquisition-history feedback mechanism does not inherently depend on shell-command strings. We have integrated EvidenceCard into our own agent framework, which exposes structured file tools such as readFile(file_path, offset, limit). In this integration, explicit adapters extract the acquisition target and read range directly from structured arguments. File enumeration and search tools are handled through analogous adapters.

Our experience also revealed a limitation of applying the same feedback indiscriminately to all tools. Our integration recorded a broader range of operations, including GUI interaction tools. We observed cases in which repeat-related feedback discouraged necessary repeated clicks and interfered with task completion. Such interactions may be required to advance a changing interface state; repeated actions do not necessarily represent redundant information acquisition. This influence arose through the agent's interpretation of advisory feedback, rather than through a hard restriction on tool execution.

We consequently narrowed this deployment adapter to explicitly supported acquisition tools, principally file reading and search. GUI actions and unsupported tools remain available to the agent but do not enter evidencecard ledger. The structured-tool adaptation is summarized in Table 1.

Table 1. Adaptation of structured acquisition tools.

Tool categoryInformation used by the adapterLedger treatment
readFileFile path, offset, and limitRecord the file and read range
extractTextFile pathRecord text extraction
globFilesSearch root and pattern argumentsRecord file enumeration
grepFiles / semanticSearchSearch scope and query argumentsRecord search acquisition

This adaptation does not require assigning progress semantics to every tool in the framework. Instead, it explicitly defines which operations belong to the information-acquisition process being managed. Structured arguments make the accessed resource and range explicit, while semantic interpretation and evidence-sufficiency judgments remain with the original agent.

Table 2 reports coding-task results for the shell and structured-tool configurations. Resource reductions are calculated against the corresponding configuration without EvidenceCard.

Table 2. EvidenceCard under shell and structured acquisition interfaces.

Acquisition interfaceConfigurationInput tokens ↓Tool tokens ↓Requests ↓Success rate (%) ↑
Shell toolsWithout EvidenceCard89.70M54.63M5.2K80.6
Shell toolsWith EvidenceCard12.45M3.44M1.7K81.7
Structured file and search toolsWithout EvidenceCard115.23M68.80M3.4K81.7
Structured file and search toolsWith EvidenceCard36.50M18.29M1.6K81.7

This deployment demonstrates a concrete adaptation to structured coding tools and identifies a failure mode of extending acquisition-reuse feedback to state-changing actions. It does not establish generalization to arbitrary MCP tools or non-coding domains. General MCP tools require explicit interpretation of their schemas and state semantics, while writing and research require further adapter development and independent evaluation. Our empirical claims remain scoped to the evaluated coding and terminal settings.

The underlying agent framework is already open source. To preserve anonymity, we omit identifying links during review. Upon acceptance, we will provide the EvidenceCard reference implementation, structured-tool adapters, and the corresponding reproduction details.

Response to Q1: Contribution of Per-Call Ledger Entries

We added the requested K=0. This variant retains the header counters and normal background ledger updates but displays no per-call entries. The stable policy and all other experimental settings remain unchanged.

VariantInput tokens ↓Tool tokens ↓Requests ↓Success rate (%) ↑
EvidenceCard, K=065.83M42.91M4.8K80.6
EvidenceCard, K=1013.69M3.77M1.9K81.7
EvidenceCard, K=2012.45M3.44M1.7K81.7

Compared with K=0, the full configuration (K=20) reduces input tokens by 81.1% and tool-observation tokens by 92.0%, while achieving an observed success rate of 81.7% versus 80.6%. K=10 also recovers most of these efficiency gains.These results support the incremental value of recent per-call entries beyond aggregate header feedback.

Response to Q2: Contribution of Phase and Evidence-Kind Labels

Thank you for suggesting this control. We added a variant with shuffled phase and evidence-kind labels and compared it with both the base agent and full EvidenceCard.

VariantInput tokens ↓Tool tokens ↓Requests ↓Success rate (%) ↑
Base agent89.70M54.63M5.2K80.6
EvidenceCard with shuffled phase and evidence-kind labels15.04M4.42M1.9K81.7
Full EvidenceCard12.45M3.44M1.7K81.7

The shuffled-label variant retains substantial efficiency gains. We therefore agree that the overall gains should not be attributed solely to the correctness of these labels.

Nevertheless, full EvidenceCard uses 17.2% fewer input tokens and 22.2% fewer tool-observation tokens than the shuffled-label variant, while both achieve an observed success rate of 81.7%. The original label assignments therefore provide an additional efficiency benefit in this comparison.

Response to Weakness 1 and Q3 — Rule precedence and parser limitations.

Thank you for requesting this audit. We examined the supplied GLM-5 / SWE-Bench-93 EvidenceCard run, containing 1,695 attempted tool calls across 93 tasks. We found that 1,129 calls (66.61%) contained at least one shell-level &&, ;, or pipe operator. We agree that assigning a single label to a compound call can leave some operations unrepresented. The key point, however, is that substantial efficiency gains remain possible when the parser correctly captures the file-reading operations responsible for much of the context cost, even if other labels are imperfect.

Our original description did not correctly specify the rule precedence. The nine rows of Table 6 have priority values [3, 7, 8, 5, 4, 6, 1, 2, 9], with lower values evaluated first. Thus, bounded file reads have priority 1 and unbounded file reads have priority 2. We will make these priorities explicit in the table and correct the accompanying description.

File-reading patterns receive early priority because source-code inspection is a major source of context consumption in the coding tasks we study. This already creates substantial opportunities for token savings without requiring every operation in a compound call to receive a perfectly accurate fine-grained label.

This precedence also addresses: ls src/ && cat src/models.py is recognized as a full-file read. Other aspects of compound calls, including additional operations and targets, may still be missed; our emphasis is on retaining the high-cost file-acquisition signal that is most directly relevant to context efficiency.

We also manually annotated a random sample of 100 ledger-recorded calls:

SampleCallsPhase agreementEvidence-kind agreementTarget agreementActual file-reading callsRecognized as reads
All calls10057/100 (57.00%)51/100 (51.00%)52/78 (66.67%)2929
Compound6228/62 (45.16%)22/62 (35.48%)18/40 (45.00%)44
Non-compound3829/38 (76.32%)29/38 (76.32%)34/38 (89.47%)2525

The important distinction in these results is that phase and evidence-kind agreement are imperfect, yet all 29 manually identified file-reading calls were recognized as reads

We additionally evaluated EvidenceCard without taxonomy, weak-observation, and repeat/reuse feedback:

VariantInput tokens ↓Tool tokens ↓Requests ↓Success rate (%) ↑
Base agent without EvidenceCard89.70M54.63M5.2K80.6
EvidenceCard w/o taxonomy, weak-observation, and repeat/reuse feedback18.34M5.01M2.1K81.7
Full EvidenceCard12.45M3.44M1.7K81.7

The result also show that: EvidenceCard therefore does not require every phase and evidence-kind label to be correct to deliver substantial practical gains. In our coding tasks, correctly identifying token-intensive file acquisition already enables useful guidance on whether further reading is necessary. The audit shows that this file-reading signal is retained in the sampled calls, while the efficiency results demonstrate large cost reductions despite imperfect fine-grained classification.

Dear Reviewer,

As the discussion period is drawing to a close, we would like to briefly follow up on our rebuttal. Your feedback has been very valuable to us, and we are eager to address any remaining concerns and further improve the quality of our work. Please let us know if anything remains unresolved; we would be happy to provide further clarification. If our responses have addressed your concerns, we would be grateful if you could reconsider your assessment in light of them.

Thank you for your time and effort in reviewing our paper.

Best regards,

Authors

Reviewer_x9hdOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces EvidenceCard, a training-free framework for reducing coding-agent token consumption by discouraging unnecessary exploration, repeated reads, and excessive post-patch verification. The main contribution is to treat context acquisition as a trajectory-level evidence-management problem by prompting then agent to reconsider what information it still needs before taking another action. The method combines a stable instruction prompt, a rule-based ledger of tool observations, and a compact dynamic card presented before each model request. The ledger tracks inspected targets, token costs, repetition, task phases, and heuristic observation-quality signals. Before doing any actions, these signals are fed to the original coding agent to decide whether to acquire more context, reuse observations, or proceed with implementation and verification. The framework does not require additional training nor auxiliary LLM calls.

Being evaluated on swe-bench-93 and compared against unmodified gant and 4 context-management methods (OpenWolf, Repoformer, SWE-Pruner, and LightMem), EvidenceCard achieves the lowest request, input-token, tool-observation, and repeated-read costs across GLM-5, Qwen3.7-Plus, DeepSeek-V4-Pro, and GPT-5.2 while maintaining the highest or comparable task success rates. Ablation studies also confirmed the contribution of both the fixed prompt and dynamic ledger. Per-task comparisons on GLM-5 show that EvidenceCard reduces input-token usage on nearly every evaluated task, rather than achieving its aggregate savings through only a few cases.

Critical strengths
  1. the improvements on efficiency is distributed across 4 llms and over 6 metrics. The method reduces input tokens by 60.2% - 86.1% and requests by 29.7% - 66.5% compared against baseline while keeping comparable task success. Also shows significantly reductions compared to other context management methods.

  2. the method is light-weight (training free and no auxiliary llm calls). It combines fixed instructions with a compact, rule-based activity ledger, requiring neither model training nor auxiliary LLM calls. The ablation removing the dynamic card increases input consumption from 12.45M to 51.40M tokens, supporting its contribution beyond the fixed prompt alone.

Critical weaknesses
  1. The evaluation does not fully establish whether the detailed evidence ledger offers substantial benefits over a simpler dynamic efficiency feedback. The authors already test using fixed instructions alone and adding the dynamic card significantly reduces input tokens. Therefore, it has evidence that dynamic feedback matters. However, it is unclear if using a much simpler dynamic message could still achieve similar savings (such as "You have made 15 tool calls. Before reading more, check whether you can already make the patch or run a targeted test."). If a simple reminder works similarly, the contribution may mainly be persistent efficiency prompting. If the detailed ledger performs substantially better, that strengthens the case for the proposed mechanism. The paper does not directly distinguish these possibilities.

  2. The baseline adaptations are insufficiently documented to fully assess the strength of the reported comparisons. The authors integrate OpenWolf, Repoformer, SWE-Pruner, and LightMem into the coding agent used in the paper. These integrations require choices that can strongly affect cost. For example, a memory method that calls an LLM to summarize after every interaction can consume considerably more requests than one that summarizes less frequently. The paper reports substantial memory-maintenance overhead, but does not fully explain the configuration choices or adaptations. (the reported savings compare EvidenceCard with particular implementation details that are missing)

Questions for authors
  1. clarification on baseline integration: How were the baselines adapted? In particular, what triggers LightMem’s memory-extraction calls, and how is Repoformer’s selective retrieval integrated into the coding trajectory? Do these choices match the official implementations? This would help determine whether the reported overhead reflects the methods themselves or particular integration choices.

  2. Additional ablation: Table 4 shows that the dynamic card improves over fixed instructions alone. Could you compare it with a simpler dynamic efficiency reminder, such as the same guidance supplemented only by tool-call and token counts? This would help establish the additional value of the detailed evidence ledger.

4 comments

Response to Weakness 2 and Q1 — Baseline integration

Thank you. We agree that baseline adaptations to a tool-using agent need clearer documentation.

SWE-Pruner. We used its official Python inference and serving code to prune individual source-code observations, retaining lines relevant to the current step.

OpenWolf. OpenWolf tracks repository file state and warns against rereading inspected files. Because our agent uses shell commands, we added command matching for operations corresponding to its Read, Write, Edit, and MultiEdit interfaces, enabling file-state tracking and repeated-read feedback. The remaining logic follows its source implementation.

LightMem. We reused its official code-assistant extraction prompt, structured memory format, and merge semantics from UPDATE_PROMPT, retaining extraction, retrieval, injection, and updating.

The main adaptation is the extraction trigger. The original workflow normalizes and optionally precompresses user–assistant exchanges, accumulates approximately 512 user tokens in a sensory buffer, applies topic segmentation, then accumulates approximately 512 tokens from selected roles in a short-memory buffer before calling the extraction LLM. Our episodes contain one initial user instruction followed mainly by assistant/tool interactions, so the original user-token threshold would rarely activate memory within a task.

We therefore use within-task events: once memory is available, extraction is triggered after at least four new events accumulate. This enables competitive context costs. On Qwen3.7-Plus / SWE-Bench-93:

MethodInput tokens ↓Tool tokens ↓
LightMem11.39M3.30M
EvidenceCard7.96M2.46M

Extraction adds auxiliary request cost. We will report the trigger alongside results to distinguish this integration from the original buffering policy.

Repoformer. Its original selective retrieval uses left/right code context around a completion location. Our agent instead receives a task and evolving tool observations, often without a completion location. We adapt the query to the latest user message concatenated with the tail of the most recent tool result, giving the gate task and repository context to decide whether further cross-file retrieval is useful. This differs from the official completion setup.

We will add a baseline-integration appendix specifying official components reused, adapted interfaces/execution policies, and extraction/retrieval triggers, particularly LightMem's event trigger and Repoformer's query, so readers can relate costs to the configurations used.

Response to Weakness 1 and Q2 — Simpler dynamic efficiency reminder

Thank you for suggesting this control. We evaluated a generic efficiency reminder at the context tail, replacing EvidenceCard's dynamic tail, while retaining the original base-agent system policy:

# Runtime Efficiency
- Complete the coding task correctly while minimizing unnecessary
  tool calls and context tokens.
- Reuse information already available instead of repeating work.
- Keep capacity for implementation and verification.
- Prefer targeted inspection; expand only when needed.
- Use the runtime feedback to decide whether another action
  is worth its cost.

This tests repeated efficiency guidance at action selection, supplied at the context tail rather than in the system prompt. We also tested an evidence-policy-only reminder there, without ledger feedback.

Dynamic-reminder controls: GLM-5 / SWE-Bench-93. All controls retain the original base system policy; Full EvidenceCard additionally includes the EvidenceCard policy in the system policy and supplies the full ledger at the context tail.

MethodStatic policyDynamic feedback at the context tailInput tokens ↓Tool tokens ↓Requests ↓Success rate (%) ↑
Base agentOriginal base policyNone89.70M54.63M5.2K80.6
Generic efficiencyOriginal base policyGeneral efficiency44.68M26.04M4.1K80.6
Evidence policy onlyOriginal base policyEvidenceCard31.57M12.65M2.9K80.6
Full EvidenceCardEvidenceCard policyEvidenceCard12.45M3.44M1.7K81.7

The generic reminder substantially reduces costs, confirming the value of persistent efficiency guidance. Full EvidenceCard further reduces input tokens by 72.1%, tool tokens by 86.8%, and requests by approximately 58.5% (4.1K to 1.7K), with observed success of 81.7% versus 80.6%.

The evidence-policy-only reminder also improves over generic guidance, but still uses 31.57M versus 12.45M input tokens and 12.65M versus 3.44M tool tokens. Neither reminder reproduces the full method's savings. These results support the practical value of combining the evidence policy with a structured, continuously updated acquisition-history ledger, beyond repeated efficiency reminders.

Dear Reviewer,  As the discussion period is drawing to a close, we would like to briefly follow up on our rebuttal. Your feedback has been very valuable to us, and we are eager to address any remaining concerns and further improve the quality of our work. Please let us know if anything remains unresolved; we would be happy to provide further clarification.  Thank you for your time and effort in reviewing our paper.  Best regards,  Authors

I thank the authors for providing detailed responses to address my questions. My concerns have been mostly addressed.

However, I notice that the "Evidence policy only" row from "Dynamic-reminder controls" section in response is identical to the "w/o stable prefix" row in Table 4 in the paper. My understanding of "w/o stable prefix" is that the dynamic ledger is still remained, which seems to be the opposite of what "evidence policy only" means. Can you clarity?

Thank you for your careful follow-up and for pointing out this inconsistency. Your understanding is correct: the dynamic evidence ledger is retained in the “w/o stable prefix” ablation. The row labeled “Evidence policy only” in our previous response reports this same configuration and should have been labeled “Ledger only (w/o stable prefix).” We apologize for the inaccurate labeling and description.

For clarity, the exact configurations are described below. All four conditions use the following base system policy. In particular, we used the template you proposed in the Generic efficiency experiment:

Base agent:

system prompt:
You are a minimal CLI agent. Complete the user's task using the available tools when needed. Use the current workspace by default. When finished, answer concisely with the result.
Dynamic feedback at the context tail(Return every time following the tool call):

None

Generic efficiency

system prompt:
You are a minimal CLI agent. Complete the user's task using the available tools when needed. Use the current workspace by default. When finished, answer concisely with the result.

# Runtime Efficiency
- Complete the coding task correctly while minimizing unnecessary
  tool calls and context tokens.
- Reuse information already available instead of repeating work.
- Keep capacity for implementation and verification.
- Prefer targeted inspection; expand only when needed.
- Use the runtime feedback to decide whether another action
  is worth its cost.
Dynamic feedback at the context tail(Return every time following the tool call):
You have made XXX tool calls. Before reading more, check whether you can already make the patch or run a targeted test.

Evidence policy only (w/o stable prefix)

system prompt:
You are a minimal CLI agent. Complete the user's task using the available tools when needed. Use the current workspace by default. When finished, answer concisely with the result.
Dynamic feedback at the context tail(Return every time following the tool call):

# EvidenceCard Runtime
- enabled: true
- current_query: <latest user query, at most 240 characters>
- context_shape: messages(user=<u>, assistant=<a>, compact_boundaries=<c>)
- evidence_debt_count: <weak-evidence count>
- repeated_evidence_count: <repeat count>
- phase_counts: <nonzero phase=count pairs>

## Recent Evidence Ledger
- <id>: tool phase=<phase> evidence=<kind> target=<target>
  tokens=<visible tokens> score=<score> decision=<decision>
  status=<status> [repeated]

Full EvidenceCard

system prompt:
You are a minimal CLI agent. Complete the user's task using the available tools when needed. Use the current workspace by default. When finished, answer concisely with the result.

# EvidenceCard
- Method: Plan-Evidence Token Budgeting for Coding Agents.
- Maintain a plan-evidence ledger. Reuse prior evidence before
  rereading the same file, symbol, test, or log.
- Keep patch and verification reserves; do not exhaust the
  budget on broad exploration or verbose explanations.
- If evidence is weak, escalate context one level at a time;
  if evidence is sufficient, stop reading and patch/verify.
Dynamic feedback at the context tail(Return every time following the tool call):

# EvidenceCard Runtime
- enabled: true
- current_query: <latest user query, at most 240 characters>
- context_shape: messages(user=<u>, assistant=<a>, compact_boundaries=<c>)
- evidence_debt_count: <weak-evidence count>
- repeated_evidence_count: <repeat count>
- phase_counts: <nonzero phase=count pairs>

## Recent Evidence Ledger
- <id>: tool phase=<phase> evidence=<kind> target=<target>
  tokens=<visible tokens> score=<score> decision=<decision>
  status=<status> [repeated]

The results with the corrected label are as follows:

MethodInput tokens ↓Tool tokens ↓Requests ↓Success rate (%) ↑
Base agent89.70M54.63M5.2K80.6
Generic efficiency44.68M26.04M4.1K80.6
Ledger only (w/o stable prefix)31.57M12.65M2.9K80.6
Full EvidenceCard12.45M3.44M1.7K81.7

The generic efficiency condition substantially reduces costs, confirming the value of persistent efficiency guidance. Compared with this control, Full EvidenceCard further reduces input tokens by 72.1%, tool tokens by 86.8%, and requests by approximately 58.5%, with an observed success rate of 81.7% versus 80.6%. Results support the practical benefit of combining the EvidenceCard policy with the continuously updated evidence ledger beyond the simpler reminder tested here. The comparison between the ledger-only and full configurations additionally shows the contribution of the stable policy prefix. These comparisons do not isolate the ledger as the sole source of the full method’s gains.

Thank you again for identifying this inconsistency.

Comments on the paper

1 comment

Dear Area Chair,

Thank you for your time and careful consideration of our submission. To provide a clearer understanding of our work, we summarize the main contributions, additional evidence provided during rebuttal:

EvidenceCard addresses the cost of acquiring context across an agent's trajectory. It combines a stable evidence policy with a lightweight acquisition-history ledger, presented before the agent's next action. The original agent interprets this feedback relative to its current plan and decides whether to acquire more information, reuse observations, patch, or verify. The method requires neither model training nor auxiliary LLM calls. Empirical results show that EvidenceCard significantly reduces file read tokens and requests, reducing input tokens by up to 86% compared to the baseline, while maintaining robust task resolution performance.

The reviewers recognized several strengths:

  • Practical importance and clear motivation [AbnZ, cHwG].
  • Substantial efficiency gains supported by multiple models, benchmarks, ablations, and per-task analyses [x9hd, AbnZ, cHwG].
  • Lightweight deployment without training or auxiliary LLM calls [x9hd, cHwG].
  • Clear presentation [x9hd, AbnZ, cHwG].

The main questions concerned the contribution beyond generic efficiency prompting, the value and reliability of the ledger signals, baseline integration, robustness, and applicability beyond shell commands. We have provided detailed responses to these questions and included the following additional experiments in the revised PDF:

  • Baseline integration details [x9hd]. Appendix C specifies the official components reused and the adaptations applied, including LightMem's within-task extraction trigger and Repoformer's agent-context retrieval query. This makes the reported costs interpretable in relation to the evaluated configurations.

  • Stronger prompting controls [x9hd, cHwG]. Against a generic efficiency policy plus a dynamic tool-count reminder, full EvidenceCard reduces input and tool-observation tokens by 72.1% and 86.8%, respectively. With the same full ledger supplied to both policies, EvidenceCard's policy further reduces these costs by 51.0% and 69.8% relative to generic efficiency instructions. These complementary controls support the combined policy-and-ledger mechanism. The simple-reminder configurations are documented in Appendix D.

  • Structured-tool deployment and explicit scope limits [AbnZ]. Using structured file and search tools on GLM-5 / SWE-Bench-93, EvidenceCard reduces input and tool-observation tokens by 68.3% and 73.4% against that deployment's own baseline, with 81.7% observed success in both conditions. Appendix E documents the adapters and a failure mode involving necessary repeated GUI actions, motivating acquisition-specific ledger coverage. We limit empirical claims to coding and terminal tasks; arbitrary MCP tools and non-coding domains require further development and evaluation.

Although we have not yet received further feedback from Reviewer AbnZ, we particularly appreciate their concerns about tool semantics and generality, as these closely reflect challenges we encountered when deploying EvidenceCard in our open-source agent framework. This deployment showed that restricting the ledger to a small set of supported acquisition tools (only readFile、extractText、globFiles、grepFiles) can already deliver substantial benefits. On GLM-5 / SWE-Bench-93, input consumption decreases from 115.23M to 36.50M tokens, a 68.3% reduction, while observed task success remains 81.7%. The higher input cost compared to the minimal agent in the paper is due to the significant overhead of system prompt and tool descriptions prompt in our agent framework. This experience helped us clarify that ledger coverage should follow acquisition semantics: repeated GUI actions, for example, may be necessary to advance the interface state and should not automatically receive reuse feedback. The EvidenceCard implementation is already included in the framework's publicly released source code. We omit identifying links during review to preserve anonymity. Upon acceptance, we are happy to provide the code repository link for our open-source agent, the complete code for EvidenceCard, the reference implementation, the structured tool adapter, and the corresponding reproduction steps.

We thank the reviewers for their valuable suggestions, which have helped us improve our work. We also appreciate their positive feedback on our research and the discussion. We would also be most grateful if the paper could earn your recognition.

Thank you once again for your valuable time.

Sincerely,

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.