- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes EvidenceCard, a lightweight method for reducing coding-agent context costs. It combines a stable prompt policy with a rule-based ledger and a compact runtime card to discourage unnecessary reads and exploration. Experiments across several models and benchmarks show substantial reductions in input tokens, tool-observation tokens, and API requests, with comparable single-run task success rates.
Critical strengths
-
The problem is practically important, and the method requires neither additional training nor auxiliary LLM calls.
-
The resource savings are substantial across multiple models and benchmarks, supported by ablations and per-task analyses.
Critical weaknesses
-
Novelty and positioning. Prior work has explored plan-aware context engineering and subgoal-driven working memory management, including PAACE and HiAgent. Although EvidenceCard differs in its lightweight, prompt-based design for reducing future information acquisition, the paper does not sufficiently establish why its evidence-specific formulation is necessary beyond existing context-management strategies. A clearer comparison of the underlying mechanisms, ideally supported by matched prompting baselines, would better establish the methodological novelty.
-
Insufficient controls to isolate the gains. The ablations do not compare EvidenceCard with a generic efficiency-oriented prompt paired with the same runtime statistics. It therefore remains unclear whether the evidence-specific formulation is necessary for the reported improvements.
-
Heuristic signals are not validated. The scoring and repeat-detection rules are simple proxies for evidence quality and redundancy. Although their limitations are described, the paper does not validate whether these signals reliably reflect semantic relevance or evidence sufficiency.
-
Limited evidence on task-success robustness. Each method–task pair is evaluated once. The authors acknowledge this and avoid statistical claims about accuracy improvements, but repeated runs would still provide stronger evidence that the resource savings do not compromise solution quality.
Questions for authors
Refer to the weaknesses.
5 comments
Response to Concern 3: Interpretation and Role of the Heuristic Signals
Thank you for raising this concern. We would like to clarify the intended role of these signals and distinguish the ledger's bookkeeping from the original agent's semantic judgments.
EvidenceCard records previous tool interactions and presents acquisition-history statistics together with heuristic observation cues. The score is derived from deterministic content rules, and the repeated-target flag records whether an extracted target has appeared previously. These signals are advisory: they do not block tool execution, automatically skip observations, or determine when information acquisition must stop.
The original coding agent interprets this feedback together with the task, its current plan, and the actual observations. It remains responsible for judging semantic relevance, identifying outstanding information needs, and deciding whether the available evidence is sufficient to patch, verify, or finish. The runtime does not maintain an explicit requirement-closure state or automatically certify evidence sufficiency.
This distinction is already stated in the method: the ledger “does not itself establish the semantic relevance or sufficiency of an observation,” and its decision labels “are feedback for the next turn and do not block tool execution.” Nevertheless, we recognize that phrases such as “making evidence sufficiency visible” and “irrelevant-read aware” can suggest a stronger capability. A more precise description of our intended claim is:
EvidenceCard exposes acquisition history and heuristic observation cues to the original coding agent, which uses them when assessing whether further information is needed for its current plan.
In particular, repeated targets should not be equated with semantic redundancy. Reading a modified file or rerunning a test after a code change may provide new information despite revisiting the same target. Similarly, a low heuristic score does not establish that an observation is irrelevant. The feedback makes these acquisition patterns available for the agent to interpret rather than resolving their semantic meaning automatically.
To assess the operational contribution of the cues, we additionally evaluated their joint removal alongside the individual ablations.
Table 3. Operational contribution of the heuristic feedback signals on GLM-5 / SWE-Bench-93.
| Variant | Input tokens ↓ | Tool tokens ↓ | Requests ↓ | Success rate (%) ↑ |
|---|---|---|---|---|
| Full EvidenceCard | 12.45M | 3.44M | 1.7K | 81.7 |
| Without weak-observation signals | 15.16M | 4.50M | 1.9K | 81.7 |
| Without repeat/reuse signals | 16.44M | 4.83M | 2.0K | 81.7 |
| Without both signal groups | 17.84M | 4.92M | 2.1K | 81.7 |
These results support the operational usefulness of the heuristic feedback in the evaluated setting. They are not presented as ground-truth validation of semantic relevance or evidence sufficiency. The claim supported here is that inexpensive acquisition-history cues can help the original agent manage its information-gathering behavior, while semantic interpretation and sufficiency judgments remain with that agent.
Response to Concern 4: Repeated-Run Task-Success Robustness
Thank you for this suggestion. We agree that the original single-run evaluation does not characterize run-to-run variability. The existing task bootstrap measures sensitivity to task-set composition and does not substitute for repeated agent episodes.
To provide additional evidence, we evaluated the base agent and full EvidenceCard over three runs on the same 93 tasks in GLM-5 / SWE-Bench-93. Table 4. Repeated-run evaluation on GLM-5 / SWE-Bench-93.
| Method | Run | Input tokens ↓ | Tool tokens ↓ | Requests ↓ | Success rate (%) ↑ |
|---|---|---|---|---|---|
| Base agent | 1 | 89.70M | 54.63M | 5.2K | 80.6 |
| Base agent | 2 | 85.32M | 50.19M | 5.1K | 80.6 |
| Base agent | 3 | 93.84M | 59.31M | 5.3K | 80.6 |
| Full EvidenceCard | 1 | 12.45M | 3.44M | 1.7K | 81.7 |
| Full EvidenceCard | 2 | 11.96M | 3.41M | 1.7K | 80.6 |
| Full EvidenceCard | 3 | 13.99M | 3.45M | 1.7K | 81.7 |
Observed task success remains comparable. The base agent achieves 80.6% success in all three runs, while EvidenceCard achieves 81.7%, 80.6%, and 81.7%. Thus, none of the additional runs shows a lower aggregate success rate for EvidenceCard, and the substantial resource reductions persist alongside comparable observed success.
These repeated-run results strengthen the evidence that the efficiency gains are reproducible in this setting. However, three runs on 93 tasks do not establish statistical equivalence or rule out small changes in success probability. We therefore interpret the results as substantial resource savings with comparable observed task success, rather than as proof of an accuracy improvement or a guarantee of no degradation.
Dear Reviewer,
As the discussion period is drawing to a close, we would like to briefly follow up on our rebuttal. Your feedback has been very valuable to us, and we are eager to address any remaining concerns and further improve the quality of our work. Please let us know if anything remains unresolved; we would be happy to provide further clarification. If our responses have addressed your concerns, we would be grateful if you could reconsider your assessment in light of them.
Thank you for your time and effort in reviewing our paper. Best regards, Authors
Response to Concern 1: Novelty and Positioning
Thank you for highlighting the relationship to PAACE and HiAgent. We agree that plan-aware context engineering and subgoal-based working memory are established research directions. EvidenceCard's contribution concerns its specific acquisition-feedback mechanism and its empirical value in coding-agent trajectories.
Mechanism-level comparison. PAACE compresses the current context conditioned on upcoming plan steps. HiAgent organizes working memory around explicit subgoals, replacing completed subgoal trajectories with summaries and retrieving their details when needed. EvidenceCard uses deterministic bookkeeping to expose acquisition history and heuristic cues to the original coding agent before its next action. Table 1 summarizes these differences.
Table 1. Mechanism-level comparison with related context-management approaches.
Our distinction concerns the primary intervention. EvidenceCard provides explicit acquisition-history feedback without introducing an auxiliary compressor or summarizer. Plan interpretation and evidence-sufficiency judgments remain with the original agent; the ledger does not maintain an explicit requirement-to-observation mapping or automatically certify requirement closure.
Response to Concern 2: Insufficient Controls
We agree that removing individual components does not adequately distinguish evidence-specific guidance from a generic instruction to be efficient. We therefore added a generic efficiency prompt both without feedback and with the same full EvidenceCard ledger.
The generic efficiency system prompt is:
This control includes substantive efficiency guidance: reusing available information, preferring targeted inspection, and preserving capacity for implementation and verification.
The results show that: First, generic efficiency prompting already substantially reduces resource consumption relative to the base agent. Without dynamic feedback, the evidence policy uses more input tokens and requests than the generic prompt, although slightly fewer tool tokens, with the same observed success rate. These results negate that evidence-specific wording alone explains the gains.
Second, ledger feedback benefits both policies. This supports the value of dynamic acquisition feedback beyond static instructions in the evaluated setting.
Third, the decisive matched comparison is between Generic efficiency prompt + same ledger and Full EvidenceCard. With the same ledger mechanism, full EvidenceCard uses approximately 51.0% fewer input tokens, 69.8% fewer tool tokens, and 37.0% fewer requests. Its observed success rate is 81.7%, compared with 80.6% for the matched control.