acceptodds
Under review as a conference paper at ICLR 2027

Adaptive Consistency Graph for Long-Horizon Agents

Abstract

Large language model agents can often make reasonable local decisions on short tasks, yet their performance degrades when success requires long sequences of dependent actions and tool calls. During execution, task requirements, historical evidence, and the current execution state may gradually become disconnected, so later decisions can drift from the original objective. We study this problem by introducing the Adaptive Consistency Graph (ACG) for long-horizon execution. ACG incrementally organizes execution evidence and its provenance in a persistent graph, then constructs a temporary requirement-centered view for each decision under a bounded context budget. Rather than replacing the base agent's planner or tool executor, ACG provides a structured and traceable context view for each decision. In the matched evaluation, ACG improves GPT-5.6-luna's average success from 44.5% with ReAct to 50.2%, with the largest gain on BrowseComp-Plus (73.5% versus 62.4%). We further analyze trajectory structure and inference cost to characterize this improvement.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Large Language Model Agents, Long-Horizon Reasoning, Context Management

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_h5GBOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a mechanism named Adaptive Consistency Graph, designed specifically for LLM agents handling long-horizon tasks. The framework maintains a persistent graph structure grounded in execution history to retrieve evidence relevant to current needs within a limited context budget. By integrating dual-query retrieval, graph expansion, and need-driven context construction, it aims to help agents retain relevant historical information and avoid execution failures. Experimental results across three benchmarks indicate performance improvements in certain settings, but the results lack consistency.

Critical strengths

Good setting (long-horizon)

The paper studies an potential problem for long-horizon agents, where execution evidence and task requirements may become disconnected over time. The proposed approach provides a structured way to maintain and retrieve relevant evidence across long execution histories, though it is complex and not interesting to me.

Easy to apply

The method is training-free and does not require modifying the underlying LLM, making it relatively easy to integrate. The experiments also evaluate the approach across multiple benchmarks and model settings, providing some evidence of its potential applicability beyond a single task.

Critical weaknesses

The performance gain is limited

It seems that ACG improves over ReAct in some settings, but the gains are not consistent across benchmarks and models. The paper would benefit from a more careful analysis of when and why the proposed approach helps, as well as stronger evidence that the observed improvements are robust.

The mechanism is super complicated, but I cannot see which component really helps

ACG integrates multiple design components, including graph-based persistent memory, dual-query retrieval, graph expansion, need-based evidence filtering, and context presentation constrained by a perception budget. However, existing experiments have not shown that each component is beneficial. I am quite confused that, the author design several intuitive components, but the gain is only ~5% in average, does it mean that some of the components are actually useless?

Questions for authors

I want to discuss about the significance of the problem with the authors. AdaPlanBench has shown that in interactive agentic settings, memory retrieval is no longer a serious problem, current models are capable of tracking constraints. Could you justify why the problem itself still a serious problem?

My thoughts. From my understanding, the whole thing is a training-free pipeline. Just an open question: I think it would be hard to distill these capabilities into the model itself. But if we use it as a separate harness, do you think it could still provide meaningful improvements over mature existing harnesses like Codex?

5 comments
AuthorsOpenReview ↗

Response to Reviewer h5GB

Dear Reviewer h5GB, we appreciate your review. Our responses to your questions are as below.

Response to W1

  1. When and why ACG helps. ACG helps when evidence already obtained is missing from the next decision context. In the paper-editor example, ACON’s summary omits the passage naming the editor, whereas ACG presents it under the corresponding requirement and answers correctly. Its benefit is limited when the difficulty lies elsewhere. In Browse task 3, ACG with the original DeepSeek model retains the correct candidate’s name but never finds the report needed to verify the answer; ACG with luna finds that report and answers correctly. In a DeepPlanning shopping task, the model mistakes a product’s association with a requirement for proof that it satisfies the requirement. These cases illustrate why organizing evidence can help without consistently correcting search or reasoning errors across models and tasks.
  2. Evidence across models and runs. As the original model is no longer available, we compare ACG and ReAct using Deepseek-v4.1-flash, with three runs on the same 50 tasks per benchmark. ACG achieves higher mean success rates on all three benchmarks, including Browse (66.7% versus 62.7%), with a macro-average of 54.4% versus 49.1% (Table R1). ACG also scores higher in each corresponding run on every benchmark. These results support repeatability on the evaluated subsets and show that the original Browse regression does not recur in every evaluated setting. Table R1. ACG and ReAct over three runs with Deepseek-v4.1-flash | Method | SWE-bench Lite | BrowseComp-Plus | DeepPlanning | Avg | |---|---:|---:|---:|---:| | Full ACG | 66.0 ± 2.0 | 66.7 ± 1.2 | 30.7 ± 1.2 | 54.4 ± 0.8 | | ReAct | 59.3 ± 2.3 | 62.7 ± 1.2 | 25.3 ± 4.2 | 49.1 ± 1.0 |

Response to W2

  1. What the components contribute. Graph expansion adds connected evidence that the initial retrieval may miss. Provenance grouping associates evidence with task requirements through its recorded source. Dual-query retrieval uses both the task and the latest interaction, so retrieval can reflect the current stage of execution. These components address different parts of constructing the next decision context.
  2. Component ablation. We ablate graph expansion, provenance grouping, and dual-query retrieval individually on the same 50 tasks per benchmark using Deepseek-v4.1-flash, as the original model is no longer available. Full ACG achieves a macro-average success rate of 54.0%; removing these components lowers it to 48.0%, 47.3%, and 45.3%, respectively (Table R2). The decreases of 6.0, 6.7, and 8.7 percentage points support the contribution of all three components. Unlike the overall gain over ReAct, these comparisons directly examine what is lost when each component is removed. Table R2. Component ablation | Method | SWE-bench Lite | BrowseComp-Plus | DeepPlanning | Avg | |---|---:|---:|---:|---:| | Full ACG | 33/50(66%) | 33/50(66%) | 15/50(30%) | 54.0% | | w/o graph expansion | 27/50(54%) | 32/50(64%) | 13/50(26%) | 48.0% | | w/o provenance grouping | 28/50(56%) | 31/50(62%) | 12/50(24%) | 47.3% | | w/o dual-query retrieval | 28/50(56%) | 29/50(58%) | 11/50(22%) | 45.3% |

Response to Q1

  1. The remaining problem. Even if a model remembers every task constraint, it may lack the evidence needed for its next decision. AdaPlanBench probes constraint forgetting by supplying previously disclosed constraints at every turn and finds limited gains in final accuracy. ACG addresses how execution context is organized: which tool evidence the next decision needs, which requirement it relates to, and where it came from.
  2. Why this matters in practice. In the paper-editor example, ACON retains the task goal but its summary omits the editor’s name already found in the search results. It continues searching until its budget is exhausted. ACG presents the original passage and its source under the relevant requirement, then answers correctly. The task requirement was available, but the evidence needed to complete it was lost during context updates. This shows why ensuring that later decisions can use collected evidence remains important even when models can track constraints.

Response to Q2

We believe ACG has the potential to provide further gains. Its context-organization mechanism is designed to be decoupled from both the base model and the surrounding harness, allowing it to complement mature agents such as Codex. It organizes tool evidence by task requirement, preserves source connections, and supplies relevant material to subsequent decisions. We expect this to be particularly useful in long tasks that repeatedly depend on earlier evidence. This role does not require first distilling the entire pipeline into model weights. We plan to explore integrating ACG into mature agent frameworks such as Codex and compare performance before and after integration using the same model and tasks.

Dear Reviewer h5GB,

Thank you again for your constructive feedback. We would like to briefly follow up on our response. In particular, we have:

  • Clarified why remembering task constraints does not ensure access to previously collected evidence, with a concrete example of evidence lost during summarization.

  • Added component ablations and three-run comparisons using Deepseek-v4.1-flash on 50 tasks per benchmark, with ACG achieving higher mean success than ReAct on all three.

  • Analyzed when ACG helps and where it remains limited, and discussed integration with mature agent frameworks as future work.

We would be happy to clarify any remaining points. If our response and additional evidence have addressed your concerns, we would appreciate it if you could reconsider your assessment.

Thank you for your time!

Dear Reviewer h5GB,

Thank you again for your constructive feedback. We would like to briefly follow up on our response. In particular, we have:

  • Clarified why remembering task constraints does not ensure access to previously collected evidence, with a concrete example of evidence lost during summarization.

  • Added component ablations and three-run comparisons using Deepseek-v4.1-flash on 50 tasks per benchmark, with ACG achieving higher mean success than ReAct on all three.

  • Analyzed when ACG helps and where it remains limited, and discussed integration with mature agent frameworks as future work.

We would be happy to clarify any remaining points. If our response and additional evidence have addressed your concerns, we would appreciate it if you could reconsider your assessment.

Thank you for your time!

Reviewer_yDxROpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper addresses the management of execution histories in long-running agents to support decisions that integrate historical evidence with current requirements. The proposed Adaptive Consistency Graph (ACG) preserves the content, provenance, and observation history of evidence in persistent memory. At each decision step, ACG retrieves evidence using the original task and recent interactions, constructs a temporary context organized by requirement within a token budget, and provides it to the underlying agent. Evaluated on three benchmarks with two models against five baselines, ACG achieves improved average performance.

Critical strengths

S1. The design clearly separates persistent evidence storage from temporary decision contexts. This separation offers practical value as a concrete system design for reusing long execution histories under context constraints.

S2. The reported results demonstrate the practical potential of ACG across different types of agent tasks. Table 1 reports higher average success rates across the three benchmarks compared with the baselines. Improvements across multiple tasks motivate further investigation of this memory architecture.

Critical weaknesses

W1. The connection between existing methods’ failures and ACG’s design is insufficiently demonstrated. For example, ACON (Kang et al., ICML 2026) also preserves important states and action-outcome information. However, the paper lacks concrete analyses and component-level ablations of graph retrieval and requirement-specific context construction. It remains unclear which information losses or missing relations these components address. Quantitative evidence would be preferable, but qualitative trends would also help. Without this evidence, their added value beyond access to original evidence is difficult to assess.

W2. The analysis does not adequately explain why ACG’s effectiveness varies across models and tasks or characterize its limitations. In Table 1, the success rate of DeepSeek-v4-flash on BrowseComp-Plus decreases from 68.2% with ReAct to 62.7% with ACG, while success rates on DeepPlanning remain approximately 12% for both models. However, the execution diagnostics in Sections 4 do not distinguish whether these outcomes arise from retrieval failures, evidence placement, information omission, or reasoning errors by the base model. Without such analysis, it remains difficult to identify the conditions under which ACG is beneficial and those under which its gains are limited or its effects are detrimental.

Questions for authors

1. [Clarification] Could the authors present a step-by-step example where ACON fails but ACG succeeds? Please identify the evidence needed before the failure and explain how the baseline context omitted that evidence or its connections. Then show what ACG retrieved and how its placement enabled a different decision. A specific failure mechanism beyond context length limitations would help address W1.

2. [Qualitative or Quantitative Analysis] Have the authors analyzed why ACG underperforms ReAct on BrowseComp-Plus with DeepSeek-v4-flash, or identified representative failure cases?

9 comments
AuthorsOpenReview ↗

Response to Reviewer yDxR (1/2)

Dear Reviewer yDxR, we appreciate your review. Our responses to your questions are as below.

Response to W1 & Q1

  1. Added value beyond evidence storage. ACG connects tool evidence to its source and the task requirement it supports. Dual-query retrieval uses the task and latest interaction; graph expansion adds connected evidence; provenance grouping places evidence under a requirement only when its recorded origin supports that association. These steps make relevant evidence and its connections explicit in the next decision context.

  2. Observed information loss. DeepSeek/BrowseComp-Plus task 37 asks for a paper’s editor. ACON retrieved document 85088 containing “Edited by Satoshi Tabata,” and this passage entered its compression input. The summary omitted the paper–editor link and still described the paper as unidentified. Once the passage left the latest interaction, the next decision input also lacked it; ACON exhausted its budget after 99 searches without an answer. ACG placed the same document under requirement R1 with its source and editor information, then submitted Satoshi Tabata in its next action and passed official evaluation. This identifies a concrete information loss during context updates.

  3. Component ablation. We ablate graph expansion, provenance grouping, and dual-query retrieval individually on the same 50 tasks per benchmark using Deepseek-v4.1-flash, as the original model is no longer available. Full ACG achieves a macro-average success rate of 54.0%; removing these components lowers it to 48.0%, 47.3%, and 45.3%, respectively (Table R1). The decreases of 6.0, 6.7, and 8.7 percentage points support the contribution of all three components.

Table R1. Component ablation

MethodSWE-bench LiteBrowseComp-PlusDeepPlanningAvg
Full ACG33/50(66%)33/50(66%)15/50(30%)54.0%
w/o graph expansion27/50(54%)32/50(64%)13/50(26%)48.0%
w/o provenance grouping28/50(56%)31/50(62%)12/50(24%)47.3%
w/o dual-query retrieval28/50(56%)29/50(58%)11/50(22%)45.3%
AuthorsOpenReview ↗

Response to Reviewer yDxR (2/2)

Response to W2 & Q2

  1. Why performance decreases on BrowseComp-Plus. Some failed runs continue searching around an incorrect candidate without changing direction. In task 3, ACG encounters the correct answer, Amr Zaki, but never finds the report confirming that he scored twice in a semifinal. It ultimately selects another player. ReAct finds that report and answers correctly. This example shows that retaining a candidate’s name is insufficient: the agent must also obtain evidence that verifies the answer. ACG helps organize and use collected material, but organizing that material may not correct a mistaken search direction. Some runs consequently continue searching without submitting an answer. Across all 830 tasks, ACG leaves 103 unanswered within budget, compared with 10 for ReAct. Among the 87 ReAct-only successes, ACG also retrieves annotated sources on fewer tasks despite making more searches (Table R2).

  2. Why DeepPlanning remains difficult. Some failures occur after the relevant information has been obtained. In one shopping task, the requested item must be suitable for Spring/Autumn. The model identifies a qualifying product and explicitly recognizes that another candidate is winter clothing. However, because the winter item appears under the corresponding requirement in ACG’s context, the model treats “related to this requirement” as “satisfies this requirement.” It removes the qualifying product, adds the winter item, and fails evaluation. This case shows that the model can misinterpret ACG’s grouping information. Placing evidence under a requirement helps the agent find and compare it, but does not replace checking each condition, such as season and sales volume. Relevant information can therefore remain available while the final plan still violates the requirements.

  3. When ACG helps and why its effects vary. The paper-editor example in W1 illustrates a problem ACG can address: the answer appears in the search results but is omitted when the history is summarized, leaving it unavailable to the next decision. ACG explicitly presents the passage, and the model answers correctly. The failure cases above identify limits to this benefit. ACG can help when available evidence is not being used effectively; its gains are limited when decisive evidence has not been found or the model misinterprets the material. Different models can also make different decisions: on the same Browse question, ACG with luna finds the decisive report and answers correctly, whereas the original DeepSeek run does not. We additionally compare ACG and ReAct using Deepseek-v4.1-flash on the same 50 tasks per benchmark. ACG achieves higher mean success rates on all three benchmarks, including 66.7% versus 62.7% on Browse and a macro-average of 54.4% versus 49.1%. Thus, the Browse regression observed with the original DeepSeek model does not recur in every evaluated setting.

Table R2. BrowseComp-Plus outcome and source-retrieval analysis

PopulationMetricReActACG
All 830 tasksCorrect submissions566520
All 830 tasksIncorrect submissions254207
All 830 tasksNo answer within budget10103
All 830 tasksRetrieved at least one annotated source710670
All 87 ReAct-only successesRetrieved at least one annotated source8636
All 87 ReAct-only successesRetrieved all annotated sources6918
All 87 ReAct-only successesMedian search calls3875

Dear Reviewer yDxR,

Thank you again for your constructive feedback. We would like to briefly follow up on our response. In particular, we have:

  • Provided a step-by-step case showing how ACON lost an already retrieved paper–editor link during summarization, while ACG explicitly presented it and answered correctly.

  • Added component ablations on 50 tasks per benchmark using Deepseek-v4.1-flash; removing each component reduced macro-average success by 6.0–8.7 percentage points.

  • Analyzed BrowseComp-Plus and DeepPlanning failures, distinguishing missing decisive sources from misinterpreted requirement grouping to clarify ACG’s benefits and limitations.

We would be happy to clarify any remaining points. If our response and additional analyses have addressed your concerns, we would appreciate it if you could reconsider your assessment.

Thank you for your time!

Reviewer_D7ntOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes the Adaptive Consistency Graph (ACG), an execution-memory layer for long-horizon LLM agents. ACG stores observations as source-linked memory units in a graph, retrieves evidence using both the task and the latest event, groups evidence under the requirement it supports (only when an explicit provenance link exists), and renders a requirement-centered context view under a token budget, without changing the base planner or tools. On DeepPlanning, BrowseComp-Plus and SWE-bench Lite with GPT-5.6-luna and DeepSeek-v4-flash, ACG reports the highest average success against five baselines (50.2% vs 44.5% for ReAct with luna; 45.9% vs 43.8% with DeepSeek).

Critical strengths

ACG ranks first in five of six model–benchmark settings, with its largest gain on BrowseComp-Plus with luna (73.5% vs 62.4% for ReAct, Table 1).

Critical weaknesses
  • None of ACG's components (graph expansion, provenance-qualified grouping, dual-query retrieval, budgeted rendering) are ablated.
  • The paper never states which base agent ACG wraps or whether its prompts and tool schemas match ReAct's.
  • Most baselines appear broken: on SWE-bench Lite with luna, COMPASS reaches 12.7%, TDP* 22.3% and CUGA 3.7% vs 60.3% for ReAct (Table 1), and roughly 70–90% of COMPASS, TDP* and CUGA tasks on DeepSeek/SWE-bench Lite end with no patch submitted (Appendix C, Table 2).
  • Against ReAct, the only seemingly working baseline, the gain is +5.7 (luna) and +2.1 (DeepSeek) points from single runs, and ACG loses on DeepSeek/BrowseComp-Plus (62.7% vs 68.2%, Table 1).
  • On DeepSeek, the gain over ReAct may come from tool-call formatting rather than memory: 80.0% of ReAct's failed SWE-bench Lite tasks contain rejected tool calls vs 1.8% for ACG, and 72.3% vs 0.0% on BrowseComp-Plus (Appendix C, Table 2).
Questions for authors
  1. Is ACG built on the same ReAct agent with identical prompts and tool schemas? Do rejected tool calls count toward the 100-action budget?
  2. How were the COMPASS, TDP*, CUGA and ACON integrations validated? Were any checked against their originally reported numbers?
  3. Can you ablate each component (graph expansion, provenance grouping, dual-query retrieval) and sweep the action budget (e.g., 50/100/200)?
  4. Can you report variance over repeated runs, at least for ACG vs ReAct?
5 comments
AuthorsOpenReview ↗

Response to Reviewer D7nt (1/2)

Dear Reviewer D7nt, we appreciate your review. Our responses to your questions are as below.

Response to W1 & Q3

  1. Ablation study. We ablate graph expansion, provenance grouping, and dual-query retrieval individually on 50 tasks per benchmark using Deepseek-v4.1-flash, as the original model is no longer available. As shown in Table R1, Full ACG achieves a macro-average success rate of 54.0%. Removing each component lowers this to 48.0%, 47.3%, and 45.3%, respectively, a decrease of 6.0, 6.7, and 8.7 percentage points. These results support the contribution of all three components.
  2. Action budget. As shown in Table R2, Full ACG achieves macro-average success rates of 53.3%, 54.0%, and 54.0% within 50, 100, and 200 actions, respectively. Increasing the budget from 100 to 200 actions produces no additional successes.

Table R1. Component ablation

MethodSWE-bench LiteBrowseComp-PlusDeepPlanningAvg
Full ACG33/50 (66%)33/50 (66%)15/50 (30%)54.0%
w/o graph expansion27/50 (54%)32/50 (64%)13/50 (26%)48.0%
w/o provenance grouping28/50 (56%)31/50 (62%)12/50 (24%)47.3%
w/o dual-query retrieval28/50 (56%)29/50 (58%)11/50 (22%)45.3%

Table R2. Action-budget results

Action thresholdSWE-bench LiteBrowseComp-PlusDeepPlanningAvg
5032/50(64%)33/50(66%)15/50(30%)53.3%
10033/50(66%)33/50(66%)15/50(30%)54.0%
20033/50(66%)33/50(66%)15/50(30%)54.0%

Response to W2 & Q1

  1. Base agent and shared settings. ACG is built on ReAct and uses the same base system instructions, task descriptions, and tool schemas. ACG only adds retrieved evidence with source references to the decision context.
  2. Action counting. Rejected tool calls count toward the 100-action budget.

Response to W3 & Q2

  1. Code sources. COMPASS, CUGA, and ACON use their public implementations; TDP* is our reproduction, marked with an asterisk in the paper.
  2. Integration validation. We checked tool execution, final outputs, and evaluation records across all three benchmarks: patch submission and official tests on SWE-bench Lite; search results, answer submission, and judging on BrowseComp-Plus; and tool interactions, final carts or itineraries, and constraint evaluation on DeepPlanning. All four baselines submitted patches that passed official SWE-bench evaluation under both models (Table R3). We also verified that ACON’s compression runs and its summaries enter subsequent model inputs.
  3. No-patch failure analysis. The inspected trajectories identify failures in the baselines’ own execution: TDP* submitted an empty patch without modifying code, and the submission interface returned normally; CUGA read the same file 100 times without editing or submitting, exhausting its action budget. In these cases, the missing patches resulted from the baselines’ failure to make code changes, not from disconnected tools or patches being dropped by our integration. The broader statistics show that 143/210 TDP*, 108/206 CUGA, and 89/157 ACON no-patch runs contained no interface rejection. Most rejection episodes were also followed by an accepted action: 69.2% for COMPASS and over 95% for the other three methods (Table R3).
  4. Checks against original scores. We checked our TDP* reproduction on 100 HotpotQA tasks with DeepSeek-V3.2: overall accuracy was 67.0% versus the paper’s 73.0%; accuracy among submitted answers was 83.8% versus 85.9%. The retrieval and judging settings differed. For the methods using public implementations, validation focused on their execution in our benchmarks; we did not separately rerun their original benchmark settings.

Table R3. DeepSeek/SWE-bench Lite patch delivery and rejection diagnostics

MethodNonempty patchesNo rejection / all no-patch runsRecovered / all rejection episodes
COMPASS29/300(9.7%)29/271(10.7%)380/549(69.2%)
TDP*90/300(30.0%)143/210(68.1%)139/146(95.2%)
CUGA94/300(31.3%)108/206(52.4%)263/264(99.6%)
ACON143/300(47.7%)89/157(56.7%)219/222(98.6%)
AuthorsOpenReview ↗

Response to Reviewer D7nt (2/2)

Response to W4

  1. Value of the improvement. ACG improves context management in a ReAct agent, using the same base model and benchmark tools without additional model training. On DeepSeek/SWE-bench Lite, it reduces unresolved issues by 15.4% (130 to 110); on DeepPlanning, successful plans increase from 23 to 42. The practical benefit is fewer failed repairs and more feasible plans. The +2.1-point average gain is smaller because the BrowseComp-Plus loss offsets part of these improvements.

  2. BrowseComp-Plus decline. In the original DeepSeek runs, ACG exhausts the action budget without submitting an answer on 103 of the 830 tasks, compared with 10 for ReAct (Table R4). Some inspected failure trajectories continue searching around an incorrect candidate until the budget runs out. ACG helps organize and reuse collected evidence, but once the model has chosen an incorrect search direction, ACG alone may not redirect it: ACG organizes and retrieves existing material, while the base model still decides whether to change direction, what to search next, and when to stop. In this setting, the additional unanswered tasks offset ACG’s gains on other tasks, resulting in a lower overall success rate than ReAct.

  3. Results with Deepseek-v4.1-flash. Using the model Deepseek-v4.1-flash, we compare ACG and ReAct on the same 50 tasks per benchmark. ACG achieves higher mean success rates on all three benchmarks, including BrowseComp-Plus, with an average of 54.4% versus 49.1% for ReAct (+5.3 percentage points; Table R5).

Table R4. BrowseComp-Plus outcome and source-retrieval analysis

PopulationMetricReActACG
All 830 tasksCorrect submissions566520
All 830 tasksIncorrect submissions254207
All 830 tasksNo answer within budget10103
All 830 tasksRetrieved at least one annotated source710670
All 87 ReAct-only successesRetrieved at least one annotated source8636
All 87 ReAct-only successesRetrieved all annotated sources6918
All 87 ReAct-only successesMedian search calls3875

Response to Q4

We repeated Full ACG and ReAct three times on the same 50 tasks per benchmark using Deepseek-v4.1-flash. Their macro-average success rates are 54.4 ± 0.8% and 49.1 ± 1.0%, respectively, a gain of 5.3 percentage points. Table R5 reports mean ± sample standard deviation.

Table R5. Repeated-run success rates

MethodSWE-bench LiteBrowseComp-PlusDeepPlanningAvg
Full ACG66.0 ± 2.066.7 ± 1.230.7 ± 1.254.4 ± 0.8
ReAct59.3 ± 2.362.7 ± 1.225.3 ± 4.249.1 ± 1.0

Response to Q5

  1. What the statistic measures. The 80.0% means that 80.0% of failed ReAct SWE tasks contained at least one rejected call. It measures occurrence within tasks, not the fraction of actions rejected or the cause of failure.

  2. Comparison across methods. Table R6 includes all six methods. Rejections also occur in successful tasks: the rate is 78.8% for ReAct’s successful SWE tasks. On Browse, both ReAct and CUGA have higher rejection-occurrence rates among successful tasks than among failed tasks. ReAct also outperforms ACG on Browse despite more rejections.

  3. Recovery and a rejection-free comparison. In ReAct’s failed SWE traces, 145 of 8,077 actions were rejected (1.8%), and the next attempt was accepted after each rejection. More directly, on the 62 SWE tasks where ReAct had no rejected calls, ACG solves 42 versus ReAct’s 36. Thus, ACG’s gains also occur without ReAct formatting failures; the reported occurrence rates do not establish formatting as the main explanation.

Table R6. Success rates and rejected-call occurrence across methods SR is overall success rate; F/S is the percentage of failed/successful tasks containing at least one interface rejection, rather than the percentage of actions rejected.

BenchmarkMethodSRFS
SWE-bench LiteReAct56.7%80.0%78.8%
SWE-bench LiteCOMPASS4.0%87.8%58.3%
SWE-bench LiteTDP*10.7%32.5%40.6%
SWE-bench LiteCUGA3.7%56.2%63.6%
SWE-bench LiteACON31.3%41.3%30.9%
SWE-bench LiteACG63.3%1.8%2.1%
BrowseComp-PlusReAct68.2%72.3%82.5%
BrowseComp-PlusCOMPASS8.7%95.8%59.7%
BrowseComp-PlusTDP*18.1%87.6%79.3%
BrowseComp-PlusCUGA8.6%11.2%50.7%
BrowseComp-PlusACON48.4%3.0%17.4%
BrowseComp-PlusACG62.7%0.0%0.2%

Dear Reviewer D7nt,

Thank you again for your constructive feedback. We would like to briefly follow up on our response. In particular, we have:

  • Added component ablations and action-budget analysis, with performance decreasing when any of the three ablated components is removed.

  • Reported three-run ACG–ReAct comparisons using Deepseek-v4.1-flash on 50 tasks per benchmark (54.4 ± 0.8% versus 49.1 ± 1.0% macro-average success).

  • Clarified the shared ReAct setup, checked baseline integrations, and shown that ACG’s gains also occur on tasks without ReAct tool-call rejections.

We would be happy to clarify any remaining points. If our response and additional results have addressed your concerns, we would appreciate it if you could reconsider your assessment.

Thank you for your time!

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Author Responses

We thank the Area Chair and reviewers for their careful assessment and constructive feedback. We summarize the main clarifications and additional evidence provided during the discussion.

1. Clarified contribution and positioning. ACG is a training-free execution-memory layer that preserves collected evidence and source links, then organizes relevant material by task requirement for subsequent decisions. It wraps the same ReAct agent with shared base system instructions, task descriptions, and tool schemas. Our responses distinguish keeping execution evidence available from remembering task constraints: an agent may retain the goal while losing the evidence needed to complete it.

2. Component ablations support the design choices. Because the original model is no longer available, the component ablations and repeated-run comparisons use Deepseek-v4.1-flash on the same 50 tasks per benchmark. Full ACG achieves 54.0% macro-average success; removing graph expansion, provenance grouping, or dual-query retrieval reduces this to 48.0%, 47.3%, and 45.3%, respectively.

3. Repeated runs provide additional evidence of consistency. Across three runs on these fixed subsets, ACG achieves 54.4 ± 0.8% macro-average success versus 49.1 ± 1.0% for ReAct, a gain of 5.3 percentage points (mean ± sample standard deviation). ACG has higher means on all three benchmarks, including 66.7% versus 62.7% on BrowseComp-Plus. These results support repeatability within the evaluated setting.

4. Inspected no-patch failures arose during baseline execution; gains also occur without ReAct call rejections. We checked tool execution, submissions, and evaluation records across all three benchmarks. In inspected no-patch cases, TDP* submitted an empty patch without editing code, while CUGA repeatedly read a file until its budget was exhausted. The tools functioned normally; these failures arose because the baselines did not complete the code changes. Separately, on the 62 original DeepSeek/SWE-bench Lite tasks where ReAct had no rejected calls, ACG solved 42 versus ReAct’s 36. This shows that ACG’s gains also occur in the absence of ReAct tool-call formatting errors.

5. ACG helps when collected evidence is lost, but search and constraint errors can still cause failure. In the paper-editor case, ACON had retrieved the editor’s name but omitted the paper–editor link during summarization; ACG presented that evidence under the relevant requirement and answered correctly. Conversely, some original DeepSeek/BrowseComp-Plus failures followed incorrect search directions without finding the decisive source. A DeepPlanning case shows the model mistaking a winter item’s placement under a requirement for satisfaction of a Spring/Autumn constraint. ACG therefore helps when available evidence would otherwise be lost during context updates; missing decisive sources or misinterpreting its grouping can limit or offset that benefit. These concrete mechanisms help explain why gains vary across models and tasks.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.