- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes a mechanism named Adaptive Consistency Graph, designed specifically for LLM agents handling long-horizon tasks. The framework maintains a persistent graph structure grounded in execution history to retrieve evidence relevant to current needs within a limited context budget. By integrating dual-query retrieval, graph expansion, and need-driven context construction, it aims to help agents retain relevant historical information and avoid execution failures. Experimental results across three benchmarks indicate performance improvements in certain settings, but the results lack consistency.
Critical strengths
Good setting (long-horizon)
The paper studies an potential problem for long-horizon agents, where execution evidence and task requirements may become disconnected over time. The proposed approach provides a structured way to maintain and retrieve relevant evidence across long execution histories, though it is complex and not interesting to me.
Easy to apply
The method is training-free and does not require modifying the underlying LLM, making it relatively easy to integrate. The experiments also evaluate the approach across multiple benchmarks and model settings, providing some evidence of its potential applicability beyond a single task.
Critical weaknesses
The performance gain is limited
It seems that ACG improves over ReAct in some settings, but the gains are not consistent across benchmarks and models. The paper would benefit from a more careful analysis of when and why the proposed approach helps, as well as stronger evidence that the observed improvements are robust.
The mechanism is super complicated, but I cannot see which component really helps
ACG integrates multiple design components, including graph-based persistent memory, dual-query retrieval, graph expansion, need-based evidence filtering, and context presentation constrained by a perception budget. However, existing experiments have not shown that each component is beneficial. I am quite confused that, the author design several intuitive components, but the gain is only ~5% in average, does it mean that some of the components are actually useless?
Questions for authors
I want to discuss about the significance of the problem with the authors. AdaPlanBench has shown that in interactive agentic settings, memory retrieval is no longer a serious problem, current models are capable of tracking constraints. Could you justify why the problem itself still a serious problem?
My thoughts. From my understanding, the whole thing is a training-free pipeline. Just an open question: I think it would be hard to distill these capabilities into the model itself. But if we use it as a separate harness, do you think it could still provide meaningful improvements over mature existing harnesses like Codex?
5 comments
Dear Reviewer h5GB,
Thank you again for your constructive feedback. We would like to briefly follow up on our response. In particular, we have:
-
Clarified why remembering task constraints does not ensure access to previously collected evidence, with a concrete example of evidence lost during summarization.
-
Added component ablations and three-run comparisons using Deepseek-v4.1-flash on 50 tasks per benchmark, with ACG achieving higher mean success than ReAct on all three.
-
Analyzed when ACG helps and where it remains limited, and discussed integration with mature agent frameworks as future work.
We would be happy to clarify any remaining points. If our response and additional evidence have addressed your concerns, we would appreciate it if you could reconsider your assessment.
Thank you for your time!
Dear Reviewer h5GB,
Thank you again for your constructive feedback. We would like to briefly follow up on our response. In particular, we have:
-
Clarified why remembering task constraints does not ensure access to previously collected evidence, with a concrete example of evidence lost during summarization.
-
Added component ablations and three-run comparisons using Deepseek-v4.1-flash on 50 tasks per benchmark, with ACG achieving higher mean success than ReAct on all three.
-
Analyzed when ACG helps and where it remains limited, and discussed integration with mature agent frameworks as future work.
We would be happy to clarify any remaining points. If our response and additional evidence have addressed your concerns, we would appreciate it if you could reconsider your assessment.
Thank you for your time!
Response to Reviewer h5GB
Dear Reviewer h5GB, we appreciate your review. Our responses to your questions are as below.
Response to W1
Response to W2
Response to Q1
Response to Q2
We believe ACG has the potential to provide further gains. Its context-organization mechanism is designed to be decoupled from both the base model and the surrounding harness, allowing it to complement mature agents such as Codex. It organizes tool evidence by task requirement, preserves source connections, and supplies relevant material to subsequent decisions. We expect this to be particularly useful in long tasks that repeatedly depend on earlier evidence. This role does not require first distilling the entire pipeline into model weights. We plan to explore integrating ACG into mature agent frameworks such as Codex and compare performance before and after integration using the same model and tasks.