- Rating
- 2
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces MemSteer, a framework that combines state-conditioned memory selection with outcome-validated memory evolution for frozen tool-using agents. A lightweight controller selects natural-language guidance at each reasoning step and learns from grouped rollouts using noise-aware advantages, while a slower loop generates memory edits from clustered failures and evaluates them on held-out tasks. Experiments across nine benchmarks and four frozen solvers assess its effectiveness in visual search, long-document reasoning, and interactive task execution.
Critical strengths
MemSteer introduces a lightweight controller to select natural-language guidance at each reasoning step while keeping the solver frozen, and evaluates the framework across visual, document, and interactive tasks, showing improvements over tool-use and memory-based baselines.
Critical weaknesses
Benchmark-specific adaptation limits evidence for general system improvement. The controller and memory are adapted using examples from the benchmarks on which performance is reported. This setup can produce strategies tailored to those benchmarks and question types, raising concerns about benchmark-level overfitting. The paper does not demonstrate that a single controller–memory system trained on a separate dataset can improve performance across multiple held-out benchmarks without further benchmark-specific adaptation. Consequently, the results do not establish whether MemSteer learns broadly reusable strategies that improve the system’s capabilities beyond the distributions used for adaptation.
Questions for authors
- General capability improvement and cross-benchmark generalization. Can the authors demonstrate that training on a common set of tasks produces a single controller–memory system that improves performance across multiple benchmarks without benchmark-specific adaptation? Additionally, how well does this system transfer to benchmarks excluded from training and validation? These evaluations would help establish whether MemSteer improves the system’s general reasoning capabilities and learns transferable strategies, rather than primarily optimizing guidance for individual benchmarks.
- Independent contribution of memory evolution. Table 5 does not include a trained controller operating on fixed seed memory. Can the authors compare this configuration with the full system while matching training data, controller-training budget, and tool access? This would clarify whether memory evolution provides additional benefits beyond learning to select existing guidance and decide when to stop.
- Relationship to skill libraries. Section 3 defines memory entries through applicability, guidance, success conditions, and recovery, while Figure 7 presents reusable procedural instructions. To what extent should Policy Memory be understood as an evolving natural-language skill or workflow library? Under this interpretation, what is MemSteer’s methodological contribution beyond learning when to select and apply such skills, and why is the additional evolution machinery necessary?
- Memory content versus tool routing. Section 3 couples policy selection with operation families and a separate STOP head, while Figure 6 shows changes in tool use and stopping behavior. Can the authors compare against a similarly trained controller that directly selects tool families or STOP without supplying memory text to the solver? With matched tools and training budgets, this would clarify whether the gains depend on natural-language experience or primarily arise from better action routing and termination.
2 comments
Rebuttal (continued)
Q4 [Memory content versus tool routing]
Each entry couples guidance with a routing family by design, so routing is one channel through which the memory acts, and the generated entries add 2.4 points on HR-Bench 4K with tools unchanged. The natural-language content also acts through the controller, which scores every entry from the embedding of its text (Sec. 3).
Tool access alone does not explain the gain. Random selection and solver self-selection expose the same routing families and tools as MemSteer and reach 51.3 and 66.3 on QuALITY with Qwen3.5-27B, against 73.7 for MemSteer (Tab. 3). On HR-Bench 4K, the controller trained on the seed memory performs on par with tool use without memory, so the gain there requires the generated entries (Table R2 in our response to your Q2).
We additionally keep the trained controller and remove the entry text from the solver prompt, leaving only the identifier and routing family of each selected entry. On the 55 QuALITY test items completed with Qwen3.5-27B before access to the solver became unavailable, this variant answers 37 correctly, the full system 38 and ReAct-style tool use 35 (exact McNemar between the first two), so both variants remain above ReAct-style tool use on these items. The memory in this evaluation is the seed memory, since no update was adopted on QuALITY (Tab. 13). A controller retrained with no entry text has not been evaluated.
Rebuttal
W1 & Q1 [Benchmark-specific adaptation and cross-benchmark generalization]
MemSteer and the experience-based baselines adapt on the same training split and are tested on a disjoint test split, so the comparison is between adaptation methods under one protocol. The claim of the paper is adaptation of a frozen solver to a task distribution, not benchmark-independent capability. We address this from two aspects below:
(1) Adaptation protocol. ACE builds its playbook in one pass over the training tasks, and ExpeL and AWM induce their memory from MemSteer's round-one training rollouts (App. B.2). The image and document benchmarks have 25 to 100 training tasks and 121 to 640 test rows (Tab. 12). The adapted components are a memory of at most 12 entries, from which candidates containing sample identifiers, sample-specific literals or answer letters are excluded (App. A.2), and a 0.86M-parameter controller. The four seed entries are shared by all benchmarks of an environment (Tab. 10).
(2) Transfer to a held-out benchmark. We apply a controller and memory trained on one benchmark, including its adopted updates, to another benchmark of the same modality with no further adaptation and with the tools of ReAct-style tool use on the target. Table R1 below reports the results.
Table R1. Test accuracy (%) of a controller and memory trained on a source benchmark and applied unchanged to a target benchmark, with ReAct-style tool use and in-domain MemSteer on the target.
Both rows use the same items for all three systems. The transferred system matches ReAct-style tool use on V$^\ast$ (exact McNemar p=1.000). On QuALITY it is 18.5 points below ReAct-style tool use (p=2.5×10−8). This pair also changes the tool set, since the LongBench v2 controller was trained with document search that QuALITY does not provide, so the drop is not attributable to the memory alone. In-domain adaptation on QuALITY, with 100 training tasks, reaches 73.49. A third transfer, HR-Bench 4K → MME-RW, reaches 56.8 on a different item set, against 54.2 and 56.1 for ReAct-style tool use and in-domain MemSteer in Tab. 2.
The paper claims adaptation per benchmark, and one controller and memory trained on a pooled multi-benchmark set is beyond the scope of its experiments. We will state this scope and add Table R1 in the final version.
Q2 [Independent contribution of memory evolution]
On HR-Bench 4K, a trained controller on the fixed seed memory reaches 80.1, 2.4 points below the full system. We additionally train the controller for the same rounds on the seed memory with the same training data, rollout budget and tools. Table R2 below compares this variant with the existing rows of Tab. 5.
Table R2. Test accuracy (%) of MemSteer component variants and the no-memory reference on HR-Bench 4K with Qwen3-VL-32B.
On this benchmark, learning to select and stop on the seed memory alone performs on par with tool use without memory, and only the full system exceeds it. Both adopted updates have b>c on validation, at 6/2 and 7/2 (Tab. 13).