acceptodds
Under review as a conference paper at ICLR 2027

MemSteer: Self-Evolving Memory for Long-Horizon Vision and Language Reasoning

Abstract

Self-evolving natural-language memory lets frozen models draw on experience during long-horizon visual and textual reasoning, but its value depends on selecting useful guidance at each step. Given the same memory and the same tools, a frozen solver that keeps every policy in context or picks one for itself falls below tool use without any memory in most of our settings. We propose MemSteer, a framework that couples state-conditioned memory selection with outcome-validated memory evolution across high-resolution images, long documents, and interactive environments. A lightweight controller selects at every decision the natural-language policies that the frozen solver acts on, learning from groups of rollouts whose advantages are discounted by how widely identical choices already scatter rewards. On a slower timescale, failed trajectories are clustered, a language model writes one abstract policy per cluster, and the update is evaluated through a paired comparison on unseen tasks. Extensive experiments on nine benchmarks spanning long-horizon visual search, long-document reasoning, and interactive task execution with four frozen solvers show MemSteer improves average performance over strong tool-use and experience-based baselines.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Large Language Model Agents, Self-Evolving Agents, Agent Memory, Tool Use, Long-Horizon Reasoning

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_boQ2OpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces MemSteer, a framework that combines state-conditioned memory selection with outcome-validated memory evolution for frozen tool-using agents. A lightweight controller selects natural-language guidance at each reasoning step and learns from grouped rollouts using noise-aware advantages, while a slower loop generates memory edits from clustered failures and evaluates them on held-out tasks. Experiments across nine benchmarks and four frozen solvers assess its effectiveness in visual search, long-document reasoning, and interactive task execution.

Critical strengths

MemSteer introduces a lightweight controller to select natural-language guidance at each reasoning step while keeping the solver frozen, and evaluates the framework across visual, document, and interactive tasks, showing improvements over tool-use and memory-based baselines.

Critical weaknesses

Benchmark-specific adaptation limits evidence for general system improvement. The controller and memory are adapted using examples from the benchmarks on which performance is reported. This setup can produce strategies tailored to those benchmarks and question types, raising concerns about benchmark-level overfitting. The paper does not demonstrate that a single controller–memory system trained on a separate dataset can improve performance across multiple held-out benchmarks without further benchmark-specific adaptation. Consequently, the results do not establish whether MemSteer learns broadly reusable strategies that improve the system’s capabilities beyond the distributions used for adaptation.

Questions for authors
  1. General capability improvement and cross-benchmark generalization. Can the authors demonstrate that training on a common set of tasks produces a single controller–memory system that improves performance across multiple benchmarks without benchmark-specific adaptation? Additionally, how well does this system transfer to benchmarks excluded from training and validation? These evaluations would help establish whether MemSteer improves the system’s general reasoning capabilities and learns transferable strategies, rather than primarily optimizing guidance for individual benchmarks.
  2. Independent contribution of memory evolution. Table 5 does not include a trained controller operating on fixed seed memory. Can the authors compare this configuration with the full system while matching training data, controller-training budget, and tool access? This would clarify whether memory evolution provides additional benefits beyond learning to select existing guidance and decide when to stop.
  3. Relationship to skill libraries. Section 3 defines memory entries through applicability, guidance, success conditions, and recovery, while Figure 7 presents reusable procedural instructions. To what extent should Policy Memory be understood as an evolving natural-language skill or workflow library? Under this interpretation, what is MemSteer’s methodological contribution beyond learning when to select and apply such skills, and why is the additional evolution machinery necessary?
  4. Memory content versus tool routing. Section 3 couples policy selection with operation families and a separate STOP head, while Figure 6 shows changes in tool use and stopping behavior. Can the authors compare against a similarly trained controller that directly selects tool families or STOP without supplying memory text to the solver? With matched tools and training budgets, this would clarify whether the gains depend on natural-language experience or primarily arise from better action routing and termination.
2 comments

W1 & Q1 [Benchmark-specific adaptation and cross-benchmark generalization]

MemSteer and the experience-based baselines adapt on the same training split and are tested on a disjoint test split, so the comparison is between adaptation methods under one protocol. The claim of the paper is adaptation of a frozen solver to a task distribution, not benchmark-independent capability. We address this from two aspects below:

(1) Adaptation protocol. ACE builds its playbook in one pass over the training tasks, and ExpeL and AWM induce their memory from MemSteer's round-one training rollouts (App. B.2). The image and document benchmarks have 25 to 100 training tasks and 121 to 640 test rows (Tab. 12). The adapted components are a memory of at most 12 entries, from which candidates containing sample identifiers, sample-specific literals or answer letters are excluded (App. A.2), and a 0.86M-parameter controller. The four seed entries are shared by all benchmarks of an environment (Tab. 10).

(2) Transfer to a held-out benchmark. We apply a controller and memory trained on one benchmark, including its adopted updates, to another benchmark of the same modality with no further adaptation and with the tools of ReAct-style tool use on the target. Table R1 below reports the results.

Table R1. Test accuracy (%) of a controller and memory trained on a source benchmark and applied unchanged to a target benchmark, with ReAct-style tool use and in-domain MemSteer on the target.

SolverSource → TargetItemsReAct-style toolsMemSteer in-domainMemSteer transferred
Qwen3-VL-32BTreeBench → V$^\ast$12089.1790.0089.17
Qwen3.5-27BLongBench v2 → QuALITY29869.1373.4950.67

Both rows use the same items for all three systems. The transferred system matches ReAct-style tool use on V$^\ast$ (exact McNemar p=1.000p=1.000). On QuALITY it is 18.5 points below ReAct-style tool use (p=2.5×10−8p=2.5\times10^{-8}). This pair also changes the tool set, since the LongBench v2 controller was trained with document search that QuALITY does not provide, so the drop is not attributable to the memory alone. In-domain adaptation on QuALITY, with 100 training tasks, reaches 73.49. A third transfer, HR-Bench 4K → MME-RW, reaches 56.8 on a different item set, against 54.2 and 56.1 for ReAct-style tool use and in-domain MemSteer in Tab. 2.

The paper claims adaptation per benchmark, and one controller and memory trained on a pooled multi-benchmark set is beyond the scope of its experiments. We will state this scope and add Table R1 in the final version.


Q2 [Independent contribution of memory evolution]

On HR-Bench 4K, a trained controller on the fixed seed memory reaches 80.1, 2.4 points below the full system. We additionally train the controller for the same rounds on the seed memory with the same training data, rollout budget and tools. Table R2 below compares this variant with the existing rows of Tab. 5.

Table R2. Test accuracy (%) of MemSteer component variants and the no-memory reference on HR-Bench 4K with Qwen3-VL-32B.

VariantMemoryControllerAccuracy
MemSteer (full)evolvedtrained82.5
w/o memory evolutionseedtrained80.1
w/o controller trainingevolvedwarm start79.4
ReAct-style toolsnonenone80.2

On this benchmark, learning to select and stop on the seed memory alone performs on par with tool use without memory, and only the full system exceeds it. Both adopted updates have b>cb>c on validation, at 6/2 and 7/2 (Tab. 13).

AuthorsOpenReview ↗

Rebuttal (continued)

Q4 [Memory content versus tool routing]

Each entry couples guidance with a routing family by design, so routing is one channel through which the memory acts, and the generated entries add 2.4 points on HR-Bench 4K with tools unchanged. The natural-language content also acts through the controller, which scores every entry from the embedding of its text (Sec. 3).

Tool access alone does not explain the gain. Random selection and solver self-selection expose the same routing families and tools as MemSteer and reach 51.3 and 66.3 on QuALITY with Qwen3.5-27B, against 73.7 for MemSteer (Tab. 3). On HR-Bench 4K, the controller trained on the seed memory performs on par with tool use without memory, so the gain there requires the generated entries (Table R2 in our response to your Q2).

We additionally keep the trained controller and remove the entry text from the solver prompt, leaving only the identifier and routing family of each selected entry. On the 55 QuALITY test items completed with Qwen3.5-27B before access to the solver became unavailable, this variant answers 37 correctly, the full system 38 and ReAct-style tool use 35 (exact McNemar p=1.000p=1.000 between the first two), so both variants remain above ReAct-style tool use on these items. The memory in this evaluation is the seed memory, since no update was adopted on QuALITY (Tab. 13). A controller retrained with no entry text has not been evaluated.

Reviewer_T5TCOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper proposes MemSteer, a framework for improving a frozen tool-using solver through an external natural-language policy memory and a trainable 0.86M-parameter controller. The proposed set-scoring controller uses noise-aware and group-relative advantages to learn which memory entries are useful at each decision, together with a validation gate that turns recurring failures into candidate entries and retains only updates that improve performance on unseen tasks. The evaluation covers four image benchmarks, four document benchmarks, and ALFWorld. Most experiments across most benchmarks show consistent gains over strong baselines, while component ablations confirm the value of learned policy selection and memory evolution.

Critical strengths
  1. A useful decomposition for frozen agents. MemSteer separates state-conditioned policy selection during each reasoning step from slower, outcome-validated memory evolution, allowing the frozen solver to benefit from experience without changing its parameters.

  2. MemSteer turns recurring failure patterns into candidate policies, filters sample-specific content and near-duplicates, then uses paired comparisons on held-out tasks to decide whether to retain each update. This provides a concrete safeguard against accumulating plausible-sounding but ineffective guidance.

Critical weaknesses
  1. The noise-aware update does not establish step-level credit assignment. Eq. 6 assigns one terminal-outcome advantage to every decision in a rollout, so it cannot isolate which memory choice helped. The variance contrast between identical- and mixed-signature groups also does not cleanly separate solver randomness from policy effects; estimates from 2–6 binary-outcome rollouts may be unstable, and the factor is set to 1 when no single-signature group exists. Table 6 reports contrast rates but does not validate the estimator against a controlled policy effect.

  2. The validation protocol and test-set use need clarification. Table 13 includes adopted updates with b < c, although Eq. 8 would reject them; the paper does not identify which rows use the alternative harm screen. The gate evaluates edits with the cloned warm-start controller π_bc, not clearly with the learned controller paired with the final memory. In addition, Table 8 reports test accuracy for hyperparameter sweeps, so it is unclear whether test results influenced parameter selection. Main comparisons lack reported paired-test results or confidence intervals, and API failure counts are omitted.

  3. Cross-modal transfer and end-to-end efficiency remain unsubstantiated. Each modality uses separately authored seed policies and maps shared routing families to different tools; no experiment transfers learned policies across modalities without target-specific authoring. Some document comparisons also use extra search access, complicating attribution of gains. The reported 34% token reduction covers selected settings, but total training and validation calls, monetary cost, and end-to-end latency are not reported.

  4. Writting in Method section needs to improve. It uses many largely independent symbols to describe the workflow without clearly explaining how the stages connect. The authors should reorganize the section around the workflow, with notation consolidated and introduced where it is used.

Questions for authors
  1. What exact likelihood is used for the selected action in Eq. 7 after top-n truncation, forced nonempty selection, and separate STOP sampling?

  2. Can you validate the noise attenuation factor in a controlled experiment that independently varies solver randomness and controller choices?

  3. Can you provide a cross experiment in which a policy memory learned in one environment or modality is reused in another with no hand-authored target-domain seed policies, and report results under matched tool access?

2 comments

W1 [Credit assignment in Eq. 6 and the noise attenuation estimator]

Eq. 6 is a rollout-level advantage by design, and the paper does not claim credit assignment to a single decision. We address this from two aspects below:

(1) Scope of the credit-assignment claim. Sec. 1 states that a terminal outcome "cannot be attributed to a single reasoning step or memory entry", and Sec. 3 states that the advantage is "shared across the decisions of that rollout". The phrase "step-level utility" in the contribution list refers to the controller output, a score for each entry conditioned on the state sts_t. We will rename it "state-conditioned selection scores" in the final version. The shared advantage updates a policy that is a function of sts_t, as in episodic policy-gradient training. The learned selections depend on the state: on image tasks, stopping accounts for 16 to 19% of decisions at low solver confidence and 66 to 72% at high confidence (Fig. 3). On HR-Bench 4K, controller training raises accuracy from 79.4 to 82.5 with the same memory (Tab. 5).

(2) The noise attenuation estimator. The factor is a round-level scaling of the advantage and not a per-task variance decomposition. It assumes that the outcome variance measured under identical selections also holds for groups with several signatures. vnoisev_{\mathrm{noise}} and vmixv_{\mathrm{mix}} are means over all eligible groups of a round, with 25 to 100 training tasks per round (Tab. 12), so the factor is not estimated from one group of 2 to 6 rollouts. It scales every advantage of a round by one value in [0,1][0,1], so an estimation error changes the strength of the update and never reverses the sign or the ordering of the advantages. When vmix>0v_{\mathrm{mix}}>0 and no single-signature group exists, the noise term cannot be estimated, and αnoise=1\alpha_{\mathrm{noise}}=1 keeps the leave-one-out advantage unattenuated.


W2 [Validation protocol and test-set use]

The adopted rows with b<cb<c follow the harm screen stated in Sec. 3 and App. D.1, and within each training run the gate and the choice of the final controller use the validation split. We address this from four aspects below:

(1) Adopted rows with b<cb<c. The screen rejects a candidate only when c−b≥max⁡(3,⌈0.075Nval⌉)c-b\ge\max(3,\lceil0.075N_{\mathrm{val}}\rceil) and pharm<0.05p_{\mathrm{harm}}<0.05. The five adopted rows with b<cb<c in Tab. 13 have pharm≥0.113p_{\mathrm{harm}}\ge0.113, and the one candidate that met both conditions (pharm=0.011p_{\mathrm{harm}}=0.011) was rejected. The adoptions with b≤cb\le c occur for Qwen3-VL-32B on MME-RW and V$^\ast$ and for Mistral-Large-2512 on LongBench v2 and QuALITY. Every other adoption has b>cb>c and satisfies Eq. 8. We will indicate the rule applied to each row of Tab. 13 in the final version.

(2) Gate controller. The gate evaluates M\mathcal{M} and M′\mathcal{M}' on the validation split with identical fixed controller parameters, which separates the effect of the memory edit from controller adaptation. The learned controllers are paired with the final memory in the last step of Alg. 1, which selects the controller most accurate on validation with that memory among the warm start and all rounds.

(3) Test accuracy in Tab. 8. This table is a sensitivity analysis and not a selection procedure. The default values of kk, nn and λc\lambda_c are shared by all settings (Tab. 11), and the rollout scheme is specified per solver (Tab. 6). The sensitivity experiments were conducted on two settings after the default configuration had been evaluated, each varying one hyperparameter with all other configuration, data splits and prompts held at the default. We will state this in the final version.

(4) Paired tests, intervals, and API failures. Table R1 below reports the test coverage of MemSteer and the paired comparison for the Qwen3-VL-32B image settings.

Table R1. Test coverage of MemSteer and accuracy (%) of MemSteer and ReAct-style tool use on the items completed by both methods, for Qwen3-VL-32B.

BenchmarkPlannedCompletedFailedPaired itemsMemSteerReAct-style tools
HR-Bench 4K6406231762382.5080.10
MME-RW4003851538556.1055.58
V$^\ast$121120112090.0089.17

The 33 failed episodes are 2.8% of the 1,161 planned in these three settings. The exact McNemar p-values are 0.896 on MME-RW and 1.000 on V$^\ast$, and those of the document settings are in (2) of our response to your W3 & Q3. We will add paired tests and confidence intervals to the main comparisons, grouped by source question on HR-Bench 4K, in the final version.

AuthorsOpenReview ↗

Rebuttal (continued)

W3 & Q3 [Cross-modal transfer, search access, and end-to-end cost]

The paper claims one framework applied to each environment, with the same controller architecture, training objective and validation gate, not transfer of learned memory across environments. We address this from three aspects below:

(1) Transfer within a modality. We apply a controller and memory trained on one benchmark to another benchmark of the same modality without further adaptation, with the tools of ReAct-style tool use on the target. No entry is authored for the target, since seed entries are shared within an environment (Tab. 10). Table R2 below reports the results.

Table R2. Test accuracy (%) on the target benchmark of ReAct-style tool use, in-domain MemSteer, and MemSteer transferred unchanged from the source benchmark.

SolverSource → TargetItemsReActIn-domainTransferred
Qwen3-VL-32BTreeBench → V$^\ast$12089.1790.0089.17
Qwen3.5-27BLongBench v2 → QuALITY29869.1373.4950.67

Both rows use the same items for all three systems. The transferred system matches ReAct-style tool use on V$^\ast$. On QuALITY it is 18.5 points below ReAct-style tool use (exact McNemar p=2.5×10−8p=2.5\times10^{-8}). This pair also changes the tool set, since the LongBench v2 controller was trained with document search that QuALITY does not provide, so the drop is not attributable to the memory alone.

(2) Search access. We evaluate ReAct-style tool use with MemSteer's search tool and budgets on the three Qwen3.5-27B settings that use search. On common items, MemSteer achieves 52.73 against 50.78 on LongBench v2 (p=0.603p=0.603), 81.82 against 83.12 on ∞Bench (p=0.791p=0.791) and 79.00 against 64.67 on AmongUs-X (p=1.7×10−7p=1.7\times10^{-7}). The advantage on AmongUs-X remains, and the other two differences are not statistically distinguishable. We will report this comparison in place of the one without search in the final version.

(3) Cost. The 34% token reduction compares the warm start with the controller after the third round on the validation tasks of seven settings without document search (Fig. 5). Complete token, monetary and latency totals for training and validation were not recorded. Training comprises three rounds of at most six rollouts (Tab. 11) on 25 to 100 tasks per setting (Tab. 12), and at inference each decision adds one state embedding and one forward pass of the 0.86M-parameter controller.


W4 [Writing of the method section]

At inference, the controller selects entries for the next action of the frozen solver from the current state. In controller optimization, grouped rollouts of one task provide the terminal returns that update the controller. In policy evolution, failed rollouts yield candidate memory edits, which the gate compares with the current memory on validation tasks before the next round. We will reorganize the method section around these three stages, add a notation table, and introduce each symbol at first use in the final version.


Q1 [Exact likelihood in Eq. 7]

The likelihood is that of the sampling process in Sec. 3, a Bernoulli STOP decision followed by independent Bernoulli draws over the candidate entries. If STOP is sampled, log⁡πθ(At∣st,M)=log⁡σ(ℓSTOP)\log\pi_\theta(A_t\mid s_t,\mathcal{M})=\log\sigma(\ell_{\text{STOP}}). Otherwise, with AtA_t the selected set and Ct\mathcal{C}_t the candidate set,

log⁡πθ(At∣st,M)=log⁡σ(−ℓSTOP)+∑m∈Atlog⁡σ(ℓm)+∑m∈Ct∖Atlog⁡σ(−ℓm).\log\pi_\theta(A_t\mid s_t,\mathcal{M})=\log\sigma(-\ell_{\text{STOP}})+\sum_{m\in A_t}\log\sigma(\ell_m)+\sum_{m\in\mathcal{C}_t\setminus A_t}\log\sigma(-\ell_m).

Truncation to the nn most probable entries and the non-empty fallback are deterministic maps applied after sampling (App. A.2) and are not part of the likelihood. The expression is therefore the factorized Bernoulli likelihood of the sampled decisions and not the marginal probability of the executed set, which would sum over all draws mapped to that set. ρt\rho_t uses the same expression. We will state this under Eq. 7 in the final version.


Q2 [Controlled validation of the noise attenuation factor]

In Tab. 6, the minimum of the factor across rounds, the second value per solver, is 1.00, 0.81, 0.47 and 0.00 where 0.0%, 0.3%, 1.4% and 2.5% of rollout groups combine identical selections with different outcomes. The factor is 1 where no such group is observed and lowest where the share is highest. All solvers decode at temperature 0 (Sec. 4), so solver randomness is measured and not controlled, and this evidence is observational.

Reviewer_bBDDOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes MemSteer, a framework for improving frozen tool-using models through self-evolving natural-language memory. It combines a lightweight, state-conditioned controller for selecting relevant memory policies with an outcome validated mechanism for evolving those policies. The controller is trained using noise-aware group-relative advantages, while memory updates are generated from clustered failures and evaluated on held out tasks. Experiments across nine benchmarks and four frozen solvers demonstrate improvements over several tool-use and memory based baselines.

Critical strengths
  1. Well-motivated and practical framework. The paper addresses an important limitation of existing memory-based agents: the utility of memory depends on the evolving reasoning state. The separation between step-level memory selection and slower memory evolution is well motivated. The lightweight controller also makes the approach practical for frozen models.
  2. Broad experimental evaluation. The paper evaluates multiple solvers across visual, document, and interactive tasks, with relevant baselines and ablations. The improvements on Mistral document benchmarks and the matched-tool QuALITY setting are encouraging. The reported reductions in tool calls and token consumption further support the practical value of learned memory selection.
Critical weaknesses
  1. Potentially unfair comparisons due to different tool access. In the Qwen3.5 document experiments, MemSteer receives additional document-search capabilities and budgets that are unavailable to several baselines. Table 7 shows substantial improvements from enabling search alone, suggesting that part of the reported gains may come from stronger tool access. Although the authors acknowledge this limitation, comparisons under fully matched tool and computation budgets are needed to establish the contribution of memory selection.
  2. Insufficient isolation of the key mechanisms. The ablations do not separately quantify the contributions of noise-aware advantage weighting and outcome-validated memory evolution. For example, Table 6 reports a noise-attenuation factor of 1.0 for Qwen3.5, raising questions about the practical impact of this component. Furthermore, Table 13 shows that some memory edits are accepted despite more validation losses than wins. Stronger ablations and statistical evidence would help substantiate the proposed mechanisms, especially given the relatively modest improvements in some settings.
Questions for authors
  1. Can the authors provide comparisons where all document baselines use the same search tools and inference budgets as MemSteer? How much of the reported improvement remains?
  2. Can the authors separately ablate the noise-aware weighting and memory-evolution components while keeping the controller and tool configurations fixed?
  3. Can the authors provide statistical significance results or multiple-seed evaluations for the main improvements? In particular, how do memory updates accepted despite negative paired validation outcomes perform on an independent test set?
2 comments

W1 & Q1 [Tool access on documents and comparison under identical search]

With identical search access, MemSteer exceeds ReAct-style tool use by 14.3 points on AmongUs-X (p=1.7×10−7p=1.7\times10^{-7}) and is statistically indistinguishable from it on LongBench v2 and ∞Bench. We evaluate ReAct-style tool use, which ExpeL and AWM also adopt for acting, with the same document search tool and the same search and token budgets as MemSteer (Tab. 11) on the three Qwen3.5-27B settings that use search. Table R1 below reports the paired comparison on the test items completed by both methods.

Table R1. Accuracy (%) of ReAct-style tool use and MemSteer with identical document search on Qwen3.5-27B, with two-sided exact McNemar p-values.

BenchmarkPaired itemsReAct-style tools + searchMemSteerΔ (points)p
LongBench v225650.7852.73+1.950.603
∞Bench15483.1281.82−1.300.791
AmongUs-X30064.6779.00+14.331.7×10−71.7\times10^{-7}

MemSteer also outperforms ReAct-style tool use in the settings whose tools already match in the paper. QuALITY with Qwen3.5-27B uses no document search, and MemSteer reaches 73.7 against 69.0 for ReAct-style tool use (Tab. 3). On the four image benchmarks with Qwen3-VL-32B, MemSteer shares tools, episode limits and answer prompt with ReAct-style tool use (Sec. 4) and averages 70.2 against 68.9 (Tab. 2).

We will report the comparison of Table R1 in place of the comparison without search for these three settings in the final version.


W2 & Q2 [Separate ablations of memory evolution and noise-aware weighting]

Memory evolution contributes 2.4 points on HR-Bench 4K under the same controller-training protocol and tools, and the noise attenuation factor is inactive for Qwen3.5-27B, where no solver noise is observed. We address this from two aspects below:

(1) Memory evolution. We additionally train the controller for the same rounds on the fixed seed memory, with the same training data, rollout budget and tools. Table R2 below compares this variant with the existing rows of Tab. 5 for HR-Bench 4K with Qwen3-VL-32B.

Table R2. Test accuracy (%) of MemSteer component variants and the no-memory reference on HR-Bench 4K with Qwen3-VL-32B.

VariantMemoryControllerAccuracy
MemSteer (full)evolvedtrained82.5
w/o memory evolutionseedtrained80.1
w/o controller trainingevolvedwarm start79.4
ReAct-style toolsnonenone80.2

On this benchmark, both reduced variants perform on par with tool use without memory, and only the full system exceeds it. Both adopted updates have b>cb>c on validation, at 6/2 and 7/2 (Tab. 13).

(2) Noise-aware weighting. The Outcome column of Tab. 6 is the share of rollout groups with identical selections and different outcomes, the groups that make vnoisev_{\mathrm{noise}} positive in Eq. 5. This share is 0.0% for Qwen3.5-27B, decoded at temperature 0 (Sec. 4), and the factor equals 1.00 in every round. Eq. 6 therefore reduces to the leave-one-out advantage, and the document results reported for this solver coincide with training at αnoise≡1\alpha_{\mathrm{noise}}\equiv1. The two values per solver in Tab. 6 are the mean and the minimum of the factor across rounds, and the minimum follows the observed noise across the four solvers, at 1.00, 0.81, 0.47 and 0.00 for shares of 0.0%, 0.3%, 1.4% and 2.5%. A variant retrained without the factor on the solvers with observed noise is not included in the paper. On those solvers the factor scales every advantage of a round by one value in [0,1][0,1], so it changes the strength of an update and never reverses the sign or the ordering of the advantages.

AuthorsOpenReview ↗

Rebuttal (continued)

Q3 [Statistical significance, seeds, and test performance of updates adopted with b<cb<c]

Under paired item-level tests, the gain on AmongUs-X under identical search is significant (p<10−6p<10^{-6}). We address this from two aspects below:

(1) Significance. The paper specifies exact McNemar tests for paired comparisons (Sec. 4) without reporting their values. Table R1 in our response to your W1 & Q1 reports them for the Qwen3.5-27B document settings, and Table R3 below reports the paired comparison for the Qwen3-VL-32B image settings.

Table R3. Accuracy (%) of MemSteer and ReAct-style tool use with Qwen3-VL-32B on the test items completed by both methods.

BenchmarkPaired itemsMemSteerReAct-style toolsΔ (points)
HR-Bench 4K62382.5080.10+2.41
MME-RW38556.1055.58+0.52
V$^\ast$12090.0089.17+0.83

The differences are positive on all three benchmarks and largest on HR-Bench 4K, the largest test set. The two-sided exact McNemar p-values against ReAct-style tool use are 0.896 on MME-RW and 1.000 on V$^\ast$, where test sets of 385 and 120 items provide limited statistical power for differences below one point. The reported results are not averaged over seeds, and all solvers decode at temperature 0. We will add the paired tests to the main comparisons, grouped by source question on HR-Bench 4K, in the final version.

(2) Updates adopted with b<cb<c. The five such rows of Tab. 13 follow the one-sided harm screen stated in Sec. 3 and App. D.1, not Eq. 8. With 30 to 60 validation tasks, the screen rejects a candidate only for a statistically supported loss, namely c−b≥max⁡(3,⌈0.075Nval⌉)c-b\ge\max(3,\lceil0.075N_{\mathrm{val}}\rceil) and pharm<0.05p_{\mathrm{harm}}<0.05. They have pharm≥0.113p_{\mathrm{harm}}\ge0.113, and the one candidate that met both conditions (Mistral-L / LongBench v2, round 1, pharm=0.011p_{\mathrm{harm}}=0.011) was rejected. We additionally evaluate the three such updates of Qwen3-VL-32B on test, with the final controller and the memory before and after each update, on common completed items. None of them shows a significant test loss. The first MME-RW update leaves accuracy at 53.65 on 384 items (exact McNemar p=1.000p=1.000). The second raises it from 54.13 to 56.00 on 375 items, with the two memories evaluated through different API providers. The first V$^\ast$ update gives 89.08 before and 86.55 after on 119 items (p=0.549p=0.549). The final systems that contain these updates achieve 56.10 and 90.00 against 55.58 and 89.17 for ReAct-style tool use. We will indicate the rule applied to each row of Tab. 13 in the final version.

Comments on the paper

1 comment
Area_Chair_SygrOpenReview ↗

Reminder: Please Follow Up on the Author Rebuttal

Dear Reviewers,

As we are approaching the end of the discussion period, I would like to remind you to review the authors' rebuttal and assess whether your concerns have been adequately addressed.

If you have any remaining concerns, please raise them as soon as possible to allow the authors sufficient time to respond. If your concerns have been resolved, please leave a brief acknowledgment to let the authors know that you have read and considered their responses.

Thank you for your time and efforts in the review process.

Best regards, Area Chair

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.