acceptodds
Withdrawn from ICLR 2027

MuSEAgent: A Multimodal Reasoning Agent with Stateful Experiences

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Multimodal agents have shown growing ability to interleave tool use with multimodal observations for complex visual reasoning, yet effectively reusing experiences from past multimodal interactions remains challenging. Existing memory paradigms struggle in complex multimodal environments: coarse trajectories flood the context window with visual redundancy while obscuring fine-grained errors at intermediate steps; ungrounded scalar feedback offers no procedural guidance for visual tool execution; and holistic representations of multimodal states conflate visual appearance with tool histories, misguiding experience retrieval. To resolve these dilemmas, we introduce MuSEAgent, a multimodal reasoning agent with stateful experiences that reframes multimodal experience reuse across three foundational dimensions: (1) granularity, decomposing multimodal interaction trajectories into atomic state-action transitions anchored to intermediate decision states; (2) supervision, synthesizing outcome-conditioned natural-language visual tactics via hindsight reasoning to provide explicit procedural rules; and (3) representation, organizing heterogeneous multimodal states into a compositional multi-view space, queried via an iterative Deep-and-Wide search to dynamically align visual observations, task semantics, and tool execution histories. Extensive evaluations across four demanding multimodal benchmarks demonstrate that MuSEAgent consistently elevates diverse base models across parameter scales, surpassing trajectory-level baselines by up to 7.97 points and outperforming state-level scalar-value alternatives by 10.42 to 12.00 points. Remarkably, stateful experiences transfer robustly to unseen benchmarks without in-domain interaction, outperforming matched trajectory-level transfer by 7.8 points.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Multimodal Agent, Stateful experience learning, Fine-grained visual reasoning

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_d1t3OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces MuSEAgent, a framework for improving multimodal tool-using agents through reusable, state-conditioned experiences. Offline trajectories are decomposed into atomic transitions, and a hindsight model produces a score and concise natural-language guidance for each transition. The framework filters experiences using graph-smoothed scores and indexes each state through 15 combinations of its instruction, visual input, task descriptor, and execution history. At inference time, Deep-and-Wide retrieval aggregates guidance across multiple queries before generating the next action.

Experiments on four high-resolution visual question-answering benchmarks show improvements over ReAct, Reflexion, ExpeL, and Memento across three Qwen-based agents. The paper also investigates cross-dataset transfer, experience sources, and alternative hindsight models.

Critical strengths
  1. The problem and motivation are clear, and the novelty is reasonable. Experience itself does present the problem of being long and containing useless information. Atom transition can actually make it more concise and more useful for generating high-quality experience.

  2. The reported experiment results show improvement across models and datasets.

Critical weaknesses
  1. The experiments are mostly done on Qwen series models. And more importantly, I could not understand why the authors mainly do experiments on multiple-choice benchmarks. This should do experiments on more general tasks, not multiple choice such easy tasks.

  2. The claimed iterative refinement is not established by the algorithm or the comparisons. In Algorithm 2, every viewpoint is sampled conditional on the same state. Previously retrieved guidance does not explicitly enter subsequent viewpoint selection, and the aggregated guidance is consumed only after the retrieval loop.

Questions for authors
  1. Can you do more experiments on general evaluation benchmark and other series models?

  2. Does the score represent action quality, error responsibility, or the expected usefulness of the resulting guidance? How are successful and failed transitions made comparable before smoothing and filtering?

  3. Does each retrieval round observe the results of previous rounds? Please provide an execution trace showing viewpoint selection, retrieval, and action generation, including model-call counts.

Reviewer_KKmpOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper targets experience reuse in multimodal reasoning agents. The authors argue that existing memory paradigms fail along three axes: granularity, supervision, and representation, and propose MuSEAgent, which decomposes multimodal trajectories into atomic state-action transitions, supervises each transition with outcome-conditioned natural-language guidance produced by hindsight reasoning, and indexes states across 15 compositional viewpoints queried by an iterative Deep-and-Wide search. Experiments on four multiple-choice VQA benchmarks with three base-model scales report consistent gains over trajectory-level baselines and state-level scalar-value memory.

Critical strengths
  • The problem is timely, and the three bottlenecks the authors identify are real and clearly articulated.
  • The empirical execution is strong: baselines are re-implemented under matched retrieval infrastructure, results are reported across model scales with multi-seed confidence intervals, and the authors include a human verification study, a web-search shortcut audit, and a qualitative failure analysis.
Critical weaknesses
  • The contribution is incremental. The three dimensions map one-to-one onto the three baselines the paper positions against. Describing this as a "principled paradigm shift along three foundational dimensions" overstates it: the axes are defined after the fact by what the method happens to do, and no argument is given for why these three are the right basis for the design space, nor whether other decompositions (e.g., visual grounding of stored experience, retrieval policy, memory lifecycle) would be more appropriate.
  • The evaluation is too narrow and too small to support the generality claims. All four benchmarks are multiple-choice VQA, the three "diverse base models" are three scales of the same Qwen family, and the evaluation sets for the two benchmarks driving the headline gains are very small (V* Bench: 96 questions; HR-Bench: 100).
  • The central claim that all three components are jointly necessary is not supported by the ablation. The reported component gains on V* Bench are +5.21 (multi-view indexing), +10.42 (iterative search), +4.17 (transition granularity), and +6.25 (NL over scalar), yet the total difference between general RAG (68.75) and the full model (85.42) is only 16.67 points. The components therefore interact strongly, and the "successive improvements" framing is misleading. More importantly, the ablation is a set of heterogeneous variants rather than a factorial design, so no interaction term is estimated and no evidence is provided that removing any one axis degrades the others.
  • The strongest baseline appears to be weakened, which undermines the headline comparison. The paper reports that Memento "degenerates" to trajectory-end transitions under sparse rewards and implements it accordingly as Trajectory-end + Scalar value.
Questions for authors
  • The four reported component gains sum to substantially more than the total improvement over general RAG, which indicates strong interaction between the proposed axes. Can you provide a full factorial ablation with interaction effects, and show that these three axes, rather than an alternative decomposition, are the right basis for the design space?
  • Your strongest baseline, Memento, is implemented as trajectory-end transitions with scalar values. Can you demonstrate that this faithfully reproduces the published method's behavior, so that the reported gap reflects a limitation of scalar-value supervision rather than of the re-implementation?
Reviewer_MhVsOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies how multimodal agents that interleave tool use with observations can reuse experiences from past multimodal interactions. MuSEAgent decomposes the exploration trajectories into atomic state-action transitions. A hindsight reasoning model, GPT-4o by default, evaluates each transition conditioned on the trajectory outcome. It outputs a quality score in [0, 10] and a natural language guidance that summarizes the key decision tactic. The scores are propagated over an affinity graph of candidate transitions (Eq. 2), and transitions with a smoothed score of at least 5.0 are retained. Each state has four parts: the user instruction, the current visual input, an optional task descriptor and the execution history. Each experience is embedded under each of the 15 non-empty compositions of these parts, which the paper calls semantic viewpoints. In the Deep-and-Wide search at inference, the agent chooses one viewpoint in each of L rounds and retrieves the top K experiences under it (L = K = 3). Experiments use V* Bench, MME-RealWorld-Lite, Zoom-Bench and HR-Bench, each partitioned into equally sized exploration and evaluation splits. The agent accesses 13 multimodal tools and uses three Qwen base models, and it is compared with five baselines: Vanilla CoT, ReAct, Reflexion, Expel and Memento. MuSEAgent has the highest average accuracy on all three base models (Table 1). Ablations on V* Bench and HR-Bench vary the components, the experience source and the hindsight model (Table 3). In the transfer test across datasets (Table 2), the bank for each benchmark is pooled from the exploration trajectories of the other three benchmarks.

Critical strengths

S1. The paper proposes an experience bank that stores natural language guidance for each transition and indexes it under multiple semantic viewpoints.

Its average accuracy is 65.30, 57.23 and 69.76 on the 32B, 235B and 397B models (Table 1). The strongest baseline on each model, Memento, reaches 58.19, 54.97 and 68.16. MuSEAgent also has the highest score in all 12 combinations of benchmark and model. The baselines include memory at the trajectory level (Reflexion, Expel) and at the state level (Memento), and these memory baselines use the same tools as MuSEAgent.

S2. The ablations separate transition granularity from natural language supervision. The bank also beats ReAct, which uses no experience, on average when GPT-4o is replaced and when no data from the target benchmark is used.

With only the transitions at the trajectory end, natural language guidance beats Memento's scalar values: 81.25 vs. 75.00 on V* Bench and 74.00 vs. 66.00 on HR-Bench (Table 3a). The full model, which uses all transitions, reaches 85.42 and 78.00. With the 32B model as its own hindsight model, MuSEAgent reaches 78.12 and 75.00, above ReAct (70.83, 66.00) and Memento (75.00, 66.00). With a bank built only from the other three benchmarks, the average accuracy is 59.64 (Table 2). Under the same protocol, Reflexion reaches 52.60 and Expel 51.86, and ReAct without any bank reaches 55.75 (Table 1).

Critical weaknesses

W1. In Table 1, MuSEAgent leads the best baseline by more than 10 points only on V* Bench and HR-Bench with the 32B model. The statistical claims also go beyond what is tested.

Location: Abstract L029-L030, L090-L093, L357-L366, Table 1, Table 3.

Quote (L029-L030): "outperforming state-level scalar-value alternatives by 10.42 to 12.00 points" ... (L091): "statistically significant gains".

(a) The 10.42 and 12.00 points in the abstract are the V* Bench and HR-Bench gaps to Memento on the 32B model. The average gap to Memento is 7.11, 2.26 and 1.60 points on the three models (L320-L321). Here the best baseline is the strongest of the five baselines on each benchmark for each model. On the 32B model, MuSEAgent leads it by +10.42 (V* Bench), +1.57 (MME-RealWorld-Lite), +3.21 (Zoom-Bench) and +12.00 (HR-Bench). On the 235B model, the leads are +4.16, +0.31, +1.29 and +2.00. On the 397B model, they are +2.08, +1.04, +1.28 and +2.00. MME-RealWorld-Lite has the largest evaluation set (960 questions), and its leads (1.57, 0.31 and 1.04) are the smallest on every model. The abstract should give the average gaps, or state which model and benchmarks the range comes from.

(b) V* Bench and HR-Bench have only 96 and 100 evaluation questions, so one question changes accuracy by about 1 point. Table 7 repeats MuSEAgent and Expel with three seeds, and the std is 1.41 to 3.07 points. The MuSEAgent-Expel gap in Table 7 (11 to 12 points) is much larger than this, so that comparison holds. The other comparisons report one number per setting, with no std or number of seeds: Table 1 (including Memento), Table 3 and Fig. 2. Multiview indexing adds +5.21 on V* Bench (5 of 96) and +3.00 on HR-Bench (3 of 100). Using all transitions in place of the transitions at the trajectory end adds +4.17 on V* Bench (4 of 96) and +4.00 on HR-Bench (4 of 100). The gaps for natural language guidance over scalar values are larger (+6.25 and +8.00). The gaps of 3 to 5 points are close to the run-to-run std in Table 7. Without repeated runs, they cannot be separated from noise. No significance test is reported. The claim of “statistically significant gains” for each component (L090-L091) is therefore not backed by a test.

W2. Part of the gain may come from using GPT-4o as the hindsight model, and Table 1 does not separate this from the method.

Location: L299, L321-L323, L465-L471, Table 1, Table 3(c).

Quote (L299): "with GPT-4o (Hurst et al., 2024) as the default hindsight reasoning model" ... (L465): "Stateful experience gains hold under self-hindsight, ruling out a distillation confound."

All MuSEAgent rows in Table 1 use GPT-4o to score transitions and write the guidance. The main text does not say which model writes the verbal reflections for Reflexion or the insights for Expel. Replacing GPT-4o with the 32B model itself lowers MuSEAgent from 85.42 to 78.12 on V* Bench and from 78.00 to 75.00 on HR-Bench. When the 32B model is its own hindsight model, the V* Bench margin over Memento falls from 10.42 to 3.12 points, which is 3 questions. The HR-Bench margin falls from 12.00 to 9.00. GPT-4o may still account for much of the headline margin. L321-L323 also attributes the Table 1 gap over Memento to natural language tactics. Memento retains only the transitions at the trajectory end (L295). The full MuSEAgent uses all transitions (Table 3a). The Table 1 gap therefore mixes two factors. Table 3(a) separates these factors only with GPT-4o, one base model and the two smallest evaluation sets. If they do not already, the trajectory level baselines could use the same GPT-4o summarizer to separate this effect.

Questions for authors
  1. Baseline details. L296 points to Appendix D. Appendix D covers Vanilla CoT, ReAct, Reflexion and Expel, and has no section on Memento. How are Memento's scalar values computed, and by which model?

  2. Variance. Please report three or more seeds for Memento, for the Table 3 rows, and for the 235B and 397B models.

  3. Hyperparameter selection. Fig. 2 reports accuracy for different L and K on V* Bench and HR-Bench, and L308-L309 states that accuracy is evaluated on the evaluation splits. The paper reports that accuracy peaks at the defaults L = K = 3. The appendix sweeps δ on the same two benchmarks (Table 6), where HR-Bench ranges from 61.00 to 78.00. Were L, K, δ and α chosen on the evaluation splits? If so, could you report results with these values chosen on a separate part of the exploration set? What value of α is used in Eq. (2), and what happens without smoothing (α = 0)?

  4. Split construction. How are the benchmarks split into exploration and evaluation sets (L286)? Can the same image appear in both splits? The viewpoint that combines the query with the visual observation is selected in 23.97% of retrieval steps (L409-L410). If an image appears in both splits, retrieval under this viewpoint could match it directly.

  5. Meaning of the hindsight score. The main text says qtq_t "reflects estimated decision quality" (L178-L179). For failed trajectories, the prompt in Appendix N.2 defines a high score differently: "A high score means this step was a critical cause of the error." (L2301). Both kinds of scores are then smoothed together on one graph (Eq. 2) and filtered by the same threshold. Are failure scores rescaled before smoothing, or is the graph split by outcome?

  6. Transfer baseline. MuSEAgent's transfer results differ from ReAct by +7.29, -1.46, +6.76 and +3.00 on the four benchmarks. Could you add ReAct and Memento to Table 2, so the margin of 7.8 points in the abstract can be compared with a baseline that uses no experience?

  7. Related work. Please consider citing ICAL (Sarch et al., NeurIPS 2024), Agent Workflow Memory (Wang et al., ICML 2025) and ReasoningBank (Ouyang et al., ICLR 2026).

Minor issues:

  • L208 and L657: "Eq. equation 2" and "Eq. equation 3" should be "Eq. (2)" and "Eq. (3)".
  • The method of Zhao et al. (2024) should be written "ExpeL", as in its title.
  • L577: "V?: Guided visual search" should be "V*: Guided visual search".
  • L546-L547: the MemGPT entry has stray underscores ("Shishir_G Patil", "Joseph_E Gonzalez") and no venue.
  • L593: ReAct appeared at the eleventh ICLR in 2023. The entry gives the year as 2022 and should give 2023.
  • Table 1 reports "Micro Avg." for MME-RealWorld-Lite, and its values match the mean of Percep. and Reas. weighted by question count. L769 describes the score as "average task accuracy". The two descriptions should match.
  • Table 2 names the benchmark "MME-RealWorld" and should be "MME-RealWorld-Lite" as in Table 1.
  • Fig. 1 sends qtq_t straight to the "> δ" filter and does not show the graph smoothing of Eq. (2).
  • The proof of Lemma 3.2 states that S is row stochastic (L653), which requires Wij≥0W_{ij} \ge 0. Please state which similarity is used in W and how negative values are handled.
  • Algorithm 1, line 5 writes Qϕ(st,at,st+1)Q_\phi(s_t, a_t, s_{t+1}). It should be Qϕ(st,at,st+1∣τ,o)Q_\phi(s_t, a_t, s_{t+1} | \tau, o), as in L177-L178.
  • L363-L364 calls the mean ± 1 std ranges of Table 7 "confidence intervals". They should be called "mean ± 1 std".
  • L374 states that "every configuration still beats the experience-free baseline". In Fig. 2(a), V* Bench accuracy at L = 1 is about 69, below ReAct at 70.83. The single-view, single-round row in Table 3(a) is also below ReAct on V* Bench (68.75 vs. 70.83).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.