acceptodds
Under review as a conference paper at ICLR 2027

TraceMem: Trajectory-Induced Reusable Memory for Multimodal Reasoning

Abstract

Multimodal reasoning is a key capability for general-purpose AI systems, requiring models to jointly interpret visual evidence, language instructions, and multi-step inference processes. However, multimodal reasoning models often repeat similar mistakes across related samples, suggesting a limited ability to transform past reasoning outcomes into reusable experience. While memory-based methods preserve prior episodes, they typically store them as weakly structured text, making retrieved memories less actionable. Test-time trajectory exploration can reveal successful and failed reasoning patterns, but these trajectories are usually discarded after the current prediction. We propose TraceMem, a trajectory-induced memory framework that converts reasoning trajectories into structured memory units with explicit triggers, procedural guidance, and avoidance rules. These units are maintained through four lifecycle-aware memory banks, allowing experience from successful and failed trajectories to be accumulated, updated, and reused. Experiments on four multimodal reasoning benchmarks show that TraceMem consistently improves performance, demonstrating the effectiveness of trajectory-induced structured memory for multimodal reasoning. The code is available at https://anonymous.4open.science/r/TraceMem-57C6/.

Primary area:
foundation or frontier models, including LLMs
Keywords:
multimodal reasoning, memory, large language model

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_k3NyOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary
  • TraceMem organizes the successful and failed reasoning trajectories of multi-modal models into problem-solving experiences, divided into four types of memory: visual success, visual failure, logical success and logical failure.
  • Retrieve relevant experience when answering new questions and update memory based on subsequent answer performance. The method does not require modification of model parameters and improves first-time answer accuracy on four multi-modal benchmarks.
Critical strengths
  • The idea of ​​accumulating experience across topics is valuable. Additional reasoning generates information that can help with subsequent tasks, rather than just revising the current answer.
  • Memory management is relatively complete. At the same time, consider successful experiences, failure lessons, applicable conditions and historical effects, which is more targeted than simply saving reasoning texts.
  • There is some evidence of migration. The memory generated by the small model can also help the large model, indicating that the extracted experience is not completely dependent on the model that generated it.
Critical weaknesses
  • The evaluation process is unclear. Validation and memory updates rely on standard answers, but how memory construction is isolated from test data and how it is updated during testing is not adequately explained.

  • The base model is outdated. Experiments only cover Qwen3-VL and GPT-4.1, lacking validation on newer, stronger bases. It's unclear whether the gains will diminish as model capabilities improve.

  • The additional gains for some tasks are small. For example, in the HallusionBench experiment with the 8B model, the improvement compared to ViLoMem is only 0.40 percentage points. The significant improvement compared to direct answers is mainly achieved through step-by-step prompts.

  • Efficiency analysis is lacking. The method incorporates task analysis, multi-track generation, and memory extraction, but complete computational costs, latency, and comparisons under equivalent budgets are not reported.

  • The paragraphs are too fragmented, resulting in a poor reading experience. The methodology section is frequently broken into short paragraphs, with repetitive explanations in adjacent paragraphs, affecting narrative coherence. It is recommended to merge and streamline these sections.

Questions for authors
  • How should the memory construction set and test set be divided? During testing, are standard answers used to update the memory, and how can we ensure that the evaluation results are not affected by information leakage?

  • Could you supplement the experiments on models such as Qwen3.5 and analyze the relationship between method benefits and foundation capabilities?

  • What is the total cost after accounting for memory construction and maintenance? Under the same computational budget, does it still have an advantage compared to multiple sampling or reflection methods?

  • If there is no standard answer and we can only rely on validators with errors, will erroneous experiences accumulate and affect subsequent answers?

Reviewer_FatdOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces TraceMem, a reusable memory framework for multimodal reasoning that is driven by reasoning traces. The approach aims to enhance reasoning performance by transforming successful and unsuccessful reasoning traces into structured, reusable memory units. Each memory unit comprises trigger conditions, procedural guidance, avoidance rules, task labels, utility scores, and a lifecycle state. The framework organizes these memories into four repositories based on modality and reasoning results. During the reasoning process, the system retrieves task-related memories to guide the model's initial predictions; if the verification fails, it generates and analyzes additional reasoning trajectories to extract new memories, while continuously updating them based on the actual utility of existing memories. Experimental results based on the Qwen3-VL and GPT-4.1 backbone models in four multimodal benchmarks demonstrate that this method achieves continuous performance improvements compared to direct prompting and existing memory-based baseline models.

Critical strengths

S1. A practical and rigorously structured framework. The proposed framework offers a systematic approach for accumulating and reusing reasoning experience. By integrating structured memory, trajectory-based induction, and progressive memory updates, it presents a promising direction for research in multimodal reasoning.

S2. Comprehensive experimental evaluation. The paper conducts extensive experiments across multiple multimodal reasoning benchmarks and various backbone models. The results consistently demonstrate performance improvements over existing baseline models.

Critical weaknesses

W1. Insufficient justification for the proposed memory mechanism. Although the framework introduces structured memory and a lifecycle-based update mechanism, it is not yet clear whether its underlying mechanism is sufficient to support the "memory reliability" and "conditional activation" capabilities claimed in the paper.

W2. Lack of clarity regarding the transition from the proposed framework to practical deployment. The process of memory induction and updating seems to depend on verifying the correctness of the results; this raises questions about the applicability of this method in real-world scenarios where ground-truth answers are lacking.

Questions for authors

Thank you for your work. I acknowledge the importance of the issues explored in this paper and appreciate the comprehensive experimental evaluation provided. However, I have some reservations regarding whether the proposed mechanisms sufficiently support the paper's core claims, as well as the framework's suitability for practical deployment scenarios. I would be happy to reconsider my rating if the following questions can be addressed satisfactorily.

1. Theoretical basis for memory utility and lifecycle management. The paper claims that lifecycle-aware memory updates enable the identification and prioritization of reliable memories. However, the proposed utility function assigns identical episode-level feedback to all retrieved memories, regardless of their individual contributions. Consequently, a memory may receive positive feedback simply because the model produces a correct answer, even if that memory provides no actual benefit. Could the authors clarify how this utility formulation supports the claims regarding memory reliability and progressive memory refinement? Furthermore, how can the resulting lifecycle states be shown to reflect genuine memory effectiveness rather than merely correlations between memory retrieval and successful predictions?

2. Alignment between structured memory representations and the actual retrieval process. The paper highlights "trigger-conditioned" and "executable" memories as core components of TraceMem. However, the retrieval mechanism appears to rely primarily on semantic similarity and historical utility, without clearly specifying how these trigger conditions are evaluated or enforced during inference. Could the authors clarify how these conditions influence memory activation during the reasoning process? Specifically, what fundamentally distinguishes the proposed trigger-conditioned memory from conventional similarity-based retrieval followed by prompt-level guidance?

3. Applicability of the proposed framework to real-world deployment scenarios. The trajectory induction and memory evolution processes rely on correctness verification, which, according to the implementation details, appears to involve reference-based evaluation. However, ground-truth feedback is typically unavailable in real-world multimodal reasoning scenarios. How does TraceMem operate under these conditions? Is the framework primarily designed for offline memory construction using labeled data, or can it support online memory evolution without access to ground-truth answers?

Reviewer_jLz1OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

TraceMem aims to turn successful strategies and recurring mistakes in multimodal reasoning into memories that can be reused on subsequent tasks. It maintains four memory banks, separating visual from logical guidance and successful from failed experiences. Each memory records trigger conditions, procedural steps, avoidance rules, and usage status. When the initial answer fails verification, the system generates four additional reasoning trajectories and extracts new memories from them. Subsequent outcomes are used to update memory utility and retrieval priority. The paper evaluates the method on MathVista, MathVision, HallusionBench, and MMStar using two sizes of Qwen3-VL and GPT-4.1, with component ablations and experiments on memory transfer between models.

Critical strengths
  1. The main results cover different model sizes and show improvements over existing memory methods. In Table 1, TraceMem outperforms the listed baselines across all combinations of three models and four benchmarks. The gains on MathVision are particularly notable. Relative to ViLoMem, accuracy increases from 49.34 to 53.55 with Qwen3-VL-8B and from 62.83 to 65.46 with the 235B model. These comparisons are more convincing than comparisons against direct answering or a simple step-by-step prompt alone, and suggest that the benefits are not confined to small models.
  2. The paper examines whether the memories themselves can be reused, beyond reporting final accuracy. Table 2 studies the roles of structured representations, lifecycle management, and success and failure memories. The transfer experiment also applies memories constructed by the 8B model to the 235B model, improving results on all four benchmarks over the setting where the 235B model uses its own memories. Although the gains are modest, this provides useful preliminary evidence that the memories can transfer between models.
Critical weaknesses
  1. The relationship between memory construction and evaluation is insufficiently specified, making the pass@1 results difficult to assess. Appendix A.3 explicitly states that the internal trajectory verifier uses reference answers. Although Section 3 describes an online memory-update loop and Appendix A.3 mentions held-out calibration splits, the exact data partitions and the timing of memory updates relative to scored predictions remain insufficiently specified. The main text emphasizes first-answer accuracy, while the appendix mentions memory-conditioned second-pass reasoning. The relationship between these needs clarification. Using reference answers as feedback on previous tasks can be a valid setting, but the current description does not establish when this feedback enters memory during evaluation or whether the baselines receive the same information. This is the main reason I cannot yet place full confidence in the comparisons in the main table. The appendix also needs more careful proofreading and layout checks. I cannot read Table 3 on page 14 clearly. Was the manuscript carefully checked before submission?
  2. The comparisons do not adequately separate the benefits of additional trajectory sampling from those of the proposed memory design. In Section 4.3, w/o failure traces disables the entire additional sampling stage after an incorrect initial answer. It therefore also removes any successful trajectories that this stage might produce. This experiment alone cannot support the interpretation that failure information provides an additional benefit. Table 2 does include w/o failure memory, but removing the failure memory bank does not resolve the sampling confound in the former ablation. ReasoningBank and its MaTTS method [1] already investigate extracting reusable memories from multiple successful and failed trajectories, so the paper needs to establish its contribution relative to this line of work more clearly. Moreover, Table 6 in version 1 of ViLoMem [2] reports 74.87 on HallusionBench for its 8B attention variant, above the 73.59 reported here, yet this variant is absent from the comparisons. I would like to see whether the full method retains an advantage over these more direct alternatives under matched feedback and computational budgets. I also looked at the anonymous repository provided by the authors. Unfortunately, it appears to contain only a brief README and no implementation. Please provide the implementation needed to reproduce the reported results, or revise the code-availability statement to accurately describe what is currently accessible.

[1] ReasoningBank: Scaling Agent Self-Evolving with Reasoning Memory. [2] Agentic Learner with Grow-and-Refine Multimodal Semantic Memory.

Questions for authors
  1. Clarification of the evaluation protocol. Could you specify the data splits and sample counts used for memory construction, calibration, and evaluation, along with when memory is initialized, updated, and reset? If memory is updated online after each example, please confirm that the answer counted toward pass@1 is generated before that example's reference answer is used for verification and memory updates. Does the second-pass reasoning mentioned in the appendix contribute to the main results? Please also clarify whether the ViLoMem and Dynamic Cheatsheet results were reproduced or taken from prior work, and whether they use the same feedback conditions. If these stages are already strictly separated, documenting the protocol would substantially address my first concern.
  2. Additional comparisons that would affect my assessment of the contribution. Please clarify whether w/o failure memory reuses the full method's candidate trajectories and verification results, while controlling the memory context budget. If so, that experiment could provide the basis for a revised interpretation of the role of failure information. Otherwise, I suggest adding a comparison with these factors controlled. Please also include the attention variant of ViLoMem and a baseline that adapts the memory induction approach of ReasoningBank/MaTTS to the same multimodal inputs, explicitly describing any changes to its verification procedure. These comparisons could initially focus on MathVision and HallusionBench with the 8B model, reporting token usage or model calls including memory construction. Clear gains under these conditions would strengthen my assessment of the value of the proposed design.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.