- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
- TraceMem organizes the successful and failed reasoning trajectories of multi-modal models into problem-solving experiences, divided into four types of memory: visual success, visual failure, logical success and logical failure.
- Retrieve relevant experience when answering new questions and update memory based on subsequent answer performance. The method does not require modification of model parameters and improves first-time answer accuracy on four multi-modal benchmarks.
Critical strengths
- The idea of accumulating experience across topics is valuable. Additional reasoning generates information that can help with subsequent tasks, rather than just revising the current answer.
- Memory management is relatively complete. At the same time, consider successful experiences, failure lessons, applicable conditions and historical effects, which is more targeted than simply saving reasoning texts.
- There is some evidence of migration. The memory generated by the small model can also help the large model, indicating that the extracted experience is not completely dependent on the model that generated it.
Critical weaknesses
-
The evaluation process is unclear. Validation and memory updates rely on standard answers, but how memory construction is isolated from test data and how it is updated during testing is not adequately explained.
-
The base model is outdated. Experiments only cover Qwen3-VL and GPT-4.1, lacking validation on newer, stronger bases. It's unclear whether the gains will diminish as model capabilities improve.
-
The additional gains for some tasks are small. For example, in the HallusionBench experiment with the 8B model, the improvement compared to ViLoMem is only 0.40 percentage points. The significant improvement compared to direct answers is mainly achieved through step-by-step prompts.
-
Efficiency analysis is lacking. The method incorporates task analysis, multi-track generation, and memory extraction, but complete computational costs, latency, and comparisons under equivalent budgets are not reported.
-
The paragraphs are too fragmented, resulting in a poor reading experience. The methodology section is frequently broken into short paragraphs, with repetitive explanations in adjacent paragraphs, affecting narrative coherence. It is recommended to merge and streamline these sections.
Questions for authors
-
How should the memory construction set and test set be divided? During testing, are standard answers used to update the memory, and how can we ensure that the evaluation results are not affected by information leakage?
-
Could you supplement the experiments on models such as Qwen3.5 and analyze the relationship between method benefits and foundation capabilities?
-
What is the total cost after accounting for memory construction and maintenance? Under the same computational budget, does it still have an advantage compared to multiple sampling or reflection methods?
-
If there is no standard answer and we can only rely on validators with errors, will erroneous experiences accumulate and affect subsequent answers?