- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper studies “visual forgetting”, where large multimodal models gradually lose access to initially perceived visual evidence during extended reasoning. It identifies three principles for reasoning-time visual memory—global contextualization, representation compression, and reasoning-state-adaptive access—and proposes ReMAP, which combines query-conditioned Global Memory with state-guided Local Memory and learns when and which memory to invoke through branched rollouts. Experiments covering single-image perception, multi-image reasoning, and general visual understanding demonstrate the complementary roles of the two memory types, with ReMAP outperforming prior visual-memory methods on all four evaluated multi-image benchmarks.
Critical strengths
The method is well targeted. ReMAP combines question-conditioned Global Memory with reasoning-state-driven Local Memory and learns when to invoke each type of memory through branched rollouts. This design enables the model to actively retrieve global relationships or local details during extended reasoning, rather than relying solely on the initial visual representations, thereby addressing the visual forgetting problem identified in the paper.
Critical weaknesses
-
The scope of the efficiency claims is limited. A reduction in the number of visual tokens does not necessarily translate into faster end-to-end inference. Region localization, local image re-encoding, and memory compression all introduce additional computational overhead, yet the paper lacks a comprehensive comparison of end-to-end latency, peak GPU memory usage, and total computational cost. This weakens the paper’s conclusions regarding its efficiency advantages.
-
The training pipeline is complex and costly to reproduce. The method involves multiple stages of supervised training and reinforcement learning, and relies on additional data construction and teacher supervision, resulting in substantial data preparation and computational costs. It remains difficult to independently reproduce the method and assess whether the performance gains justify the additional training costs.
Questions for authors
-
Could the authors report end-to-end latency, peak GPU memory, and total compute for ReMAP, Native, and Coarse under matched hardware and decoding settings, together with the average number of memory calls and a module-level cost breakdown?
-
Could the authors quantify the total GPU-hours and teacher-generation or annotation costs of the training pipeline, and clarify whether any existing ablations isolate the contributions of student-state adaptation, interface warm-up, and access learning?
4 comments
Response to Reviewer pRvm (2/3): W1 & Q1 (continued) — Module-level breakdown
Table R11. Module-level cost of ReMAP (per-question means; time in s, share of total latency in parentheses).
| Module | Params | V* calls | V* time | V* TFLOPs | Muir calls | Muir time | Muir TFLOPs |
|---|---|---|---|---|---|---|---|
| Preprocessing + vision encoder (coarse visual prefix) | 0.67B (backbone) | 1 | 0.06 (1.0%) | 5.4 | 1 | 0.15 (1.7%) | 12.4 |
| Decoder prefill | 7.6B (backbone) | 1 | 0.04 (0.7%) | 16.7 | 1 | 0.08 (0.9%) | 36.0 |
| Global construction (Image Weaver + Relation Weaver) | 0.32B | 1 | 0.03 (0.5%) | 0.7 | 1 | 0.08 (0.9%) | 1.5 |
| Global insertion | – | 0.6 | 0.02 (0.3%) | 0.1 | 0.9 | 0.04 (0.4%) | 0.3 |
| Source Selector + Twig | 0.70B | 0.2 | 0.01 (0.2%) | 1.4 | 0.1 | <0.01 (0.1%) | 0.4 |
| ROI re-encoding | frozen vision encoder | 0.2 | 0.01 (0.2%) | 0.9 | 0.1 | 0.01 (0.1%) | 0.6 |
| Local Weaver + insertion | 0.35B | 0.2 | <0.01 (0.1%) | 0.1 | 0.1 | <0.01 (<0.1%) | 0.1 |
| Decoding | 7.6B (backbone) | 221 tokens | 5.75 (97.1%) | 3.4 | 331 tokens | 8.61 (96.0%) | 5.0 |
| Total | 1.4B added | 5.92 | 29 | 8.97 | 56 |
The breakdown shows where the time goes and why latent memory pays off.
- Decoding dominates. It takes 97% of the latency on V*Bench and 96% on MuirBench. Inference becomes faster when the decoding phase is shorter and cheaper, and this is what latent memory provides.
- Memory access is almost free. Source selection, Twig, re-encoding, the Local Weaver and insertion together take less than 1% of the latency. A Global request only re-inserts the cached block.
- Global memory is built once and is cheap. The Image and Relation Weavers run once before decoding and take less than 1% of the latency (0.03 s on V*Bench, 0.08 s on MuirBench).
A single question shows this directly. On a two-image difference-spotting question from MuirBench, we run ReMAP with memory on and off (same seed). ReMAP makes one Global and one Local request and answers correctly. The memory-off run takes longer to reason and answers incorrectly.
Table R12. Timeline of one MuirBench question (seconds).
| Stage | ReMAP (1 Global + 1 Local request) | Memory-OFF |
|---|---|---|
| Preprocessing, coarse encoding, prefill (2 images) | 0.14 | 0.14 |
| Global construction | 0.05 | 0.05 |
| Text decoding | 3.28 (126 tokens) | 5.12 (197 tokens) |
| Global request (re-insert cached block) | 0.04 | – |
| Local request (selection, Twig, re-encoding, Local Weaver, insertion) | 0.19 | – |
| Total | 3.70 | 5.31 |
Both runs incur the same Global construction cost. The two requests add 0.23 s, and the shorter chain saves 1.84 s of decoding, so the question finishes about 30% faster.
Response to Reviewer pRvm (3/3): W2 & Q2 — Why the training has several stages, and what it costs
W2 / Q2. The training pipeline is complex and costly. Please quantify GPU-hours and teacher cost, and clarify which ablations isolate student-state adaptation, interface warm-up and access learning.
Why the pipeline has several stages. Each stage solves one problem that the next stage depends on. We target visual forgetting: as the text reasoning grows, the model relies less on the visual evidence. Our analyses fix what the remedy must look like. Memory has to be compact and split into Global and Local parts (Figure 1(a), (b)), and it cannot be inserted at fixed positions, because mistimed evidence hurts (Figure 1(c); Fixed in Table 3). This gives four problems that have to be solved in order.
- Form the memory (Global and Local memory formation). Without this step, the model has no useful visual evidence to retrieve.
- Locate regions from the reasoning state, including the student's own states (training Twig, then student-state adaptation). Otherwise, Local returns are built from the wrong regions at inference.
- Read inserted memory and issue requests (interface warm-up). Before this step, the model does not request at all.
- Request the right memory at the right state (Stage II access learning). This is where the gain over Stage I comes from.
No existing dataset provides this supervision. Existing data give questions, images and answers or full chains of thought, but not what a compact Global memory should hold or when a region should be re-inspected, so the teacher constructs it (Appendix A.5).
Cost. The cost sits in one step: Stage II takes about 95% of the roughly 6,400 A800 GPU-hours, and all of Stage I takes about 350. The teacher (Qwen3-VL-32B-Instruct) runs locally, so there is no API or annotation cost.
Table R13. Training cost per step (A800 GPU-hours).
| Step | Data (Appendix A.5) | GPU-hours, range | Median |
|---|---|---|---|
| Teacher reference CoT | 80,000 examples | 28–39 | 33 |
| 1–2. Global and Local memory formation (Weavers, Source Selector, Twig) | 30,000 Global + 50,000 Local | 68–92 | 79 |
| 3a. Student-state adaptation: student rollouts | 50,000–80,000 | 48–66 | 57 |
| 3b. Student-state adaptation: teacher review | 2–3 states per example | 131–187 | 158 |
| 4. Interface warm-up | 10,000–20,000 | 20–29 | 24 |
| 5. Access learning (Stage II) | 40,000 | 5,270–6,730 | 6,060 |
| Total | 5,570–7,140 | ≈6,410 |
Stage II is expensive because each of its 2,500 steps samples four root trajectories per question, together with their sibling branches and continuation controls. This is the usual cost of reinforcement learning for visual reasoning.
What the ablations isolate. In Table 3, SFT vs. Stage I_sft isolates memory formation, and Stage I_sft vs. ReMAP isolates access learning. Warm-up is the step that makes the model request at all: before it the model almost never requests, after it the model requests too often, and Stage II makes the requests selective (Table R14). Removing warm-up is therefore effectively the "w/o Memory" row. Student-state adaptation has no separate row: the Source Selector, Twig and Local Weaver are frozen in Stage II, so it is their only training on the student's own states (in the manner of DAgger; Ross et al., 2011). Without it, Stage II would learn on mismatched returns, and the two effects could not be separated.
Table R14. Request behavior across training stages.
| Checkpoint | Questions with at least one request | Requests per question |
|---|---|---|
| Before warm-up (after student-state adaptation) | ~0% | 0 |
| After warm-up | 57–88% | 1.3–2.4 |
| After Stage II (ReMAP) | 36–79% | 0.84 |
Reproducibility. Appendix A.4 lists all optimization settings, Appendix A.5 the data sources and sizes, and Appendix A.6 the evaluation protocol. We will release the code, the trained weights and the data sources upon acceptance.
Summary of Responses to Reviewer pRvm
Dear Reviewer pRvm
Thank you again for your time and constructive feedback. As the discussion period draws to a close, we would be grateful to hear whether the additional measurements resolve your concerns or whether further clarification would be helpful. We would be happy to respond promptly. If you find that the new evidence satisfactorily addresses your concerns and strengthens the paper, we would greatly appreciate your consideration of an updated rating. Your recognition of these improvements would mean a great deal to us.
Following your suggestions, we have added matched end-to-end measurements of latency, peak GPU memory, and compute, a module-level cost breakdown, and a step-by-step account of the training cost. The revised manuscript includes the cost analysis (Sec. 4.6, Fig. 9), with changes marked in blue. For your convenience, we summarize the responses below:
[W1/Q1] End-to-End Inference Cost
- ReMAP is the fastest of the compared reasoning methods and uses less compute than Native. On one H100 with greedy decoding, it takes 5.92 s / 8.97 s per question on V*Bench / MuirBench (Native: 7.07 s / 10.78 s) and uses 30% / 53% of Native's TFLOPs. Peak GPU memory is 2.7 GiB above Native on V*Bench and 3.5 GiB below it on MuirBench, with 0.8 / 1.0 memory requests per question. Compared with Coarse at the same input budget, ReMAP adds only 3 / 1 TFLOPs, is 0.4 s / 1.1 s faster, and is 15.7 / 8.2 points more accurate.
- The memory modules cost under 2% of the latency. Decoding takes 96–97%; Global construction, region localization, re-encoding, and Local compression together take less than 2%. Latent memory also shortens reasoning, making ReMAP about 20% faster than its Memory-OFF variant with the same weights.
[W2/Q2] Training Cost and Ablations
- Stage II accounts for about 95% of the training cost. The full pipeline takes about 6,410 A800 GPU-hours, of which Stage II takes about 6,060 and all of Stage I, including teacher generation and review, about 350. The teacher (Qwen3-VL-32B-Instruct) runs locally, with no API or annotation cost.
- Each stage solves a prerequisite of the next. The pipeline forms the memory, locates regions from the student's own states, teaches the model to read memory and issue requests, and finally learns when and which memory to request.
- The ablations isolate the main stages. SFT vs. Stage I_sft isolates memory formation, and Stage I_sft vs. ReMAP isolates access learning (57.81 → 65.38 on MuirBench). Warm-up is what enables requests at all (from almost none to 57–88% of questions), so removing it equals the w/o Memory row. Student-state adaptation is the only training of the Source Selector, Twig, and Local Weaver on student states, as they are frozen in Stage II.
- We will release the code, trained weights, and data sources upon acceptance. Appendices A.4–A.6 list the optimization settings, data, and evaluation protocol.
Thank you again for your valuable suggestions and for your continued consideration.
The Authors
Response to Reviewer pRvm (1/3): W1 & Q1 — End-to-end cost
We thank the reviewer for the positive assessment of the method design and for the concrete questions on cost and reproducibility. We answer each point below and add measurements where needed. Changes to the paper are marked in blue.
W1 / Q1. Fewer visual tokens need not mean faster inference. Please report end-to-end latency, peak GPU memory, total compute, the number of memory calls and a module-level breakdown under matched settings.
We now report all of them. Measured end-to-end, ReMAP needs less compute than Native and has the lowest latency of the compared methods on both benchmarks, including Coarse and its own Memory-OFF variant.
Setup. One H100, greedy decoding, each baseline with its released implementation (VisMem: our reproduction). Accuracy and cost are measured on the same V*Bench and MuirBench questions. ReMAP requests memory about once per question (0.8 times on V*Bench, 1.0 on MuirBench; Global / Local: 0.6 / 0.2 and 0.9 / 0.1).
Table R1. End-to-end inference cost. Latency (s) and TFLOPs are per-question means; memory is peak GPU memory (GiB, weights included). Native and Coarse answer with reasoning; Native (direct) answers without reasoning, as in Table 1, and is listed for reference.
Why ReMAP is cheaper and faster. What enters the decoding phase in ReMAP is latent memory, not image tokens. This has three effects.
The overhead of the memory modules is small. Building Global memory, region localization, re-encoding, and memory compression together take less than 2% of the latency (Table R11, next comment). Peak memory is 2.7 GiB above Native on single-image V*Bench and 3.5 GiB below it on multi-image MuirBench.
Compared with Coarse, which has the same input budget, ReMAP adds little compute (3 TFLOPs on V*Bench, 1 on MuirBench), is 0.4 s and 1.1 s faster because its reasoning is shorter, and is 15.7 and 8.2 points more accurate.
Revision. Sec. 4.6 now reports the end-to-end cost with this analysis (new Figure 9).