acceptodds
Under review as a conference paper at ICLR 2027

REMAP: RESTORING THE PERCEPTUAL CYCLE WITH REASONING-TIME LATENT VISUAL MEMORY

Abstract

As multimodal large language models (MLLMs) reason for longer, attention to the initial visual input diminishes, weakening visual grounding. Visual memory reintroduces visual evidence during reasoning. We conduct a controlled analy- sis of visual memory along three axes: curation, organization, and access. We find that local evidence benefits from global context, compact latent representa- tions balance accuracy and visual-context cost, and the utility of memory access depends on the reasoning state. Guided by these findings, we propose ReMAP (Reasoning-Time Memory-Augmented Perception), which couples two comple- mentary latent memories: a static, question-conditioned Global memory that pre- serves scene and cross-image context, and a dynamic Local memory that uses this context as an anchor while selecting and re-encoding region-level evidence according to the current reasoning state and the question. Both memories re- turn compact latent tokens that are inserted into the reasoning sequence, and a reinforcement-learning access policy trained with branched rollouts decides when to continue reasoning or invoke Global or Local memory. On ten benchmark fam- ilies, ReMAP outperforms prior visual-memory methods on all four multi-image benchmarks, exceeding the strongest prior results on MuirBench and MIMIC by 8.38 and 14.84 percentage points, respectively. Across four backbone families, enabling memory access improves over the same trained model with memory dis- abled, and on shared V*Bench, CV-Bench-2D, and MuirBench questions ReMAP reduces the visual tokens entering the reasoning sequence by 51.0–76.8% relative to the native-resolution backbone. Further analyses show that Global and Local memory form distinct yet complementary latent representations and play coordi- nated roles in interleaved vision-language reasoning. Together, these components restore the perceptual cycle by letting the reasoning state trigger targeted visual retrieval, with the retrieved evidence guiding subsequent reasoning.

Primary area:
foundation or frontier models, including LLMs
Keywords:
vision-language models, visual reasoning, visual memory, active perception, latent representations, reinforcement learning

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_pRvmOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper studies “visual forgetting”, where large multimodal models gradually lose access to initially perceived visual evidence during extended reasoning. It identifies three principles for reasoning-time visual memory—global contextualization, representation compression, and reasoning-state-adaptive access—and proposes ReMAP, which combines query-conditioned Global Memory with state-guided Local Memory and learns when and which memory to invoke through branched rollouts. Experiments covering single-image perception, multi-image reasoning, and general visual understanding demonstrate the complementary roles of the two memory types, with ReMAP outperforming prior visual-memory methods on all four evaluated multi-image benchmarks.

Critical strengths

The method is well targeted. ReMAP combines question-conditioned Global Memory with reasoning-state-driven Local Memory and learns when to invoke each type of memory through branched rollouts. This design enables the model to actively retrieve global relationships or local details during extended reasoning, rather than relying solely on the initial visual representations, thereby addressing the visual forgetting problem identified in the paper.

Critical weaknesses
  1. The scope of the efficiency claims is limited. A reduction in the number of visual tokens does not necessarily translate into faster end-to-end inference. Region localization, local image re-encoding, and memory compression all introduce additional computational overhead, yet the paper lacks a comprehensive comparison of end-to-end latency, peak GPU memory usage, and total computational cost. This weakens the paper’s conclusions regarding its efficiency advantages.

  2. The training pipeline is complex and costly to reproduce. The method involves multiple stages of supervised training and reinforcement learning, and relies on additional data construction and teacher supervision, resulting in substantial data preparation and computational costs. It remains difficult to independently reproduce the method and assess whether the performance gains justify the additional training costs.

Questions for authors
  1. Could the authors report end-to-end latency, peak GPU memory, and total compute for ReMAP, Native, and Coarse under matched hardware and decoding settings, together with the average number of memory calls and a module-level cost breakdown?

  2. Could the authors quantify the total GPU-hours and teacher-generation or annotation costs of the training pipeline, and clarify whether any existing ablations isolate the contributions of student-state adaptation, interface warm-up, and access learning?

4 comments
AuthorsOpenReview ↗

Response to Reviewer pRvm (1/3): W1 & Q1 — End-to-end cost

We thank the reviewer for the positive assessment of the method design and for the concrete questions on cost and reproducibility. We answer each point below and add measurements where needed. Changes to the paper are marked in blue.

W1 / Q1. Fewer visual tokens need not mean faster inference. Please report end-to-end latency, peak GPU memory, total compute, the number of memory calls and a module-level breakdown under matched settings.

We now report all of them. Measured end-to-end, ReMAP needs less compute than Native and has the lowest latency of the compared methods on both benchmarks, including Coarse and its own Memory-OFF variant.

Setup. One H100, greedy decoding, each baseline with its released implementation (VisMem: our reproduction). Accuracy and cost are measured on the same V*Bench and MuirBench questions. ReMAP requests memory about once per question (0.8 times on V*Bench, 1.0 on MuirBench; Global / Local: 0.6 / 0.2 and 0.9 / 0.1).

Table R1. End-to-end inference cost. Latency (s) and TFLOPs are per-question means; memory is peak GPU memory (GiB, weights included). Native and Coarse answer with reasoning; Native (direct) answers without reasoning, as in Table 1, and is listed for reference.

MethodV* Acc.LatencyMem.TFLOPsMuir Acc.LatencyMem.TFLOPs
Native (direct)76.440.7519.09357.850.9531.298
Native74.877.0719.19656.5810.7831.3106
Coarse66.496.3615.92657.1910.1018.955
Q-Zoom79.066.2317.03450.389.2517.528
DeepEyes84.298.8920.016648.5011.5220.588
VisMem60.217.1719.519357.0010.8831.5207
ReMAP, Memory-OFF79.067.4121.82761.0011.2327.856
ReMAP82.205.9221.82965.388.9727.856

Why ReMAP is cheaper and faster. What enters the decoding phase in ReMAP is latent memory, not image tokens. This has three effects.

  1. Less compute. The image enters at coarse resolution, and each return adds only 32 Local or 4N+4 Global latents. ReMAP uses 30% of Native's TFLOPs on V*Bench and 53% on MuirBench.
  2. Lower latency. Fewer tokens enter the decoder, so every decoding step costs less.
  3. Shorter reasoning. A return carries the needed evidence in compressed form, so the model writes fewer reasoning tokens. With the same weights, ReMAP is about one fifth faster than Memory-OFF.

The overhead of the memory modules is small. Building Global memory, region localization, re-encoding, and memory compression together take less than 2% of the latency (Table R11, next comment). Peak memory is 2.7 GiB above Native on single-image V*Bench and 3.5 GiB below it on multi-image MuirBench.

Compared with Coarse, which has the same input budget, ReMAP adds little compute (3 TFLOPs on V*Bench, 1 on MuirBench), is 0.4 s and 1.1 s faster because its reasoning is shorter, and is 15.7 and 8.2 points more accurate.

Revision. Sec. 4.6 now reports the end-to-end cost with this analysis (new Figure 9).

AuthorsOpenReview ↗

Response to Reviewer pRvm (2/3): W1 & Q1 (continued) — Module-level breakdown

Table R11. Module-level cost of ReMAP (per-question means; time in s, share of total latency in parentheses).

ModuleParamsV* callsV* timeV* TFLOPsMuir callsMuir timeMuir TFLOPs
Preprocessing + vision encoder (coarse visual prefix)0.67B (backbone)10.06 (1.0%)5.410.15 (1.7%)12.4
Decoder prefill7.6B (backbone)10.04 (0.7%)16.710.08 (0.9%)36.0
Global construction (Image Weaver + Relation Weaver)0.32B10.03 (0.5%)0.710.08 (0.9%)1.5
Global insertion–0.60.02 (0.3%)0.10.90.04 (0.4%)0.3
Source Selector + Twig0.70B0.20.01 (0.2%)1.40.1<0.01 (0.1%)0.4
ROI re-encodingfrozen vision encoder0.20.01 (0.2%)0.90.10.01 (0.1%)0.6
Local Weaver + insertion0.35B0.2<0.01 (0.1%)0.10.1<0.01 (<0.1%)0.1
Decoding7.6B (backbone)221 tokens5.75 (97.1%)3.4331 tokens8.61 (96.0%)5.0
Total1.4B added5.92298.9756

The breakdown shows where the time goes and why latent memory pays off.

  1. Decoding dominates. It takes 97% of the latency on V*Bench and 96% on MuirBench. Inference becomes faster when the decoding phase is shorter and cheaper, and this is what latent memory provides.
  2. Memory access is almost free. Source selection, Twig, re-encoding, the Local Weaver and insertion together take less than 1% of the latency. A Global request only re-inserts the cached block.
  3. Global memory is built once and is cheap. The Image and Relation Weavers run once before decoding and take less than 1% of the latency (0.03 s on V*Bench, 0.08 s on MuirBench).

A single question shows this directly. On a two-image difference-spotting question from MuirBench, we run ReMAP with memory on and off (same seed). ReMAP makes one Global and one Local request and answers correctly. The memory-off run takes longer to reason and answers incorrectly.

Table R12. Timeline of one MuirBench question (seconds).

StageReMAP (1 Global + 1 Local request)Memory-OFF
Preprocessing, coarse encoding, prefill (2 images)0.140.14
Global construction0.050.05
Text decoding3.28 (126 tokens)5.12 (197 tokens)
Global request (re-insert cached block)0.04–
Local request (selection, Twig, re-encoding, Local Weaver, insertion)0.19–
Total3.705.31

Both runs incur the same Global construction cost. The two requests add 0.23 s, and the shorter chain saves 1.84 s of decoding, so the question finishes about 30% faster.

AuthorsOpenReview ↗

Response to Reviewer pRvm (3/3): W2 & Q2 — Why the training has several stages, and what it costs

W2 / Q2. The training pipeline is complex and costly. Please quantify GPU-hours and teacher cost, and clarify which ablations isolate student-state adaptation, interface warm-up and access learning.

Why the pipeline has several stages. Each stage solves one problem that the next stage depends on. We target visual forgetting: as the text reasoning grows, the model relies less on the visual evidence. Our analyses fix what the remedy must look like. Memory has to be compact and split into Global and Local parts (Figure 1(a), (b)), and it cannot be inserted at fixed positions, because mistimed evidence hurts (Figure 1(c); Fixed in Table 3). This gives four problems that have to be solved in order.

  1. Form the memory (Global and Local memory formation). Without this step, the model has no useful visual evidence to retrieve.
  2. Locate regions from the reasoning state, including the student's own states (training Twig, then student-state adaptation). Otherwise, Local returns are built from the wrong regions at inference.
  3. Read inserted memory and issue requests (interface warm-up). Before this step, the model does not request at all.
  4. Request the right memory at the right state (Stage II access learning). This is where the gain over Stage I comes from.

No existing dataset provides this supervision. Existing data give questions, images and answers or full chains of thought, but not what a compact Global memory should hold or when a region should be re-inspected, so the teacher constructs it (Appendix A.5).

Cost. The cost sits in one step: Stage II takes about 95% of the roughly 6,400 A800 GPU-hours, and all of Stage I takes about 350. The teacher (Qwen3-VL-32B-Instruct) runs locally, so there is no API or annotation cost.

Table R13. Training cost per step (A800 GPU-hours).

StepData (Appendix A.5)GPU-hours, rangeMedian
Teacher reference CoT80,000 examples28–3933
1–2. Global and Local memory formation (Weavers, Source Selector, Twig)30,000 Global + 50,000 Local68–9279
3a. Student-state adaptation: student rollouts50,000–80,00048–6657
3b. Student-state adaptation: teacher review2–3 states per example131–187158
4. Interface warm-up10,000–20,00020–2924
5. Access learning (Stage II)40,0005,270–6,7306,060
Total5,570–7,140≈6,410

Stage II is expensive because each of its 2,500 steps samples four root trajectories per question, together with their sibling branches and continuation controls. This is the usual cost of reinforcement learning for visual reasoning.

What the ablations isolate. In Table 3, SFT vs. Stage I_sft isolates memory formation, and Stage I_sft vs. ReMAP isolates access learning. Warm-up is the step that makes the model request at all: before it the model almost never requests, after it the model requests too often, and Stage II makes the requests selective (Table R14). Removing warm-up is therefore effectively the "w/o Memory" row. Student-state adaptation has no separate row: the Source Selector, Twig and Local Weaver are frozen in Stage II, so it is their only training on the student's own states (in the manner of DAgger; Ross et al., 2011). Without it, Stage II would learn on mismatched returns, and the two effects could not be separated.

Table R14. Request behavior across training stages.

CheckpointQuestions with at least one requestRequests per question
Before warm-up (after student-state adaptation)~0%0
After warm-up57–88%1.3–2.4
After Stage II (ReMAP)36–79%0.84

Reproducibility. Appendix A.4 lists all optimization settings, Appendix A.5 the data sources and sizes, and Appendix A.6 the evaluation protocol. We will release the code, the trained weights and the data sources upon acceptance.

AuthorsOpenReview ↗

Summary of Responses to Reviewer pRvm

Dear Reviewer pRvm

Thank you again for your time and constructive feedback. As the discussion period draws to a close, we would be grateful to hear whether the additional measurements resolve your concerns or whether further clarification would be helpful. We would be happy to respond promptly. If you find that the new evidence satisfactorily addresses your concerns and strengthens the paper, we would greatly appreciate your consideration of an updated rating. Your recognition of these improvements would mean a great deal to us.

Following your suggestions, we have added matched end-to-end measurements of latency, peak GPU memory, and compute, a module-level cost breakdown, and a step-by-step account of the training cost. The revised manuscript includes the cost analysis (Sec. 4.6, Fig. 9), with changes marked in blue. For your convenience, we summarize the responses below:

[W1/Q1] End-to-End Inference Cost

  • ReMAP is the fastest of the compared reasoning methods and uses less compute than Native. On one H100 with greedy decoding, it takes 5.92 s / 8.97 s per question on V*Bench / MuirBench (Native: 7.07 s / 10.78 s) and uses 30% / 53% of Native's TFLOPs. Peak GPU memory is 2.7 GiB above Native on V*Bench and 3.5 GiB below it on MuirBench, with 0.8 / 1.0 memory requests per question. Compared with Coarse at the same input budget, ReMAP adds only 3 / 1 TFLOPs, is 0.4 s / 1.1 s faster, and is 15.7 / 8.2 points more accurate.
  • The memory modules cost under 2% of the latency. Decoding takes 96–97%; Global construction, region localization, re-encoding, and Local compression together take less than 2%. Latent memory also shortens reasoning, making ReMAP about 20% faster than its Memory-OFF variant with the same weights.

[W2/Q2] Training Cost and Ablations

  • Stage II accounts for about 95% of the training cost. The full pipeline takes about 6,410 A800 GPU-hours, of which Stage II takes about 6,060 and all of Stage I, including teacher generation and review, about 350. The teacher (Qwen3-VL-32B-Instruct) runs locally, with no API or annotation cost.
  • Each stage solves a prerequisite of the next. The pipeline forms the memory, locates regions from the student's own states, teaches the model to read memory and issue requests, and finally learns when and which memory to request.
  • The ablations isolate the main stages. SFT vs. Stage I_sft isolates memory formation, and Stage I_sft vs. ReMAP isolates access learning (57.81 → 65.38 on MuirBench). Warm-up is what enables requests at all (from almost none to 57–88% of questions), so removing it equals the w/o Memory row. Student-state adaptation is the only training of the Source Selector, Twig, and Local Weaver on student states, as they are frozen in Stage II.
  • We will release the code, trained weights, and data sources upon acceptance. Appendices A.4–A.6 list the optimization settings, data, and evaluation protocol.

Thank you again for your valuable suggestions and for your continued consideration.

The Authors

Reviewer_gVvYOpenReview ↗
Rating
3(+1)
Confidence
2(−1)
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates how reasoning-time visual memory can improve visual grounding in multimodal reasoning. It proposes ReMAP, which combines a static, question-conditioned Global memory capturing scene-level and cross-image context with a dynamic Local memory that selects and re-encodes image regions based on the current reasoning state. Learnable queries compress visual evidence into compact latent tokens, which are inserted directly into the ongoing reasoning sequence. A reinforcement-learning access policy, trained using shared-prefix branched rollouts, learns when and which memory to invoke. The authors evaluate ReMAP across ten benchmark families and multiple backbone architectures, reporting improved multi-image reasoning and reduced decoder visual-context costs.

Critical strengths

S1. A well-motivated integration of global context and local visual evidence. The paper addresses the important and timely problem of maintaining visual grounding during multimodal reasoning. Combining question-conditioned global context with state-dependent local reinspection is a natural and technically sensible design: global information can help contextualize local evidence, while renewed perception can recover details absent from the initial encoding. This gives the framework a clear motivation and potential relevance across visual reasoning tasks.

S2. Broad empirical validation with informative controls. The evaluation spans ten benchmark families and multiple backbone architectures, supported by component ablations and same-weight memory-on/off comparisons. These controls provide meaningful evidence that reasoning-time memory access benefits the trained system beyond changes to model weights alone, strengthening the empirical case for the approach. The appendix also explicitly acknowledges the limits of several diagnostic probes and cost measurements.

Critical weaknesses

W1. The motivation and key figures are unclear. The introduction does not clearly connect its motivating analyses to the proposed design. For example, Figure 1(a) uses an explicit textual global hint, but its relationship to the learned latent Global memory is insufficiently explained. Several elements of Figures 1 and 2 cannot be understood from the figures and captions alone, particularly the colored bars and positive/negative gains in Figure 1(c).

W2. The VisMem baseline requires validation. The paper reports 57.00% on MuirBench for VisMem, whereas its original paper reports 69.8% with Qwen2.5-VL-7B. Protocol differences may explain this gap, but they are not sufficiently reconciled.

W3. The benefit of the latent representation is not sufficiently isolated. Table 2 compares memory access on versus off, establishing the benefit of additional evidence access rather than the specific advantage of latent returns. Despite the pixel-based baselines and preliminary comparisons, a matched comparison within the final framework—using latent tokens versus crop-token replay or textual evidence—is missing. Such a comparison should control evidence selection, training, and access conditions while reporting accuracy and computational cost.

Questions for authors

See weaknesses.

7 comments
AuthorsOpenReview ↗

Response to Reviewer gVvY (1/4): W1 — Motivation and key figures

We thank the reviewer for recognizing the motivation of combining global context with local re-inspection and the value of the same-weight Memory-ON/OFF comparison. We answer each point below and add experiments where needed. Changes to the paper are marked in blue.

W1. The motivating analyses are not clearly connected to the design, and Figures 1 and 2 are hard to read from the figures and captions alone.

We agree that our description of the two figures was not clear enough. The design of ReMAP does come directly from the three analyses in Figure 1: each panel leads to one design choice. Figure 2 then shows the resulting design: how latent memory is built and used during inference. We explain both figures below.

Figure 1(a) → Global memory that conditions Local memory. A crop from a low-resolution MuirBench image (#393) shows densely packed cylinders, and Qwen2.5-VL-7B guesses plastic or wood. One sentence of scene context ("the background looks like a kitchen") changes the answer to spaghetti. The text hint is only a probe: it shows that local evidence needs global context. In ReMAP this context is not text. It is the learned latent Global memory, and every Local return is conditioned on it (Eq. 2). The ablation in Table R5 confirms this link. With Global conditioning, Local-only access is better than no memory access on both benchmarks. Without it, Local-only access falls below no memory access (60.4 vs. 61.8 on MuirBench), far below the 66.4 of the full model.

Table R5. Local memory with and without Global conditioning (accuracy %; 400 CV-Bench and 500 MuirBench questions).

SettingExplicit Global requestsLocal reads ZGZ_GCV-BenchMuirBench
ReMAP (full)✓✓81.0066.40
No Global conditioning in Local✓✗79.7563.40
Local only (Table 3, w/ Local memory)✗✓80.0062.60
Local only, no Global conditioning✗✗78.5060.40
No memory access (w/o Memory)––79.2561.80

Figure 1(b) → re-encode the region, then compress it. This is a preliminary experiment on a quarter of V*Bench. We return the same region in three ways: frozen latents from the initial encoding, crop pixels, or fresh latents obtained by re-encoding and compressing the region. Dark bars show accuracy, and light bars the added GPU memory relative to crop pixels. Crop pixels are the most accurate and the most expensive. Frozen latents are cheap but lose the detail. Fresh latents recover part of the accuracy at about half the memory. ReMAP takes this trade-off and returns compact latents built from re-encoded pixels. Our new results in the final framework show the same trade-off between accuracy and cost: the trained latent return is as accurate as a crop at one eighth of the time per request (Tables R8 and R9, W3 below).

Figure 1(c) → learned access. Each band runs along the chain of thought. Its color shows the gain from inserting the illustrated memory at each position: green is positive, red is negative, white is close to zero. The arrow marks the illustrated insertion point. An easy recognition question gains nothing anywhere, a V*Bench detail question gains only in a narrow window, and a two-image question gains from two memories at two different positions. Memory is therefore not uniformly useful, so ReMAP learns when to request it. Table 10 quantifies this with log-probabilities, and a new answer-level test confirms it (Table R10, W3 below).

Figure 2: how latent memory is used at inference. The figure follows the order of operations.

  1. Before decoding, the Image and Relation Weavers build the Global memory once and cache it.
  2. During reasoning, when the model emits a Local request, the current hidden state selects sources and regions, the pixels are re-encoded, and the Local Weaver compresses them together with the cached Global memory.
  3. Each return is inserted as a block of latent tokens right after its request marker, and decoding continues.

Revision. The caption of Figure 1 now explains the bands in panel (c). The new Figure 8 (Sec. 4.5) gives quantitative evidence, in the final framework, for the choices motivated by Figure 1(b) and (c).

AuthorsOpenReview ↗

Response to Reviewer gVvY (2/4): W2 — Validation of the VisMem baseline

W2. VisMem scores 57.00 on MuirBench in the paper but 69.8 in its original report.

The evaluation protocol explains only a small part of this gap. The main reason is that the numbers reported in the VisMem paper rest on training that its public release does not contain, so they cannot be reproduced from that release.

Table R7. VisMem: reported scores and scores under our protocol (%).

SourceBase, MuirBenchBase, BLINKVisMem, MuirBenchVisMem, BLINK
VisMem paper (Qwen2.5-VL-7B)57.455.469.864.5
Our protocol (Table 1)57.85 (Native)56.62 (Native)57.00 (reproduced)56.95 (reproduced)

The protocol is not the main cause. Under both protocols the base model scores almost the same (0.5 points apart on MuirBench, 1.2 on BLINK), while the VisMem scores differ by 12.8 and 7.6 points.

The public release does not contain the training described in the paper. The official repository (github.com/xlyu0106/VisMem) has no checkpoint and no training data, and its training code differs from the paper. Each point below can be checked in the named file. Open issues in the repository ask for the checkpoints (#10), the complete GRPO training pipeline (#13) and the training data (#14), and one reports discrepancies between the paper and the released Stage I and Stage II code (#12).

  • Stage I is not the described method. The paper trains both stages with GRPO ("We update the models based on reinforcement learning, i.e., GRPO strategy") and updates both memory formers in Stage I. The released Stage I (main/trainer/stage1_memory_formation.py) is a supervised cross-entropy loss on the reference answer and forms only short-term memory; no released script trains the long-term memory former.
  • Stage II gives no learning signal. The released script (main/cli/train_stage2.py) stops with an UnboundLocalError on its first example, because reward_ema is assigned inside main() without global. With that fixed, the advantage is still zero: the paper uses groups of 16 trajectories (its Table 4), but the released code samples one trajectory per prompt, and its trainer (main/trainer/grpo.py) subtracts the batch mean from this single reward.

Our reproduction. We therefore implemented the training as described in the VisMem paper, on top of the public code, and evaluated it under the same protocol as every other method. This implementation will be part of our code release.

Revision. The caption of Table 8 marks the VisMem row as our reproduction, and Appendix A.6 now states how it was obtained (in blue). We will replace the row as soon as an official checkpoint is released.

AuthorsOpenReview ↗

Response to Reviewer gVvY (3/4): W3 — Latent, crop and text returns in the final framework

W3. The benefit of latent returns is not isolated from that of evidence access.

We ran the matched comparison as requested. It shows two things.

  1. The region that ReMAP localizes is a good source of evidence, in whatever form it is returned.
  2. A latent return is about as accurate as a crop, with its advantage lying in cost. As in Figure 1(b), the latent form is a trade-off between accuracy and latency.

Setup. Weights, request policy and Global memory are fixed, and only the content of a Local return changes. Crop replay follows DeepEyes: tool-call text, a native-resolution crop, and a re-prefill. Text inserts the backbone's own description of the region (up to 64 tokens). To control evidence selection, crop and text use three regions: ReMAP's Twig box, a random box of the same size, and a box annotated with GPT. All variants reuse ReMAP's weights. Training each form with its own Stage II cannot be finished within the discussion period.

Table R8. Local return forms (accuracy %; 400 CV-Bench and 500 MuirBench questions).

Local returnRegionCV-BenchMuirBench
Latent (ReMAP)Twig81.7565.60
Crop replayTwig82.2564.40
Crop replayRandom78.7561.60
Crop replayGPT box83.2566.20
TextTwig80.0062.80
TextRandom79.0061.80
TextGPT box80.7563.40
No Local return–80.2564.20

ReMAP's localization helps. With the Twig box, a crop is about 3 points better than with a random box, and it is within 1–2 points of the GPT-annotated box. Random boxes fall below returning nothing, so the gain comes from the selected region and not from extra tokens.

Latent and crop are on par in accuracy. With the same Twig box, the latent is 0.5 points below the crop on CV-Bench and 1.2 above it on MuirBench, where a crop alone lacks the cross-image context that the latent takes from Global memory. Text is below both.

Table R9. Cost of the return forms (one H100, Twig box; latency in s per question; "162 Q": the questions on which ReMAP issues a Local request).

Local returnTokens / requests / requestLatency CVLatency MuirLatency 162 QPeak GiB (CV / Muir)TFLOPs (CV / Muir)
Latent (ReMAP)320.173.898.974.5620.6 / 27.817 / 56
Crop replay3601.334.569.366.3621.2 / 29.025 / 60
Text621.855.249.747.9320.7 / 28.021 / 58
No Local return0–4.159.214.7920.5 / 27.816 / 55

Latent returns are much cheaper. A latent uses one eleventh of a crop's tokens and one eighth of its time per request, because it needs neither tool-call text nor a re-prefill. Text is the slowest, since the description has to be generated first. On the questions with Local requests, crop replay and text are 39% and 74% slower than the latent. The latent is even faster than no Local return, because it shortens the reasoning.

Revision. The new Figure 8(a) (Sec. 4.5) shows this trade-off.

Reviewer_KfghOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces ReMAP, a framework that allows multimodal language models to revisit visual evidence during reasoning through compact latent memories. The method combines a static, question-conditioned Global memory with a dynamic Local memory that selects and re-encodes image regions based on the current reasoning state. A two-stage training procedure first learns memory formation and then optimizes memory access using branched rollouts that compare requesting memory with continuing from the same reasoning state. The evaluation covers ten benchmark families, multiple backbone families, and ablations of memory composition, access policies, and latent capacity. ReMAP achieves particularly strong results on multi-image reasoning, while same-model Memory-OFF comparisons provide evidence that reasoning-time memory access contributes beyond the effects of training alone.

Critical strengths
  1. The method is well motivated and coherently designed.

    The organization around memory curation, representation, and access gives the paper a clear structure. The distinction between Global context and Local detail is sensible, particularly for multi-image questions where a crop may be ambiguous without its scene or cross-image context. Conditioning Local memory on both the reasoning state and cached Global memory provides a concrete implementation of this motivation.

  2. The empirical evaluation is broad and generally convincing.

    The experiments cover single-image perception, multi-image reasoning, and broader visual understanding. The improvements are especially substantial on MuirBench and MIMIC, where ReMAP exceeds the strongest prior-method results in Table 1 by 8.38 and 14.84 points, respectively. The results across additional backbone families further suggest that the approach is not restricted to one particular model.

  3. The paper includes useful controls for attributing the gains.

    The Memory-OFF comparisons retain the trained model weights and disable memory invocation only at inference. This is a valuable control because it separates the contribution of reasoning-time memory access from improvements already absorbed into the trained parameters. The comparisons with SFT and Stage I training, together with the memory-type and invocation-policy ablations, provide a reasonably comprehensive picture of the system.

  4. The access-learning formulation addresses a meaningful credit-assignment problem.

    Comparing memory invocation with continuation from the same prefix is more directly aligned with the utility of a request than assigning every request the final trajectory reward. The paper also correctly notes that the continuation control may request memory later, so the comparison concerns requesting now rather than prohibiting memory altogether.

  5. The presentation is clear and polished.

    The main text explains the architecture and training stages effectively, while the appendices provide substantial detail on objectives, memory interfaces, data construction, evaluation protocols, and qualitative traces. The distinction between decoder visual-context cost and total inference compute is explicitly documented, which helps interpret the efficiency results.

Critical weaknesses
  1. Reduced decoder visual context does not establish an end-to-end efficiency advantage.

    The visual-token reductions are encouraging, but Appendix A.12 excludes computation inside the vision encoder and memory weavers. ReMAP also constructs Global memory before decoding and re-encodes crops during Local requests. These operations may offset some of the savings from shorter decoder context. Furthermore, the token counts are measured on relatively small shared subsets, whereas the performance curves use full-benchmark scores. Reporting end-to-end latency, peak GPU memory, and average memory-call counts would clarify the practical accuracy–cost trade-off.

  2. The invocation ablations do not fully isolate the benefit of the proposed access-learning objective.

    The Fixed, Random, and Entropy controls modify invocation behavior at inference using the fully trained ReMAP weights. These comparisons support the usefulness of the learned policy, but do not isolate the training contribution of the same-state invocation gain, branch-point selection, and auxiliary objectives. The controls also differ in their realized number of requests, and the heuristic policies do not provide an equally informed choice between Global and Local memory. Appendix A.10 provides a useful branching diagnostic, but routing diversity does not directly establish improved final performance after policy training. A focused comparison against access learning without the same-state invocation objective would strengthen the attribution.

  3. The evidence for Global–Local complementarity should be interpreted more narrowly.

    Appendix A.9 clarifies that the Local-only variant still uses cached Global memory when constructing its Local returns. This comparison therefore tests the value of explicit Global requests in addition to globally conditioned Local requests, rather than Local memory with versus without Global context. Similarly, the t-SNE separation in Figure 4 is suggestive but does not establish the semantic roles attributed to the representations. An ablation removing Global conditioning from the Local Weaver would more directly test the proposed contextual anchoring mechanism.

Questions for authors
  1. What are the end-to-end latency, peak GPU memory, and average Global/Local request counts relative to the Native backbone and representative memory baselines?

  2. How much does the same-state invocation-gain objective contribute beyond ordinary final-answer policy optimization under a matched rollout budget?

  3. What happens when cached Global context is removed from the Local Weaver? This would directly test whether contextual anchoring improves Local evidence acquisition.

  4. Can the authors clarify the differing effects of Stage II across benchmarks, particularly the strong improvements on MuirBench and MM-Vet versus the nearly unchanged MIMIC average?

8 comments
AuthorsOpenReview ↗

Response to Reviewer Kfgh (1/5): W1 & Q1 — End-to-end efficiency

We thank the reviewer for the detailed and constructive comments, and for recognizing the motivation, the breadth of the evaluation, the Memory-OFF control and the same-state credit assignment. We answer each point below and add experiments where needed. Changes to the paper are marked in blue.

W1 / Q1. Does a shorter decoder visual context translate into end-to-end efficiency?

Yes. Measured end to end, ReMAP is faster than Native, Coarse and all memory baselines on both benchmarks, and also faster than its own Memory-OFF variant. It uses about one third to one half of Native's compute.

Setup. One H100, greedy decoding, each baseline with its released implementation (VisMem: our reproduction). The reviewer is right that the token counts in Appendix A.12 were measured on the questions profiled for all five methods, while the performance lines used full-benchmark scores. Here accuracy and cost are measured on the same V*Bench and MuirBench questions. ReMAP requests memory about once per question (0.8 times on V*Bench, 1.0 on MuirBench), mostly Global (Global / Local: 0.6 / 0.2 and 0.9 / 0.1).

Table R1. End-to-end inference cost. Latency (s) and TFLOPs are per-question means; memory is peak GPU memory (GiB, weights included). Native and Coarse answer with reasoning; Native (direct) answers without reasoning, as in Table 1, and is listed for reference.

MethodV* Acc.LatencyMem.TFLOPsMuir Acc.LatencyMem.TFLOPs
Native (direct)76.440.7519.09357.850.9531.298
Native74.877.0719.19656.5810.7831.3106
Coarse66.496.3615.92657.1910.1018.955
Q-Zoom79.066.2317.03450.389.2517.528
DeepEyes84.298.8920.016648.5011.5220.588
VisMem60.217.1719.519357.0010.8831.5207
ReMAP, Memory-OFF79.067.4121.82761.0011.2327.856
ReMAP82.205.9221.82965.388.9727.856

Why ReMAP is faster. What enters the decoding phase in ReMAP is latent memory, not image tokens. This saves time in two ways.

  1. Compressed evidence makes reasoning shorter. A return carries the needed visual evidence in a few latent tokens, so the model writes fewer text tokens to describe and re-infer details. With the same weights, ReMAP is therefore faster than Memory-OFF.
  2. Fewer tokens enter decoding, so each step costs less. The image enters at coarse resolution, and each return adds only 32 Local or 4N+4 Global latents. The decoder's visual context is 51.0–76.8% smaller than Native's.

The memory modules themselves cost little: building Global memory once and serving all requests take under 2% of the latency, while decoding takes 96–97%. Per module (s per question; share of latency in parentheses):

ModuleV* timeMuir time
Preprocessing + vision encoder0.06 (1.0%)0.15 (1.7%)
Decoder prefill0.04 (0.7%)0.08 (0.9%)
Global construction (Image Weaver + Relation Weaver)0.03 (0.5%)0.08 (0.9%)
Global insertion0.02 (0.3%)0.04 (0.4%)
Source Selector + Twig0.01 (0.2%)<0.01 (0.1%)
ROI re-encoding0.01 (0.2%)0.01 (0.1%)
Local Weaver + insertion<0.01 (0.1%)<0.01 (<0.1%)
Decoding5.75 (97.1%)8.61 (96.0%)
Total5.928.97

Comparison with Q-Zoom and DeepEyes. Q-Zoom needs the least compute on MuirBench, but it runs two forward passes, is slightly slower than ReMAP on both benchmarks, and is far less accurate on multi-image questions. DeepEyes is slower: every access writes tool-call text and re-prefills the history with a new crop, while ReMAP emits one request token and inserts the latents directly.

Case analysis on questions with calls. None of the three methods issues a call on every question: Q-Zoom and DeepEyes call on only part of the questions, and ReMAP requests memory only when needed (0.8 requests per question on V*Bench). Averages over all questions therefore dilute the cost of a call. For a direct comparison, we also use a V*Bench subset on which Q-Zoom, DeepEyes and ReMAP generally all issue calls.

Table R2. V*Bench questions with calls (per-question means, seconds).

MethodCalls / questionLatencyGlobal constructionEvidence acquisitionPer call
Native–8.1–––
Q-Zoom0.8 RoI7.2–0.100.12
DeepEyes1.2 crops10.3–1.621.35
ReMAP0.9 Global / 1.4 Local6.90.030.270.12

The conclusion is the same. ReMAP issues about twice as many requests as DeepEyes, yet it spends one sixth of the time on acquiring evidence, and it has the lowest latency of the four methods.

Revision. Sec. 4.6 now reports the end-to-end cost with this analysis (new Figure 9).

AuthorsOpenReview ↗

Response to Reviewer Kfgh (2/5): W2 — Realized request counts and the Global/Local choice

W2. The controls differ in realized request counts, and the heuristics lack an equally informed Global/Local choice.

ReMAP's memory access is agentic: it decides each memory request from the reasoning generated so far, and each memory return changes the reasoning that follows. Once two policies make one different decision, their chains of thought diverge, so we cannot fix the request points and vary only the number of requests or only the memory type. We therefore control the two separately: the request count in Experiment 1 and the memory type in Experiment 2. All other settings follow the paper.

Experiment 1: the same request budget. Every policy gets the same budget: a uniform cap of K requests per chain, or a per-question quota equal to ReMAP's own count on that question.

Table R3. Invocation policies under the same budget (100 MuirBench questions; accuracy % / realized requests per question).

PolicyK = 1K = 2K = 5Per-question quota
Random61 / 0.4758 / 1.0354 / 2.4457 / 1.02
Entropy65 / 0.7660 / 1.1561 / 1.3963 / 0.91
ReMAP62 / 0.6961 / 0.9766 / 1.1666 / 1.16

ReMAP's advantage comes from where it requests, not from how often.

  • ReMAP is the only policy that benefits from a larger budget. As K grows, Random requests more and loses accuracy, and Entropy does not improve. ReMAP is the most accurate from K = 2 on and reaches its best accuracy at K = 5. We attribute this to ReMAP having two memory types that work together. For example, it can retrieve global relations first and check a local detail afterwards. A single request cannot explicitly use both, so the advantage does not show under this cap. The K = 1 setting therefore limits ReMAP's ability to fully use its learned access policy.
  • The request count does not explain the gain. At K = 5, ReMAP issues the fewest requests and is the most accurate. Under the per-question quota, it issues the most and is still the most accurate. Whether it requests less or more than the heuristics, it stays ahead.

Experiment 2: an equally informed Global/Local choice. We cross the request timing (learned or Entropy) with the request type (ReMAP's own p(G), p(L), or 50:50).

Table R4. Request timing × request type (accuracy %; 400 CV-Bench questions, 200 each from 2D and 3D, and 500 MuirBench questions; rows with sampled types average three seeds). 50:50: Global or Local memory is requested with equal probability; p(G), p(L): the type is sampled from the probabilities of ReMAP's learned policy.

TimingTypeCV-BenchMuirBenchMean
LearnedReMAP81.7565.6073.68
Learned50:5077.7563.8770.81
EntropyReMAP's p(G), p(L)78.2562.9370.59
Entropy50:5079.0062.2770.63

Timing and type work together. ReMAP chooses the type for the state at which it requests: a request expresses a need at that moment, either global relations or a local detail. Replacing one side breaks this match. Learned timing with a random type, and Entropy timing with ReMAP's type preference, both stay at the level of the pure heuristic. ReMAP's own combination is clearly better, by about 3 points on average. Giving a heuristic a better-informed type choice is therefore not enough, and timing and type have to be learned together.

Revision. The new Figure 8(b) (Sec. 4.5) shows the budget comparison.

AuthorsOpenReview ↗

Response to Reviewer Kfgh (3/5): Q2 — Contribution of the same-state invocation gain

Q2. How much does the same-state invocation gain contribute beyond final-answer policy optimization under a matched rollout budget?

The request policy trained with the same-state gain adds 4.4 points on MuirBench and 6.5 on MM-Vet beyond final-answer policy optimization. We first explain why requests need their own credit, and then give the comparison.

A final-answer reward cannot credit a single request. Request tokens are less than 1% of the generated tokens, and the final reward also depends on all the text reasoning and on later requests. GRPO gives every token of a trajectory the same advantage, so it cannot tell a well-timed request from a mistimed one that later reasoning repairs. The same-state gain branches from a shared prefix and compares "request now" with "continue" at the same state, so the reward difference is the value of this one request. ReMAP keeps the final-answer objective for everything else: the decoder LoRA is trained with the GRPO advantage, and only the request-token embeddings and output rows use the same-state gain (Appendix A.3.4).

Contribution beyond final-answer optimization. The two objectives train different parameters, so their contributions can be separated in the same trained model. With all requests masked, only what the final-answer objective trained remains, and the request policy, trained with the same-state gain, accounts for the rest. All rows below use the same Stage II weights (Table 3).

Requests in the Stage II modelMuirBenchMM-Vet
Masked (final-answer optimization only)61.0057.80
Fixed, Random or Entropy rule (best of the three)62.0059.63
Learned request policy (ReMAP)65.3864.27

Final-answer optimization alone lifts Stage I from 57.81 to 61.00 on MuirBench and from 48.35 to 57.80 on MM-Vet. The learned requests add a further 4.38 and 6.47 points. Rule-based requests with the same weights and the same memory recover at most 1.00 and 1.83 of them, so 3.4 and 4.6 points come from the request decisions themselves, which are what the same-state gain trains.

The objective also changes how the model requests. After the interface warm-up, the model requests on 57–88% of the questions, 1.3–2.4 times per question. After Stage II, it requests on 36–79% of them, 0.84 times per question, and accuracy is higher. Timing is what matters: the same crop lowers accuracy from 69.8% to 67.3% when it is inserted at the start of reasoning, and raises it to 72.8% at the point where ReMAP requests.

Retraining with trajectory-level credit. A variant that trains the request tokens with the trajectory-level advantage, from the same Stage I checkpoint and with the same rollout budget, needs a second Stage II run (about 6,000 A800 GPU-hours). We cannot complete it within the discussion period and will add it to the final version.

Comments on the paper

1 comment

Dear Area Chair

Thank you for overseeing the review of our submission and for your time and effort.

We have posted detailed responses to all three reviewers, together with additional experiments and a revised manuscript (changes marked in blue). We fully understand that reviewers have many commitments during this busy period, and we are sincerely grateful for the time they have already devoted to our paper.

As the discussion period draws to a close, we would greatly appreciate it if you could kindly bring our responses to the attention of Reviewers Kfgh, gVvY, and pRvm. We would welcome any further feedback and would be happy to provide additional clarification if helpful.

Thank you again for your kind help and consideration.

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.