acceptodds
Under review as a conference paper at ICLR 2027

Look Back, Think Ahead: Visual Memory on Demand for Efficient Multimodal Reasoning

Abstract

Processing long visual token sequences from high-resolution images makes multi-step reasoning computationally expensive for multimodal Large Language Models (MLLMs). Existing one-shot pruning and aggregation methods compress visual tokens into a fixed context before decoding. However, visual evidence needs can shift as reasoning unfolds, making it difficult for a fixed compressed context to retain all the details needed across stages. To address this challenge, we propose ViMoD, a lightweight framework that maintains a compact visual context while preserving access to original fine-grained evidence as reasoning needs evolve. Deformable Aggregation of Region-wise Tokens (DART) learns content-adaptive groups and aggregation capacities, constructing compact Coarse representations linked to recoverable original Fine tokens. Temporal Routing for Adaptive Contextual Evidence (TRACE) integrates decoding history to anticipate upcoming evidence needs and select, retain, or replace active Fine-token groups. Selected Fine tokens augment the persistent Coarse context in the frozen backbone, enabling stage-specific evidence access without continuously attending to all visual tokens. On Qwen3-VL-4B, ViMoD outperforms all evaluated baselines on all eight reasoning benchmarks at a **20%** target visual-token budget, improving the mean normalized score by **39.0%** over the strongest evaluated one-shot baseline. These gains require additional trainable parameters amounting to only **0.0546%** of the frozen backbone's parameter count.

Primary area:
foundation or frontier models, including LLMs
Keywords:
multimodal large language models, multimodal reasoning, visual token compression, efficient inference

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_TajyOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
3(+1)
Presentation
2
Soundness
3(+1)
Read review
Paper summary

This paper proposes ViMoD, a framework for dynamic visual token compression in multi-step multimodal reasoning. By combining content-adaptive visual memory (DART) with history-aware evidence routing (TRACE), the method enables selective access to fine-grained visual information during decoding. The motivation is well supported, and the experimental results demonstrate substantial improvements across multiple benchmarks. My main concerns lie in the experimental validation, particularly regarding efficiency trade-offs, design choices, and the robustness of the method. Further clarification or additional experiments addressing these concerns would strengthen the paper's rigor and persuasiveness and increase my confidence in its contributions.

Critical strengths

(1) The paper identifies a limitation of existing visual token compression methods, which typically retain a fixed visual context throughout decoding. Figures 1–2 provide empirical evidence that visual evidence requirements change across reasoning stages, motivating dynamic evidence access.

(2) DART adaptively adjusts token grouping and compression granularity based on image content while preserving access to the original Fine tokens. This allows the model to retrieve fine-grained visual evidence when needed without repeatedly encoding the image.

(3) TRACE uses decoding history to predict upcoming visual evidence needs and employs a HOLD/UPDATE mechanism to retain or replace active Fine-token groups. This allows visual evidence access to change during reasoning rather than relying on a fixed compressed context.

Critical weaknesses

(1) ViMoD retains the full Fine KV cache alongside additional Coarse KV representations, resulting in higher peak GPU memory usage than the uncompressed backbone. While the method demonstrates reduced inference latency, could the authors evaluate its accuracy–throughput trade-off under a fixed GPU memory budget, allowing each method to use its maximum feasible batch size? This would help clarify whether the computational benefits translate into practical efficiency gains under comparable memory constraints.

(2) TRACE extracts hidden states from backbone layers 27, 31, and 35, but the paper does not explain the rationale behind this selection. Since these representations directly determine the temporal routing state, the choice of layers may substantially affect evidence prediction. Although prior studies have provided some insights into layer selection, we would appreciate further justification for why this specific configuration is suitable for the proposed task. Additionally, it would be valuable to investigate how different teacher models or backbone model families affect the final performance, providing further evidence of the method's robustness and generalizability.

(3) TRACE aims to predict upcoming visual evidence needs by integrating decoding history. However, its effectiveness is primarily demonstrated through downstream task accuracy, without directly evaluating the quality of the predicted evidence regions. Although Fig. 8 qualitatively illustrates the dynamic evidence access process, it remains unclear how accurately TRACE predicts future evidence requirements. We suggest that the authors provide quantitative evaluations (e.g., evidence prediction precision based on annotated data) to validate TRACE's predictive capability.

Questions for authors

Please refer to the Critical Weaknesses section.

4 comments
AuthorsOpenReview ↗

Evidence Prediction, Readout Design, and Efficiency Across Backbones

Thank you for recognizing the motivation and design of ViMoD. We have addressed your concerns with the following evaluations. References refer to the revised manuscript.

W1 — Fixed-memory efficiency. Fig. 6 now compares methods at their maximum feasible batch sizes under an 80 GiB budget on a mixed MMMU-Pro/DynaMath workload. At 40% occupancy, maximum batches are 51 for ViMoD, 52 for Full, and 56 for one-shot methods, accounting for Fine and Coarse storage. At 40% visual occupancy, ViMoD reduces the time to complete the same workload by 9.6% relative to Full and 12.8–24.2% relative to one-shot methods, while achieving 87.5% normalized accuracy versus their 60.4–79.8%. This demonstrates a better accuracy–efficiency trade-off under the same memory budget. Sec. 4.4 and App. C.6 provide the measurements and protocol; Table 14 lists the 40% operating points: 2,326 s for ViMoD, 2,573 s for Full, and 2,668–3,068 s for one-shot methods.

W2 — Readout layers. Zero-based decoder layers 27, 31, and 35 sample evenly spaced depths in the decoder’s final third. Prior studies identify distinct reasoning and answer-decoding stages across layers and find later hidden states informative about subsequent response content, motivating the fusion of multiple late readouts. Our added ablation in Table 13 directly compares Early, Middle, Late, and Mixed layouts under the same training setup. Late achieves the highest evidence-prediction F1, 11.67% versus 11.22–11.27%, and answer accuracy, 32.20% versus 20.60–25.60%, at matched visual occupancy. App. C.4 discusses the motivation and evaluation.

W2 — Backbone generalization. We have added InternVL3-9B, changing both family and scale from Qwen3-VL-4B while using the same training procedure and training data. Evaluation uses InternVL3’s official decoding configuration and otherwise follows VLMEvalKit. At 20% occupancy, ViMoD achieves a normalized reasoning average of 93.53%, versus 78.93% for the strongest one-shot baseline. It leads both the reasoning and general-understanding averages at every evaluated budget, extending the main finding to a larger backbone from another family. Sec. 4.5 and Table 7 present the main results; App. C.10, Tables 17–18, provide all benchmarks. Table 9 compares the backbone-specific DART/TRACE architectures and parameter counts.

W3 — Evidence-prediction quality. Table 4 directly evaluates predicted Fine-token sets against evidence for the next decoding stage on 100 unseen MMMU-Pro trajectories, annotated by a separate, stronger evaluation teacher, DeepSeek-V4.1-Flash. Predictors receive only the generated prefix and available history. Under a common 10% Fine-token cap, TRACE achieves 22.88% precision, 23.19% recall, and 23.03% F1, compared with F1 scores of 10.45% for random DART selection and 6.33% for attention-based selection. These results directly demonstrate TRACE’s ability to anticipate the next decoding stage’s evidence. Next-stage supervision improves evidence-prediction F1 by 2.73 points over current-stage supervision. With next-stage supervision, SSM exceeds the parameter-matched MLP by 0.62 points, and RL further improves SSM F1 from 15.86% to 23.03%. These comparisons support anticipating upcoming evidence, retaining recurrent history, and optimizing the access policy with task feedback. Sec. 4.2 provides a complementary downstream comparison: with the same 9:1 DART memory and frozen backbone, TRACE exceeds the DSTP adaptation by 8.48–11.19 points in normalized reasoning average. Sec. 4.3 analyzes evidence prediction, with evaluation details in App. C.3. In Table 12, both teachers independently annotate the same prefixes using the same prompt and only the image, question, and available reasoning prefix, without future reasoning or the final answer. Their evidence-set F1 is 55.65%. The same TRACE predictions achieve 21.92% F1 against Qwen3.8 and 23.03% against DeepSeek, showing similar prediction quality across the two annotation references.

AuthorsOpenReview ↗

Key Results: Fixed-Memory Efficiency and Direct Evidence Prediction

For easier reference, we include the key results from our revised manuscript below, complementing our earlier response, Evidence Prediction, Readout Design, and Efficiency Across Backbones.

W1 — Accuracy and efficiency under the same memory budget. Each method uses its maximum feasible batch under an 80 GiB budget on the same mixed MMMU-Pro/DynaMath workload. Compressed methods use 40% visual occupancy; Full retains all visual tokens. Accuracy averages Full-normalized task scores. From Table 14:

MethodMaximum batchTotal time (s)Normalized accuracy (%)
Full522,573100.0
FastV562,82860.4
VisionZip562,66879.8
DivPrune563,06864.4
ViMoD512,32687.5

ViMoD completes the workload faster than all compared methods while retaining substantially more accuracy than one-shot compression. Its smaller maximum batch already accounts for persistent Fine/Coarse storage.

W2 — Readout-layer selection. On 500 annotated MMMU-Pro examples unseen during training, all layouts use the same supervised training setup without RL. Values are mean ± standard deviation over three seeds; layer indices are zero-based. From Table 13:

LayoutLayersEvidence F1 (%)New-evidence recall (%)Answer accuracy (%)Occupancy (%)
Early3, 7, 1111.26 ± 0.0314.93 ± 0.0725.60 ± 2.0020.30
Middle15, 19, 2311.22 ± 0.0315.03 ± 0.0120.60 ± 1.6420.20
Late27, 31, 3511.67 ± 0.0115.39 ± 0.0332.20 ± 2.3120.10
Mixed11, 23, 3511.27 ± 0.0215.22 ± 0.0122.40 ± 1.6420.20

Late readouts achieve both the highest evidence-prediction F1 and answer accuracy at comparable occupancy, directly supporting the chosen configuration.

W3 — Direct evaluation of upcoming evidence. On 100 unseen MMMU-Pro trajectories, a separate, stronger evaluation teacher, DeepSeek-V4.1-Flash, annotates evidence for the next decoding stage. All methods use a common 10% Fine-token cap. Current means the current decoding stage; Future means the next decoding stage. From Table 4:

PredictorSFT targetFine used (%)Precision (%)Recall (%)F1 (%)
Attention—9.267.645.406.33
Random DART—9.3311.629.4910.45
SSM (SFT)Current9.3313.2513.0213.13
SSM (SFT)Future9.3315.8815.8315.86
Window MLP (SFT)Current9.3313.4112.9813.19
Window MLP (SFT)Future9.3315.0615.4215.24
SSM (SFT+RL)Future9.3622.8823.1923.03

Next-stage supervision improves both architectures; recurrent SSM outperforms the parameter-matched MLP with Future supervision, and RL further improves F1 to 23.03%. Thus, the evaluation measures evidence prediction directly, beyond downstream accuracy.

For annotation-reference consistency, Table 12 evaluates the same TRACE predictions against both teachers:

EvaluationPrecision (%)Recall (%)F1 (%)
Qwen3.8 vs. DeepSeek-V4.1-Flash58.4153.1355.65
TRACE vs. Qwen3.822.1221.7321.92
TRACE vs. DeepSeek-V4.1-Flash22.8823.1923.03

Both teachers independently annotate identical prefixes using the same prompt, without future reasoning or the final answer. TRACE's similar F1 against the two references supports consistent evidence-prediction quality.

W2 — Cross-backbone generalization. InternVL3-9B uses the same training procedure and data as Qwen3-VL-4B, its official decoding configuration, and otherwise VLMEvalKit evaluation. Full retains all tokens; compressed methods use 20% occupancy and ViMoD uses 9:1 memory. From Table 7:

MethodDynaMath (%)MMMU-Pro (%)LogicVista (%)MathVista (%)
Full, 100%51.9232.7244.0768.40
FastV33.2516.9434.6856.10
DivPrune32.5114.1037.5855.50
VisionZip32.0813.9333.1147.80
ViMoD 9:148.6622.2542.9566.90

ViMoD leads every compressed baseline on these four tasks. Its normalized reasoning average is 93.53%, versus the best one-shot average of 78.93%, supporting generalization across model families and scales.

We would appreciate your assessment of whether these direct measurements address your concerns about practical efficiency, layer selection, and predictive capability.

Thank you again for taking the time to review our paper and for the constructive comments in your review. As the discussion period draws to a close, we’d be grateful to hear whether our responses and additional experiments have addressed the concerns you raised. If there’s anything you’d like us to clarify, we’d be happy to do so.

I apologize for the delayed response. We have been handling a large volume of reviews recently, and your manuscript covers several aspects that required additional time for careful consideration. Thank you for your detailed response and clarifications, particularly regarding the generalization capability and the quality of the supporting evidence. My previous concerns have been largely addressed, and I will increase my rating accordingly.

Reviewer_EFreOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper argues that one-shot visual-token compression suits multi-step reasoning poorly. The visual evidence a model needs shifts during decoding, and per-token savings can be offset by longer outputs. ViMoD keeps Qwen3-VL-4B frozen and adds two modules. DART learns content-adaptive groups of visual tokens and pools them into a persistent Coarse KV cache, while keeping the original Fine KV entries recallable. TRACE, a multi-timescale state-space router, decides every 16 generated tokens which Fine groups to activate.

Critical strengths
  1. The motivation is backed by clear diagnostics. Fig. 2 links shifting evidence demand to failures of fixed compression. Fig. 3 shows per-token savings offset by longer outputs, which argues for measuring cost over whole responses.
  2. The comparison with DSTP†\text{DSTP}^\dagger keeps the DART memory and frozen backbone fixed, so it isolates the access policy. TRACE gains 7.37–9.73 points in normalized reasoning average across budgets.
  3. The design is lightweight and well ablated. The backbone stays frozen and only 0.0546% parameters are added. The ablations target repeated vs. single recall, DART vs. mean pooling, and RL vs. SFT. A router trained with the 9:1 compressor transfers to the 4:1 compressor without retraining.
Critical weaknesses
  1. The evaluation protocol may inflate the gap to one-shot baselines. With greedy decoding, presence_penalty=0 and a 4,096-token cap, Full scores only 30.92 on MMMU-Pro and 19.18 on VisualPuzzles. The MMMU-Pro figure is 53.2 in the technical report and above 40% in the 8,192-token run of Fig. 6. The VisualPuzzles figure is below the 25% chance level. Both suggest heavy truncation. One-shot baselines generate 15.9–29.9% more tokens than Full, so truncation likely hurts them more, and all averages are normalized by these Full scores.
  2. The margin over one-shot baselines is confounded. ViMoD prefills with all visual tokens and is trained with distillation, a 27B teacher and RL. The baselines prune before or during prefill and need no training. Coarse-only ViMoD, at a nominal 11.1% occupancy, already beats DivPrune at 30% on DynaMath and MMMU-Pro (47.72/12.20 vs. 45.71/11.79). Much of the margin may therefore not come from dynamic access.
  3. The efficiency gains are modest and cost memory. Peak memory rises from 38.40 to 42.67 GiB, and total time drops by at most about 10% (Fig. 6(a)). Efficiency is measured on only one benchmark, one GPU and batch size 8.
Questions for authors
  1. For each method and benchmark, what are the truncation rate, the answer-extraction failure rate and the output length? How do results change under the recommended decoding settings or a higher output cap, at least on MMMU-Pro, VisualPuzzles and one math benchmark?
  2. How do one-shot baselines perform when applied after full prefill, at matched decoding occupancy? How does a trained static baseline at matched realized occupancy compare with ViMoD? One example is the 4:1 DART Coarse memory without recall.
  3. In Tab. 4, the SFT-only variant scores 47.66 on DynaMath at all three budgets. Is this a reporting error, or does the δν\delta_{\nu} offset not change this policy's behavior? What occupancy does it actually reach?
  4. Are the Coarse keys in Eq. (5) pooled before or after the rotary position encoding (MRoPE)? How are positions handled for groups that cross grid cells?
6 comments
AuthorsOpenReview ↗

Decoding Robustness, Controlled Comparisons, and Efficiency

Thank you for recognizing the diagnostics and shared-memory comparison. We have addressed the evaluation concerns and added controls to isolate the contribution of dynamic access. References refer to the revised manuscript.

W1/Q1 — Decoding and truncation. The original greedy setup followed DSTP. Our companion reply, Generation Audit: Output Length, Truncation, and Answer Extraction, reports output length, truncation, and extraction failure for three tasks under greedy decoding, and truncation/extraction failure across benchmarks under the revised settings.

To test whether the accuracy advantage persists under recommended decoding, we reran the eight reasoning benchmarks, V*, and OCRBench v2 with Qwen3-VL’s recommended sampling configuration, a 16,384-token limit, and natural EOS. All methods share each benchmark’s settings; evaluation otherwise follows VLMEvalKit.

Full now scores 43.58% on MMMU-Pro and 39.90% on VisualPuzzles. At 20% occupancy, ViMoD scores 28.67/33.39/53.33 on MMMU-Pro/VisualPuzzles/DynaMath, versus the strongest one-shot scores of 14.97/30.57/38.70. Its normalized reasoning average is 83.75%, exceeding the best one-shot average at 40%, 82.45%, with half the visual budget. The advantage therefore persists in raw accuracy and with normalization against the updated Full scores. Tables 1–2 report the results; App. C.1 specifies the protocol. Table 11 lists per-benchmark output limits and truncation/extraction failure for Full, FastV, VisionZip, DivPrune, and ViMoD; compressed methods use 20%. On MMMU-Pro, ViMoD reaches 14.40% truncation / 14.72% extraction failure, versus Full’s 15.78% / 18.09%; on VisualPuzzles, the rates are 17.81% / 18.32%, close to Full’s 16.95% / 17.81%.

W2/Q2 — Full prefill and trained static controls. Unlike training-free pruning, ViMoD trains DART/TRACE offline to learn memory construction and evidence access, adding 0.0546% trainable parameters with a frozen backbone. The teacher is not used at inference. See Secs. 3.4 and 4.1. Table 3 evaluates DynaMath and MMMU-Pro at matched visual occupancy. Applying DivPrune after full-image prefill improves accuracy by 4.28–6.31 points. ViMoD exceeds this baseline by 3.01–5.00 points even when its prompt-text KV is recomputed using only Coarse context, with Fine memory available for subsequent recall.

To isolate dynamic access, we compare continued updates with fixing the first recalled Fine set using the same trained DART/TRACE checkpoint. Continued updates improve accuracy by 7.14–8.21 points. We also train a static policy from the same SFT checkpoint, restricting RL to at most one recall per response with the same training steps. Dynamic access exceeds this Static RL policy by 3.18–7.12 points. Both static controls retain Coarse memory and an active Fine set at matched occupancy, isolating the benefit of updating evidence during decoding. Sec. 4.3 and App. C.3 describe these controls; realized occupancy differs from target by −0.51 to +0.46 percentage points.

W3 — Efficiency including memory costs. Fig. 6 replaces the fixed-batch experiment with a mixed MMMU-Pro/DynaMath workload under recommended decoding, allowing each method its maximum feasible batch within 80 GiB. At 40% occupancy, maximum batches are 51 for ViMoD, 52 for Full, and 56 for one-shot methods, accounting for persistent Fine/Coarse storage. At 40% occupancy, ViMoD reduces the time to complete the same workload by 9.6% versus Full and 12.8–24.2% versus one-shot methods, while achieving higher accuracy than the latter. Total time includes memory processing and routing. Table 14 lists the 40% operating points: ViMoD takes 2,326 s, Full 2,573 s, and one-shot methods 2,668–3,068 s. Sec. 4.4 reports the results; Fig. 8 and App. C.7 additionally examine output limits, resolution, and batch size.

Q3 — Repeated SFT-only score. The three 47.66 entries reused one SFT operating point with 12.93% actual occupancy. Its gate rarely triggered UPDATE, so changing the candidate-selection offset alone did not reach the nominal budgets. We recalibrated the gate and replaced these entries in revised Table 5, matching target occupancy within 0.37 percentage points. SFT-only DynaMath scores are now 52.60/52.20/55.60 at 20%/30%/40%, versus 53.33/57.66/59.20 after RL. The corrected comparison supports higher accuracy with RL at every budget on both evaluated tasks.

Q4 — MRoPE. Coarse keys are pooled after MRoPE, so each Fine key carries its original positional encoding before weighted aggregation. Groups spanning grid locations receive no new coordinate or second rotation. The query’s pre-softmax score against a Coarse key equals the weighted sum of its scores against the member keys. Recalled Fine KV retains its original positions, and generated-text positions continue along the original sequence. App. A.1.2 now provides the implementation and identities.

AuthorsOpenReview ↗

Generation Audit: Output Length, Truncation, and Answer Extraction

Additional data for W1/Q1. This reply accompanies our main response and reports the generation measurements requested by the reviewer. References refer to the revised manuscript.

Original greedy configuration. All three methods use greedy decoding and max_new_tokens=4096. Full retains all visual tokens; DivPrune and ViMoD use 20% budgets, with 9:1 memory for ViMoD.

BenchmarkMethodAvg. tokensTruncationExtraction failure
MMMU-ProFull1,750.023.988%24.971%
MMMU-ProDivPrune2,510.040.983%43.988%
MMMU-ProViMoD2,040.027.977%30.983%
VisualPuzzlesFull2,620.445.377%46.490%
VisualPuzzlesDivPrune2,980.852.226%54.024%
VisualPuzzlesViMoD2,750.647.945%49.401%
DynaMathFull510.33.992%0.180%
DynaMathDivPrune610.74.790%0.040%
DynaMathViMoD530.44.192%0.000%

Revised main-table configurations. The following reproduces Table 11. Each cell is truncation / answer-extraction failure, in percent. Full uses 100% visual tokens; the other methods use 20%, with 9:1 memory for ViMoD. Sampling uses Qwen3-VL’s recommended configuration with a 16,384-token limit; the remaining benchmarks use greedy decoding with the limits listed below. Within each benchmark, all methods use the same decoding settings and output limit.

BenchmarkDecoding / token limitFullFastVVisionZipDivPruneViMoD
We-MathSampling / 16,3840.63 / 0.110.69 / 0.000.63 / 0.170.46 / 0.060.63 / 0.06
DynaMathSampling / 16,3841.94 / 0.002.14 / 0.062.48 / 0.003.05 / 0.021.32 / 0.00
MathVerse CoTSampling / 16,3843.17 / 2.283.12 / 2.393.76 / 2.394.09 / 2.793.63 / 2.59
MathVistaSampling / 16,3841.10 / 0.201.10 / 0.001.60 / 0.000.90 / 0.101.80 / 0.20
MATH-VisionSampling / 16,38412.14 / 0.5610.20 / 0.3311.64 / 0.4913.98 / 0.3010.07 / 0.33
LogicVistaSampling / 16,3845.15 / 4.033.58 / 3.588.95 / 6.719.62 / 6.949.17 / 5.59
VisualPuzzlesSampling / 16,38416.95 / 17.8113.27 / 13.9616.95 / 17.0417.47 / 17.6417.81 / 18.32
MMMU-ProSampling / 16,38415.78 / 18.0912.79 / 14.0516.23 / 17.2917.88 / 18.2114.40 / 14.72
GQAGreedy / 320.00 / —0.00 / —0.01 / —0.00 / —0.00 / —
POPEGreedy / 320.00 / 0.000.00 / 0.000.00 / 0.000.00 / 0.000.00 / 0.00
V*Sampling / 16,3840.00 / 4.571.05 / 6.090.52 / 5.180.52 / 5.420.00 / 4.51
MMEGreedy / 20.21 / 0.000.25 / 0.000.38 / 0.000.38 / 0.000.29 / 0.00
MMBenchGreedy / 1,0240.00 / 0.000.00 / 0.000.02 / 0.000.00 / 0.000.00 / 0.00
OCRBench v2Sampling / 16,3840.54 / —5.42 / —6.32 / —3.93 / —6.62 / —
OCR ENSampling / 16,3840.57 / —3.96 / —4.77 / —2.95 / —5.88 / —
OCR CNSampling / 16,3840.46 / —9.58 / —10.73 / —6.73 / —8.73 / —
ChartQAGreedy / 640.00 / —0.84 / —0.12 / —0.16 / —0.08 / —
DocVQAGreedy / 640.00 / —1.72 / —0.32 / —0.17 / —1.01 / —

Both rates use all evaluated examples as the denominator. Outputs reaching the token limit without an extractable answer count toward both rates; extraction failure therefore includes incomplete outputs. “—” means that the evaluation pipeline does not report a separate extraction-failure metric.

Under recommended decoding, ViMoD's MMMU-Pro truncation and extraction-failure rates are below Full's, while its VisualPuzzles rates are close to Full's. Together with the accuracy results in our main response and Table 1, these measurements show that the reasoning advantage persists under the revised configuration.

AuthorsOpenReview ↗

Complete Revised Qwen Results: Reasoning

For easier reference, we reproduce the complete revised Qwen3-VL-4B reasoning table below. These results supplement our earlier response, Decoding Robustness, Controlled Comparisons, and Efficiency.

W1/Q1 — All eight reasoning benchmarks under recommended decoding. All methods use Qwen3-VL's recommended sampling configuration, a 16,384-token limit, and natural EOS; evaluation otherwise follows VLMEvalKit. From Table 1. Parenthesized percentages are relative to Full; Avg. equally averages the eight normalized scores. DSTP denotes the adaptation using the same DART memory as ViMoD. We-Math, DynaMath, MathVerse, VisualPuzzles, and MMMU-Pro use explicit CoT prompting.

MethodBudgetWe-MathDynaMathMathVerseMathVistaMathVisionLogicVistaVisualPuzzlesMMMU-ProAvg. (%)
Full100%48.38(100.0%)65.45(100.0%)66.09(100.0%)74.40(100.0%)53.19(100.0%)58.61(100.0%)39.90(100.0%)43.58(100.0%)100.00
FastV20%17.43(36.0%)33.25(50.8%)43.73(66.2%)45.90(61.7%)41.88(78.7%)36.24(61.8%)30.57(76.6%)13.35(30.6%)57.81
VisionZip20%19.33(40.0%)35.47(54.2%)45.18(68.4%)47.60(64.0%)40.76(76.6%)34.45(58.8%)27.31(68.4%)13.82(31.7%)57.76
DivPrune20%23.81(49.2%)38.70(59.1%)48.76(73.8%)53.20(71.5%)43.32(81.4%)38.93(66.4%)27.23(68.2%)14.97(34.4%)63.01
DSTP 9:120%32.01(66.2%)49.02(74.9%)52.00(78.7%)61.80(83.1%)45.03(84.7%)43.18(73.7%)32.02(80.3%)17.05(39.1%)72.56
ViMoD 9:120%42.29(87.4%)53.33(81.5%)58.65(88.7%)66.80(89.8%)48.42(91.0%)48.10(82.1%)33.39(83.7%)28.67(65.8%)83.75
FastV30%24.67(51.0%)37.58(57.4%)50.86(77.0%)49.20(66.1%)42.76(80.4%)35.57(60.7%)27.91(69.9%)19.42(44.6%)63.39
VisionZip30%28.10(58.1%)45.61(69.7%)55.43(83.9%)51.10(68.7%)44.11(82.9%)42.95(73.3%)32.36(81.1%)24.05(55.2%)71.60
DivPrune30%31.90(65.9%)46.61(71.2%)56.04(84.8%)56.50(75.9%)45.66(85.8%)41.39(70.6%)29.20(73.2%)17.57(40.3%)70.98
DSTP 9:130%35.00(72.3%)53.07(81.1%)58.02(87.8%)62.20(83.6%)46.61(87.6%)45.19(77.1%)33.99(85.2%)23.87(54.8%)78.69
DSTP 4:130%36.09(74.6%)57.70(88.2%)59.75(90.4%)63.80(85.8%)47.04(88.4%)48.10(82.1%)34.85(87.3%)25.72(59.0%)81.97
ViMoD 9:130%43.71(90.3%)57.66(88.1%)61.37(92.9%)66.40(89.2%)48.95(92.0%)52.13(88.9%)34.85(87.3%)31.79(72.9%)87.73
ViMoD 4:130%45.52(94.1%)59.24(90.5%)62.61(94.7%)69.80(93.8%)49.18(92.5%)50.34(85.9%)35.36(88.6%)35.55(81.6%)90.21
FastV40%28.86(59.7%)43.45(66.4%)54.82(82.9%)54.60(73.4%)45.07(84.7%)37.14(63.4%)28.77(72.1%)23.70(54.4%)69.62
VisionZip40%39.14(80.9%)53.77(82.2%)59.14(89.5%)60.20(80.9%)48.19(90.6%)46.09(78.6%)31.08(77.9%)34.45(79.1%)82.45
DivPrune40%36.57(75.6%)51.02(78.0%)60.00(90.8%)59.90(80.5%)47.43(89.2%)42.51(72.5%)31.34(78.5%)21.97(50.4%)76.94
DSTP 9:140%39.08(80.8%)54.97(84.0%)62.00(93.8%)66.10(88.8%)48.95(92.0%)50.11(85.5%)35.02(87.8%)23.06(52.9%)83.20
DSTP 4:140%39.25(81.1%)59.24(90.5%)62.03(93.9%)67.90(91.3%)49.41(92.9%)49.66(84.7%)35.96(90.1%)28.21(64.7%)86.16
ViMoD 9:140%46.57(96.3%)59.20(90.5%)62.72(94.9%)69.50(93.4%)49.64(93.3%)53.02(90.5%)35.70(89.5%)37.11(85.2%)91.68
ViMoD 4:140%46.67(96.5%)61.80(94.4%)62.94(95.2%)69.90(94.0%)50.39(94.7%)49.66(84.7%)36.73(92.1%)37.80(86.7%)92.29

Under recommended decoding, ViMoD leads the reasoning-suite average at every budget and exceeds the strongest one-shot average with half the visual budget.

Generation statistics are already provided in our reply, Generation Audit: Output Length, Truncation, and Answer Extraction.

Reviewer_tzQpOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes ViMoD, a framework for efficient multimodal reasoning that combines persistent coarse visual memory with on-demand access to fine-grained visual evidence. DART learns adaptive visual grouping and aggregation, while TRACE uses decoding history to retain or replace active evidence. Lightweight modules are trained with the backbone frozen, improving performance under limited active visual-token budgets.

Critical strengths
  1. Practical problem. The paper addresses changing visual evidence requirements during multi-step reasoning. Its analysis also highlights that lower per-token computation does not necessarily translate into lower total inference cost, particularly when compression increases response length.

  2. Strong results under tight visual budgets. At the 20% target budget, ViMoD achieves a normalized reasoning average of 80.33, compared with 57.78 for the strongest one-shot baseline and 72.96 for the adapted DSTP using the same DART memory. Ablations further support the usefulness of repeated recall, learned memory construction, and policy optimization.

Critical weaknesses
  1. Unclear robustness to decoding settings. The Full baseline falls substantially below the reported reference score on MMMU-Pro, and robustness to alternative decoding settings remains untested.

  2. Additional resource costs. ViMoD retains full prefill and increases peak memory from 38.40 to 42.67 GiB. Its efficiency benefits for shorter responses remain insufficiently established.

  3. Limited backbone coverage. All experiments use Qwen3-VL-4B-Instruct, leaving generalization to other model families and scales unverified.

  4. Incomplete discussion of related work. The paper lacks a sufficiently comprehensive review and analysis of closely related studies, such as [1, 2, 3].

[1] Shang Y, Cai M, Xu B, et al. Llava-prumerge: Adaptive token reduction for efficient large multimodal models[C]//2025 IEEE/CVF International Conference on Computer Vision (ICCV). IEEE, 2025: 22857-22867.

[2] Wang H, Yu Z, Spadaro G, et al. Folder: Accelerating multi-modal large language models with enhanced performance[C]//2025 IEEE/CVF International Conference on Computer Vision (ICCV). IEEE, 2025: 23614-23625.

[3] Zhang Y, Fan C K, Ma J, et al. Sparsevlm: Visual token sparsification for efficient vision-language model inference[J]. arXiv preprint arXiv:2410.04417, 2024.

Questions for authors
  1. Do the accuracy and end-to-end efficiency gains persist under the backbone’s recommended decoding settings?

  2. How do the latency and memory trade-offs change with response length, image resolution, and batch size?

  3. Can the authors provide results on another backbone or model scale to assess generalization?

5 comments
AuthorsOpenReview ↗

Decoding Robustness, Resource Trade-offs, and Cross-Backbone Generalization

Thank you for recognizing the motivation and tight-budget results. References refer to the revised manuscript.

W1/Q1 — Recommended decoding. The original greedy setup followed DSTP. We have rerun the eight reasoning benchmarks, V*, and OCRBench v2 with Qwen3-VL’s recommended sampling configuration, a 16,384-token limit, and natural EOS. All methods share each benchmark’s settings; evaluation otherwise follows VLMEvalKit.

Full now scores 43.58% on MMMU-Pro and 39.90% on VisualPuzzles. At 20% visual occupancy, ViMoD achieves 28.67% and 33.39%, versus the strongest one-shot scores of 14.97% and 30.57%. Its normalized reasoning average is 83.75%, exceeding the best one-shot average at 40%, 82.45%, with half the visual budget. The advantage persists under recommended decoding and updated Full normalization. Tables 1–2 report the results, and App. C.1 specifies the protocol. Table 11 lists per-benchmark output limits, truncation, and answer-extraction failure under the main-table configurations. On MMMU-Pro, ViMoD’s rates are 14.40% / 14.72%, below Full’s 15.78% / 18.09%; on VisualPuzzles, they are close to Full’s.

W2/Q2 — Resource costs and workload dependence. Fig. 6 now compares methods at their maximum feasible batch sizes under an 80 GiB budget on a mixed MMMU-Pro/DynaMath workload. At 40% occupancy, maximum batches are 51 for ViMoD, 52 for Full, and 56 for one-shot methods, accounting for persistent Fine/Coarse storage. At 40% occupancy, ViMoD reduces the time to complete the same workload by 9.6% relative to Full and 12.8–24.2% relative to one-shot methods, while achieving higher accuracy than the latter. Table 14 lists these measurements. At 40% occupancy, batch-amortized TTFT is 57 ms for ViMoD, 56 ms for Full, and 42–44 ms for one-shot methods.

Fig. 8 additionally examines workload dependence on MMMU-Pro at 20% occupancy:

  • Response length: With natural EOS, ViMoD/Full throughput is approximately 1.02× at a 128-token output limit, 1.12× at 1–2K, and 1.07× at 16K. Longer decoding initially amortizes memory construction; at larger limits, accumulated text KV and feasible concurrency also affect throughput.
  • Image resolution: Relative throughput rises from 0.98× at 300 visual tokens to 1.45× near 4.9K, reaching 1.84× with upsampled 15.8K-token inputs. Selective access becomes more valuable as visual attention grows, while larger visual banks reduce feasible concurrency for both methods.
  • Batch size: Relative throughput rises from 0.88–0.94× at batches 1–4 to 1.05× at batch 32 and 1.07× at maximum batch, as attention savings contribute more to overall throughput.

ViMoD targets multi-step visual reasoning, where changing evidence needs favor selective access during decoding. For short-answer DocVQA, ViMoD roughly matches Full’s throughput, while one-shot methods are approximately twice as fast. Separately, Table 2 reports higher DocVQA accuracy for ViMoD at 20%: 75.72 versus the best one-shot score of 56.13. This separates the short-answer accuracy–throughput trade-off from savings during multi-step reasoning. Sec. 4.4 and Apps. C.6–C.7 provide the measurements and protocols.

W3/Q3 — Cross-backbone generalization. We have added InternVL3-9B, changing both family and scale from Qwen3-VL-4B while using the same training procedure and training data. Evaluation uses InternVL3’s official decoding configuration and otherwise follows VLMEvalKit. At 20% occupancy, ViMoD achieves a normalized reasoning average of 93.53%, versus 78.93% for the strongest one-shot baseline. It leads both suite averages at every budget, extending the finding across backbone families and scales. Sec. 4.5 and Table 7 present the main results; App. C.10, Tables 17–18, provide all benchmarks. Table 9 compares the backbone-specific DART/TRACE architectures and parameter counts.

W4 — Related work. We have cited and discussed all three suggested papers in Sec. 2. LLaVA-PruMerge adapts token counts through attention sparsity and merges similar tokens. FOLDER analyzes compression-error propagation and uses iterative merging near the vision-encoder output. SparseVLM uses text raters for question-aware, layer-adaptive pruning and recycles discarded information.

ViMoD couples compact visual context with changing evidence access: DART learns groups shared by Coarse aggregation and original Fine recall, while TRACE uses decoding history to predict the next decoding stage’s evidence and decide when to update. Sec. 4.3 separates these contributions: Table 3 tests continued updates, while Table 5 shows DART improves DynaMath/MMMU-Pro accuracy by 8.26/9.42 points at 30% over regular mean pooling, with Fine recall enabled in both variants. We also discuss decoding-time access in DyCoke and joint compression and selection in text-LLM Native Sparse Attention.

AuthorsOpenReview ↗

Complete Revised Qwen Results: Reasoning

For easier reference, we reproduce the complete revised Qwen3-VL-4B reasoning table below. These results supplement our earlier response, Decoding Robustness, Resource Trade-offs, and Cross-Backbone Generalization.

W1/Q1 — All eight reasoning benchmarks under recommended decoding. All methods use Qwen3-VL's recommended sampling configuration, a 16,384-token limit, and natural EOS; evaluation otherwise follows VLMEvalKit. From Table 1. Parenthesized percentages are relative to Full; Avg. equally averages the eight normalized scores. DSTP denotes the adaptation using the same DART memory as ViMoD. We-Math, DynaMath, MathVerse, VisualPuzzles, and MMMU-Pro use explicit CoT prompting.

MethodBudgetWe-MathDynaMathMathVerseMathVistaMathVisionLogicVistaVisualPuzzlesMMMU-ProAvg. (%)
Full100%48.38(100.0%)65.45(100.0%)66.09(100.0%)74.40(100.0%)53.19(100.0%)58.61(100.0%)39.90(100.0%)43.58(100.0%)100.00
FastV20%17.43(36.0%)33.25(50.8%)43.73(66.2%)45.90(61.7%)41.88(78.7%)36.24(61.8%)30.57(76.6%)13.35(30.6%)57.81
VisionZip20%19.33(40.0%)35.47(54.2%)45.18(68.4%)47.60(64.0%)40.76(76.6%)34.45(58.8%)27.31(68.4%)13.82(31.7%)57.76
DivPrune20%23.81(49.2%)38.70(59.1%)48.76(73.8%)53.20(71.5%)43.32(81.4%)38.93(66.4%)27.23(68.2%)14.97(34.4%)63.01
DSTP 9:120%32.01(66.2%)49.02(74.9%)52.00(78.7%)61.80(83.1%)45.03(84.7%)43.18(73.7%)32.02(80.3%)17.05(39.1%)72.56
ViMoD 9:120%42.29(87.4%)53.33(81.5%)58.65(88.7%)66.80(89.8%)48.42(91.0%)48.10(82.1%)33.39(83.7%)28.67(65.8%)83.75
FastV30%24.67(51.0%)37.58(57.4%)50.86(77.0%)49.20(66.1%)42.76(80.4%)35.57(60.7%)27.91(69.9%)19.42(44.6%)63.39
VisionZip30%28.10(58.1%)45.61(69.7%)55.43(83.9%)51.10(68.7%)44.11(82.9%)42.95(73.3%)32.36(81.1%)24.05(55.2%)71.60
DivPrune30%31.90(65.9%)46.61(71.2%)56.04(84.8%)56.50(75.9%)45.66(85.8%)41.39(70.6%)29.20(73.2%)17.57(40.3%)70.98
DSTP 9:130%35.00(72.3%)53.07(81.1%)58.02(87.8%)62.20(83.6%)46.61(87.6%)45.19(77.1%)33.99(85.2%)23.87(54.8%)78.69
DSTP 4:130%36.09(74.6%)57.70(88.2%)59.75(90.4%)63.80(85.8%)47.04(88.4%)48.10(82.1%)34.85(87.3%)25.72(59.0%)81.97
ViMoD 9:130%43.71(90.3%)57.66(88.1%)61.37(92.9%)66.40(89.2%)48.95(92.0%)52.13(88.9%)34.85(87.3%)31.79(72.9%)87.73
ViMoD 4:130%45.52(94.1%)59.24(90.5%)62.61(94.7%)69.80(93.8%)49.18(92.5%)50.34(85.9%)35.36(88.6%)35.55(81.6%)90.21
FastV40%28.86(59.7%)43.45(66.4%)54.82(82.9%)54.60(73.4%)45.07(84.7%)37.14(63.4%)28.77(72.1%)23.70(54.4%)69.62
VisionZip40%39.14(80.9%)53.77(82.2%)59.14(89.5%)60.20(80.9%)48.19(90.6%)46.09(78.6%)31.08(77.9%)34.45(79.1%)82.45
DivPrune40%36.57(75.6%)51.02(78.0%)60.00(90.8%)59.90(80.5%)47.43(89.2%)42.51(72.5%)31.34(78.5%)21.97(50.4%)76.94
DSTP 9:140%39.08(80.8%)54.97(84.0%)62.00(93.8%)66.10(88.8%)48.95(92.0%)50.11(85.5%)35.02(87.8%)23.06(52.9%)83.20
DSTP 4:140%39.25(81.1%)59.24(90.5%)62.03(93.9%)67.90(91.3%)49.41(92.9%)49.66(84.7%)35.96(90.1%)28.21(64.7%)86.16
ViMoD 9:140%46.57(96.3%)59.20(90.5%)62.72(94.9%)69.50(93.4%)49.64(93.3%)53.02(90.5%)35.70(89.5%)37.11(85.2%)91.68
ViMoD 4:140%46.67(96.5%)61.80(94.4%)62.94(95.2%)69.90(94.0%)50.39(94.7%)49.66(84.7%)36.73(92.1%)37.80(86.7%)92.29

Under recommended decoding, ViMoD leads the reasoning-suite average at every budget and exceeds the strongest one-shot average with half the visual budget.

AuthorsOpenReview ↗

Complete Cross-Backbone Reasoning Results: InternVL3-9B

For easier reference, we reproduce the complete InternVL3-9B reasoning results from the revised manuscript below, complementing our earlier response, Decoding Robustness, Resource Trade-offs, and Cross-Backbone Generalization.

W3/Q3 — Another model family and scale. InternVL3-9B uses the same training procedure and training data as Qwen3-VL-4B. Evaluation uses InternVL3's official decoding configuration and otherwise follows VLMEvalKit. ViMoD uses 9:1 memory. From Table 17; parenthesized percentages are relative to Full, and Avg. equally averages the eight normalized scores.

MethodBudgetWe-MathDynaMathMathVerseMathVistaMathVisionLogicVistaVisualPuzzlesMMMU-ProAvg. (%)
Full100%33.62(100.0%)51.92(100.0%)45.13(100.0%)68.40(100.0%)28.52(100.0%)44.07(100.0%)33.22(100.0%)32.72(100.0%)100.00
FastV20%26.03(77.4%)33.25(64.0%)39.52(87.6%)56.10(82.0%)27.99(98.1%)34.68(78.7%)30.48(91.8%)16.94(51.8%)78.93
DivPrune20%27.41(81.5%)32.51(62.6%)38.83(86.0%)55.50(81.1%)27.86(97.7%)37.58(85.3%)31.16(93.8%)14.10(43.1%)78.90
VisionZip20%22.95(68.3%)32.08(61.8%)36.04(79.9%)47.80(69.9%)27.73(97.2%)33.11(75.1%)29.28(88.1%)13.93(42.6%)72.86
ViMoD 9:120%32.57(96.9%)48.66(93.7%)42.21(93.5%)66.90(97.8%)29.64(103.9%)42.95(97.5%)32.19(96.9%)22.25(68.0%)93.53
FastV30%30.29(90.1%)40.08(77.2%)39.90(88.4%)61.60(90.1%)28.06(98.4%)37.81(85.8%)31.76(95.6%)20.29(62.0%)85.94
DivPrune30%30.52(90.8%)39.80(76.7%)40.38(89.5%)60.20(88.0%)27.53(96.5%)38.48(87.3%)31.51(94.9%)18.73(57.2%)85.11
VisionZip30%29.52(87.8%)38.96(75.0%)38.88(86.2%)53.20(77.8%)27.70(97.1%)34.23(77.7%)32.19(96.9%)17.69(54.1%)81.57
ViMoD 9:130%33.62(100.0%)51.32(98.8%)43.48(96.3%)68.30(99.9%)29.38(103.0%)46.31(105.1%)33.22(100.0%)26.24(80.2%)97.92
FastV40%32.99(98.1%)45.01(86.7%)42.51(94.2%)63.10(92.3%)28.82(101.1%)42.95(97.5%)33.82(101.8%)24.86(76.0%)93.44
DivPrune40%32.19(95.7%)46.65(89.8%)42.01(93.1%)62.70(91.7%)28.49(99.9%)42.28(95.9%)33.90(102.0%)22.72(69.4%)92.21
VisionZip40%32.76(97.4%)43.49(83.8%)41.62(92.2%)58.20(85.1%)28.45(99.8%)38.70(87.8%)32.28(97.2%)24.51(74.9%)89.77
ViMoD 9:140%35.14(104.5%)52.38(100.9%)44.57(98.8%)69.70(101.9%)29.51(103.5%)46.76(106.1%)33.22(100.0%)28.96(88.5%)100.52

ViMoD leads the normalized reasoning average at all three budgets, supporting effectiveness across backbone families and scales.

Comments on the paper

2 comments

We thank all three reviewers for their constructive feedback. We have added evaluations under recommended decoding, fixed-memory efficiency comparisons, trained static controls, direct evidence-prediction and cross-teacher analyses, readout-layer ablations, and cross-backbone results. These experiments support ViMoD’s advantages in multi-step visual reasoning and clarify the contribution of dynamic evidence access. In the revised PDF, added text and the labels of new or updated tables and figures are marked in blue for easy reference.

AuthorsOpenReview ↗

Follow-up on Our Author Responses

Dear Reviewers,

Thank you again for taking the time to review our paper and share your feedback. As the discussion period draws to a close, we’d appreciate hearing whether our responses have addressed your concerns. If any questions remain, we’d be happy to clarify them.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.