- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 3(+1)
- Presentation
- 2
- Soundness
- 3(+1)
Read review
Paper summary
This paper proposes ViMoD, a framework for dynamic visual token compression in multi-step multimodal reasoning. By combining content-adaptive visual memory (DART) with history-aware evidence routing (TRACE), the method enables selective access to fine-grained visual information during decoding. The motivation is well supported, and the experimental results demonstrate substantial improvements across multiple benchmarks. My main concerns lie in the experimental validation, particularly regarding efficiency trade-offs, design choices, and the robustness of the method. Further clarification or additional experiments addressing these concerns would strengthen the paper's rigor and persuasiveness and increase my confidence in its contributions.
Critical strengths
(1) The paper identifies a limitation of existing visual token compression methods, which typically retain a fixed visual context throughout decoding. Figures 1–2 provide empirical evidence that visual evidence requirements change across reasoning stages, motivating dynamic evidence access.
(2) DART adaptively adjusts token grouping and compression granularity based on image content while preserving access to the original Fine tokens. This allows the model to retrieve fine-grained visual evidence when needed without repeatedly encoding the image.
(3) TRACE uses decoding history to predict upcoming visual evidence needs and employs a HOLD/UPDATE mechanism to retain or replace active Fine-token groups. This allows visual evidence access to change during reasoning rather than relying on a fixed compressed context.
Critical weaknesses
(1) ViMoD retains the full Fine KV cache alongside additional Coarse KV representations, resulting in higher peak GPU memory usage than the uncompressed backbone. While the method demonstrates reduced inference latency, could the authors evaluate its accuracy–throughput trade-off under a fixed GPU memory budget, allowing each method to use its maximum feasible batch size? This would help clarify whether the computational benefits translate into practical efficiency gains under comparable memory constraints.
(2) TRACE extracts hidden states from backbone layers 27, 31, and 35, but the paper does not explain the rationale behind this selection. Since these representations directly determine the temporal routing state, the choice of layers may substantially affect evidence prediction. Although prior studies have provided some insights into layer selection, we would appreciate further justification for why this specific configuration is suitable for the proposed task. Additionally, it would be valuable to investigate how different teacher models or backbone model families affect the final performance, providing further evidence of the method's robustness and generalizability.
(3) TRACE aims to predict upcoming visual evidence needs by integrating decoding history. However, its effectiveness is primarily demonstrated through downstream task accuracy, without directly evaluating the quality of the predicted evidence regions. Although Fig. 8 qualitatively illustrates the dynamic evidence access process, it remains unclear how accurately TRACE predicts future evidence requirements. We suggest that the authors provide quantitative evaluations (e.g., evidence prediction precision based on annotated data) to validate TRACE's predictive capability.
Questions for authors
Please refer to the Critical Weaknesses section.
4 comments
Key Results: Fixed-Memory Efficiency and Direct Evidence Prediction
For easier reference, we include the key results from our revised manuscript below, complementing our earlier response, Evidence Prediction, Readout Design, and Efficiency Across Backbones.
W1 — Accuracy and efficiency under the same memory budget. Each method uses its maximum feasible batch under an 80 GiB budget on the same mixed MMMU-Pro/DynaMath workload. Compressed methods use 40% visual occupancy; Full retains all visual tokens. Accuracy averages Full-normalized task scores. From Table 14:
| Method | Maximum batch | Total time (s) | Normalized accuracy (%) |
|---|---|---|---|
| Full | 52 | 2,573 | 100.0 |
| FastV | 56 | 2,828 | 60.4 |
| VisionZip | 56 | 2,668 | 79.8 |
| DivPrune | 56 | 3,068 | 64.4 |
| ViMoD | 51 | 2,326 | 87.5 |
ViMoD completes the workload faster than all compared methods while retaining substantially more accuracy than one-shot compression. Its smaller maximum batch already accounts for persistent Fine/Coarse storage.
W2 — Readout-layer selection. On 500 annotated MMMU-Pro examples unseen during training, all layouts use the same supervised training setup without RL. Values are mean ± standard deviation over three seeds; layer indices are zero-based. From Table 13:
| Layout | Layers | Evidence F1 (%) | New-evidence recall (%) | Answer accuracy (%) | Occupancy (%) |
|---|---|---|---|---|---|
| Early | 3, 7, 11 | 11.26 ± 0.03 | 14.93 ± 0.07 | 25.60 ± 2.00 | 20.30 |
| Middle | 15, 19, 23 | 11.22 ± 0.03 | 15.03 ± 0.01 | 20.60 ± 1.64 | 20.20 |
| Late | 27, 31, 35 | 11.67 ± 0.01 | 15.39 ± 0.03 | 32.20 ± 2.31 | 20.10 |
| Mixed | 11, 23, 35 | 11.27 ± 0.02 | 15.22 ± 0.01 | 22.40 ± 1.64 | 20.20 |
Late readouts achieve both the highest evidence-prediction F1 and answer accuracy at comparable occupancy, directly supporting the chosen configuration.
W3 — Direct evaluation of upcoming evidence. On 100 unseen MMMU-Pro trajectories, a separate, stronger evaluation teacher, DeepSeek-V4.1-Flash, annotates evidence for the next decoding stage. All methods use a common 10% Fine-token cap. Current means the current decoding stage; Future means the next decoding stage. From Table 4:
| Predictor | SFT target | Fine used (%) | Precision (%) | Recall (%) | F1 (%) |
|---|---|---|---|---|---|
| Attention | — | 9.26 | 7.64 | 5.40 | 6.33 |
| Random DART | — | 9.33 | 11.62 | 9.49 | 10.45 |
| SSM (SFT) | Current | 9.33 | 13.25 | 13.02 | 13.13 |
| SSM (SFT) | Future | 9.33 | 15.88 | 15.83 | 15.86 |
| Window MLP (SFT) | Current | 9.33 | 13.41 | 12.98 | 13.19 |
| Window MLP (SFT) | Future | 9.33 | 15.06 | 15.42 | 15.24 |
| SSM (SFT+RL) | Future | 9.36 | 22.88 | 23.19 | 23.03 |
Next-stage supervision improves both architectures; recurrent SSM outperforms the parameter-matched MLP with Future supervision, and RL further improves F1 to 23.03%. Thus, the evaluation measures evidence prediction directly, beyond downstream accuracy.
For annotation-reference consistency, Table 12 evaluates the same TRACE predictions against both teachers:
| Evaluation | Precision (%) | Recall (%) | F1 (%) |
|---|---|---|---|
| Qwen3.8 vs. DeepSeek-V4.1-Flash | 58.41 | 53.13 | 55.65 |
| TRACE vs. Qwen3.8 | 22.12 | 21.73 | 21.92 |
| TRACE vs. DeepSeek-V4.1-Flash | 22.88 | 23.19 | 23.03 |
Both teachers independently annotate identical prefixes using the same prompt, without future reasoning or the final answer. TRACE's similar F1 against the two references supports consistent evidence-prediction quality.
W2 — Cross-backbone generalization. InternVL3-9B uses the same training procedure and data as Qwen3-VL-4B, its official decoding configuration, and otherwise VLMEvalKit evaluation. Full retains all tokens; compressed methods use 20% occupancy and ViMoD uses 9:1 memory. From Table 7:
| Method | DynaMath (%) | MMMU-Pro (%) | LogicVista (%) | MathVista (%) |
|---|---|---|---|---|
| Full, 100% | 51.92 | 32.72 | 44.07 | 68.40 |
| FastV | 33.25 | 16.94 | 34.68 | 56.10 |
| DivPrune | 32.51 | 14.10 | 37.58 | 55.50 |
| VisionZip | 32.08 | 13.93 | 33.11 | 47.80 |
| ViMoD 9:1 | 48.66 | 22.25 | 42.95 | 66.90 |
ViMoD leads every compressed baseline on these four tasks. Its normalized reasoning average is 93.53%, versus the best one-shot average of 78.93%, supporting generalization across model families and scales.
We would appreciate your assessment of whether these direct measurements address your concerns about practical efficiency, layer selection, and predictive capability.
Thank you again for taking the time to review our paper and for the constructive comments in your review. As the discussion period draws to a close, we’d be grateful to hear whether our responses and additional experiments have addressed the concerns you raised. If there’s anything you’d like us to clarify, we’d be happy to do so.
I apologize for the delayed response. We have been handling a large volume of reviews recently, and your manuscript covers several aspects that required additional time for careful consideration. Thank you for your detailed response and clarifications, particularly regarding the generalization capability and the quality of the supporting evidence. My previous concerns have been largely addressed, and I will increase my rating accordingly.
Evidence Prediction, Readout Design, and Efficiency Across Backbones
Thank you for recognizing the motivation and design of ViMoD. We have addressed your concerns with the following evaluations. References refer to the revised manuscript.
W1 — Fixed-memory efficiency.
Fig. 6now compares methods at their maximum feasible batch sizes under an 80 GiB budget on a mixed MMMU-Pro/DynaMath workload. At 40% occupancy, maximum batches are 51 for ViMoD, 52 for Full, and 56 for one-shot methods, accounting for Fine and Coarse storage. At 40% visual occupancy, ViMoD reduces the time to complete the same workload by 9.6% relative to Full and 12.8–24.2% relative to one-shot methods, while achieving 87.5% normalized accuracy versus their 60.4–79.8%. This demonstrates a better accuracy–efficiency trade-off under the same memory budget.Sec. 4.4andApp. C.6provide the measurements and protocol;Table 14lists the 40% operating points: 2,326 s for ViMoD, 2,573 s for Full, and 2,668–3,068 s for one-shot methods.W2 — Readout layers. Zero-based decoder layers 27, 31, and 35 sample evenly spaced depths in the decoder’s final third. Prior studies identify distinct reasoning and answer-decoding stages across layers and find later hidden states informative about subsequent response content, motivating the fusion of multiple late readouts. Our added ablation in
Table 13directly compares Early, Middle, Late, and Mixed layouts under the same training setup. Late achieves the highest evidence-prediction F1, 11.67% versus 11.22–11.27%, and answer accuracy, 32.20% versus 20.60–25.60%, at matched visual occupancy.App. C.4discusses the motivation and evaluation.W2 — Backbone generalization. We have added InternVL3-9B, changing both family and scale from Qwen3-VL-4B while using the same training procedure and training data. Evaluation uses InternVL3’s official decoding configuration and otherwise follows VLMEvalKit. At 20% occupancy, ViMoD achieves a normalized reasoning average of 93.53%, versus 78.93% for the strongest one-shot baseline. It leads both the reasoning and general-understanding averages at every evaluated budget, extending the main finding to a larger backbone from another family.
Sec. 4.5andTable 7present the main results;App. C.10,Tables 17–18, provide all benchmarks.Table 9compares the backbone-specific DART/TRACE architectures and parameter counts.W3 — Evidence-prediction quality.
Table 4directly evaluates predicted Fine-token sets against evidence for the next decoding stage on 100 unseen MMMU-Pro trajectories, annotated by a separate, stronger evaluation teacher, DeepSeek-V4.1-Flash. Predictors receive only the generated prefix and available history. Under a common 10% Fine-token cap, TRACE achieves 22.88% precision, 23.19% recall, and 23.03% F1, compared with F1 scores of 10.45% for random DART selection and 6.33% for attention-based selection. These results directly demonstrate TRACE’s ability to anticipate the next decoding stage’s evidence. Next-stage supervision improves evidence-prediction F1 by 2.73 points over current-stage supervision. With next-stage supervision, SSM exceeds the parameter-matched MLP by 0.62 points, and RL further improves SSM F1 from 15.86% to 23.03%. These comparisons support anticipating upcoming evidence, retaining recurrent history, and optimizing the access policy with task feedback.Sec. 4.2provides a complementary downstream comparison: with the same 9:1 DART memory and frozen backbone, TRACE exceeds the DSTP adaptation by 8.48–11.19 points in normalized reasoning average.Sec. 4.3analyzes evidence prediction, with evaluation details inApp. C.3. InTable 12, both teachers independently annotate the same prefixes using the same prompt and only the image, question, and available reasoning prefix, without future reasoning or the final answer. Their evidence-set F1 is 55.65%. The same TRACE predictions achieve 21.92% F1 against Qwen3.8 and 23.03% against DeepSeek, showing similar prediction quality across the two annotation references.