acceptodds
Withdrawn from ICLR 2027

From Visual Token Reduction to Realized Edge Efficiency: A Stage-Wise Study with CIVIC

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Reducing visual tokens is a common strategy for accelerating vision–language models, yet token counts and operation counts do not establish faster batch-one inference on edge hardware. Compression after visual encoding leaves expensive backbone layers untouched, while early compression may be undone at the multimodal interface or offset by aggregation overhead. This paper presents Compact Inference for Vision-Language Integrated Compression (CIVIC), which aggregates visual states at an intermediate encoder layer and carries the compact representation through the remaining visual blocks, projector, and language-model prefill. A configurable visual budget controls aggregation, and text-aligned distillation transfers predictions across unequal visual-token sequences. The evaluation adapts CIVIC to Qwen3-VL-4B-Instruct and profiles batch-one inference on Jetson AGX Orin, tracing interface lengths and timing aggregation, the encoder, the projector, and prefill separately. The median final prefix falls from 816 to 470 tokens; derived response-length-standardized latency decreases by 26.6%, with lower measured encoder and prefill stage times. In a fixed-configuration control, restoring dense downstream tokens increases derived latency, whereas optional visual key–value pooling adds device cost. CIVIC reduces derived latency on the four scored benchmarks and within each Video-MME duration group. Task scores show benchmark-dependent quality trade-offs. Together, the stage traces and fixed-budget controls connect persistent compactness with realized edge acceleration despite the measured aggregation overhead.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Vision-Language Models, Large Multimodal Models, Efficient Inference, Visual Token Compression, Hardware-Aware AI, End-to-End Acceleration, Multimodal Reasoning, Model Compression

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_XTcxOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper introduces CIVIC, a method that aggregates tokens within the intermediate layers of a vision encoder. These compact representations are then propagated through subsequent vision layers, the projector, and the language model’s prefill phase, complemented by text-alignment distillation. The authors evaluate the approach on Qwen3-VL-4B-Instruct using the Jetson AGX Orin platform, measuring both raw execution time and normalized latency. While latency is reduced, performance varies significantly by task: scores decline on MMMU and MathVision, improve on RealWorldQA, and remain largely stable on Video-MME.

Critical strengths
  1. The paper focuses on actual hardware costs, measuring the overheads associated with encoding, projection, prefill, and aggregation, rather than relying solely on token counts or theoretical computational complexity to estimate speedups.

  2. Results on RealWorldQA and Video-MME (Table 2) demonstrate favorable quality-latency trade-offs; furthermore, comparisons involving the restoration of dense tokens and visual KV pooling provide insight into the sources of these practical gains.

Critical weaknesses
  1. In Table 2, the MMMU score drops from 0.714 to 0.285, and the MathVision score from 0.480 to 0.305, representing declines of 42.9 and 17.5 percentage points, respectively. In this context, the reduction in latency is insufficient to justify a superior quality-efficiency trade-off. The paper lacks task-specific quality-latency curves and fails to specify the extent to which acceleration is maintained when performance approaches that of the dense model.

  2. The evaluation is limited to a single model backbone, one device, and a batch-size-of-one configuration. Given the authors' emphasis that practical gains depend on model interfaces and the computational costs of hardware operators, this scope is insufficient to demonstrate generalizability. Furthermore, the systematic comparison of aggregation layer placement is inadequate, making it difficult to determine whether the 16th layer represents an optimal balance between information retention and computational savings.

Questions for authors
  1. Could you provide native score-vs.-latency curves for MMMU and MathVision across multiple compression budgets? When quality degradation is kept within a narrow range, is there still a significant speedup compared to the dense model and key baselines?

  2. Why is there a marked performance degradation on MMMU and MathVision, while RealWorldQA shows improvement and Video-MME remains largely stable? Could you explain these discrepancies through error analysis, considering factors such as the location of compression, compression intensity, and the use of distillation?

  3. Could you include representative experiments using a different model configuration or a different type of hardware, reporting both quality and latency, to determine whether the observed gains are dependent on the current setup? If this is not feasible, please clearly define the scope and limitations of your conclusions.

Reviewer_pwvQOpenReview ↗
Rating
1
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

CIVIC aggregates visual tokens at vision layer 16 of Qwen3-VL-4B into a configurable set of learned-anchor states, carries them through the remaining encoder, projector, and LM prefill, and distills a dense teacher at text positions. It profiles batch-one inference on Jetson AGX Orin and reports stage times plus a response-length-standardized latency.

Critical strengths
  • The central observation — token/FLOP reduction does not imply device-latency reduction — is well taken, and the stage traces support it.
  • The dense-restoration control cleanly isolates the value of persistent compactness; the negative result for visual-KV pooling is reported honestly.
  • The Qwen3-VL interface details (grid, merger groups, placeholders, DeepStack) are a useful engineering contribution.
  • The paper is candid about coverage and cross-method comparability.
Critical weaknesses
  1. Quality collapse. MMMU 0.714 → 0.285 and MathVision 0.480 → 0.305 is not a "trade-off"; it makes the model unusable for reasoning. RealWorldQA rising 0.625 → 0.835 while MMMU collapses strongly suggests an evaluation/parsing artifact. This must be diagnosed before efficiency claims are meaningful.
  2. Derived metric. The headline 26.6% comes from Eq. 8, not measured wall-clock latency. There is no measured end-to-end latency at fixed output length, no run-to-run variance, and no energy. "Realized edge acceleration" is therefore not established.
  3. Limited novelty. Intermediate-layer query aggregation plus distillation is close to TokenLearner/Perceiver/LLaVA-Mini/TokenPacker; the contribution is mainly an interface adapter and a single-device measurement study.
  4. Unfair comparisons. CIVIC uses per-benchmark checkpoints while DiffRate/DynamicViT/HiDrop use pooled training; OOM and timing coverage differ across methods. The paper concedes this, which undercuts the "largest latency reduction" claim.
  5. Configuration ambiguity. §3.4 says the headline setting uses Ce = 256; §4 and Appendix A say 128. A 470-token final prefix is also hard to reconcile with a 128-state local budget. State the exact configuration for each table.
Questions for authors
  1. Is the MMMU/MathVision drop due to answer parsing or genuine information loss? Provide per-category results and a qualitative example.
  2. What is the measured end-to-end latency at fixed max-new-tokens, with variance?
  3. Which budget (128 or 256) produced each reported number?
  4. How does CIVIC compare at iso-accuracy with VisionTrim/ZOOPrune?
Reviewer_CpqyOpenReview ↗
Rating
2
Confidence
2
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

Summary

  • This paper considers on device acceleration of one batch VLM inference. It compress all the way from vision encoder intermediate layer to pre-fill stage, targeting at "persistent compactness".
  • This paper propose an integrated pipeline of many previously used acceleration component scattered in different locations of VLM, combining them into one acceleration pipeline.
Critical strengths

Strength

  • This paper consider a potential useful application: low latency inference of batch one VLM on edge device.
Critical weaknesses

Weakness

  • The empirical advantage of the proposed approach is not quite significant. In Fig 1. A, the proposed approach seems to achieve a different work vs latency trade-off compared with DiffRate, but does not dominate it. Is it possible to tune DiffRate to achieve the same point as proposed approach? Note that DiffRate is proposed in 2023. In Fig 2. B, the number of proposed approach looks quite close to DiffRate. Besides, is it possible to combine the advantage of different methods? i.e., the vision encoder of the proposed approach, projector output of DiffRate and prefill of AutoPrune? In Fig 2. C, does this figure really show overall advantage of proposed approach? In Tab 2, why the latency is only reported for base (dense) and proposed? Are there other valid baselines? In Tab 2, is a task score from 0.71 -> 0.28 still usable?
  • The proposed method seems to be an effort of system engineering. In Fig 2, it is shown that the proposed method combines several post-hoc runtime speed up method and making them into one inference pipeline. Then the main contribution seems to be the integration of all the components. Besides, the proposed components seem to require quite complex training and distillation. It is not clear whether the advantage of the proposed approach really comes from effective method, or just additional complexity.
Questions for authors

Questions

  • see weakness

Justification

  • Overall I find this paper a solid application for accelerating one specific model (Qwen3-VL-4B-Instruct) for one scenario ( batch-one inference on Jetson AGX Orin) and target (persistent compactness). However, I am not really sure how this paper would interest the general machine learning community. The technical contribution of this paper is less significant considering it looks like a combination of previous acceleration methods (Fig 2). The significance of result is limited by its nitch application. Very limited end to end result is presented and it remains unclear whether the proposed approach will harm task performance significantly or not (Tab 2).
  • I have so many questions about this paper and I think its quality is clearly below the bar of ICLR. The reason why I am giving a weak, not clear rejection is I do not have direct research experience in this area.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.