acceptodds
Under review as a conference paper at ICLR 2027

Attend, Transform, or Silence: Operator-Level Visual Skipping for Efficient Multimodal LLM Inference

Abstract

Multimodal large language models (MLLMs) increasingly process long visual-token sequences, making decoder prefill computationally expensive. Existing acceleration methods typically remove visual tokens or skip their updates at the layer level, but these coarse strategies may discard fine-grained evidence or suppress useful operators together with redundant ones. We study visual computation from an answer-observable perspective and find that late visual-token updates can remain large while having little effect on answer representations and generated outputs. By decomposing each Transformer layer into attention and FFN operators, we further show that useful visual computation is operator-dominant and varies across layers. We therefore propose OpSkip, an operator-level visual-token skipping framework that preserves the full visual sequence while selectively bypassing visual attention, FFN, or both. Experiments across four MLLMs and 10 VQA benchmarks reduce computation to 55%–67% of the vanilla TFLOPs while retaining 95.0%–99.5% average performance. OpSkip achieves the highest average retention on three of the four architectures and also provides consistent prefill and time-to-first-token speedups.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Multimodal Large Language Models, Efficient Inference, Visual Token Redundancy, Operator-Level Skipping, Training-Free Acceleration

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_2PUrOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes OpSkip, a training-free framework for reducing the decoder prefill cost of multimodal LLMs through operator-level visual-computation skipping. The authors identify answer-silent visual updates: visual representations can undergo substantial changes in later Transformer layers while having limited influence on answer generation. Using a small calibration set, OpSkip measures output-distribution KL divergence under visual-operator interventions to determine whether to skip visual attention, FFN, or both in selected layers. The resulting policy preserves the complete visual-token sequence and leaves text-token computation unchanged. Experiments across versatile architectures and 10 benchmarks show that OpSkip uses 55%–67% of baseline TFLOPs while retaining 95.0%–99.5% of baseline performance on average. Additional experiments examine operator selection, calibration sensitivity, inference latency, and compatibility with token pruning.

Critical strengths
  • The paper is well-motivated and provides an interesting observation that substantial visual hidden-state updates do not necessarily translate into meaningful contributions to answer generation. The detailed analysis of VUM, LRC and IRC together with operator-level analysis provides a reasonable motivation for the proposed method and offers a useful perspective on efficient MLLM inference.

  • The method is evaluated across multiple MLLM architectures and 10 VQA benchmarks, with comparisons against representative visual-token pruning and layer-skipping approaches. The results demonstrate that selectively skipping operators can effectively preserve model accuracy more effectively and is comparable to coarse-grained skipping strategies at similar computational budgets.

Critical weaknesses
  • OpSkip derives a fixed per-model execution policy from a small calibration set, but the reliability of these discrete layer and operator decisions remains insufficiently studied. Figure 5 provides useful evidence by comparing mixed-domain and ChartQA calibration. However, this comparison is limited, and substantial task-specific differences emerge under aggressive budgets. Appendix C.1 varies calibration sample count but does not report variability across independently sampled calibration sets at a fixed size. Therefore, neither the stability of the selected policies nor the uncertainty in their downstream performance is established. This matters because the method relies on transferring a calibration-derived policy across tasks.

  • OpSkip retains visual K/V at every layer and primarily saves computation when visual tokens are first processed. Its practical benefit therefore depends on the balance between new-image prefill, decoding, and accumulated cache costs. More importantly, low influence on the current answer does not establish low influence on later questions that revisit earlier images. New images in later turns can also attend to preceding dialogue, changing the computational setting underlying the single-turn analysis. The absence of such validation limits the demonstrated scope of the method for interactive multimodal inference.

Questions for authors
  • The risks RA(l)R_A(l), RF(l)R_F(l), and RAF(l)R_{AF}(l) are measured by intervening on one layer while leaving the remaining layers unchanged, whereas the deployed policy modifies multiple layers simultaneously. Although the final benchmark results validate the selected policies, how reliable are these independently measured risks under joint execution? Is there any evidence quantifying cross-layer interactions or comparing the current selection rule with sequential calibration that re-estimates risks after each selected modification, at matched final computational budgets?

  • Were the calibration experiments repeated with independent random subsets at a fixed sample count? Beyond the mixed-domain versus ChartQA comparison in Figure 5, what evidence is available regarding cross-domain variation in operator risks, selected layers, and operator assignments? Please report any available seed-level performance variation and clarify whether the observed differences persist under matched computational budgets.

  • Is the fixed policy intended to generalize to interleaved multi-turn conversations with newly introduced images and later references to earlier images? If so, what evidence supports that claim, given that policy calibration measures only the first answer-token distribution? If such evaluation is unavailable, please clarify the intended scope of the generalization claims.

5 comments
AuthorsOpenReview ↗

Response 1

Dear Reviewer 2PUr,

Thank you very much for your recognition of our work and for your insightful questions. Following your suggestions, we have revised the paper (new content is highlighted in blue). Specifically, we have added an analysis of cross-layer interactions and sequential recalibration, experiments with independent draws and cross-domain calibration, and an evaluation on multi-turn, multi-image dialogue. Your questions have helped us substantially broaden and solidify the paper. We address each of them below.

Q1: The risks RA(l)R_A(l), RF(l)R_F(l), and RAF(l)R_{AF}(l) are measured by intervening on one layer while leaving the remaining layers unchanged, whereas the deployed policy modifies multiple layers simultaneously. Although the final benchmark results validate the selected policies, how reliable are these independently measured risks under joint execution? Is there any evidence quantifying cross-layer interactions or comparing the current selection rule with sequential calibration that re-estimates risks after each selected modification, at matched final computational budgets?

Thank you for raising this insightful question. Following your suggestion, we have added an analysis of cross-layer interactions (Page 22, §C.6). Because each layer has three possible actions and the number of combinations grows exponentially with the number of layers, fully characterizing cross-layer interactions is infeasible. We therefore compare two layer-selection schemes:

  1. Independent is OpSkip itself: the RAR_A and RFR_F of each layer are measured individually while all other layers remain unchanged, and the BB layers with the lowest RA+RFR_A + R_F are selected in a single ranking pass.
  2. Sequential performs greedy layer-by-layer recalibration: each time a layer is added, the RA+RFR_A + R_F of every remaining layer is re-measured with all already-selected layers skipped, and the layer with the smallest value is added, until the FLOPs match those of OpSkip at B=28B=28.

Since Sequential must re-measure all candidate layers at every step, its calibration takes 39 minutes (versus about 1 minute for OpSkip). We compare the first-token KL when the entire policy is executed jointly (Joint KL) with the sum of the KLs obtained when each selected layer is executed alone (Σ single-layer KL, i.e., an additive reference based on isolated interventions). Their difference reflects the magnitude of cross-layer interactions. Both schemes are calibrated on the same 48 calibration samples of Qwen3-VL-8B. At every budget, the joint KL does not exceed the sum of single-layer KLs; that is, summing independent risks does not underestimate the error of joint execution, although the two quantities gradually approach each other as the budget increases. The layers selected by Sequential have a much larger sum of single-layer risks (0.412) than those selected by Independent (0.262), yet their joint KL is not lower (0.191 vs. 0.174).

BSelectionJoint KL ↓Σ single-layer KLAgreement (%) ↑
20Independent0.0450.07293.8
28Independent0.1740.26289.6
32Independent0.4060.43981.2
28Sequential0.1910.41283.3
AuthorsOpenReview ↗

Response 2

Q1 (continued).

We further compare the downstream performance of Independent (OpSkip) and Sequential at the FLOPs corresponding to B=28B=28. Since Reviewer 2uFH (Q3) also reminded us to rule out the effect of computational cost, we additionally include three types of controls: skipping only one type of operator (Attention-only and FFN-only), freezing only (Frozen), and two single-pass rules that replace only the ranking score:

  1. Action-risk ranks layers solely by the risk of the operator that is actually skipped, min⁡(RA,RF)\min(R_A, R_F), without counting the risk of the other operator.
  2. Cost-aware ranks layers by this risk divided by the FLOPs saved by the corresponding action.

All methods are calibrated with seed 101 (48 calibration samples newly drawn following the recipe in the paper) and evaluated on the holdout set, which consists of the 42,136 questions (out of the original 44,605) that remain in the ten evaluation tasks after removing all calibration questions of seeds 101–105 and all questions sharing an image with them.

MethodFLOPs ↓GQATextVQAMMEMMBMMMUPOPESQAAI2DOCRBVizWizRetention (%) ↑
Vanilla1.00061.280.12383.384.853.089.194.483.882.869.4100.00
OpSkip0.5660.576.42149.581.951.988.890.180.576.870.396.42
Attention-only0.5658.262.21987.779.050.387.587.377.926.662.985.06
FFN-only0.6752.657.51566.576.744.879.678.868.943.863.779.82
Frozen0.5658.868.31905.376.849.387.384.375.562.366.889.37
Action-risk0.5660.276.02111.481.751.689.190.980.479.168.996.26
Cost-aware0.5760.276.52161.280.251.289.190.880.779.569.596.44
Sequential0.5659.975.92034.681.650.588.687.879.576.869.795.00

At matched FLOPs, OpSkip (96.42%) clearly outperforms the fixed-operator policies with independently selected layers (Attention-only 85.06%, Frozen 89.37%, and FFN-only only 79.82% despite its higher cost). Meanwhile, Action-risk and Cost-aware remain within 0.2 points of OpSkip, and Sequential is lower (95.00%). These results indicate that the gain comes from per-layer operator selection rather than from the ranking rule or per-step re-estimation.

Q2: Were the calibration experiments repeated with independent random subsets at a fixed sample count? Beyond the mixed-domain versus ChartQA comparison in Figure 5, what evidence is available regarding cross-domain variation in operator risks, selected layers, and operator assignments? Please report any available seed-level performance variation and clarify whether the observed differences persist under matched computational budgets.

We sincerely thank you for this valuable feedback. Following your suggestion, we performed five non-overlapping unlabeled draws on each of four calibration domains and compared the resulting policies at matched FLOPs (Page 23, §C.7):

Holdout denotes the average ten-task retention (%). TextVQA, DYSM, and ChartQA report accuracy (%).

66% FLOPs57% FLOPs
Calib. domainFLOPsHoldoutTextVQADYSMChartQAFLOPsHoldoutTextVQADYSMChartQA
Vanilla1.00100.0080.148.185.41.00100.0080.148.185.4
Original0.66100.12 ± 0.5179.5 ± 0.444.6 ± 3.482.4 ± 2.60.5795.33 ± 1.6574.8 ± 2.132.3 ± 4.258.7 ± 11.6
TextVQA0.6699.68 ± 0.5379.4 ± 0.642.5 ± 4.480.8 ± 3.10.5892.95 ± 3.2475.1 ± 3.327.9 ± 4.548.5 ± 11.4
DYSM0.6699.87 ± 0.4579.2 ± 0.346.2 ± 0.782.0 ± 3.60.5793.44 ± 1.7573.9 ± 1.532.4 ± 3.953.6 ± 9.7
ChartQA0.6699.94 ± 0.7179.0 ± 0.445.0 ± 2.283.5 ± 1.30.5893.83 ± 2.9074.1 ± 2.031.0 ± 6.365.2 ± 6.3
AuthorsOpenReview ↗

Response 2

Q2 (continued).

Layer Jaccard is the mean pairwise Jaccard index between the selected layer sets, and Same operator is the fraction of commonly selected layers that receive the same action.

FLOPsComparisonLayer JaccardSame operator
0.66Within Original0.7977%
Within TextVQA0.7276%
Within DYSM0.8174%
Within ChartQA0.7066%
Across domains0.7473%
0.57Within Original0.8479%
Within TextVQA0.8077%
Within DYSM0.9182%
Within ChartQA0.7378%
Across domains0.7879%

With the sample count fixed at 48, the layers selected across five independent draws overlap heavily for every model (Layer Jaccard ≥0.82\geq 0.82 at B=20B=20, with a retention standard deviation of at most 0.29 points, Table 14 of the revised paper). Under the two compute budgets, the cross-domain Layer Jaccard is 0.74 and 0.78, respectively, both within the corresponding within-domain ranges (0.70–0.81 and 0.73–0.91). This indicates that the low-risk operators identified by OpSkip transfer across datasets.

At matched FLOPs, diverse calibration sets yield stable overall performance, and a small number of in-domain unlabeled samples can further improve performance on target domains that differ substantially from the calibration data. Under aggressive budgets, however, the loss stems mainly from the skipped computation itself, which changing the calibration set can mitigate but not eliminate.

Q3: Is the fixed policy intended to generalize to interleaved multi-turn conversations with newly introduced images and later references to earlier images? If so, what evidence supports that claim, given that policy calibration measures only the first answer-token distribution? If such evaluation is unavailable, please clarify the intended scope of the generalization claims.

We sincerely thank you for this question, which offers a distinctive perspective. Following your suggestion, to examine whether OpSkip extends to multi-turn, multi-image question answering, we evaluated it on MMDU (110 dialogues, 1,645 turns), applying OpSkip to every prefill that introduces new images (Page 24, §C.8).

Scores of Qwen3-VL-8B on MMDU. Δ denotes the paired difference relative to Vanilla with its 95% bootstrap confidence interval. Reference conditions on the reference answers (all 110 dialogues), and Own conditions on the model's own responses (10 dialogues, 174 turns) so that errors can propagate.

HistoryMethodOverallVisual perceptionImage relationshipAccuracyΔ OverallΔ RevisitΔ Other
ReferenceVanilla72.8870.7676.9468.34–––
OpSkip (B=20)72.1269.7375.9067.87-0.77 [-1.34, -0.16]-0.53 [-1.14, +0.08]-1.98 [-3.33, -0.62]
OpSkip (B=32)69.0366.1072.6565.16-3.85 [-4.56, -3.12]-2.68 [-3.38, -2.00]-9.89 [-11.84, -7.90]
OwnVanilla64.0262.8268.7957.76–––
OpSkip (B=20)64.0861.9568.1658.28+0.06 [-2.91, +2.87]+0.21 [-3.05, +3.36]-0.61 [-3.10, +1.43]
OpSkip (B=32)59.4355.8661.3254.31-4.60 [-9.54, +0.00]-4.33 [-9.52, +0.20]-5.76 [-12.37, +0.45]

Truncation rate and TTFT (ms) by turn type for Qwen3-VL-8B on MMDU. No new img. denotes turns that introduce no new images (including turns that revisit earlier images).

HistoryMethodTurnsTrunc. (%)First turnLater new img.All img.No new img.
ReferenceVanilla1,6452.31287.0295.3292.185.3
OpSkip (B=20)1,6451.70229.3219.3223.184.7
OpSkip (B=32)1,6452.92208.6199.3202.985.0
OwnVanilla17420.1192.2254.4205.2150.8
OpSkip (B=20)17418.9779.7185.5153.4141.3
OpSkip (B=32)17413.2274.6175.4144.9139.6

At B=20B=20, OpSkip is essentially on par with Vanilla (Reference −0.77, Own +0.06). Under both protocols, the confidence intervals for turns that revisit earlier images include 0, indicating that reusing the KV produced by OpSkip in later turns yields no statistically significant difference; meanwhile, the TTFT of new-image turns decreases from 292.1 ms to 223.1 ms. At B=32B=32, the Overall score drops by about 4 points. Therefore, regarding the intended scope, first-token calibration transfers to long answers, multiple turns, and self-generated histories at moderate budgets, whereas the risk increases under aggressive budgets.

Reviewer_dBjzOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates redundant visual-token computation in multimodal large language models. Through representation-level interventions and output-distribution analysis, the authors find that late-layer visual representations can undergo substantial changes while having limited influence on answers. Further analysis reveals that the contributions of visual attention and FFNs vary across layers. Based on these findings, OpSkip uses calibration-derived, layer-wise policies to selectively bypass visual attention, FFNs, or both, while retaining all visual tokens and allowing text tokens to continue accessing visual information.

Critical strengths
  1. The paper distinguishes between the magnitude of visual representation updates and their actual influence on answers, and separately analyzes the contributions of attention and FFNs. Compared with skipping entire visual layers, this design allows finer-grained allocation of computation while preserving text tokens’ access to the complete visual sequence.

  2. The experiments cover a broad range of settings and include actual efficiency measurements.The authors evaluate the trade-off between performance and computation across four models and ten benchmarks, reporting prefill latency and TTFT rather than only theoretical FLOPs. Experiments varying the calibration sample size and calibration data source also provide some support for the applicability of the offline policy.

Critical weaknesses
  1. Data separation and selection stability in offline calibration are insufficiently documented. The default policy is determined using only eight samples from each of six benchmarks. However, the paper does not clearly specify which splits these samples come from, whether they are excluded from the final evaluation, or whether policy-related hyperparameters were selected using test performance. The sample-size and cross-domain experiments are informative, but they cannot replace multiple independent sampling runs with a fixed sample size. Consequently, it remains difficult to assess how strongly the results depend on the particular calibration samples, especially under aggressive skipping.

  2. Compatibility with FlashAttention remains unclear. As an inference acceleration method, OpSkip does not clearly state whether it is compatible with FlashAttention. If implementing the skipping policy requires disabling FlashAttention, the computational savings may be offset by the loss of low-level optimizations, potentially making it slower than the original model with FlashAttention enabled. Its compatibility therefore needs to be clarified.

Questions for authors
  1. Which data splits are the default calibration samples drawn from? Are they completely disjoint from the final evaluation samples? Were settings such as the skipping budget and freezing threshold selected using final test performance? If there is any overlap, please provide results obtained using an independent calibration set.

  2. With the calibration sample size fixed at 48, how much do the selected layers and operator policies vary across independent samples drawn using different random seeds? Please report policy consistency and the mean and variability of downstream performance, especially under aggressive skipping.

  3. Is OpSkip compatible with FlashAttention?

3 comments
AuthorsOpenReview ↗

Response 1

Dear Reviewer dBjz,

Thank you very much for your careful review and valuable comments. Following your suggestions, we have revised the paper (new content in blue) by clarifying the calibration data source, re-evaluating five independent calibrations on a strictly deduplicated holdout set, and clarifying that OpSkip is compatible with FlashAttention, which was enabled in all our latency measurements.

Q1 & Q2: Calibration isolation and stability at a fixed sample size.

We sincerely thank you for raising this point. The skipping budget BB is a user-specified efficiency knob, and following ShortV [1] and V-Skip [2] we report results over a range of budgets rather than one selected point, while the Frozen threshold is fixed at 0.7% of each model's total RAFR_{AF} for all models and tasks. The default calibration set is drawn from the evaluation splits of GQA, POPE, TextVQA, MME, MMMU, and ScienceQA (eight questions each). The original results were computed on the complete evaluation splits, in which the 48 calibration samples account for approximately 0.11% of the 44,605 questions. Calibration uses only the model's own output distributions and no labels. To rule out any possible leakage, we constructed a holdout set that strictly excludes all calibration questions and all questions sharing an image with them, and re-evaluated five independently calibrated policies. Specifically, to analyze the robustness of our method to the choice of calibration samples under different random seeds, we drew five non-overlapping 48-sample calibration sets for each model (seeds 101–105, from the same source tasks as in the paper), and removed all calibration questions involved in these draws as well as all questions sharing an image with any calibration sample (Page 19, §C.4):

POPEGQAMMMUTextVQASQAMMBMMEAI2DVizWizOCRBTotal
Original9,00012,5789005,0002,0174,3292,3743,0884,3191,00044,605
Holdout8,31611,0168604,9381,9764,3292,2943,0884,3191,00042,136
AuthorsOpenReview ↗

Response 2

Q1 & Q2 (continued).

Layer Jaccard is the mean pairwise Jaccard index between the selected layer sets, Layers in all 5 counts the layers selected by all five policies, and Same operator is the fraction of commonly selected layers that receive the same action. Full splits reports the paper's policies on the complete evaluation splits.

ModelBLayer JaccardLayers in all 5Same operatorRetention (%)Full splits
LLaVA-1.5-7B200.8215/2072%99.14 ± 0.1398.8
240.9122/2475%98.75 ± 0.1398.5
280.9326/2878%97.70 ± 0.2297.4
321.0032/3278%93.73 ± 1.9194.5
LLaVA-NeXT-7B201.0020/2079%99.02 ± 0.0898.8
240.8821/2476%97.35 ± 0.8596.7
280.9225/2873%94.41 ± 0.6894.3
321.0032/3271%87.83 ± 2.6787.9
Qwen2.5-VL-7B200.8316/2069%95.49 ± 0.2995.0
240.8217/2471%91.13 ± 1.3192.7
281.0028/2875%81.87 ± 2.8279.2
Qwen3-VL-8B200.8518/2077%99.88 ± 0.2599.5
240.8219/2478%98.65 ± 0.4397.6
280.8323/2878%95.32 ± 1.8795.7
320.9027/3277%91.65 ± 2.6191.9

At B=20B=20, the layers selected under different seeds overlap heavily (mean Layer Jaccard ≥0.82\geq 0.82 for every model), and the standard deviation of retention is at most 0.29. For the few layers with inconsistent operator assignments, RAR_A and RFR_F are already close (relative gap ∣RA−RF∣/(RA+RF)|R_A - R_F| / (R_A + R_F) of 0.19, versus 0.30 for layers with consistent assignments), so skipping either operator incurs a similar cost and has little effect on performance. The variance increases under aggressive budgets (e.g., ±2.82\pm 2.82 for Qwen2.5-VL at B=28B=28), but for B≤24B \leq 24 the results of the paper's policies on the complete splits differ from the holdout means by at most 1.6 points.

Q3: Is OpSkip compatible with FlashAttention?

Thank you for pointing out this issue. OpSkip does not require disabling FlashAttention. Skipping visual attention removes only the visual queries, while text queries still attend to all keys and values under the original causal mask. We implement this by invoking FlashAttention on the retained contiguous query block with a bottom-right-aligned causal mask. K, V, and the KV cache remain unchanged, and the decoding stage uses the standard FlashAttention path. In fact, all measured speedups reported in our paper were obtained with FlashAttention enabled. We apologize for not emphasizing this point in the original submission, and we have now clarified it in the revised manuscript (Page 9, §4.2, RQ4). The measured results on Qwen3-VL-8B are as follows:

BPrefillTTFT
Vanilla (ms)OpSkip (ms)SpeedupVanilla (ms)OpSkip (ms)Speedup
1250.941.21.24×80.670.81.14×
1650.940.01.27×80.569.51.16×
2050.939.01.31×80.568.61.17×
2450.937.21.37×80.666.81.21×
2850.935.71.43×80.565.31.23×
3250.934.41.48×80.563.91.26×
3650.933.51.52×80.563.11.28×

References

[1] Yuan et al. ShortV: Efficient multimodal large language models by freezing visual tokens in ineffective layers. ICCV 2025.

[2] Ma et al. Look less, reason more: Block-wise attention skipping for efficient multimodal LLMs. arXiv:2606.08511, 2026.

We hope that these revisions and responses have adequately addressed your questions. Thank you again for your valuable time and insightful feedback, and we would be glad to provide any further clarification you may need.

Reviewer_2uFHOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes OpSkip, a training-free framework that selectively skips visual-token attention, FFN computation, or both across decoder layers while preserving all visual tokens. Policies are constructed using output-distribution sensitivity on a small unlabeled calibration set. Experiments across four MLLMs and ten benchmarks report retaining 95.0%-99.5% of average task performance at 55%-67% of the original FLOPs, together with pre-fill and time-to-first-token improvements.

Critical strengths
  1. Fine-grained and practical computation control. Separately controlling visual attention and FFNs provides greater flexibility than freezing entire visual branches. Preserving visual K/V also allows textual tokens to continue accessing the complete visual sequence. The design is intuitive and its execution is clearly specified.

  2. Useful empirical analysis and encouraging efficiency results. The paper combines representation-level interventions with output-distribution and generation-level diagnostics. Results across the tested backbones, particularly Qwen3-VL, and the combination with token pruning provide encouraging evidence that heterogeneous operator skipping can offer useful efficiency gains.

Critical weaknesses
  1. Insufficient validation on current models and visually intensive benchmarks. The evaluation is restricted to LLaVA-1.5, LLaVA-NeXT, Qwen2.5-VL, and Qwen3-VL at the 7B/8B scale. These backbones do not adequately establish applicability to more recent and larger open-weight VLMs. For an inference-efficiency method, visual redundancy and practical speedups may depend substantially on model scale and architecture. I would therefore expect validation on at least one more recent model, such as Qwen3.6-27B, with any required architectural adaptations clearly explained. The benchmark suite should also include more targeted, visually intensive evaluation of small objects, dense counting, spatial relations, or high-resolution details, for example Do You See Me or comparable benchmarks. Although TextVQA and OCRBench already test visual detail, the current results do not sufficiently substantiate the broader claim of preserving fine-grained visual evidence in demanding perception settings.

  2. The benefit of the allocation mechanism requires stronger controlled evidence. Table 3 fixes the selected layers but does not match computational cost: attention skipping, FFN skipping, and freezing remove different amounts of work. Without per-variant FLOPs or latency, this ablation establishes an accuracy difference rather than a clear performance-efficiency advantage. The R_A + R_F selection rule also lacks comparisons against simpler action-risk or cost-aware alternatives. Furthermore, calibration/evaluation separation is insufficiently specified, and some gains are small: Table 2 implies approximately 98.761% versus 98.719% retention for OpSkip and APET on LLaVA-1.5. Matched-cost controls and repeated independent calibration draws are needed to establish a reliable advantage.

Questions for authors
  1. Can the authors validate the method on a more recent, larger open-weight VLM, such as Qwen3.6-27B, and clarify whether its architectural differences affect the skipping mechanism and efficiency gains?

  2. Does the method preserve performance on visually intensive benchmarks targeting small objects, dense counting, spatial relations, and high-resolution details? Please report task-level degradation rather than only average retention.

  3. What are the FLOPs and latency of each Table 3 variant? Does the heterogeneous policy retain its advantage over independently optimized fixed-operator policies at matched cost?

  4. Are calibration examples strictly disjoint from scored examples, and are the reported improvements stable across independent calibration draws? How does R_A + R_F compare with simpler action-risk or cost-aware selection?

4 comments
AuthorsOpenReview ↗

Response 1

Dear Reviewer 2uFH,

Thank you very much for your careful review and constructive comments. Following your suggestions, we have revised the paper (new content in blue) with validation on a newer and larger model, per-subtask results on visually intensive benchmarks, matched-cost comparisons, and independent calibration draws.

Q1: Can the authors validate the method on a more recent, larger open-weight VLM, such as Qwen3.6-27B, and clarify whether its architectural differences affect the skipping mechanism and efficiency gains?

We sincerely thank you for this forward-looking suggestion. Following your suggestion, we have validated OpSkip on Qwen3.8-27B (Page 18, §C.3), a more recent model that, like Qwen3.6-27B, adopts a hybrid-attention architecture. It has 64 layers, consisting of 48 Gated DeltaNet layers and 16 full-attention layers. Full-attention layers admit all three actions (FFN-only, Attention-only, and Frozen), whereas DeltaNet layers admit only Attention-only and Frozen, so that the recurrent state read by text tokens is left unchanged. At matched FLOPs, OpSkip clearly outperforms both whole-layer and fixed-operator methods:

MethodFLOPs ↓POPESQAMMBOCRBMMEMMMUAI2DDYSMV*Retention (%) ↑
Vanilla1.00089.295.184.083.72381.855.783.351.482.7100.00
OpSkip0.48087.688.678.067.72075.449.974.943.679.690.28
ShortV0.48387.882.074.849.41997.349.872.327.769.681.20
V-Skip0.48785.279.472.345.11702.148.168.729.172.878.20

Moreover, OpSkip delivers measured speedups across a range of budgets (Page 18, §C.3):

MethodFLOPs ↓POPESQAMMBOCRBMMEMMMUAI2DDYSMV*Retention (%) ↑Prefill ↑TTFT ↑
Vanilla1.00089.295.184.083.72381.855.783.351.482.7100.001.00×1.00×
OpSkip (B=28)0.6688.893.882.077.42354.454.080.146.582.296.671.28×1.23×
OpSkip (B=36)0.5888.692.681.276.82268.952.179.245.382.295.141.38×1.30×
OpSkip (B=42)0.5387.890.980.275.42191.951.977.343.980.693.371.45×1.35×
OpSkip (B=48)0.4887.688.678.067.72075.449.974.943.679.690.281.53×1.41×
OpSkip (B=54)0.4285.283.774.554.61892.451.671.734.775.483.811.63×1.48×
OpSkip (B=60)0.3681.577.967.424.21619.545.067.628.471.272.631.74×1.55×

Q2: Does the method preserve performance on visually intensive benchmarks targeting small objects, dense counting, spatial relations, and high-resolution details? Please report task-level degradation rather than only average retention.

Thank you for suggesting a more targeted evaluation of fine-grained visual perception. We have added per-subtask results on DYSM and V* for all four models (Page 20, §C.5), so that task-level degradation is reported rather than only average retention. On all four models, OpSkip achieves the highest overall DYSM and V* scores among the acceleration methods:

LLaVA-1.5-7BLLaVA-NeXT-7B
SubtasknVanillaOpSkipShortVV-SkipV2DropVanillaOpSkipShortVV-SkipV2Drop
FLOPs1.000.660.660.760.661.000.670.670.740.67
DYSM overall2,61311.711.711.510.58.612.313.512.113.211.7
Form constancy27026.725.927.825.921.123.723.323.723.028.1
Closure16828.026.827.426.816.720.228.625.025.626.8
Figure–ground9027.826.724.422.221.118.926.722.220.028.9
Spatial relation8064.33.84.84.71.25.25.84.86.25.2
Color & shape4087.88.17.13.44.217.919.917.919.611.5
Geometric2405.46.25.45.05.47.56.76.76.27.5
Letter13511.111.18.910.48.13.01.53.03.73.0
3D color & shape1208.310.810.06.75.023.323.318.326.711.7
3D shape1204.25.85.05.80.810.013.36.710.09.2
3D letter960.00.00.00.00.00.00.00.00.00.0
3D form constancy8062.562.557.556.275.028.828.828.828.825.0
3D spatial803.85.00.01.22.57.57.57.55.02.5
V* overall19147.147.145.546.646.158.156.053.954.552.9
V* attribute11544.343.541.740.941.753.950.447.848.750.4
V* position7651.352.651.355.352.664.564.563.263.256.6
AuthorsOpenReview ↗

Response 2

Q2 (continued).

Qwen2.5-VL-7BQwen3-VL-8B
SubtasknVanillaOpSkipShortVV-SkipV2DropVanillaOpSkipShortVV-SkipV2Drop
FLOPs1.000.550.550.670.551.000.660.660.760.66
DYSM overall2,61341.736.433.630.535.948.439.237.831.534.1
Form constancy27054.820.027.417.031.190.456.767.842.261.9
Closure16841.131.026.811.927.469.651.851.241.756.5
Figure–ground9027.818.921.120.027.838.932.236.736.737.8
Spatial relation80626.123.927.221.220.746.538.534.626.929.5
Color & shape40879.478.263.770.676.545.836.533.628.431.4
Geometric24020.018.316.75.817.928.325.824.620.824.2
Letter1355.95.95.23.73.710.46.76.72.24.4
3D color & shape12084.282.560.075.877.570.074.266.773.347.5
3D shape12060.865.850.049.271.771.769.265.864.245.0
3D letter960.00.00.00.00.05.22.11.01.02.1
3D form constancy8071.280.070.083.870.018.817.516.231.228.8
3D spatial8033.828.831.222.525.042.546.236.237.536.2
V* overall19170.768.665.463.464.477.077.075.475.470.7
V* attribute11572.269.667.063.561.778.379.177.478.371.3
V* position7668.467.163.263.268.475.073.772.471.169.7

On fine-grained visual perception, OpSkip generally outperforms the other acceleration methods. A notable exception is the three Gestalt subtasks of DYSM (2D Form constancy, Closure, and Figure–ground), which require perceiving a shape as a whole under transformation, fragmentation, or clutter. On these three subtasks combined, OpSkip outperforms V-Skip but falls behind ShortV and V2Drop. To identify the cause, we drew five non-overlapping samples (48 questions each) from each of four calibration domains on Qwen3-VL. We report the Gestalt accuracy of each resulting policy. DYSM is scored on the 2,214 questions that remain after excluding the calibration questions, on which Vanilla achieves 75.6.

Calib. domainDraw 1Draw 2Draw 3Draw 4Draw 5
Original74.474.673.953.2†76.5
TextVQA74.874.452.6†62.8†50.9†
DYSM72.070.969.771.673.9
ChartQA71.274.676.175.469.9

Among the 20 policies evaluated, every pronounced Gestalt degradation coincides with skipping the visual attention in layer 18: the four policies that skip this operator (†) achieve 50.9–62.8%, whereas the remaining 16 achieve 69.7–76.5%. The B=20B=20 policy in the paper also skips this operator, and its Gestalt accuracy is 51.3%. One plausible explanation is that general VQA relies mostly on local object evidence, so calibration on such data may underestimate the risk of this layer, whereas Gestalt tasks require mid-layer visual attention to integrate contours across the whole image.

Keeping the method unchanged, calibrating with only 48 unlabeled DYSM questions consistently preserves this operator and restores Gestalt accuracy to 71.6±1.571.6 \pm 1.5 (vs. 51.3 for the paper's policy and 57.7 for ShortV), while the general holdout retention remains 99.87±0.4599.87 \pm 0.45. These results suggest that the visual attention in layer 18 may be particularly important for Gestalt tasks, and that a small number of unlabeled target-domain samples can improve operator selection and mitigate the observed degradation, which whole-layer skipping or fixed-operator schemes cannot address.

Q3: What are the FLOPs and latency of each Table 3 variant? Does the heterogeneous policy retain its advantage over independently optimized fixed-operator policies at matched cost?

We sincerely thank you for this suggestion. Following your suggestion, we have re-compared per-layer operator selection with fixed-operator policies at matched FLOPs (Page 21, §C.6). We also thank you for pointing out (Q4) that the advantage of OpSkip might stem from the design choice of the RA+RFR_A + R_F ranking score. To examine this, together with the independence assumption raised by Reviewer 2PUr (Q1), we designed three ablation variants:

  1. Action-risk ranks layers solely by the risk of the operator that is actually skipped, min⁡(RA,RF)\min(R_A, R_F), without counting the risk of the other operator.
  2. Cost-aware ranks layers by this risk divided by the FLOPs saved by the corresponding action.
  3. Sequential adds one layer at a time: at each step, the RA+RFR_A + R_F of every candidate layer is re-measured on the 48 calibration samples with all previously selected layers already skipped, and the layer with the smallest value is added, until the saved FLOPs match those of OpSkip.
AuthorsOpenReview ↗

Response 3

Q3 (continued).

All variants are evaluated on the holdout set, which consists of the 42,136 questions (out of the original 44,605) that remain in the ten evaluation tasks after removing all calibration questions of seeds 101–105 and all questions sharing an image with them.

MethodFLOPs ↓GQATextVQAMMEMMBMMMUPOPESQAAI2DOCRBVizWizRetention (%) ↑
Vanilla1.00061.280.12383.384.853.089.194.483.882.869.4100.00
OpSkip0.5660.576.42149.581.951.988.890.180.576.870.396.42
Attention-only0.5658.262.21987.779.050.387.587.377.926.662.985.06
FFN-only0.6752.657.51566.576.744.879.678.868.943.863.779.82
Frozen0.5658.868.31905.376.849.387.384.375.562.366.889.37
Action-risk0.5660.276.02111.481.751.689.190.980.479.168.996.26
Cost-aware0.5760.276.52161.280.251.289.190.880.779.569.596.44
Sequential0.5659.975.92034.681.650.588.687.879.576.869.795.00

At matched FLOPs, OpSkip (96.42%) clearly outperforms the fixed-operator policies with independently selected layers (Attention-only 85.06%, Frozen 89.37%, and FFN-only only 79.82%). Meanwhile, Action-risk and Cost-aware remain within 0.2 points of OpSkip, and Sequential is lower (95.00%). These results indicate that the gain comes from per-layer operator selection rather than from the ranking rule or per-step re-estimation.

Q4: Are calibration examples strictly disjoint from scored examples, and are the reported improvements stable across independent calibration draws? How does RA+RFR_A + R_F compare with simpler action-risk or cost-aware selection?

We sincerely thank you for raising this point. The original results were computed on the complete evaluation splits, in which the 48 calibration samples account for approximately 0.11% of the 44,605 questions. Calibration uses only the model's own output distributions and no labels. To rule out any possible leakage, we constructed a holdout set that strictly excludes all calibration questions and all questions sharing an image with them, and re-evaluated five independently calibrated policies. Specifically, to analyze the robustness of our method to the choice of calibration samples under different random seeds, we drew five non-overlapping 48-sample calibration sets for each model (seeds 101–105, from the same source tasks as in the paper), and removed all calibration questions involved in these draws as well as all questions sharing an image with any calibration sample (Page 19, §C.4):

POPEGQAMMMUTextVQASQAMMBMMEAI2DVizWizOCRBTotal
Original9,00012,5789005,0002,0174,3292,3743,0884,3191,00044,605
Holdout8,31611,0168604,9381,9764,3292,2943,0884,3191,00042,136

Layer Jaccard is the mean pairwise Jaccard index between the selected layer sets, Layers in all 5 counts the layers selected by all five policies, and Same operator is the fraction of commonly selected layers that receive the same action. Full splits reports the paper's policies on the complete evaluation splits.

ModelBLayer JaccardLayers in all 5Same operatorRetention (%)Full splits
LLaVA-1.5-7B200.8215/2072%99.14 ± 0.1398.8
240.9122/2475%98.75 ± 0.1398.5
280.9326/2878%97.70 ± 0.2297.4
321.0032/3278%93.73 ± 1.9194.5
LLaVA-NeXT-7B201.0020/2079%99.02 ± 0.0898.8
240.8821/2476%97.35 ± 0.8596.7
280.9225/2873%94.41 ± 0.6894.3
321.0032/3271%87.83 ± 2.6787.9
Qwen2.5-VL-7B200.8316/2069%95.49 ± 0.2995.0
240.8217/2471%91.13 ± 1.3192.7
281.0028/2875%81.87 ± 2.8279.2
Qwen3-VL-8B200.8518/2077%99.88 ± 0.2599.5
240.8219/2478%98.65 ± 0.4397.6
280.8323/2878%95.32 ± 1.8795.7
320.9027/3277%91.65 ± 2.6191.9

At B=20B=20, the layers selected under different seeds overlap heavily (mean Layer Jaccard ≥0.82\geq 0.82 for every model), and the standard deviation of retention is at most 0.29. For the few layers with inconsistent operator assignments, RAR_A and RFR_F are already close (relative gap ∣RA−RF∣/(RA+RF)|R_A - R_F| / (R_A + R_F) of 0.19, versus 0.30 for layers with consistent assignments), so skipping either operator incurs a similar cost and has little effect on performance. The variance increases under aggressive budgets (e.g., ±2.82\pm 2.82 for Qwen2.5-VL at B=28B=28), but for B≤24B \leq 24 the results of the paper's policies on the complete splits differ from the holdout means by at most 1.6 points.

We hope that these revisions and responses have adequately addressed your questions. Thank you again for your valuable time and insightful feedback, and we would be glad to provide any further clarification you may need.

Comments on the paper

2 comments

Dear Reviewers,

The authors have submitted their rebuttal and tried to address the concerns raised in your reviews. We would appreciate it if you could take the time to read their responses and engage in the discussion, particularly to assess whether your concerns have been adequately addressed.

Thank you for your time and valuable contributions to the review process!

Best, AC

AuthorsOpenReview ↗

Summary of Rebuttal

Dear Program Chairs, Senior Area Chairs, Area Chairs, and Reviewers:

We thank all reviewers for their constructive feedback. We have responded to all three reviewers and uploaded a revised paper (new content in blue). Reviewer 2PUr stated that "the newly added analyses provide evidence that further strengthens the paper" and is "satisfied with the responses." As Reviewers 2uFH and dBjz have not yet responded, we summarize the paper and the rebuttal below.

Key Findings and Contributions

  1. Answer-silent visual updates. We design a set of answer-observable metrics to empirically trace visual-token updates layer by layer: Visual Update Magnitude (VUM), Linearized and Interventional Representation Change (LRC/IRC) of the final prompt representation, answer-token KL divergence, and generation agreement. Across multiple MLLMs and benchmarks, VUM grows in late layers while LRC, IRC, and KL decline and generations remain largely unchanged, so update magnitude does not measure useful computation. Freezing late visual states causes only minor changes on seven benchmarks (Table 1).
  2. Redundancy is operator-dominant and layer-dependent. Decomposing each layer into attention and FFN reveals attention-dominant, FFN-dominant, and nearly silent layers, which token- and layer-level methods cannot exploit. Operator choice matters both with the same 28 layers fixed, where changing only the skipped operator moves TextVQA from 25.6 to 75.6 (Table 3), and at matched FLOPs, where OpSkip retains 96.42% vs. ≤ 89.37% for every fixed-operator policy (§C.6).
  3. OpSkip: training-free, label-free, keeps every visual token. Calibrated on 48 unlabeled samples in about 1 minute, OpSkip retains 95.0–99.5% performance at 55–67% FLOPs across four MLLMs and 10 benchmarks, with the best average retention on three of the four. It gives real prefill/TTFT speedups with FlashAttention and complements token pruning: at the same cost, VisionZip (10%) + OpSkip beats VisionZip (3%) by +14.1 TextVQA and +18.4 MMB (Table 4).

All reviewers recognized these strengths, including the "interesting observation" (2PUr), the "fine-grained and practical computation control" (2uFH), and the "finer-grained allocation of computation" (dBjz).

Summary of Resolved Concerns

Shared concern (all reviewers): calibration isolation and stability. We built a strict holdout set (42,136 of 44,605 questions) that excludes every calibration question and every question sharing an image with one, and re-evaluated five independent 48-sample draws per model [§C.4, Page 19]. At B=20, the selected layers overlap heavily (Layer Jaccard ≥ 0.82 for every model) and retention std is ≤ 0.29.

Reviewer 2uFH (Rating: 2)

Concern 1 (Newer and larger models): We added Qwen3.8-27B [§C.3, Page 18], which has the same hybrid architecture as Qwen3.6 (48 Gated DeltaNet + 16 full-attention layers). At 0.48 FLOPs, OpSkip retains 90.28% vs. 81.20% (ShortV) and 78.20% (V-Skip), with 1.28×–1.74× prefill speedups.

Concern 2 (Visually intensive benchmarks): We added per-subtask DYSM and V* results for all four models [§C.5, Page 20]. OpSkip achieves the best overall DYSM and V* scores among acceleration methods on every model. We traced the one weaker case (Gestalt subtasks) to the layer-18 visual attention. Calibrating on 48 unlabeled DYSM questions restores Gestalt accuracy from 51.3 to 71.6 ± 1.5 (ShortV: 57.7).

Concern 3 (Matched-cost ablation, selection rule): At matched FLOPs [§C.6], OpSkip beats all fixed-operator policies (Contribution 2). Action-risk (96.26%) and Cost-aware (96.44%) rules perform on par, so the gain comes from per-layer operator selection.

Reviewer dBjz (Rating: 2)

Concern 1 (Calibration splits, hyperparameters): Calibration uses 48 label-free samples (≈0.11% of evaluation questions). B is a user-specified budget reported over its full range, as in ShortV and V-Skip, and the Frozen threshold is fixed for all models and tasks. Leakage is ruled out by the holdout above.

Concern 2 (Stability at fixed size): Addressed by the five-seed experiment above.

Concern 3 (FlashAttention): OpSkip is fully compatible, and all reported speedups were measured with FlashAttention enabled [§4.2, Page 9].

Reviewer 2PUr (Rating: 3, retained)

Concern 1 (Cross-layer interactions): Joint KL never exceeds the sum of single-layer KLs. Sequential recalibration costs 39× more calibration time and performs worse (95.00% vs. 96.42%) [§C.6].

Concern 2 (Cross-domain calibration): Over 4 domains × 5 draws, cross-domain Layer Jaccard falls within the within-domain range [§C.7, Page 23].

Concern 3 (Multi-turn dialogue): On MMDU, OpSkip (B=20) matches Vanilla (−0.77 / +0.06) and cuts new-image TTFT from 292.1 to 223.1 ms [§C.8, Page 24].

The reviewer confirmed satisfaction and maintained Weak Accept.

We believe these additions address the reviewers' concerns. Thank you for your time and consideration.

Sincerely, Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.