- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper proposes OpSkip, a training-free framework for reducing the decoder prefill cost of multimodal LLMs through operator-level visual-computation skipping. The authors identify answer-silent visual updates: visual representations can undergo substantial changes in later Transformer layers while having limited influence on answer generation. Using a small calibration set, OpSkip measures output-distribution KL divergence under visual-operator interventions to determine whether to skip visual attention, FFN, or both in selected layers. The resulting policy preserves the complete visual-token sequence and leaves text-token computation unchanged. Experiments across versatile architectures and 10 benchmarks show that OpSkip uses 55%–67% of baseline TFLOPs while retaining 95.0%–99.5% of baseline performance on average. Additional experiments examine operator selection, calibration sensitivity, inference latency, and compatibility with token pruning.
Critical strengths
-
The paper is well-motivated and provides an interesting observation that substantial visual hidden-state updates do not necessarily translate into meaningful contributions to answer generation. The detailed analysis of VUM, LRC and IRC together with operator-level analysis provides a reasonable motivation for the proposed method and offers a useful perspective on efficient MLLM inference.
-
The method is evaluated across multiple MLLM architectures and 10 VQA benchmarks, with comparisons against representative visual-token pruning and layer-skipping approaches. The results demonstrate that selectively skipping operators can effectively preserve model accuracy more effectively and is comparable to coarse-grained skipping strategies at similar computational budgets.
Critical weaknesses
-
OpSkip derives a fixed per-model execution policy from a small calibration set, but the reliability of these discrete layer and operator decisions remains insufficiently studied. Figure 5 provides useful evidence by comparing mixed-domain and ChartQA calibration. However, this comparison is limited, and substantial task-specific differences emerge under aggressive budgets. Appendix C.1 varies calibration sample count but does not report variability across independently sampled calibration sets at a fixed size. Therefore, neither the stability of the selected policies nor the uncertainty in their downstream performance is established. This matters because the method relies on transferring a calibration-derived policy across tasks.
-
OpSkip retains visual K/V at every layer and primarily saves computation when visual tokens are first processed. Its practical benefit therefore depends on the balance between new-image prefill, decoding, and accumulated cache costs. More importantly, low influence on the current answer does not establish low influence on later questions that revisit earlier images. New images in later turns can also attend to preceding dialogue, changing the computational setting underlying the single-turn analysis. The absence of such validation limits the demonstrated scope of the method for interactive multimodal inference.
Questions for authors
-
The risks , , and are measured by intervening on one layer while leaving the remaining layers unchanged, whereas the deployed policy modifies multiple layers simultaneously. Although the final benchmark results validate the selected policies, how reliable are these independently measured risks under joint execution? Is there any evidence quantifying cross-layer interactions or comparing the current selection rule with sequential calibration that re-estimates risks after each selected modification, at matched final computational budgets?
-
Were the calibration experiments repeated with independent random subsets at a fixed sample count? Beyond the mixed-domain versus ChartQA comparison in Figure 5, what evidence is available regarding cross-domain variation in operator risks, selected layers, and operator assignments? Please report any available seed-level performance variation and clarify whether the observed differences persist under matched computational budgets.
-
Is the fixed policy intended to generalize to interleaved multi-turn conversations with newly introduced images and later references to earlier images? If so, what evidence supports that claim, given that policy calibration measures only the first answer-token distribution? If such evaluation is unavailable, please clarify the intended scope of the generalization claims.
5 comments
Response 2
Q1 (continued).
We further compare the downstream performance of Independent (OpSkip) and Sequential at the FLOPs corresponding to . Since Reviewer 2uFH (Q3) also reminded us to rule out the effect of computational cost, we additionally include three types of controls: skipping only one type of operator (Attention-only and FFN-only), freezing only (Frozen), and two single-pass rules that replace only the ranking score:
- Action-risk ranks layers solely by the risk of the operator that is actually skipped, , without counting the risk of the other operator.
- Cost-aware ranks layers by this risk divided by the FLOPs saved by the corresponding action.
All methods are calibrated with seed 101 (48 calibration samples newly drawn following the recipe in the paper) and evaluated on the holdout set, which consists of the 42,136 questions (out of the original 44,605) that remain in the ten evaluation tasks after removing all calibration questions of seeds 101–105 and all questions sharing an image with them.
| Method | FLOPs ↓ | GQA | TextVQA | MME | MMB | MMMU | POPE | SQA | AI2D | OCRB | VizWiz | Retention (%) ↑ |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Vanilla | 1.000 | 61.2 | 80.1 | 2383.3 | 84.8 | 53.0 | 89.1 | 94.4 | 83.8 | 82.8 | 69.4 | 100.00 |
| OpSkip | 0.56 | 60.5 | 76.4 | 2149.5 | 81.9 | 51.9 | 88.8 | 90.1 | 80.5 | 76.8 | 70.3 | 96.42 |
| Attention-only | 0.56 | 58.2 | 62.2 | 1987.7 | 79.0 | 50.3 | 87.5 | 87.3 | 77.9 | 26.6 | 62.9 | 85.06 |
| FFN-only | 0.67 | 52.6 | 57.5 | 1566.5 | 76.7 | 44.8 | 79.6 | 78.8 | 68.9 | 43.8 | 63.7 | 79.82 |
| Frozen | 0.56 | 58.8 | 68.3 | 1905.3 | 76.8 | 49.3 | 87.3 | 84.3 | 75.5 | 62.3 | 66.8 | 89.37 |
| Action-risk | 0.56 | 60.2 | 76.0 | 2111.4 | 81.7 | 51.6 | 89.1 | 90.9 | 80.4 | 79.1 | 68.9 | 96.26 |
| Cost-aware | 0.57 | 60.2 | 76.5 | 2161.2 | 80.2 | 51.2 | 89.1 | 90.8 | 80.7 | 79.5 | 69.5 | 96.44 |
| Sequential | 0.56 | 59.9 | 75.9 | 2034.6 | 81.6 | 50.5 | 88.6 | 87.8 | 79.5 | 76.8 | 69.7 | 95.00 |
At matched FLOPs, OpSkip (96.42%) clearly outperforms the fixed-operator policies with independently selected layers (Attention-only 85.06%, Frozen 89.37%, and FFN-only only 79.82% despite its higher cost). Meanwhile, Action-risk and Cost-aware remain within 0.2 points of OpSkip, and Sequential is lower (95.00%). These results indicate that the gain comes from per-layer operator selection rather than from the ranking rule or per-step re-estimation.
Q2: Were the calibration experiments repeated with independent random subsets at a fixed sample count? Beyond the mixed-domain versus ChartQA comparison in Figure 5, what evidence is available regarding cross-domain variation in operator risks, selected layers, and operator assignments? Please report any available seed-level performance variation and clarify whether the observed differences persist under matched computational budgets.
We sincerely thank you for this valuable feedback. Following your suggestion, we performed five non-overlapping unlabeled draws on each of four calibration domains and compared the resulting policies at matched FLOPs (Page 23, §C.7):
Holdout denotes the average ten-task retention (%). TextVQA, DYSM, and ChartQA report accuracy (%).
| 66% FLOPs | 57% FLOPs | |||||||||
|---|---|---|---|---|---|---|---|---|---|---|
| Calib. domain | FLOPs | Holdout | TextVQA | DYSM | ChartQA | FLOPs | Holdout | TextVQA | DYSM | ChartQA |
| Vanilla | 1.00 | 100.00 | 80.1 | 48.1 | 85.4 | 1.00 | 100.00 | 80.1 | 48.1 | 85.4 |
| Original | 0.66 | 100.12 ± 0.51 | 79.5 ± 0.4 | 44.6 ± 3.4 | 82.4 ± 2.6 | 0.57 | 95.33 ± 1.65 | 74.8 ± 2.1 | 32.3 ± 4.2 | 58.7 ± 11.6 |
| TextVQA | 0.66 | 99.68 ± 0.53 | 79.4 ± 0.6 | 42.5 ± 4.4 | 80.8 ± 3.1 | 0.58 | 92.95 ± 3.24 | 75.1 ± 3.3 | 27.9 ± 4.5 | 48.5 ± 11.4 |
| DYSM | 0.66 | 99.87 ± 0.45 | 79.2 ± 0.3 | 46.2 ± 0.7 | 82.0 ± 3.6 | 0.57 | 93.44 ± 1.75 | 73.9 ± 1.5 | 32.4 ± 3.9 | 53.6 ± 9.7 |
| ChartQA | 0.66 | 99.94 ± 0.71 | 79.0 ± 0.4 | 45.0 ± 2.2 | 83.5 ± 1.3 | 0.58 | 93.83 ± 2.90 | 74.1 ± 2.0 | 31.0 ± 6.3 | 65.2 ± 6.3 |
Response 2
Q2 (continued).
Layer Jaccard is the mean pairwise Jaccard index between the selected layer sets, and Same operator is the fraction of commonly selected layers that receive the same action.
| FLOPs | Comparison | Layer Jaccard | Same operator |
|---|---|---|---|
| 0.66 | Within Original | 0.79 | 77% |
| Within TextVQA | 0.72 | 76% | |
| Within DYSM | 0.81 | 74% | |
| Within ChartQA | 0.70 | 66% | |
| Across domains | 0.74 | 73% | |
| 0.57 | Within Original | 0.84 | 79% |
| Within TextVQA | 0.80 | 77% | |
| Within DYSM | 0.91 | 82% | |
| Within ChartQA | 0.73 | 78% | |
| Across domains | 0.78 | 79% |
With the sample count fixed at 48, the layers selected across five independent draws overlap heavily for every model (Layer Jaccard at , with a retention standard deviation of at most 0.29 points, Table 14 of the revised paper). Under the two compute budgets, the cross-domain Layer Jaccard is 0.74 and 0.78, respectively, both within the corresponding within-domain ranges (0.70–0.81 and 0.73–0.91). This indicates that the low-risk operators identified by OpSkip transfer across datasets.
At matched FLOPs, diverse calibration sets yield stable overall performance, and a small number of in-domain unlabeled samples can further improve performance on target domains that differ substantially from the calibration data. Under aggressive budgets, however, the loss stems mainly from the skipped computation itself, which changing the calibration set can mitigate but not eliminate.
Q3: Is the fixed policy intended to generalize to interleaved multi-turn conversations with newly introduced images and later references to earlier images? If so, what evidence supports that claim, given that policy calibration measures only the first answer-token distribution? If such evaluation is unavailable, please clarify the intended scope of the generalization claims.
We sincerely thank you for this question, which offers a distinctive perspective. Following your suggestion, to examine whether OpSkip extends to multi-turn, multi-image question answering, we evaluated it on MMDU (110 dialogues, 1,645 turns), applying OpSkip to every prefill that introduces new images (Page 24, §C.8).
Scores of Qwen3-VL-8B on MMDU. Δ denotes the paired difference relative to Vanilla with its 95% bootstrap confidence interval. Reference conditions on the reference answers (all 110 dialogues), and Own conditions on the model's own responses (10 dialogues, 174 turns) so that errors can propagate.
| History | Method | Overall | Visual perception | Image relationship | Accuracy | Δ Overall | Δ Revisit | Δ Other |
|---|---|---|---|---|---|---|---|---|
| Reference | Vanilla | 72.88 | 70.76 | 76.94 | 68.34 | – | – | – |
| OpSkip (B=20) | 72.12 | 69.73 | 75.90 | 67.87 | -0.77 [-1.34, -0.16] | -0.53 [-1.14, +0.08] | -1.98 [-3.33, -0.62] | |
| OpSkip (B=32) | 69.03 | 66.10 | 72.65 | 65.16 | -3.85 [-4.56, -3.12] | -2.68 [-3.38, -2.00] | -9.89 [-11.84, -7.90] | |
| Own | Vanilla | 64.02 | 62.82 | 68.79 | 57.76 | – | – | – |
| OpSkip (B=20) | 64.08 | 61.95 | 68.16 | 58.28 | +0.06 [-2.91, +2.87] | +0.21 [-3.05, +3.36] | -0.61 [-3.10, +1.43] | |
| OpSkip (B=32) | 59.43 | 55.86 | 61.32 | 54.31 | -4.60 [-9.54, +0.00] | -4.33 [-9.52, +0.20] | -5.76 [-12.37, +0.45] |
Truncation rate and TTFT (ms) by turn type for Qwen3-VL-8B on MMDU. No new img. denotes turns that introduce no new images (including turns that revisit earlier images).
| History | Method | Turns | Trunc. (%) | First turn | Later new img. | All img. | No new img. |
|---|---|---|---|---|---|---|---|
| Reference | Vanilla | 1,645 | 2.31 | 287.0 | 295.3 | 292.1 | 85.3 |
| OpSkip (B=20) | 1,645 | 1.70 | 229.3 | 219.3 | 223.1 | 84.7 | |
| OpSkip (B=32) | 1,645 | 2.92 | 208.6 | 199.3 | 202.9 | 85.0 | |
| Own | Vanilla | 174 | 20.11 | 92.2 | 254.4 | 205.2 | 150.8 |
| OpSkip (B=20) | 174 | 18.97 | 79.7 | 185.5 | 153.4 | 141.3 | |
| OpSkip (B=32) | 174 | 13.22 | 74.6 | 175.4 | 144.9 | 139.6 |
At , OpSkip is essentially on par with Vanilla (Reference −0.77, Own +0.06). Under both protocols, the confidence intervals for turns that revisit earlier images include 0, indicating that reusing the KV produced by OpSkip in later turns yields no statistically significant difference; meanwhile, the TTFT of new-image turns decreases from 292.1 ms to 223.1 ms. At , the Overall score drops by about 4 points. Therefore, regarding the intended scope, first-token calibration transfers to long answers, multiple turns, and self-generated histories at moderate budgets, whereas the risk increases under aggressive budgets.
Response 1
Dear Reviewer 2PUr,
Thank you very much for your recognition of our work and for your insightful questions. Following your suggestions, we have revised the paper (new content is highlighted in blue). Specifically, we have added an analysis of cross-layer interactions and sequential recalibration, experiments with independent draws and cross-domain calibration, and an evaluation on multi-turn, multi-image dialogue. Your questions have helped us substantially broaden and solidify the paper. We address each of them below.
Q1: The risks RA(l), RF(l), and RAF(l) are measured by intervening on one layer while leaving the remaining layers unchanged, whereas the deployed policy modifies multiple layers simultaneously. Although the final benchmark results validate the selected policies, how reliable are these independently measured risks under joint execution? Is there any evidence quantifying cross-layer interactions or comparing the current selection rule with sequential calibration that re-estimates risks after each selected modification, at matched final computational budgets?
Thank you for raising this insightful question. Following your suggestion, we have added an analysis of cross-layer interactions (Page 22, §C.6). Because each layer has three possible actions and the number of combinations grows exponentially with the number of layers, fully characterizing cross-layer interactions is infeasible. We therefore compare two layer-selection schemes:
Since Sequential must re-measure all candidate layers at every step, its calibration takes 39 minutes (versus about 1 minute for OpSkip). We compare the first-token KL when the entire policy is executed jointly (Joint KL) with the sum of the KLs obtained when each selected layer is executed alone (Σ single-layer KL, i.e., an additive reference based on isolated interventions). Their difference reflects the magnitude of cross-layer interactions. Both schemes are calibrated on the same 48 calibration samples of Qwen3-VL-8B. At every budget, the joint KL does not exceed the sum of single-layer KLs; that is, summing independent risks does not underestimate the error of joint execution, although the two quantities gradually approach each other as the budget increases. The layers selected by Sequential have a much larger sum of single-layer risks (0.412) than those selected by Independent (0.262), yet their joint KL is not lower (0.191 vs. 0.174).