- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper studies how output length affects VLM inference efficiency and proposes a cascade combining short answers from large and small models. Agreement permits early exit; disagreements trigger small-model reasoning and, when necessary, transfer of that reasoning to the large model for a short answer. Experiments cover several model families and six benchmarks, including two measured end-to-end deployments. A single-attention-layer analysis and a separate sparse-transfer study provide additional motivation.
Critical strengths
- The component profiling and response-policy comparisons provide useful evidence that model size alone is an inadequate efficiency proxy. They identify settings where a larger model’s short response outperforms a smaller model’s substantially longer reasoning trace.
- The appendix includes genuine end-to-end measurements incorporating the small-model stages. On ChartQA, the matched H100/vLLM experiment reduces total completion time from 1,122 to 468 seconds, a 2.40× speedup (Table 21). A Gemma experiment provides an additional 1.65× result, accompanied by a small accuracy decrease (Table 22).
Critical weaknesses
- The measured quality–cost comparison omits an important single-model alternative. On ChartQA, the 8B Explain configuration achieves 87.68% with 249.8 output tokens, versus 86.20% and 294.3 tokens for Reasoning (Tables 5–6), but the measured timing comparison includes only Reasoning and MAI. These numbers do not establish that Explain is faster than MAI; they show that the relevant comparison is incomplete. Furthermore, seconds per sample at batch size 32 measure amortized dataset completion time rather than individual request latency.
- The theoretical analysis provides limited explanation of end-to-end reasoning transfer. Its bounds concern attention-weighted averages with fixed representations, whereas actual multi-layer inference recomputes representations, positions, and queries. The required small Wasserstein discrepancy is not established experimentally. The proof can be correct under its assumptions while offering limited support for the broader mechanism claim; the sparse-transfer experiment also leaves its incremental deployment cost unresolved.
Questions for authors
- Can Simple, Explain, Reasoning, Full RT, and MAI be compared on the same run’s accuracy–time frontier? Please distinguish batch throughput from individual request latency and identify the targeted deployment regime.
- Which model, layer, and query supply the sparse-selection attention, and what additional computation does selection require? What evidence connects the fixed-representation bound to the actual multi-layer transfer mechanism?
3 comments
Dear Authors,
Thank you for the detailed response and the additional experiments. I appreciate the inclusion of the Explain baselines, batch-size-one latency measurements, and further clarification of the sparse-selection mechanism.
The new results address part of my original concerns and provide stronger evidence for the practical efficiency of MAI. However, I remain concerned about the reliability of the accuracy–latency comparisons and the connection between the theoretical analysis and the actual reasoning-transfer mechanism.
1. The revised accuracy–latency comparison remains insufficiently controlled.
Table R1 is a useful addition, especially the comparison between MAI (8B w/ 4B-Reasoning) and 8B Explain + S2.
However, the table combines accuracy results reported elsewhere in the paper with latency measurements obtained on the first 200 ChartQA questions at batch size 1. A matched evaluation reporting both accuracy and end-to-end latency on the same examples would provide stronger evidence for the claimed quality–efficiency tradeoff.
Furthermore, the best MAI configuration achieves 87.84% accuracy, compared with 87.68% for Explain + S2, a difference of only 0.16 percentage points. While MAI demonstrates a substantial latency reduction (1.03 s versus 2.99 s), the accuracy difference is sufficiently small that paired uncertainty estimates would be important for interpreting the claimed performance advantage.
2. The practical efficiency tradeoff requires more careful characterization.
The added results show that MAI can provide a favorable accuracy–latency tradeoff relative to longer-response baselines.
At the same time, the 8B Simple configuration achieves 84.72% accuracy at 0.10 s, whereas MAI (8B w/ 4B-Reasoning) achieves 87.84% at 1.03 s.
This represents an approximately tenfold increase in latency for a 3.12-percentage-point accuracy improvement. Both configurations may be useful under different deployment requirements, but the practical efficiency advantage depends strongly on the targeted accuracy and latency budget.
A clearer characterization of the Pareto frontier, including uncertainty in accuracy and end-to-end inference cost, would make the main efficiency claims more convincing.
3. The theoretical justification for reasoning transfer remains incomplete.
I appreciate the clarification that sparse token selection uses attention from a single layer and that the reported Top-K ranking overhead is below 1 ms.
My original concern, however, concerns the relationship between the theoretical assumptions and actual reasoning transfer.
The analysis considers attention-weighted representations under a simplified setting, whereas transferred reasoning tokens subsequently affect multi-layer autoregressive inference. The response clarifies the selection procedure but provides limited additional evidence that the assumed representation similarity, including the Wasserstein discrepancy condition, explains the observed downstream improvements.
Moreover, the reported Top-K timing characterizes the ranking operation. A fuller accounting of attention extraction and any associated implementation overhead would help establish the practical cost of the complete sparse-selection procedure.
Overall, I recognize the value of the empirical observations and acknowledge that the additional latency measurements strengthen the paper. The reported speedups relative to reasoning-intensive baselines are promising.
Nevertheless, the central claims about accuracy–efficiency advantages and theoretically motivated reasoning transfer still require stronger matched empirical validation and clearer evidence connecting the theoretical assumptions to the implemented mechanism.
For these reasons, I maintain my original rating of 2 (Weak Rejection).
Dear Reviewer 115V,
Due to the limited time left, we just would like to clarify that we reuse the attention map for selection without any additional cost. The current empirical implementation for reasoning transfer follows the theoretical analysis without any differences.
Response to Reviewer 115V
We would like to thank the reviewer for the valuable time and insightful comments, and we will add the following measurements to the revision.
W1: The measured comparison omits 8B Explain, and seconds per sample at batch size 32 reflect throughput rather than request latency.
Thanks for your suggestion and we added the following experiment for Explain. On ChartQA, Explain prompt actually uses the similar number of reasoning tokens as Reasoning, i.e., 249.8 tokens vs. 294.3 tokens, which leads to the similar performance and latency. Moreover, our method is not limited to Reasoning prompt and we can also use reasoning tokens from Explain prompt for transfer. The difference is how reasoning tokens is obtained. The empirical results confirm that our method is independent from specific prompt and reasoning tokens from a different Explain prompt can be transferred well, e.g., MAI (8B w/ 2B-Explain) and MAI (8B w/ 4B-Explain) are better than 8B-S. We will include the discussion in revision. For latency estimation, since the inference type is mixed, e.g., S/E/R, the avg over individual requests is thus utilized.
W2: The single-layer, fixed-representation analysis does not establish the multi-layer transfer mechanism, and the deployment cost of sparse selection is unclear.
Thanks for pointing this out. Please kindly note that our proposed method is applicable to a single layer (also any single layer) and thus we strictly follow our theoretical analysis to select key tokens from a single layer in the experiments as discussed in Section 5.3. To the best of our knowledge, there is no multi-layer approach, which will be an interesting future work and we will clarify this.
Regarding deployment, the selection is a top-K rank given an attention map, which is less than 1 ms and is negligible.
Q1: Can Simple, Explain, Reasoning, Full RT and MAI be compared on one accuracy–time frontier, separating batch throughput from request latency?
Thanks for the suggestion. We include the full comparison in the Table as follows. The observation on Explain is similar to Reasoning, where 8B-E + S2 is better than 8B-S but slower, and MAI (8B w/ 2B/4B-Reasoning) can further improve the performance with mild overhead: MAI (8B w/ 4B-Reasoning) slightly exceeds 8B-E + S2 (87.84 vs.\ 87.68) at about 3× lower latency. We will add this discussion in the revision.
Table R1. Accuracy–time frontier on ChartQA (Qwen3-VL, one H100). Accuracy as reported in the paper (8B Simple, Explain + S2 and Reasoning + S2: Table 6; Reasoning: Table 21; Full RT and MAI (8B w/ 2B-Reasoning): Table 9). Latency: mean latency of a single request (batch size 1, first 200 questions).
Q2: Which model, layer and query supply the sparse-selection attention, and what does the selection cost?
Thank you for pointing this out. Please kindly note our approach is applicable to any single layer. We have the ablation on which layer is good for transfer as in Section 5.3 and more detailed discussion can be found in Appendix H. The computational cost for selection is to rank the attention map from a single layer, which is negligible in our experiments. We will make it clear in revision.