acceptodds
Under review as a conference paper at ICLR 2027

Stage-adaptive Token Selection for Efficient Omni-modal LLMs

Abstract

The unified audio-visual understanding capability of omni-modal large language models (om-LLMs) comes at the high cost of processing tens of thousands of visual and audio tokens throughout the LLM. To reduce the cost, training-free methods for om-LLM token selection are being actively developed, operating either pre-LLM or inner-LLM. Recognizing that the non-textual token redundancy varies across stages and LLM layers, we propose in this paper a novel stage-adaptive token selection (SEATS) method that operates in both pre-LLM and inner-LLM stages. Given an overall token budget measured by the token retention rate (TRR), SEATS performs top-down token budget allocation in an input-dependent manner. Specifically, a larger TRR is used for pre-LLM token selection, whilst within the LLM, all tokens are retained in shallow layers, increasingly decreased TRRs are applied in middle layers, and the non-textual tokens are fully removed in late layers. Extensive experiments on three representative om-LLMs and six benchmarks verify the viability of the proposed method. By retaining only 10% of visual and audio tokens, SEATS preserves 94.9% of the full-token performance on Qwen2.5-Omni while achieving a FLOPs reduction and a prefill speedup.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Omni-modal LLM, Efficient MLLM, token selection

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 4 reviews, above 85% of the others.
Reviewer_knfAOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes SEATS, a training-free token selection method to accelerate Omni model inference. It works in three stages: first, it filters out redundant audio-visual tokens before they enter the LLM; then, in the middle layers of the LLM, it continues pruning tokens at a decreasing retention rate, weighted by their relevance to the question; finally, the deeper layers drop the remaining audio-visual tokens entirely.

Critical strengths
  1. The experimental observations and the design of the method are coherent, and the writing is clear. The paper separately tests the effects of removing visual and audio tokens, then sets layer-wise retention rates based on those observations, and allocates token budgets by time window and modality, so the logic is throughout.

  2. This paper validates the effectiveness of SEATS across multiple models. SEATS retains 94.9% of the average score performance through token selection, achieving a 9.3× reduction in FLOPs, a 4.9× speedup in prefill, and a 1.9× improvement in TTFT.

Critical weaknesses

The novelty of the core layer-wise depth observations is limited. FastV (Chen et al., 2024) already identified the inefficiency of processing visual tokens in deep layers. PyramidDrop (Xing et al., 2024) already reported that visual tokens matter in shallow layers but become increasingly redundant in deeper layers. And the paper's experimental baselines do not include a direct comparison with the closely related work OmniFit (Wang et al., 2026), leaving the gains relative to the nearest method unclear.

Questions for authors
  1. Could you compare SEATS against more closely related layer-wise methods such as OmniFit (Wang et al., 2026) under the same model, token budget, and timing conventions? If not feasible, what are the specific obstacles?

  2. Under what kinds of inputs would the fixed layer boundaries break down? If the starting and ending layers for pruning were determined dynamically per input sample, could performance be improved?

  3. At 10% TRR, prefill achieves a 4.9× speedup and TTFT a 1.9× speedup. Could you further break down the time cost into encoding, pre-LLM filtering, and in-LLM filtering, and show how these vary with different video lengths?

[1] FastV(Chen et al., 2024):An Image is Worth 1/2 Tokens After Layer 2: Plug-and-Play Inference Acceleration for Large Vision-Language Models

[2] PyramidDrop(Xing et al., 2024):PyramidDrop: Accelerating Your Large Vision-Language Models via Pyramid Visual Redundancy Reduction

[3] OmniFit(Wang et al., 2026):OmniFit: Bridging Modalities via Layer-Adaptive Token Compression for Omnimodal Large Language Models

1 comment

Thank the authors for the detailed response. I have raised my score from 2 to 3, as the rebuttal has strengthened the empirical validation of SEATS. However, I still regard the layer-wise depth observation itself as incremental relative to prior token selection methods; the contribution is a carefully designed and thoroughly validated combination rather than a new insight about token redundancy.

Reviewer_XuN9OpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

SEATS is a training-free method for pruning audio-visual tokens. It first applies an attention-weighted, window-based extension of DivPrune before the LLM. It keeps these tokens in shallow layers, prunes them at fixed middle-layer boundaries, and removes all remaining non-textual tokens in late layers. Attention from the last textual token guides window and modality budgets and token ranking. The depth schedule is fixed; the budget allocation is intended to adapt to the input.

The reported retention ratio averages the non-textual token budget across layers. At 10% retention, Qwen2.5-Omni-7B scores 48.6 versus 51.2 with full tokens, retaining 94.9% of the mean score across six benchmarks. WorldSense timing shows approximately 9.3× fewer FLOPs, 4.9× faster prefill, and 1.9× faster time to first token (TTFT). The evaluation covers single-turn QA on complete audio-visual inputs; streaming and multi-turn use remain outside its scope.

Critical strengths
  1. The depth analysis supports delaying pruning: removing non-textual tokens causes less performance loss in later layers across three models and different video durations. Shifting the block boundaries also has little effect on the mean score. These results support the schedule, though they do not show that cross-modal fusion is complete at a fixed depth.

  2. SEATS outperforms the evaluated alternatives across three models and six benchmarks. At 10% layer-averaged retention, its means are 48.6, 55.6, and 50.4 on Qwen2.5-Omni, Qwen3-Omni, and OmniVinci, versus 47.3, 53.6, and 48.8 for the strongest reported alternatives; the full-token means are 51.2, 58.7, and 52.9.

  3. The paper measures latency and memory as well as FLOPs. At 10% retention on Qwen2.5-Omni, prefill is 4.9× faster, TTFT is 1.9× faster, and peak memory drops from 22.83 GB to 17.68 GB. Prefill includes selection overhead, and TTFT shows how much of the computational saving reaches end-to-end latency.

Critical weaknesses
  1. The written allocation rule can produce nearly uniform window budgets. In Section 4.2.2, attention weights are averaged within each window and passed through another softmax across windows; Appendix C.2 states that the token weights are already normalized and averaged across heads. For token weights aia_i and a window Wt,mW_{t,m} with nt,mn_{t,m} tokens of modality mm,

    \qquad 0\leq u_{t,m}\leq\frac{1}{n_{t,m}}.$$ For nonempty windows with $n_{t,m}\geq n_{\min}$, the modality-specific allocation weights satisfy $$\frac{\max_t S_{t,m}}{\min_t S_{t,m}} =\exp\!\left(\max_t u_{t,m}-\min_t u_{t,m}\right) \leq\exp(1/n_{\min}).$$ With 100 tokens per window, the ratio is at most about 1.010 under this rule; this is a calculated example, not a measured allocation. Table 4 gives 51.4 for the full method, 51.0 without inter-window allocation, and 51.1 without intra-window reallocation. These gains alone do not show that the budgets differ meaningfully from uniform allocation. Specify any temperature, rescaling, or use of logits that would change this analysis. Report budget distributions and compare with uniform budgets while keeping the schedule and within-window ranking fixed.
  2. Neither Section 2 nor the baseline description in Appendix C.3 discusses FastAV, and the comparisons in Section 5.1 omit it. First public on 19 January 2026, before this submission, FastAV also uses training-free audio-visual pruning, with global pruning in intermediate layers and fine pruning in later layers. Explain how SEATS differs from this schedule. Compare under the same backbone, inputs, budgets, and timing settings if an implementation is available, or state the implementation constraint. The published scores use different backbones and cannot be compared directly. Clarify whether an implementation was available at submission time.

  3. The pre-LLM ablations change more than the components they claim to test. In Table 4, replacing winDivPrune with DivPrune-om lowers the mean from 51.4 to 51.1, and replacing it with VisionZip-om lowers it to 51.0. Section 5.3 attributes these drops to removing weighting and diversity. Section 4.1 defines winDivPrune selection within temporal windows, whereas Appendix C.3 describes DivPrune-om selection per frame; VisionZip-om also changes the selection and merging rules. Hold grouping, token budget, and aggregation fixed when testing weighting and diversity separately; otherwise the scores do not isolate either component.

  4. The schedule controls in Section 5.3 do not separate progressive pruning from delayed pruning or complete late-token removal. Single Decay and Equal Step score 50.8 and 51.1 versus 51.4, but Table 4 gives neither the Single Decay layer nor each variant's realized FLOPs or latency. These ablations cover only Qwen2.5-Omni at 35% retention. Specify the pruning layer and compare with a delayed one-shot prune with the same preprocessing and realized token budget. Test late-token removal against a schedule that keeps some late tokens and rebalances the earlier budgets. Include the 10% setting to check whether the schedule gains hold at the reported high-compression setting.

  5. The latency comparison does not show whether SEATS is faster at matched accuracy. In Table 3, alternatives are timed only at 35% retention: SEATS scores 51.4 versus DivPrune-om's 50.5, but prefill takes 0.424 s versus 0.349 s and TTFT takes 1.011 s versus 0.952 s. This shows an accuracy–latency trade-off. Report lower-retention timing for DivPrune-om and VisionZip-om under the same execution settings, with timing variability, and compare at matched accuracy where their ranges overlap. These measurements are needed to assess whether the selection overhead pays off.

  6. The tuning protocol and uncertainty of small gains are unclear. Appendix B.3 selects λ=1.40\lambda=1.40 from the six-benchmark mean; Table 7 reports 51.44 versus 51.37 at λ=1.45\lambda=1.45. State whether tuning and evaluation use disjoint examples. Report paired uncertainty estimates for the 51.4 versus 51.2 gain over full tokens in Table 2 and the 0.3–0.6-point ablation gains in Table 4, accounting for questions from the same video. If tuning used the reported evaluation examples, use a separate evaluation set before claiming these gains.

Questions for authors
  1. Specify the attention-to-budget transformation in Section 4.2.2 and Appendix C.2: probabilities or logits, mean or sum, and any temperature or rescaling. Show the resulting budget distributions and the uniform-budget comparison requested in Weakness 1.

  2. Does the weighting ablation in Table 4 preserve window grouping when using DivPrune-om as described in Appendix C.3? Does removing late tokens help at an equal overall budget? Report the controls in Weaknesses 3–4 at 35% and 10% retention, with realized budgets and FLOPs, and explain the distinction from FastAV.

  3. Extend the timing comparison in Table 3 to lower retention for DivPrune-om and VisionZip-om, and compare at matched accuracy. Specify the attention backend, precision, batch size, and timing repetitions.

  4. State whether the examples used to select λ\lambda in Appendix B.3 and Table 7 are disjoint from final evaluation. Provide paired uncertainty estimates for the comparisons in Tables 2 and 4, accounting for questions from the same video.

  5. Do the budgets in Appendix C.2 use each example's actual token counts? Explain how audio and visual windows remain aligned under the frame caps and modality settings in Appendix C.3 and Table 11. Report realized layer-averaged retention.

  6. Specify the encoder-attention source and weighting rule for winDivPrune in Section 4.1. For the pruning procedure in Appendix C.2, specify the attention layer used at each transition and how position indices and KV caches are handled after selection.

2 comments

Thank you for the detailed rebuttal and additional experiments. I appreciate the authors' efforts to address my concerns, particularly through the uniform-budget ablation, FastAV comparison, controlled pruning experiments, and extended latency analysis. These results help clarify the technical contributions and strengthen the experimental evidence.

However, I remain concerned that the hyperparameter λ was tuned on the evaluation benchmarks, which may introduce evaluation bias. In addition, the efficiency advantages at matched accuracy remain relatively modest, and the additional selection overhead limits the practical speedup. While the rebuttal has addressed several of my concerns, I do not think the remaining issues have been fully resolved. Therefore, I will maintain my current rating.

Thank you for the additional experiments and clarifications. The robustness analysis across different λ values and models, together with the pre-encoding efficiency results, provides stronger evidence for the effectiveness of SEATS. I still have some concerns regarding evaluation-set tuning, the lack of statistical significance compared with full tokens, and the deployment-dependent efficiency gains. Nevertheless, I acknowledge that the rebuttal has substantially strengthened the paper and addressed most of my original concerns. I have therefore raised my rating to 3 (Weak Acceptance).

Reviewer_YX3vOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This manuscript proposes SEATS, a training-free token selection method that used to improve efficiency of omni models. It's motivated by an empirical observation that the dependence on visual and audio tokens varies across LLM layers: shallow layers are highly sensitive to non-textual token removal, middle layers gradually become more tolerant, and late layers can discard non-textual tokens with minimal performance degradation. SEATS integrates multi stages of token reduction: pre-LLM (winDivPrune), mid layers (query-guided, top-down token budget allocation), and late layers (remove all non-textual tokens).

Experiments on Qwen2.5-Omni-7B, OmniVinci-9B, and Qwen3-Omni-30B across six audio-visual understanding benchmarks show its efficiency-performance trade-offs.

Critical strengths
  1. SEATS integrates pre-LLM diversity-based selection, depth-aware token budget scheduling, and query-guided cross-modal token allocation into a unified framework.
  2. The reported results demonstrate substantial computational savings while retaining most of the original model performance.
Critical weaknesses
  1. Novelty and insight are a little bit limited. The design of unified framework is well-motivated and is supported by empirical observations, however, pre-LLM stage extends DivPrune, depth-dependent retention schedule follows the general observation that non-textual token redundancy increases with LLM depth, and query-attention-based token selection and multi-stage token compression have also been investigated in prior works.
  2. Comparison could be improved. OmniFit is discussed in related work while not included in experiments. The current evaluation primarily compares SEATS against pre-LLM compression methods or methods originally developed for image/video LLMs. It would strengthen the contribution to provide a more direct comparison with OmniFit or a comparable depth-adaptive inner-LLM baseline under matched computational budgets.
  3. Unclear compatibility with FlashAttention. SEATS requires explicit query-to-token attention scores, which are not directly available from standard FlashAttention implementations (most attention-based token compression methods have such limitations). The paper does not clarify how these scores are obtained or whether the associated overhead is included in the latency measurements. This raises concerns about the practical efficiency and deployability of the proposed method.
  4. Potential issue with double normalization. SEATS applies an additional softmax across temporal windows to already softmax-normalized attention scores (Section 4.2.2 and Appendix C.2). Since these scores are typically very small, the second softmax may produce nearly uniform window-level relevance scores, potentially weakening the effectiveness of query-adaptive token allocation. The authors should clarify this design and provide evidence that it produces meaningful differences in token budgets across windows.
Questions for authors

Refer to weaknesses.

1 comment

Thank you for the detailed response. The OmniFit comparison and implementation clarifications address most of my concerns. I still see the novelty as incremental, since both depth-dependent pruning and query-guided token selection build on prior work. The stronger empirical results make a convincing case for their combination in omni-modal models, and I have raised my score to 3.

Reviewer_F7TKOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Omni-modal large language models (om-LLMs) encode video and audio into tens of thousands of tokens, incurring substantial prefill computation and memory costs. This paper proposes SEATS, a training-free, stage-adaptive token selection method operating in both the pre-LLM and inner-LLM stages. The key empirical finding is a block-wise dependence pattern: removing all visual/audio tokens at a given layer causes performance collapse in shallow layers (<50% depth), rapidly diminishing losses in middle layers (50%–80%), and virtually no impact in late layers (>80%). The pattern is verified across three om-LLMs (Qwen2.5-Omni-7B, OmniVinci-9B, Qwen3-Omni-30B) and across video durations (Appendix A). The authors note that existing methods run counter to this pattern — FastV prunes once at a shallow layer, while HiDrop retains more tokens in deeper layers — and build their design on a depth-decaying retention schedule. The method has three stages: (1) Pre-LLM: winDivPrune extends DivPrune to per-window, per-modality max-min diversity selection with attention re-weighting, at a higher retention rate of λR (λ=1.4). (2) Inner-LLM: no pruning in shallow layers; linearly decaying TRRs across middle sub-blocks (block boundaries at fixed depth proportions 50%/60%/70%/80%, with the step δ solved analytically from the token budget); query-guided top-down budget allocation first across temporal windows and then across modalities within each window. (3) Late layers: all non-textual tokens are removed.

Critical strengths
  1. A solid observation→design→validation loop, with clear empirical support for the decay direction. The layer-wise removal study (Fig. 2) consistently supports the claim that non-textual tokens are dispensable in late layers, and the proposed decaying schedule directly opposes FastV (one-shot shallow pruning) and HiDrop (a concave schedule that retains more tokens deeper). The schedule involves very few hyperparameters (fixed depth proportions, an analytically solved step δ, a single scale factor λ) and is robust to ±2-layer boundary shifts (ΔMean ≤ 0.21), so the aggressive "full removal in late layers" design is well grounded and directly accounts for the efficiency advantage.

  2. Comprehensive and fairly conducted experimentation relative to training-free peers. Three models (including a 30B MoE, with synchronized pruning of DeepStack-injected features) × six benchmarks (average duration 43s–35min, covering audio-centric, vision-centric, and long-video settings) × four TRRs. Baselines are uniformly adapted to the omni setting under a shared budget constraint and evaluated with LMMs-Eval. The efficiency table honestly reports SEATS's own token-selection overhead (0.034s + 0.083s at R=35%) and peak memory, and ten ablations isolate each component's contribution. Critical Weaknesses

Critical weaknesses
  1. Missing direct comparison with OmniFit (ICML'26), the closest prior work for the inner-LLM stage of SEATS. Among the omni-LLM token selection methods listed in Table 1, OmniFit is the only one operating inside the LLM, and it likewise allocates layer-wise, depth-decaying retention budgets — architecturally the nearest counterpart to SEATS's inner-LLM component (SEATS itself spans both pre-LLM and inner-LLM stages, so the comparison concerns its inner stage). The paper's central novelty claim — input-dependent top-down budget allocation vs. OmniFit's offline, input-invariant scheme — is supported only qualitatively, as OmniFit is excluded from the experiments on the grounds that it is not open-source (Table 1). Notably, the paper's own ablations (Tab. 4, Setups 9–10) attribute only ~0.3–0.4 mean points to the input-dependent allocation at R=35% — within the noise range discussed in the next weakness — so the practical significance of this central distinction remains to be established empirically.

  2. Statistical rigor is insufficient to support the headline claims. All tables appear to be single runs, without variance, confidence intervals, or significance tests. The claim of surpassing full-token performance at R=35% (51.4 vs 51.2 on Qwen2.5-Omni; 53.0 vs 52.9 on OmniVinci) lies within the typical noise range of multiple-choice benchmarks and poses an over-claim risk. Moreover, λ=1.40 is grid-searched only on Qwen2.5-Omni at R=35% (Fig. 5 / Tab. 7) and then applied unchanged to all other models and more aggressive TRRs, so the tuning evidence does not match the claimed generality.

Questions for authors
  1. Could you provide a direct or reproduced comparison with OmniFit, even on a single model/benchmark? If not feasible, a more detailed feature-level comparison quantifying where input-dependent budget allocation concretely helps would strengthen the paper. Also, why is EchoingPixels (ICML'26), another omni-LLM compression method, absent from the comparisons?
  2. Are the reported results single runs? Please provide multi-seed variance or significance analysis for the near-full-token conclusions (e.g., 51.4 vs 51.2 at R=35%).
  3. λ is grid-searched only on Qwen2.5-Omni at R=35% over a narrow grid [1.25, 1.45] (Fig. 5 / Tab. 7), and then applied unchanged to all other models and TRRs; even at the tuned setting the peak is within noise (51.44 at λ=1.40 vs 51.37 at λ=1.45, which is in fact better on LVOmniBench). Please provide λ sweeps on OmniVinci / Qwen3-Omni and at low TRRs (e.g., R=10%), extending the grid to include λ=1.0 (no shallow-layer buffer at all — a direct test of whether the λ mechanism is load-bearing) and values ≥ 1.6. The risk is most concrete at low TRR: at R=10%, λ=1.4 leaves shallow layers only 14% of non-textual tokens to carry all cross-modal fusion, precisely the regime where the method's lead over baselines is largest (94.9% vs 92.4%; 94.7% vs 91.3% on Qwen3-Omni). If the optimal λ there differs, the headline advantage is overstated; if the curve is flat everywhere, the "tuned optimum" characterization loses meaning.
  4. Daily-Omni, where audio evidence is critical, shows the largest drop among the six benchmarks (62.1 vs 64.0 at R=35%), yet the text dismisses it as an "exception". Does this indicate that audio budget allocation (especially at low TRRs) is the current weak point? Any case analysis of erroneously pruned audio tokens?
  5. Full removal in late layers structurally guarantees KV-cache and decoding savings, yet the efficiency analysis covers only prefill/TTFT, measured on a single benchmark (WorldSense, ~141 s videos) on one A800 GPU — please quantify decode throughput and KV-cache savings. Moreover, Table 3 itself locates the bottleneck: at full tokens, TTFT ≈ 0.53 s of encoder time + 0.937 s prefill, while at R=10% the prefill drops to 0.192 s and the encoder side is untouched, raising its share of TTFT from ~36% to ~70% — the arithmetic reason the TTFT speedup (1.9×) falls far below the FLOPs reduction (9.3×). Given that the limitations section concedes that encoding cost is not reduced, should the headline efficiency claims (9.3× FLOPs, 4.9× prefill) be restated in end-to-end terms? On long-video benchmarks (LVOmniBench, ~34.5 min on average, up to 768 frames) the untouched encoder share grows further — could you report at least one long-video latency measurement?
  6. The block-wise dependence observation is established on only three datasets (WorldSense, Daily-Omni, Video-MME Long; Figs. 2/7/8), while the remaining three benchmarks lack the corresponding validation. Does the pattern — and the 50%/80% boundaries — hold on those benchmarks and on other om-LLM architectures?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.