- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes Stability-Based Halting (SBH) for efficient long-video question answering. It uses a lightweight linear predictor to estimate whether the current answer will remain stable under an expanded video view, incorporating confidence, observation progress, and remaining-video information. A video-level conformal calibration strategy provides a statistical guarantee on early-stop disagreement with the reference answer. Experiments across multiple benchmarks and MLLM backbones demonstrate inference speedups with comparable accuracy.
Critical strengths
-
The paper addresses an important limitation of confidence-based early stopping. A confident answer may change when additional video evidence becomes available. The proposed reference-consistency formulation is clearly defined, and the video-level conformal calibration provides a meaningful finite-sample guarantee while accounting for dependencies across observation steps and questions.
-
The experiments include multiple backbones, benchmarks, controlled stopping baselines, component ablations, and end-to-end latency measurements. The authors also report paired bootstrap confidence intervals and distinguish reference consistency from answer correctness. The demonstrated inference speedups provide practical evidence for the proposed stopping framework.
Critical weaknesses
-
The theoretical guarantee controls agreement with a fixed 128 frame reference view rather than correctness or consistency with the complete video. Although the authors acknowledge this distinction, the reliability of the reference itself is not sufficiently investigated. It remains unclear whether the reference answers and stopping decisions would remain stable under larger observation budgets. Since the proposed guarantee fundamentally depends on the reference definition, this limits its practical interpretation for long-video understanding.
-
While the controlled experiments demonstrate meaningful latency reductions, the accuracy improvements are small and statistically uncertain. The contribution of the remaining-video cue is also relatively modest, removing it changes accuracy from 68.02% to 67.78% and latency from 7.74 to 8.05 seconds (Table 6). Moreover, the method requires substantial offline reference scoring and calibration, including 26.7 GPU-hours for Qwen2.5-VL on Video-MME. The paper does not sufficiently discuss how these costs are amortized or how well the calibrated stopping rule transfers across different deployment settings.
Questions for authors
Q1. The proposed guarantee is defined relative to a fixed 128 frame reference view. How sensitive are the reference answers and early-stopping decisions to this budget? Would the conclusions remain consistent if the reference budget were increased to 256 or 512 frames?
Q2. Table 6 shows that the remaining-video cue provides relatively small improvements in final accuracy (68.02% vs. 67.78%) and latency (7.74 vs. 8.05 seconds). Could the authors provide stronger evidence that these improvements are consistent and statistically meaningful, particularly compared with simpler calibrated stopping predictors?
Q3. Table 8 reports substantial offline scoring costs, including 26.7 GPU-hours for Qwen2.5-VL on Video-MME. How many online queries are required to amortize this overhead, and to what extent can the fitted predictor and calibration parameters be reused across datasets or deployment environments?
Q4. The qualitative examples in Figures 6-8 use AI-generated scenes with manually specified answers and stopping-bound signs. Could the authors provide actual inference trajectories from real benchmark videos, including the predicted margins, calibrated bounds, stopping decisions, and final answer correctness?
5 comments
Response to Reviewer Wbzc (2/4): Statistical evidence for the cue
W2(a) / Q2. Accuracy and statistical evidence for the remaining-video cue.
Table 9 shows a 3.10 s latency reduction with a paired confidence interval entirely below zero. The accuracy difference is +0.37 percentage points, with its confidence interval reported alongside latency:
| Original Table 9: SBH minus endpoint | Difference [95% paired CI] |
|---|---|
| Accuracy (percentage points) | +0.37 [−0.74, +1.48] |
| Latency (s) | −3.10 [−3.48, −2.74] |
For the remaining-video cue, the direct mechanism is better prediction of answer stability. Original Table 6 reports:
| Original Table 6 | Margin MAE | Positive bounds (%) | Early (%) | Accuracy (%) | Latency (s) | |
|---|---|---|---|---|---|---|
| SBH without cue | 0.103 | 0.251 | 35.91 | 88.15 | 67.78 | 8.05 |
| SBH with cue | 0.084 | 0.214 | 42.63 | 92.35 | 68.02 | 7.74 |
The cue reduces prediction MAE by 18.4%. The positive-bound frequency rises by 6.72 percentage points and the early-stop rate rises by 4.20 percentage points. Positive-bound frequency uses all eligible prefixes, with question- and video-balanced averaging. It measures how often the predictor can certify a stop. The training-only donor cues in Table 6 give higher MAE (0.105–0.110) and latency (8.11–8.22 s), supporting the value of the remaining video's content.
Paired cue analysis with 95% confidence intervals. Each no-cue predictor is separately refitted and recalibrated. Differences are with-cue minus without-cue. The three seeds keep the evaluation videos fixed and vary the fitting/calibration split.
| Dataset / backbone | Latency difference (s) [95% CI] | Accuracy difference (pp) [95% CI] | Margin-MAE difference [95% CI] | Latency-difference range across three seeds (s) |
|---|---|---|---|---|
| Video-MME / Qwen2.5-VL | −0.31 [−0.50, −0.12] | +0.24 [−0.49, +0.97] | −0.019 [−0.025, −0.013] | [−0.39, −0.23] |
| Video-MME / InternVL2.5 | −0.36 [−0.59, −0.13] | +0.24 [−0.49, +0.97] | −0.018 [−0.024, −0.012] | [−0.44, −0.25] |
| LongVideoBench / Qwen2.5-VL | −0.30 [−0.57, −0.03] | +0.22 [−0.69, +1.13] | −0.018 [−0.026, −0.010] | [−0.38, −0.18] |
Simpler calibrated predictors. Qwen / Video-MME, identical video-level calibration at . Margin and entropy use and , respectively. Each is calibrated against the binary reference-agreement target using the maximum residual across a video's questions and prefixes. The table reports their observed video-level risks .
| Predictor | (%) | Accuracy (%) | Latency (s) |
|---|---|---|---|
| SBH with cue | 5.19 | 68.02 | 7.74 |
| Calibrated classifier with cue | 5.56 | 67.78 | 7.99 |
| Calibrated margin | 5.93 | 67.65 | 9.12 |
| Calibrated entropy | 6.30 | 67.28 | 9.38 |
| Calibrated TV | 2.96 | 67.53 | 12.47 |
The cue reduces prediction error and latency in all three settings. The paired latency and MAE intervals are below zero in each setting, and all three fitting/calibration seeds give lower latency. The accuracy intervals include zero, so the evidence supports faster stopping at similar observed accuracy.
Response to Reviewer Wbzc (3/4): Amortization and cross-domain reuse
W2(b) / Q3. Amortization of offline scoring and reuse across deployments.
The offline scoring cost is amortized after approximately 31,000–37,400 queries under the Table 8 resource settings. The break-even query count is
Amortization calculated from Table 8. The calculation uses comparable GPU resources and one GPU allocated throughout each online query. Fitting and calibration times are reported separately in seconds.
| Backbone | Offline scoring (GPU-h) | Fit + calibration (s) | Endpoint (s) | SBH (s) | Break-even queries |
|---|---|---|---|---|---|
| Qwen2.5-VL | 26.7 | 1.36 | 10.84 | 7.74 | 31,007 |
| LLaVA-OneVision | 29.8 | 1.44 | 11.30 | 8.43 | 37,380 |
| InternVL2.5 | 28.5 | 1.39 | 12.13 | 8.79 | 30,719 |
For Qwen, 50,000 and 100,000 queries yield calculated net savings of 16.36 and 59.41 GPU-h after the 26.7 GPU-h scoring cost. These GPU allocation-time savings make repeated deployment workloads the main use case.
Predictor reuse and calibration reuse have different conditions. A fixed source predictor can receive a target-domain guarantee by calibrating on independent target videos exchangeable with deployment videos. The source-only rows below evaluate direct reuse empirically, and the target-calibrated rows evaluate adaptation to the new domain.
Qwen2.5-VL transfer from Video-MME, . Each dataset uses the same target evaluation videos across rows. Source- rows reuse the fitted predictor and change only calibration. Target- rows use target fitting and calibration. measures any early-stop reference disagreement per video. measures coverage across all questions and eligible prefixes in a video, including prefixes after stopping. The GPU-h column reports additional calibration scoring for source-predictor reuse.
| Target | Calibration videos | (%) | (%) | Early (%) | Latency (s) | New scoring (GPU-h) | ||
|---|---|---|---|---|---|---|---|---|
| LongVideoBench | Source | Source 180 | 0.214 | 86.34 | 11.45 | 94.10 | 9.14 | 0.0 |
| LongVideoBench | Source | Target 50 | 0.258 | 92.51 | 7.05 | 86.40 | 9.58 | 2.8 |
| LongVideoBench | Source | Target 150 | 0.239 | 91.19 | 6.17 | 88.60 | 9.26 | 8.3 |
| LongVideoBench | Target | Target 150 | 0.226 | 93.39 | 5.29 | 90.72 | 8.89 | — |
| MLVU | Source | Source 180 | 0.214 | 86.69 | 10.36 | 93.70 | 7.04 | 0.0 |
| MLVU | Source | Target 50 | 0.236 | 91.42 | 6.51 | 87.60 | 7.52 | 2.4 |
| MLVU | Source | Target 224 | 0.230 | 89.64 | 5.62 | 89.10 | 7.33 | 10.6 |
| MLVU | Target | Target 224 | 0.198 | 94.08 | 4.73 | 91.28 | 7.12 | — |
Reusing with 50 target calibration videos reduces observed risk from 11.45% to 7.05% on LongVideoBench and from 10.36% to 6.51% on MLVU, at 2.8/2.4 GPU-h of additional scoring. Larger target calibration sets yield 6.17%/5.62% risk. Theorem 1 gives marginal validity over calibration and new videos. Calibration uses model-generated reference scores without GT labels. A hardware-only change permits rule reuse when predictions are unchanged. Changes to the model, input protocol, observation policy, reference, or deployment distribution call for calibration under the updated setting.
Response to Reviewer Wbzc (1/4): Reference-budget sensitivity
Thank you for your careful review and the time you devoted to assessing our theory and experiments. We address your constructive comments below with analyses of reference budgets, cue effectiveness, deployment costs, and inference trajectories.
Response. Theorem 1 formalizes preservation of a specified reference prediction. With the predictor and protocol fixed before calibration, exchangeability of calibration and test videos gives a marginal bound of α on any early-stop reference disagreement within a new video. To assess the reference itself, the tables below report ground-truth (GT) accuracy and sensitivity to larger observation budgets. The all-return metric includes budget-endpoint disagreements.
Preserving reference behavior is an established acceleration objective in CATs [1] and CALM [2]. SBH applies this objective to video evidence acquisition through simultaneous video-level calibration.
Reference-budget sensitivity: Qwen2.5-VL / Video-MME. Nested 128/256/512-frame references contain the same 64-frame online endpoint. Every frame is processed in full at the same 448×448 resolution. The first comparison freezes the original stopping rule and its stopping times. J is the percentage of videos with any early-stop disagreement. All-return disagreement is measured per question.
Stopping sensitivity after refitting and recalibrating for each reference, α=0.10. J is measured per video. Stopping changes and accuracy are measured per question.
Expanding the reference changes 3.70–4.20% of answers and raises reference accuracy from 68.40% to 69.14–69.38%. The increase from 256 to 512 frames changes only 14/810 answers (1.73%). After refitting and recalibration, stopping steps change for 9.38–13.33% of questions, while accuracy stays within 68.02–68.15%. The larger-reference rules retain 1.35× and 1.33× speedups over the 10.84 s endpoint. This supports 128 frames as a practical reference in this setting. The frozen-rule rows measure sensitivity to a changed reference. Each refitted rule is calibrated against its own reference for the guarantee.
[1] Consistent Accelerated Inference via Confident Adaptive Transformers.
[2] Confident Adaptive Language Modeling.