- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper proposes C2I-CasITI-VTG, a confidence-to-IoU cascaded inference-time intervention framework for video temporal grounding. The method is motivated by the mismatch between prediction confidence and localization accuracy, which may introduce noise when constructing IoU-based steering directions. To address this issue, the authors first apply confidence-based intervention during the prefill stage to obtain confidence-conditioned representations. The resulting samples are then filtered according to their confidence scores, and an IoU-based steering direction is constructed from the high-confidence subset. During inference, only the resulting IoU direction is applied to the decoding activations. Experiments on three VTG benchmarks with four MLLMs demonstrate consistent improvements over the corresponding frozen backbones.
Critical strengths
-
The paper investigates the relationship between prediction confidence and localization accuracy, which is an interesting perspective for improving video temporal grounding.
-
The experiments cover multiple MLLMs and three VTG benchmarks. The reported improvements, particularly under strict IoU thresholds, are encouraging. The cross-dataset evaluations also provide some evidence of transferability.
Critical weaknesses
-
Although the proposed method only injects a single steering direction during inference, it still relies on a relatively large MLLM for temporal localization. The paper does not report inference latency, computational cost, or memory consumption. Considering that efficiency is an important factor for video temporal grounding, especially for long videos and large-scale applications, the authors should provide a more comprehensive efficiency analysis. Moreover, Table 1 mainly compares MLLM-based approaches, without including representative conventional VTG methods.
-
In Section 3.2, confidence is estimated using the weighted entropy of timestamp-generation tokens. However, low token entropy does not necessarily indicate accurate temporal localization, especially when a model confidently generates incorrect timestamps. Furthermore, Table 2 shows that pure confidence intervention substantially decreases mIoU from 53.93% to 44.66%. This raises an important question about whether the confidence intervention actually improves the reliability of the internal representations. Although Fig. 3 provides some qualitative observations, quantitative analysis of the correlation between confidence and IoU is missing. The authors should further investigate this relationship and compare the proposed weighted entropy with other confidence estimation strategies.
Questions for authors
-
The paper argues that independently constructed confidence and IoU steering directions may interfere with each other. Is there any more direct evidence of such interference? Moreover, how does the proposed method compare with a simple parallel combination of confidence and IoU steering?
-
According to Section 3.4, only the final IoU direction is applied during test-time inference, while confidence intervention is used during vector construction. Why is the confidence intervention not retained during inference?
-
I also suggest that the authors provide visualization examples comparing the predicted temporal boundaries of the baseline, pure IoU intervention, and the proposed method against the ground truth. Both successful and failure cases would help illustrate how the proposed intervention improves temporal localization. In addition, visualization of the hidden representations before and after confidence intervention could provide further insights into the effectiveness of the proposed cascaded design.
5 comments
W2: Confidence estimation and its role in the cascade. Thank you for raising this important distinction.Our method estimates confidence through uncertainty in timestamp generation, using weighted entropy computed from the model’s internal token probability distributions. The decrease from 53.93 to 44.66 mIoU under confidence-only intervention illustrates why confidence is used as an auxiliary signal in our method, rather than the final localization objective.
To quantitatively examine the relationship between confidence and localization quality, we computed the Spearman correlation between negative weighted entropy and IoU on 7,000 paired samples. After confidence intervention, the correlation increases from 0.197 to 0.258, indicating a modest improvement in their alignment. We also visualize 500 randomly sampled pairs using a jointly fitted PCA, which shows a shift in the activation distribution after intervention. Together, these observations support the role of confidence intervention in conditioning representations for subsequent IoU-vector construction.
Our aim is to link confidence to the model’s own timestamp-generation behavior, so that confidence and the behavior it describes remain consistent. We therefore need to obtain confidence from signals provided by the model itself. Two common approaches are to derive confidence scores from token prediction probabilities or log-probabilities, or to prompt the model to report its confidence directly, known as verbalized confidence. Our weighted entropy follows the first approach, using uncertainty in timestamp generation to measure confidence.
To evaluate verbalized confidence, we first prompted the model to generate a temporal interval together with its confidence. Across the tested 100-point, 5-point, and binary scales, the model consistently reported maximum confidence, providing little distinction between predictions. We therefore adopted a two-pass self-evaluation procedure: the model first predicts the interval and then assesses its confidence in that prediction in a separate pass.
We used these self-reported scores to construct the confidence direction and select high-confidence samples for subsequent IoU-vector construction. The results on Charades-STA using Qwen3.5-9B are reported below; all metrics are percentages.
| Confidence estimator | Intervention | mIoU | [email protected] | [email protected] | [email protected] |
|---|---|---|---|---|---|
| — | Backbone | 53.93 | 79.70 | 62.63 | 32.72 |
| Two-pass self-evaluation | Confidence → IoU | 56.79 | 81.29 | 67.02 | 39.09 |
| Weighted entropy | Confidence → IoU (Ours) | 57.47 | 82.12 | 67.15 | 41.16 |
With either estimator, the confidence-to-IoU cascade outperforms the corresponding confidence-only intervention. The cascade using weighted entropy achieves 57.47 mIoU, compared with 56.79 using two-pass self-evaluation. Weighted entropy therefore yields better localization performance in this comparison and eliminates the second inference pass needed for self-evaluation.
These results support weighted entropy as a useful confidence measure for our framework: it provides a model-derived signal for vector construction, achieves better localization performance than the tested self-evaluation alternative, and avoids an additional inference pass.
Q1: Parallel combination of confidence and IoU steering. Thank you for suggesting this comparison. We evaluated a weighted-sum baseline on Charades-STA using Qwen3.5-9B, combining the independently constructed IoU and confidence vectors into a single intervention vector. We swept the IoU:confidence ratio from 1:0 to 0:1 in steps of 0.1 under otherwise identical settings. All metrics below are percentages.
| IoU:Conf | mIoU | [email protected] | [email protected] | [email protected] |
|---|---|---|---|---|
| 1.0:0.0 | 56.13 | 80.91 | 65.67 | 38.12 |
| 0.9:0.1 | 56.09 | 80.94 | 65.11 | 37.72 |
| 0.8:0.2 | 55.74 | 80.86 | 64.89 | 37.07 |
| 0.7:0.3 | 55.07 | 80.46 | 64.11 | 35.75 |
| 0.6:0.4 | 54.26 | 79.97 | 62.74 | 33.87 |
| 0.5:0.5 | 52.99 | 78.71 | 61.37 | 31.59 |
| 0.4:0.6 | 51.82 | 77.98 | 59.11 | 29.22 |
| 0.3:0.7 | 50.19 | 76.53 | 55.97 | 26.64 |
| 0.2:0.8 | 48.87 | 75.51 | 53.25 | 24.84 |
| 0.1:0.9 | 47.42 | 74.19 | 51.08 | 22.61 |
| 0.0:1.0 | 44.66 | 71.91 | 44.92 | 18.12 |
| Ours | 57.47 | 82.12 | 67.15 | 41.16 |
The best mIoU in the sweep occurs at the pure-IoU endpoint. Increasing the confidence weight progressively reduces mIoU, [email protected], and [email protected], showing that adding the independently constructed confidence direction can weaken localization performance under the tested settings.
These results provide empirical support for our concern about combining the two steering objectives directly. Our cascade instead uses confidence to condition the representations from which the IoU direction is constructed, achieving 57.47 mIoU. We will include these results in the revision to show why we use confidence during vector construction instead of combining it directly with IoU steering.
Q2: Why is confidence intervention not retained at inference? Confidence intervention is used to condition the representations from which the final IoU direction is constructed. The deployed direction is then explicitly guided by localization quality, so confidence does not need to be applied again as a separate steering signal.
We directly tested retaining confidence intervention during prefill while applying the constructed IoU direction during decoding. Results on Charades-STA using Qwen3.5-9B are shown below; all metrics are percentages.
| Test-time intervention | mIoU | [email protected] | [email protected] | [email protected] |
|---|---|---|---|---|
| Backbone | 53.93 | 79.70 | 62.63 | 32.72 |
| Confidence during prefill + final IoU direction during decoding | 29.83 | 47.72 | 28.79 | 11.21 |
| Final IoU direction only (Ours) | 57.47 | 82.12 | 67.15 | 41.16 |
Retaining confidence intervention substantially reduces localization performance in this experiment. One possible explanation is that the two interventions shift the model’s representations in different directions: confidence steering changes the prefill representations, while the subsequent IoU steering shifts the decoding representations toward localization quality. These successive shifts may interfere with each other and weaken the final prediction. Confidence first conditions the representations used for vector construction, while IoU guides the final intervention toward the downstream localization objective. We therefore apply confidence intervention only during vector construction and retain a single IoU direction at inference.
Q3: Qualitative examples and representation visualization. We sincerely thank the reviewer for these helpful suggestions. We have added qualitative case studies to the appendix and will expand them to compare the baseline, pure IoU intervention, and our method against the ground truth, including both successful and failure cases. We will show the predicted temporal boundaries and corresponding IoU values to make the improvements and remaining errors easier to inspect.
We also appreciate the suggestion to visualize hidden representations before and after confidence intervention. We will include this analysis to help illustrate how confidence intervention changes the representations used for subsequent IoU-vector construction. We will present these visualizations as supporting observations, alongside the quantitative ablations, to provide a clearer understanding of the cascaded design.
Thank you again for the constructive suggestions, which help us better communicate both the effectiveness and limitations of our method.
W1: Efficiency analysis and comparison with conventional VTG methods. Thank you for highlighting these omissions. We have expanded Table 1 to include representative conventional VTG methods: TR-DETR, UVCOM, Sim-DETR, and SE-DETR+. Their published results are included below alongside the existing comparisons.
Updated Table 1. Comparison on Charades-STA (C), ActivityNet-Captions (A), and QVHighlights (Q). All values are percentages. A dash indicates an unreported result.
Conventional methods use their respective task-specific training settings. The reported TR-DETR, UVCOM, and Sim-DETR results use SlowFast+CLIP video features and CLIP text features, while SE-DETR+ uses multi-layer CLIP video features (K=4) and CLIP text features.
For efficiency, we measured Qwen3.5-9B with and without our intervention on the same Charades-STA samples using a single NVIDIA RTX 3090 GPU:
At inference time, our method simply adds a precomputed vector to the model’s hidden states, without additional model passes. The measured latency increase is only 0.040%, while generation length and peak memory remain unchanged at the reported precision. Offline vector construction requires two activation-extraction rounds using forward passes, without training or backpropagation.
The MLLM backbone remains the main source of computational and memory cost. These results show that our method improves localization while maintaining essentially the same inference latency and GPU memory usage as the backbone. We will include these efficiency measurements in the revision.