- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper introduces the problem of active test-time adaptation for temporal action segmentation and adopts an episodic setting where the model is reset after adapting independently to each video. To select frames for labeling, the authors propose an uncertainty-guided frame selection method that divides the video into K regions with equal cumulative prediction entropy and selects one frame per segment that minimizes the uncertainty-weighted distance. During adaptation, the method employs a sparse classification loss for labeled frames and a temporal smoothing loss for the full sequence. Dynamic trajectory regularization based on KL divergence from the prediction of the previous epoch is employed, with the regularization strength adjusted both globally and segment-wise depending on the degree of agreement between the current prediction and the ground-truth label frames. Experiments demonstrate performance improvements over existing active TTA methods designed for image classification and simple selection baselines.
Critical strengths
The paper experimentally demonstrated the importance of which specific parts are designated as labels. Performance improved compared to pre-trained models, and the fact that the performance gain was better than other frame selection methods is particularly noteworthy. This trend remained consistent even when the model was changed.
Critical weaknesses
The most significant weakness is the lack of a sufficient justification for the setup involving active TTA in TAS. In active TTA for image classification, the labeled sample and the sample being evaluated are distinct images; however, in this paper’s setup, the labeled frame and the frame evaluated after adaptation belong to the same video. Given the strong correlation between adjacent frames in TAS, a ground-truth label at a specific time point serves as a strong cue for the entire surrounding segment, rather than providing information limited to just that single frame. In fact, the proposed method aims to propagate the label’s influence to neighboring frames via a smoothing loss; consequently, it is difficult to distinguish whether the substantial performance improvement over the pre-trained model stems from the test-time adaptation effect or simply from the direct provision of ground-truth cues. In this regard, the setup appears closer to interactive segmentation or timestamp supervision, where a user designates specific frames and the model propagates this information across the video, than to standard TTA, yet the paper fails to discuss or compare the method against these related approaches. Furthermore, the paper lacks an explanation regarding which real-world applications would require human labeling for every test video after deployment, nor does it explain why the “stronger model” mentioned in the introduction is not used directly if it is available as an oracle. I believe the authors should present specific, realistic scenarios where this setup is necessary and clearly articulate the distinctions between their approach and related methods.
Additionally, regarding the methodology, the reliance on predictive uncertainty, a concept already widely used in existing active learning and active TTA studies, limits the novelty of the frame selection component. Although the authors frame their contribution as distributing uncertainty along the temporal axis, the resulting selection rule is fairly simple in practice. With the default L1 distance, minimizing Eq. (7) within each region yields the entropy-weighted median of that region, and since every region contains an equal share of the total entropy mass, the procedure essentially amounts to selecting frames at evenly spaced quantiles of the cumulative entropy.
Questions for authors
-
What are the practical scenarios where active TTA is applied to TAS?
-
In experiments comparing episodic and online settings, how is the sequence of videos determined for the online setting? I imagine the results could vary significantly depending on the order.
-
I would like to know more details about the implementation of SimATTA and BiTTA.
2 comments
W2.1 Use of predictive uncertainty limits the novelty.
We respectfully disagree that using predictive uncertainty limits the novelty of the frame selection. Predictive uncertainty is the widely used informativeness signal in active learning and active TTA (e.g., SimATTA), so the novelty of a selection strategy lies in how uncertainty is exploited, not in whether the measure has been used before. As you also recognize, one of our key contributions is distributing uncertainty along the temporal axis through our uncertainty-aware temporal allocation. The other concerns the adaptation stage, where the active queries are also used as adaptation signals. To the best of our knowledge, neither has been explored in previous active TTA work.
W 2.2 Simplicity of the selection rule
We agree that using the cost in Eq. (7) admits a weighted-median solution. However, we argue this is an advantage rather than a limitation: a selection rule that is simple yet effective is what test-time adaptation calls for. This is also acknowledged by reivewer CDTV as a strength. Concretely:
-
Using for Eq.7 can be implemented efficiently without iterative numerical optimization, which matters at test time when a video contains thousands of frames.
-
It is also effective, it outperforms more elaborate strategies such as clustering-based and MC-dropout-based selection (Table 1).
-
Importantly, evenly paced quantiles of cumulative uncertainty (with l1) should not be confused with equi-distant temporal sampling, as the resulting sampling locations adapt to the nonuniform uncertainty distribution across the video.
-
Last but not least, our formulation is not restricted to l1 distance. As shown in Table 4, alternative distance functions give competitive results, which shows that the formulation is more general than the weighted-median interpretation.
Q2. Video order in the online setting
The online results reported in the paper process the test videos in the order listed in each dataset's test-split file, without additional shuffling. Following your comment, we additionally ran three random permutations of the test order (Breakfast, MS-TCN).
| Setting | Acc | Edit | F1@10 | F1@25 | F1@50 |
|---|---|---|---|---|---|
| Online, 3 random | |||||
| Episodic | 81.9 | 79.5 | 81.8 | 77.8 | 65.5 |
As we can see, the variation across permutations is small: the standard deviations of accuracy and F1@50 are only 0.2 and 0.3, respectively, so the online performance does not depend strongly on a particular random ordering.
We attribute this to the per-video feedback: each video receives its own queried labels and adaptation epochs, so the model re-specializes to the current video regardless of the state inherited from the previous one. The remaining gap to episodic adaptation is present under every order, which suggests it stems from parameter drift accumulated across videos and not from a particular ordering. This supports what we dicsussed this in L. 402-412 and Appendix A.3.
Q3. Implementation of SimATTA and BiTTA
Both methods were proposed for image classification, where each batch contains images. For TAS, we treat each test video as one batch and its frames as the samples.
SimATTA. Frames whose entropy lies in the top 20% of the video form the high-entropy candidate set. We use this within-video percentile instead of the fixed threshold , because entropy scales vary across videos and datasets. We cluster the last-stage features of these candidates with K-means into clusters and annotate the frame closest to each centroid with its ground-truth label. Following SimATTA's low-entropy sample selection, the lowest-entropy frames within the bottom 20% are assigned the source model's prediction. is the number of frames to be labeled and they are kept consistent with ours. Annotated and pseudo-labeled frames are used for adaptation.
BiTTA. We select frames by MC-dropout uncertainty. We run 4 forward passes of the source model with dropout enabled and average the softmax outputs to obtain . The certainty of each frame is , the MC-averaged probability of the deterministic prediction. The frames with the lowest certainty are selected for annotation. In addition, most certain are assigned as pseudo-labels. One difference from the original setting is the form of feedback: BiTTA's oracle only indicates whether is correct. To compare all methods under the same annotation protocol, the selected frames receive their ground-truth semantic labels, as in SimATTA and our method.
Thank you for your comments. W1 raises several related points, which we address in turn as W1.1–W1.5.
W1.1 Labeled and evaluated frames belong to the same video
Yes, and this is by design. In TAS the test sample is the whole video, and the shift we adapt to is specific to each video.
In image classification, each test image is one sample with one label, so a queried label belongs to a different sample from the ones it helps. TAS is different. A TAS model takes a video of T frames and outputs T labels as a single structured prediction. For each video, we query a small fraction of its frames (0.16% on average on Breakfast with MS-TCN), adapt the model, and evaluate on all T frames. A queried frame is therefore a partial label on the test video and should not be considered as the label on an image.
In addition, active adaptation from the same video is needed because the shift is video-specific. Each test video has its own action composition, durations, and transitions, which differ from what the globally trained model has learned. On Breakfast:
W1.2 Adaptation or label propagation?
Thank you for raising this point. Adaptation and label propagation are not contradictory. Propagating information from a few labeled samples to many unlabeled ones is the purpose of active and semi-supervised learning in general, and combining queried labels with label propagation or pseudo-labeling is common practice. Existing active TTA methods such as SimATTA and BiTTA use the queried feedback to update the model and additionally train on model-generated labels for unlabeled samples.
Ours follows the same principle, using only sparse label supervision, and the propagation likewise takes place through the model instead of spreading labels directly over the outputs. Specifically, the queried labels provide supervision where the model is most uncertain, the model weights are updated, and the updated model re-predicts all T frames, including those far from any queried frame.
W1.3 Relation to timestamp supervision and interactive segmentation
The key difference between our setting and these two is who decides where to label: in ours it is the model, based on its uncertainty, which is the "active" part.
In timestamp supervision, the locations are normally sampled at random and do not depend on the model.
In interactive segmentation, the location of each interaction is typically simulated in benchmarks from the error region, which requires access to the ground truth to decide where to select.
In contrast, in our setting the model decides where to query from its own predictive uncertainty, with no access to the ground truth or to its errors. The goal is to select frames better, so that the same annotation budget yields a larger improvement from adaptation. Our main contributions are deisgned for this, making them distinct from the other settings.
W1.4 & Q1. Realistic usecase
A concrete application is human-in-the-loop annotation of extensively long procedural videos. The objective is to reduce the amount of inspection needed to obtain an accurate segmentation: the annotator only needs to provide feedback on the model-selected queries, which is far cheaper than inspecting the full video for dense annotation.
W1.5 Why not use the stronger model directly?
In our experiments the oracle is simulated with ground-truth labels. Simulating is the standard convention, and most existing active learning and active TTA work follows it, including SimATTA and BiTTA.
As the first work in this setting, our primary objective is to establish the problem formulation and to evaluate how sparse feedback can be selected and used for video-specific adaptation. Investigating external feedback sources, including imperfect annotations from stronger models, is an important direction that we leave for future research.