- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
per proposes an active test-time adaptation framework for temporal action segmentation (TAS). The method selects informative predicted action boundaries for sparse annotation and introduces agnostic and semantic directional overwrite strategies to construct adaptation labels. The model is subsequently adapted using sparse classification supervision and temporal smoothing. Experiments on Breakfast, GTEA, and 50Salads with MS-TCN and ASFormer demonstrate overall performance improvements under limited annotation budgets in the evaluated transductive setting.
Critical strengths
1.The paper addresses a meaningful problem by introducing active human feedback into test-time adaptation for temporal action segmentation. The boundary-centric query strategy is motivated by the temporal structure of TAS, and the directional overwrite mechanisms provide a practical way to construct supervision from sparse feedback.
2.The experimental evaluation covers three established TAS benchmarks and two representative backbones. Comparisons with active selection and adaptation baselines, annotation-budget analyses, and a user study provide useful empirical evidence for the effectiveness and practical potential of sparse boundary supervision in the evaluated setting.
Critical weaknesses
1.Potentially incorrect supervision induced by directional boundary overwrite. Equations (11) and (13) assign the same adaptation label to both frames adjacent to every queried predicted boundary. However, if a predicted boundary already coincides with the ground-truth transition, these two frames should have different labels under the standard frame-wise segmentation protocol. Consequently, even with correct oracle feedback, the overwrite rule necessarily assigns an incorrect target to at least one of the two frames. This raises a fundamental concern about the reliability of the proposed adaptation supervision. The paper does not quantify how often this situation occurs or whether the benefits of local label consistency outweigh the resulting supervision errors.
2.Insufficient validation of the distribution-shift motivation. The paper emphasizes temporal distribution shifts during deployment, but the main experiments evaluate adaptation on standard test splits without explicitly examining controlled temporal shifts or cross-domain scenarios. The results support the effectiveness of sparse test-set supervision in the evaluated setting, but do not sufficiently establish robustness under the distribution changes motivating the broader deployment claims. The transductive adaptation protocol itself is valid; the concern is the limited empirical support for the broader deployment claims.
Questions for authors
1.When a predicted boundary coincides with the ground-truth transition, Eqs. (11) and (13) still assign the same label to both adjacent frames, making at least one adaptation target incorrect under frame-wise ground-truth labels. How does the proposed method address this issue? Could the authors quantify the correctness of the generated adaptation labels for both agnostic and semantic overwrite, and clarify whether preserving correctly localized boundaries would affect the reported results?
2.How robust is the proposed framework to incorrect human feedback? Given that the main experiments use simulated oracle responses and the user study evaluates annotation time rather than feedback accuracy, can the authors report results using actual human annotations or analyze the effect of imperfect feedback?
3.Can the authors provide experimental evidence under explicit temporal distribution shifts, such as changes in action-duration distributions or cross-domain evaluation, to substantiate the claimed deployment robustness?
W1 & Q1. Already-correct boundaries
Thank you for asking about this detail. The short answer is: our implementation imposes no agnostic overwrite when a predicted boundary matches the ground truth. We did not mention this in the paper for two reasons: 1) the case is extremely rare (<1%), and 2) judging the exact end frame of an action is ill-posed for annotators.
Specifically, over 99% of predicted boundaries across all datasets do not coincide exactly with the ground-truth transition, so nearly every query concerns a boundary that does need to move. The exact-match case is very rare, which is why we did not explicitly discuss it in the submission and focused instead on the far more common case of misaligned boundaries. In our implementation, when a predicted boundary does coincide exactly with the ground truth, it is kept as it is and no agnostic overwrite is applied.
Moreover, the directional query is closer to how annotation works in practice. Action transitions are ambiguous at the frame level, so asking an annotator whether a predicted boundary is exactly right is an ill-posed question, whereas judging whether it comes too early or too late is easy and consistent.
We will mention how we handle it explicitly in the revision.
W2 & Q3. Validation of the shift motivation
Unfortunately, we are unable to evaluate under a cross-domain shift: TAS models are trained on a fixed set of action classes, and we could not find videos recorded under a different distribution that share the action classes of these benchmarks.
Although training and test videos come from the same benchmark, the standard splits already contain a substantial temporal shift. We measured it on Breakfast:
The test sets therefore substantially differ from the training sets in both action durations and temporal patterns, and the improvement validate the effectiveness of our approach.
Q2. Noisy feedback
Thank you for your great question. We would like to first point out that simulating the annotator with ground-truth labels follows the standard protocol in existing active test-time adaptation (what SimATTA, BiTTA also adopt). A full human-in-the-loop evaluation is prohibitive at this scale: it would need human answers at every queried boundary across all test videos, splits, backbones and settings.
We therefore simulated imperfect overwrite on Breakfast with MS-TCN and agnostic overwrite. For each video, we randomly flip 10%, 20%, 30% of the overwrite direction of queried boundaries and report the performance in the table below. As we can see, performance generally declines as the error rate rises, but the method still manages to stay above the pretrained model. The method is therefore robust to moderate annotation error.