acceptodds
Under review as a conference paper at ICLR 2027

Where to Label and How to Adapt at Test Time for Temporal Action Segmentation

Abstract

This paper studies where to request frame-level annotations and how to use them to adapt a pretrained temporal action segmentation model at test time. To determine where to label, we propose an uncertainty-guided selection strategy that partitions each test video into consecutive regions with approximately equal uncertainty mass and queries one representative frame from each region. This design allocates annotations more densely to uncertain portions while retaining broad temporal coverage. To determine how to adapt, we use the queried labels as sparse semantic supervision and exploit their agreement with model predictions to dynamically regulate the adaptation trajectory across epochs. Experiments on standard benchmarks show consistent improvements under highly sparse annotation budgets. On Breakfast with MS-TCN, our approach improves accuracy from 65.4% to 81.9%, while querying only 0.16% of video frames on average.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
temporal action segmentation, test time adaptation, video understanding

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_Hj8EOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper introduces the problem of active test-time adaptation for temporal action segmentation and adopts an episodic setting where the model is reset after adapting independently to each video. To select frames for labeling, the authors propose an uncertainty-guided frame selection method that divides the video into K regions with equal cumulative prediction entropy and selects one frame per segment that minimizes the uncertainty-weighted distance. During adaptation, the method employs a sparse classification loss for labeled frames and a temporal smoothing loss for the full sequence. Dynamic trajectory regularization based on KL divergence from the prediction of the previous epoch is employed, with the regularization strength adjusted both globally and segment-wise depending on the degree of agreement between the current prediction and the ground-truth label frames. Experiments demonstrate performance improvements over existing active TTA methods designed for image classification and simple selection baselines.

Critical strengths

The paper experimentally demonstrated the importance of which specific parts are designated as labels. Performance improved compared to pre-trained models, and the fact that the performance gain was better than other frame selection methods is particularly noteworthy. This trend remained consistent even when the model was changed.

Critical weaknesses

The most significant weakness is the lack of a sufficient justification for the setup involving active TTA in TAS. In active TTA for image classification, the labeled sample and the sample being evaluated are distinct images; however, in this paper’s setup, the labeled frame and the frame evaluated after adaptation belong to the same video. Given the strong correlation between adjacent frames in TAS, a ground-truth label at a specific time point serves as a strong cue for the entire surrounding segment, rather than providing information limited to just that single frame. In fact, the proposed method aims to propagate the label’s influence to neighboring frames via a smoothing loss; consequently, it is difficult to distinguish whether the substantial performance improvement over the pre-trained model stems from the test-time adaptation effect or simply from the direct provision of ground-truth cues. In this regard, the setup appears closer to interactive segmentation or timestamp supervision, where a user designates specific frames and the model propagates this information across the video, than to standard TTA, yet the paper fails to discuss or compare the method against these related approaches. Furthermore, the paper lacks an explanation regarding which real-world applications would require human labeling for every test video after deployment, nor does it explain why the “stronger model” mentioned in the introduction is not used directly if it is available as an oracle. I believe the authors should present specific, realistic scenarios where this setup is necessary and clearly articulate the distinctions between their approach and related methods.

Additionally, regarding the methodology, the reliance on predictive uncertainty, a concept already widely used in existing active learning and active TTA studies, limits the novelty of the frame selection component. Although the authors frame their contribution as distributing uncertainty along the temporal axis, the resulting selection rule is fairly simple in practice. With the default L1 distance, minimizing Eq. (7) within each region yields the entropy-weighted median of that region, and since every region contains an equal share of the total entropy mass, the procedure essentially amounts to selecting frames at evenly spaced quantiles of the cumulative entropy.

Questions for authors
  • What are the practical scenarios where active TTA is applied to TAS?

  • In experiments comparing episodic and online settings, how is the sequence of videos determined for the online setting? I imagine the results could vary significantly depending on the order.

  • I would like to know more details about the implementation of SimATTA and BiTTA.

2 comments

Thank you for your comments. W1 raises several related points, which we address in turn as W1.1–W1.5.

W1.1 Labeled and evaluated frames belong to the same video

Yes, and this is by design. In TAS the test sample is the whole video, and the shift we adapt to is specific to each video.

In image classification, each test image is one sample with one label, so a queried label belongs to a different sample from the ones it helps. TAS is different. A TAS model takes a video of TT frames and outputs TT labels as a single structured prediction. For each video, we query a small fraction of its frames (0.16% on average on Breakfast with MS-TCN), adapt the model, and evaluate on all TT frames. A queried frame is therefore a partial label on the test video and should not be considered as the label on an image.

In addition, active adaptation from the same video is needed because the shift is video-specific. Each test video has its own action composition, durations, and transitions, which differ from what the globally trained model has learned. On Breakfast:

  • 42.1% of test videos follow an action ordering that never occurs in the training data, and
  • 37.2% of actions differ in mean duration between training and testing by a factor of at least 1.25.

W1.2 Adaptation or label propagation?

Thank you for raising this point. Adaptation and label propagation are not contradictory. Propagating information from a few labeled samples to many unlabeled ones is the purpose of active and semi-supervised learning in general, and combining queried labels with label propagation or pseudo-labeling is common practice. Existing active TTA methods such as SimATTA and BiTTA use the queried feedback to update the model and additionally train on model-generated labels for unlabeled samples.

Ours follows the same principle, using only sparse label supervision, and the propagation likewise takes place through the model instead of spreading labels directly over the outputs. Specifically, the queried labels provide supervision where the model is most uncertain, the model weights are updated, and the updated model re-predicts all TT frames, including those far from any queried frame.


W1.3 Relation to timestamp supervision and interactive segmentation

The key difference between our setting and these two is who decides where to label: in ours it is the model, based on its uncertainty, which is the "active" part.

SelectionHow locations are decided
TimestampPassiveRandom, independent of the model
InteractiveCorrectiveUser finds errors, requires access to GT
OursActiveModel uncertainty, no access to GT

In timestamp supervision, the locations are normally sampled at random and do not depend on the model.

In interactive segmentation, the location of each interaction is typically simulated in benchmarks from the error region, which requires access to the ground truth to decide where to select.

In contrast, in our setting the model decides where to query from its own predictive uncertainty, with no access to the ground truth or to its errors. The goal is to select frames better, so that the same annotation budget yields a larger improvement from adaptation. Our main contributions are deisgned for this, making them distinct from the other settings.


W1.4 & Q1. Realistic usecase

A concrete application is human-in-the-loop annotation of extensively long procedural videos. The objective is to reduce the amount of inspection needed to obtain an accurate segmentation: the annotator only needs to provide feedback on the model-selected queries, which is far cheaper than inspecting the full video for dense annotation.


W1.5 Why not use the stronger model directly?

In our experiments the oracle is simulated with ground-truth labels. Simulating is the standard convention, and most existing active learning and active TTA work follows it, including SimATTA and BiTTA.

As the first work in this setting, our primary objective is to establish the problem formulation and to evaluate how sparse feedback can be selected and used for video-specific adaptation. Investigating external feedback sources, including imperfect annotations from stronger models, is an important direction that we leave for future research.

W2.1 Use of predictive uncertainty limits the novelty.

We respectfully disagree that using predictive uncertainty limits the novelty of the frame selection. Predictive uncertainty is the widely used informativeness signal in active learning and active TTA (e.g., SimATTA), so the novelty of a selection strategy lies in how uncertainty is exploited, not in whether the measure has been used before. As you also recognize, one of our key contributions is distributing uncertainty along the temporal axis through our uncertainty-aware temporal allocation. The other concerns the adaptation stage, where the active queries are also used as adaptation signals. To the best of our knowledge, neither has been explored in previous active TTA work.


W 2.2 Simplicity of the selection rule

We agree that using the ℓ1\ell_1 cost in Eq. (7) admits a weighted-median solution. However, we argue this is an advantage rather than a limitation: a selection rule that is simple yet effective is what test-time adaptation calls for. This is also acknowledged by reivewer CDTV as a strength. Concretely:

  1. Using ℓ1\ell_1 for Eq.7 can be implemented efficiently without iterative numerical optimization, which matters at test time when a video contains thousands of frames.

  2. It is also effective, it outperforms more elaborate strategies such as clustering-based and MC-dropout-based selection (Table 1).

  3. Importantly, evenly paced quantiles of cumulative uncertainty (with l1) should not be confused with equi-distant temporal sampling, as the resulting sampling locations adapt to the nonuniform uncertainty distribution across the video.

  4. Last but not least, our formulation is not restricted to l1 distance. As shown in Table 4, alternative distance functions give competitive results, which shows that the formulation is more general than the weighted-median interpretation.


Q2. Video order in the online setting

The online results reported in the paper process the test videos in the order listed in each dataset's test-split file, without additional shuffling. Following your comment, we additionally ran three random permutations of the test order (Breakfast, MS-TCN).

SettingAccEditF1@10F1@25F1@50
Online, 3 random79.0±0.279.0 \pm 0.276.7±0.476.7 \pm 0.479.8±0.279.8 \pm 0.275.2±0.375.2 \pm 0.361.8±0.361.8 \pm 0.3
Episodic81.979.581.877.865.5

As we can see, the variation across permutations is small: the standard deviations of accuracy and F1@50 are only 0.2 and 0.3, respectively, so the online performance does not depend strongly on a particular random ordering.

We attribute this to the per-video feedback: each video receives its own queried labels and adaptation epochs, so the model re-specializes to the current video regardless of the state inherited from the previous one. The remaining gap to episodic adaptation is present under every order, which suggests it stems from parameter drift accumulated across videos and not from a particular ordering. This supports what we dicsussed this in L. 402-412 and Appendix A.3.


Q3. Implementation of SimATTA and BiTTA

Both methods were proposed for image classification, where each batch contains images. For TAS, we treat each test video as one batch and its frames as the samples.

SimATTA. Frames whose entropy lies in the top 20% of the video form the high-entropy candidate set. We use this within-video percentile instead of the fixed threshold ehe_h, because entropy scales vary across videos and datasets. We cluster the last-stage features of these candidates with K-means into KK clusters and annotate the frame closest to each centroid with its ground-truth label. Following SimATTA's low-entropy sample selection, the KK lowest-entropy frames within the bottom 20% are assigned the source model's prediction. KK is the number of frames to be labeled and they are kept consistent with ours. Annotated and pseudo-labeled frames are used for adaptation.

BiTTA. We select frames by MC-dropout uncertainty. We run 4 forward passes of the source model with dropout enabled and average the softmax outputs to obtain π(c∣xt)\pi(c \mid x_t). The certainty of each frame is Ct=π(y^t∣xt)C_t = \pi(\hat{y}_t \mid x_t), the MC-averaged probability of the deterministic prediction. The KK frames with the lowest certainty are selected for annotation. In addition, KK most certain are assigned y^t\hat{y}_t as pseudo-labels. One difference from the original setting is the form of feedback: BiTTA's oracle only indicates whether y^t\hat{y}_t is correct. To compare all methods under the same annotation protocol, the selected frames receive their ground-truth semantic labels, as in SimATTA and our method.

Reviewer_CDTVOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces active test-time adaptation for temporal action segmentation. For each test video, a pretrained model may query ground-truth labels for a small number of frames and then adapt to that video before being reset for the next one. Two components are proposed. For where to label, frame-wise entropy is accumulated over time and the video is partitioned into K regions of equal uncertainty mass. Within each region the frame minimizing the uncertainty-weighted temporal distance to all other frames is queried. For how to adapt, the queried labels supply a sparse cross-entropy loss alongside the standard smoothing loss. A KL term to the previous epoch's predictions is added, with its weight set per frame according to whether the queried frame in that region is currently predicted correctly and how many queried frames overall agree with the model. Experiments on GTEA, 50Salads, and Breakfast with MS-TCN and ASFormer compare against random, equidistant, boundary-based, SimATTA, and BiTTA selection under the same budget. The method gives the best results in all settings. Additional experiments show that episodic adaptation beats online adaptation, that the dynamic KL also helps other selection strategies, and that the method matches ground-truth boundary annotation at lower annotation cost.

Critical strengths
  1. The setting is practical and clearly motivated. Per-video adaptation with a handful of oracle labels is realistic for deployed TAS systems, and the episodic formulation is justified empirically by showing that cross-video shifts in Breakfast are large and that online adaptation underperforms.
  2. The equal-uncertainty-mass partition is a simple and sensible selection rule. It is equivalent to inverse-CDF sampling over the entropy distribution, which guarantees coverage while concentrating queries where uncertainty is high, and it has no hyperparameters.
  3. The within-region selector has a clean interpretation. Minimizing the uncertainty-weighted L1 distance picks the uncertainty-weighted median of the region, and Table 4 shows the choice of distance barely matters.
  4. Using queried labels as a progress signal for regularization is a reasonable idea. Agreement on annotated frames is a free, label-grounded proxy for whether a region has converged, and Table 9 shows the dynamic KL improves Edit and F1 for every selection strategy tested.
  5. The ground-truth boundary comparison (Tables 5, 10) is an interesting control showing that the chosen frames are as informative as boundary-adjacent labels.
Critical weaknesses
  1. Table 1 conflates selection and adaptation. The baselines use plain sparse adaptation while "Ours" also uses the dynamic KL, so the headline margins mix the two contributions.
  2. The selection contribution is small once adaptation is equalized. From Table 9, the proposed selection adds 0.8 accuracy, 1.4 Edit, and 2.1 F1@50 over equidistant sampling, on one dataset and one backbone. The "where to label" claim rests on this single fair comparison.
  3. The dynamic weighting adds little over a plain moving-reference KL. In Table 3 the step from moving KL to dynamic KL is 0.7 accuracy and 0.5 Edit, and the three-level weights with a 5x jump at high agreement are hand set.
  4. Most of the reported gain comes from the labels themselves. Random selection with plain adaptation already lifts Breakfast MS-TCN from 65.4 to 75.8 and Equi to 78.2, so the abstract's 65.4 to 81.9 framing overstates the method's contribution.
  5. Hyperparameters appear to be selected on test data.
  6. The claim that ground-truth boundary annotation costs 6x more borrows a dense-labeling estimate from Ma et al. (2020). The GTB control labels two frames per anchor, not dense labels, so the cost argument is overstated.
Questions for authors
  1. Please confirm whether the baselines in Table 1 use the dynamic KL.
  2. How were the per-dataset learning rates and the α and threshold values chosen? Was any held-out data used, or were they selected on test performance?
  3. Why does BiTTA underperform the pretrained model on two datasets when it has access to correct labels? Was its learning rate tuned separately?
  4. What is the wall-clock adaptation time per video on Breakfast for MS-TCN and ASFormer?
3 comments

Thank you for your comments. We address each point below.

W1 & Q1. Table 1 conflates selection and adaptation

Thank you for your question. Table 1 does not apply dynamic KL to other baselines for three concrete reasons.

  • Both selection and adaptation are our contributions, and they are linked: the actively queried labels are used not only as supervision but also to guide the adaptation. To our knowledge, a regularizer whose reference moves across adaptation epochs and whose strength is set by the queried labels has not been used in active TTA or in TAS, so the adaptation strategy as a whole is our contribution, SimATTA and BiTTA are implemented with their own adaptation objectives, as published. Adding our regularizer to them would no longer evaluate those methods, but a hybrid of theirs and ours.
  • It is designed together with our selection. The frame-wise weights in Eq. (14) are defined on the uncertainty regions produced by our selection, each of which contains exactly one queried frame. Other strategies (random, top-KK, clustering) define no such regions, so applying the dynamic KL to them requires additional design choices.
  • Table 1 follows the standard practice of comparing complete methods, with the components isolated in ablations.

Ablation study in Table 3 shows that two components each bring consistent gains. On Breakfast with ASFormer, active labels raise Acc by 7.6 and F1@50 by 3.0, and dynamic KL adds additional 2.7 and 8.0, respectively.


W2. The selection contribution is small once adaptation is equalized

We agree that the overall gain over "Equi + dynamic KL" is modest once dynamic KL is applied.

However, our adaptation contribution is the whole trajectory regularization: the KL term, its moving reference, and its dynamic weighting. So "Equi + dynamic KL" should be regarded as one variant of our approach, and it shows dynamic KL effectively improves the "Equi" baseline significantly.

More interestingly, this variant is the strongest alternative, while other strategies driven by uncertainty alone are worse under the same dynamic KL. This shows that temporal coverage is essential when adapting TAS models, supporting the design choice of our selection, which keeps full temporal coverage and adds uncertainty awareness on top. We believe this provides insight into how uncertainty should be used in TAS: on its own it concentrates the queries in a few regions, and it becomes effective when combined with temporal coverage.


W3. The dynamic weighting adds little over a plain moving-reference KL

As we mentioned in W2, our adaptation contribution is the whole trajectory regularization: the KL term, its moving reference, and its dynamic weighting. To the best of our knowledge, a reference that moves across adaptation epochs with a strength set by the agreement between queried labels and predictions, has not been studied. So the moving-reference KL should be considered an intermediate form of our own method. The regularizer as a whole improves over plain sparse adaptation by 2.7 Acc, 4.0 Edit, and 8.0 F1@50, and the moving reference and dynamic weighting together add 1.9 Acc, 1.2 Edit, and 2.0 F1@50 over a fixed-reference KL (see Table 3).

Furthermore, the dynamic weighting helps most on segment-level quality what it was deisgned for: it adds 1.0 F1@25 and 1.3 F1@50 over the moving reference, and it improves all five metrics. It comes at no extra cost, since it reuses the queried labels that are already available and needs no additional annotation or forward pass.


W4. Abstract performance statement

Thank you for raising this point.

We would like to clarify the purpose of the abstract sentence. As this is the first work on active adaptation of TAS models, this sentence is meant to show what the newly proposed setting can achieve, with the annotation cost stated explicitly. Starting from a pretrained model, querying 0.16% of the frames brings accuracy from 65.4% to 81.9%. It is not intended as a comparison with alternative methods to highlight our contribution.


W5 & Q2. Hyperparameter selection

  • α\alpha and the thresholds (τlow,τhigh)(\tau_{low}, \tau_{high}) were selected from the sensitivity analysis on a single dataset and backbone (Fig. 5) and then fixed for all other datasets and backbones without further tuning. The analysis also shows low sensitivity: performance varies only between 81.4 and 81.9 over a wide range of values.
  • Learning rates were set manually per dataset and backbone to ensure a stable adaptation process, judged from the adaptation loss curve. We did not run a grid search on test performance, and the same learning rates are shared by all compared methods.

TAS benchmarks provide standard cross-validation splits but no held-out validation set, so we selected the hyperparameters on one split and applied them to the others.

W6. The 6×6\times annotation cost of ground-truth boundaries

At first glance, GTB might seem to require far less effort than dense labeling, since it needs only two frame labels per anchor. However, annotation cost is determined not only by the number of labeled frames, but by the effort required to obtain them.

In practice, dense annotation is not done by labeling every frame individually. Annotators locate the action boundaries and then assign one label to each resulting segment, so the main effort lies in finding each boundary precisely.

GTB requires the same: to label the two frames adjacent to a boundary, the annotator must first locate that boundary exactly (L.462–467). Its cost is therefore comparable to that of dense labeling, despite the small number of labeled frames.


Q3. Why does BiTTA underperform the pretrained model on two datasets?

BiTTA's learning rate was not tuned separately. Learning rates were specified per dataset and backbone and shared across all methods.

BiTTA estimates each frame's uncertainty with Monte Carlo dropout and queries the top-KK most uncertain frames. Top-KK selection tends to concentrate the annotations on a single local peak instead of distributing them over time. We quantify this on Breakfast with MS-TCN by normalizing each video's length to [0,1][0, 1] and measuring the standard deviation of the queried positions. For BiTTA, the median standard deviation is only 0.005, compared with 0.29 for our sampler and 0.27 for Equi. The queries are therefore largely redundant and leave most of the video unlabeled, which can degrade performance (L.397–401).


Q4. Wall-clock adaptation time per video

On a single NVIDIA GeForce RTX 4080 SUPER GPU, the average time per video on Breakfast is approximately 0.2 s for MS-TCN and 8.4 s for ASFormer, using five adaptation epochs with our method in the episodic setting.

Area_Chair_73s3OpenReview ↗

Response needed

Dear Reviewer CDTV,

Now that the reviewer-author interaction deadline is approaching, please respond to the authors' rebuttal ASAP.

Best, AC

Reviewer_Pd71OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes active test-time adaptation for temporal action segmentation. Problem setup: for each test video, a few frames are selected for ground-truth annotation, the pretrained model is adapted on that single video, and then the model is reset before the next video. The proposed method splits the video into regions of equal entropy mass and picking one frame per region. Adaptation combines sparse classification loss, temporal smoothing, and a KL regularizer whose strength depends on agreement with the queried labels. Experiments use MS-TCN and ASFormer on GTEA, 50Salads, and Breakfast, showing gains over the pretrained model and several selection baselines.

Critical strengths
  1. Part of the research question, temporal action segmentation, is an important and promising topic.
  2. The method is easy to follow. The selection rule (equal uncertainty mass per region) is intuitive and provides a natural way to balance uncertainty with temporal coverage.
Critical weaknesses
  1. The problem setup is not motivated. I struggled to understand why this setting makes sense. Test-time adaptation is designed for distribution shift, but the experiments use standard in-domain splits with no shift at all. Essentially, the model is fine-tuned on oracle labels of each individual test video, then those labels are thrown away when the model resets. This is more like a weaker active learning setting rather than an test-time adaptation problem. More specifically, it just includes a new partial labeled same domain video from testing to training then throw it away. The paper does not explain when the proposed setting would be preferable in practice.
  2. limited novelty for the method. The frame selection based on sampling by entropy. Then the adaptation loss combines the timestamp-supervision objective from Li et al. (2021) with a KL regularizer toward the previous epoch. Neither component is new, and the combination does not introduce a clear technical insight.
Questions for authors

Could the authors test a setting with actual distribution shift (e.g., cross-dataset or cross-environment) to demonstrate where test-time adaptation is needed? This is more similar to few shot setting for TAS.

1 comment

Thank you for your comments, and we address your concerns below.

W1.1 No shift at all in TAS

We respectfully disagree that our setup has no shift at all. Conventional TTA is often studied under specific domain shifts in image classification. TAS has a different source of shift: each test video differs from the pooled training distribution in ways that are specific to that video.

The reason lies in how TAS models are trained. TAS is formulated as frame-wise action classification, where a model is trained jointly on videos with diverse execution patterns and even multiple activities. Although temporal dependencies are modeled, the resulting classifier is globally optimized across this mixture of distributions.

At test time, each test video corresponds to a particular activity or execution, exhibiting its own action composition, duration distribution, and transition characteristics. This creates a discrepancy between the globally trained model and the video-specific temporal structure. On Breakfast,

  • 42.1% of test videos show an action ordering that never occurs in the training data, and
  • 37.2% of actions differ in mean duration between training and testing by a factor of at least 1.25.

Such shift in TAS is video-specific and has been little explored, and to the best of our knowledge, we are the first to address it with active adaptation at test time.


W1.2 Relation to active learning, and why the model is reset

Our setting differs from active learning in its goal. Active learning selects samples to label so that a model retrained on the enlarged training set improves on future data. Our goal is the test video itself: the queried labels are used at inference to correct the prediction on that video, without access to the source data and without retraining.

Resetting the model after each test sample is the episodic test-time adaptation setting, an established problem formulation in which every test sample is adapted independently from the source model (e.g., MEMO, Zhang et al., 2022).


W1.3 Practical application

A concrete application is human-in-the-loop annotation of extensively long procedural videos. The objective is to reduce the amount of inspection needed to obtain an accurate segmentation: the annotator only needs to provide feedback on the model-selected queries, which is far cheaper than inspecting the full video.


W2. Use of predictive uncertainty limits the novelty.

We respectfully disagree that using predictive uncertainty limits the novelty of the frame selection. Predictive uncertainty is the widely used informativeness signal in active learning and active TTA (e.g., SimATTA), so the novelty of a selection strategy lies in how uncertainty is exploited, not in whether the measure has been used before. As reviewer Hj8E recognizes, one of our key contributions is distributing uncertainty along the temporal axis through our uncertainty-aware temporal allocation. The other concerns the adaptation stage, where the active queries are also used as adaptation signals. To the best of our knowledge, neither has been explored in previous active TTA work.

However, we do not claim the sparse objective as a contribution, as stated in L.78–81: "Inspired by (Li et al., 2021; Souri et al., 2022), we ...".

Lastly, we believe our key insights are twofold.

  1. We show that the actively queried labels can play two roles. They serve as the semantic supervision for adaptation and also as a signal that guides the adaptation itself, and this second role comes at no additional annotation cost.
  2. We show that entropy alone is not enough in TAS; temporal coverage matters. Uncertainty is most useful when the queries it selects are also spread over the whole video.

Q1. Testing under actual distribution shift, and the relation to few-shot TAS

Unfortunately, we are unable to evaluate under a cross-dataset or cross-environment shift. Our setting keeps the action classes fixed, and we could not find videos recorded in a different dataset or environment that share the action classes of these benchmarks.

The shift we study is the one present within the standard splits, where the predictions on a specific test video can be unreliable (see response to W1.)

Few-shot TAS addresses a different scope from ours. The most significant difference is that it extends what a model can recognize, generalizing to novel tasks from a few labeled support videos, whereas our setting adapts a model to a test video whose action classes it already knows.

Combining test-time adaptation with few-shot TAS, for example adapting to novel tasks at inference, is an interesting direction, but it lies outside the scope of this paper and we leave it for future work.

Comments on the paper

2 comments

Thank you for your follow-up. We appreciate your comment, but we see these points differently.


1. this paper needs to label each test video, train the model for that testing video, then discard it. [...] how is that different from just memorizing each video?

We believe these comments conflate training with adaptation.

Training aims to produce a model that is to be reused on other data: what it delivers is a better model.

Adaptation takes a trained model and adjusts its weights to a given test sample at inference time, with the purpose of predicting that sample better: what it delivers is the prediction for that sample. This is why the adapted weights can be discarded afterwards without any loss. In fact, adapting on a test sample, predicting it, and resetting the model is an established protocol which is called episodic TTA (e.g., MEMO, Zhang et al., 2022).

That being said, we argue ours is not memorization, but adaptation. Memorization typically refers to a behavior observed in model training, where a model fits its training labels without generalizing to new data. In our setting, the model is given labels for 0.16% of the frames on Breakfast, and its aim is to improve predictions on the remaining 99.84%. Such improvements have been extensively validated by our experiments.

Furthermore, regardless of “memorization” framing, one would still need to decide which 0.16% of the frames to label for better performance, and this is also one of our main contributions (where to label).


2. action order variations are common characters within standard TAS setting. Modeling long time temporal dependency for various ordering is the goal of TAS model.

We agree with you and we also acknowleged this point in our previous response, but we see no contradiction with our conclusions from the paper. What we show in the paper is that temporal dependencies and order variations can be handled better with test-time adaptation: adapting to each test video improves both MS-TCN and ASFormer consistently on all three benchmarks (Table 1). Test-time adaptation is thus complementary to better temporal modeling, not a replacement for it.

If you would like further clarification on the difference between training and test-time adaptation, we are happy to provide it.

Area_Chair_73s3OpenReview ↗

rebuttal, feedback and discussion

Dear Reviewers,

Thank you for your efforts so far. The authors have responded to your reviews with detailed feedbacks. Please check them carefully (you may also check other reviewers' comments) and provide your feedbacks.

Best, AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.