acceptodds
Under review as a conference paper at ICLR 2027

Boundary-Aware Progress-Balanced Prefix Learning for Strictly Online Temporal Action Segmentation

Abstract

Temporal action segmentation assigns an action label to every frame of an untrimmed video. Most segmentation models are trained with complete videos, whereas strictly online inference can access only a growing observed prefix. Directly applying a full-sequence model to prefix inference substantially degrades segment quality, and causal constraints recover only part of the loss. We characterize this mismatch as temporal context distribution shift and address it by reconstructing training prefixes and their supervision. Our method first balances prefix endpoints across observation progress, then increases sampling near action transitions, and finally weights frame-level supervision toward the prefix endpoint. The approach modifies training but introduces no additional inference branch. With FACT as the backbone, it reaches [email protected] scores of 68.42% on GTEA and 39.68% on 50Salads, improving over causal adaptation by 9.32 and 26.86 points. Applying the strategy to ASFormer also improves [email protected] over causal adaptation on both datasets, indicating that its benefits can extend to other attention-based backbones. These results show that aligning prefix coverage and supervision is important when adapting attention-based segmentation models to strictly online inputs.

Primary area:
learning on time series and dynamical systems
Keywords:
Online Temporal Action Segmentation, Prefix Learning, Distribution Shift, Temporal Modeling, Video Representation Learning, Deep Learning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_THXFOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper aims to adapt the offline tas model to the online setup. It argues that simply making the model causal is not enough because models trained on full videos has a distrubution shitf of prefix length during testing. The authors terms this as the temporal context distribution shift and propose a input sampling startegy to mitigate this. Specifically, the authors sample prefix length endpoints across progress bins, nudge some endpoints towards action boundaries, and also weight the loss towards the end of the prefix. Experiments on two action segmentation datasets show improvements compared to other existing works.

Critical strengths
  1. The temporal context distribution shift (TCDS) is an interesting and valid point to consider when adapting in offline models to online setup. This paper makes the observation that even with causality enforced, a model trained only on complete videos has never seen the prefix states it will encounter online, and its intermediate representations are shaped by that.

  2. The method is training-only and remains clean at test time. By prefix length sampling and loss weighting during training, the model gains capability to recognize actions during online testing without additional branches. This likely make the contribution more applicable to any other backbone.

Critical weaknesses
  1. The most significant issue is its testing protocol. At each time of a streaming video, making prediction for a single frame would require the model to access and process all the previous frames. This has two implications: 1) the method is with complexity of O(T2)\mathcal{O}(T^2) per video of length T, yet the paper did not reports latency nor FLOPs. 2) the method has access to the unbounded history in its raw input format does not form a fair comparsion with methods like OnlineTAS which is memory bounded. The authors should acknowledge this when performing comparsion.

  2. The paper only evaluates on two small-sized TAS benchmarks, missing Breakfast. Breakfast is a larger benchmark which is also widely used in the TAS community, having far more videos and more varies temporal structure. It is very important to have the comparsion on this dataset to show its sampling and loss weighting can generalize to more difficult settings.

Questions for authors
  1. Since each prefix is recomputed independently, it naturally ensures the causality, i.e., making predictions on t does not rely on access to frames >t. It does not require the every frame indexed prior to the prefix be blind to its own future. Frames 1...t-1 are already observed, letting i see j (1<i<j<t-1) should be okay. What is the rationale in keep the causal conv/attention mask given the prefix training paradigm? Please justify.

  2. Add evalutions on breakfast.

  3. What is the inference cost of the methods? The authors should report latency between frames as well as the FLOPs aligned with the video length.

1 comment

Dear Reviewer THXF,

As the author-reviewer interaction period is ending, please respond to the authors' feedback ASAP.

Best, AC

Reviewer_W8CoOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a training strategy for adapting offline Temporal Action Segmentation (TAS) models to strictly online settings. The authors introduce the concept of Temporal Context Distribution Shift (TCDS), arguing that causal architectural adaptation alone is insufficient to address the mismatch between full-video training and prefix-based online inference.

To address this issue, the paper combines progress-balanced prefix sampling, transition-guided endpoint sampling, and prefix-end-weighted supervision. These components aim to balance training across different observation stages, emphasize challenging action transitions, and prioritize predictions near the current observation endpoint.

Experiments on GTEA and 50Salads demonstrate substantial improvements over causal adaptation baselines, particularly with FACT. The approach also improves performance with ASFormer.

Critical strengths
  1. Simple and effective approach. The proposed training strategy is straightforward and achieves substantial improvements on GTEA and 50Salads, particularly in segment-level metrics.

  2. Generalizability across backbones. The method improves performance with both FACT and ASFormer, demonstrating its applicability to multiple attention-based TAS architectures.

  3. Practical offline-to-online adaptation. The approach provides a simple way to adapt existing offline TAS models to online settings without introducing additional inference branches.

Critical weaknesses

Weaknesses

  1. The motivation behind TCDS is not convincing.

The paper argues that causal adaptation alone cannot address TCDS. However, existing causal online TAS methods, such as ProTAS, already supervise predictions at every temporal position, effectively covering all prefix endpoints in a single forward pass. Therefore, it is unclear why TCDS would remain an issue for these methods or why explicit prefix sampling is necessary.

  1. Progress-balanced sampling appears similar to uniform random truncation.

The proposed strategy uniformly samples a temporal interval before selecting an endpoint. When the intervals have approximately equal lengths, this is nearly equivalent to uniformly sampling an endpoint from the entire video. A direct comparison with uniform random truncation is needed to justify this design.

  1. Limited methodological novelty.

Beyond progress-balanced sampling, the main contributions are transition-guided prefix sampling and prefix-end-weighted supervision. While effective, both are relatively straightforward techniques based on boundary oversampling and linear loss reweighting. The paper does not introduce a substantially new modeling or optimization mechanism.

Moreover, the ablations combine progress balancing and transition-guided sampling, making it difficult to assess their individual contributions.

Questions for authors

Please see the weakness.

2 comments

Dear Reviewer W8Co,

As the author-reviewer interaction period is ending, please respond to the authors' feedback ASAP.

Best, AC

AuthorsOpenReview ↗

Follow-up to Reviewer W8Co

Dear Reviewer W8Co:

Thank you again for your valuable comments. We carefully addressed each of your concerns through additional controlled experiments, clearer explanations, and corresponding revisions to the manuscript.

As the discussion period is approaching its end, we would like to kindly follow up and invite you to review our response and revised manuscript when convenient. We sincerely hope that the added evidence and clarifications have addressed your concerns. If any questions remain, we would be grateful for the opportunity to discuss them further.

Thank you very much for your time and consideration.

Reviewer_EyGFOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper looks at strictly online temporal action segmentation, where the model must label each frame using only the frames seen so far. The authors argue that a model trained on full videos sees different inputs than the growing prefixes it gets at test time. They call this temporal context distribution shift (TCDS). Method changes only the training. It cuts each training video at a sampled endpoint, spreads the endpoints over the early, middle and late parts of the video,picks more end points near action changes, and gives more loss weight to the frames near the end of the prefix.

Critical strengths

S1. The idea is simple, and it changes only the training, so the model at test time stays the same. The gains over CAS with FACT are large in Table 1, especially on 50Salads (12.82 to 39.68 [email protected]). S2. The paper is open about its limits. It reports that MS-TCN gets worse with the method, and it says that the comparison with published methods in Table 5 is not controlled. I appreciated this.

Critical weaknesses

The paper does not show which part of the method causes the gains. Table 2 is progressive and only on split 1, and progress balancing and transition sampling are added together in one row (BPB). The paper also says that this table does not give the isolated effect of each part (L324-326), So I could not find proof for the claim that "the two-stage choice is important" (L196). I am also not sure that progress balancing is different from uniform sampling. I could not find the widths of the K intervals. If they are equal, picking an interval first and then a frame inside it gives every endpoint the same chance, which seems to be the same as uniform sampling. In that case, it is possible that the gain comes mainly from the transition part.

The main baseline looks too weak. causal FACT baseline has only 44.42 Acc on 50Salads. other causal baselines in the same paper have 75.2(ASFormer) and 68.86(MS-TCN). On [email protected], the plain ASFormer baseline is equal to or better than FACT with the new method.

Some parts seem close to ideas that already exist. I think Training on prefixes is already the CAS baseline ("ordinary prefix training", L290), so I think the new part is mainly how the prefixes are sampled and weighted. Progress is also used in ProTAS (L094-095), although in a different way, and the focus on action boundaries is motivated by ASRF and other cited work (L239-241). The end weighting also seems close in spirit to MiniROAD (An et al., ICCV 2023), which is not cited. MiniROAD also says that the mismatch between training and inference is the main problem when RNNs are trained for online action detection, and it uses non-uniform loss weights over the time steps to reduce it.

Questions for authors
  1. How is the "ordinary prefix training" of CAS done? For example, how are its endpoints sampled, and does it use the same Lmin?
  2. Do the K progress intervals have equal width? If yes, how is progress balancing different from uniform sampling?
  3. Do you know why CAS with FACT is so much weaker on 50Salads (44.42 Acc) than CAS with ASFormer or MS-TCN?
  4. Could you give the mean and standard deviation over the runs, and an ablation of each part alone (progress balancing, transition sampling and end weighting)?
  5. How is the prefix-end weighting different from the loss weighting in MiniROAD?
2 comments
Area_Chair_5jGXOpenReview ↗

rebuttal and feedback

Dear Reviewers,

Thank you for your efforts so far. The authors have responded to your reviews, please check them carefully (you may also check other reviewers' comments) and provide your feedback.

Best, AC

AuthorsOpenReview ↗

Follow-up to Reviewer EyGF

Dear Reviewer EyGF:

Thank you again for your detailed and constructive feedback. We carefully responded to the concerns raised in your review and conducted additional experiments, including independent component ablations and further controlled analyses. We also revised the manuscript to clarify the experimental protocol, methodological contribution, and relationship to prior work.

As the discussion period is approaching its end, we would like to kindly follow up and invite you to review our response and revised manuscript when convenient. We sincerely hope that our revisions have addressed your concerns. If any questions remain, we would be grateful for the opportunity to provide further clarification.

Thank you very much for your time and consideration.

Comments on the paper

1 comment
Area_Chair_5jGXOpenReview ↗

Response to rebuttal

Dear Reviewers,

Thank you for your efforts so far. The authors have responded to your reviews with detailed feedbacks. Please check them carefully (you may also check other reviewers' comments) and provide your feedbacks.

Best, AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.