Boundary-Aware Progress-Balanced Prefix Learning for Strictly Online Temporal Action Segmentation

Temporal action segmentation assigns an action label to every frame of an untrimmed video. Most segmentation models are trained with complete videos, whereas strictly online inference can access only a growing observed prefix. Directly applying a full-sequence model to prefix inference substantially degrades segment quality, and causal constraints recover only part of the loss. We characterize this mismatch as temporal context distribution shift and address it by reconstructing training prefixes and their supervision. Our method first balances prefix endpoints across observation progress, then increases sampling near action transitions, and finally weights frame-level supervision toward the prefix endpoint. The approach modifies training but introduces no additional inference branch. With FACT as the backbone, it reaches [email protected] scores of 68.42% on GTEA and 39.68% on 50Salads, improving over causal adaptation by 9.32 and 26.86 points. Applying the strategy to ASFormer also improves [email protected] over causal adaptation on both datasets, indicating that its benefits can extend to other attention-based backbones. These results show that aligning prefix coverage and supervision is important when adapting attention-based segmentation models to strictly online inputs.