- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The paper aims to adapt the offline tas model to the online setup. It argues that simply making the model causal is not enough because models trained on full videos has a distrubution shitf of prefix length during testing. The authors terms this as the temporal context distribution shift and propose a input sampling startegy to mitigate this. Specifically, the authors sample prefix length endpoints across progress bins, nudge some endpoints towards action boundaries, and also weight the loss towards the end of the prefix. Experiments on two action segmentation datasets show improvements compared to other existing works.
Critical strengths
-
The temporal context distribution shift (TCDS) is an interesting and valid point to consider when adapting in offline models to online setup. This paper makes the observation that even with causality enforced, a model trained only on complete videos has never seen the prefix states it will encounter online, and its intermediate representations are shaped by that.
-
The method is training-only and remains clean at test time. By prefix length sampling and loss weighting during training, the model gains capability to recognize actions during online testing without additional branches. This likely make the contribution more applicable to any other backbone.
Critical weaknesses
-
The most significant issue is its testing protocol. At each time of a streaming video, making prediction for a single frame would require the model to access and process all the previous frames. This has two implications: 1) the method is with complexity of per video of length T, yet the paper did not reports latency nor FLOPs. 2) the method has access to the unbounded history in its raw input format does not form a fair comparsion with methods like OnlineTAS which is memory bounded. The authors should acknowledge this when performing comparsion.
-
The paper only evaluates on two small-sized TAS benchmarks, missing Breakfast. Breakfast is a larger benchmark which is also widely used in the TAS community, having far more videos and more varies temporal structure. It is very important to have the comparsion on this dataset to show its sampling and loss weighting can generalize to more difficult settings.
Questions for authors
-
Since each prefix is recomputed independently, it naturally ensures the causality, i.e., making predictions on t does not rely on access to frames >t. It does not require the every frame indexed prior to the prefix be blind to its own future. Frames 1...t-1 are already observed, letting i see j (1<i<j<t-1) should be okay. What is the rationale in keep the causal conv/attention mask given the prefix training paradigm? Please justify.
-
Add evalutions on breakfast.
-
What is the inference cost of the methods? The authors should report latency between frames as well as the FLOPs aligned with the video length.
Dear Reviewer THXF,
As the author-reviewer interaction period is ending, please respond to the authors' feedback ASAP.
Best, AC