- Rating
- 3
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies temporal action segmentation using automatically generated sparse timestamps and incomplete action sets under a fixed token budget for MLLM-generated supervision. MS-ADD combines a FACT-based two-branch architecture with independent noise levels for each frame and action, allowing available supervision to condition pseudo-label generation. It also uses pseudo-label confidence to set noise levels and loss weights, reducing the model’s reliance on uncertain pseudo-labels. CrossTask uses MLLM-generated supervision, whereas Breakfast and GTEA use sparse supervision derived from ground-truth annotations. The paper also investigates how to allocate the budget between the two supervision formats.
Critical strengths
-
The supervision setting is practically relevant, and annotation quality is quantified. Unlike standard timestamp supervision, this setting allows segments to be omitted and labels to contain errors. Table 2 shows that, under the selected setting, timestamp accuracy on CrossTask is 74%, while the action set has a precision of 90.7% and a recall of 74.8%. Treating omitted candidate actions as UNKNOWN is appropriate for handling incomplete action sets.
-
The model’s gains do not arise solely from additional supervision. Table 1 compares methods under the same single-supervision setting; Table 3 evaluates pseudo-label quality; and Table 4 isolates the confidence-based input-noise design while retaining confidence-weighted loss. The ablation of asynchronous inference in Supplementary Table S1 provides further evidence in support of the design.
Critical weaknesses
- The claimed general allocation plan is supported by a narrower setting than its wording suggests. The recommendation to spend about 80% of the budget on timestamps is derived from one model, one reported cost ratio, and split 1 of three datasets. Only CrossTask uses real MLLM labels; Breakfast and GTEA use clean, ground-truth-derived supervision. Allocation curves are evaluated using F1@10, although other metrics can behave differently. For example, on CrossTask, adding action sets to full timestamp supervision improves F1@10 and Edit but decreases F1@50 from 15.7 to 14.9 and accuracy from 72.3 to 59.3. The evidence supports an empirical allocation heuristic for the tested configuration, rather than a generally optimal plan.
- The experiments do not fully isolate the contribution of the proposed asynchronous diffusion formulation from that of the dual-branch architecture and other ways of exploiting the same supervision. The timestamp baselines are evaluated under highly incomplete supervision, and TS-SEG is run without label expansion. Although this choice is motivated by the incomplete timestamp setting, it changes an important component of the baseline. More informative controls would include a dual-branch model trained with ordinary self-training and a synchronously trained diffusion variant, both using the same supervision and comparable training budgets. Table S1 compares different inference schedules of the proposed model, including synchronous inference; it does not isolate the effect of asynchronous noise levels during training. These additional comparisons would strengthen the attribution of performance gains to the proposed diffusion formulation.
Questions for authors
- How are allocation plans selected without using test labels? Please clarify the validation protocol and whether a plan selected on one dataset transfers to another without retuning.
- Can you compare against the same dual-branch architecture using ordinary self-training, or a synchronously trained diffusion variant, with identical mixed supervision? These controls would materially strengthen the architectural contribution.
- How is the sampling rate described as “one per segment on average” determined without access to segment annotations? Please specify whether it uses a fixed interval, dataset-level statistics, or per-video segment counts.
- The clean-supervision experiments are useful, but they should be presented as complementary evidence rather than additional validation under real MLLM noise. Repeated supervision-allocation draws are particularly important for GTEA, which has only 21 training videos and seven test videos.
- The cost analysis transparently accounts for supervision-generation tokens. Please distinguish this from total computational cost, which also includes model training and repeated pseudo-label generation.
- Because confident predictions are permanently committed during inference, an analysis of incorrect commitments and threshold sensitivity would help characterize error propagation.