acceptodds
Desk rejected at ICLR 2027

MS-ADD: MLLM-Supervised Asynchronous Dual-Branch Diffusion for Temporal Action Segmentation

Desk rejected at ICLR 2027, so left out of the paper lists and the map.

Abstract

Weak supervision makes temporal action segmentation cheaper to annotate than frame-wise labels, yet every format of it still requires a person to watch and annotate every video. We use a multimodal large language model (MLLM) instead of a person to generate the supervision, which eliminates the human labour cost. Because the labels an MLLM generates inevitably contain errors, we select sparse timestamps and action sets, two formats of supervision that are easy for the MLLM to generate and that tolerate its errors. Neither format is perfect, so we use both: when one contains an error, the other may still provide valid supervision. The two formats differ in the computation they require, so under a fixed budget an allocation plan decides which share of the videos receives each format. Training on such supervision poses two challenges: a format may be missing from a video, and on average the timestamps of a video cover less than 1% of its frames. We propose MLLM-Supervised Asynchronous Dual-Branch Diffusion (MS-ADD). Its two-branch architecture takes the two formats of supervision as separate inputs and lets its branches exchange information. Its asynchronous diffusion assigns an independent noise level to every frame and every action, and thereby unifies pseudo-label generation, training and inference with the same forward pass. On CrossTask, Breakfast and GTEA, our model outperforms each baseline on nearly every segmental metric under the same single format of supervision, and a suitable allocation enables it to outperform the single-format state of the art at a lower budget. These results yield a general allocation plan.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
video understanding, temporal action segmentation, diffusion

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_3UPLOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies temporal action segmentation using automatically generated sparse timestamps and incomplete action sets under a fixed token budget for MLLM-generated supervision. MS-ADD combines a FACT-based two-branch architecture with independent noise levels for each frame and action, allowing available supervision to condition pseudo-label generation. It also uses pseudo-label confidence to set noise levels and loss weights, reducing the model’s reliance on uncertain pseudo-labels. CrossTask uses MLLM-generated supervision, whereas Breakfast and GTEA use sparse supervision derived from ground-truth annotations. The paper also investigates how to allocate the budget between the two supervision formats.

Critical strengths
  1. The supervision setting is practically relevant, and annotation quality is quantified. Unlike standard timestamp supervision, this setting allows segments to be omitted and labels to contain errors. Table 2 shows that, under the selected setting, timestamp accuracy on CrossTask is 74%, while the action set has a precision of 90.7% and a recall of 74.8%. Treating omitted candidate actions as UNKNOWN is appropriate for handling incomplete action sets.

  2. The model’s gains do not arise solely from additional supervision. Table 1 compares methods under the same single-supervision setting; Table 3 evaluates pseudo-label quality; and Table 4 isolates the confidence-based input-noise design while retaining confidence-weighted loss. The ablation of asynchronous inference in Supplementary Table S1 provides further evidence in support of the design.

Critical weaknesses
  1. The claimed general allocation plan is supported by a narrower setting than its wording suggests. The recommendation to spend about 80% of the budget on timestamps is derived from one model, one reported cost ratio, and split 1 of three datasets. Only CrossTask uses real MLLM labels; Breakfast and GTEA use clean, ground-truth-derived supervision. Allocation curves are evaluated using F1@10, although other metrics can behave differently. For example, on CrossTask, adding action sets to full timestamp supervision improves F1@10 and Edit but decreases F1@50 from 15.7 to 14.9 and accuracy from 72.3 to 59.3. The evidence supports an empirical allocation heuristic for the tested configuration, rather than a generally optimal plan.
  2. The experiments do not fully isolate the contribution of the proposed asynchronous diffusion formulation from that of the dual-branch architecture and other ways of exploiting the same supervision. The timestamp baselines are evaluated under highly incomplete supervision, and TS-SEG is run without label expansion. Although this choice is motivated by the incomplete timestamp setting, it changes an important component of the baseline. More informative controls would include a dual-branch model trained with ordinary self-training and a synchronously trained diffusion variant, both using the same supervision and comparable training budgets. Table S1 compares different inference schedules of the proposed model, including synchronous inference; it does not isolate the effect of asynchronous noise levels during training. These additional comparisons would strengthen the attribution of performance gains to the proposed diffusion formulation.
Questions for authors
  1. How are allocation plans selected without using test labels? Please clarify the validation protocol and whether a plan selected on one dataset transfers to another without retuning.
  2. Can you compare against the same dual-branch architecture using ordinary self-training, or a synchronously trained diffusion variant, with identical mixed supervision? These controls would materially strengthen the architectural contribution.
  3. How is the sampling rate described as “one per segment on average” determined without access to segment annotations? Please specify whether it uses a fixed interval, dataset-level statistics, or per-video segment counts.
  4. The clean-supervision experiments are useful, but they should be presented as complementary evidence rather than additional validation under real MLLM noise. Repeated supervision-allocation draws are particularly important for GTEA, which has only 21 training videos and seven test videos.
  5. The cost analysis transparently accounts for supervision-generation tokens. Please distinguish this from total computational cost, which also includes model training and repeated pseudo-label generation.
  6. Because confident predictions are permanently committed during inference, an analysis of incorrect commitments and threshold sensitivity would help characterize error propagation.
Reviewer_yEoyOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies temporal action segmentation with sparse timestamps and action sets under a supervision-generation budget. MS-ADD combines a frame–action architecture with asynchronous diffusion and confidence-aware pseudo-label training. Experiments report useful performance and budget trade-offs. CrossTask uses MLLM-generated supervision, while Breakfast and GTEA use supervision derived from ground truth.

Critical strengths
  1. The budgeted use of complementary supervision formats is practically relevant, and the model provides a coherent way to handle incomplete supervision.
  2. The experiments provide positive evidence, including single-format comparisons and the confidence-dependent noise-sampling ablation in Table 4.
Critical weaknesses
  1. The contribution of the proposed diffusion mechanism is not sufficiently isolated from the backbone, network depth, matching and losses. The three-block configuration lacks a depth–cost comparison. Table 3 may also be affected by how its comparator excludes actions missing from an incomplete action set.
  2. The checkpoint and hyperparameter selection procedures need clarification, and variability across training and supervision-allocation runs is not reported. Evidence for robustness to erroneous timestamps and broader budget-allocation recommendations is limited: only CrossTask uses actual MLLM labels, and Figure 5 illustrates a single training video.
Questions for authors
  1. Can you provide a matched comparison isolating the conditioning/corruption mechanism, and explain the choice of three diffusion blocks with a limited depth–cost comparison at fixed denoising steps?
  2. Which data were used to select checkpoints, annotation settings, thresholds and allocations? How is “one per segment on average” sampling determined without per-video ground-truth boundaries or counts? What variability is observed across independent runs?
  3. In Table 3, does the advantage remain if actions missing from the MLLM set are treated as unknown rather than excluded?
  4. What quantitative evidence supports robustness to erroneous timestamps? How should the allocation recommendation be limited given the different supervision sources across datasets?
Reviewer_SA4EOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper proposes a temporal action segmentation framework utilizing a multimodal large language model to generate the supervision. Considering that the generated labels inevitably contain errors, this paper selects sparse timestamps and action sets as supervision forms, and proposes MLLM-Supervised Asynchronous Dual-Branch Diffusion (MS-ADD) to adapt to this training setting. The experimental results on several datasets show the proposed method outperforms the single-format state of the art at a lower budget.

Critical strengths
  1. This paper focuses on the problem of manual annotation reliance in weakly supervised temporal action segmentation, and proposes to use MLLM to address the problem. This is indeed an important and urgent problem in the field of video analysis. The motivation is strong and reasonable.

  2. Asynchronous diffusion assigns an independent noise level to every frame and every action by using different noise schedules for different types of labels. This design is reasonable and inspiring.

  3. Good experimental results are achieved on several datasets.

Critical weaknesses
  1. The main focus of the paper is unclear. The paper’s major contribution lies in generating labels with MLLM and budget allocation, yet no comparisons with alternative methods are provided, nor are analyses of prompt stability and different sampling strategies included. Most experiments focus on comparisons involving MS-ADD. However, in my opinion, this is more of an implementation with some specific designs under this experimental setting rather than the core novelty. Moreover, its main framework is adapted from FACT. In addition, the number of compared methods is limited, and all were proposed in 2023 or earlier. If no other comparable methods exist, there should be clarifications.

  2. The MLLM-generated supervision is only validated on one of three datasets; the two standard benchmarks bypass the MLLM entirely. In Table 1, the CrossTask column is labeled "(MLLM)", while the Breakfast and GTEA columns are labeled "(ground truth)". Moreover, the MLLM-generated label quality is not high according to Table 2. The experimental support for the claim is weak.

  3. Some details remain unclear, so it is questionable whether human intervention is involved to assist MLLM label generation. For detailed questions, please refer to “Questions For Authors”.

Questions for authors

For MLLM label generation:

-In Lines 193 -- 194, regarding “the complete subtitle track and the set of actions the video’s task may contain”, it is unclear how the scope of “may contain” is determined. It is also ambiguous whether the subtitle track refers only to captions or includes audio transcripts, and how cases are handled when a video has no subtitles.

-In Line 195-196, regarding "so an action it is uncertain about is omitted rather than guessed", the criterion for judging when the MLLM is uncertain about an action is unclear.

-In Section S3 of the supplementary material, the prompt contains many “e.g.” examples, which, in my view, may substantially affect the output of the MLLM.

-(Minor) In Line 199, for “sampled at uniform intervals”, it is unclear whether this means a fixed number of frames per video or a fixed sampling frequency. How many frames are sampled on average per video, and what is the sampling frequency?

For MS-ADD:

-Are there any existing methods that can be compared under exactly the same mixed allocation setting? If the answer is no, how to validate the effectiveness of the proposed approach?

-In Figure 2, the shapes of “frame labels with added noise” and “action labels with added noise” appear identical. However, I think the frame label should be T-dimensional? Since the number of actions in a video is not known in advance, how is this handled here?

-(Discussion) Since the training labels for MS-ADD are predicted by the MLLM, how does the MLLM perform when directly predicting full temporal labels? Alternatively, is it feasible to implement iterative interaction between MS-ADD and the MLLM?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.