- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces TraceCut, a framework for action segmentation and annotation in robot manipulation videos. Its core concept involves decoupling the generation of temporal boundary candidates, semantic description, and the final decoding of the action sequence. First, visual features are extracted using a pre-trained VJEPA 2, and a multi-scale candidate action tree is constructed based on temporal constraints. Second, a VLM is fine-tuned, which is employed to generate descriptions for candidate segments across various temporal scales. Finally, a language model is used to integrate the multi-scale semantic information contained within the action tree. On the RoboActionTrace dataset, TraceCut outperforms the baselines in both temporal segmentation and automated semantic metrics; however, on the WGO Robot75 dataset, its semantic annotation performance still lags behind certain baselines that leverage task instructions.
Critical strengths
Decoupling candidate temporal segmentation from semantic action selection is a sound design with practical value. A complete semantic action does not necessarily correspond to visual motion of a fixed scale. This approach helps avoid missing actions that might otherwise be overlooked if the segmentation granularity were rigidly fixed.
Critical weaknesses
(1) The absolute performance of end-to-end semantic labeling remains low. The authors point out that the baseline method utilizes additional task instructions, whereas TraceCut relies solely on RGB video. The paper also presents experiments conducted after removing the task instructions from the baseline. However, these results do not fully resolve concerns regarding fairness, as the various methods still employ different VLMs, visual sampling techniques, context lengths, and temporal localization strategies.
(2) The independent contributions of the core methods lack sufficient validation. The ablation study in Table 4 focuses solely on post-processing operations within the decoder; it fails to systematically evaluate the individual contributions of the multi-scale candidate hierarchy, the specialized VLM, and the LLM-based aggregation mechanism. Table 3 merely compares the original Qwen3 model with a version fine-tuned on 30,000 samples of TraceCut supervision data. Consequently, the observed performance gains could stem from the additional training data, changes to the training objectives, or the fine-tuning process itself.
Questions for authors
Q1. Could you provide a fairer, end-to-end comparison to validate the benefits of multi-scale decoding over direct VLM-based segmentation? Specifically, compare the performance of directly predicting a sequence of atomic actions against the hierarchical decoding approach used in TraceCut, while ensuring the use of the same videos and, as closely as possible, the same underlying vision models.
Q2. Could you provide additional ablation studies? Please compare fixed-interval segmentation, time-constrained flat clustering, and multi-scale candidate hierarchies, while controlling for the number of candidates or the computational budget. Furthermore, compare the performance of the original Qwen3 against the fine-tuned VLM, and evaluate the results when components such as text-based LLM aggregation or actionability-specific refinement are removed.
Q3. Demonstrate that the gains in downstream planning performance stem from the quality of the segmentation and annotation provided by TraceCut. Using the same 30K source videos, construct three separate datasets. Via direct VLM annotation, one using an alternative segmentation pipeline, and one using TraceCut, then compare them under identical training configurations. This will provide a more direct validation of the paper’s proposed link between data quality and planning performance.