acceptodds
Withdrawn from ICLR 2027

TraceCut: Atomic Action Segmentation and Labeling for Robot Manipulation Videos via Actionability-Aware Hierarchy Decoding

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Long-horizon tasks and their corresponding planning models represent a prominent yet challenging frontier in robotics research. While collecting diverse long-horizon task videos has become a key focus, fine-grained action segmentation and annotation remain fundamental to constructing high-quality datasets. Although temporal grounding based on vision-language models (VLMs) offers a natural solution, it often struggles inferior accuracy due to the burden of simultaneously handling action perception, temporal localization, and semantic labeling. To address this, we propose TraceCut, a video atomic action segmentation and annotation pipeline based on hierarchical segment decomposition. TraceCut builds a hierarchical action tree for robot videos to enable accurate multi-level segmentation. It achieves precise atomic action splitting and annotation by generating and aggregating segment-level descriptions. Furthermore, we develop a lightweight video captioning model to fulfill the critical requirement for efficient and accurate caption generation within the pipeline. We also introduce RoboActionTrace, a standardized benchmark for evaluating robot video action segmentation and annotation. Experiments demonstrate that both the TraceCut pipeline and the lightweight captioning model achieve competitive performance in atomic action segmentation and annotation. Moreover, data constructed via this pipeline significantly enhances performance in core long-horizon task planning embodied task, demonstrating substantial commercial potential amid the growing demand for automated annotation.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Atomic Action, Segmentation and Labeling, Actionability-Aware Hierarchy Decoding

Reviews

Among:
Shape:
1234this paper, Bottom 9%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.50 over 2 reviews, above 9% of the others.
Reviewer_SsFCOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces TraceCut, a framework for action segmentation and annotation in robot manipulation videos. Its core concept involves decoupling the generation of temporal boundary candidates, semantic description, and the final decoding of the action sequence. First, visual features are extracted using a pre-trained VJEPA 2, and a multi-scale candidate action tree is constructed based on temporal constraints. Second, a VLM is fine-tuned, which is employed to generate descriptions for candidate segments across various temporal scales. Finally, a language model is used to integrate the multi-scale semantic information contained within the action tree. On the RoboActionTrace dataset, TraceCut outperforms the baselines in both temporal segmentation and automated semantic metrics; however, on the WGO Robot75 dataset, its semantic annotation performance still lags behind certain baselines that leverage task instructions.

Critical strengths

Decoupling candidate temporal segmentation from semantic action selection is a sound design with practical value. A complete semantic action does not necessarily correspond to visual motion of a fixed scale. This approach helps avoid missing actions that might otherwise be overlooked if the segmentation granularity were rigidly fixed.

Critical weaknesses

(1) The absolute performance of end-to-end semantic labeling remains low. The authors point out that the baseline method utilizes additional task instructions, whereas TraceCut relies solely on RGB video. The paper also presents experiments conducted after removing the task instructions from the baseline. However, these results do not fully resolve concerns regarding fairness, as the various methods still employ different VLMs, visual sampling techniques, context lengths, and temporal localization strategies.

(2) The independent contributions of the core methods lack sufficient validation. The ablation study in Table 4 focuses solely on post-processing operations within the decoder; it fails to systematically evaluate the individual contributions of the multi-scale candidate hierarchy, the specialized VLM, and the LLM-based aggregation mechanism. Table 3 merely compares the original Qwen3 model with a version fine-tuned on 30,000 samples of TraceCut supervision data. Consequently, the observed performance gains could stem from the additional training data, changes to the training objectives, or the fine-tuning process itself.

Questions for authors

Q1. Could you provide a fairer, end-to-end comparison to validate the benefits of multi-scale decoding over direct VLM-based segmentation? Specifically, compare the performance of directly predicting a sequence of atomic actions against the hierarchical decoding approach used in TraceCut, while ensuring the use of the same videos and, as closely as possible, the same underlying vision models.

Q2. Could you provide additional ablation studies? Please compare fixed-interval segmentation, time-constrained flat clustering, and multi-scale candidate hierarchies, while controlling for the number of candidates or the computational budget. Furthermore, compare the performance of the original Qwen3 against the fine-tuned VLM, and evaluate the results when components such as text-based LLM aggregation or actionability-specific refinement are removed.

Q3. Demonstrate that the gains in downstream planning performance stem from the quality of the segmentation and annotation provided by TraceCut. Using the same 30K source videos, construct three separate datasets. Via direct VLM annotation, one using an alternative segmentation pipeline, and one using TraceCut, then compare them under identical training configurations. This will provide a more direct validation of the paper’s proposed link between data quality and planning performance.

Reviewer_dJdsOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The problem aims to solve the problem of automatically segmenting robot video doing a multi-step task into meaningful temporal sgements, each with a short text decription. The overall pipeline comprises three stages, 1) a temporally constrained agglomerative clustering is used based on the V-JEPA 2 features to form a overcomplete semantic hierarchy, 2) a fine-tuned captioner labels each node, and 3) a text-only llm decodes the hierarchy to actions, followed by hand-written refinement. They also introduce RoboActionTrace that provides action segments and language annotations, which could serve as the benchmark. Experiments show that after fine-tuning on 30K pipeline-generated samples improves EgoPlan accuracy.

Critical strengths
  1. I quite like the idea of ''actionability-aware decoding'' over an overcomplete hierarchy and consider it as a clean articulation of what's hard about the undestanding the action granularity in generic procedural videos. And the results on robot data shows that the hierarchy does contain good candidates and selection is the bottleneck.

  2. Manually annotating the benchmark is another contribution.

Critical weaknesses
  1. One major concern is that the core mechnism is not stated or validated. I agree with the authors saying that depths is not semantic granularity, but they did not explictly mention what is. Based on the overall understanding of the paper, the design seems to rely on the trained captioner's output cardinality. The trick is that the caption is trained so that how it describes a segment reveals whether the node of interest should be considered as an action. If so, the authors should explicitly mention this as the central hypothesis, and discusses its implications. For example, 1) selection can only be as good as the captioning calibration. 2) how is overlapping nodes aross different levels determine the action boundary, say pour water from different levels is captioned for overlapping segments, 13-16s, 8-19s, 6-20s. i think this also relates to how the llm reads them across the tree, which i consider important yet not clearly explained.

  2. Another major concern is regarding fair comparsion. Baselines are zero-shot pipelines, yet the tracecut is finetuned on 2m samples from robot data with annotations. Even the authors mention that test-set overlap is excluded, the same scene, task and annotation convention (relates to the captioner) can still leak information and favor Tracecut.

  3. More ablation is required. For example, hierarchy vs. uniform windows or shot detection; fine-tuned captioner vs. base Qwen3-VL-8B vs. Seed inside the full pipeline; choice of aggregation LLM. These are necessary to show which part plays the most signifient roles.

Questions for authors

The major issues are listed above. Yet I still have the following concerns.

Temporal mIOU is mentioned in L335, but not reported in result tables.

What is Table 3 for? There's no text refering to Table 3 in the main paper.

Please remove human videos from figure 1 as the first column captioned as robot video.

Unfilled template text from AI in reproducibility statement

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.