acceptodds
Under review as a conference paper at ICLR 2027

RoboAtom: Scalable Atomic-Action Supervision for Robot Learning

Abstract

Fine-grained action annotations can support skill reuse and compositional generalization in robot policies. However, robot datasets typically provide task-level instructions without subtask annotations. Although some include subtask labels, inconsistent formats and conventions hinder cross-source joint training. We propose RoboAtom-Data, which adds unified, time-aligned atomic-action annotations to existing data from multiple sources, covering 1.36 million trajectories, 6.6 million skill-labeled segments, and 15,736.6 hours. Our pipeline combines a unified protocol, large-scale manual annotation, propagation across repeated executions with the same ordered skill sequence, and multi-stage quality control. We evaluate the dataset through automatic atomic-action annotation and robot control. We construct RoboAtom-Bench to standardize evaluation of skill recognition and temporal boundary localization. Trained on the annotated data, RoboAtom-Anno achieves higher annotation quality at lower inference cost than all evaluated open- and closed-source models on both RoboAtom-Bench and the external WGO-Bench. For robot control, atomic-action supervision paired with hierarchical execution improves task success over task-level baselines on LIBERO-Long, LIBERO-Plus, and RoboTwin. We will release the data, benchmark, model, and construction pipeline to support further research on skill-level robot learning.

Primary area:
applications to robotics, autonomy, planning
Keywords:
robot learning, atomic-action supervision, robot datasets, temporal action segmentation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_9cSBOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

To address missing subtask annotations and inconsistencies in action categories, segmentation granularity, and temporal boundaries across robotic datasets, this paper introduces RoboAtom-Data, a unified atomic-action supervision dataset, alongside RoboAtom-Bench and the automatic annotation model RoboAtom-Anno. Its primary contributions lie in standardizing cross-source annotations, designing a scalable annotation pipeline, and constructing a large-scale supervision resource, not proposing a new model architecture.

Critical strengths
  1. The unified annotation resource has clear practical value: By providing the dataset, construction pipeline, and evaluation benchmark, the paper offers a potentially useful resource for the robotics research community.
  2. Practical annotation pipeline: The pipeline combines model-assisted annotation, human correction, constrained same-task propagation, and targeted quality control to reduce repetitive annotation effort; experiments on RoboAtom-Bench and WGO-Bench support its practical utility.
Critical weaknesses
  1. Limited methodological novelty: PhysExtend segments skills using a physically informed scheme(state–action, motion change, gripper events) already explored in prior embodied-learning work on atomic actions. Therefore, this work as an engineering refinement of a unified annotation pipeline and the construction of a large-scale datasets, not a methodological advance.
  2. The requirements stipulate that segmentation should occur "only when the dominant atomic skill changes," while also mandating the decomposition of composite behaviors. How are these rules reconciled in continuous tasks (such as folding laundry)? How should one define boundaries in data where such atomic transitions are inherently ambiguous, and how is the appropriate granularity for segmentation determined?
  3. RoboAtom-Anno achieves results superior to the evaluated baselines on two annotation benchmarks, existing metrics primarily focus on the alignment between temporal segments and skill categories. The accuracy of success/failure classification for atomic skills, the precision and recall of online completion detection are not reported.
  4. In the downstream experiments, the approach simultaneously modifies the training language conditions and incorporates a high-level planner. While the performance gains shown in Table 4 confirm the efficacy of the complete system, the improvements are modest—only 2%–5% on tasks that not yet reached saturation. Moreover, RoboAtom-Anno has already been trained to identify atomic skills and their temporal boundaries, why does "System 2" employ a separate Qwen3-27B model instead of reusing the existing model to directly evaluate its performance?
Questions for authors

As noted above, My main concerns are the following:

  1. How does RoboAtom-Anno perform on continuous tasks with ambiguous boundaries, and what is the granularity at which it segments atomic actions?
  2. Please provide quantitative metrics for annotation quality, such as precision and recall? When applied to a specific, independent dataset, the primary considerations should be the cost of human labor and the accuracy of the automated pipeline.
  3. For the downstream "System 2" component, which utilizes a standalone Qwen3.8-27B model, would performance on downstream tasks improve if incorporates RoboAtom-Anno?
  4. The granularity of "atomic actions" is partly determined by manually defined rules. However, given that some tasks involve strong sequential dependencies, is the current approach superior to other reasonable granularities? How does it compare to supervision applied at the level of coarser sub-tasks, or the current atomic-action level?
3 comments
AuthorsOpenReview ↗

Response to Reviewer 9cSB

We sincerely thank the reviewer for the careful review and constructive feedback. We first clarify two points:

  1. The two segmentation rules are not contradictory (Weakness 2). The rule “segment only when the dominant skill changes” applies to the data we annotate ourselves, primarily to prevent overly fine segmentation. “Decomposing composite behaviors” mainly applies to existing datasets with overly coarse annotations, with the aim of standardizing annotation criteria. See our response to Question 1 for details.
  2. RoboAtom-Anno is an offline annotation model (Weaknesses 3 and 4 and Question 3). It takes a complete video as input and outputs atomic-action labels and their temporal segments. It does not perform online completion detection and is distinct from the downstream System 2 planning module.

Weakness 1: Methodological Novelty

The revised Introduction presents this work as a data resource and system integration effort that builds on prior work in segmentation using physical signals.

  1. A unified cross-source atomic-action standard. Existing datasets differ in subtask annotation formats, skill vocabularies, granularity, and temporal coverage. To our knowledge, this is the first work to unify 16 data sources under a common atomic-action standard, covering 1.45 million trajectories, 6.88 million segments, and 16,882.1 hours.
  2. A scalable annotation pipeline. Segmentation based on physical signals alone is sensitive to pauses and speed variations and can be difficult to align with skill semantics. PhysExtend combines reference skill sequences with skill-specific physical cues to propagate temporal boundaries across repeated executions of the same task at low cost.
  3. Benchmarking and downstream validation. RoboAtom-Bench standardizes evaluation of skill recognition and temporal boundary localization. Our experiments validate the utility of the annotations for automatic annotation and robot control.

Question 1 & Weakness 2: Ambiguous Boundaries and Segmentation Granularity

Boundaries in continuous tasks such as folding are clearer than they may appear. Because robot data collection is relatively standardized, most atomic-action boundaries are marked by physical events such as grasp, release, and contact changes. Consecutive folds can be reliably separated (see the demo at 28–38 s). To support consistent boundary annotation and resolve uncertain cases, we use the following three measures:

  1. Detailed annotation guidelines. We provide annotators with segmentation rules for atomic skills and video demonstrations of each skill. The revised manuscript describes the segmentation rules in Section 3.1 and Appendices A.1 and A.3.
  2. A question-reporting mechanism. Annotators are required to ask us whenever they encounter uncertainty. Of the 56 questions raised by 18 annotators, 10 (17.9%) concerned ambiguous boundaries. All were resolved, after which annotators compared their annotations on the same episode to ensure that the resolution was applied consistently.
  3. Consistency checks. We annotate multiple episodes for each task, with at least two per task. We compare the consistency of skill sequences across episodes of the same task. If inconsistencies arise, a third annotator reviews them and determines the final annotations, ensuring consistency in skill boundaries.

Question 2: Annotation Quality

On 100 tasks across 10 datasets, human evaluation gave PhysExtend 74.6% segment-level precision and recall and 69.0% episode-level accuracy. After quality control, a manual audit of 740 episodes found 90.6% subtask accuracy and 82.6% episode-level accuracy.

Question 3 & Weaknesses 3 and 4: Dual-System Ablations

  1. With System 1 fixed, replacing Qwen3.8-27B with RoboAtom-Anno-9B slightly reduced success on LIBERO-Long and LIBERO-Plus Overall.
  2. With dual-system inference fixed, replacing RoboAtom-Data training annotations with Gemini 3.6 Flash annotations removed most of the gains over the end-to-end baseline on both benchmarks (Appendix C.8, Table C.7).

See Downstream Robot Control Ablations for Questions 3 and 4.

Question 4: Comparison with Coarser-Grained Supervision

We merged adjacent Pick–Place actions in LIBERO into “place [object] at [location]” and retrained the control policy. Success rates were close to those with atomic supervision on LIBERO-Long and LIBERO-Plus Overall. We favor atomic segmentation for two reasons:

  1. Our atomic-action definitions are consistent with conventions widely used in works such as π0.5, Cortex, RoboInter, and AgiBot World, facilitating alignment with existing datasets, joint training, and extension.
  2. Coarser-grained supervision is less conducive to reuse and compositional generalization across tasks. For example, Pick is not always followed by Place; it may instead be followed by Pour, Stack, or Insert.

Thank you for providing the additional experiments and clarifications, and this addresses some of our concerns. I find that PhysExtend still exhibits limited accuracy and relies on manual verification, mostly. Combined with my assessment that this work is primarily engineering refinement, as you mentioned in Answer 1, I will keep my current rating.

AuthorsOpenReview ↗

Manual Verification and the Contributions of RoboAtom

We thank the reviewer for the additional feedback. We briefly address the two concerns below.

On "PhysExtend still exhibits limited accuracy and relies on manual verification"

We agree with the first half of this assessment: measured against 100% accuracy, PhysExtend indeed "still exhibits limited accuracy." Under the same annotation extension task and evaluation protocol, however, PhysExtend ranks first on all four accuracy metrics (Table 2 of the main paper).

ModelmIoU[email protected]Skill [email protected]Episode Accuracy (%)Cost (USD/h)
GPT-4o48.220.711.1100.888
Gemini 3.1 Pro61.144.939.7392.877
GPT-5.6-sol58.937.835.0463.510
Qwen3.8-27B46.125.419.1250.051
Cortex66.549.848.5374.8 × 10⁻⁴
PhysExtend76.871.671.0692.8 × 10⁻⁵

Precisely because we agree that absolute accuracy matters more than relative ranking, we follow automatic annotation with manual quality checks and corrections. Used as a standalone automatic annotator, PhysExtend achieves an Episode Accuracy of 69.0%. After manual quality checking and iteration, Episode Accuracy reaches 82.6% and subtask accuracy reaches 90.6%.

We therefore regard this manual step as a strength of the paper rather than a weakness: it makes RoboAtom-Data a dataset with a known error distribution and a committable quality floor, not merely "another set of automatic annotation results." By comparison, a pipeline that relies entirely on a general-purpose model has no explicit acceptance criteria, yet costs five orders of magnitude more (3.510 vs. 2.8 × 10⁻⁵ USD per source-video hour) while achieving less than half the Skill [email protected] (35.0 vs. 71.0).

In other words, we accept that automatic precision is "limited" — and that is precisely why a verifiable quality-control stage is needed; the existence of this stage is exactly what makes the data safe to use.

On "this work is primarily engineering refinement"

We agree with this description, and it has been our consistent positioning throughout: the first page of the Introduction already presents the work as "a unified resource and construction pipeline." This is precisely the contribution we claim. Datasets and benchmarks are established categories of research contribution, and their evaluation criteria are scale, consistency, verifiability, and downstream utility — not the algorithmic novelty of any individual component. Under that standard, our claims are as follows.

  • First, cross-source unification. Few robot datasets provide subtask annotations, and where they exist (AgiBot World, RoboInter-Data, BEHAVIOR-1K) vocabularies, granularity, and temporal coverage differ, making them impossible to combine directly. To our knowledge, this is the first work to unify 16 heterogeneous sources under one 29-category atomic-action vocabulary and temporal standard (1.45M trajectories, 6.88M segments, 16,882 hours, 3,334 tasks, 32 embodiments).

  • Second, a verifiable quality floor. To our knowledge, this is the first robot manipulation dataset to declare thresholds before release (subtask ≥ 85%, episode ≥ 75%) and report measured results against them: 90.6% / 82.6% on the 740-trajectory audit, reported by source (Appendix A.6) so users can judge the distribution of errors.

  • Third, a benchmark for multi-source unification. Comparable benchmarks use free-text labels and few source families (WGO-Bench: 3 families, 100 episodes, 743 segments), so scores are hard to compare across pipelines. RoboAtom-Bench uses a fixed 15-category vocabulary and unified temporal protocol covering 99.2% of skill-labeled segments across 16 sources (500 trajectories, 4,018 segments), and can serve as a common standard for later cross-source work.

  • Fourth, downstream utility. In automatic annotation, RoboAtom-Anno-9B trained on RoboAtom-Data outperforms GPT-5.6-sol on RoboAtom-Bench and WGO-Bench Robot at ~1/720 the inference cost (Table 3). In robot control, with the dual-system setup fixed and only the annotation source varied, RoboAtom annotations yield higher LIBERO-Long and LIBERO-Plus success rates than Gemini 3.6 Flash annotations, by 2.80 and 3.61 points (Table C.7).

We acknowledge that this work does not introduce an entirely new model architecture. We believe its contribution should be judged on the merits of the data resource and benchmark it provides. We thank the reviewer again for their time and feedback.

Reviewer_zFnxOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents RoboAtom, addressing the lack of consistent, fine-grained atomic-action annotations across heterogeneous robot manipulation datasets. The core contributions are threefold: (1) RoboAtom-Data, a unified dataset spanning 14 sources (real + simulation) with 1.36M trajectories, 6.6M skill-labeled segments, and 15,736.6 hours, using a 28-class skill vocabulary where each segment contains a skill label, action description, and start/end frames; (2) PhysExtend, a propagation method that transfers temporal boundaries from a few reference trajectories to other executions of the same task using only state-action signals (no video frames), significantly outperforming Cortex; and (3) RoboAtom-Bench (500 expert-reviewed trajectories) and RoboAtom-Anno, an annotation model built from Qwen3.5 via robot-domain mid-training, SFT, and GRPO, which surpasses GPT-5.6-sol, Gemini 3.1 Pro, and other baselines on both internal and external evaluations at roughly 700× lower inference cost. Downstream, atomic-action supervision in a dual-system architecture improves success rates on LIBERO-Long, LIBERO-Plus, and RoboTwin over task-level baselines.

Critical strengths
  1. The problem is well-chosen.​ Everyone working on skill reuse or hierarchical policies (RT-H, Hi Robot, Cortex, etc.) knows how painful it is to have inconsistent subtask annotations. The paper makes a strong case for why unifying across sources matters.

  2. Large-scale, unified dataset is the main selling point.​ 1.36M trajectories / 6.6M segments / 15.7K hours is genuinely impressive. The 28-class vocabulary (with a 15-class closed set for eval), the LeRobot v2.1 conversion, and how they handle Idle/Recover — all nicely documented and practical.

  3. PhysExtend is pragmatic and cost-effective.​ State-only features, runs on a single CPU, costs less than $0.001 per hour — that's a very attractive alternative to burning money on API calls. Multi-reference fusion gives solid gains too.

  4. Relatively thorough evaluation.​ The authors have Short/Medium/Long/Multiview splits internally, plus an external benchmark (WGO-Bench), and they compare cost alongside accuracy. That's good practice.

  5. Engineering transparency is high.​ The authors include prompts for pre-annotation, human review steps, and quality control details in the appendix. It actually feels reproducible.

Critical weaknesses
  1. Honestly, the novelty is mostly in the integration, not in individual components.​ PhysExtend's DP + event correction, the QC pipeline, and the SFT+GRPO annotation model — none of these are new by themselves. What's new is putting them together at scale and releasing a benchmark. The paper should be upfront about this.

  2. The control experiment doesn't really isolate what's causing the improvement.​ In Table 4, "Our method" changes two things at once: the training language (from task instruction to atomic-action instruction) and the inference structure (from end-to-end to hierarchical with System 2). The baseline π0.5 is pure end-to-end. So we can't tell if the gain comes from the supervision signal or from having a separate System 2. An ablation keeping the inference mode fixed and only changing the supervision would be necessary.

  3. Quality control feels a bit circular.​ Human annotations are checked by Qwen3-VL, then text fixes are done by GPT-5.6 — there's potential for bias. They report spot-check numbers but never show before-and-after error rates. Private data is included but not released. And I couldn't find any inter-annotator agreement numbers (like IoU or skill consistency) for any source.

  4. PhysExtend assumes the same skill sequence repeats within a task.​ They mention in Sec. A.6 that some tasks get excluded because of structural mismatch, but they don't say how many tasks or trajectories fail that assumption. Feels like the pipeline might only work on "well-behaved" tasks.

Questions for authors
  1. Could you run an ablation where the baseline π0.5 is fine-tuned with atomic-action instructions but still executed end-to-end (no System 2)? That would let us see the actual impact of the supervision signal alone.

  2. Do you have inter-annotator agreement numbers (segment IoU or skill accuracy) on, say, 50 trajectories?

1 comment
AuthorsOpenReview ↗

Response to Reviewer zFnx

We thank the reviewer for recognizing the value of the dataset, annotation pipeline, and evaluation design, and for raising concerns about the positioning of our contributions, attribution of control improvements, annotation consistency, and the applicability of propagation. We agree that these aspects require clearer explanation and have revised the manuscript accordingly.

Weakness 1: Methodological Novelty

We agree that the main contributions of this work are a unified atomic-action data resource, a scalable construction pipeline, and standardized evaluation. PhysExtend combines skill-specific physical cues to propagate temporal boundaries at low cost for tasks with a shared action sequence. We use SFT and GRPO to validate the value of the constructed data for automatic annotation. Our focus is on unifying skill semantics, action granularity, and temporal annotations across sources and integrating human annotation, propagation, and quality control into a scalable pipeline. We have clarified this positioning in the Introduction (Section 1).

Question 1 & Weakness 2: Insufficient Dual-System Ablations

Thank you for pointing out that the original comparison changed both the training supervision and the inference structure. We added three sets of experiments:

  1. Changing the training annotations. With the dual-system inference setup fixed, we retrained System 1 using Gemini 3.6 Flash annotations in place of RoboAtom-Data annotations. Success rates dropped from 98.20% to 95.40% on LIBERO-Long and from 88.18% to 84.57% on LIBERO-Plus Overall. This removed most of the observed gains over the end-to-end baseline on both benchmarks. See our Downstream Robot Control Ablations comment and Appendix C.8 (Table C.7).
  2. Removing System 2 at inference. System 1 received only the task instruction at inference, although it was trained with task, skill, and subtask instructions. This input mismatch makes it difficult to isolate the effect of subtask supervision. Removing System 2 lowered success rates on LIBERO-Long, LIBERO-Plus Overall, and RoboTwin relative to the end-to-end baseline (see Downstream Robot Control Ablations for the LIBERO results; RoboTwin: 62.03% vs. 64.37%).
  3. Replacing the planner. With System 1 fixed, using RoboAtom-Anno-9B as the planner reduced success from 98.20% to 96.60% on LIBERO-Long and from 88.18% to 87.45% on LIBERO-Plus Overall.

Question 2 & Weakness 3: Quality Control and Annotation Consistency

  1. Annotation consistency. We re-annotated 50 tasks from 11 datasets covering 23 embodiment labels. Temporal mIoU, Skill [email protected], and Edit Score between the two rounds of annotation were 74.0, 65.8, and 89.3, respectively (Appendix A.7).
  2. Quality control. We agree that model-based checking and revision alone cannot serve as independent evidence of annotation correctness. During correction, human checks identified systematic problems in each dataset and guided adjustments to reference annotations and the propagation method. In the final manual audit, we sampled different tasks within each dataset–embodiment group for human evaluation, covering 740 episodes in total. Subtask accuracy was 90.6% and episode-level accuracy was 82.6%. The corresponding accuracies were 93.7% / 88.9% for simulation data, 89.7% / 79.3% for real-robot data, and 90.7% / 86.0% for reprocessed data (Appendix A.7, Table A.6).

Weakness 4: Number of Tasks or Trajectories That Violate the Assumption

For the 53 of 3,334 tasks (1.59%) with differing skill sequences, we skipped PhysExtend propagation and retained all 18,356 affected trajectories (1.27% of RoboAtom-Data) by annotating them individually with model assistance. See Appendix A.6 and our response to Reviewer rNK1, Question 3.

Reviewer_rNK1OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces a framework for standardized, time-aligned atomic-action supervision across a lot of heterogenous robot datasets. RoboAtom-Data contains approximately 1.36 million trajectories and 6.61 million skill-labeled segments from 14 different sources. For constructing this dataset, the framework combines human annotation and semi-automated multi-stage quality control protocols. The paper also introduces RoboAtom-Bench and an automatic annotation model, RoboAtom-Anno. Experiments evaluate annotation quality and downstream robot control using atomic-action supervision within a hierarchical execution framework. All the robot experiments are in simulation environments.

Critical strengths

Substantial and potentially useful dataset contribution: The paper addresses an important obstacle to using heterogeneous robot datasets: inconsistent action vocabularies, segmentation granularity, and temporal annotations.

The paper describes a shared representation of skill categories, action descriptions, and temporal boundaries for many robotic manipulation tasks. This could be a valuable contribution to the community that could facilitate training policies across tasks, and anylyzing skills across tasks.

Scalable annotation pipeline. PhysExtend enables efficient annotation propagation through a combination of human review and multi-stage quality control.

Broad evaluation. Results show stronger annotation performance on internal and external benchmarks, alongside improved control performance in simulation - Libero and RoboTwin environments.

Critical weaknesses

There are no real robot manipulation experiments. All downstream control experiments are conducted in simulation, leaving the benefits on physical robots unverified. This is unsatisfying because the dataset also has real robot rollouts, but it remains unclear whether the dataset would be useful for robotics.

The key contributions of the paper are not clearly and independently analyzed. The simulated robot control experiments alter both the type of supervision and planning, and thus it is difficult to attribute gains to the annotations alone. it would be helpful to analyze the quality of annotations in more detail, separately so that it is clear how useful are they likely to be beyond the tasks demonstrated in the paper.

Restrictive propagation assumption. PhysExtend requires a shared skill sequence, limiting its applicability to executions with additional or reordered actions.

Poor performance on long sequences. As evident from the results, annotation quality declines substantially as sequences become more complex, raising concerns about reliability on long-horizon tasks.

Questions for authors

Can you provide real-robot control experiments to demonstrate that the improvements extend beyond simulation?

How much do atomic-action annotations contribute to downstream performance? it would be helpful to provide more detailed perspectives on the quality of the annotations themselves.

What proportion of tasks or trajectories is excluded from the propagation stage because their skill sequences differ?

Why is the annotation performance low in long sequences? How do these errors affect downstream control?

1 comment
AuthorsOpenReview ↗

Response to Reviewer rNK1

Thank you for your careful review and constructive feedback.

Question 1 & Weakness 1: Lack of Real-Robot Experiments

Thank you for pointing out the need for real-robot validation. We added a five-subtask toaster task with randomized initial positions of the bread and toaster. In 20 trials per method, the dual-system method completed the full task in 18 trials (90%), compared with 15 trials (75%) for the end-to-end baseline. This comparison provides preliminary evidence of a benefit on this real-robot task (Table 4 and Appendix B.4).

Question 2 & Weakness 2: Annotation Quality and Control Gains

Thank you for pointing out that the original control experiments changed both the supervision and the inference structure. We added two analyses to address this concern.

Quality of the annotations themselves. In a manual audit of 740 trajectories covering all dataset–embodiment groups, we found 90.6% subtask accuracy and 82.6% episode-level accuracy. The subtask and episode-level accuracies, respectively, were 93.7% and 88.9% for simulation data, 89.7% and 79.3% for real-robot data, and 90.7% and 86.0% for reprocessed data (Appendix A.7, Table A.6).

Impact of annotation quality on downstream control. With the dual-system inference setup fixed, we retrained System 1 using Gemini 3.6 Flash annotations in place of RoboAtom-Data annotations. Success rates dropped from 98.20% to 95.40% on LIBERO-Long and from 88.18% to 84.57% on LIBERO-Plus Overall. This change removed most of the observed gains over the end-to-end baseline on both benchmarks. See our Downstream Robot Control Ablations comment and Appendix C.8 (Table C.7).

Question 3 & Weakness 3: Propagation Assumption and Exclusion Rates

PhysExtend requires the reference and target trajectories to share the same ordered skill sequence. In large-scale robot data collection, standardized protocols typically specify the steps for each task, so repeated demonstrations usually follow the same skill sequence.

Even under standardized collection protocols, demonstrations of the same task can vary in which skills they use or in what order. We observed such variation in 53 of 3,334 tasks (1.59%) and skipped PhysExtend propagation for those tasks. This affected 6 of 1,424 simulation tasks (0.42%) and 47 of 1,910 real-robot tasks (2.46%), covering 6,446 and 11,910 trajectories, respectively (18,356 in total, 1.27% of RoboAtom-Data).

We retained these 53 tasks in the dataset and annotated them trajectory by trajectory. Using the human reference subtask sequence as auxiliary information, GLM-5.3-Flash generated an initial segmentation from the target video. Qwen3.8-27B checked it, and GPT-5.6-sol reviewed and corrected uncertain cases. We combined these annotations with those generated by PhysExtend and included both in the same manual audit (Section 3.4 and Appendix A.6).

Question 4 & Weakness 4: Long-Sequence Annotation Performance and Downstream Effects

  1. Why performance drops on long sequences. Fitting a long video into a fixed visual token budget can obscure brief skill transitions and make temporal boundaries harder to locate (Appendix C.7). Processing long videos in shorter clips may help preserve these details, and we plan to test this approach.
  2. Implications for downstream control. Our control experiments used RoboAtom-Data annotations, which are constructed from human reference annotations, state–action-based propagation, and quality checks. The quality of propagated annotations depends mainly on how closely the target trajectory matches the reference trajectory. So far, we have not observed a clear relationship between propagation quality and sequence length.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.