- Rating
- 2
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
To address missing subtask annotations and inconsistencies in action categories, segmentation granularity, and temporal boundaries across robotic datasets, this paper introduces RoboAtom-Data, a unified atomic-action supervision dataset, alongside RoboAtom-Bench and the automatic annotation model RoboAtom-Anno. Its primary contributions lie in standardizing cross-source annotations, designing a scalable annotation pipeline, and constructing a large-scale supervision resource, not proposing a new model architecture.
Critical strengths
- The unified annotation resource has clear practical value: By providing the dataset, construction pipeline, and evaluation benchmark, the paper offers a potentially useful resource for the robotics research community.
- Practical annotation pipeline: The pipeline combines model-assisted annotation, human correction, constrained same-task propagation, and targeted quality control to reduce repetitive annotation effort; experiments on RoboAtom-Bench and WGO-Bench support its practical utility.
Critical weaknesses
- Limited methodological novelty: PhysExtend segments skills using a physically informed scheme(state–action, motion change, gripper events) already explored in prior embodied-learning work on atomic actions. Therefore, this work as an engineering refinement of a unified annotation pipeline and the construction of a large-scale datasets, not a methodological advance.
- The requirements stipulate that segmentation should occur "only when the dominant atomic skill changes," while also mandating the decomposition of composite behaviors. How are these rules reconciled in continuous tasks (such as folding laundry)? How should one define boundaries in data where such atomic transitions are inherently ambiguous, and how is the appropriate granularity for segmentation determined?
- RoboAtom-Anno achieves results superior to the evaluated baselines on two annotation benchmarks, existing metrics primarily focus on the alignment between temporal segments and skill categories. The accuracy of success/failure classification for atomic skills, the precision and recall of online completion detection are not reported.
- In the downstream experiments, the approach simultaneously modifies the training language conditions and incorporates a high-level planner. While the performance gains shown in Table 4 confirm the efficacy of the complete system, the improvements are modest—only 2%–5% on tasks that not yet reached saturation. Moreover, RoboAtom-Anno has already been trained to identify atomic skills and their temporal boundaries, why does "System 2" employ a separate Qwen3-27B model instead of reusing the existing model to directly evaluate its performance?
Questions for authors
As noted above, My main concerns are the following:
- How does RoboAtom-Anno perform on continuous tasks with ambiguous boundaries, and what is the granularity at which it segments atomic actions?
- Please provide quantitative metrics for annotation quality, such as precision and recall? When applied to a specific, independent dataset, the primary considerations should be the cost of human labor and the accuracy of the automated pipeline.
- For the downstream "System 2" component, which utilizes a standalone Qwen3.8-27B model, would performance on downstream tasks improve if incorporates RoboAtom-Anno?
- The granularity of "atomic actions" is partly determined by manually defined rules. However, given that some tasks involve strong sequential dependencies, is the current approach superior to other reasonable granularities? How does it compare to supervision applied at the level of coarser sub-tasks, or the current atomic-action level?
3 comments
Thank you for providing the additional experiments and clarifications, and this addresses some of our concerns. I find that PhysExtend still exhibits limited accuracy and relies on manual verification, mostly. Combined with my assessment that this work is primarily engineering refinement, as you mentioned in Answer 1, I will keep my current rating.
Manual Verification and the Contributions of RoboAtom
We thank the reviewer for the additional feedback. We briefly address the two concerns below.
On "PhysExtend still exhibits limited accuracy and relies on manual verification"
We agree with the first half of this assessment: measured against 100% accuracy, PhysExtend indeed "still exhibits limited accuracy." Under the same annotation extension task and evaluation protocol, however, PhysExtend ranks first on all four accuracy metrics (Table 2 of the main paper).
| Model | mIoU | [email protected] | Skill [email protected] | Episode Accuracy (%) | Cost (USD/h) |
|---|---|---|---|---|---|
| GPT-4o | 48.2 | 20.7 | 11.1 | 10 | 0.888 |
| Gemini 3.1 Pro | 61.1 | 44.9 | 39.7 | 39 | 2.877 |
| GPT-5.6-sol | 58.9 | 37.8 | 35.0 | 46 | 3.510 |
| Qwen3.8-27B | 46.1 | 25.4 | 19.1 | 25 | 0.051 |
| Cortex | 66.5 | 49.8 | 48.5 | 37 | 4.8 × 10⁻⁴ |
| PhysExtend | 76.8 | 71.6 | 71.0 | 69 | 2.8 × 10⁻⁵ |
Precisely because we agree that absolute accuracy matters more than relative ranking, we follow automatic annotation with manual quality checks and corrections. Used as a standalone automatic annotator, PhysExtend achieves an Episode Accuracy of 69.0%. After manual quality checking and iteration, Episode Accuracy reaches 82.6% and subtask accuracy reaches 90.6%.
We therefore regard this manual step as a strength of the paper rather than a weakness: it makes RoboAtom-Data a dataset with a known error distribution and a committable quality floor, not merely "another set of automatic annotation results." By comparison, a pipeline that relies entirely on a general-purpose model has no explicit acceptance criteria, yet costs five orders of magnitude more (3.510 vs. 2.8 × 10⁻⁵ USD per source-video hour) while achieving less than half the Skill [email protected] (35.0 vs. 71.0).
In other words, we accept that automatic precision is "limited" — and that is precisely why a verifiable quality-control stage is needed; the existence of this stage is exactly what makes the data safe to use.
On "this work is primarily engineering refinement"
We agree with this description, and it has been our consistent positioning throughout: the first page of the Introduction already presents the work as "a unified resource and construction pipeline." This is precisely the contribution we claim. Datasets and benchmarks are established categories of research contribution, and their evaluation criteria are scale, consistency, verifiability, and downstream utility — not the algorithmic novelty of any individual component. Under that standard, our claims are as follows.
-
First, cross-source unification. Few robot datasets provide subtask annotations, and where they exist (AgiBot World, RoboInter-Data, BEHAVIOR-1K) vocabularies, granularity, and temporal coverage differ, making them impossible to combine directly. To our knowledge, this is the first work to unify 16 heterogeneous sources under one 29-category atomic-action vocabulary and temporal standard (1.45M trajectories, 6.88M segments, 16,882 hours, 3,334 tasks, 32 embodiments).
-
Second, a verifiable quality floor. To our knowledge, this is the first robot manipulation dataset to declare thresholds before release (subtask ≥ 85%, episode ≥ 75%) and report measured results against them: 90.6% / 82.6% on the 740-trajectory audit, reported by source (Appendix A.6) so users can judge the distribution of errors.
-
Third, a benchmark for multi-source unification. Comparable benchmarks use free-text labels and few source families (WGO-Bench: 3 families, 100 episodes, 743 segments), so scores are hard to compare across pipelines. RoboAtom-Bench uses a fixed 15-category vocabulary and unified temporal protocol covering 99.2% of skill-labeled segments across 16 sources (500 trajectories, 4,018 segments), and can serve as a common standard for later cross-source work.
-
Fourth, downstream utility. In automatic annotation, RoboAtom-Anno-9B trained on RoboAtom-Data outperforms GPT-5.6-sol on RoboAtom-Bench and WGO-Bench Robot at ~1/720 the inference cost (Table 3). In robot control, with the dual-system setup fixed and only the annotation source varied, RoboAtom annotations yield higher LIBERO-Long and LIBERO-Plus success rates than Gemini 3.6 Flash annotations, by 2.80 and 3.61 points (Table C.7).
We acknowledge that this work does not introduce an entirely new model architecture. We believe its contribution should be judged on the merits of the data resource and benchmark it provides. We thank the reviewer again for their time and feedback.
Response to Reviewer 9cSB
We sincerely thank the reviewer for the careful review and constructive feedback. We first clarify two points:
Weakness 1: Methodological Novelty
The revised Introduction presents this work as a data resource and system integration effort that builds on prior work in segmentation using physical signals.
Question 1 & Weakness 2: Ambiguous Boundaries and Segmentation Granularity
Boundaries in continuous tasks such as folding are clearer than they may appear. Because robot data collection is relatively standardized, most atomic-action boundaries are marked by physical events such as grasp, release, and contact changes. Consecutive folds can be reliably separated (see the demo at 28–38 s). To support consistent boundary annotation and resolve uncertain cases, we use the following three measures:
Question 2: Annotation Quality
On 100 tasks across 10 datasets, human evaluation gave PhysExtend 74.6% segment-level precision and recall and 69.0% episode-level accuracy. After quality control, a manual audit of 740 episodes found 90.6% subtask accuracy and 82.6% episode-level accuracy.
Question 3 & Weaknesses 3 and 4: Dual-System Ablations
See Downstream Robot Control Ablations for Questions 3 and 4.
Question 4: Comparison with Coarser-Grained Supervision
We merged adjacent Pick–Place actions in LIBERO into “place [object] at [location]” and retrained the control policy. Success rates were close to those with atomic supervision on LIBERO-Long and LIBERO-Plus Overall. We favor atomic segmentation for two reasons: