- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces RoboSplitBench, a benchmark for evaluating subtask boundary detection in open-world robot demonstrations, together with RoboSplit, a training-free subtask annotation method. RoboSplitBench provides manually annotated subtask transitions from two independent annotators and introduces an evaluation protocol that accounts for annotation variability. RoboSplit combines proprioceptive event detection with VLM-based semantic reasoning. It uses gripper events and end-effector motion cues to guide temporal windowing, selectively incorporates wrist-camera observations, and applies cross-seam re-splitting to recover transitions near window boundaries. Experiments show that RoboSplit achieves an F1 score of 0.713 on RoboSplitBench, compared with 0.479 for the strongest reported baseline. The paper also includes component ablations, analyses of different evaluation settings, and additional evaluations against individual annotators.
Critical strengths
-
A meaningful benchmark for a practically important robot learning problem. RoboSplitBench addresses the challenge of identifying subtask transitions in diverse robot demonstrations. Its collection of 585 demonstrations across 12 scenes, together with manually annotated transition boundaries, provides a useful resource for studying and evaluating subtask segmentation in robot learning. I consider this the main contribution of the paper, as reliable segmentation is relevant to organizing robot demonstration data and understanding temporal task structure.
-
Thoughtful annotation design with detailed experiment results and analysis. The use of two independent annotators acknowledges the inherent ambiguity of semantic subtask boundaries. The paper also provides comparisons with existing approaches, component ablations, and conventional per-annotator F1 evaluations. Importantly, RoboSplit maintains its relative advantage under the conventional per-annotator evaluation, which supports the robustness of the main empirical conclusion.
Critical weaknesses
-
Limited methodological novelty and unclear attribution of the reported performance gains. RoboSplit mainly combines existing techniques, including temporal windowing, VLM-based segmentation, and cross-seam refinement, rather than introducing a fundamentally new algorithmic mechanism. The ablation results in Table 2 raise further questions about the contribution of the proposed design: a VLM operating on fixed 60-second windows, without proprioceptive guidance, wrist-camera observations, or cross-seam re-splitting, already achieves an F1 score of 0.614, compared with 0.414 for the whole-video VLM baseline. In contrast, the additional components contribute relatively modest improvements. Furthermore, the primary evaluation protocol uses an asymmetric definition of precision and recall, although the conventional per-annotator results (F1 = 0.615/0.641) provide additional support for the method's effectiveness. Without a stronger baseline that combines windowed VLM inference with simple proprioceptive event hints, it remains difficult to determine how much of the reported 48.9% relative improvement can be attributed specifically to the proposed complementary use of proprioception and visual reasoning.
-
Limited validation of the downstream utility of the proposed segmentation benchmark. The paper motivates automated subtask segmentation as a way to facilitate robot learning, but does not directly evaluate whether improvements in boundary-detection F1 lead to better policy learning or task success. Consequently, it remains unclear whether the evaluation choices, such as the ±1-second boundary tolerance and the minimum segment length of 3 seconds, are well aligned with the requirements of downstream manipulation tasks. In addition, the benchmark is based on human-held demonstrations rather than trajectories collected from autonomous robot executions, which may limit its applicability to certain robot learning settings. A small-scale downstream experiment, for example comparing policies trained with segmentations of different quality and examining the relationship between segmentation F1 and task success, would provide stronger evidence of the benchmark's practical value.
Questions for authors
-
Contribution of the proposed components in RoboSplit. Table 2 shows that fixed-window VLM inference already provides a substantial improvement over whole-video inference. Could the authors clarify how much of RoboSplit's improvement comes specifically from the proposed complementary use of proprioception and VLM reasoning? In particular, how would RoboSplit compare with a fixed-window VLM baseline that receives simple gripper-event and end-effector motion hints?
-
Downstream utility of the benchmark. Does improved subtask boundary-detection F1 translate into better downstream robot learning performance? Could the authors provide evidence that the proposed segmentation granularity and evaluation tolerance are appropriate for downstream manipulation tasks? Even a small-scale validation experiment, such as comparing policies trained using different segmentation methods under the same training budget, would help demonstrate the practical value of the benchmark.
-
Applicability to contact-rich manipulation. The benchmark enforces a minimum subtask duration of 3 seconds, and RoboSplit mainly relies on visual observations and proprioceptive motion cues. However, contact-rich manipulation tasks, such as insertion and assembly, may involve short but important transitions between different contact states that are difficult to identify from these signals. How applicable is the proposed segmentation framework to such tasks? In particular, could the 3-second minimum duration cause meaningful contact transitions to be merged into a single subtask? It would be useful to discuss the limitations of the current annotation granularity or evaluate its sensitivity to the minimum segment duration.
6 comments
Response to Reviewer XJeu, Part 2
-
Question 3: Applicability to contact-rich manipulation and annotation granularity. We discussed the limitations and future extensions of the benchmark, conducted additional experiments on segmentation granularity, and provided supporting evidence from an external benchmark.
Thank you for suggesting that we clarify the applicability of our benchmark and annotation granularity. The current version of RoboSplitBench covers 12 everyday scenes, with limited coverage of contact-rich tasks such as precision insertion and assembly. Our annotations focus on semantic subtasks, so brief contact-state transitions may be included within a larger subtask. Transitions without clear visual or kinematic changes also remain challenging for the current framework.
The 3-second minimum duration was chosen to provide consistent semantic granularity for the demonstrations in our benchmark and was also applied in RoboSplit. To examine its influence, we additionally evaluated RoboSplit with the minimum segment duration reduced to 1 second, keeping the reference annotations unchanged. This setting achieves an F1 of 0.720, Precision of 0.709, and Recall of 0.732, close to the original results. This comparison shows that RoboSplit's performance on the current benchmark is not sensitive to this duration setting.
To further address your concern about the benchmark's relevance to robot learning, we additionally evaluated RoboSplit on the AgiBot-based RoboSegData benchmark[2], which is specifically designed for robot learning. RoboSplit achieves an F1@±1s score of 0.681 across 21 tasks, comparable to 0.713 on RoboSplitBench. This consistency suggests that RoboSplitBench provides a meaningful evaluation of subtask segmentation quality, with results comparable to those obtained on an existing robot learning benchmark.
Our data collection already includes factory environments, and these recordings are awaiting curation. Incorporating them will enrich the coverage of contact-rich manipulation. We will also adapt the minimum annotation duration to the requirements of these tasks to preserve meaningful transitions at the appropriate granularity.
Thank you again for your feedback. We hope these clarifications address your concerns and encourage you to reconsider the overall contribution of this paper. We will update the manuscript after the discussion, incorporating all of the feedback received.
[1] Yakefu, Adina, et al. "Robochallenge: Large-scale real-robot evaluation of embodied policies." arXiv preprint arXiv:2510.17950 (2025).
[2] Kaiyuan Chen et al. Learning Multi-Task Robot Trajectory Segmentation from Visual and Kinematic Streams. CVPR Findings, 2026.
Invitation for Discussion
Dear Reviewer XJeu,
We would greatly appreciate it if you could take a moment to review our response and let us know whether our responses have addressed your concerns. We would be happy to provide any further clarification.
Thank you for your time and consideration.
Response to Reviewer XJeu, Part 1
Thank you for recognizing the practical value of our RoboSplitBench, its annotation design, and our experimental analysis. This is particularly encouraging for our submission in the Datasets and Benchmarks Primary Area. Regarding your concerns about the contribution of RoboSplit’s components, downstream utility of the benchmark, and applicability to contact-rich manipulation, we respond point by point below.
Weakness 1 & Question 1: Methodological contribution and attribution of performance gains. We further explained the methodological contributions, analyzed existing ablation results, and conducted additional baseline experiments.
RoboSplit complements our proposed RoboSplitBench as a strong, training-free reference solution, illustrating how proprioceptive cues and VLM reasoning can be combined effectively.
As shown in Table 2, windowing provides the largest improvement and is an integral component of RoboSplit. It allows the VLM to focus on shorter temporal contexts. Building on this, proprioception-guided window adjustment, selective wrist-view observations, and cross-seam re-splitting further improve performance, as demonstrated by the component ablations in Table 2.
For your suggested comparison, we additionally evaluated State-Prompted VLMs with fixed 60-second windows, providing the VLM with timestamped gripper-event and end-effector motion hints within each window. It achieves an F1 of 0.513, compared with 0.363 for the original State-Prompted VLMs and 0.713 for RoboSplit. Although windowing improves this State-Prompted VLMs baseline, its F1 remains below fixed-window VLM inference without event hints (F1 0.614, in Table 2), consistent with the trend observed in the whole-video setting. These results highlight the importance of how proprioceptive cues are incorporated.
Weakness 2 & Question 2: Downstream utility and practical value of RoboSplitBench. We conducted preliminary downstream real-robot experiments
Thank you for highlighting the importance of downstream validation. During this discussion period, we conducted a preliminary real-robot experiment using the watering task from RoboChallenge[1]. The official splitting method gives a single instruction for the entire task, while RoboSplit divides the task into three meaningful subtasks: picking up the watering can, watering the plant, and putting down the watering can. We observed that the original policy frequently hesitated before acting, occasionally grasped the watering can, and repeatedly performed the pouring motion without stopping. In contrast, the policy trained on segmented demonstrations executed individual subtasks more reliably when their instructions were manually switched. These observations suggest that a good subtask splitting helps policies distinguish different manipulation stages, reduce hesitation caused by similar initial and final states, and avoid repetitive behaviors, highlighting its practical value for downstream robot learning.
Nevertheless, we emphasize that for efficient evaluation, RoboSplitBench directly evaluates whether predicted boundaries recover the subtask structure identified by human annotators. This evaluation approach is also commonly adopted in related work such as RoboSegNet[2], which evaluates predicted segmentations against annotated transitions and segments. Our benchmark extends this direction to diverse open-world demonstrations, with two independent annotators and an evaluation protocol that accounts for annotation variability.