acceptodds
Under review as a conference paper at ICLR 2027

Where to Split? Benchmarking Subtask Splitting in Open-World Robot Data

Abstract

Subtask splitting is crucial for structuring long robot demonstrations into semantically meaningful units for robot learning. However, existing benchmarks and methods largely focus on constrained settings, limiting their applicability to rapidly growing and increasingly diverse open-world robot data. We introduce RoboSplitBench, a benchmark for systematically evaluating subtask splitting in open-world robot data. It contains demonstrations from diverse open-world scenes, with proprioception, multi-view observations, and human-annotated temporal subtask transition points. To establish a strong reference solution on this benchmark, we further propose RoboSplit. It is a training-free approach that explicitly assigns complementary roles to proprioception and Vision-Language Model (VLM) reasoning. Proprioception guides temporal windowing and selective use of different views, while VLMs predict subtask transition points. We systematically evaluate different methods on RoboSplitBench across diverse scenes and transition types, revealing the challenges of open-world subtask splitting. The strongest baseline achieves an F1 score of 0.479, while RoboSplit establishes a strong reference at 0.713 and consistently outperforms representative baselines across diverse scenes. As robot datasets continue to grow rapidly, our work provides a foundation for their structuring. We provide an anonymous link to RoboSplitBench's data, and an example data sample in supplementary materials.

Primary area:
datasets and benchmarks
Keywords:
Robotics Data Splitting, Benchmark

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_XJeuOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces RoboSplitBench, a benchmark for evaluating subtask boundary detection in open-world robot demonstrations, together with RoboSplit, a training-free subtask annotation method. RoboSplitBench provides manually annotated subtask transitions from two independent annotators and introduces an evaluation protocol that accounts for annotation variability. RoboSplit combines proprioceptive event detection with VLM-based semantic reasoning. It uses gripper events and end-effector motion cues to guide temporal windowing, selectively incorporates wrist-camera observations, and applies cross-seam re-splitting to recover transitions near window boundaries. Experiments show that RoboSplit achieves an F1 score of 0.713 on RoboSplitBench, compared with 0.479 for the strongest reported baseline. The paper also includes component ablations, analyses of different evaluation settings, and additional evaluations against individual annotators.

Critical strengths
  1. A meaningful benchmark for a practically important robot learning problem. RoboSplitBench addresses the challenge of identifying subtask transitions in diverse robot demonstrations. Its collection of 585 demonstrations across 12 scenes, together with manually annotated transition boundaries, provides a useful resource for studying and evaluating subtask segmentation in robot learning. I consider this the main contribution of the paper, as reliable segmentation is relevant to organizing robot demonstration data and understanding temporal task structure.

  2. Thoughtful annotation design with detailed experiment results and analysis. The use of two independent annotators acknowledges the inherent ambiguity of semantic subtask boundaries. The paper also provides comparisons with existing approaches, component ablations, and conventional per-annotator F1 evaluations. Importantly, RoboSplit maintains its relative advantage under the conventional per-annotator evaluation, which supports the robustness of the main empirical conclusion.

Critical weaknesses
  1. Limited methodological novelty and unclear attribution of the reported performance gains. RoboSplit mainly combines existing techniques, including temporal windowing, VLM-based segmentation, and cross-seam refinement, rather than introducing a fundamentally new algorithmic mechanism. The ablation results in Table 2 raise further questions about the contribution of the proposed design: a VLM operating on fixed 60-second windows, without proprioceptive guidance, wrist-camera observations, or cross-seam re-splitting, already achieves an F1 score of 0.614, compared with 0.414 for the whole-video VLM baseline. In contrast, the additional components contribute relatively modest improvements. Furthermore, the primary evaluation protocol uses an asymmetric definition of precision and recall, although the conventional per-annotator results (F1 = 0.615/0.641) provide additional support for the method's effectiveness. Without a stronger baseline that combines windowed VLM inference with simple proprioceptive event hints, it remains difficult to determine how much of the reported 48.9% relative improvement can be attributed specifically to the proposed complementary use of proprioception and visual reasoning.

  2. Limited validation of the downstream utility of the proposed segmentation benchmark. The paper motivates automated subtask segmentation as a way to facilitate robot learning, but does not directly evaluate whether improvements in boundary-detection F1 lead to better policy learning or task success. Consequently, it remains unclear whether the evaluation choices, such as the ±1-second boundary tolerance and the minimum segment length of 3 seconds, are well aligned with the requirements of downstream manipulation tasks. In addition, the benchmark is based on human-held demonstrations rather than trajectories collected from autonomous robot executions, which may limit its applicability to certain robot learning settings. A small-scale downstream experiment, for example comparing policies trained with segmentations of different quality and examining the relationship between segmentation F1 and task success, would provide stronger evidence of the benchmark's practical value.

Questions for authors
  1. Contribution of the proposed components in RoboSplit. Table 2 shows that fixed-window VLM inference already provides a substantial improvement over whole-video inference. Could the authors clarify how much of RoboSplit's improvement comes specifically from the proposed complementary use of proprioception and VLM reasoning? In particular, how would RoboSplit compare with a fixed-window VLM baseline that receives simple gripper-event and end-effector motion hints?

  2. Downstream utility of the benchmark. Does improved subtask boundary-detection F1 translate into better downstream robot learning performance? Could the authors provide evidence that the proposed segmentation granularity and evaluation tolerance are appropriate for downstream manipulation tasks? Even a small-scale validation experiment, such as comparing policies trained using different segmentation methods under the same training budget, would help demonstrate the practical value of the benchmark.

  3. Applicability to contact-rich manipulation. The benchmark enforces a minimum subtask duration of 3 seconds, and RoboSplit mainly relies on visual observations and proprioceptive motion cues. However, contact-rich manipulation tasks, such as insertion and assembly, may involve short but important transitions between different contact states that are difficult to identify from these signals. How applicable is the proposed segmentation framework to such tasks? In particular, could the 3-second minimum duration cause meaningful contact transitions to be merged into a single subtask? It would be useful to discuss the limitations of the current annotation granularity or evaluate its sensitivity to the minimum segment duration.

6 comments
AuthorsOpenReview ↗

Response to Reviewer XJeu, Part 1

Thank you for recognizing the practical value of our RoboSplitBench, its annotation design, and our experimental analysis. This is particularly encouraging for our submission in the Datasets and Benchmarks Primary Area. Regarding your concerns about the contribution of RoboSplit’s components, downstream utility of the benchmark, and applicability to contact-rich manipulation, we respond point by point below.

  • Weakness 1 & Question 1: Methodological contribution and attribution of performance gains. We further explained the methodological contributions, analyzed existing ablation results, and conducted additional baseline experiments.

    RoboSplit complements our proposed RoboSplitBench as a strong, training-free reference solution, illustrating how proprioceptive cues and VLM reasoning can be combined effectively.

    As shown in Table 2, windowing provides the largest improvement and is an integral component of RoboSplit. It allows the VLM to focus on shorter temporal contexts. Building on this, proprioception-guided window adjustment, selective wrist-view observations, and cross-seam re-splitting further improve performance, as demonstrated by the component ablations in Table 2.

    For your suggested comparison, we additionally evaluated State-Prompted VLMs with fixed 60-second windows, providing the VLM with timestamped gripper-event and end-effector motion hints within each window. It achieves an F1 of 0.513, compared with 0.363 for the original State-Prompted VLMs and 0.713 for RoboSplit. Although windowing improves this State-Prompted VLMs baseline, its F1 remains below fixed-window VLM inference without event hints (F1 0.614, in Table 2), consistent with the trend observed in the whole-video setting. These results highlight the importance of how proprioceptive cues are incorporated.

  • Weakness 2 & Question 2: Downstream utility and practical value of RoboSplitBench. We conducted preliminary downstream real-robot experiments

    Thank you for highlighting the importance of downstream validation. During this discussion period, we conducted a preliminary real-robot experiment using the watering task from RoboChallenge[1]. The official splitting method gives a single instruction for the entire task, while RoboSplit divides the task into three meaningful subtasks: picking up the watering can, watering the plant, and putting down the watering can. We observed that the original policy frequently hesitated before acting, occasionally grasped the watering can, and repeatedly performed the pouring motion without stopping. In contrast, the policy trained on segmented demonstrations executed individual subtasks more reliably when their instructions were manually switched. These observations suggest that a good subtask splitting helps policies distinguish different manipulation stages, reduce hesitation caused by similar initial and final states, and avoid repetitive behaviors, highlighting its practical value for downstream robot learning.

    Nevertheless, we emphasize that for efficient evaluation, RoboSplitBench directly evaluates whether predicted boundaries recover the subtask structure identified by human annotators. This evaluation approach is also commonly adopted in related work such as RoboSegNet[2], which evaluates predicted segmentations against annotated transitions and segments. Our benchmark extends this direction to diverse open-world demonstrations, with two independent annotators and an evaluation protocol that accounts for annotation variability.

AuthorsOpenReview ↗

Response to Reviewer XJeu, Part 2

  • Question 3: Applicability to contact-rich manipulation and annotation granularity. We discussed the limitations and future extensions of the benchmark, conducted additional experiments on segmentation granularity, and provided supporting evidence from an external benchmark.

    Thank you for suggesting that we clarify the applicability of our benchmark and annotation granularity. The current version of RoboSplitBench covers 12 everyday scenes, with limited coverage of contact-rich tasks such as precision insertion and assembly. Our annotations focus on semantic subtasks, so brief contact-state transitions may be included within a larger subtask. Transitions without clear visual or kinematic changes also remain challenging for the current framework.

    The 3-second minimum duration was chosen to provide consistent semantic granularity for the demonstrations in our benchmark and was also applied in RoboSplit. To examine its influence, we additionally evaluated RoboSplit with the minimum segment duration reduced to 1 second, keeping the reference annotations unchanged. This setting achieves an F1 of 0.720, Precision of 0.709, and Recall of 0.732, close to the original results. This comparison shows that RoboSplit's performance on the current benchmark is not sensitive to this duration setting.

    To further address your concern about the benchmark's relevance to robot learning, we additionally evaluated RoboSplit on the AgiBot-based RoboSegData benchmark[2], which is specifically designed for robot learning. RoboSplit achieves an F1@±1s score of 0.681 across 21 tasks, comparable to 0.713 on RoboSplitBench. This consistency suggests that RoboSplitBench provides a meaningful evaluation of subtask segmentation quality, with results comparable to those obtained on an existing robot learning benchmark.

    Our data collection already includes factory environments, and these recordings are awaiting curation. Incorporating them will enrich the coverage of contact-rich manipulation. We will also adapt the minimum annotation duration to the requirements of these tasks to preserve meaningful transitions at the appropriate granularity.

Thank you again for your feedback. We hope these clarifications address your concerns and encourage you to reconsider the overall contribution of this paper. We will update the manuscript after the discussion, incorporating all of the feedback received.

[1] Yakefu, Adina, et al. "Robochallenge: Large-scale real-robot evaluation of embodied policies." arXiv preprint arXiv:2510.17950 (2025).

[2] Kaiyuan Chen et al. Learning Multi-Task Robot Trajectory Segmentation from Visual and Kinematic Streams. CVPR Findings, 2026.

AuthorsOpenReview ↗

Invitation for Discussion

Dear Reviewer XJeu,

We would greatly appreciate it if you could take a moment to review our response and let us know whether our responses have addressed your concerns. We would be happy to provide any further clarification.

Thank you for your time and consideration.

Reviewer_mnJDOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces RoboSplitBench, a benchmark for detecting semantic subtask boundaries in long-horizon, open-world robot demonstrations, and RoboSplit, a training-free method for splitting trajectories into meaningful subtasks. The benchmark contains 585 trajectories (about 30 hours) across 12 real-world environments and 50+ manipulation skills, with synchronized multi-view video, end-effector proprioception, and transition annotations from two independent annotators, and its evaluation protocol accounts for annotation ambiguity using consensus and single-annotator transition points. RoboSplit combines proprioception and VLM reasoning in three stages: proprioception-guided temporal windowing (using gripper events and motion pauses), VLM-based transition prediction from main-view and selectively included wrist-view images, and cross-seam re-splitting to recover transitions missed at window boundaries. It achieves an F1 of 0.713, well above the strongest baseline (0.479), and the authors support this with ablations, scene-wise evaluations, and efficiency studies on video sampling rates and image resolutions.

Critical strengths
  1. RoboSplit uses physical signals to structure visual context and leaves semantic boundary decisions to the VLM. It improves F1 from 0.479 to 0.713, gains across all 12 scenes, and is backed by ablations. Selective wrist-view use nearly matches using all wrist views while cutting input tokens by 41.6%.
  2. RoboSplitBench pairs long-horizon demos, diverse scenes, multi-view video, proprioception, and dual annotations, filling the gap between constrained segmentation settings and open-world data. Its analysis of transitions with and without proprioceptive events makes it more than just a larger dataset.
Critical weaknesses
  1. The F1 score is nonstandard and asymmetric: a prediction counts toward precision if either annotator supports it, but recall is computed only on consensus transitions. That makes it difficult to compare against conventional boundary-detection metrics. The individual-annotator results do preserve the method ranking, but the authors should justify more directly why this is the preferred primary score, and report conventional metrics against a clearly defined aggregate reference more prominently. As it stands, I’m not sure how much weight to put on the absolute improvement.
  2. This work is evaluated only on RoboSplitBench, and strong learned alternatives like RoboSegNet are left out because they need training data. That’s reasonable for a training-free comparison, but it undercuts the broader claim of establishing a strong reference solution for open-world subtask splitting. I’d ask the authors to separate claims about tuning-free methods from claims about the task as a whole, and ideally add a training-based setting or an evaluation on an external benchmark.
Questions for authors

How much of RoboSplit’s advantage holds if precision and recall are both computed against a single fixed aggregate reference (please specify how it is built)? Also, Appendix D keeps only trajectories with inter-annotator F1 ≥ 0.6. What fraction was excluded, and how do results and method rankings change on a sample of the excluded trajectories? How were the development scenes chosen, and how do they differ from the 12 benchmark scenes in tasks, environments, and camera setup? Is there any evidence, even preliminary, from a different embodiment or collection platform, given that all trajectories come from the same handheld bimanual device?

6 comments
AuthorsOpenReview ↗

Response to Reviewer mnJD, Part 1

Thank you for recognizing the value of our benchmark, RoboSplitBench, for its diverse real-world demonstrations, dual-annotator protocol, and analysis of proprioceptive events. This is particularly encouraging for our submission in the Datasets and Benchmarks Primary Area. We also appreciate your recognition of the effectiveness of our reference solution, RoboSplit. Regarding your concerns about the evaluation metrics, comparisons with learning-based methods, annotation filtering, development-scene selection, and cross-embodiment generalization, we respond point by point below.

  • Weakness 1 & Question 1: Metric design and fixed-reference evaluation. We explained the evaluation metric design, clarified existing experimental results, and conducted additional evaluations using conventional metrics.

    Reason for using this metric. This metric accounts for human annotation ambiguity. As explained in Section 3.4, Precision credits transitions supported by either annotator, while Recall measures recovery of consensus transitions. This avoids penalizing plausible boundaries or requiring recovery of annotator-specific boundaries.

    Explanation of individual-annotator evaluations. Our individual-annotator evaluations already use conventional Precision, Recall, and F1 with a consistent reference. Table 4 in Appendix D.1 computes Precision and Recall against each annotator's complete transition set independently. These Precision and Recall metrics, together with a ±1s matching tolerance, are commonly used in the community, for example in RoboSegNet[1].

    Additional aggregate reference results. Following Section 3.4, we match the two annotations within ±1\pm1s and average matched timestamps to obtain consensus transitions. We evaluate against these alone and against their union with all single-annotator transitions, using the same reference for Precision and Recall. Here we report the standard reference F1 score:

Reference setRoboSplitState OnlyWhole-Video VLMsUVDState-Prompted VLMsSuzuki et al.State-to-VLMs Refine
Consensus transitions0.65190.40140.37970.22210.33930.22210.4028
Consensus + single-annotator transitions0.61930.45260.35840.31960.30570.30380.4499
  • Weakness 2: Learned alternatives and external evaluation. We conducted additional experiments with a learning-based baseline and evaluated our method on an external benchmark.

    Training-based method. As the RoboSegNet[1] checkpoint has not been publicly released, we evaluated FLORA[2], a model specifically fine-tuned for subtask splitting, on RoboSplitBench. FLORA achieves an F1 of 0.572, Recall of 0.688, and Precision of 0.490. Its F1 exceeds all other baselines in Figure 5, while remaining below RoboSplit's 0.713.

    Other benchmark evaluation. We additionally evaluated RoboSplit on the AgiBot-based RoboSegData benchmark[1] to assess its performance beyond RoboSplitBench. Their reported RoboSegNet is trained on 18 tasks from RoboSegData and evaluated on both these 18 tasks and 3 additional unseen tasks, achieving F1@±1s scores of 0.387 and 0.303, respectively.

    Since this benchmark consists of real-world robot demonstrations that are generally short (rarely exceeding 60 seconds), we adapt our prompt to inform the VLM that the inputs are real-world robot data and adjust the window size to 15 seconds. These are the only two modifications to our method.

    Our method achieves F1@±1s scores of 0.686 on the 18 tasks and 0.654 on the other 3 tasks. And our representative State-Only baseline achieves F1 scores of 0.242 and 0.256, respectively. These results are comparable to those obtained on our RoboSplitBench, demonstrating the consistency of our method's performance across the two benchmarks and establishing a connection between RoboSplitBench and this existing benchmark.

AuthorsOpenReview ↗

Response to Reviewer mnJD, Part 2

  • Question 2: Filtering rate and evaluation on excluded trajectories. We provided additional information on the data filtering process and conducted additional experiments on previously excluded trajectories.

    Approximately 50% of the candidate trajectories were excluded by the inter-annotator F1 threshold of 0.6 in Appendix D. Below, we additionally compare RoboSplit with State Only, the strongest baseline in Figure 5, on these excluded trajectories.

    RoboSplit achieves an F1 of 0.670 (Recall: 0.649, Precision: 0.692), outperforming State Only with an F1 of 0.463 (Recall: 0.479, Precision: 0.448). RoboSplit maintains its advantage, with a slight F1 decrease primarily driven by lower precision, potentially due to greater annotation ambiguity.

  • Question 3: Development scenes and benchmark separation. We provided details on the development data and explained its separation from the benchmark data.

    As stated in Appendix M, RoboSplit was developed in separate bedroom environments outside RoboSplitBench. The development data came from a separate collection batch. Under our crowdsourced collection process, we used separate groups of collectors for development and benchmark data. Although the development environments and the benchmark's bedroom environments share the same scene category, their collection locations, environments, and tasks differ. The camera setup is identical because the same data-collection device was used.

  • Question 4: Evidence from another collection platform. We conducted additional experiments.

    As detailed in our response to Weakness 2, we additionally evaluated RoboSplit on RoboSegData[1], an external benchmark collected using AgiBot robots rather than our handheld bimanual device.

Thank you again for your feedback. We hope these clarifications address your concerns and encourage you to reconsider the overall contribution of this paper. We will update the manuscript after the discussion, incorporating all of the feedback received.

[1] Kaiyuan Chen et al. Learning Multi-Task Robot Trajectory Segmentation from Visual and Kinematic Streams. CVPR Findings, 2026.

[2] Sicong Leng et al. FLORA: An Open Model Suite for Scaling Robotics Data Engines. arXiv:2610.31779, 2026.

AuthorsOpenReview ↗

Invitation for Discussion

Dear Reviewer mnJD,

We would greatly appreciate it if you could take a moment to review our response and let us know whether our responses have addressed your concerns. We would be happy to provide any further clarification.

Thank you for your time and consideration.

Reviewer_BNUmOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

In this paper, the author present a self-annotated trajectory segmentation benchmark and propose a VLM-based method to detect the subtask transition timestamps. Specifically, this paper collects a 30 hours human egocentric dataset with UMI-style robot data collection tool to record proprioceptive state. The dataset contains diverse scene setup and background with rich visual observation. On this dataset, the author annotate subtask transition timestamps. The paper also propose a training free method that use proprioceptive as a coarse guidance to localize the candidate event time window. Experiment result on propose robosplitbench shows that the robosplit method outperforms baselines.

Critical strengths
  1. The human annotated dataset could be valuable to the community. Propose benchmark has two human annotator to cross validate the timestamps and provide the agreement statistic.
  2. Directly using proprioceptive state event is a simple but potentially effective design for subtask transition.
  3. The paper is well organized and the overall presentation is clear.
Critical weaknesses
  1. Despite proprioceptive state event as a key design of this paper, the author does not rigorously explain this critical concept: a. how does these state is selected. b. Except from gripper open/close, what are other metrics that is used for determine the transition event. c. how to differentiate the event to noise given the complicated and high degree of freedom signal from data collection process? i.e. how to differentiate a pause is required by the task itself or just the noise from data collector.
  2. For the benchmark: There is only one VLM being evaluated in this benchmark. The author does not explain why this model and does not include any state of the art VLM such as Qwen, Gemma, ChatGPT and Gemini. Since the paper consider benchmark as core contribution, the overall evaluation need to cover a subset of representative models.
  3. This robosplit method's evaluation is mostly based on the proposed benchmark. Since the proposed trajectory segmentation method does not need any training, the author should compare the method on different dataset to show the method's generalization ability. The benchmark and paper would be more technical convincing if the author can establish correlation between the proposed benchmark and other existing benchmark such as Agibot's subtask detection.
Questions for authors

Please address questions in weaknesses section. Please includes more details about method, benchmark correlation and evaluating the state-of-the-art model for completeness of benchmark.

2 comments
AuthorsOpenReview ↗

Response to Reviewer BNUm

Thank you for recognizing the value of our benchmark, RoboSplitBench, and its human-annotated data. This is particularly encouraging for our submission in the Datasets and Benchmarks Primary Area. Regarding your concerns about the proprioceptive-event design of our reference solution RoboSplit, representative VLM coverage, and cross-dataset evaluation and benchmark correlation, we respond point by point below.

  • Weakness 1: Proprioceptive events and noise handling. We clarified the methodological details, provided further explanations, and highlighted supporting evidence from existing experiments.

    (a, b) Event selection and detection. When adjusting window boundaries, we consider not only gripper events but also motion pauses. For gripper events, we apply 0.5-second moving-average smoothing to gripper signals, followed by thresholding at the midpoint of their 5th–95th percentile range and a 0.8-second minimum-duration filter. And for motion pauses, we detect pauses based on the combined translational motion of both end-effectors (EEFs). Specifically, we sum their frame-to-frame translation magnitudes and smooth the resulting signal over up to 15 frames. We then identify local minima separated by at least 2 seconds, with a minimum prominence of 10% of the signal range.

    (c) Proprioception noise handling. A detected pause does not automatically become a subtask transition. As explained in Section 4 and Figure 4, physical cues ONLY organize temporal windows and visual evidence, while VLM reasoning determines semantic boundaries. The prompt explicitly keeps within-action adjustments and failed attempts immediately followed by successful retries within the same segment. At unannotated points near physical events (which may correspond to the noise), Table 9 reports a false-transition match rate of 0.133 for RoboSplit versus 0.447 for State Only. Thus, filtered physical cues and visual semantic reasoning jointly reduce spurious splitting.

  • Weakness 2: VLM coverage and model selection. We clarified where the existing experimental results are reported and conducted additional experiments.

    Thank you for emphasizing the importance of representative VLM coverage. We chose Doubao-Seed-2.1-Pro (thinking disabled) for its lower price and competitive performance. And Appendix L has reported results for Doubao-Seed-2.1-Turbo (thinking disabled), Doubao-Seed-2.1-Pro (thinking enabled), and GPT-5.6-Sol (thinking disabled), providing a broader evaluation across different VLMs.

    During this discussion, we additionally evaluated Qwen-3.8-Max (thinking disabled), obtaining Precision of 0.645, Recall of 0.727, and F1 of 0.684 at a 1-second matching tolerance. Overall, Turbo and Qwen achieve lower F1 than our used Seed-Pro, while GPT performs better. These results broaden model coverage.

  • Weakness 3: Cross-dataset evaluation. We conducted additional experiments.

    Thank you for suggesting evaluation beyond RoboSplitBench. We supplemented the evaluation of our method on RoboSegData[1], an benchmark based on the AgiBot dataset. Their method, RoboSegNet, is trained on 18 tasks from RoboSegData and evaluated on both these 18 tasks and 3 additional unseen tasks, achieving F1@±1s scores of 0.387 and 0.303, respectively.

    Since this benchmark consists of real-world robot demonstrations that are generally short (rarely exceeding 60 seconds), we adapt our prompt to inform the VLM that the inputs are real-world robot data and adjust the window size to 15 seconds. These are the only two modifications to our method.

    Our method achieves F1@±1s scores of 0.686 on the 18 tasks and 0.654 on the other 3 tasks. And our representative State-Only baseline achieves F1 scores of 0.242 and 0.256, respectively. These results are comparable to those obtained on our RoboSplitBench, demonstrating the consistency of our method's performance across the two benchmarks and establishing a connection between RoboSplitBench and this existing benchmark.

Thank you again for your feedback. We hope these clarifications address your concerns and sincerely hope you will reconsider the overall contribution of this paper, especially in the context of its primary area, Datasets and Benchmarks. We will update the manuscript after the discussion, taking all feedback into account.

[1] Kaiyuan Chen et al. Learning Multi-Task Robot Trajectory Segmentation from Visual and Kinematic Streams. CVPR Findings, 2026.

AuthorsOpenReview ↗

Invitation for Discussion

Dear Reviewer BNUm,

We would greatly appreciate it if you could take a moment to review our response and let us know whether our responses have addressed your concerns. We would be happy to provide any further clarification.

Thank you for your time and consideration.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Author Response Summary

We sincerely thank all reviewers for their thoughtful and constructive feedback, as well as their recognition of the value of our work. We are particularly encouraged that all three reviewers acknowledge the quality and contribution of our benchmark, RoboSplitBench, highlighting its diverse real-world demonstrations and carefully designed human annotated data. This shared recognition is especially encouraging given that our paper is submitted to the Datasets and Benchmarks primary area, where the quality and utility of the benchmark are central to its contribution.

The reviewers also raised several important constructive feedback, primarily regarding:

  • methodological clarity and the contribution of individual components, particularly the role of proprioceptive events and their complementary use with VLM reasoning;
  • the comprehensiveness of benchmark evaluation, including stronger VLM baselines, conventional evaluation metrics, and comparisons with existing benchmarks;
  • the generalizability of RoboSplit, especially across different datasets and collection platforms;
  • the downstream utility and applicability of the benchmark, including its evaluation tolerance, segmentation granularity, and limitations in contact-rich manipulation.

In our responses to individual reviewers, we have carefully clarified potential misunderstandings and provided further explanations of the questions raised. Beyond these clarifications, we have made additional efforts to strengthen the paper, including:

  • Expanded baseline evaluations, incorporating additional VLMs and training-based method FLORA[1].
  • Cross-dataset evaluations, including experiments on RoboSegData Benchmark[2].
  • Additional robustness analyses, including evaluations on previously excluded trajectories with lower inter-annotator agreement.
  • Downstream real-robot experiments, providing preliminary evidence of the practical benefits of subtask segmentation for robot policy learning.

We sincerely thank the reviewers again for their valuable feedback and constructive suggestions, which have helped us further strengthen our work. We greatly appreciate the time and effort devoted to reviewing our paper. We hope our responses adequately address the reviewers' concerns and encourage them to reconsider the overall contribution of our paper. We will carefully revise the manuscript after receiving further feedback from the reviewers.

[1] Sicong Leng et al. FLORA: An Open Model Suite for Scaling Robotics Data Engines. arXiv:2610.31779, 2026.

[2] Kaiyuan Chen et al. Learning Multi-Task Robot Trajectory Segmentation from Visual and Kinematic Streams. CVPR Findings, 2026.


Summary of Updates Before the Discussion Deadline

Based on the reviewers' feedback, we have revised the manuscript to address their comments and concerns. We sincerely thank all reviewers for their constructive feedback and valuable suggestions. We have corrected a transcription error and incorporated the following additional experiments, explanations, and clarifications into the revised manuscript:

  1. Revised Section 5.2: Added a windowed State-Prompted VLM baseline (F1: 0.513) and updated the Abstract, Introduction, and related performance comparisons.
  2. Added Section 5.6: Evaluated RoboSplit on RoboSegData, demonstrating its effectiveness on an existing benchmark.
  3. Revised Section 6 Paragraph "Future Work": Clarified limitations regarding contact-rich manipulation and outlined future improvements.
  4. Revised Appendix D: Evaluated RoboSplit on excluded trajectories to examine its robustness to annotation ambiguity.
  5. Revised Appendix D.1: Added Table R1 to report conventional F1 scores using the same reference set for Precision and Recall.
  6. Revised Appendix L: Added evaluations with the additional VLM backbone, Qwen-3.8-Max.
  7. Revised Appendix M: Clarified the separation between method development and benchmark evaluation.
  8. Revised Appendix N: Added implementation details for proprioceptive event selection and algorithm pseudocode.
  9. Added Appendix O: Compared RoboSplit with FLORA, a concurrent training-based subtask splitting method.
  10. Added Appendix P: Conducted a preliminary downstream real-robot experiment to demonstrate the practical benefits of subtask splitting.

These additions are highlighted in green in the revised manuscript.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.