acceptodds
Under review as a conference paper at ICLR 2027

SCHEMA: Scalable Harness-Guided Temporal Quality Metadata Construction for Robotic Manipulation Learning

Abstract

Robotic manipulation trajectories contain temporally heterogeneous behavior, including productive progress, inefficient attempts, recoveries, and failures, yet standard Vision-Language-Action (VLA) model training largely treats these segments alike. We introduce **SCHEMA**, a scalable harness-guided framework that uses a Vision-Language Model (VLM) to construct sparse temporal quality metadata through iterative evidence seeking, contextual quality assessment, and boundary refinement, enabling better use of trajectory quality during VLA training. We further introduce **RTQ-Bench** (Robot Temporal Quality Benchmark), built from multiple public manipulation datasets with human-annotated quality intervals. On RTQ-Bench, SCHEMA substantially improves temporal quality localization over single-pass inference with the same backbone and performs competitively against diverse VLM- and score-based baselines. Pretraining on approximately 1,500 hours of real-world manipulation data with SCHEMA-guided sampling improves downstream VLA performance over uniform sampling, with the training data, model initialization, action-learning objective, and optimization schedule held fixed. Across four real-world tasks, SCHEMA-generated metadata improves both sampling-based and advantage-conditioned learning, with larger gains when failed policy rollouts are included.

Primary area:
applications to robotics, autonomy, planning
Keywords:
harness, metadata, robot learning

Reviews

Among:
Shape:
1234this paper, Bottom 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.25 over 4 reviews, above 49% of the others.
Reviewer_UmV5OpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces SCHEMA, a harness-guided framework that uses a VLM to iteratively acquire visual evidence, assess context, and refine temporal boundaries to produce sparse optimal/suboptimal interval metadata from robot manipulation trajectories. It also presents RTQ-Bench, a human-annotated benchmark for temporal quality localization. The authors show that SCHEMA substantially improves temporal localization over single-pass inference with the same backbone and delivers competitive results against strong VLM and score-based baselines; furthermore, SCHEMA-guided sampling and conditioning yield consistent gains in large-scale VLA pretraining on RoboTwin 2.0 and in real-world bimanual tasks, particularly when failed rollouts are included.

Critical strengths
  1. SCHEMA combines budgeted evidence seeking, contextual quality assessment, and boundary refinement into a structured framework for temporal quality annotation, improving localization over single-pass inference with the same VLM backbone.

  2. The generated metadata supports both sampling-based and quality-conditioned policy learning without requiring the annotation harness at deployment, providing a reusable interface between trajectory assessment and VLA training.

  3. Evaluation spans RTQ-Bench, downstream RoboTwin 2.0 tasks, and four real-world manipulation tasks. The real-world results suggest improved use of mixed-quality experience, with larger average benefits when failed rollouts are included.

Critical weaknesses
  1. There is no ablation of SCHEMA’s main components, such as iterative evidence seeking and boundary refinement, making it difficult to determine which design choices account for the reported gains.

  2. The definition of “optimal/suboptimal” relies on a rubric and human judgment. Independent inter-annotator agreement is not reported, leaving uncertainty about the consistency of event selection, quality labels, and temporal boundaries.

  3. SCHEMA achieves higher “All” macro BS-IoU but lower GT+ macro BS-IoU and CC-Recall than GPT-5.6. This mixed ranking warrants further analysis of how sparse annotations and empty-reference episodes influence the comparison.

  4. Progress-based baselines are evaluated through a shared progress-to-label conversion that may miss stagnation and repeated ineffective attempts. Their reported performance may therefore partly reflect limitations of the conversion rather than the underlying models.

Questions for authors
  1. How much of the localization improvement comes from adaptive evidence seeking and boundary refinement, rather than additional observations and model calls? Can you provide a budget-matched comparison or a targeted ablation to clarify this?

  2. Since unannotated regions are not verified negatives, have you checked whether predictions outside the reference intervals correspond to valid but unannotated events? A manual audit on a representative subset would help interpret the mixed ranking between SCHEMA and GPT-5.6 across All, GT+, and recall metrics.

5 comments
AuthorsOpenReview ↗

Response to Reviewer UmV5 [1/4]

Thank you for your positive assessment and constructive suggestions on component ablations and benchmark interpretation.

Response to Weakness 1 and Question 1: Components and inference budget

To clarify the roles of the individual components and their inference budgets, we have added targeted analyses on all 1,350 RTQ-Bench episodes in Appendix A.9:

ConfigurationBS-IoU All Macro (%)BS-IoU GT+ Macro (%)CC-Recall pooled (%)
Full SCHEMA50.5444.6441.55
Fixed uniform local scan39.8137.0525.67
Repeated global review38.4933.3024.80
Target-only local windows40.8431.8824.01
One adaptive local inspection34.9026.4916.98
One inspection, without refinement34.9126.4117.54
Without final boundary refinement (trace reconstruction)50.5044.5541.05

Recorded inference usage (million tokens).

Configuration / stageInputOutputTotal% of Full
Full SCHEMA108.935.43114.36100.00%
Fixed uniform local scan116.305.11121.42106.17%
Repeated global review125.546.85132.38115.76%
Target-only local windows97.295.43102.7389.83%
One adaptive local inspection51.382.6053.9847.20%
One inspection, without refinement29.902.0831.9927.97%
Without final boundary refinement (same traces)77.954.6882.6372.26%
Full SCHEMA: final refinement only30.980.7531.7327.74%

Counts are rounded independently; fixed-scan and repeated-review values retain their original reporting precision. The no-final-refinement cost is derived from the same Full SCHEMA traces by excluding final-refinement calls. The last row is a stage included in Full SCHEMA. All rows follow the recorded-call accounting over the same 1,350 episodes.

Adaptive inspection and token efficiency. The fixed-scan and repeated-review controls use Full SCHEMA's per-episode observation counts. Full SCHEMA achieves higher scores on all three localization metrics while using 114.36M tokens, compared with 121.42M and 132.38M for the two controls—approximately 5.82% and 13.61% fewer tokens, respectively. Target-only windows use 89.83% of Full's tokens; restoring surrounding local context improves All Macro BS-IoU, GT+ Macro BS-IoU, and pooled CC-Recall by 9.70, 12.76, and 17.54 percentage points, respectively.

Complementary roles of evidence acquisition and refinement. Contextual, iterative inspection identifies quality events and gathers supporting evidence; final boundary refinement provides finer temporal adjustment of those events. The result reconstructed before final refinement from the same traces reaches 50.50 / 44.55 / 41.05 on the three metrics with 82.63M tokens (72.26% of Full). It outperforms fixed scanning and repeated review while using approximately 31.95% and 37.58% fewer tokens, respectively. The final boundary-refinement stage accounts for 31.73M tokens (27.74%) and is dedicated to examining local action details and adjusting temporal endpoints.

Boundary refinement: finer temporal granularity. Boundary refinement is designed to resolve action details at a finer temporal scale. After contextual inspection identifies a quality event, denser local observations fine-tune its start and end times while preserving its category and semantic explanation (Appendix A.4). These endpoints determine which training frames receive quality-dependent sampling weights or conditioning labels (Appendix A.5). We retain this dedicated stage to meet SCHEMA's objective of producing detailed temporal quality annotations.

The paired comparison between Full SCHEMA and the result reconstructed before final refinement from the same 1,350 episode traces provides concrete evidence of these local adjustments (Appendix A.9):

  • BS-IoU: 388 episodes attain a higher mean BS-IoU across the optimal and suboptimal categories after refinement.
  • CC-Recall: 35 previously unmatched original GT segments satisfy the matching criteria after refinement, yielding a net increase of 18 matched original GT segments across all changes.

These paired results show the contribution of finer endpoint adjustment to temporal overlap and recovery of existing reference events. Appendix A.9 reports the complete paired breakdown.

AuthorsOpenReview ↗

Response to Reviewer UmV5 [2/4]

Response to Weakness 2: Annotation reliability

RTQ-Bench uses primary human annotation followed by a second human review of every episode. Human experts determine event selection, quality categories, and temporal boundaries (Appendix D).

To assess annotation reliability, we have completed independent verification with three additional human evaluators, complemented by GPT-6 Astra (Max) verification, across all 1,350 videos (Appendix D.1). These checks directly assess event selection, quality categories, and temporal boundaries. A separate uncovered-region audit examines additional task-significant events outside the reference intervals (Appendix D.2).

Human reference review (Appendix D.1). Each evaluator reviews the same 4,550 reference items, assessing event inclusion, quality category, temporal boundaries, and the appropriateness of empty category lists. Overall acceptance ranges from 95.65% to 97.67%; acceptance of annotated optimal and suboptimal intervals ranges from 93.49% to 99.70%.

Verification targetItemsHuman 1 accepted (%)Human 2 accepted (%)Human 3 accepted (%)
Annotated optimal intervals2,9732,964 (99.70%)2,873 (96.64%)2,901 (97.58%)
Annotated suboptimal intervals584565 (96.75%)578 (98.97%)546 (93.49%)
Empty optimal reference lists174142 (81.61%)91 (52.30%)138 (79.31%)
Empty suboptimal reference lists819773 (94.38%)810 (98.90%)807 (98.53%)
All verification items4,5504,444 (97.67%)4,352 (95.65%)4,392 (96.53%)

Acceptance rate = accepted reference items / items; overall rates are weighted by item count.

Complementary verification with GPT-6 Astra (Max). The model reviews the same reference items using video evidence, with the original annotation reasons withheld. Its results are reported separately below.

Verification targetItemsAcceptedAcceptance rate
Annotated optimal intervals2,9732,87696.74%
Annotated suboptimal intervals58443273.97%
Empty optimal reference lists1747945.40%
Empty suboptimal reference lists81976393.16%
All verification items4,5504,15091.21%

Response to Weakness 3 and Question 2: Sparse references and benchmark interpretation

Uncovered-region audit (Question 2)

We have completed an uncovered-region audit with three human evaluators and GPT-6 Astra (Max) to examine valid events beyond the sparse references (Appendix D.2). The audit examines video regions outside all retained optimal and suboptimal intervals, with annotation reasons and previous verification results withheld. It records clearly visible, task-significant events distinct from the retained annotations; boundary adjustments, ordinary preparatory actions, and minor pauses are excluded.

EvaluatorCategory with nonempty GTEligible videosVideos with detected omissionsDetection rate
GPT-6 Astra (Max)Optimal1,17627523.38%
GPT-6 Astra (Max)Suboptimal531112.07%
Human 1Optimal1,17619816.84%
Human 1Suboptimal53140.75%
Human 2Optimal1,17621618.37%
Human 2Suboptimal53191.69%
Human 3Optimal1,17614212.07%
Human 3Suboptimal531132.45%

For each category, eligible videos have nonempty GT. Detection rate is the percentage of eligible videos with at least one qualifying omitted event of that category.

Audit findings. The audits by the human evaluators and GPT-6 Astra (Max) consistently show that detected omissions are concentrated in optimal events and are limited to a minority of eligible videos; suboptimal omissions are rare. The three human evaluators detect additional optimal events in 12.07–18.37% of eligible videos, compared with only 0.75–2.45% for suboptimal events. GPT-6 Astra (Max) shows the same pattern, with rates of 23.38% and 2.07%, respectively. For every evaluator, more than three quarters of eligible optimal-category videos have no additional optimal event detected, and over 97.5% of eligible suboptimal-category videos have no additional suboptimal event detected.

Together with the 93.49–99.70% human acceptance of retained intervals, these findings provide complementary support for the reference annotations: the retained events receive strong human support, and additional detections predominantly concern optimal behavior, with suboptimal omissions detected in very few eligible videos. All reported method comparisons continue to use the same original reference annotations and unchanged benchmark scores.

AuthorsOpenReview ↗

Response to Reviewer UmV5 [3/4]

Response to Weakness 3: Empty-reference sensitivity and mixed rankings

Appendix B.6 reports both the original metrics and sensitivity to empty-reference handling (%):

MethodAll Macro BS-IoU (primary)All Macro BS-IoU (full credit)GT+ Macro BS-IoUCC-Recall pooled
SCHEMA50.5463.7644.6441.55
GPT-5.6 Sol34.3866.8347.9452.49

Full credit assigns score 1 to every empty-reference pair irrespective of predictions; it is a sensitivity analysis, not the primary metric. Among 819 episodes with empty suboptimal references, SCHEMA predicts events in 301 and GPT in 762. SCHEMA better matches reference sparsity, while GPT recovers more annotated events overall. The uncovered-region audit in Part 2 identifies optimal events as the main source of additional detections in its eligible videos, with suboptimal omissions detected in very few of them.

Response to Weakness 4: Progress-to-label decoding

Appendix C.2.4 evaluates an episode-wise top/bottom-30% progress-rate decoder using predictions from Robometer [1] and Robo-Dopamine 2.0 [2]. This decoder can select stagnation without negative progress. The comparison uses the original benchmark's BS-IoU metrics (%):

MethodDecoderAll MacroGT+ OptimalGT+ SuboptimalGT+ Macro
Robometer-4BThreshold31.9643.7420.6532.19
Robometer-4BRank28.8046.8142.7744.79
Robo-DopamineThreshold31.2747.9721.4834.73
Robo-DopamineRank28.2745.1543.7544.45

All columns report BS-IoU under the original evaluation protocol. All includes every episode; GT+ includes category-positive episodes.

The rank decoder improves GT+ overlap for both progress models, demonstrating how decoder choice affects interval localization. Interpreting low-progress regions as necessary pauses or ineffective repetition additionally requires contextual event assessment.

References

[1] Liang et al. Robometer: Scaling General-Purpose Robotic Reward Models via Trajectory Comparisons. Robotics: Science and Systems (RSS), 2026.

[2] Xu et al. Robo-Dopamine 2.0: History-Conditioned and OOD-Aware Process Reward Modeling for Robotic Manipulation. arXiv:2608.15680, 2026.

Reviewer_mqDZOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces an offline framework for constructing temporal quality metadata from heterogeneous robotic manipulation trajectories. A vision-language model operates within a stateful harness that supports global video inspection, targeted local inspection with temporal context, evidence accumulation, and boundary refinement. The resulting annotations identify sparse productive and suboptimal intervals, accompanied by explanations and evidence references. Uncovered regions remain unlabeled rather than being assigned a negative label.

The metadata is used through two interfaces: quality-dependent sampling during policy training and quality conditioning within the existing RECAP framework [1]. The paper also introduces RTQ-Bench for evaluating temporal quality localization. Experiments cover annotation quality, downstream transfer following VLA pretraining, and real-world manipulation with Qwen-PI and π₀.₅. The results demonstrate useful empirical improvements, particularly when learning from trajectories containing both successful and failed behavior.

Critical strengths

SCHEMA integrates contextual video inspection, evidence management, and temporal quality annotation into a reusable pipeline. Its metadata supports both sampling and RECAP-style conditioning [1], without requiring the annotation system during policy deployment.

Results on RTQ-Bench, RoboTwin 2.0, and real-world manipulation support the approach’s usefulness. The matched training comparisons strengthen the evidence that the metadata benefits policy learning, particularly with mixed-quality trajectories.

Critical weaknesses

The distinctive contribution remains insufficiently isolated. Related ideas already appear in transition-level curation with SCIZOR [2], rollout segmentation with HELP [3], and iterative video inspection with VideoAgent [4]. Although SCHEMA offers a useful integration, the lack of computation-matched comparisons and component ablations leaves unclear whether its additional complexity is necessary to achieve the reported benefits.

Benchmark comparisons and annotation reliability require stronger validation. Sparse references can penalize valid but unannotated predictions, while baseline decoding choices further affect comparability. Independent annotation agreement and better-aligned decoders would strengthen the conclusions. Recent ROBORMBENCH findings [5] also motivate checking sensitivity to equivalent task descriptions, which remains untested here.

[1] π*₀.₆: a VLA That Learns From Experience

[2] SCIZOR: A Self-Supervised Approach to Data Curation for Large-Scale Imitation Learning

[3] HELP: Human-Efficient Large-Scale Robot Post-Training with Rollout Segmentation

[4] VideoAgent: Long-form Video Understanding with Large Language Model as Agent

[5] Same Trajectory, Contradictory Rewards (ROBORMBENCH): Paraphrase Fragility in Vision Language Reward Models

Questions for authors

NA

5 comments
AuthorsOpenReview ↗

Response to Reviewer mqDZ [1/5]

Thank you for recognizing the value of SCHEMA's reusable metadata and for your suggestions on clarifying its methodological contribution and strengthening benchmark validation.

Response to Weakness 1: Methodological contribution

SCHEMA's technical contribution is a harness-guided procedure for constructing temporal quality annotations. The controller uses accumulated evidence to select further local observations, while the harness preserves findings and their evidence references across rounds. Contextual assessment identifies quality events, and denser inspection refines their endpoints. The resulting intervals specify which training frames receive quality-dependent sampling weights or conditioning labels.

The comparison below distinguishes how quality decisions are produced from their temporal output and downstream use (Sections 2–3; Appendices A.3–A.5).

MethodHow quality decisions are producedTemporal output and downstream use
SCIZOR [1]Self-supervised progress prediction and state–action deduplicationRetained transitions for imitation learning
HELP (VLAC-CUT) [2]A process-supervised critic uses trajectory context to estimate signed progress, diagnose local changes, and predict cut pointsLabeled rollout segments; progress-making and recovery segments retained for post-training
SCHEMAAdaptive evidence acquisition, accumulated findings, contextual event assessment, and dedicated boundary refinementSparse intervals with categories, explanations, and evidence references; reused for sampling and conditioning

VideoAgent [3] provides a precedent for iterative visual evidence acquisition. SCHEMA organizes this principle around robot-quality event assessment and boundary refinement, producing temporal supervision for policy learning.

Appendix A.9 connects these design choices to targeted comparisons: fixed scanning and repeated global review test the inspection strategy; target-only windows test surrounding temporal context; reconstruction from the same traces tests final boundary refinement. Across 1,350 episodes, refinement improves mean BS-IoU in 388 episodes and newly matches 35 original GT segments, with a net increase of 18 matches. Part 2 reports the component scores and token usage; Part 3 presents the downstream policy-learning comparisons.

References

[1] Zhang et al. SCIZOR: A Self-Supervised Approach to Data Curation for Large-Scale Imitation Learning. ICRA, 2026.

[2] Zhai et al. HELP: Human-Efficient Large-Scale Robot Post-Training with Rollout Segmentation. arXiv:2607.09776, 2026.

[3] Wang et al. VideoAgent: Long-form Video Understanding with Large Language Model as Agent. ECCV, 2024.

AuthorsOpenReview ↗

Response to Reviewer mqDZ [2/5]

Response to Weakness 1: Component evidence and inference budget

Appendix A.9 evaluates SCHEMA's contextual evidence acquisition and boundary refinement through the following interventions on all 1,350 RTQ-Bench episodes.

Component results (%):

ConfigurationBS-IoU All Macro ↑BS-IoU GT+ Macro ↑CC-Recall pooled ↑
Full SCHEMA50.5444.6441.55
Fixed uniform local scan39.8137.0525.67
Repeated global review38.4933.3024.80
Target-only local windows40.8431.8824.01
One adaptive local inspection34.9026.4916.98
One inspection, without refinement34.9126.4117.54
Without final boundary refinement (trace reconstruction)50.5044.5541.05

Recorded inference usage (million tokens).

Configuration / stageInputOutputTotal% of Full
Full SCHEMA108.935.43114.36100.00%
Fixed uniform local scan116.305.11121.42106.17%
Repeated global review125.546.85132.38115.76%
Target-only local windows97.295.43102.7389.83%
One adaptive local inspection51.382.6053.9847.20%
One inspection, without refinement29.902.0831.9927.97%
Without final boundary refinement (same traces)77.954.6882.6372.26%
Full SCHEMA: final refinement only30.980.7531.7327.74%

Counts are rounded independently; fixed-scan and repeated-review values retain their original reporting precision. The no-final-refinement cost is derived from the same Full SCHEMA traces by excluding final-refinement calls. The last row is a stage included in Full SCHEMA. All rows follow the recorded-call accounting over the same 1,350 episodes.

Adaptive inspection and token efficiency. The fixed-scan and repeated-review controls use Full SCHEMA's per-episode observation counts. Full SCHEMA achieves higher scores on all three localization metrics while using 114.36M tokens, compared with 121.42M and 132.38M for the two controls—approximately 5.82% and 13.61% fewer tokens, respectively. Target-only windows use 89.83% of Full's tokens; restoring surrounding local context improves All Macro BS-IoU, GT+ Macro BS-IoU, and pooled CC-Recall by 9.70, 12.76, and 17.54 percentage points, respectively.

Complementary roles of evidence acquisition and refinement. Contextual, iterative inspection identifies quality events and gathers supporting evidence; final boundary refinement provides finer temporal adjustment of those events. The result reconstructed before final refinement from the same traces reaches 50.50 / 44.55 / 41.05 on the three metrics with 82.63M tokens (72.26% of Full). It outperforms fixed scanning and repeated review while using approximately 31.95% and 37.58% fewer tokens, respectively. The final boundary-refinement stage accounts for 31.73M tokens (27.74%) and is dedicated to examining local action details and adjusting temporal endpoints.

Boundary refinement: finer temporal granularity. Boundary refinement is designed to resolve action details at a finer temporal scale. After contextual inspection identifies a quality event, denser local observations fine-tune its start and end times while preserving its category and semantic explanation (Appendix A.4). These endpoints determine which training frames receive quality-dependent sampling weights or conditioning labels (Appendix A.5). We retain this dedicated stage to meet SCHEMA's objective of producing detailed temporal quality annotations.

The paired comparison between Full SCHEMA and the result reconstructed before final refinement from the same 1,350 episode traces provides concrete evidence of these local adjustments (Appendix A.9):

  • BS-IoU: 388 episodes attain a higher mean BS-IoU across the optimal and suboptimal categories after refinement.
  • CC-Recall: 35 previously unmatched original GT segments satisfy the matching criteria after refinement, yielding a net increase of 18 matched original GT segments across all changes.

These paired results show the contribution of finer endpoint adjustment to temporal overlap and recovery of existing reference events. Appendix A.9 reports the complete paired breakdown.

AuthorsOpenReview ↗

Response to Reviewer mqDZ [3/5]

Response to Weakness 1: Downstream policy-learning evidence

The updated Figure 4 and Appendices F.2–F.3 provide complementary downstream comparisons with Robometer [4] and RECAP [5].

Real-world success rates (%):

Policy and trainingSFT dataPickPlaceFastenTwoLongCutInsertFourAverage
Qwen-PI: Uniform SamplingSuc684048841
Qwen-PI: Uniform SamplingSuc+Fail684452442
Qwen-PI: Robometer-Guided SamplingSuc684048440
Qwen-PI: Robometer-Guided SamplingSuc+Fail724448843
Qwen-PI: SCHEMA-Guided SamplingSuc724044841
Qwen-PI: SCHEMA-Guided SamplingSuc+Fail765256848
π₀.₅: Advantage Conditioning (RECAP)Suc7232481241
π₀.₅: Advantage Conditioning (RECAP)Suc+Fail6032481639
π₀.₅: Advantage Conditioning (Robometer)Suc7636401642
π₀.₅: Advantage Conditioning (Robometer)Suc+Fail6840441642
π₀.₅: Advantage Conditioning (SCHEMA)Suc7636441242
π₀.₅: Advantage Conditioning (SCHEMA)Suc+Fail8448522051

Each task success rate is based on 25 trials per method and SFT data regime. Suc denotes successful-only training; Suc+Fail additionally includes failed rollouts. Average equally weights the four tasks. Bold marks the best result within each policy family and data regime, including ties.

All real-world Qwen-PI variants start from the same SCHEMA-guided pretrained checkpoint. Within each policy family and data regime, all methods use identical policy-training trajectories, batch sizes, and update budgets. On the shared Suc+Fail pools, SCHEMA improves mean success by 6 / 12 percentage points over Uniform / RECAP and 5 / 9 points over Robometer for sampling / conditioning. Its larger gains when all methods receive the same failed rollouts support more effective use of mixed-quality experience, including failed trajectories.

Robometer-guided pretraining

We have completed Robometer-guided Qwen-PI pretraining and added the downstream comparison to Section 4.3 and Figure 3. All three methods share initialization, corpora, dataset/task mixture, optimization schedules, and downstream SFT/evaluation settings; Robometer uses the continuous progress-weighted sampling adapter in Appendix F.2.1. Appendix E reports the per-task results for all 50 RoboTwin 2.0 tasks. The table below summarizes mean success under Clean and Randomized evaluation at each SFT budget.

RoboTwin 2.0 mean success rates (%).

SFT stepsEvaluationUniform pretrainingRobometer-guided pretrainingSCHEMA-guided pretrainingSCHEMA − Robometer (pp)
2.5KClean27.3631.8033.56+1.76
2.5KRandomized27.8831.0431.36+0.32
10KClean47.7248.9249.36+0.44
10KRandomized48.0447.8848.04+0.16
25KClean65.4467.4469.04+1.60
25KRandomized63.6467.5668.92+1.36

Each mean equally weights all 50 tasks. Bold marks the highest success rate in each row, including ties; pp denotes percentage points.

SCHEMA achieves higher mean success than Robometer-guided pretraining at all three SFT budgets under both evaluation conditions. At 25K steps, SCHEMA reaches 69.04% / 68.92%, exceeding Robometer by 1.60 / 1.36 percentage points and Uniform by 3.60 / 5.28 points, respectively. These results extend the evidence for SCHEMA's policy-learning utility from real-world fine-tuning to large-scale pretraining.

References

[4] Liang et al. Robometer: Scaling General-Purpose Robotic Reward Models via Trajectory Comparisons. Robotics: Science and Systems (RSS), 2026.

[5] Physical Intelligence et al. π*₀.₆: A VLA That Learns From Experience. arXiv:2511.14759, 2025.

Reviewer_xDqbOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces SCHEMA, an agentic-style framework for sparse temporal quality annotation of robot trajectories. It identifies optimal and suboptimal segments through harness-guided evidence seeking, contextual quality assessment, and boundary refinement. The authors also introduce RTQ-Bench, a human-annotated benchmark for evaluating temporal quality localization. For downstream policy learning, SCHEMA-derived annotations are used to guide data sampling and to replace value-based advantage labels in RECAP-style advantage-conditioned training. Experiments show improvements over uniform sampling and RECAP-based conditioning in several settings, particularly when training with mixed-quality trajectories.

Critical strengths
  1. The paper addresses an important problem in robot learning by introducing a framework for identifying sparse, segment-level quality information from robot trajectories. The proposed RTQ-Bench also provides a useful benchmark for evaluating temporal quality annotations which may benefit further experiment.
  2. The paper evaluates the practical utility of the generated annotations in both simulation and real-world robot learning, covering quality-guided sampling and advantage-conditioned training. The experiments show promising improvements, particularly when learning from mixed-quality trajectories.
Critical weaknesses
  1. Limited comparisons with existing baselines. The downstream experiments mainly compare against uniform sampling and RECAP-based conditioning. Comparisons with simpler quality-aware sampling strategies and related data curation methods are missing. Moreover, the improvements in real-world experiments are more evident when successful and failed trajectories are combined, while the gains with successful-only data are limited. Since adding failed trajectories also increases the training data size, additional data-size-matched comparisons would help clarify the benefits of incorporating failed experience.
  2. Computational overhead and insufficient justification of the agentic design. SCHEMA requires substantially more tokens and computational resources than single-pass prediction. However, the paper lacks sufficient ablation studies to demonstrate the individual contributions of its agentic components, such as iterative evidence seeking, contextual assessment, and boundary refinement. It remains unclear whether these components are necessary or whether similar improvements could be achieved with simpler approaches under comparable computational budgets.
Questions for authors

Q1. Could the authors provide comparisons with other data selection methods discussed in the Related Work, or include a simpler baseline, such as sampling based on episode-level success or failure? This would help clarify whether the improvements mainly come from SCHEMA's segment-level quality annotations. Q2. The improvements over RECAP vary across the four real-world tasks. Could the authors clarify whether the RECAP baseline was sufficiently tuned, particularly regarding the value function and advantage conditioning? Also, since adding failed trajectories increases the training data size, could the authors provide a data-size-matched comparison to better isolate the benefits of failed experience?

Q3. Could the authors provide more detailed ablations of the agentic components, particularly iterative evidence seeking and boundary refinement? Given the additional computational overhead, how does SCHEMA compare with simpler multi-pass approaches under similar inference budgets?

4 comments
AuthorsOpenReview ↗

Response to Reviewer xDqb [1/4]

Thank you for your constructive suggestions on baseline comparisons, learning from failed experience, and the agentic design.

Response to Weakness 1 and Question 1: Additional quality-aware baselines

Following your suggestion, we have added comparisons using Robometer [1] to guide sampling and advantage conditioning on the same task-specific SFT pools. The RECAP baseline uses the advantage-conditioning formulation of π*₀.₆ [2]. Appendix F.2 specifies the adapters; the updated Figure 4 and Appendix F.3 report the following complete results.

Real-world success rates (%).

Policy and trainingSFT dataPickPlaceFastenTwoLongCutInsertFourAverage
Qwen-PI: Uniform SamplingSuc684048841
Qwen-PI: Uniform SamplingSuc+Fail684452442
Qwen-PI: Robometer-Guided SamplingSuc684048440
Qwen-PI: Robometer-Guided SamplingSuc+Fail724448843
Qwen-PI: SCHEMA-Guided SamplingSuc724044841
Qwen-PI: SCHEMA-Guided SamplingSuc+Fail765256848
π₀.₅: Advantage Conditioning (RECAP)Suc7232481241
π₀.₅: Advantage Conditioning (RECAP)Suc+Fail6032481639
π₀.₅: Advantage Conditioning (Robometer)Suc7636401642
π₀.₅: Advantage Conditioning (Robometer)Suc+Fail6840441642
π₀.₅: Advantage Conditioning (SCHEMA)Suc7636441242
π₀.₅: Advantage Conditioning (SCHEMA)Suc+Fail8448522051

Each task success rate is based on 25 trials per method and SFT data regime. Suc denotes successful-only data; Suc+Fail additionally includes failed rollouts. Average equally weights the four tasks. Bold marks the best result within each policy family and data regime, including ties.

Episode-level filtering and segment-level guidance. Qwen-PI with Uniform sampling + Suc is the episode-level success-filtering baseline: it trains uniformly on successful trajectories. Within each fixed SFT pool (Suc or Suc+Fail), Uniform, Robometer, and SCHEMA sampling use identical trajectories and update budgets, isolating the effect of segment-level guidance (Appendices F.2–F.3). We also compare the methods on mixed-quality pools subsampled to the original Suc episode counts; the protocol and results are reported in Part 4 and in Appendix F.4.

Completion time in seconds (speedup).

Policy and trainingPickPlaceFastenTwoLongCutInsertFourAverage
Qwen-PI: Uniform Sampling25.58 (1.00×)33.85 (1.00×)98.77 (1.00×)210.11 (1.00×)92.08 (1.00×)
Qwen-PI: Robometer-Guided Sampling24.84 (1.03×)32.48 (1.04×)79.63 (1.24×)203.99 (1.03×)85.24 (1.08×)
Qwen-PI: SCHEMA-Guided Sampling24.36 (1.05×)31.34 (1.08×)65.48 (1.51×)194.55 (1.08×)78.93 (1.17×)
π₀.₅: Advantage Conditioning (RECAP)30.25 (1.00×)36.31 (1.00×)95.13 (1.00×)160.30 (1.00×)80.50 (1.00×)
π₀.₅: Advantage Conditioning (Robometer)29.52 (1.02×)30.63 (1.19×)80.27 (1.19×)155.41 (1.03×)73.96 (1.09×)
π₀.₅: Advantage Conditioning (SCHEMA)28.01 (1.08×)33.62 (1.08×)73.83 (1.29×)152.67 (1.05×)72.03 (1.12×)

All policies use Suc+Fail training. Timing evaluations are separate from the 25-trial success-rate evaluations and continue until 10 successful rollouts are collected per method and task; each task time averages these 10 rollouts. Speedup is relative to Uniform for Qwen-PI and RECAP for π₀.₅; the Average speedup is the ratio of four-task mean times. Bold marks the shortest time within each policy family.

All real-world Qwen-PI variants start from the same SCHEMA-guided pretrained checkpoint. Within each policy family and data regime, all methods use identical policy-training trajectories, batch sizes, and update budgets. On the shared Suc+Fail pools, SCHEMA improves mean success by 6 / 12 percentage points over Uniform / RECAP and 5 / 9 points over Robometer for sampling / conditioning, with lower mean successful completion times. Its larger gains when all methods receive the same failed rollouts support more effective use of mixed-quality experience, including failed trajectories.

References

[1] Liang et al. Robometer: Scaling General-Purpose Robotic Reward Models via Trajectory Comparisons. Robotics: Science and Systems (RSS), 2026.

[2] Physical Intelligence et al. π*₀.₆: A VLA That Learns From Experience. arXiv:2511.14759, 2025.

AuthorsOpenReview ↗

Response to Reviewer xDqb [2/4]

Response to Weakness 1 and Question 1: Robometer-guided pretraining

We have completed Qwen-PI pretraining guided by Robometer [1] and added the downstream comparison to Section 4.3 and Figure 3. All three methods share initialization, corpora, dataset/task mixture, optimization schedules, and downstream SFT/evaluation settings; Robometer uses the continuous progress-weighted sampling adapter in Appendix F.2.1. Appendix E reports the per-task results for all 50 RoboTwin 2.0 tasks. The table below summarizes mean success under Clean and Randomized evaluation at each SFT budget.

RoboTwin 2.0 mean success rates (%).

SFT stepsEvaluationUniform pretrainingRobometer-guided pretrainingSCHEMA-guided pretrainingSCHEMA − Robometer (pp)
2.5KClean27.3631.8033.56+1.76
2.5KRandomized27.8831.0431.36+0.32
10KClean47.7248.9249.36+0.44
10KRandomized48.0447.8848.04+0.16
25KClean65.4467.4469.04+1.60
25KRandomized63.6467.5668.92+1.36

Each mean equally weights all 50 tasks. Bold marks the highest success rate in each row, including ties; pp denotes percentage points.

SCHEMA achieves higher mean success than Robometer-guided pretraining at all three SFT budgets under both evaluation conditions. At 25K steps, SCHEMA reaches 69.04% / 68.92%, exceeding Robometer by 1.60 / 1.36 percentage points and Uniform by 3.60 / 5.28 points, respectively. These results extend the evidence for SCHEMA's policy-learning utility from real-world fine-tuning to large-scale pretraining.

Response to Question 2: RECAP training and conditioning

RECAP-based data collection and fair comparison. Every rollout iteration follows RECAP [2]: the value function is fitted on accumulated task data, its advantage estimates condition policy training, and the resulting policy collects the next rollouts (Appendix F). The RECAP baseline therefore learns from rollouts collected through its own value-guided learning loop. The final trajectory pool is frozen and shared across methods; comparisons within each policy family and data regime use identical policy-training data and budgets.

RECAP initialization. Appendix C.1.1 now specifies the value-model pretraining: 367,960 updates on Open X-Embodiment and AgiBot World using 24 GPUs and global batch 1,536, followed by 75,000 updates on RoboCOIN using 8 GPUs and global batch 512. Both stages freeze SigLIP2 and train Gemma, the visual projector, and the 201-bin value head with two-hot cross-entropy. Both use AdamW at a fixed learning rate of 5e-5; the remaining architecture and optimization settings are detailed in the appendix.

Task-specific value fitting. Appendix F.1 specifies the final value-model training on each task's full Suc+Fail pool, initialized from the pretrained value checkpoint. Both the visual and language backbones are updated. All four models were trained to convergence using 4 GPUs, 32 examples per GPU, and global batch 128, with the same AdamW settings as pretraining:

TaskValue-model updates
PickPlace1,200
FastenTwo1,200
LongCut2,900
InsertFour1,500

Policy conditioning and parameter choice. The RECAP baseline follows the advantage-conditioning formulation and classifier-free-guidance (CFG) procedure of π*₀.₆ [2]. The RECAP, Robometer, and SCHEMA π₀.₅ variants share the same policy architecture and use a fixed CFG weight of 2.0 across all tasks and data regimes, within the moderate guidance range described in the original paper’s Appendix A-E; RECAP supplies value-derived advantage conditions, while SCHEMA supplies metadata-derived quality conditions. Appendix F.1 now makes this parameter provenance explicit. The real-world value models use a separate branch from the RTQ-Bench-specific 80K-step adaptation and frame-selection decoder. The value-model fitting budgets above are separate from the matched policy-training budgets.

References

[1] Liang et al. Robometer: Scaling General-Purpose Robotic Reward Models via Trajectory Comparisons. Robotics: Science and Systems (RSS), 2026.

[2] Physical Intelligence et al. π*₀.₆: A VLA That Learns From Experience. arXiv:2511.14759, 2025.

AuthorsOpenReview ↗

Response to Reviewer xDqb [3/4]

Response to Weakness 2 and Question 3: Component contributions and computational cost

To clarify each component's role and computational cost, we report the component and token-usage results below, with intervention details in Appendix A.9.

Component analysis on all 1,350 RTQ-Bench episodes (%).

ConfigurationBS-IoU All Macro ↑BS-IoU GT+ Macro ↑CC-Recall pooled ↑
Full SCHEMA50.5444.6441.55
Fixed uniform local scan39.8137.0525.67
Repeated global review38.4933.3024.80
Target-only local windows40.8431.8824.01
One adaptive local inspection34.9026.4916.98
One inspection, without refinement34.9126.4117.54
Without final boundary refinement (trace reconstruction)50.5044.5541.05

Recorded inference usage (million tokens).

Configuration / stageInputOutputTotal% of Full
Full SCHEMA108.935.43114.36100.00%
Fixed uniform local scan116.305.11121.42106.17%
Repeated global review125.546.85132.38115.76%
Target-only local windows97.295.43102.7389.83%
One adaptive local inspection51.382.6053.9847.20%
One inspection, without refinement29.902.0831.9927.97%
Without final boundary refinement (same traces)77.954.6882.6372.26%
Full SCHEMA: final refinement only30.980.7531.7327.74%

Counts are rounded independently; fixed-scan and repeated-review values retain their original reporting precision. The no-final-refinement cost is derived from the same Full SCHEMA traces by excluding final-refinement calls. The last row is a stage included in Full SCHEMA. All rows follow the recorded-call accounting over the same 1,350 episodes.

Adaptive inspection and token efficiency. The fixed-scan and repeated-review controls use Full SCHEMA's per-episode observation counts. Full SCHEMA achieves higher scores on all three localization metrics while using 114.36M tokens, compared with 121.42M and 132.38M for the two controls—approximately 5.82% and 13.61% fewer tokens, respectively. Target-only windows use 89.83% of Full's tokens; restoring surrounding local context improves All Macro BS-IoU, GT+ Macro BS-IoU, and pooled CC-Recall by 9.70, 12.76, and 17.54 percentage points, respectively.

Complementary roles of evidence acquisition and refinement. Contextual, iterative inspection identifies quality events and gathers supporting evidence; final boundary refinement provides finer temporal adjustment of those events. The result reconstructed before final refinement from the same traces reaches 50.50 / 44.55 / 41.05 on the three metrics with 82.63M tokens (72.26% of Full). It outperforms fixed scanning and repeated review while using approximately 31.95% and 37.58% fewer tokens, respectively. The final boundary-refinement stage accounts for 31.73M tokens (27.74%) and is dedicated to examining local action details and adjusting temporal endpoints.

Boundary refinement: finer temporal granularity. Boundary refinement is designed to resolve action details at a finer temporal scale. After contextual inspection identifies a quality event, denser local observations fine-tune its start and end times while preserving its category and semantic explanation (Appendix A.4). These endpoints determine which training frames receive quality-dependent sampling weights or conditioning labels (Appendix A.5). We retain this dedicated stage to meet SCHEMA's objective of producing detailed temporal quality annotations.

The paired comparison between Full SCHEMA and the result reconstructed before final refinement from the same 1,350 episode traces provides concrete evidence of these local adjustments (Appendix A.9):

  • BS-IoU: 388 episodes attain a higher mean BS-IoU across the optimal and suboptimal categories after refinement.
  • CC-Recall: 35 previously unmatched original GT segments satisfy the matching criteria after refinement, yielding a net increase of 18 matched original GT segments across all changes.

These paired results show the contribution of finer endpoint adjustment to temporal overlap and recovery of existing reference events. Appendix A.9 reports the complete paired breakdown.

AuthorsOpenReview ↗

Response to Reviewer xDqb [4/4]

Response to Weakness 1 and Question 2: Matched-data comparison

Matched episode counts (Weakness 1 and Question 2)

To address the data-size question, we added a comparison in which the mixed-quality pool has the same number of episodes as the original Suc pool for each task (Appendix F.4). Starting from the full Suc+Fail pools in Table 19, we randomly subsample episodes within the three source groups, preserving their proportions up to integer rounding. The selected episodes are shared by all six methods for policy training. RECAP [2] reuses the task-specific value checkpoint fitted on the full Suc+Fail pool (Appendix F.1) to generate advantage conditions. Episode matching applies to the policy-training subsets, with this value checkpoint held fixed. Each method retains its original initialization, batch size, training steps, and conditioning/CFG settings, with 25 evaluation trials per task.

Episode counts for the matched comparison.

TaskTeleoperation SuccessfulRollout (+HITL) SuccessfulRollout (+HITL) FailedSuccessful + Failed (matched)
PickPlace20 → 149 → 711 → 829
FastenTwo20 → 1613 → 117 → 633
LongCut117 → 87283 → 211136 → 102400
InsertFour20 → 1723 → 207 → 643

Source categories follow Table 19. The three source columns report original → retained episode counts; Rollout (+HITL) includes human-in-the-loop interventions. The final column gives the matched Successful + Failed total, equal to the original Successful Only count for each task.

Success rates (%) on the episode-matched Suc+Fail pools.

Policy and training (episode-matched Suc+Fail)PickPlaceFastenTwoLongCutInsertFourAverage
Qwen-PI: Uniform Sampling64 (−4)28 (−12)48 (0)4 (−4)36 (−5)
Qwen-PI: Robometer-Guided Sampling60 (−8)40 (0)40 (−8)8 (+4)37 (−3)
Qwen-PI: SCHEMA-Guided Sampling76 (+4)32 (−8)44 (0)8 (0)40 (−1)
π₀.₅: Advantage Conditioning (RECAP)68 (−4)36 (+4)36 (−12)8 (−4)37 (−4)
π₀.₅: Advantage Conditioning (Robometer)72 (−4)40 (+4)40 (0)4 (−12)39 (−3)
π₀.₅: Advantage Conditioning (SCHEMA)76 (0)32 (−4)48 (+4)8 (−4)41 (−1)

Each parenthesized value is the change in percentage points relative to the same method's original Suc result in Appendix F.3. Average equally weights the four tasks. Bold marks the highest success rate within each policy family, including ties.

With episode counts fixed, SCHEMA achieves the highest mean success in both policy families: 40% (−1 pp vs. Suc) for sampling and 41% (−1 pp vs. Suc) for conditioning. On these shared mixed-quality pools, it exceeds Uniform / RECAP by 4 / 4 percentage points and Robometer [1] by 3 / 2 points, respectively. This controlled comparison supports SCHEMA's use of mixed-quality experience under the same episode budget. Tables 24–25 in Appendix F.4 provide the sampling composition and complete results.

Utilizing the full mixed-quality pools. In the full-pool comparison, all methods receive the same additional failed rollouts, while policy-training batch sizes and update budgets remain unchanged within each policy family. Moving from Suc to Suc+Fail raises SCHEMA's mean success by 7 / 9 percentage points for sampling / conditioning, compared with +1 / −2 for Uniform / RECAP and +3 / 0 for Robometer. On the shared Suc+Fail pools, SCHEMA reaches 48% / 51%, versus 42% / 39% and 43% / 42%, respectively. These matched comparisons support SCHEMA's ability to make better use of mixed-quality experience, including failed rollouts. Appendix F.3 reports the comparison.

References

[1] Liang et al. Robometer: Scaling General-Purpose Robotic Reward Models via Trajectory Comparisons. Robotics: Science and Systems (RSS), 2026.

[2] Physical Intelligence et al. π*₀.₆: A VLA That Learns From Experience. arXiv:2511.14759, 2025.

Reviewer_SaUDOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces SCHEMA, a harness-guided framework for temporal quality metadata construction. The motivation mainly lies in the conventional approach which evenly samples from a full robot trajectory, whereas there is no guarantee that every step in the trajectory is optimal, i.e. there may exist suboptimal actions that may lead to worse states but recovered later in the trajectory. SCHEMA consists of a harness that takes in trajectories and outputs intervals labeled optimal or suboptimal. This enables advantage-weighted training loss or weighted sampling. This was further accompanied by RTQ-Bench, which evaluates localization accuracy and downstream learning utility from expert-annotated trajectories. Results include improved performance on RTQ-Bench and better VLA performance with sampling with SCHEMA-labeled data.

Critical strengths
  1. Experiments setups are reasonable and well-executed. I like the experiments of this paper best, which clearly showcases the strength of this paper. This was done through mainly three parts, improved localization performance, improved training of Qwen-PI based on SCHEMA metadata sampling, and improved RECAP-style improved through advantage-conditioned regression.
Critical weaknesses
  1. For downstream performance, the experiments mainly cover SCHEMA versus uniform sampling, which does not clearly show the contribution of this paper, using an agentic pipeline for progress labeling rather than progress models. I would like to better understand the performance of this harness-based labeling system versus existing progress reward models, such as RoboMeter [1].

[1] Liang et al. Robometer: Scaling General-Purpose Robotic Reward Models via Trajectory Comparisons, 2026.

Questions for authors
  1. Following Weakness 1, I would like to see Robometer or other progress value models used to label data, followed by Qwen-PI pretraining with progress-weighted sampling.

  2. One thing that confuses me is that within-trajectory level data filtering is not a new thing, as shown in [1] and [2]. Could the authors explicitly compare the SCHEMA approach against existing works in this domain, is it more of a empirical result improvement or is there some technical contribution?

[1] Zhang et al., SCIZOR: A Self-Supervised Approach to Data Curation for Large-Scale Imitation Learning, 2025.

[2] Zhai et al., HELP: Human-Efficient Large-Scale Robot Post-Training with Rollout Segmentation, 2026.

3 comments
AuthorsOpenReview ↗

Response to Reviewer SaUD [1/3]

Thank you for your positive assessment of the experiments and your suggestion to include stronger progress-model comparisons.

Response to Weakness 1 and Question 1: Comparisons with Robometer

Real-world fine-tuning (Weakness 1)

Following your suggestion, we have added comparisons using Robometer [1] to guide sampling and advantage conditioning on the same task-specific SFT pools. The RECAP baseline uses the advantage-conditioning formulation of π*₀.₆ [2]. Appendix F.2 specifies the adapters; the updated Figure 4 and Appendix F.3 report the following complete results.

Real-world success rates (%).

Policy and trainingSFT dataPickPlaceFastenTwoLongCutInsertFourAverage
Qwen-PI: Uniform SamplingSuc684048841
Qwen-PI: Uniform SamplingSuc+Fail684452442
Qwen-PI: Robometer-Guided SamplingSuc684048440
Qwen-PI: Robometer-Guided SamplingSuc+Fail724448843
Qwen-PI: SCHEMA-Guided SamplingSuc724044841
Qwen-PI: SCHEMA-Guided SamplingSuc+Fail765256848
π₀.₅: Advantage Conditioning (RECAP)Suc7232481241
π₀.₅: Advantage Conditioning (RECAP)Suc+Fail6032481639
π₀.₅: Advantage Conditioning (Robometer)Suc7636401642
π₀.₅: Advantage Conditioning (Robometer)Suc+Fail6840441642
π₀.₅: Advantage Conditioning (SCHEMA)Suc7636441242
π₀.₅: Advantage Conditioning (SCHEMA)Suc+Fail8448522051

Each task success rate is based on 25 trials per method and SFT data regime. Suc denotes successful-only data; Suc+Fail additionally includes failed rollouts. Average equally weights the four tasks. Bold marks the best result within each policy family and data regime, including ties.

Completion time in seconds (speedup).

Policy and trainingPickPlaceFastenTwoLongCutInsertFourAverage
Qwen-PI: Uniform Sampling25.58 (1.00×)33.85 (1.00×)98.77 (1.00×)210.11 (1.00×)92.08 (1.00×)
Qwen-PI: Robometer-Guided Sampling24.84 (1.03×)32.48 (1.04×)79.63 (1.24×)203.99 (1.03×)85.24 (1.08×)
Qwen-PI: SCHEMA-Guided Sampling24.36 (1.05×)31.34 (1.08×)65.48 (1.51×)194.55 (1.08×)78.93 (1.17×)
π₀.₅: Advantage Conditioning (RECAP)30.25 (1.00×)36.31 (1.00×)95.13 (1.00×)160.30 (1.00×)80.50 (1.00×)
π₀.₅: Advantage Conditioning (Robometer)29.52 (1.02×)30.63 (1.19×)80.27 (1.19×)155.41 (1.03×)73.96 (1.09×)
π₀.₅: Advantage Conditioning (SCHEMA)28.01 (1.08×)33.62 (1.08×)73.83 (1.29×)152.67 (1.05×)72.03 (1.12×)

All policies use Suc+Fail training. Timing evaluations are separate from the 25-trial success-rate evaluations and continue until 10 successful rollouts are collected per method and task; each task time averages these 10 rollouts. Speedup is relative to Uniform for Qwen-PI and RECAP for π₀.₅; the Average speedup is the ratio of four-task mean times. Bold marks the shortest time within each policy family.

All real-world Qwen-PI variants start from the same SCHEMA-guided pretrained checkpoint. Within each policy family and data regime, all methods use identical policy-training trajectories, batch sizes, and update budgets. On the shared Suc+Fail pools, SCHEMA improves mean success by 6 / 12 percentage points over Uniform / RECAP and 5 / 9 points over Robometer for sampling / conditioning, with lower mean successful completion times. Its larger gains when all methods receive the same failed rollouts support more effective use of mixed-quality experience, including failed trajectories.

References

[1] Liang et al. Robometer: Scaling General-Purpose Robotic Reward Models via Trajectory Comparisons. Robotics: Science and Systems (RSS), 2026.

[2] Physical Intelligence et al. π*₀.₆: A VLA That Learns From Experience. arXiv:2511.14759, 2025.

AuthorsOpenReview ↗

Response to Reviewer SaUD [2/3]

Response to Question 1: Robometer-guided Qwen-PI pretraining

We have completed Qwen-PI pretraining guided by Robometer [1] and added the downstream comparison to Section 4.3 and Figure 3. All three methods share initialization, corpora, dataset/task mixture, optimization schedules, and downstream SFT/evaluation settings; Robometer uses the continuous progress-weighted sampling adapter in Appendix F.2.1. Appendix E reports the per-task results for all 50 RoboTwin 2.0 tasks. The table below summarizes mean success under Clean and Randomized evaluation at each SFT budget.

RoboTwin 2.0 mean success rates (%).

SFT stepsEvaluationUniform pretrainingRobometer-guided pretrainingSCHEMA-guided pretrainingSCHEMA − Robometer (pp)
2.5KClean27.3631.8033.56+1.76
2.5KRandomized27.8831.0431.36+0.32
10KClean47.7248.9249.36+0.44
10KRandomized48.0447.8848.04+0.16
25KClean65.4467.4469.04+1.60
25KRandomized63.6467.5668.92+1.36

Each mean equally weights all 50 tasks. Bold marks the highest success rate in each row, including ties; pp denotes percentage points.

SCHEMA achieves higher mean success than Robometer-guided pretraining at all three SFT budgets under both evaluation conditions. At 25K steps, SCHEMA reaches 69.04% / 68.92%, exceeding Robometer by 1.60 / 1.36 percentage points and Uniform by 3.60 / 5.28 points, respectively. These results extend the evidence for SCHEMA's policy-learning utility from real-world fine-tuning to large-scale pretraining.

Response to Question 2: Methodological contribution

SCHEMA's technical contribution is a harness-guided procedure for constructing temporal quality annotations. The controller uses accumulated evidence to select further local observations, while the harness preserves findings and their evidence references across rounds. Contextual assessment identifies quality events, and denser inspection refines their endpoints. The resulting intervals specify which training frames receive quality-dependent sampling weights or conditioning labels.

The comparison below distinguishes how quality decisions are produced from their temporal output and downstream use (Sections 2–3; Appendices A.3–A.5).

MethodHow quality decisions are producedTemporal output and downstream use
SCIZOR [3]Self-supervised progress prediction and state–action deduplicationRetained transitions for imitation learning
HELP (VLAC-CUT) [4]A process-supervised critic uses trajectory context to estimate signed progress, diagnose local changes, and predict cut pointsLabeled rollout segments; progress-making and recovery segments retained for post-training
SCHEMAAdaptive evidence acquisition, accumulated findings, contextual event assessment, and dedicated boundary refinementSparse intervals with categories, explanations, and evidence references; reused for sampling and conditioning

VideoAgent [5] provides a precedent for iterative visual evidence acquisition. SCHEMA organizes this principle around robot-quality event assessment and boundary refinement, producing temporal supervision for policy learning.

Appendix A.9 connects these design choices to targeted comparisons: fixed scanning and repeated global review test the inspection strategy; target-only windows test surrounding temporal context; reconstruction from the same traces tests final boundary refinement. Across 1,350 episodes, refinement improves mean BS-IoU in 388 episodes and newly matches 35 original GT segments, with a net increase of 18 matches. Part 3 reports the component scores and token usage.

References

[1] Liang et al. Robometer: Scaling General-Purpose Robotic Reward Models via Trajectory Comparisons. Robotics: Science and Systems (RSS), 2026.

[3] Zhang et al. SCIZOR: A Self-Supervised Approach to Data Curation for Large-Scale Imitation Learning. ICRA, 2026.

[4] Zhai et al. HELP: Human-Efficient Large-Scale Robot Post-Training with Rollout Segmentation. arXiv:2607.09776, 2026.

[5] Wang et al. VideoAgent: Long-form Video Understanding with Large Language Model as Agent. ECCV, 2024.

AuthorsOpenReview ↗

Response to Reviewer SaUD [3/3]

Response to Question 2: Component evidence, computational cost, and boundary refinement

Following the mechanism comparison in Part 2, we report the component results, token usage, and paired refinement statistics from Appendix A.9.

Component analysis on all 1,350 RTQ-Bench episodes (%).

ConfigurationBS-IoU All Macro ↑BS-IoU GT+ Macro ↑CC-Recall pooled ↑
Full SCHEMA50.5444.6441.55
Fixed uniform local scan39.8137.0525.67
Repeated global review38.4933.3024.80
Target-only local windows40.8431.8824.01
One adaptive local inspection34.9026.4916.98
One inspection, without refinement34.9126.4117.54
Without final boundary refinement (trace reconstruction)50.5044.5541.05

Recorded inference usage (million tokens).

Configuration / stageInputOutputTotal% of Full
Full SCHEMA108.935.43114.36100.00%
Fixed uniform local scan116.305.11121.42106.17%
Repeated global review125.546.85132.38115.76%
Target-only local windows97.295.43102.7389.83%
One adaptive local inspection51.382.6053.9847.20%
One inspection, without refinement29.902.0831.9927.97%
Without final boundary refinement (same traces)77.954.6882.6372.26%
Full SCHEMA: final refinement only30.980.7531.7327.74%

Counts are rounded independently; fixed-scan and repeated-review values retain their original reporting precision. The no-final-refinement cost is derived from the same Full SCHEMA traces by excluding final-refinement calls. The last row is a stage included in Full SCHEMA. All rows follow the recorded-call accounting over the same 1,350 episodes.

Adaptive inspection and token efficiency. The fixed-scan and repeated-review controls use Full SCHEMA's per-episode observation counts. Full SCHEMA achieves higher scores on all three localization metrics while using 114.36M tokens, compared with 121.42M and 132.38M for the two controls—approximately 5.82% and 13.61% fewer tokens, respectively. Target-only windows use 89.83% of Full's tokens; restoring surrounding local context improves All Macro BS-IoU, GT+ Macro BS-IoU, and pooled CC-Recall by 9.70, 12.76, and 17.54 percentage points, respectively.

Evidence acquisition and refinement. Contextual inspection identifies events; refinement fine-tunes their temporal extent. Before final refinement, the same traces yield 50.50 / 44.55 / 41.05 on the three metrics using 82.63M tokens (72.26% of Full). This exceeds fixed scanning and repeated review while using 31.95% and 37.58% fewer tokens, respectively. Final refinement uses 31.73M tokens (27.74%) for finer local observations and endpoint adjustment.

Boundary refinement: finer action detail. Denser observations adjust an event's start and end while preserving its category and explanation (Appendix A.4). These endpoints determine which training frames receive quality-dependent sampling weights or conditioning labels (Appendix A.5). We retain this stage to capture action details at the finer temporal granularity needed by these training interfaces.

Comparing Full SCHEMA with its pre-refinement reconstruction on the same 1,350 episode traces shows these local benefits (Appendix A.9):

  • BS-IoU: 388 episodes attain a higher mean BS-IoU across the optimal and suboptimal categories after refinement.
  • CC-Recall: 35 previously unmatched original GT segments satisfy the matching criteria after refinement, yielding a net increase of 18 matched original GT segments across all changes.

These counts measure improved overlap and recovery of existing reference events. Appendix A.9 reports the complete paired breakdown.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.