- Rating
- 3
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper introduces SCHEMA, a harness-guided framework that uses a VLM to iteratively acquire visual evidence, assess context, and refine temporal boundaries to produce sparse optimal/suboptimal interval metadata from robot manipulation trajectories. It also presents RTQ-Bench, a human-annotated benchmark for temporal quality localization. The authors show that SCHEMA substantially improves temporal localization over single-pass inference with the same backbone and delivers competitive results against strong VLM and score-based baselines; furthermore, SCHEMA-guided sampling and conditioning yield consistent gains in large-scale VLA pretraining on RoboTwin 2.0 and in real-world bimanual tasks, particularly when failed rollouts are included.
Critical strengths
-
SCHEMA combines budgeted evidence seeking, contextual quality assessment, and boundary refinement into a structured framework for temporal quality annotation, improving localization over single-pass inference with the same VLM backbone.
-
The generated metadata supports both sampling-based and quality-conditioned policy learning without requiring the annotation harness at deployment, providing a reusable interface between trajectory assessment and VLA training.
-
Evaluation spans RTQ-Bench, downstream RoboTwin 2.0 tasks, and four real-world manipulation tasks. The real-world results suggest improved use of mixed-quality experience, with larger average benefits when failed rollouts are included.
Critical weaknesses
-
There is no ablation of SCHEMA’s main components, such as iterative evidence seeking and boundary refinement, making it difficult to determine which design choices account for the reported gains.
-
The definition of “optimal/suboptimal” relies on a rubric and human judgment. Independent inter-annotator agreement is not reported, leaving uncertainty about the consistency of event selection, quality labels, and temporal boundaries.
-
SCHEMA achieves higher “All” macro BS-IoU but lower GT+ macro BS-IoU and CC-Recall than GPT-5.6. This mixed ranking warrants further analysis of how sparse annotations and empty-reference episodes influence the comparison.
-
Progress-based baselines are evaluated through a shared progress-to-label conversion that may miss stagnation and repeated ineffective attempts. Their reported performance may therefore partly reflect limitations of the conversion rather than the underlying models.
Questions for authors
-
How much of the localization improvement comes from adaptive evidence seeking and boundary refinement, rather than additional observations and model calls? Can you provide a budget-matched comparison or a targeted ablation to clarify this?
-
Since unannotated regions are not verified negatives, have you checked whether predictions outside the reference intervals correspond to valid but unannotated events? A manual audit on a representative subset would help interpret the mixed ranking between SCHEMA and GPT-5.6 across All, GT+, and recall metrics.
5 comments
Response to Reviewer UmV5 [2/4]
Response to Weakness 2: Annotation reliability
RTQ-Bench uses primary human annotation followed by a second human review of every episode. Human experts determine event selection, quality categories, and temporal boundaries (Appendix D).
To assess annotation reliability, we have completed independent verification with three additional human evaluators, complemented by GPT-6 Astra (Max) verification, across all 1,350 videos (Appendix D.1). These checks directly assess event selection, quality categories, and temporal boundaries. A separate uncovered-region audit examines additional task-significant events outside the reference intervals (Appendix D.2).
Human reference review (Appendix D.1). Each evaluator reviews the same 4,550 reference items, assessing event inclusion, quality category, temporal boundaries, and the appropriateness of empty category lists. Overall acceptance ranges from 95.65% to 97.67%; acceptance of annotated optimal and suboptimal intervals ranges from 93.49% to 99.70%.
| Verification target | Items | Human 1 accepted (%) | Human 2 accepted (%) | Human 3 accepted (%) |
|---|---|---|---|---|
| Annotated optimal intervals | 2,973 | 2,964 (99.70%) | 2,873 (96.64%) | 2,901 (97.58%) |
| Annotated suboptimal intervals | 584 | 565 (96.75%) | 578 (98.97%) | 546 (93.49%) |
| Empty optimal reference lists | 174 | 142 (81.61%) | 91 (52.30%) | 138 (79.31%) |
| Empty suboptimal reference lists | 819 | 773 (94.38%) | 810 (98.90%) | 807 (98.53%) |
| All verification items | 4,550 | 4,444 (97.67%) | 4,352 (95.65%) | 4,392 (96.53%) |
Acceptance rate = accepted reference items / items; overall rates are weighted by item count.
Complementary verification with GPT-6 Astra (Max). The model reviews the same reference items using video evidence, with the original annotation reasons withheld. Its results are reported separately below.
| Verification target | Items | Accepted | Acceptance rate |
|---|---|---|---|
| Annotated optimal intervals | 2,973 | 2,876 | 96.74% |
| Annotated suboptimal intervals | 584 | 432 | 73.97% |
| Empty optimal reference lists | 174 | 79 | 45.40% |
| Empty suboptimal reference lists | 819 | 763 | 93.16% |
| All verification items | 4,550 | 4,150 | 91.21% |
Response to Weakness 3 and Question 2: Sparse references and benchmark interpretation
Uncovered-region audit (Question 2)
We have completed an uncovered-region audit with three human evaluators and GPT-6 Astra (Max) to examine valid events beyond the sparse references (Appendix D.2). The audit examines video regions outside all retained optimal and suboptimal intervals, with annotation reasons and previous verification results withheld. It records clearly visible, task-significant events distinct from the retained annotations; boundary adjustments, ordinary preparatory actions, and minor pauses are excluded.
| Evaluator | Category with nonempty GT | Eligible videos | Videos with detected omissions | Detection rate |
|---|---|---|---|---|
| GPT-6 Astra (Max) | Optimal | 1,176 | 275 | 23.38% |
| GPT-6 Astra (Max) | Suboptimal | 531 | 11 | 2.07% |
| Human 1 | Optimal | 1,176 | 198 | 16.84% |
| Human 1 | Suboptimal | 531 | 4 | 0.75% |
| Human 2 | Optimal | 1,176 | 216 | 18.37% |
| Human 2 | Suboptimal | 531 | 9 | 1.69% |
| Human 3 | Optimal | 1,176 | 142 | 12.07% |
| Human 3 | Suboptimal | 531 | 13 | 2.45% |
For each category, eligible videos have nonempty GT. Detection rate is the percentage of eligible videos with at least one qualifying omitted event of that category.
Audit findings. The audits by the human evaluators and GPT-6 Astra (Max) consistently show that detected omissions are concentrated in optimal events and are limited to a minority of eligible videos; suboptimal omissions are rare. The three human evaluators detect additional optimal events in 12.07–18.37% of eligible videos, compared with only 0.75–2.45% for suboptimal events. GPT-6 Astra (Max) shows the same pattern, with rates of 23.38% and 2.07%, respectively. For every evaluator, more than three quarters of eligible optimal-category videos have no additional optimal event detected, and over 97.5% of eligible suboptimal-category videos have no additional suboptimal event detected.
Together with the 93.49–99.70% human acceptance of retained intervals, these findings provide complementary support for the reference annotations: the retained events receive strong human support, and additional detections predominantly concern optimal behavior, with suboptimal omissions detected in very few eligible videos. All reported method comparisons continue to use the same original reference annotations and unchanged benchmark scores.
Response to Reviewer UmV5 [3/4]
Response to Weakness 3: Empty-reference sensitivity and mixed rankings
Appendix B.6 reports both the original metrics and sensitivity to empty-reference handling (%):
| Method | All Macro BS-IoU (primary) | All Macro BS-IoU (full credit) | GT+ Macro BS-IoU | CC-Recall pooled |
|---|---|---|---|---|
| SCHEMA | 50.54 | 63.76 | 44.64 | 41.55 |
| GPT-5.6 Sol | 34.38 | 66.83 | 47.94 | 52.49 |
Full credit assigns score 1 to every empty-reference pair irrespective of predictions; it is a sensitivity analysis, not the primary metric. Among 819 episodes with empty suboptimal references, SCHEMA predicts events in 301 and GPT in 762. SCHEMA better matches reference sparsity, while GPT recovers more annotated events overall. The uncovered-region audit in Part 2 identifies optimal events as the main source of additional detections in its eligible videos, with suboptimal omissions detected in very few of them.
Response to Weakness 4: Progress-to-label decoding
Appendix C.2.4 evaluates an episode-wise top/bottom-30% progress-rate decoder using predictions from Robometer [1] and Robo-Dopamine 2.0 [2]. This decoder can select stagnation without negative progress. The comparison uses the original benchmark's BS-IoU metrics (%):
| Method | Decoder | All Macro | GT+ Optimal | GT+ Suboptimal | GT+ Macro |
|---|---|---|---|---|---|
| Robometer-4B | Threshold | 31.96 | 43.74 | 20.65 | 32.19 |
| Robometer-4B | Rank | 28.80 | 46.81 | 42.77 | 44.79 |
| Robo-Dopamine | Threshold | 31.27 | 47.97 | 21.48 | 34.73 |
| Robo-Dopamine | Rank | 28.27 | 45.15 | 43.75 | 44.45 |
All columns report BS-IoU under the original evaluation protocol. All includes every episode; GT+ includes category-positive episodes.
The rank decoder improves GT+ overlap for both progress models, demonstrating how decoder choice affects interval localization. Interpreting low-progress regions as necessary pauses or ineffective repetition additionally requires contextual event assessment.
References
[1] Liang et al. Robometer: Scaling General-Purpose Robotic Reward Models via Trajectory Comparisons. Robotics: Science and Systems (RSS), 2026.
[2] Xu et al. Robo-Dopamine 2.0: History-Conditioned and OOD-Aware Process Reward Modeling for Robotic Manipulation. arXiv:2608.15680, 2026.
Response to Reviewer UmV5 [1/4]
Thank you for your positive assessment and constructive suggestions on component ablations and benchmark interpretation.
Response to Weakness 1 and Question 1: Components and inference budget
To clarify the roles of the individual components and their inference budgets, we have added targeted analyses on all 1,350 RTQ-Bench episodes in Appendix A.9:
Recorded inference usage (million tokens).
Counts are rounded independently; fixed-scan and repeated-review values retain their original reporting precision. The no-final-refinement cost is derived from the same Full SCHEMA traces by excluding final-refinement calls. The last row is a stage included in Full SCHEMA. All rows follow the recorded-call accounting over the same 1,350 episodes.
Adaptive inspection and token efficiency. The fixed-scan and repeated-review controls use Full SCHEMA's per-episode observation counts. Full SCHEMA achieves higher scores on all three localization metrics while using 114.36M tokens, compared with 121.42M and 132.38M for the two controls—approximately 5.82% and 13.61% fewer tokens, respectively. Target-only windows use 89.83% of Full's tokens; restoring surrounding local context improves All Macro BS-IoU, GT+ Macro BS-IoU, and pooled CC-Recall by 9.70, 12.76, and 17.54 percentage points, respectively.
Complementary roles of evidence acquisition and refinement. Contextual, iterative inspection identifies quality events and gathers supporting evidence; final boundary refinement provides finer temporal adjustment of those events. The result reconstructed before final refinement from the same traces reaches 50.50 / 44.55 / 41.05 on the three metrics with 82.63M tokens (72.26% of Full). It outperforms fixed scanning and repeated review while using approximately 31.95% and 37.58% fewer tokens, respectively. The final boundary-refinement stage accounts for 31.73M tokens (27.74%) and is dedicated to examining local action details and adjusting temporal endpoints.
Boundary refinement: finer temporal granularity. Boundary refinement is designed to resolve action details at a finer temporal scale. After contextual inspection identifies a quality event, denser local observations fine-tune its start and end times while preserving its category and semantic explanation (Appendix A.4). These endpoints determine which training frames receive quality-dependent sampling weights or conditioning labels (Appendix A.5). We retain this dedicated stage to meet SCHEMA's objective of producing detailed temporal quality annotations.
The paired comparison between Full SCHEMA and the result reconstructed before final refinement from the same 1,350 episode traces provides concrete evidence of these local adjustments (Appendix A.9):
These paired results show the contribution of finer endpoint adjustment to temporal overlap and recovery of existing reference events. Appendix A.9 reports the complete paired breakdown.