- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies task-definition conflicts in robot benchmarks. It introduces executable task contracts to trace declarations, policy inputs, supervision, rewards, and success checks. It also proposes Contract-Guided Supervision Repair (CGSR), which searches from late to earlier admissible phases, preserves action prefixes, regenerates suffixes, and verifies complete repaired executions. The experiments combine benchmark audits with an evaluation of CGSR and an exploratory student comparison.
Critical strengths
- The paper identifies clear inconsistencies between declarations, policy inputs, and success checks. Its distinction between rescoring existing behavior and improving learned behavior is useful.
- The geometric calibration and blinded human review make the analysis of success checks more convincing.
Critical weaknesses
- The paper does some useful checks, but the methodological contribution is less clear. The search in CGSR is fairly straightforward, while the verifier and controller depend on the task. How much would need to be redesigned for a new task? It is unclear what the contract framework itself lets us reuse.
- The repair experiments are limited in scope. The two tasks share the same scene template, and the frozen extension includes only nine cases that need repair, all involving geometry failures. These results show that the current implementation works on this set of cases, but are not enough to judge how well the method would perform on other tasks or failure types.
- The evidence for improved student performance is still preliminary. Table 3 shows gains across three training seeds, but the evaluation uses previously examined starts (Section 4.5). Would it possible to see whether these gains hold on a separate set of evaluation starts under the same protocol. The independent validation would make the learning results more convincing.
Questions for authors
Please see the weaknesses.