acceptodds
Desk rejected at ICLR 2027

Task-Definition Conflicts in Robot Benchmarks: Measurement and Learning

Desk rejected at ICLR 2027, so left out of the paper lists and the map.

Abstract

Robot benchmarks distribute task definitions across instructions, demonstrations, rewards, and success checks. These components can agree at runtime yet implement a goal different from the declaration. We introduce executable task contracts to trace this correspondence and separate measurement changes from learning interventions. The same reference guides Contract-Guided Supervision Repair (CGSR), which searches late-to-early rollback boundaries, preserves action prefixes, and verifies complete repaired executions. In LIBERO, native and calibrated completion checks differ by 38.0 percentage points on identical executions. In a frozen extension, CGSR repairs nine of nine fresh-execution failures versus six for full recollection, while halving candidate-execution cost per verified repair. A subsequent exploratory student comparison improves calibrated completion by 13.3 points under equal training updates and unequal acquired-data yields. Blinded human review and RoboCasa monitor diagnostics clarify the interpretation of these outcomes. Together, explicit task-reference tracing and verified repair connect benchmark claims to the behaviors actually measured and taught.

Primary area:
applications to robotics, autonomy, planning
Keywords:
robot benchmarks, task specification, benchmark auditing, evaluation validity, vision-language-action models

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_LKJFOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies task-definition conflicts in robot benchmarks. It introduces executable task contracts to trace declarations, policy inputs, supervision, rewards, and success checks. It also proposes Contract-Guided Supervision Repair (CGSR), which searches from late to earlier admissible phases, preserves action prefixes, regenerates suffixes, and verifies complete repaired executions. The experiments combine benchmark audits with an evaluation of CGSR and an exploratory student comparison.

Critical strengths
  • The paper identifies clear inconsistencies between declarations, policy inputs, and success checks. Its distinction between rescoring existing behavior and improving learned behavior is useful.
  • The geometric calibration and blinded human review make the analysis of success checks more convincing.
Critical weaknesses
  • The paper does some useful checks, but the methodological contribution is less clear. The search in CGSR is fairly straightforward, while the verifier and controller depend on the task. How much would need to be redesigned for a new task? It is unclear what the contract framework itself lets us reuse.
  • The repair experiments are limited in scope. The two tasks share the same scene template, and the frozen extension includes only nine cases that need repair, all involving geometry failures. These results show that the current implementation works on this set of cases, but are not enough to judge how well the method would perform on other tasks or failure types.
  • The evidence for improved student performance is still preliminary. Table 3 shows gains across three training seeds, but the evaluation uses previously examined starts (Section 4.5). Would it possible to see whether these gains hold on a separate set of evaluation starts under the same protocol. The independent validation would make the learning results more convincing.
Questions for authors

Please see the weaknesses.

Reviewer_UED5OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The authors point out that in some simulated benchmarks such as LIBERO, the language instruction does not align with the demonstrations (as well as the simulator success check). They propose a way to fix these unmatched demonstrations by removing the problematic part of the trajectory and fill them in with a hand-designed controller with privilege simulator information. Through experiments, the authors show that this mismatch influences scores, and is more efficience than recollecting data.

Critical strengths

n/a

Critical weaknesses
  1. The paper is extremely hard to understand. Many of the phrasing is weird and is hard to parse. For example, the third sentence says “we introduce executable task contracts to trace this correspondence and separate measurement changes from learning interventions”, but it is unclear what “this correspondence”, “measurement changes”, and “learning interventions” refer to because it is not introduced in the two previous sentences. In addition, central examples are used before they are introduced and explained. For example, the caddy example from LIBERO is used throughout the paper, but nowhere in the paper do the authors introduce what the example actually is. It first appears on line 32 without any introduction. A major rewrite is needed.
  2. The significance of the work is questionable. The authors propose a way to fix existing simulated demonstrations using a hand-designed controller with privileged simulator information. However, it is not clear to me why this is needed instead of just using the same controller (or human teleoperators) to collect new demonstrations in simulation. An even simpler fix of the problem pointed out in the paper would be just editing the language task instruction to align with the demonstrations.
Questions for authors
  1. Why is it important to edit the action trajectory if it doesn’t align with language instructions? Wouldn’t it be easier to fix language instruction instead?
  2. If we need a hand-designed controller with privilege simulator information to correct the “corrupted” trajectories, why not use the controller to collect entirely new trajectories? Why is it important to keep as much of the original demonstrations as possible? The experiment in Section 4.4 argues that this is more efficient because it keeps more of the original demonstrations, but both methods seem be to on the same scale of cost.
Reviewer_EpT8OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper argues that robot benchmarks spread a task definition over declaration, policy-facing instruction, demonstrations, reward and success check, which can agree with each other yet differ from the declaration. To trace this correspondence, the paper proposes executable task contracts, a framework designed to trace task intent across every layer of execution. The paper establishes a clean boundary between measurement adjustments (relabelling existing executions with corrected checkers) and learning interventions (providing modified demonstrations to train new policies). To repair flawed demonstrations without discarding entire trajectories, the authors propose Contract-Guided Supervision Repair (CGSR) which searches late-to-early rollback, keeps the action prefix, regenerates the suffix and verifies the full execution.

To support these claims, an audit of the standard LIBERO and RoboCasa benchmarks is conducted. On the LIBERO caddy tasks the declared goal hit 0/100 for official demos and three released policies, yet the internal code instructs the policy and rewards it for placing the mug outside on the tabletop. Moving beyond binary labels, the authors show that metric calibration causes measured success on 150 recorded runs to drop from 65.3% down to 27.3%. The findings are also supported by a human review. The paper also uncovers race conditions in RoboCasa, where querying environment state counters can advance the success logic before the physical task is actually completed.

The proposed CGSR method diagnoses the point of failure, rolls execution back to an earlier valid phase, generates a new action suffix using a privileged controller, and verifies the full trajectory against the contract. On a frozen cohort of nine failures, CGSR recovers all 9 executions compared to only 6 recovered by full recollection, cutting simulator execution cost per repair in half. In an exploratory downstream experiment, student policies trained on the CGSR-augmented dataset show improved strict completion over recollection, demonstrating how explicit task contracts can both diagnose benchmark drift and repair training data.

Critical strengths
  • The paper reveals a verifiable defect in established benchmarks plus a clean measurement/learning separation.
  • The rescoring vs. retraining distinction is useful and applied consistently.
  • By combining same-execution rescoring with tolerance sweeps, the paper provides empirical proof that the benchmark's native evaluation marks physical failures as successes.
  • The claims are supported by a blinded human review.
Critical weaknesses
  1. The paper packs too many concepts in just 9 pages (plus an extensively long appendix). This results in the main text lacking clarity as it stripped out of narrative transitions and intuitive explanations. It also pushes critical context into the massive appendix, resulting in the main text sufficiently lacking completeness. This is a major shortcoming.

  2. CGSR evidence is small and its components are weakly isolated, using 9 geometry failures from one scene template with privileged geometry and a scripted controller.

  3. Regarding student-learning, the +13.3 pts compares 58 vs 51 demonstrations on previously examined starts with seeds 0/3/4. The authors label it exploratory and acknowledge the size difference.

  4. In the equal-size study (Table 4), no gain is achieved from filtering, while native completion falls under replacement. Moreover, the replacements are not CGSR outputs.

  5. Verification uses one host and a fixed seed.

Questions for authors
  1. Does the CGSR advantage hold at an equal dataset size against a “keep replay-passing + recollect failures” arm and on starts not previously examined?
  2. How robust are CGSR-verified demonstrations across hosts and seeds?
  3. How feasible is CGSR without privileged simulator state and an analytical suffix controller, and how much manual phase engineering do other task types need?
  4. Does any published benchmark conclusion change under the declared reference, given that SIMPLER and reset rankings are preserved? Have the LIBERO maintainers confirmed the intended reference?
  5. The 38.0-pt gap corresponds to the authors’ models on these two cuddy tasks. Is this magnitude of gap expected to generalize beyond this specific scene template?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.