- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper investigates how stop-on-success evaluation protocols can introduce outcome leakage through episode duration, allowing robot policy evaluators to achieve high accuracy without necessarily recognizing task success from visual observations. The authors formalize this problem through an evidence budget and statistical certification criteria, and demonstrate its prevalence across multiple robot policies and simulation environments. They further investigate fixed-horizon logging as a potential remedy, while identifying remaining challenges involving outcome definitions and information unavailable to evaluators.
The paper's strongest contribution is demonstrating how robot evaluation protocols can introduce shortcuts that make conventional evaluator accuracy metrics insufficient for establishing genuine task-recognition capability. The authors provide both theoretical and empirical evidence of this problem, along with an alternative evaluation protocol. The central claims are well supported, although the generality of the proposed certification framework and the practical implications of fixed-horizon evaluation require further clarification.
Critical strengths
-
The paper addresses whether conventional evaluation metrics actually establish the capabilities they are intended to measure. This has meaningful implications for benchmarking robot policies and automated evaluators.
-
The combination of theoretical analysis and controlled experiments is compelling. In particular, the termination-channel intervention provides strong evidence that the observed outcome leakage arises from the evaluation protocol rather than the evaluator itself.
-
The authors go beyond identifying the problem to investigate a potential solution and its limitations. I particularly appreciate the distinction between task achievement and terminal success, and the observation that ground-truth labels may depend on information unavailable to visual evaluators.
Critical weaknesses
-
Clarify the scope of evaluator non-identifiability. The manuscript makes strong claims about evaluators being unverifiable under certain protocols. However, the frame-blind witness is constructed using the original evaluator's scores and outcome labels, rather than operating as an independent evaluator. While the authors acknowledge this distinction, I encourage them to more explicitly distinguish the inability to validate genuine task recognition from evidence that an evaluator lacks such capabilities. Similarly, the proposed certification floor depends on a particular statistical testing framework, and its generality should be carefully qualified.
-
Clarify the fixed-horizon evaluation comparison. The authors argue that fixed-horizon logging restores opportunities for evaluator validation without increasing the number of rollouts. However, the main comparison reports 40-50 episodes per condition under stop-on-success and 200 under fixed-horizon logging. It is unclear how much of the reported improvement in certification is attributable to the protocol rather than differences in sample size. A clearer comparison using equivalent evaluation budgets, or further explanation of the existing matched-size experiments, would strengthen this conclusion. Furthermore, the implications of continuing robot execution after task completion deserve greater consideration, particularly for physical deployments.
-
Strengthen reproducibility and statistical reporting. The manuscript acknowledges several unresolved aspects of its evaluation, including incomplete scorer-protocol records, repeated initial states, and statistical corrections that have not been recomputed at the appropriate clustering level. These limitations are especially relevant because the paper's contribution concerns the reliability of evaluation procedures themselves. I encourage the authors to clarify which results remain robust under these limitations and ensure that the central certification claims are supported by fully specified evaluation procedures.
Questions for authors
See areas for improvement ('critical weaknesses') above.
2 comments
W3. Reproducibility and statistical robustness.
We distinguish the structural results from evaluator-specific certification. The propositions, census, termination-mask intervention, admissibility screen, and achievement/terminal-label counts require no judge scores. Admissibility already uses initial-state-disjoint units, and deduplicating episodes does not change its verdicts (§4.1).
We recovered the W1 scorer protocol from the scoring code and archived rows. Re-scoring with the open-weight judge exactly reproduces all 1,800 recorded scores. The original dated served judge is no longer available; scoring its current version under the recovered protocol yields matched-unit certifications on 7/9 conditions, versus 8/9 with the archived scores. We also re-scored C1 with the open-weight judge under its specified protocol: all three arms remain below the derived clock (margins −0.08 to −0.50). D1/D2 provenance and D2 scorer-protocol details remain unresolved. We retain those analyses as scoped downstream evidence, not support for the central structural claims.
The statistical corrections materially narrow the calibration-based certification results. The initial-state-disjoint matched-unit test (App. C.4) compares raw success/failure score rankings using an exact one-sided sign test, with Holm correction across 18 arms at α = 0.1; it rejects in 16/18 arms. The certification ladder instead tests recalibrated decisions against a calibration-majority comparator. It rejects in 12/18 arms at the episode level. In a new cluster-aware sensitivity analysis, selecting one episode per initial state and repeating the ladder over 100 random collapses yields a median of 2/18 rejecting arms after Holm correction (range 2–4). This is a sensitivity distribution, not one confirmatory 2/18 certificate. The 16/18 ranking result does not rescue the original 12/18 ladder count; the tests address different questions.
We separately assessed calibration-split variability over 20 seeds: the N1 > 0.5 verdict changes for at most one arm, while the other 17 remain stable. This checks split-seed sensitivity, not the effect of choosing different initial-state representatives. The split-sample analysis for γ* remains unrun; no conclusion relies on that selected column (App. C.2).
Thus, the structural non-identifiability result remains unchanged, while the cluster-aware analysis warrants a narrower interpretation of evaluator certification. We will reflect this distinction in the revision and release per-episode tables, matching code, and analysis scripts upon acceptance.
We thank the reviewer again for highlighting these distinctions.
We thank the reviewer for the careful assessment. We address the scope of our claims and the protocol-versus-sample-size comparison below.
W1. Non-identifiability and the scope of the floor.
Our claim concerns verifiability, not an evaluator's intrinsic ability. When duration perfectly determines the outcome, agreement with ground truth cannot establish that the evaluator recognized the task from visual evidence. This does not imply that the evaluator lacks that ability. Indeed, on the three witness conditions, the stop-on-success evidence budget is zero, whereas Qwen3-VL-Plus wins 11/12, 10/12, and 12/12 initial-state-disjoint pairs under fixed-horizon logging (App. C.4).
The frame-blind witness is deliberately conditional: it reassigns the judge's scores within duration-defined outcome classes. It is not an independent or deployable evaluator. Its purpose is to show that AUROC, ROC curves, calibration, and other statistics determined by class-conditional score multisets cannot distinguish these assignments from the original judge's scores. Episode-level statistics can distinguish them (Prop. 1). Independently, a duration-only clock achieves AUROC 1.000 on these three conditions, compared with 0.9888, 0.8926, and 0.9951 for the judge (§4.2).
The floor in Prop. 3 applies to tests under the stated matched-unit label-randomization null, not to all possible statistical models. Within this randomization framework, no nonempty rejection region exists at level δ unless m ≥ ⌈log₂(1/δ)⌉: four units at δ = 0.1. The paper also reports a more conservative seven-unit requirement under its e-value criterion. Initial-state-disjoint matching addresses repeated states but does not remove the need for exchangeability assumptions (App. A). The zero-budget conclusion of Corollary 1 is sharper: under its harness assumptions, and provided no success ends exactly at H, there are no clock-discordant pairs regardless of rollout count.
W2. Protocol versus sample size.
We agree that Table 1 alone cannot separate these effects: it juxtaposes 40–50 stop-on-success episodes with 200 fixed-horizon episodes. The claim of "no additional rollouts" refers to the logging rule itself, not to an equal-size comparison already established by Table 1.
We therefore added two analyses on the fixed-horizon pool. First, we counterfactually truncate the same 200 trajectories per condition at the first success or H for the seven Octo conditions, which have no termination channel. The resulting stop-on-success view supplies 0–1 initial-state-disjoint matched units, versus 12–38 under fixed-horizon logging. Second, across 1,000 random subsamples of 40 episodes per condition, the probability of reaching the four-unit floor is 0 for these seven stop-on-success views, versus 0.88–1.00 across all nine fixed-horizon conditions. These comparisons measure the evidence produced by each protocol, including its label definition; they do not isolate duration from all other changes.
For RT-1-X, policy-issued termination already produces early failures, so both protocol and sample size matter. The paper's P1/P2 collections provide a separate equal-size, but not same-seed, comparison (200 episodes each): Qwen3-VL-Plus AUROC is 0.914 under stop-on-success (clock: 0.788) and 0.830 under fixed-horizon logging; duration matching removes only about a tenth of this gap (§4.3).
Fixed-horizon logging uses 39.6% more simulator steps on average (12.7%–140.5% across conditions), despite requiring no additional rollouts. We do not recommend unrestricted continuation on hardware: the recommendation is explicitly conditional on safety (§5). If continuation is unsafe, the admissibility screen still applies, and agreement-based certification should be withheld when its evidence requirement is unmet.