acceptodds
Under review as a conference paper at ICLR 2027

Every Failure Is a Timeout: Failure-Mode Homogeneity Makes Robot Policy Evaluators Unverifiable

13 views

Abstract

Automatic robot policy evaluators are commonly validated by their agreement with ground-truth success labels. We show that such agreement can provide no evidence of genuine evaluation capability when the evaluation protocol itself reveals the outcome. Under stop-on-success evaluation, policies without a termination channel fail exclusively by reaching the time horizon, allowing episode duration alone to distinguish success from failure. We establish this failure mode through a controlled intervention on the termination channel and construct a frame-blind witness that exactly reproduces a vision–language judge's evaluation metrics. We further formalize an evidence budget based on success–failure pairs misordered by duration and derive a randomization floor below which evaluator certification is impossible. Across 47 screened conditions, only two satisfy this floor, and neither remains admissible after suppressing the termination channel. To address this identifiability failure, we investigate fixed-horizon logging, which eliminates duration-based outcome leakage without increasing the number of rollouts. This protocol also distinguishes achievement from terminal success, revealing discrepancies concealed by stop-on-success evaluation. However, removing duration leakage does not guarantee evaluator verifiability: the first success time retains predictive information, while some terminal success labels depend on historical states inaccessible to frame-based evaluators. Together, our findings establish that reliable evaluator validation requires both sufficient evidence beyond protocol-induced shortcuts and alignment between ground-truth labels and observable information.

Primary area:
datasets and benchmarks
Keywords:
Robot Policy Evaluation, Evaluator Identifiability, Protocol-Induced Shortcuts, Duration Leakage, Fixed-Horizon Logging

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_mKh2OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates how stop-on-success evaluation protocols can introduce outcome leakage through episode duration, allowing robot policy evaluators to achieve high accuracy without necessarily recognizing task success from visual observations. The authors formalize this problem through an evidence budget and statistical certification criteria, and demonstrate its prevalence across multiple robot policies and simulation environments. They further investigate fixed-horizon logging as a potential remedy, while identifying remaining challenges involving outcome definitions and information unavailable to evaluators.

The paper's strongest contribution is demonstrating how robot evaluation protocols can introduce shortcuts that make conventional evaluator accuracy metrics insufficient for establishing genuine task-recognition capability. The authors provide both theoretical and empirical evidence of this problem, along with an alternative evaluation protocol. The central claims are well supported, although the generality of the proposed certification framework and the practical implications of fixed-horizon evaluation require further clarification.

Critical strengths
  1. The paper addresses whether conventional evaluation metrics actually establish the capabilities they are intended to measure. This has meaningful implications for benchmarking robot policies and automated evaluators.

  2. The combination of theoretical analysis and controlled experiments is compelling. In particular, the termination-channel intervention provides strong evidence that the observed outcome leakage arises from the evaluation protocol rather than the evaluator itself.

  3. The authors go beyond identifying the problem to investigate a potential solution and its limitations. I particularly appreciate the distinction between task achievement and terminal success, and the observation that ground-truth labels may depend on information unavailable to visual evaluators.

Critical weaknesses
  1. Clarify the scope of evaluator non-identifiability. The manuscript makes strong claims about evaluators being unverifiable under certain protocols. However, the frame-blind witness is constructed using the original evaluator's scores and outcome labels, rather than operating as an independent evaluator. While the authors acknowledge this distinction, I encourage them to more explicitly distinguish the inability to validate genuine task recognition from evidence that an evaluator lacks such capabilities. Similarly, the proposed certification floor depends on a particular statistical testing framework, and its generality should be carefully qualified.

  2. Clarify the fixed-horizon evaluation comparison. The authors argue that fixed-horizon logging restores opportunities for evaluator validation without increasing the number of rollouts. However, the main comparison reports 40-50 episodes per condition under stop-on-success and 200 under fixed-horizon logging. It is unclear how much of the reported improvement in certification is attributable to the protocol rather than differences in sample size. A clearer comparison using equivalent evaluation budgets, or further explanation of the existing matched-size experiments, would strengthen this conclusion. Furthermore, the implications of continuing robot execution after task completion deserve greater consideration, particularly for physical deployments.

  3. Strengthen reproducibility and statistical reporting. The manuscript acknowledges several unresolved aspects of its evaluation, including incomplete scorer-protocol records, repeated initial states, and statistical corrections that have not been recomputed at the appropriate clustering level. These limitations are especially relevant because the paper's contribution concerns the reliability of evaluation procedures themselves. I encourage the authors to clarify which results remain robust under these limitations and ensure that the central certification claims are supported by fully specified evaluation procedures.

Questions for authors

See areas for improvement ('critical weaknesses') above.

2 comments

We thank the reviewer for the careful assessment. We address the scope of our claims and the protocol-versus-sample-size comparison below.

W1. Non-identifiability and the scope of the floor.

Our claim concerns verifiability, not an evaluator's intrinsic ability. When duration perfectly determines the outcome, agreement with ground truth cannot establish that the evaluator recognized the task from visual evidence. This does not imply that the evaluator lacks that ability. Indeed, on the three witness conditions, the stop-on-success evidence budget is zero, whereas Qwen3-VL-Plus wins 11/12, 10/12, and 12/12 initial-state-disjoint pairs under fixed-horizon logging (App. C.4).

The frame-blind witness is deliberately conditional: it reassigns the judge's scores within duration-defined outcome classes. It is not an independent or deployable evaluator. Its purpose is to show that AUROC, ROC curves, calibration, and other statistics determined by class-conditional score multisets cannot distinguish these assignments from the original judge's scores. Episode-level statistics can distinguish them (Prop. 1). Independently, a duration-only clock achieves AUROC 1.000 on these three conditions, compared with 0.9888, 0.8926, and 0.9951 for the judge (§4.2).

The floor in Prop. 3 applies to tests under the stated matched-unit label-randomization null, not to all possible statistical models. Within this randomization framework, no nonempty rejection region exists at level δ unless m ≥ ⌈log₂(1/δ)⌉: four units at δ = 0.1. The paper also reports a more conservative seven-unit requirement under its e-value criterion. Initial-state-disjoint matching addresses repeated states but does not remove the need for exchangeability assumptions (App. A). The zero-budget conclusion of Corollary 1 is sharper: under its harness assumptions, and provided no success ends exactly at H, there are no clock-discordant pairs regardless of rollout count.

W2. Protocol versus sample size.

We agree that Table 1 alone cannot separate these effects: it juxtaposes 40–50 stop-on-success episodes with 200 fixed-horizon episodes. The claim of "no additional rollouts" refers to the logging rule itself, not to an equal-size comparison already established by Table 1.

We therefore added two analyses on the fixed-horizon pool. First, we counterfactually truncate the same 200 trajectories per condition at the first success or H for the seven Octo conditions, which have no termination channel. The resulting stop-on-success view supplies 0–1 initial-state-disjoint matched units, versus 12–38 under fixed-horizon logging. Second, across 1,000 random subsamples of 40 episodes per condition, the probability of reaching the four-unit floor is 0 for these seven stop-on-success views, versus 0.88–1.00 across all nine fixed-horizon conditions. These comparisons measure the evidence produced by each protocol, including its label definition; they do not isolate duration from all other changes.

For RT-1-X, policy-issued termination already produces early failures, so both protocol and sample size matter. The paper's P1/P2 collections provide a separate equal-size, but not same-seed, comparison (200 episodes each): Qwen3-VL-Plus AUROC is 0.914 under stop-on-success (clock: 0.788) and 0.830 under fixed-horizon logging; duration matching removes only about a tenth of this gap (§4.3).

Fixed-horizon logging uses 39.6% more simulator steps on average (12.7%–140.5% across conditions), despite requiring no additional rollouts. We do not recommend unrestricted continuation on hardware: the recommendation is explicitly conditional on safety (§5). If continuation is unsafe, the admissibility screen still applies, and agreement-based certification should be withheld when its evidence requirement is unmet.

W3. Reproducibility and statistical robustness.

We distinguish the structural results from evaluator-specific certification. The propositions, census, termination-mask intervention, admissibility screen, and achievement/terminal-label counts require no judge scores. Admissibility already uses initial-state-disjoint units, and deduplicating episodes does not change its verdicts (§4.1).

We recovered the W1 scorer protocol from the scoring code and archived rows. Re-scoring with the open-weight judge exactly reproduces all 1,800 recorded scores. The original dated served judge is no longer available; scoring its current version under the recovered protocol yields matched-unit certifications on 7/9 conditions, versus 8/9 with the archived scores. We also re-scored C1 with the open-weight judge under its specified protocol: all three arms remain below the derived clock (margins −0.08 to −0.50). D1/D2 provenance and D2 scorer-protocol details remain unresolved. We retain those analyses as scoped downstream evidence, not support for the central structural claims.

The statistical corrections materially narrow the calibration-based certification results. The initial-state-disjoint matched-unit test (App. C.4) compares raw success/failure score rankings using an exact one-sided sign test, with Holm correction across 18 arms at α = 0.1; it rejects in 16/18 arms. The certification ladder instead tests recalibrated decisions against a calibration-majority comparator. It rejects in 12/18 arms at the episode level. In a new cluster-aware sensitivity analysis, selecting one episode per initial state and repeating the ladder over 100 random collapses yields a median of 2/18 rejecting arms after Holm correction (range 2–4). This is a sensitivity distribution, not one confirmatory 2/18 certificate. The 16/18 ranking result does not rescue the original 12/18 ladder count; the tests address different questions.

We separately assessed calibration-split variability over 20 seeds: the N1 > 0.5 verdict changes for at most one arm, while the other 17 remain stable. This checks split-seed sensitivity, not the effect of choosing different initial-state representatives. The split-sample analysis for γ* remains unrun; no conclusion relies on that selected column (App. C.2).

Thus, the structural non-identifiability result remains unchanged, while the cluster-aware analysis warrants a narrower interpretation of evaluator certification. We will reflect this distinction in the revision and release per-episode tables, matching code, and analysis scripts upon acceptance.

We thank the reviewer again for highlighting these distinctions.

Reviewer_VtWPOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper focuses on automatic evaluators for robot policies and, specifically, in understanding when such evaluators do actually recognize task success or over-fit spurious features. The authors identify in the stop-on-success protocol a key challenge for automatic evaluation and propose novel metrics to determine when these evaluator can be trusted.

Critical strengths
  1. I find the paper interesting, and the experimental results seem convincing. The authors identify an important problem related to automatic robot policy evaluators, analyze its implications, and investigate possible solutions to address it.
Critical weaknesses
  1. The paper is difficult to read. The presentation and prose should be substantially improved to make the contributions more accessible. Several key concepts, including stop-on-success, frame-blind witness, evidence budget, and fixed-horizon logging, would benefit from clearer, more intuitive explanations before introducing the mathematical formalism. The paper also introduces several closely related concepts whose differences are not always immediately apparent.
Questions for authors
  1. Could the authors clarify exactly what information is available to the automatic evaluators? In particular, do they have access to episode duration, timestamps, or only sampled visual observations?

  2. Could the authors provide a more intuitive explanation of how the frame-blind witness is constructed and what it demonstrates? In particular, since the witness relies on the original evaluator's scores, how should we interpret its equivalence to the evaluator in terms of evaluation metrics?

  3. Could the authors better explain the differences between identifiability, admissibility, and certification? These concepts appear central to the theoretical analysis, but their practical interpretation and relationship are not immediately clear.

  4. Could the authors clarify how the proposed fixed-horizon protocol changes the evaluation objective? In particular, what is the intended distinction between task achievement and terminal success, and under what circumstances should one be preferred over the other?

1 comment

We thank the reviewer for the questions. We agree that the presentation should be more accessible, and we are revising the manuscript to introduce each key concept intuitively before its formal definition, using the explanations below.

The paper in plain terms

An automatic judge of robot rollouts is usually validated by how well its scores agree with ground-truth success labels. Many benchmarks end an episode as soon as the task succeeds, a protocol we call stop-on-success. Under this protocol, a policy that cannot end its own episode fails only by running out of time. In the Octo-Small close-drawer example of §1, all 23 failures end at the horizon of 114 steps, while all 17 successes end earlier. Episode length alone therefore separates the outcomes, so high agreement does not by itself show that the judge used the video. In the paper, we formalize this limitation, measure how common it is, and study fixed-horizon logging, in which every episode runs to the horizon H, as a remedy. Below, the clock is the scorer that uses duration alone, and the evidence budget* is the set of success–failure pairs that the clock orders wrongly or ties.

Q1. What information the evaluators receive

The evaluators receive sampled frames and a fixed text prompt. They receive no duration, no timestamp and no frame count. Duration may still enter the input indirectly, because the served judges score the first and the last three frames of an episode and the frame index of Qwen2.5-VL-7B is clamped on short episodes, "so its input may encode duration" (App. C.1). In any case, our argument does not rely on the judge observing duration, because when duration separates the outcomes in the evaluation set, agreement with the labels can be reached without the frames.

Q2. How the frame-blind witness is built and interpreted

To build the witness, we shuffle the judge's own scores within each outcome class (Figure 1a). When duration separates the outcomes, the duration of each episode identifies its outcome class, so failures exchange scores with failures and successes with successes. The resulting witness therefore uses only durations and a list of scores.

The judge and the witness obtain identical aggregate metrics, because AUROC, the ROC curve, calibration, the Brier score and ECE depend only on which scores fall in each class, and the shuffle leaves the score distribution within each class unchanged (§1). The score of a single episode, by contrast, changes by up to 0.956 on close-drawer.

This equivalence indicates that, on an evaluation set where duration separates the outcomes, the aggregate metrics alone do not distinguish a judge that reads the frames from a score assignment based on duration only. The witness is neither a deployable rival evaluator (§3.2) nor evidence that the judge ignored the frames.

Q3. How identifiability, admissibility and certification differ

These concepts correspond to successive questions (§3.1). Identifiability holds when the evaluation set separates the evaluator from the clock. Admissibility holds when the evaluation set contains enough evidence for a valid test, and at δ = 0.1 admissibility requires four matched units, which are pairs from the evidence budget that share no episode and no initial state. This minimum comes from the exact sign test, in which a judge that wins all m matched units obtains p=2−mp = 2^{-m}, a value below 0.1 only when m ≥ 4. Finally, certification holds for one judge on an admissible set when a valid test rejects the hypothesis that the judge is no better than the clock. As we state in the paper, "Admissibility is necessary for certification but not sufficient." In the census, two of the 47 conditions are admissible (§1).

Q4. Whether fixed-horizon logging changes the evaluation objective

Fixed-horizon logging does not change the underlying task goal or success predicate, but it allows two distinct evaluation targets to be measured. Achievement means that the success predicate fires at least once, whereas terminal success requires the predicate to hold at H (Remark 4). These labels differ on 93 of 1,800 fixed-horizon episodes (§4.3). Neither definition is universally preferable: achievement suits event-based objectives, while terminal success suits objectives requiring a final state. We therefore recommend reporting both labels (§5).

We thank the reviewer again for highlighting these presentation issues. We will incorporate these clarifications into the revised manuscript and hope they address the reviewer's concerns.

Reviewer_XpmvOpenReview ↗
Rating
4
Confidence
2
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

The paper studies automatic evaluators for robot policies, especially vision–language judges that label rollouts as successful or failed. Its core claim is that the standard validation recipe, comparing a judge’s scores with ground-truth labels using AUROC, accuracy, calibration, or Brier score, can be fundamentally insufficient.

Under stop-on-success logging, success terminates an episode early, while a policy without an explicit termination action can fail only by timing out at horizon H. If no success lasts to H, duration perfectly reveals the label. In that case, a judge’s apparently strong agreement cannot establish that it used visual evidence. The paper makes this an identifiability claim rather than merely a warning about shortcut learning. It constructs a conditional frame-blind witness: by permuting a judge’s scores within duration-defined outcome classes, it preserves the judge’s AUROC, full ROC curve, calibration, Brier score, and ECE while changing the episode-level score assignments. Thus, conventional aggregate metrics cannot distinguish visual competence from a score assignment compatible with duration alone. The paper provides a great number of tests and statistics to support the critique.

Critical strengths
  • The paper offers a relevant, mathematically and experimentally grounded critique of evaluators that rely solely on statistical agreement with ground-truth labels on held-out rollouts, measured by accuracy, AUROC, or calibration.

  • The paper introduces a statistically defined identifiability that, together with the evidence budget and the randomization floor, provides a concrete mathematical proof that an evaluator cannot be certified under standard logging conditions. When a policy lacks a termination channel and the harness stops on success, every failure is forced to end at the horizon.

  • The authors build a trivial baseline script that is completely stripped of any vision or language capabilities. It does not look at the robot’s video frames, nor does it process language instructions. Instead, it takes only one single feature as input: the episode duration (the number of frames or seconds in the logged trial). This shows that the agreement between success/failure labels and human ground-truth labels is not what is going on. The judge is simply exploiting the fact that short episodes = success and long episodes = failure.

  • The evidence budget counts success–failure pairs for which Tsuccess ≥ Tfailure, including ties. These are the only comparisons on which an evaluator can outperform a duration-based clock. If duration perfectly orders labels, b = 0: collecting more rollouts of the same kind cannot create evidence beyond duration. The authors evaluate the evidence budget across 47 environments systematically; they empirically demonstrate that failure-mode homogeneity is not an isolated bug found in a single dataset; it is systematic through several reinforcement learning and imitation learning benchmarks such as RT-1 / RT-1-X.

  • The randomization floor imposes a condition for an evaluator to be considered. Running every rollout to H makes observed duration constant, so the realized duration clock is chance-level. It also preserves information that stop-on-success hides: first-success time, terminal state, and the difference between ever achieving the task and still satisfying it at the horizon. Under these conditions, the authors show that the performance metrics (F1-score, accuracy) of the VLM evaluators collapsed dramatically. This empirical drop directly proved that the VLMs were not using robust visual reasoning to identify success; they were heavily relying on the protocol-induced duration shortcut to make their predictions.

Critical weaknesses

The paper clearly discusses limitations.

When evaluators have richer context, such as variations in language instruction, multi-camera views, or force-torque data, reliance on temporal shortcuts may shift. The paper’s critique applies cleanly to vision-language models evaluated purely on end-to-end binary outcome labels, but it spends less time analysing how dense step-level supervision or intermediate goal-checking alters identifiability.

The paper focuses primarily on in-distribution statistical identifiability rather than exploring active adversarial stress-testing as an alternative verification avenue and out-of-distribution counterfactual tests.

Fixed-horizon logging removes observed episode length, but first-success time remains highly predictive. The paper shows that the derived clock beats judges on 14 of 18 arms, yet it does not provide a full method to validate visual evaluation independently of all predicate-trace-derived timing information.

Questions for authors

You report your tests on real robot log data for manipulation tasks. Have you tested or looked into different tasks?

Maybe you have explored out-of-distribution tests?

1 comment

We thank the reviewer for the carefully evaluation. The following content is a response to your question.

W1. Richer inputs and step-level labels

We agree that reliance on temporal shortcuts may shift with richer context. The identifiability result does not depend on the evaluator's input, because Proposition 1 conditions only on labels, durations and the scorer's own scores per outcome class. Therefore, whenever the duration supports of the two outcomes are disjoint, the witness would reproduce the agreement statistics of a multi-camera or force-torque evaluator. This is an inference; we have not tested such evaluators. The same reasoning extends to step-level labels. Under stop-on-success the success predicate holds only at the final step of a successful episode, so a step-level label is determined by the position of the step within the episode.

Richer input can still change whether the label is visible. In an input ablation, the AUROC of Qwen2.5-VL-7B on pick-coke-can, whose label depends on earlier events, rises from 0.71-0.75 (last frame) to 0.78-0.86 (16 frames), while eggplant stays at 0.96-1.00.

W2 and Q2. Adversarial and out-of-distribution tests

Such tests complement identifiability, and we have run several. For termination, zeroing only the decoded termination component of RT-1-X on its two admissible cells removes early failure in all 34 failures and drives the evidence budget to zero (§4.2). For inputs, changing only the frames shown for fixed episodes moves AUROC by +0.1049 [0.0527, 0.1624] on move-near and +0.0187 [-0.0274, 0.0615] on pick-coke-can (§4.3), and over all nine fixed-horizon conditions the AUROC of Qwen2.5-VL-7B is 0.031 higher (95% CI [0.009, 0.056]) on the frames stop-on-success would record. For scenes, ten LIBERO-Plus cells in new rollouts perturb lighting, background and table textures, object layout, initial robot state and instruction, and all 22 failures end at the horizon. For policies, four action-perturbation variants of one OpenVLA-OFT checkpoint give 357 failures, none early (§4.2), and the nine judge–condition pairs of OpenVLA and SpatialVLA all fall below the derived clock (§4.4).

Camera viewpoint, sensor noise and adversarial perturbation of the judge's input remain untested; in future work we will extend the study to a more comprehensive out-of-distribution and adversarial evaluation.

W3. Validation beyond the first-success time

We agree that the first-success time remains highly predictive; the derived clock, rebuilt from this time with the predicate trace no judge receives, beats the judges in 14 of 18 arms (§4.4). Our test nevertheless extends to this signal, since against the derived clock the judge is tested only on success–failure pairs that this clock orders wrongly or ties, where the first-success time does not favour the correct answer. This restriction removes the main temporal shortcut, without identifying which visual evidence the judge uses. We pre-registered the test (exact sign test of App. C.4) and excluded pick-coke-can, whose terminal label depends on predicate history. Two conditions supply the four matched units required.

Judgespoon-on-towel (7 units)move-near (4 units)
Qwen3-VL-Plus7 of 72 of 4
Qwen2.5-VL-7B5 of 71 of 4

Validation beyond first-success timing thus appears feasible, at least for one judge on one condition.

Q1. Other tasks and benchmarks

All rollouts are simulated (Ethics statement). The census covers 47 conditions (nine SimplerEnv and ten LIBERO tasks, five policies), and for policies without a termination channel none of the 646 Octo and 360 LIBERO failures ends before the horizon (§4.2, App. D). We also screened RoboCasa with Cosmos Policy and LIBERO-Plus with OpenVLA-OFT in new rollouts; neither policy has a termination channel, and both harnesses stop on success. As Corollary 1 predicts (registered beforehand), every failure ends at the horizon and no cell has b>0b > 0.

BenchmarkCellsRolloutsFailuresBefore HCells with b>0b > 0
RoboCasa1040011500
LIBERO-Plus102482200

Under fixed-horizon logging (one pre-selected cell per benchmark, n=100n = 100), duration no longer separates the outcomes; the minority outcome has 37 episodes on RoboCasa and 16 on LIBERO-Plus, and Qwen2.5-VL-7B (AUROC 0.627, 0.695) is certified on neither. In future work, we will apply the screen and fixed-horizon logging to real-robot rollouts.

We hope these answers help.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.