acceptodds
Desk rejected at ICLR 2027

Beyond Repeatable Scores: Experimental Fidelity Auditing in Robot Benchmarks

Desk rejected at ICLR 2027, so left out of the paper lists and the map.

Abstract

A repeatable benchmark score is meaningful when its scenes, attempts, and observations preserve the declared experiment. Yet aggregation can conceal changes to these units, masking a different evaluation. We study this gap through experimental-unit fidelity and trace how execution and reporting can alter the declared units. This perspective leads to EFA-Replay, which aligns records with declarations, tests evaluator changes with matched controls, and determines which results the evidence supports. Paired-label accounting exposes changes hidden by aggregation, while sharp compatible-record bounds identify what reduced records preserve. In an ACT evaluation, scene count and 80% success remain unchanged despite 20% request-label disagreement. For SimplerEnv, exact pooled success leaves condition variance unidentified in 64/78 cells. A separate LIBERO audit flags a stale first observation before scoring; correcting it changes ACT task-8 success from 11/30 to 27/30 on frozen follow-up states. These results motivate preserving the unit-level evidence that gives benchmark scores their meaning.

Primary area:
applications to robotics, autonomy, planning
Keywords:
robot-policy evaluation, experimental fidelity, evidence integrity, benchmark auditing, causal testing, statistical inference

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_ASrJOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper argues that a repeatable robot-benchmark score can still describe a different experiment from the one declared, a property it calls experimental-unit fidelity. It identifies two failure mechanisms. At the audited LeRobot revision, per-slot seeds receive a second worker-index offset, so request s+i runs scene s+i+(i mod B). In a five-worker ACT run, scene count (20 distinct) and success (16/20) are identical before and after repair, yet 4 request labels change, and in controlled SmolVLA 9/64 labels change while success moves by only −1.56 pp. On the reporting side, the SimplerEnv helper rebuilds a rounded binary pseudo-trial vector from aggregate success, so pseudo-trial variance exceeds condition-mean variance in 64 of 78 cells.

The authors propose EFA-Replay, which aligns declared units with executed records, tests evaluator changes with matched controls and paired-label accounting, and uses sharp identification ranges over compatible records plus dependency-based invalidation to decide whether to recompute, reuse, replay or leave an evidence gap. They also report a LIBERO audit that flags a stale first observation before scoring, where correcting it changes ACT task-8 success from 11/30 to 27/30. Supporting results include a small diagnostic suite (6 deviations, 5 legal configurations, 1 insufficient input), component removals and archived-replay case studies.

Critical strengths
  1. The central observation is correct, concrete and checkable. Eq. 2 cleanly separates net score change from changed-label fraction. Dlabels − |Δ| = 2·min(n01, n10)/N is exactly the disagreement that averaging cancels, and the ACT example (identical 20 scenes and 16/20, four changed labels) makes the point vividly. The seed-offset mechanism is stated precisely enough to reproduce, and the controls are sensible: all 128 same-arm and 32 unchanged-scene controls pass, the execution is repeated, and a frozen forecast drops errors from 27/80 to 3/80 once the correct effective-scene ledger is used.

  2. The reporting analysis uses partial identification in a principled way. Distinguishing what a retained record determines (pooled success, 78/78) from what it does not (condition table 9/78, variance 14/78) is a useful framing. Attainable bounds with witness records are provided, and the paper separates structural loss (Eq. 3) from rounding effects. Calling out the 64/78 variance inflation, while also reporting the one decrease and the 13 ties, is honest.

  3. The paper is careful about scope and reports negative or limiting cases. It retains two descriptive exclusions, reports a non-official LIBERO adapter, notes that 8 of 10 planned comparisons pass the controls, and states that the second LIBERO check is separate from the exploratory one. The LIBERO timing result (stale vs. current observation, 11/30 → 27/30 with 17 vs. 1 discordant) is large and independently motivated.

Critical weaknesses
  1. The problems found are narrow, and the generality claim is not supported. The execution failure is a seed-offset bug in one library revision, and the reporting failure is the behavior of one released SimplerEnv helper. The paper does not show that either appears in published results, other harnesses or later revisions, and it does not report whether maintainers confirmed them. The abstract and conclusion present these as general properties of robot benchmarking, but the evidence is two library behaviors plus one timing mismatch in LIBERO. The SimplerEnv finding concerns the variance of a reconstructed vector, while the headline mean is unchanged by construction, so it is unclear which published conclusion it alters. The fixed-checkpoint shifts (−12 to +16 pp) are on 50 requests per arm with no uncertainty, and the paper itself says winner identity is preserved in the first cohort.

  2. Statistical support is thin, and key claims rest on small, post hoc or single-checkpoint evidence. Most controlled results have n = 20–64 paired requests, with no confidence intervals or tests for the 9/64 and 9/20 disagreements. The LIBERO result is a single public ACT checkpoint on one task, run through a native adapter that the paper says does not give official scores. The SimplerEnv analysis and the scene-reweighting decomposition are explicitly post hoc, and the cells in Figures 3–5 share scenes, widths and noise strata, so counts such as 64/78 and 52/66 overstate independence. The forecast improvement (27/80 → 3/80) is computed on 80 comparisons that share only 40 task/scene IDs.

  3. The EFA-Replay evaluation is author-designed, and the added value over standard tools is unclear. The diagnostic suite (6 deviations, 5 legal controls, 1 insufficient input) and its presets were built by the authors, EFA-Replay merely ties the “paired oracle”, and the baselines (B0 repeat/total, B1 nominal IDs) are weak by design. There is no held-out or blind set of deviations and no comparison with existing audit tools such as Rollout Cards, ABE-Ralph or metamorphic testing. The recovery decisions and replay case studies are demonstrations on archived data where the plan is derived from the same evidence. Several results restate known identities: Eq. 3 is the standard variance relation and the Eq. 6 bounds follow from convexity, so the sharp ranges mostly quantify a known information loss. The per-benchmark effort of writing adapters, declarations and dependency graphs is not measured.

Questions for authors
  1. Can you show that these failures matter beyond the audited code paths? For example: were the LeRobot seed-offset issue and the SimplerEnv pseudo-trial behavior confirmed by maintainers, do they exist in current revisions, and do they affect any published number or ranking (for example, a leaderboard or a paper that used these tools)? The paper says winner identity was preserved in the first cohort. Is there any case where a conclusion actually flips?

  2. Can you add uncertainty to the key quantities? Specifically, confidence intervals or paired tests for the label-change fractions and fixed-checkpoint shifts, and for the LIBERO 11/30 → 27/30 result across more checkpoints, tasks and suites. How many of the eight control-admitted contrasts remain meaningful once the n = 50 sampling error is taken into account, and how do the 64/78 variance results look when cells sharing scenes and widths are treated as dependent?

  3. How would you evaluate EFA-Replay against an external benchmark of deviations that you did not design? For example: a blind set of injected or naturally occurring evaluator faults from other harnesses, comparison with an existing audit tool or a metamorphic-testing baseline, and a measure of adapter effort and false-flag rate on benchmarks it was not built for. Please also clarify which components (pairing, bounds, dependency closure) are new relative to standard provenance and partial-identification tools.

Reviewer_66YiOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper proposes EFA-REPLAY, a framework for auditing the experimental fidelity of robotic evaluation pipelines. EFA-REPLAY verifies the consistency between requested and executed experimental units, compares original and corrected evaluators to assess the effects of inconsistencies, and determines which experimental results can be recomputed, reused, or require re-execution.

The paper examines three robotic evaluation pipelines and finds that: (1) in RoboCasa, incorrect scene assignments by the LeRobot evaluator can change individual episode outcomes while aggregate success rates remains unchanged, concealing the evaluation error; (2) in SimplerEnv, reconstructing rollout-level success/failure labels from aggregated success rates can distort variance estimates, and introduce rounding errors. (3) in LIBERO, an observation-timing bug causes the policy to receive outdated observations, affecting its measured performance.

Finally, the authors conduct experiments demonstrating that EFA-REPLAY can identify previous rollouts that are valid for reuse and determine which missing rollouts must be executed to complete a full evaluation.

Critical strengths

This paper:

  1. Identifies issues in robotic benchmark evaluation pipelines relevant to the field, including LeRobot, SimplerEnv, and LIBERO.

  2. Provides empirical evidence that these bugs can affect measured policy performance, including controlled comparisons between original and corrected evaluators.

  3. Examines discrepancies hidden by aggregate metrics, showing that individual episode outcomes can change substantially even when overall success rates remain unchanged.

Critical weaknesses

1. Substantial clarity and presentation issues

I found the paper exceptionally difficult to follow due to several presentation issues, including but not limited to: convoluted language; dense combinations of undefined terms and jargon; concepts that are insufficiently defined when introduced; explanations that too often defer details to the appendices; description of relatively straightforward ideas through unnecessary dense terminology.

In my assessment, these issues are substantial enough to warrant rejection, as they prevent a reliable evaluation of the methods, results, contributions, and the paper's overall significance.

I provide representative examples at the end. I strongly encourage the authors to revise the manuscript extensively, especially if substantive parts of the text was written with the assistance of AI.

2. Insufficient specification of EFA-Replay and unclear methodological contribution The paper would substantially benefit from a more concrete explanation of what EFA-Replay is and how it operates.

For example, Figure 2 describes the framework in a very high-level and, in my perspective, unclear language:

A claim-specific audit connects identity, evidence, and recovery. Alignment binds declarations to execution and reporting records. Matched controls assess evaluator changes, and compatible records characterize retained information; eligibility and registered boundaries then determine the action. Outputs distinguish the deviation, supported effect, and recovery decision, linking each to its own evidence requirements.

Section 3 introduces mathematical formulations and describes individual checks and recovery rules, but does not provide a clear end-to-end algorithm, pseudocode, or software architecture explaining how these components work together. Furthermore, much of the methodology appears to formalize established practices, such as checking execution records against specifications or comparing evaluations before and after bug fixes.

3. Unclear generalizability of EFA-Replay It remains unclear whether EFA-Replay constitutes a general auditing system applicable to arbitrary robotic benchmarks or primarily a collection of benchmark-specific checks developed for the studied cases. The methodology appears to rely on manually specified evaluation choices, benchmark-specific adapters, and predefined repair operations.

Could the authors clarify which components are reusable across benchmarks and which must be designed separately? Demonstrating the framework on a previously unseen benchmark would provide stronger evidence of its generalizability.

4. Unclear novelty and practical significance of the aggregation analysis A central claim is that reconstructing pseudo-trials from aggregate success rates can discard information about individual outcomes and distort statistical estimates. However, this is a well-known statistical property, and the issue appears straightforward to address by using the original rollout-level data (which are available in SimplerEnv).

Could the authors motivate why is this a relevant issue? How widespread is the practice of reconstructing individual outcomes from aggregate statistics in robotic benchmarks? Under what realistic circumstances would the proposed identification bounds and recovery analysis be necessary or useful?


Examples of clarity issues

The following are some excerpts that I found difficult to follow. They represent only a subset of the issues; I strongly encourage a complete revision of the manuscript.

Section 4.4

a separately implemented internal verifier checks target values and witnesses.

What exactly is a witness in this context? What does the "separately implemented internal verifier" do, and how does it relate to EFA-Replay? These terms are introduced without sufficient explanation at this point in the manuscript.

Section 4.4

Exact averages can coexist with missing structure. R1 and R2 preserve pooled success in all 78 cells, yet identify only 9 condition tables and 14 condition variances (Figure 6a). Thus grouping loss alone leaves variance non-identifiable in 64/78 cells, even before rounding enters the analysis.

This passage is unnecessarily difficult to interpret. Also: condition tables and condition variances receive little explanation in the main text, definitions deferred to Appendix B.3 (not directly referenced).

Section 3.3

Align and test. Each adapter emits a UnitRecord binding the chain in Eq. (1) to the recorded context and provenance. Shared checks detect substitutions, collisions, omissions, and altered accounting. The contracts cover initial-state validity, attempt isolation, scorer non-interference, and result-population accounting; the latter and attempt isolation govern our main cases. Matched repair comparisons b→b′ fix population, policy, unrelated operations, and randomness. Trajectory repeats and unchanged-scene controls support attribution; complete accounting supports record analysis.

This is a particularly important passage because it is supposed to explain the proposed methodology. However, it introduce numerous technical concepts in sequence without sufficiently explaining their concrete meaning or implementation. A simple example could help substantially.

Introduction

Consider an ACT evaluation with five workers: both original and repaired runs contain twenty distinct scenes and report 16/20 successes, yet four request labels change. (ACT: undefined acronym; also, this is assuming a high degree of familiarity with the paper's evaluation protocol in the second paragraph)

We call this property experimental-unit fidelity. The mechanisms act at different stages, but both weaken the link between a summary and its underlying units. Opposing label changes cancel when averaged; many-to-one transformations erase grouping.

Section 4.1

Pairing and analysis units. Labels are paired by requested identity within task, checkpoint, cohort, width, and noise stratum. The first formal execution supplies scores; the second verifies labels and changed-request identities. We run the pinned SimplerEnv helper on all 78 cells, preserving both resolutions of one conflicting record (Li et al., 2025; xuanlinli17, 2024), and analyze reductions of these saved outcomes post hoc. The summaries describe fixed, paired records with shared requests, repeated widths, and overlapping cells across the retained configurations (Appendices A–B).

This paragraph introduces several experimental details without sufficient context.

Section 4.6

The generating context determines the remedy. Shared state can require process-wide repair. A conservative chain expands 39 changed records to 51, and the supported boundary covers all 52, including two unscored tails. The remaining candidate is regenerated with the process. Missing historical identity calls for a different response: an aggregate may determine a statistic, but named attempts require their original binding. New collection instead defines a fresh evaluation population.

This passage is particularly difficult to interpret without reconstructing the experimental setup from earlier sections and appendices. Also: what is an "unscored tail"? What is the purpose of the last sentence?

Questions for authors

See weaknesses.

Reviewer_NXZ7OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This paper audits a number of existing simulated robot manipulation benchmarks, finds that there are various problems in the evaluation scripts of these benchmarks, and then proposes some statistical techniques for re-using prior reported results on these (demonstrated flawed) benchmarks. Concretely, the authors find bugs in the evaluation code of LeRobot, SimplerEnv, and LIBERO. For LeRobot, the authors find that LeRobot’s evaluation code does not correctly evaluate on for example 50 scenes when 50 scenes worth of evaluations are requested; instead only 26 distinct scenes are evaluated on. For SimplerEnv, the authors find that SimplerEnv reports only the mean success rate (this is not a bug, but it seems the authors are arguing here that SimplerEnv should report more than just the mean). And for LIBERO the authors find a bug where during evaluation at the beginning of the episode, policies can receive observations that are stale, effecting model performance. Then, the authors outline the following phenomena: (1) these benchmarks can be flawed but still reproducible, i.e., evaluating a policy again can yield the same result, and just because the benchmark is flawed does not mean it’s not reproducible, (2) fixing the flaws in the benchmark can result in the same or similar overall mean success rate, but the authors argue that despite this it is important to recognize that the benchmarks are still flawed, (3) just looking at the mean success rate often does not tell you about other important things, like whether a 50% score is due to 50% performance on each of the tasks, or 100% on half of them and 0% on the other half, and so more statistics other than the mean should be reported. Finally, after outlining these phenomena, the authors study ways to recover true information from prior flawed evals, and whether/how much information can be recovered at all given that in many cases only the mean success rate was reported.

Critical strengths

(1) The paper finds bugs in prior benchmarks like LeRobot and LIBERO, which is useful to the community to know about and fix. A broader takeaway from the fact that these bugs exist is that much care needs to be taken in benchmarks to make sure that the benchmark is testing what the authors of the benchmark intended for it to test.

(2) The authors also emphasize that more fine-grained metrics other than averaged success rate should be reported in benchmarks, which I agree with.

Critical weaknesses

(1) The writing of the paper is near incomprehensible, and it took substantial effort to parse the writing. I suspect that the reason for this incomprehensibility is the fact that the paper was, perhaps entirely, written by AI. AI written text is not necessarily bad by itself if the paper can be understood clearly and reads well, but this is not the case, and it is likely due to the AI that the paper reads as it does.



(2) The paper spends a lot of time explaining points that are quite obvious. For example, the paper emphasizes and repeats multiple times phenomena (1) and (2) (as listed in my paper summary), but these are quite obvious and not very consequential observations. Expanding more on (2), whether or not the mean success rate stays the same after the bug fix does not matter; what matters is whether the relative ranking of policies remains the same before and after the bug fix, as this tells us whether the bugs were misrepresenting policy performance in a significant way. However the authors to not study this.

(3) The actual contribution of the paper is minimal. The authors find bugs in prior benchmarks, which in my opinion is the main contribution of this paper, but finding bugs is not something I would say is sufficient of a contribution. The authors also study whether prior reported results on these buggy benchmarks can still be extracted to get some signal about true model performance, but it is unclear why the authors study this when it is very easy to just re-run the simulated evals with the bug fixes made.

(4) The authors only run experiments with ACT and SmolVLA policies, instead of the many prior policies evaluated on these benchmarks. It would have been more informative if benchmark results for these many policies were re-run to see if fixing the bugs had noticeable ranking order changes.

Questions for authors

(1) Why is it important to try to repair previous benchmark results run with buggy evaluation code, or to try to extract information from prior bug-present eval results when after fixing the bugs it’s very easy to just re-run the full evaluation?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.