- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper argues that a repeatable robot-benchmark score can still describe a different experiment from the one declared, a property it calls experimental-unit fidelity. It identifies two failure mechanisms. At the audited LeRobot revision, per-slot seeds receive a second worker-index offset, so request s+i runs scene s+i+(i mod B). In a five-worker ACT run, scene count (20 distinct) and success (16/20) are identical before and after repair, yet 4 request labels change, and in controlled SmolVLA 9/64 labels change while success moves by only −1.56 pp. On the reporting side, the SimplerEnv helper rebuilds a rounded binary pseudo-trial vector from aggregate success, so pseudo-trial variance exceeds condition-mean variance in 64 of 78 cells.
The authors propose EFA-Replay, which aligns declared units with executed records, tests evaluator changes with matched controls and paired-label accounting, and uses sharp identification ranges over compatible records plus dependency-based invalidation to decide whether to recompute, reuse, replay or leave an evidence gap. They also report a LIBERO audit that flags a stale first observation before scoring, where correcting it changes ACT task-8 success from 11/30 to 27/30. Supporting results include a small diagnostic suite (6 deviations, 5 legal configurations, 1 insufficient input), component removals and archived-replay case studies.
Critical strengths
-
The central observation is correct, concrete and checkable. Eq. 2 cleanly separates net score change from changed-label fraction. Dlabels − |Δ| = 2·min(n01, n10)/N is exactly the disagreement that averaging cancels, and the ACT example (identical 20 scenes and 16/20, four changed labels) makes the point vividly. The seed-offset mechanism is stated precisely enough to reproduce, and the controls are sensible: all 128 same-arm and 32 unchanged-scene controls pass, the execution is repeated, and a frozen forecast drops errors from 27/80 to 3/80 once the correct effective-scene ledger is used.
-
The reporting analysis uses partial identification in a principled way. Distinguishing what a retained record determines (pooled success, 78/78) from what it does not (condition table 9/78, variance 14/78) is a useful framing. Attainable bounds with witness records are provided, and the paper separates structural loss (Eq. 3) from rounding effects. Calling out the 64/78 variance inflation, while also reporting the one decrease and the 13 ties, is honest.
-
The paper is careful about scope and reports negative or limiting cases. It retains two descriptive exclusions, reports a non-official LIBERO adapter, notes that 8 of 10 planned comparisons pass the controls, and states that the second LIBERO check is separate from the exploratory one. The LIBERO timing result (stale vs. current observation, 11/30 → 27/30 with 17 vs. 1 discordant) is large and independently motivated.
Critical weaknesses
-
The problems found are narrow, and the generality claim is not supported. The execution failure is a seed-offset bug in one library revision, and the reporting failure is the behavior of one released SimplerEnv helper. The paper does not show that either appears in published results, other harnesses or later revisions, and it does not report whether maintainers confirmed them. The abstract and conclusion present these as general properties of robot benchmarking, but the evidence is two library behaviors plus one timing mismatch in LIBERO. The SimplerEnv finding concerns the variance of a reconstructed vector, while the headline mean is unchanged by construction, so it is unclear which published conclusion it alters. The fixed-checkpoint shifts (−12 to +16 pp) are on 50 requests per arm with no uncertainty, and the paper itself says winner identity is preserved in the first cohort.
-
Statistical support is thin, and key claims rest on small, post hoc or single-checkpoint evidence. Most controlled results have n = 20–64 paired requests, with no confidence intervals or tests for the 9/64 and 9/20 disagreements. The LIBERO result is a single public ACT checkpoint on one task, run through a native adapter that the paper says does not give official scores. The SimplerEnv analysis and the scene-reweighting decomposition are explicitly post hoc, and the cells in Figures 3–5 share scenes, widths and noise strata, so counts such as 64/78 and 52/66 overstate independence. The forecast improvement (27/80 → 3/80) is computed on 80 comparisons that share only 40 task/scene IDs.
-
The EFA-Replay evaluation is author-designed, and the added value over standard tools is unclear. The diagnostic suite (6 deviations, 5 legal controls, 1 insufficient input) and its presets were built by the authors, EFA-Replay merely ties the “paired oracle”, and the baselines (B0 repeat/total, B1 nominal IDs) are weak by design. There is no held-out or blind set of deviations and no comparison with existing audit tools such as Rollout Cards, ABE-Ralph or metamorphic testing. The recovery decisions and replay case studies are demonstrations on archived data where the plan is derived from the same evidence. Several results restate known identities: Eq. 3 is the standard variance relation and the Eq. 6 bounds follow from convexity, so the sharp ranges mostly quantify a known information loss. The per-benchmark effort of writing adapters, declarations and dependency graphs is not measured.
Questions for authors
-
Can you show that these failures matter beyond the audited code paths? For example: were the LeRobot seed-offset issue and the SimplerEnv pseudo-trial behavior confirmed by maintainers, do they exist in current revisions, and do they affect any published number or ranking (for example, a leaderboard or a paper that used these tools)? The paper says winner identity was preserved in the first cohort. Is there any case where a conclusion actually flips?
-
Can you add uncertainty to the key quantities? Specifically, confidence intervals or paired tests for the label-change fractions and fixed-checkpoint shifts, and for the LIBERO 11/30 → 27/30 result across more checkpoints, tasks and suites. How many of the eight control-admitted contrasts remain meaningful once the n = 50 sampling error is taken into account, and how do the 64/78 variance results look when cells sharing scenes and widths are treated as dependent?
-
How would you evaluate EFA-Replay against an external benchmark of deviations that you did not design? For example: a blind set of injected or naturally occurring evaluator faults from other harnesses, comparison with an existing audit tool or a metamorphic-testing baseline, and a measure of adapter effort and false-flag rate on benchmarks it was not built for. Please also clarify which components (pairing, bounds, dependency closure) are new relative to standard provenance and partial-identification tools.