acceptodds
Under review as a conference paper at ICLR 2027

ROBOLEAK: CLAIM-CONDITIONED OVERLAP AUDITING FOR EMBODIED AI BENCHMARKS

Abstract

Robot-learning benchmarks often reuse objects, scenes, and skills across training and evaluation to probe transfer under controlled changes. Yet a benchmark score alone does not reveal which aspects of the evaluated behavior are novel relative to the training collection. We introduce ROBOLEAK, a claim-conditioned auditing framework that first records cross-boundary relations at five levels—stored files, decoded observations, near-duplicate views, object and scene structure, and task semantics—and then applies a declared evaluation contract specifying which observed relations violate the benchmark’s intended novelty. Auditing LIBERO with ROBOLEAK shows that all seven Goal-task object models already appear in the training suites, while no scene composition is repeated; under a strict novelty rule a single familiar task is excluded, and this one exclusion raises SmolVLA’s score from 76% to 80%, leaves X-VLA unchanged at 100%, and slightly lowers VLA-JEPA’s score. The same cleaning moves different models in opposite directions because it removes a task each model finds differently hard. A complementary controlled exposure study finds no consistent average gain from exact-task demon- strations, while revealing substantial seed-level variation under a fixed training budget. ROBOLEAK turns a benchmark score into an answerable question: what was shared, what was required to be new, and which tasks does the number average over?

Primary area:
applications to robotics, autonomy, planning
Keywords:
Embodied AI, Robot Learning, Dataset Overlap, Benchmark Contamination, Train–Test Leakage, Benchmark Auditing, Generalization Evaluation, Data Deduplication

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_nvk9OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents a claim-conditioned framework to reduce or mitigate the visual or skill reuse problem in evaluation. It characterizes overlap at five levels, ranging from exact file duplication to object, scene, and task-semantic reuse, and introduces evaluation contracts that determine whether observed overlaps violate a benchmark's intended generalization claim. Experiments on LIBERO demonstrate that different novelty definitions lead to substantially different retained evaluation sets, while equal-count subset comparisons show that benchmark cleaning can shift model scores due to task selection rather than changes in model capability.

Critical strengths
  1. Comparing to traditional embodied ai evaluation, this paper offer a hierachical frame to reduce reuse factors like visual items, skills, scene composition/layout, which can reflect the real capability of VLA on evaluation tasks.

  2. In this paper, authors introduce equal-count controls to provide a more principled statistical comparison of benchmark scores after data cleaning. By comparing the cleaned evaluation set against alternative subsets of the same size, the method accounts for changes in task count and helps contextualize score shifts caused by differences in task composition and difficulty.

Critical weaknesses

1. Limited Scope of Evidence

The main audit focuses on a single LIBERO training–evaluation boundary, while CALVIN and DROID-100 receive only limited analysis. Moreover, the controlled exposure experiment in Sec. 5.5 considers only fine-tuning with pretrained models, leaving the impact of overlap on models trained from scratch or with weaker priors unexplored.

2. Coarse-grained Task-level Exclusion

Sec. 3.2 removes entire tasks whenever any overlap violates the evaluation contract. This coarse-grained approach may unnecessarily discard valid evaluation signals, particularly in complex long-horizon tasks containing both overlapping and novel subtasks.

3. Heavy Reliance on Human Review

ROBOLEAK relies heavily on manual verification for visual overlap, task semantics, and contract construction, limiting scalability to large-scale datasets. The reported low initial agreement (33.3%) in L4 rule formulation further raises concerns about annotation subjectivity and reproducibility.

4. Insufficient Disentanglement of Generalization Dimensions

Different transfer settings require different novelty criteria, yet the retained tasks vary substantially across contracts. This sensitivity raises concerns about contract reliability and whether detected overlap reflects actual evaluation contamination or insufficient disentanglement of generalization dimensions in benchmark design.

** 5. Figure quality (Minor)**

Text and visual item overlap in Fig 2.

Questions for authors
  1. Can the authors provide more comprehensive evaluations on recent and diverse embodied AI benchmarks to demonstrate the generalizability of ROBOLEAK beyond the LIBERO setting?

  2. How would the conclusions from Sec. 5.5 change for models trained from scratch (denote to visual alignment) or with weaker pretrained priors? Could pretraining knowledge mask the actual impact of data overlap?

  3. Why does ROBOLEAK exclude entire tasks rather than individual subtasks or overlapping components? Could a finer-grained exclusion strategy preserve more valid evaluation signals, especially for long-horizon tasks?

  4. How can the authors ensure that evaluation contracts consistently reflect the intended generalization objectives rather than subjective interpretations? Would the conclusions remain stable under different reasonable contract definitions?

Reviewer_sBcMOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes RoboLeak for checking whether robot benchmarks reuse content from training data, and whether that reuse actually matters. Instead of treating all overlap as cheating, it defines a multi-level relation inventory covering identical files, identical decoded observations, near-duplicate visual views, shared objects or scene structure, and related task semantics. Confirmed matches are converted into decisions through an evaluation contract, which specifies which shared relations conflict with the novelty required by the test. The paper audits LIBERO and briefly examines CALVIN and a small SmolVLA training study. It finds that overlap depends heavily on the chosen rule, and that removing a task can raise or lower a score depending on how hard that task was. Overall, RoboLeak offers a clearer way to report and interpret reuse in robot benchmarks.

Critical strengths

This paper targets at addressing the benchmark overlap issue in embodied-AI evaluation. In many existing discussions of benchmark contamination, any detected overlap is implicitly treated as problematic. RoboLeak instead argues that the relevance of an overlap depends on what the benchmark claims to hold out. The multi-level taxonomy also provides a useful vocabulary for future benchmark analysis.

Critical weaknesses

The main technical contribution appears relatively limited, and a substantial portion of RoboLeak relies on manually defined or manually reviewed relations. Most individual components are relatively standard: byte-level comparison, decoded-pixel matching, DINOv2 retrieval, object identifiers, manually structured task semantics, and subset-based score analysis. The main novelty lies in organizing these components around an evaluation contract rather than introducing a substantially new detection methodology. Meanwhile, these components relies heavily on hand-crafted designs and predefined instructions or thresholds. This raises concerns about scalability and subjectivity when moving from the relatively small and structured LIBERO benchmark to larger embodied datasets.

Questions for authors

1.How sensitive are the final retained task sets to the construction of the evaluation contract? The large difference in retained rate (Figure 2b) suggests that the final conclusion may depend heavily on how the benchmark claim is translated into a semantic rule. Could the authors provide a sensitivity analysis showing how alternative reasonable interpretations of the same benchmark claim affect the final conclusions?

2.Several stages currently involve manual inspection or manually assigned semantic structures. How would RoboLeak scale to benchmarks containing thousands or millions of episodes and much less structured task descriptions?

3.The authors correctly clarify that zero accepted pairs among top-k candidates does not imply zero duplicates globally. However, this also makes it difficult to interpret a negative result. How sensitive are the conclusions to the number of retrieved candidates, sampling density, and choice of visual encoder?

Reviewer_MpUrOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper audits the overlap between training and evaluation data in embodied benchmarks. It uses evaluation contracts to decide which overlaps require excluding tasks, then compares the resulting scores against sized task subsets. Experiments mainly use LIBERO, with additional analyses on CALVIN and controlled task exposure.

Critical strengths
  1. The paper correctly emphasizes that overlap should be interpreted relative to the benchmark's intended generalization goal.
  2. The subset comparisons clearly illustrate how task removal can change scores without changing model capabilities
Critical weaknesses
  1. X-VLA and VLA-JEPA already achieve 100% and 98% on the evaluated LIBERO suite. This ceiling effect limits how informative these models are for validating the approach
  2. Limited contribution beyond task filtering. The approach largely filters evaluation tasks and recomputes scores. Its added value beyond ordinary subset analysis needs to be demonstrated more clearly
  3. Unclear criteria for removing tasks: Different rules remove very different numbers of tasks, so the choice of rule needs better justification.
  4. Limited validation through VLA performance. Score changes after filtering reflect the difficulty of the removed tasks and do not independently validate the filtering decisions.
Questions for authors

What evidence supports the chosen exclusion rules, and how do they distinguish invalid overlap from intended skill transfer? Beyond changes in average scores, what evidence shows that the filtered evaluation better measures the intended generalization ability? Other concerns are under the weakness section.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.