- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper presents a claim-conditioned framework to reduce or mitigate the visual or skill reuse problem in evaluation. It characterizes overlap at five levels, ranging from exact file duplication to object, scene, and task-semantic reuse, and introduces evaluation contracts that determine whether observed overlaps violate a benchmark's intended generalization claim. Experiments on LIBERO demonstrate that different novelty definitions lead to substantially different retained evaluation sets, while equal-count subset comparisons show that benchmark cleaning can shift model scores due to task selection rather than changes in model capability.
Critical strengths
-
Comparing to traditional embodied ai evaluation, this paper offer a hierachical frame to reduce reuse factors like visual items, skills, scene composition/layout, which can reflect the real capability of VLA on evaluation tasks.
-
In this paper, authors introduce equal-count controls to provide a more principled statistical comparison of benchmark scores after data cleaning. By comparing the cleaned evaluation set against alternative subsets of the same size, the method accounts for changes in task count and helps contextualize score shifts caused by differences in task composition and difficulty.
Critical weaknesses
1. Limited Scope of Evidence
The main audit focuses on a single LIBERO training–evaluation boundary, while CALVIN and DROID-100 receive only limited analysis. Moreover, the controlled exposure experiment in Sec. 5.5 considers only fine-tuning with pretrained models, leaving the impact of overlap on models trained from scratch or with weaker priors unexplored.
2. Coarse-grained Task-level Exclusion
Sec. 3.2 removes entire tasks whenever any overlap violates the evaluation contract. This coarse-grained approach may unnecessarily discard valid evaluation signals, particularly in complex long-horizon tasks containing both overlapping and novel subtasks.
3. Heavy Reliance on Human Review
ROBOLEAK relies heavily on manual verification for visual overlap, task semantics, and contract construction, limiting scalability to large-scale datasets. The reported low initial agreement (33.3%) in L4 rule formulation further raises concerns about annotation subjectivity and reproducibility.
4. Insufficient Disentanglement of Generalization Dimensions
Different transfer settings require different novelty criteria, yet the retained tasks vary substantially across contracts. This sensitivity raises concerns about contract reliability and whether detected overlap reflects actual evaluation contamination or insufficient disentanglement of generalization dimensions in benchmark design.
** 5. Figure quality (Minor)**
Text and visual item overlap in Fig 2.
Questions for authors
-
Can the authors provide more comprehensive evaluations on recent and diverse embodied AI benchmarks to demonstrate the generalizability of ROBOLEAK beyond the LIBERO setting?
-
How would the conclusions from Sec. 5.5 change for models trained from scratch (denote to visual alignment) or with weaker pretrained priors? Could pretraining knowledge mask the actual impact of data overlap?
-
Why does ROBOLEAK exclude entire tasks rather than individual subtasks or overlapping components? Could a finer-grained exclusion strategy preserve more valid evaluation signals, especially for long-horizon tasks?
-
How can the authors ensure that evaluation contracts consistently reflect the intended generalization objectives rather than subjective interpretations? Would the conclusions remain stable under different reasonable contract definitions?