- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper proposes DREAM-CAL, a decision-regret-based framework for determining when a robot should refresh visual observations instead of relying on world-model predictions. The method policy-conditioned decision regret from paired continuations and selects a refresh horizon using a block-calibrated upper bound. However, this paper lacks substantial experiments on broader comparisons against other baselines and open benchmarks, as well as suspicious real-world experiments.
Critical strengths
The paper addresses a problem in WAM and reports some improvements in its own evaluation set.
Critical weaknesses
- Lack of standard benchmarks and comprehensive comparisons.
The paper evaluates 37 custom tasks (Section 5.1), without reporting results on established public manipulation benchmarks such as RoboTwin, CALVIN, or a reproducible LIBERO-based evaluation suite. More importantly, the experiments only compare against their internal ablation models, without direct comparisons to ANY existing published methods, making it difficult to evaluate the results. - Insufficient evidence and documentation of real-robot experiments.
Appendix I claims real-robot experiments on Franka, but provides no photographs, execution videos, or qualitative visualizations of the actual robot experiments. The hardware and evaluation protocols are also insufficiently documented for independent verification. Figure 11 only presents aggregate statistics, without demonstrating the experimental execution. Given the importance of real-world validation, the lack of supporting experimental evidence is a serious concern.
Questions for authors
See weakness