- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes a benchmark for long horizon manipulation with a focus on tasks which require specific types of memory.
Critical strengths
Overall, the paper is well written and easy to follow. The fact that successful demonstrations makes this benchmark easier to adopt and. Real-world experiments are always good to see.
Critical weaknesses
The position of this paper is in a slightly odd place, if the focus is on the memory and reasoning, then there are major omissions of the literature, if the focus is on the control, then the evaluation is somewhat lacking.
For reasoning, PARTNR [1] or ALFRED [2] (which are omitted from discussion) are highly relevant. PARTNR test similar reasoning at a much larger scale (around 100k tasks). With only nine tasks, and visually hidden attributes appearing mainly in SweetCup and TempReport, it isn't clear to me what this benchmark adds beyond these. PARTNR can also be run with low-level control, so building on RoboCasa doesn't by itself set this work apart. This paper would benefit discussing how this work should be viewed in relation to these existing benchmarks.
If the focus is control, then some demonstration of sim-to-real transfer is needed to show that the simulation is high enough fidelity for benchmark scores to correlate with real-world performance. The choice to perform a real-world task which is not one of the benchmark tasks undercuts the value of the benchmark. Real-world versions of even two or three benchmark tasks, with the method ranking compared against Table 3, would make the paper stronger.
Also, the real-world results come from 10 trials per method, and ACT and DP were trained but aren't reported in Table 4. Confidence intervals following Agarwal et al. [3] would help here.
Finally, the average SR improvement over π0.5 comes almost entirely from TempReport. This almost makes it seem like this task is an outlier or pathological in some way. This paper would benefit from more tasks so the results can better differentiate methods.
Questions for authors
"let Nj denote the number of valid evaluation episodes" What exactly makes an evaluation episode invalid?
"All methods are trained on the same task demonstrations." How are ACT and DP conditioned on the language instruction?