- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper evaluates Fable 5 and GPT-6 Astra for robot control and demonstration generation, compares human-only and mixed human–model data for training π0.5, studies hierarchical and multi-aspect language supervision on RoboCasa 365 and BEHAVIOR-1K, and combines some of these components in a robotic crossword task.
Critical strengths
Table 2 provides clear ablations, with the combined supervision achieving the best performance across both benchmarks and instructor settings. The 27,113 primitive action segments(line 303) could provide a useful annotation resource for the community if publicly released. Section 3.1 provides real-world evidence for frontier-model control and the benefit of corrective demonstration segments. The paper explores a timely research direction: connecting frontier-model reasoning with robot execution.
Critical weaknesses
The paper compares success rates between policies trained on 20 human demonstrations plus 20 generated demonstrations and those trained on 40 human demonstrations. However, could 20 human demonstrations already be sufficient? If they bring the policy close to its current performance ceiling, adding either type of data may make little difference. The existing results do not rule out this possibility. Without a 20-human-demonstration baseline or a comparison between 20 generated and 20 human demonstrations, the value of autonomously generated successful trajectories remains unclear.
The paper presents several experimental components, but their connections and the motivation for combining them require stronger experimental support and clearer explanation. For example, it does not explicitly report the total number of training demonstrations for the 6×6 crossword task or the proportion of human versus autonomously generated demonstrations. If this proportion differs from the 1:1 mixture of 20 human-teleoperated demonstrations and 20 GPT-6 Astra-generated trajectories, why was the 1:1 ratio specifically chosen as the only mixture evaluated in the earlier experiment (lines 235–236)?
The core ideas behind both supervision methods have substantial precedent in prior work. The methodological contribution therefore appears to lie primarily in the construction, combination, and empirical evaluation of the three-level annotations. The paper should more clearly articulate its additional contributions beyond existing hierarchical and multi-aspect supervision methods.
Questions for authors
What is the total number of training demonstrations for the 6×6 crossword task, and what is the ratio of human to autonomously generated demonstrations?