- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper proposes Qoboter, a robot-centric scaffold for VLM-based manipulation, consisting of three components, including Loop, Tools, and Skills. The VLM plans task-level subgoals, observes the environment, selects and executes actions through tool calls, and adjusts its decisions based on feedback. Experiments across four simulators and a real robot demonstrate improved success rates over existing coding scaffolds, with additional cross-model evaluations and ablation studies.
Critical strengths
- This work focuses on the design of Coding Scaffolds for robotic manipulation, which is a relatively new research direction for both robotics and Coding Scaffold-based agents. The authors achieve promising results, demonstrating that the robot can adjust its actions and correct its mistakes in response to environmental feedback.
- The authors organize their framework into three levels: Loop, Tools, and Skills. This provides a clear separation of responsibilities and makes the overall system architecture easier to understand and analyze. (I have some concerns about the description and motivation of these components, which I will discuss later.)
Critical weaknesses
- Admittedly, this paper appears to achieve quite good results (based on the anonymous video link provided by the authors), but it's overshadowed by low quality writing. The authors show their design intentions and ideas, but do not explain the details of their work in sufficient depth. At the very least, a pipeline example such as Figure 8 should be clearly explained in the main text, even if it is only a demo (ideally covering a broader range of situations or examples). After reading the main text, I do not think readers can clearly understand how the authors actually implemented this work.
- The story showed by the authors feels rather forced. In particular, the authors take certain typical characteristics of the environments in which Coding Agents operate and generalize them into inherent assumptions of Coding Scaffolds such as Codex and Claude Code. They then use argument that these assumptions do not hold in robotic environments to motivate the three design components of Qoboter. But the authors don't sufficiently justify whether these three assumptions are valid in the first place. For example, the authors claim that Coding Scaffolds rely on the reversibility of actions, but Codex can execute shell commands (e.g., sudo rm -rf ~/), and such operations are not necessarily reversible. The use of "rely on" in the abstract may also be inaccurate or overly absolute, or there may even be a logical flaw in this argument. Furthermore, the authors do not establish a clear causal relationship between these assumptions and the observed failures of the baselines in their experiments.
- The authors should pay more attention to details, including but not limited to reporting the specific version numbers of Codex and Claude Code, as both are continuously updated. In Figure 2, the legend for "Loop" does not match the actual diagram. Several layers use boxes that appear to correspond to the "Loop" legend, but all of them have dashed borders, and there is not a single white box with a solid border as indicated in the legend. Additionally, there appears to be a missing arrow below "RoboDojo · insert tubes", or is there a missing connection in the flow?
Questions for authors
- Could the authors provide a more detailed explanation of their pipeline and include it in the main text?
- I would like to discuss with the authors the claimed shortcomings of Coding Scaffolds (codex or cluade code, maybe) in embodied AI about their claim. At least for now, I find the authors' story insufficiently rigorous. It's a major motivation for this paper, I think it a relatively important concern. I cannot be entirely sure that my interpretation is correct.
- The authors should pay more attention to their figures. In my opinion, most of them need revision. For example, Figure 3 still shows noticeable signs of AI generation, raising concerns that the use of AI may go beyond what is disclosed in the "AI USE STATEMENT."
- It would be helpful if the authors could provide some comparisons with methods based on other paradigms, such as VLA-based approaches. I am interested in understanding the performance differences between these paradigms on the same tasks. (just a suggestion)