- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper proposed JARVIS benchmark and JARVIS agent. JARVIS benchmark is built to thoroughly assess the current VLM agent's ability to generatign correct plan for the autonomous robotics LEGO assembly tasks as well as to accurately execute the plan and to recover from failure. JARVIS agent is also built to evaluate the execution ability of the VLM agents with their coding policy. The experimental verifies the effectiveness of the proposed benchmark, and the idea is overall solid, but several points still need to be answered and the method still need to be polished.
Critical strengths
- The proposed benchmark has filled substantial gap for exisiting long-horizon autonomous robotic assembly assessment benchmarks, as the eixisting ones only evaluate the planning ability, not include the actual policy execution. This is an important close-loop for the field.
- The JARVIS has strong reproducibility, and rigorously controlled difficulty level.
- The proposed definition of task completion is policy agnostic, which is a reasonable design.
Critical weaknesses
- For a long-horizon task, the overall system will get more and more organized as the task going on, therefore, the sub-task complexity will drop as time goes on. However, a simple task difficulty Di is a constant for each task, it alone cannot reflect this change. For example, an "easy" task with more sub-tasks might has the same difficulty as a "hard" task with less sub-tasks.
- The JARVIS suite does not have suites for real-world robot execution performance even though the agent can correctly finish the task in simulation, therefore, there is a potential gap that the proposed benchmark does not cover.
- The benchmark seems do not limit the same retry and token budgets for the tested agents.
- The JARVIS agent is more like a "code as policy" style pipeline, so the proposed method will not cover the end-to-end policy situation, which will limit the impact of the work.
- Since commercial VLM performs task decomposition, repeated evaluation is generally necessary as a single run does not distinguish model quality from sampling luck. I didn't see this in the paper, correct me if I was wrong.
Questions for authors
See weakness
6 comments
W3:
Thank you for raising this point. In the standard execution experiments, all models share the same budget: a limit of 20,480 tokens per generation, and either a single execution round (without reflection) or up to three rounds (with reflection), i.e., the initial execution followed by at most two corrective executions. Skills attempts are capped at three per skill invocation. We will state these settings explicitly in the revised manuscript.
W4:
Thank you for this valuable comment. We would like to respectfully clarify the distinction between the benchmark and the method. JARVIS-agentic is indeed a Code-as-Policy method, but this does not preclude incorporating end-to-end policies. Such policies can serve as callable skills, extending the tool library to support manipulation scenarios beyond those covered by the current primitives. The framework can therefore accommodate end-to-end control within a broader process of manual interpretation, task decomposition, verification, and recovery.
Besides, JARVIS benchmark scores assembly outcomes rather than requiring a particular policy interface. To demonstrate this, we evaluated two pretrained end-to-end VLA policies, π₀.₅-DROID and OpenVLA-7B, on the same 50 tasks with the same scoring, without fine-tuning, under two configurations: fixed language instructions and an external visual planner.
Table R1. Assembly scores (out of 100) of end-to-end VLA baselines on JARVIS.
| Model | Fixed language instructions | External visual planner |
|---|---|---|
| π₀.₅-DROID | 2.258 / 100 | 2.003 / 100 |
| OpenVLA-7B | 1.856 / 100 | 1.846 / 100 |
| ours | 14.71/100 (Model average) | 15.90/100 (GPT-OSS-120B) |
| Improvement | +12.45 (6.5×) | +13.90 (7.9×) |
Neither model fully completed any task in either configuration. These results show that end-to-end policies can be integrated and evaluated in JARVIS under the same tasks and scoring, and that their zero-shot performance on long-horizon, precision-critical assembly is currently very limited. We hope this broadens the impact of the benchmark: it offers a common testbed on which code-generating agents and end-to-end policies can be compared, and on which fine-tuned VLA policies can be studied in future work.
W5:
Thank you for raising this important point. The results reported in the paper are the median of three runs under identical settings, but we inadvertently omitted this statement. We will clarify this in the revised manuscript and provide the individual results for all three runs in Table R3.
Table R3. Overall score (out of 100) of each model over three independent runs, with generic and task-oriented tool libraries.
| Model | Tool library | Run 1 | Run 2 | Run 3 | Median | Range (max − min) |
|---|---|---|---|---|---|---|
| Gemini-3-Pro | Generic | 8.19 | 8.16 | 8.22 | 8.19 | 0.06 |
| Gemini-3-Pro | Task-oriented | 17.47 | 17.06 | 17.13 | 17.13 | 0.41 |
| Claude-Opus-4.5 | Generic | 13.77 | 13.86 | 13.83 | 13.83 | 0.09 |
| Claude-Opus-4.5 | Task-oriented | 17.04 | 17.05 | 16.97 | 17.04 | 0.08 |
| Claude-Haiku-4.5 | Generic | 5.88 | 5.93 | 5.95 | 5.93 | 0.07 |
| Claude-Haiku-4.5 | Task-oriented | 11.68 | 11.63 | 11.81 | 11.68 | 0.18 |
| DeepSeek-V4.1 | Generic | 3.01 | 8.49 | 8.51 | 8.49 | 5.50 |
| DeepSeek-V4.1 | Task-oriented | 17.39 | 17.41 | 17.33 | 17.39 | 0.08 |
| Qwen3-VL-235B | Generic | 8.27 | 7.21 | 8.35 | 8.27 | 1.14 |
| Qwen3-VL-235B | Task-oriented | 15.67 | 15.36 | 15.54 | 15.54 | 0.31 |
| Kimi-K2.5 | Generic | 4.31 | 5.91 | 5.58 | 5.58 | 1.60 |
| Kimi-K2.5 | Task-oriented | 11.07 | 11.36 | 11.21 | 11.21 | 0.29 |
| GPT-4o | Generic | 4.62 | 3.90 | 4.77 | 4.62 | 0.87 |
| GPT-4o | Task-oriented | 11.55 | 11.81 | 11.51 | 11.55 | 0.30 |
| GPT-5.1 | Generic | 7.63 | 8.03 | 7.38 | 7.63 | 0.65 |
| GPT-5.1 | Task-oriented | 15.90 | 14.99 | 14.81 | 14.99 | 1.09 |
| GPT-OSS-120B | Generic | 6.31 | 8.38 | 8.83 | 8.38 | 2.52 |
| GPT-OSS-120B | Task-oriented | 15.90 | 16.21 | 15.39 | 15.90 | 0.82 |
Dear Reviewer w1J2,
Thank you for your valuable feedback and for taking the time to review our work.
We hope that our rebuttal and additional experimental results have adequately addressed your concerns. We would be grateful if you could let us know whether you have any remaining questions or concerns.
Your further feedback before the rebuttal deadline would be greatly appreciated, as it would allow us to provide any necessary clarifications.
Thank you again for your time and consideration.
We thank the reviewer for the detailed evaluation and insightful comments. Please find our point-by-point responses below.
W1:
Thank you for this thoughtful comment. We would like to respectfully clarify that, in manual-guided assembly, sub-task difficulty does not generally decrease as the task progresses, and Di=α⋅Siscene+(1−α)⋅Siact (Eqs. 1–2) was designed with this in mind. The bricks-to-assemble term Siact receives the larger weight because errors propagate across steps and a longer horizon demands more state tracking and memory, while Siscene reflects that a larger existing structure leaves less free workspace and raises collision risk.
Our results support this design. Increasing the horizon from 1–2 to 3–5 bricks lowers the macro-average score from 40.01 to 18.29, and growing the initial structure from 0–29 to 79+ bricks lowers it from 40.01 to 16.64 (Table 3): a larger, more complete structure makes steps harder, not easier. The average score also falls from 20.62 (Easy) to 15.17 (Medium) and 8.35 (Hard) (Table 2).
W2:
Thank you for the suggestion. We agree that real-world execution is an essential long-term goal. At the current stage, we would respectfully suggest that simulation is a reasonable and well-suited choice for this benchmark. It follows common practice in embodied reasoning and planning benchmarks [1–5], enabling controlled, reproducible, and scalable evaluation of long-horizon tasks, which is hard to achieve with real robots across many models and tasks. In addition, the task is still far from solved in simulation: the best Overall score across nine models is only 17.39 out of 100 (Table 2). The current bottleneck therefore appears to lie in reasoning, grounding, and long-horizon execution, which simulation already exposes clearly.
To narrow the gap, our simulator models a standard Franka Panda arm with a parallel-jaw gripper and captures grasp failures, placement errors, and collision-induced structural changes (Appendix C). We also added perception-noise experiments to approximate real-world sensing uncertainty (Table R2): image noise lowers the mean score by less than one point (5–7%), and moving-target perturbation by 1.2–2.8 points (11–24%), providing preliminary evidence of robustness. Real-robot validation is planned as part of our upcoming open challenge and future work.
Table R2. Effect of perception noise on DeepSeek-driven JARVIS (scores out of 100).
We sampled two tasks from each difficulty tier of Benchmark-50 and ran them with three noise seeds under five conditions (clean plus the four noise conditions), 90 episodes in total. The table reports valid clean–noisy pairs matched on task and seed for DeepSeek-driven JARVIS (deepseek/deepseek-v4.1-flash-1017); scores include a time penalty. Image noise is zero-mean Gaussian on the RGB channels; target perturbation uses the listed position and rotation standard deviations, with the scoring ground truth unchanged.
References
[1] Srivastava et al. BEHAVIOR: Benchmark for Everyday Household Activities in Virtual, Interactive, and Ecological Environments. CoRL 2021.
[2] Li et al. BEHAVIOR-1K: A Benchmark for Embodied AI with 1,000 Everyday Activities and Realistic Simulation. CoRL 2022.
[3] Ge et al. BEHAVIOR Vision Suite: Customizable Dataset Generation via Simulation. CVPR 2024.
[4] Li et al. Embodied Agent Interface: Benchmarking LLMs for Embodied Decision Making. NeurIPS 2024.
[5] Yang et al. EmbodiedBench: Comprehensive Benchmarking Multi-modal Large Language Models for Vision-Driven Embodied Agents. ICML 2025.