acceptodds
Under review as a conference paper at ICLR 2027

JARVIS: From Visual Manuals to Autonomous Robotic Assembly

Abstract

Visual manuals communicate procedural knowledge through stepwise diagrams, but translating these instructions into reliable long-horizon robotic execution remains challenging. We introduce JARVIS, a method-agnostic benchmark for autonomous visual-manual-guided execution without additional language instructions. JARVIS comprises 271,879 LEGO assembly tasks across 8,960 scenes, with a carefully curated JARVIS-v1 suite of 50 tasks for interactive robotic evaluation. Tasks span varying initial structural complexities and assembly horizons, requiring agents to interpret manuals, ground actions in an evolving workspace, and execute the intended procedure. A structure-aware evaluation protocol captures geometric accuracy, structural relations, construction order, partial progress, and completion. We further introduce JARVIS-agentic, a tool-grounded embodied agent that composes robot primitives into executable programs and uses execution feedback for verification and recovery, without task-specific action trajectories or model parameter updates. Across nine multimodal and reasoning models, task-oriented primitives increase the average assembly score. Performance nevertheless declines sharply with assembly horizon, and execution errors remain difficult to correct. Prior-task experience improves initial performance, while repeated reflection provides inconsistent benefits, highlighting persistent limitations in connecting visual procedural reasoning with reliable physical execution.

Primary area:
datasets and benchmarks
Keywords:
Visual Manual Understanding, Long-Horizon Manipulation, Multimodal Reasoning, Robotic Assembly, Agentic Systems

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_w1J2OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposed JARVIS benchmark and JARVIS agent. JARVIS benchmark is built to thoroughly assess the current VLM agent's ability to generatign correct plan for the autonomous robotics LEGO assembly tasks as well as to accurately execute the plan and to recover from failure. JARVIS agent is also built to evaluate the execution ability of the VLM agents with their coding policy. The experimental verifies the effectiveness of the proposed benchmark, and the idea is overall solid, but several points still need to be answered and the method still need to be polished.

Critical strengths
  1. The proposed benchmark has filled substantial gap for exisiting long-horizon autonomous robotic assembly assessment benchmarks, as the eixisting ones only evaluate the planning ability, not include the actual policy execution. This is an important close-loop for the field.
  2. The JARVIS has strong reproducibility, and rigorously controlled difficulty level.
  3. The proposed definition of task completion is policy agnostic, which is a reasonable design.
Critical weaknesses
  1. For a long-horizon task, the overall system will get more and more organized as the task going on, therefore, the sub-task complexity will drop as time goes on. However, a simple task difficulty Di is a constant for each task, it alone cannot reflect this change. For example, an "easy" task with more sub-tasks might has the same difficulty as a "hard" task with less sub-tasks.
  2. The JARVIS suite does not have suites for real-world robot execution performance even though the agent can correctly finish the task in simulation, therefore, there is a potential gap that the proposed benchmark does not cover.
  3. The benchmark seems do not limit the same retry and token budgets for the tested agents.
  4. The JARVIS agent is more like a "code as policy" style pipeline, so the proposed method will not cover the end-to-end policy situation, which will limit the impact of the work.
  5. Since commercial VLM performs task decomposition, repeated evaluation is generally necessary as a single run does not distinguish model quality from sampling luck. I didn't see this in the paper, correct me if I was wrong.
Questions for authors

See weakness

6 comments

We thank the reviewer for the detailed evaluation and insightful comments. Please find our point-by-point responses below.

W1:

Thank you for this thoughtful comment. We would like to respectfully clarify that, in manual-guided assembly, sub-task difficulty does not generally decrease as the task progresses, and Di=α⋅Siscene+(1−α)⋅SiactD_i = \alpha \cdot S_i^{\text{scene}} + (1 - \alpha) \cdot S_i^{\text{act}} (Eqs. 1–2) was designed with this in mind. The bricks-to-assemble term SiactS_i^{\text{act}} receives the larger weight because errors propagate across steps and a longer horizon demands more state tracking and memory, while SisceneS_i^{\text{scene}} reflects that a larger existing structure leaves less free workspace and raises collision risk.

Our results support this design. Increasing the horizon from 1–2 to 3–5 bricks lowers the macro-average score from 40.01 to 18.29, and growing the initial structure from 0–29 to 79+ bricks lowers it from 40.01 to 16.64 (Table 3): a larger, more complete structure makes steps harder, not easier. The average score also falls from 20.62 (Easy) to 15.17 (Medium) and 8.35 (Hard) (Table 2).

W2:

Thank you for the suggestion. We agree that real-world execution is an essential long-term goal. At the current stage, we would respectfully suggest that simulation is a reasonable and well-suited choice for this benchmark. It follows common practice in embodied reasoning and planning benchmarks [1–5], enabling controlled, reproducible, and scalable evaluation of long-horizon tasks, which is hard to achieve with real robots across many models and tasks. In addition, the task is still far from solved in simulation: the best Overall score across nine models is only 17.39 out of 100 (Table 2). The current bottleneck therefore appears to lie in reasoning, grounding, and long-horizon execution, which simulation already exposes clearly.

To narrow the gap, our simulator models a standard Franka Panda arm with a parallel-jaw gripper and captures grasp failures, placement errors, and collision-induced structural changes (Appendix C). We also added perception-noise experiments to approximate real-world sensing uncertainty (Table R2): image noise lowers the mean score by less than one point (5–7%), and moving-target perturbation by 1.2–2.8 points (11–24%), providing preliminary evidence of robustness. Real-robot validation is planned as part of our upcoming open challenge and future work.

Table R2. Effect of perception noise on DeepSeek-driven JARVIS (scores out of 100).

Noise conditionCleanNoisyΔ
Light image noise (σ = 8)10.97410.441−0.533
Heavy image noise (σ = 20)10.97410.225−0.749
Light moving-target perturbation (1 mm, 0.5°)11.5748.792−2.782
Heavy moving-target perturbation (3 mm, 2°)10.9749.759−1.216

We sampled two tasks from each difficulty tier of Benchmark-50 and ran them with three noise seeds under five conditions (clean plus the four noise conditions), 90 episodes in total. The table reports valid clean–noisy pairs matched on task and seed for DeepSeek-driven JARVIS (deepseek/deepseek-v4.1-flash-1017); scores include a time penalty. Image noise is zero-mean Gaussian on the RGB channels; target perturbation uses the listed position and rotation standard deviations, with the scoring ground truth unchanged.

References

[1] Srivastava et al. BEHAVIOR: Benchmark for Everyday Household Activities in Virtual, Interactive, and Ecological Environments. CoRL 2021.

[2] Li et al. BEHAVIOR-1K: A Benchmark for Embodied AI with 1,000 Everyday Activities and Realistic Simulation. CoRL 2022.

[3] Ge et al. BEHAVIOR Vision Suite: Customizable Dataset Generation via Simulation. CVPR 2024.

[4] Li et al. Embodied Agent Interface: Benchmarking LLMs for Embodied Decision Making. NeurIPS 2024.

[5] Yang et al. EmbodiedBench: Comprehensive Benchmarking Multi-modal Large Language Models for Vision-Driven Embodied Agents. ICML 2025.

W3:

Thank you for raising this point. In the standard execution experiments, all models share the same budget: a limit of 20,480 tokens per generation, and either a single execution round (without reflection) or up to three rounds (with reflection), i.e., the initial execution followed by at most two corrective executions. Skills attempts are capped at three per skill invocation. We will state these settings explicitly in the revised manuscript.

W4:

Thank you for this valuable comment. We would like to respectfully clarify the distinction between the benchmark and the method. JARVIS-agentic is indeed a Code-as-Policy method, but this does not preclude incorporating end-to-end policies. Such policies can serve as callable skills, extending the tool library to support manipulation scenarios beyond those covered by the current primitives. The framework can therefore accommodate end-to-end control within a broader process of manual interpretation, task decomposition, verification, and recovery.

Besides, JARVIS benchmark scores assembly outcomes rather than requiring a particular policy interface. To demonstrate this, we evaluated two pretrained end-to-end VLA policies, π₀.₅-DROID and OpenVLA-7B, on the same 50 tasks with the same scoring, without fine-tuning, under two configurations: fixed language instructions and an external visual planner.

Table R1. Assembly scores (out of 100) of end-to-end VLA baselines on JARVIS.

ModelFixed language instructionsExternal visual planner
π₀.₅-DROID2.258 / 1002.003 / 100
OpenVLA-7B1.856 / 1001.846 / 100
ours14.71/100 (Model average)15.90/100 (GPT-OSS-120B)
Improvement+12.45 (6.5×)+13.90 (7.9×)

Neither model fully completed any task in either configuration. These results show that end-to-end policies can be integrated and evaluated in JARVIS under the same tasks and scoring, and that their zero-shot performance on long-horizon, precision-critical assembly is currently very limited. We hope this broadens the impact of the benchmark: it offers a common testbed on which code-generating agents and end-to-end policies can be compared, and on which fine-tuned VLA policies can be studied in future work.

W5:

Thank you for raising this important point. The results reported in the paper are the median of three runs under identical settings, but we inadvertently omitted this statement. We will clarify this in the revised manuscript and provide the individual results for all three runs in Table R3.

Table R3. Overall score (out of 100) of each model over three independent runs, with generic and task-oriented tool libraries.

ModelTool libraryRun 1Run 2Run 3MedianRange (max − min)
Gemini-3-ProGeneric8.198.168.228.190.06
Gemini-3-ProTask-oriented17.4717.0617.1317.130.41
Claude-Opus-4.5Generic13.7713.8613.8313.830.09
Claude-Opus-4.5Task-oriented17.0417.0516.9717.040.08
Claude-Haiku-4.5Generic5.885.935.955.930.07
Claude-Haiku-4.5Task-oriented11.6811.6311.8111.680.18
DeepSeek-V4.1Generic3.018.498.518.495.50
DeepSeek-V4.1Task-oriented17.3917.4117.3317.390.08
Qwen3-VL-235BGeneric8.277.218.358.271.14
Qwen3-VL-235BTask-oriented15.6715.3615.5415.540.31
Kimi-K2.5Generic4.315.915.585.581.60
Kimi-K2.5Task-oriented11.0711.3611.2111.210.29
GPT-4oGeneric4.623.904.774.620.87
GPT-4oTask-oriented11.5511.8111.5111.550.30
GPT-5.1Generic7.638.037.387.630.65
GPT-5.1Task-oriented15.9014.9914.8114.991.09
GPT-OSS-120BGeneric6.318.388.838.382.52
GPT-OSS-120BTask-oriented15.9016.2115.3915.900.82

Dear Reviewer w1J2,

Thank you for your valuable feedback and for taking the time to review our work.

We hope that our rebuttal and additional experimental results have adequately addressed your concerns. We would be grateful if you could let us know whether you have any remaining questions or concerns.

Your further feedback before the rebuttal deadline would be greatly appreciated, as it would allow us to provide any necessary clarifications.

Thank you again for your time and consideration.

Reviewer_Ym6sOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

JARVIS introduces a benchmark for autonomous visual-manual-guided robotic assembly with 236,679 LEGO tasks and a curated 50-task interactive suite, using structure-aware evaluation beyond binary success. Its JARVIS-agentic framework composes robot primitives into executable programs without task-specific action trajectories. Across 9 models, task-oriented tools improve scores from 7.88 to 14.71, but performance drops sharply with longer assembly horizons, and reflection provides inconsistent benefits while prior-task experience proves more useful than online self-correction.

Critical strengths

1.This is a very interesting and practical benchmark. LEGO is an excellent task domain because it offers nearly infinite object combinations and randomizations within a limited workspace, providing high task randomness. At the same time, by relying on visual instructions as input, the benchmark can test an agent's visual understanding, spatial reasoning, long-horizon planning, and self-reflection abilities.

2.​ The figures clearly illustrate the agent workflow in JARVIS. The writing is also very clear.

3.​ The experiments are very comprehensive, including large‑scale benchmarking (236,679 tasks across 8,987 scenes), evaluation across 9 multimodal/reasoning models, detailed ablation on the use of task‑oriented primitives (score improvement from 7.88 to 14.71), analysis of how assembly horizon affects performance, and comparisons between prior‑task experience versus online self‑correction.

Critical weaknesses

1.​ Limited novelty: Similar to Manual2Skill and other works that use visual instruction manuals to guide robot assembly.

2.​ Limited work content: The entire system builds a task scenario in Isaac Sim and a set of agent workflows. The overall difficulty and contribution of the work is relatively low.

  1. Monotonous scenario and tasks: Although it includes many LEGO objects, the action skills are limited to pick-and-place (no rotation), the scene is monotonous, and the chain-of-thought reasoning follows the same pattern throughout. Beyond LEGO, more diverse scenarios could be added, such as furniture assembly, construction, cooking, etc."
Questions for authors

1.How are the current brick locations, brick identities, and target poses obtained from the RGB-D observations and visual manuals, and which parts of this information come from the environment API rather than being inferred by the model?

  1. I noticed that the difficulty score appears to be based mainly on initial-scene complexity and the number of bricks that must be added. However, different brick types, the gap between the current and target spatial structural states (e.g., in a large space vs. a crowded space), and structural stability may make the actual difficulty vary. Have these aspects been considered?

3.How does an agent like GPT6 Astra directly outputting actions compare the agents outputting code or python function?

6 comments

We are grateful to the reviewer for the thoughtful assessment and helpful suggestions. Our detailed responses follow:

W1 & W2:

Thank you for these comments. JARVIS starts from a practical requirement: a robot must follow a visual manual to build a structure and verify its progress through physical interaction. This requires more than understanding the manual or executing a given instruction in isolation. Each step must be grounded in the evolving workspace, and execution outcomes must inform subsequent interpretation, planning, and action. These interdependent challenges motivate JARVIS’s full loop from manual understanding to workspace grounding, autonomous execution, and verification.

Pretrained VLAs achieve only about 2 out of 100 in this setting (Table 1). With no ground-truth action trajectories available for task-specific training or adaptation, we designed JARVIS-agentic to decouple agentic reasoning from robot control (Sec. 4). The backbone model interprets visual manuals, grounds actions in the evolving workspace, and generates executable programs, while a skill library handles low-level manipulation, and execution memory and feedback support verification, recovery, and replanning. This full loop requires both an agentic pipeline that operates without being trained on task-specific action trajectories, and an evaluation protocol that captures assembly correctness and partial progress.

Table 1. Assembly scores (out of 100) of end-to-end VLA baselines on JARVIS.

ModelFixed language instructionsExternal visual planner
π₀.₅-DROID2.258 / 1002.003 / 100
OpenVLA-7B1.856 / 1001.846 / 100
ours14.71/100 (Model average)15.90/100 (GPT-OSS-120B)
Improvement+12.45 (6.5×)+13.90 (7.9×)

The challenge goes beyond constructing a simulation scenario. The agent must locate each illustrated step within a complex existing structure, translate it into grounded tool calls, and handle execution-induced changes. Evaluation is equally central: since no reference trajectories exist, outcomes must be judged from the assembled structure itself, so we design structure-aware metrics within a unified, method-agnostic execution framework that enables fair comparison across models. Across nine models, the highest Easy and Hard scores are only 25.66 and 10.35, respectively (Table 2), highlighting the difficulty of this setting. Our work therefore makes three complementary contributions: (1) the JARVIS benchmark and evaluation protocol, with 236,679 tasks across 8,987 simulation-evaluable scenes; (2) the JARVIS-agentic framework, which performs these tasks without task-specific training; and (3) extensive experiments and findings on long-horizon execution and recovery.

W3:

Thank you for raising this point. We would like to respectfully clarify that actions in JARVIS-agentic are not limited to translation-only pick-and-place: the placement primitive also executes in-plane yaw adjustments. For example, in task 000027/task_002_ops_002_002, the primitive applied a −90° yaw adjustment to block 2 and returned success, with a post-release yaw error of only 0.457°. (This example illustrates the rotation capability at the primitive level; the overall assembly in this task did not meet the final success criterion.) Beyond rotation, the agent composes robot skills by generating programs, and the skill library is extensible: skills such as light tapping or pressing can be added to improve connection stability, with their benefit verified directly through the assembly score. We have also reserved skill and control interfaces to support more complex assembly actions, such as bimanual manipulation. We agree that richer manipulation skills are a valuable direction, and we will make the current action space and its extensibility clearer in the revised manuscript.

Q1:

In the visual-grounding experiments, the model receives only the manual image, scene RGB, a depth visualization, and camera calibration; nothing is read from simulator state.

  • Current locations: In the RGB-D + 3D-descriptor setting, geometric utilities segment regions by color and back-project depth to report world-coordinate centers, extents, and orientations. The RGB-D baseline receives no descriptors.
  • Brick identities: No simulator instance or part IDs are provided. Region labels are generated locally, and we evaluate step-indexed target-pose predictions rather than brick identification.
  • Target poses: The model predicts world-coordinate positions and quaternions itself; ground truth is used only for scoring.

Q2:

Thanks for your insightful question. Yes, these factors are considered, but they enter through task sampling and execution-based evaluation rather than the difficulty score itself. The score uses the numbers of initial bricks and bricks to be added because they give a uniform, computable measure of structural complexity and execution horizon. During sampling, we additionally prioritize coverage of different source LEGO models, so the benchmark spans diverse part combinations, structural forms, and spatial layouts. Brick type, local crowding, target placement location, and structural stability do affect actual difficulty. In interactive simulation they surface as grasping, collision, placement, and structural-disturbance failures, and are therefore reflected in the final assembly score (see Supplementary Section C for collision cases).

Q3:

Thank you for this interesting question. We were not able to include GPT-6 Astra in our experiments because of its cost: recent reports put a single episode at roughly USD 2 with direct action output and USD 36 in an agentic setup [1], and at about 1–9M tokens and one to two hours for a long-horizon episode [2], which is beyond our current budget for Benchmark-50. We will add this comparison once the budget allows. Recent reports nevertheless give an indication. In one direct comparison, Astra outputting end-effector targets reaches 66.3% success on RoboLab-120, versus 74.8% when wrapped in an agentic coding session, though at a much higher token cost [1]. Direct action output is strongest on short-horizon pick-and-place and weakest on contact-rich, high-precision tasks: plug insertion succeeds in 0 of 11 real-world trials [2], and a systematic study reports a gap between sound task-level decisions and reliable physical control [3]. Since JARVIS requires long-horizon, precise brick placement, we expect code or function outputs grounded in a skill library to remain advantageous in our setting, although this remains to be verified empirically.

References

[1] Yang et al. Astranomy: An Empirical Study of GPT-6 Astra as a Native Robot Policy. CoRL 2026 LBR.

[2] Joint Research Team. Has GPT-6-Astra Achieved Embodied Intelligence? arXiv:2610.22471, 2026.

[3] Galbot Team et al. Systematically Exploring the Capabilities of GPT-6 Astra as Embodied Policies. arXiv:2609.38537, 2026.

Dear Reviewer Ym6s,

Thank you for your valuable and constructive feedback. As the discussion period is coming to an end, we would like to kindly follow up and ask whether our responses have adequately addressed your concerns.

Thank you again for your time and consideration.

Reviewer_dicWOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces JARVIS, a benchmark for autonomous robotic assembly guided by visual manuals. The benchmark contains 236,679 LEGO assembly tasks across 8,987 scenes, with a curated set of 50 tasks for interactive robotic evaluation. The authors also propose JARVIS-agentic, a tool-grounded framework that integrates multimodal reasoning, executable robot primitives, and execution feedback. Experiments across nine multimodal and reasoning models investigate the effects of task difficulty, tool abstraction, spatial grounding, and reflection. The results highlight substantial challenges in translating visual procedural understanding into reliable long-horizon robotic execution.

Critical strengths

The paper addresses an important gap between visual instruction understanding and autonomous physical execution, moving beyond conventional static assembly planning benchmarks.

JARVIS provides a large task collection with varying assembly horizons and structural complexities. Its structure-aware evaluation captures partial progress and geometric accuracy beyond binary success.

The results reveal that stronger planning and grounding do not necessarily translate into successful execution, and that repeated reflection may provide limited benefits. These observations offer useful insights for future embodied agents.

Critical weaknesses

JARVIS-agentic primarily integrates existing multimodal models, code generation, robot primitives, and feedback mechanisms. The technical contribution lies mainly in benchmark construction rather than a novel robotic learning or control algorithm.

Despite the large task collection, only 50 tasks are included in the interactive benchmark, potentially limiting the representativeness and statistical reliability of the reported results.

Experiments are conducted entirely in a LEGO assembly simulation environment. The absence of real-world robotic experiments leaves the robustness of the framework to perception noise, physical uncertainty, and hardware constraints unclear.

Questions for authors

none

4 comments

We sincerely thank the reviewer for the careful reading and constructive feedback. Our detailed responses follow.

W1:

We would like to clarify the scope of our contributions, which comprise three complementary parts: (1) the JARVIS benchmark, (2) the JARVIS agentic framework, and (3) extensive experiments and findings. We chose the benchmark track because our primary emphasis is on establishing a new practical problem and enabling its systematic evaluation.

The technical contribution of JARVIS-agentic lies in formulating manual-guided assembly as decoupled agentic reasoning and robot control (Sec. 4). The backbone model interprets visual manuals, grounds actions in the evolving workspace, and generates executable programs, while a skill library handles low-level manipulation, and execution memory and feedback support verification, recovery, and replanning. This design lets models perform diverse assembly tasks without parameter updates or task-specific action trajectories, which are costly to collect. Its practical value is supported by Table 1, where task-oriented tools raise the average assembly score from 7.88 to 14.71. In contrast, pretrained VLA policies score only about 2 out of 100 (π₀.₅-DROID: 2.26; OpenVLA-7B: 1.86).

Table 1. Assembly scores (out of 100) of end-to-end VLA baselines on JARVIS.

ModelFixed language instructionsExternal visual planner
π₀.₅-DROID2.258 / 1002.003 / 100
OpenVLA-7B1.856 / 1001.846 / 100
ours14.71/100 (Model average)15.90/100 (GPT-OSS-120B)
Improvement+12.45 (6.5×)+13.90 (7.9×)

W2:

Thank you for raising this concern. The 50 tasks were systematically selected to represent the candidate pool across assembly horizon, initial structural complexity, and source-model diversity. Section 3 outlines how this selection balances representative coverage with interactive evaluation cost, and Appendix B.3 details the full selection procedure. Specifically, from 1,067 valid candidate tasks derived from 48 LEGO source models, we selected 20 Easy, 15 Medium, and 15 Hard tasks, prioritizing distinct source models to reduce structural redundancy. The selected tasks span 1–29 assembly steps and 2–236 initial bricks. As shown in Appendix B.3 and Figure 5, the benchmark closely matches the candidate pool in mean assembly horizon (5.18 vs. 4.82 steps), proportion of 3–5-step tasks (50.0% vs. 50.7%), and median initial brick count (58.5 vs. 54). These comparisons support the suite's representativeness along these dimensions, and including the longest, 29-step task ensures coverage of challenging long-horizon assembly.

W3:

Thank you for the suggestion. Manual-guided assembly already poses substantial visual understanding and planning challenges in simulation [6, 7]. Requiring actual robotic execution within the simulator adds further difficulties in grounding actions, handling physical interactions, and recovering from errors. Following established embodied reasoning and planning benchmarks [1–5], we therefore use simulation for controlled, reproducible evaluation of these combined challenges. Even with task-oriented tools, current models achieve an average assembly score of only 14.71 out of 100 (Table 2), highlighting the difficulty of connecting visual understanding, reasoning, and robotic execution.

We therefore see costly real-robot evaluation as the next step rather than a prerequisite. Our simulator captures grasp failures, placement errors, and collision-induced structural changes (Appendix C). To further approximate real-world sensing uncertainty, we added perception-noise experiments (Table R2). Image noise lowers the mean score by less than one point (5–7%), and moving-target perturbation by 1.2–2.8 points (11–24%), so JARVIS-agentic retains most of its assembly capability under the tested conditions. These results provide preliminary evidence of robustness to perception noise. Real-robot validation is planned as part of our upcoming open challenge and future work.

Table 2. Effect of perception noise on DeepSeek-driven JARVIS (scores out of 100).

Noise conditionCleanNoisyΔ
Light image noise (σ = 8)10.97410.441−0.533
Heavy image noise (σ = 20)10.97410.225−0.749
Light moving-target perturbation (1 mm, 0.5°)11.5748.792−2.782
Heavy moving-target perturbation (3 mm, 2°)10.9749.759−1.216

We sampled two tasks from each difficulty tier of Benchmark-50 and ran them with three noise seeds under five conditions (clean plus the four noise conditions), 90 episodes in total. The table reports valid clean–noisy pairs matched on task and seed for DeepSeek-driven JARVIS (deepseek/deepseek-v4.1-flash-1017); scores include a time penalty. Image noise is zero-mean Gaussian on the RGB channels; target perturbation uses the listed position and rotation standard deviations, with the scoring ground truth unchanged.

References

[1] Srivastava et al. BEHAVIOR: Benchmark for Everyday Household Activities in Virtual, Interactive, and Ecological Environments. CoRL 2021.

[2] Li et al. BEHAVIOR-1K: A Benchmark for Embodied AI with 1,000 Everyday Activities and Realistic Simulation. CoRL 2022.

[3] Ge et al. BEHAVIOR Vision Suite: Customizable Dataset Generation via Simulation. CVPR 2024.

[4] Li et al. Embodied Agent Interface: Benchmarking LLMs for Embodied Decision Making. NeurIPS 2024.

[5] Yang et al. EmbodiedBench: Comprehensive Benchmarking Multi-modal Large Language Models for Vision-Driven Embodied Agents. ICML 2025.

[6] Wang et al. Translating a Visual LEGO Manual to a Machine-Executable Plan. ECCV 2022.

[7] Wang et al. IKEA-Manual: Seeing Shape Assembly Step by Step. NeurIPS 2022.

Dear Reviewer dicW,

Thank you for your valuable feedback and for taking the time to review our work.

We hope that our rebuttal and additional experimental results have adequately addressed your concerns. We would be grateful if you could let us know whether you have any remaining questions or concerns.

Your further feedback before the rebuttal deadline would be greatly appreciated, as it would allow us to provide any necessary clarifications.

Thank you again for your time and consideration.

Dear Reviewer dicW,

We appreciate how busy you are with multiple submissions. If you have a moment before the rebuttal deadline, we would be grateful if you could let us know whether our responses have addressed your concerns.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Response to the Area Chair

We thank all reviewers for their careful reviews and constructive suggestions. We appreciate their recognition of the following strengths:

  1. Bridging the gap between visual-manual understanding and robotic execution, covering planning, execution, and recovery (Reviewers dicW and w1J2).
  2. A practical and diverse LEGO assembly benchmark evaluating spatial reasoning, long-horizon planning, and self-reflection across varying task complexities (Reviewers Ym6s and dicW).
  3. Reproducible, policy-agnostic, and structure-aware evaluation measuring partial progress and geometric accuracy across controlled difficulty levels (Reviewers w1J2 and dicW).
  4. Comprehensive experiments and insightful analyses across nine models, tool-library ablations, task complexity, and online reflection, with clear presentation (Reviewers Ym6s and dicW).

Reviewer dicW initially rated the paper Weak Reject. Our rebuttal clarified the contributions of JARVIS and JARVIS-agentic, supported the representativeness of the 50-task suite through comparison with the candidate pool, and explained the rationale for simulation-based evaluation, including results from the requested perception-noise tests. The reviewer has raised no further questions to date.

Reviewer Ym6s initially rated the paper Weak Accept. In their follow-up, they confirmed that our response provided the information they had requested.

Reviewer w1J2 initially rated the paper Weak Reject. In their follow-up, the reviewer acknowledged that explicitly recognized the paper’s novelty and the impact of the problem setting and acknowledged that our rebuttal had addressed their questions. They encouraged us to pursue real-robot experiments. We explained why JARVIS starts with a reproducible simulation benchmark and identified real-robot evaluation as the next step once methods can consistently solve these tasks in simulation.

Thank you again for your time and consideration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.