acceptodds
Withdrawn from ICLR 2027

IndustryNav: Evaluating Spatial Reasoning and Safety of Embodied Agents in Dynamic Industrial Navigation

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Visual large language models (VLLMs) are increasingly used as embodied agents, yet industrial warehouses demand more than reaching a goal: agents must navigate safely around moving workers and vehicles. To evaluate embodied agents' goal reaching and safety, we introduce IndustryNav, a simulation environment with 24 warehouse scenes for closed-loop navigation to specified destinations. Alongside task success and efficiency, we introduce safety-aware metrics that capture collisions and near-collision warnings along each trajectory. Experiments with eleven leading open- and closed-source VLLMs and three navigation baselines show that agents can reach their goals yet collide with moving obstacles, exposing a gap between success and safe navigation. This gap matters in safety-critical settings, where a single collision can end the task. IndustryNav also supports policy learning through automated oracle trajectories and simulator interaction, with held-out scenes for generalization tests. These experiments show that policies can succeed on new routes in familiar warehouses yet fail to transfer to held-out layouts, making cross-scene generalization an open challenge.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Embodied agent, closed-loop navigation, unity simulation, safe navigation

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_NRfUOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

IndustryNav introduces a Unity-based benchmark for dynamic warehouse navigation, containing 24 scenes with four fixed start–goal pairs each: 96 evaluation tasks. It evaluates VLLMs alongside A*, DAgger, and PPO, examining task completion, risk proxies, interaction history, and transfer to unseen layouts. Its principal contribution is an environment and evaluation framework rather than a new navigation algorithm.

Critical strengths
  1. The platform integrates dynamic warehouse environments, agent evaluation, expert-trajectory generation, and policy training, providing a potentially useful resource for embodied navigation research.
  2. The experiments offer useful diagnostic findings. Under fixed RGB input, adding interaction history improves Gemini’s success rate from 9.38% to 59.38%. Static–dynamic comparisons also show that similar success rates can conceal differences in collision and warning proxies.
  3. Static/dynamic comparisons show that success alone can conceal changes in process-level risk. Cross-layout experiments also keep test routes fixed across training configurations.
Critical weaknesses
  1. The evaluation coverage is limited for the paper’s benchmark and generalization claims. IndustryNav contains 24 scenes but only four fixed start–goal pairs per scene, yielding 96 evaluation tasks. The cross-layout study is further restricted to 24 tasks across six scenes, where a single outcome changes the success rate by approximately 4.17 percentage points. Although the authors acknowledge the descriptive nature of these results, the limited route coverage and lack of reported variation across repeated runs or scene splits leave the robustness of model rankings and transfer conclusions unclear. Broader task sampling is important for establishing the benchmark’s reliability beyond the particular routes evaluated.
  2. The safety evaluation does not adequately establish safe navigation under continuous dynamic interaction. CR measures blocked forward progress rather than direct physical contacts, while WR uses depth thresholds without accounting for relative motion. Validation against physical contact events and reporting collision-free success would better connect these proxies to safe task completion. In addition, as acknowledged in Appendix F.3, the environment remains static between action steps. This excludes risks caused by model deliberation latency: obstacles do not continue moving while the agent decides. The reported results therefore characterize safety under a synchronized protocol rather than continuous real-time interaction.
  3. The significance of the benchmark’s incremental contribution is not sufficiently demonstrated. The paper acknowledges prior work on industrial safety understanding, training-free embodied agents, and interactive embodied evaluation. Its main distinction is combining active navigation, moving obstacles, and safety assessment in warehouse environments. This is a useful extension, but the experiments do not clearly establish which new capability requirements or failure modes arise specifically from this combination. The contribution would be stronger with a more direct comparison to existing navigation benchmarks and evidence that the industrial setting reveals challenges beyond those already studied.
Questions for authors

Can the evaluated agents improve through online adaptation across attempts? Specifically, after an unsuccessful episode, could an agent use its trajectory and execution feedback to revise its strategy and retry the same task? Comparing feedback-informed retries with independent restarts and a longer single attempt under the same total interaction budget would help distinguish limitations in initial exploration from an inability to learn from failure.

1 comment
AuthorsOpenReview ↗

Response to Reviewer NRfU

Thank you for the thoughtful review. We agree that more testing and a closer analysis of safety would strengthen the study. We address your points below.

1. Task coverage and statistical reliability (W1)

To check how results vary between runs, we evaluated GPT-Terra three times on the same 96 tasks: 24 scenes, with four start-goal pairs per scene. The table reports the mean ± standard deviation across these three runs. All values are percentages.

MetricMean ± SD (%)
SR@2m18.75 ± 0.95
SR@5m23.96 ± 1.47
SR@10m38.02 ± 2.21
DR59.83 ± 0.96
ES12.55 ± 1.48
CR11.94 ± 0.81
WR21.99 ± 2.68

These repetitions show how GPT-Terra's results vary when the tasks stay the same. We agree that we also need to test more routes and repeat evaluations of other models before drawing stronger conclusions about model rankings. The separate cross-scene study covers only 24 tasks, so its findings should be treated as preliminary. Testing more training/test scene splits is also important.

2. Successful navigation without collisions, and real-time safety (W2)

Following your suggestion, we calculated collision-free success@2m from the existing records. A run counts only if it finishes within 2 m of the goal and has no recorded collisions. We divide this count by the model's total number of available runs.

ModelSR@2m (%)Collision-free success (%)
A*10047.92
DAgger52.0810.42
PPO (gallery replay)76.0418.75
Claude-Sonnet-519.790
GPT-5-Mini11.831.08
GPT-5.6-Luna17.717.29
GPT-5.6-Terra18.482.17
GPT-5.6-Sol20.8310.42
GPT-6-Astra51.0418.75
Gemini-3.8-Flash59.3825
Qwen3.8-Flash3.191.06
MiniMax-M317.711.04
DeepSeek-V4-Flash-Vision-Exp21.887.29
GLM-5.3-Flash41.6712.50

Reaching the goal does not mean getting there without collisions. Gemini succeeds in 57 of 96 tasks, but only 24 of those successes have no recorded collisions: its rate falls from 59.38% to 25%. Astra similarly drops from 51.04% to 18.75%. These differences show why success and safety should be evaluated together.

PPO uses a separately recorded gallery replay, so its results differ from Table 2. Here, a collision is still detected when a forward move travels less than 95% of its expected distance. We agree that this rule needs to be checked against actual physical contacts, which we leave to future work. WR also checks obstacle distance without considering how fast or in which direction the obstacle is moving.

We also agree with your point about the environment pausing while the model decides (Appendix F.3). In real-time navigation, workers and vehicles keep moving. An action chosen from an earlier image may therefore be unsafe by the time it is executed. We see asynchronous navigation, where the environment keeps running while the agent thinks and acts, as an important direction for future work. We plan to study how decision delays affect safety and how agents can respond to changes that happen while they are deciding.

3. What the industrial setting adds (W3)

We agree that our experiments should explain more clearly what makes warehouse navigation challenging compared with existing benchmarks. They show that agents struggle, but do not fully separate the effects of warehouse layout and moving traffic. Your suggestion is valuable. In future work, we plan to compare with existing benchmarks and vary layout and traffic separately. This would help explain, for example, how narrow aisles and crossing traffic affect an agent's ability to wait, find another route, or escape a blocked area.

4. Learning from failed attempts (Q1)

We agree that this is a very promising direction. An agent could review what went wrong, change its strategy, and try again. This could be a starting point for studying recursive self-improvement (RSI) in embodied agents. Our current memory is used only within one attempt. In future work, we plan to compare retries that retain feedback, restarts that discard it, and one longer attempt, as you suggest. Giving all three the same total number of action steps would help test whether improvement comes from using feedback rather than simply having more attempts. We would measure both success and safety.

Reviewer_Zic1OpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

This work aims to study warehouse ("industry") safety where you have a large room, rows of stock, and equipment (e.g. forklifts) moving around that need to be avoided. In principle this is a navigation task with a safety requirement -- get to the goal, but don't collide.

Critical strengths

The task is a real problem that needs to be addressed. The environments are human authored. I think it's important for the field to seriously engage with scenarios that are relevant to deployed systems.

Critical weaknesses
  1. The core claims of the paper are about safety and dynamics but ... they are actually not well studied in this work. The actual results are mostly focused on the point-goal setup and there's very little (or no) discussion of how the dynamic and safety components are integrated or modeled. We really only get collision and warning so running into a static object would be collision, and warning isn't really a danger...
  2. Under the metrics (1) all the models are already safe? nearly oracle safe? So the problem... is solved?
  3. I appreciate the interest in training a model and testing generalization, etc but the off the shelf models already beat them, particular if presented with common enhancements like memory
  4. I am very confused about the action space, it seems as though I have to teleport 1.5m at a time and can only turn in 45 degree increments. Why?
  5. (4) is particularly concerning since it's directly died to the safety definition. WR has 0.4m margin but I have to move in 1.5m increments. I have similar confusion about the 0.6m success vs the 2m radius and how to interpret the example in Fig 4
  6. Why are we using a distance ratio? Why not L2 distance to the goal?
Questions for authors

See weaknesses.

1 comment
AuthorsOpenReview ↗

Response to Reviewer Zic1

Thank you for the detailed review and for recognizing the value of human-authored, deployment-relevant scenes.

1. Dynamics and safety analysis (W1)

The appendix provides additional analyses of navigation behavior and safety beyond goal attainment. Appendix A.3 (Fig. 10) illustrates warning detection through 12 paired RGB and depth examples. Appendix C.1 (Figs. 16–19) examines obstacle avoidance and recovery, while Appendices C.2.3 (Figs. 23–27) and D.1 (Fig. 29) analyze failure modes and scene-level safety. Fig. 7 in the main text further compares navigation performance in static and dynamic environments.

Warnings highlight potential collision risks in both static and dynamic scenes, since a moving agent can also collide with stationary obstacles. They indicate a need for caution, not that a collision will necessarily occur. In future work, we will record physical contacts and the objects involved, validate CR against these events, and report contact-free success.

Separately, Appendix F.3 notes that the simulator pauses while the model selects its next action. The current results therefore do not account for risks caused by obstacles moving during that time.

2. Are models already nearly oracle-safe? (W2)

A low CR does not mean that an agent can reliably reach its goal while avoiding obstacles. In Scene 22, Gemini has only 1.77% CR but fails all four tasks (Appendix D.1, Fig. 29). Conversely, Claude-Sonnet-5 repeatedly attempts blocked forward moves on S24/P4, reaching 95% CR (Fig. 26). These examples show that agents still struggle both to find a viable route and to move around obstacles.

A* is our map-based planning baseline, but it does not guarantee collision-free navigation. Matching its CR therefore does not establish that an agent is safe. In future work, we will record actual contacts and measure how often agents reach their goals without contact.

3. Trained policies versus off-the-shelf models (W3)

The VLLM results in Table 2 already include five-step memory. The minimap experiments provide additional map information and are reported separately.

Our policy-learning study asks whether policies can navigate warehouses they have not seen during training. We train DAgger and PPO on either scenes 1–18 or all 24 scenes, then test both settings on the same tasks in scenes 19–24. The test routes are excluded from both training settings. This lets us compare performance when the warehouse layout is familiar versus unfamiliar. A VLLM achieving higher success than a learned policy does not answer this generalization question, which is the focus of Section 3.2.

4. Discrete actions and teleportation (W4)

The nominal 1.5 m forward moves and 45-degree turns provide a simple, consistent interface for VLLM evaluation. They are configurable parameters, not fixed simulator limits: finer forward distances and turning angles are supported.

Discrete choices do not mean teleportation. Commands become Unity controller signals; forward displacement passes through CharacterController.Move or its physics wrapper rather than directly setting the final position. Obstacles can limit actual movement below the nominal command.

5. Warning margin and success radius (W5)

The warning distance includes the planned movement plus a 0.4 m buffer. For a 1.5 m forward action, we check for obstacles within 1.9 m, rather than only 0.4 m. For actions without forward movement, the distance is 0.4 m. A warning is triggered when at least 0.5% of valid depth pixels in the forward region fall within this distance. It indicates nearby obstacles that may pose a collision risk. Depth is used to calculate this metric and is not given to the default VLLM.

An episode succeeds when the agent finishes within 2 m of the goal. The 0.60 m shown in the example is simply how close that particular run finished; it is not a different success threshold. We also report success within 5 m and 10 m to show how often agents get near the goal.

6. Distance Ratio versus final L2 distance (W6)

DR is intended to measure how much of the initial distance to the goal an agent has covered. For example, two agents may both finish 10 m from the goal, but one started 20 m away and the other 100 m away. Their final L2 distance is identical, while their distance reductions are 50% and 90%. DR helps distinguish these cases, and final L2 distance remains useful for showing how far the agent still needs to go.

However, the current formula uses the absolute change in distance, so it can also give a positive score when an agent moves farther from the goal. We need to review this definition and recompute the affected results before interpreting DR as progress toward the goal.

Reviewer_uEwtOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This is a Unity benchmark evaluating PointGoal navigation (with given goal coordinates) and safety of VLLM (vision LLM) agents in warehouses with moving workers and vehicles. It defines success rate (SR), goal progress (DR), efficiency (ES), collision ratio (CR), and near-collision warning ratio (WR), and compares 11 VLLMs with A*, DAgger, and PPO.

Critical strengths
  • The paper compares VLLM agents with classical and learned policies under the same conditions in warehouses with moving workers and vehicles. It evaluates 11 VLLMs and A*, DAgger, and PPO on the same tasks and metrics. Although there are problems with the interpretation of the conclusions (1 below), this provides comparative evidence on where VLLM agents stand in navigation in dynamic environments.
Critical weaknesses
  1. It is unclear why this benchmark needs to be solved, and the comparison setup does not support the main conclusions. A*, given the full map, already reaches the goal within 2m (SR@2m) in 100% of episodes on this task. Agents solve a PointGoal task with exact self-localization and goal coordinates at every step. This is a problem directly addressed by classical navigation stacks with depth-based costmaps and dynamic obstacle avoidance (e.g., Nav2), even without the full map, and the task has no elements requiring VLLMs, such as language or semantic goals. Yet there is no such classical stack baseline, and the comparisons are also unfair. The A* collisions used to support "Success can conceal collisions" may reflect the limitations of a planner that does not avoid dynamic obstacles, and "Task-trained policies outperform most VLLMs" (p.6) is based on a comparison where VLLMs receive RGB while learned policies receive depth.
  2. The new contribution over existing benchmarks is unclear, and the core safety metric is unvalidated. Table 1 compares only with spatial VQA, embodied QA (EQA), and household task benchmarks, but benchmarks that evaluate the collisions and safety of navigation policies among moving people and obstacles already exist (e.g., Arena 3.0, arXiv:2406.00837; MetaUrban, arXiv:2407.08725; HA-VLN 2.0, arXiv:2503.14229; CostNav, arXiv:2511.20216). In addition, CR is a proxy that counts a step as a collision if the agent moves less than 95% of the commanded distance, rather than measuring actual contact (p.4), and it has not been checked against actual contacts in the simulator.
Questions for authors
  1. What are the SR, CR, and WR of a classical stack such as Nav2 using a depth-based costmap and dynamic obstacle avoidance without the full map?
  2. Why is it important to evaluate VLLMs on a task that classical stacks can solve? Have you considered settings such as language instructions or semantic goals?
  3. How closely does CR agree with actual contacts in the simulator? Also, PPO's DR of 42.12% is lower than its SR@2m of 73.56%; how is this consistent with Eq. 3?
  4. What does IndustryNav newly measure compared with dynamic-obstacle and social navigation benchmarks such as CostNav, Arena 3.0, and MetaUrban?
1 comment
AuthorsOpenReview ↗

Response to Reviewer uEwt

Thank you for the detailed review. We address the concerns and questions below.

1. Motivation and task choice (Concern 1; Q2)

We do not claim that classical methods cannot solve navigation. Their strong capabilities motivate our question: can existing general-purpose VLLM agents make similarly reliable spatial decisions? Progress in manipulation using embodied agents such as Astra or Fable 5.2 does not establish navigation competence. IndustryNav evaluates eleven open- and closed-source VLLMs on judging traversable space, selecting routes, and avoiding moving workers, forklifts, and robots. The default input is local RGB, numerical odometry, and memory, without depth, a global map, or an external planner. RGB tests spatial decisions from visual cues rather than a supplied geometric obstacle representation.

Language instructions and semantic goals are future directions we intend to pursue. They combine navigation with language understanding, target recognition, and goal grounding. We first use explicit goal coordinates to reduce ambiguity about the destination and evaluate navigation more precisely. This helps distinguish navigation failures from goal-interpretation failures and provides a foundation for these extensions.

2. Baselines and Nav2 (Concern 1; Q1)

IndustryNav uses Unity ML-Agents and currently has no native ROS 2 interface or Nav2 bridge. Running Nav2 requires additional sensor, coordinate-transform, and control integration; we therefore have no Nav2 SR/CR/WR results to report.

Table 2 provides PPO and DAgger results. Both use depth and numerical odometry without a global map at inference; DAgger learns from a privileged A* teacher. These are local-sensing policy references, not substitutes for a classical costmap stack. A* uses privileged geometry, while VLLMs use RGB, so comparisons describe these system configurations rather than isolating architecture or spatial reasoning as the sole cause of performance differences.

A*'s blocked moves can reflect its controller and obstacle-response behavior; they alone do not establish limitations of classical dynamic avoidance. Goal reaching and obstacle interaction remain distinct, with the safety qualification below.

3. Related benchmarks (Concern 2)

  • Arena 3.0 benchmarks navigation planners in human-populated settings. We focus on eleven general-purpose VLLMs' spatial reasoning and dynamic obstacle avoidance under a common RGB-odometry-memory interface.
  • MetaUrban studies urban policy learning and generalization; its reported benchmarks use LiDAR. We study RGB-based VLLMs in warehouses, including memory, map access, and dynamics.
  • HA-VLN 2.0 emphasizes human-aware language grounding and social navigation. We use coordinate goals to reduce instruction ambiguity and examine spatial action selection around moving equipment and workers.
  • CostNav evaluates delivery through physical and economic consequences. We diagnose general-purpose VLLM navigation and policy transfer, rather than economic viability.

These works deserve direct comparison. We do not claim that dynamic obstacles, safety evaluation, or policy training are individually new.

4. Safety proxies and the PPO table error (Concern 2; Q3)

CR flags forward commands with displacement below 95% of the commanded distance. It measures obstructed progress, not physical contact directly. CR-contact agreement has not been measured, so we cannot report it. Validation requires paired contact events and proxy flags, false-positive/negative analysis, and contact-free success. WR indicates action-dependent geometric proximity, not a calibrated accident probability.

Thank you for catching the PPO error: DR and ES were assigned to the wrong columns; 42.12% is ES, not DR. Eq. 3 assigns successful episodes DR=1 and failures nonnegative DR, implying DR≥SR\mathrm{DR}\geq\mathrm{SR} for the same episode set. Remaining PPO entries still require reconciliation with consistent samples and definitions before reporting a corrected row. This table error does not resolve contact validation.

5. Additional evaluation contribution (Q4)

IndustryNav couples VLLM diagnosis with expert-action collection, policy learning, and held-out-layout evaluation. Section 3.2 compares DAgger/PPO trained on scenes 1-18 or all 24 scenes on the same 24 tasks in scenes 19-24, excluding evaluation start-goal pairs from both training regimes. This distinguishes new routes in familiar layouts from transfer to held-out layouts, evaluating success and safety jointly. Our contribution is this warehouse resource, protocol, and empirical study, not the first study of safety or generalization.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.