- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
IndustryNav introduces a Unity-based benchmark for dynamic warehouse navigation, containing 24 scenes with four fixed start–goal pairs each: 96 evaluation tasks. It evaluates VLLMs alongside A*, DAgger, and PPO, examining task completion, risk proxies, interaction history, and transfer to unseen layouts. Its principal contribution is an environment and evaluation framework rather than a new navigation algorithm.
Critical strengths
- The platform integrates dynamic warehouse environments, agent evaluation, expert-trajectory generation, and policy training, providing a potentially useful resource for embodied navigation research.
- The experiments offer useful diagnostic findings. Under fixed RGB input, adding interaction history improves Gemini’s success rate from 9.38% to 59.38%. Static–dynamic comparisons also show that similar success rates can conceal differences in collision and warning proxies.
- Static/dynamic comparisons show that success alone can conceal changes in process-level risk. Cross-layout experiments also keep test routes fixed across training configurations.
Critical weaknesses
- The evaluation coverage is limited for the paper’s benchmark and generalization claims. IndustryNav contains 24 scenes but only four fixed start–goal pairs per scene, yielding 96 evaluation tasks. The cross-layout study is further restricted to 24 tasks across six scenes, where a single outcome changes the success rate by approximately 4.17 percentage points. Although the authors acknowledge the descriptive nature of these results, the limited route coverage and lack of reported variation across repeated runs or scene splits leave the robustness of model rankings and transfer conclusions unclear. Broader task sampling is important for establishing the benchmark’s reliability beyond the particular routes evaluated.
- The safety evaluation does not adequately establish safe navigation under continuous dynamic interaction. CR measures blocked forward progress rather than direct physical contacts, while WR uses depth thresholds without accounting for relative motion. Validation against physical contact events and reporting collision-free success would better connect these proxies to safe task completion. In addition, as acknowledged in Appendix F.3, the environment remains static between action steps. This excludes risks caused by model deliberation latency: obstacles do not continue moving while the agent decides. The reported results therefore characterize safety under a synchronized protocol rather than continuous real-time interaction.
- The significance of the benchmark’s incremental contribution is not sufficiently demonstrated. The paper acknowledges prior work on industrial safety understanding, training-free embodied agents, and interactive embodied evaluation. Its main distinction is combining active navigation, moving obstacles, and safety assessment in warehouse environments. This is a useful extension, but the experiments do not clearly establish which new capability requirements or failure modes arise specifically from this combination. The contribution would be stronger with a more direct comparison to existing navigation benchmarks and evidence that the industrial setting reveals challenges beyond those already studied.
Questions for authors
Can the evaluated agents improve through online adaptation across attempts? Specifically, after an unsuccessful episode, could an agent use its trajectory and execution feedback to revise its strategy and retry the same task? Comparing feedback-informed retries with independent restarts and a longer single attempt under the same total interaction budget would help distinguish limitations in initial exploration from an inability to learn from failure.
Response to Reviewer NRfU
Thank you for the thoughtful review. We agree that more testing and a closer analysis of safety would strengthen the study. We address your points below.
1. Task coverage and statistical reliability (W1)
To check how results vary between runs, we evaluated GPT-Terra three times on the same 96 tasks: 24 scenes, with four start-goal pairs per scene. The table reports the mean ± standard deviation across these three runs. All values are percentages.
These repetitions show how GPT-Terra's results vary when the tasks stay the same. We agree that we also need to test more routes and repeat evaluations of other models before drawing stronger conclusions about model rankings. The separate cross-scene study covers only 24 tasks, so its findings should be treated as preliminary. Testing more training/test scene splits is also important.
2. Successful navigation without collisions, and real-time safety (W2)
Following your suggestion, we calculated collision-free success@2m from the existing records. A run counts only if it finishes within 2 m of the goal and has no recorded collisions. We divide this count by the model's total number of available runs.
Reaching the goal does not mean getting there without collisions. Gemini succeeds in 57 of 96 tasks, but only 24 of those successes have no recorded collisions: its rate falls from 59.38% to 25%. Astra similarly drops from 51.04% to 18.75%. These differences show why success and safety should be evaluated together.
PPO uses a separately recorded gallery replay, so its results differ from Table 2. Here, a collision is still detected when a forward move travels less than 95% of its expected distance. We agree that this rule needs to be checked against actual physical contacts, which we leave to future work. WR also checks obstacle distance without considering how fast or in which direction the obstacle is moving.
We also agree with your point about the environment pausing while the model decides (Appendix F.3). In real-time navigation, workers and vehicles keep moving. An action chosen from an earlier image may therefore be unsafe by the time it is executed. We see asynchronous navigation, where the environment keeps running while the agent thinks and acts, as an important direction for future work. We plan to study how decision delays affect safety and how agents can respond to changes that happen while they are deciding.
3. What the industrial setting adds (W3)
We agree that our experiments should explain more clearly what makes warehouse navigation challenging compared with existing benchmarks. They show that agents struggle, but do not fully separate the effects of warehouse layout and moving traffic. Your suggestion is valuable. In future work, we plan to compare with existing benchmarks and vary layout and traffic separately. This would help explain, for example, how narrow aisles and crossing traffic affect an agent's ability to wait, find another route, or escape a blocked area.
4. Learning from failed attempts (Q1)
We agree that this is a very promising direction. An agent could review what went wrong, change its strategy, and try again. This could be a starting point for studying recursive self-improvement (RSI) in embodied agents. Our current memory is used only within one attempt. In future work, we plan to compare retries that retain feedback, restarts that discard it, and one longer attempt, as you suggest. Giving all three the same total number of action steps would help test whether improvement comes from using feedback rather than simply having more attempts. We would measure both success and safety.