- Rating
- 4(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
IR-SIM is a lightweight 2D simulator for navigation learning and benchmarking, including multi-robot collision avoidance and robot navigation among pedestrians. YAML configurations specify robots, obstacles, kinematics, behaviors, and sensors, while custom controllers can be integrated through Python. The system provides CPU and batched GPU implementations, together with skills that help coding agents construct and run scenarios. Experiments cover cross-simulator execution speed, scenario construction with two language models, benchmarking of eight navigation controllers, and SARL policy training.
Critical strengths
- The system has practical value for navigation research requiring repeated scenario construction and execution. Scenario configurations, common motion behaviors, sensors, and learning interfaces are integrated into one tool, potentially reducing repeated development work in multi-robot navigation experiments.
- Tests cover different robot counts, LiDAR and sensor-free settings, single-core CPU execution, and batched GPU execution, with shared layouts and recorded input actions. IR-SIM is faster in the evaluated large-scale multi-robot settings
- Scenario construction is evaluated through execution and requirement satisfaction. The validator checks execution, scenario requirements, task completion, and collisions. On the evaluated tasks, skills reduce Opus's mean authoring time from 204 to 98 seconds and achieve approximately 95% scenario validity with both models.
Critical weaknesses
- Section 4.1 emphasizes cross-simulator speed advantages. However, Appendix B.1 states that MVSim uses octagonal robots and Box2D contact forces, whereas IR-SIM uses circular robots and a collision-stop rule in this test. Matching layouts and input actions does not ensure the same simulation workload. The timing impact of these differences is not assessed.
- Appendix C.3 states that all conditions receive documentation and source code, but only the IR-SIM skill condition additionally receives the authors' skill package. No corresponding skill-package condition is reported for VMAS or PyRoboSim. Table 2 therefore supports the practical advantage of the supplied IR-SIM skills package, but does not establish how much of the authoring-time and validity gap would remain if other tools received similar support.
- The second contribution highlights approximately 95% scenario validity. However, Appendix C.6 explicitly states that skill version 0.9.1 was refined using failures from these brief families, and describes the result as an upper bound rather than an independent measurement. No results are reported on new tasks after freezing the skills.
Questions for authors
- For the comparison with MVSim, can collision-handling requirements be aligned, or timings be reported separately for motion updates, collision handling, and LiDAR? How much do octagonal versus circular footprints and contact forces versus stopping affect the results?
- Can at least one baseline, VMAS or PyRoboSim, receive comparable API guides, scenario templates, and behavior-parameter guidance, followed by an authoring-time and validity comparison under the same model and tool budgets?
- After freezing the skill version, can scenario validity and authoring time be evaluated on new tasks not used for development or debugging?
7 comments
Author Response Part 2
W3 & Q3 Agent skill generalization: We clarify that the IR-SIM skills were refined based on failures by adding general rules to improve the skill performance, rather than tailoring the skills to the specific evaluated task families. To validate the effectiveness of the IR-SIM skills in unseen scenario families, we have added an experiment on six unseen scenarios (U1–U6), which were not used during the skill design and the skill is frozen, with three models (Opus 5.5, Sonnet 5.5, and Haiku 5.5) and three baselines (IR-SIM, VMAS, and PyRoboSim). The results are summarized in the following table:
Unseen scenarios U1–U6
| Simulator | Executable (%) | Valid (%) | Spec. acc. (%) | Turns | Time (s) | Cost (USD) | Lines Python | Lines YAML |
|---|---|---|---|---|---|---|---|---|
| Opus 5.5 | ||||||||
| IR-SIM + skills | 100.0 | 100.0 | 100.0 | 16.2 | 192 | 0.89 | 116 | 59 |
| IR-SIM | 83.3 | 83.3 | 83.3 | 20.8 | 191 | 0.88 | 156 | 65 |
| VMAS | 100.0 | 100.0 | 100.0 | 23.1 | 214 | 0.84 | 275 | 0 |
| PyRoboSim | 94.4 | 72.2 | 92.8 | 21.9 | 209 | 0.87 | 235 | 19 |
| Sonnet 5.5 | ||||||||
| IR-SIM + skills | 94.4 | 66.7 | 91.8 | 10.1 | 64 | 0.22 | 81 | 55 |
| IR-SIM | 83.3 | 77.8 | 82.9 | 12.1 | 67 | 0.19 | 107 | 53 |
| VMAS | 88.9 | 55.6 | 85.9 | 13.9 | 85 | 0.23 | 187 | 0 |
| PyRoboSim | 88.9 | 66.7 | 83.2 | 18.9 | 90 | 0.28 | 166 | 12 |
| Haiku 5.5 | ||||||||
| IR-SIM + skills | 100.0 | 66.7 | 96.9 | 19.6 | 196 | 0.04 | 130 | 60 |
| IR-SIM | 100.0 | 72.2 | 97.8 | 20.8 | 204 | 0.04 | 144 | 56 |
| VMAS | 77.8 | 50.0 | 72.8 | 20.0 | 258 | 0.06 | 254 | 0 |
| PyRoboSim | 88.9 | 50.0 | 85.7 | 23.8 | 239 | 0.06 | 234 | 17 |
From the unseen scenarios U1–U6 results, we can see that IR-SIM + skills consistently achieves the highest executable percentages across all three models, which shows that the skills carry over to unseen scenarios in producing executable scenarios. We acknowledge that, for the weaker models, the validity on the unseen scenarios is lower than the approximately 95% reported on the original briefs, and we narrow our claim accordingly.
In addition, we acknowledge that the validity improvement with Opus 5.5 over the other simulators is not statistically significant. The reason is that Opus 5.5 is a stronger model, which can construct valid scenarios directly without strongly relying on the skills or the simulator itself.
However, for the weaker models, Sonnet 5.5 and Haiku 5.5, IR-SIM, with or without the skills, shows higher validity than VMAS, and its cost and time are also the lowest among all the simulators, which indicates that our simulator design can effectively enhance the performance of weaker models in unseen scenarios. For these models, the skills do not raise the validity further; they reduce the turns and the code, and raise the executable percentage with Sonnet 5.5.
Moreover, when VMAS receives a comparable support package, it reaches the same or higher validity than IR-SIM + skills (100.0 / 100.0 / 88.9% against 100.0 / 66.7 / 66.7% with Opus 5.5 / Sonnet 5.5 / Haiku 5.5; see W2 & Q2 above), so skill-style support also works for other simulators.
In the revision, we will narrow our agent skill contribution accordingly: we will not claim a validity advantage from the skills, and we will emphasize the compact declarative scenarios and the lower authoring time and cost of IR-SIM with the weaker models. We will also report the results on the unseen scenarios.
I would like to thank the authors for their detailed supplementary experiments; the more objective comparison has given me a deeper understanding of this method’s strengths and weaknesses.
Author Response Part 1
We appreciate the reviewer’s feedback regarding the fair comparison and skill generalization. W1 & Q1 Fair Comparison with MVSim: We acknowledge the reviewer’s concern regarding the fair comparison with MVSim: the stop rule and the circular robot footprint in IR-SIM lead to a lighter simulation workload. Thus, we have added an experiment that compares IR-SIM with MVSim under the same robot footprint and collision handling requirements, and we report the time per component as requested. The results are summarized in the following table:
Time per component (ms per step at 10 / 100 / 1,000 robots)
From the results, under the same robot footprint and collision handling requirements, on the CPU, MVSim is 1.4 to 2.7 times faster than IR-SIM, which confirms the reviewer’s concern. Between the default and the aligned IR-SIM rows, the octagonal footprint accounts for most of the difference: it raises the IR-SIM step time by 1.3 / 2.4 / 2.2 times at 10 / 100 / 1,000 robots, mainly in the LiDAR, and the contact handling adds a further 6-17%.
On the GPU, under the same requirements, IR-SIM is 1.56x and 11.7x faster than MVSim at 100 and 1,000 robots, while MVSim is faster at 10 robots, where the fixed GPU overhead dominates. MVSim is compared in its CPU mode, which is its fastest configuration. This demonstrates the advantage of the IR-SIM batching mechanism accelerated by the GPU at large numbers of robots. We will modify our claim on MVSim comparison and add these results to the revised paper.
W2 & Q2 Comparable API Guides and Support for VMAS: We provide IR-SIM-skill-style support for VMAS: a support package with API guides, scenario and runner templates, behavior guidance, and review lists. As we are not the developers of VMAS, we built this package with a coding agent (Fable 5.2) and checked every statement in it against VMAS. It was written without access to the unseen scenarios, frozen before they were run, and evaluated with the same models and harness. The results are summarized in the following table:
Unseen scenarios U1–U6, with a support package for VMAS
With the support package, VMAS achieves the same validity as IR-SIM + skills with Opus 5.5, and a higher validity with Sonnet 5.5 and Haiku 5.5. We therefore acknowledge that skill-style support also works for other simulators. The advantage of IR-SIM + skills is a shorter deliverable with all three models and. In addition, VMAS does not support declarative configuration: its scenario is task specific Python code (155–221 lines here), which is hard to reuse for another task or controller, while by design the IR-SIM scenario is a short YAML file (55–60 lines) that is separate from the control code and can be reused.
We will add these results to the revised paper and narrow our agent skill contribution accordingly: we will no longer claim a validity advantage from the skills over other simulators, we will state that the skills mainly reduce the turns and the code of IR-SIM scenarios, and we will emphasize the compact, reusable declarative scenarios and the lower authoring time and cost with the weaker models.