acceptodds
Under review as a conference paper at ICLR 2027

ForagerSim: Evaluating Embodied Map-Based Reasoning in Large-Scale Dynamic Worlds

Abstract

Animals build cognitive maps of vast environments, use them to perform map-based navigation, and might even be reusing the same circuitry used in spatial reasoning for creating and using maps of arbitrary task-spaces. In contrast, current embodied AI systems either rely on hand-engineered mapping pipelines, which break in the presence of unmodeled dynamics and are restricted to metric space, or on learned policies that rarely form map-like representations at all. Developing architectures that could learn to build and use map-like representations as flexibly as animals is, however, difficult because few embodied AI benchmarks actually require map-based reasoning. To address this, we present ForagerSim — an interdisciplinary research platform and simulation environment focused on embodied tasks that require map-based reasoning, with scenarios ranging from mobile-robot search and rescue to animal foraging in both complex mazes and vast, potentially infinite open areas. ForagerSim is highly modular and allows modifying independent axes of difficulty related to perception (abstract 2D sensing to full RGB or LiDAR), scale (room-scale to multi-km scale), topology (open-fields to complex mazes), dynamics (smoke, debris, sensorimotor failures and others) and tasks (from simple 2D coverage to foraging-inspired well-alternation tasks). We compare reinforcement learning baselines with a simple engineered robotics stack and a human player on 8 example worlds which show how different difficulty axes affect existing methods. Our results show that, when sensing is simplified to 2D, learned policies can perform multi-object search and well-alternation tasks surprisingly well in simple topologies, but they fall short of human performance in larger or more complex-topology environments where map-based baselines perform well, suggesting that map-based reasoning is the bottleneck in these scenarios. Code and data are shared in supplementary materials and will be made open-source upon acceptance.

Primary area:
reinforcement learning
Keywords:
cognitive maps, reinforcement learning, embodied search, mapping, planning, benchmark

Reviews

Among:
Shape:
1234this paper, Top 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.67 over 3 reviews, above 99% of the others.
Reviewer_H9gTOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

ForagerSim is a Unity benchmark for embodied search and foraging with configurable perception, scale, topology, dynamics and task, plus Gym and ROS 2 interfaces. The authors evaluate PPO, DreamerV3, a greedy frontier explorer and one human on eight worlds. They conclude that learned agents fall short in larger or maze-like topologies, so map-based reasoning is the bottleneck.

Critical strengths
  1. It fills a real gap: fast, configurable, dynamic worlds with identical scoring for RL policies and ROS 2 stacks. The home-cue-withheld well-alternation task is a clever test of return-to-known-location behavior. This is the main reason I lean toward acceptance.

  2. It is candid and reproducible: the Dreamer position shortcut and the failed curriculum are reported, and code, configs and seeds are shared. This makes the benchmark trustworthy as a community resource.

Critical weaknesses
  1. The claim that map-based reasoning is the bottleneck is not isolated. PPO is memoryless and Dreamer runs near defaults, with no oracle-map test or learned-memory baseline. Scale, topology and budget vary together, and the multi-km, failure and 3D claims are untested.

  2. The comparisons are weak. The frontier explorer fails in dynamic worlds by construction and is absent from the foraging tasks. The human is a single person, and with 3 seeds and standard deviations that sometimes exceed the means, claims like "near-human" are within noise.

Questions for authors

Clarifications:

  1. How are the ± values computed, and are the evaluation worlds held out from training?
  2. What happens if PPO and Dreamer receive an oracle occupancy grid? This would directly test the central bottleneck claim.

Suggestions: 3. How do recurrent or spatial-memory agents perform? 4. Single-axis sweeps (scale, topology and budget varied independently) and results with more human participants would strengthen the conclusions.

2 comments
AuthorsOpenReview ↗

Response to Reviewer H9gT (Part 1)

Thank you for the constructive feedback and for recognizing the value of ForagerSim, particularly the well-alternation task and the comparison of learned and engineered approaches. We hope the following responses address your concerns:

Q1: How are the ± values computed, and are the evaluation worlds held out from training?

  • The ± values show the standard deviation of the per-episode returns. The mean and its deviation are computed across all evaluation runs of all training seeds per method and scenario (5 per training seed).
  • Evaluation splits: The evaluation worlds are held out from training. Since we also test variations of a task in some scenarios while keeping the world topology the same (e.g., 4arm-submazes and 4arm-submazes-foraging, and the new dynamic variants of memorymaze15x15), we also vary the metaseed used for generating the actual world-generation seeds for these instances, so that the agents (and humans) do not see the same mazes twice during evaluation when only the task is varied and the layout generation is the same. We have reworded the table captions and a part in the experiments description in Sec.4.1 to make this clearer.

W1, Q2: Testing the bottleneck claim by providing an explicit map (oracle-map) Good question. Following your suggestion, we have added new experiments with explicit map observations (Appendix A, Fig. 7-8) to investigate whether providing agents with a spatial representation improves performance. Rather than supplying a complete oracle map at the start, we give the agents a map that is incrementally constructed during the experiment from the agent’s original observation data (similar to the explicit map built by the frontier explorer), since the continuously built map conveys better information for the search task (i.e., it shows where the agent has not been yet, which allows for discovering strategies akin to frontier-based exploration, which the complete map would not allow) and for the foraging task, it can also aid navigation since it shows the already visited passages. We evaluated the effect of providing an explicit map on the most challenging 4arm-submazes topology, for both search and well-alternation tasks. Visualizations and results of the runs are in the new section, and in summary, they show that having the explicit map does help Dreamer greatly in the search task, but it did not help it with the foraging task (even though the map shows the way to the home well, the agent’s position, and the well positions). This suggests that the challenge of these scenarios lies not only in the construction of maps, but also in the ability to use them to perform goal-directed navigation.

Q3: Performance of recurrent and spatial-memory baselines. We have tested recurrent (the RecurrentPPO implementation in StableBaselines) in preliminary experiments, but even with some parameter tuning, its performance was always near zero and it was not able to train well. For this reason, we have instead focused on Dreamer as a strong RL baseline with memory (and evaluated it with multiple hyperparameter options in the curriculum experiment). If by spatial-memory you mean the SRU spatial recurrent unit presented in Yang et al. 2026 (IJRR), we have not tested this, but future investigations could definitely test whether the spatially-enhanced memory systems can perform well in ForagerSim.

AuthorsOpenReview ↗

Response to Reviewer H9gT (Part 2)

W1–W2, Q4: Single-axis sweeps and human and multi-km evaluation.

  • Single-axis sweeps: Thank you for the suggestion. The current experiments already contain sweeps (in the sense of controlled changes between a set of scenarios) across different axes, which we have now separated more clearly in the reorganized section 4.1: a) Dynamics: we test 2 varying levels of dynamics in the 5houses-dynamic and 5houses-sar (same topology, different dynamics) and now also in the new dynamic variant of the memorymaze15x15 world (same search task and maze as in the memorymaze15x15 scenario, just with passage closing); b) Task complexity: we also test the independent effect of task complexity (4arm-submazes and 4arm-submazes-foraging have the same topology and scale, just a different task specified by swapping reward objects for wells in the world config and the episode length in the task config); and c) Topology/environment complexity: we test various topologies for both the search and foraging tasks. The episode length changes between some scenarios that have a different scale or a different task, but all baselines always have the same budget within a scenario and we draw our conclusion not by comparing absolute returns between scenarios, but from looking at how the differences between the baselines change across scenarios. As another example, the curriculum experiment does a smooth sweep of the environment scale while keeping the topology generation, budget, reward signal, actions, and observations the same (although the number of rooms and the distances between them do change), and users can easily define their own similar sweeps for any other environment through config-file edits with no code changes.
  • Human reference and multi-km world evaluations. Regarding the human baseline, please note that the single participant is intended as a feasibility reference and an approximate upper bound, demonstrating that substantially higher performance is achievable with the same observations, rather than providing a statistical characterization of human performance. While we agree that more participants would provide a more reliable estimate of human performance, we do not expect this to change our main conclusions, and such an evaluation was not feasible within the rebuttal period. We also acknowledge that the multi-kilometer 3D scenarios (supported by the simulator) and some sensorimotor-failure settings remain outside the current quantitative evaluation. The tested agents already struggle in several low-dimensional 2D scenarios, which provide substantial room for improvement of the approaches (both learned and engineered) without adding visual-perception complexity.
Reviewer_yRAtOpenReview ↗
Rating
4
Confidence
4
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

ForagerSim is a simulation environment focused on embodied tasks that require map-based reasoning, a skill that is engrained in animals since one of the key skills for survival is to search for rare objects (e.g. food) in large dynamic environments before returning to known locations (e.g. nest). It addresses the need for fine-grained benchmarking tools to help develop embodied AI systems with a map-based reasoning system that can be learned and improved from data and experiences. Current systems have partially learnable mapping and planning modules but their interactions and their hierarchy levels are often hand-designed and not learnable end-to-end, which can be brittle in large dynamic environments and domain shifts. The main contributions of the paper are (a) the open-sourced simulation environment for embodied research that has both a fast 2D depth+semantic observation mode and a realistic 3D RGB/LiDAR sensing mode, (b) model-free and model-based RL agent baselines, (c) a classic ROS2 stack baseline, and (d) a human player baseline.

Critical strengths
  • ForagerSim has scalability via world generation and streaming on the fly, modularity that allows independent control over the task difficulty of topology, scale, observations, dynamics and switching between 2D and 3D modes, and having both ROS2 and Gym interfaces that allow the robotics and ML research communities to have a shared benchmark.
  • Has an option to simulate sensory failures and observation noise.
  • The study and benchmarking work is thorough and comprehensive in its scope, with model-free (PPO)/model-based RL (Dreamer V3), ROS2 and human player baselines provided. The simulation and benchmarking code is open-sourced.
  • As a reviewer it is delightful to finally have a paper that is not AI slop and reads well. Thank you for putting in the effort of making the paper readable with clear delivery.
Critical weaknesses
  • The reward design seems to be fixed and not very configurable. (Collision penalty is surprisingly low which could lead to reward hacking behavior)
  • The reward object is a simple red cube, which is rather unrealistic and stands out too much from the background. It would be nice to have more variety and customizability over the reward objects as well to make it more realistic.
  • There aren’t any composite tasks such as retrieving objects of different types in a particular sequence. Environments such as Minecraft are better suited as a benchmark for evaluating composite tasks and long-term planning tasks, since there are interdependencies between subtasks such that one item must be retrieved before another because something needs to be crafted in advance.
  • The action space is limited to navigation and no further actions such as tool use.
  • [Minor typos] space in front of comma at L355, “expeirment” -> “experiment” in L227 and L481, “abstraciton” -> “abstraction” in L281, “trunacted” -> truncated in L444, “betweeen” -> “between” in L449.
Questions for authors
  • Would it be possible to make the reward object and reward values customizable (not just red cube), and have some settings where the user can define a custom task of retrieving objects in a particular sequence? (e.g. the agent seeing a blue sphere, but you need to find at least 3 cubes of either (“green”, “red”) to be able to carry back the blue sphere as well) Ideally the user should be able to provide a config that defines the dependency tree of the order/condition the agent can pick up objects.
  • The “per-episode return” score that is used for benchmarking hugely differs depending on the environment and is difficult to interpret unless compared against the human baseline. Would it be possible to come up with an additional standardized metric to measure the navigation efficiency or planning ability of agents, which are more robust to changes in the world setting? (Especially for custom designed worlds, it would be beneficial to have these normalized scores where the human baseline is not yet available)
1 comment
AuthorsOpenReview ↗

Response to Reviewer yRAt

Thank you very much for your detailed review! We are happy that the paper reads well and that you appreciate ForagerSim's interdisciplinarity, modularity and the benchmarking work. We hope the following responses address your concerns:

W1-W2, Q1: Reward signal and objects customizability, collision penalty:

  • Customizability of rewards and objects (Sec.3.2) We apologize that the original wording was unclear. The rewarding actions, their values, and also termination conditions are already configurable through the ‘task’ config file for any given scenario (each scenario is composed of three configuration files: agents+world+task). We used identical reward settings across experiments to isolate the effects of environment and task complexity. The ability to change the reward objects is available in the simulator - one can swap the reward object prefab by editing a world configuration file (‘reward_objects/prefab_name’ field in world config files). We have clarified this in Sec. 3.2. The current ForagerSim build contains only 2 prefabs (the red cube for the 2D environments and a semi-realistic model of a human for the more realistic scenarios), but a user can easily create new objects inside the Unity Editor and can start using them in new world configs without any code changes necessary.
  • Collision penalty. Regarding the collision penalty, we have thus far not observed any reward hacking behavior, which is why we kept the collision penalty low (and also so that human players are not penalized too much for colliding since it can sometimes be challenging to maneuver in tight spaces using the keyboard). The only related behavior we observed is that the Dreamer agent can sometimes repeatedly push into the wells in the well-alternation tasks instead of waiting near the well to dispense the reward, so we will try increasing the penalty in future experiments.

W3–W4, Q1: Composite tasks and action space: We agree that the action space is small, but this was a deliberate design decision to focus on building and using map-like representations for navigation in large/complex worlds, while minimizing other sources of complexity (e.g., high-dimensional actions/observations). We also agree that configurable sequential retrieval tasks could be a valuable extension and will consider them for future development (potentially in the form of search and rescue tasks with some priority definition to stay grounded in robotics/animal scenarios). General sequential retrieval tasks are already implemented in some other environments (e.g., MemoryMaze tasks or MIKASA’s Numpad task) and so we focused more on the map-based navigation in isolation in this version. Please also note that while ForagerSim does not currently support arbitrary object-dependency trees, the well-alternation task does introduce sequential dependencies (home → random well → home) grounded in animal navigation experiments. Regarding more complex object manipulation and composite tasks, we see extending ForagerSim with those as a natural next step which we plan to pursue in the near future. Specifically, we plan to first implement a simple abstract gripping mechanism for 2D scenarios, and then move on to full articulated robots with grippers in 3D. This would enable richer robotics- and animal-inspired tasks (e.g., clearing rubble or assisting in rescue operations, manipulating objects or pulling levers) that would combine map-based reasoning with tool use/manipulation and sequential planning.

Q2: Standardized evaluation metrics Thank you for this suggestion! You are right that the score can be hard to interpret without the human or frontier-explorer baseline data and can vary a lot between worlds. Thanks to your and Reviewer h8uM’s suggestion, we have added a new (normalized) navigation optimality metric (Sec. 4.1, Table 3) for foraging scenarios, which compares the path lengths the agents traveled between wells against ground-truth optimal paths. This metric enables evaluating navigation efficiency in foraging scenarios without requiring a human baseline. For example, DreamerV3 achieves 84% optimality when navigating toward cued wells in forest-foraging, but only 21% when returning to the uncued home well, compared with ~80% for the human participant. The optimal path computation uses Unity’s native pathfinding package, can be run after the evaluation runs, and can even account for dynamic objects and passage closures in the world. In addition, Table 4 in the appendix shows a breakdown of the objects found and the area explored for each world, which should be easier to interpret than the combined score.

W5: Minor typos: Thank you for catching these! We have fixed them in the revised version.

Reviewer_h8uMOpenReview ↗
Rating
4(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This work introduces ForagerSim, a modular simulation platform for studying embodied map-based reasoning. The simulator is motivated by robotics and animal foraging and supports configurable scenes, dynamics, sensing and tasks. It exposes both a Gym and Ros2 interface and abstract 2D or richer 3D observations. To demonstrate the capabilities of this simulation platform, the authors compare memoryless PPO, DreamerV3, a manually engineered exploration stack and a human participant in eight scenarios covering object search and well alternation. The results show that explicit mapping and planning perform well on complex static search tasks where the tested RL agents struggle, while learned agents handle obstacle interactions that defeat the classical baseline.

Critical strengths
  1. The platform offers a useful combination of capabilities. ForagerSim allows for easy and modular configuration of different scenarios, including procedural world generation, configurable dynamics and sensing. A shared Gym/ROS2 interfaces make it possible to compare learned policies and engineered systems within the same environment. Simplified semantic-depth observations also allow navigation to be studied separately from high-dimensional visual perception. This is a potentially valuable infrastructure contribution.

  2. The static search experiments provide meaningful benchmark validation. In 4arm-submazes, the explicit mapping-and-planning baseline collects 35.4 of 36 objects on average, compared with Dreamer’s 14.9 and PPO’s 9.3 (Table 3). These results show that the benchmark includes solvable tasks on which a map-based system substantially outperforms the tested learned policies. The dynamic scenarios reveal complementary limitations of the classical baseline, while the curriculum experiments provide additional evidence that the difficult foraging task remains challenging under the tested training modifications.

Critical weaknesses
  1. The additional evaluation value over existing benchmarks could be articulated more clearly. Table 1 and Section 2 motivate ForagerSim’s combination of scalability, modularity, and configurable dynamics, and the experiments demonstrate tasks on which explicit mapping and planning are effective. However, the connection between these results and the limitations of existing benchmarks remains less explicit. Identifying which specific evaluation questions the proposed scenarios enable, and why those questions are difficult to study in the closest existing platforms, would strengthen the positioning of the contribution.

  2. Validation is strongest for static search, while the distinctive dynamic and large-scale capabilities receive more limited evaluation. In the dynamic scenarios, the frontier explorer fails because entering houses requires pushing rubble, which its navigation logic does not support. This demonstrates an interaction challenge, but the results do not isolate the ability to update a map and replan after environmental changes. Finally, the multi-kilometer 3D worlds are presented as future evaluation settings rather than quantitatively validated in this paper.

Questions for authors
  1. Which specific capability is insufficiently evaluated by the closest existing benchmarks, and which ForagerSim task and metric directly address that gap?

  2. Do any evaluated dynamic scenarios require revising a previously useful map or route after an environmental change? If so, can you show an example episode in which an agent encounters a changed route and successfully replans, rather than simply pushing obstacles aside or handling degraded sensing?

3 comments
AuthorsOpenReview ↗

Response to Reviewer h8uM (Part 1)

Thank you for the detailed and constructive feedback and for recognizing the value of ForagerSim's modular design and the search experiments as a meaningful validation of the benchmark. Your comments helped us clarify the benchmark's motivation and strengthen the evaluation of dynamic environments and navigation optimality. We hope the following responses address your concerns:

W1, Q1: Benchmark positioning and evaluation metrics:

  • We have reworded and extended the positioning section in Sec. 2 (final paragraph), explicitly identifying the gap we aim to address: evaluating map-based reasoning independently of other sources of complexity. Existing benchmarks often feature either relatively simple-topology environments in which even memoryless policies can perform well and thus do not require map-based reasoning (e.g., POPGym navigation tasks) or large, slow-to-learn (high-dimensional observation and action spaces), and computationally demanding worlds where navigation performance is confounded by other types of complexity (perception and control) (e.g., UnrealZoo, MineDojo). ForagerSim addresses this by offering environments and tasks that are explicitly designed to be difficult without map-based reasoning and by independently varying environment topology, scale, dynamics, and task complexity, all with the option to run with efficient, low-dimensional observations. We believe the rat trial-inspired well-alternation task specifically provides a strong test of map-based reasoning, as maintaining and using a spatial map offers substantial advantages over random-walk strategies compared to basic search scenarios. The per-episode return’s main component is the number of rewards picked up, which grows linearly with the number of wells visited, so it is a metric that reflects the overall performance at this task. However, we have also added an additional metric to better analyze agent performance on this task:
  • We have introduced a new ‘navigation optimality’ metric (Sec. 4.1, Table 3) to quantify how efficiently agents navigate compared to optimal paths. The results provide quantitative evidence that DreamerV3 navigates relatively efficiently toward cued wells (e.g., 84% vs. 79% for the human in forest-foraging), but performs substantially worse when returning to the uncued home well (21% vs. 82% for the human).
AuthorsOpenReview ↗

Response to Reviewer h8uM (Part 2)

W2, Q2: Dynamic worlds testing ability to react to replan after environmental changes, large-scale world evaluation: Thank you for the suggestion! You are right that in the dynamic world we have shown, there were no mechanics for explicit blocking of previously taken paths, which would nicely allow evaluating an agent’s capability to update its internal map. We have addressed this limitation in three ways:

  1. New mechanics allowing dynamic passage-closing in any maze environment (Sec. 3.1). This allows config-controllable definition of a set of cells that can become blocked based on specified triggers during a trial, and it also performs a simple connectivity check on each passage-closing trigger, so that the maze stays fully connected at all times, and the tasks stay completable.
  2. Added 2 new scenarios to the simulator and to the evaluations in the paper (Sec.4.1, Appendices B2-B3): 1) memorymaze15x15-dyn’ (doors closing in front of agent), investigating how collapsing the passages right in front of the agent affects the search task, and 2) ‘memorymaze15x15-dyn-for’ (doors closing behind + well-alternation), investigating how closing the passages after an agent passes them affects the foraging tasks (revising the route when returning to the known location). The memorymaze15x15 world, which was simple to solve in a static-search setting, has now become a great topology to test this capability due to its high number of loops and thus a high amount of potential passage closures. The new Fig. 10 and Fig.11 illustrate the passage-closing mechanics in these scenarios. We also made sure that the new navigation optimality metric computation works even when the world topology changes mid-trial (Sec. 4.1, Table 3). This works by computing the A* optimal paths not once per world but by replaying the path of the agent in Unity, triggering the correct door closings, and computing the optimal paths between any traveled well-well segment with the correct doors closed).
  3. Added Figure 11 showing the Frontier-Explorer agent encountering a door closing that blocks its planned path, and replanning to a newly selected goal. PPO and Dreamer architectures do not perform any online planning, and thus it is not possible to show them performing replanning on environment change (although their reactive policies they converge to still achieve nonzero overall return even without explicit path planning).

Regarding evaluations on multi-kilometer 3D worlds, the simulator offers these scenarios, but we did not run training on these because the tested RL baselines have low performance even in the simpler low-dimensional 2D scenarios. We therefore believe these 2D scenarios provide a meaningful benchmark, with substantial room for improvement of the RL methods before moving to more computationally demanding 3D environments with complex visual perception.

Thank you for the clarifications and additional experiments. The clearer positioning and new dynamic topology scenarios address my concerns. I update my recommendation to clear acceptance.

Comments on the paper

1 comment

We thank the reviewers for taking the time to review our paper. We are happy that you appreciate the value of ForagerSim as an interdisciplinary, modular platform for evaluating embodied map-based reasoning. We have made the following additions to the paper based on the reviewers’ suggestions, indicated in blue in the revised manuscript and described in more detail in responses to individual reviewers:

  • New navigation optimality metric (Sec.4.1, Table 3, Figs. 4, 16): We have implemented automatic evaluation of a new ‘navigation optimality’ metric that compares the paths between wells taken by an agent during a well-alternation trial and compares them to an optimal path (computed using Unity’s built-in pathfinding). We have computed this metric for the baselines on all foraging scenarios and note our findings in the revised paper, which now provide quantitative evidence that the RL baselines navigate in a highly inefficient manner without directional cues.
  • New dynamic-topology scenarios (Secs. 3.1, 4.1; Figs. 10–11): We have implemented new simulation mechanics for automatic passage-closing in maze worlds to test the effects of dynamic topology on agent performance. For this, we have added 2 new scenarios and ran all the training and evaluations for them as for the previous 8 scenarios.
  • New map-observation experiments (Appendix A, Fig.7): We have added an experiment testing whether giving RL baselines an explicit map as a direct observation would help performance and found that while it helps with the search task, it did not help RL baselines with the foraging task.
  • Improved clarity and presentation: We have rewritten and reorganized the sections that were indicated as unclear by the reviewers.

We believe that the manuscript has been significantly improved thanks to your insightful feedback and we hope that our revisions and answers adequately address your concerns and questions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.