acceptodds
Under review as a conference paper at ICLR 2027

ProcTD: A Procedural Tower Defense Benchmark for Spatial Reasoning and Long-Horizon Planning

Abstract

Tower Defense (TD) games combine long-horizon economic planning with short-horizon spatial reasoning, yet remain largely absent from reinforcement learning benchmarks. We introduce ProcTD, a procedurally generated, Gymnasium-compatible TD environment with partial observability, configurable spatial scale, heterogeneous tower–enemy matchups, technology progression, and an optional tower-selling mechanic. Its JIT-compiled implementation exceeds 10,000 environment steps per second on a single CPU core, making large-scale experimentation feasible on commodity hardware. The raw action space grows quadratically with map size, from 1,540 discrete actions on to 6,148 on , yet only a small subset is valid at any step. We benchmark general-purpose RL methods and find that performance degrades substantially as spatial scale increases. As a feasibility baseline, we introduce Macro-Action Policy with Symbolic Placement Proximal Policy Optimization (MAPS-PPO), which combines a learned high-level policy with handcrafted spatial placement that reduces the action space by more than 80%. MAPS-PPO reaches a median of waves on with an IQR of , and again a median of on but with an IQR of , while PPO reaches and waves, respectively. ProcTD therefore provides a lightweight benchmark for studying large, structured, spatially grounded action spaces and is currently not solved by general-purpose methods. We release the environment open-source with baseline implementations and an asymmetric attacker–defender extension that allows for competitive benchmarking.

Primary area:
reinforcement learning
Keywords:
Reinforcement Learning, Benchmark, Neurosymbolic, Games

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_Tir8OpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper presents a new tower-defense benchmark, ProcTD, for reinforcement learning agents. The benchmark is intended to test a combination of spatial reasoning, long-horizon planning, and complex action selection. The benchmark can also be used as an asymmetric competitive multi-agent environment with a learned defender and attacker agent. To provide a starting point for researchers, the paper evaluates a combination of heuristic and learned agents, finding that a learned high-level controller with low-level heuristic tower placement performs the best and is more robust to increasing map size. The main contributions are the new benchmark, initial baselines (including DQN and Dreamer in the appendix), and the hierarchical MAPS-PPO model which outperforms standard PPO. They validate these results with a single seed of each run, as well as a 3 seed sensitivity analysis. There are also additional analyses of the impact of various environment settings (such as enabling/disabling selling, map size, and reward shaping) in the appendix.

Critical strengths

The paper presents a useful new benchmark with many features that would be of interest to the reinforcement learning community. The environment provides controls over the map size to increase the overall complexity of the state and action spaces. Modifying the sell price can change how permanent decisions are and affects how important long-horizon planning is. The benchmark even supports a learned adversary, enabling an asymmetric multiplayer game and presenting opportunities to incorporate unsupervised environment design methods.

Although some details are missing from the main paper, the full paper including the appendix describes the environment in extremely clear detail, with auxiliary results testing the importance of several game mechanics and the performance of multiple RL methods. In particular, the paper introduces several interesting heuristic baselines, and ships with a strong hierarchical neuro-symbolic baseline to demonstrate that strong performance is achievable with strong priors and domain knowledge, but not yet achieved by standard RL methods. This, combined with the environment's fast interaction speed, places the benchmark in a good state to provide immediate value to the research community.

Critical weaknesses

I have some concerns regarding the clarity, comparisons to prior work, and experimental design in the main paper.

First, the main paper does not explain a few crucial details of the environment that are necessary for evaluating the claims of the paper. For instance, the main paper does not describe how long an episode is, the frequency with which the agent interacts with the environment, or how sparse feedback is for those actions. All of these factors impact the degree to which long horizon planning is necessary in the environment. It is also not clear until the appendix that the cost of each tower type increases geometrically as you purchase them, so fact that agents "must compose a portfolio" is actually forced by the environment rather than an optimal strategy for agents to discover. Similarly, XP is never explained, nor are the upgrade cards. I recognize that it would be impossible to explain everything in the main paper, but there should be at least a sentence explaining each mechanic so that it is possible for the reader to reason about the environment without going through the full appendix.

Second, the paper states that ProcTD combines spatial reasoning, partial observability, and procedural variation while prior benchmarks "emphasize different challenges", but NetHack, Craftax, and microRTS, for instance, tests all of these capabilities. NetHack and Craftax also test complex action spaces that this paper emphasizes. A more precise comparison to prior work seems necessary to convey what value ProcTD adds to the current research ecosystem. Similarly, the paper is missing a qualitative analysis of the trained agents. It's not clear whether any of these capabilities are actually necessary, or if the agent is learning a very simple and unremarkable strategy. The paper should provide evidence to defend the claim that this is a good benchmark for the specific capabilities.

Third, the main results in the paper use an insufficient number of seeds and the statistical analysis in the appendix is largely invalid. My understanding is that all of the results in the paper are reported from individual runs, except for the seed sensitivity analysis which uses only 3 seeds. It is not possible to produce uncertainty measures from a single seed, so these results all could be attributed to random noise. If the method were extremely expensive it may be excusable, but a key selling point of the benchmark is fast iteration speed. The costs listed in Table F2 suggest that repeating the main experiments for 5 seeds would not be particularly costly. On a similar note, the statistical analysis in Appendix G is not informative. By treating evaluation seeds as individual experimental units for the test, you are testing the claim that the particular MAPS-PPO agent that you trained outperforms your particular PPO and MAPS-RND agents. To test the claim that MAPS-PPO trains better agents in ProcTD, you need to train multiple agents, evaluate each for a large number of seeds, and treat each one as an individual experimental unit.

Questions for authors

It's unclear to me how difficult the environment is from the main paper. How often does the agent interact with the environment? Is one episode 1 wave or 50 waves?

How was the environment balanced, and how were the heuristic components developed and tuned?

This is not a request that will affect my score in any way, but a suggestion that you may want to incorporate to strengthen the paper. The results in Table 2 suggest that replacing both the high-level and low-level components of the hierarchical model improves performance. This seems to suggest that a learned high-level and learned low-level controller would outperform the current baselines, which only learns the high-level controller. I don't believe the fully learned model is necessary to defend the claims made by the paper, but it would be a valuable addition.

4 comments
AuthorsOpenReview ↗

Rebuttal to Tir8 [1/2]

We thank the reviewer for their thorough and constructive evaluation. We particularly appreciate the detailed feedback on the environment's mechanics, its positioning relative to existing benchmarks, and the statistical robustness of our experiments. These comments have helped us identify several important areas for improvement, which we address below.

W1. Missing environment details. We agree that space limitations led us to omit some important environment mechanics from the main paper, making it more difficult to understand how the environment functions. With the additional page, we have expanded Section 3 (The ProcTD Benchmark) to clarify the episode structure, interaction frequency, reward feedback, XP progression, and technology upgrade cards. The changes are highlighted in the revised manuscript provided in the supplementary materials, while more comprehensive descriptions remain in the appendix.

Regarding tower costs, the geometric increase in construction costs does not force agents to diversify, as they remain free to invest exclusively in one tower type if they consider it optimal. Rather, this mechanic is intended to discourage indiscriminate tower construction, particularly in later waves when more gold becomes available. Similar cost-scaling mechanisms are also used in existing Tower Defense games.

W2. Comparison to existing benchmarks and qualitative analysis. We agree that existing benchmarks such as NetHack, Craftax, and microRTS already combine several of these capabilities. ProcTD's main distinction is its emphasis on persistent spatial commitments: unlike environments centered on reactive control, towers cannot be relocated or changed to another type once built. Early placement decisions therefore have long-term consequences, while tower selling, when enabled, introduces an economic cost for correcting them. ProcTD also provides an asymmetric attacker–defender setting, where the two agents have distinct objectives, observations, and action spaces, enabling the study of adaptive offensive and defensive strategies. We have expanded Section 2 (Related Work) to clarify these distinctions.

We have also added a qualitative analysis of learned MAPS-PPO strategies in Appendix F.3. This examines the distribution of tower-building and upgrading actions across waves and individual maps. We observe a clear progression from building towers early to upgrading them later. While the overall tower-action distributions are similar across maps, as expected given the tower–enemy matchup mechanics (Avery et al., 2011), the actions taken at specific waves vary considerably between maps. These findings indicate that MAPS-PPO learns temporally structured and map-dependent strategies rather than simply following a fixed action sequence.

W3. Statistical robustness. We agree with the reviewer's criticism of the original statistical analysis, particularly the distinction between evaluation episodes and independent training runs. We have therefore trained six additional independent runs for both PPO and MAPS-PPO, bringing the total to seven runs per method and map size, each evaluated on the same 100 unseen maps. We have also revised Appendix G.2 to include a hierarchical bootstrap that resamples training runs and evaluation episodes, accounting for both sources of variability. The updated analysis confirms that MAPS-PPO consistently outperforms PPO across both map sizes.

AuthorsOpenReview ↗

Rebuttal to Tir8 [2/2]

Q1. Environment difficulty and episode structure. An episode lasts up to 50 waves, ending earlier if the defender's base is destroyed. During each wave, the agent can build, upgrade, or wait, with each action advancing the simulation by 50 internal game ticks. Between waves, the agent selects a technology card. We have clarified the episode structure, interaction frequency, and reward feedback in Section 3.1.

Quantifying difficulty more broadly is challenging. As a proxy, we evaluated seven independently trained MAPS-PPO agents on the same 100 evaluation maps and found that none cleared all 50 waves on approximately 6% of them. This suggests that some maps remain challenging even for our strongest baseline, although it does not establish that they are unwinnable. Further details are provided in Appendix B.2.

Q2. Environment balancing and heuristic development. Environment balancing was performed iteratively. The mechanics were inspired by existing Tower Defense games, while initial parameters, including enemy health, wave budgets, and gold economy, were selected based on our intuition of balanced gameplay.

We first evaluated the Advanced Heuristic over 100 episodes, aiming for a mean survival of approximately 25 waves while keeping the fraction of full 50-wave clears below 5%. This ensured that the environment was challenging but not impossible to complete. We then trained PPO and subsequently MAPS-PPO, adjusting mainly the coefficients of existing mechanics to increase difficulty while preserving solvability.

The placement heuristics were developed based on our experience with Tower Defense games. They prioritize strategically valuable locations, such as chokepoints and elevated terrain, with additional tower-specific rules. For example, Sniper towers favor central positions to maximize map coverage. Their implementation is detailed in Appendix E.1.

Q3. Two-stage learner. Thank you for the suggestion. We agree that combining learned high-level and low-level controllers would be an interesting extension, and we are currently exploring how best to implement this approach.

We hope that the additional experiments, expanded methodological descriptions, and revised statistical analysis address the reviewer's main concerns. We thank the reviewer again for their constructive suggestions and hope these improvements will encourage a positive reassessment of our submission.

Thank you for the detailed response and answers to my questions. I believe W1, Q1, Q2, and Q3 have been sufficiently addressed.

Regarding W1, the geometrically increasing cost per tower type seems like a strong incentive to build a roughly uniform tower distribution. This is an acceptable choice, but it may hide balance issues between the towers and limit strategic diversity. You may consider testing a smaller geometric increase shared across all towers, to see what portfolio the agents develop without an incentive toward uniformity. However, it seems that the claim regarding agents needing to "compose a portfolio" is no longer in the paper, so this is no longer a concern.affecting my rating.

For W2, the current language is an improvement but still does not seem to clearly define the unique challenges of a tower defense game. For instance, Craftax and NetHack both require long term planning and commitments. It's still not clear to me what different challenges a tower defense presents, or ProcTD in particular. The term "persistent spatial commitments" does not sound meaningfully different from long-term plannin. The Related Works section also implies that Craftax and NetHack do not require strategic planning or resource management, while those are likely the two most significant challenges in those environments. Similarly, it suggests that SMAC and Neural MMO do not present challenges like exploration, resource management, etc. Overall, the related benchmark comparison feels imprecise, which makes it less clear what new challenge ProcTD presents.

Regarding W3, I thank the authors for conducting additional runs of the main PPO baselines, and it is encouraging to see that the main claims have not changed. However, additional runs of the heuristic baselines would also be necessary to provide a fair comparison against the 6 PPO runs, particularly for statistical tests to defend claims such as "Random and Simple Heuristic perform similarly, whereas Advanced Heuristic and MAPS-RND improve substantially when structured placement is introduced. PPO further improves over the non-learned baselines on the 16 × 16 environment, while MAPS-PPO achieves the strongest overall performance and consistently reaches the full episode horizon." Without these results, these comparisons are difficult to trust.

I will raise my score to a 3, because based on prior work, it is not surprising that PPO and hierarchical PPO outperform heuristic baselines, and because the new experiments sufficiently demonstrate that MAPS-PPO outperforms PPO, which was the least predictable comparison. However, I strongly encourage the authors to expand their experiments to include additional seeds for all baselines.

Thank you for your constructive feedback and for raising your score. We appreciate the time and effort you have invested in reviewing our work. Your comments have helped us substantially improve the manuscript.

Regarding tower costs (W1), our initial version of the environment did not include cost scaling at all. However, this led to the agent spamming towers in every buildable tile (on the 16x16 map) and made balancing a lot more challenging. Additionally, it did not let us study the build vs. upgrade tradeoff properly, since building more towers came with no additional cost. The environment itself is inspired by Rogue Tower, as noted in our paper, and the game uses a similar scaling to circumvent this. As such, we decided to go the route the game goes. Regardless, we do agree that a smaller geometric increase shared across towers would be an interesting test; but since this would require major rebalancing, we keep it open for the future.

Regarding statistical robustness (W3), the heuristic baselines are deterministic and do not involve training, so there is no seed variability to account for. However, the random baseline and MAPS-RND do include a random component. Varying this seed results in a slightly different result than we initially had (the only change is the significance of the results). The manuscript has been updated with the results and the additional information on the seeds. Thank you for bringing this to our attention.

Thank you again for your thoughtful suggestions and positive reassessment.

Reviewer_dfxWOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Procedural Tower Defense (ProcTD) is the result of the author's work on constructing a benchmark for evaluating in the tower defense game a single-agent or attacker-defender mode algorithm dealing with gymansium compatible api environment. Alongside a standard evaluation of the PPO algorithm playing the game, a new Macro-Action Policy with Symbolic Placement Proximal Policy Optimization (MAPS-PPO) methods is developed to test the performance of the environment against a hierarchical method.

Critical strengths
  • The construction of the environment is clear, well defined, and logically implemented. The presence of multimodality of the agent interaction is highly appreciated and a nice contribution in the direction of the adoption of this benchmark for many kind of possible algorithms.
  • The idea of adding partial observability features in the definition of the environment states is foundamental for a benchmark that pretends to be useful for algorithm useful in more complex scenario.
Critical weaknesses

Despite its main association with "reinforcement learning", i found that the primary area of this paper should belongs to datasets and benchmarks, reducing a little the contribution that the paper addresses. For this reason, for this paper to be accepted in my opinion, a couple of missing aspects should be addressed:

  • A symmetrically single-agent mode for agent attackers, to be trained within ProcTD, allowing both single agent mode side possible among the same framework
  • A more expressive campaign simulation, respect to table 1 and figure 3, where a single seed is used to validate the target of algorithm that could be beneficial to test with such kind of environment.
Questions for authors
  • Can you clarify the "Win" column in the table 2, is the attacker winning rate or the defender?
  • Have you tried to paralellize the environment? There are some effects on the learning?
3 comments
AuthorsOpenReview ↗

Rebuttal to dfxW

We thank the reviewer for recognizing the main contributions and strengths of our paper, particularly the development of an environment with partial observability that can be used to benchmark “many kinds of possible algorithms.” The reviewer is correct that the primary contribution of our work is a new benchmark environment for RL methods, and that “Datasets and Benchmarks” would probably have been a more appropriate primary area.

W1. Single-agent attacker mode. ProcTD is inspired by traditional Tower Defense games, where the player controls the defender against waves of enemies generated by the game. Accordingly, our primary single-agent setting focuses on the defender. Nevertheless, we also provide a learned attacker in the multi-agent extension (Table 3), trained against a frozen, pretrained defender to ensure a consistent opponent. Since an attacker necessarily requires a defender to interact with, we are unsure what additional functionality a standalone single-agent attacker mode would provide. We would appreciate clarification if the reviewer had a different setting in mind.

W2. Campaign simulations and training seeds. We interpret this comment as a concern about the limited number of independent training runs in Table 1 and Figure 3. We agree that the original single-seed evaluation was insufficient to establish statistical robustness. We have therefore conducted six additional independent training runs for both PPO and MAPS-PPO and updated the manuscript accordingly, as detailed in our general response. If we have misunderstood the concern, we would appreciate further clarification.

Q1. Win rate. We assume the reviewer is referring to Table 3, as Table 2 doesn’t have a “Win” column. As indicated in the table caption, "Att. Win" denotes the attacker win rate, defined as the fraction of episodes in which the attacker destroys the defender's base before Wave 50.

Q2. Environment parallelization. ProcTD achieves approximately 10,000 environment steps/s (500,000 simulation ticks/s) on a single CPU core. Given this efficiency, we have not explored parallelizing the simulation itself. However, all our RL training experiments already use multiple environment instances running in parallel across CPU cores (e.g., 64 parallel workers for PPO and MAPS-PPO, as detailed in Appendix E.3). The throughput for these training experiments is reported in Table C2.

We hope that our responses address the reviewer's concerns and questions. We would be happy to continue the discussion and clarify any points we may have misunderstood, and hope that our responses will encourage the reviewer to reassess our work more positively and reconsider their score.

Reviewer_dfxWOpenReview ↗

Response to the authors

Thank you for your detailed response and for running the six additional training seeds along with the hierarchical bootstrap analysis,this resolves my statistical soundness concern regarding Table 1 and Figure 3. Thank you also for clarifying the attacker win rate metric and the 64-worker CPU parallelization setup.

Regarding the symmetric attacker mode, I understand that an attacker requires a defender opponent, though providing standardized built-in heuristic defenders out-of-the-box would make single-agent attacker benchmarking much more accessible.

However, my primary reservation remains: as a submission to the Reinforcement Learning track rather than a Datasets & Benchmarks track, the contribution bar requires either stronger algorithmic novelty (beyond pairing PPO with handcrafted placement heuristics) or a clearer demonstration of what distinct RL challenges ProcTD isolates compared to existing procedural/spatial benchmarks. While the engineering and statistical rigor are much improved, I believe the paper is better suited for a dedicated benchmarks track and will maintain my current rating.

AuthorsOpenReview ↗

Additional Attacker Results and Primary Subject Area Clarification

Thank you again for your constructive feedback and for acknowledging the improvements made during the rebuttal.

Regarding the attacker benchmark, we have additionally trained a learned attacker against the Advanced Heuristic defender and updated Table 3 accordingly. As expected, given the relatively poor single-agent performance of Advanced Heuristic (Table 1), the learned attacker achieves a 100% win rate. Interestingly, however, Advanced Heuristic achieves higher mean and minimum survival (24 and 11 waves, respectively) against its learned attacker than either PPO or MAPS-PPO, suggesting greater robustness to changes in attacker behavior. We have updated the corresponding discussion in the manuscript.

We acknowledge that Datasets and Benchmarks would have been a more appropriate primary subject area for our submission (ICLR does not have a separate D&B track), and we have contacted the Area Chair to ask whether this classification can still be changed. We nevertheless hope that the contribution can be evaluated primarily on its merits as an RL benchmark.

We appreciate your consideration and hope you might reconsider your assessment in light of this distinction.

Reviewer_PrwcOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes a new RL benchmark, a tower defense game, where long horizon economic planning and short horizon spacial reasoning matters. They use numpy and python numba jit compile for fast CPU based simulations.

Critical strengths
  1. Interesting benchmark combining multiple RL challenges

The environment combines long-horizon planning, spatial reasoning, resource management, partial observability, and large state-dependent action spaces. Early tower placements can affect outcomes many waves later, making this a potentially useful benchmark for studying long-term credit assignment.

  1. Procedural generation encourages generalization

Unlike fixed-map Tower Defense environments, ProcTD generates different maps across episodes. This reduces reliance on memorized placement strategies and encourages agents to learn policies that generalize across spatial layouts.

  1. Computationally efficient and accessible

The Numba-based implementation achieves over 10,000 environment steps per second on a single CPU core.

  1. Configurable difficulty and extensibility

The authors provide multiple map sizes, tower types, enemy configurations, technology progression, optional tower selling, and an attacker–defender mode. This allows researchers to investigate different aspects of RL within a common environment.

  1. MAPS-PPO demonstrates that the benchmark is tractable

The proposed MAPS-PPO baseline combines a learned high-level policy with handcrafted spatial placement. Its performance suggests that the environment is not simply too difficult to solve and that spatial structure can substantially improve learning.

  1. Open-source implementation and reproducibility

The authors release the environment, baseline implementations, configurations, and evaluation scripts. They also provide supplementary experiments on reward formulations, selling, training seeds, and additional algorithms.

Critical weaknesses
  • Limited statistical evaluation of training performance - The main experiments use only one training seed, although the authors evaluate each trained policy on 100 unseen episodes. Given it is a benchmark/simulation paper, I would highly appreciate if authors can provide statistical results with multiple seeds.

  • The main experiments focus heavily on PPO and its MAPS-based variant. The authors also provide DQN and DreamerV3 experiments in Appendix F.1, so it would be inaccurate to say they only evaluate PPO. However, the evidence supporting the broader claim that general-purpose RL algorithms struggle remains limited. In particular, the benchmark has a structured spatial action space, yet the evaluation lacks stronger approaches designed to exploit this structure.

  • Only CPU based implementation (critical) - The authors emphasize computational efficiency and report over 10,000 environment steps per second using Numba on a single CPU core. Could the authors clarify why a CPU-based implementation was chosen over a GPU-based approach, such as JAX? It would also be useful to discuss whether the environment supports parallel execution and how its throughput scales with multiple environments. Given the complexity of the environment, recent trends in RL simulation have increasingly favored GPU-compatible implementations, allowing both environment simulation and policy training to run on the GPU without transferring data between CPU and GPU memory. I believe the benchmark could benefit significantly from a GPU-compatible implementation using frameworks such as JAX. This would better align the benchmark with current trends in the RL community and potentially improve its scalability for large-scale training.

Questions for authors
  1. Computational scalability: The authors report over 10,000 environment steps per second on a single CPU core. Could the authors provide end-to-end training throughput measurements with different numbers of parallel environments? Also, have the authors considered a GPU-compatible implementation, such as JAX, to support large-scale RL training without CPU–GPU data transfers?

  2. Procedural generation quality: The authors acknowledge that some generated maps may be unusually difficult or potentially unwinnable. How frequently does this happen? Is there any mechanism to validate the difficulty or feasibility of generated maps?

  3. Generalization: Since procedural generation is a major contribution, could the authors evaluate generalization across different map distributions rather than only unseen seeds from the same generator? For example, training on one range of map topologies and evaluating on another.

  4. Difficulty control: The benchmark provides several configurable difficulty parameters, but most experiments focus on map size. Could the authors demonstrate how other parameters, such as enemy strength, partial observability, and tower-selling mechanics, affect task difficulty?

  5. Simulation efficiency: Appendix C measures throughput using random actions. Since random policies typically terminate early, these measurements may not reflect the computational cost of successful policies that reach later waves. Could the authors report throughput using trained policies or across different stages of an episode?

  6. Benchmark mechanics: The environment includes several mechanics intended to test different RL capabilities. Could the authors clarify which specific capabilities each mechanic is designed to evaluate and whether these can be tested independently?

3 comments
AuthorsOpenReview ↗

Rebuttal to Prwc [1/2]

We thank the reviewer for recognizing our main contributions and the many strengths of our paper. Below, we address the raised weaknesses and questions in detail.

W1. Replicates and statistical robustness.. We acknowledge that evaluating training performance using only a single seed was not statistically sound. We have now performed six independent runs for each learned model and updated the manuscript accordingly. Please see our general response for further details.

W2. Focus on PPO and MAPS. Our primary goal is to provide a benchmark environment for the community, rather than to exhaustively evaluate all available RL methods. Nevertheless, we aimed to cover the major families of modern RL algorithms with our baselines: policy-gradient actor-critic (PPO), value-based (DQN), and model-based (DreamerV3). Since DQN and DreamerV3 performed poorly, we focused the main paper and subsequent experiments on PPO, as it was the only general-purpose approach that demonstrated meaningful performance on the benchmark. MAPS, on the other hand, serves a crucial role in demonstrating the feasibility of the benchmark using an expert-designed approach. This explains the stronger emphasis on PPO and MAPS throughout the paper.

W3/Q1/Q5 Numba-implementation & Computational efficiency Our decision to use Numba rather than JAX was primarily motivated by our familiarity with the former. We nevertheless considered a JAX-based implementation. ProcTD's simulation involves dynamically changing numbers of towers and enemies, which Numba handles efficiently through loops whose computational cost depends on the current board state. A JAX implementation would require substantial restructuring to accommodate these dynamics efficiently, particularly under vectorized execution.

Given the high throughput already achieved with Numba, we did not consider a complete reimplementation justified at this stage. Nevertheless, we recognize the potential advantages of GPU acceleration and consider a JAX-compatible implementation a possible future extension.

To address the concerns about computational scalability and the representativeness of random-policy benchmarks, we have expanded Appendix C.3 with measurements using trained policies. Table C2 reports end-to-end training throughput, including its evolution throughout training, while Table C3 reports evaluation throughput on different hardware configurations using one and sixteen CPU cores

AuthorsOpenReview ↗

Rebuttal to Prwc [2/2]

Q2. Procedural generation quality We checked 25,000 procedurally generated maps at each of the three sizes (16×16, 32×32, and 64×64) for structural validity, and none failed. Determining whether a map is unwinnable or unusually difficult requires an agent to actually play it, so we cannot reliably quantify the frequency of such cases.

As a proxy, we evaluated all seven independently trained MAPS-PPO agents on the same 100 evaluation maps and found that none cleared all 50 waves on approximately 6% of the maps. This does not establish that these maps are unwinnable, only that none of our trained agents could solve them. Moreover, since our criterion for solving the environment is a median of 50 waves survived with an IQR of 0, this residual 6% has limited bearing on our reported conclusions.

We have added these findings to Appendix B.2 (Map Validity and Feasibility).

Q3. Generalization. We agree that evaluating generalization across different map distributions would be valuable. However, the current procedural generator uses fixed parameters governing map topology, including branch structure, elevation frequency, and buildable terrain radius. While different seeds produce diverse maps, systematically shifting the underlying topology distribution would require modifying the generator itself.

Q4 Difficulty control. Partial observability is a built-in feature of the environment, and fog-of-war cannot currently be disabled. However, enemy strength is configurable, and we have added an experiment examining its effect on agent performance in Appendix F.6. The impact of tower selling on training was already evaluated in Appendix F.5. We have additionally added an evaluation of the trained models with the tower-selling mechanic in Table F.3. Enabling tower selling reduces performance for both agents by a similar margin. Increasing the enemy HP multiplier degrades both models similarly, with performance holding steady until around the default multiplier before declining.

Q6 Benchmark mechanics. We agree that this distinction is worth clarifying. Each mechanic is intended to stress a distinct RL capability; partial observability, for instance, is designed to evaluate decision-making under incomplete information, and cannot be isolated from the environment. Several mechanics are already evaluated independently in the paper via their corresponding ablations. Others are deliberately left untested in isolation by design: they are exposed as tunable configuration parameters so that future work can study them independently if needed, rather than being exhaustively pre-evaluated by us, which would preempt the targeted experimentation the benchmark is intended to support.

We hope these clarifications address the reviewer’s main concerns. We would be glad to discuss further and hope these additions are helpful for positively reassessing our work.

Dear Reviewer,

We would appreciate it if you could take a moment to review our rebuttal and the additional experiments and revisions. We would be happy to clarify any remaining concerns or questions.

Thank you for your time and consideration.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response to Reviews

Dear Reviewers,

We thank you for your thoughtful and constructive reviews. The depth of your comments reflects the substantial effort invested in evaluating our submission, and your feedback has helped us strengthen the manuscript.

We particularly appreciate the reviewers' recognition of the strengths of our work, including its combination of challenging RL capabilities, procedural generation, computational efficiency, configurable difficulty, and strong baselines that demonstrate its potential as a useful and accessible benchmark for the RL community.

While we address each reviewer's weaknesses and questions individually, one important concern was raised by all three reviewers: the insufficient number of independent training runs and the resulting lack of statistical robustness in our main results. We therefore address this issue collectively here.

We agree that this was a significant weakness of the original submission. To address it, we have now trained six additional models for both PPO and MAPS-PPO using different seeds and revised the manuscript accordingly, particularly Table 1, Figure 3 and Appendix G (statistical analysis). Each run is controlled by two independent seeds, governing the training process and procedural environment generation, respectively, as detailed in Section 4.1. The additional experiments confirm the robustness of our main findings. Variability across independent training runs is substantially smaller than variability across evaluation episodes, and none of the paper's original conclusions are affected.

To facilitate verification of all modifications, we provide (i) a revised manuscript with all edits highlighted in the supplementary material and (ii) a clean updated version as the main submission.

We invite the reviewers to examine our detailed responses and consider the additional analyses and revisions in their evaluations.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.