- Rating
- 3(+1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper presents a new tower-defense benchmark, ProcTD, for reinforcement learning agents. The benchmark is intended to test a combination of spatial reasoning, long-horizon planning, and complex action selection. The benchmark can also be used as an asymmetric competitive multi-agent environment with a learned defender and attacker agent. To provide a starting point for researchers, the paper evaluates a combination of heuristic and learned agents, finding that a learned high-level controller with low-level heuristic tower placement performs the best and is more robust to increasing map size. The main contributions are the new benchmark, initial baselines (including DQN and Dreamer in the appendix), and the hierarchical MAPS-PPO model which outperforms standard PPO. They validate these results with a single seed of each run, as well as a 3 seed sensitivity analysis. There are also additional analyses of the impact of various environment settings (such as enabling/disabling selling, map size, and reward shaping) in the appendix.
Critical strengths
The paper presents a useful new benchmark with many features that would be of interest to the reinforcement learning community. The environment provides controls over the map size to increase the overall complexity of the state and action spaces. Modifying the sell price can change how permanent decisions are and affects how important long-horizon planning is. The benchmark even supports a learned adversary, enabling an asymmetric multiplayer game and presenting opportunities to incorporate unsupervised environment design methods.
Although some details are missing from the main paper, the full paper including the appendix describes the environment in extremely clear detail, with auxiliary results testing the importance of several game mechanics and the performance of multiple RL methods. In particular, the paper introduces several interesting heuristic baselines, and ships with a strong hierarchical neuro-symbolic baseline to demonstrate that strong performance is achievable with strong priors and domain knowledge, but not yet achieved by standard RL methods. This, combined with the environment's fast interaction speed, places the benchmark in a good state to provide immediate value to the research community.
Critical weaknesses
I have some concerns regarding the clarity, comparisons to prior work, and experimental design in the main paper.
First, the main paper does not explain a few crucial details of the environment that are necessary for evaluating the claims of the paper. For instance, the main paper does not describe how long an episode is, the frequency with which the agent interacts with the environment, or how sparse feedback is for those actions. All of these factors impact the degree to which long horizon planning is necessary in the environment. It is also not clear until the appendix that the cost of each tower type increases geometrically as you purchase them, so fact that agents "must compose a portfolio" is actually forced by the environment rather than an optimal strategy for agents to discover. Similarly, XP is never explained, nor are the upgrade cards. I recognize that it would be impossible to explain everything in the main paper, but there should be at least a sentence explaining each mechanic so that it is possible for the reader to reason about the environment without going through the full appendix.
Second, the paper states that ProcTD combines spatial reasoning, partial observability, and procedural variation while prior benchmarks "emphasize different challenges", but NetHack, Craftax, and microRTS, for instance, tests all of these capabilities. NetHack and Craftax also test complex action spaces that this paper emphasizes. A more precise comparison to prior work seems necessary to convey what value ProcTD adds to the current research ecosystem. Similarly, the paper is missing a qualitative analysis of the trained agents. It's not clear whether any of these capabilities are actually necessary, or if the agent is learning a very simple and unremarkable strategy. The paper should provide evidence to defend the claim that this is a good benchmark for the specific capabilities.
Third, the main results in the paper use an insufficient number of seeds and the statistical analysis in the appendix is largely invalid. My understanding is that all of the results in the paper are reported from individual runs, except for the seed sensitivity analysis which uses only 3 seeds. It is not possible to produce uncertainty measures from a single seed, so these results all could be attributed to random noise. If the method were extremely expensive it may be excusable, but a key selling point of the benchmark is fast iteration speed. The costs listed in Table F2 suggest that repeating the main experiments for 5 seeds would not be particularly costly. On a similar note, the statistical analysis in Appendix G is not informative. By treating evaluation seeds as individual experimental units for the test, you are testing the claim that the particular MAPS-PPO agent that you trained outperforms your particular PPO and MAPS-RND agents. To test the claim that MAPS-PPO trains better agents in ProcTD, you need to train multiple agents, evaluate each for a large number of seeds, and treat each one as an individual experimental unit.
Questions for authors
It's unclear to me how difficult the environment is from the main paper. How often does the agent interact with the environment? Is one episode 1 wave or 50 waves?
How was the environment balanced, and how were the heuristic components developed and tuned?
This is not a request that will affect my score in any way, but a suggestion that you may want to incorporate to strengthen the paper. The results in Table 2 suggest that replacing both the high-level and low-level components of the hierarchical model improves performance. This seems to suggest that a learned high-level and learned low-level controller would outperform the current baselines, which only learns the high-level controller. I don't believe the fully learned model is necessary to defend the claims made by the paper, but it would be a valuable addition.
4 comments
Rebuttal to Tir8 [2/2]
Q1. Environment difficulty and episode structure. An episode lasts up to 50 waves, ending earlier if the defender's base is destroyed. During each wave, the agent can build, upgrade, or wait, with each action advancing the simulation by 50 internal game ticks. Between waves, the agent selects a technology card. We have clarified the episode structure, interaction frequency, and reward feedback in Section 3.1.
Quantifying difficulty more broadly is challenging. As a proxy, we evaluated seven independently trained MAPS-PPO agents on the same 100 evaluation maps and found that none cleared all 50 waves on approximately 6% of them. This suggests that some maps remain challenging even for our strongest baseline, although it does not establish that they are unwinnable. Further details are provided in Appendix B.2.
Q2. Environment balancing and heuristic development. Environment balancing was performed iteratively. The mechanics were inspired by existing Tower Defense games, while initial parameters, including enemy health, wave budgets, and gold economy, were selected based on our intuition of balanced gameplay.
We first evaluated the Advanced Heuristic over 100 episodes, aiming for a mean survival of approximately 25 waves while keeping the fraction of full 50-wave clears below 5%. This ensured that the environment was challenging but not impossible to complete. We then trained PPO and subsequently MAPS-PPO, adjusting mainly the coefficients of existing mechanics to increase difficulty while preserving solvability.
The placement heuristics were developed based on our experience with Tower Defense games. They prioritize strategically valuable locations, such as chokepoints and elevated terrain, with additional tower-specific rules. For example, Sniper towers favor central positions to maximize map coverage. Their implementation is detailed in Appendix E.1.
Q3. Two-stage learner. Thank you for the suggestion. We agree that combining learned high-level and low-level controllers would be an interesting extension, and we are currently exploring how best to implement this approach.
We hope that the additional experiments, expanded methodological descriptions, and revised statistical analysis address the reviewer's main concerns. We thank the reviewer again for their constructive suggestions and hope these improvements will encourage a positive reassessment of our submission.
Thank you for the detailed response and answers to my questions. I believe W1, Q1, Q2, and Q3 have been sufficiently addressed.
Regarding W1, the geometrically increasing cost per tower type seems like a strong incentive to build a roughly uniform tower distribution. This is an acceptable choice, but it may hide balance issues between the towers and limit strategic diversity. You may consider testing a smaller geometric increase shared across all towers, to see what portfolio the agents develop without an incentive toward uniformity. However, it seems that the claim regarding agents needing to "compose a portfolio" is no longer in the paper, so this is no longer a concern.affecting my rating.
For W2, the current language is an improvement but still does not seem to clearly define the unique challenges of a tower defense game. For instance, Craftax and NetHack both require long term planning and commitments. It's still not clear to me what different challenges a tower defense presents, or ProcTD in particular. The term "persistent spatial commitments" does not sound meaningfully different from long-term plannin. The Related Works section also implies that Craftax and NetHack do not require strategic planning or resource management, while those are likely the two most significant challenges in those environments. Similarly, it suggests that SMAC and Neural MMO do not present challenges like exploration, resource management, etc. Overall, the related benchmark comparison feels imprecise, which makes it less clear what new challenge ProcTD presents.
Regarding W3, I thank the authors for conducting additional runs of the main PPO baselines, and it is encouraging to see that the main claims have not changed. However, additional runs of the heuristic baselines would also be necessary to provide a fair comparison against the 6 PPO runs, particularly for statistical tests to defend claims such as "Random and Simple Heuristic perform similarly, whereas Advanced Heuristic and MAPS-RND improve substantially when structured placement is introduced. PPO further improves over the non-learned baselines on the 16 × 16 environment, while MAPS-PPO achieves the strongest overall performance and consistently reaches the full episode horizon." Without these results, these comparisons are difficult to trust.
I will raise my score to a 3, because based on prior work, it is not surprising that PPO and hierarchical PPO outperform heuristic baselines, and because the new experiments sufficiently demonstrate that MAPS-PPO outperforms PPO, which was the least predictable comparison. However, I strongly encourage the authors to expand their experiments to include additional seeds for all baselines.
Thank you for your constructive feedback and for raising your score. We appreciate the time and effort you have invested in reviewing our work. Your comments have helped us substantially improve the manuscript.
Regarding tower costs (W1), our initial version of the environment did not include cost scaling at all. However, this led to the agent spamming towers in every buildable tile (on the 16x16 map) and made balancing a lot more challenging. Additionally, it did not let us study the build vs. upgrade tradeoff properly, since building more towers came with no additional cost. The environment itself is inspired by Rogue Tower, as noted in our paper, and the game uses a similar scaling to circumvent this. As such, we decided to go the route the game goes. Regardless, we do agree that a smaller geometric increase shared across towers would be an interesting test; but since this would require major rebalancing, we keep it open for the future.
Regarding statistical robustness (W3), the heuristic baselines are deterministic and do not involve training, so there is no seed variability to account for. However, the random baseline and MAPS-RND do include a random component. Varying this seed results in a slightly different result than we initially had (the only change is the significance of the results). The manuscript has been updated with the results and the additional information on the seeds. Thank you for bringing this to our attention.
Thank you again for your thoughtful suggestions and positive reassessment.
Rebuttal to Tir8 [1/2]
We thank the reviewer for their thorough and constructive evaluation. We particularly appreciate the detailed feedback on the environment's mechanics, its positioning relative to existing benchmarks, and the statistical robustness of our experiments. These comments have helped us identify several important areas for improvement, which we address below.
W1. Missing environment details. We agree that space limitations led us to omit some important environment mechanics from the main paper, making it more difficult to understand how the environment functions. With the additional page, we have expanded Section 3 (The ProcTD Benchmark) to clarify the episode structure, interaction frequency, reward feedback, XP progression, and technology upgrade cards. The changes are highlighted in the revised manuscript provided in the supplementary materials, while more comprehensive descriptions remain in the appendix.
Regarding tower costs, the geometric increase in construction costs does not force agents to diversify, as they remain free to invest exclusively in one tower type if they consider it optimal. Rather, this mechanic is intended to discourage indiscriminate tower construction, particularly in later waves when more gold becomes available. Similar cost-scaling mechanisms are also used in existing Tower Defense games.
W2. Comparison to existing benchmarks and qualitative analysis. We agree that existing benchmarks such as NetHack, Craftax, and microRTS already combine several of these capabilities. ProcTD's main distinction is its emphasis on persistent spatial commitments: unlike environments centered on reactive control, towers cannot be relocated or changed to another type once built. Early placement decisions therefore have long-term consequences, while tower selling, when enabled, introduces an economic cost for correcting them. ProcTD also provides an asymmetric attacker–defender setting, where the two agents have distinct objectives, observations, and action spaces, enabling the study of adaptive offensive and defensive strategies. We have expanded Section 2 (Related Work) to clarify these distinctions.
We have also added a qualitative analysis of learned MAPS-PPO strategies in Appendix F.3. This examines the distribution of tower-building and upgrading actions across waves and individual maps. We observe a clear progression from building towers early to upgrading them later. While the overall tower-action distributions are similar across maps, as expected given the tower–enemy matchup mechanics (Avery et al., 2011), the actions taken at specific waves vary considerably between maps. These findings indicate that MAPS-PPO learns temporally structured and map-dependent strategies rather than simply following a fixed action sequence.
W3. Statistical robustness. We agree with the reviewer's criticism of the original statistical analysis, particularly the distinction between evaluation episodes and independent training runs. We have therefore trained six additional independent runs for both PPO and MAPS-PPO, bringing the total to seven runs per method and map size, each evaluated on the same 100 unseen maps. We have also revised Appendix G.2 to include a hierarchical bootstrap that resamples training runs and evaluation episodes, accounting for both sources of variability. The updated analysis confirms that MAPS-PPO consistently outperforms PPO across both map sizes.