- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
TAPE makes the claim that standard RL algorithms suffer from OOD generalization issues, but that the specific sensitivity of algorithms to such issues is often confounded because existing benchmarks simultaneously change dynamics, goals, rewards, and observations. They claim this conflates the issues of OOD performance with task difficulties in many cases. To combat this, the authors develop the TAPE benchmark that holds all but the transition dynamics fixed, enabling a low computational cost benchmark to test OOD generalization capabilities of a variety of algorithms at scale.
Critical strengths
The paper compares many seeds and many algorithms under unseen dynamics rule shift. The variety and scale of experimentation provide a substantial evaluation of TAPE as a benchmark. This, alongside the coverage of many algorithms, provides significant evidence for the potential utility of TAPE.
The paper also introduces a simple, computationally cheap, and relatively controlled environment for rapidly testing the OOD generalization capabilities of RL algorithms under transition dynamics. This makes systematic testing of RL algorithms and comparisons much simpler, and the lightweight nature of the benchmark makes the evaluation of many algorithms over many seeds and transition rules very feasible even with limited computational resources.
Critical weaknesses
One of the major motivations of this work is the pressing need to isolate the OOD generalization capabilities of algorithms from other confounding factors that accompany a shift in benchmark environments for evaluation. The authors aim to do this by isolating the changes in their TAPE benchmark to only the transition rules of the cellular automaton, but this also changes the difficulty of the control task. As a result, this introduces an additional confounder in the form of the underlying difficulty of the problem. The authors provide additional analysis to address this entanglement, but it is unclear to me how much of the performance changes to attribute to generalization issues and how much to attribute to changes in the underlying task difficulty. Given that the work is all about mitigating confounders, this seems like a serious confounder that is unresolved and makes me question whether the central claims of this work have been sufficiently backed by evidence.
Although this paper studies many algorithms under dynamics rule shift, the simplicity of the testing protocol is certainly a limiting factor when considering the impact of this work. This is not to say the simplicity itself is the issue here, but the same simplicity that makes this paper interesting also makes me wonder how broadly conclusions from this work apply to the practical problems faced by the RL community. The simple case of evaluating under stable, periodic, and chaotic dynamics and a one-dimensional elementary cellular automaton makes the takeaways cheap to compute, but also makes it unclear what scientific insights from this controlled experiment can be lifted and applied elsewhere in the field. This is a good start for helping isolate variables and helps us conduct better science in RL, but I am not convinced of the general utility of TAPE.
Questions for authors
Is there any way to change the transition rules in a way to control for underlying task difficulty? Is there some way to provide some kind of difficulty-adjusted analysis? I think this would greatly enhance the contributions of the paper.
Given the low success rates of all the algorithms, alongside the low success rate even for an MPC method with access to the true dynamics, is the choice of the automaton environment appropriate to measure for OOD generalization capabilities? Would a simpler environment with higher OOD generalization success rates for the oracle be a better choice to make your conclusions stronger?
3 comments
Response to Reviewer 3cck (2)
2. Full 20-Seed Type-Balanced Split (, 48 Train / 24 Test)
We also expanded the type-balanced split (16 train and 8 test rules per class: stable, periodic, chaotic) to all 20 seeds across our full model suite:
Table B: Type-balanced split (, 20 seeds, 100 initial states/rule)
| Method | ID Strict Success | OOD Strict Success | Method | ID Strict Success | OOD Strict Success |
|---|---|---|---|---|---|
| PEARL-style | 26.08% | 26.71% | DRQN | 14.80% | 20.58% |
| DQN+RAD | 25.12% | 26.81% | PPO | 14.41% | 19.98% |
| DQN | 25.09% | 26.22% | PPO-LSTM | 14.35% | 20.06% |
| CURL | 20.55% | 22.89% | PPO+DR | 14.20% | 19.93% |
| Dreamer-style | 15.11% | 20.70% | History Full / Reset (400k) | 31.44% / 30.90% | 30.97% / 30.19% |
When rule-type difficulty is balanced between train and test, the large negative ID$\to$OOD drop disappears (ID and OOD rates are closely matched, and overall success rises substantially). Moreover, on this balanced split, retaining recurrent history (History Full) reliably outperforms resetting state (History Reset) on OOD rules by ** (95% CI strictly ), reversing the pattern seen on the chaotic-heavy split.
3. Swap Split Check
Training on the original 30 test rules and testing on the original 60 training rules yields ID / OOD for DQN and ID / OOD for DRQN. Because the two pools differ in size and rule composition, success rates do not simply invert—confirming why our same-rule 3-fold cross-pairing is the most rigorous way to isolate true generalization from rule difficulty.
Response to Reviewer 3cck (3)
Response to W2 and Q2: Oracle Solvability Calibration and Generalizable Scientific Insights
- Reviewer's Points (W2, Q2): Given the low success rates of all algorithms and even the true-dynamics MPC reference, is the CA environment appropriate for measuring OOD generalization, or would a simpler environment with higher oracle success be better? What broader scientific insights from this controlled setting apply elsewhere in RL?
1. Re-Calibrated Solvability & Higher-Oracle-Success Regimes (Addressing Q2)
Under our re-verified evaluation across all 90 rules and 100 frozen initial states ( episodes), the environment is substantially more solvable than earlier preliminary numbers suggested:
- At : Known-dynamics random-shooting MPC (512 candidates, depth 8) achieves 44.57% mean strict success across all 90 rules (solving of episodes on 53/90 rules; and 43.06% on the 72-rule balanced split), compared to 12.47% for passive unperturbed CA evolution (non-zero on 20/90 rules), 23.29% for a simple rule-agnostic first-mismatch greedy heuristic, and 7.16% for uniform random actions. Increasing the planning budget (2,048 candidates, depth 16, ) on 3 designated diagnostic rules raises MPC success to .
- At (Simpler Regime with Higher Oracle Success): We explicitly include the regime where the exact same MPC budget reaches 56.51% across the 90 rules. In this higher-solvability regime,
PEARL-styleachieves 33.96% ID and 33.48% OOD—beating the first-mismatch greedy heuristic on both splits ( ID and OOD, 95% CIs strictly ).
Thus, the low RL success rates at () are not an artifact of an unsolvable environment; rather, they expose a sharp failure of current RL pipelines relative to both known-dynamics planning () and simple heuristics ().
2. Transferable Scientific Insights for the Broader RL Community (Addressing W2)
While 1D cellular automata are intentionally minimal, isolating transition-law shift under an invariant interface yields four concrete mechanistic lessons that generalize to multi-task RL, meta-RL, and world-model research:
- Cross-Task Policy Interference Dominates Single-Task Learnability: When trained in isolation at , standard
DQNand action-equivariantDQN+RADsolve the identity rule (Rule 204) to 100.0% strict success across all 20 seeds. Yet when the exact same Rule 204 is placed inside the 60-rule mixed training pool, strict success collapses to 0.0% (DQN) and 0.60% (RAD). Even when every individual task is trivially solvable under an identical interface, naive multi-task optimization suffers severe representation and gradient interference across conflicting dynamics. - Standard RL Pipelines Underperform Simple Heuristics Under Dynamics Mixing: At ( steps), every evaluated RL family—including domain randomization (
PPO+DR), contrastive learning (CURL), and latent world models (Dreamer-style)—performs significantly worse than a rule-agnostic greedy heuristic (), showing that observation augmentations and horizon randomization do not confer robustness to discrete transition-law shifts. - End-to-End Recurrent RL Fails at Implicit System Identification from Reward Alone: Adding recurrent memory (
DRQNandPPO-LSTM) without explicit dynamics-inference supervision fails to improve over memoryless baselines even after environment steps ( ID / OOD, lagging -step feedforwardDQNat / ). Architectural recurrence alone does not automatically induce latent-law identification under sparse/shaped control rewards. - Posterior Miscalibration in Chaotic Regimes: Explicit rule-bit inference (
HistoryQMDP) boosts OOD success to , yet retaining recurrent state across steps (Full) hurts performance on chaotic-dominated splits ( vs. step-wiseReset) while helping on balanced splits (). This highlights a general risk for Bayes-adaptive and belief-state controllers: when out-of-distribution dynamics are chaotic, accumulating history in an approximate posterior can amplify overconfident model misspecification.
We hope these difficulty-controlled experiments and calibrated solvability analyses address your concerns and clarify TAPE's value as a diagnostic tool for RL generalization.
Response to Reviewer 3cck (1)
We sincerely apologize for the delayed response. Designing and running the new difficulty-deconfounding experiments (including the 3-fold same-rule cross-pairing, full 20-seed type-balanced splits, and 90-rule planner calibrations) across all 20 seeds required substantial compute time, alongside concurrent rebuttal commitments for other submissions in our group. We deeply appreciate your thoughtful review—your core critique regarding confounding task difficulty with OOD generalization and the broader scientific takeaways of a minimal CA benchmark directly motivated our strongest new experiments.
Below is our point-by-point response addressing W1, W2, and Q1–Q2:
Response to W1 and Q1: Deconfounding Task Difficulty from True Generalization
You identified a genuine confounder in our original default split (
make_splits(60, 30, seed=0)), where farthest-point sampling placed 22/30 chaotic rules in the test set versus 32/60 in the training set. To cleanly separate intrinsic rule difficulty from true rule-shift generalization, we conducted three complementary analyses across 20 training seeds (0–19) and 100 frozen Bernoulli(0.5) initial states per rule at L=32,H=32:1. Same-Rule 3-Fold Cross-Pairing & Planner-Normalized Gap (Direct Difficulty Control)
Instead of comparing different rule sets, we partitioned the 90 rules into three balanced 30-rule groups (A,B,C) and trained three 60-rule models (
ABholding outC,ACholding outB, andBCholding outA) across 20 seeds each (plus a second independent partitioning seed for robustness).DQNonAB→Cgets 8.31% seen vs. 3.42% unseen, whereas onAC→Bit gets 6.15% seen vs. 9.20% unseen because foldBis intrinsically easier than foldC.Table A: Same-rule paired difference (Seen minus Unseen on identical rules, L=32, 20 seeds)
(Note: For the planner-normalized column, each rule's gap is normalized by its known-dynamics MPC success rate, keeping raw values when the denominator is <0.05 to prevent ratio blow-up; excluding Rule 204 leaves
History Fullat +1.95 pp, 95% CI strictly >0.)DQNand end-to-endDRQNexhibit virtually zero generalization gap (+0.07 pp and −0.01 pp)—meaning the large ID–OOD drop on the original split was predominantly driven by test-set chaotic rule concentration rather than memorization of seen rules. By contrast, explicit rule-inference (History Full) exhibits a genuine, difficulty-adjusted seen-rule advantage of ∼1–2 pp (∼4 pp MPC-normalized).