acceptodds
Under review as a conference paper at ICLR 2027

CrossMaze: Benchmarking Zero-Shot Structural Generalization in Offline Continuous Control

Abstract

Policies trained on offline data are typically evaluated in familiar configurations represented in the training data, leaving their ability to transfer to unseen environments uncertain. We introduce CrossMaze, a benchmark for zero-shot structural generalization in offline continuous control. CrossMaze combines M16, a fixed diagnostic suite, with SeedMap, a procedural track for scaling training-layout diversity, in both PointMaze and AntMaze. We use CrossMaze to systematically evaluate eight imitation and offline reinforcement learning (RL) methods, including behavior cloning and offline goal-conditioned reinforcement learning (GCRL) across algorithm families and training-layout scales. As a strong reference baseline for CrossMaze, we introduce PIDA, a pretrained language-conditioned behavior-cloning policy that maps serialized maze layouts, states, and goals directly to continuous control actions. On M16, PIDA achieves held-out success rate of on PointMaze and on AntMaze, compared with and for the best evaluated non-PIDA baselines, respectively. Scaling experiments reveal that broader training-layout coverage yields uneven gains across algorithms and embodiments, while PIDA remains a strong reference with fewer layouts under the stated protocol. CrossMaze provides a controlled, reproducible testbed for measuring structural generalization in offline continuous control. Source code: https://anonymous.4open.science/r/llm_offline-B54D/

Primary area:
reinforcement learning
Keywords:
Offline reinforcement learning, Structural generalization, Zero-shot generalization, Continuous control, Reinforcement learning benchmarks

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 4 reviews, above 27% of the others.
Reviewer_eoz9OpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces CrossMaze, a benchmark for zero-shot structural generalization in offline continuous control. Layouts act as contexts in a goal-conditioned contextual MDP, with disjoint training and held-out layouts and no policy updates during evaluation. The benchmark includes two tracks for PointMaze and AntMaze: M16, a fixed diagnostic suite with 16 training and 10 held-out layouts per embodiment, and SeedMap, a deterministic procedural generator for scaling the number of training layouts. Eight behavior-cloning, offline RL and offline GCRL baselines are evaluated alongside PIDA, a behavior-cloning policy using Qwen3-0.6B with LoRA that receives the state, goal and layout as text and outputs continuous actions. On M16, PIDA achieves 69.8% and 72.7% held-out success on PointMaze and AntMaze, compared with 17.6% and 4.4% for the best non-PIDA baselines. Further experiments vary the number of training layouts and the width and depth of an MLP BC policy, and test the effects of physical observation descriptions and action descriptions in PIDA's prompt.

Critical strengths
  1. A clear evaluation methodology. Training and held-out layouts are disjoint, each layout contributes equally to the reported success rate, and model selection uses only training-side data. The paper also specifies versioned maps, seeds, datasets and start–goal pairs to support reproducibility.

  2. A strong reference policy. PIDA uses standard components and behavior cloning, yet achieves much higher held-out success than the evaluated baselines. On AntMaze, its 72.7% held-out success is close to its 76.7% average over the 12 custom training layouts. On PointMaze, it reaches 34.2% held-out success after training on a single layout.

Critical weaknesses
  1. The authors do not explain the failures on AntMaze training layouts. In Table 2, all eight non-PIDA methods score 0.0% on the U-Maze and Medium training layouts, despite the dataset containing 5,000 trajectories per training layout. Their average success over the 12 custom training layouts is at most 18.0%. The paper describes hyperparameter tuning but does not explain these failures. Since the baselines already perform poorly on training layouts, their low held-out success does not clearly show how much additional difficulty comes from encountering new layouts.

  2. The comparisons do not explain PIDA's advantage. The numerical baselines receive a flattened, padded map through an MLP, while PIDA receives a row-wise text map through a pretrained transformer. There is no comparison with a CNN map encoder or a transformer trained from scratch on the same text input. The paper acknowledges that the comparison cannot separate pretraining, architecture, representation and optimization. These experiments would help establish whether a different map encoder improves the baselines and how much pretraining contributes to PIDA's performance.

  3. The evaluation focuses on high-quality trajectories. The reported experiments use successful, high-quality navigation trajectories collected by controllers. The paper does not examine whether the comparison between behavior cloning and offline RL changes when the data include failed or suboptimal trajectories. This limits the conclusions that can be drawn about the methods across different offline datasets. More varied offline data are listed as future work, leaving this question unanswered in the current evaluation.

Questions for authors
  1. Could you explain why all eight baselines score 0% on the AntMaze U-Maze and Medium training layouts in Table 2?
  2. Have you tested whether a CNN map encoder or a transformer trained from scratch on PIDA's text input reduces the gap to PIDA?
3 comments
AuthorsOpenReview ↗

Response to Reviewer eoz9 Part-1

We thank the reviewer for recognizing the clarity of the evaluation protocol and the strength of PIDA as a reference policy. To address your concerns, we have thoroughly reviewed each comment and provided comprehensive responses, supported by further clarifications and additional experimental results. The single-layout baseline checks are reported in the orange-marked Appendix G, Table 8 and Figure 6. If any issues remain, we would be more than glad to clarify them further.

W1: The authors do not explain the failures on AntMaze training layouts.
Q1: Could you explain why all eight baselines score 0% on the AntMaze U-Maze and Medium training layouts in Table 2?

A1: Thank you for highlighting the zero-success results on the two AntMaze training layouts. The revision now explains them. These zeros appear only when a single baseline policy is trained jointly on the M16 AntMaze layouts. M16 trains one shared layout-conditioned policy across multiple layouts, rather than a separate policy for each layout, so these entries indicate failures to learn particular training tasks under the joint-training setting.

The following additional and existing experiments support this interpretation.

  • Our implementations achieve strong single-layout results. In the additional single-layout checks, our BC and offline RL implementations (TD3+BC, IQL, and ReBRAC) achieve 76.1%–92.6% success on CrossMaze AntMaze U-Maze, compared with 0% under joint M16 training. For GCRL, our QRL, CRL, and HIQL implementations achieve 80.2%, 93.8%, and 94.2% on the official OGBench AntMaze Medium benchmark, respectively. These findings are consistent with prior evidence that offline RL and GCRL methods can learn single-layout AntMaze tasks [1, 2]. Full settings, results, and training curves are provided in the orange-marked Appendix G, Table 8 and Figure 6.

  • The new map-encoder and sensing experiments retain the BC training pipeline. CNN and Transformer map encoders improve PointMaze training performance, while CNN-BC with sensing reaches 75.4% training and 71.9% held-out success on AntMaze (Section 5.4, Table 3). These results show that the shared BC implementation can learn effective AntMaze policies when spatial information is made easier to use. Part 2 of this response discusses this experiment in more detail.

  • M16 and SeedMap use the same baseline implementations. Several baselines improve with broader layout coverage on SeedMap. For example, AntMaze CRL's held-out success increases from 3.3% with 100 training layouts to 43.3% with 500 layouts (Section 5.3 and Appendix E, Table 7). Thus, the same implementation can learn policies that transfer to unseen AntMaze layouts under broader structural coverage.

  • PointMaze and AntMaze share the algorithm implementations, with observation and action dimensions adapted to each embodiment. On PointMaze M16, BC achieves 60.7% training success, including 97.3% on U-Maze and 96.0% on Medium, although held-out success remains at 12.2% (Tables 1 and 3). This demonstrates learning on the training layouts while exposing a substantial transfer gap, rather than a general inability of the implementation to learn.

Single-layout results and implementation details. The table below reproduces the single-layout checks in Table 8 of the revised manuscript.

Values are cross-checkpoint trimmed mean success rates (%), excluding one highest and one lowest evaluation value per run. OGBench success averages the five official tasks equally.

Project U-Maze algorithmMean success (%)OGBench Medium algorithmMean success (%)
BC92.0GCBC23.5
TD3+BC92.6QRL80.2
IQL76.1CRL93.8
ReBRAC85.0HIQL94.2

We also agree with the reviewer’s reading of Table 2: on AntMaze, the low baseline scores mainly reflect the difficulty of learning 16 layouts jointly, so they do not show on their own how much difficulty new layouts add. The new Table 3 answers that question for policies that do learn the training layouts. For CNN-BC with sensing, held-out success is 3.5 points below training success on AntMaze, and 32.2 points below on PointMaze (82.2% vs. 50.0%). We have clarified in Section 5.2 and Appendix G, marked in orange, that the AntMaze M16 results include difficulty learning the training tasks themselves.

[1] Tarasov, D., Kurenkov, V., Nikulin, A., and Kolesnikov, S. Revisiting the Minimalist Approach to Offline Reinforcement Learning. NeurIPS, 2023.

[2] Park, S., Frans, K., Eysenbach, B., and Levine, S. OGBench: Benchmarking Offline Goal-Conditioned RL. ICLR, 2025.

AuthorsOpenReview ↗

Response to Reviewer eoz9 Part-2

W2: The comparisons do not explain PIDA’s advantage.
Q2: Have you tested whether a CNN map encoder or a transformer trained from scratch on PIDA's text input reduces the gap to PIDA? These experiments would help establish whether a different map encoder improves the baselines and how much pretraining contributes to PIDA’s performance.

A2: Thank you for suggesting these comparisons. We have added CNN-BC and Transformer-BC, along with diagnostics that supply explicit spatial features to MLP-BC and CNN-BC. These comparisons help distinguish the benefit of changing the map encoder from that of making spatial relations explicit, probing a potential source of PIDA's advantage.

CNN and Transformer map encoders. The CNN processes the two-dimensional maze through spatial convolutions. The Transformer treats grid cells as tokens, uses two-dimensional positional encodings, and masks padded cells. Both produce a 256-dimensional map feature and use the same downstream feature-fusion and policy-backbone architecture. Implementation details are in the blue-marked Appendix D.

Explicit spatial features. Sensing adds the agent's two-dimensional grid-cell index and four adjacent-wall indicators while retaining the original map and continuous state. These externally supplied features make spatial relations explicit; they are not learned by BC and contain no shortest path, subgoal, recommended direction, or expert action. These are feature-assisted diagnostics, not identical-input encoder comparisons.

Table 3 reports these results alongside MLP-BC and PIDA. Success rates (%) are mean ± standard deviation over five training seeds, with equal layout weighting.

Policy / map encoderSensingPointMaze trainPointMaze held-outAntMaze trainAntMaze held-out
BC / MLPNo60.7 ± 0.812.2 ± 1.39.5 ± 1.81.8 ± 1.4
BC / TransformerNo68.8 ± 1.219.8 ± 1.25.8 ± 2.52.2 ± 2.4
BC / CNNNo73.8 ± 1.028.7 ± 1.76.7 ± 2.03.9 ± 1.3
BC / MLPYes79.6 ± 2.041.7 ± 2.067.1 ± 11.449.4 ± 11.8
BC / CNNYes82.2 ± 1.650.0 ± 1.575.4 ± 3.171.9 ± 2.3
PIDANo78.5 ± 5.869.8 ± 1.875.6 ± 2.472.7 ± 2.4

Changing the encoder helps, but does not close the gap on its own. On PointMaze, CNN-BC improves held-out success from 12.2% to 28.7%, while Transformer-BC reaches 19.8%. These improvements support the value of spatial inductive biases, but both remain below PIDA's 69.8%. On AntMaze, CNN-BC and Transformer-BC without sensing reach 3.9% and 2.2% held-out success, respectively.

Explicit spatial information produces substantially larger improvements. Adding sensing to MLP-BC raises held-out success to 41.7% on PointMaze and 49.4% on AntMaze, gains of 29.5 and 47.6 percentage points over MLP-BC without sensing. Combining sensing with the CNN further raises success to 50.0% and 71.9%, respectively. The latter approaches PIDA's 72.7% on AntMaze, while a gap remains on PointMaze. These results indicate that access to useful spatial information is a major factor among the tested BC modifications, with map encoding providing complementary benefits. They are consistent with the hypothesis that PIDA benefits from pretrained spatial-semantic priors. The blue-marked Section 5.4 and revised conclusion discuss these findings.

A Transformer trained from scratch on PIDA's text input. A controlled text-input comparison would further clarify the roles of architecture and pretraining. Transformer-BC is trained from scratch on grid-cell tokens, not PIDA's serialized text; the latter comparison is not included. For a from-scratch version of the full PIDA backbone, the training protocol also matters. PIDA freezes its 0.6B-parameter pretrained backbone and trains LoRA adapters and the action-decoding components. Training the full random backbone would change the trainable-parameter budget and may require different optimization and training budgets. Thus, comparing PIDA with a fully trained, randomly initialized backbone makes it difficult to vary initialization alone while keeping the training protocol fixed. The new experiments directly establish the benefit of structured map encoding and provide evidence on how much pretraining contributes. Even with a CNN or Transformer map encoder, numerical BC stays below 29% held-out success on PointMaze and below 4% on AntMaze. It approaches PIDA only when explicit spatial features are supplied from outside (71.9% vs. 72.7% on AntMaze), and a gap still remains on PointMaze (50.0% vs. 69.8%). PIDA receives none of these features and reaches 69.8% and 72.7% from the serialized map alone. This suggests that its pretrained backbone provides the kind of spatial knowledge that numerical BC has to be given from outside, and points to pretraining as a key factor in its advantage.

AuthorsOpenReview ↗

Response to Reviewer eoz9 Part-3

W3: The evaluation focuses on high-quality trajectories. The paper does not examine whether the comparison between behavior cloning and offline RL changes when the data include failed or suboptimal trajectories.

Q3: Thank you for raising this point. We agree that dataset quality can affect the relative performance of behavior cloning and offline RL, and that this is distinct from the layout-generalization axis evaluated here. The current experiments use successful, high-quality controller-generated trajectories, as specified in Section 3, and all methods use the same fixed dataset within each experimental condition. This provides a controlled starting point for studying whether policies trained on successful behavior can transfer that behavior to unseen structures. With high-quality data, low held-out success cannot be explained by poor demonstrations: even in this setting, which is the most favorable one for behavior cloning, numerical BC reaches only 12.2% on PointMaze and 1.8% on AntMaze held-out layouts.

In particular, successful demonstrations are well aligned with the behavior-cloning objective, whereas learning from mixed-quality behavior can make reward-based selection and improvement more important. We therefore do not interpret the present results as showing that behavior cloning generally outperforms offline RL across dataset qualities. The comparison between PIDA and numerical BC, and the encoder and sensing diagnostics within BC, remain informative about policy representations in the evaluated high-quality-data regime. These comparisons use the same behavior-cloning objective and the same data, so they do not depend on how BC ranks against offline RL.

CrossMaze includes an offline data generator in addition to the fixed datasets, allowing further datasets to be constructed, and the conclusion identifies more varied offline data as an extension. Our empirical conclusions are consequently specific to the reported data regime; whether failed or suboptimal trajectories change the relative ranking of BC and offline RL remains an open question for this benchmark.

Reviewer_qtajOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper addresses an important gap in the evaluation of zero-shot structural generalization in offline continuous control. The authors introduce CrossMaze, a benchmark that provides standardized environments (PointMaze and AntMaze), fixed and procedural layout tracks (M16 and SeedMap), and reproducible evaluation protocols. They also introduce PIDA, a pretrained language-conditioned behavior-cloning policy that serves as a strong reference baseline and achieves substantially higher held-out success rates than the evaluated offline RL and goal-conditioned RL methods. The results reveal significant structural generalization limitations in existing approaches and demonstrate that strong zero-shot transfer to unseen layouts is achievable, while leaving considerable room for further improvement.

Critical strengths
  1. The paper introduces CrossMaze, a benchmark for evaluating zero-shot structural generalization in offline continuous control, formalized through held-out layout evaluation in a goal-conditioned contextual MDP.

  2. CrossMaze provides a shared dynamic-map interface that gives all methods access to the current layout without oracle information, along with deterministic layout generators, versioned datasets, and standardized evaluation protocols to support reproducibility and controlled scaling of layout diversity.

  3. The paper introduces PIDA, a pretrained language-conditioned behavior-cloning policy built on Qwen3-0.6B with LoRA adaptation. PIDA substantially outperforms eight evaluated offline RL and goal-conditioned RL baselines on held-out layouts, demonstrating the feasibility of strong zero-shot structural generalization and establishing a useful reference point for future research.

Critical weaknesses
  1. All eight baselines are evaluated through adapted implementations (a d3rlpy-based adapter for BC, TD3+BC, IQL, and ReBRAC, and OGBench-derived implementations for the GCRL methods), but the paper provides no sanity check demonstrating that these pipelines reproduce published results under standard benchmark settings. This is particularly concerning because every non-PIDA baseline achieves 0.0% success on the AntMaze U-Maze and Medium training layouts (Table 2). Consequently, their low held-out success rates may reflect training or implementation failures rather than limitations in structural generalization. The authors should validate their implementations against published results under a standard single-layout protocol, such as OGBench, and distinguish failures to learn the training tasks from failures to generalize to unseen layouts.

  2. Section 4 presents PIDA as an investigation of whether pretrained sequence models can translate unseen structures into continuous control. Furthermore, two of the four research questions (RQ3 and RQ4) focus on explaining PIDA's advantages. However, PIDA differs from the numerical baselines in pretraining, architecture, and map representation. In particular, PIDA uses a pretrained language model with a text-grid representation, whereas the baselines use flattened numerical maps. The MLP capacity experiments do not isolate these factors. Therefore, the results establish PIDA as a strong reference baseline but cannot explain whether its advantage arises from language pretraining, the architecture, or the input representation. The authors should clarify PIDA's intended contribution and consider comparing it with a randomly initialized version of the same architecture or a non-language model equipped with a structured spatial map encoder.

  3. Since CrossMaze is primarily a benchmark contribution, it is important to establish that its layouts capture meaningful and diverse structural variations. However, the topology-difficulty metric used to characterize M16's structural coverage (Section 3 and Appendix A) relies on manually specified weights, normalization constants, and transformations (Eqs. 5 and 8–10), without empirical justification or sensitivity analysis. Although the paper claims that held-out layouts have difficulty distributions similar to those of training layouts, it reports difficulty ranges only for the training layouts. Furthermore, SeedMap generates only tree-structured mazes, where each pair of cells has a unique simple path. This excludes environments containing alternative routes between the same start and goal, limiting the structural diversity examined in the SeedMap scaling experiments. Reporting held-out difficulty distributions, analyzing per-layout success rates against structural characteristics, and considering a SeedMap variant with loops would strengthen the benchmark's validity and generality.

Questions for authors
  1. On AntMaze M16, all eight non-PIDA baselines achieve 0.0% success on the U-Maze and Medium training layouts (Table 2). Could the authors provide training-success curves for at least CRL and HIQL, along with a single-layout sanity check (e.g., using OGBench's antmaze-medium-navigate dataset and evaluation protocol) demonstrating that their implementations reproduce published results? This would help distinguish training failures from structural generalization failures.

  2. Have the authors evaluated PIDA with a randomly initialized backbone under the same architecture, input representation, and training budget? Such a comparison would help isolate the contribution of language pretraining from the effects of model architecture and text-based inputs.

  3. Have the authors evaluated non-language baselines equipped with structured spatial map encoders, such as CNNs or Transformers operating over grid cells? Since the numerical baselines receive flattened, padded map representations while PIDA processes a structured textual grid, such comparisons would help determine whether the observed performance gap is partly attributable to differences in map representation.

  4. Could the authors report the topology-difficulty scores of all held-out M16 layouts, together with their per-layout success rates? This would help verify the claim that training and held-out layouts have similar difficulty distributions and clarify how structural difficulty relates to generalization performance. Additionally, how were the weights in Equation (5) and the constants in Equations (8–10) selected, and how sensitive is the M16 layout selection to these choices?

  5. How does PIDA differ conceptually and algorithmically from Li et al. (2022), Pre-Trained Language Models for Interactive Decision-Making, which also investigates initializing decision-making policies from pretrained language models and adapting them through behavior cloning?

5 comments
AuthorsOpenReview ↗

Response to Reviewer qtaj Part-1

We thank the reviewer for recognizing the benchmark's structural-generalization setting, reproducible interface, and strong reference policy. We have reviewed each concern and provide detailed responses, clarifications, and additional experiments. In the revised manuscript, encoder and sensing experiments are marked in blue, difficulty-score reporting and analysis in green, and single-layout checks in orange. If any issues remain, we would be more than glad to clarify them further.

W1: Consequently, their low held-out success rates may reflect training or implementation failures rather than limitations in structural generalization.
Q1: Could the authors provide training-success curves for at least CRL and HIQL, along with a single-layout sanity check (e.g., using OGBench's antmaze-medium-navigate dataset and evaluation protocol) demonstrating that their implementations reproduce published results?

A1: Thank you for this suggestion. We agree that training-task failures should be distinguished from failures to generalize. We ran small-scale single-layout checks with our repository implementations; settings, results, and training curves are in the orange-marked Appendix G, Table 8 and Figure 6. All eight implementations learn single-layout AntMaze tasks, and CRL and HIQL match published OGBench results.

Offline RL and BC implementations. We trained the BC, TD3+BC, IQL, and ReBRAC implementations from our repository separately on the project's AntMaze U-Maze dataset, using the project's evaluation protocol. All four achieve high success rates (76.1%–92.6%), consistent with prior evidence that offline methods can learn single-layout AntMaze tasks [1].

Goal-conditioned implementations. We tested our repository's OGBench-based GCBC, QRL, CRL, and HIQL implementations on the official antmaze-medium-navigate-v0 dataset, using OGBench's evaluation tasks and success criterion. Each evaluation used 30 episodes for each of the five official tasks, with a 1,000-step horizon, and averaged success equally across tasks.

Figure 6 shows single-layout evaluation-success curves, including CRL and HIQL, with evaluations every 100k gradient updates. The table reports trimmed mean success rates, excluding one highest and one lowest evaluation value per run.

Project U-Maze algorithmMean success (%)OGBench Medium algorithmMean success (%)
BC92.0GCBC23.5
TD3+BC92.6QRL80.2
IQL76.1CRL93.8
ReBRAC85.0HIQL94.2

CRL and HIQL reach 93.8% and 94.2%, respectively, consistent with published OGBench results (approximately 95% and 96%; Table 2 in [2]). QRL's stronger and GCBC's weaker results also follow the published pattern.

Additional evidence from the existing experiments. Existing experiments also show that these implementations can learn effective policies.

  • M16 and SeedMap use the same baseline implementations. Several baselines improve with broader layout coverage on SeedMap. For example, AntMaze CRL's held-out success increases from 3.3% with 100 training layouts to 43.3% with 500 layouts (Section 5.3 and Appendix E, Table 7). Thus, broader structural coverage lets the same implementation learn policies that transfer to unseen AntMaze layouts.
  • PointMaze and AntMaze share the algorithm implementations, with observation and action dimensions adapted to each embodiment. On PointMaze M16, numerical BC achieves 60.7% training success, although held-out success remains at 12.2% (Table 3). This shows training-layout learning alongside a substantial transfer gap, not a general inability to learn.
  • CNN, Transformer, and sensing diagnostics retain the BC training pipeline and downstream policy-backbone implementation, changing only map encoders or explicit spatial features. The structured encoders improve PointMaze training performance, and CNN-BC with sensing reaches 75.4% training and 71.9% held-out success on AntMaze (Section 5.4, Table 3). Thus, the shared BC implementation learns effective AntMaze policies when spatial information is easier to use.

Together, these checks support the shared implementations and locate the difficulty in M16 multi-layout learning and transfer, not general implementation failure. M16 trains a shared layout-conditioned policy jointly across 16 layouts. Zero entries indicate failures on particular training layouts, not an inability to learn AntMaze at all. The orange-marked Section 5.2 and Appendix G clarify that AntMaze M16 baselines fail to learn all training layouts jointly and to generalize effectively to unseen layouts.

[1] Tarasov, D., Kurenkov, V., Nikulin, A., and Kolesnikov, S. Revisiting the Minimalist Approach to Offline Reinforcement Learning. NeurIPS, 2023.

[2] Park, S., Frans, K., Eysenbach, B., and Levine, S. OGBench: Benchmarking Offline Goal-Conditioned RL. ICLR, 2025.

AuthorsOpenReview ↗

Response to Reviewer qtaj Part-2

W2: The authors should clarify PIDA's intended contribution and consider comparing it with a randomly initialized version of the same architecture or a non-language model equipped with a structured spatial map encoder.
Q2: Have the authors evaluated PIDA with a randomly initialized backbone under the same architecture, input representation, and training budget?

A2: Thank you for these suggestions.

PIDA’s intended contribution. PIDA is intended as a reference policy for CrossMaze. It shows that strong zero-shot structural generalization is achievable from the same offline data: it reaches 69.8% and 72.7% held-out success, against 17.6% and 4.4% for the best numerical baseline in the original submission. It also gives a reference point for studying what the numerical baselines lack. As Section 4 states, its differences from the baselines should be read as differences in policy representation, pretraining, capacity, and objective together.

Which factor explains PIDA’s advantage. The reviewer suggests two comparisons. In this revision, we added non-language baselines with structured spatial map encoders and further diagnostic experiments with explicit spatial features. Specifically, we added CNN-BC and Transformer-BC, which process the maze using spatial convolutions or attention over grid cells. We also tested MLP-BC and CNN-BC with sensing, which externally supplies the agent's two-dimensional grid-cell index and four adjacent-wall indicators while retaining the original map and continuous state. Part 3 provides the detailed experimental settings and full results.

The results address the three factors the reviewer lists.

  • Input representation. Replacing the flattened map with a CNN or Transformer encoder over grid cells helps on PointMaze (12.2% to 28.7% held-out), but AntMaze stays below 4%. The representation alone does not explain the gap.
  • Architecture. The Transformer encoder uses attention over the map without pretraining and reaches only 19.8% and 2.2% held-out success. The architecture alone does not produce the advantage.
  • Pretraining. Numerical BC approaches PIDA only when explicit spatial features are supplied from outside (71.9% vs. 72.7% on AntMaze), and a gap still remains on PointMaze (50.0% vs. 69.8%). PIDA receives none of these features and reaches 69.8% and 72.7% from the text map alone. It also learns from very few layouts: trained on a single PointMaze layout, it reaches 34.2% held-out success, higher than CNN-BC trained on all 16 layouts (28.7%).

Taken together, these results suggest that its pretrained backbone provides the kind of spatial knowledge that numerical BC has to be given from outside, and they point to pretraining as a key factor in its advantage.

Randomly initialized backbone. We agree that a matched comparison would help assess the contribution of pretraining. PIDA freezes its 0.6B-parameter backbone and trains LoRA adapters and action-decoding components. Keeping a randomly initialized backbone frozen would test low-rank adaptation of random representations; training the entire backbone from scratch would change the trainable-parameter budget and could require different optimization settings and training budgets. In the latter setting, it is difficult to vary only initialization without introducing other training-protocol changes. For this reason, the revision examines the role of pretraining through the controlled comparisons above.

The current revision instead adds structured map encoders and controlled spatial-feature interventions, reported in response 3 below. These experiments show that making spatial relations explicit substantially improves numerical BC and support the importance of accessible spatial information. They are consistent with the interpretation that PIDA benefits from pretrained spatial-semantic priors.

AuthorsOpenReview ↗

Response to Reviewer qtaj Part-3

Q3: Have the authors evaluated non-language baselines equipped with structured spatial map encoders, such as CNNs or Transformers operating over grid cells?

A3: Yes. We added CNN-BC and Transformer-BC, along with separate feature-assisted diagnostics, in the blue-marked Section 5.4, newly added Table 3, and Appendix D. These diagnostics supply sensing features (the agent's two-dimensional grid-cell index and four adjacent-wall indicators) to MLP-BC and CNN-BC. The CNN processes the two-dimensional map with convolutions, while the Transformer uses grid-cell tokens, two-dimensional positional encodings, and padding masks. Both produce a 256-dimensional map feature and use the same downstream feature-fusion and policy-backbone architecture. The CNN and Transformer variants without sensing change only the map encoder, without adding explicit spatial features.

Table 3 is reproduced below, including the MLP-BC and PIDA references. Success rates are percentages, reported as mean ± standard deviation over five training seeds, with layouts weighted equally.

Policy / map encoderSensingPointMaze trainPointMaze held-outAntMaze trainAntMaze held-out
BC / MLPNo60.7 ± 0.812.2 ± 1.39.5 ± 1.81.8 ± 1.4
BC / TransformerNo68.8 ± 1.219.8 ± 1.25.8 ± 2.52.2 ± 2.4
BC / CNNNo73.8 ± 1.028.7 ± 1.76.7 ± 2.03.9 ± 1.3
BC / MLPYes79.6 ± 2.041.7 ± 2.067.1 ± 11.449.4 ± 11.8
BC / CNNYes82.2 ± 1.650.0 ± 1.575.4 ± 3.171.9 ± 2.3
PIDANo78.5 ± 5.869.8 ± 1.875.6 ± 2.472.7 ± 2.4

Structured encoding helps, but does not close the gap alone. CNN-BC and Transformer-BC without sensing improve PointMaze held-out success from 12.2% to 28.7% and 19.8%, respectively. On AntMaze, however, held-out success remains at 3.9% and 2.2%, with training success below 7%.

Explicit spatial information yields substantially larger improvements. The sensing features supplement, rather than replace, the original map and continuous state. These features are externally provided, not learned by BC, and contain no shortest path, subgoal, recommended direction, or expert action. They are reported separately as feature-assisted diagnostics, not identical-input encoder comparisons. Adding them to MLP-BC raises held-out success to 41.7% on PointMaze and 49.4% on AntMaze. Combining them with the CNN further raises success to 50.0% and 71.9%, respectively, bringing AntMaze close to PIDA's 72.7%. Thus, among the tested BC modifications, access to useful spatial information provides larger gains than changing the encoder alone, while a structured encoder adds complementary benefits. These results are also consistent with the interpretation that PIDA benefits from pretrained spatial-semantic priors.

Reviewer_WbXnOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces CrossMaze, a benchmark for evaluating zero-shot structural generalization in offline continuous control. The paper evaluates eight existing imitation learning and offline RL methods, including goal-conditioned approaches, and introduces PIDA as an additional reference baseline. PIDA uses a pretrained Qwen3 language model with LoRA adaptation and a continuous action prediction head, taking serialized maze layouts, robot states, and goals as inputs.

The main experimental finding is that conventional offline learning methods exhibit substantial performance degradation on unseen layouts, whereas PIDA achieves considerably higher held-out success rates. Overall, the paper contributes a structured evaluation protocol for an important generalization problem and provides empirical evidence that policy representation may play a significant role in transferring to unseen environment structures.

Critical strengths
  1. I appreciate the paper's focus on structural generalization rather than generalization to new goals or initial states within a familiar environment. These are meaningfully different problems, and success on standard offline RL benchmarks does not necessarily imply that a policy can navigate an environment with previously unseen connectivity. The distinction is relevant for robotics, where changes in obstacle configurations can invalidate previously successful behaviors. I think the paper does a good job motivating why this deserves more systematic evaluation.

  2. The combination of a fixed diagnostic benchmark and a scalable procedural benchmark is a useful design choice. The evaluation covers several representative algorithm families, two control embodiments, multiple training-layout scales, and several ablations. I particularly appreciate that the authors report results across multiple seeds and explicitly separate training and held-out layouts. The inclusion of reproducibility details, such as layout manifests, dataset generation procedures, and evaluation protocols, could make CrossMaze a useful resource for future research.

Critical weaknesses
  1. My primary concern is the interpretation of the substantial performance gap between PIDA and the other methods. Although the authors acknowledge that these differences prevent the attribution to language pretraining alone, the current experiments still leave the most interesting question unanswered: what actually enables PIDA to generalize so much better?

  2. The MLP capacity in section 5.4 is useful, but I do not think it adequately addresses this concern. Increasing the width or depth of a MLP does not control for architectural inductive bias, pretraining, or the way spatial structure is represented. In particular, a flattened maze representation may be a poor choice for learning spatial relationships across layouts of different sizes and configurations.

  3. I would find the results more convincing with at least one stronger structured baseline, such as a CNN-based or transformer-based policy that explicitly processes the maze layout. An architecture-matched comparison between pretrained and randomly initialized PIDA backbones would also help determine whether the improvement comes from pretraining or simply from a more suitable architecture. This matters because the paper's most interesting result is the large advantage of PIDA, but the current evidence does not establish which of its design choices is responsible.

  4. The reported held-out success rates combine several potentially different failure modes. A policy may fail because it cannot identify a valid route through an unseen maze, because it cannot execute the necessary low-level control, or because it has not learned a sufficiently effective policy even on the training layouts. I think the paper would benefit substantially from separating these factors. For example, the authors could report generalization gaps conditional on successful training-layout performance, or evaluate a hierarchical baseline that combines an explicit planner operating on the provided map with a learned low-level controller. Since the benchmark already provides complete maze layouts, such a comparison seems particularly relevant. It would help establish whether the challenge is primarily learning to interpret unseen structures, executing continuous control, or integrating the two.

Questions for authors

See the weaknesses above

3 comments
AuthorsOpenReview ↗

Response to Reviewer WbXn Part-1

We thank the reviewer for recognizing the importance of structural generalization and the value of CrossMaze's fixed and procedural tracks. To address your concerns, we have thoroughly reviewed each comment and provided comprehensive responses, supported by further clarifications and additional experimental results. This part addresses concerns 1 and 2; Part 2 details the structured baselines and discusses the pretraining comparison requested in concern 3. The additional experiments and their interpretation appear in the blue-marked Section 5.4, newly added Table 3, and Appendix D of the revised manuscript. If any issues remain, we would be more than glad to clarify them further.

W1: Although the authors acknowledge that these differences prevent the attribution to language pretraining alone, the current experiments still leave the most interesting question unanswered: what actually enables PIDA to generalize so much better?
W2: The MLP capacity in section 5.4 is useful, but I do not think it adequately addresses this concern. Increasing the width or depth of a MLP does not control for architectural inductive bias, pretraining, or the way spatial structure is represented. In particular, a flattened maze representation may be a poor choice for learning spatial relationships across layouts of different sizes and configurations.

A1–2: Thank you for raising these concerns. We agree that MLP capacity scaling alone does not address architectural inductive bias, pretraining, or spatial representation. We have therefore extended the analysis to structured map encoders and explicit spatial features, using BC to keep the actor-learning objective aligned with PIDA's behavior-cloning objective. The new diagnostics point to the effective use of spatial information as a major factor in the performance gap, rather than model capacity alone. The evidence is as follows.

  • Increasing MLP capacity alone does not close the gap. Wider or deeper MLPs remain far below PIDA in the tested capacity study (Section 5.4). This limits a capacity-only explanation but does not rule out architectural or pretraining effects.

  • Structured map encoding helps, but is not sufficient on its own. CNN-BC and Transformer-BC improve PointMaze held-out success from 12.2% to 28.7% and 19.8%, respectively, while AntMaze remains below 4%. Part 2 provides the encoder details and full results.

  • Making spatial relations explicit yields larger gains than changing the encoder alone. Supplying the agent's grid-cell index and adjacent-wall indicators, termed sensing, raises MLP-BC held-out success to 41.7% on PointMaze and 49.4% on AntMaze. Combining sensing with a CNN further raises success to 50.0% and 71.9%, respectively, approaching PIDA's 72.7% on AntMaze while leaving a gap to its 69.8% on PointMaze. Thus, explicit spatial features and structured map encoding provide complementary benefits (Table 3).

  • PIDA generalizes without these externally supplied spatial features. PIDA receives the serialized map, state, and goal, but no grid-cell index or adjacent-wall indicators. It achieves 69.8% and 72.7% held-out success on PointMaze and AntMaze. CNN-BC with sensing fits the training layouts slightly better (82.2% vs. 78.5%), yet loses 32.2 percentage points on held-out layouts, whereas PIDA loses 8.7.

PIDA also generalizes from few training layouts. Trained on a single PointMaze layout, it reaches 34.2% held-out success, exceeding CNN-BC trained on all 16 layouts (28.7%) and all eight original baselines (at most 17.6%). With 32 SeedMap layouts, it exceeds every baseline trained on 100 to 1,000 layouts (Section 5.3).

Together, these findings are consistent with the hypothesis that PIDA benefits from pretrained spatial-semantic priors, making useful spatial relations more accessible for policy learning. Part 2 discusses the architecture-matched pretrained-versus-randomly-initialized comparison, which is not included in the current revision. The blue-marked Section 5.4 and revised conclusion discuss this interpretation.

AuthorsOpenReview ↗

Response to Reviewer WbXn Part-2

W3: I would find the results more convincing with at least one stronger structured baseline, such as a CNN-based or transformer-based policy that explicitly processes the maze layout. An architecture-matched comparison between pretrained and randomly initialized PIDA backbones would also help determine whether the improvement comes from pretraining or simply from a more suitable architecture. This matters because the paper's most interesting result is the large advantage of PIDA, but the current evidence does not establish which of its design choices is responsible.

A3: Thank you for suggesting these comparisons. We have added CNN-BC and Transformer-BC, along with sensing diagnostics for MLP-BC and CNN-BC. These comparisons distinguish the benefit of changing the map encoder from that of explicitly supplying spatial relations. All variants use BC, aligning the actor-learning objective with PIDA's.

Structured map encoders. The CNN processes the two-dimensional maze through spatial convolutions. The Transformer uses grid-cell tokens with two-dimensional positional encodings and padding masks. Both produce a 256-dimensional map feature and share the same downstream feature-fusion and policy-backbone architecture. Implementation details are in the blue-marked Appendix D.

Explicit spatial features. Sensing supplies the agent's two-dimensional grid-cell index and four adjacent-wall indicators, while retaining the original map and continuous state. These features are externally computed, not learned by BC, and provide no shortest path, subgoal, recommended direction, or expert action. They can be computed from the state and map already available to every method. We therefore report these as feature-assisted diagnostics, not identical-input encoder comparisons.

Table 3 reports the following results alongside MLP-BC and PIDA. Success rates (%) are mean ± standard deviation over five training seeds, with layouts weighted equally.

Policy / map encoderSensingPointMaze trainPointMaze held-outAntMaze trainAntMaze held-out
BC / MLPNo60.7 ± 0.812.2 ± 1.39.5 ± 1.81.8 ± 1.4
BC / TransformerNo68.8 ± 1.219.8 ± 1.25.8 ± 2.52.2 ± 2.4
BC / CNNNo73.8 ± 1.028.7 ± 1.76.7 ± 2.03.9 ± 1.3
BC / MLPYes79.6 ± 2.041.7 ± 2.067.1 ± 11.449.4 ± 11.8
BC / CNNYes82.2 ± 1.650.0 ± 1.575.4 ± 3.171.9 ± 2.3
PIDANo78.5 ± 5.869.8 ± 1.875.6 ± 2.472.7 ± 2.4

Structured encoding helps, but does not close the gap alone. CNN-BC and Transformer-BC improve PointMaze held-out success from 12.2% to 28.7% and 19.8%, respectively, while AntMaze remains at 3.9% and 2.2%.

Sensing yields larger gains, and complements the CNN. Adding sensing to MLP-BC raises held-out success to 41.7% on PointMaze and 49.4% on AntMaze, gains of 29.5 and 47.6 percentage points. Combining sensing with the CNN further raises success to 50.0% and 71.9%. The latter approaches PIDA's 72.7% on AntMaze, while a gap remains on PointMaze. These findings support the interpretation in Part 1 that making spatial relations accessible matters beyond changing the encoder alone. The blue-marked Section 5.4 and revised conclusion discuss these findings.

Architecture-matched pretraining comparison. We agree that comparing pretrained and randomly initialized PIDA backbones would clarify the contribution of pretraining. For a from-scratch version of the full PIDA backbone, the training protocol also matters. PIDA freezes its 0.6B-parameter pretrained backbone and trains LoRA adapters and the action-decoding components. Training the full random backbone would change the trainable-parameter budget and may require different optimization and training budgets. Thus, comparing PIDA with a fully trained, randomly initialized backbone makes it difficult to vary initialization alone while keeping the training protocol fixed.

The new diagnostics nevertheless provide evidence relevant to interpreting PIDA's advantage. Even with a CNN or Transformer map encoder, BC stays below 29% held-out success on PointMaze and below 4% on AntMaze. Among the tested BC variants, it approaches PIDA only when explicit spatial features are supplied externally (71.9% vs. 72.7% on AntMaze), while a gap remains on PointMaze (50.0% vs. 69.8%). PIDA receives none of these features and reaches 69.8% and 72.7% using the serialized map, state, and goal. These results suggest that its pretrained backbone may make useful spatial knowledge accessible without the external features supplied to BC, pointing to pretrained spatial-semantic priors as a plausible source of its advantage.

AuthorsOpenReview ↗

Response to Reviewer WbXn Part-3

W4: A policy may fail because it cannot identify a valid route through an unseen maze, because it cannot execute the necessary low-level control, or because it has not learned a sufficiently effective policy even on the training layouts. The paper would benefit substantially from separating these factors.

A4: Thank you for this suggestion. We agree that low held-out success alone cannot distinguish these possibilities. Following the reviewer’s first suggestion, we report training and held-out results together in Table 3 (shown in response 2), which provides a more informative diagnosis than held-out results alone. In short, for the tested policies, low-level control is not the main obstacle. The difficulty lies in getting spatial information from the map and using it for control.

  • Some AntMaze baselines have not learned effective training-layout behavior. Without sensing, MLP-BC, Transformer-BC, and CNN-BC achieve only 9.5%, 5.8%, and 6.7% training success, respectively. Their low held-out success should therefore not be interpreted solely as a failure to transfer an otherwise successful training policy.
  • Explicit spatial features improve both training-task learning and held-out performance. On AntMaze, MLP-BC with sensing reaches 67.1% training and 49.4% held-out success. CNN-BC with sensing reaches 75.4% and 71.9%, respectively, leaving a 3.5-percentage-point difference between the reported means. Thus, the benefit is not confined to unseen layouts; the features also help the policy learn effective behavior on the training layouts.
  • Higher training performance does not automatically eliminate the transfer gap. On PointMaze, CNN-BC with sensing reaches 82.2% training success but 50.0% held-out success, a difference of 32.2 percentage points. A likely main reason for the smaller gap on AntMaze is its lower and more concentrated layout topology difficulty (Appendix A.1).

We tested the idea behind the suggested planner–controller baseline with our sensing intervention. Both give the policy part of the spatial information directly, while the continuous actions are still learned. The difference is how much is given: a planner supplies a full route or subgoals, whereas sensing supplies only the agent’s current grid cell and its adjacent walls. The policy still has to choose the route and execute the actions by itself. Even with this small amount of help, CNN-BC reaches 71.9% on AntMaze held-out layouts, up from 3.9% without it. This shows that continuous control is not the main obstacle: once the policy is told where it is in the map, it can navigate unseen layouts. The main challenge is getting spatial information from the map and using it for control, which is the integration question the reviewer raises.

Reviewer_zRfnOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces CrossMaze, a benchmark to evaluate zero-shot structural generalization in offline continuous control. The benchmark consists of two complementary tracks: M16, a fixed suite containing 16 training layouts and 10 held-out evaluation layouts, and SeedMap, a procedural maze generation framework that supports scaling the diversity of training environments. Both tracks are instantiated in PointMaze and AntMaze, where policies must navigate unseen maze structures using offline training data without additional environment interaction or test-time adaptation. The authors evaluate eight representative imitation learning, offline RL, and offline goal-conditioned RL methods, including BC, IQL, TD3+BC, ReBRAC, GCBC, QRL, CRL, and HIQL. They additionally introduce PIDA, a pretrained language-model-based behavior cloning policy that serializes maze layouts, observations, and goals into textual inputs and predicts continuous actions through a learned action decoder. The main finding is that PIDA substantially outperforms the evaluated conventional baselines on unseen maze layouts, achieving 69.8% and 72.7% held-out success rates on PointMaze and AntMaze, respectively, compared with 17.6% and 4.4% for the strongest non-PIDA baselines.

Critical strengths
  1. A well-defined benchmark for evaluating structural generalization. By explicitly separating training and evaluation maze layouts, CrossMaze provides a controlled setting for investigating whether policies can transfer across changes in environment connectivity and feasible trajectories.

  2. Reproducible benchmark design and relatively comprehensive empirical evaluation. The combination of a fixed diagnostic benchmark (M16) and a procedurally generated benchmark (SeedMap) is a useful design choice. M16 facilitates controlled comparisons across algorithms, while the SeedMap enables experiments on scaling training-layout diversity.

Critical weaknesses

A major motivation of this paper is that strong offline RL performance on previously observed environment configurations does not necessarily imply successful generalization to unseen environment structures. However, this observation is largely expected and does not constitute a particularly surprising insight. In particular, it is intuitive that a policy trained on a limited set of maze layouts may fail when the obstacle configuration, connectivity, and feasible navigation routes change. The broader distinction between in-distribution performance and out-of-distribution generalization has also been extensively investigated in prior RL generalization literature, including work on procedurally generated environments and offline RL generalization, some of which is already discussed in the paper.

Besides, it lacks strong structure-aware baselines. As described in Appendix C, the numerical baselines receive maze layouts as row-major flattened matrices, padded to a fixed size, and are implemented using MLP-based architectures. Such representations do not explicitly preserve spatial locality, neighborhood relationships, or graph connectivity. Thus, they may be poorly suited to learning transferable navigation strategies across unseen maze structures. In contrast, PIDA receives a row-wise textual representation of the maze and processes it using a large pretrained language model. Although the raw environment information is shared across methods, the architectural capabilities and inductive biases are different.

Questions for authors

Please see the weakness part above.

2 comments
AuthorsOpenReview ↗

Response to Reviewer zRfn Part-1

We thank the reviewer for recognizing CrossMaze's controlled evaluation of structural generalization and its reproducible combination of fixed and procedural tracks. To address your concerns, we have thoroughly reviewed each comment and provided comprehensive responses, supported by further clarifications and additional experimental results. The additional map-encoder and spatial-feature experiments are reported in the blue-marked Section 5.4, Table 3, and Appendix D of the revised manuscript. If any issues remain, we would be more than glad to clarify them further.

W1: However, this observation is largely expected and does not constitute a particularly surprising insight. It is intuitive that a policy trained on a limited set of maze layouts may fail when the obstacle configuration, connectivity, and feasible navigation routes change. The broader distinction between in-distribution performance and out-of-distribution generalization has also been extensively investigated in prior RL generalization literature

A1: Thank you for raising this point. We agree that a gap between training and held-out layouts is expected, and that prior work on procedurally generated environments has established this distinction. What was not known is how large the drop is under continuous control, whether it can be closed, and what causes it. CrossMaze’s contribution is to answer these three questionsin a setting that existing benchmarks do not cover, and it yields results that were not predictable in advance.

  • Structural changes coupled with continuous control. Within each embodiment, CrossMaze varies obstacle layouts and route connectivity while keeping the control embodiment fixed. Such changes alter collision outcomes and feasible continuous trajectories, rather than only changing goals within a fixed environment or physical parameters such as mass and friction. Prior work on offline Procgen studies transfer to unseen levels with discrete actions [1], whereas continuous-control maze tasks in OGBench evaluate goal-conditioned policies within fixed layouts [2]. CrossMaze brings these two settings together by extending PointMaze and AntMaze with diverse layouts and explicit train/held-out layout splits for zero-shot evaluation, while retaining their native continuous-control dynamics. This difference matters in practice. With the same layout generator and the same method, transfer changes sharply once control becomes harder.

  • Offline, zero-shot transfer with an explicit layout interface. Policies learn from fixed training-layout trajectories and are evaluated on disjoint layouts without parameter updates or test-time adaptation. The current layout is available to all methods, without shortest paths or oracle actions. This tests whether a policy can use a description of an unfamiliar structure to produce continuous actions, rather than requiring it to discover the map through visual exploration. Sections 2.2 and 3 distinguish this protocol from unseen-goal generalization and offline data-coverage generalization within a fixed MDP. Under this protocol, all eight standard methods stay below 18% held-out success on PointMaze and below 5% on AntMaze, while PIDA reaches 69.8% and 72.7% from the same offline data (Tables 1 and 2). The low baseline scores are therefore not an unavoidable cost of distribution shift: the task can be solved, and the benchmark separates methods that solve it from methods that do not.

  • A diagnostic and scalable benchmark, rather than a single negative result. M16 provides a fixed suite of 16 training and 10 held-out layouts per embodiment for algorithm comparison and per-layout diagnosis, while SeedMap supports controlled scaling of training-layout diversity through deterministic generation. Together with the shared layout interface and equal-layout evaluation metric, this enables reproducible comparisons of how training coverage, representations, and policy classes affect structural transfer. Two results show what this reveals. More training layouts do not steadily help: AntMaze CRL rises from 3.3% to 43.3% between 100 and 500 layouts, then falls to 36.0% at 1,000 (Section 5.3). A stronger map encoder is not enough on its own: with a CNN or Transformer encoder, BC stays below 4% on AntMaze held-out layouts, whereas adding six simple spatial features raises it to 49.4%, and to 71.9% with a CNN [point to the new added content].

Together, these results show that CrossMaze measures a gap that existing benchmarks do not expose and helps identify their reasons.

[1] Mazoure, B., Kostrikov, I., Nachum, O., and Tompson, J. Improving Zero-Shot Generalization in Offline Reinforcement Learning Using Generalized Similarity Functions. NeurIPS, 2022.

[2] Park, S., Frans, K., Eysenbach, B., and Levine, S. OGBench: Benchmarking Offline Goal-Conditioned RL. ICLR, 2025.

AuthorsOpenReview ↗

Response to Reviewer zRfn Part-2

W2: Besides, it lacks strong structure-aware baselines. Sucn representation..... Thus, they may be poorly suited to learning transferable navigation strategies across unseen maze structures. Although the raw environment information is shared across methods, the architectural capabilities and inductive biases are different.

A2: Thank you for this suggestion. We agree that sharing raw information does not make the representations or inductive biases equivalent. We have therefore added BC baselines with CNN and Transformer map encoders. In separate diagnostics, we also supply sensing features (the agent's two-dimensional grid-cell index and four adjacent-wall indicators) to both MLP-BC and CNN-BC. Using BC also keeps the actor-learning objective aligned with PIDA's behavior-cloning objective.

The CNN applies convolutions to the two-dimensional map, which preserves spatial locality. The Transformer treats grid cells as tokens with two-dimensional positional encodings, so it can relate any two cells, and it masks padded cells. Both encoders produce a 256-dimensional map feature and use the same downstream feature-fusion and policy-backbone architecture, described in the blue-marked Appendix D. We refer to CNN-BC and Transformer-BC without sensing as encoder-only variants.

Table 3 of the revised manuscript reports these results alongside MLP-BC and PIDA. Values are success rates in percent, reported as mean ± standard deviation over five training seeds, with layouts weighted equally.

Policy / map encoderSensingPointMaze trainPointMaze held-outAntMaze trainAntMaze held-out
BC / MLPNo60.7 ± 0.812.2 ± 1.39.5 ± 1.81.8 ± 1.4
BC / TransformerNo68.8 ± 1.219.8 ± 1.25.8 ± 2.52.2 ± 2.4
BC / CNNNo73.8 ± 1.028.7 ± 1.76.7 ± 2.03.9 ± 1.3
BC / MLPYes79.6 ± 2.041.7 ± 2.067.1 ± 11.449.4 ± 11.8
BC / CNNYes82.2 ± 1.650.0 ± 1.575.4 ± 3.171.9 ± 2.3
PIDANo78.5 ± 5.869.8 ± 1.875.6 ± 2.472.7 ± 2.4

Structured map encoding helps, but does not close the gap on its own. On PointMaze, CNN-BC and Transformer-BC without sensing improve held-out success from 12.2% to 28.7% and 19.8%, respectively, supporting the value of spatial inductive biases. These results nevertheless remain below PIDA's 69.8%. On AntMaze, the same encoder-only variants achieve only 3.9% and 2.2% held-out success, respectively, and training success remains below 7%. Thus, in AntMaze, the difficulty is not solely transfer from successfully learned training tasks; the encoder-only variants also struggle on the training layouts.

Making spatial relations explicit yields much larger improvements. The sensing diagnostics test whether explicitly supplying spatial relations helps beyond changing the map encoder alone. These features supplement, rather than replace, the original map and continuous state. They are not learned by BC and provide no shortest path, subgoal, recommended direction, or expert action. The wall indicators give the local connectivity around the agent directly. All six features can be computed from the continuous state and the map that every method already receives, so they add no new information. They are therefore identified separately as feature-assisted diagnostics, not presented as identical-input encoder comparisons.

Adding these features to MLP-BC raises held-out success to 41.7% on PointMaze and 49.4% on AntMaze, gains of 29.5 and 47.6 percentage points over MLP-BC without sensing. Combining them with the CNN further increases success to 50.0% and 71.9%, respectively. The latter nearly matches PIDA's 72.7% on AntMaze, while a gap remains on PointMaze.

Among the tested BC variants, adding external sensing features to MLP-BC yields substantially larger gains than using a CNN or Transformer map encoder alone. Combining sensing with a CNN further improves performance, indicating that explicit spatial features and structured map encoding provide complementary benefits. On PointMaze, PIDA still generalizes better than the strongest structured baseline: CNN-BC with sensing fits the training layouts slightly better (82.2% vs. 78.5%), but PIDA keeps far more on held-out layouts (69.8% vs. 50.0%). They are consistent with the hypothesis that PIDA benefits from pretrained spatial-semantic priors. The blue-marked Section 5.4 and revised conclusion discuss these findings, while Table 3 and Appendix D provide the quantitative results and encoder details.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Follow-up on our responses

Dear Reviewers and Area Chair,

As the discussion period is nearing its end, we would like to briefly follow up on our responses and additional experiments. We would appreciate your feedback on whether these additions address your concerns. If any points remain unclear or unresolved, please let us know, and we would be happy to provide further clarification.

Thank you for your time and effort in reviewing our paper.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.