acceptodds
Under review as a conference paper at ICLR 2027

When Does Navigation Pretraining Help? The Role of Action Selection

Abstract

Does the way a reinforcement-learning agent chooses actions change the measured benefit of pretraining? We study recurrent PPO agents adapting from local maze grids to directional wall distances and goal-visibility signals. Transfer retains the trained controller behind a fresh input encoder. Two settings compare maze pretraining, starting from scratch and equally long obstacle-field pretraining, with eight training seeds per condition. We evaluate the same saved networks by always taking the most probable action (greedy selection) or drawing from the learned probabilities (sampling). In the original setting, sampling improves scratch more than maze pretraining. The pretrained-minus-scratch difference in average success during adaptation on a zero-to-one scale changes from +0.053 to −0.054: a shift of −0.107 (descriptive 95% bootstrap interval [−0.182, −0.044]). However, neither ordering establishes superiority, and the task inventories overlap. An exploratory follow-up on larger mazes with structurally separate training and test sets does not reproduce this reversal: pretraining stays ahead on average, and the change in its advantage is uncertain. A further control preserves the probability of leaving the greedy action but chooses the alternatives equally. Sampling improves performance beyond this control, without clearly changing the pretraining advantage. Action selection can change a transfer comparison, but that sensitivity does not necessarily carry over to another task setting.

Primary area:
reinforcement learning
Keywords:
reinforcement learning, transfer learning, navigation, recurrent policies, policy evaluation, action selection

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_xnXkOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper tests whether deployment-time action selection changes the apparent benefit of navigation pretraining. Re-evaluating recurrent PPO checkpoints under greedy and sampled actions reverses the matched-minus-scratch mean AUC contrast in the original maze setting. A structurally disjoint follow-up does not reproduce that reversal. Matched-departure controls further distinguish the probability of leaving the greedy action from the weighting of alternative actions.

Critical strengths

S1. The estimand is well chosen. Re-evaluating identical checkpoints and measuring the difference in transfer contrasts isolates sensitivity to the deployed action rule (Equation 2). Full-curve AUC and all eight seed blocks are reported, without equating a reversal of means with established superiority.

S2. The follow-up tests the initial result's limits. The authors audit task overlap, construct a split by unlabelled tree structure and report that the reversal does not recur. The matched-departure control usefully separates exploration frequency from the choice of non-greedy alternatives.

Critical weaknesses

W1. The result remains a setting-specific interaction. The original interaction is -0.1074, whereas the structural holdout gives +0.0388 with an interval spanning zero. Size, task weighting, inventory and seeds change together. Beyond showing that evaluation policy can matter, the paper identifies no property that explains or predicts when the pretraining comparison changes (Sections 4.2-4.4).

W2. The strongest diagnostic does not address the main interaction. The fixed-epsilon and matched-departure controls apply to the structural holdout, where the transfer interaction is already uncertain (Appendix C). They show that weighting alternatives improves absolute performance there, but do not explain the original reversal.

Assessment. I recommend weak rejection (2/4). The reporting is sound, but the connection between the main interaction and its proposed diagnostic remains too limited to provide a clear new explanation of transfer sensitivity.

Questions for authors

Q1 (W2). Are fixed-epsilon or matched-departure results available for the original 9-by-9 checkpoints? If so, please report the matched-minus-scratch interaction. If not, which existing result supports an explanation of the original reversal beyond generic benefits of stochastic evaluation?

Q2 (W1). Do the existing action-probability or trajectory records identify a difference between scratch and pretrained policies that tracks the interaction across seeds? Please give the specific statistic and its relationship to transfer, rather than another aggregate success comparison.

1 comment
AuthorsOpenReview ↗

Response: original-setting controls and behavioral comparison

New and revised text appears in blue in the revised manuscript.

Q1. Do the action controls explain the original result? We repeated the controls on all 32 original agents at all 21 checkpoints. All greedy and sampled baseline success scores reproduced. With fixed ε=0.1, we draw uniformly among all three actions with probability 0.1 and otherwise take the greedy action. With matched departure, we keep the greedy action's learned probability and divide the remainder equally between the other two actions.

Maze-pretrained minus scratch AUC is +0.0533 under greedy selection, −0.0541 under sampling, −0.0174 under fixed ε and −0.0428 under matched departure. AUC measures average success across adaptation. Both controls reproduce the reversal of the mean ordering. All four individual contrast intervals include zero.

Relative to matched departure, learned sampling adds 0.0514 AUC for scratch and 0.0402 for maze pretraining. The difference between those gains is −0.0113 (descriptive 95% interval [−0.0379, 0.0136]). The original mean reversal therefore occurs even with equal weights on the alternatives. Whether the learned weights provide a different extra gain for the two training histories remains uncertain. The corresponding sampling-minus-fixed-ε interaction is −0.0366 [−0.0571, −0.0164]. Fixed ε also changes how often the policy departs from greedy, so that comparison cannot isolate the alternatives' weights. See Section 4.4 (pp. 7–8), Table 4 (p. 8) and Appendix Q (p. 33).

Q2. Which behavior tracks the sampling effect? For each of eight seed pairs, we measured the fraction of greedy steps spent revisiting a position and heading, averaged across adaptation. We compared the maze-pretrained minus scratch difference in this measure with the change in their success gap under sampling. Pearson r was −0.595. Where maze-pretrained agents repeated poses more relative to scratch, sampling tended to help scratch more relative to maze-pretrained agents. The corresponding correlations for collisions and coverage were −0.069 and +0.287. Appendix Q (p. 33) gives every pair.

Longer unsuccessful episodes allow more opportunities to revisit a pose. The same position and heading can also occur with different recurrent states. These eight observations do not establish that sampling breaks a cycle or predict where the interaction will occur.

Why do the settings differ? In the larger study, maze-pretrained agents had higher mean AUC under both rules. Size, action cap, task inventory, task weighting and training seeds all changed. The comparison tests whether the observed reversal recurs under those changes. It cannot identify which change accounts for the difference (Section 4.3, pp. 5–7, and Section 5, p. 9).

Reviewer_cmueOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
3
Read review
Paper summary

Authors tested whether encoder-controller architectures trained with RL for navigation 1) are aided by pretraining on the environment with a different encoder, and 2) how this interacts with greedy vs. faithful sampling of actions from the learned PPO policy. In particular, they train an agent to navigate a maze with one local representation, passed to a controller via a bespoke encoder network, and then see what happens when they change the representation modality (and the corresponding encoder).

Critical strengths

None

Critical weaknesses
  1. The paper is nearly unreadable. The text is simultaneously extremely terse while also following every meaningful statement with an LLM-style hedge or equivocation. Nearly every claim is followed by at least one sentence that limits its own scope, often against objections that no one raised except the authors themselves. The setup is not clearly explained at any point, nor the motivations of the study, nor the outcomes. I can only conjecture that no human eyes looked at this text.
  2. The stated main results don’t generalize. Sections 4.1 and 4.2 establish on a given dataset that pretraining seems to maybe, but not certainly, be better than starting from scratch, but that switching from greedy action selection to following the PPO policy aids the scratch agent more than one (but not both) of the pretrained agents. Then, Section 4.3 shows that actually 4.1 and 4.2 accidentally trained on test (why are they included at all?) and if one fixes this (and also makes some other unrelated changes like increasing the maze size) then the effect reverses and sampling helps the pretrained agents more than the scratch agents. At best, the authors’ results don’t even generalize within their own study, at worst sections 4.1 and 4.2 are entirely voided.
  3. The remaining results on various sampling regimes are largely trivial and often statistically insignificant. For one, that the greedy policy is worse than the sampled policy in the transfer context is trivial given that the encoder is likely quite bad (and the controller probably gets quite bad, too) so the noise can avoid getting stuck in loops or exploration failures.
Questions for authors

Have authors considered freezing the controller during the transfer test (either totally or in graded fashion)? Right now introducing an essentially random encoder is likely to destroy the controller policy before any meaningful encoder representation is learned.

2 comments
AuthorsOpenReview ↗

Response: readability, pretrained comparisons and controller freezing

New and revised text appears in blue in the revised manuscript.

Motivation and setup. We revised the opening around one question. If sampling helps both policies navigate, does it also increase the measured advantage of pretraining? A local-grid agent first learns the task. We copy its recurrent controller and action/value heads, attach a fresh encoder for ray inputs and train the whole network. Section 3.2 (pp. 3–4) spells out this sequence. Section 4.2 (p. 5) distinguishes a pretraining difference under one rule from the change in that difference across rules.

Why keep the overlapping study? Its prespecified greedy criterion required maze pretraining to outperform both scratch and equal-budget obstacle-field pretraining. It failed. The 9×9 evaluation inventory shares 94/100 grids and 48/100 complete task configurations with training. It still tells us what happened on that distribution, including where the exploratory reversal arose. The 11×11 follow-up uses disjoint tree structures, but also changes maze size, the action cap, task weighting and seeds. It cannot isolate the effect of removing overlap (Sections 3.2 and 4.3, pp. 3–4 and 5–7).

Is the sampling result expected? Which groups does it distinguish? Sampling can help an agent escape repetitive choices in a partially observed maze. An increase in absolute success for both agents is compatible with a larger, smaller or unchanged pretraining advantage. Originally, sampling increased average success across adaptation from 0.1940 to 0.4492 for scratch, and from 0.2474 to 0.3951 for maze pretraining. Across eight paired seeds, sampling helped scratch more by 0.1074 AUC (maze-pretrained minus scratch interaction −0.1074, descriptive 95% interval [−0.1821, −0.0439]). Neither individual ordering established superiority. Obstacle-field agents also gained less than scratch on average, rising from 0.2754 to 0.3936. We report a paired interval for maze versus scratch, but none for obstacle-field versus scratch. Comparing the two pretrained histories directly gives a maze-minus-obstacle-field interaction of +0.0296 [−0.0734, 0.1298] (Section 4.2, p. 5).

Equal weighting of non-greedy alternatives also reversed the original mean ordering. The repeated-pose comparison uses all eight seed pairs, but cannot establish loop breaking (Section 4.4, pp. 7–8, and Appendix Q, p. 33). In the larger study the maze-minus-scratch interaction was +0.0388 [−0.0847, 0.1537]. The negative point estimate and mean reversal did not recur there, and the interval still admits a negative interaction (Section 4.3, pp. 5–7).

Would freezing the controller improve transfer? A fresh encoder initially supplies unfamiliar inputs, and joint training might disrupt a useful controller before the encoder adapts. In the related same-sensor control, each of eight pretrained source agents received a fresh local encoder. Training only that encoder gave mean AUC 0.0689 versus 0.4885 for joint training. A second random start gave 0.1749 versus 0.6858. Averaged within the eight source agents, encoder-only minus joint training was −0.4653 [−0.6127, −0.2991] (Section 4.5, pp. 8–9, and Appendix K, pp. 25–28). The sensor stayed local and the controller and heads stayed frozen throughout repair. We have no completed cross-sensor comparison that establishes what happens when the controller is frozen throughout, for an initial period, or to different degrees. Your proposed adaptation test remains open.

Thank you for the response.

As far as I can tell, none of my previous reservations were addressed.

  1. The paper remains nearly unreadable -- as another reviewer said, it reads like a lab notebook and not like a prepared submission.
  2. There remains no reason to include the section of the paper in which the model was trained on test -- beside the fact that it is the only setting, as far as I can tell, where the claimed effect actually occurs.
  3. The results (to the extent that any are claimed!) remain mostly trivial and the question itself remains uninteresting.

With respect to the authors, a substantial rewrite is needed for me to change my review from a strong reject to even a weak reject.

Reviewer_wvG5OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The authors devise an RL maze navigation transfer setup with a GRU policy learned via PPO, where the sensor is swapped for a different one (source: local 5x5 grid field of view, target: rays in 3 directions that measure distance and whether a goal is present along the ray). Maze size is 9x9. They set out to test whether pretraining the GRU with the source sensor, replacing the sensor, and then finetuning beats training the whole system GRU+encoder from scratch.

First, they don't find any significant benefit to pre-training used in this way. Variations in results are not significant. Then, they notice that 94 out of their 100 test mazes are part of the training set. So they decide to do another experiment with larger 11x11 mazes where training and test sets are disjoint (I assume the change in size is to make it easier to avoid the overlap).

Then, in the 11x11 mazes, they study whether using the agent taking greedy steps vs taking sampling steps changes the conclusion about whether pretraining helps, because in the first study it did (although within the random variation margins). They find that it doesn't, and pretraining helps regardless.

Critical strengths
  1. The paper is honest about the research process, including what the original research question was, how the failure to produce an answer caused a pivot in the paper aim and brought action selection into the picture, and how an audit found an unexpected result that resulted in a second experiment being created (the 11x11 mazes). The original research question and significance criterion is expressed upfront, and reported as failed, which is direct and honest. The 94/100 overlap between train and test is reported and addressed. Reporting on negative results is useful too.

  2. Several interventions are well designed to isolate effects, e.g.:

  • Using identical checkpoints for the greedy and sampling versions of the policy
  • Adding a control based on an obstacle field to isolate maze pretraining specifically
Critical weaknesses
  1. The paper doesn't really answer the question in the title, and the results are small scale and weak, not much can be concluded from them, and what we can conclude to some extent, we already knew: pretraining helps somewhat on hard problems if done right, and sampling from the policy helps memoryless or imperfect policies in partially observable problems (Singh, Jaakkola & Jordan, ICML 1994). In more detail:
  • In the small 9x9 mazes, the pre-training vs scratch results are basically within random variation: The "flip" found by the authors is between two results that are deemed by the paper as not statistically significant.
  • In the bigger 11x11 mazes, in 5 out of 8 runs from scratch the policy doesn't learn anything meaningful, so sampling vs greedy doesn't really tell us much.
  • Pretraining does help in the 11x11 mazes. Wasn't proved to help in the 9x9 mazes. This is clearly specific to the details of each setup. None of this is tested at scale.
  • Any claim is extremely narrow, and only applies to the specific, small-scale setup tested: One maze family at two small sizes, one sensor pair, one PPO configuration, 8 seeds.
  1. The setup is contrived and the goal of the paper is unclear, it reads more like a worklog that reports minor findings along the way of some basic experimentation.
  • The setup considers policy transfer by replacing only the encoder from a source 5x5 local sensor to a target ray sensor that can sense goals at a distance. This is a toy and contrived transfer setup. First, there is no reason why the best policy (which has to rely on local information only) should be similar to the best policy using distant information. Second, there's little motivation given for the choice of this particular sensor pair and transfer method, or for what is truly being pursued by pre-training with a most likely inadequate policy.
  • The task is mostly search, since the maze is unknown and the agent has no information about position or direction. This task is already solved well by wall following in perfect mazes like the ones in this paper.
  • The problem is very easy, both in terms of its scale, and the task itself: even hand-made walkers with no training solve it.
  • As the authors acknowledge, there's no alignment between the features of the old and new encoder.
  • There is no testing of what could transfer, like sensor-independent search behaviors (e.g., keep moving, avoid backtracking, etc).
  • As the authors show, replacing the encoder, even while keeping the same sensor, leads AUC (mean success) to drop from 0.999 to 0.628. So the transfer mechanism is weak, and what is measured conflates how fast the agent recovers from the encoder switch with what actually transfers.
Questions for authors
  1. Is there a motivation for your concrete choice of sensor swap? What should transfer, how do you expect it to? Which behaviors do you think should carry over between such different information being sensed?
  2. In the 11x11 mazes pretraining does help. How? Is it learning to explore in a sensor agnostic way? Is it specific to mazes? Is it covering only, or aiming for the specific goal location? (aside: is the goal at a fixed position in the 11x11 mazes? or how do you determine goal and start position?)
  3. What do you think can be claimed based on the results of the paper that applies more widely and isn't known (known: sampling can help in partially observed RL, among other things to avoid getting stuck in loops, and pre-training properly for a task helps in that task)?
  4. Does anything hold in situations where hand-coded walkers (wall followers) fail? E.g., mazes with loops. What about larger mazes, or goals and starts placed at random?
1 comment
AuthorsOpenReview ↗

Response: sensor reuse, novelty and benchmark limitations

New and revised text appears in blue in the revised manuscript.

Q1. What is reused, and why switch sensors? We copy the trained recurrent controller and action/value heads, attach a fresh encoder and train the whole network. Switching from a local grid to directional wall distances and goal cues holds actions, movement and rewards fixed while changing what the agent can observe. The sensors reveal different information, so we cannot assume they support the same optimal policy. We test one form of controller reuse. Success can reflect recovery from an unfamiliar encoder as well as useful prior learning (Introduction, p. 1, and Section 3.2, pp. 3–4).

Even with the sensor held local, replacing the encoder reduced mean adaptation AUC from 0.9986 to 0.6276. In a separate comparison behind the same fixed new local encoder, inherited minus fresh controller AUC was +0.1918 [−0.0224, 0.4033]. Six of eight fresh controllers eventually reached 95–100% success. Recovery alone leaves the content of any inheritance unidentified (Section 4.5, pp. 8–9, and Appendix O, p. 32).

Q2. Can exploration account for goal reaching? How are the 11×11 endpoints chosen? The generator carves from (1,1), chooses a farthest cell as start, then a farthest cell from the start as goal. The goal is at (1,1) in 499/512 training, 32/32 validation and 96/100 test mazes (Appendix J, pp. 24–25). The task rewards reaching the goal and gives a training-only bonus for first visits to cells. Persistent movement or a reusable search strategy could score well without directed goal seeking. When we relocated the goal on each original maze, frozen maze-pretrained local policies reached 99.25% of targets. Their original trajectories already crossed 93.375% of those new targets, and the mean shortest route fell from 35.04 to 18.03 actions. This original-setting probe cannot establish which behavior produced the 11×11 pretraining advantage, or whether it depends on maze structure rather than sensor-independent exploration (Appendix H, pp. 23–24, and Section 5, p. 9).

Q3. What is new, and what can generalize? Singh, Jaakkola and Jordan (1994) established that stochastic action choice can help memoryless policies under partial observation. Jang and colleagues compare post-training action rules in PPO. We ask how changing that rule alters the estimated benefit of pretraining across the same saved recurrent policies' adaptation curves (Section 2, p. 2). We agree that the individual original contrasts are small and uncertain. On a 0–1 AUC scale, maze-pretrained minus scratch is +0.0533 [−0.0432, 0.1383] under greedy and −0.0541 [−0.2118, 0.0721] under sampling. For the same trained agents, the paired change is better estimated, −0.1074 [−0.1821, −0.0439]. That is evidence that the rule affected this comparison, but it cannot establish which training history is better (Sections 4.1–4.2, p. 5).

Equal weights on the non-greedy actions also reverse the original mean ordering. On 11×11 mazes, the maze-pretraining advantage was +0.1790 [0.0647, 0.2832] under greedy selection and +0.2179 [0.0097, 0.3886] under sampling. The title asks when pretraining helps. Our results suggest reporting the action rule and checking whether the pretraining contrast changes under another rule. We tested one sensor change and one recurrent PPO architecture. We have no validated predictor for another task or agent (Sections 4.3–4.4, pp. 5–8, and Section 5, p. 9).

Q4. What happens with cyclic mazes and random endpoints? The larger 11×11 target mazes are still trees with the endpoint bias described above. The cyclic obstacle fields are a pretraining control. We have no result on a cyclic target with randomized starts and goals where the strong simple policy fails (Section 4.3, pp. 5–7).

How strong are the benchmark and scratch baseline? The goal-cued ray walker succeeds on 99.508% of held-out attempts against 0.061% for uniform random actions. A short-history rule can solve these tasks. Of eight scratch agents, four had exactly zero greedy AUC and a fifth had 0.0030 over 500,000 target interactions. All are retained (Section 4.5, pp. 8–9, and Appendix C, Table 7, p. 17). Checks of PPO's recorded action probabilities, value predictions and episode boundaries do not settle whether training was adequate. We have no audited longer-budget comparison or hyperparameter sweep that establishes a competitive scratch baseline. Whether the finding survives adequate baseline learning and a harder benchmark remains open.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.