acceptodds
Withdrawn from ICLR 2027

Executability, Not Optimality: Offline Planning on Learned Cognitive Maps

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Animals reach places they were never shown how to reach, and hippocampal place cells are thought to support this by encoding a predictive map of where the animal can get to next. One computational form of that map is the multi-scale random-walk transition kernel, whose non-negative spectral embeddings reproduce localized place fields, compose across scales by matrix squaring, and yield a route by climbing a similarity field rather than searching a graph. Whether it survives being fitted to offline trajectories, which supply neither the known lattice nor the exact kernel its demonstrations assume, and what it is then worth as a planner, has been open. Here we show that it survives the fit and that what its ascent buys over shortest-path search on the same support is executability, not optimality: it leads A on 13 of 16 embodied OGBench datasets, and support geometry alone recovers three quarters of what that preference is worth to search. Two received rules fail there: greedy ascent, sufficient on the exact kernel, falls on the fitted map into limit cycles that one sampled alternative escapes; and distance-optimal search loses to the map, while fast marching told what the ascent prefers, as a cost on the visit count , beats everything here, 20 points past the published baselines. Offline planning on a recovered support is therefore not a shortest-distance problem: the support's boundary is not the world's, and the same cost improves every planner we tested. A predictive map is a usable offline planner whose value lies less in the routes it produces than in what it reveals about which routes are worth producing.

Primary area:
reinforcement learning
Keywords:
cognitive map, place cells, predictive map, offline goal-conditioned reinforcement learning, planning, fast marching, OGBench

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_iyRGOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper investigates whether the place-cell model of Zhao et al. (2025), non-negative embeddings of multi-scale random-walk transition kernels, can serve as an effective planner when fitted to offline trajectories rather than an analytical ground-truth map. The proposed method discretizes visited (x,y)(x, y) positions into a spatial grid, trains a coordinate network on the multi-scale kernel over this grid, and plans via sampled ascent on the learned similarity representation following Lin et al. (2026), termed Preplay.

Critical strengths
  1. The paper explains why the greedy read-out of Zhao et al. (2025) fails when the kernel is fitted to offline data.
Critical weaknesses

Overall, as the authors disclosed in the AI use statement, the paper appears to be mostly written by AI, which is allowed at this conference. However, the authors should have revised the paper for better readability and chosen proper experiments.

  1. First of all, the paper is very hard to read, and the writing must be improved.

    1.1 Almost every sentence contains a colon or a semicolon (157 colons and 103 semicolons, including the semicolons in citations), e.g., The pair weight is not: removing ... in L184 and The map is a network: a shared trunk ... in L170.

    1.2 The symbol hh in Figure 1 is not defined. I recommend making each figure and its caption self-contained.

    1.3 Several sentences are not written in an academic tone, e.g., Table 2 varies nothing but where the waypoints come from, ... in L373.

    1.4 In L99, it is hard to understand what 14–7–3 means.

    1.5 In L107, Because the analysis needed it, and the analysis is what we defend. is not grammatically correct.

    1.6 The first sentences of many paragraphs are hard to understand. For example, the authors begin Section 3.1 with Only the planning is search-free, not the pipeline. (L160). However, it is unclear what the authors want to argue.

    1.7 Some terms are not properly explained, such as cov. Although readers can infer that it is related to coverage, it is better to give the full name when a term first appears.

    1.8 The free set F\mathcal{F} is used in L161 before it is defined in L166.

  2. The paper heavily relies on Zhao et al. (2025), but there is no visualization or analysis of place cells.

  3. The novelty of the paper compared with Zhao et al. (2025) and Lin et al. (2026) is not well addressed, which could be due to the writing. If the authors clearly explain the novelty, it would be easier to understand the contribution of the paper.

  4. The authors omitted the count-based exploration literature. Equation 6 is exactly the same as the standard count-based exploration bonus in reinforcement learning [R1, R2], but the authors did not acknowledge it. Also, regarding the choice of α\alpha, the authors state We report $\alpha = 1/2$ because it was the first value we tried (L432). This is not a valid reason for choosing a hyperparameter.

  5. Multiple experiments in the Appendix are not well justified, and most of them might be unnecessary.

    5.1 The main argument of the paper is related to neither hardware nor inference time. Hence, I do not understand the contribution of Table A3, which compares the evaluation on CPU and GPU.

    5.2 The authors show the results of each seed in Table A4, which is appreciated. However, it would be better to explain why seeds 7, 14, and 1234 were chosen. Otherwise, I recommend simply labeling them Seed 1, Seed 2, and Seed 3.

  6. The comparison with the OGBench baselines in Table 1 is not fair. Preplay discretizes the x-y positions of the state (L165-167), which the baselines (e.g., HIQL) do not use.

Minor In L354, 'like for like' -> 'like-for-like'.

[R1] Bellemare, Marc, et al. "Unifying count-based exploration and intrinsic motivation." NeurIPS. 2016.

[R2] Tang, Haoran, et al. "#Exploration: A study of count-based exploration for deep reinforcement learning." NeurIPS. 2017.

Questions for authors

N/A

Reviewer_XTaKOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

I wasn't able to understand almost anything despite my best efforts.

Critical strengths

None, given that I couldn't understand the paper.

Critical weaknesses

I found the paper extremely difficult to follow, and I wasn’t able to understand almost anything. Technical terms, mechanisms, and empirical claims are introduced before the problem is even formulated. E.g., terms like multi-scale transition kernels, non-negative spectral embeddings, similarity-field ascent, support geometry, and “executability” are introduced without explanation.

Given the unreadability of the paper, I wasn't able to judge the significance of the results.

I do not think the current manuscript meets the clarity bar for ICLR. I believe this paper needs a major writing revision before resubmission.

Questions for authors

None

Reviewer_cn1BOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Preplay, a planning method that reads out routes from learned cognitive maps fitted to offline trajectories. It recovers accessible regions, constructs multi-scale random-walk kernels on that support, and fits them with nonnegative neural representations. Preplay uses similarity ascent and switches to stochastic sampling when deterministic ascent cycles. I see the main contribution as the empirical finding that, with the same map and controller, favoring frequently visited regions improves execution success over distance minimization alone. The authors study 24 OGBench maze datasets and use cycle diagnostics and geometric controls to examine this finding. On the 16 embodied datasets, visitation-weighted FMM-cov improves mean success over unweighted FMM by 30.7 percentage points.

Critical strengths
  1. The executability finding has targeted controls. I appreciate that Table 2 holds the map, controller, and executor fixed across planners. Table A10 then compares visitation counts, boundary clearance, and shuffled counts. Clearance alone recovers 23.2 of the 30.7-point gain. I read this as evidence that route geometry explains much of the benefit, with visitation patterns potentially adding information. The authors appropriately avoid treating this comparison as a causal decomposition.
  2. The failure analysis is thorough and includes unfavorable results. I found the comparisons of exact and fitted kernels, cycle statistics, controller experiments, and analyses of data size, grid resolution, and transfer across embodiments informative (Table A6; Appendices H, L, and P). I also appreciate the authors' candor about failed closed-loop control from pixels, the fact that their strongest planner does not require a learned map, and the absence of a claim to outperform GAS. These results make the empirical analysis more useful to me.
Critical weaknesses
  1. Search effects are not fully separated from route post-processing. My main concern is that cov-A* and FMM-cov use the same cost but differ by an average of 13.3 percentage points across all 24 datasets. The former applies line-of-sight shortcutting; the latter retains cell-by-cell routes. Appendix M acknowledges that the gap remains unexplained. Shortcuts may cross low-visitation regions that the cost was intended to avoid. I also could not determine whether selecting the neighbor with the lowest remaining cost includes the current edge cost. With nonuniform costs, minimizing D(v) is not equivalent to minimizing c(u,v)+D(v). This does not invalidate the gain from weighting FMM, but it leaves me uncertain about how to interpret the planner rankings and the sources of their differences.
  2. Local agreement does not yet explain how fitting errors trigger cycles. The 99.2% neighbor agreement applies only to states where the exact field has a strict maximum, which Appendix H says account for about 71% of states. I think the remaining states and their tie-breaking rules deserve closer attention. I recognize that the authors already acknowledge that fitting can break the original readout and show that all logged failures form cycles. What I still miss is an explanation of how those cycles begin. I therefore find the inference in §4.1 from high local agreement to a readout failure too strong. An analysis of cycle entry that includes tied states would help; otherwise, I would narrow this interpretation.
Questions for authors
  1. Clarification and additional control: Could you clarify how FMM-cov handles edge costs during route reconstruction? I would like to know whether the 13.3-point gap remains when A* and Dijkstra both reconstruct routes under the same weighted objective, with shortcutting disabled.
  2. Mechanism clarification: At the first deviation from the exact-kernel route in failed trajectories, how often does the exact field have a strict maximum versus a tie? How do cycling and success rates change under a common tie-breaking rule or a simple rule that prohibits immediate backtracking? These results would affect my assessment of the need for stochastic sampling and the proposed mechanism.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.