acceptodds
Under review as a conference paper at ICLR 2027

Cost-to-Go Estimation by Decomposed Latent Path Integration

Abstract

Estimating the effort required to reach a goal from a given state is a fundamental problem in planning and control. Many recent approaches estimate this cost by encoding the start and goal observations independently and computing a metric or quasimetric between their representations. This paper introduces decomposed latent path integration (DeLPI), an architectural primitive that estimates distance through accumulated motion in latent space. In our instantiation, the two observations are jointly encoded into a spatial grid of latent vectors, which is iteratively updated by a depth-recurrent module. The predicted distance is computed as the sum of the latent vector displacements produced over these updates. We compare our method against metric, quasimetric, and joint-encoding baselines on four environments with nontrivial combinatorial structure: Switchyard, a gridworld with a controllable degree of factor coupling; Sokoban; Rubik's cube; and Craftax. Across these environments, it either outperforms the baselines or achieves comparable performance with substantially fewer parameters. More broadly, our experiments show that, in these environments, architectures that jointly process the start and goal states consistently outperform approaches that embed the two states independently.

Primary area:
reinforcement learning
Keywords:
Cost-to-Go Estimation, Architectural primitive, Latent path Integration, Gridworlds, Goal-conditioned value function, Depth-recurrent models

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_QcA7OpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies how to estimate the number of steps needed to get from a start state to a goal state, a quantity used as a heuristic in planning and as a value in goal-conditioned RL. It proposes DeLPI, which feeds the start and goal into one network together, refines an internal representation step by step, and reads the distance as the total amount the representation moved during those steps, instead of predicting a number from the final representation. It also introduces Switchyard, a gridworld where the coupling between mechanisms can be controlled. On Switchyard, Sokoban, Rubik’s cube, and Craftax, DeLPI beats or matches both separate-encoding and joint-encoding baselines, with a readout ablation and an analysis of which cells contribute to the predicted distance.

Critical strengths
  • The path-integral readout is a simple and, to my knowledge, new architectural idea, and the paper provides a readout ablation against a linear readout on the same encoder and recurrent body. It also demonstrates improved performance over both separate-encoding and joint-encoding baselines in estimation accuracy and in downstream planning.

  • The paper includes a fairly complete set of ablations over how the distance is read out, the number of iterations, the encoder, the parameter budget, the label quality and the training objective. This makes it possible to attribute several of the reported gains to specific design choices.

Critical weaknesses
  • The paper should provide more analysis of why path integration works better than decoding. The only justification given is that recurrence and integration “complement each other”, which restates the result rather than explaining it. More analysis is also needed on the assumptions behind the readout and on when it applies, such as whether it relies on the cost being a simple count of unit steps or on the latent vectors being arranged by grid position. The failed slot variant in Appendix D suggests that such conditions exist, and the paper does not state them. In addition, the readout ablation is run only on Switchyard against a weak alternative head, and the recurrent body is never separated from the readout, so the gains over CoAt cannot be attributed to path integration alone.

  • Parameter efficiency is reported, but computational cost is not. A small parameter count does not imply cheap inference, since DeLPI reuses the same block over many recurrent updates, and the search results report nodes generated rather than time. Per-query inference time and total planner time should be reported, especially since the method is intended to be used inside a search loop.

Questions for authors
  • Can the authors provide more analysis of why the path-integral readout works and when it applies, and report inference time per query and total planner time for DeLPI and the baselines, as stated in the weaknesses above?

  • Can the authors isolate the effect of the recurrent body from that of the readout, for example by adding recurrence to CoAt or using untied weights in DeLPI? Can they also repeat the readout ablation on the established benchmarks such as Sokoban, using a stronger alternative readout than mean pooling followed by a linear layer?

5 comments

Thank you for reading the paper and suggesting the improvements. The replies are bellow and the changes in the PDF which were prompted by your comments are colored in green.

W1. Why path integration works, when it applies, and separating the recurrent body from the readout.

We agree that "complement each other" restates the result. Below we give the motivation for the readout, the evidence we have, and the limits of what we can claim.

Motivation. A recurrent network unrolls a vector in latent space, so it is natural to read the prediction from the trajectory itself rather than from its final point. Our original motivation was that planning environments decompose into interacting factors (an agent, a key, a door), and that a network could simulate the trajectory of each factor in latent space; the sum of the trajectory lengths would then reflect the total movement needed. In a slot-based version where the factorization is given as input, this holds closely: the per-slot displacement tracks the number of moves of its factor along a shortest plan (correlation 0.880 to 0.998 over three seeds), and decoded trajectories visit the required unlocking positions in 96% to 100% of cases. Our response to Reviewer H8ob explain this in more details ( https://openreview.net/forum?id=n0ISmEOIDF&noteId=4WhkJXWnSo ). When the slots need to discover the factors from images, this version is less accurate than both DeLPI and CoAt (Table 21, Appendix D). The grid-based architecture of the paper keeps the readout and is more accurate, but loses this clean per-factor interpretation. We therefore do not claim to know exactly why the readout works on the grid model, and we report what we can measure.

What we observe.

  • The readout has no parameters, yet none of six learned heads that aggregate the intermediate latent vectors is more accurate (Tables 16 and 17, Appendix C).
  • With the same encoder and recurrent body, the integration readout benefits more from additional iterations than a linear head on the final state (Figure 6).
  • The latent motion concentrates on the cells of a shortest path and on the mechanisms the plan relies on. The first iteration singles out 57% of the mechanisms on the shortest path, against 31% for the most selective layer of CoAt (Section 5 and Appendix C, Figure 13).

Separating the recurrent body from the readout. We added two controls on Switchyard 11×11 (Table 15, Appendix C). Making CoAt recurrent raises its correlation from 0.940 to 0.958, still far from DeLPI (0.985), and doubling its iterations does not change this (0.961). Untying the weights of DeLPI achieves 0.980 with 6.7 times the parameters. Together with the readout comparisons above, which keep DeLPI's encoder and recurrent body fixed, this indicates that recurrence alone does not explain the gains over CoAt.

When the readout applies. We did not find a clean condition which would give us an indication for the applicability of the architecture. The readout itself does not require latent vectors arranged by grid position, since it works on slots when the factorization is given, but the slot version that learns its factors from images is less accurate than the grid model. Depth-recurrent architectures are becoming popular and we believe that the integration readout is a natural choice for them and moreover does not require additional parameters. This paper demonstrated that it is competitive on standard benchmarks which are being used for cost-to-go prediction.

AuthorsOpenReview ↗

W2. Computational cost.

Thank you again for pointing this out. The paper reported only the number of parameters, and we agree that this does not measure the cost of a weight-tied model. We now report FLOPs, time per query, planner time and training time in Appendix C.

It is true that the recurrence makes a query more expensive. With the 8 iterations of two blocks used on the gridworlds, a query of DeLPI costs 2.2 to 2.5 times the FLOPs of CoAt, the strongest baseline, and takes about 5 times as long. A training step takes about 4 times as long, but at equal training time DeLPI is still ahead (correlation 0.982 against 0.940 after 0.6 hours on Switchyard 11×11). In planning, the total time also depends on how many nodes the search generates. On Sokoban, where the simulator dominates the cost of a node, DeLPI generates 4.7 times fewer nodes than CoAt and solves a board 3.5 times faster (11.4 against 39.6 s). On Switchyard, with the 8 iterations of the paper, its planner takes 1.7 times as long and solves more pairs.

The number of iterations sets a trade-off between accuracy and computation. Each iteration applies the same two blocks once more, so FLOPs and time per query grow linearly with it, while accuracy saturates. On Switchyard 11×11, with four seeds and time per query on one H100, the numbers are as follows:

modelMFLOPs per queryms per querycorrelationMAEgreedy success
DeLPI, 1 iteration1451.20.9332.560.33
DeLPI, 2 iterations2642.10.9611.810.47
DeLPI, 4 iterations5034.00.9781.240.67
DeLPI, 8 iterations (paper)9807.60.9850.920.78
CoAt4361.40.9402.240.38

With 2 iterations DeLPI is already more accurate than CoAt with 60% of its FLOPs. Each further doubling lowers the error by about a third, and greedy success more than doubles from 1 to 8 iterations. On the Rubik's cube, DeLPI with its 16 iterations costs 308 MFLOPs per query, less than the 459 of CoAt with 1.83M parameters, which it matches in accuracy. With 2 iterations it needs 43 MFLOPs, about half of CoAt with 520k parameters, and comes within 0.01 of it in correlation.

We add these measurements and the table to Appendix C, and Section 4 now explicitly states next to the parameter count that fewer parameters do not mean less computation.

AuthorsOpenReview ↗

Answers to questions

Q1. Analysis of the readout, inference time and planner time. Both are answered above: why the readout works and when it applies in W1, and the time per query and the planner time in W2.

Q2. Separating the recurrent body from the readout. We ran both controls you suggest, on Switchyard 11×11 with four seeds. All rows are new runs, so DeLPI and CoAt differ slightly from the paper:

modelparameterscorrelationgreedy success
CoAt1.81M0.9400.38
CoAt with recurrence (its blocks tied and repeated)0.70M0.9580.49
DeLPI with untied weights3.69M0.9800.74
DeLPI0.55M0.9850.78

Recurrence helps CoAt, but leaves it far from DeLPI, and doubling its iterations does not change this (0.961). Untying DeLPI's weights does not help either: with 6.7 times the parameter count it reaches 0.980.

So the recurrence alone does not explain the gain over CoAt. The gain comes from the jointly encoded recurrent network and its readout, and among the readouts we tried the path length is the most accurate and needs no parameters.

We added these controls to Appendix C (Extended results).

Q3. A stronger readout, also on Sokoban. As suggested also by reviewer H8ob, we trained six stronger heads on DeLPI's encoder and recurrent network on Switchyard 11×11, each at its tuned learning rate. They are per-cell MLPs on the latent vector after every update, on every update (the delta between states), or on several intermediate states at once, and an MLP on the latent vectors averaged over the cells. We also trained the per-cell MLP on the latent vector after every update on Sokoban, with four seeds on both benchmarks.

Switchyard 11×11correlationgreedy successpairs solved with a shortest plan
path length (DeLPI)0.9850.780.90
the six learned heads0.976 to 0.9840.69 to 0.750.85 to 0.89
SokobanMAE in stepsgreedy successsearch successnodes per board
path length (DeLPI)0.850.830.983262
per-cell MLP on the latent vector after every update1.000.770.966377

No learned head is more accurate than the path length, which adds no parameters. On Switchyard four of the six heads are significantly less accurate (p ≤ 0.027), and the closest comes within 0.001 in correlation with 157k added parameters (p = 0.19). The path length also finds shortest plans more often than every head. On Sokoban it is ahead on every metric (p ≤ 0.004).

We added both tables to Appendix C (Tables 16 and 17), and the Boxoban comparison is in the text above Table 17.

Reviewer_srBZOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

Heuristic search and goal-conditioned RL rely on an estimate of how far a state is from the goal to guide their search. The start and the goal are usually encoded separately, and then the distance between them is computed using a metric or a quasimetric. The paper argues that this split approach is well suited if the environment can be split into independent factors, but for more complex tasks that involve coupling, such as going to a place to collect a key and then opening a door, this decoupling loses relational information. To tackle this problem, the paper proposes an algorithm, DeLPI, which jointly encodes the start and goal states into a grid of latent vectors, predicts the distance as the total length of the path these vectors trace across updates, and shows that DeLPI outperforms or achieves comparable performance across four considered tasks. Additionally, it proposes a new “Switchyard” environment, which is a Sokoban-like gridworld with a controllable degree of coupling.

Critical strengths
  1. The paper provides clear reasoning for why encoding the state and the goal together is meaningful for sequential tasks. Figure 1 concretely summarizes the entire approach, and Figure 8 helps in explaining the architectural differences between DeLPI and the considered baselines.
  2. Training procedures and hyperparameters are well-detailed in the paper to ensure reproducibility.
  3. The paper is honest about how the current approach mostly works in the supervised setting, where the true distance from start to goal is known beforehand (computed using BFS in this case), and it outlines future work to address this.
Critical weaknesses
  1. The introduction begins by explaining how embedding struggles arise when environmental factors interact, and how DeLPI can handle this coupling better. But this is not tested fully on its own in any of the experiments. The main results include different maze sizes, and the appendix shows how the mechanism changes with different levers and doors, but the maze size still changes there as well. Adding a cleaner experiment in which the maze size remains fixed, and coupling is increased one step at a time, can help determine whether DeLPI handles coupling properly.
  2. The presentation of the algorithm (Section 2) itself is quite concise; it feels almost like a series of bullet points; I would have preferred a more narrative structure explaining the intuition of many of those choices. Also, the use of some synonyms is quite confusing. For example: “Estimating the effort required to reach a goal from a given state is a fundamental”. What is effort? I think the paper means cost, but is effort a common word used by the community?The paper mentions that DeLPI has fewer parameters, but this doesn't mean that it is faster. In fact, DeLPI has only two blocks, but it applies them 16 times, so this recurrence can potentially make it slower. A better comparison will be to report FLOPs or the wall-clock time.
  3. The paper's main claim is path integration, but DeLPI differs from other baselines in many ways: the start/goal is reinjected at every block, the weight-tied recurrence, where the attention block sits, and the readout itself. So, it's a bit difficult to pinpoint the main cause of the observed gains.
  4. Another key claimed contribution in the paper is that “We show that architectures that process start and goal states jointly in most cases outperform embedding models that encode them separately.”. It is unclear to me that the paper presents results that support the claim that processing start and goal states jointly is beneficial. Is there a clear experiment about that? It seems to me that what is being shown is that, overall, the whole algorithm works.
  5. Most of the claims are based on 3-4 seeds, and no statistical tests are reported. Adding some tests will strengthen the claims. I also have concerns about the measures of spread used, and the used for choosing checkpoints (see questions below).
Questions for authors
  1. Hyperparameter tuning is somewhat uneven, and training details vary across tasks. Is there any specific reason for this? Also, in the ranking experiment, why do DeLPI and CoAT receive different input encodings (seven vs five channels)?
  2. Why does the paper report the checkpoint with the lowest/highest validation instead of the last checkpoint? Isn’t the training process robust enough that more training always improves results? Could performance collapse?
  3. Across all results, are you reporting the standard deviation? This is not stated in Figure 2, for example. Also, the reported spread seems quite small. Are you reporting the spread over 4 samples, measuring the variance across training procedures, or over the evaluations (e.g., 1000 evaluations), using a denominator that is much higher to compute the variance?
  4. Craftax environment comes with creatures, randomness, and partial observability, but that’s not the case in the environment being used. Is there any specific reason for that? How will DeLPI apply to stochastic and partially observable settings?
  5. In your ablation (end of Section 5), did you tune the hyperparameters for DeLPI-I, or did you maintain the original hyperparameters you found for DeLPI?
  6. Will the solution from DeLPI be admissible and optimal? Current search results report only the success and node counts, but not the solution length.
12 comments
AuthorsOpenReview ↗

W1. Coupling at a fixed grid size

We thank the reviewer for a careful reading and the suggested improvements. We adress them with new experiments and updates of the text.

We agree that this is a clean test, and we report the results bellow. On a fixed 11×11 grid we added one mechanism at a time and trained all seven models of Figure 2 at every level, with four seeds each and every model at its tuned learning rate. These are new runs, so the last row differs slightly from Figure 2. The table gives the correlation between predicted and true distances:

mechanismsDeLPICoAts-CRTR (reg)s-CRTR (class)SymIQEMRN
agent only1.0000.9990.9990.9990.9960.9940.993
+ pushable box0.9980.9850.9910.9880.9460.9650.975
+ gates, one lever each0.9930.9750.9350.9260.8820.8730.919
+ arbitrary lever wiring0.9900.9610.9270.9200.8610.8510.873
+ pressure plate0.9880.9450.9250.9170.8590.8520.870
+ one-way door (the paper's 11×11)0.9850.9400.9210.9180.8550.8430.864
total drop0.0150.0590.0790.0810.1420.1510.129

The results support the argument of the introduction. DeLPI's accuracy stays almost unchanged as coupling increases, with its correlation going from 1.000 to 0.985. The embedding models (Sym, IQE and MRN) are affected most. They are nearly perfect when only the agent moves, but fall to about 0.85 once all five mechanisms interact. The baselines that see both states jointly (CoAt and the two s-CRTR networks) fall in between, at 0.92 to 0.94. The difference also shows in planning, where with all five mechanisms the greedy policy solves 78% of the test pairs with DeLPI and at most 42% with any baseline.

We added this experiment to Appendix C (Extended results), with the full table, and refer to it in Section 4.

AuthorsOpenReview ↗

W2. Presentation and FLOPs.

The presentation of the algorithm (Section 2) We agree that Section 2 described the components of DeLPI without explaining why they were chosen. We have revised the section to open with the motivation behind the architecture. In short, planning environments often decompose into interacting factors, such as an agent, a key and a door. Each factor has a much smaller state space than their joint configuration, but their shortest paths depend on one another: the agent must collect the key before the door can open. We therefore wanted a network that represents the factors with separate latent vectors and moves them in latent space as if simulating a plan, including the detours one factor makes to unlock another. If the network learns such a simulation, the distance to the goal is the total motion the factors require, which DeLPI reads off as the summed length of the latent trajectories. The revised section also explains why the main experiments use one latent vector per grid cell rather than the slot-based variant we started from: the grid version kept the trajectory readout and was more accurate, at the cost of vectors that no longer correspond to single factors. For a more detailed account of how the architecture was conceived, including our exploratory results with the slot-based variant, we refer the reviewer to our response to Reviewer H8ob https://openreview.net/forum?id=n0ISmEOIDF&noteId=4WhkJXWnSo or the TL;DR below that response.

Effort Thank you for pointing this out. We initially wrote the motivation in more loose and aspirational terms and claimed that estimating effort is a fundamental ability which every biological creature has to deal with. We eventually rewrote it to more grounded language but the term effort stayed there and we agree that it was confusing. It is now replaced by the term “cost”.

FLOPs and wall-clock time. Thank you again for pointing this out. The paper reported only the number of parameters, and we agree that this does not measure the cost of a weight-tied model. We now report FLOPs, time per query, planner time and training time in Appendix C.

It is true that the recurrence makes a query more expensive. With the 8 iterations of two blocks used on the gridworlds (the 16 applications the you mention), a query of DeLPI costs 2.2 to 2.5 times the FLOPs of CoAt, the strongest baseline, and takes about 5 times as long. A training step takes about 4 times as long, but at equal training time DeLPI is still ahead (correlation 0.982 against 0.940 after 0.6 hours on Switchyard 11×11). In planning, the total time also depends on how many nodes the search generates. On Sokoban, where the simulator dominates the cost of a node, DeLPI generates 4.7 times fewer nodes than CoAt and solves a board 3.5 times faster (11.4 against 39.6 s). On Switchyard, with the 8 iterations of the paper, its planner takes 1.7 times as long and solves more pairs.

The number of iterations sets a trade-off between accuracy and computation. Each iteration applies the same two blocks once more, so FLOPs and time per query grow linearly with it, while accuracy saturates. On Switchyard 11×11, with four seeds and time per query on one H100, the numbers are as follows:

modelMFLOPs per queryms per querycorrelationMAEgreedy success
DeLPI, 1 iteration1451.20.9332.560.33
DeLPI, 2 iterations2642.10.9611.810.47
DeLPI, 4 iterations5034.00.9781.240.67
DeLPI, 8 iterations (paper)9807.60.9850.920.78
CoAt4361.40.9402.240.38

With 2 iterations DeLPI is already more accurate than CoAt with 60% of its FLOPs. Each further doubling lowers the error by about a third, and greedy success more than doubles from 1 to 8 iterations. On the Rubik's cube, DeLPI with its 16 iterations costs 308 MFLOPs per query, less than the 459 of CoAt with 1.83M parameters, which it matches in accuracy. With 2 iterations it needs 43 MFLOPs, about half of CoAt with 520k parameters, and comes within 0.01 of it in correlation.

We add these measurements and the table to Appendix C, and Section 4 now explicitly states next to the parameter count that fewer parameters do not mean less computation.

AuthorsOpenReview ↗

W3. The main cause of the observed gains

We believe that the combination of path integration and recurrence is the main conceptual contribution of the paper and we mainly wanted to demonstrate that this combination is competitive with SOTA baselines. Depth-recurrent architectures are becoming more standard and the path integration readout is natural fit for them because it does not need training parameters. As explained in the previous comment, we iterated over several versions of the architecture until we found a version that we present in the paper. The start/goal reinjection was one of the tricks which made the training more stable.

Below we present several ablations which make the role of individual components more transparent.

Separating the recurrent body from the readout. We ran two controls which separate the role of recurrence, on Switchyard 11×11 with four seeds. All rows are new runs, so DeLPI and CoAt differ slightly from the paper:

modelparameterscorrelationgreedy success
CoAt1.81M0.9400.38
CoAt with recurrence (its blocks tied and repeated)0.70M0.9580.49
DeLPI with untied weights3.69M0.9800.74
DeLPI0.55M0.9850.78

Recurrence helps CoAt, but leaves it far from DeLPI, and doubling its iterations does not change this (0.961). Untying DeLPI's weights does not help either: with 6.7 times the parameter count it reaches 0.980.

So the recurrence alone does not explain the gain over CoAt. The gain comes from the jointly encoded recurrent network and its readout, and among the readouts we tried the path length is the most accurate and needs no parameters (Q3).

A stronger readout. To test whether a learned readout does better than the path length, we trained six learned heads that aggregate the intermediate latent vectors, on DeLPI's encoder and recurrent network on Switchyard 11×11, each at its tuned learning rate. Five are MLPs applied per cell to the latent vector after every update, to every update, or to several latent vectors at once, and one is an MLP on the latent vectors averaged over the cells. One of them we also trained on Sokoban, with four seeds on both benchmarks.

Switchyard 11×11added parameterscorrelationgreedy successpairs solved with a shortest plan
path length (DeLPI)00.9850.780.90
the six learned heads9k to 157k0.976 to 0.9840.69 to 0.750.85 to 0.89
Sokobanadded parametersMAE in stepsgreedy successsearch successnodes per board
path length (DeLPI)00.850.830.983262
per-cell MLP on the latent vector after every update9k1.000.770.966377

No learned head is more accurate than the path length, which adds no parameters. We added more detailed comparisons for the results presented above to Appendix C (Extended results).

Re-injection. We trained DeLPI on Switchyard 11×11 with and without re-injecting the start and goal at every block, with T = 1, 2, 4 and 8 iterations, four seeds each, and all other settings unchanged. Re-injection mainly stabilizes the recurrence as the number of iterations grows. At T = 4 it halves the spread across seeds (sd of the correlation 0.003 against 0.006). A model trained with T = 4 and run for 12 iterations at test time keeps a correlation of 0.943 ± 0.007 with re-injection, against 0.910 ± 0.040 without. At T = 8, the setting of the paper, the difference in accuracy is small:

Switchyard 11×11, T = 8correlationMAE in steps
with re-injection (DeLPI)0.9850.92
without re-injection0.9840.97
CoAt0.9402.24
Reviewer_H8obOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a new approach for estimating the cost-to-go between a start state and a goal state. Instead of directly predict the distance from static embeddings, DeLPI jointly encodes the states and estimates the distance by accumulating changes in latent representations over recurrent updates. The method is evaluated on four planning environments, and it shows that the method generally outperforms existing distance estimation methods or achieves comparable performance, while also improving planning efficiency.

Critical strengths

The author introduces a simple but interesting idea of estimating cost-to-go through accumulated latent displacements rather than directly predicting distance from static representations.

The method performs well across four planning environments. And the analysis shows that latent displacements tend to concentrate on path-relevant regions and actionable objects, providing some insight into the model's behavior.

Critical weaknesses

The proposed method accumulates latent displacements across recurrent network updates, but these updates are not explicitly related to physical movements, valid state transitions, or planning trajectories. In conventional spatial path integration, latent representations track physical movement and can often be decoded into spatial positions. Here, the latent updates are driven by neural network computation rather than environment dynamics. Therefore, I feel it is still unclear whether the model learns meaningful planning representations or simply uses latent displacement as an alternative way to predict a scalar distance.

I'm not sure if there's a clear reason why accumulated latent displacement should correspond to optimal cost-to-go, beyond being fitted through supervision.Maybe author can explain more about this logic.

The paper only compares path integration with a final-state readout. Stronger baselines that aggregate intermediate representations would better isolate the contribution of path integration. Also, the experiments focus on structured environments with supervised distance labels. I'm worried about the performance of generalization to longer horizons and more complex planning task.

Questions for authors

N/A

4 comments
AuthorsOpenReview ↗

Explanation of the logic behind the architecture.

We thank the reviewer for asking us to clarify the motivation behind DeLPI and the interpretation of its latent trajectories. Spatial path integration was exactly the concept which motivated the architecture. This response explains the original motivation and the evidence from the slot-based variant. We address the remaining points in separate responses.

Our original motivation was that planning environments often decompose into interacting factors, each with a much smaller state space than their joint configuration space. For example, an agent, a key, and a door can be represented separately, but their trajectories depend on one another: the agent must collect the key before it can unlock the door. This motivated us to seek representations that compose across factors. We hypothesized that a network could represent these factors and simulate their trajectories in latent space, including intermediate states needed to enable changes in other factors. Summing latent trajectory lengths was intended to reflect the total movement required across the factors.

This motivated our initial slot-based DeLPI architecture, where slots were intended to represent individual factors. We first tested this idea in a symbolic environment where the factorization was supplied as input, so the slots did not have to learn which factors to represent (slot of the agent would be given position of the agent as an input). In this version, the accumulated per-slot displacement closely tracked the corresponding factor's number of moves along a shortest plan, with correlations from 0.880 to 0.998 (in three seeds). Over the three seeds, decoded trajectories visited required unlocking positions (e.g. position of a key) in 96% to 100% of cases. So in this case, the latent trajectories can be viewed as simulations of shortest paths in the underlying state-space.

We then explored image observations, where the correspondence between slots and factors had to be learned. To examine the agent's representation, we selected the slot with the highest attention to the agent's starting cell and trained a linear decoder to predict its grid position from the initial slot vector (before the recurrent updates). The decoder was trained using the agent's starting position as the label and evaluated on separate maps. We then applied the same decoder to the same slot at each recurrent update. The decoder localized the initial agent position within a 3×3 neighborhood in 94.9% of held-out examples, averaged over five decoder fits.

To test whether the trajectory corresponding to a given factor visits the unlocking state, we constructed controlled tasks in which the agent had to visit a specific lever. Each starting state was paired with two gate goals, one requiring lever A and the other requiring lever B. Changing the goal from requiring lever B to requiring lever A increased the decoded probability assigned to the position of lever A relative to the position of lever B in 70.8% of paired comparisons, averaged over intermediate updates. This measures a shift of the trajectory toward the relevant location which the agent needs to visit in the environment, but the observation is not as clean as in the case of symbolic representations. We therefore regard these exploratory results as limited evidence of goal-dependent spatial information within the trajectories of slot-based DeLPI.

Comparisons with the reported baselines subsequently motivated the grid-based DeLPI architecture used in the main paper. It retained the displacement readout and achieved better predictive performance, while its representations were harder to interpret as individual factor trajectories. We therefore selected it for the main experiments and retained the conceptually more interesting slot-based variant as an auxiliary comparison in Appendix D. We will add a detailed explanation of this logic in the paper during the discussion period.

TL;DR for the response above: DeLPI was motivated by representing interacting factors and simulating their trajectories in latent space, including intermediate states needed to enable other factors to move so spatial path integration was exactly the concept which motivated the architecture. Accumulated latent movement was intended to reflect planning cost. The initial experiments with the hard-coded slot model supported this motivation. The per-factor displacement tracked shortest-plan moves, and decoded trajectories visited required unlocking states in 96% to 100% of cases.

Evidence for spatial tracking in the slot-based DELPI where the factors need to be learned is weaker. Changing the required lever needed for the goal shifted decoded probability toward its position in 70.8% of matched comparisons. This suggests goal-dependent spatial information in the latent trajectories, but does not establish reliable trajectory simulation.

The main model of the paper was developed later. It retains the displacement readout and achieves better predictive performance, while individual factor trajectories become harder to interpret. We will clarify this motivation and evidence in Appendix D.

AuthorsOpenReview ↗

Stronger readouts that aggregate intermediate states

We thank the reviewer for suggesting to compare the integration readout to other readouts that aggregate the intermediate states. Below we show 6 variants which keep DeLPI's encoder and recurrent network and changes only how the distance is computed from the latent states:

  • per-cell MLPs whose outputs are summed over cells, applied to every intermediate state or to every update (the difference between consecutive states);
  • per-cell MLPs on concatenated intermediate states (states 8, 12 and 16, every fourth state, or all 17 states), and an MLP on all 17 states averaged over cells.

The learning rate of every readout was tuned on validation maps. The results are for Switchyard 11×11, the mean ± standard deviation are reported over 4 seeds:

readoutadded parameterscorrelationgreedy successsolved with a shortest path
path length (DeLPI)00.9850.780.90
per-cell MLP on the latent vector after every update9k0.9760.690.86
per-cell MLP on every update9k0.9830.750.89
per-cell MLP on the latent vectors after updates 8, 12 and 1628k0.9810.740.86
per-cell MLP on the latent vectors after every fourth update46k0.9820.730.88
per-cell MLP on the latent vectors after all updates157k0.9840.750.88
MLP on the latent vectors averaged over the cells157k0.9780.690.85
CoAt (for reference)0.9400.380.72

The learned readouts come close to path integration: they are within 0.001 to 0.009 of its correlation, and all are clearly ahead of CoAt. None of them is more accurate, but each adds 9k to 157k parameters (up to 28% of DeLPI's 552k), whereas path integration adds none. Four of the six are significantly less accurate (Welch's t-test, p ≤ 0.027). The closest readout, the per-cell MLP on the latent vectors after all updates, is also the largest, and its difference from DeLPI is not significant (p = 0.19). The learned readouts also need their own learning rate: at DeLPI's learning rate of 1e-3 they reach only 0.848 to 0.945. Path integration thus matches the strongest learned readouts without parameters or tuning of its own. We add these results to Appendix C of the revised paper, and Section 5 refers to them after the comparison with the linear head on the final state (the revised paper will be uploaded with more changes by tomorrow).

AuthorsOpenReview ↗

Generalization to longer horizons and more complex planning tasks

We thank the reviewer for suggesting an analysis of length and difficulty generalization. Length generalization was in fact the main aspect that we hoped our architecture would enable. We hoped that the iterative nature of the model would allow us to generalize to longer horizons through test-time scaling and the first experiments we ran were testing length generalization. The results for length generalization were eventually not as significant as we hoped (even though we still generalize better than other baselines), and we therefore did not focus on them in this paper.

To demonstrate how DELPI compares to other baselines in terms of length and complexity generalization, we ran two tests without retraining the models: larger Switchyard grids, and harder Sokoban levels from a public benchmark which were not seen during training.

Larger grids. We evaluated the models trained on Switchyard 11×11 on the larger grids of the paper, from 13×13 to 19×19. These grids have the same five mechanisms and up to three times the area, and up to 46% of their test pairs are longer than any pair seen in training. DeLPI has no positional input and runs unchanged, and for CoAt we generated its positional encoding at the new size. All seven models are at their tuned learning rates, with four seeds. The table gives the correlation between predicted and true distances:

gridDeLPICoAts-CRTR (reg)s-CRTR (class)SymIQEMRN
11×11 (trained here)0.9850.9400.9210.9180.8550.8430.864
13×130.9800.9310.9170.9010.8400.8260.838
15×150.9490.9000.8820.8270.8080.7960.798
17×170.9360.8810.8640.7070.7870.7570.755
19×190.9140.8480.8270.5370.7580.7030.710

DeLPI is the most accurate model at every larger size.

Harder Sokoban levels. We also evaluated the models trained on Sokoban, unchanged, on the hard set of Boxoban (Guez et al., 2018). The table below gives the search success, the fraction of the first 500 hard levels solved within 1,000 generated nodes. Each cell shows two training seeds. For comparison, the right column gives the same models on the Sokoban boards of the paper, with the same limit:

modelsearch success, Boxoban hard levelssearch success, Sokoban boards of the paper
DeLPI0.42 / 0.390.96 / 0.96
CoAt0.12 / 0.150.76 / 0.79
s-CRTR (reg)0.16 / 0.150.78 / 0.79
s-CRTR (class)0.16 / 0.170.79 / 0.81
Sym0.17 / 0.140.79 / 0.77

DeLPI solves 2.3 to 3.4 times as many hard levels as any baseline, in both seeds.

We added these experiments to Appendix C (Extended results).

Future work: longer horizons with subgoals. Since length generalization was our original motivation, we are also experimenting with an extension of DeLPI, which we plan to present in a separate paper because it is orthogonal to the contributions here. The idea is to never ask the distance model about a pair that is longer than what it saw in training by splitting the path to multiple segments. A second network, with the same trunk as DeLPI, predicts an intermediate subgoal on the way from the start to the goal, such as the next state where a lever is pulled. Repeating this splits a long pair into a chain of short segments, and the distance is estimated as the sum of DeLPI's estimates over the segments. Both networks are trained only on pairs up to a fixed distance, and the chain lets the model handle pairs beyond this distance. In a preliminary test on Switchyard 11×11, we trained with a maximal distance of 12 and 18 steps and evaluated on pairs up to 36 steps on held-out maps (three seeds). The table compares the same DeLPI used directly and with a chain of subgoals:

max. dist. 12: MAE / corrmax. dist. 18: MAE / corr
DeLPI (direct)11.74 / 0.418.76 / 0.22
DeLPI + subgoal chain4.05 / 0.682.63 / 0.67

The chain reduces the error on pairs beyond the maximal distance by a factor of about three, and the error on pairs within the maximal distance stays the same (MAE 0.08 and 0.19).

Comments on the paper

1 comment
AuthorsOpenReview ↗

PDF revisions

We have uploaded a revised version of the paper. To make the changes easy to find, new or changed text, tables and figures are coloured by the reviewer whose comment prompted them:

blue: Reviewer H8ob

orange: Reviewer srBZ

green: Reviewer QcA7

Where two reviewers raised the same point, the change is coloured for one of them, and our response to the other reviewer says where to find it. Each response also gives the sections, tables and figures that contain the changes for that reviewer.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.