- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies how to estimate the number of steps needed to get from a start state to a goal state, a quantity used as a heuristic in planning and as a value in goal-conditioned RL. It proposes DeLPI, which feeds the start and goal into one network together, refines an internal representation step by step, and reads the distance as the total amount the representation moved during those steps, instead of predicting a number from the final representation. It also introduces Switchyard, a gridworld where the coupling between mechanisms can be controlled. On Switchyard, Sokoban, Rubik’s cube, and Craftax, DeLPI beats or matches both separate-encoding and joint-encoding baselines, with a readout ablation and an analysis of which cells contribute to the predicted distance.
Critical strengths
-
The path-integral readout is a simple and, to my knowledge, new architectural idea, and the paper provides a readout ablation against a linear readout on the same encoder and recurrent body. It also demonstrates improved performance over both separate-encoding and joint-encoding baselines in estimation accuracy and in downstream planning.
-
The paper includes a fairly complete set of ablations over how the distance is read out, the number of iterations, the encoder, the parameter budget, the label quality and the training objective. This makes it possible to attribute several of the reported gains to specific design choices.
Critical weaknesses
-
The paper should provide more analysis of why path integration works better than decoding. The only justification given is that recurrence and integration “complement each other”, which restates the result rather than explaining it. More analysis is also needed on the assumptions behind the readout and on when it applies, such as whether it relies on the cost being a simple count of unit steps or on the latent vectors being arranged by grid position. The failed slot variant in Appendix D suggests that such conditions exist, and the paper does not state them. In addition, the readout ablation is run only on Switchyard against a weak alternative head, and the recurrent body is never separated from the readout, so the gains over CoAt cannot be attributed to path integration alone.
-
Parameter efficiency is reported, but computational cost is not. A small parameter count does not imply cheap inference, since DeLPI reuses the same block over many recurrent updates, and the search results report nodes generated rather than time. Per-query inference time and total planner time should be reported, especially since the method is intended to be used inside a search loop.
Questions for authors
-
Can the authors provide more analysis of why the path-integral readout works and when it applies, and report inference time per query and total planner time for DeLPI and the baselines, as stated in the weaknesses above?
-
Can the authors isolate the effect of the recurrent body from that of the readout, for example by adding recurrence to CoAt or using untied weights in DeLPI? Can they also repeat the readout ablation on the established benchmarks such as Sokoban, using a stronger alternative readout than mean pooling followed by a linear layer?
5 comments
W2. Computational cost.
Thank you again for pointing this out. The paper reported only the number of parameters, and we agree that this does not measure the cost of a weight-tied model. We now report FLOPs, time per query, planner time and training time in Appendix C.
It is true that the recurrence makes a query more expensive. With the 8 iterations of two blocks used on the gridworlds, a query of DeLPI costs 2.2 to 2.5 times the FLOPs of CoAt, the strongest baseline, and takes about 5 times as long. A training step takes about 4 times as long, but at equal training time DeLPI is still ahead (correlation 0.982 against 0.940 after 0.6 hours on Switchyard 11×11). In planning, the total time also depends on how many nodes the search generates. On Sokoban, where the simulator dominates the cost of a node, DeLPI generates 4.7 times fewer nodes than CoAt and solves a board 3.5 times faster (11.4 against 39.6 s). On Switchyard, with the 8 iterations of the paper, its planner takes 1.7 times as long and solves more pairs.
The number of iterations sets a trade-off between accuracy and computation. Each iteration applies the same two blocks once more, so FLOPs and time per query grow linearly with it, while accuracy saturates. On Switchyard 11×11, with four seeds and time per query on one H100, the numbers are as follows:
| model | MFLOPs per query | ms per query | correlation | MAE | greedy success |
|---|---|---|---|---|---|
| DeLPI, 1 iteration | 145 | 1.2 | 0.933 | 2.56 | 0.33 |
| DeLPI, 2 iterations | 264 | 2.1 | 0.961 | 1.81 | 0.47 |
| DeLPI, 4 iterations | 503 | 4.0 | 0.978 | 1.24 | 0.67 |
| DeLPI, 8 iterations (paper) | 980 | 7.6 | 0.985 | 0.92 | 0.78 |
| CoAt | 436 | 1.4 | 0.940 | 2.24 | 0.38 |
With 2 iterations DeLPI is already more accurate than CoAt with 60% of its FLOPs. Each further doubling lowers the error by about a third, and greedy success more than doubles from 1 to 8 iterations. On the Rubik's cube, DeLPI with its 16 iterations costs 308 MFLOPs per query, less than the 459 of CoAt with 1.83M parameters, which it matches in accuracy. With 2 iterations it needs 43 MFLOPs, about half of CoAt with 520k parameters, and comes within 0.01 of it in correlation.
We add these measurements and the table to Appendix C, and Section 4 now explicitly states next to the parameter count that fewer parameters do not mean less computation.
Answers to questions
Q1. Analysis of the readout, inference time and planner time. Both are answered above: why the readout works and when it applies in W1, and the time per query and the planner time in W2.
Q2. Separating the recurrent body from the readout. We ran both controls you suggest, on Switchyard 11×11 with four seeds. All rows are new runs, so DeLPI and CoAt differ slightly from the paper:
| model | parameters | correlation | greedy success |
|---|---|---|---|
| CoAt | 1.81M | 0.940 | 0.38 |
| CoAt with recurrence (its blocks tied and repeated) | 0.70M | 0.958 | 0.49 |
| DeLPI with untied weights | 3.69M | 0.980 | 0.74 |
| DeLPI | 0.55M | 0.985 | 0.78 |
Recurrence helps CoAt, but leaves it far from DeLPI, and doubling its iterations does not change this (0.961). Untying DeLPI's weights does not help either: with 6.7 times the parameter count it reaches 0.980.
So the recurrence alone does not explain the gain over CoAt. The gain comes from the jointly encoded recurrent network and its readout, and among the readouts we tried the path length is the most accurate and needs no parameters.
We added these controls to Appendix C (Extended results).
Q3. A stronger readout, also on Sokoban. As suggested also by reviewer H8ob, we trained six stronger heads on DeLPI's encoder and recurrent network on Switchyard 11×11, each at its tuned learning rate. They are per-cell MLPs on the latent vector after every update, on every update (the delta between states), or on several intermediate states at once, and an MLP on the latent vectors averaged over the cells. We also trained the per-cell MLP on the latent vector after every update on Sokoban, with four seeds on both benchmarks.
| Switchyard 11×11 | correlation | greedy success | pairs solved with a shortest plan |
|---|---|---|---|
| path length (DeLPI) | 0.985 | 0.78 | 0.90 |
| the six learned heads | 0.976 to 0.984 | 0.69 to 0.75 | 0.85 to 0.89 |
| Sokoban | MAE in steps | greedy success | search success | nodes per board |
|---|---|---|---|---|
| path length (DeLPI) | 0.85 | 0.83 | 0.983 | 262 |
| per-cell MLP on the latent vector after every update | 1.00 | 0.77 | 0.966 | 377 |
No learned head is more accurate than the path length, which adds no parameters. On Switchyard four of the six heads are significantly less accurate (p ≤ 0.027), and the closest comes within 0.001 in correlation with 157k added parameters (p = 0.19). The path length also finds shortest plans more often than every head. On Sokoban it is ahead on every metric (p ≤ 0.004).
We added both tables to Appendix C (Tables 16 and 17), and the Boxoban comparison is in the text above Table 17.
W1
Thank you for reading the paper and suggesting the improvements. The replies are bellow and the changes in the PDF which were prompted by your comments are colored in green.
W1. Why path integration works, when it applies, and separating the recurrent body from the readout.
We agree that "complement each other" restates the result. Below we give the motivation for the readout, the evidence we have, and the limits of what we can claim.
Motivation. A recurrent network unrolls a vector in latent space, so it is natural to read the prediction from the trajectory itself rather than from its final point. Our original motivation was that planning environments decompose into interacting factors (an agent, a key, a door), and that a network could simulate the trajectory of each factor in latent space; the sum of the trajectory lengths would then reflect the total movement needed. In a slot-based version where the factorization is given as input, this holds closely: the per-slot displacement tracks the number of moves of its factor along a shortest plan (correlation 0.880 to 0.998 over three seeds), and decoded trajectories visit the required unlocking positions in 96% to 100% of cases. Our response to Reviewer H8ob explain this in more details ( https://openreview.net/forum?id=n0ISmEOIDF¬eId=4WhkJXWnSo ). When the slots need to discover the factors from images, this version is less accurate than both DeLPI and CoAt (Table 21, Appendix D). The grid-based architecture of the paper keeps the readout and is more accurate, but loses this clean per-factor interpretation. We therefore do not claim to know exactly why the readout works on the grid model, and we report what we can measure.
What we observe.
Separating the recurrent body from the readout. We added two controls on Switchyard 11×11 (Table 15, Appendix C). Making CoAt recurrent raises its correlation from 0.940 to 0.958, still far from DeLPI (0.985), and doubling its iterations does not change this (0.961). Untying the weights of DeLPI achieves 0.980 with 6.7 times the parameters. Together with the readout comparisons above, which keep DeLPI's encoder and recurrent body fixed, this indicates that recurrence alone does not explain the gains over CoAt.
When the readout applies. We did not find a clean condition which would give us an indication for the applicability of the architecture. The readout itself does not require latent vectors arranged by grid position, since it works on slots when the factorization is given, but the slot version that learns its factors from images is less accurate than the grid model. Depth-recurrent architectures are becoming popular and we believe that the integration readout is a natural choice for them and moreover does not require additional parameters. This paper demonstrated that it is competitive on standard benchmarks which are being used for cost-to-go prediction.