- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces EmbedPlan, a lightweight transition model for text-described planning states. A frozen language encoder represents the current state and action, learned projection heads map them to a 128-dimensional latent space, and a small learned transition network predicts the embedding of the successor state. The predicted vector is then mapped back to an actual state using nearest-neighbor retrieval.
The motivation is to replace expensive autoregressive LLM calls during planning search with a cheap latent transition operation.
Critical strengths
The central idea—whether an expensive pretrained language representation can be used once and then support a cheap learned dynamics model is interesting. The separation between a frozen semantic encoder and a lightweight transition function is conceptually clean and potentially useful for systems that repeatedly plan in the same environment.
The comparison against character n-grams, bag-of-words, bag-of-literals, offsets, random representations, and lifted STRIPS is useful. The result that very simple sparse text features outperform the 70B LLM representation on held-out problems is striking and potentially an important diagnostic observation.
The fact-order experiment is also informative: a semantically meaningless reordering can move the BGE embedding more than an actual action transition does. This provides concrete evidence rather than merely asserting that pretrained embeddings have the "wrong geometry."
Critical weaknesses
A concern I have is how that pool is obtained for unseen problems.
In the Extrapolation protocol, each query is evaluated against the true successor plus distractors drawn from the same held-out problem. Thus the model is effectively asked to identify the successor among states already available from the test problem, rather than to construct a successor whose representation may not previously exist.
This distinction seems important for the intended use as a cheap transition model. For a genuinely new planning problem, it is unclear how the system obtains an enumerable bank of states that both covers the problem and already contains the correct future state. The issue is therefore less candidate size than candidate coverage: the reported Hit@k evaluates ranking conditional on the ground-truth successor being present, but does not measure the probability that such a candidate would be available in the first place.
Questions for authors
In an unseen problem, where does the candidate state bank come from? In particular, can EmbedPlan predict transitions when the correct successor has not previously been enumerated, or is the intended setting one in which the reachable state set is already available?
4 comments
Comment 2/2: Scientific contribution, manuscript revisions, and reassessment
Does the candidate-coverage limitation undermine the transition-learning results?
The limitation affects practical applicability, but does not invalidate the controlled transition-ranking experiments or the scientific findings they support.
As you highlighted in your strengths, the comparison with simple sparse representations and the fact-order experiment provide important diagnostic evidence. These findings do not depend on how candidates would be generated in a deployed system.
In Table 7, we compare representations using the same transition head, training data, and candidate protocol. On unseen problems across Ferry, Logistics, and Goldminer, character n-grams reach 79.4% Hit@5, compared with 56.8% for Llama-3.3-70B embeddings. Because the candidate pool is identical, coverage cannot explain this difference.
The fact-order experiment provides an explanation for the representation gap: reordering semantically identical state facts moves BGE-M3 embeddings 1.13x further than applying an actual action (Appendix N). This identifies a concrete limitation of frozen text representations rather than merely observing lower accuracy.
Appendix G.2 further shows that learning action-conditioned dynamics matters. Removing the action-disambiguation loss reduces mean Hit@5 on unseen problems from 54.6% to 35.3%, and action Acc@5 from 16.2% to 4.8%.
These are controlled findings about learned transitions and representations. Candidate coverage limits their direct application to a complete planner, but does not undermine the comparisons or their conclusions.
We believe this distinction matters for assessing Soundness. A missing operational candidate generator is a legitimate limitation of applicability, but not evidence that the controlled transition-learning experiments are methodologically invalid.
What have we changed in the revised manuscript?
We have already incorporated the clarifications, with all changes highlighted in blue.
- Section 3, p.4: Added an explicit sentence stating that retrieval cannot return states outside the candidate pool.
- Section 3, p.5: Added "Candidate-pool construction and coverage", clarifying the evaluation pools, conditional ranking, generator coverage, and absence of a combined proposer-and-ranker evaluation.
- Section 3, p.5: Included the full-pool Hit@1 results on unseen problems, 3.8% for frozen BGE-M3 and 9.3% after LoRA adaptation, averaged over three domains. These results were already reported in Section 4.3 and Appendix M.
- Table 1: Added explanations of what is new at test time for each exposure level and clarified two protocol descriptions to improve presentation.
- Limitations: Made the candidate-availability assumption more explicit.
These revisions directly address the coverage and presentation concerns without changing the experimental results or expanding our claims.
Our request
We agree with the practical limitation you identified, and have addressed its presentation in the revised manuscript. We believe the scientific contributions you highlighted remain valid and informative within their stated scope.
Does the revised Section 3 resolve your central concern about the interpretation and validity of the evaluation?
If so, we would appreciate a reassessment of the paper, particularly its Soundness, Presentation, and overall recommendation, in light of the revisions and the scientific findings you recognized.
If a methodological issue remains beyond the acknowledged candidate-coverage limitation, we would appreciate knowing what it is so that we can address it during the discussion.
i raised my score to 3 given the rebuttal has addressed part of my concern
It's great to see authors have identified the problem and make some revisions carefully, however, the limitation might not be easily addressed given the revised content.
Thank you for reconsidering your assessment and for engaging constructively with our rebuttal. We genuinely appreciate it!
We are glad the revisions helped clarify the scope of our evaluation and the scientific contribution of our controlled experiments. We believe these findings offer valuable insights into learning transitions over frozen language representations, including where current representations can be improved.
We appreciate your thoughtful consideration and the opportunity for meaningful scientific discussion.
Comment 1 of 2: Candidate generation and coverage
Thank you for recognizing the clean separation between the frozen semantic encoder and the lightweight transition model, and for highlighting the sparse-feature comparison and fact-order experiment as important diagnostic findings. We appreciate that you see the potential value of this approach for systems that repeatedly plan in the same environment.
Your question about candidate coverage is important because it determines where these findings can be applied. We agree with this concern and have already revised the manuscript to clarify the distinction (Section 3).
Q1/W1. Where do candidate successors come from, and does Hit@k measure coverage?
Hit@k measures how well EmbedPlan ranks the correct successor, conditional on that successor being in the candidate pool. It does not measure candidate generation or coverage.
The original submission stated this assumption in Section 3, the limitations, and Appendix B.4. We agree that it deserved more prominence, and have now added a dedicated paragraph, "Candidate-pool construction and coverage" (p.5).
The distinction depends on the evaluation setting.
1. Observed problems: the pool contains states collected offline in the benchmark domain.
In Interpolation, training and test transitions come from the same planning problems. The standard evaluation includes the true successor and 127 distractors.
We also evaluate ranking against every state observed in the benchmark domain (Table 2), rather than only 128 candidates: 46,205 states on Ferry and 13,373 on Logistics. The true successor is included by construction, with no distractor sampling in this full-pool setting.
The results show both the capability and its limitation. Full-pool Hit@5 is 86.9% on Ferry and 74.9% on Logistics, while Hit@1 falls to 35.2% and 32.6%. We report this decrease explicitly.
The states were collected offline for the benchmark. We do not claim that an equivalent pool would automatically be available in deployment.
2. Unseen problems: the evaluation pool is constructed from the held-out problem, not generated by EmbedPlan.
In Extrapolation, the model is trained on other problems from the same domain. Each evaluation query includes the true successor and 127 distractors drawn from the held-out problem (Table 1).
These distractors are challenging because they share the problem's objects and often differ from the target by only a few facts (Appendix B.4).
The resulting 54.6% mean Hit@5 across nine domains (Table 5) measures generalization of transition ranking to new problem configurations. It does not establish the coverage of an independently generated candidate pool.
3. Can EmbedPlan return a successor absent from the pool?
No, not as a textual state under the evaluated retrieval formulation.
The network predicts a next-state embedding, but retrieving a discrete textual successor requires an available candidate. An external proposer or a collection of previously visited states could supply candidates, but neither integration has been evaluated here.
We also test whether intermediate retrieval can be avoided. On observed Ferry problems, step-wise Hit@1 falls from 94.2% with per-step snapping to 68.3% without it, using a 1,000-state evaluation pool (Table 4). This shows that latent-only rollout is not currently a complete alternative.
4. What do we know about the coverage of a candidate generator?
Table 3 reports that GPT-5.4 includes the true successor among its top five generated states for 56% of Ferry queries and 100% of Logistics queries. These numbers measure the generator alone, not a combined generator-and-EmbedPlan system.
We now report this distinction explicitly in Section 3 (p.5), and state that such a combined pipeline has not been evaluated.
The boundary is therefore clear: EmbedPlan learns to rank successor states, but does not solve the separate problem of making the correct successor available for retrieval. This limitation is now explicit in the revised manuscript.