The paper has severe methodological and experimental flaws. While I find the
contributions conceptually interesting and worth pursuing, I cannot properly
assess the practical relevance because of these experimental flaws. Therefore, I
am negative about this paper.
Overall, I think your paper would benefit tremendously if you applied the advice
listed in [1]. A few immediate issues worth pointing out:
-
The paper compares methods that were evaluated on vastly different hardware
and limits. While I understand that this will happen at some level (e.g., you
need a GPU for some methods), I think there must be a larger effort to run all
the methods in similar limits. In your main results, we have methods running
with 15m and methods running with 1h time limit.
-
The baselines used are not the best baselines for this test set from the
competition itself. In fact, LAMA was only 4th in performance for coverage
(the one you focused on) in the IPC learning track. If you want to keep a
non-learning baseline, then please run Fast Downward Stone Soup or one of the
winners of the last classical planning IPC.
One more relevant issue is regarding the properties and novelty of LIW. First,
LIW is very similar "in spirit" to famous algorithms like BFWS(f5) or BFWS(R),
see [2,3]. The core idea is to use "relevant" atoms to guide width-search
algorithms in a more "goal-oriented" fashion. LIW does use it in a different
way, of course, but using a set of relevant atoms to make width-search more
"goal-oriented" is well known.
The more relevant issue is that LIW and IW are incomplete search algorithms!
So saying that "it is polynomial" is misleading when stated without the
incompleteness caveat. You do hint at that in the appendix, but this is crucial
to the method. More importantly, other existing methods (like BFWS) make the
search complete by ranking the nodes instead of pruning. This matters because
LIW(1) failures are considered as subgoal failures in training, if I understand
correctly (Appendix C4).
The landmark extraction part is also a bit fuzzy to me. From what is reported in
section 5, the median time for hard Sokoban tasks to extract the landmarks was
about half of total time limit. For some other domains (such as Rovers) the
time is still non-negligible. Not taking into account this time into the total
time is very arbitrary, and it may impact the results (although in Sokoban the
performance is already 0 on hard tasks).
Last, but not least, I strongly recommend the authors to re-read the paper and
try to smooth the text. In its current version, it has too many sharp turns and
it assumes a deeper knowledge about symbolic planning than the average ICLR
reader would have. For example, the introduction drops terms which are known
only to planning folks (e.g., it mentions "childsnack" in the third paragraph as
if this was a worldwide famous domain), and in the background it assumes that a
reader knows what an action schema is, or what the difference is between a
ground action and a schema.
[1] Michael Katz, Harsha Kokel, Christian Muise, Shirin Sohrabi, Sarath Sreedharan.
"Make Planning Research Rigorous Again!".
arXiv:2505.21674. 2025.
[2] Nir Lipovetzky and Hector Geffner.
"Best-First Width Search: Exploration and Exploitation in Classical Planning."
AAAI 2017.
[3] Guillem Francès, Miquel Ramírez, Nir Lipovetzky, and Hector Geffner.
"Purely Declarative Action Descriptions are Overrated: Classical Planning with Simulators."
IJCAI 2017.
Thank you for the positive assessment and questions. Since the submission we fixed a bug in the manager's training and retrained Landmarker (please see the general response). It now solves many more instances than every baseline.
Extraction time: Every Landmarker result now counts landmark extraction toward the hour. Coverage is not affected by the landmark extraction time: all 751 instances are solved within the hour including extraction. The extraction takes the longest in sokoban instances which our model could not solve regardless.
Sokoban: It is correct that the manager is trained with LIW(1) and never sees the learned worker, and hence the manager cannot learn jointly with the test-time worker. The paper lists this as a limitation. In sokoban, however, the results point to the worker learning as the issue:
In sokoban the worker therefore does not learn to execute its subgoals well enough. A manager adapted to this worker could avoid the subgoals it fails, which we have not tested, but it could not make the worker better at them.
Please note that not using LIW(1) as proxy worker during training carries two dangers. First, similar to a complete-search worker (see our reply to Reviewer Lwbq), the manager may come to rely on a worker that achieves more and more complex subgoals, and so overfit its decomposition to subgoals that do not generalize. Second, a worker that initially reaches no subgoal can prevent learning altogether. Both is avoided by the proxy LIW(1).
In the other domains, the step from search to the learned worker is what makes the large instances solvable: over all domains the learned worker solves 218 hard instances and LIW(1) 35. Childsnack and floortile are the clearest cases. Both have dead ends, so the manager has to order its subgoals to avoid them and the worker has to carry them out without taking an action back. Landmarker solves all 90 childsnack instances and 82 in floortile. No baseline solves a single medium or hard floortile instance.
Camera-ready. The revision reports the retrained model in its abstract, main results, ablations and subgoal analysis. Three analyses need new training runs that do not fit into the rebuttal period. The camera-ready version will contain them and every further update needed for the whole paper to reflect the retrained model.