- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper asks whether the standard rejection-sampling-plus-SFT loop, run across model generations, improves end-to-end LLM planning in PDDL domains. Starting from a base model (Qwen3 4B Thinking or Gemma 4 E4B), each generation samples one trace per task on a fixed pool of 1000 training tasks per domain (Blocksworld, Rovers, Sokoban), keeps the traces whose final plan passes the VAL validator, aggregates them across generations keeping only the shortest plan per task, and fine-tunes the next generation from the base model with LoRA.
Critical strengths
S1. The controlled design is genuinely clean. Holding the base model, task pools, validator, and tuning recipe fixed while restarting every generation from the base model isolates the training pool as the only changing variable. This is the right way to run a sufficiency study.
Critical weaknesses
W1. The "harder distribution" claim rests on the wrong axis: object counts do not guarantee harder instances, only optimal plan length does.
The paper's central generalization claim (Contribution 2) is that later generations solve held-out tasks from a harder parameter distribution. Table 4 defines hardness through generator parameters: Blocksworld 2-10 blocks in training versus 3-12 in test, Sokoban 1-7 boxes versus 2-10, and similar shifts for Rovers. This construction does not ensure that test instances are harder in the dimension that matters for planning, namely the depth of reasoning required to solve them. Each action in these domains involves at most two objects (a block and its target, a box and a direction), so a plan of length k can causally involve at most about 2k objects. Objects beyond that bound are decoration. A 12-block Blocksworld instance whose goal mentions three blocks may have an optimal plan of six steps, no harder than a small training instance. Huang et al. (ICAPS 2025, cited by the authors) make exactly this point and therefore construct their OOD test sets by plan length directly (training plans of 3-16 steps, test plans of 17-32), verified with a planner. The present paper shifts the parameter axis without measuring the length axis.
Three consequences follow. First, the training pool is compressed toward short plans from both ends: easy tasks are solved and admitted first, and the per-task selection keeps only the shortest plan. Without optimal-length measurements on either side, neither the training distribution nor the test distribution has a horizon yardstick, so "generalization to harder tasks" is currently unverifiable rather than demonstrated. Second, the plan-length evidence in Figures 4 and 5 cannot be interpreted as stated. Plans are satisficing, so longer returned plans on newly solved tasks are consistent with two opposite readings: the newly solved tasks genuinely require deeper search, or the model more often stumbles into long, wandering, but valid plans. A comparison against planner-computed optimal lengths on the same instances would separate these; the paper's own sentence that same-task plan lengths stay constant across generations rules out only a global verbosity shift, not conditional wandering on tasks the base model could not solve. Third, the three domains are not equally exposed. Sokoban box count does scale required horizon almost linearly, since every box must reach a goal cell, while Blocksworld block count need not (the goal sampling procedure is not described). Sokoban, the one domain where the parameter shift most reliably increases horizon, shows by far the smallest absolute gains (7.6 to 43.0 out of 1000 for Qwen), a pattern consistent with the gains being concentrated in the same-horizon regime rather than reflecting length generalization.
The authors already have every tool needed to close this gap: VAL, the task generators, and the PDDL files. What is missing is one classical planner call per instance. We suggest the following revisions, ordered by effort:
- Measure the horizon axis directly. Run an optimal or near-optimal planner on all training and test instances and report the two optimal-length distributions alongside Table 4. Also describe the goal sampling procedure for each domain (for Blocksworld, how many blocks the goal mentions), since this determines whether object counts translate into required depth at all.
- Report per-length-bucket solve rates. For each generation, break test accuracy down by optimal-length bucket, with a separate row for instances whose optimal length exceeds the longest plan in the training pool. This single table would show whether later generations genuinely extend the horizon or only consolidate reliability inside the training length range. The training pool side is already available from Table 3 and the stored traces.
- Separate horizon growth from wandering. Scatter returned plan length against optimal plan length per solved test instance. If returned lengths track optimal lengths, the plan-length shift in Figures 4 and 5 supports real horizon generalization. If they are systematically inflated, the shift reflects longer wandering plans, and the corresponding claims should be revised.
- If the length distributions overlap heavily, downgrade the claim. "Tasks from a harder parameter distribution" should then read as a parameter shift with unverified horizon shift, and the contribution should be reframed as improved reliability on same-horizon tasks, which is still a meaningful and honest result. Stronger still: add a length-controlled test split in the style of Huang et al. (2025), where test instances are selected so that their optimal plans are strictly longer than any training plan, and report whether the loop's gains survive there. This is the version of the experiment that would directly support Contribution 2.
W2. No methodological novelty, and the empirical increment over the closest prior work is narrow.
The loop is the STaR / ReST-EM recipe, which the authors state openly, so the paper's value must come from evidence. Weighed against the closest prior work, the increment is thin.
Questions for authors
Willingness to revise the score. If the authors provide the length-controlled evaluation from W1 (test instances whose optimal plans, computed with a planner, strictly exceed the longest training plan, with per-bucket solve rates per generation) and the gains survive there, I will raise the score substantially
I want to be transparent that the existing numbers point the other way. Sokoban is the one domain where the parameter shift reliably forces longer horizons, and it is exactly where the loop improves least: 7.6 to 43.0 of 1000 for Qwen at generation 4, and 26.3 to 50.3 over ten generations for Gemma. Blocksworld shows the largest gains and is simultaneously the domain where object count least constrains optimal plan length. If the length-controlled split confirms this pattern, the correct framing of the result is reliability consolidation within the training horizon, which remains publishable but with a different title, abstract, and contribution statement.