acceptodds
Withdrawn from ICLR 2027

Iterative Deployment Improves Planning Skills in LLMs

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Frontier LLMs have become much better at end-to-end planning across recent model generations, but their training pipelines are not public, so the exact cause of this progress is unknown. Earlier attempts to improve planning skills of LLMs relied on planning-specific supervision, such as optimal plans from a classical planner, carefully shaped rewards, or structured validator feedback. We test whether a much weaker signal suffices: the model's own attempts, filtered by a pass/fail check of each final plan. Starting from Qwen3 4B Thinking or Gemma 4 E4B, we repeatedly prompt the model on PDDL tasks, check each final plan with a symbolic validator, keep the traces whose plans pass, and fine-tune the next generation on the pool of successful traces. Across three different domains, later generations solve substantially more held-out test tasks, including tasks from a harder parameter distribution, and find valid plans with longer horizons. Over ten generations, performance plateaus without model collapse. Our study also shows that curation is essential: training on all generated traces performs much worse despite using more data. Because we hold the base model, tasks, validator, and tuning recipe fixed, these results show that this minimal mechanism is sufficient to produce large planning gains. It is therefore a candidate explanation for part of the progress of frontier models, although our experiments cannot show how those models were trained.

Primary area:
neurosymbolic & hybrid AI systems (physics-informed, logic & formal reasoning, etc.)
Keywords:
planning, neurosymbolic AI

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_1BqvOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper asks whether the standard rejection-sampling-plus-SFT loop, run across model generations, improves end-to-end LLM planning in PDDL domains. Starting from a base model (Qwen3 4B Thinking or Gemma 4 E4B), each generation samples one trace per task on a fixed pool of 1000 training tasks per domain (Blocksworld, Rovers, Sokoban), keeps the traces whose final plan passes the VAL validator, aggregates them across generations keeping only the shortest plan per task, and fine-tunes the next generation from the base model with LoRA.

Critical strengths

S1. The controlled design is genuinely clean. Holding the base model, task pools, validator, and tuning recipe fixed while restarting every generation from the base model isolates the training pool as the only changing variable. This is the right way to run a sufficiency study.

Critical weaknesses

W1. The "harder distribution" claim rests on the wrong axis: object counts do not guarantee harder instances, only optimal plan length does.

The paper's central generalization claim (Contribution 2) is that later generations solve held-out tasks from a harder parameter distribution. Table 4 defines hardness through generator parameters: Blocksworld 2-10 blocks in training versus 3-12 in test, Sokoban 1-7 boxes versus 2-10, and similar shifts for Rovers. This construction does not ensure that test instances are harder in the dimension that matters for planning, namely the depth of reasoning required to solve them. Each action in these domains involves at most two objects (a block and its target, a box and a direction), so a plan of length k can causally involve at most about 2k objects. Objects beyond that bound are decoration. A 12-block Blocksworld instance whose goal mentions three blocks may have an optimal plan of six steps, no harder than a small training instance. Huang et al. (ICAPS 2025, cited by the authors) make exactly this point and therefore construct their OOD test sets by plan length directly (training plans of 3-16 steps, test plans of 17-32), verified with a planner. The present paper shifts the parameter axis without measuring the length axis.

Three consequences follow. First, the training pool is compressed toward short plans from both ends: easy tasks are solved and admitted first, and the per-task selection keeps only the shortest plan. Without optimal-length measurements on either side, neither the training distribution nor the test distribution has a horizon yardstick, so "generalization to harder tasks" is currently unverifiable rather than demonstrated. Second, the plan-length evidence in Figures 4 and 5 cannot be interpreted as stated. Plans are satisficing, so longer returned plans on newly solved tasks are consistent with two opposite readings: the newly solved tasks genuinely require deeper search, or the model more often stumbles into long, wandering, but valid plans. A comparison against planner-computed optimal lengths on the same instances would separate these; the paper's own sentence that same-task plan lengths stay constant across generations rules out only a global verbosity shift, not conditional wandering on tasks the base model could not solve. Third, the three domains are not equally exposed. Sokoban box count does scale required horizon almost linearly, since every box must reach a goal cell, while Blocksworld block count need not (the goal sampling procedure is not described). Sokoban, the one domain where the parameter shift most reliably increases horizon, shows by far the smallest absolute gains (7.6 to 43.0 out of 1000 for Qwen), a pattern consistent with the gains being concentrated in the same-horizon regime rather than reflecting length generalization.

The authors already have every tool needed to close this gap: VAL, the task generators, and the PDDL files. What is missing is one classical planner call per instance. We suggest the following revisions, ordered by effort:

  1. Measure the horizon axis directly. Run an optimal or near-optimal planner on all training and test instances and report the two optimal-length distributions alongside Table 4. Also describe the goal sampling procedure for each domain (for Blocksworld, how many blocks the goal mentions), since this determines whether object counts translate into required depth at all.
  2. Report per-length-bucket solve rates. For each generation, break test accuracy down by optimal-length bucket, with a separate row for instances whose optimal length exceeds the longest plan in the training pool. This single table would show whether later generations genuinely extend the horizon or only consolidate reliability inside the training length range. The training pool side is already available from Table 3 and the stored traces.
  3. Separate horizon growth from wandering. Scatter returned plan length against optimal plan length per solved test instance. If returned lengths track optimal lengths, the plan-length shift in Figures 4 and 5 supports real horizon generalization. If they are systematically inflated, the shift reflects longer wandering plans, and the corresponding claims should be revised.
  4. If the length distributions overlap heavily, downgrade the claim. "Tasks from a harder parameter distribution" should then read as a parameter shift with unverified horizon shift, and the contribution should be reframed as improved reliability on same-horizon tasks, which is still a meaningful and honest result. Stronger still: add a length-controlled test split in the style of Huang et al. (2025), where test instances are selected so that their optimal plans are strictly longer than any training plan, and report whether the loop's gains survive there. This is the version of the experiment that would directly support Contribution 2.

W2. No methodological novelty, and the empirical increment over the closest prior work is narrow.

The loop is the STaR / ReST-EM recipe, which the authors state openly, so the paper's value must come from evidence. Weighed against the closest prior work, the increment is thin.

Questions for authors

Willingness to revise the score. If the authors provide the length-controlled evaluation from W1 (test instances whose optimal plans, computed with a planner, strictly exceed the longest training plan, with per-bucket solve rates per generation) and the gains survive there, I will raise the score substantially

I want to be transparent that the existing numbers point the other way. Sokoban is the one domain where the parameter shift reliably forces longer horizons, and it is exactly where the loop improves least: 7.6 to 43.0 of 1000 for Qwen at generation 4, and 26.3 to 50.3 over ten generations for Gemma. Blocksworld shows the largest gains and is simultaneously the domain where object count least constrains optimal plan length. If the length-controlled split confirms this pattern, the correct framing of the result is reliability consolidation within the training horizon, which remains publishable but with a different title, abstract, and contribution statement.

Reviewer_ddVNOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies whether iterative deployment can improve LLM planning on PDDL tasks. The proposed method repeatedly samples plans from the model, uses a symbolic validator to keep only successful traces, and fine-tunes the next model on the growing pool of validated traces. The authors report that this loop improves planning performance on several benchmarks and argue that it may help explain part of the recent progress in LLM planning.

Critical strengths

The paper tests the same generate-validate-finetune loop on two models and three PDDL domains, and includes an ablation showing that filtering by a symbolic validator is important for improving planning abilities, though the method or analysis is not novel in my opinion.

Critical weaknesses
  1. Contribution is too weak.

Repeatedly applying SFT on successful traces and then using the updated model to expand the data pool can not be considered as a novel method. The authors themselves acknowledge prior work such as Self-Taught Reasoner and ReST^EM, and the paper does not show a clear difference from these prior methods.

  1. The theoretical section is not a substantive contribution.

The connection between SFT on successful samples and REINFORCE with binary rewards is fairly standard, and the paper does not use this connection to design a new algorithm or explain a nontrivial property of the method. The off-policy discussion is also only interpretive, since the actual training does not estimate or apply importance weights. Therefore, this section does not materially strengthen the paper’s claim about improving LLM planning.

  1. The paper lacks experimental baselines for comparison.

Since the authors already have a reliable verifier for evaluating plans, the same verifier could be used to provide a verifiable reward for a basic RL training baseline. This would allow a direct comparison between the proposed method and standard RL-based training. And the curation ablation is only reported for Qwen on Blocksworld, and other design choices such as shortest-plan selection, self selection are not ablated.

Questions for authors

Q1: Could you establish some strong baselines for the experiments to better evaluate the effectiveness of the proposed method? Please refer to W3 for specific selection criteria.

Q2: What specific capabilities does the proposed data construction method actually endow the model with? In what aspects does it enhance the model's planning ability? It is possible that the observed performance gain simply stems from the model being better aligned with the output format, rather than a genuine improvement in capability, given that the length of the plan itself does not necessarily explain such gains. Furthermore, why is the improvement on Sokoban significantly less obvious for both Qwen3 and Gemma4?

Reviewer_TURtOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies whether an established self-training loop can improve end-to-end classical planning. At each round, the model generates traces on a fixed pool of PDDL tasks, and VAL checks the final plans. Successful traces are accumulated across generations. For each task, the method selects the trace with the shortest plan and breaks ties by fewer reasoning tokens. Each generation is fine-tuned from the original base model. Experiments with two model families across three domains show improvements on held-out tasks. The paper also examines returned plan lengths, the effect of curation, and performance over ten generations.

Critical strengths
  1. The experimental setting is straightforward. The base checkpoint and training task pool are fixed, and VAL provides a clear criterion for whether the final plan succeeds. This makes it possible to test the full procedure without planner-generated training solutions or detailed error feedback.
  2. The main gains are clear and occur across both model families and all three domains. In Table 5, Qwen's average number of solved Rovers tasks increases from 33.6 to 161.6, while Gemma's Blocksworld result increases from 113.0 to 204.6, out of 1,000 test tasks per domain. These tasks are excluded from fine-tuning and trace selection, so the results support improvement beyond the training instances.
  3. The ten-generation experiments with Qwen in Blocksworld and Gemma in Rovers and Sokoban provide useful information about performance after the initial gains. They show plateaus and fluctuations, with performance remaining above the base model in each case. The paper reports these limits rather than suggesting that improvement continues steadily.
Critical weaknesses
  1. The methodological novelty is limited. The paper describes its procedure as “an established generate, validate, and fine-tune loop” and acknowledges in Section 2.1 that it follows the same recipe as ReSTᴱᴹ. The results provide evidence that this procedure works for PDDL planning, but the contribution is mainly an empirical extension, with limited additional insight into the existing self-training method.
  2. The effects of aggregating successful traces across generations and selecting the trace with the shortest plan for each task are not separately evaluated. The authors explicitly state, “We do not ablate aggregation separately,” and provide no comparison with using only successful traces from the current generation. The selection mechanism keeps the trace with the shortest plan and breaks ties by fewer reasoning tokens, but this rule is not compared with randomly selecting one successful trace per task. It remains unclear how much these two design choices contribute to the reported gains.
  3. The results on held-out tasks from a harder parameter distribution need a more detailed breakdown. In Table 4, Blocksworld training instances contain 2–10 blocks, while test instances contain 3–12. The authors acknowledge that the ranges overlap, but Table 5 reports the average number of solved tasks for the whole test pool. It remains unclear whether later generations solve more tasks beyond the largest training instances. Reporting the number of test instances and the average number of solved tasks separately for instances with 11 or 12 blocks would clarify this point.
  4. The interpretation of the longer returned plans needs more evidence. The introduction argues that the gains are “unlikely to come only from better formatting or better following of instructions.” Appendix A.2 states that malformed outputs, or outputs whose final plan cannot be extracted in the expected format, are counted as invalid, but the paper does not report how often these cases occur across generations, or how often traces hit the maximum token limit. Reporting these numbers would provide more direct evidence for assessing this interpretation.
Questions for authors
  1. What new understanding of end-to-end classical planning do these experiments provide beyond showing that an established generate, validate, and fine-tune loop improves performance? How do the findings extend those of ReSTᴱᴹ on mathematics and code?
  2. On one model and domain, could you compare aggregating successful traces across generations with using only successful traces from the current generation? For the selection mechanism, could you compare choosing the shortest plan with randomly selecting one successful trace per task from the same candidate pool?
  3. How many Blocksworld test instances contain 11 or 12 blocks, and how many are solved at each generation? Could you report these results separately from the test instances with 3–10 blocks to show whether the gains extend beyond the largest training instances?
  4. Could you report how often each generation produces malformed outputs, outputs without an extractable final plan, or traces that hit the maximum token limit? How do these numbers change alongside the number of solved tasks? This would help assess the explanation that the gains are “unlikely to come only from better formatting or better following of instructions.”

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.