- Rating
- 3
- Confidence
- 4(+1)
- Informativeness
- 3(+1)
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper mainly present a testbed called PlanWorlds to study how supervision interfaces and post-training strategies shape LLM planning transfer. They made systematic comparison to show that similar in-environment success can mask substantial differences in downstream performance and robustness.
Critical strengths
I find the paper overall is convincing, with supportive experimental details and in-depth explaination.
-
Well-motivated research question. The paper addresses an important question: whether strong performance in synthetic planning environments means transferable planning capabilities.
-
Valuable insights and thoughtful analysis. The paper provides useful insights into planning transfer and carefully interprets the experiment results. The author honestly present both possible explanations and the limitations of the evidence.
-
Detailed diagnostic analysis. The interface controls and trajectory perturbation experiments provide further insights into the observed performance gaps.
Critical weaknesses
-
Limited methodological guidance. Despite the detailed analyses, the paper provides limited actionable guidance on how to improve transferability through post-training. The findings looks more like an evaluation of transfer, but the conclusion remains unclear on what post-training strategies should be preferred and which properties may predict successful transfer.
-
Limited validation of robustness to training choices. The observed differences may depend substantially on the RL reward design, SFT training configuration, CoT construction/quality, in addition to the learning objective itself. The current experiments do not establish whether the main findings remain consistent across reasonable variations in these choices. Ablations would help assess the robustness of the conclusions and identify the factors driving the transfer differences.
-
Limited model coverage. The experiments are restricted to two earlier-generation 7–8B models, leaving the applicability of the findings to newer and larger models unclear. Recent evaluations report improved planning performance for newer models with larger sizes, such as Qwen3.5-27B. These improvements raise the question of whether models with stronger initial planning capabilities exhibit similar transfer patterns. Consequently, conclusions drawn from these earlier-generation smaller models cannot be readily generalized to newer models without further validation.
Questions for authors
-
Can the authors conclude their analyses of the observed phenomena into actionable methodological guidance? Specifically answer the question on when to prefer different post-training strategies and what properties may predict successful transfer?
-
Can the authors evaluate alternative reward designs and different levels of chain-of-thought supervision quality to assess whether the conclusions remain consistent across these choices?
-
Can the authors extend the experiments to newer or larger models, such as Qwen3.5-4B or Qwen3.5-27B? If the evaluation remains restricted to relatively small models, the claims should be narrowed accordingly.
I will raise the score accordingly if the questions were properly answered.
4 comments
Author Response (2/2)
Q2. Reward Design and Chain-of-Thought Supervision
We conducted additional reward ablations and clarified the supervision settings used in PlanWorlds.
Reward Design. We tested two additional configurations on Qwen2.5-7B, using the same Interactive SFT initialization, training seed, and RL budget as the original SFT + RL setting:
- Terminal-only: Retains only the success bonus, removing progress shaping, step costs, and invalid-action penalties.
- Without KL: Retains the original shaped reward but removes KL regularization.
| Metric | Interactive SFT | Shaped + KL | Terminal-only | Without KL |
|---|---|---|---|---|
| In-distribution success (%) | 98 | 100 | 96 | 98 |
| Unseen-theme success (%) | 92 | 96 | 94 | 92 |
| Difficulty generalization (%) | 46 | 54 | 42 | 44 |
| ALFWorld success (%) | 47.8 | 50.7 | 38.1 | 44.0 |
| WebShop score | 61.8 | 61.3 | 59.7 | 60.9 |
| Perturbation-induced success drop (pp) ↓ | 14.4 | 6.9 | 12.5 | 15.0 |
The full setting achieves the highest ALFWorld success and the smallest perturbation-induced drop. Compared with Terminal-only, it improves ALFWorld success by 12.6 percentage points and reduces the perturbation-induced drop by 5.6 points. Removing KL also lowers ALFWorld success and increases the drop. Differences on WebShop are relatively small.
These results suggest that the robustness gains depend partly on reward design and KL regularization, rather than RL alone. We will extend both ablations to two training seeds and report the updated results in the appendix of the revised paper.
SFT Configuration. Full-Plan SFT and Interactive SFT use the same planning problems, expert actions, and optimization settings. They differ in supervision format: complete plans versus stepwise actions conditioned on intermediate observations. Corrective SFT and Self-Imitation SFT also share the supervised optimization settings but differ in how training trajectories are collected and selected. We did not sweep SFT hyperparameters, and the continued-training methods have different data-collection procedures and training budgets.
Chain-of-Thought Supervision. PlanWorlds does not use chain-of-thought supervision. Training targets contain executable actions rather than intermediate reasoning traces, so CoT quality is not a variable in our experiments. We will clarify this in the revision and identify CoT-augmented supervision as a direction for future work.
Q3. Generalization to Newer and Larger Models
Our experiments focus on Qwen2.5-7B and Llama3.1-8B to enable controlled comparisons across training strategies and evaluation settings. We agree that evaluating newer and larger models would help establish how broadly our findings apply. We will therefore clarify the scope of our claims and explicitly acknowledge this limitation in the revised paper. At the same time, PlanWorlds supports adjustable task difficulty, providing a foundation for future studies of planning transfer across models with different capabilities and scales.
Thank you again for your valuable feedback. We hope our responses address your concerns and would be happy to discuss any remaining questions.
Thank you for the clarifications and additional experiments. These substantially address my experimental concerns. However, two main reservations remain about the overall scope of the paper:
-
The main conclusions offer limited new insight beyond expected effects of distribution and interface mismatch. Matching supervision to the execution interface and observing that in-environment success does not guarantee downstream transfer are not, by themselves, substantive advances in understanding planning transfer. The central limitation is that the analysis remains largely descriptive: it documents performance differences without establishing explanatory principles that predict when or why a post-training strategy will transfer successfully. For a paper whose primary contribution is understanding these phenomena, this substantially limits its conceptual contribution.
-
Some claims remain broader than the evidence supports. For example, “what language-model agents actually acquire” suggests identifying learned capabilities, whereas the experiments primarily characterize differences in execution, transfer, and robustness. I suggest narrowing this wording.
Overall, the additional evidence strengthens my confidence in the findings, but does not substantially change my assessment of the contribution. I still believe the paper’s strengths outweigh its weaknesses and maintain my original rating.
Thank you for your constructive feedback and positive assessment of our work. We are glad that the additional experiments have addressed your experimental concerns.
We share your interest in understanding when and why planning transfer succeeds. Our study contributes toward this goal by using PDDL’s explicit execution rules and verifiable solutions to support controlled comparisons. These comparisons show how transfer gaps respond differently to interface controls, how further training can change transfer, and how recovery can differ among policies with nearly identical clean performance. We hope this supports a more fine-grained evaluation of planning transfer and gives future work more specific questions to investigate about when and why transfer succeeds.
We also agree that our claims should closely reflect the evidence. We have revised the abstract to focus on execution, transfer, and robustness, replacing the broader wording about what agents “actually acquire.”
Author Response (1/2)
Thank you for your constructive review. We appreciate the opportunity to clarify our findings and strengthen our analysis. Below, we address your questions point by point.
Our results offer several insights for post-training strategy selection and transfer evaluation.
Guidance for Post-Training
Match supervision to the target execution setting. Full-Plan SFT achieves high plan-generation success but performs poorly in stepwise execution, despite using the same tasks and expert actions as Interactive SFT. This suggests that full-plan supervision is useful for generating complete plans, while interactive supervision is more appropriate for agents that must act on changing observations.
Choose continued-training strategies based on the target capability. Even when source-task performance is nearly saturated, further training can have different effects. On Qwen, Corrective SFT reduces ALFWorld performance, while Self-Imitation SFT largely preserves it. SFT + RL, in contrast, improves recovery from controlled perturbations despite similar performance on unperturbed tasks.
Guidance for Evaluation
Use interface controls to interpret transfer gaps. On ACPBench, answer re-elicitation changes response correctness, whereas on ALFWorld, substantial gaps persist under constrained decoding. Reporting both original and interface-controlled results helps distinguish protocol-sensitive performance from differences that remain under the tested controls.
Align capability claims with the evaluation measure. Source-task success measures competence in the training environment, perturbation tests measure local recovery, and external benchmarks measure downstream transfer. These measures can yield different rankings of training methods. Improved recovery should therefore be reported as a distinct benefit rather than taken as evidence of broader transfer.