acceptodds
Under review as a conference paper at ICLR 2027

What Transfers from Synthetic Planning? A Controlled Study of LLM Post-Training

Abstract

Synthetic planning offers verifiable supervision for language-model agents, yet strong task performance leaves open what these agents learn and transfer. We introduce PlanWorlds, a controlled testbed that connects executable symbolic worlds with natural-language interaction, enabling systematic study of supervision interfaces, training trajectories, and learning objectives. Across two model families and six post-training strategies, we find that planning post-training produces distinct execution and transfer profiles. Holding expert solutions fixed, full-plan and interactive supervision lead to sharply different execution behaviors. Moreover, policies that achieve similarly strong, often near-saturated performance within PlanWorlds can diverge substantially on downstream planning tasks, and further post-training from the same SFT checkpoint can reshape transfer without changing training-environment success. We further show that these transfer gaps have different interpretations. Some largely contract under target-interface controls, while others persist and are accompanied by differences in robustness to local trajectory deviations despite nearly identical clean performance. These findings motivate evaluating planning transfer through complementary views of task success, interface compatibility, and sequential robustness, and enable controlled investigation of what language-model agents actually acquire from synthetic planning post-training.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM, Planning, Post training

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_ygmHOpenReview ↗
Rating
3
Confidence
4(+1)
Informativeness
3(+1)
Presentation
3
Soundness
2
Read review
Paper summary

This paper mainly present a testbed called PlanWorlds to study how supervision interfaces and post-training strategies shape LLM planning transfer. They made systematic comparison to show that similar in-environment success can mask substantial differences in downstream performance and robustness.

Critical strengths

I find the paper overall is convincing, with supportive experimental details and in-depth explaination.

  1. Well-motivated research question. The paper addresses an important question: whether strong performance in synthetic planning environments means transferable planning capabilities.

  2. Valuable insights and thoughtful analysis. The paper provides useful insights into planning transfer and carefully interprets the experiment results. The author honestly present both possible explanations and the limitations of the evidence.

  3. Detailed diagnostic analysis. The interface controls and trajectory perturbation experiments provide further insights into the observed performance gaps.

Critical weaknesses
  1. Limited methodological guidance. Despite the detailed analyses, the paper provides limited actionable guidance on how to improve transferability through post-training. The findings looks more like an evaluation of transfer, but the conclusion remains unclear on what post-training strategies should be preferred and which properties may predict successful transfer.

  2. Limited validation of robustness to training choices. The observed differences may depend substantially on the RL reward design, SFT training configuration, CoT construction/quality, in addition to the learning objective itself. The current experiments do not establish whether the main findings remain consistent across reasonable variations in these choices. Ablations would help assess the robustness of the conclusions and identify the factors driving the transfer differences.

  3. Limited model coverage. The experiments are restricted to two earlier-generation 7–8B models, leaving the applicability of the findings to newer and larger models unclear. Recent evaluations report improved planning performance for newer models with larger sizes, such as Qwen3.5-27B. These improvements raise the question of whether models with stronger initial planning capabilities exhibit similar transfer patterns. Consequently, conclusions drawn from these earlier-generation smaller models cannot be readily generalized to newer models without further validation.

Questions for authors
  1. Can the authors conclude their analyses of the observed phenomena into actionable methodological guidance? Specifically answer the question on when to prefer different post-training strategies and what properties may predict successful transfer?

  2. Can the authors evaluate alternative reward designs and different levels of chain-of-thought supervision quality to assess whether the conclusions remain consistent across these choices?

  3. Can the authors extend the experiments to newer or larger models, such as Qwen3.5-4B or Qwen3.5-27B? If the evaluation remains restricted to relatively small models, the claims should be narrowed accordingly.

I will raise the score accordingly if the questions were properly answered.

4 comments
AuthorsOpenReview ↗

Author Response (1/2)

Thank you for your constructive review. We appreciate the opportunity to clarify our findings and strengthen our analysis. Below, we address your questions point by point.

Q1. Actionable Methodological Guidance

Our results offer several insights for post-training strategy selection and transfer evaluation.

Guidance for Post-Training

  • Match supervision to the target execution setting. Full-Plan SFT achieves high plan-generation success but performs poorly in stepwise execution, despite using the same tasks and expert actions as Interactive SFT. This suggests that full-plan supervision is useful for generating complete plans, while interactive supervision is more appropriate for agents that must act on changing observations.

  • Choose continued-training strategies based on the target capability. Even when source-task performance is nearly saturated, further training can have different effects. On Qwen, Corrective SFT reduces ALFWorld performance, while Self-Imitation SFT largely preserves it. SFT + RL, in contrast, improves recovery from controlled perturbations despite similar performance on unperturbed tasks.

Guidance for Evaluation

  • Use interface controls to interpret transfer gaps. On ACPBench, answer re-elicitation changes response correctness, whereas on ALFWorld, substantial gaps persist under constrained decoding. Reporting both original and interface-controlled results helps distinguish protocol-sensitive performance from differences that remain under the tested controls.

  • Align capability claims with the evaluation measure. Source-task success measures competence in the training environment, perturbation tests measure local recovery, and external benchmarks measure downstream transfer. These measures can yield different rankings of training methods. Improved recovery should therefore be reported as a distinct benefit rather than taken as evidence of broader transfer.

AuthorsOpenReview ↗

Author Response (2/2)

Q2. Reward Design and Chain-of-Thought Supervision

We conducted additional reward ablations and clarified the supervision settings used in PlanWorlds.

Reward Design. We tested two additional configurations on Qwen2.5-7B, using the same Interactive SFT initialization, training seed, and RL budget as the original SFT + RL setting:

  • Terminal-only: Retains only the success bonus, removing progress shaping, step costs, and invalid-action penalties.
  • Without KL: Retains the original shaped reward but removes KL regularization.
MetricInteractive SFTShaped + KLTerminal-onlyWithout KL
In-distribution success (%)981009698
Unseen-theme success (%)92969492
Difficulty generalization (%)46544244
ALFWorld success (%)47.850.738.144.0
WebShop score61.861.359.760.9
Perturbation-induced success drop (pp) ↓14.46.912.515.0

The full setting achieves the highest ALFWorld success and the smallest perturbation-induced drop. Compared with Terminal-only, it improves ALFWorld success by 12.6 percentage points and reduces the perturbation-induced drop by 5.6 points. Removing KL also lowers ALFWorld success and increases the drop. Differences on WebShop are relatively small.

These results suggest that the robustness gains depend partly on reward design and KL regularization, rather than RL alone. We will extend both ablations to two training seeds and report the updated results in the appendix of the revised paper.

SFT Configuration. Full-Plan SFT and Interactive SFT use the same planning problems, expert actions, and optimization settings. They differ in supervision format: complete plans versus stepwise actions conditioned on intermediate observations. Corrective SFT and Self-Imitation SFT also share the supervised optimization settings but differ in how training trajectories are collected and selected. We did not sweep SFT hyperparameters, and the continued-training methods have different data-collection procedures and training budgets.

Chain-of-Thought Supervision. PlanWorlds does not use chain-of-thought supervision. Training targets contain executable actions rather than intermediate reasoning traces, so CoT quality is not a variable in our experiments. We will clarify this in the revision and identify CoT-augmented supervision as a direction for future work.

Q3. Generalization to Newer and Larger Models

Our experiments focus on Qwen2.5-7B and Llama3.1-8B to enable controlled comparisons across training strategies and evaluation settings. We agree that evaluating newer and larger models would help establish how broadly our findings apply. We will therefore clarify the scope of our claims and explicitly acknowledge this limitation in the revised paper. At the same time, PlanWorlds supports adjustable task difficulty, providing a foundation for future studies of planning transfer across models with different capabilities and scales.

Thank you again for your valuable feedback. We hope our responses address your concerns and would be happy to discuss any remaining questions.

Thank you for the clarifications and additional experiments. These substantially address my experimental concerns. However, two main reservations remain about the overall scope of the paper:

  1. The main conclusions offer limited new insight beyond expected effects of distribution and interface mismatch. Matching supervision to the execution interface and observing that in-environment success does not guarantee downstream transfer are not, by themselves, substantive advances in understanding planning transfer. The central limitation is that the analysis remains largely descriptive: it documents performance differences without establishing explanatory principles that predict when or why a post-training strategy will transfer successfully. For a paper whose primary contribution is understanding these phenomena, this substantially limits its conceptual contribution.

  2. Some claims remain broader than the evidence supports. For example, “what language-model agents actually acquire” suggests identifying learned capabilities, whereas the experiments primarily characterize differences in execution, transfer, and robustness. I suggest narrowing this wording.

Overall, the additional evidence strengthens my confidence in the findings, but does not substantially change my assessment of the contribution. I still believe the paper’s strengths outweigh its weaknesses and maintain my original rating.

Thank you for your constructive feedback and positive assessment of our work. We are glad that the additional experiments have addressed your experimental concerns.

We share your interest in understanding when and why planning transfer succeeds. Our study contributes toward this goal by using PDDL’s explicit execution rules and verifiable solutions to support controlled comparisons. These comparisons show how transfer gaps respond differently to interface controls, how further training can change transfer, and how recovery can differ among policies with nearly identical clean performance. We hope this supports a more fine-grained evaluation of planning transfer and gives future work more specific questions to investigate about when and why transfer succeeds.

We also agree that our claims should closely reflect the evidence. We have revised the abstract to focus on execution, transfer, and robustness, replacing the broader wording about what agents “actually acquire.”

Reviewer_nARjOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces PlanWorlds, a synthetic, executable planning testbed built on STRIPS-style problems rendered in natural language, and uses it to study how different post-training strategies transfer to downstream planning benchmarks. The headline results: (1) full-plan vs. interactive supervision of the same expert solutions produces starkly different execution profiles; (2) models that are near-saturated inside PlanWorlds diverge substantially on external tasks (WebShop, ALFWorld, ACPBench) after further post-training; and (3) a careful analysis shows some of these gaps shrink under target-interface controls (ACPBench) while others persist (ALFWorld), with matched perturbations revealing differences in robustness to local deviations.

Critical strengths

The experimental design is genuinely controlled: same task pool, same execution semantics, varied only in interface, trajectory source, or objective, and the authors are refreshingly upfront about what the comparisons don't isolate (budgets aren't equated, methods differ in multiple ways at once). I particularly liked the ACPBench re-elicitation analysis.
The paper is also clearly written, the experiments are extensive, and the authors are candid about confounds (unequal budgets, strategies differing in multiple factors).

Critical weaknesses

My main concern is novelty. The core observations: SFT overfits to its training interface, RL generalizes a bit better out of distribution, and downstream metrics are sensitive to decoding protocols — will not surprise anyone who has followed the SFT-vs-RL literature (Chu et al. 2025 is cited, but the paper's conclusions land close to theirs). The PlanWorlds testbed itself is nice, but I hope to see clearly what it enables that, say, PDDL-Instruct or PLAN-TUNING's setup doesn't. Right now it reads mostly as "another synthetic planner," and the eight classical domains are the same ones the field has been using for years. Relatedly, two small models, one perturbation probe with 160 problems, and two interface controls. Each individual result is a bit thin, and the paper leans on the accumulation of small effects rather than one decisive demonstration. I'd also like to see any analysis of what the models are doing differently — the behavioral story is entirely black-box success rates; even a qualitative look at failure modes in the perturbation probe would help.

Questions for authors
  1. Why is the robustness probe Qwen-only?
  2. The reward function mixes goal progress, step cost, invalid-action penalty, and success bonus, so was there any reward ablations or tuning?
  3. Can the authors report variance across the full pipeline (data generation seeds, not just training seeds)? With 2 seeds per continued-training condition, some of these gaps could be noise.
4 comments
AuthorsOpenReview ↗

Author Response (1/2)

W1. Regarding the Novelty

Thank you for suggesting we clarify our contribution.

Research question. Chu et al. (2025) compare SFT and RL under rule and visual variations. Our study examines transfer from synthetic planning training to independent tasks and interactive environments. PDDL-Instruct and PLAN-TUNING provide important context: the former evaluates generalization across symbolic planning domains, while the latter evaluates generalization on two mathematical reasoning benchmarks (OlympiadBench and AIME). These studies highlight the potential of synthetic planning data, while motivating a closer examination of how transfer should be measured and interpreted.

The role of PlanWorlds. We position PlanWorlds as a controlled experimental tool for studying transfer. The eight classical domains provide explicit execution semantics and verifiable reference solutions. These allow the same expert solutions to support different supervision interfaces, matched initializations to support different post-training paths, and executable intermediate states to support paired perturbations with controlled solvability and remaining distance to the goal. For example, Chu et al.’s multi-turn setting uses verifier feedback to revise outputs, and their V-IRL setting advances to the next scene once an action is judged correct. Our perturbation probe first executes a legal action that departs from the reference trajectory, moving the environment into another solvable state. It then tests whether the policy can complete the task while controlling the remaining difficulty. This directly examines recovery after a state deviation, providing evidence beyond overall task success.

Main findings. We show that even when models have learned the source tasks, downstream performance, interface sensitivity, and local recovery can lead to different assessments of the training methods. These findings go beyond the broad characterization that “SFT overfits while RL generalizes”: some apparent degradations need to be reassessed under interface controls, some transfer gaps persist after those controls, and local recovery cannot substitute for downstream transfer evaluation.

W2. Further Analysis of Planning Robustness and Failure Modes

To strengthen the behavioral evidence, we extended the perturbation evaluation to both model families and conducted a new trajectory-level failure analysis.

Extended Perturbation Evaluation. We evaluated 160 matched perturbation problems per model family, with two training seeds for each trained condition. We further classified failed perturbed episodes into three behavioral patterns:

  • Mostly-invalid: At least 50% of actions are rejected by the environment.
  • Loop: The model repeatedly issues the same command at least four times.
  • Wander: The model continues acting without reaching the goal before exhausting its step budget.

The tables below report failure rates and the distribution of failure modes among failed episodes.

Qwen2.5-7B-Instruct

ConditionFailure Rate (%) ↓Mostly-invalid (%)Wander (%)Loop (%)
Interactive SFT14.160382
Corrective SFT14.7533611
Self-Imitation SFT14.151427
SFT + RL8.4305615

Llama-3.1-8B-Instruct

ConditionFailure Rate (%) ↓Mostly-invalid (%)Wander (%)Loop (%)
Base59.4275914
Interactive SFT8.419784
Corrective SFT6.210900
Self-Imitation SFT8.877914
SFT + RL7.8127216

Failure-mode percentages are conditional on failed perturbed episodes and may not sum to 100 due to rounding.

Key Findings

The analysis reveals distinct behavioral patterns. On Qwen, failures under the SFT conditions are predominantly caused by invalid actions, consistent with policies continuing to follow actions that are no longer applicable after perturbation. SFT + RL reduces the overall failure rate from 14.1% to 8.4%, with a smaller proportion of failures dominated by invalid actions. Its remaining failures more often involve executable actions that fail to reach the goal.

On Llama, the Base and RL Only conditions remain vulnerable to perturbations, while all SFT-initialized conditions achieve similarly low failure rates (6.2–8.8%). Thus, the robustness advantage of SFT + RL observed on Qwen does not consistently extend to Llama.

We will add the cross-model evaluation, failure-mode analysis, and two illustrative trajectories to the appendix of the revised paper.

AuthorsOpenReview ↗

Author Response (2/2)

Q1. Extending the Robustness Probe to Llama

Thank you for this helpful suggestion. We have extended the perturbation probe to Llama to examine whether the observed recovery patterns generalize across model families.

On Llama, the Base model has a high perturbation failure rate (59%), whereas the four SFT-initialized conditions achieve 91–94% recovery success. Differences among these conditions are small (within 3 percentage points across 160 matched problems), with performance approaching the ceiling. The probe therefore does not clearly distinguish SFT from SFT + RL on Llama.

On Qwen, SFT + RL shows a clearer recovery advantage, reducing the perturbation failure rate from 14% to 8%, with a paired improvement of 5.6 percentage points (95% CI [+1.6, +10.0]).

We will include these results in the revision and clarify that the observed recovery advantage is supported on Qwen, while the near-ceiling Llama results remain inconclusive.

Q2. Reward Design and Ablation Studies

The reward weights were fixed throughout our original experiments without additional tuning. To investigate the effects of reward design and regularization, we conducted two additional ablations:

  1. Terminal-only: Retaining only the success bonus while removing progress shaping, step costs, and invalid-action penalties.
  2. Without KL: Retaining the original shaped reward while removing KL regularization.

Both ablations use the same Interactive SFT initialization, random seed, and training budget as the original SFT + RL run.

The table below reports the single-seed results for Qwen2.5-7B-Instruct.

MetricInteractive SFTShaped + KLTerminal-onlyWithout KL
In-distribution success (%)981009698
Unseen-theme success (%)92969492
Difficulty generalization (%)46544244
ALFWorld success (%)47.850.738.144.0
WebShop score61.861.359.760.9
Perturbation-induced success drop (pp) ↓14.46.912.515.0

The results highlight the importance of reward design for execution robustness. Compared with the terminal-only reward, the full setting improves ALFWorld success by 12.6 percentage points and reduces the perturbation-induced success drop by 5.6 points. Removing KL also reduces ALFWorld success (50.7% to 44.0%) and increases sensitivity to perturbations (6.9 to 15.0 pp).

Due to the limited rebuttal period, these additional ablations were conducted with a single training seed. We will extend them to two seeds, consistent with our main experiments, and include the complete results and corresponding discussion in the appendix of the revised paper.

Q3. Data-Generation Seeds and Variability

Experimental Design. We deliberately use a fixed training pool across all conditions to enable controlled comparisons of post-training strategies without confounding differences in sampled tasks. Training and evaluation pools are generated independently using different master seeds. We also verified zero overlap in PDDL problem texts between training and evaluation pools.

Statistical Robustness. As reported in Appendix D (Tables 8–9), we provide means and standard deviations across training seeds for both model families. We have additionally computed paired bootstrap 95% confidence intervals over evaluation instances, including paired differences relative to Interactive SFT.

While using a shared training pool ensures comparability across methods, we acknowledge that our analysis does not measure variability across independently regenerated training pools. We will clarify the scope of our conclusions in the revised paper.

Thank you again for your thoughtful questions. We hope these additional experiments and clarifications address your concerns, and we would welcome any further questions or discussion.

Reviewer_nARjOpenReview ↗

Thank you for the detailed rebuttal

Thank you for the detailed rebuttal and for running the additional experiments These experiments clarify the behavioral picture and improve the depth of the paper. However, I decided to maintain my current rating for the following reasons:

  1. Extending the perturbation probe to Llama-3.1-8B and adding the failure-mode breakdown are helpful. However, the new Llama results show that all SFT-initialized conditions cluster near the ceiling (91–94% recovery, within 3 percentage points), meaning the local robustness advantage of SFT + RL does not hold across both tested model families. Taken together with the downstream transfer results—where relative rankings shift depending on both the model family and the target benchmark—the empirical findings remain somewhat fragmented rather than establishing a clear, generalizable takeaway.

  2. Novelty is still the main concern. The revised paper positions against Chu et al. (2025), PDDL-Instruct, and PLAN-TUNING and better articulates the role of PlanWorlds as an executable testbed. However, the central observations—that SFT overfits to its training interface, evaluation metrics are sensitive to answer-elicitation or decoding protocols, and transfer varies across target tasks—remain close to known findings in the post-training literature.

AuthorsOpenReview ↗

Clarifying Our Contribution

Thank you for the follow-up. We appreciate your recognition of the additional experiments. We would like to offer further clarification on these two points.

1. The variation that remains under controlled comparisons reveals the conditions and limits of transfer.

Our experiments control expert solutions, initialization, and perturbation difficulty to isolate the effects of supervision interfaces, post-training strategies, and recovery. Interface controls explain some apparent transfer losses, but other gaps persist. SFT+RL improves recovery for Qwen, while Llama’s SFT-initialized conditions show similar recovery. These controlled comparisons show that source-task success, interface compatibility, and local recovery cannot individually establish downstream transfer. The findings clarify the conditions and limits of transfer, rather than establish a ranking of training methods.

2. Our novelty lies in identifying an unresolved transfer question and designing a controlled investigation to answer it.

Synthetic planning data and verifiable rewards are increasingly used to improve LLM capabilities, yet out-of-domain evaluations often cover only one or two independent benchmarks. As detailed in our previous response, PDDL-Instruct and PLAN-TUNING illustrate this pattern. This motivates us to study the transferability of synthetic planning training itself.

The research question itself is novel in the context of interactive planning. The closest work we are aware of, which we already cite, examines whether mathematical reasoning training improves broader LLM capabilities [1]. Our study investigates transfer in interactive planning, where actions change the environment and subsequent decisions depend on the resulting states.

Our methodological contribution is to use PDDL’s explicit execution semantics, verifiable solutions, and controllable task structure to investigate transfer through matched comparisons of supervision interfaces, post-training strategies, and recovery after state deviations. PlanWorlds serves as the experimental testbed for this study; we do not claim it as a novel benchmark.

We believe this line of work helps the community better understand when and how training gains transfer beyond the source tasks. This aligns with ICLR’s explicit recognition of contributions through new, relevant and impactful empirical knowledge.

[1] Huan et al. Does Math Reasoning Improve General LLM Capabilities? Understanding Transferability of LLM Reasoning. ICML, 2026.

Reviewer_V44XOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces PlanWorlds, a testbed that renders procedurally generated PDDL planning tasks in natural language, and uses it to study what LLMs learn from synthetic planning post-training. It compares six post-training strategies on Qwen2.5-7B and Llama-3.1-8B, including full-plan SFT, interactive SFT, corrective SFT, self-imitation, and RL. The authors find that models with similar in-environment success can transfer very differently to downstream tasks such as ALFWorld and WebShop, and they use interface controls and perturbation tests to explain these gaps.

Critical strengths
  1. PlanWorlds is a clean and controlled testbed. It separates symbolic dynamics from their language rendering and keeps the tasks and reference solutions fixed across training strategies, which makes the comparisons easy to interpret.

  2. I think the diagnostic analyses are useful. The interface controls show that raw downstream scores can mix decision quality with output-format compatibility. The perturbation test also reveals robustness differences that are hidden by near-identical success on clean trajectories.

  3. Evaluation is broad. The study covers two model families, several internal generalization settings, four external planning benchmarks, and three general capability benchmarks. The authors are also explicit that the comparisons concern complete training strategies rather than isolated objectives.

Critical weaknesses
  1. The Corrective SFT condition differs from standard DAgger. It collects data for only one round and leaves out the action that triggered the correction, so the model may not learn to recover from its own mistakes. Including a standard DAgger baseline could make the comparison more complete.

  2. The general-capability results might be influenced by the evaluation setup. Qwen's base scores on GSM8K and MBPP (74 and 39) are lower than the 91.6 and 79.2 reported in the Qwen2.5 technical report, while Llama's scores are close to its reported numbers. This is strange. It may be worth checking whether the MBPP gains of the Qwen variants partly come from output formatting.

  3. The current evidence may not be sufficient to answer the broad question in the title. For example, RL Only reaches 63~67% in-distribution success, compared with 98–100% for the SFT conditions, so its weaker transfer could reflect limited training rather than what RL learns. Adding training curves and comparisons at matched internal success might help.

Questions for authors

See weaknesses.

5 comments
AuthorsOpenReview ↗

Author Response (1/2)

We appreciate your careful review and insightful comments. Please find our detailed responses to each point below.

W1. Clarifying Corrective SFT and Its Differences from DAgger

Rationale for Corrective SFT. In PlanWorlds, multiple optimal execution paths may exist, so disagreement with the expert planner does not necessarily indicate an error. Corrective SFT therefore uses planning progress to determine when intervention is needed. Specifically, it retains valid student trajectory prefixes and switches to expert continuations only when an action is invalid, fails to reduce the estimated remaining steps, or makes the task unsolvable. This design allows us to examine how imitation learning from corrected trajectories affects execution and transfer.

Additional DAgger Experiment. Following the reviewer's suggestion, we implemented a three-round DAgger-style baseline. In each round, the student collects trajectories, and the expert planner labels actions at states visited by the student. The training examples preserve the student's actions and environment feedback as context, while applying the supervision loss only to the expert's next-action label. We verified this implementation through loss-mask checks.

Each round uses 16,000 randomly sampled current-round states and 6,000 complete expert trajectories. From the second round, we additionally include 3,000 previously labelled states to retain historical supervision. This controls the training volume per round while retaining expert demonstrations.

The table below reports results on Qwen2.5-7B-Instruct. DAgger R1–R3 denote successive training rounds, with DAgger results averaged over two random seeds.

MetricInteractive SFTCorrective SFTDAgger R1DAgger R2DAgger R3
In-distribution success (%)989910010099
Unseen-theme success (%)9397644550
Unseen-theme invalid action rate (%)88314542
Difficulty generalization (%)4447484244
ALFWorld success (%)45.829.910.83.02.2
WebShop score61.461.655.552.248.4
Perturbation-induced success drop (pp)14.114.76.94.44.7

Results and Analysis. The DAgger-style baseline improves robustness to local perturbations, reducing the success-rate drop from 14.1 to 4.7 percentage points by round three, while maintaining near-perfect in-distribution performance. However, this improvement does not translate to broader generalization. Unseen-theme success falls to 50%, ALFWorld success drops from 45.8% to 2.2%, and the WebShop score declines from 61.4 to 48.4. Performance on both external benchmarks deteriorates progressively across DAgger rounds.

To investigate the ALFWorld decline, we further evaluated the models with constrained decoding, restricting outputs to valid environment commands.

ModelALFWorld Success with Constrained Decoding (%)
Base34.3
Interactive SFT48.5 / 44.8 / 49.3
Corrective SFT40.3 / 39.6
DAgger R325.4 / 22.4

Values separated by slashes represent individual training seeds.

Although constrained decoding substantially improves DAgger's performance, it remains well below Interactive SFT, indicating that command-interface mismatch alone does not explain the transfer gap.

Trajectory analysis provides further insight. In DAgger, 70–83% of actions repeat the immediately preceding action, compared with 8–10% for Interactive SFT, and 98% of episodes reach the 50-step limit. These patterns suggest that repeated actions and limited adaptation to environment feedback contribute to the poor transfer performance.

We also audited disagreements between student and planner actions. Notably, 79–85% involve different actions that are both certified optimal. Thus, disagreement with the planner does not necessarily indicate a task-level error, and expert-action supervision may encourage the student to imitate one of several equally valid choices.

AuthorsOpenReview ↗

Author Response (2/2)

W2. Clarifying Qwen's MBPP Evaluation and Results

We thank the reviewer for this insightful observation, which prompted us to examine the evaluation pipeline more closely.

Evaluation Protocol. To ensure fair comparisons, we used the benchmark's default evaluation configuration consistently across all models and training conditions. Upon closer inspection, we found that unexpected formatting markers in some Qwen outputs caused valid solutions to be incorrectly extracted. We manually inspected the outputs, improved the extractor, and re-evaluated all conditions on both the 100- and 500-problem MBPP sets.

The corrected results are shown below (mean ± standard deviation across seeds).

ConditionOriginal 100Full 500
Base64.065.0
Full-Plan SFT69.3 ± 1.566.1 ± 0.6
Interactive SFT68.3 ± 0.666.5 ± 0.1
Corrective SFT70.0 ± 0.066.8 ± 0.0
Self-Imitation SFT66.0 ± 2.866.1 ± 0.1
RL Only65.5 ± 0.765.2 ± 0.3
SFT + RL69.0 ± 1.465.9 ± 0.7

Comparison with Official Evaluation Settings. We also appreciate the reviewer's comparison with the Qwen2.5 technical report. Our evaluation uses consistent instances, prompts, and scoring procedures across all conditions. However, our 5-shot GSM8K and 3-shot MBPP settings differ from Qwen's official zero-shot configurations, and some evaluation details are not publicly specified. Therefore, the official scores serve as useful references but are not directly comparable.

We also checked the other benchmarks and found no similar extraction issues. In the revision, we will report the corrected MBPP results, and refine our conclusions on general-capability retention.

W3. Understanding RL-Only Training and Transfer Performance

We further examined RL Only's training dynamics and failure patterns to better understand its generalization and transfer behavior.

Training Dynamics. The training reward plateaued after approximately 120 steps, with no consistent improvement in external transfer at later checkpoints. This suggests that continued training under the current configuration offers limited additional gains, although alternative training or reward settings may yield further improvements.

Failure Analysis. Our domain-level analysis shows that RL Only performs comparably to Interactive SFT on six of the eight PlanWorlds domains, with most of the remaining performance gap concentrated in Depot and Logistics. In these domains, RL Only typically generates valid actions but fails to reach the goal. One possible explanation is that the goal-progress reward provides limited guidance for intermediate actions whose benefits emerge only after several steps.

These findings suggest that RL Only can acquire effective planning behaviors through verifiable rewards, but its learning may be constrained by reward design and exploration challenges in tasks involving multi-step dependencies and delayed feedback. We will include the training curves and domain-level failure analysis in the appendix.

We thank the reviewer again for helping us identify these issues and improve our analysis. We hope these additional experiments and clarifications address your concerns.

AuthorsOpenReview ↗

Follow-up on Our Responses and Revisions

Thank you again for your constructive feedback, which has helped us improve the paper. We would be grateful to hear whether our responses address your concerns. For W1 / Q1, we added a multi-round DAgger-style baseline. For W2 / Q2, we reviewed the MBPP evaluation procedure and re-evaluated all conditions, leading to more consistent conclusions. For Q3, we added training curves and failure analyses to the appendix to examine RL Only’s transfer. If any points remain open, we would be happy to clarify them before the discussion closes.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.