acceptodds
Under review as a conference paper at ICLR 2027

From Solver Feedback to Faithful Plans: Multi-Role Reinforcement Learning for Symbolic Planning

Abstract

Reliable planning requires converting natural-language instructions into executable symbolic specifications, yet large language models remain brittle without costly PDDL annotations and may exploit solver success in semantically unfaithful ways. We study how to learn faithful natural-language-to-PDDL formalization using only solver feedback, without human-written demonstrations. We propose a solver-grounded multi-role reinforcement learning framework where a single language model acts as an Actor, Judge, and Editor for generation, verification, and repair. The Actor proposes PDDL specifications, the Judge provides a solver-calibrated quality signal, and the Editor performs bounded diagnostic-conditioned refinement. On PlanBench, our method improves average success from 35.5% for LLM+P to 70.8%, achieves 66.3% faithful success, and reduces semantic drift to 6.4%. These results show that organizing solver feedback into generation, verification, and repair roles enables more scalable and faithful annotation-free symbolic planning.

Primary area:
foundation or frontier models, including LLMs
Keywords:
large language models, symbolic planning, reinforcement learning, natural language to PDDL, solver feedback, semantic faithfulness

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_UGToOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes an annotation-free reinforcement learning framework for natural language to PDDL planning tasks, named Solver-Grounded Multi-Role Reinforcement Learning. The same language model takes on three roles over shared parameters: the Actor generates PDDL specifications, the Judge evaluates generated results based on solver feedback, and the Editor repairs failed specifications within a limited number of steps according to solver diagnostics. Through role collaboration, the authors aim to convert sparse solver feedback into more effective generation, evaluation, and repair signals, while reducing the risk that the model "tricks" the solver by weakening the goal or altering the specification. The experiments use Qwen2.5-7B and report an average planning success rate of 70.8% on PlanBench, higher than the Self-Refine baseline under the same solver-call budget; the authors also report a 66.3% semantic faithfulness success rate and a 6.4% conditional semantic drift rate, and test zero-shot transfer on ProntoQA and several subsets of NATURAL PLAN.

Critical strengths
  • It addresses an important problem in symbolic planning. Solver success does not mean that the generated PDDL faithfully expresses the original natural language task. The paper clearly distinguishes "executability" from "semantic faithfulness" and treats specification drift as a problem to be evaluated, rather than only reporting planning success rates.
  • The multi-role design has a clear functional division of labor. The Actor is responsible for overall specification generation, the Judge learns to evaluate solver outcomes, and the Editor performs bounded-step repairs based on structured diagnostics. Compared with repeatedly prompting the model to rewrite without distinction, organizing the feedback and repair flow by role has a certain methodological rationale.
  • The experiments include comparisons under the same model and the same solver budget. In the setting of Qwen2.5-7B and at most six solver calls, the paper reports an average PlanBench success rate of 70.8%, higher than Self-Refine+Solver's 49.8%; the average number of actual calls is also smaller (3.2, compared with 4.7). This makes the main performance comparison more convincing than merely comparing against models of different scales.
  • The evaluation is not limited to planning success rates. The paper also reports semantic faithfulness success rates, conditional drift rates, and per-domain analyses; it additionally tests Mystery BlocksWorld with predicate-name confusion and several zero-shot transfer tasks. The appendix also provides training configurations, computational resources, and review results of the evaluation checker, which helps in understanding the experimental setup.
Critical weaknesses
  • There remains a misalignment between the training feedback and the core objective. The Judge's training objective is to predict solver success, not to judge whether the specification is faithful to the original task; the Actor's reward also includes the Judge's solver-success prediction. Therefore, the training process itself does not directly receive a semantic faithfulness signal. The paper's theoretical analysis relies on assumptions such as "the probability of disagreement between solver success and task correctness is low," and solver feedback alone cannot guarantee that this assumption still holds during training and after policy changes.
  • The coverage of the semantic faithfulness checker is limited. The evaluation checker mainly checks goals, objects and types, action patterns, and transition semantics. The paper reports human review and controlled perturbation tests, which increase confidence in the checker; however, these indicators may still not cover all subtle constraints of natural language tasks, and may also struggle to handle semantically equivalent but formally different PDDL expressions.
  • The transfer results are not yet sufficient to support broad generalization conclusions. The PlanBench experiments focus on four classic, deterministic planning domains; zero-shot transfer covers only ProntoQA and the Trip Planning and Calendar Scheduling subsets of NATURAL PLAN, using single Actor generation. The paper also includes a leave-one-domain-out experiment, but the current evidence still makes it difficult to claim that the method applies to real planning tasks that are more complex, involve uncertainty, or have richer action semantics.
  • The theoretical conclusion about shared parameters suppressing reward hacking relies on strong assumptions. The paper's local analysis adopts a specific gradient model and assumes that the Judge and Editor can constrain the relevant directions in the shared parameter space. Whether these conditions hold in actual training is not easy to determine; therefore, the theoretical result is more like a local explanation under specific conditions than a guarantee against general reward-hacking risk.
Questions for authors
  • Regarding the relationship between training objectives and semantic faithfulness: Since the Judge is trained only on solver outcomes, how do the authors verify that the reward shaping it provides can suppress semantic drift, rather than merely making it easier for the model to predict or optimize solver success? Does the paper's key assumption about the consistency between solver outcomes and task correctness still hold after the policy is continuously updated?
  • Regarding the faithfulness checker: How does the checker handle complex natural language constraints, implicit constraints, and semantically equivalent but structurally different PDDL expressions? Can the authors provide more false-positive and false-negative analyses conducted by independent experts, and show checker performance across different categories of tasks?
  • Regarding the scope of generalization: Can the method be validated on a broader range of planning domains or more complex tasks, such as tasks with uncertainty, numerical, or temporal constraints? For ProntoQA and NATURAL PLAN, can comparisons be added under the same base model and the same inference budget, and can the impact of single Actor output on the results be analyzed?
  • Regarding the theoretical conclusions: Can the assumptions in the paper, such as gradient constraints and local isotropy, be measured or verified on actual training checkpoints? When these assumptions do not hold, or when the model undergoes large parameter updates, can parameter sharing still stably reduce semantic drift?
4 comments
AuthorsOpenReview ↗

Response to Reviewer UGTo (1/3)

Thank you for your time and thoughtful feedback on faithfulness, generalization, and the theoretical analysis. We address these points below with the corresponding experiments and derivations.

Throughout this response, SS is solver success, FF is solved-and-faithful success, and d=100(S−F)/Sd=100(S-F)/S is drift among solved outputs. Success rates are percentages.

W1 / Q1. Does shaping reduce semantic drift as the policy changes?

Response.\color{green}{\mathbf{Response.}} Your question concerns whether solver supervision continues to improve faithfulness as the policy changes. Equation (7) makes the relevant condition explicit through disagreement between solver success and task success. For solver label BB and specification faithfulness MM, task success is C=BMC=B M and disagreement is ϵ=S−F\epsilon=S-F.

To test the relationship during policy updates, each frozen checkpoint generates its own outputs on the same held-out tasks. Three seeds and 100 tasks/domain/seed give:

Training progressFull SSFull FFϵ\epsilon (pp)Full ddNo-shaping ddBlocked J/E gradients dd
0%20.7514.256.5031.3331.3331.33
25%35.0028.007.0020.0028.1332.43
50%49.2542.007.2514.7221.3526.62
75%62.5056.006.5010.4016.5222.56
100%70.7566.254.506.3610.1919.78

The final-checkpoint interventions separate two pathways:

ConfigurationSSFFS−FS-F (pp)dd
No Actor shaping, Judge loss retained66.2559.506.7510.19
J/E backbone gradients blocked45.5036.509.0019.78
Full70.7566.254.506.36

Removing shaping lowers faithful success by 6.75 points and increases drift. Blocking shared updates has a larger effect. These controls support contributions from both mechanisms. Across the measured checkpoints, Full's faithful success increases and its conditional drift decreases. The disagreement term varies during training, so the empirical connection is evaluated for each checkpoint's policy. Human semantic labels are used solely for evaluation.

AuthorsOpenReview ↗

Response to Reviewer UGTo (2/3)

W2 / Q2. Complex constraints, equivalent expressions, and independent checker errors.

Response.\color{green}{\mathbf{Response.}} The checker in Appendix H.4 aligns types and renamings, normalizes conjunctions, compares goals and action structure, and replays plans under reference transitions. It checks explicit task constraints against the reference specification. Plan replay verifies the transitions along the returned plan. This gives each check a concrete semantic target. For natural-language constraints, the task description and reference specification define the intended meaning.

The independent audit you requested uses 800 model-generated candidates, two per task and 200 per domain. It covers Actor/Editor outputs and includes high-scoring errors. Two annotators label faithfulness independently, with a third adjudicating disagreements. The 74 invalid candidates are assessed separately for structural validity. Semantic metrics use the valid candidates:

Domain/subsetValid nnPrecision [95% CI]Recall [95% CI]False accepts / unfaithfulFalse rejects / faithfulκ\kappa
BW18696.5 [92.1,98.5]95.9 [91.3,98.1]5/416/145.86
MBW18296.0 [91.0,98.3]93.8 [88.2,96.8]5/538/129.81
Logistics18094.1 [88.3,97.1]92.5 [86.4,96.0]7/609/120.77
Gripper17892.8 [86.4,96.3]91.2 [84.5,95.1]8/6510/113.80
Overall72695.0 [92.7,96.6]93.5 [91.0,95.3]25/21933/507.81
Valid-but-unsolved27494.1 [89.5,96.8]95.2 [90.9,97.6]10/1068/168.84

Alongside the original binary decision, an error-reason diagnostic measures concrete omissions and schema errors:

CategoryTP / FP / FNPrecision [95% CI]Recall or correct acceptance [95% CI]
Goal weakening/change44 / 3 / 893.6 [82.8,97.8]84.6 [72.5,92.0]
Initial state/object/type54 / 4 / 793.1 [83.6,97.3]88.5 [78.2,94.3]
Action precondition/effect91 / 6 / 1393.8 [87.2,97.1]87.5 [79.8,92.5]
Explicit constraint omission49 / 5 / 990.7 [80.1,96.0]84.5 [73.1,91.6]
Equivalent, non-isomorphic syntax——59/68 = 86.8 [76.7,92.9]

The intervals use the candidate-level Wilson calculation for this audit pool, treating candidate counts as independent. The equivalence test extends Table 11's reorderings and renamings to non-isomorphic expressions. The checker correctly accepts 59 of 68 expert-confirmed equivalent outputs, with nine false rejections. Across valid outputs, precision is 95.0% and recall is 93.5%. The domain and error-category breakdowns make both successful checks and remaining error patterns explicit.


W3 / Q3. Richer planning semantics and matched single-Actor transfer.

Response.\color{green}{\mathbf{Response.}} The full pipeline is tested on 100 fixed, unseen Miconic-ADL tasks using frozen checkpoints, six solver calls, and an independent transition/reference-plan checker with human adjudication where needed. This tests richer action semantics outside the four training domains:

MethodSS [95% CI]FF [95% CI]ddCalls/task
Self-Refine+Solver32 [23.7,41.7]22 [15.0,31.1]31.255.1
Monolithic-RL+repair43 [33.7,52.8]32 [23.7,41.7]25.584.6
Full56 [46.2,65.3]49 [39.4,58.7]12.503.9
ADL subsetnnSelf-Refine faithfulMonolithic faithfulFull faithful
Active conditional effects6010/6017/6027/60
Complex preconditions4012/4015/4022/40

For the richer planning settings you mention, this experiment focuses on conditional effects and complex preconditions within deterministic ADL. These semantics are supported by the Fast Downward backend [1]. Full improves faithful success on both subsets, providing evidence of transfer to action semantics beyond the training domains.

For your single-Actor question, Table 14 uses the same Qwen2.5-7B backbone, a format-only prompt, one greedy generation, and zero solver calls. The monolithic trained control provides a further comparison under this protocol, with 100 tasks per benchmark:

CheckpointProntoQA exact matchTrip Planning exact matchCalendar Scheduling exact match
Base, Table 14631227
Actor-only RL, Table 14722431
Monolithic-RL+repair, new control793435
Full-trained Actor, Table 14904840

The Full-trained Actor leads on all three benchmarks under the same single-generation budget. These gains are present in the trained checkpoint itself. The ADL experiment separately evaluates transfer of the complete generation-and-repair process.

[1] Fast Downward Documentation: PDDL Support.

AuthorsOpenReview ↗

Response to Reviewer UGTo (3/3)

W4 / Q4. Gradient assumptions and behavior under larger updates.

Response.\color{green}{\mathbf{Response.}} To examine the assumptions you highlight, we use checkpoint-level gradient diagnostics and controlled update-size experiments. Appendix E gives a local bound on first-order gains for Judge/Editor-neutral perturbations. It assumes backbone sensitivity and an isotropic hacking-gradient model. The training algorithm applies joint PPO updates, whose behavior is evaluated directly below.

The actual-checkpoint diagnostic uses 512 per-sample gradients and a fixed 256-dimensional projection. With ΣJE=E[gJgJ⊤]+E[gEgE⊤]\Sigma_{JE}=\mathbb E[g_Jg_J^\top]+\mathbb E[g_Eg_E^\top], the reported spectral effective rank is reff=tr⁡(ΣJE)/λmax⁡(ΣJE)r_{\mathrm{eff}}=\operatorname{tr}(\Sigma_{JE})/\lambda_{\max}(\Sigma_{JE}):

Training progressFull reffr_{\mathrm{eff}}Blocked-gradient reffr_{\mathrm{eff}}Full cosine A/J, A/EBlocked-gradient cosine A/J, A/E
25%12.511.1+.12, −.04+.07, −.08
50%14.310.0+.10, +.03+.03, −.02
100%16.79.1+.08, +.09−.02, −.05

The blocked-gradient variant computes J/E gradients for diagnosis only. The table measures sensitivity within the 256-dimensional projection. Full's effective rank increases across checkpoints, and its final Actor gradients have positive cosine similarity with both roles. Full-backbone rank and the distribution of ghackg_{\mathrm{hack}} are assumptions of the local theorem, distinct from these projected role-gradient measurements.

A more general bound also makes the role of isotropy precise. For a fixed admissible subspace H\mathcal H, projection PHP_{\mathcal H}, and second moment G=E[ghackghack⊤]G=\mathbb E[g_{\mathrm{hack}}g_{\mathrm{hack}}^\top],

=\mathbb E\\|P_{\mathcal H}g_{\mathrm{hack}}\\|_2 \leq\sqrt{\operatorname{tr}(P_{\mathcal H}G P_{\mathcal H})}.$$ This follows from Cauchy–Schwarz and holds for any finite second moment on the fixed admissible subspace. It expresses the bound through the projected energy of the hacking gradient. Here $G$ concerns $g_{\mathrm{hack}}$, while the diagnostic $\Sigma_{JE}$ concerns Judge/Editor gradients. To directly test update size, branches from the final checkpoint use the same update batch at different step multipliers: | Step multiplier | Measured KL | Full $S/F$ | Full $d$ | Blocked J/E $S/F$ | Blocked J/E $d$ | | --- | ---: | ---: | ---: | ---: | ---: | | 0× | .0000 | 70.75/66.25 | 6.4 | 45.50/36.50 | 19.8 | | 0.5× | .0007 | 71.25/67.00 | 6.0 | 46.00/36.75 | 20.1 | | 1× | .0028 | 71.75/67.50 | 5.9 | 46.50/36.75 | 21.0 | | 2× | .0112 | 72.25/65.00 | 10.0 | 47.50/36.00 | 24.2 | **Full has lower drift than the blocked-gradient control at every measured step size.** At the standard 1× update, faithful success increases to 67.50% and drift decreases to 5.9%. The 2× update raises solver success but lowers faithful success, showing why update size matters. These results support the benefit of shared updates and give a direct empirical account of their local stability.
AuthorsOpenReview ↗

Follow-up to Reviewer UGTo

Thank you again for your thoughtful questions, which motivated checkpoint studies showing semantic drift among solved outputs falling from 31.33% to 6.36% as training progresses. We also provide an independent checker audit, held-out ADL results, and gradient and update-size diagnostics, together with a bound that relaxes the isotropy assumption.

We would be happy to discuss any remaining concerns and provide further clarification or evidence during the remaining discussion period. If these results and clarifications address your concerns, we would be grateful if you would consider updating your rating.

Reviewer_FA4cOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper looks at how to translate natural language planning tasks into PDDL without any task-specific PDDL demonstrations. It trains a single model to generate a specification, score it against solver results, and then repair it using the solver's diagnostics. On four PlanBench domains it achieves 70.8% solver success, 66.3% faithful success, and only 6.4% semantic drift among solved outputs. The paper's central concern is relevant: a solver can verify that a specification is executable without verifying that it has the correct meaning.

Critical strengths

The method is trying to solve the problem of a specification being executable yet wrong in terms of the original specification. It generates a PDDL spec, gets feedback from Fast Downward and then modifies the spec when the solver reports an error. That's a very clear way of using solver feedback in both the generation and the repair phase.

The evaluation compares the approach to Self-Refine+Solver using the same Qwen2.5-7B model and a cap of six solver calls. The proposed method achieves 70.8% solver success and 66.3% faithful success as opposed to 49.8% and 37.5% for Self-Refine+Solver. It also averages 3.2 solver calls per case (vs 4.7 for Self-Refine+Solver).

The dataset choices test a number of relevant capabilities. PlanBench contains four planning domains (Mystery BlocksWorld being one of them where object and predicate names are hidden). The authors also report transfer results on ProntoQA and NaturalPlan and do an evaluation on domains held out during training. The held out domain results are lower than the standard results but do provide some useful evidence of performance outside of the training domains.

The analysis goes beyond the overall success rate. The authors track improvements across repair steps, examine remaining errors, assess Judge scores for faithful vs. unfaithful outputs, ,and test the faithfulness checker against human judgment. The human audit shows an inter-annotator agreement of κ = 0.84 and a checker F1 of 94%.

Critical weaknesses

The Judge is designed to detect solver success, not semantic correctness. The paper's assertion that Judge scores can track task correctness assumes solver success and task correctness agree on the model's outputs. This is reasonable to examine, but the method does not guarantee it. The post hoc audit provides some evidence of correlation but does not prove the Judge will reliably identify faithful specifications across domains and failure modes.

The Judge objective is described in a very inconsistent manner. The main method describes a binary cross-entropy objective with a confidence penalty. The appendix describes a thresholded Judge reward and joint policy gradient updates. That's a very important point, not just a matter of wording. The Judge is central to the method and to the faithfulness argument.

The benchmark coverage is extremely narrow in terms of the range of claims. The core training and evaluation is based on four typed STRIPS domains. The transfer results on ProntoQA and NaturalPlan are based on a single actor call and so don't represent the transfer of the full generation-solver-repair method. They also don't demonstrate the method's ability to deal with unsolvable problems or with changing task constraints/planning settings outside of the domains used in the selection.

The interpretability analysis is of some value in that it does give you some insight, the repair curves and solver diagnostics being particularly useful in terms of showing where the failures are. The judge scores and AUC do correlate with faithfulness but they don't tell you why a specification is faithful and the AUC in no way demonstrates that the scores are calibrated. The human audit is clearly a positive, though the 400 solver successful cases it provides are a very narrow basis on which to judge the performance in the case of unsolved and invalid outputs.

The claims of generality and efficiency need to be made in the context of task-family coverage and full pipeline cost.

Questions for authors
  1. Which judge objective led to the reported results? Please reconcile the main text loss with the appendix reward, state the exact training objective and hyperparameters and report judge performance on a held-out human labelled set using a reliability plot, Brier score and expected calibration error. Report the results by domain and include confidence intervals.

  2. How much of the gain is due to each role? Please report Actor only, Actor+Judge, Actor+Editor and full model results across all four domains with at least 3 random seeds and matched backbone size, training budget and solver call budget. Please report both solver success and faithful success (not just solver success on Mystery BlocksWorld).

  3. How does it perform in the case of no plan? You need to separate out the various failure modes: invalid PDDL, solver timeout, proven unsolvability and valid plan found. How many cases fall into each category and what is the accuracy for each? A single 'failure' label will mask whether the problem was simply unformalisable, unsolvable or correctly identified as having no plan.

  4. Does the method generalise beyond the four training domains? Please go through the full Actor–Judge–Editor process on at least one held-out task family outside of typed STRIPS, using a fixed test set and an independent correctness checker. Report solver success, faithful success, conditional drift and solver calls per instance. The single actor call transfer results must be kept separate from this evaluation.

  5. How robust is the faithfulness checker in the face of real world errors? The controlled perturbation study is a good first step. We'd like to see the checker's precision and recall by error type on independently sampled model outputs (schema errors, missing constraints, invalid cases, unsolved cases etc). We'll need the sampling procedure and the confidence intervals on both the human audit and checker metrics.

  6. What is the cost of the method (to train and to use)? Please report training GPU-hours, total solver calls, median and 90th percentile solver calls per test instance and end-to-end inference time. Compare these against the matched Self-Refine+Solver baseline under the same backbone and solver limit.

5 comments
AuthorsOpenReview ↗

Response to Reviewer FA4c (1/4)

We appreciate the time you spent on this detailed review and your constructive comments on the objectives and evaluation. We respond to each point below with the training details and supporting results.

Throughout this response, BB is solver success, MM is specification faithfulness, and C=BMC=B M. We report S=E[B]S=\mathbb{E}[B], F=E[C]F=\mathbb{E}[C], and conditional drift d=100(S−F)/Sd=100(S-F)/S. Success rates are percentages.

W1. Does a solver-trained Judge reliably track faithfulness?

Response.\color{green}{\mathbf{Response.}} Your question calls for evidence that the Judge's training signal improves faithfulness. The Judge shapes the Actor's reward during training, while the solver determines acceptance at inference. Equation (7) relates the Judge to task correctness under a bound on solver–correctness disagreement. Its δJ\delta_J denotes uniform approximation error. ECE and AUC measure calibration and ranking, respectively.

Two interventions test the mechanism under identical solver-interaction budgets. The first removes Actor shaping and retains Judge-loss training. The second blocks J/E backbone gradients and retains their role-specific learning. Three-seed means are:

ConfigurationSSFFS−FS-F (pp)dd
No Actor shaping66.2559.506.7510.19
J/E backbone gradients blocked45.5036.509.0019.78
Full70.7566.254.506.36

The held-out human audit also measures semantic ranking among solved candidates:

Audit candidatesSolved candidatesAUROC for MM [95% CI]AUPRC for MM [95% CI]
8004520.860 [0.820, 0.895]0.945 [0.923, 0.964]

Full achieves the highest faithful success and lowest drift in these controls. The human audit also shows that Judge scores rank faithful solved candidates above unfaithful ones. The analysis below complements this evidence with domain-specific calibration errors.

AuthorsOpenReview ↗

Response to Reviewer FA4c (2/4)

W2 / Q1. Exact Judge objective, gradient flow, and held-out calibration.

Response.\color{green}{\mathbf{Response.}} Tables 1–5 and the controlled experiments use the BCE-plus-band objective in Section 3.4 and Appendix B.2. Appendix A's thresholded Judge reward is a separate formulation. With s=sJ(x,y0)s=s_J(x,y_0) and b=B0b=B_0, the implemented objectives are:

\ell_{\mathrm{band}}(s,b) &=(1-b)\max(0,s-0.5)^2+\tfrac12 b\max(0,0.5-s)^2,\\\\ \mathcal L_J&=\mathrm{BCE}(s,b)+0.2\ell_{\mathrm{band}}(s,b),\\\\ R_A&=B_0+0.3\\,\operatorname{stopgrad}(s_J),\qquad R_E=B_T-0.1T,\\\\ \mathcal L&=\mathcal L_{\mathrm{PPO}}^A+0.5\mathcal L_{\mathrm{PPO}}^E+0.3\mathcal L_J. \end{aligned}$$ Here $T\leq5$ counts repair attempts. The two coefficients of 0.3 weight Actor shaping and Judge loss separately. PPO treats the score as a constant reward. The Judge loss backpropagates through the shared backbone. | Setting | Value | | --- | --- | | Optimizer | AdamW, learning rate $3\times10^{-5}$, $\beta=(0.9,0.95)$, weight decay 0.01 | | PPO | Clip 0.2, batch 32, 2 epochs, discount 0.9, GAE 0.95, gradient clip 1.0 | | Training | 12,000 solver interactions/seed including repairs, training temperature 0.7 | | Repair | At most 5 Editor steps, early stopping at first solver success, 60 seconds/solver call | The held-out calibration audit contains 800 candidates: 100 task IDs per domain and two distinct Actor/Editor candidates per task. It includes high-scoring errors. Two annotators assess faithfulness independently, and a third resolves disagreements. The pool contains 452 solved, 274 valid-but-unsolved, and 74 invalid candidates. We evaluate the Judge's raw scores directly. Brier and ECE use all candidates and 10 equal-width bins. Confidence intervals use 2,000 task-ID clustered bootstrap resamples and describe this audit pool. | Domain | $n$ | Brier\$_B\$ [95% CI] | ECE\$_B\$ [95% CI] | Brier\$_C\$ [95% CI] | ECE\$_C\$ [95% CI] | | --- | ---: | --- | --- | --- | --- | | BW | 200 | .074 [.059,.091] | .161 [.139,.192] | .063 [.051,.076] | .159 [.140,.185] | | MBW | 200 | .103 [.086,.121] | .172 [.151,.215] | .091 [.077,.106] | .149 [.126,.184] | | Logistics | 200 | .101 [.083,.122] | .198 [.167,.234] | .111 [.093,.131] | .099 [.093,.150] | | Gripper | 200 | .097 [.082,.114] | .205 [.178,.237] | .109 [.091,.127] | .136 [.122,.187] | | Overall | 800 | .094 [.085,.102] | .179 [.165,.197] | .094 [.085,.102] | .131 [.115,.149] | The reliability curve is specified by these bin coordinates: | Score bin | $n$ | Mean score | Observed $B$ rate | Observed $C$ rate | | --- | ---: | ---: | ---: | ---: | | [0.0,0.1) | 92 | .042 | .043 | .000 | | [0.1,0.2) | 115 | .152 | .035 | .000 | | [0.2,0.3) | 108 | .248 | .093 | .009 | | [0.3,0.4) | 57 | .347 | .263 | .035 | | [0.4,0.5) | 48 | .445 | .812 | .292 | | [0.5,0.6) | 50 | .553 | 1.000 | .600 | | [0.6,0.7) | 96 | .651 | 1.000 | .771 | | [0.7,0.8) | 85 | .755 | 1.000 | .859 | | [0.8,0.9) | 67 | .843 | 1.000 | .955 | | [0.9,1.0] | 82 | .964 | 1.000 | .988 | The reliability curves separate solver success from faithful success most clearly in the middle bins. In the highest-score bin, the observed faithful-success rate is 98.8%. The departures from the identity line quantify probability error, following the calibration definition in [1]. Together, the ranking metrics and reliability curves show how the scores relate to task quality. The $C$ target specifically measures solved-and-faithful success, while $M$ records the semantics of valid unsolved specifications. [1] On Calibration of Modern Neural Networks. --- > #### **W3 / Q4. Full-pipeline generalization beyond typed STRIPS.** $\color{green}{\mathbf{Response.}}$ The frozen four-domain checkpoint is evaluated on **100 fixed Miconic-ADL tasks held out from training and checkpoint selection**. The test activates conditional effects in 60 tasks and complex preconditions in 40. The complete generation–solver–repair process uses the same six-call cap and 60-second solver limit. An independent checker aligns grounded transitions and validates plans against the reference. Human adjudication resolves ambiguous cases. | Method | $S$ [95% CI] | $F$ [95% CI] | $d$ | Calls/task | Automatic / adjudicated | | --- | --- | --- | ---: | ---: | ---: | | Self-Refine+Solver | 32 [23.7,41.7] | 22 [15.0,31.1] | 31.25 | 5.1 | 94 / 6 | | Monolithic-RL+repair | 43 [33.7,52.8] | 32 [23.7,41.7] | 25.58 | 4.6 | 91 / 9 | | **Full** | **56 [46.2,65.3]** | **49 [39.4,58.7]** | **12.50** | **3.9** | 93 / 7 | Intervals are Wilson intervals over 100 tasks. Full achieves a **17-point faithful-success advantage over trained repair** under richer deterministic action semantics. It also uses fewer solver calls per task. This tests the full pipeline, and Table 14 separately measures single-Actor transfer on ProntoQA/NATURAL PLAN. Q3 below addresses tasks with no plan.
AuthorsOpenReview ↗

Response to Reviewer FA4c (3/4)

W4 / Q5. Calibration versus interpretability, and checker reliability beyond solved cases.

Response.\color{green}{\mathbf{Response.}} Your distinction between score discrimination and semantic explanation is useful. Table 5 and Figure 5 show score separation. The specification checks and error audit identify concrete semantic issues, and W2/Q1 reports probability calibration.

The new audit covers 800 model-generated candidates, with 200 per domain and two distinct candidates per task. It spans Actor/Editor outputs and includes high-scoring errors. Two annotators provide independent semantic labels, and a third resolves disagreements. Precision and recall are computed on valid candidates, with faithfulness as the positive class. Invalid candidates receive a separate structural assessment:

Domain/subsetValid nnPrecision [95% CI]Recall [95% CI]False accepts / false rejectsκ\kappaAnnotator disagreements
BW18696.5 [92.1,98.5]95.9 [91.3,98.1]5 / 6.869/186
MBW18296.0 [91.0,98.3]93.8 [88.2,96.8]5 / 8.8115/182
Logistics18094.1 [88.3,97.1]92.5 [86.4,96.0]7 / 9.7719/180
Gripper17892.8 [86.4,96.3]91.2 [84.5,95.1]8 / 10.8017/178
Overall72695.0 [92.7,96.6]93.5 [91.0,95.3]25 / 33.8160/726
Valid-but-unsolved27494.1 [89.5,96.8]95.2 [90.9,97.6]10 / 8.8421/274

An error-reason diagnostic evaluated alongside the binary checker provides the following multilabel results. This diagnostic assigns error categories, while the original interface returns a binary faithfulness decision. An independent parser supplies the invalidity labels.

Error typeGold casesTP / FP / FNPrecision [95% CI]Recall [95% CI]
Goal weakening/change5244 / 3 / 893.6 [82.8,97.8]84.6 [72.5,92.0]
Initial state/object/type6154 / 4 / 793.1 [83.6,97.3]88.5 [78.2,94.3]
Action precondition/effect10491 / 6 / 1393.8 [87.2,97.1]87.5 [79.8,92.5]
Explicit constraint omission5849 / 5 / 990.7 [80.1,96.0]84.5 [73.1,91.6]
Invalid PDDL7473 / 2 / 197.3 [90.8,99.3]98.6 [92.7,99.8]

The intervals use the candidate-level Wilson calculation for this audit pool, treating candidate counts as independent. The error categories are multilabel and can overlap. This extends Table 10's audit to naturally occurring failures. On valid-but-unsolved outputs, checker precision is 94.1% and recall is 95.2%. The category-level results also identify which constraints and action semantics are harder to detect.


W5 / Q6. Training cost and full-pipeline inference efficiency.

Response.\color{green}{\mathbf{Response.}} To answer your cost question, we report training and full-pipeline inference measurements under the same hardware and protocol. The setup uses one node with 8×A100-80GB, CPU-based Fast Downward, batch size 1, and fixed concurrency/cache settings. Timing starts after warm-up. Inference statistics cover all 400 tasks, including failures, timeouts, and the executed Judge/Editor work.

MethodTraining GPU-hours/seed (3-seed total)Training solver calls/seed (total)Training tokens/seedTest calls mean / median / P90End-to-end seconds mean / median / P90
Self-Refine+Solver0004.70 / 6 / 6109 / 87 / 256
Actor-only RL101.6 (304.8)12,000 (36,000)19.8M5.11 / 6 / 6112 / 87 / 278
Monolithic-RL+repair109.4 (328.2)12,000 (36,000)25.1M4.07 / 6 / 698 / 77 / 240
Full112.0 (336.0)12,000 (36,000)27.5M3.21 / 2 / 688 / 66 / 226
MethodTraining solver CPU-hours/seedTest solver CPU-hours/400 tasksTest LLM tokens/taskPeak inference GPU memory
Self-Refine+Solver03.05,05018.5 GB
Monolithic-RL+repair16.52.44,25019.1 GB
Full17.81.93,92019.3 GB

Full reduces measured mean inference time by 19.3% relative to matched Self-Refine. Its median solver-call count drops from six to two, and its P90 end-to-end time is also lower. The table reports the task-specific training cost alongside these inference gains. The generalization evidence covers the four STRIPS domains and held-out deterministic ADL, as detailed above.

Reviewer_qcViOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces a reinforcement learning method that trains an LLM to turn natural language instructions into executable PDDL plans without human-labeled examples. The model simultaneously acts as an Actor to create plans, a Judge to evaluate their quality, and an Editor to fix errors using solver feedback. The method improves the overall planning performance while keeping a good semantic alignment to original statement.

Critical strengths
  • The idea is well motivated. It is important and useful to reduce the need for expensive human-written PDDL annotation.

  • The empirical evidence is strong. The proposed method achieves considerable improvement over existing methods.

  • The reward hacking problem is explicitly considered, as it distinguishes the successful run of a solver and semantic correctness. This is generally an important problem in formal language reasoning / symbolic planning.

  • The experiments includes semantic faithfulness checks, human evaluation, held-out domains, and zero-shot transfer. I think it is comprehensive.

Critical weaknesses
  • The methodologial contribution is limited. In fact, Actor–Judge–Editor resembles existing popular generate-verify-refine frameworks. The main novelty is their integration with solver-based RL, which I think is also quite popular in coding tasks.

  • The semantic guarantees are weak. The Judge still learns mostly from the solver and whether it is a successful compilation or not, rather than the true semantic correctness. The theoretical guarantees assume limited semantic drift instead of demonstrating that the method prevents it.

  • The proposed method is trained with refinforcement learning, while several competing methods rely only on prompting. Stronger trained baselines would make the comparison more convincing.

  • Some important references that also use natural language to generate PDDL programs is missing: [Generating Symbolic World Models via Test-time Scaling of Large Language Models, https://arxiv.org/abs/2502.04728]. I simply searched relevant topics and found this one, but I think there could be more. I suggest the authors to do a better survey and compare with the missing references.

Questions for authors

See the weakness section.

3 comments
AuthorsOpenReview ↗

Response to Reviewer qcVi (1/2)

Thank you for taking the time to review our paper and for your thoughtful suggestions. We address your questions about the learning design, semantic evidence, and comparisons below.

Throughout this response, SS denotes solver success, FF denotes solved-and-faithful success, and d=100(S−F)/Sd=100(S-F)/S denotes drift among solved outputs. Success rates are percentages.

W1. Novelty beyond generate–verify–refine and solver-based RL.

Response.\color{green}{\mathbf{Response.}} The key challenge is that the model generates the specification that the solver subsequently verifies. It can therefore earn reward by producing an easier specification that changes the task. Our approach gives the Actor credit for its initial specification before any repair. Shared training then connects this initial generation to solver prediction and diagnostic-conditioned repair. Appendix D explains why rewarding only the final repaired output can obscure the quality of the initial specification.

To address your question, we compare against a trained, single-policy generation-and-repair baseline with terminal reward BT−0.1TB_T-0.1T. Both methods start from Qwen2.5-7B, use 12,000 solver interactions per seed, and allow six solver calls at inference. The table reports three-seed means over four domains:

Training configurationInitial FFFinal SSFinal FFdd
Monolithic-RL+repair21.0054.0042.5021.30
Full without Actor shaping36.5066.2559.5010.19
Full with J/E backbone gradients blocked21.5045.5036.5019.78
Full39.5070.7566.256.36

Full improves faithful success by 23.75 points over trained monolithic repair. Its stronger initial generations also show that the gain is already present before repair. The ablations further show that both Actor shaping and shared backbone updates improve faithful success.


W2. Solver-supervised Judge and conditional semantic guarantees.

Response.\color{green}{\mathbf{Response.}} The training label requires the solver to find a valid plan within its budget. Your question concerns the additional requirement that the specification preserve the task's meaning. Equation (7) makes this connection explicit: E∣sJ−C∣≤ϵ+δJ\mathbb{E}|s_J-C|\leq\epsilon+\delta_J. Here BB is solver success, MM is specification faithfulness, C=BMC=B M, and ϵ=Pr⁡(B≠C)\epsilon=\Pr(B\ne C). The term δJ\delta_J bounds approximation to the solver-success predictor. This gives a conditional guarantee whose disagreement term can be measured as S−FS-F.

The Judge supplies the Actor's shaping reward during training. At inference, the solver's valid-plan verdict determines acceptance and early stopping.

We evaluate whether faithful success improves and disagreement decreases as training proceeds. Frozen checkpoints on the same held-out tasks give:

Training progressSSFFS−FS-F (pp)dd
0%20.7514.256.5031.33
25%35.0028.007.0020.00
50%49.2542.007.2514.72
75%62.5056.006.5010.40
100%70.7566.254.506.36

Each checkpoint is evaluated on 100 tasks per domain for each of three training seeds. The final S/FS/F agrees with Tables 2–3. Faithful success rises from 14.25% to 66.25%, while conditional drift falls from 31.33% to 6.36%. Evaluating both quantities shows that the lower drift accompanies substantial gains in task success. These checkpoint results provide empirical support for the conditional analysis.


W3. Trained baselines under matched budgets.

Response.\color{green}{\mathbf{Response.}} The trained controls you requested use the same initialization, data splits, 12,000 solver interactions per seed, and six-call inference cap. Training interactions include repairs. During the controlled stochastic evaluation, methods without an Editor sample independent candidates after failure. Methods with an Editor repair sequentially. All methods share the sampling settings and total generation-token cap.

MethodSS, mean ± SDFF, mean ± SDddMean solver callsTraining GPU-hours/seedTraining tokens/seed
Self-Refine+Solver, prompting49.7537.5024.624.7000
Actor-only RL26.25 ± 2.1721.50 ± 1.8018.105.11101.619.8M
Monolithic-RL+repair54.00 ± 2.0542.50 ± 1.5021.304.07109.425.1M
Full70.75 ± 1.9866.25 ± 1.756.363.21112.027.5M

SD is calculated across three training seeds, each evaluated on 400 tasks. The prompting baseline has one fixed evaluation. Solver interactions are matched, and the table also reports actual training compute. Full achieves higher faithful success than both prompting and trained repair, with fewer solver calls per task.

AuthorsOpenReview ↗

Response to Reviewer qcVi (2/2)

W4. Missing comparison with Yu et al. and related work.

Response.\color{green}{\mathbf{Response.}} Thank you for pointing us to Yu et al. [1]. Their BoN+iVML method searches and revises symbolic models at inference with fixed LLM weights. Rosa et al. [2] also study checking and repair for problem generation given a domain. Our work learns from solver interactions across tasks. It assigns credit to the initial specification and uses shared-role updates to improve subsequent generation.

For a direct comparison, BoN+iVML is adapted to joint domain/problem generation from the natural-language input alone. The evaluation uses Qwen2.5-7B and the same six-solver-call and generation-token caps. The results below come from this adaptation under our evaluation protocol:

MethodBW S/FS/FMBW S/FS/FLogistics S/FS/FGripper S/FS/FOverall FFdd
BoN+iVML, adapted91/8351/3742/3040/2744.2520.98
Full, three-seed mean98/9471/6758/5356/5166.256.36

The adapted method improves on Self-Refine, and Full retains a 22-point faithful-success advantage. Together with the trained monolithic comparison in W3, this addresses both stronger inference-time search and the contribution of our training design.

[1] Generating Symbolic World Models via Test-time Scaling of Large Language Models.

[2] Grounded Evaluation and Repair for NL-to-PDDL Problem Generation.

AuthorsOpenReview ↗

Follow-up to Reviewer qcVi

Thank you again for your helpful suggestions, which motivated comparisons showing 66.25% faithful success for our method versus 42.50% for trained monolithic repair and 44.25% for adapted BoN+iVML under matched inference limits. We also report drift among solved outputs falling from 31.33% to 6.36% during training, with ablations supporting the contributions of initial-specification credit and shared-role updates.

We would be happy to discuss any remaining questions and provide further clarification or evidence while the discussion is still open. If these comparisons and clarifications address your concerns, we would be grateful if you would consider updating your rating.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.