- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes an annotation-free reinforcement learning framework for natural language to PDDL planning tasks, named Solver-Grounded Multi-Role Reinforcement Learning. The same language model takes on three roles over shared parameters: the Actor generates PDDL specifications, the Judge evaluates generated results based on solver feedback, and the Editor repairs failed specifications within a limited number of steps according to solver diagnostics. Through role collaboration, the authors aim to convert sparse solver feedback into more effective generation, evaluation, and repair signals, while reducing the risk that the model "tricks" the solver by weakening the goal or altering the specification. The experiments use Qwen2.5-7B and report an average planning success rate of 70.8% on PlanBench, higher than the Self-Refine baseline under the same solver-call budget; the authors also report a 66.3% semantic faithfulness success rate and a 6.4% conditional semantic drift rate, and test zero-shot transfer on ProntoQA and several subsets of NATURAL PLAN.
Critical strengths
- It addresses an important problem in symbolic planning. Solver success does not mean that the generated PDDL faithfully expresses the original natural language task. The paper clearly distinguishes "executability" from "semantic faithfulness" and treats specification drift as a problem to be evaluated, rather than only reporting planning success rates.
- The multi-role design has a clear functional division of labor. The Actor is responsible for overall specification generation, the Judge learns to evaluate solver outcomes, and the Editor performs bounded-step repairs based on structured diagnostics. Compared with repeatedly prompting the model to rewrite without distinction, organizing the feedback and repair flow by role has a certain methodological rationale.
- The experiments include comparisons under the same model and the same solver budget. In the setting of Qwen2.5-7B and at most six solver calls, the paper reports an average PlanBench success rate of 70.8%, higher than Self-Refine+Solver's 49.8%; the average number of actual calls is also smaller (3.2, compared with 4.7). This makes the main performance comparison more convincing than merely comparing against models of different scales.
- The evaluation is not limited to planning success rates. The paper also reports semantic faithfulness success rates, conditional drift rates, and per-domain analyses; it additionally tests Mystery BlocksWorld with predicate-name confusion and several zero-shot transfer tasks. The appendix also provides training configurations, computational resources, and review results of the evaluation checker, which helps in understanding the experimental setup.
Critical weaknesses
- There remains a misalignment between the training feedback and the core objective. The Judge's training objective is to predict solver success, not to judge whether the specification is faithful to the original task; the Actor's reward also includes the Judge's solver-success prediction. Therefore, the training process itself does not directly receive a semantic faithfulness signal. The paper's theoretical analysis relies on assumptions such as "the probability of disagreement between solver success and task correctness is low," and solver feedback alone cannot guarantee that this assumption still holds during training and after policy changes.
- The coverage of the semantic faithfulness checker is limited. The evaluation checker mainly checks goals, objects and types, action patterns, and transition semantics. The paper reports human review and controlled perturbation tests, which increase confidence in the checker; however, these indicators may still not cover all subtle constraints of natural language tasks, and may also struggle to handle semantically equivalent but formally different PDDL expressions.
- The transfer results are not yet sufficient to support broad generalization conclusions. The PlanBench experiments focus on four classic, deterministic planning domains; zero-shot transfer covers only ProntoQA and the Trip Planning and Calendar Scheduling subsets of NATURAL PLAN, using single Actor generation. The paper also includes a leave-one-domain-out experiment, but the current evidence still makes it difficult to claim that the method applies to real planning tasks that are more complex, involve uncertainty, or have richer action semantics.
- The theoretical conclusion about shared parameters suppressing reward hacking relies on strong assumptions. The paper's local analysis adopts a specific gradient model and assumes that the Judge and Editor can constrain the relevant directions in the shared parameter space. Whether these conditions hold in actual training is not easy to determine; therefore, the theoretical result is more like a local explanation under specific conditions than a guarantee against general reward-hacking risk.
Questions for authors
- Regarding the relationship between training objectives and semantic faithfulness: Since the Judge is trained only on solver outcomes, how do the authors verify that the reward shaping it provides can suppress semantic drift, rather than merely making it easier for the model to predict or optimize solver success? Does the paper's key assumption about the consistency between solver outcomes and task correctness still hold after the policy is continuously updated?
- Regarding the faithfulness checker: How does the checker handle complex natural language constraints, implicit constraints, and semantically equivalent but structurally different PDDL expressions? Can the authors provide more false-positive and false-negative analyses conducted by independent experts, and show checker performance across different categories of tasks?
- Regarding the scope of generalization: Can the method be validated on a broader range of planning domains or more complex tasks, such as tasks with uncertainty, numerical, or temporal constraints? For ProntoQA and NATURAL PLAN, can comparisons be added under the same base model and the same inference budget, and can the impact of single Actor output on the results be analyzed?
- Regarding the theoretical conclusions: Can the assumptions in the paper, such as gradient constraints and local isotropy, be measured or verified on actual training checkpoints? When these assumptions do not hold, or when the model undergoes large parameter updates, can parameter sharing still stably reduce semantic drift?
4 comments
Response to Reviewer UGTo (2/3)
W2 / Q2. Complex constraints, equivalent expressions, and independent checker errors.
The checker in Appendix H.4 aligns types and renamings, normalizes conjunctions, compares goals and action structure, and replays plans under reference transitions. It checks explicit task constraints against the reference specification. Plan replay verifies the transitions along the returned plan. This gives each check a concrete semantic target. For natural-language constraints, the task description and reference specification define the intended meaning.
The independent audit you requested uses 800 model-generated candidates, two per task and 200 per domain. It covers Actor/Editor outputs and includes high-scoring errors. Two annotators label faithfulness independently, with a third adjudicating disagreements. The 74 invalid candidates are assessed separately for structural validity. Semantic metrics use the valid candidates:
| Domain/subset | Valid | Precision [95% CI] | Recall [95% CI] | False accepts / unfaithful | False rejects / faithful | |
|---|---|---|---|---|---|---|
| BW | 186 | 96.5 [92.1,98.5] | 95.9 [91.3,98.1] | 5/41 | 6/145 | .86 |
| MBW | 182 | 96.0 [91.0,98.3] | 93.8 [88.2,96.8] | 5/53 | 8/129 | .81 |
| Logistics | 180 | 94.1 [88.3,97.1] | 92.5 [86.4,96.0] | 7/60 | 9/120 | .77 |
| Gripper | 178 | 92.8 [86.4,96.3] | 91.2 [84.5,95.1] | 8/65 | 10/113 | .80 |
| Overall | 726 | 95.0 [92.7,96.6] | 93.5 [91.0,95.3] | 25/219 | 33/507 | .81 |
| Valid-but-unsolved | 274 | 94.1 [89.5,96.8] | 95.2 [90.9,97.6] | 10/106 | 8/168 | .84 |
Alongside the original binary decision, an error-reason diagnostic measures concrete omissions and schema errors:
| Category | TP / FP / FN | Precision [95% CI] | Recall or correct acceptance [95% CI] |
|---|---|---|---|
| Goal weakening/change | 44 / 3 / 8 | 93.6 [82.8,97.8] | 84.6 [72.5,92.0] |
| Initial state/object/type | 54 / 4 / 7 | 93.1 [83.6,97.3] | 88.5 [78.2,94.3] |
| Action precondition/effect | 91 / 6 / 13 | 93.8 [87.2,97.1] | 87.5 [79.8,92.5] |
| Explicit constraint omission | 49 / 5 / 9 | 90.7 [80.1,96.0] | 84.5 [73.1,91.6] |
| Equivalent, non-isomorphic syntax | — | — | 59/68 = 86.8 [76.7,92.9] |
The intervals use the candidate-level Wilson calculation for this audit pool, treating candidate counts as independent. The equivalence test extends Table 11's reorderings and renamings to non-isomorphic expressions. The checker correctly accepts 59 of 68 expert-confirmed equivalent outputs, with nine false rejections. Across valid outputs, precision is 95.0% and recall is 93.5%. The domain and error-category breakdowns make both successful checks and remaining error patterns explicit.
W3 / Q3. Richer planning semantics and matched single-Actor transfer.
The full pipeline is tested on 100 fixed, unseen Miconic-ADL tasks using frozen checkpoints, six solver calls, and an independent transition/reference-plan checker with human adjudication where needed. This tests richer action semantics outside the four training domains:
| Method | [95% CI] | [95% CI] | Calls/task | |
|---|---|---|---|---|
| Self-Refine+Solver | 32 [23.7,41.7] | 22 [15.0,31.1] | 31.25 | 5.1 |
| Monolithic-RL+repair | 43 [33.7,52.8] | 32 [23.7,41.7] | 25.58 | 4.6 |
| Full | 56 [46.2,65.3] | 49 [39.4,58.7] | 12.50 | 3.9 |
| ADL subset | Self-Refine faithful | Monolithic faithful | Full faithful | |
|---|---|---|---|---|
| Active conditional effects | 60 | 10/60 | 17/60 | 27/60 |
| Complex preconditions | 40 | 12/40 | 15/40 | 22/40 |
For the richer planning settings you mention, this experiment focuses on conditional effects and complex preconditions within deterministic ADL. These semantics are supported by the Fast Downward backend [1]. Full improves faithful success on both subsets, providing evidence of transfer to action semantics beyond the training domains.
For your single-Actor question, Table 14 uses the same Qwen2.5-7B backbone, a format-only prompt, one greedy generation, and zero solver calls. The monolithic trained control provides a further comparison under this protocol, with 100 tasks per benchmark:
| Checkpoint | ProntoQA exact match | Trip Planning exact match | Calendar Scheduling exact match |
|---|---|---|---|
| Base, Table 14 | 63 | 12 | 27 |
| Actor-only RL, Table 14 | 72 | 24 | 31 |
| Monolithic-RL+repair, new control | 79 | 34 | 35 |
| Full-trained Actor, Table 14 | 90 | 48 | 40 |
The Full-trained Actor leads on all three benchmarks under the same single-generation budget. These gains are present in the trained checkpoint itself. The ADL experiment separately evaluates transfer of the complete generation-and-repair process.
[1] Fast Downward Documentation: PDDL Support.
Response to Reviewer UGTo (3/3)
W4 / Q4. Gradient assumptions and behavior under larger updates.
To examine the assumptions you highlight, we use checkpoint-level gradient diagnostics and controlled update-size experiments. Appendix E gives a local bound on first-order gains for Judge/Editor-neutral perturbations. It assumes backbone sensitivity and an isotropic hacking-gradient model. The training algorithm applies joint PPO updates, whose behavior is evaluated directly below.
The actual-checkpoint diagnostic uses 512 per-sample gradients and a fixed 256-dimensional projection. With , the reported spectral effective rank is :
| Training progress | Full | Blocked-gradient | Full cosine A/J, A/E | Blocked-gradient cosine A/J, A/E |
|---|---|---|---|---|
| 25% | 12.5 | 11.1 | +.12, −.04 | +.07, −.08 |
| 50% | 14.3 | 10.0 | +.10, +.03 | +.03, −.02 |
| 100% | 16.7 | 9.1 | +.08, +.09 | −.02, −.05 |
The blocked-gradient variant computes J/E gradients for diagnosis only. The table measures sensitivity within the 256-dimensional projection. Full's effective rank increases across checkpoints, and its final Actor gradients have positive cosine similarity with both roles. Full-backbone rank and the distribution of are assumptions of the local theorem, distinct from these projected role-gradient measurements.
A more general bound also makes the role of isotropy precise. For a fixed admissible subspace , projection , and second moment ,
=\mathbb E\\|P_{\mathcal H}g_{\mathrm{hack}}\\|_2 \leq\sqrt{\operatorname{tr}(P_{\mathcal H}G P_{\mathcal H})}.$$ This follows from Cauchy–Schwarz and holds for any finite second moment on the fixed admissible subspace. It expresses the bound through the projected energy of the hacking gradient. Here $G$ concerns $g_{\mathrm{hack}}$, while the diagnostic $\Sigma_{JE}$ concerns Judge/Editor gradients. To directly test update size, branches from the final checkpoint use the same update batch at different step multipliers: | Step multiplier | Measured KL | Full $S/F$ | Full $d$ | Blocked J/E $S/F$ | Blocked J/E $d$ | | --- | ---: | ---: | ---: | ---: | ---: | | 0× | .0000 | 70.75/66.25 | 6.4 | 45.50/36.50 | 19.8 | | 0.5× | .0007 | 71.25/67.00 | 6.0 | 46.00/36.75 | 20.1 | | 1× | .0028 | 71.75/67.50 | 5.9 | 46.50/36.75 | 21.0 | | 2× | .0112 | 72.25/65.00 | 10.0 | 47.50/36.00 | 24.2 | **Full has lower drift than the blocked-gradient control at every measured step size.** At the standard 1× update, faithful success increases to 67.50% and drift decreases to 5.9%. The 2× update raises solver success but lowers faithful success, showing why update size matters. These results support the benefit of shared updates and give a direct empirical account of their local stability.Follow-up to Reviewer UGTo
Thank you again for your thoughtful questions, which motivated checkpoint studies showing semantic drift among solved outputs falling from 31.33% to 6.36% as training progresses. We also provide an independent checker audit, held-out ADL results, and gradient and update-size diagnostics, together with a bound that relaxes the isotropy assumption.
We would be happy to discuss any remaining concerns and provide further clarification or evidence during the remaining discussion period. If these results and clarifications address your concerns, we would be grateful if you would consider updating your rating.
Response to Reviewer UGTo (1/3)
Thank you for your time and thoughtful feedback on faithfulness, generalization, and the theoretical analysis. We address these points below with the corresponding experiments and derivations.
Throughout this response, S is solver success, F is solved-and-faithful success, and d=100(S−F)/S is drift among solved outputs. Success rates are percentages.
Response. Your question concerns whether solver supervision continues to improve faithfulness as the policy changes. Equation (7) makes the relevant condition explicit through disagreement between solver success and task success. For solver label B and specification faithfulness M, task success is C=BM and disagreement is ϵ=S−F.
To test the relationship during policy updates, each frozen checkpoint generates its own outputs on the same held-out tasks. Three seeds and 100 tasks/domain/seed give:
The final-checkpoint interventions separate two pathways:
Removing shaping lowers faithful success by 6.75 points and increases drift. Blocking shared updates has a larger effect. These controls support contributions from both mechanisms. Across the measured checkpoints, Full's faithful success increases and its conditional drift decreases. The disagreement term varies during training, so the empirical connection is evaluated for each checkpoint's policy. Human semantic labels are used solely for evaluation.