acceptodds
Withdrawn from ICLR 2027

DiGnosis: Verification and Repair of Task Plans via Conditional Graph Diffusion Models

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Task planning frameworks solve complex user queries by selecting a set of available tools and their dependencies, naturally represented as a plan graph. Whether they rely on large language models (LLMs), prompted or fine-tuned, or augment them with graph-based components, task planners can produce incorrect plan graphs, with steps of the query left uncovered, missing tool dependencies, or tools absent from the catalog. Verifying plan graphs before execution is therefore essential, yet most existing verifiers still rely on LLMs, which may be just as prone to errors as the planners they are meant to check. In this work, we propose DiGnosis, a planner-agnostic verifier that exploits the graph structure of the produced plans and recasts their verification as a graph denoising task. Concretely, we train a discrete graph diffusion model only on correct plan graphs, conditioned on the query that generated them. From this single denoiser, DiGnosis derives three signals for a candidate plan graph: a brokenness score, a per-node and per-edge diagnosis with preferred alternatives, and a one-shot repair. Score and diagnosis never involve an LLM, which merely proposes a repair candidate guided by the diagnosis, accepted only if it lowers the brokenness score. Extensive experiments on three datasets and four task planners of different nature show that DiGnosis improves the raw plan graph more often and by larger margins than four existing verifiers. Moreover, when the planner is fine-tuned on the same correct plan graphs, DiGnosis recognizes that its plans are largely correct and therefore abstains from repairing them.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Task Planning, Verifier, Graph Neural Networks, Large Language Models, Graph Diffusion Models

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_jTdfOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper "DIGNOSIS: VERIFICATION AND REPAIR OF TASK PLANS VIA CONDITIONAL GRAPH DIFFUSION MODELS" presents a framework for verifying and repairing task plans using conditional graph diffusion models. It aims to improve the reliability of tool-use and task planning by treating plans as graphs.

Critical strengths
  1. Principled LLM-Free Verification and Objective Scoring Gate: Unlike conventional verifiers that rely entirely on LLM prompting, DiGnosis derives both its brokenness score and per-node/edge diagnosis directly from the diffusion denoiser. The strict acceptance criterion (Σrepaired<Σraw\Sigma_{repaired} < \Sigma_{raw}) prevents the widespread issue where LLM verifiers degrade already correct plans (e.g., Self-Refine drops ReAct accuracy from 45.4% to 24.8% on HuggingFace, whereas DiGnosis maintains or improves accuracy across nearly all settings).
  2. Outstanding Efficiency and Compact Footprint: With only 6.43M trainable parameters and an 84-second training time on a single NVIDIA H200 (Table 7), DiGnosis achieves sub-second verification latency (0.02s for score and diagnosis alone; ~1.1s for the full pipeline including LLM repair). This provides an orders-of-magnitude efficiency and cost advantage over multi-turn agentic or large-LLM verifiers.
Critical weaknesses
  1. Unequal Chain Rewriting and Confounded Gain Attribution Across Planners: A major methodological nuance is the framework's use of chain rewriting: for graph-based verifiers (DiGnosis and GNNVerifier), returned plan graphs are deterministically rewritten as linear chains implied by tool order, whereas LLM-based verifiers return plans unmodified. On planners like DIRECT, unequal chain rewriting alone accounts for the vast majority of accuracy gains on HuggingFace and Multimedia by eliminating spurious edge topologies. Conversely, on planners that already generate chain-shaped plan graphs (GNN4Plan and ReAct), chain rewriting has zero effect. This unequal treatment creates an asymmetric advantage over LLM baselines and confounds how much gain stems from diffusion graph denoising versus deterministic topological linearization.
  2. Evaluation Limits Imposed by Chain-Shaped Reference Graphs and Planner-Specific Calibration:
    • Chain-Shaped Reference-Graph Evaluation Limits: In Section 3.4 and Appendix D.1, every reference test plan graph across all three benchmarks is strictly a chain-shaped plan of at most 8 tools (100% of UltraTool and all test splits). Consequently, DiGnosis's performance on genuine directed acyclic graphs (DAGs) with complex branching, concurrency, conditional forks, or multi-parent dependencies remains unvalidated on benchmark test sets despite being trained on general graph representations.
    • Planner-Specific Calibration Requirement: DiGnosis's brokenness gate σ\sigma and repair policy are not universal constants; they require planner-specific calibration on 300 validation plan graphs produced by each specific planner. As acknowledged in the text, transferring calibration across planners (e.g., applying GNN4Plan-tuned thresholds to GTool) causes the score gate to misclassify valid plans and degrade accuracy, necessitating a separate calibration pool whenever a planner or environment changes.
  3. Fixed Tool Capacity and Semantic Tool Equivalence Sensitivity: The fixed matrix capacity d=12d=12 slots limits application to long-horizon plans. Furthermore, as shown in Appendix F (Table 11), one-hot categorical node encodings cannot capture functional near-synonyms (e.g., image upscaling vs. resolution enhancement), leading the denoiser to penalize semantically valid alternative tools.
Questions for authors
  1. Isolating Chain Rewriting from Diffusion Denoising on DIRECT: In the ablation study (Section 5.4, RQ2), chain rewriting accounts for most of the accuracy gains on DIRECT. Could the authors report a baseline where LLM verifiers (e.g., VeriPlan, VeriCoder) are also equipped with identical post-hoc chain rewriting to isolate the pure graph-diffusion repair gain from topological linearization?
  2. Generalization Beyond Chain-Shaped Reference Graphs: Since the test splits across HuggingFace, Multimedia, and UltraTool contain exclusively chain-shaped reference graphs, how does DiGnosis perform when evaluated on synthetic or real-world DAG reference graphs with non-trivial diamond structures or parallel branches?
  3. Sensitivity to Planner-Specific Calibration Drift: How sensitive is the brokenness threshold σ\sigma when the underlying planner undergoes prompt modifications, few-shot demonstration updates, or decoding temperature shifts? Does DiGnosis require re-calibration on new validation query sets whenever planner generation distributions drift?
  4. Tool Catalog Expansion and Open-Vocabulary Scaling: Since node features are one-hot encodings over a fixed extended catalog T′=T∪{∅,unk}T' = T \cup \{\emptyset, \text{unk}\}, how does DiGnosis adapt when tool catalogs are dynamically modified at runtime?
Reviewer_bnpDOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes DiGnosis, a verifier and repair framework for tool-use plans based on query-conditioned discrete graph diffusion. The model is trained by corrupting correct reference plans and learning to reconstruct their node and edge attributes. At inference time, the denoiser provides a graph-level brokenness score, local diagnoses, and a one-shot repair candidate. An LLM can propose an additional repair, and the learned score selects among candidates. Experiments evaluate the framework on three benchmarks, four planner types, and two LLM backbones, with additional analysis of a fine-tuned planner.

Critical strengths
  1. A coherent formulation unifying scoring, diagnosis, and repair. Representing plan errors as changes to categorical node and edge attributes allows the same denoiser to produce a global score, localized alternatives, and a direct repair candidate. Empty slots provide a simple mechanism for inserting missing tools within the fixed-capacity representation. This is a useful formulation for studying learned plan verification, and the ability to generate repairs directly from the verifier is a meaningful aspect of the approach.

  2. Evidence of genuine repair gains, accompanied by transparent failure analysis. The GNN4Plan–HuggingFace result improves from 35.60 to 42.40 accuracy. Since these plans are already chain-shaped, the paper's explanation indicates that chain rewriting does not account for this improvement. The variant without the LLM edit still reaches 39.53, supporting a useful contribution from the denoising repair mechanism itself. The additional backbone, runtime measurements, and explicit examples of damaged correct plans make the empirical study more informative.

Critical weaknesses
  1. The gains from chain rewriting are insufficiently separated from the contribution of the diffusion model. According to Appendix D.1, all three test sets contain only chain-shaped plans with at most eight tools. Appendix D.5 applies chain rewriting to DiGnosis and GNNVerifier, while other LLM-based verifiers return their generated graphs directly. The authors further acknowledge that chain rewriting accounts for most of the DIRECT improvements on HuggingFace and Multimedia. This means that some of the main improvements may come from a simple deterministic graph post-processing rule rather than the proposed conditional diffusion verification model. The same rewriting should be applied to the raw planners and all verifiers, alongside a standalone chain-rewriting baseline.

  2. Positive-only training and automatic abstention depend substantially on reference-based calibration. Appendix D.6 selects tdiagt_{\mathrm{diag}} and α\alpha using validation AUROC, and selects ρ\rho, trepairt_{\mathrm{repair}}, and the repair policy using n-F1+l-F1\text{n-F1}+\text{l-F1} on 300 validation queries. Thus, the complete system depends on reference-labeled calibration despite training the denoiser only from correct plans. More importantly, Appendix E.1 states that every benchmark and seed in RQ3 selects a global no-repair policy. Consequently, preserving the fine-tuned planner’s outputs provides limited evidence of accurate instance-level abstention. The gate should be evaluated independently of this global policy, with clarification of performance without reference plans for calibration.

  3. The ablation results provide limited support for the independent value of detailed diagnosis. Table 2 reports accuracy of 42.40±1.4442.40 \pm 1.44 for the full method, 39.53±0.4239.53 \pm 0.42 without the LLM edit, 42.27±1.2942.27 \pm 1.29 without diagnostic information Δ\Delta, and 38.67±0.5038.67 \pm 0.50 with random candidate selection. Removing the LLM edit reduces accuracy by 2.87 percentage points, whereas removing Δ\Delta reduces it by only 0.13 points, much smaller than the reported run-to-run variation. These results support the importance of LLM repair and candidate selection, while providing limited evidence that detailed node/edge diagnosis improves upon simply informing the LLM that the plan may be incorrect.

Questions for authors

Q1. After applying chain rewriting to the raw planners and all verifiers, how much improvement does DiGnosis retain over a chain-only baseline?

Q2. How well does instance-level abstention work independently of the global no-repair policy, and how dependent is it on reference-labeled calibration?

Q3. What evidence supports the independent value of detailed diagnosis, given that removing Δ\Delta reduces accuracy by only 0.13 percentage points?

Reviewer_kmegOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper proposes DiGnosis, a query-conditioned discrete graph diffusion model for verifying and repairing task plans. A Graph Transformer learns to reconstruct correct plan graphs from synthetically corrupted inputs. Its predictions provide a brokenness score, node and edge diagnoses, and a direct repair candidate. The full pipeline also generates an LLM repair candidate, selects candidates using the learned score, and applies chain rewriting. Experiments cover three benchmarks, four planners, and two LLM backbones, with an additional fine-tuned planner. The reported results show improved reference-plan matching in most evaluated configurations.

Critical strengths

A unified model provides verification signals and a direct repair path. The denoiser produces global scores, local diagnoses, and repair candidates without requiring changes to the planner’s parameters. This offers a useful alternative to relying exclusively on an LLM for repair generation.

The experiments demonstrate benefits beyond LLM editing and chain rewriting. In Table 2, removing LLM editing still improves Accuracy from 35.60 to 39.53 on HuggingFace with GNN4Plan, compared with 42.40 for the full method. GNN4Plan already produces chains, so this improvement cannot be attributed to chain rewriting. The evaluation also reuses each benchmark’s denoiser across multiple planners and backbones, while recalibrating the inference policy.

Critical weaknesses
  1. The presentation makes the method and its contribution difficult to assess. Sections 2–3 require substantial effort to follow. The Problem Formulation relies on lengthy verbal definitions without compactly summarizing the main input–output relationships. Section 3.1 then introduces numerous sets, indices, special categories, and tensor dimensions before showing how a concrete plan becomes the model input. Readers must repeatedly cross-reference these definitions to connect the representation, denoising model, diagnosis, and repair pipeline. This makes it harder to assess how the method addresses the stated planning failures. Compact mappings such as G^qP=Planner⁡(q,GT)\widehat{G}_q^P=\operatorname{Planner}(q,G^T) and (q,G^qP)→(ΣqP,ΔqP,ΩqP)(q,\widehat{G}_q^P)\rightarrow(\Sigma_q^P,\Delta_q^P,\Omega_q^P) would clarify the formulation. A notation table and a worked example following one plan through slot encoding, diagnosis, coordinated node and edge edits, and final selection would substantially improve readability.

  2. The motivation also needs a clearer distinction from the closest prior work. GNNVerifier (Hao et al., 2026) already performs graph-level scoring, local diagnosis, and repair acceptance using a GNN. The Introduction acknowledges this work, but emphasizes its reliance on artificial corruptions and LLM-based repair. DiGnosis also constructs synthetic corruptions during training, and its full evaluated pipeline includes LLM-generated repairs. Key differences include the corruption process, reconstruction objective, and direct repair capability. These are meaningful distinctions, but the paper does not clearly establish which specific limitation of the previous approach they resolve. Consequently, the broad motivation about unreliable LLM verification does not adequately substantiate the incremental contribution over the closest prior work.

  3. The experiments do not sufficiently isolate the contribution of diffusion to the reported gains. Section 5 acknowledges that chain rewriting accounts for most DIRECT gains on HuggingFace and Multimedia. Appendix D.5 applies this operation to both graph-based verifiers, but not to the LLM-based verifiers. This complicates attribution when comparing against raw planners and LLM-based methods, although it does not explain the difference between DiGnosis and GNNVerifier. Table 2 provides evidence of learned repair benefits, but does not compare diffusion against a matched, simpler conditional graph denoiser. The marginal-versus-uniform noise ablation cannot establish that distinction.

    The abstention result also requires a narrower interpretation. Appendix E.1 reports that calibration selects the no-repair policy for every benchmark and seed with SFT-Direct, while chain rewriting remains active. This supports conservative policy selection on validation data, but does not establish instance-level recognition of already-correct plans. Together, these issues leave the specific contribution of diffusion-based verification less clear than the overall pipeline results.

Questions for authors
  1. Diffusion-specific contribution. What evidence distinguishes the benefit of diffusion training from that of a simpler conditional graph denoising objective? A matched comparison keeping the architecture, training data, and downstream pipeline fixed would help resolve this concern.

  2. Contribution of chain rewriting. Could the authors report a chain-rewriting-only control and results before this post-processing step, particularly for DIRECT on HuggingFace and Multimedia? This would clarify how much improvement comes from learned verification and repair.

  3. Repair consistency. After the entrywise predictions in Eq. (8), what mechanisms coordinate tool replacements or insertions with the corresponding dependency changes? Are compatibility checks applied beyond chain rewriting and rescoring?

  4. Self-confirmation in the repair loop. The same denoiser determines whether a plan should be edited, proposes a repair, and scores candidates for acceptance. How do the authors mitigate self-confirming errors, where a candidate receives a lower brokenness score without becoming more correct? Table 11 already illustrates this failure mode. Beyond the reported examples and selected counts, could the authors quantify across configurations how often accepted repairs lower ΣqP\Sigma_q^P but worsen reference-based plan quality? Are any independent checks used to detect such cases?

  5. Valid but unfamiliar plans. How does the brokenness score behave on correct tool combinations that were absent from training? Have the authors examined false alarms or harmful repairs on such plans within the same tool catalog? This would help determine whether the model distinguishes unfamiliarity from incorrectness.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.