- Rating
- 1
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes DUAL-BN, a framework for extracting probabilistic graphs from natural-language descriptions. The method combines bidirectional supervised fine-tuning (SFT) with Group Relative Policy Optimization (GRPO). The reinforcement learning reward integrates reference-graph fidelity, text–graph–text reconstruction consistency, and a complexity penalty.
Critical strengths
-
Meaningful research problem. Extracting structured probabilistic representations from natural language is a challenging and potentially useful task.
-
Promising empirical improvements. The proposed method improves Node F1 from 0.7745 to 0.9106 and mapped Edge F1 from 0.6762 to 0.8415 over SFT.
Critical weaknesses
-
The discrepancy between training and evaluation. During GRPO, the model generates all graph components jointly, whereas evaluation relies on a staged extraction procedure with reference-assisted filtering. Consequently, the reported scores do not directly measure the quality of complete graphs generated from text alone.
-
The individual contributions of the proposed components are insufficiently isolated. The experiments do not adequately establish which components are responsible for the observed improvements. GRPO without the complexity penalty yields only modest improvements over SFT, especially for Edge F1 (0.6762 to 0.6855). Most of the reported structural gains appear after introducing the complexity penalty.
-
An overly simplified task formulation. DUAL-BN is built directly upon PRISM-BN [5], adopting much of its evaluation protocol. PRISM-BN identifies accurate conditional probability estimation as one of the central challenges of the task. However, rather than directly addressing this challenge, DUAL-BN simplifies the original full multi-parent conditional probability tables into parent-specific pairwise tables, thereby reducing the complexity of probability reconstruction. Under this simplified setting, although DUAL-BN achieves lower KL divergence than SFT, it is difficult to determine whether the proposed method truly addresses the key challenges of the original benchmark.
-
Unclear substantive novelty and contributions over prior work. DUAL-BN primarily combines existing techniques, including bidirectional learning, cycle-consistency feedback, GRPO, and complexity regularization. Bidirectional text–graph transformation and cycle-consistency learning have already been explored in CycleGT [1] and Semantic Parsing with Dual Learning [2], while GRPO [3] and complexity regularization in Bayesian network learning [4] are also established techniques. Therefore, the paper should clarify what substantive methodological novelty or new mechanism is introduced beyond applying these existing techniques to the PRISM-BN task.
-
Although the authors disclose the use of generative AI to assist with manuscript writing, the paper has still not undergone sufficient human review and revision. First, the research motivation in the Introduction lacks clear logical connections. The second paragraph emphasizes the challenges of maintaining consistency among the components of probabilistic graphs, while the third paragraph directly shifts to the limitations of SFT's token-level loss without explaining the relationship between the two. Although the fourth paragraph introduces methods such as bidirectional supervision, cycle-consistency feedback, and GRPO, it does not clearly explain how these designs correspond to and address the specific challenges identified earlier. Second, the Introduction contains considerable defensive language. For example, before adequately establishing the research motivation, the authors emphasize that local pairwise conditional probability tables cannot constitute a complete Bayesian network. Furthermore, the Introduction reports detailed changes in multiple experimental metrics, which would be more appropriate in the experimental section.
[1] Guo, Q., Jin, Z., Qiu, X., Zhang, W., Wipf, D., & Zhang, Z. (2020). CycleGT: Unsupervised Graph-to-Text and Text-to-Graph Generation via Cycle Training. Proceedings of the 3rd International Workshop on Natural Language Generation from the Semantic Web (WebNLG+), 77–88.
[2] Cao, R., Zhu, S., Liu, C., Li, J., & Yu, K. (2019). Semantic Parsing with Dual Learning. Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, 51–64.
[3] Shao, Z., et al. (2024). DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models. arXiv preprint arXiv:2402.03300.
[4] Lam, W., & Bacchus, F. (1994). Learning Bayesian Belief Networks: An Approach Based on the MDL Principle. Computational Intelligence, 10(3), 269–293.
[5] Bhattacharya, A., Singh, N., Pokhriyal, N., & Vosoughi, S. (2026). PRISM-BN: A Controlled Corpus and Benchmark for Text-to-Parameterized Bayesian Network Extraction. arXiv preprint arXiv:2609.21673.
Questions for authors
-
Can the authors provide an end-to-end evaluation without reference-assisted filtering to verify whether the reported improvements still hold?
-
Why does DUAL-BN simplify full multi-parent conditional probability tables into pairwise tables, and how does this address the challenges identified by PRISM-BN [5]?
-
What is the substantive novelty of DUAL-BN over prior work, and why are the original PRISM-BN [5] extraction baselines and simpler structural regularization methods not included?
-
Can the authors provide ablations to demonstrate the independent contributions of bidirectional supervision, cycle-consistency feedback, and complexity regularization?
3 comments
4. Component ablations
We evaluated all configurations on the same 73 descriptions from five held-out PRISM-BN backbones using the original staged evaluation. The table includes the penalty-off run from Table 3.
| SFT | GRPO rewards | Excess penalty | Node F1 ↑ | Conditional State F1 ↑ | Mapped Edge F1 ↑ | Forward Pairwise KL ↓ | Bidir. KL ↓ | Coverage ↑ |
|---|---|---|---|---|---|---|---|---|
| Forward only | — | — | 0.7322 | 0.8126 | 0.6578 | 0.8712 | 0.7441 | 67.20% |
| Bidirectional | — | — | 0.7745 | 0.8738 | 0.6762 | 0.8707 | 0.6160 | 66.18% |
| Forward only | Fidelity | 0.05 | 0.7350 | 0.8562 | 0.7691 | 0.7229 | 0.5989 | 74.55% |
| Bidirectional | Fidelity | 0.05 | 0.8622 | 0.8371 | 0.7819 | 0.7332 | 0.5441 | 74.67% |
| Bidirectional | Fidelity + cycle | 0 | 0.8082 | 0.8837 | 0.6855 | 0.7189 | 0.5302 | 74.22% |
| Bidirectional (full DUAL-BN) | Fidelity + cycle | 0.05 | 0.9106 | 0.8486 | 0.8415 | 0.6594 | 0.5357 | 82.30% |
Coverage is the percentage of reference probability columns with comparable predictions.
- Bidirectional supervision. With the GRPO rewards and penalty fixed, Node F1 rose from 0.7350 to 0.8622 and mapped Edge F1 from 0.7691 to 0.7819, with trade-offs in conditional State F1 and forward KL.
- Cycle feedback. With the SFT checkpoint, fidelity reward, and penalty fixed, adding cycle feedback improved all six metrics. Coverage increased from 74.67% to 82.30%.
- Complexity regularization. With fidelity and cycle feedback fixed, adding the penalty improved mapped Edge F1 (0.6855 → 0.8415) and forward KL (0.7189 → 0.6594), while conditional State F1 decreased and bidirectional KL rose slightly.
We will restructure the introduction so that each challenge maps to a design choice:
- Cross-component consistency → a joint graph-level reward
- A token-level loss that is blind to graph errors → GRPO over whole graphs
- Information loss → cycle reconstruction
The pairwise-table discussion will move to the task definition, and detailed metrics will move to Results.
We hope these results address your concerns.
Thank you for the feedback. We added a matched SFT–DUAL-BN evaluation without reference filtering and controlled ablations of bidirectional supervision, cycle feedback, and complexity regularization.
1. Graph reconstruction without reference filtering
We evaluated SFT and DUAL-BN on the same 50 descriptions. Each stage passed its own predictions directly to the next. We used reference graphs only after generation for matching and scoring.
Without reference filtering, DUAL-BN improved Node F1 by 0.0721 and mapped Edge F1 by 0.0186, while lowering both KL scores. KL measures probability agreement on recovered edges with comparable tables.
Joint generation. We also compared SFT and DUAL-BN when both models generated all graph components together in a single pass from the input description.
Under joint generation, DUAL-BN improved Node F1 by 0.0533 and reduced both KL scores relative to SFT. State F1 decreased by 0.0220. Thus, the node recovery and probability agreement gains persisted in one-pass generation, with a trade-off in state recovery.
Mapped Edge F1 is the Edge F1 metric reported in the paper.
We align predicted nodes to reference nodes, then score directed edges after mapping their endpoints. Predicted edges with an unaligned endpoint are excluded, and reversed edges do not count as matches. In this evaluation, alignment is used only for scoring after generation.
2. Why DUAL-BN uses pairwise tables
DUAL-BN trains on the parent-specific pairwise tables provided by PRISM-BN [5, §3.2]. For a child with four binary parents, a full conditional probability table requires 16 distributions, one per joint parent assignment. The pairwise targets require eight, two per parent. This reduces the number of distributions to predict and lets us evaluate recovered edges when the model identifies only part of a node’s parent set. Our target captures parent-specific probabilities. Full Bayesian-network parameterization would also require modeling joint parent effects.
3. Novelty and baseline comparisons
CycleGT [1] uses back-translation to learn from unpaired text and graphs. Cao et al. [2] use validity and reconstruction rewards for semantic parsing. DUAL-BN builds on these ideas by training a model to recover graph structure and pairwise probabilities together. Each GRPO candidate jointly predicts variables, states, directed edges, and probability tables. The same candidate receives three scores:
The cycle ablation shows the value of the reconstruction signal. With the starting checkpoint, fidelity weight, and excess penalty fixed, adding cycle feedback raised mapped Edge F1 from 0.7819 to 0.8415 and lowered forward pairwise KL from 0.7332 to 0.6594.
We prioritized a comparison against the same extractor’s SFT starting checkpoint to measure what DUAL-BN training adds. The gains persisted when the extractor had to pass its own predictions between stages. PRISM-BN’s prompted results provide context, and its full-CPD KL scores measure a different target from our pairwise KL. For regularization, we used a simple excess-count penalty and switched it on and off while holding the other rewards fixed. A size cap at decoding or pruning after generation would test a different intervention: controlling graph size at inference. To summarize Prior work. CycleGT uses cycle training to exploit unpaired data through back-translation. Cao et al. apply reconstruction rewards to logical forms. What is new here. DUAL-BN uses text reconstruction as a reward on a jointly generated probabilistic graph that contains continuous parameters. That reward is scored together with structural and numerical fidelity on the same GRPO candidate. Why it matters. Token-level SFT does not penalize a small edit that reverses a dependency or shifts a probability, while a graph-level reward does. Evidence. The cycle ablation isolates this signal's contribution beyond fidelity.