- Rating
- 2
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
The paper studies joint training of a generator and a verifier for structured tasks. The verifier either accepts an answer or raises a safety assurance case (SAC) containing a claim, argument, and evidence. The generator can then keep its answer or adopt a revision. DPA-GRPO trains both roles using paired SAC/no-SAC and KEEP/REVISE decisions, correctness rewards, and KL regularization. The paper analyzes local equilibrium conditions and training dynamics under stated assumptions. Experiments on TaxCalcBench TY24 and a FinQA test subset report higher final accuracy than generator-only RL, with additional paired-training controls, ablations, and analysis of intervention and revision behavior.
Critical strengths
-
The FinQA results provide evidence that the trained interaction can improve final answers. On the 100-question test subset, DPA-GRPO reaches 37% accuracy, compared with 29% for GRPO, the strongest generator-only baseline (Table 1). The reported paired-bootstrap 95% confidence interval for the gain is +2 to +14 percentage points (Appendix E.5). This supports the practical value of the full procedure on the evaluated subset.
-
Learning when to intervene and when to accept a revision is a useful design choice. The paired SAC/no-SAC and KEEP/REVISE rewards explicitly train these decisions using answer correctness (Sections 2.2–2.3). This addresses the risk that verifier feedback introduces errors into a correct answer. Section 4.3 illustrates why this matters: adding the untrained interaction loop reduces TaxCalcBench accuracy from 41.4% to 38.8%.
Critical weaknesses
-
The added value over prior joint training methods is unclear. PAG (Jiang et al., 2025) already trains generation, verification, and selective revision. RL Tango (Zha et al., 2025) jointly trains a generator and a generative verifier with RL, while MALT (Motwani et al., 2024) trains generation, critique, and refinement. These closely related works are missing from the Introduction and Appendix A and are not compared experimentally. This makes it hard to assess what the paired actions, KEEP/REVISE decision, and SAC format add beyond existing methods. The contribution of training each role is also unclear. Section 4.3 compares the untrained interaction loop with joint training, while the generator-only baselines in Table 1 use a different inference procedure. There are no matched comparisons that train only G with V frozen or only V with G frozen. The results therefore do not show how much training each role contributes, or whether updating both is needed to obtain the reported gains.
-
The verifier reward does not assess SAC quality. In Section 2.2, raising an SAC receives reward 1 whenever the initial answer is wrong, regardless of whether the claim, argument, or evidence is valid. An incorrect explanation can therefore accompany a rewarded intervention. The intervention precision in Table 3 only measures whether the flagged initial answer is wrong. It does not establish that SACs provide reliable explanations, which is central to the paper's assurance framing.
Questions for authors
-
What does DPA-GRPO add beyond PAG, RL Tango, and MALT? Can you compare against a close baseline such as PAG under matched conditions, and isolate the gains from paired-action updates and the additional KEEP/REVISE decision?
-
Can you compare (i) training only G with V frozen, (ii) training only V with G frozen, and (iii) training both, using the same backbone, data, and inference procedure with comparable training budgets? How much does training each role contribute to final accuracy?
-
Does any reward or loss directly assess the content of an SAC? Beyond detecting that the initial answer is wrong, what evidence shows that the SAC's claim, argument, and supporting evidence are correct?
2 comments
Thank you for the additional experiments. They help clarify the contributions of the generator and verifier. However, my main concern about novelty remains. Joint generator–verifier training has been explored in closely related work, and I consider DPA-GRPO’s additional contribution limited. I will maintain my original score.
We thank the reviewer for the careful review and questions.
Response to what DPA-GRPO adds beyond PAG, RL Tango and MALT.
All three are now discussed in Section 2. DPA-GRPO sits in the jointly trained category but differs in three respects. First, intervention and acceptance are both explicitly trained decisions — the verifier can stay silent, and the generator chooses KEEP or REVISE after seeing the candidate revision; in PAG and MALT the revision always replaces the answer, and in RL Tango the verifier only supplies a training signal. Second, both decisions are updated from paired counterfactual rewards on the same draft via a two-sample GRPO step with a role-specific KL anchor. Third, the feedback takes the form of a structured claim–argument–evidence record that can be checked against the input.
To isolate the KEEP/REVISE decision, we replay a "revise-whenever-flagged" rule (the PAG policy) on identical interactions from our runs — same drafts, same SACs, same candidate revisions — changing only the final decision. On TaxCalcBench (Qwen3-4B), the learned policy gets 35 more decision units correct than always-revise, because it keeps correct drafts that the verifier challenged.
Table 3(a) compares DPA-GRPO with Paired-CE and Paired-DPO over the same decisions, supporting the paired-action update specifically. We did not reproduce PAG's full training pipeline in the rebuttal period; the replay isolates the inference-time difference on identical interactions.
Manuscript revision. Section 2 now discusses PAG, RL Tango and MALT.
Response to training only the generator, only the verifier, or both.
Table 3(b) reports this matrix on 300 FinQA test questions. All rows use the same backbone, training data, learning rate, and inference loop. Training both roles gives the highest accuracy (63.0% after interaction, +3.0 pp over the untrained loop). Training one role alone gives +1.0 pp. The verifier-only run raises intervention precision from 67.6% to 74.2%; the generator-only run changes the acceptance decision (keeps its draft after 27 of 34 SACs vs. 3 of 34 untrained) and loses no correct draft to a revision.
On GSM8K the trained verifier is able to find the correct answer on 1,132 of 1,319 questions versus 1,093 for the untrained verifier with Qwen3-4B, and 1,079 versus 1,033 with Ministral-8B.
Manuscript revision. Table 3(b) and the role-ablation discussion in Section 4.3 are new.
Response to whether any reward assesses SAC content.
No reward term scores the SAC text. The verifier's paired reward depends only on whether the intervention decision is correct (Section 3.2). SAC content enters the game indirectly: the revision is conditioned on the SAC and scored by S_z, and KEEP/REVISE compares S_x with S_z.
To measure whether the content is nonetheless correct, we audited every SAC programmatically. On held-out FinQA (77 SACs over 300 questions), 93.5% of claims correctly identify a wrong draft and 100% of the 272 cited evidence values occur in the source report. On GSM8K with Ministral-8B (1,319 questions), the revision is correct for 100 of 127 flagged wrong drafts whose SAC contains the correct answer (79%).
The intervention precision in Appendix D.2 measures the claim; the audit above adds evidence and argument measurements. A content-aware reward is a direct extension we have not yet run.