acceptodds
Under review as a conference paper at ICLR 2027

A Two-Player Agentic Game for Verifiable Assurance Reasoning and Interactive Training

Abstract

In structured decision-making workflows such as form filling, compliance checking, and maintenance reporting, LLM outputs must be locally correct, globally consistent, and auditable against task-specific rules. Existing refinement methods often rely on heuristic debate, self-play, or LLM-generated supervision, creating a second-order assurance problem. We propose DPA-GRPO (Dual Paired-Action Group-Relative Policy Optimization), a paired-action training method for a two-player generator–verifier game with structured verifier interventions. The generator proposes outputs and may revise them when challenged; the verifier either remains silent or raises a safety assurance case (SAC) containing a claim, argument, and evidence. These SAC/no-SAC and KEEP/REVISE decisions induce paired counterfactual action groups, which DPA-GRPO uses for role-specific KL-regularized GRPO updates. We analyze the unregularized game and show that positive probability on strictly lower-reward intervention or revision actions creates a profitable unilateral deviation. Under standard stochastic-approximation assumptions, DPA-GRPO tracks the corresponding game ODE, whose isolated asymptotically stable limit points are stationary and candidate local equilibria under role-wise local optimality. Experiments on TaxCalcBench TY24 and FinQA show that DPA-GRPO improves structured decision accuracy over zero-shot generation and generator-only RL, with additional TaxCalcBench controls against static paired-training baselines. Training increases correct silent acceptance, reduces missed errors, and improves revision behavior.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM, Decision Making, Planning, Optimization

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_KjidOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
3
Read review
Paper summary

The paper studies joint training of a generator and a verifier for structured tasks. The verifier either accepts an answer or raises a safety assurance case (SAC) containing a claim, argument, and evidence. The generator can then keep its answer or adopt a revision. DPA-GRPO trains both roles using paired SAC/no-SAC and KEEP/REVISE decisions, correctness rewards, and KL regularization. The paper analyzes local equilibrium conditions and training dynamics under stated assumptions. Experiments on TaxCalcBench TY24 and a FinQA test subset report higher final accuracy than generator-only RL, with additional paired-training controls, ablations, and analysis of intervention and revision behavior.

Critical strengths
  1. The FinQA results provide evidence that the trained interaction can improve final answers. On the 100-question test subset, DPA-GRPO reaches 37% accuracy, compared with 29% for GRPO, the strongest generator-only baseline (Table 1). The reported paired-bootstrap 95% confidence interval for the gain is +2 to +14 percentage points (Appendix E.5). This supports the practical value of the full procedure on the evaluated subset.

  2. Learning when to intervene and when to accept a revision is a useful design choice. The paired SAC/no-SAC and KEEP/REVISE rewards explicitly train these decisions using answer correctness (Sections 2.2–2.3). This addresses the risk that verifier feedback introduces errors into a correct answer. Section 4.3 illustrates why this matters: adding the untrained interaction loop reduces TaxCalcBench accuracy from 41.4% to 38.8%.

Critical weaknesses
  1. The added value over prior joint training methods is unclear. PAG (Jiang et al., 2025) already trains generation, verification, and selective revision. RL Tango (Zha et al., 2025) jointly trains a generator and a generative verifier with RL, while MALT (Motwani et al., 2024) trains generation, critique, and refinement. These closely related works are missing from the Introduction and Appendix A and are not compared experimentally. This makes it hard to assess what the paired actions, KEEP/REVISE decision, and SAC format add beyond existing methods. The contribution of training each role is also unclear. Section 4.3 compares the untrained interaction loop with joint training, while the generator-only baselines in Table 1 use a different inference procedure. There are no matched comparisons that train only G with V frozen or only V with G frozen. The results therefore do not show how much training each role contributes, or whether updating both is needed to obtain the reported gains.

  2. The verifier reward does not assess SAC quality. In Section 2.2, raising an SAC receives reward 1 whenever the initial answer is wrong, regardless of whether the claim, argument, or evidence is valid. An incorrect explanation can therefore accompany a rewarded intervention. The intervention precision in Table 3 only measures whether the flagged initial answer is wrong. It does not establish that SACs provide reliable explanations, which is central to the paper's assurance framing.

Questions for authors
  1. What does DPA-GRPO add beyond PAG, RL Tango, and MALT? Can you compare against a close baseline such as PAG under matched conditions, and isolate the gains from paired-action updates and the additional KEEP/REVISE decision?

  2. Can you compare (i) training only G with V frozen, (ii) training only V with G frozen, and (iii) training both, using the same backbone, data, and inference procedure with comparable training budgets? How much does training each role contribute to final accuracy?

  3. Does any reward or loss directly assess the content of an SAC? Beyond detecting that the initial answer is wrong, what evidence shows that the SAC's claim, argument, and supporting evidence are correct?

2 comments

We thank the reviewer for the careful review and questions.

Response to what DPA-GRPO adds beyond PAG, RL Tango and MALT.

All three are now discussed in Section 2. DPA-GRPO sits in the jointly trained category but differs in three respects. First, intervention and acceptance are both explicitly trained decisions — the verifier can stay silent, and the generator chooses KEEP or REVISE after seeing the candidate revision; in PAG and MALT the revision always replaces the answer, and in RL Tango the verifier only supplies a training signal. Second, both decisions are updated from paired counterfactual rewards on the same draft via a two-sample GRPO step with a role-specific KL anchor. Third, the feedback takes the form of a structured claim–argument–evidence record that can be checked against the input.

To isolate the KEEP/REVISE decision, we replay a "revise-whenever-flagged" rule (the PAG policy) on identical interactions from our runs — same drafts, same SACs, same candidate revisions — changing only the final decision. On TaxCalcBench (Qwen3-4B), the learned policy gets 35 more decision units correct than always-revise, because it keeps correct drafts that the verifier challenged.

Table 3(a) compares DPA-GRPO with Paired-CE and Paired-DPO over the same decisions, supporting the paired-action update specifically. We did not reproduce PAG's full training pipeline in the rebuttal period; the replay isolates the inference-time difference on identical interactions.

Manuscript revision. Section 2 now discusses PAG, RL Tango and MALT.

Response to training only the generator, only the verifier, or both.

Table 3(b) reports this matrix on 300 FinQA test questions. All rows use the same backbone, training data, learning rate, and inference loop. Training both roles gives the highest accuracy (63.0% after interaction, +3.0 pp over the untrained loop). Training one role alone gives +1.0 pp. The verifier-only run raises intervention precision from 67.6% to 74.2%; the generator-only run changes the acceptance decision (keeps its draft after 27 of 34 SACs vs. 3 of 34 untrained) and loses no correct draft to a revision.

On GSM8K the trained verifier is able to find the correct answer on 1,132 of 1,319 questions versus 1,093 for the untrained verifier with Qwen3-4B, and 1,079 versus 1,033 with Ministral-8B.

Manuscript revision. Table 3(b) and the role-ablation discussion in Section 4.3 are new.

Response to whether any reward assesses SAC content.

No reward term scores the SAC text. The verifier's paired reward depends only on whether the intervention decision is correct (Section 3.2). SAC content enters the game indirectly: the revision is conditioned on the SAC and scored by S_z, and KEEP/REVISE compares S_x with S_z.

To measure whether the content is nonetheless correct, we audited every SAC programmatically. On held-out FinQA (77 SACs over 300 questions), 93.5% of claims correctly identify a wrong draft and 100% of the 272 cited evidence values occur in the source report. On GSM8K with Ministral-8B (1,319 questions), the revision is correct for 100 of 127 flagged wrong drafts whose SAC contains the correct answer (79%).

The intervention precision in Appendix D.2 measures the claim; the audit above adds evidence and argument measurements. A content-aware reward is a direct extension we have not yet run.

Thank you for the additional experiments. They help clarify the contributions of the generator and verifier. However, my main concern about novelty remains. Joint generator–verifier training has been explored in closely related work, and I consider DPA-GRPO’s additional contribution limited. I will maintain my original score.

Reviewer_CuAyOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper introduces DPA-GRPO. This is a method that jointly trains a generator and a verifier model using GRPO-style approach with programmatic rewards and KL regularization to improve accuracy through structured challenging proposals and following revisions (or acceptance of the initial draft). The verifier learns when to challenge an answer using arguments; the generator learns whether to accept or reject a suggested correction. Experiments show improvements on FinQA and only small gains on TaxCalcBench.

Critical strengths
  • Interesting new approach under the assumptions that the verifier can make incorrect or misleading suggestions with the system being trained end-to-end.
  • Mathematically rigorous. The conceptual side of the algorithm is well explained and detailed. The appendix contains a detailed pseudo-code.
  • The empirical results are supported with rigorous discussions on theoretical guarantees and local Nash equilibrium. At the same time the theoretical discussions are appropriately bounded and the authors do not make strong claims on global convergence.
  • Decent additional analysis and ablation studies (including an ablation without KL regularization, and structured versus unstructured verifier feedback). The sections on Paired Uncertainty Analysis, Additional Training-Time Diagnostics and Verifier Intervention Statistic bring deeper understanding of the methodology.
Critical weaknesses
  • Unclear headline of the paper. Probably the main weakness of the paper: it is not clear what the main contribution of the paper is. Is it the joint training framework or the verifier model? Or a specific financial application demonstrated on FinQA and TaxCalcBench? Or theoretical guarantees?
  • The main paper is very dense: the method walkthrough and theoretical guarantees dilute the headline contribution (especially given that there are no strong global convergence guarantees). I'd prefer a more standard paper organization (introduction, related works, methodology, experiments, conclusion) as it allows to understand the contribution of the paper much faster.
  • Limited number of benchmarks studied (FinQA, TaxCalcBench). The technique with suggesting the revisions and accepting/rejecting them could be potentially applied in other domains too. I would expect more common datasets from RLVR literature such as GSM8K, AIME, HumanEval.
  • One family of models studied: Qwen3. How does the approach transfer to other models?
  • The improvement on TaxCalcBench is modest (but the authors acknowledge it in the paper).
Questions for authors
  • Why do scores for "Case 1 + 3" in Table 1 have +0 in all rows except "(a) Ours"?
  • Does the oracle only provide training rewards? Or does it also select the best-of-5 sampled revisions used in the KEEP/REVISE decision?
2 comments

We thank the reviewer for the detailed reading and suggestions.

Response to the headline contribution and paper organization.

The contribution is a training method: DPA-GRPO trains the two decisions of a generator–verifier interaction: whether the verifier should intervene and whether the generator should accept the resulting revision, from paired counterfactual rewards. The SAC format, the theory, and the benchmarks each serve this method rather than standing as separate contributions.

Manuscript revision. We have moved Related Work to the main text (Section 2) and extended it with trained multi-agent methods.

Response to the request for more benchmarks and a second model family.

We added GSM8K and MATH (Table 2), each evaluated with both Qwen3-4B and Ministral-8B-Instruct. DPA-GRPO gives the highest post-interaction accuracy in every setting.

Manuscript revision. Table 2, the Ministral results and the corresponding discussion in Section 4.2 are new.

Response to Q1.

Case 3 counts correct drafts that the verifier challenged and the generator kept. Zero-shot generation and generator-only RL have no verifier, so no SAC is raised and Case 3 is zero by construction. For DPA-GRPO, Case 3 is 0.01 (Qwen3-4B) and 0.13 (Qwen3-8B) on TaxCalcBench. On FinQA it is 0 because the trained verifier raised no SAC on a correct draft (false-positive rate 0.0%).

Manuscript revision. The Table 1 caption now states that Case 3 is zero because no verifier is present in those rows.

Response to Q2

The oracle provides training rewards only. The KEEP/REVISE decision is always made by the generator without oracle access. The candidate selection procedure is described in Appendix D.1.

Manuscript revision. Appendix D.1 now states this procedure explicitly.

Dear Reviewer CuAy,

Thank you for your feedback. We have addressed your concerns regarding the headline contribution, paper organization, benchmark and model coverage, and the two questions on Case 1+3 and oracle usage.

We would appreciate it if you could review these revisions and let us know whether your principal concerns have been addressed or if any further clarification would be helpful.

Thank you for your consideration.

Reviewer_YhofOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The paper proposes a variant of GRPO, called Dual Paired-Action GRPO, for training a two-player language model architecture, in which one player generates proposed solutions and the second player can optionally raise "safety assurance cases" that identify problems with the initial generation. There is some theory showing convergence (theorem 3). Empirically, this procedure significantly improves the performance of Qwen-8B on FinQA, and leads to small and statistically insignificant improvements on TaxCalcBench, which is a smaller dataset.

Critical strengths
  • Training for multi-agent architectures like debate and verification is currently underexplored, and the proposed approach seems basically sound.
  • The paper is clearly written and is forthright about limitations, such as large confidence intervals for TaxCalcBench and higher inference-time compute cost.
  • The taxonomy of cases is interesting, and provides some more qualitative explanation for the results table.
Critical weaknesses
  • The empirical support is pretty weak; it would be great to see another benchmark
  • Conceptually, I'm unclear on the scope of problems to which the method is applicable. Could any type of output be subject to an SAC critique? If so, why not try some conventional reasoning datasets like math?
  • When key sections like related work are put in the appendix, the page limit becomes meaningless. Research publication requires discussion of related work in the main text.

update: the revision fixes the related work issue and adds some math benchmarks

Questions for authors
  • Could any type of output be subject to an SAC critique? If so, why not try some conventional reasoning datasets like math?
  • Is there any prior work on fine-tuning for multi-agent critique / verification / iterative refinement? How does the DPA-GRPO relate?
3 comments

We thank the reviewer for the constructive feedback.

Response to the empirical support and the request for another benchmark.

We added GSM8K and MATH with two model families (Table 2) and a 300-question FinQA role ablation (Table 3(b)). DPA-GRPO has the highest post-interaction accuracy in every new setting, and training both roles outperforms training either one alone.

Manuscript revision. Table 2, Table 3(b) and Sections 4.2–4.3 are new.

Response to whether any output can be subject to an SAC critique, and why not math.

The method applies wherever the output decomposes into decision units that can be scored programmatically during training — a form line, a numeric answer, a function call, etc. It also requires a citable context so the SAC evidence can point to specific inputs. No oracle is needed at inference.

GSM8K and MATH satisfy these requirements. The game, paired rewards, and update rule carry over unchanged; only the prompts, evidence type, and answer checker are adapted to each task. Free-form outputs without a checkable unit are outside the current scope.

Manuscript revision. The Introduction now states this applicability requirement explicitly.

Response to related work placement and prior fine-tuning for multi-agent critique.

We agree that related work belongs in the main text. The revised Section 2 covers the trained multi-agent methods closest to ours: SCoRe (Kumar et al., 2024) and PAG (Jiang et al., 2025) for self-correction with RL; RL Tango (Zha et al., 2025) and prover–verifier games (Kirchner et al., 2024) for jointly trained generator–verifier pairs; MALT (Motwani et al., 2025) and multiagent finetuning (Subramaniam et al., 2025) for role-specialized pipelines.

DPA-GRPO sits in the jointly trained category but differs in three respects. First, intervention and acceptance are both explicit trained decisions — the verifier can stay silent, and the generator chooses KEEP or REVISE after seeing the candidate revision; in PAG and MALT the revision always replaces the answer, and in RL Tango the verifier only supplies a training signal. Second, both decisions are updated from paired counterfactual rewards on the same draft via a two-sample GRPO step with a role-specific KL anchor. Third, the feedback takes the form of a structured claim–argument–evidence record that can be checked against the input.

Manuscript revision. Related Work is now Section 2 of the main text.

References

These additional results and revisions address my main concerns, I will increase my score to weak accept.

Thank you for your comments and the effort you put into the rebuttal. Unfortunately, I stick to my rating.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.