acceptodds
Under review as a conference paper at ICLR 2027

Reinforcement Learning via Structured Deliberation and Fine-Grained Attribution

Abstract

Reinforcement learning with verifiable rewards (RLVR) improves the reasoning capabilities of large language models by optimizing policies against a single reward signal derived from the final outcome. However, real-world reasoning tasks often involve multiple valid answers or more open-ended objectives that a single outcome signal cannot adequately capture or distinguish. In such settings, outcome-only RLVR assigns a homogeneous advantage to the entire reasoning trajectory, ignoring both the complex reasoning structure and the heterogeneous contributions of individual components. We introduce **s**tructured **o**ptimization via **d**eliberation and **a**ttribution (SODA), a framework that addresses these limitations through structured deliberation and fine-grained attribution. SODA first makes the structure of complex reasoning explicit by decomposing intermediate reasoning into semantically meaningful, locally evaluable decision units, and then attributes credit to each individual unit based on its contribution in the context of the remaining response. These credits are computed via local edits, yielding differentiated training signals without additional policy rollouts or a learned critic. We instantiate SODA on set-valued QA, where each question may have a different number of semantically distinct answers. Reasoning is structured into complementary stages of candidate proposal and selective refinement to produce correct and non-redundant answers. Experiments across multiple datasets and base models show consistent improvements over state-of-the-arts, with ablations demonstrating complementary gains from structured deliberation and fine-grained attribution.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Reinforcement Learning with Verifiable Rewards, Fine-Grained Credit Assignment, Structured Deliberation, Set-Valued Generation, Large Language Models

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_2kgKOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes SODA, a reinforcement learning framework that combines structured deliberation with fine-grained credit assignment. It organizes reasoning into candidate generation and selection stages, assigning decision-specific rewards based on changes in task utility under local edits. The method is evaluated on set-valued question answering tasks, where experiments across multiple datasets and language models show improvements over GRPO and other baselines. Ablation studies further demonstrate the benefits of fine-grained credit assignment in both stages.

Critical strengths

The paper addresses the important challenge of assigning fine-grained rewards to different decisions within a reasoning process, rather than relying solely on outcome-level rewards.

Critical weaknesses
  1. The method relies on explicitly structured and independently editable decision units, making its applicability to complex reasoning tasks with interdependent steps unclear.
  2. The approach primarily combines structured reasoning with local counterfactual reward assignment. While effective, its technical novelty is relatively limited, and the experiments focus exclusively on set-valued QA tasks.
Questions for authors
  1. Can the authors provide evidence that SODA is effective beyond set-valued question answering, particularly in tasks involving multi-step reasoning?
  2. How does SODA handle tasks with multiple valid reasoning paths or intermediate decisions that cannot be independently evaluated?
  3. How sensitive is SODA to the predefined reasoning structure and the number of candidate answers?
5 comments

We thank the reviewer for the careful reading and constructive feedback. We have conducted additional analyses and revised our explanations to clarify the scope and practical requirements of SODA. We have incorporated these changes into the revised manuscript and highlighted all revisions in blue for ease of reference. Below, we respond to each comment in detail.

W2 & Q1: Technical novelty and generalization beyond QA.

Technical contribution. Our manuscript and the proposed method discuss a comparatively underexplored problem in RLVR. Classic RLVR often focuses on producing a single correct answer, while the desired output of some problems is a set whose size varies across inputs. The elements in the set can be interdependent and must be evaluated jointly. For example, a redundant answer adds no new coverage, while including more candidates can improve coverage at the cost of precision. Our motivating study in Section 2 shows that prompting the base model alone does not reliably adapt the set size to the input to achieve a balance between coverage and precision.

We contribute by formulating the learning problem of set-valued generation, empirically characterizing the base model’s difficulty in cardinality adaptation, and introducing SODA, a GRPO variant that addresses this problem by assigning separate credits to proposal and selection decisions. While classic GRPO assigns a homogeneous advantage to the entire reasoning trajectory, we first consider fine-grained reasoning structure and develop the heterogeneous contribution assignment for individual decision components in SODA.

Evidence beyond QA. SODA can apply directly to tasks beyond QA whenever the goal is to generate a set of answers and the task provides a verifiable utility for scoring that set. Our MIMIC-III experiment demonstrates this in clinical diagnosis coding, where the model reads a patient’s clinical record and selects the applicable diagnoses from a predefined ICD code dictionary. We observed the model integrating multiple clinical findings to propose a diagnosis and then weighing competing explanations before deciding whether to retain it. Thus, a single proposal or verdict can reflect several reasoning steps, even when those steps are expressed in free-form text. We include examples in the appendix of the revised manuscript.

W1 & Q2: Applicability to other reasoning tasks.

We thank the reviewer for raising this question. SODA’s two-stage structure for candidate proposal and selection is compatible with complex, multi-step reasoning within either stage. As in the MIMIC-III example discussed above, the model can integrate several clinical findings during thinking to propose a diagnosis, then weigh competing explanations during rethinking. The units SODA evaluates through local edits are the outcomes of these reasoning processes: a candidate proposal or a keep/drop verdict. Below, we explain how SODA applies in each of the two settings raised by the reviewer.

(1) Intermediate decisions that cannot be independently evaluated. Whether SODA applies depends on what the task asks the model to produce and how a candidate is defined. If the task requires a single executable trajectory and each action is treated as a candidate, removing an action may make later actions infeasible. Such an edit fails the admissible-edit condition in Section 3.1, so SODA cannot use it to assign local credit to that action. This reflects a broader challenge for outcome-based RLVR: a verifiable final reward can supervise an entire trajectory, but does not by itself establish the contribution of each intermediate action. SODA addresses local attribution when admissible edits are available; it does not resolve arbitrary sequential dependencies. If the task instead calls for a set of verifiable action sequences, each complete sequence can serve as a candidate. Removing one sequence preserves the dependencies within every remaining sequence, allowing its contribution to the set-level utility to be evaluated without downstream regeneration. Examples include proposing alternative plans for implementing a solution and generating a set of test sequences that collectively covers the required behaviors of a system.

(2) Multiple valid reasoning paths. In SODA, reasoning paths are expressed in the THINKING and RETHINKING spans. The model is flexible to approach the problem in different ways, so SODA does not require a unique reasoning path. Different paths can lead to the same candidate, and a single path can yield several distinct candidates. Our method provides fine-grained supervision for the proposals and verdicts the model produces through reasoning, without prescribing how the model generates the reasoning trace.

Q3: Sensitivity to reasoning structure and candidate count.

To assess sensitivity to these design choices, we added comparisons of alternative reasoning structures and an additional training candidate count.

(1) Sensitivity to the reasoning structure. We evaluated two additional reasoning structures: one in which the model verifies each candidate immediately after generating it, and one in which the model generates the full candidate pool and issues keep/drop verdicts without a separate reflection passage. For both variants, we use the training configuration from Table 1 and adjusted the prompt and credit-to-span mapping to match the output structure. Among these structures, SODA’s proposed two-stage structure achieved the highest final-set F1F_1.

While pool recall differed by only 1.15 points across the three structures, the main difference emerged during the selection stage. These results provide evidence that while fine-grained credit assignment brings benefits to multiple reasoning structures, the proposed organization of SODA offers additional advantages.

Our motivation for separating proposal from reflection is that candidates must be evaluated jointly: their contributions depend on overlapping coverage, redundancy, and the precision–recall tradeoff. Generating the complete pool before reflection allows selection decisions to account for these interactions. Omitting reflection retains credit on individual verdicts but removes the explicit reasoning passage through which the model can learn to reconsider candidate interactions before making those decisions. These results and analyses are included in the revised manuscript in Appendix E.4, Table E.4.

StructurePool recallF1F_1PrecisionFinal-set recallMean ∣S∣\lvert S\rvertSpearman ρ\rho
Proposed two-stage structure62.80%65.70%82.10%58.00%6.480.89
Immediate verification62.62%61.81%78.98%54.21%6.070.84
No separate reflection61.65%52.63%78.76%43.54%4.830.70

(2) Sensitivity to the candidate count. To investigate the sensitivity to the predefined candidate count, we additionally train SODA with M=10M{=}10 and evaluate it with candidate budgets ranging from 5 to 30. F1F_1 remains stable across different inference budgets, except for Minference=5M_{inference}{=}5. Although the M=10M{=}10 model performs below the original M=15M{=}15 setting, using only 10 candidates still exceeds all baselines reported in the main table, showing that SODA retains its performance advantage with different candidate counts and does not require precise inference-time tuning once the candidate budget is sufficient. These results are reported in the revised manuscript in Appendix E.5, Table E.6.

Test MPool recallF1F_1PrecisionRecallMean ∣S∣\lvert S\rvertSpearman ρ\rho
50.4500.4990.8410.3883.450.72
100.5530.5840.7810.4975.460.86
150.5690.5890.7560.5135.990.86
200.5700.5870.7400.5206.230.85
300.5690.5800.7310.5166.280.84

We sincerely hope that these clarifications and discussions satisfactorily respond to your comments. All these discussions and additional results are now part of the revised paper. We welcome any further questions or insights, and thank you once again for your vital contributions to our work!

Dear Reviewer,

As the discussion period is nearing its end, we wanted to check whether we have adequately addressed your concerns. If you have any remaining questions or feedback, please let us know. We would be happy to address them.

Thank you again for your time and valuable feedback!

Reviewer_86DuOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces Structured Optimization via Deliberation and Attribution (SODA), instantiated on set-valued question answering. Responses are organized into candidate proposal and selective refinement. Local credits are computed by deleting candidates or flipping keep/drop decisions and re-evaluating the resulting set, while stage-level utilities provide credit for broader reasoning segments. The procedure requires neither additional policy continuations nor a learned critic. Validation includes DDXPlus, ProtoQA, BioASQ, and MIMIC-III, additional backbones, component ablations, and a theoretical analysis of fixed candidate sets. The strongest evidence is on DDXPlus, where SODA improves over direct and structured GRPO and where the local-credit ablation shows a substantial gain. Some implementation and external-comparison details remain consequential to the assessment.

Critical strengths
  1. An operationally concrete form of fine-grained attribution. Candidate-deletion and selection-flip utilities define reproducible local signals tied to proposal coverage and final set quality. The approach avoids both extra policy continuations and a learned process critic. The theoretical analysis also explicitly stops short of claiming policy-optimization convergence. This is a useful methodological contribution for tasks with locally editable, verifiable output structure.

  2. Strong main-task evidence and an informative local-credit ablation. Table 1 reports DDXPlus F1 of 0.657 for SODA, compared with 0.576 for direct GRPO and 0.531 for structured GRPO. Table 4 retains stage utilities while removing both local-credit components and obtains 0.570; restoring the local credits yields 0.657. These comparisons provide evidence beyond a prompt-format change or stage-level reward decomposition, and they are the strongest reason to consider acceptance.

Critical weaknesses
  1. The manuscript does not establish that training attribution and evaluation operate on the same final set. Equations (5)–(6) on p.6 define the selected set through keep verdicts, whereas Appendix B.4 on p.18 states that evaluation reads the last FINAL SET line. A prompt requesting consistency is not itself a guarantee that these objects agree. The paper does not sufficiently specify mismatch handling or report the mismatch rate. If they differ, the locally attributed utility may not be the utility ultimately evaluated. This may be only a documentation omission if the implementation deterministically reconstructs the final set, but that needs to be established because it is central to the claimed attribution mechanism.

  2. The evidence for broader superiority is weaker than the main-task result. Cross-task and additional-backbone comparisons use structured GRPO, although direct GRPO is stronger on the primary benchmark. Thus, those results do not establish that SODA outperforms the strongest simple alternative outside DDXPlus. For open-ended answers, normalized exact matching also leaves uncertainty about how much gain reflects semantic answer-set improvement versus canonical wording; exact matching is a legitimate protocol, but an independent semantic check would strengthen that interpretation. The single training seed further limits the stability claims, and the paper correctly notes that checkpoint-conditional bootstrap intervals do not cover retraining variability. Together, these limitations constrain the breadth of the claimed reasoning improvement.

Questions for authors
  1. Clarification: What exact object supplies the training outcome reward, each local credit, and the final evaluation score when keep verdicts and FINAL SET disagree? Is the final set deterministically reconstructed? Please report disagreement, malformed-output, and truncation rates, and whether consistent re-scoring changes the main comparison.

  2. Additional evidence that could change my rating: How does SODA compare with direct GRPO on at least one non-DDXPlus task and one additional backbone under matched training and generation budgets? A targeted comparison is more decision-relevant than adding further structured-GRPO-only results.

  3. Additional evidence that could change my rating: Do the main gains survive repeated training and an independent alias-aware or human semantic check of the predictions that differ between methods? I am particularly interested in whether the local-credit benefit remains after separating canonical-string matches from semantically correct answers.

3 comments

We thank the reviewer for the careful reading and constructive feedback. We have conducted additional analyses and revised our explanations to clarify the scope and practical requirements of SODA. We have incorporated these changes into the revised manuscript and highlighted all revisions in blue for ease of reference. Below, we respond to each comment in detail.

W1 & Q1: Final-set consistency.

In our implementation, both Fε(S)F_\varepsilon(S) and Δjrethink\Delta_j^{\text{rethink}} use the set reported in FINAL SET. Thus, the two calculations thus share the same reference set, helping maintain consistency between the rethinking and output stages.

Still, we agree that without constrained decoding or an explicit consistency check, a model’s verdicts may disagree with its FINAL SET line and we thank the reviewer for raising this point. To assess the frequency and impact of these disagreements, we audited the final Qwen3-8B checkpoint on the DDXPlus evaluate set and report this audit in Appendix E.7, Table E.8. For each output, we reconstructed the output set from their keep/drop verdicts, compared it with the reported set, and recomputed F1F_1.

MethodSet disagreementReported-set F1F_1Reconstructed-set F1F_1
SODA2.95%64.46%64.60%
Structured GRPO1.35%50.88%50.82%

Disagreements occurred in fewer than 3% of outputs for each method. Reconstructing the sets changed F1F_1 by at most 0.14 percentage points and preserved SODA’s advantage. While we did not retrain the model after filtering out inconsistent rollouts given the computational cost, this inconsistency can be solved by direct check at inference time. We report this audit and clarify the potential impact and solution to this generation inconsistency.

W2 & Q2: Direct GRPO comparisons beyond the main setting

We thank the reviewer for suggesting these comparisons. We added direct GRPO on MIMIC-III with Qwen3-8B and on DDXPlus with Qwen3-30B-A3B, covering a non-DDXPlus task and an additional backbone. As shown below, SODA achieves higher F1F_1 in both settings, reflecting improved recall and output-cardinality adaptation. These additional comparisons are now reported in Appendix E.8, Table E.9.

SettingPool recallF1F_1PrecisionFinal-set recallMean ∣S∣\lvert S\rvertSpearman ρ\rho
Direct GRPO on MIMIC-III with Qwen3-8B—0.5930.5810.6735.400.706
SODA on MIMIC-III with Qwen3-8B0.8600.6060.5990.6795.140.699
Direct GRPO on DDXPlus with Qwen3-30B-A3B—0.2160.4900.1662.460.061
SODA on DDXPlus with Qwen3-30B-A3B0.5670.5000.6220.4806.350.589

W2 & Q3: Robustness to semantic evaluation

We thank the reviewer for suggesting an independent semantic check. Across our datasets, surface wording is less of a concern for MIMIC-III, where outputs are diagnosis codes, and for DDXPlus and BioASQ, where alias tables account for common variants of the reference answers. We therefore focused on ProtoQA, which lacks an alias table and admits open-ended answers.

We re-scored the predictions from the untrained model, structured GRPO, and SODA with DeepSeek-v4-pro as an independent judge to assess whether a predicted answer and a reference answer expressed the same meaning.

While semantic matching raised F1F_1 for all models, SODA’s lead over structured GRPO and its gain over the untrained model remained substantial. These results indicate that SODA’s gains on ProtoQA persist when semantically equivalent answers receive credit, and that the gain reflects improved answer-set quality rather than merely strict adherence to canonical answer wording. We include the evaluation protocol and results in Appendix E.6 in the revised manuscript.

MethodExact-match F1Semantic-match F1
untrained19.73%34.90%
SODA33.16%47.42%
Structured GRPO31.20%44.38%

W2 & Q3: Stability across training seeds

Within the rebuttal time and computational budget, we provide another two seeds to assess the stability of SODA’s gains and include these results in the revised manuscript in Appendix E.9, Table E.10. The completed comparison reproduces SODA’s F1 advantage over direct GRPO, providing further evidence that the improvement is not specific to the original seed.

MethodSeedF1F_1PrecisionRecallMean ∣S∣\lvert S\rvertSpearman ρ\rho
SODA420.6570.8210.5806.480.890
SODA430.6360.7390.5967.260.867
SODA440.6350.8080.5546.190.883
Direct GRPO420.5760.7600.4926.010.870
Direct GRPO430.5580.7530.4725.710.849
Direct GRPO440.5330.7010.4615.990.828

We sincerely hope that these clarifications and discussions satisfactorily respond to your comments. All these discussions and additional results are now part of the revised paper. We welcome any further questions or insights, and thank you once again for your vital contributions to our work!

Dear Reviewer,

As the discussion period is nearing its end, we wanted to check whether we have adequately addressed your concerns. If you have any remaining questions or feedback, please let us know. We would be happy to address them.

Thank you again for your time and valuable feedback!

Reviewer_KTPyOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes SODA, which structures responses into candidate proposal and subsequent selection. It assigns local training signals using changes in answer coverage after candidate removal and changes in F1 after reversing keep/drop decisions. Experiments on four set-valued QA datasets show improvements in candidate coverage and final answer quality.

Critical strengths

The method distinguishes useful proposals from incorrect selection decisions, providing more targeted feedback than a shared response-level advantage.

Local credits require only rescoring, without additional policy rollouts or a learned critic.

Results across multiple datasets and base models support its effectiveness. Ablations show complementary benefits from proposal-level and selection-level credits.

Critical weaknesses

The method relies on tasks that fit the proposal–selection structure. All evaluated tasks involve answer-set prediction, leaving its applicability to sequentially dependent decisions unclear. For example, in a robotic task involving collecting a key, unlocking a door, and retrieving an object, removing the first action invalidates subsequent actions. These dependencies cannot be evaluated by simply editing one action while holding the rest fixed.

Requiring multiple candidates may obscure errors in the preceding reasoning. An incorrect analysis could still produce a candidate pool that happens to contain the correct answer. Although this candidate genuinely improves coverage, it does not validate the analysis, which may nevertheless receive positive stage-level credit. The paper does not examine the frequency or impact of such cases.

The choice of the two-stage structure remains somewhat heuristic. Existing ablations establish the usefulness of local credits within this structure, but do not establish its advantage over alternative structures, such as verifying each candidate immediately after generation or omitting the separate reflection passage.

Questions for authors

How would SODA define valid sequentially dependent decisions while retaining its advantage of avoiding downstream regeneration? 

How often does incorrect reasoning yield a candidate pool containing correct answers, and how often do these reasoning spans receive positive advantages? 

Under comparable sampling and token budgets, how does the proposed structure compare with immediate candidate verification or a variant without the separate reflection passage?

4 comments

We thank the reviewer for the careful reading and constructive feedback. We have conducted additional analyses and revised our explanations to clarify the scope and practical requirements of SODA. We have incorporated these changes into the revised manuscript and highlighted all revisions in blue for ease of reference. Below, we respond to each comment in detail.

W1 & Q1: Validity of local edits under sequential dependencies.

We thank the reviewer for raising this question. SODA focuses on RLVR tasks whose desired output is a set of varying cardinality. Its decision units are candidate proposals and keep/drop verdicts, each of which may be the outcome of several interdependent reasoning steps. Local edits change the resulting scoring object without deleting or rewriting the reasoning that produced it.

Therefore, whether SODA can be adapted to the sequentially dependent setting depends on how a candidate is defined and what the task asks the model to produce. If the task asks for a single executable trajectory and each action is treated as a candidate, removing each individual action can make the subsequent action infeasible. This edit fails the admissible-edit condition in Section 3.1, and SODA therefore cannot attribute credit to the action. This reflects a broader challenge for outcome-based RLVR: a verifiable final reward can supervise an entire trajectory, but does not by itself establish the contribution of each intermediate action. SODA addresses local attribution when admissible edits are available; it does not resolve arbitrary sequential dependencies. Instead, in settings where the task asks for multiple sequences, SODA could be adapted by treating each complete sequence as a candidate. This edit preserves the action dependencies within every remaining sequence, allowing the set-level utility to be recomputed without downstream regeneration. Examples include proposing alternative plans for implementing a solution and generating a set of test sequences that collectively covers the required behaviors of a system.

In response, we revised Section 3.1 to clarify how the choice of decision unit determines which local edits are admissible.

W2 & Q2: Positive credit for incorrect reasoning

We agree that an incorrect analysis can produce a useful candidate and that the preceding reasoning span can receive a positive stage-level advantage. However, this is an inherent limitation of outcome-based supervision shared by all RLVR methods that rely solely on outcome rewards. Prior work in mathematical reasoning documents cases where flawed reasoning yields a correct final answer, highlighting the need for process-level feedback when the goal is to verify the reasoning itself [1, 2].

While SODA provides finer-grained credit across response spans and enables the model to learn which candidates to explore and retain, these credits still derive entirely from outcome-based utilities. Quantifying how often this incorrect analysis occurs would require independently assessing the reasoning spans. Such process-level verification falls outside the scope of this study, which focuses on fine-grained credit assignment for verifiable rewards.

[1] Uesato, Jonathan, et al. (2022) "Solving math word problems with process-and outcome-based feedback." arXiv preprint arXiv:2211.14275.

[2] Lightman, Hunter, et al. (2024) "Let's verify step by step." International Conference on Learning Representations. Vol. 2024.

W3 & Q3: Other reasoning structures and justification for the two-stage structure

To examine the role of the proposed two-stage structure, we additionally consider two alternatives mentioned by the reviewer, verifying each candidate immediately after generation and omitting the separate reflection passage. Both variants use the exact same training configuration as in the main experiment (Table 1). We modify the requested output structure in the prompt and adapt the credit-to-output mapping accordingly.

Immediate verification achieves 61.81% F1F_1, exceeding the strongest baseline in the main table while remaining below SODA’s proposed structure. Omitting the separate reflection passage achieves comparable candidate-pool recall, but lower final-set recall and F1F_1, accompanied by a smaller average size of the output sets. These results and analyses are included in Appendix E.4, Table E.4.

StructurePool recallF1F_1PrecisionRecallMean ∣S∣\lvert S\rvertSpearman ρ\rho
Proposed two-stage structure62.80%65.70%82.10%58.00%6.480.89
Immediate verification62.62%61.81%78.98%54.21%6.070.84
No separate reflection61.65%52.63%78.76%43.54%4.830.70

Our motivation for separating proposal from reflection is that candidates must be evaluated jointly: their contributions depend on overlapping coverage, redundancy, and the precision–recall tradeoff. Generating the complete pool before reflection allows selection decisions to account for these interactions. Omitting reflection retains credit on individual verdicts but removes the explicit reasoning passage through which the model can learn to reconsider candidate interactions before making those decisions.

Together with our credit ablations, these results provide evidence that the benefits of fine-grained credit assignment extend to an alternative reasoning structure, while the proposed organization of SODA offers additional advantages.

We sincerely hope that these clarifications and discussions satisfactorily respond to your comments. All these discussions and additional results are now part of the revised paper. We welcome any further questions or insights, and thank you once again for your vital contributions to our work!

Dear Reviewer,

As the discussion period is nearing its end, we wanted to check whether we have adequately addressed your concerns. If you have any remaining questions or feedback, please let us know. We would be happy to address them.

Thank you again for your time and valuable feedback!

Thank you to the authors for their response! The method relies on a predefined propose–select structure, limiting its applicability. In multi-hop search QA, earlier findings determine subsequent queries, making local edits with downstream decisions held fixed insufficient to assess a decision’s contribution. While I appreciate the authors’ clarification, I remain concerned about the method’s generalizability and therefore maintain my score.

Comments on the paper

2 comments

We thank the reviewers for their thoughtful feedback. We notice several common questions about the scope of SODA, and would like to clarify the learning problem SODA addresses and the level at which it assigns credit.

Contribution and Scope. SODA is a variant of group relative policy optimization (GRPO) for reinforcement learning with verifiable rewards (RLVR), and is particularly well suited when the desired output is a set whose size varies across inputs. In this setting, the model must balance coverage and precision by learning the appropriate set size for each input. This challenge receives less attention in standard RLVR settings, where success is often defined by the correctness of a single answer. However, many reasoning tasks require several complementary answers, such as explaining an observation, identifying potential causes of a failure, or proposing feasible solutions to a problem. Such tasks require the model to go beyond generating individually correct answers and assess whether those answers, taken together, adequately address the question. Our motivating study in Section 2 shows that prompting the base model alone does not reliably resolve this trade-off or adapt the size of the final set to the input. SODA rewards proposals for the coverage they add and selection decisions for their effect on final-set quality, allowing the model to explore broadly while learning which candidates to retain. In the revised manuscript, we emphasize why this setting matters for RLVR, and clarify the range of tasks SODA supports and improves performance over GRPO.

Decision Units in SODA. In the set-valued setting described above, SODA structures deliberation around two decisions for each candidate: proposing it and deciding whether to retain it. Each proposal or verdict is a decision unit and it can be the outcome of several interdependent reasoning steps. This structure specifies where SODA assigns local credit while leaving the model free to reason through each decision. In MIMIC-III, for example, the model may integrate several clinical findings to propose a diagnosis, then reconsider it in light of competing explanations before deciding whether to retain it.

This formulation extends beyond QA to reasoning tasks in which the desired output is a set and the effect of adding or removing an element can be evaluated with the other elements held fixed. The edit changes which elements the set contains while leaving intact every step of the reasoning that produced them. In the revised manuscript, we clarify what constitutes a decision unit in SODA and how this definition accommodates multi-step reasoning for each candidate. We illustrate this relationship with examples in Appendix F in the revised manuscript.

Additional Results. In response to the reviewers’ comments, we conducted additional analyses and summarized them as follows:

(1) Reasoning structure (to reviewer 2kgK, KTPy). Complementary ablations in Appendix E.4 report the performance of immediate candidate verification and omitting reflection, illustrating the necessity of our proposed reasoning structure.

(2) Generalization (to reviewer 86Du). Additional direct-GRPO comparisons test whether SODA’s gains extend to another task and backbone. SODA achieved higher final-set F1 than direct GRPO in both comparisons.

(3) Evaluation robustness (to reviewer 86Du). We reconstructed predictions from keep/drop verdicts and independently re-scored ProtoQA predictions for semantic equivalence. Both checks preserved SODA’s F1 advantage over structured GRPO. The results are provided in Appendix E.6 and E.7.

(4) Sensitivity to seeds and candidate budgets (to reviewer 86Du, 2kgK). Retraining with two new seeds reproduced SODA’s gain over direct GRPO. Training with fewer candidates and varying the inference budget yielded similar F1 across all but the smallest tested budget.

These analyses clarify the role of SODA’s reasoning structure and show how its gains vary across tasks, backbones, evaluation protocols, and training and inference settings.

We sincerely thank the reviewers for their careful reading and thoughtful suggestions. Their feedback helped us clarify SODA’s scope and strengthen the empirical evaluation. We have incorporated these changes into the revised manuscript and highlighted all revisions in blue for ease of reference.

We thank all reviewers for their thoughtful feedback and Reviewer 2kgK for confirming that our response addressed their concerns. We believe our clarifications and additional experiments have also addressed the main concerns raised by the other reviewers.

As the discussion period is nearing its end, we would greatly appreciate hearing your thoughts on our responses before the discussion closes.

Thank you again for your time and consideration!

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.