- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper proposes SODA, a reinforcement learning framework that combines structured deliberation with fine-grained credit assignment. It organizes reasoning into candidate generation and selection stages, assigning decision-specific rewards based on changes in task utility under local edits. The method is evaluated on set-valued question answering tasks, where experiments across multiple datasets and language models show improvements over GRPO and other baselines. Ablation studies further demonstrate the benefits of fine-grained credit assignment in both stages.
Critical strengths
The paper addresses the important challenge of assigning fine-grained rewards to different decisions within a reasoning process, rather than relying solely on outcome-level rewards.
Critical weaknesses
- The method relies on explicitly structured and independently editable decision units, making its applicability to complex reasoning tasks with interdependent steps unclear.
- The approach primarily combines structured reasoning with local counterfactual reward assignment. While effective, its technical novelty is relatively limited, and the experiments focus exclusively on set-valued QA tasks.
Questions for authors
- Can the authors provide evidence that SODA is effective beyond set-valued question answering, particularly in tasks involving multi-step reasoning?
- How does SODA handle tasks with multiple valid reasoning paths or intermediate decisions that cannot be independently evaluated?
- How sensitive is SODA to the predefined reasoning structure and the number of candidate answers?
5 comments
Q3: Sensitivity to reasoning structure and candidate count.
To assess sensitivity to these design choices, we added comparisons of alternative reasoning structures and an additional training candidate count.
(1) Sensitivity to the reasoning structure. We evaluated two additional reasoning structures: one in which the model verifies each candidate immediately after generating it, and one in which the model generates the full candidate pool and issues keep/drop verdicts without a separate reflection passage. For both variants, we use the training configuration from Table 1 and adjusted the prompt and credit-to-span mapping to match the output structure. Among these structures, SODA’s proposed two-stage structure achieved the highest final-set .
While pool recall differed by only 1.15 points across the three structures, the main difference emerged during the selection stage. These results provide evidence that while fine-grained credit assignment brings benefits to multiple reasoning structures, the proposed organization of SODA offers additional advantages.
Our motivation for separating proposal from reflection is that candidates must be evaluated jointly: their contributions depend on overlapping coverage, redundancy, and the precision–recall tradeoff. Generating the complete pool before reflection allows selection decisions to account for these interactions. Omitting reflection retains credit on individual verdicts but removes the explicit reasoning passage through which the model can learn to reconsider candidate interactions before making those decisions. These results and analyses are included in the revised manuscript in Appendix E.4, Table E.4.
| Structure | Pool recall | Precision | Final-set recall | Mean | Spearman | |
|---|---|---|---|---|---|---|
| Proposed two-stage structure | 62.80% | 65.70% | 82.10% | 58.00% | 6.48 | 0.89 |
| Immediate verification | 62.62% | 61.81% | 78.98% | 54.21% | 6.07 | 0.84 |
| No separate reflection | 61.65% | 52.63% | 78.76% | 43.54% | 4.83 | 0.70 |
(2) Sensitivity to the candidate count. To investigate the sensitivity to the predefined candidate count, we additionally train SODA with and evaluate it with candidate budgets ranging from 5 to 30. remains stable across different inference budgets, except for . Although the model performs below the original setting, using only 10 candidates still exceeds all baselines reported in the main table, showing that SODA retains its performance advantage with different candidate counts and does not require precise inference-time tuning once the candidate budget is sufficient. These results are reported in the revised manuscript in Appendix E.5, Table E.6.
| Test M | Pool recall | Precision | Recall | Mean | Spearman | |
|---|---|---|---|---|---|---|
| 5 | 0.450 | 0.499 | 0.841 | 0.388 | 3.45 | 0.72 |
| 10 | 0.553 | 0.584 | 0.781 | 0.497 | 5.46 | 0.86 |
| 15 | 0.569 | 0.589 | 0.756 | 0.513 | 5.99 | 0.86 |
| 20 | 0.570 | 0.587 | 0.740 | 0.520 | 6.23 | 0.85 |
| 30 | 0.569 | 0.580 | 0.731 | 0.516 | 6.28 | 0.84 |
We sincerely hope that these clarifications and discussions satisfactorily respond to your comments. All these discussions and additional results are now part of the revised paper. We welcome any further questions or insights, and thank you once again for your vital contributions to our work!
Dear Reviewer,
As the discussion period is nearing its end, we wanted to check whether we have adequately addressed your concerns. If you have any remaining questions or feedback, please let us know. We would be happy to address them.
Thank you again for your time and valuable feedback!
We thank the reviewer for the careful reading and constructive feedback. We have conducted additional analyses and revised our explanations to clarify the scope and practical requirements of SODA. We have incorporated these changes into the revised manuscript and highlighted all revisions in blue for ease of reference. Below, we respond to each comment in detail.
Technical contribution. Our manuscript and the proposed method discuss a comparatively underexplored problem in RLVR. Classic RLVR often focuses on producing a single correct answer, while the desired output of some problems is a set whose size varies across inputs. The elements in the set can be interdependent and must be evaluated jointly. For example, a redundant answer adds no new coverage, while including more candidates can improve coverage at the cost of precision. Our motivating study in Section 2 shows that prompting the base model alone does not reliably adapt the set size to the input to achieve a balance between coverage and precision.
We contribute by formulating the learning problem of set-valued generation, empirically characterizing the base model’s difficulty in cardinality adaptation, and introducing SODA, a GRPO variant that addresses this problem by assigning separate credits to proposal and selection decisions. While classic GRPO assigns a homogeneous advantage to the entire reasoning trajectory, we first consider fine-grained reasoning structure and develop the heterogeneous contribution assignment for individual decision components in SODA.
Evidence beyond QA. SODA can apply directly to tasks beyond QA whenever the goal is to generate a set of answers and the task provides a verifiable utility for scoring that set. Our MIMIC-III experiment demonstrates this in clinical diagnosis coding, where the model reads a patient’s clinical record and selects the applicable diagnoses from a predefined ICD code dictionary. We observed the model integrating multiple clinical findings to propose a diagnosis and then weighing competing explanations before deciding whether to retain it. Thus, a single proposal or verdict can reflect several reasoning steps, even when those steps are expressed in free-form text. We include examples in the appendix of the revised manuscript.
We thank the reviewer for raising this question. SODA’s two-stage structure for candidate proposal and selection is compatible with complex, multi-step reasoning within either stage. As in the MIMIC-III example discussed above, the model can integrate several clinical findings during thinking to propose a diagnosis, then weigh competing explanations during rethinking. The units SODA evaluates through local edits are the outcomes of these reasoning processes: a candidate proposal or a keep/drop verdict. Below, we explain how SODA applies in each of the two settings raised by the reviewer.
(1) Intermediate decisions that cannot be independently evaluated. Whether SODA applies depends on what the task asks the model to produce and how a candidate is defined. If the task requires a single executable trajectory and each action is treated as a candidate, removing an action may make later actions infeasible. Such an edit fails the admissible-edit condition in Section 3.1, so SODA cannot use it to assign local credit to that action. This reflects a broader challenge for outcome-based RLVR: a verifiable final reward can supervise an entire trajectory, but does not by itself establish the contribution of each intermediate action. SODA addresses local attribution when admissible edits are available; it does not resolve arbitrary sequential dependencies. If the task instead calls for a set of verifiable action sequences, each complete sequence can serve as a candidate. Removing one sequence preserves the dependencies within every remaining sequence, allowing its contribution to the set-level utility to be evaluated without downstream regeneration. Examples include proposing alternative plans for implementing a solution and generating a set of test sequences that collectively covers the required behaviors of a system.
(2) Multiple valid reasoning paths. In SODA, reasoning paths are expressed in the
THINKINGandRETHINKINGspans. The model is flexible to approach the problem in different ways, so SODA does not require a unique reasoning path. Different paths can lead to the same candidate, and a single path can yield several distinct candidates. Our method provides fine-grained supervision for the proposals and verdicts the model produces through reasoning, without prescribing how the model generates the reasoning trace.