acceptodds
Under review as a conference paper at ICLR 2027

RACA: Role-Aware Credit Assignment for Multi-Agent LLM Reasoning

Abstract

Recent research has sought to enhance LLM reasoning through collaboration among specialized agents. Training such systems, however, requires accurate credit assignment across roles and decisions. Existing outcome-level reinforcement learning assigns the same terminal reward to every turn, allowing an incorrect action to receive positive credit when later agents repair it. We introduce Role-Aware Credit Assignment (RACA), which aligns credit with each role's responsibility and decision scale. RACA retains trajectory-level credit for controller decisions, evaluates worker actions using locally observable outcomes, and computes relative advantages only among turns matched by role, blackboard-derived context, and reward channel. For primary proposer responses, it separately routes solution-quality and assistance-seeking advantages to their corresponding token spans. All signals come from the original rollout, without an additional value model or counterfactual execution. Using Qwen3-8B, RACA achieves 64.5% accuracy on MATH Level 5 and 16.0% on AIME 2022–2026, outperforming all compared baselines on both benchmarks. Under the same adaptive multi-agent framework, it improves over Outcome GRPO by 6.4 and 6.0 percentage points, respectively. On MATH, RACA averages 7.4 model calls per problem, compared with 8.7–10.7 for the other adaptive methods.

Primary area:
reinforcement learning
Keywords:
multi-agent, post-traing, agentic rl

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_1UTWOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes RACA, a role-aware credit assignment method for multi-agent LLM reasoning, to address inaccurate reward assignment across collaborating agents. It combines role-specific local rewards, context-aware advantage grouping, and separate credit assignment for different token spans. Using Qwen3-8B, RACA achieves 64.5% accuracy on MATH Level 5 and 16.0% on AIME, outperforming the compared baselines.

Critical strengths
  1. Reasonable motivation and method design.

Traditional Outcome GRPO assigns the same final reward to different agents, which may not reflect their individual contributions. RACA assigns credit based on agent roles and reasoning contexts, and separates solution generation from assistance-seeking decisions. The design is reasonable and has some novelty.

  1. Relatively comprehensive experimental comparisons.

The paper compares RACA with relevant methods, including AT-GRPO and GiGPO, and provides ablation studies for its main components. RACA achieves a clear accuracy improvement on MATH Level 5 while using fewer model calls than other adaptive multi-agent methods.

Critical weaknesses
  1. Limited performance gains and generalization evidence.

RACA improves performance on MATH, but its accuracy is only 2.1 percentage points higher than Single-Agent GRPO + Self-Consistency. On AIME, the improvement over the strongest baseline is only 1.3 percentage points on a test set of 150 problems, making it difficult to establish whether the gain is stable and statistically significant. In addition, the experiments only use Qwen3-8B on mathematical reasoning tasks, without evaluation across different model sizes or other tasks.

  1. Reliance on manually designed rewards and insufficient evidence of effective collaboration.

RACA relies on manually designed reward rules and weights, which may not accurately reflect each agent's actual contribution. For example, when the Proposer gives an incorrect answer, requesting assistance receives a higher reward (−0.02) than not requesting assistance (−0.10), even if the request fails to correct the answer. This may encourage unnecessary interactions rather than decisions based on their actual value. Similarly, the Critic's reward partly depends on whether the answer is later corrected, although a successful correction does not necessarily mean that the Critic made a causal contribution.

Although the ablation studies support the effectiveness of the credit assignment components, RACA still requests assistance more frequently for correct answers (63%) than for incorrect answers (58%). This suggests that the learned interaction policy remains imperfect. Therefore, while the method improves accuracy, the evidence is insufficient to establish that it achieves accurate credit assignment and effective selective collaboration.

Questions for authors

The improvement on AIME over the strongest baseline is relatively small. Could the authors provide additional statistical analysis to clarify whether this improvement is stable and significant?

Could the authors compare RACA with stronger single-model reasoning baselines under a similar inference compute budget, such as Qwen3-8B Thinking, to better demonstrate the practical benefit of multi-agent collaboration?

3 comments

W01: Performance gains and generalization evidence

Thank you for your careful review and constructive feedback.

We have strengthened the AIME evaluation with an expanded 450-problem benchmark, three independent training seeds, and paired significance tests. We also clarify the accuracy–cost comparison with the strongest single-agent baseline.

On MATH, the 2.1-point improvement over Single-Agent GRPO + Self-Consistency is accompanied by fewer model calls and generated tokens. Table 3 reports:

MethodMATH-L5 accuracyLLM callsInput tokensOutput tokens
Single-Agent GRPO + Self-Consistency62.4%8.02,278.44,208.4
RACA64.5%7.43,617.42,014.3

Thus, RACA’s improvement is achieved with fewer calls and approximately 52% fewer output tokens, although it processes more input tokens. Within the same four-role framework, RACA additionally improves over Outcome GRPO by 6.4 percentage points, providing a controlled comparison of credit-assignment mechanisms.

For AIME, we agree that the original 1.3-point difference on 150 problems requires further statistical evidence. The expanded evaluation gives:

Training seedRACAGiGPODifferenceHolm-adjusted pp
4287/450 (19.3%)71/450 (15.8%)+3.56 pp0.0017
12397/450 (21.6%)79/450 (17.6%)+4.00 pp0.0016
202691/450 (20.2%)81/450 (18.0%)+2.22 pp0.0309
Mean ± SD20.4 ± 1.1%17.1 ± 1.2%+3.26 ± 0.93 pp—

The improvement is positive across all three seeds and statistically significant for each seed under two-sided exact McNemar tests with Holm correction across the three comparisons. Repeated evaluations of the same questions are not counted as additional independent problems.

Regarding generalization, AIME evaluates zero-shot transfer beyond the MATH training dataset: AIME evaluation problems are excluded from SFT and RL, and no solved AIME demonstrations are supplied at inference time. This provides cross-dataset evidence within mathematical reasoning. RACA’s credit-assignment formulation does not depend on Qwen-specific architectural features, but our empirical validation currently covers Qwen3-8B and mathematical tasks; broader cross-model and cross-task evaluation remains future work.

W02: Reward design and evidence of effective collaboration

The interaction advantage is computed separately for initially correct and incorrect answers, because its comparison groups are conditioned on ptp_t (Section 3.3). The reviewer’s comparison between −0.02 and −0.10 therefore concerns only the incorrect-answer groups. It does not determine the reward ordering for correct answers.

Initial answerInteraction outcomeInteraction reward
IncorrectRequest, successfully repaired+0.28
IncorrectRequest, repair unsuccessful−0.02
IncorrectNo request−0.10
CorrectRequest−0.07
CorrectNo request0

Within incorrect-answer groups, an unsuccessful request receives −0.02, compared with −0.10 for making no request; successful repair receives +0.28. This intentionally favors attempting assistance when the current answer is wrong, while rewarding successful assistance more strongly. Within correct-answer groups, the preference is reversed: requesting receives −0.07, whereas not requesting receives 0. These groups are normalized separately, so the reward design does not impose a uniform preference for interaction across both correctness conditions.

For the critic, Section 3.2 explicitly defines the repair signal as reflecting “observed local outcomes, not a counterfactual causal effect.” Eq. (2) combines correct error detection with the subsequent short-horizon repair outcome: false alarms are penalized, and the repair bonus is available only after a correctly flagged error. This provides locally grounded feedback from the original rollout without requiring counterfactual executions. A subsequent correction is therefore used as a training signal, not as proof that the critic independently caused the correction.

The empirical contribution of the credit-assignment mechanism is examined while holding the reward rules fixed. In Table 4, removing context conditioning reduces MATH/AIME accuracy by 4.7/2.7 points, and replacing separate functional channels with a single turn-level advantage reduces accuracy by 3.9/2.0 points. These comparisons show that the gains depend on how feedback is compared and applied, beyond introducing manually specified rewards.

Regarding interaction selectivity, Table 2 shows that RACA achieves the highest interaction rate on incorrect answers (58% versus 23–39%) and the smallest gap between correctness conditions (5 points versus 17–47 points) among the compared methods.

To further examine the remaining gap, we separate proposer-initiated requests across all reasoning rounds by their target, excluding forced interactions and mechanically scheduled calls:

Request targetRequest rate given an incorrect answerRequest rate given a correct answerFix rateBreak rate
Critic50.6%49.3%25.7%3.6%
Verifier7.4%13.7%12.0%1.2%

The request rates are conditioned on answer correctness before interaction. Fix and break rates measure correctness changes by the end of the requested interaction chain within the same reasoning round.

The higher overall interaction rate after correct answers is associated primarily with verifier requests, which are 6.3 percentage points more frequent after correct answers. Critic requests are instead 1.3 points more frequent after incorrect answers. Thus, the aggregate 63% versus 58% combines different target-selection behaviors. The outcomes also differ: critic-initiated chains exhibit a higher fix rate, while verifier-initiated chains exhibit a lower break rate.

Figure 3(b) provides complementary evidence from training: requests after correct answers decrease substantially, while requests after incorrect answers initially decline and then partially recover. The overall interaction rate also declines. The narrowing gap therefore accompanies reduced interaction on already-correct answers, rather than a uniform increase in assistance-seeking.

These results support improved interaction behavior relative to the baselines. We retain the limitation explicitly acknowledged in Section 4.3: RACA’s assistance policy remains imperfect.

Q01: AIME statistical analysis

We have added three-seed evaluation and question-level paired significance tests on the expanded 450-problem AIME benchmark. The complete results are:

Training seedRACAGiGPODifferenceRACA-only / GiGPO-only correctMcNemar ppHolm-adjusted pp
4287/450 (19.3%)71/450 (15.8%)+3.56 pp19 / 30.00090.0017
12397/450 (21.6%)79/450 (17.6%)+4.00 pp22 / 40.00050.0016
202691/450 (20.2%)81/450 (18.0%)+2.22 pp14 / 40.03090.0309
Mean ± SD20.4 ± 1.1%17.1 ± 1.2%+3.26 ± 0.93 pp———

We use two-sided exact McNemar tests, comparing the two methods’ predictions on the same questions within each seed. Holm correction is applied to the unrounded p-values across the three seed-specific comparisons. Means and sample standard deviations are computed across training seeds; repeated evaluations of the same questions are not treated as additional independent problems.

RACA improves over GiGPO for all three seeds, with gains ranging from 2.22 to 4.00 percentage points, and every comparison remains significant after correction. These results provide evidence of both training-seed consistency and statistical significance on the expanded benchmark.

These tests concern the expanded evaluation; we do not infer significance for the original 150-problem comparison from its aggregate means and standard deviations. Further evaluation details are provided in our response to Weakness 1.

Q02: Stronger single-model reasoning baseline

We have added the requested Qwen3-8B Thinking comparisons, evaluating both the SFT policy and its Self-Consistency variant with N=8N=8. The MATH Level-5 results are:

MethodAccuracyLLM callsInput tokensOutput tokens
Qwen3-8B Thinking (SFT)47.1%1.0284.8715.5
Qwen3-8B Thinking + Self-Consistency (N=8N=8)58.4%8.02,278.44,683.2
RACA64.5%7.43,617.42,014.3

Calls and token counts are averaged per problem.

Compared with Thinking + Self-Consistency, RACA improves accuracy by 6.1 percentage points, while using 7.5% fewer model calls and approximately 57.0% fewer generated tokens. RACA processes more input tokens because subsequent agents receive accumulated reasoning and feedback through the blackboard; the table reports this cost explicitly.

These additional evaluations address the requested Thinking-mode comparison and support RACA’s practical accuracy–cost benefits under the evaluated settings, alongside the controlled credit-assignment comparisons within the shared multi-agent framework.

Reviewer_AwzXOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Role-Aware Credit Assignment (RACA) to address the credit misassignment problem in multi-agent LLM reasoning, where outcome-based reinforcement learning assigns the same terminal reward to agents with different responsibilities. RACA introduces role-specific local rewards, computes relative advantages within groups matched by role, reasoning context, and reward channel, and further separates solution-generation and interaction-decision advantages at the token level for the proposer. The approach does not require an additional value model or counterfactual rollouts.

Experiments with Qwen3-8B demonstrate that RACA achieves 64.5% accuracy on MATH Level 5 and 16.0% on AIME, outperforming the compared baselines. Ablation studies, interaction analyses, and inference-cost comparisons further support the effectiveness of the proposed components.

Critical strengths
  • The paper presents a reasonable combination of role-specific rewards, context-conditioned advantage normalization, and functional-channel decomposition to improve credit assignment in multi-agent LLM reasoning. Separating solution-generation and interaction-decision advantages within the same proposer response is a useful design choice.
  • Under the same adaptive multi-agent framework, RACA improves over Outcome GRPO by 6.4% and 6.0% on MATH Level 5 and AIME, respectively. The ablation studies support the effectiveness of the proposed components, particularly context conditioning and functional-channel decomposition. RACA also achieves higher accuracy with fewer LLM calls than the other adaptive multi-agent baselines.
Critical weaknesses
  • All experiments are conducted with Qwen3-8B on math reasoning tasks, while RACA's reward design relies on verifiable intermediate answers and predefined agent responsibilities. It remains unclear how well the approach generalizes to other models or more open-ended tasks. Moreover, the improvement over the strongest AIME baseline, GiGPO, is relatively modest. Given the limited evaluation set of 150 problems and the reported variance, the robustness of this improvement is not fully established.
  • Several components of RACA are closely related to prior work on agent-wise and state-conditioned credit assignment. While the combination is reasonable, the functional decomposition also relies on predefined output structures and manually designed rewards.
  • Table 2 shows that the interaction rate is still higher for correct initial answers than for incorrect ones. This suggests that the model has not fully learned when additional assistance is necessary. While the results convincingly demonstrate the benefits of collaborative error correction, the evidence that interaction-specific credit assignment leads to appropriately selective collaboration is less clear.
  • It remains unclear whether the additional complexity and training cost of RACA are justified by these performance gains.
Questions for authors
  • What is the main methodological contribution of RACA beyond combining existing role-specific rewards and context-conditioned advantage estimation? In particular, how does functional-channel decomposition differ from applying separate reward signals to predefined output segments?
  • RACA uses fine-grained role-context-channel grouping. What proportion of turns are excluded from advantage estimation due to insufficient group size or near-zero reward variance? Does this sparsity affect different roles disproportionately?
  • Why does RACA still request interaction more frequently for correct answers than incorrect ones? Could the authors provide a breakdown of critic versus verifier requests and their respective contributions to correcting final answers?
  • Given the relatively small improvement over the strongest baseline on AIME, could the authors clarify how consistently this improvement holds across random seeds and discuss the method's sensitivity to reward design and predefined role configurations?
6 comments

W01: Generalization and robustness of the AIME improvement

Thank you for your careful review and constructive feedback.

RACA studies role-specialized multi-agent reasoning with verifiable feedback (Section 3.1). Predefined responsibilities and locally verifiable outcomes define this setting; our experiments test whether aligning credit with roles, contexts, and decision functions improves learning.

The mechanism is not specific to Qwen3-8B: role-specific rewards, context-conditioned groups, and token-level advantage routing require identifiable role outputs and suitable feedback, not Qwen-specific architecture. However, empirical validation currently covers Qwen3-8B on mathematical reasoning; cross-model and open-ended-task generalization remains to be evaluated.

AIME tests zero-shot transfer beyond the training dataset: training uses MATH, AIME evaluation problems are excluded from SFT/RL, and no solved AIME demonstrations are provided at evaluation.

The original 150-problem results, 16.0 ± 1.3% versus 14.7 ± 0.7%, do not alone establish significance. We retain the original AIME I/II problems from 2022–2026 and add 300 historical problems from 2010–2021, excluding SFT/RL overlaps, for 450 unique problems. Both methods use the same questions:

Training seedRACAGiGPODifferenceRACA-only / GiGPO-only correctMcNemar ppHolm-adjusted pp
4287/450 (19.3%)71/450 (15.8%)+3.56 pp19 / 30.00090.0017
12397/450 (21.6%)79/450 (17.6%)+4.00 pp22 / 40.00050.0016
202691/450 (20.2%)81/450 (18.0%)+2.22 pp14 / 40.03090.0309
Mean ± SD20.4 ± 1.1%17.1 ± 1.2%+3.26 ± 0.93 pp———

RACA gains 2.22–4.00 percentage points across all three seeds. Each comparison remains significant under two-sided exact McNemar tests with Holm correction across the three comparisons. Means and sample SDs are computed across seeds; repeated evaluations do not count as additional independent problems. These results support consistent AIME gains, while broader generalization claims remain limited to the evaluated models and tasks.

W02: Distinction from prior work and dependence on predefined structures and rewards

RACA differs in the unit of comparison and credit assignment (Section 2). Agent-wise normalization distinguishes agents; AT-GRPO groups by agent and turn position; GiGPO combines episode-level credit with recurring-state comparisons. RACA additionally separates decision functions within one response: a correct solution can accompany an unnecessary request, or an incorrect solution a useful request. Agent- or turn-level distinctions alone do not separate them.

Sections 3.2–3.3 introduce function-specific rewards, comparison conditions, and optimization spans. The solution channel evaluates pre-interaction correctness and excludes ptp_t from grouping, since conditioning on it would make binary rewards constant. The interaction channel includes ptp_t to compare assistance decisions under the same initial correctness condition. Advantages are independently normalized and routed to disjoint token spans. Eq. (8) averages each channel over its own span, preventing longer solution text from diluting the short interaction block.

Output structure identifies the decisions; rewards supply feedback. Neither specifies the comparison groups or policy updates. Applying one turn-level advantage to the same structured response still couples solution generation and assistance-seeking.

Table 4 tests this distinction while keeping architecture, turn-level rewards, training data, and optimization unchanged (Section 4.4):

VariantMATH-L5AIME
Without role conditioning62.4 ± 0.415.3 ± 0.7
Without context conditioning59.8 ± 1.013.3 ± 1.1
Without channel decomposition60.6 ± 0.614.0 ± 1.3
RACA64.5 ± 0.616.0 ± 1.3

With identical structures and rewards, removing channel decomposition costs 3.9 MATH / 2.0 AIME points; removing context conditioning costs 4.7 / 2.7 points. These comparisons support the contribution of credit assignment beyond the predefined structure and reward rules.

W03: Evidence for selective collaboration

Section 4.3 already acknowledges that RACA requests interaction more frequently after correct answers than after incorrect ones (63% versus 58%). However, this observation should be considered together with the corresponding results for the other methods:

MethodP(Int∣Wrong)P(\mathrm{Int}\mid\mathrm{Wrong})P(Int∣Correct)P(\mathrm{Int}\mid\mathrm{Correct})Correct − Wrong
Outcome GRPO23%40%17 pp
Role-Wise Reward GRPO39%86%47 pp
AT-GRPO34%80%46 pp
GiGPO31%53%22 pp
RACA58%63%5 pp

RACA achieves the highest interaction rate on incorrect answers and the smallest gap between the two correctness conditions among the compared methods. It requests assistance on 58% of incorrect answers, compared with 23–39% for the baselines. Meanwhile, its excess interaction rate on correct answers is 5 percentage points, compared with 17–47 points. These results support improved interaction behavior, while the remaining 5-point gap indicates that the policy is still imperfect.

Figure 3(b) provides complementary evidence from training: requests after correct answers decrease substantially, while requests after incorrect answers initially decline and then partially recover. The overall interaction rate also declines. Thus, the narrowing gap is accompanied by reduced interaction on already-correct answers, rather than achieved through a uniform increase in assistance-seeking.

Our claim is therefore that RACA improves assistance-seeking behavior relative to the compared methods, as reflected in its greater coverage of incorrect answers and smaller disparity between correctness conditions. We explicitly acknowledge that learning when assistance is necessary remains an unresolved limitation.

W04: Additional complexity and training cost

The additional complexity of RACA consists of role-specific reward computation, matched-group advantage normalization, and channel-specific token masks during training. As specified in Sections 3.2–3.3, all feedback comes from the original rollouts. RACA requires no learned value function, process reward model, or counterfactual execution, and introduces no additional inference-time component beyond the shared four-role framework.

The paper explicitly reports inference costs in Table 3 and training costs in Appendix E.9, Table 7. The comparison within the same multi-agent framework is:

MethodTraining GPU-hoursMATH-L5 accuracyLLM calls per problem
Outcome GRPO251.3958.1%8.7
Role-Wise Reward GRPO318.6060.0%10.7
AT-GRPO349.5260.8%10.7
GiGPO265.4559.1%9.3
RACA294.6464.5%7.4

RACA’s training cost is lower than AT-GRPO and Role-Wise Reward GRPO, while its MATH accuracy is higher by 3.7 and 4.5 percentage points, respectively. Compared with GiGPO, RACA uses approximately 11% more training GPU-hours, improves MATH accuracy by 5.4 points, and reduces average inference calls from 9.3 to 7.4. Section 4.1 specifies that these methods are trained to convergence under the same validation-based stopping and checkpoint-selection protocol; Table 7 reports the resulting training expenditure.

The practical comparison also includes Single-Agent GRPO + Self-Consistency: RACA achieves 64.5% versus 62.4% MATH accuracy using 7.4 versus 8.0 calls and 2,014 versus 4,208 output tokens per problem, although it processes more input tokens. Thus, the paper provides explicit accuracy–cost comparisons rather than relying on accuracy alone to justify the method.

We will make this trade-off more prominent: RACA incurs additional training bookkeeping and does not minimize every resource metric, but achieves higher accuracy with fewer inference calls than the compared multi-agent credit-assignment methods.

Q01: Main methodological contribution and functional-channel decomposition

RACA’s main methodological contribution is to align the evaluation scope, comparison group, and optimization target with the responsibility of each decision. Sections 3.2–3.3 distinguish episode-level controller decisions from local worker actions, construct role–context–channel-matched worker comparison groups, and further separate solution generation from assistance-seeking within primary proposer responses.

For the proposer, functional-channel decomposition includes both advantage estimation and token-level optimization. It is therefore more specific than assigning separate reward values to two predefined segments:

  • Different comparison conditions. The solution channel excludes initial answer correctness ptp_t from its grouping key, because conditioning on ptp_t would make its binary reward constant within each group. The interaction channel includes ptp_t, comparing assistance decisions under the same initial correctness condition.
  • Independent advantage normalization. Each reward is standardized within its own matched group, preserving separate feedback for solution quality and assistance-seeking.
  • Separate, span-normalized updates. Eqs. (7)–(8) apply the two advantages to disjoint token spans and average each channel over its own span before combining their contributions. This prevents the longer solution text from diluting the short interaction block.

For example, a correct answer followed by an unnecessary request receives a positive solution reward and a negative interaction reward. RACA maintains this distinction through group construction, normalization, and optimization. Merely specifying segment-level rewards leaves these choices unspecified.

The predefined structure identifies the relevant token spans; the methodological content lies in how their distinct feedback is converted into policy updates. Table 4 evaluates this distinction while keeping the architecture and turn-level rewards unchanged: replacing the functional channels with a single turn-level advantage reduces accuracy by 3.9 points on MATH and 2.0 points on AIME. Further comparisons with existing methods are provided in our response to Weakness 2.

Q02: Filtering rates and differences across roles

We report diagnostics from two randomly selected training updates, containing 64 question groups, 1,024 trajectories, and 10,481 worker turns. Pooling the three reasoning contexts gives the following channel-level statistics. A channel is filtered when its comparison group has fewer than two representatives or fails the reward-variance threshold.

Role / channelCredited / eligible turnsCoverageFiltered
Primary proposer / solution1,749/2,93959.5%40.5%
Primary proposer / interaction, pt=0p_t=01,327/1,67879.1%20.9%
Primary proposer / interaction, pt=1p_t=1839/1,19870.0%30.0%
Proposer correction917/1,65655.4%44.6%
Critic3,409/3,48197.9%2.1%
Verifier2,169/2,40590.2%9.8%

Filtering affects roles and channels differently. It is more frequent for proposer solution and correction channels, whereas critic and verifier turns retain valid advantages in 97.9% and 90.2% of cases, respectively. Both correctness-conditioned interaction channels also retain substantial coverage. Forced interactions are excluded from interaction-channel eligibility by design and are not counted as failures of group validity.

Importantly, channel filtering does not necessarily exclude an entire proposer turn: a primary response can retain solution credit, interaction credit, or both. The counts establish that at least 8,661/10,481 worker turns (82.6%) receive a valid advantage, so at most 17.4% receive no valid channel advantage in these sampled updates. This is a conservative bound: it counts primary turns credited through interaction channels together with credited correction, critic, and verifier turns; additional solution-only primary turns can increase coverage.

The grouping mechanism also provides more samples than treating each trajectory as a single comparison item would suggest. Matching worker turns are pooled across trajectories and rounds of the same question, using semantic contexts rather than exact blackboard states or round indices. Across 1,018 nonempty groups, the median representative count after deduplication is 13, with an interquartile range of 7–17; only 31 groups (3.0%) contain exactly two representatives. Deduplication controls the normalization statistics, after which valid advantages are assigned back to all represented turns.

These diagnostics quantify the unequal filtering rates while showing that the grouping scheme preserves training signals for the large majority of worker turns in the sampled updates.

Reviewer_htBKOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

A four-role LLM system (controller/proposer/critic/verifier, LoRA adapters on frozen Qwen3-8B, blackboard communication) is trained with verifiable-reward RL. RACA replaces outcome-broadcast GRPO with: episode-level credit for the controller, role-specific local rewards for workers, group-relative advantages computed within (role, blackboard context, reward channel)-matched groups, and separate solution/interaction advantages for proposer turns routed to disjoint token spans. Reports 64.5 on MATH-L5 and 16.0 on AIME, +6.4/+6.0 over same-framework Outcome GRPO, with fewer model calls than adaptive baselines.

Critical strengths

Well-motivated problem and thoughtful mechanism design (excluding p_t from the solution anchor while conditioning the interaction anchor on it; span-routed advantages). Careful protocol: shared SFT initialization, test-once discipline, 3 seeds, full compute accounting, call-matched Self-Consistency baseline, component ablations. The +6.4/+6.0 over framework-matched Outcome GRPO shares identical execution and voting and is the solid core. Honest about imperfect selectivity.

Critical weaknesses
  • Eq. (4) broadcasts one episode-level advantage to every controller turn, including deferred premature stops, justified by an argument that holds equally for the worker turns where broadcast is rejected.

  • The stop gate is also vacuous (any verifier score unlocks stopping, premature stop proposals cost nothing), so the learned controller approaches "always propose stop" while the mechanical gate does the work. And not acknowledged or ablated.

  • The grouping statistics are unreported. Groups are carved from 16 rollouts of one question, sliced by (role, 3 contexts, channel, p_t), deduplicated, and dropped below 2 representatives; late-context and p_t-conditioned interaction groups will routinely be empty, and the paper never reports what fraction of turns actually receive credit. With n=2 the normalized advantage is exactly ±1 regardless of effect size, so the tuned reward magnitudes (g, o, m, c_int) are washed out — the flat Table 6 sensitivity may mean the reward values are inert, not robust. The reward tables also admit gaming: the critic earns more for a terminal flag with no repair (0.2) than for a failed repair (0.1), rewarding late flagging; the verifier's Brier reward has a v=0.5 hedging equilibrium that the sigma > delta validity rule would lock in by dropping the channel.

  • The selectivity narrative is overstated against the paper's own metrics: Table 2 wants P(Int|Correct) down, but RACA's 63% is worse than Outcome GRPO (40%) and GiGPO (53%). The +9.0 pre-to-post gain is confounded with majority voting (the paper's own baselines gain +7.5 to +11.3 from voting alone; no no-vote ablation; fix/break rates not reported). The AIME headline is within noise: 16.0±1.3 vs 14.7±0.7 on 150 problems is 2 problems (p ~ 0.2), so "outperforms all baselines on both benchmarks" holds only on MATH; no significance tests anywhere; AIME 2022-2024 contamination with Qwen3 pretraining is undiscussed. Baseline fidelity is unverifiable: Appendix E.3 gives no implementation detail for the AT-GRPO/GiGPO adaptations, and the strongest single-agent baseline had 8x less compute (37 vs 295 GPU-h) with no matched control. Related work misses the nearest neighbors: ADPO, M-GRPO, MATPO/MARFT (shared-backbone role training), learned when-to-ask; GiGPO and ReMA are cited with wrong years.

Questions for authors

What fraction of worker turns receive a valid channel advantage per update, by context and by p_t-conditioned interaction group? Can you add a no-vote / last-candidate ablation and fix/break rates to separate repair from candidate-accumulation voting?

How were AT-GRPO/GiGPO mapped onto your framework, and how do the adaptations compare to their published results? Why does the critic's b_terminal (0.2) exceed the failed-repair reward (0.1)? How do you rule out the verifier hedging equilibrium given the sigma > delta rule?

6 comments

W01: Controller credit assignment

Thank you for your careful review and constructive feedback.

We agree that the explanation in Appendix C.3 needs clarification: worker actions also affect subsequent execution, so this fact alone does not justify the different credit scopes.

RACA's principle is to match credit assignment to each role's responsibility, rather than to replace episode-level rewards for every role.

The controller is responsible for managing the entire reasoning episode: it decides when the team should continue reasoning and when it should stop. Its decisions must therefore be evaluated in terms of the episode's final correctness and computational cost, rather than the correctness of an individual intermediate answer. Equation (3) explicitly captures this objective, and the resulting episode-level advantage is used to train the controller's decisions.

Workers, by contrast, produce outputs that can be evaluated locally. For example, an incorrect proposer answer remains incorrect even if another agent subsequently repairs it. Assigning terminal success to that answer would conflate its quality with downstream recovery. RACA therefore uses locally verifiable rewards for these role-specific outputs.

Retaining episode-level credit for the controller while assigning local credit to workers is thus a deliberate application of the same role-aware principle.

This also applies to deferred stop proposals: they are sampled controller actions evaluated under the specified execution protocol. Specifically, when a stop proposal is deferred, the round continues; if no verifier score is available after the interaction chain, an auxiliary verifier call is executed before the next controller decision. During training, when an interaction is injected following a deferred stop, its target is the verifier (Appendix B). Thus, although immediate termination is blocked, the proposal can still affect the subsequent verification path. Training these proposals with the realized episode return does not assume that each proposal independently contributed to success. We will clarify this rationale and the execution consequences in the revision.

W02: Stop gate and controller behavior

We thank the reviewer for raising this concern. The original paper did not report how the learned controller actually uses the gate. We clarify the mechanism and provide the corresponding diagnostics below.

The gate checks whether verification evidence exists; the controller decides whether that evidence justifies termination. Evaluating the available evidence is the controller's responsibility.

Specifically, termination requires both a controller decision to stop and an available verifier score (Eq. 13). Any score satisfies the availability requirement, but it does not prescribe the controller's decision. A low score can instead inform a decision to continue. The gate therefore does not determine when the answer is sufficiently reliable to stop.

Regarding the lack of a separate penalty for premature stop proposals, the controller is trained against the episode-level objective in Eq. (3). Unused budget earns a bonus when the final answer is correct, whereas an incorrect answer incurs a penalty that increases with unused budget. Thus, the objective rewards efficient successful termination and penalizes unsuccessful early termination.

We examine two randomly selected training updates, at steps 23 and 24, containing 1,024 trajectories:

DiagnosticObserved value
Stop proposals / all controller decisions241/3,180 (7.6%)
Stop proposals blocked by the gate0/241 (0%)
Continue decisions when a verifier score is available1,536/1,777 (86.4%)
Mean active reasoning rounds per trajectory2,939/1,024 = 2.87

In particular, when verification evidence is already available, the gate would permit a stop proposal, yet the controller chooses to continue in 86.4% of these decisions. Moreover, none of the observed stop proposals requires the gate to override it. These statistics show that the controller does not collapse to always proposing stop in the sampled updates: continuation is predominantly selected by the controller itself, rather than enforced by repeated gate rejection.

W03: Grouping statistics, normalization, and reward design

We report diagnostics from two randomly selected updates (steps 23 and 24): 64 question groups, 1,024 trajectories, and 10,481 worker turns.

The 16 rollouts are trajectories, each containing multiple worker turns. Matching turns are pooled across trajectories and rounds of the same question. Context means explore, refine, or verify, not an exact blackboard state or round index. Only the primary interaction channel is additionally split by ptp_t. Deduplication selects representatives for normalization; valid advantages are assigned back to all represented turns.

Their channel-level credit coverage is:

Role / channelExploreRefineVerify
Primary proposer / solution608/1,024 (59.4%)415/668 (62.1%)726/1,247 (58.2%)
Primary proposer / interaction, pt=0p_t=0402/502 (80.1%)337/479 (70.4%)588/697 (84.4%)
Primary proposer / interaction, pt=1p_t=1312/499 (62.5%)115/176 (65.3%)412/523 (78.8%)
Proposer correction239/446 (53.6%)288/504 (57.1%)390/706 (55.2%)
Critic1,156/1,201 (96.3%)886/900 (98.4%)1,367/1,380 (99.1%)
Verifier631/709 (89.0%)521/619 (84.2%)1,017/1,077 (94.4%)

Each cell reports credited / eligible turns; forced primary turns are excluded from interaction-channel eligibility.

The solution channel retains 1,749/2,939 turns (59.5%). Its binary correctness reward has zero variance in all-correct or all-incorrect groups, which the validity rule intentionally filters. The same proposer turn can still receive interaction credit.

At least 8,661/10,481 worker turns (82.6%) receive a valid advantage, counting interaction-credited primary turns plus credited correction, critic, and verifier turns; solution-only primary turns can increase this bound. Both ptp_t-conditioned channels retain substantial coverage across all contexts. These observations do not support the inference that such groups routinely lose their training signals.

Across 1,018 nonempty groups, the median representative count after deduplication is 13, with an interquartile range of 7–17 and a maximum of 37. Only 31 groups (3.0%) have n=2n=2, counting both valid and filtered groups. Thus, widespread two-sample normalization is not supported by these diagnostics.

Separately, standardization removes common positive affine transformations, but does not generally remove changes to individual reward terms. For an incorrect-answer group containing successful, unsuccessful, and missed requests, the rewards are g−cintg-c_{\mathrm{int}}, −cint-c_{\mathrm{int}}, and −m-m. Their successive gap ratio, g/(m−cint)g/(m-c_{\mathrm{int}}), is preserved by standardization. Changing gg from 0.2 to 0.4 at the default m=0.1m=0.1 and cint=0.02c_{\mathrm{int}}=0.02 changes this ratio from 2.5 to 5.0. Conversely, binary-reward groups can remain scale-invariant regardless of group size. We therefore interpret Table 6 as showing limited performance sensitivity within the tested settings, without claiming that every coefficient changes advantages in every group.

The critic reward requires a separate distinction between an observed failed repair and an unavailable repair outcome. Under Equation (2) and Appendix C.2, a correctly flagged error receives 0.1 or 0.3 when a local repair outcome is observed. The fixed 0.2 applies only when neither a within-round correction nor a subsequent active round is available. This intermediate value preserves feedback for correct detection without treating missing repair feedback as an observed failure; it is not a bonus for termination. The critic does not make the controller’s stop decision, so these are not two freely selectable feedback conditions within the same critic call. In the sampled batches, only 213/3,481 critic calls (6.12%) enter this terminal fallback branch.

Finally, the Brier reward does not generally favor v=0.5v=0.5. If the conditional probability of correctness is μ\mu, expected reward is maximized at v=μv=\mu, with an expected reward loss of (v−μ)2(v-\mu)^2 for reporting vv. We acknowledge that an entirely constant-0.5 group has constant reward and supplies no policy-gradient signal. However, filtering concerns individual comparison groups in the current update, rather than permanently disabling the verifier channel. Persistent constant predictions remain a possible degeneracy to monitor, not a general optimum induced by the Brier reward. In these Brier-trained batches, only 21/2,405 valid verifier scores (0.87%) equal 0.5, and 2,169/2,405 verifier turns (90.2%) receive valid advantages. Thus, the sampled batches show neither concentration at 0.5 nor widespread verifier-channel inactivity.

W04: Interaction evidence, evaluation, and baseline comparisons

We address the concerns about interaction behavior, answer aggregation, AIME evaluation, baseline comparisons, and related work below.

1. Interaction behavior.

The interaction behavior should be assessed using both conditional interaction rates reported in Table 2. RACA achieves the highest interaction rate on incorrect answers, and the smallest excess interaction rate on correct over incorrect answers among all compared adaptive methods:

MethodP(Int∣Wrong)P(\mathrm{Int}\mid\mathrm{Wrong})P(Int∣Correct)P(\mathrm{Int}\mid\mathrm{Correct})Difference: Correct − Wrong
Outcome GRPO23%40%17 pp
Role-Wise Reward GRPO39%86%47 pp
AT-GRPO34%80%46 pp
GiGPO31%53%22 pp
RACA58%63%5 pp

Specifically, RACA requests assistance on 58% of incorrect answers, compared with 23–39% for the baselines. Its excess interaction rate on correct answers is only 5 percentage points, compared with 17–47 points for the baselines. Thus, comparing the 63% correct-answer interaction rate alone overlooks RACA’s substantially greater coverage of incorrect answers and its smallest disparity between the two conditions. Section 4.3 already acknowledges that interaction remains more frequent on correct answers; the evidence supports these comparative improvements rather than fully resolved interaction selectivity.

To distinguish the assistance targets, we additionally break down proposer-chosen requests across all reasoning rounds, excluding forced interactions and mechanically scheduled verification calls:

Request targetRequest rate given an incorrect answerRequest rate given a correct answerFix rateBreak rate
Critic50.6%49.3%25.7%3.6%
Verifier7.4%13.7%12.0%1.2%

Request rates are conditioned on the primary answer’s correctness before interaction. Here fix/break rates are round-level outcomes, comparing that answer with the answer remaining at the end of the requested interaction chain within the same reasoning round.

The higher correct-answer interaction rate is associated primarily with verifier requests (13.7% versus 7.4%); critic requests are slightly more frequent on incorrect answers (50.6% versus 49.3%). Critic-initiated chains exhibit a higher fix rate, while verifier-initiated chains exhibit a lower break rate. This breakdown distinguishes target-selection behavior and the observed repair outcomes during reasoning, supplementing the aggregate interaction statistics above.

2. Separating pre-aggregation improvement from voting.

The 9.0-point improvement in Table 2 measures the total change from the initial proposer answer to the final system output, including subsequent reasoning, interaction, and answer aggregation. It should therefore be distinguished from the effect of repair alone. Likewise, the gains from Self-Consistency include both additional candidate generation and aggregation; their magnitude alone does not establish how much of RACA’s improvement comes from voting. Separately, RACA and Outcome GRPO use the same execution framework and final aggregation rule, so their 6.4-point performance difference cannot be attributed to using different voting mechanisms.

To directly examine whether improvements persist without final voting, we compare the initial answer, the last generated candidate without voting, and the final system output on the same evaluation trajectories:

OutputAccuracyFix rateBreak rate
Initial proposer answer55.5%——
Last candidate, without voting62.5%19.5%3.0%
Final system output64.5%22.7%2.0%

Fix rate is the fraction of initially incorrect answers that become correct; break rate is the fraction of initially correct answers that become incorrect.

Without voting, the last candidate improves accuracy by 7.0 percentage points, retaining approximately 78% of the total 9.0-point improvement. Final aggregation provides a further 2.0-point increase. These results show that substantial net improvement already occurs before final voting, while aggregation contributes additional benefits.

The fix/break analysis further characterizes this improvement. The no-vote output corrects 19.5% of initially incorrect answers while changing 3.0% of initially correct answers to incorrect ones. Final aggregation increases the fix rate to 22.7% and reduces the break rate to 2.0%. Thus, the results support both error recovery during subsequent reasoning and improved candidate selection at the final stage. We attribute the 9.0-point gain to the complete downstream reasoning process, rather than treating it as a direct measure of repair alone.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.