- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
This paper introduces Co-RL, a label-free multi-agent reinforcement learning framework that addresses the reliance on ground-truth answers in reasoning-oriented RL and the tendency of self-rewarding training to reinforce existing errors, homogenize responses, and collapse. Multiple models with separate parameters independently solve the same problem. Each model uses the majority-vote answer from a designated peer’s sampled responses as a pseudo-label, receives rewards based on agreement with that pseudo-label, and is updated independently using GRPO. To reduce correlated errors, the authors introduce diversity through different model families, parameter scales, and semantically equivalent question reformulations. The paper investigates how cross-agent supervision and cohort diversity affect performance and training stability through a simplified theoretical analysis, ablation studies, and budget-matched comparisons. Experiments cover seven text-only reasoning benchmarks and four multimodal reasoning benchmarks, reporting average improvements of 3.0–7.2 and 1.6–4.4 percentage points over the initial models, respectively, and outperforming several self-rewarding and multi-agent RL baselines under the evaluated settings.
Critical strengths
The paper’s core strength is its combination of independent peer feedback and model diversity to provide complementary supervision for label-free reinforcement learning. Unlike methods that derive rewards from a model’s own predictions, Co-RL obtains pseudo-labels from independently optimized peers and reduces correlated errors through cross-family pairing and question reformulation, mitigating self-reinforcing errors and training collapse. The mechanism is simple and requires no additional reward judge during training. Ablation studies and budget-matched comparisons support its effectiveness in improving training stability and reasoning performance.
Critical weaknesses
The paper’s core limitation is that cross-agent supervision directly treats a peer’s majority-vote answer as the reward target, without a mechanism for assessing pseudo-label reliability during training. Its effectiveness therefore depends on the models’ existing capabilities and complementary error patterns. Incorrect peer predictions or highly correlated errors may still reinforce mistakes or degrade the stronger model. Although the authors provide theoretical analysis and demonstrate several effective model combinations, the paper does not establish a practical model-pairing criterion for predicting when mutual supervision will yield consistent net benefits.
Questions for authors
- How should models be paired for effective mutual supervision? The paper demonstrates several effective model combinations, but what criteria should guide peer selection in practice? Could the authors systematically control initial capability gaps and error complementarity to establish pairing criteria that predict the training gains of both models?
- How can incorrect pseudo-labels be identified and negative transfer prevented?The current method directly uses a peer’s majority-vote answer to construct rewards. Could consistently incorrect peer predictions or shared misconceptions degrade model performance? Could the authors analyze these failure cases and compare against pseudo-label filtering or adaptive supervision mechanisms?
- Is updating both models necessary to obtain the reported gains?Could the authors include a baseline in which a frozen peer provides pseudo-labels and compare it with simultaneous updates of both models? This would help distinguish gains attributable to the peer’s supervision quality from those arising from co-evolution.
- How does the theoretical condition translate to practical LLM training?Under simplifying assumptions, the theoretical analysis establishes convergence to the correct answer when the per-problem correctness probabilities satisfy . To what extent does this result hold when there are multiple distinct incorrect answers, correlated errors between models, and practical GRPO updates?
- Are the improvements over the strongest baselines statistically robust?The main text-only experiments report average gains of 0.7–2.0 percentage points over the strongest label-free baselines. Could the authors report means, standard deviations, or confidence intervals across multiple independent training seeds to establish the consistency of these gains, particularly for models with smaller improvements?
- Is the citation to VideoOrion in Appendix A appropriate? In the “RLVR for multimodal reasoning” paragraph, Feng et al. (2025), corresponding to VideoOrion, is cited among multimodal RL methods using R1-style rule-based rewards. However, that paper primarily studies the representation and encoding of object dynamics in videos. Could the authors clarify how this reference supports the statement or correct the citation, and check the remaining citations for consistency with the claims they support?
4 comments
Response to Reviewer 2
>>> Q2 Incorrect pseudo-labels and confidence-based rewards We thank the reviewer for this question.
How can incorrect pseudo-labels be identified and negative transfer prevented?
We address this concern from three perspectives.
First, model pairing. Our analysis in Q1 suggests that selecting peers with complementary strengths and fewer shared errors can reduce negative transfer.
Second, theoretical understanding. Theorem 1 shows that both agents can converge correctly when , even if one initially performs below chance. Mutual training also allows weaker peers to improve their supervision over time, as supported by our frozen-peer experiments in Q3.
Third, limitations. Shared misconceptions may reinforce incorrect predictions, and reliably identifying incorrect pseudo-labels without ground truth remains challenging. Future advances in pseudo-label reliability estimation and error mitigation could be incorporated into Co-RL.
Comparison with Pseudo-Label Filtering and Adaptive Rewards.
To address this concern, we evaluate two additional reward designs: (1) Confidence-gated supervision, which serves as pseudo-label filtering and applies majority-vote supervision only when at least 62.5% of peer responses agree, while penalizing rarely predicted answers; and (2) Frequency-weighted supervision, which assigns soft rewards proportional to each answer's frequency among peer responses, adapting the reward to the peer's response distribution.
Neither consistently improves over standard Co-RL. All three achieve similar pseudo-label accuracy, agreement, and validation performance. To this end, we remain our current simple yet effective design within Co-RL.
| Agent | Reward | Pseudo-label accuracy | Agreement | Validation accuracy, mean | Validation accuracy, best |
|---|---|---|---|---|---|
| Qwen2.5-3B | Co-RL, peer majority vote | 63.8 | 62.1 | 61.6 | 64.4 |
| Qwen2.5-3B | Confidence-gated supervision | 63.1 | 61.2 | 61.7 | 64.2 |
| Qwen2.5-3B | Frequency-weighted supervision | 63.6 | 61.9 | 61.7 | 64.4 |
| Llama-3.2-3B-Instruct | Co-RL, peer majority vote | 67.3 | 62.1 | 50.2 | 52.4 |
| Llama-3.2-3B-Instruct | Confidence-gated supervision | 67.5 | 61.2 | 50.2 | 53.0 |
| Llama-3.2-3B-Instruct | Frequency-weighted supervision | 67.4 | 61.9 | 49.3 | 52.8 |
>>> Q3 Comparing mutual training with a frozen peer We thank the reviewer for this constructive suggestion. To address this concern, we introduce an additional frozen-peer baseline, where the peer generates responses and provides majority-vote pseudo-labels at each training step, as in Co-RL, but its parameters remain frozen throughout training. We evaluate two settings, freezing either Qwen2.5-3B or Llama-3.2-3B-Instruct, to isolate the benefits of simultaneous mutual training.
| Agent | Method | GSM8K | MATH500 | AMC | HEval | GPQA | MBPP | LCB | Avg |
|---|---|---|---|---|---|---|---|---|---|
| Qwen2.5-3B | TTRL | 80.4 | 66.4 | 31.3 | 63.4 | 22.2 | 51.8 | 15.9 | 47.3 |
| Qwen2.5-3B | Co-RL, frozen peer | 78.4 | 64.4 | 31.3 | 64.6 | 24.2 | 53.2 | 18.3 | 47.8 (+0.5) |
| Qwen2.5-3B | Co-RL, both models updated | 80.1 | 66.8 | 33.7 | 64.0 | 22.7 | 56.8 | 15.2 | 48.5 (+1.2) |
| Llama-3.2-3B-Instruct | TTRL | 77.9 | 50.2 | 26.5 | 59.2 | 24.8 | 51.2 | 12.0 | 43.1 |
| Llama-3.2-3B-Instruct | Co-RL, frozen peer | 76.0 | 50.0 | 31.3 | 61.0 | 23.7 | 48.4 | 12.0 | 43.2 (+0.1) |
| Llama-3.2-3B-Instruct | Co-RL, both models updated | 80.5 | 56.2 | 27.7 | 59.1 | 21.2 | 50.4 | 11.0 | 43.7 (+0.6) |
Co-RL achieves the highest average performance for both agents. These results further support the importance of simultaneous mutual training over fixed peer supervision.
Beyond empirical evidence, our theoretical analysis also provides insights into why simultaneous mutual training can be more effective than learning from a fixed peer. Specifically, Proposition 1 shows that an agent's learning direction depends on the quality of its peer's supervision. When one peer is frozen, its predictions remain unchanged throughout training, meaning that any incorrect pseudo-labels may repeatedly misguide the other agent. In contrast, simultaneous training allows both agents to continuously improve (as shown in Figure 3), leading to better supervision for each other.
Response to Reviewer 2
>>> Q4 Applicability of Theorem 1 to practical GRPO training We thank the reviewer for this important question.
-
Multiple distinct incorrect answers do not violate our theoretical framework, as correctness is modeled as a binary outcome. Similarly, shared misconceptions do not violate our conditional independence assumption, since the two agents generate rollouts independently for each problem. These two factors do not invalidate the underlying assumptions and our analysis still applies.
-
While practical GRPO introduces additional complexities, our empirical results exhibit trends consistent with the theoretical predictions. First, Proposition 2 predicts that self-rewarding RL can reinforce incorrect majority answers. Figure 8 illustrates this on Qwen2.5-7B, where TTRL reinforces its own incorrect predictions and collapses late in training. Second, Theorem 1 predicts that both agents can learn from complementary strengths when . For Qwen2.5-7B and Llama-3.1-8B-Instruct on MATH-500, the fraction of problems solved by exactly one agent decreases from 34.0% to 24.8%, while joint correctness increases from 42.8% to 52.8% after Co-RL. Furthermore, our theoretical analysis provides insights into the role of mutual training versus freezing one peer, as further examined in Q3. Thus, although our analysis does not provide exact convergence guarantees for practical GRPO, it offers meaningful insights into the learning dynamics and is supported by empirical evidence.
>>> Q5 Multi-seed results of Co-RL+ and TTRL We thank the reviewer for this important question. We evaluate Co-RL+ over three training seeds on both Qwen2.5-3B and Llama-3.2-3B-Instruct. We also rerun TTRL, the strongest baseline for Qwen-3B in the upper part of Table 1 with multiple seeds. The table below reports the mean and standard deviation across seeds.
| Model | Method (seeds) | GSM8K | MATH500 | AMC | HEval | GPQA | MBPP | LCB | Avg |
|---|---|---|---|---|---|---|---|---|---|
| Qwen2.5-3B | Co-RL+ (3 seeds) | 80.7 ± 0.8 | 67.0 ± 1.6 | 36.5 ± 1.9 | 63.2 ± 3.7 | 23.9 ± 1.8 | 54.7 ± 0.9 | 16.9 ± 0.5 | 49.0 ± 0.6 |
| Qwen2.5-3B | TTRL (2 seeds) | 79.5 ± 1.3 | 64.6 ± 2.5 | 31.3 ± 0.0 | 65.6 ± 3.0 | 22.7 ± 0.7 | 53.6 ± 2.5 | 15.7 ± 0.4 | 47.6 ± 0.3 |
| Llama-3.2-3B-Instruct | Co-RL+ (3 seeds) | 78.0 ± 0.7 | 54.3 ± 0.8 | 27.3 ± 3.0 | 60.6 ± 1.5 | 25.1 ± 2.5 | 50.2 ± 1.7 | 12.1 ± 0.3 | 43.9 ± 0.5 |
| Llama-3.2-3B-Instruct | TTRL (1 seed) | 77.9 | 50.2 | 26.5 | 59.2 | 24.8 | 51.2 | 12.0 | 43.1 |
Our results show consistent improvements across training seeds. On Qwen2.5-3B, Co-RL+ achieves 49.0 ± 0.6%, compared with 47.6% for TTRL. On Llama-3.2-3B-Instruct, Co-RL+ achieves 43.9 ± 0.5%, compared with 43.1% for TTRL. In both cases, the improvements exceed the observed standard deviations, providing encouraging evidence of reproducibility across independent training runs.
>>> Q6 Correcting the VideoOrion citation Thank you for pointing this out. While VideoOrion is relevant to multimodal reasoning, we agree that it does not directly support the specific claim about R1-style rule-based rewards. We will correct this citation and carefully review the remaining references to ensure that they appropriately support the corresponding claims.
Response to Reviewer 2
Thank you again for your constructive review.
We have addressed all the questions and concerns in our responses above. As the discussion period closes soon, we would be grateful to hear whether these responses resolve your concerns or whether any point still needs clarification. If the new results change your assessment, we would sincerely appreciate it if you could reflect this in your final score. We appreciate your time and the constructive discussion.
Best regards,
Authors
Response to Reviewer 2
We sincerely thank the reviewer for the constructive feedback and recognition of Co-RL's contributions. We have conducted additional experiments to address the concerns and hope these new results will encourage the reviewer to reconsider their assessment and raise the score.
>>> W1, Q1Choosing a peer by complementarity and shared errors To investigate practical model-pairing criteria, we conducted an additional pre-training analysis of nine model instances from five base models on MATH-500, following the evaluation protocol in Table 1. We measure complementarity (the fraction of problems where exactly one agent is correct), shared errors (the fraction where both agents produce the same incorrect answer), and initial model accuracy, which reflects the reliability of peer-generated pseudo-labels at the beginning of training. We then examine how these metrics relate to downstream Co-RL performance.We observe that different-family model pairs exhibit higher complementarity (31.0–34.0%) and fewer shared incorrect answers (1.0–2.4%) than same-model pairs with different sampling seeds and achieve higher or comparable downstream performance on all four models.
Importantly, higher peer accuracy does not necessarily lead to greater downstream gains. For example, when training Qwen2.5-7B, we observe:
These results show that a weaker but more complementary peer can provide more effective supervision. This is consistent with Theorem 1, which shows how complementary strengths can expand the basin of correct convergence. Our findings suggest that high complementarity and low shared-error rates are useful indicators for model pairing, beyond individual model accuracy.