acceptodds
Under review as a conference paper at ICLR 2027

Co-RL: Unsupervised Reasoning Emerges from Diverse Cohort in Multi-agent RL

Abstract

Reinforcement Learning (RL) has emerged as a powerful approach for improving reasoning in language and vision-language models, yet its strongest successes still depend heavily on ground-truth supervision (e.g., verifiable reward). Such annotations are costly to obtain and become increasingly scarce as reasoning capabilities advance beyond what humans can reliably evaluate. Self-rewarding RL reduces this dependence by enabling models to derive reward signals from their own completions. However, training solely on self-generated feedback can reinforce existing biases, reduce response diversity, and ultimately lead to homogenized responses and training collapse. In this work, we show that unsupervised reasoning can emerge through cooperative multi-agent training. We introduce Co-RL, a framework in which multiple decoupled models, sharing no parameters, are simultaneously optimized through RL using rewards derived from their peers. We further show that increasing cohort diversity, through heterogeneous model families, sizes, and rephrased training samples, reduces the correlated errors that drive self-reinforcing feedback loops. This diversity consistently improves reasoning performance and mitigates training collapse. Without any ground-truth labels, Co-RL improves the base models by 3.0–7.2% on average across seven benchmarks and by 1.6–4.4% across four multimodal benchmarks. Compared with prior state-of-the-art methods, it also outperforms the multi-agent RL method CoMAS by 4.0% and the self-rewarding vision-language RL method MM-UPT by 2.0%.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Reinforcement Learning, Large Language Models, Vision-Language Models, Reasoning, Label-Free RL, Self-Rewarding RL, Multi-Agent RL, Cross-Agent Supervision, Co-Training

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 2 reviews, above 27% of the others.
Reviewer_V2xWOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper introduces Co-RL, a label-free multi-agent reinforcement learning framework that addresses the reliance on ground-truth answers in reasoning-oriented RL and the tendency of self-rewarding training to reinforce existing errors, homogenize responses, and collapse. Multiple models with separate parameters independently solve the same problem. Each model uses the majority-vote answer from a designated peer’s sampled responses as a pseudo-label, receives rewards based on agreement with that pseudo-label, and is updated independently using GRPO. To reduce correlated errors, the authors introduce diversity through different model families, parameter scales, and semantically equivalent question reformulations. The paper investigates how cross-agent supervision and cohort diversity affect performance and training stability through a simplified theoretical analysis, ablation studies, and budget-matched comparisons. Experiments cover seven text-only reasoning benchmarks and four multimodal reasoning benchmarks, reporting average improvements of 3.0–7.2 and 1.6–4.4 percentage points over the initial models, respectively, and outperforming several self-rewarding and multi-agent RL baselines under the evaluated settings.

Critical strengths

The paper’s core strength is its combination of independent peer feedback and model diversity to provide complementary supervision for label-free reinforcement learning. Unlike methods that derive rewards from a model’s own predictions, Co-RL obtains pseudo-labels from independently optimized peers and reduces correlated errors through cross-family pairing and question reformulation, mitigating self-reinforcing errors and training collapse. The mechanism is simple and requires no additional reward judge during training. Ablation studies and budget-matched comparisons support its effectiveness in improving training stability and reasoning performance.

Critical weaknesses

The paper’s core limitation is that cross-agent supervision directly treats a peer’s majority-vote answer as the reward target, without a mechanism for assessing pseudo-label reliability during training. Its effectiveness therefore depends on the models’ existing capabilities and complementary error patterns. Incorrect peer predictions or highly correlated errors may still reinforce mistakes or degrade the stronger model. Although the authors provide theoretical analysis and demonstrate several effective model combinations, the paper does not establish a practical model-pairing criterion for predicting when mutual supervision will yield consistent net benefits.

Questions for authors
  1. How should models be paired for effective mutual supervision? The paper demonstrates several effective model combinations, but what criteria should guide peer selection in practice? Could the authors systematically control initial capability gaps and error complementarity to establish pairing criteria that predict the training gains of both models?
  2. How can incorrect pseudo-labels be identified and negative transfer prevented?The current method directly uses a peer’s majority-vote answer to construct rewards. Could consistently incorrect peer predictions or shared misconceptions degrade model performance? Could the authors analyze these failure cases and compare against pseudo-label filtering or adaptive supervision mechanisms?
  3. Is updating both models necessary to obtain the reported gains?Could the authors include a baseline in which a frozen peer provides pseudo-labels and compare it with simultaneous updates of both models? This would help distinguish gains attributable to the peer’s supervision quality from those arising from co-evolution.
  4. How does the theoretical condition translate to practical LLM training?Under simplifying assumptions, the theoretical analysis establishes convergence to the correct answer when the per-problem correctness probabilities satisfy pA+pB>1p_A+p_B>1. To what extent does this result hold when there are multiple distinct incorrect answers, correlated errors between models, and practical GRPO updates?
  5. Are the improvements over the strongest baselines statistically robust?The main text-only experiments report average gains of 0.7–2.0 percentage points over the strongest label-free baselines. Could the authors report means, standard deviations, or confidence intervals across multiple independent training seeds to establish the consistency of these gains, particularly for models with smaller improvements?
  6. Is the citation to VideoOrion in Appendix A appropriate? In the “RLVR for multimodal reasoning” paragraph, Feng et al. (2025), corresponding to VideoOrion, is cited among multimodal RL methods using R1-style rule-based rewards. However, that paper primarily studies the representation and encoding of object dynamics in videos. Could the authors clarify how this reference supports the statement or correct the citation, and check the remaining citations for consistency with the claims they support?
4 comments
AuthorsOpenReview ↗

Response to Reviewer 2

We sincerely thank the reviewer for the constructive feedback and recognition of Co-RL's contributions. We have conducted additional experiments to address the concerns and hope these new results will encourage the reviewer to reconsider their assessment and raise the score.

>>> W1, Q1 Choosing a peer by complementarity and shared errors To investigate practical model-pairing criteria, we conducted an additional pre-training analysis of nine model instances from five base models on MATH-500, following the evaluation protocol in Table 1. We measure complementarity (the fraction of problems where exactly one agent is correct), shared errors (the fraction where both agents produce the same incorrect answer), and initial model accuracy, which reflects the reliability of peer-generated pseudo-labels at the beginning of training. We then examine how these metrics relate to downstream Co-RL performance.

Model A × Model BMATH-500 Before → After (Gain, A/B)Complementarity / Shared ErrorsFinal 7-Benchmark Avg. (Gain vs. TTRL)
Different-family pairs
Qwen2.5-3B × Llama-3.2-3B-Instruct54.2/44.0 → 66.8/56.2 (+12.6/+12.2)31.0 / 2.448.5 (+1.2) / 43.7 (+0.6)
Qwen2.5-7B × Llama-3.1-8B-Instruct70.2/49.4 → 75.2/54.8 (+5.0/+5.4)34.0 / 1.056.8 (+1.7) / 46.8 (+0.7)
Same-model pairs (different seeds)
Qwen2.5-3B × Qwen2.5-3B54.2/56.2 → 66.0/– (+11.8)27.2 / 4.648.7 (+1.4)
Llama-3.2-3B-Instruct × Same44.0/43.4 → 52.4/– (+8.4)23.8 / 5.242.7 (−0.4)
Qwen2.5-7B × Qwen2.5-7B70.2/68.0 → 74.6/– (+4.4)22.2 / 3.655.5 (+0.4)
Llama-3.1-8B-Instruct × Same49.4/47.4 → 51.4/– (+2.0)24.0 / 4.046.8 (+0.7)

We observe that different-family model pairs exhibit higher complementarity (31.0–34.0%) and fewer shared incorrect answers (1.0–2.4%) than same-model pairs with different sampling seeds and achieve higher or comparable downstream performance on all four models.

Importantly, higher peer accuracy does not necessarily lead to greater downstream gains. For example, when training Qwen2.5-7B, we observe:

PeerPeer accuracy on MATH-500 (%)Complementarity (%)Co-RL gain on MATH-500 (%)Co-RL gain on 7 benchmarks over TTRL (%)
Qwen2.5-7B (new seed)68.022.2+4.4+0.4
Llama-3.1-8B-Instruct49.434.0+5.0+1.7

These results show that a weaker but more complementary peer can provide more effective supervision. This is consistent with Theorem 1, which shows how complementary strengths can expand the basin of correct convergence. Our findings suggest that high complementarity and low shared-error rates are useful indicators for model pairing, beyond individual model accuracy.

AuthorsOpenReview ↗

Response to Reviewer 2

>>> Q2 Incorrect pseudo-labels and confidence-based rewards We thank the reviewer for this question.

How can incorrect pseudo-labels be identified and negative transfer prevented?

We address this concern from three perspectives.

First, model pairing. Our analysis in Q1 suggests that selecting peers with complementary strengths and fewer shared errors can reduce negative transfer.

Second, theoretical understanding. Theorem 1 shows that both agents can converge correctly when pA+pB>1p_A+p_B>1, even if one initially performs below chance. Mutual training also allows weaker peers to improve their supervision over time, as supported by our frozen-peer experiments in Q3.

Third, limitations. Shared misconceptions may reinforce incorrect predictions, and reliably identifying incorrect pseudo-labels without ground truth remains challenging. Future advances in pseudo-label reliability estimation and error mitigation could be incorporated into Co-RL.

Comparison with Pseudo-Label Filtering and Adaptive Rewards.

To address this concern, we evaluate two additional reward designs: (1) Confidence-gated supervision, which serves as pseudo-label filtering and applies majority-vote supervision only when at least 62.5% of peer responses agree, while penalizing rarely predicted answers; and (2) Frequency-weighted supervision, which assigns soft rewards proportional to each answer's frequency among peer responses, adapting the reward to the peer's response distribution.

Neither consistently improves over standard Co-RL. All three achieve similar pseudo-label accuracy, agreement, and validation performance. To this end, we remain our current simple yet effective design within Co-RL.

AgentRewardPseudo-label accuracyAgreementValidation accuracy, meanValidation accuracy, best
Qwen2.5-3BCo-RL, peer majority vote63.862.161.664.4
Qwen2.5-3BConfidence-gated supervision63.161.261.764.2
Qwen2.5-3BFrequency-weighted supervision63.661.961.764.4
Llama-3.2-3B-InstructCo-RL, peer majority vote67.362.150.252.4
Llama-3.2-3B-InstructConfidence-gated supervision67.561.250.253.0
Llama-3.2-3B-InstructFrequency-weighted supervision67.461.949.352.8

>>> Q3 Comparing mutual training with a frozen peer We thank the reviewer for this constructive suggestion. To address this concern, we introduce an additional frozen-peer baseline, where the peer generates responses and provides majority-vote pseudo-labels at each training step, as in Co-RL, but its parameters remain frozen throughout training. We evaluate two settings, freezing either Qwen2.5-3B or Llama-3.2-3B-Instruct, to isolate the benefits of simultaneous mutual training.

AgentMethodGSM8KMATH500AMCHEvalGPQAMBPPLCBAvg
Qwen2.5-3BTTRL80.466.431.363.422.251.815.947.3
Qwen2.5-3BCo-RL, frozen peer78.464.431.364.624.253.218.347.8 (+0.5)
Qwen2.5-3BCo-RL, both models updated80.166.833.764.022.756.815.248.5 (+1.2)
Llama-3.2-3B-InstructTTRL77.950.226.559.224.851.212.043.1
Llama-3.2-3B-InstructCo-RL, frozen peer76.050.031.361.023.748.412.043.2 (+0.1)
Llama-3.2-3B-InstructCo-RL, both models updated80.556.227.759.121.250.411.043.7 (+0.6)

Co-RL achieves the highest average performance for both agents. These results further support the importance of simultaneous mutual training over fixed peer supervision.

Beyond empirical evidence, our theoretical analysis also provides insights into why simultaneous mutual training can be more effective than learning from a fixed peer. Specifically, Proposition 1 shows that an agent's learning direction depends on the quality of its peer's supervision. When one peer is frozen, its predictions remain unchanged throughout training, meaning that any incorrect pseudo-labels may repeatedly misguide the other agent. In contrast, simultaneous training allows both agents to continuously improve (as shown in Figure 3), leading to better supervision for each other.

AuthorsOpenReview ↗

Response to Reviewer 2

>>> Q4 Applicability of Theorem 1 to practical GRPO training We thank the reviewer for this important question.

  1. Multiple distinct incorrect answers do not violate our theoretical framework, as correctness is modeled as a binary outcome. Similarly, shared misconceptions do not violate our conditional independence assumption, since the two agents generate rollouts independently for each problem. These two factors do not invalidate the underlying assumptions and our analysis still applies.

  2. While practical GRPO introduces additional complexities, our empirical results exhibit trends consistent with the theoretical predictions. First, Proposition 2 predicts that self-rewarding RL can reinforce incorrect majority answers. Figure 8 illustrates this on Qwen2.5-7B, where TTRL reinforces its own incorrect predictions and collapses late in training. Second, Theorem 1 predicts that both agents can learn from complementary strengths when pA+pB>1p_A+p_B>1. For Qwen2.5-7B and Llama-3.1-8B-Instruct on MATH-500, the fraction of problems solved by exactly one agent decreases from 34.0% to 24.8%, while joint correctness increases from 42.8% to 52.8% after Co-RL. Furthermore, our theoretical analysis provides insights into the role of mutual training versus freezing one peer, as further examined in Q3. Thus, although our analysis does not provide exact convergence guarantees for practical GRPO, it offers meaningful insights into the learning dynamics and is supported by empirical evidence.

>>> Q5 Multi-seed results of Co-RL+ and TTRL We thank the reviewer for this important question. We evaluate Co-RL+ over three training seeds on both Qwen2.5-3B and Llama-3.2-3B-Instruct. We also rerun TTRL, the strongest baseline for Qwen-3B in the upper part of Table 1 with multiple seeds. The table below reports the mean and standard deviation across seeds.

ModelMethod (seeds)GSM8KMATH500AMCHEvalGPQAMBPPLCBAvg
Qwen2.5-3BCo-RL+ (3 seeds)80.7 ± 0.867.0 ± 1.636.5 ± 1.963.2 ± 3.723.9 ± 1.854.7 ± 0.916.9 ± 0.549.0 ± 0.6
Qwen2.5-3BTTRL (2 seeds)79.5 ± 1.364.6 ± 2.531.3 ± 0.065.6 ± 3.022.7 ± 0.753.6 ± 2.515.7 ± 0.447.6 ± 0.3
Llama-3.2-3B-InstructCo-RL+ (3 seeds)78.0 ± 0.754.3 ± 0.827.3 ± 3.060.6 ± 1.525.1 ± 2.550.2 ± 1.712.1 ± 0.343.9 ± 0.5
Llama-3.2-3B-InstructTTRL (1 seed)77.950.226.559.224.851.212.043.1

Our results show consistent improvements across training seeds. On Qwen2.5-3B, Co-RL+ achieves 49.0 ± 0.6%, compared with 47.6% for TTRL. On Llama-3.2-3B-Instruct, Co-RL+ achieves 43.9 ± 0.5%, compared with 43.1% for TTRL. In both cases, the improvements exceed the observed standard deviations, providing encouraging evidence of reproducibility across independent training runs.

>>> Q6 Correcting the VideoOrion citation Thank you for pointing this out. While VideoOrion is relevant to multimodal reasoning, we agree that it does not directly support the specific claim about R1-style rule-based rewards. We will correct this citation and carefully review the remaining references to ensure that they appropriately support the corresponding claims.

AuthorsOpenReview ↗

Response to Reviewer 2

Thank you again for your constructive review.

We have addressed all the questions and concerns in our responses above. As the discussion period closes soon, we would be grateful to hear whether these responses resolve your concerns or whether any point still needs clarification. If the new results change your assessment, we would sincerely appreciate it if you could reflect this in your final score. We appreciate your time and the constructive discussion.

Best regards,

Authors

Reviewer_gNtNOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Co-RL, a label-free multi-agent reinforcement learning framework designed to improve reasoning capabilities without ground-truth supervision or external reward models. Unlike conventional self-rewarding approaches that derive rewards from a model's own predictions, Co-RL employs multiple independently optimized agents that supervise one another through majority-vote pseudo-labels. The authors further explore cohort diversity across model families, model scales, and rephrased inputs to reduce correlated errors and prevent training collapse. A theoretical analysis suggests that cross-agent supervision enlarges the basin of convergence toward correct solutions compared with self-rewarding RL. Extensive experiments on language and vision-language models demonstrate improvements over existing self-rewarding and multi-agent RL baselines across mathematical, coding, scientific, and multimodal reasoning benchmarks.

Critical strengths

S1. Simple and effective cross-agent reward mechanism.

The proposed framework replaces self-generated rewards with peer-generated pseudo-labels, addressing a fundamental limitation of self-rewarding RL. The method is straightforward to implement, requires neither external judges nor parameter sharing, and can be integrated into existing RL optimization frameworks such as GRPO.

S2. Comprehensive empirical evaluation and convincing ablations.

The paper evaluates Co-RL across multiple model families, scales, and both text-only and multimodal reasoning tasks. Comparisons against TTRL, RENT, Intuitor, Co-rewarding, CoMAS, and MM-UPT demonstrate consistent improvements in many settings. The matched-budget comparisons and ablation studies on model diversity further strengthen the empirical evidence.

S3. Meaningful theoretical analysis of cross-agent learning dynamics.

The theoretical analysis provides an intuitive explanation for why cross-agent supervision can outperform self-rewarding approaches. In particular, the authors show that complementary agents can enlarge the basin of convergence toward correct predictions, offering theoretical motivation for the proposed diversity-based design.

Critical weaknesses

W1. Limited methodological novelty beyond existing co-training and self-rewarding approaches.

The core mechanism essentially combines conventional majority-vote pseudo-labeling, cross-model supervision, and GRPO. Similar principles have been extensively investigated in co-training, mutual learning, and self-rewarding RL. While the integration is effective, the paper introduces relatively few novel algorithmic components beyond applying these established ideas to independently trained LLMs.

W2. The theoretical analysis relies on restrictive assumptions that limit its applicability.

The analysis reduces reasoning to binary correct/incorrect answers, assumes independent rollouts and simplified continuous-time optimization dynamics, and omits practical effects such as clipping and KL regularization. Moreover, the theoretical conclusions concern a two-agent system, while the proposed framework supports larger cohorts. It remains unclear how well these theoretical guarantees characterize actual multi-agent RL training.

W3. Limited exploration of scalability and computational efficiency.

Although Co-RL is evaluated with two and three agents, the paper does not systematically investigate larger cohorts, alternative interaction topologies, or the trade-off between model diversity and computational cost. Furthermore, the framework requires simultaneously training multiple independent models, which may limit its practicality for larger-scale foundation models. The experiments are also predominantly conducted on relatively small models and mathematical reasoning training data.

Questions for authors

Q1. How does Co-RL behave when peer models have substantially different reasoning capabilities?

The paper demonstrates that weak-to-strong supervision can be beneficial, but what happens when the performance gap between agents becomes significantly larger? Could incorrect pseudo-labels from a substantially weaker agent negatively affect a stronger model? Would confidence-weighted or adaptive peer selection improve robustness?

Q2. Is model diversity itself sufficient to guarantee useful cross-agent supervision?

The paper establishes a strong correlation between agent disagreement and performance improvement. However, disagreement does not necessarily imply that the peer provides correct supervision. Could the authors further analyze the relationship between disagreement, peer pseudo-label accuracy, and downstream learning gains? In particular, how does Co-RL distinguish productive disagreement from disagreement caused by unreliable predictions?

Q3. How does Co-RL scale beyond three agents, and how sensitive is it to the supervision topology?

The current approach uses a ring-based supervision topology, where each agent receives rewards from one designated peer. Would aggregating pseudo-labels from multiple peers improve robustness? How would alternative topologies, such as all-to-all supervision or dynamically selected peers, affect convergence, diversity, and computational efficiency?

4 comments
AuthorsOpenReview ↗

Response to Reviewer 1

We sincerely thank the reviewer for the thoughtful and encouraging assessment of our work. We have conducted additional experiments to address the concerns and hope these new results will encourage the reviewer to reconsider their assessment and raise the score.

>>> W1 Contribution beyond combining existing components We thank the reviewer for this comment. We agree that majority voting, cross-model supervision, and GRPO are established techniques. However, our contribution goes beyond combining these components. To our knowledge, we are the first to demonstrate that co-training principles can enable stable, fully label-free reasoning improvement in autoregressive LLMs through multi-agent RL.

While co-training has been extensively studied in conventional prediction tasks, including image classification and segmentation, extending these principles to LLM reasoning presents fundamentally different challenges. Unlike classification tasks with a fixed set of output categories, LLM reasoning involves autoregressive generation of long, open-ended trajectories, where supervision is provided through sparse, outcome-level rewards. Moreover, both the learning policies and their peer-generated supervision signals evolve throughout RL training, creating non-stationary feedback that can amplify incorrect predictions and lead to training collapse. Therefore, whether independently trained LLMs can provide sufficiently reliable supervision to improve each other's reasoning without external labels is far from established. Our work provides an affirmative answer and identifies cohort diversity as a key factor in making such learning effective and stable.

This contribution is supported by both theoretical and empirical findings. Theorem 1 shows that, under a simplified setting, cross-agent supervision expands the basin of jointly correct convergence from (pA>0.5, pB>0.5p_A>0.5,\ p_B>0.5) under self-rewarding to pA+pB>1p_A+p_B>1 under Co-RL. This demonstrates how complementary agents can correct errors that self-rewarding would otherwise reinforce. Empirically, our controlled ablations show that progressively introducing decoupled supervision, heterogeneous model families, and input rephrasing reduces error correlation and consistently improves performance. Furthermore, under matched training budgets, Co-RL outperforms TTRL ensembles in most settings, demonstrating that the improvements cannot be attributed simply to using additional models.

We will revise the Introduction and Related Work to better distinguish our contribution from conventional co-training and clarify the challenges and findings associated with extending these principles to label-free autoregressive reasoning RL.

>>> W2 Applicability of the theory to actual training We thank the reviewer for this insightful comment. We acknowledge that our theoretical analysis adopts simplifying assumptions. However, we would like to emphasize that our goal is not to fully characterize practical GRPO training, but rather to provide theoretical insights into how peer supervision can facilitate learning without ground-truth rewards.

Our empirical findings are consistent with the theoretical insights. First, Proposition 2 shows that self-rewarding RL reinforces a model's majority answer, even when incorrect. This is reflected in Figure 8, where TTRL on Qwen2.5-7B suffers performance collapse late in training. Second, Theorem 1 shows that complementary model strengths can enable both agents to converge to the correct answer when pA+pB>1p_A+p_B>1. Consistent with this, for Qwen2.5-7B and Llama-3.1-8B-Instruct on MATH-500, complementarity decreases from 34.0% to 24.8%, while joint correctness increases from 42.8% to 52.8% after Co-RL, suggesting that initially complementary capabilities become shared strengths through collaborative learning.

Regarding the extension of our theoretical guarantees to multi-agent RL, the analysis can be readily extended to three or more agents under a ring topology (as adopted in our current setup) using the same assumptions, since each agent receives supervision from a single designated peer, preserving the pairwise structure of the learning dynamics. We have added this extension in Appendix D.4 of the revised manuscript.

AuthorsOpenReview ↗

Response to Reviewer 1

>>> W3 Scalability, computational efficiency and generalization We thank the reviewer for these constructive suggestions. We clarify Co-RL's scalability, computational efficiency, and generalization below.

(1) Scalability and topology. Co-RL's ring topology naturally extends to larger cohorts, with each agent supervised by one peer and total computation scaling linearly with the number of agents. Our additional experiments across five supervision topologies show that the ring achieves the highest average accuracy (see Q3).

(2) Computational efficiency. Each Co-RL agent requires comparable computation to a single TTRL run, without additional reward-model inference. Under matched budgets, Co-RL outperforms independently trained TTRL ensembles by 2.0–3.1 percent (Section 5.3). It also outperforms MM-UPT by 2.0 percent with one-quarter of its training budget (Table 3) and CoMAS by 4.0 percent with one-third of its inference budget up to the reported checkpoint.

(3) Generalization. Our experiments extend beyond mathematical reasoning to mixed math, coding, and general reasoning training data (Table 2), as well as multimodal training on MMR1-Math and multimodal-open-r1 (Table 8). Even when trained only on MATH, Co-RL transfers to coding and science benchmarks, including an 8.1% improvement on GPQA over the strongest baseline.

>>> Q1 Peers with different capabilities and confidence-based rewards We thank the reviewer for this important question. Impact of Larger Agent Performance Gaps. Our experiments do cover substantial capability gaps across LLMs and VLMs. The table reports stronger-agent performance before and after Co-RL, averaged over seven LLM or four VLM benchmarks (Tables 1 and 8; Figures 4 and 5).

Stronger agent × Weaker peerInitial gapTraining dataBefore → After (Gain)
Qwen2.5-7B × Qwen2.5-3B10.6MATH52.7 → 56.1 (+3.4)
Qwen2.5-7B × Llama-3.1-8B8.0MATH52.7 → 56.8 (+4.1)
Qwen2.5-3B × Llama-3.2-3B3.4MATH42.1 → 48.5 (+6.4)
InternVL3.5-2B × Qwen2.5-VL-3B3.78open-r1 / MMR143.49 → 47.87 / 47.76 (+4.38 / +4.27)
InternVL3.5-8B × Qwen2.5-VL-7B19.50open-r1 / MMR171.43 → 73.05 / 74.20 (+1.62 / +2.77)

The stronger agent improves in all seven settings (+1.6–6.4%). Even with a peer 19.5 points weaker, InternVL3.5-8B outperforms GT-Reward on both datasets (73.05 vs. 72.96; 74.20 vs. 73.97). While these results demonstrate robustness to capability gaps, negative transfer remains possible, motivating better model-pairing strategies.

Comparison with Confidence-Based Reward Designs. We evaluate two variants: (1) confidence-gated supervision, requiring ≥62.5% peer agreement, and (2) frequency-weighted supervision, assigning rewards based on answer frequencies. Neither consistently improves pseudo-label accuracy, agreement, or validation performance over standard Co-RL. We retain the simpler design and leave improved reliability estimation for future work.

AgentRewardPseudo-label accuracyAgreementValidation accuracy, meanValidation accuracy, best
Qwen2.5-3BCo-RL63.862.161.664.4
Qwen2.5-3BConfidence-gated supervision63.161.261.764.2
Qwen2.5-3BFrequency-weighted supervision63.661.961.764.4
Llama-3.2-3B-InstructCo-RL67.362.150.252.4
Llama-3.2-3B-InstructConfidence-gated supervision67.561.250.253.0
Llama-3.2-3B-InstructFrequency-weighted supervision67.461.949.352.8
AuthorsOpenReview ↗

Response to Reviewer 1

>>> Q2 Diversity, peer accuracy and downstream gains We thank the reviewer for this insightful question. Model diversity alone does not guarantee useful cross-agent supervision, as effective collaboration requires complementary strengths and sufficiently reliable peer predictions. To further investigate this, we analyze three factors: peer pseudo-label accuracy, complementarity (the fraction of problems where exactly one agent is correct), and shared errors (the fraction where both agents produce the same incorrect answer). We then examine how these factors relate to downstream Co-RL gains across different model pairings:

Model A × Model BMATH-500 Before → After (Gain, A/B)Complementarity / Shared ErrorsFinal 7-Benchmark Avg. (Gain vs. TTRL)
Different-family pairs
Qwen2.5-3B × Llama-3.2-3B-Instruct54.2/44.0 → 66.8/56.2 (+12.6/+12.2)31.0 / 2.448.5 (+1.2) / 43.7 (+0.6)
Qwen2.5-7B × Llama-3.1-8B-Instruct70.2/49.4 → 75.2/54.8 (+5.0/+5.4)34.0 / 1.056.8 (+1.7) / 46.8 (+0.7)
Same-model pairs (different seeds)
Qwen2.5-3B × Qwen2.5-3B54.2/56.2 → 66.0/– (+11.8)27.2 / 4.648.7 (+1.4)
Llama-3.2-3B-Instruct × Same44.0/43.4 → 52.4/– (+8.4)23.8 / 5.242.7 (−0.4)
Qwen2.5-7B × Qwen2.5-7B70.2/68.0 → 74.6/– (+4.4)22.2 / 3.655.5 (+0.4)
Llama-3.1-8B-Instruct × Same49.4/47.4 → 51.4/– (+2.0)24.0 / 4.046.8 (+0.7)

We observe that different-family model pairs exhibit higher complementarity (31.0–34.0%) and fewer shared incorrect answers (1.0–2.4%) than same-model pairs with different sampling seeds and achieve higher or comparable downstream performance on all four models. Importantly, higher peer accuracy does not necessarily lead to greater downstream gains. For example, when training Qwen2.5-7B, we observe:

PeerPeer accuracy on MATH-500 (%)Complementarity (%)Co-RL gain on MATH-500 (%)Co-RL gain on 7 benchmarks over TTRL (%)
Qwen2.5-7B (new seed)68.022.2+4.4+0.4
Llama-3.1-8B-Instruct49.434.0+5.0+1.7

While Co-RL does not explicitly distinguish productive from unproductive disagreement during training, its reward mechanism avoids updates when no sampled responses match the peer's pseudo-label. Meanwhile, complementary strengths further enable both agents to learn from each other's correct predictions, facilitating mutual improvement.

>>> Q3 Supervision topology and scaling beyond three agents We thank the reviewer for this important question. To investigate supervision topology, we conducted additional experiments with three agents (Qwen2.5-3B, Llama-3.2-3B-Instruct, and Qwen3-1.7B) under five strategies: ring, pooled majority, random peer selection, self-inclusive majority, and consensus-based supervision. All settings follow the same training configuration. The table reports each agent's average accuracy across seven benchmarks.

Supervision topologyQwen2.5-3BLlama-3.2-3BQwen3-1.7B
Ring (ours)48.544.747.3
Pooled majority48.343.847.1
Random peer48.043.247.2
Self-inclusive majority48.242.947.1
Consensus47.742.447.0

The ring topology achieves the highest average accuracy for all three agents. Neither aggregating multiple peers nor randomly selecting peers improves performance, while consensus-based supervision performs worst. Nevertheless, the average performance across the three agents varies by only 1.1 percent, suggesting that Co-RL is relatively robust to topology choices. Regarding scalability, the ring topology naturally extends to larger cohorts, with each agent supervised by one peer and total training computation scaling linearly with the number of agents. All five strategies use the same number of generated responses in our experiments. While our results support the effectiveness and efficiency of ring supervision, evaluating larger cohorts remains an interesting direction for future work.

AuthorsOpenReview ↗

Response to Reviewer 1

Thank you again for your constructive review.

We have addressed all the questions and concerns in our responses above. As the discussion period closes soon, we would be grateful to hear whether these responses resolve your concerns or whether any point still needs clarification. If the new results change your assessment, we would sincerely appreciate it if you could reflect this in your final score. We appreciate your time and the constructive discussion.

Best regards,

Authors

Comments on the paper

4 comments
AuthorsOpenReview ↗

Response to Reviewer 3

We sincerely thank the reviewer for the positive feedback and recognition of Co-RL’s simplicity, generalizability, and rigorous empirical validation.

>>> W1 Isolating mutual training from peer teaching We thank the reviewer for this insightful comment. We agree that isolating the benefits of simultaneous mutual training is important. To address this, we conducted additional controlled experiments, as detailed in our response to Q1. Our empirical results demonstrate that Co-RL with simultaneous mutual training outperforms the setting where one agent is frozen, highlighting the benefits of jointly evolving agents and bidirectional supervision.

Besides, our theoretical analysis in Section 4 further explains the advantage of mutual training over a frozen peer. Specifically, Proposition 1 shows that an agent's learning direction depends on the quality of its peer's supervision. When one peer is frozen, its predictions remain unchanged throughout training, meaning that any incorrect pseudo-labels may repeatedly misguide the other agent. In contrast, simultaneous training allows both agents to continuously improve (as shown in Figure 3), leading to better supervision for each other.

Theorem 1 establishes that both agents converge to the correct answer whenever pA(0)+pB(0)>1p_A(0)+p_B(0)>1. For example, with pA(0)=0.9p_A(0)=0.9 and pB(0)=0.2p_B(0)=0.2, freezing B drives A toward the incorrect answer, whereas mutual training enables both agents to converge to the correct answer. This illustrates why jointly improving peer supervision offers an advantage beyond simply using predictions from another pretrained model.

>>> Q1 Comparison with a frozen peer and a fixed pool We thank the reviewer for raising this important question. To address this concern, we compare Co-RL against two additional baselines suggested by the reviewer, using matched rollout and update budgets for the trained agent.

(1) Frozen-peer: The peer generates responses and provides majority-vote pseudo-labels at each training step, as in Co-RL, but its parameters remain frozen throughout training. We consider two settings, freezing either Qwen2.5-3B or Llama-3.2-3B-Instruct.

(2) Fixed-pool: Before training, Llama-3.2-3B-Instruct generates 12 responses for each training question. The corresponding majority-vote pseudo-labels are then fixed and used to train Qwen2.5-3B throughout training.

AgentMethodGSM8KMATH500AMCHEvalGPQAMBPPLCBAvg
Qwen2.5-3BTTRL80.466.431.363.422.251.815.947.3
Qwen2.5-3BFrozen peer78.464.431.364.624.253.218.347.8 (+0.5)
Qwen2.5-3BFixed pool79.267.034.961.623.753.816.948.2 (+0.9)
Qwen2.5-3BMutual Co-RL80.166.833.764.022.756.815.248.5 (+1.2)
Llama-3.2-3B-InstructTTRL77.950.226.559.224.851.212.043.1
Llama-3.2-3B-InstructFrozen peer76.050.031.361.023.748.412.043.2 (+0.1)
Llama-3.2-3B-InstructMutual Co-RL80.556.227.759.121.250.411.043.7 (+0.6)

Co-RL achieves the highest average performance for both agents, consistent with our theoretical analysis discussed above, which highlights the benefits of progressively improving peer supervision. Furthermore, our training logs show that the accuracy of the pseudo-labels provided by each agent improves by 4.0–6.0 percentage points from the first to the last quarter of training. These results further support the importance of simultaneous mutual training over fixed peer supervision. We will include both additional baselines in the revised paper.

AuthorsOpenReview ↗

Response to Reviewer 3

>>> Q2 Reproducibility across seeds and checkpoint selection We thank the reviewer for the questions on reproducibility across seeds and on checkpoint selection.

We evaluate Co-RL+ over three training seeds on both Qwen2.5-3B and Llama-3.2-3B-Instruct. We also rerun TTRL, the strongest baseline for both models in the upper part of Table 1. The table below reports the mean and standard deviation across seeds.

ModelMethod (seeds)GSM8KMATH500AMCHEvalGPQAMBPPLCBAvg
Qwen2.5-3BCo-RL+ (3 seeds)80.7 ± 0.867.0 ± 1.636.5 ± 1.963.2 ± 3.723.9 ± 1.854.7 ± 0.916.9 ± 0.549.0 ± 0.6
Qwen2.5-3BTTRL (2 seeds)79.5 ± 1.364.6 ± 2.531.3 ± 0.065.6 ± 3.022.7 ± 0.753.6 ± 2.515.7 ± 0.447.6 ± 0.3
Llama-3.2-3B-InstructCo-RL+ (3 seeds)78.0 ± 0.754.3 ± 0.827.3 ± 3.060.6 ± 1.525.1 ± 2.550.2 ± 1.712.1 ± 0.343.9 ± 0.5
Llama-3.2-3B-InstructTTRL (1 seed)77.950.226.559.224.851.212.043.1

Our results show consistent improvements across training seeds. On Qwen2.5-3B, Co-RL+ achieves 49.0 ± 0.6%, compared with 47.6% for TTRL. On Llama-3.2-3B-Instruct, Co-RL+ achieves 43.9 ± 0.5%, compared with 43.1% for TTRL. In both cases, the improvements exceed the observed standard deviations, providing encouraging evidence of reproducibility across independent training runs.

For checkpoint selection, we use MATH-500 as the validation set and evaluate each run every 10 training steps. We report the checkpoint achieving the highest validation accuracy, applying the same selection criterion consistently across all methods. We choose not to report the final checkpoint to ensure a fair comparison with the baselines, as several self-rewarding methods exhibit performance degradation in later training stages (see Figures 8 and 9), resulting in substantially lower final accuracy than their best checkpoints. We will clarify this checkpoint selection protocol in the experimental setup of the revised paper.

AuthorsOpenReview ↗

Summary of Rebuttal and Changes for AC

Dear Area Chair,

We thank the reviewers for their constructive comments and for recognizing Co-RL as a simple and effective approach. We summarize our rebuttal below.

1. Mutual training (V2xW, 4FVg). The reviewers questioned the need to update both models. We compared Co-RL with a frozen peer and with a fixed pool of peer pseudo-labels. Co-RL outperforms every baseline on both models. On Qwen2.5-3B, it reaches 48.5%, compared with 47.8% for a frozen peer and 48.2% for a fixed pool. On Llama-3.2-3B-Instruct, it reaches 43.7%, compared with 43.2% for a frozen peer. Updating the peer therefore adds a consistent gain.

2. Pseudo-label reliability (gNtN, V2xW). To address the concern about wrong pseudo-labels from the peer, we first tested confidence-based pseudo-label filtering and adaptive supervision. Neither consistently improves pseudo-label accuracy during training. A wrong pseudo-label only harms training through errors shared by both models. We therefore measured complementarity and shared errors on MATH-500 before training. Different-family pairs reach 31.0–34.0% complementarity and 1.0–2.4% shared errors. Same-family pairs reach 22.2–27.2% and 3.6–5.2%. With the weaker Llama-3.1-8B-Instruct as its peer, Qwen2.5-7B gains 1.7% over TTRL against only 0.4% with a same-family peer.

3. Reproducibility (V2xW, 4FVg). Following the reviewers' suggestion, we trained Co-RL+ with three seeds and report the mean and standard deviation. Co-RL+ reaches 49.0 ± 0.6% on Qwen2.5-3B and 43.9 ± 0.5% on Llama-3.2-3B-Instruct. On both models, the variation across seeds (0.6% and 0.5%) is smaller than the gain over TTRL (1.4% and 0.8%). This demonstrates the reproducibility of the gains across training seeds.

4. Scalability and topology (gNtN). To test the ring topology of Co-RL, we compared it with four alternatives under the three-agent setting. In the ring, each agent learns from one designated peer while each alternative changes one factor of this design, namely the number of peers, the choice of peer, self-voting or agreement filtering. The ring performs best on all three agents with a cost linear in the number of agents.

5. Novelty (gNtN). Our contribution is to enable stable, fully label-free reasoning improvement in LLMs through multi-agent RL and to identify cohort diversity as a key factor. Diversity reduces the correlated errors behind self-reinforcing feedback loops. Decoupled supervision, different model families and rephrased inputs raise the four-model average step by step from 47.9% to 49.5%.

In the revision, we will add the new baselines, clarify checkpoint selection, revise the discussion of co-training, and correct the VideoOrion citation.

The discussion period ends in a few days. We would appreciate your help in encouraging further discussion among the reviewers.

AuthorsOpenReview ↗

Response to Reviewer 3

Thank you again for your constructive review.

We have addressed all the questions and concerns in our responses above. As the discussion period closes soon, we would be grateful to hear whether these responses resolve your concerns or whether any point still needs clarification. If the new results change your assessment, we would sincerely appreciate it if you could reflect this in your final score. We appreciate your time and the constructive discussion.

Best regards,

Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.