acceptodds
Under review as a conference paper at ICLR 2027

Peer-Critique On-Policy Distillation from Heterogeneous Models for LLM Reasoning

Abstract

Improving an already-capable reasoning model requires repairing its remaining failures. Verifiable rewards can reinforce successful exploration but do not explain how a particular failed trajectory should change; a critique can provide that problem-specific guidance. The model may also fail to diagnose its own error; one remedy is a much larger critic, which adds cost and presumes access to a stronger model. We instead replace the single critic with a panel of same-size peers from different families, whose critiques repair complementary failures: three 3-4B peers repair 41.8-66.0% of failures, exceeding either of two 27-32B critics on every target. **PECO** (**Pe**-**C**ritique **O**n-policy Distillation) uses the target's frozen base, conditioned on a failed attempt and a peer critique that enabled a verified repair, as the teacher for on-policy distillation. Already-solved problems enter without critiques, anchoring training to the initial policy to limit forgetting. The construction requires no reference rationale or model larger than the panel; only the verifier accesses the gold final answer. PECO achieves relative gains of 6.4% over the base models and 6.9% over on-policy self-distillation, beats the verifiable-reward baseline on three targets, and outperforms distillation from much larger teachers on all four. PECO-efficient prunes the critique pool, retaining 93-102% of PECO's four-benchmark accuracy with 36-44% fewer data-building TFLOPs, and uses 2.1 fewer generation calls than GRPO.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Language Models, On-policy distillation, Multi-models, Critique

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_fhPJOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper uses verified critiques from heterogeneous small models for on-policy self-distillation, with anchor examples to reduce forgetting.

Critical strengths

The paper evaluates multiple model families and includes useful ablations on critique sources and anchor examples.

Critical weaknesses

See Questions For Authors

Questions for authors

Data construction already produces verified repaired answers from the target model. How does directly fine-tuning on these answers, together with the initially correct answers, compare with PECO under the same anchor/rescued mixture and matched training budget? This would clarify the benefit of critique-conditioned distillation over simply learning from successful repairs.

Reviewer_UF6dOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes PeCo, which builds verified repair contexts using a heterogeneous panel of comparable-size reasoning models and distills a frozen, critique-conditioned copy of each target into its unassisted student. Initially solved problems provide empty-context anchors to limit forgetting. A pruned variant reduces repair-generation cost. Experiments on four 3-4B targets and five math benchmarks show the strongest improvements on Qwen3 4B and Gemma-3 4B, with smaller or mixed advantages on Llama and Phi. Analyses examine repair complementarity, anchor removal, panel size, cost, and transferred revision style.

Critical strengths

Useful supervision without a larger teacher. The panel repairs a substantial fraction of initial errors, and the within-pipeline larger-critic comparison supports the practical value of small heterogeneous peers. This could make critique-guided improvement accessible when a substantially stronger teacher is unavailable.

Critical weaknesses
  1. The reported comparisons do not establish that heterogeneous peers outperform a homogeneous critic ensemble or repeated critic sampling at matched generation tokens, verification opportunities and update budgets.
  2. The magnitude and robustness of improvement are insufficiently established. The main results use greedy evaluation without training-seed uncertainty; small gains on Phi and Llama are particularly sensitive. Stronger matched baselines and uncertainty could materially change the claimed advantage
Questions for authors

Under a matched total token/FLOP budget and identical anchor and row-count rules, how does the heterogeneous panel compare with multiple sampled critiques from one same-size critic and one larger critic? Please separate diversity from additional repair opportunities.

1 comment
AuthorsOpenReview ↗

Response to Reviewer UF6d (1/2)

[Update, Nov 12] We have extended the ×27 best-single-peer control to Llama-3.2-3B and Phi-3.5-mini and updated its compute (Table R3). The rest of the response is unchanged.

We thank the reviewer for the careful and constructive review, and for recognizing that PeCo provides useful supervision without a larger teacher, which could make critique-guided improvement accessible when a stronger teacher is unavailable. Below, we show that PeCo's gain comes from the diversity of its panel: PeCo outperforms single critics sampled repeatedly at the same number of critiques, while using the least compute. We also show that its gains on Qwen3-4B and Gemma-3-4B exceed one training-seed standard deviation.

W1/Q. Heterogeneous panel vs. homogeneous critics at a matched budget.

PeCo's gain comes from the diversity of its panel, not from additional repair opportunities. Peers from different families repair complementary failures: the strongest peer accounts for only 81% (Qwen3-4B) and 73% (Gemma-3-4B) of the panel's repairs, and the rest are repaired only by other peers (Fig. 2).

The controls the reviewer asked for confirm this (Table R3): PeCo outperforms every homogeneous critic at the same number of critiques and a single peer given three times as many critiques, while using the least compute.

Every condition uses one re-solve per critique, the same verifier, and the same anchor rule. The ×9 conditions match PeCo's 9 critiques per failed problem and, like PeCo, train on all their rescued rows.

  • Self-critic ×9 is the same-model case.
  • 32B critic ×9 is the single larger critic (Qwen3-32B).
  • Best single peer ×9 is the panel member with the highest repair rate for each target (Gemma-3-4B for Qwen3-4B, Llama-3.2-3B and Phi-3.5-mini; Qwen3-4B for Gemma-3-4B).

Table R3. Five-benchmark average and total compute relative to PeCo.

CriticsCritiques/problemTraining rowsQwen3-4BGemma-3-4BLlama-3.2-3BPhi-3.5-miniCompute vs. PeCo
32B critic9all rescued49.741.732.130.34.26×
Self-critic9all rescued49.241.032.030.51.05×
Best single peer9all rescued49.641.831.630.11.26×
Best single peer27matched to PeCo48.542.532.530.63.08×
PeCo9all rescued52.044.932.831.81.00×

Compute covers data construction and training, summed over the four targets (PeCo: 4,231 PFLOPs). Compute is estimated: 2N FLOPs per token for inference, and 4N plus a 2N teacher forward pass per token for LoRA training (N = number of parameters), without the attention term quadratic in sequence length. Training-rollout lengths were not stored, so we estimate them from base-model samples.

Diversity beats repeated sampling. At an equal number of critiques, PeCo outperforms the best homogeneous critic on all four targets (+2.3, +3.1, +0.7 and +1.3).

More repair opportunities do not close the gap. To separate diversity from additional repair opportunities, we give the best single peer three times as many critiques (×27) and subsample its rescued rows to PeCo's number, so it has more repair opportunities than PeCo and the same training data. It still trails PeCo on all four targets, by 3.5, 2.4, 0.3 and 1.2 points.

PeCo uses the least compute. The 32B critic uses 4.26 times and the ×27 single peer 3.08 times as much compute as PeCo. The self-critic is nearly FLOP-matched, at 1.05 times PeCo's compute, and trails PeCo on all four targets, by 2.8, 3.9, 0.8 and 1.3 points. We match critique count rather than tokens because critique length differs across models; since every baseline already uses at least as much compute as PeCo, matching FLOPs would only reduce their budget.

Reviewer_t3ukOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Authors proposed PeCo (Peer-Critique On-Policy Distillation), a method to improve llm reasoning by using critics from small-scale peer models instead of relying on a larger teacher model or self-critic. The method uses 3 peer models from different families to critic failed solutions. The target model then tries to solve the problem again using these critics, and the critics that lead to correct answers are selected. Authors use on-policy self-distillation to train the target model with these verified critics. Authors evaluated 4 ~4B models on five math benchmarks. PeCo outperforms GRPO on three of the four targets, though some improvements are small. Authors also proposed PeCo-efficient, which reduces data construction compute by 36% to 44% while maintaining the performance.

Critical strengths
  • The proposed idea seems effective, authors provided a detailed analysis of how critiques from different models help correct failed solutions. For e.g., fig. 2 shows that ~40% of failures are corrected by one peer. Table 1 shows how critiques transfer between models. These results would be helpful to understand when peer feedback helps, even though the idea of using multiple critics is not very new.
Critical weaknesses
  • Most components of PeCo have been studied in prior works. For instance, weak critic OPD (Jin et al., 2026), critiques as privileged context (Kara and Ersoy, 2026), peer-feedback for OPD (Byeon et al., 2026) and combining feedback from multiple models (Lan et al., 2025). The main contribution here is using a heterogeneous set of same-scale peers and transferring critiques across models. Authors cite these in related works, but not compared against them.

  • PeCo uses up to nine critique candidates per failed problem, while the larger-critic comparison uses a single critic. Thus, the improvement could also come from more correction attempts rather than model diversity.

Questions for authors
  • How does PeCo compare with Jin et al., 2026 and other critic-based OPD methods under similar budgets? What is the main improvement beyond using multiople critics?

  • Can you compare heterogeneous peers with same-family critics or a single critic using the same number of critiques, re-solving attempts and training examples? We should separate improvement coming from #critiques and heterogeneity.

3 comments
AuthorsOpenReview ↗

Response to Reviewer t3uk (1/2)

[Update, Nov 12] We have added OPCD results on Llama-3.2-3B and Phi-3.5-mini (Table R1) and extended the ×27 best-single-peer control to all four targets (Table R2). The rest of the response is unchanged.

We thank the reviewer for the careful review and for finding our complementarity (Fig. 2) and cross-model transfer (Table 1) analyses useful. Below, we show that PeCo outperforms OPCD [1] (Jin et al., 2026), the closest prior method, under the same training setup, and that PeCo's gain comes from the heterogeneity of its panel, not from more critiques, training data or compute.

W1/Q1. Prior work and what PeCo adds.

We first clarify that we do not claim critique-conditioned self-distillation or keeping only the critiques that lead to a correct revision as contributions, both of which appear in Jin et al. [1]. Instead, what PeCo adds goes beyond using multiple critics, and it lets PeCo outperform OPCD, which uses both components (Table R1). PeCo's contributions are:

  1. A new source of supervision. PeCo uses three critics, each from a different model family and of the same scale as the target. None of the four works uses such a panel: Jin et al. [1] use a single critic, as do Kara and Ersoy [2]; Byeon et al. [3] pair each student with one peer from the same base model; and Lan et al. [4] combine models stronger than the model they train.
  2. Evidence that heterogeneity, not the number of critiques, drives the gain. Peers repair complementary failures: 38–40% of repaired failures are repaired by only one peer (Fig. 2). Critique usefulness transfers asymmetrically across models (Table 1). The panel outperforms single critics given the same number of critiques (W2/Q2, Table R2).

Comparison with OPCD [1] (Jin et al., 2026).

PeCo outperforms OPCD on three of the four targets with either prompt, by 1.6 points on Qwen3-4B, 2.8 points on Gemma-3-4B and 1.9 points on Phi-3.5-mini over the better prompt. On Llama-3.2-3B, OPCD is ahead by 0.1 points with our prompt and 0.7 points with its original prompt.

We reimplemented OPCD on all four targets with a single smaller critic from the target's own family, as in the original (Qwen3-1.7B, Gemma-3-1B and Llama-3.2-1B-Instruct; Phi-3.5-mini has no smaller model in its family, so we use Qwen3-1.7B), and with the same training problems, training steps and evaluation as PeCo. Because the prompt format can affect the results, we trained OPCD with both its original prompt and ours.

Table R1. Five-benchmark average of OPCD and PeCo.

TargetOPCD (OPCD's prompt)OPCD (our prompt)PeCo
Qwen3-4B50.449.552.0
Gemma-3-4B41.542.144.9
Llama-3.2-3B33.532.932.8
Phi-3.5-mini29.929.631.8

Both methods distill a critique-conditioned model into its unassisted self. Besides the critic panel (contribution 1), PeCo differs from OPCD in two design choices:

  1. Anchors. Problems the target already solves are kept as empty-context anchors, so the student also learns to reproduce its own correct reasoning, which limits forgetting. OPCD has no such anchor set.
  2. Offline construction. PeCo generates attempts, critiques and re-solves once before training, whereas OPCD regenerates critiques every epoch.

OPCD appeared on arXiv and at an ICML 2026 workshop without proceedings, so ICLR's reviewer guidelines do not require a comparison; we include the comparison (Table R1) because it is informative.

How PeCo differs from the other works the reviewer mentioned.

Kara and Ersoy (2026) [2]. They study how the teacher's context affects on-policy self-distillation, and find that a step-by-step critique of the solver's attempt works better than the reference solution. Their critic is a 32B model that sees the reference (gold) solution, so a comparison would mainly measure access to stronger supervision rather than the training method. Within the rebuttal period, we therefore prioritized OPCD. The nearest condition in our setup is the 32B critic ×9 baseline (Table R2), which PeCo outperforms on all four targets.

Byeon et al. (2026) [3] and Lan et al. (2025) [4]. Both works train with feedback from other models: Byeon et al. put a peer's feedback into the teacher's context in on-policy self-distillation, and Lan et al. merge critiques from four large models. These two works differ from PeCo in their training goal and setup, so a direct comparison would not be meaningful. Byeon et al. co-train two students, each specialized in a different domain, so that both improve across domains; PeCo trains a single target on one domain with frozen peers. Lan et al. train the critic itself to write better critiques and evaluate critique quality, so there is no solver accuracy to compare against.

AuthorsOpenReview ↗

Response to Reviewer t3uk (2/2)

[Update, Nov 12] We have extended the ×27 best-single-peer control to all four targets (Table R2). The rest of the response is unchanged.

W2/Q2. Number of critiques vs. heterogeneity.

At an equal number of critiques (the ×9 conditions), PeCo outperforms every homogeneous critic on all four targets, by 2.3, 3.1, 0.7 and 1.3 points over the best one, with the largest margins on Qwen3-4B and Gemma-3-4B.

Table R2. Five-benchmark average and total compute relative to PeCo.

CriticsCritiques/problemTraining rowsQwen3-4BGemma-3-4BLlama-3.2-3BPhi-3.5-miniCompute vs. PeCo
32B critic9all rescued49.741.732.130.34.26×
Self-critic9all rescued49.241.032.030.51.05×
Best single peer9all rescued49.641.831.630.11.26×
Best single peer27matched to PeCo48.542.532.530.63.08×
PeCo9all rescued52.044.932.831.81.00×

Compute is the estimated FLOPs of data construction and training, summed over the four targets (PeCo: 4,231 PFLOPs); the estimation is detailed in our response to Reviewer UF6d.

The ×9 conditions match PeCo's 9 critiques per failed problem, and all conditions use one re-solve per critique and the same verifier, anchor rule and training steps.

  • Self-critic ×9 is the same-model case.
  • 32B critic ×9 is the single larger critic (Qwen3-32B).
  • Best single peer ×9 is the panel member with the highest repair rate for each target (Gemma-3-4B for Qwen3-4B, Llama-3.2-3B and Phi-3.5-mini; Qwen3-4B for Gemma-3-4B).

To also match the number of training examples, we give the best single peer three times as many critiques (27) and subsample its rescued rows to PeCo's number, so it has more repair attempts than PeCo and the same amount of training data. Even so, it still trails PeCo on all four targets, by 3.5, 2.4, 0.3 and 1.2 points, while using 3.08 times as much compute.

PeCo's gain therefore does not come from more critiques, training data or compute. The reason is complementarity: the strongest peer accounts for only 81% (Qwen3-4B) and 73% (Gemma-3-4B) of the panel's repairs, and the rest are repaired only by other peers (Fig. 2).

We hope this clarifies how PeCo differs from prior work and where its gain comes from. We would be happy to discuss any remaining questions during the discussion period.

References

[1] Jin et al., "Weak Critics Make Strong Learners: On-Policy Critique Distillation for Scalable Oversight," ICML 2026 Workshop on AI for Math; arXiv:2606.00424.
[2] Kara and Ersoy, "The Role of Feedback Alignment in Self-Distillation," ICML 2026 Workshop on RL from World Feedback; arXiv:2606.11173.
[3] Byeon et al., "Be My Tutor: On-Policy Co-Distillation for Mutual LLM Improvement via Peer Feedback," arXiv:2606.14368, 2026.
[4] Lan et al., "Training Language Models to Critique With Multi-agent Feedback," Findings of EMNLP 2025.

AuthorsOpenReview ↗

Follow-up on Our Response

Dear Reviewer t3uk,

We understand this is a busy period, and we thank you again for your thoughtful review. As the discussion period ends on November 18, we wanted to briefly check whether our response has addressed your concerns. In summary:

  • Relation to prior work. We clarified how PeCo differs from the closest works and compared it directly with a reimplementation of OPCD (Jin et al.), which PeCo outperforms on three of the four targets.
  • Critique count vs. heterogeneity. At the same number of critiques, PeCo outperforms a 32B critic, the target itself and the best single peer on all four targets, and the best single peer still trails PeCo with three times as many critiques and the same training rows.

If any part of our response remains unclear, we would be glad to clarify before the discussion closes.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.