- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper uses verified critiques from heterogeneous small models for on-policy self-distillation, with anchor examples to reduce forgetting.
Critical strengths
The paper evaluates multiple model families and includes useful ablations on critique sources and anchor examples.
Critical weaknesses
See Questions For Authors
Questions for authors
Data construction already produces verified repaired answers from the target model. How does directly fine-tuning on these answers, together with the initially correct answers, compare with PECO under the same anchor/rescued mixture and matched training budget? This would clarify the benefit of critique-conditioned distillation over simply learning from successful repairs.
Response to Reviewer UF6d (1/2)
[Update, Nov 12] We have extended the ×27 best-single-peer control to Llama-3.2-3B and Phi-3.5-mini and updated its compute (Table R3). The rest of the response is unchanged.
We thank the reviewer for the careful and constructive review, and for recognizing that PeCo provides useful supervision without a larger teacher, which could make critique-guided improvement accessible when a stronger teacher is unavailable. Below, we show that PeCo's gain comes from the diversity of its panel: PeCo outperforms single critics sampled repeatedly at the same number of critiques, while using the least compute. We also show that its gains on Qwen3-4B and Gemma-3-4B exceed one training-seed standard deviation.
W1/Q. Heterogeneous panel vs. homogeneous critics at a matched budget.
PeCo's gain comes from the diversity of its panel, not from additional repair opportunities. Peers from different families repair complementary failures: the strongest peer accounts for only 81% (Qwen3-4B) and 73% (Gemma-3-4B) of the panel's repairs, and the rest are repaired only by other peers (Fig. 2).
The controls the reviewer asked for confirm this (Table R3): PeCo outperforms every homogeneous critic at the same number of critiques and a single peer given three times as many critiques, while using the least compute.
Every condition uses one re-solve per critique, the same verifier, and the same anchor rule. The ×9 conditions match PeCo's 9 critiques per failed problem and, like PeCo, train on all their rescued rows.
Table R3. Five-benchmark average and total compute relative to PeCo.
Compute covers data construction and training, summed over the four targets (PeCo: 4,231 PFLOPs). Compute is estimated: 2N FLOPs per token for inference, and 4N plus a 2N teacher forward pass per token for LoRA training (N = number of parameters), without the attention term quadratic in sequence length. Training-rollout lengths were not stored, so we estimate them from base-model samples.
Diversity beats repeated sampling. At an equal number of critiques, PeCo outperforms the best homogeneous critic on all four targets (+2.3, +3.1, +0.7 and +1.3).
More repair opportunities do not close the gap. To separate diversity from additional repair opportunities, we give the best single peer three times as many critiques (×27) and subsample its rescued rows to PeCo's number, so it has more repair opportunities than PeCo and the same training data. It still trails PeCo on all four targets, by 3.5, 2.4, 0.3 and 1.2 points.
PeCo uses the least compute. The 32B critic uses 4.26 times and the ×27 single peer 3.08 times as much compute as PeCo. The self-critic is nearly FLOP-matched, at 1.05 times PeCo's compute, and trails PeCo on all four targets, by 2.8, 3.9, 0.8 and 1.3 points. We match critique count rather than tokens because critique length differs across models; since every baseline already uses at least as much compute as PeCo, matching FLOPs would only reduce their budget.