- Rating
- 3(+1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper propose a multi-teacher guided preference alignment method, which uses several teacher models to help improve a target model. It combines their scores based on how confident they are, then uses that signal to adjust how strongly the target model is trained to avoid bad responses. Experiment over phi student model and Qwen etc teacher model that trained over UltraFeedback and test over some math, coding , QA benchmark is presented.
Critical strengths
- The paper clearly discusses the scope of previous works established.
- The visual presentation is overall clear, but the language is getting less understandable moving to the appendix.
Critical weaknesses
I consider the problem basically as a alignment with multi-teacher guidance. But the title is somehow misleading, as no benchmark covers any well-known preference alignment one (AlpacaEval, MT-Bench, etc). The experiment setup that trained on UltraFeedback while testing on math and coding benchmarks without any alignment benchmark is also questionable. Taking similar method such as WRPO, it includes all those traditional alignment benchmarks.
The actual usefulness and whether the conclusion holds to stronger models remain uncertain. For stronger model, I mean the one with same size as your current base model, such as Qwen3-4B. Also, three teacher models come from one base family, which might be bias towards consensus. It remains the concern whether this pipeline can work towards general multi-teacher framework.
There are so many techniques introduced in the paper, how would adding techniques one-by-one instead of removing each change the results? Also, after zoom into the appendix of ablation, I find the results hard to follow, especially for the ablation for appendix B.5, the formalism and unclear language makes it hard to understand what's aim of the results.
The larger model experiments in Appendix might raise concerns as SFT baseline can only raise the baseline by 1pp, did you ever tune the baseline methods?
Another issue is that since you have already calculated the logit from the teacher, I currently cannot see the advantage of your method over direct distillation.
Questions for authors
I am also curious how did you implement the baseline such as WRPO, which should only use the prompt from UltraFeedback and distill the response directly from the teacher models?
I also don't quite understand the motivation. If we got 2 good and 2 bad labels, a pairwise baseline could construct pairs only between good and bad answers.
What's the teacher model in Table 5? Are you scaling to stronger teacher?
Please also address all the weaknesses.
7 comments
W3. Component ablations
A3: We thank the reviewer for suggesting a forward-addition study. The paper’s removal ablations ask what is lost when a component is taken away from full CGPO. To complement them, we added components sequentially and evaluated each resulting configuration:
| Stage | Addition relative to the preceding stage | Math Avg. | Reasoning Avg. |
|---|---|---|---|
| 0 | No source-informed data allocation; both branches use the initial-policy reference; no curriculum | 52.4 | 69.1 |
| 1 | Add source-informed data allocation; keep the undesirable branch policy-anchored | 56.2 | 71.2 |
| 2 | Add uniform, unshifted source consensus to the undesirable branch | 55.0 | 61.2 |
| 3 | Replace uniform consensus with calibrated entropy weighting | 56.3 | 70.5 |
| 4 | Add prompt-level baseline shift | 56.5 | 71.3 |
| 5 | Add D-first/U-warmup | 57.0 | 71.7 |
The sequence shows why source guidance must be calibrated. Source-informed data allocation improves both averages at Stage 1. Simply inserting uniform, unshifted consensus then reduces reasoning performance sharply; calibrated weighting recovers much of that loss, baseline alignment improves both averages further, and the curriculum yields the full CGPO result. Thus, the forward study does not suggest that every added source signal is beneficial on its own. It shows how the proposed components turn an unreliable source signal into useful guidance.
These rows should not be equated with similarly named removal controls: Stages 0–4 have no curriculum, whereas the original ablations hold their stated full-training conditions fixed. We will also make Appendix B.5 easier to follow by stating its question before the formulas: given the same source-informed training data, should consensus enter the desirable branch, the undesirable branch, or both? Its reference-assignment comparison addresses that question, rather than repeating the forward-addition study.
W4. SFT baseline for larger targets.
A4: We thank the reviewer for asking us to clarify the SFT baseline in Appendix Table 5. For each target size, we initialized SFT from the corresponding unmodified target model: Phi-3.5-mini-instruct for the main experiment, and Phi-3-small-8k-instruct or Phi-3-medium-4k-instruct for the 7B and 14B experiments. We trained SFT for three epochs at a learning rate of on desirable UltraFeedback responses selected from approximately half of the training data. The resulting checkpoint is both the reported SFT baseline and the starting point for subsequent preference optimization; the remaining data are used for that stage. The learning rate reported for preference optimization is distinct from the SFT learning rate.
The 7B and 14B targets already have substantial capabilities before SFT. Training on desirable responses can refine their response behavior, but is not expected to teach mathematical reasoning or coding from scratch. This provides context for the SFT gains of 0.7 and 0.6 points in the seven-benchmark overall average. Starting from those same SFT checkpoints, CGPO improves the averages by a further 3.3 and 1.9 points, respectively. We will add the two-stage training setup to the paper so that the SFT and preference-stage results are easier to interpret.
W5 and Q1. Distillation and the WRPO implementation.
A5: We thank the reviewer for asking us to clarify what information CGPO takes from the source models. Access to source logits does not mean that we distill their full next-token distributions. For each completed response, CGPO obtains a length-normalized sequence log probability under each source’s own tokenizer and uses the resulting scalar in its consensus-guided objective. The sources can therefore differ in architecture and vocabulary without manual token or vocabulary alignment.
This differs from token-logit distillation, which transfers distributions over possible next tokens at each position. FuseLLM can also fuse heterogeneous models, but explicitly aligns their token sequences and predictive distributions to do so. CGPO does not align those full distributions, nor does it train the target to reproduce them. Instead, verifier labels determine whether each response is desirable or undesirable, while source sequence scores guide the correction of undesirable responses. In our comparison, CGPO achieves 63.3 overall in 36 training GPU hours, compared with FuseLLM’s 60.3 in 162 training GPU hours. These figures demonstrate the observed training-efficiency difference.
For WRPO, we used UltraFeedback prompts to generate new candidates from both the source and target models. Following WRPO’s procedure, reward-based ranking selected a preferred source response and preferred and rejected target responses, forming the quadruples used by its objective. We did not turn CGPO’s independently labeled desirable and undesirable responses into WRPO pairs. The methods share the target and evaluation setting, while each retains its own training-data construction. We will clarify this in the paper.
Q2: the motivation for pointwise fusion
A6: We thank the reviewer for this thoughtful question. We agree that good–bad pairs can be formed when both labels are available; our motivation concerns what happens when a prompt’s candidate responses share the same verifier label.
The reviewer is right that good–bad pairs can be constructed when a prompt contains both kinds of response. The difficulty arises when the candidate set is homogeneous. InfiFPO selects highest- and lowest-reward candidates, while WRPO uses reward-ranked source and target candidates to construct its preference comparisons. These rankings identify relative winners and losers; they do not, by themselves, require the winner to pass an independent desirable-response test. If every candidate for a prompt is undesirable, a within-prompt ranking can still promote a relatively better but undesirable response. The symmetric issue arises when every candidate is desirable: a relatively lower-ranked response can be suppressed despite being desirable.
Restricting pairs to independently verified desirable–undesirable responses avoids those label conflicts, but then prompts containing only desirable or only undesirable candidates provide no within-prompt pair. CGPO takes the other route: it retains each completion’s independent verifier label and learns from desirable and undesirable responses separately, without requiring an opposite-label partner. The label determines whether a response is promoted or suppressed; source consensus guides the correction priority of undesirable responses. This is the pointwise motivation, not a claim that valid preference pairs can never be formed.
We thank the reviewer for the careful assessment. We address the weaknesses and related questions in their original order.
A1: We thank the reviewer for pointing out that our evaluation could better convey the scope of the paper. CGPO’s primary goal is to fuse complementary expert capabilities into a single target model. It uses response-level desirable/undesirable labels to determine the update direction and multi-source model signals to guide that update. Preference optimization is the training mechanism for capability fusion, rather than a claim that CGPO is uniformly superior on open-ended human-preference evaluations.Accordingly, our original benchmarks measure whether the fused model acquires useful capabilities across mathematics, general reasoning, code generation, and instruction following. We nevertheless appreciate the reviewer’s suggestion: an open-ended conversational benchmark provides a valuable complementary perspective. We therefore added MT-Bench and conducted a separate experiment that adds Mistral-7B-Instruct-v0.3 to the original four sources. CGPO and all trainable baselines were retrained in this five-source setting; the results below should not be mixed with the paper’s original four-source results. MT-Bench values are GPT-4 single-answer ratings.
CGPO improves the target’s MT-Bench score from 8.42 to 8.71, while WRPO scores slightly higher at 8.72. Thus, the additional evaluation shows that capability fusion does not come at the expense of conversational quality in this setting. We discuss what the same experiment reveals about source diversity in our response to W2.
A2: We thank the reviewer for raising these two distinct questions. Appendix Table 5 varies the target model, not the source pool: the 7B and 14B targets use the same four sources as the main 3.8B experiment, rather than a stronger teacher. This design tests whether CGPO’s benefit depends on a large size gap between the target and its sources. The 7B target is comparable in size to three of the sources, while the 14B target is larger than all four. CGPO still improves their seven-benchmark overall averages from 63.8 to 67.8 and from 66.1 to 68.6, respectively. Thus, the gains are not confined to a substantially smaller target learning from larger models. We will clarify the unchanged source pool in the Table 5 description.The reviewer’s observation about source diversity is also helpful. Although all four original sources belong to model families different from the Phi target, three are from the Qwen family. To examine a broader source mixture, we added Mistral-7B-Instruct-v0.3 as a fifth source and retrained CGPO and the baselines. As the complete results table in W1 shows, Mistral has a different performance profile from the existing sources, and five-source CGPO improves on the unmodified target across all five reported evaluations, including MT-Bench (8.42 to 8.71), MMLU (69.5 to 73.0), and IFEval (49.2 to 54.6). It also achieves the highest MMLU and IFEval scores among the retrained methods. These results show that CGPO remains effective with an additional non-Qwen source; they do not attribute the improvement to Mistral alone.