acceptodds
Under review as a conference paper at ICLR 2027

Consensus-Guided Preference Optimization for Multi-Source Language Model Alignment

Abstract

Large language models often possess complementary capabilities, yet combining those capabilities in one deployable model remains difficult when source models differ in architecture, specialization, and probability calibration. Most fusion approaches rely on parameter merging or supervised fine-tuning. Existing preference-stage alternatives depend critically on the quality of constructed chosen-rejected pairs. We introduce Consensus-Guided Preference Optimization (CGPO), a framework for multi-source language model alignment from unpaired preference supervision. CGPO turns source-model sequence probabilities into a confidence-calibrated consensus that directly guides each pointwise preference update. The consensus supplies a completion-specific optimization baseline, while pointwise verifier feedback identifies desirable and undesirable responses. This design transfers complementary capabilities from heterogeneous sources without parameter merging, vocabulary alignment, or artificially constructed preference pairs. Comprehensive experiments on seven benchmarks spanning mathematics, code generation, reasoning, and instruction following show that CGPO outperforms existing model-fusion and preference-optimization baselines. For Phi-3.5, CGPO raises the average performance across these benchmarks from 58.1 to 63.3. Our code is available at https://anonymous.4open.science/r/CGPO-6501/.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Large Language Models, Preference Optimization, Model Fusion, Model Alignment, Unpaired Preferences

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_aoHWOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper propose a multi-teacher guided preference alignment method, which uses several teacher models to help improve a target model. It combines their scores based on how confident they are, then uses that signal to adjust how strongly the target model is trained to avoid bad responses. Experiment over phi student model and Qwen etc teacher model that trained over UltraFeedback and test over some math, coding , QA benchmark is presented.

Critical strengths
  1. The paper clearly discusses the scope of previous works established.
  2. The visual presentation is overall clear, but the language is getting less understandable moving to the appendix.
Critical weaknesses

I consider the problem basically as a alignment with multi-teacher guidance. But the title is somehow misleading, as no benchmark covers any well-known preference alignment one (AlpacaEval, MT-Bench, etc). The experiment setup that trained on UltraFeedback while testing on math and coding benchmarks without any alignment benchmark is also questionable. Taking similar method such as WRPO, it includes all those traditional alignment benchmarks.

The actual usefulness and whether the conclusion holds to stronger models remain uncertain. For stronger model, I mean the one with same size as your current base model, such as Qwen3-4B. Also, three teacher models come from one base family, which might be bias towards consensus. It remains the concern whether this pipeline can work towards general multi-teacher framework.

There are so many techniques introduced in the paper, how would adding techniques one-by-one instead of removing each change the results? Also, after zoom into the appendix of ablation, I find the results hard to follow, especially for the ablation for appendix B.5, the formalism and unclear language makes it hard to understand what's aim of the results.

The larger model experiments in Appendix might raise concerns as SFT baseline can only raise the baseline by 1pp, did you ever tune the baseline methods?

Another issue is that since you have already calculated the logit from the teacher, I currently cannot see the advantage of your method over direct distillation.

Questions for authors

I am also curious how did you implement the baseline such as WRPO, which should only use the prompt from UltraFeedback and distill the response directly from the teacher models?

I also don't quite understand the motivation. If we got 2 good and 2 bad labels, a pairwise baseline could construct pairs only between good and bad answers.

What's the teacher model in Table 5? Are you scaling to stronger teacher?

Please also address all the weaknesses.

7 comments

We thank the reviewer for the careful assessment. We address the weaknesses and related questions in their original order.

W1. Evaluation scope and the meaning of “alignment.”

A1: We thank the reviewer for pointing out that our evaluation could better convey the scope of the paper. CGPO’s primary goal is to fuse complementary expert capabilities into a single target model. It uses response-level desirable/undesirable labels to determine the update direction and multi-source model signals to guide that update. Preference optimization is the training mechanism for capability fusion, rather than a claim that CGPO is uniformly superior on open-ended human-preference evaluations.

Accordingly, our original benchmarks measure whether the fused model acquires useful capabilities across mathematics, general reasoning, code generation, and instruction following. We nevertheless appreciate the reviewer’s suggestion: an open-ended conversational benchmark provides a valuable complementary perspective. We therefore added MT-Bench and conducted a separate experiment that adds Mistral-7B-Instruct-v0.3 to the original four sources. CGPO and all trainable baselines were retrained in this five-source setting; the results below should not be mixed with the paper’s original four-source results. MT-Bench values are GPT-4 single-answer ratings.

Role / modelMT-BenchGSM8KMMLUIFEvalMBPP
Target: Phi-3.5-mini8.4281.0569.549.268.5
Sources
Qwen2.5-7B-Base6.5184.173.533.173.3
Qwen2.5-7B-Instruct8.7490.573.471.278.9
Gemma-2-9B-IT8.5378.273.170.275.2
Qwen2.5-Math-7B-Instruct3.6691.639.736.15.3
Mistral-7B-Instruct-v0.37.6254.262.351.752.9
Retrained methods
SFT8.5182.770.450.169.3
SFT-DPO8.6883.270.051.570.2
SFT-KTO8.6383.270.950.369.1
WRPO8.7283.870.551.673.1
InfiFPO8.7084.769.252.775.5
CGPO8.7184.573.054.675.0

CGPO improves the target’s MT-Bench score from 8.42 to 8.71, while WRPO scores slightly higher at 8.72. Thus, the additional evaluation shows that capability fusion does not come at the expense of conversational quality in this setting. We discuss what the same experiment reveals about source diversity in our response to W2.

W2 and Q3. Stronger targets and source diversity.

A2: We thank the reviewer for raising these two distinct questions. Appendix Table 5 varies the target model, not the source pool: the 7B and 14B targets use the same four sources as the main 3.8B experiment, rather than a stronger teacher. This design tests whether CGPO’s benefit depends on a large size gap between the target and its sources. The 7B target is comparable in size to three of the sources, while the 14B target is larger than all four. CGPO still improves their seven-benchmark overall averages from 63.8 to 67.8 and from 66.1 to 68.6, respectively. Thus, the gains are not confined to a substantially smaller target learning from larger models. We will clarify the unchanged source pool in the Table 5 description.

The reviewer’s observation about source diversity is also helpful. Although all four original sources belong to model families different from the Phi target, three are from the Qwen family. To examine a broader source mixture, we added Mistral-7B-Instruct-v0.3 as a fifth source and retrained CGPO and the baselines. As the complete results table in W1 shows, Mistral has a different performance profile from the existing sources, and five-source CGPO improves on the unmodified target across all five reported evaluations, including MT-Bench (8.42 to 8.71), MMLU (69.5 to 73.0), and IFEval (49.2 to 54.6). It also achieves the highest MMLU and IFEval scores among the retrained methods. These results show that CGPO remains effective with an additional non-Qwen source; they do not attribute the improvement to Mistral alone.

W3. Component ablations

A3: We thank the reviewer for suggesting a forward-addition study. The paper’s removal ablations ask what is lost when a component is taken away from full CGPO. To complement them, we added components sequentially and evaluated each resulting configuration:

StageAddition relative to the preceding stageMath Avg.Reasoning Avg.
0No source-informed data allocation; both branches use the initial-policy reference; no curriculum52.469.1
1Add source-informed data allocation; keep the undesirable branch policy-anchored56.271.2
2Add uniform, unshifted source consensus to the undesirable branch55.061.2
3Replace uniform consensus with calibrated entropy weighting56.370.5
4Add prompt-level baseline shift56.571.3
5Add D-first/U-warmup57.071.7

The sequence shows why source guidance must be calibrated. Source-informed data allocation improves both averages at Stage 1. Simply inserting uniform, unshifted consensus then reduces reasoning performance sharply; calibrated weighting recovers much of that loss, baseline alignment improves both averages further, and the curriculum yields the full CGPO result. Thus, the forward study does not suggest that every added source signal is beneficial on its own. It shows how the proposed components turn an unreliable source signal into useful guidance.

These rows should not be equated with similarly named removal controls: Stages 0–4 have no curriculum, whereas the original ablations hold their stated full-training conditions fixed. We will also make Appendix B.5 easier to follow by stating its question before the formulas: given the same source-informed training data, should consensus enter the desirable branch, the undesirable branch, or both? Its reference-assignment comparison addresses that question, rather than repeating the forward-addition study.

W4. SFT baseline for larger targets.

A4: We thank the reviewer for asking us to clarify the SFT baseline in Appendix Table 5. For each target size, we initialized SFT from the corresponding unmodified target model: Phi-3.5-mini-instruct for the main experiment, and Phi-3-small-8k-instruct or Phi-3-medium-4k-instruct for the 7B and 14B experiments. We trained SFT for three epochs at a learning rate of 5×10−65\times10^{-6} on desirable UltraFeedback responses selected from approximately half of the training data. The resulting checkpoint is both the reported SFT baseline and the starting point for subsequent preference optimization; the remaining data are used for that stage. The 5×10−75\times10^{-7} learning rate reported for preference optimization is distinct from the SFT learning rate.

The 7B and 14B targets already have substantial capabilities before SFT. Training on desirable responses can refine their response behavior, but is not expected to teach mathematical reasoning or coding from scratch. This provides context for the SFT gains of 0.7 and 0.6 points in the seven-benchmark overall average. Starting from those same SFT checkpoints, CGPO improves the averages by a further 3.3 and 1.9 points, respectively. We will add the two-stage training setup to the paper so that the SFT and preference-stage results are easier to interpret.

W5 and Q1. Distillation and the WRPO implementation.

A5: We thank the reviewer for asking us to clarify what information CGPO takes from the source models. Access to source logits does not mean that we distill their full next-token distributions. For each completed response, CGPO obtains a length-normalized sequence log probability under each source’s own tokenizer and uses the resulting scalar in its consensus-guided objective. The sources can therefore differ in architecture and vocabulary without manual token or vocabulary alignment.

This differs from token-logit distillation, which transfers distributions over possible next tokens at each position. FuseLLM can also fuse heterogeneous models, but explicitly aligns their token sequences and predictive distributions to do so. CGPO does not align those full distributions, nor does it train the target to reproduce them. Instead, verifier labels determine whether each response is desirable or undesirable, while source sequence scores guide the correction of undesirable responses. In our comparison, CGPO achieves 63.3 overall in 36 training GPU hours, compared with FuseLLM’s 60.3 in 162 training GPU hours. These figures demonstrate the observed training-efficiency difference.

For WRPO, we used UltraFeedback prompts to generate new candidates from both the source and target models. Following WRPO’s procedure, reward-based ranking selected a preferred source response and preferred and rejected target responses, forming the quadruples used by its objective. We did not turn CGPO’s independently labeled desirable and undesirable responses into WRPO pairs. The methods share the target and evaluation setting, while each retains its own training-data construction. We will clarify this in the paper.

Q2: the motivation for pointwise fusion

A6: We thank the reviewer for this thoughtful question. We agree that good–bad pairs can be formed when both labels are available; our motivation concerns what happens when a prompt’s candidate responses share the same verifier label.

The reviewer is right that good–bad pairs can be constructed when a prompt contains both kinds of response. The difficulty arises when the candidate set is homogeneous. InfiFPO selects highest- and lowest-reward candidates, while WRPO uses reward-ranked source and target candidates to construct its preference comparisons. These rankings identify relative winners and losers; they do not, by themselves, require the winner to pass an independent desirable-response test. If every candidate for a prompt is undesirable, a within-prompt ranking can still promote a relatively better but undesirable response. The symmetric issue arises when every candidate is desirable: a relatively lower-ranked response can be suppressed despite being desirable.

Restricting pairs to independently verified desirable–undesirable responses avoids those label conflicts, but then prompts containing only desirable or only undesirable candidates provide no within-prompt pair. CGPO takes the other route: it retains each completion’s independent verifier label and learns from desirable and undesirable responses separately, without requiring an opposite-label partner. The label determines whether a response is promoted or suppressed; source consensus guides the correction priority of undesirable responses. This is the pointwise motivation, not a claim that valid preference pairs can never be formed.

Reviewer_XPguOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes a Consensus-Guided Preference Optimization (CGPO) method designed to transfer knowledge from heterogeneous language models through unpaired preference optimization. The method is built upon KTO and utilizes independently labeled ideal and non-ideal responses. Calibration consensus on the source model’s sequence probabilities guides the suppression of non-ideal responses. The method also incorporates entropy-based expert weights, prompt-level source-to-target baseline alignment, and a D-first/U-warmup training scheme. Experiments conducted on heterogeneous source models across seven benchmarks demonstrate that this method outperforms various fusion and preference optimization baseline methods.

Critical strengths

1.Effective unpaired multi-source alignment.

CGPO separates the update directions provided by the validator from the optimization process guided by the source model, thereby eliminating the need for paired preferences. This enables knowledge transfer between models with different parameters or vocabularies. I believe this is extremely useful in practical applications, especially when integrating models developed independently.

2.Extensive experiments.

Across seven benchmarks, CGPO achieved the highest overall average score among the trainable baseline models evaluated, including InfiFPO. The improvements at the 7B and 14B scales are equally encouraging, indicating that the method is not limited to a single target scale.

Critical weaknesses

1.Inconsistent ablations.

It`s difficult to understand these “No source consensus” ablation. Section 5.3 (Figure 3(a)) shows that performance drops significantly when source consensus is removed. However, Appendix C.1 (Table 4) reports that the “policy-based reference model” scored 62.9, while Appendix B.5 describes it as the “source-free” control group. The “no-source consensus” ablation experiment scored 59.9, while the “policy-anchored reference” scored 62.9. Under the reported matching setup, adding the source-guided reference model appears to improve performance by only 0.4 points. This is not a trivial reporting issue. I cannot determine whether CGPO’s performance improvement stems primarily from multi-source knowledge transfer or from its underlying point-by-point optimization method. This distinction is crucial to the paper’s central argument.

2.Limitation for the proposed mechanism.

Ablation experiments show that certain design choices affect performance but do not directly validate the proposed mechanism. In particular, I cannot determine whether entropy-weighted consensus actually tracks source expertise. The paper also lacks diagnostic analysis linking source weights or consensus margins to expert reliability and downstream correction behavior. I am also unable to determine whether source guidance remains effective when faced with well-matched, unsourced references. These omissions are critical to my evaluation. The mechanism sounds plausible, but the current evidence does not sufficiently demonstrate why it works or how robust it is.

Questions for authors

Q1. Could the authors explain the source results the in Figure 3(a) and Table 4?

The “No source consensus” ablation and the Policy-Anchored Reference both appear to remove source-dependent guidance. Why are their scores 59.9 and 62.9 respectively? Please clarify whether there are specific differences in the objective function, data, training plan, hyperparameters, and evaluation settings. This difference is at the heart of my assessment.

Q2. What evidence supports the entropy-based expert weighting?

Could the authors examine whether calibrated expert weights are related to source correctness or task expertise? I'd also like to understand how these weights affect gradient modulation in practice. Here it would be helpful to compare with a confidence-based or difficulty-based weighting approach based only on the target. Without these comparisons, I have a hard time distinguishing true multi-source knowledge transfer from more general example reweighting.

Q3. Could the authors clarify the exactimplementations and training configurations?

I`m willing to raise my scores if the questions can be addressed.

3 comments

Thank you for recognizing the value of unpaired multi-source alignment and the breadth of the evaluation. We address the two weaknesses together with your corresponding questions, followed by the request for implementation details.

W1 / Q1. Interpreting Figure 3(a) and Table 4.

A1: Thank you for carefully comparing Figure 3(a) with Table 4. Your apparent discrepancy makes it particularly important to clarify which part of source guidance each experiment removes. These experiments remove source guidance at different stages. Figure 3(a)’s “No source consensus” control removes source influence from both data-side allocation and the reward reference. Table 4 instead holds source-informed data allocation fixed and varies only the references assigned to the desirable (D) and undesirable (U) branches:

SettingSource-informed data allocationD referenceU referenceOverall Avg.
Figure 3(a): No Source GuidanceNoInitial policyInitial policy59.9
Table 4: Policy-AnchoredYesInitial policyInitial policy62.9
Table 4: Consensus-AnchoredYesConsensusConsensus63.1
Table 4: CGPOYesInitial policyConsensus63.3

The term “source-free” in Appendix B.5 was intended to contrast the Policy-Anchored reward reference with references that incorporate consensus, not to describe the entire training pipeline. We see how that shorter description could obscure the distinction. We will qualify it in the appendix and rename the Figure 3(a) control “No Source Guidance,” while retaining Table 4’s reference-assignment names.

The comparisons consequently answer different questions: 59.9 versus 62.9 reflects the presence of data-side source guidance in these settings; the matched 62.9 versus 63.3 comparison tests the additional label-conditional source reference. We will state the data construction, reference, and matched training and evaluation conditions alongside these results.

W2 / Q2. Does entropy-weighted consensus track expertise and affect correction?

A2:

We appreciate your question about whether entropy weighting identifies useful expertise or merely changes example importance. We address these possibilities with complementary weight analyses and controlled comparisons. Our paper defines entropy as a relative confidence signal, not as a correctness label. To test whether the resulting weights track task-level expertise, we examined the mean source weights on each benchmark:

TaskBest-performing source (score)BaseGemma-ITQwen-InstructQwen-Math
GSM8KQwen-Math (91.6)0.210.180.300.31
MATHQwen-Math (83.3)0.190.170.290.35
TheoremQAQwen-Instruct (36.7)0.260.220.290.23
BBHQwen-Instruct (70.6)0.280.240.290.19
MMLUBase (73.5)0.280.250.290.18
IFEvalQwen-Instruct (71.2)0.200.270.310.22
MBPPQwen-Instruct (78.9)0.260.270.310.16

The highest-weighted source is the best-scoring source on six of seven tasks. MMLU is the exception: Base scores 73.5 and receives weight 0.28, while Qwen-Instruct receives 0.29. The weights also respond to differing source strengths: Qwen-Math receives 0.35 on MATH but 0.16 on MBPP.

We further retrained after randomly shuffling the expert-weight assignments while fixing the other experimental conditions:

WeightingOverall Avg.
CGPO63.3
Uniform expert weights57.4
Shuffled expert-weight assignments57.2

Shuffling nearly eliminates the advantage over uniform weighting. This supports the importance of which expert receives a weight.

To address your alternative explanation more directly, we added controls using the same source-generated response pool, D/U labels, SFT initialization, training budget, β\beta, warmup, and evaluation procedure. Target-derived signals are computed offline from the frozen initial policy π0\pi_0. The weighting controls match the mean and dispersion of their weights, with weighting direction and strength selected on validation data.

U-branch guidanceMath Avg.Reasoning Avg.IFEvalMBPPOverall Avg.
Source-confidence-only weighting56.671.353.773.562.8
Target-entropy weighting56.471.253.473.462.6
Target-difficulty weighting56.371.053.373.362.5
Target-only value reference56.571.353.673.962.8
CGPO57.071.754.374.763.3

The first control retains expert-confidence information but uses π0\pi_0, rather than source log probabilities, as the reward reference; its confidence concentration only weights U examples. The next two controls use target entropy or target negative mean log probability to weight U examples. The final control uses a centered target-only difficulty signal to change the U reference itself, making it a closer test of whether changing the value boundary is sufficient. CGPO exceeds all four controls overall by 0.5–0.8 points, with gains in every reported category. This supports a contribution from source-dependent sequence-score guidance in the U reference beyond the tested forms of confidence weighting, target difficulty weighting, and target-only value guidance.

As derived in the paper, the source reference changes the U margin and therefore the gradient factor gβ(mU)=σ(βmU)[1−σ(βmU)]g_\beta(m_U)=\sigma(\beta m_U)[1-\sigma(\beta m_U)], while the verifier label fixes the correction direction. The factor increases when the reference moves an instance’s margin toward zero and decreases when it moves the margin away.

Q3. Implementation and training conditions.

A3:

Thank you for asking us to clarify the implementation details. For the original four-source experiments, we start from Phi-3.5-mini-instruct and train SFT for three epochs on desirable responses selected from approximately half of the UltraFeedback prompts, using a learning rate of 5×10−65\times10^{-6} . The resulting checkpoint is both the reported SFT baseline and the initial policy for subsequent preference optimization; the remaining prompts are used for that stage.

CGPO uses independently labeled desirable and undesirable completions. The four frozen sources score each completed text offline using mean completion-token log probability under their respective tokenizers. The preference-training learning rate is 5×10−75\times10^{-7}, distinct from the SFT learning rate. The CGPO configuration uses β=0.08\beta=0.08, desirable/undesirable weights of 1.2/1.0, an entropy temperature of 0.8, a weight floor of 0.02, and a 100-step undesirable-branch warmup.

Preference-optimization baselines start from the same SFT checkpoint but retain their method-specific objectives and data construction. In particular, WRPO generates source and target candidates from UltraFeedback prompts and uses reward-based ranking to construct its training quadruples; we do not convert CGPO’s independently labeled completions into WRPO pairs. For the reference-assignment comparison in Table 4, source-informed data allocation and the other training conditions are held fixed, while the D and U references vary. We will make these distinctions explicit in the revised experimental setup and appendix.

Reviewer_NnDSOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces Consensus-Guided Preference Optimization (CGPO), a framework for multi-source language model alignment under unpaired preference supervision. CGPO uses the initial policy as an anchor for desirable completions and leverages source consensus to guide the suppression of undesirable completions. To calibrate the consensus and stabilize optimization, it integrates entropy-based expert weighting, prompt-level baseline alignment, and a desirable-first curriculum. Evaluations on 7 benchmarks demonstrate its effectiveness across mathematics, code generation, general reasoning, and instruction following.

Critical strengths
  1. The paper explores capability fusion among heterogeneous language models, a problem of substantial practical relevance, as it integrates the complementary strengths of multiple experts while maintaining the deployment cost of a single model.

  2. The method is clear, combining the optimization direction determined by preference labels with guidance derived from multi-source consensus. This provides a natural approach to model fusion under unpaired preference supervision.

  3. The experiments cover multiple domains, including mathematics, code generation, reasoning, and instruction following, and demonstrate improvements across target models of different sizes, providing comprehensive evidence of the method’s effectiveness.

Critical weaknesses
  1. In my opinion, the key ablation results are inconsistent. Removing consensus in Figure 3 yields an overall score of 59.9, whereas the Policy-Anchored baseline without consensus in Table 4 achieves 62.9. This discrepancy requires a clear explanation.

  2. The method determines expert weights using predictive entropy, but lower entropy does not necessarily indicate stronger task capabilities. The paper lacks direct analyses and experiments demonstrating that these weights can effectively identify experts with an advantage on different tasks.

  3. Length normalization and mean shifting are insufficient to eliminate differences arising from tokenizers and model scoring scales. The comparability of sequence scores across models therefore remains insufficiently justified.

  4. Multi-source consensus is applied only to the negative-sample branch, without directly leveraging expert support for positive samples to reinforce high-quality responses. Consequently, the method makes limited use of experts’ positive preference information.

Questions for authors

As discussed in the weaknesses, my concerns are twofold. First, could the authors clarify the inconsistencies in the experimental results and explain whether the evidence adequately supports their conclusions? Second, could they provide deeper analyses or targeted experiments to substantiate the key insights behind the method and make the rationale for its design choices more intuitive and convincing?

3 comments

We thank the reviewer for recognizing the practical relevance of heterogeneous model fusion, the separation of verifier-defined update directions from source guidance, and the breadth of our evaluation. We address the four concerns in order.

W1. Why do Figure 3(a) and Table 4 report 59.9 and 62.9 for apparently source-free settings?

A1: Thank you for highlighting this ambiguity. The two experiments remove source information at different stages.

Figure 3(a) evaluates source guidance as a whole. The setting currently labeled “No source consensus” removes source influence from both training-data allocation and the reward reference, and obtains 59.9 overall. Appendix Table 4 instead isolates reference assignment: its three methods use the same source-informed data allocation and differ only in the references assigned to the desirable (D) and undesirable (U) branches.

SettingSource-informed data allocationReward referenceOverall Avg.
Figure 3(a): No Source GuidanceNoInitial policy for D and U59.9
Table 4: Policy-AnchoredYesInitial policy for D and U62.9
Table 4: Consensus-AnchoredYesConsensus for D and U63.1
Table 4: Label-Conditional (CGPO)YesInitial policy for D; consensus for U63.3

In Appendix B.5, “source-free” was used in the narrower context of the reward reference: Policy-Anchored has no source-consensus reference, in contrast to Consensus-Anchored and CGPO. It was not intended to describe its data allocation. We appreciate that, without this qualification, the phrase can be read as removing sources from the entire training pipeline, which is precisely the intervention in Figure 3(a).

We will label the Figure 3(a) control “No Source Guidance” and clarify the scope of “source-free” in the Appendix B.5–C.1 prose and figure discussion. With the experimental conditions made explicit, 59.9 versus 62.9 compares two different interventions, while the matched 62.9 versus 63.3 comparison isolates the additional contribution of the source-consensus reference on the U branch. The full 59.9 versus 63.3 comparison measures the combined contribution of source-informed data allocation and value guidance.

W2. What the entropy weights represent.

A2:

Thank you for raising this question. Our method uses calibrated predictive entropy as a relative confidence signal, we do not treat low entropy as a correctness label. The relevant empirical question is whether the resulting weights tend to favor sources with stronger task performance.

We have added the following task-level analysis. Each row reports the source with the highest benchmark score and the mean weight assigned to each source.

TaskBest-performing source (score)BaseGemma-ITQwen-InstructQwen-Math
GSM8KQwen-Math (91.6)0.210.180.300.31
MATHQwen-Math (83.3)0.190.170.290.35
TheoremQAQwen-Instruct (36.7)0.260.220.290.23
BBHQwen-Instruct (70.6)0.280.240.290.19
MMLUBase (73.5)0.280.250.290.18
IFEvalQwen-Instruct (71.2)0.200.270.310.22
MBPPQwen-Instruct (78.9)0.260.270.310.16

The highest-weighted source is also the best-performing source on six of seven tasks. The MMLU exception is narrow: Base and Qwen-Instruct differ by 0.1 points in benchmark score and by 0.01 in mean weight. The domain pattern is informative as well: Qwen-Math receives the largest weight on MATH (0.35), but substantially less weight on BBH (0.19) and MBPP (0.16), where other sources perform better.

We also retrained with the expert weights randomly shuffled while keeping the other experimental conditions fixed. Overall Avg. fell from 63.3 to 57.2, close to the 57.4 obtained with uniform weights. Thus, the gain depends on how weights are assigned to experts, rather than merely on using a nonuniform set of weight values. Together, the task-level analysis and this controlled retraining support the intended role of confidence calibration in directing source influence toward relevant expertise.

W3. The role of sequence-level scores and calibration.

A3:

Thank you for raising this concern. CGPO scores the same completed text under each model’s own tokenizer and uses the mean log probability of its completion tokens. This provides a sequence-level signal without requiring token-by-token vocabulary alignment. The prompt-level shift then aligns the source consensus and initial-policy means over the same candidate set, while preserving differences between candidates. Its purpose is to remove a prompt-common offset, rather than to assert that heterogeneous models produce identically calibrated probabilities.

The use of average sequence log probability as an optimization signal is supported by SimPO; InfiFPO also demonstrates sequence-level fusion across heterogeneous sources. We agree that these precedents do not replace a test of CGPO’s sensitivity to residual scoring differences. We therefore added three controlled score transformations:

  1. Per-expert centering: subtract each expert’s mean score over the same prompt’s candidate set before aggregation, while retaining CGPO’s original post-aggregation shift. This probes expert-specific offsets that can still matter when weights vary across completions.
  2. Per-expert scale matching: use means and standard deviations from an independent calibration split to map each expert’s scores to the initial policy’s score scale, keeping the entropy weights and training configuration fixed.
  3. Byte normalization with scale matching: divide each source’s total log probability by the UTF-8 byte length of the same completion, then apply the scale matching above. This tests sensitivity to the scoring unit; byte normalization is not assumed to calibrate probabilities exactly.
Source scoringMath Avg.Reasoning Avg.IFEvalMBPPOverall Avg.
Original CGPO57.071.754.374.763.3
Per-expert centering57.171.854.274.763.4
Per-expert scale matching57.271.954.474.763.5
Byte normalization + scale matching56.871.554.274.163.1

Overall Avg. remains between 63.1 and 63.5 across these transformations, compared with 63.3 under the original scoring rule. The byte-normalized variant shows a larger change on MBPP (74.1 versus 74.7), but the aggregate result remains close to the original. These results do not establish identical score scales across models; they show that CGPO’s observed performance is stable under the tested changes to expert offsets, score scales, and tokenization-dependent scoring units. We will clarify this intended scope of “comparability” in the paper.

References

[1] Yu Meng, Mengzhou Xia, and Danqi Chen. SimPO: Simple Preference Optimization with a Reference-Free Reward. Advances in Neural Information Processing Systems 37 (NeurIPS 2024).

[2] Yanggan Gu, Yuanyi Wang, Zhaoyi Yan, Yiming Zhang, Qi Zhou, Fei Wu, and Hongxia Yang. InfiFPO: Implicit Model Fusion via Preference Optimization in Large Language Models. Advances in Neural Information Processing Systems 38 (NeurIPS 2025).

W4. Why source consensus directly guides the U reference.

A4: Thank you for raising this question. The desirable branch already receives a positive update from its independent verifier label and is anchored to the frozen initial policy. Source information also affects which examples enter training through the shared source-informed data allocation. For the reward reference, however, placing a source score on a desirable completion does not necessarily reinforce its update: the pointwise gradient depends on the resulting margin, so changing that reference can either increase or decrease its magnitude. The U branch is where consensus has the intended role of distinguishing verified errors that sources strongly reject from those they still support.

Appendix Table 4 tests the alternative of putting consensus in both reward references. Consensus-Anchored reaches 63.1 overall, while the label-conditional assignment reaches 63.3 under the same data allocation. We will clarify that “consensus enters only the U branch” refers specifically to the reward reference, not to every way source information participates in training.

Reviewer NnDS, thanks for your review. Could you take a look at the authors’ response and comment on which of your concerns have or have not been resolved?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.