acceptodds
Under review as a conference paper at ICLR 2027

TFPO: Token-Level Objective Fusion for Stable Preference Alignment

Abstract

Preference optimization is usually applied uniformly to entire responses, although explanatory text, final answers, formatting fields, and code need not benefit from the same training signal. We propose TFPO (Token-Fused Preference Optimization), which learns a lightweight gate that routes each response token between a DPO-style preference objective and a chosen-response likelihood anchor. Training uses no token-level supervision and includes ratio, smoothness, and entropy regularization. On the matched ten-benchmark suite, TFPO scores 88.00 versus 83.08 for SimPO, the strongest baseline without NLL anchoring, and 83.52 for SimPO+NLL (three-seed means). It also improves external alignment across Qwen, Llama, and Mistral backbones and multimodal performance. Under repeated sampling, TFPO raises answer agreement and majority-answer accuracy while preserving explanation diversity. On 1,000 responses from five extractable-answer tasks, annotated blind to gate values and model identity, its content gate recovers answer spans at 0.85 AUPRC versus 0.55 for the strongest strict position-only control. These results support token-level objective routing as a practical way to improve both preference alignment and answer stability.

Primary area:
foundation or frontier models, including LLMs
Keywords:
preference optimization, large language models, token-level alignment, direct preference optimization, answer stability

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_CQMUOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Token-Fused Preference Optimization (TFPO), which combines token-level weighting with a mixture of preference and likelihood objectives. The proposed core design is a lightweight gate, learned jointly with the policy from detached decoder hidden states without token-level supervision. The gate assigns complementary weights to tokens' contributions to a DPO-style sequence-level preference comparison and a chosen response negative log-likelihood objective. The intended behavior is to apply preference shaping to explanatory content while anchoring answer-bearing or format-sensitive tokens. The paper evaluates TFPO on text capability benchmarks, external alignment across three open-source models, and multimodal tasks, reporting improvements over several preference-optimization baselines.

Critical strengths
  1. The proposed gate is lightweight and requires no token annotations during training. It provides a simple way to allocate preference and likelihood objectives within a response.

  2. The evaluation covers text capability, external alignment across three model families, and multimodal tasks. The reported improvements are broad, and the main text capability results include three training seeds.

  3. The routing analysis includes blind answer span annotations, position based controls, and answer relocation experiments. These complementary checks provide evidence that the learned gate identifies answer content rather than relying only on its position.

Critical weaknesses
  1. The experiments do not sufficiently isolate the contribution of learned routing. Differences in loss normalization and tuning budgets complicate comparisons with the NLL baselines, while the routing controls mainly evaluate span identification rather than downstream performance. It therefore remains unclear how much of the reported improvement can be attributed specifically to the learned allocation between objectives.

  2. The connection between the gate’s optimization behavior and its intended role remains insufficiently explained. Although the paper specifies the gradient computation and regularization, it does not clearly establish how their interaction produces the proposed allocation between explanations and answers. The ablations characterize gate behavior but provide limited evidence connecting these design choices to downstream improvements.

  3. The procedure used for the functional token interventions in Table 14 needs clarification. The paper describes removing selected tokens, but it is unclear whether the modified responses are directly rescored or whether answers are regenerated. If only rescoring is performed, answer failure could result from removing extractable answer text, limiting what these interventions establish about the mechanism behind improved answer stability.

Questions for authors
  1. Does TFPO retain its advantage when the NLL baselines use comparable loss normalization and tuning budgets? A comparison with a constant gate under the same training setup would help clarify the contribution of learned routing.

  2. How do the gate update and regularization jointly produce the intended allocation between explanations and answers? In particular, what motivates treating the gate gradients differently in the policy and reference branches, and how does this choice affect downstream performance?

  3. For the functional token interventions in Table 14, are the modified responses directly rescored, or are answers regenerated after token removal? Please clarify the procedure and explain what the results establish beyond identifying text needed for answer extraction.

Reviewer_hSoQOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This work proposes TFPO, an approach to strike a balance between the DPO preference loss and the likelihood-forcing loss in preference alignment, by learning a router that assigns tokens to one of them.

Critical strengths
  • preference alignment is an important research question
  • bridging two types of training objectives is interesting
Critical weaknesses
  • better motivation and justification as to why we combine the two objectives in this way
  • more insights onto the router
Questions for authors
  • Throughout the paper, the authors assume, most notably in paragraph 2 of the introduction, that tokens are and should be separable into two groups, "reasoning" and "others" tokens. 1) Is this an accurate 2-way breakdown of language model tokens? 2) How do we establish that doing preference loss on type-1 tokens and likelihood loss on type-2 tokens, and only one type of loss, is the correct way?

  • If the authors can provide a well-defined dichotomy of the 2 types of tokens, the second question arises: is the trainer router faithful in identifying the two types of tokens as the authors defined? Quantitative results would be helpful here.

  • Figure 1 is hard to read, could enlarge the text a bit.

  • The policy starts as a (pretrained/post-trained?) model to some extent, but the router is randomly initialized (correct?) with an MLP. Would the cold start of router training negatively impact things?

  • A natural idea: instead of assigning each token to one loss and one loss only, how about generalize the router to a probabilistic predictor of a mixture of two losses? Is that possible? Does any baseline already include that? Would that increase computational cost?

  • Overall the proposed method doesn't feel technically riveting: two existing losses, selected through a linear probing MLP. Perhaps that isn't a huge issue. The authors are welcome to further highlight the technical interesting points.

Reviewer_MDAnOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Token-Fused Preference Optimization (TFPO), which uses a learnable token-level gate to allocate training weights between a DPO-style preference objective and an SFT/NLL objective. The method requires no token-level supervision and incorporates three regularization terms to stabilize gate learning. Extensive experiments across multiple benchmarks, model families, and multimodal tasks demonstrate improvements in preference alignment, overall performance, and answer stability.

Critical strengths

1, The paper is well-written and clearly structured, making it easy to follow.

2, The experimental evaluation is extensive, covering multiple benchmarks and model families. The authors also provide comprehensive ablation studies and analyses of the learned gate.

Critical weaknesses

Insufficient motivation and justification for token-level objective allocation, see questions below.

Questions for authors

1, In Lines 39–42, this paper argues that reasoning tokens are better suited to preference optimization, while final-answer tokens and structured outputs benefit more from SFT/NLL. Is there any theoretical reasoning, prior literature, or empirical evidence supporting this assumption?

2, More fundamentally, the motivation behind the proposed objective function remains unclear. The method learns token-level weights without explicit supervision of reasoning or conclusion spans, but the authors do not explain why optimizing the proposed loss should lead to the intended allocation of different objectives to different token types.

Although the empirical improvements and gate analyses are encouraging, they do not fully explain why the proposed allocation is appropriate or why it emerges from the training objective. I encourage the authors to provide a clearer conceptual justification for the method, rather than relying primarily on its empirical effectiveness. Otherwise, the proposed approach appears to be more of an effective engineering design than a well-motivated methodological advancement.

3, Could the authors elaborate on the underlying intuition for each regularization term in Section 3.4 and explain why these constraints are expected to encourage meaningful token-level objective allocation? A more principled discussion would help distinguish the proposed method from a collection of engineering choices.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response to ACs and All Reviewers

Dear ACs and Reviewers,

Thank you for your careful reviews and constructive discussion. Your feedback prompted additional controlled experiments and helped us clarify what the routing analyses establish. Below, we highlight TFPO's methodological contribution and the evidence supporting it.

Methodological contribution. Preference comparisons tell the model which response is better, while chosen-response likelihood directly reinforces the observed tokens. TFPO learns how to balance these signals within a response using a lightweight gate, without token annotations. The same gate changes each token's contribution to the preference comparison and, on chosen responses, its likelihood weight. This allows different objective weights within the same response, providing flexibility that a single shared mixture coefficient cannot express.

Controlled comparisons. The fixed-gate control keeps the TFPO objective and mean gate weight matched, but removes content-dependent allocation. TFPO produces more accurate answers across repeated sampling on BBH, GSM8K, and Hendrycks MATH. Individual-response accuracy improves by 2.65 pp, and the accuracy of the highest-vote answer from four samples improves by 2.67 pp. The advantage also persists against SimPO+NLL after matching chosen-NLL reduction and coefficient-search budget. Both comparisons yield positive accuracy gains in all three paired seeds, supporting the contribution of learned allocation.

Evidence about the learned weights. Human annotators blind to model identity and gate scores marked attempted answer spans for evaluation. Higher likelihood weights tend to identify these spans and continue to follow the answer when it is moved within the response. This supports content-sensitive allocation across answer positions. Deletion tests separately locate the text needed to extract an answer, and regularizer ablations connect stable gate learning to better span recovery. Together, these analyses characterize what the gate learns, while the training controls establish its overall benefit on the evaluated tasks.

Thank you again for your time and consideration.

Best regards,

Authors of Submission 10127

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.