- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes Token-Fused Preference Optimization (TFPO), which combines token-level weighting with a mixture of preference and likelihood objectives. The proposed core design is a lightweight gate, learned jointly with the policy from detached decoder hidden states without token-level supervision. The gate assigns complementary weights to tokens' contributions to a DPO-style sequence-level preference comparison and a chosen response negative log-likelihood objective. The intended behavior is to apply preference shaping to explanatory content while anchoring answer-bearing or format-sensitive tokens. The paper evaluates TFPO on text capability benchmarks, external alignment across three open-source models, and multimodal tasks, reporting improvements over several preference-optimization baselines.
Critical strengths
-
The proposed gate is lightweight and requires no token annotations during training. It provides a simple way to allocate preference and likelihood objectives within a response.
-
The evaluation covers text capability, external alignment across three model families, and multimodal tasks. The reported improvements are broad, and the main text capability results include three training seeds.
-
The routing analysis includes blind answer span annotations, position based controls, and answer relocation experiments. These complementary checks provide evidence that the learned gate identifies answer content rather than relying only on its position.
Critical weaknesses
-
The experiments do not sufficiently isolate the contribution of learned routing. Differences in loss normalization and tuning budgets complicate comparisons with the NLL baselines, while the routing controls mainly evaluate span identification rather than downstream performance. It therefore remains unclear how much of the reported improvement can be attributed specifically to the learned allocation between objectives.
-
The connection between the gate’s optimization behavior and its intended role remains insufficiently explained. Although the paper specifies the gradient computation and regularization, it does not clearly establish how their interaction produces the proposed allocation between explanations and answers. The ablations characterize gate behavior but provide limited evidence connecting these design choices to downstream improvements.
-
The procedure used for the functional token interventions in Table 14 needs clarification. The paper describes removing selected tokens, but it is unclear whether the modified responses are directly rescored or whether answers are regenerated. If only rescoring is performed, answer failure could result from removing extractable answer text, limiting what these interventions establish about the mechanism behind improved answer stability.
Questions for authors
-
Does TFPO retain its advantage when the NLL baselines use comparable loss normalization and tuning budgets? A comparison with a constant gate under the same training setup would help clarify the contribution of learned routing.
-
How do the gate update and regularization jointly produce the intended allocation between explanations and answers? In particular, what motivates treating the gate gradients differently in the policy and reference branches, and how does this choice affect downstream performance?
-
For the functional token interventions in Table 14, are the modified responses directly rescored, or are answers regenerated after token removal? Please clarify the procedure and explain what the results establish beyond identifying text needed for answer extraction.
General Response to ACs and All Reviewers
Dear ACs and Reviewers,
Thank you for your careful reviews and constructive discussion. Your feedback prompted additional controlled experiments and helped us clarify what the routing analyses establish. Below, we highlight TFPO's methodological contribution and the evidence supporting it.
Methodological contribution. Preference comparisons tell the model which response is better, while chosen-response likelihood directly reinforces the observed tokens. TFPO learns how to balance these signals within a response using a lightweight gate, without token annotations. The same gate changes each token's contribution to the preference comparison and, on chosen responses, its likelihood weight. This allows different objective weights within the same response, providing flexibility that a single shared mixture coefficient cannot express.
Controlled comparisons. The fixed-gate control keeps the TFPO objective and mean gate weight matched, but removes content-dependent allocation. TFPO produces more accurate answers across repeated sampling on BBH, GSM8K, and Hendrycks MATH. Individual-response accuracy improves by 2.65 pp, and the accuracy of the highest-vote answer from four samples improves by 2.67 pp. The advantage also persists against SimPO+NLL after matching chosen-NLL reduction and coefficient-search budget. Both comparisons yield positive accuracy gains in all three paired seeds, supporting the contribution of learned allocation.
Evidence about the learned weights. Human annotators blind to model identity and gate scores marked attempted answer spans for evaluation. Higher likelihood weights tend to identify these spans and continue to follow the answer when it is moved within the response. This supports content-sensitive allocation across answer positions. Deletion tests separately locate the text needed to extract an answer, and regularizer ablations connect stable gate learning to better span recovery. Together, these analyses characterize what the gate learns, while the training controls establish its overall benefit on the evaluated tasks.
Thank you again for your time and consideration.
Best regards,
Authors of Submission 10127