- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper investigates why length penalties fail to resolve surface bias in preference optimization. The authors introduce token-exact counterfactual rewrites to decompose the preference alignment gap into semantic, syntactic, and length components, using twin rewrites to calibrate their magnitudes. Experiments on 1,979 TL;DR preference pairs across 41 models show that existing length-handling methods alter length sensitivity without improving directional semantic correctness. Training on surface-matched pairs improves semantic alignment, suggesting that syntactic confounding remains insufficiently addressed by length penalties.
Critical strengths
The paper introduces an interesting counterfactual framework for analyzing semantic, syntactic, and length-related effects in preference optimization. Extensive experiments reveal limitations of existing length-bias mitigation methods.
Critical weaknesses
The central analysis relies on LLM-generated counterfactual rewrites without sufficient human validation, and the low agreement among LLM judges raises concerns about their reliability. The experiments also do not conclusively establish syntactic confounding as the cause of the observed improvements.
Questions for authors
- Do the main findings hold on human-validated counterfactual rewrites?
- Does surface matching provide additional benefits over simply paraphrasing the rejected responses without length or syntax constraints?
W1. Validation: the rewrites are derived from the rejected summary and must keep its meaning without moving toward the chosen one, so they remain the dispreferred side by construction, and what has to be checked is meaning preservation. The 200 families of Sec. B.3 have been verified manually by the authors against four rules (y1′ keeps the rejected meaning, y1′′ keeps it, y1′ does not drift toward the chosen summary, and the two differ only in syntax); 144 pass all four. On these 144 the qualitative picture is unchanged (Table 4), with S above 1 for every model (1.19–1.47), but the subset is too small to separate methods. Sec. B.3, the Limitations and the AI-use statement will be updated to report this verification. We will extend the manual verification to all 1,979 families and report it in the revision. We agree that the LLM judges agree poorly (κ of 0.155–0.204); Sec. B.3 presents the filtered subsets as a check of stability, not of validity. Cause: we agree that the experiments do not establish syntactic confounding as the cause of the improvement. Sec. 6 says the mechanism "remains a hypothesis that the decomposition makes testable", and we will say so in the abstract and Sec. 1 as well. What the paper establishes is narrower: no length-handling objective raises directional correctness above plain DPO, on two backbones and at ten penalty strengths.
Q1. On the 144 manually verified families the qualitative picture is unchanged (Table 4): S stays above 1 for every model (1.19–1.47), the I4 variants keep the largest L, and SimPO and ORPO keep L at or below the untrained value. With 144 families the subset is too small to resolve differences between methods, so we will report the paired contrasts once the manual verification covers the full benchmark. The LLM-judge-filtered subsets, which are larger, give the nearest evidence on the contrasts: on the strictest one (904 families) and on the pre-registered one, the flattened-arm gains remain significant and no length-handling objective exceeds DPO, while DPO − BASE on C loses significance (Sec. B.3).
Q2. The present data do not answer this, because no arm was trained on unconstrained paraphrases. They show two things. Length-only matching gains as much as length-and-syntax matching (C 0.801 and 0.805 against 0.775, Sec. F.3), so explicit syntax matching adds nothing measurable beyond length matching. The same rewrites give no gain when used as KL-DRO views (0.781 against 0.775) and a gain of 4–6 points when used as ordinary pairs. Whether an unconstrained paraphrase would give the same gain is open, and we will state finding (iv) accordingly.