acceptodds
Under review as a conference paper at ICLR 2027

Syntactic Shadowing: Why Length Penalties Do Not Fix Preference Optimization

Abstract

Preference optimization is supposed to teach a language model what people prefer, but a training pair exposes more than meaning. The resulting shortfall has been attributed to length bias and answered with length penalties and length normalization, as if controlling length were controlling non-semantic bias. These remedies have not fixed preference optimization: models trained with them align meaning no better than models trained without them. We ask whether the reason is a confound in the training signal itself. A longer response is rarely just a longer version of a shorter one; it carries different sentence structure. Length and syntax are confounded in natural text, so a model that appears to prefer long responses may in fact prefer the syntax that long responses tend to have. To make this measurable, we use meaning-preserving rewrites in which each step changes exactly one property, yielding a length gap measured at fixed syntax and a syntactic gap measured at fixed length, both read against the scale of the discrepancy between two rewrites of the same response produced under the same specification. Building on this, we propose a framework that decomposes the alignment gap of a preference pair into a length, a syntactic and a semantic component, together with a construction that rewrites the rejected response into token-exact counterfactuals of the chosen one and computes the corresponding gaps. Applying this to 25 models spanning ten state-of-the-art methods, and to 16 models of a second backbone, reveals a consistent pattern. Whether trained or not, and whichever method is used, directional semantic preference does not track the length channel, and the syntactic channel beneath it stays where it was; what does improve it, in a controlled comparison on identical data, is training on surface-matched counterfactual pairs. We call this syntactic shadowing: the syntactic component of surface bias sits in the shadow of length, where length control does not reach it.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Preference Optimization, Length Bias, Large Language Model

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_eDgZOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates why length penalties fail to resolve surface bias in preference optimization. The authors introduce token-exact counterfactual rewrites to decompose the preference alignment gap into semantic, syntactic, and length components, using twin rewrites to calibrate their magnitudes. Experiments on 1,979 TL;DR preference pairs across 41 models show that existing length-handling methods alter length sensitivity without improving directional semantic correctness. Training on surface-matched pairs improves semantic alignment, suggesting that syntactic confounding remains insufficiently addressed by length penalties.

Critical strengths

The paper introduces an interesting counterfactual framework for analyzing semantic, syntactic, and length-related effects in preference optimization. Extensive experiments reveal limitations of existing length-bias mitigation methods.

Critical weaknesses

The central analysis relies on LLM-generated counterfactual rewrites without sufficient human validation, and the low agreement among LLM judges raises concerns about their reliability. The experiments also do not conclusively establish syntactic confounding as the cause of the observed improvements.

Questions for authors
  1. Do the main findings hold on human-validated counterfactual rewrites?
  2. Does surface matching provide additional benefits over simply paraphrasing the rejected responses without length or syntax constraints?
1 comment

W1. Validation: the rewrites are derived from the rejected summary and must keep its meaning without moving toward the chosen one, so they remain the dispreferred side by construction, and what has to be checked is meaning preservation. The 200 families of Sec. B.3 have been verified manually by the authors against four rules (y1′y'_1 keeps the rejected meaning, y1′′y''_1 keeps it, y1′y'_1 does not drift toward the chosen summary, and the two differ only in syntax); 144 pass all four. On these 144 the qualitative picture is unchanged (Table 4), with S above 1 for every model (1.19–1.47), but the subset is too small to separate methods. Sec. B.3, the Limitations and the AI-use statement will be updated to report this verification. We will extend the manual verification to all 1,979 families and report it in the revision. We agree that the LLM judges agree poorly (κ\kappa of 0.155–0.204); Sec. B.3 presents the filtered subsets as a check of stability, not of validity. Cause: we agree that the experiments do not establish syntactic confounding as the cause of the improvement. Sec. 6 says the mechanism "remains a hypothesis that the decomposition makes testable", and we will say so in the abstract and Sec. 1 as well. What the paper establishes is narrower: no length-handling objective raises directional correctness above plain DPO, on two backbones and at ten penalty strengths.

Q1. On the 144 manually verified families the qualitative picture is unchanged (Table 4): S stays above 1 for every model (1.19–1.47), the I4 variants keep the largest L, and SimPO and ORPO keep L at or below the untrained value. With 144 families the subset is too small to resolve differences between methods, so we will report the paired contrasts once the manual verification covers the full benchmark. The LLM-judge-filtered subsets, which are larger, give the nearest evidence on the contrasts: on the strictest one (904 families) and on the pre-registered one, the flattened-arm gains remain significant and no length-handling objective exceeds DPO, while DPO − BASE on C loses significance (Sec. B.3).

Q2. The present data do not answer this, because no arm was trained on unconstrained paraphrases. They show two things. Length-only matching gains as much as length-and-syntax matching (C 0.801 and 0.805 against 0.775, Sec. F.3), so explicit syntax matching adds nothing measurable beyond length matching. The same rewrites give no gain when used as KL-DRO views (0.781 against 0.775) and a gain of 4–6 points when used as ordinary pairs. Whether an unconstrained paraphrase would give the same gain is open, and we will state finding (iv) accordingly.

Reviewer_A6soOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper points out that length bias cannot adequately account for the limitations of preference optimization, and highlights the confounding effects of semantics, syntax, and length. It proposes a framework to isolate these three factors, provides findings about their relationships, and identifies a phenomenon, syntactic shadowing, whereby syntax co-varies with length, but syntactic effects persist despite length control.

Critical strengths

1.The research question is well-introduced. There is adequate context in the introduction to explain why the confounding effects of syntax should be studied. 2.The proposed decomposition provides a systematic way to examine the contributions of different linguistic components to preference alignment. 3.The paper compares a wide range of preference optimization methods and their variants.

Critical weaknesses

1.The content could be better organized across sections. Currently, the introduction and related work are 2.5 and 0.5 page long, respectively. Moving some prior studies from L44-68 to Sec. 2 may help; the methodology in L93-144 could also be simplified, with more details going to Sec. 3 as preliminaries.

2.More justifications for meaning- and syntax-preserving claims are needed to ensure that the three components are fully isolated. a)In Table 1, although both y_w and y_1’ are all 27 tokens long, y_w begins with a declarative clause about the speaker's relationship with the person, while y_1’ presents the person's situation before introducing an indirect question. b)The rewrite is performed by Claude-Opus-5 equipped with Claude Code and then validated by two Claude judges and one Qwen2.5-72B-Instruct judge, but as mentioned in L968-969, the agreement across judges is low. For syntax, some rule-based validation would be helpful; for semantics or meaning, the current validation seems insufficient given the mutual entailment requirement, as described in L197-199.

3.Other experimental controls also warrant attention. For example, a)As discussed in the limitation paragraph, an unconstrained paraphrase control is important for a clearer attribution of the observed improvement, i.e., is it because of surface matching, LLM rewriting, or both? b)Some hyperparameters, including the training sample size and the DPO beta, are not consistent across the methods.

4.Minor concerns about the framing. For example, a)The paper reports 25 + 16 models in the beginning, but it is better to describe the actual setting as two backbone models in 25 & 16 configurations, respectively. b)There are gaps between semantic/meaning alignment and real human preference. It is unclear whether the original preference labels are still valid after rewording the rejected responses. It is important to clarify this distinction, as it concerns whether semantic alignment is genuinely needed.

Questions for authors

1.Is meaning a common term in preference optimization research? From my understanding, meaning and semantics are used interchangeably in the paper. The authors are expected to cite prior studies using this term or provide a brief explanation in the beginning, as meaning appears early in both the abstract and the introduction.

2.The sentence in L69-71 needs further clarification => mitigation of what? Length sensitivity?

2 comments

W1 (organization). Agreed. We will move the method survey (L44–68) to Sec. 2, cut Sec. 1 to about 1.5 pages, and move Eq. (1)–(2) with the description of the rewrite path (L93–144) to Sec. 3.1 as preliminaries.

W2 (meaning and syntax preservation). (a) In Table 1 the match is at the level of the sentence pattern as specified (clause order, subordination, clause boundaries), not of clause content. Both ywy_w and y1′y'_1 consist of two coordinated declarative clauses, the second of the form "I'm [verb-ing] [indirect wh-question]". The first clause differs in content because y1′y'_1 must keep the rejected meaning, which contains no proposition about the speaker; that difference belongs to g1g_1 by design. We will annotate the clause structure of the four responses in Table 1 so that what is matched is visible. (b) Syntax: a sentence pattern cannot be constrained or checked by rule as directly as length. In the pipeline it is an instruction to the writer, re-checked by the auditor, and the assembler reports the character overlap of y1′y'_1 and y1′′y''_1 so that the syntax step is a rearrangement rather than a second paraphrase. This is also why the syntactic component is defined as a residual: length is matched exactly first, and syntax is what remains of the surface difference at matched length. We agree that a rule-based check would strengthen this; it is not in this version, and we will describe sentence pattern as instructed and LLM-audited. Meaning: The 200 families of Sec. B.3 have been verified manually by the authors against four rules (y1′y'_1 keeps the rejected meaning, y1′′y''_1 keeps it, y1′y'_1 does not drift toward the chosen summary, and the two differ only in syntax); 144 pass all four. On these 144 the qualitative picture is unchanged (Table 4), with S above 1 for every model (1.19–1.47), but the subset is too small to separate methods. Sec. B.3, the Limitations and the AI-use statement will be updated to report this verification. We will extend the manual verification to all 1,979 families and report it in the revision. The LLM judges, whose agreement is low, remain a check of stability only, as Sec. B.3 says.

W3 (controls and hyperparameters). (a) Agreed. The unconstrained-paraphrase control was not run, so the improvement cannot be attributed to surface matching, to LLM rewriting, or to both. We will restate the finding: training on rewritten rejected responses that match the chosen response in length, or in length and sentence pattern, raises C over training on the original pairs; whether the matching or the rewriting carries the gain is not established. (b) Sample size: the ten SOTA models share the training data and the number of steps (Table 7). The counterfactual-data controls are a separate group trained on 1,998 prompts and are compared only among themselves; we will put the two groups in separate tables so that they are not read as comparable. β\beta: DPO uses 0.05 and the other DPO-family objectives 0.1 (Sec. C.2), and we did not run DPO at 0.1. The closest existing evidence is the sweep of Table 14: LD-DPO at α=0.8\alpha = 0.8, which is close to DPO at β=0.1\beta = 0.1, has C = 0.794 against 0.790, and C is indistinguishable from DPO at all ten penalty strengths. As Sec. C.2 notes, the differences in accuracy and L between DPO and the other objectives may partly reflect β\beta; we will repeat this caveat next to Table 2.

W4 (framing). (a) Agreed: "two backbones, trained in 25 and 16 configurations". (b) The rewrites are derived from the rejected summary, must keep its meaning and must not move toward the chosen one, and the chosen summary is untouched. They therefore remain the dispreferred side by construction: the label carries over as long as the meaning is preserved, which is what the manual verification of W2(b) checks. We agree that this is a statement about preference over content. A label that was driven by length or wording is not a semantic preference, and C is built not to reward it. We will state this distinction in Sec. 4.2 and will not describe C as agreement with human preference in general.

Q1. "Meaning" is not a standard term in preference optimization. We take it from paraphrase research (mutual entailment; Androutsopoulos & Malakasiotis, 2010) and from the counterfactual-edit literature. We will define it in the first paragraph of Sec. 1 and use "meaning" as the noun and "semantic" only as its adjective.

Q2. Mitigation of length bias by the objectives listed in the preceding paragraph. We will reword the sentence: "First, these objectives mitigate length bias only incompletely: on rewrites that differ only in length, R-DPO is more length-sensitive than plain DPO…".

Thanks authors for their response. The response partially solved my concerns. The core remaining concerns are 1) the meaning- and syntax-preserving claims are too strong without clear definitions and 2) missing experimental controls. therefore, I will keep my score unchanged.

Reviewer_ng2POpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper investigates why preference objectives that account for length fail to improve semantic alignment. For each TL;DR preference pair, the authors construct a path using LLM-generated rewrites of the rejected summary, yw→y1′→y1′′→yly_w \to y'_1 \to y''_1 \to y_l. Here, y1′′y''_1 matches the chosen summary’s length, while y1′y'_1 also matches its sentence pattern. This path decomposes the sequence log-probability gap into semantic, syntactic, and length components: Gθ=g1+g2+g3G\theta = g_1 + g_2 + g_3. A second rewrite under each specification (a “twin”) provides a scale νˉ\bar\nu for normalizing the mean absolute gaps: semantic separation M=E∣g1∣/νˉM = \mathbb{E}|g_1|/\bar\nu, syntactic sensitivity S=E∣g2∣/νˉS = \mathbb{E}|g_2|/\bar\nu, and length sensitivity L=E∣g3c∣/νˉL = \mathbb{E}|g_3^c|/\bar\nu, where g3cg_3^c is the content part of the length gap. Direction is measured by C=Pr⁡(g1>0)C = \Pr(g_1 > 0) and the signed gap gˉ1=Eg1/νˉ\bar g_1 = \mathbb{E}g_1/\bar\nu.

Across 25 Qwen2.5-1.5B and 16 Llama-3.2-3B models, length-handling objectives change LL but not CC, while SS remains close to 1.5 for all models. Training on flattened, surface-matched pairs increases CC by 4–6 points. The authors describe this pattern as “syntactic shadowing” and also derive a bound on E∣Gθ∣\mathbb{E}|G_\theta|.

Critical strengths
  1. The paper addresses a relevant question in preference optimization: how to separate length from structural properties that tend to vary with it.

  2. The token-exact counterfactual benchmark, including its twin rewrites, could be a useful resource for future studies.

  3. The study reports extensive robustness checks (twin swapping, independently drawn twins, judge-filtered subsets, extra seeds, a penalty sweep, and a second backbone with pre-stated predictions) and releases code, benchmark and per-family scores.

  4. The limitations section openly acknowledges several of the concerns discussed below.

Critical weaknesses
  1. The decomposition relies on assumptions that are not tested. The identity Gθ=g1+g2+g3G_\theta = g_1 + g_2 + g_3 follows by telescoping, and Eq. (2) follows from the triangle inequality. Both hold for any intermediate texts. Their interpretation as separate semantic, syntactic, and length effects therefore depends on whether the rewrites satisfy Definition 1. This assumes that length, sentence pattern, and meaning can be changed separately. However, matching length introduces padding such as “exactly”, “absolutely”, and “at all” (Table 1), so g3g_3 also captures changes in fluency and style. The decomposition also depends on the order of changes (Section 3.2), but the paper constructs only the length-first path. Adding the missing intermediate rewrite would allow both orderings to be compared and the length–syntax interaction to be estimated. Theorem 1 does not resolve these concerns: its second term never tightens the bound (Section A.6), and the twin construction also violates its i.i.d. assumption.

  2. The key properties are checked heuristically, and the results depend on design choices. Section 3.1 formally defines sentence pattern and meaning, but the construction checks in Section B.2 cover only token count, whitespace, and distinctness. No parser or entailment model verifies the required structural and semantic properties. The method also relies on several additional choices, including the length-first ordering, co-written twins, and the Δdisp<−60\Delta_{\text{disp}} < -60 threshold. These choices matter: swapping the twin roles changes CC by 0.03–0.05 (Table 11), exceeding most differences between methods, including DPO’s gain over BASE (+0.016+0.016). It also changes the statistical significance of 6 out of 24 comparisons (Table 12).

  3. The twin scale provides a weak basis for interpreting S>1S > 1. The twins are written together, and for BASE, the first scores about 4.7 higher than the second, compared with a normalization scale of νˉ=8.7\bar{\nu} = 8.7 (Section 3.3). The change from y1′′y''_1 to y1′y'_1 involves reordering, while the changes between twins involve rewording. These edits differ in type, and their size is not controlled. Therefore, S>1S > 1 alone does not establish syntactic bias. Under the paper’s sampling assumptions, the inequality S×≥1S _{\times} \geq 1 for the cross-twin version follows directly from the energy-distance identity in Section A. Moreover, E[∣g2∣]\mathbb{E}[|g_2|] and νˉ\bar{\nu} correlate at r=0.999r = 0.999 across models (Section D), so a nearly constant SS is consistent with all score differences scaling together. Finally, because length is adjusted first, g2g_2 excludes syntactic changes associated with length, even though these changes are central to the shadowing hypothesis.

  4. There is no human validation, and the construction depends on the tokenizer. Whether the rewrites preserve meaning and the original preference labels is assessed only by LLM judges, whose agreement is low (κ=0.155\kappa = 0.155–0.2040.204, Table 5). When transferred to Llama, only 61.4% of families remain token-exact (Section F.5). Transfer to other datasets is not demonstrated.

  5. The only positive result is confounded by the training and evaluation setup. The flattened arms train on the same type of pair used to compute CC: a human-written ywy_w paired with an LLM-written rejected rewrite, using the same writer and specification. The models could therefore learn to reject rewriting artifacts rather than the rejected meaning. Length-matched pairs alone perform as well as pairs matched for both length and syntax (Section F.3), while the reference arm performs almost identically to BASE (accuracy 0.519 vs. 0.518). A control using unconstrained paraphrases is missing.

  6. The paper does not evaluate generated responses. Length penalties aim to control verbosity during generation, but all outcomes are based on log-probabilities assigned to fixed texts. In addition, LL excludes by design the token-count term that these penalties act on. Reporting generated lengths and judged generation quality is necessary to support the title’s claim.

  7. The evidence is limited in domain and model scale. The positive finding comes only from the 1,998-prompt set, using roughly one forty-sixth of the data used for the main models. All experiments concern short TL;DR summaries, although length penalties are mainly motivated by long-form chat. Both backbones are also small, at 1.5B and 3B parameters.

Questions for authors
  1. Can you construct the missing intermediate rewrite (rejected meaning, chosen sentence pattern, rejected length) and report the main results under the syntax-first ordering, together with an estimate of the length–syntax interaction? How much of g3g_3 reflects padding-induced changes in fluency rather than length itself?
  2. Can you verify the sentence-pattern constraint with a parser and meaning preservation with an entailment model on the full benchmark? How many levels of the constituency parse are retained in s(y)s(y)?
  3. What value of SS would an edit-size-matched null give, for example a reordering that preserves the parse? Is the conclusion about SS unchanged when the ratio is replaced by absolute magnitudes, given that E∣g2∣\mathbb{E}|g_2| and νˉ\bar\nu correlate at r=0.999r = 0.999?
  4. Can you report human validation of meaning and label preservation on a sizable subset, and show that the pipeline transfers to another preference dataset or tokenizer?
  5. Can you add an unconstrained-paraphrase control and a chosen-side rewrite control with the same prompts, seeds and steps, to separate the effect of surface matching from learning rewriter artifacts?
  6. Do the length-handling models and the flattened arms differ in generated output length and in judged quality of generated summaries?
  7. Do the main findings hold on a long-form chat preference dataset and on a larger backbone (for example 7B)?
2 comments

Thank you for the detailed review. We could not run new data construction or training in the response period, so we answer from the submission and restrict claims where you are right. Each question is answered with the matching weakness.

W1 / Q1 (assumptions, padding, order, Theorem 1). We agree that the identity and Eq. (2) hold for any intermediate texts; the interpretation rests on Definition 1. Padding: by definition g3g_3 is the effect of everything that changes when length is matched, inserted words included. We will say that L is not sensitivity to token count alone; we have no estimate of the padding share. Order: it is part of the definition. The components are successive residuals: length, then the surface difference that remains at matched length (syntax), then what remains at matched surface form (semantics). A syntax-first path would be a different decomposition, so we did not construct that rewrite. Length comes first because it is the only property that can be matched exactly and checked by code (a sentence pattern can only be instructed and audited), and because crediting length with everything that comes with it is conservative for a negative conclusion about length. C and gˉ1\bar g_1 do not depend on the order. Theorem 1: agreed. The refined bound does not bind here (Sec. A.6); we will shorten Sec. 3.4 and say so.

W2 / Q2 (checks, twin swap). Code enforces token count, whitespace and distinctness. Sentence pattern is an instruction to the writer, re-checked by the auditor, not a fixed parse depth. No parser or entailment model was applied, and we will describe the pattern as instructed and LLM-audited. Twin swap: the shift in C is a level shift of +0.033 to +0.054 in 24 of 25 models, BASE included (Table 11). Paired contrasts are stable: DPO − BASE on C is +0.016, +0.014 and +0.022 under the three designations, all significant, and SimPO below DPO and the flattened-arm gains also hold in all three (Table 12). One conclusion is affected: under the swapped designation alone, R-DPO exceeds DPO by 0.012 in C. We will make the symmetrized estimator primary and report this exception. The displacement threshold only annotates models.

W3 / Q3 (twin scale and S). We agree that S above 1 follows from exchangeability and is a regularity check, and that the syntax step (reordering) and the twins (rewording) differ in edit type and size; we have no edit-size-matched null. The 0.999 correlation means the claim holds for the ratio only: no objective changes sensitivity to sentence pattern relative to rewording. The ratio is not vacuous, since L varies 4.5 times as much as S (SD 0.298 vs 0.067). With independently regenerated twins (Sec. F.4) the co-writing bias disappears for the length twin and S stays above 1 for every model. That S excludes the syntax travelling with length follows from the definition: the paper places the shadowed part in the length component (Sec. F.3), and we will make this explicit.

W4 / Q4 (validation, tokenizer, transfer). The rewrites are derived from the rejected summary, must keep its meaning and must not move toward the chosen one, so they remain the dispreferred side by construction. What needs checking is meaning preservation. The 200 families of Sec. B.3 have been verified manually by the authors against its four rules; 144 pass all four, and on these the qualitative picture is unchanged (Table 4; S of 1.19–1.47). We will extend the manual verification to all 1,979 families and update Sec. B.3, the Limitations and the AI-use statement. On Llama, 61.4% of families stay token-exact and every result holds on them (Sec. F.5). Transfer to another dataset is not shown.

W5 / Q5 (confounded positive result). Agreed: the flattened arms train against LLM-written negatives, and the unconstrained-paraphrase control was not run. Existing data do not separate matching from rewriting: the same rewrites give no gain as KL-DRO views (C 0.781 vs 0.775), and length-only matching gains as much as length-and-syntax matching (0.801 and 0.805 vs 0.775). We will restate finding (iv): whether matching or rewriting carries the gain is not established. We do not rewrite ywy_w by design. It is the summary annotators preferred; rewriting it would replace annotated text with text no annotator has judged. The main negative finding is unaffected, because the models it compares never see a rewrite in training.

W6 / Q6 (generation). Agreed. All outcomes are log-probabilities of fixed texts. We will restrict the claims to the preference signal on pairs, add raw L (Sec. A.6) to Table 2, and state that generation is not evaluated.

W7 / Q7 (domain and scale). Agreed. The evidence covers one domain, 1,998 prompts for the positive finding, and backbones of 1.5B and 3B parameters. We will state these limits and make no claim about long-form chat or larger models.

Thanks for the detailed reply. The twin-swap explanation makes sense to me. On the ordering, I agree the residual definition is consistent, but it means SS can’t capture the syntax that changes with length, which is what the shadowing hypothesis is about. I also note that Section B.3 of the submission describes the 200 screened families as judged by LLM annotators without human adjudication, which differs from the manual verification described in the response. Given that the mechanism is now presented as a hypothesis, the flattened-arm finding is restated, and the claims are restricted to the preference signal on fixed pairs, I’ll keep my score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.