acceptodds
Under review as a conference paper at ICLR 2027

Diagnosing & Mitigating Constraint Induced Response Collapse in Instruction-Tuned LLMs

Abstract

Instruction-tuned language models are trained to give thorough, well-organized answers. Recent work showed that this behavior is fragile: banning a single character, such as the em dash, can remove up to 48% of an answer's substantive content. This _constraint-induced response collapse_ has been ascribed to shallow template following, but its mechanism remains unclear. We show that lexical bans are only one of many triggers: across seven models, everyday constraints on tone, formatting, and delivery cut comprehensiveness by up to 45%, even though the model obeys them. Controlled ablations identify the trigger as _form primers_, phrases in the prompt that refer to the form of the response rather than its content. Studying each post-training stage of the open OLMo family, we trace the collapse to reward-based post-training rather than template following: it emerges at preference optimization and is largest after reinforcement learning from verifiable rewards, two stages whose rewards do not measure content. A _single direction_ in the model's activations controls the collapse: adding it to unconstrained prompts induces collapse, and subtracting it from constrained prompts restores most of the lost content with the constraint still obeyed, mitigating the collapse at inference time. Because pointwise judges miss about half of this loss, we release a 160-question benchmark that scores every answer against the same model's unconstrained answer.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Instruction Tuning, LLMs, Constraints, helpfulness, alignment

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_19WHOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The authors aim to explain and mitigate “constraint-induced response collapse,” where prompted constraints can reduce the comprehensiveness of answers. They find that this can be resolved either by prompting the LLM to be thorough (Table 2) or steering (Section 3). The authors also trace the emergence of the described phenomenon to reward-based post-training (Section 4).

Critical strengths

S1: The topic is relevant and practical. Format constraints are common (e.g., in judge protocols), so the documented phenomenon has the potential to be problematic.

S2: The steering experiment in Section 3 and the results in Section 4 are interesting. I found especially interesting that subtracting the direction from constrained prompts barely reduced constraint adherence.

Critical weaknesses

Weakness 1

Concerns about the discussion of form primers and alternative hypotheses. This limits my ability to endorse the downstream conclusions at this stage.

  • The definition in Section 2.1 ("when the thing it talks about is one of the properties of the response as a piece of text rather than of its content") is imprecise. Notwithstanding the foregoing, I do not follow the "form primer" classification in Table 2, e.g., why are (1) whether em or en dashes are present or (2) whether kittens or cats are discussed properties of the response "as a piece of text" and not "its content"?

  • I find the alternative hypotheses incomplete (and, with the exception of R3, somewhat implausible), so the conclusion that Table 2 “leav[es] form priming as the likely explanation” is a bit unconvincing. For example, the following hypothesis appears substantiated by other results in the text:

Reward-based post-training primes LLMs to infer a preference from the user. Absent additional constraints, the implicit preference is “thorough response” (or something to that effect). Introducing constraints to the prompt pushes the LLM to prioritize constraint-adherence and de-prioritize any implicit preference, since these prompts do not instruct the LLM to answer in the way the judge is scoring (e.g., details, examples, etc.). This is consistent with Table 2 (e.g., no collapse is observed when the constraint is aligned with this objective).

Weakness 2

The measurement and discussion of comprehensiveness and "collapse" depend on aspects of the experimental design that are not satisfactorily justified.

  • In Section 2, the authors define comprehensiveness as “substantive content conveyed about [the prompt],” but I would say that both responses in Figure 1 are reasonable and deliver substantive content. Meanwhile, the judge explicitly calls for “examples,” which are not strictly necessary for a comprehensive answer (Appendix A).

  • The “collapse” is largely driven by reliance on a pairwise judge, and the effect appears much more modest under pointwise scoring (Table 3). The authors say this means the pointwise judge only “see[s] half of the full story,” but it is also plausible that the pairwise judges exaggerates the loss.

I believe this is a significant oversight and feel the authors do not engage sufficiently with prior work that may help clarify the issue. First, the idea that pairwise judgement exaggerates the perceived distinction between candidate items has been discussed in the social sciences for many years (i.e., "distinction bias" [1]) and is consistent with pre-existing work regarding judge/feedback protocols (e.g., "distracted evaluation" [2] and ranking vs. rating [3]).

Weakness 3

Missing details and overstated clams. For example:

  • The judge prompts are described in Appendix A but not actually provided in full.

  • The abstract claims that "Controlled ablations identify the trigger as form primers," which is not supported (i.e., causal). The second sentence of the Conclusion is more appropriate, in my view.

  • Section 3 (steering) is introduced as "a practical way to study the effect and to reduce [constraint-induced response collapse]." But simply prompting with "Be thorough, include all details" - certainly far more practical than activation steering - appears to mitigate the collapse on its own (Table 2). This by no means invalidates the section, as I find its content interesting (as mentioned in Strength 2), but it weakens the stated claim, which appears to motivate the section's existence.


References

[1] Wikipedia, "Distinction bias," https://en.wikipedia.org/wiki/Distinction_bias

[2] Tripathi et al., "Pairwise or Pointwise? Evaluating Feedback Protocols for Bias in LLM-Based Evaluation," COLM (2025).

[3] Bansal et al., "Peering Through Preferences: Unraveling Feedback Acquisition for Aligning Large Language Models," ICLR (2024).

Questions for authors
  • How were the questions selected for the benchmark? It would be nice to include some example questions. Are the experiments only on 160160 questions?

  • Do the models adhere to the constraints after SFT (but before the reward-based post-training)?

  • How are the authors using the negated turnstyle operator in Appendix G? Its usage here appears nonstandard and unintroduced.

  • Appendix F mention "released code," but I do not see it. Have I missed this somewhere?

18 comments

(1/4)

We thank the reviewer for the detailed review. In response, we made the following changes. The revised PDF shows new and revised text in blue. We summarize the answers here, with details for each following the summary.

  • W1: Section 2.1 gives a precise definition of a form primer. It also adds the reviewer's alternative account as R5 and tests it with the results of Table 2 and with a new pre-registered experiment (Appendix M).
  • W2: five checks on the pairwise judge, including a count of facts that does not ask any judge which answer is better, and a judge from another model family. The count of facts gives a loss of the same size as the pairwise judge (37.8% against 36.1%), and the judge finds no loss when nothing is missing.
  • W3: Appendix A gives every judge prompt in full, the abstract now states that form primers are enough to trigger the collapse, and the motivation of Section 3 is revised.
  • Q1 to Q4: a new appendix on how the benchmark questions were selected and how precise the estimates are with 160 questions, compliance at every post-training stage, the corrected symbol, and the code in the supplementary material.

W1. Definition of a form primer, and the kittens case. We define a form primer in a way that can be applied to any clause. A clause is a form primer when it refers to the response as a piece of text, by which we mean its characters, layout and formatting, register and tone, delivery medium, or what it must not mention, instead of the information that the question asks for. The em-dash ban refers to characters in the response. The kittens clause names a subject, but no answer to the question contains this subject, so the clause restricts nothing that the question asks for. It belongs to the no-op group in Figure 2. "Be thorough" and "address the question in the best possible way" refer to the content of the answer, that is, how well the response covers the question. Therefore, they are not form primers, and they do not cause the collapse (Table 2). We add this definition to Section 2.1.

W1. The reviewer's alternative (post-training installs an implicit "be thorough" preference, and an explicit constraint overrides it). We agree with the first half of this account, and it is the account of Section 4. Reward-based post-training makes one elaborated answer the model's default way of being thorough (Tables 5 and 6, Figure 4). Section 2.1 asks a different question, namely what moves the model away from this default. Two results in the paper show that the second half cannot explain every collapse. (i) The no-op clauses give the model nothing to put ahead of thoroughness. "Freely use any words in your response" permits everything, and "Do not discuss kittens" forbids nothing that an answer would contain. Yet these clauses cost 15 to 38% of the content (Table 2). If the model gave up thoroughness in order to obey, it would need something to obey. These clauses act only by being present in the prompt. (ii) Statements that give no instruction at all ("Your response will be read aloud", "displayed in a monospace font") cost 20 to 33% (Table 2, R2). In both kinds of clause, the cost comes from a reference to the response that asks nothing of it. We call this effect form priming. We add the reviewer's account as R5 in Section 2.1, together with these two observations.

We also tested the account directly. Before generating any answer, we wrote down what R5 and form priming each predict for nine new clauses. On Qwen3-4B-Instruct (40 questions, the setup of Table 2), R5 explains part of the effect. Instructions about content that do not ask for thoroughness also cost content: "Use an analogy to explain the main idea" (-40.9%), "Mention one common misconception" (-27.0%), and "Include at least one concrete, real-world example" (-16.3%). Other rows go as form priming predicts. "Explain why this matters in practice" leaves the answer unchanged (+0.3%), and "I am reading this on a train", which gives no instruction, costs 31.1%. Two statements that neither account predicted also cost content ("Em dashes are longer than en dashes", -32.1%; "Today is Tuesday", -15.6%). Four added sentences in the same test do not collapse the answer (+0.3%, +1.9%, -5.8%, +10.1%), so the effect does not come from a longer prompt (R4). The new Appendix M reports all twelve rows (the nine new clauses and three rows of Table 2). Section 2.1 now states which part of the effect R5 explains, and it no longer says that Table 2 leaves form priming as the likely explanation. The abstract now states that form primers are enough to trigger the collapse.

(2/4)

R1, R2, and R4 are the explanations that prior work and readers are likely to consider first. Template dependence (prior work) needs the clause to ban something that the answer uses (R1). An instruction-following account needs an order (R2). R4 is the basic null hypothesis that any added sentence causes the loss. Table 2 is built so that each row removes one of these factors and keeps the reference to the form. R3 is the alternative that the reviewer finds plausible, and Table 2 tests it most directly. "Pay attention to your formatting and punctuation as you write" asks for no change and costs 15 to 22%. "Italicize each section's heading" asks for more text and costs 28 to 30%.

W2. Comprehensiveness and the word "examples". We measure a relative quantity. We compare what a model conveys with the constraint to what the same model conveys without it. Each answer in Figure 1 looks reasonable when it is read alone. For this reason, single-answer evaluation does not notice the loss, and we compare each answer with the model's own unconstrained answer instead. Factual coverage never asks for examples. We also re-scored the answers with a pairwise prompt that asks only "how much of the information needed to answer this question does each response convey". This prompt does not mention examples, edge cases, or depth (Appendix A gives the full prompt). It reports -26.1% on Qwen3-4B-Instruct and -15.7% on Llama-3.1-8B-Instruct. The unconstrained answer wins 97.2% and 79.0% of the pairs.

W2. Could the pairwise judge exaggerate the loss? We tested this with five checks. One check does not ask any judge which answer is better. Another check uses a judge from a different model family.

  1. An independent count gives a loss of the same size. Factual coverage does not ask a judge which answer is better. It lists the claims in the unconstrained answer. It then checks, one claim at a time, whether each claim is present in the constrained answer. On the 160-question benchmark (Qwen3-4B-Instruct, all 1,760 pairs), constrained answers convey 36.1% of these claims. A second unconstrained answer to the same question conveys 58.0% (Table 7). Relative to an answer given with no constraint, this is a loss of 37.8%. We re-ran the pairwise judge on exactly these pairs for this check, and it reports -36.1%. The two measures also rank the eleven constraints in nearly the same order (Spearman 0.91). On Llama-3.1-8B-Instruct, the same comparison gives 52.4% against 63.9%, a loss of 18.0%, and the pairwise judge reports -21.6%.
  2. As a null control, we ran the same pairwise judge on two unconstrained answers to each of the 160 questions. It reports Delta = -0.4% (95% CI -2.1 to +1.3). It scores 51.2% of the pairs as exact ties. The mean absolute gap between the two scores is 0.54, against 3.13 when one of the two answers has a constraint. So the judge does not create gaps between answers that are equally complete.
  3. To test the effect of answer order, we re-judged the 440 pairs of the 40 written questions (Qwen3-4B-Instruct) with the A and B positions swapped. In 394 of the 395 pairs that received a score in both runs, the two runs agree on whether the unconstrained answer wins (Appendix A.1 gives Delta for each position).
  4. To control for length, Table 3 includes a length-discounted judge. This judge is told to ignore length, formatting, and style, and to count distinct points. It still finds most of the loss: -26.6% against -29.0% for the pairwise judge on Qwen3-4B-Instruct, and -17.0% against -21.6% on Llama-3.1-8B-Instruct. Correction: in the submission, the Llama rows for the pointwise and length-discounted judges (-14.3% and -19.1%) had been scored by Sonnet 5 on answers from a different generation run. We re-scored them with Opus 4.8 on the same answers as the pairwise row. The new values are -12.4% and -17.0%, and the revised Table 3 uses them.
  5. We re-scored 220 of the benchmark pairs (20 held-out questions, eleven constraints) with a judge from another model family, an OpenAI model (gpt-6.1-sol), and the same prompt. We also re-scored the null control on the same questions. This judge reports Delta = -29.2% (the unconstrained answer wins 92.7%), against -34.9% for Opus 4.8 on the same pairs. On the null pairs, it reports +2.2%, with 65% ties. The two judges pick the same winner in 207 of the 220 pairs, and they rank the eleven constraints in nearly the same order (Spearman 0.96).

(3/4)

Regarding the work that the reviewer cites, Tripathi et al. (2025) find that pairwise judgments change toward a response that carries a distractor (an assertive tone, wordiness, or sycophancy) more often than absolute scores do. In their study, the distractor is added by a rewrite that keeps the content the same. In our pairs, the content differs. Check 1 measures this difference without asking which answer is better, and check 4 discounts wordiness. Bansal et al. (2024) find that ratings and rankings disagree mainly on pairs that are judged close, where ratings give ties. Check 2 shows that our judge gives ties when nothing is missing. Hsee and Zhang (2004) describe the evaluation of a single option as "close to a step function". Zheng et al. (2023) note that single-answer grading "may be unable to discern subtle differences". We report the same behavior for pointwise judges, which give close scores to answers that differ in content. Finally, the omission does not depend on a comparison being made. A user who sees only the constrained answer still does not receive the 42% of essential claims that this answer drops (Appendix B). We add these citations to Appendix K, and Section 2.1 now cites Hsee and Zhang (2004) and Zheng et al. (2023).

W3. Judge prompts. Appendix A now gives the full text of every prompt: pairwise, pointwise, length-discounted, content-only, claim extraction, claim matching, importance grading, and compliance.

W3. "Controlled ablations identify the trigger". We revise the abstract to state what the ablations show: "Controlled ablations show that form primers, phrases in the prompt that refer to the form of the response rather than its content, are enough to trigger the collapse, even when they restrict nothing." The conclusion makes the same claim. Each row of Table 2 changes one appended sentence and keeps everything else fixed, so each comparison is an intervention on the prompt. Section 3 adds an intervention on the activations. When we add the direction computed from form-primed prompts to a prompt with no constraint, the collapse appears.

W3. Steering vs. "be thorough" prompting. We agree that the sentence that motivates Section 3 puts the emphasis in the wrong place, and we revise it. (The "be thorough" row of Table 2 adds that sentence to a question with no constraint. It is the control for R4.) The purpose of Section 3 is to test cause and effect. We read a single direction from the prompt at its last token, before the model generates any answer token. This direction alone is enough to produce the collapse on a prompt that has no constraint. It is also enough to remove most of the collapse from a prompt that has a constraint, while the model still follows the constraint (Table 4). A prompting experiment cannot show this, because prompting changes the input. Steering keeps the input fixed and changes only the internal state. The new held-out evaluation (Appendix C.1) shows that both effects hold on questions and constraints that were not used to build the direction. We keep the practical benefit as a secondary point. The intervention needs no change to the prompt that the user or the deployed system provides.

Reviewer_gvbQOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies the loss of substantive content when instruction-tuned language models follow additional constraints on how an answer should be expressed. The authors find that, beyond character bans, instructions concerning tone, presentation, and even constraints that impose no practical restriction can reduce response comprehensiveness. They investigate this phenomenon through controlled prompts, factual coverage measurements, and unconstrained resampling controls. They also use activation differences between constrained and unconstrained prompts to mitigate content loss through linear activation steering. Finally, comparisons across OLMo post-training stages show that the effect emerges after preference optimization and becomes more pronounced after RLVR.

Critical strengths
  1. The research question is meaningful, and the analysis goes beyond demonstrating the phenomenon. Much existing work focuses on whether models follow instructions, whereas this paper examines what may be lost when they do. In particular, the finding that seemingly irrelevant no-op prompts also reduce content provides useful evidence about models’ sensitivity to constraints.
  2. The experiments are extensive and do not rely on a single metric. The authors combine evaluations across models, controlled prompts, factual coverage, resampling, and comparisons across post-training stages to investigate content loss systematically. Bidirectional activation interventions further show that the behavior can be manipulated and mitigated, rather than merely correlated with particular representations. Although the intervention builds on existing techniques, its application to content recovery while checking instruction compliance has empirical value.
Critical weaknesses
  1. The root-cause explanation is stronger than the current evidence supports. The OLMo checkpoint comparisons and DPO experiments link the phenomenon to post-training, but do not fully separate the effects of preference data, stylistic changes, and reward objectives. In particular, preferred responses being longer or more heavily formatted does not directly establish that the preference signal disregards content. The root-cause claim should therefore be stated more cautiously.
  2. The activation-intervention protocol and result reporting need clarification. The paper does not sufficiently explain the separation between data used for direction construction, layer and strength selection, and evaluation. In addition, the Llama comprehensiveness scores and relative changes in Table 4 are inconsistent, particularly for the random-direction control and recovery condition, which affects the interpretation of intervention effectiveness.
Questions for authors
  1. Are the questions and constraints used to construct the activation direction separate from the evaluation set? How are the intervention layer and strength selected, and is an independent validation set used?
  2. Please check the Llama results in Table 4. Under the formula in the appendix, using 7.74 as the baseline, the random-direction score of 6.95 corresponds to approximately −10.2%, and the recovery score of 6.83 corresponds to approximately −11.8%, whereas the table reports −3% and −8%, respectively. Were different baselines or aggregation scopes used? Would correcting these values affect the conclusions?
  3. Regarding the explanation that content-blind rewards cause collapse, which alternative factors can the current experiments rule out? Please clarify how the 6,750 analyzed preference pairs and the instruction-following verifier relate to the complete OLMo training pipeline, and distinguish established findings from hypotheses.
4 comments

(1/3)

We thank the reviewer for the careful review. In response, we made the following changes. The revised PDF shows new and revised text in blue.

  • W1/Q3: Section 4 now separates what the experiments measure from the mechanism that we propose. A new test shows that the reward learned by DPO favors the elaborated form and not the facts that the form carries.
  • W2/Q1: we evaluate the steering directions on 100 held-out questions, including five constraints that were never used to build them. Both effects hold: adding the direction induces the collapse, and subtracting it recovers 70% of the loss on Qwen3-4B-Instruct.
  • Q2: each Delta in Table 4 was computed against a paired baseline that the table did not show. The revised table shows it, and the conclusions do not change.

W1/Q3. Separating preference data, style, and objective, and how the preference statistics relate to the pipeline. Following the reviewer's suggestion, the revised Section 4 separates two things. The first is what the experiments establish: where the collapse appears, that its size grows with the strength of optimization, that the RL offset does not depend on the constraint, and which reward pushes toward shorter answers. The second is the mechanism that we propose to explain these results: DPO ties thoroughness to one elaborated form, and the incentive of the instruction-following verifier produces the RL offset.

The DPO stage of OLMo trains on Dolci-Instruct-DPO, which has 259,922 pairs: 124,980 judge-labeled pairs, 124,942 delta-learning pairs (the chosen answer comes from a larger model and the rejected answer from a smaller one), and 10,000 multi-turn pairs. We recomputed the statistics of Section 4 on the full released set, with a script that is part of the supplementary code. Over the 259,786 pairs in which both answers are non-empty, the preferred answer is the longer one in 64.3% of pairs (the shorter one in 31.9%). It is the more formatted one in 49.1% (the less formatted one in 21.1%). Over the 124,845 judge-labeled pairs with two non-empty answers, the preferred answer is the longer one in 58.5%. Among the 16,937 of these pairs whose two answers are within 10% in length, it is the more formatted one in 41.7%, against 26.0% for the less formatted one. The submission reported these statistics on a subset of 6,750 judge-labeled pairs. The full set changes them very little (longer: 58% becomes 58.5%; more vs. less formatted: 44% vs. 27% becomes 41.7% vs. 26.0%). The revision reports the full-set numbers. The instruction-following verifier scores the 22.1% of RL prompts that carry constraints. On the checkpoint where RL starts, it is the only measured reward with a clear push toward shorter answers (its 95% CI excludes zero; details in our answer to Reviewer epWv).

The experiments rule out four explanations. (a) The collapse is not inherited from the demonstrations: the SFT model follows the constraints 91.6% of the time and does not collapse. (b) It does not follow from the preference data alone: at beta = 1.0, the same data produce no collapse. (c) It is not specific to length-normalized DPO: standard DPO collapses in the same way at matched quality. (d) The preference stage does not reward the facts that the collapse removes (see the implicit-reward test below).

(2/3)

We separate the three factors with three experiments.

  • In the first, the data are fixed and the strength of the objective varies. Table 6 retrains from the released SFT checkpoint on one fixed random subset of 30,000 preference pairs, and it changes only beta. The collapse appears only when the policy is optimized far from SFT (+3% at beta = 1.0, -4% at 0.1, -21% at 0.03).
  • In the second, the form of the objective varies at matched quality. We trained standard DPO and length-normalized DPO on the same subset. They collapse by similar amounts (-21% vs -22%), so the loss is not specific to one objective.
  • The third (new) tests style against content in the learned reward. DPO defines an implicit reward, r(x, y) = beta times the mean per-token log-ratio of the DPO policy to the SFT policy. This lets us measure directly what the trained stage rewards. We used the length-normalized objective of OLMo (beta = 5). We took the answers of the DPO checkpoint to the 40 questions and made three rewrites of each answer, with the same model and the same procedure. All three versions are rewrites, so no version differs from the others just because it was rewritten. The three versions are: a light paraphrase (the control; 99.6% of the original's claims, same markdown), a plain-prose version (99.5% of the claims, no markdown), and a reduced version (73.9% of the claims, same formatting style). We compare each version with the paraphrase. The plain version keeps the facts and drops the form. It loses 0.67 units of implicit reward, and its reward is lower for 88% of answers. The reduced version drops a quarter of the facts and keeps the form. It does not lose reward (+0.36; lower for only 17% of answers). A version of this reward that is not normalized by length gives the same order (Appendix D.4). The preference stage therefore rewards the elaborated form and not the facts that the form carries, which is the property that Section 4 attributes to it. We add this test as a new appendix (D.4).

W2/Q1. Separation of data for the direction, selection, and evaluation. In the submitted Table 4, these were not fully separate. We built the direction from 12 of the 40 written questions, and these 12 are among the 40 evaluation questions. Six of the eleven evaluated constraints match or paraphrase the seven form cues that we used to build the direction. The counterpart prompts are not the bare questions. They are the same 12 questions, each followed by one of three goal sentences that set no constraint on the form ("Answer in whatever way you think is best.", "Approach this however you find most useful.", "Be helpful, thorough, and comprehensive in your response."). The revised Section 3 states this, and Appendix C lists every sentence. We chose the Llama layer and gamma on 12 separate development questions. The Qwen layer (14 of 36) and gamma = 1 are the values of the submitted Table 4.

To remove the overlap, we evaluated the same two directions without any change (layer 14 of 36 for Qwen3-4B-Instruct, layer 8 of 32 for Llama-3.1-8B-Instruct, gamma = 1). We evaluated them on 100 benchmark questions that were not used for construction or selection. Five of the eleven constraints were never used in the construction, and we report them separately. The random controls are three random directions with the same norm as v and fixed seeds. We did not change any choice after the test answers were generated. The table below gives the results on the 100 held-out questions. Every condition is judged against the same unconstrained answers, and "Unc. wins" is the share of pairs that the unconstrained answer wins.

Held-out testQwen3-4B-Instruct: DeltaUnc. winsLlama-3.1-8B-Instruct: DeltaUnc. wins
Constrained, unsteered-35.5%95.8%-15.7%72.3%
-v on constrained prompts (rescue)-10.6% (95% CI -12.3 to -9.0)58.3%-5.5% (95% CI -7.5 to -3.5)50.6%
constraints used to build v (6)-36.8% -> -10.7%-16.3% -> -6.6%
constraints never used to build v (5)-34.0% -> -10.5%-15.0% -> -4.2%
+v on unconstrained prompts (induce)-30.3%98.0%-13.5%74.0%
+r, random with the same norm, 3 seeds+6.4%, -1.8%, +0.4%12.0%, 35.0%, 29.0%+0.1%, -0.8%, +0.7%

(3/3)

On questions that the direction never saw, adding v to a prompt with no constraint reproduces most of the constrained loss on both models, and random directions with the same norm do not. Subtracting v recovers 70% of the loss on Qwen3-4B-Instruct and about two thirds on Llama-3.1-8B-Instruct. It works about as well on the five constraints that were never used to build v as on the six that were. On Llama-3.1-8B-Instruct, compliance under the rescue is 89.5%, against 92.4% for the unsteered answers (over the 979 of 1,100 prompts with a valid compliance judgment in both conditions). It is within 2.2 points of the unsteered level on eight of the eleven constraints. Perplexity under the unsteered model is 1.303 for the rescued answers and 1.305 for the unsteered answers.

One of the three counterpart sentences asks for thoroughness, so -v could partly act like a request for thoroughness. To check this, we also built a direction with no goal sentence at all. It is the mean over constrained prompts minus the mean over the same questions with nothing added (Qwen3-4B-Instruct, the 40 build questions, six constraints; layer 14 and gamma = 1 chosen on the development questions). On the 100 test questions, adding this direction to unconstrained prompts reproduces the collapse (-33.5%, unconstrained wins 97.0%). Three random directions with the same norm do not (+5.9%, -5.3%, -1.7%). Subtracting the direction from constrained prompts halves the loss (from -35.5% to -18.6%). So the collapse direction does not depend on the thoroughness sentence. The matched counterparts of Table 4 control for the presence of an extra sentence, and they give the larger rescue.

Q2. Llama numbers in Table 4. We judged each steered condition in its own pairwise run, side by side with the same 40 unconstrained answers. Delta is computed within each run, against the unconstrained score in that run. The Comp. column of Table 4 shows only the steered score. For this reason, the Delta of a steered row cannot be computed by comparing its Comp. value with 7.74, the unconstrained score of the constrained run. The paired values are:

ConditionUnconstrained (paired)Steered/constrainedDeltaUnconstrained wins
Constrained7.7416.070-21.6%83.0%
+v (induce)7.6256.525-14.4%80.0%
+r (random)7.1756.950-3.1%37.5%
-v (rescue)7.4256.832-8.0%57.5%

The conclusions do not change. The random direction stays within 3.1% of its baseline. The collapse direction costs 14.4%. The rescue reduces the loss from 21.6% to 8.0%, which is 13.6 of the 21.6 points (the "14 of 22" in the text). The revised Table 4 shows the paired baseline in every Llama row (new "Base" column).

We believe these changes address the reviewer's questions, and we would be glad to answer any further questions during the discussion.

Dear Reviewer gvbQ,

As the discussion period ends soon, we wanted to follow up on our response and the revised PDF. If they address the reviewer's concerns, we would be grateful if the reviewer would consider raising the score. We would be glad to answer any remaining questions before the deadline.

Reviewer_epWvOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies constraint-induced response collapse, in which simple constraints, such as banning a single character, can substantially reduce an answer’s substantive content. Although this problem has been studied in previous work, the paper provides additional perspectives, particularly through no-op constraints that do not meaningfully restrict the answer but still trigger collapse. It also investigates linear activation steering as a mitigation method and examines OLMo checkpoints to understand how collapse develops during post-training.

Critical strengths

The no-op experiments provide an interesting extension of previous findings. They show that collapse can occur even when an instruction does not impose a meaningful restriction on the answer. This helps characterize the conditions that trigger collapse and broadens the relevance of the problem.

The activation-steering experiments provide a promising mitigation method. The authors demonstrate bidirectional interventions across two model families: adding the direction induces collapse, while subtracting it improves comprehensiveness with little change in constraint compliance. These experiments strengthen the paper beyond merely documenting the phenomenon.

Critical weaknesses

The attribution of the additional collapse after RL to the instruction-following reward is not sufficiently isolated. OLMo’s RL stage trains on multiple tasks using multiple reward signals. Comparing checkpoints before and after RL, together with inspecting one reward component, does not establish that this component specifically causes the additional collapse. A controlled comparison or a more cautious interpretation would strengthen this conclusion.

The content recovered through activation steering could be verified more directly. The reported comprehensiveness improvements are encouraging, but the paper does not apply its existing factual-coverage evaluation to the steered responses. Reporting whether important omitted facts are recovered would provide stronger evidence for the substantive value of the mitigation.

Questions for authors

What evidence isolates the instruction-following reward as the cause of the additional collapse after RL, given OLMo’s mixed-task training and multiple reward signals?

Does activation steering recover important omitted facts when evaluated using the paper’s existing factual-coverage metric?

3 comments

(1/2)

We thank the reviewer for the review. In response, we made the following changes. The revised PDF shows new and revised text in blue.

  • W1: Section 4 now separates what we measure from the mechanism that we propose. A new measurement on the checkpoint where RL starts shows that, of the rewards we measured, the instruction-following reward is the only one with a clear push toward shorter answers.
  • W2: we apply factual coverage to the steered answers on 100 held-out questions. Steering recovers facts, including essential ones. Measured against a second unconstrained answer, which is the level of an answer that loses nothing, steering cuts the loss of facts from 37.7% to 15.7%.

W1. Isolating the IF reward in a mixed RL stage. Our evidence on the instruction-following reward has two parts. The first is a measurement of the incentive of each reward on the checkpoint where RL starts. The second is the result of the OLMo authors' own RL run on instruction-following data only. We did not run an ablation ourselves. As the reviewer suggests, the revised Section 4 also separates what we measure from the mechanism that we propose to explain it. The reviewer is right that the stage mixes rewards. The released Dolci-Instruct-RL prompts are: instruction following (22.1%), math (37.7%), code (11.8%), and general chat (28.5%). Math and code use verifiers that check the final answer. Chat uses an LLM judge that scores one answer at a time. The instruction-following prompts use only IFEvalVerifier. For the RL stage, Table 5 listed only the last of these. The revised table lists the full mix. We add a direct measurement of which reward pushes toward shorter answers, and independent evidence from the OLMo authors.

  1. We measured which reward favors shorter answers on the policy before RL. GRPO updates the policy using the advantages within each group of samples. The sign of sum_i A_i log(length_i) over a group therefore gives the direction in which one update pushes the answer length. We sampled the released DPO checkpoint (where the RL stage starts) 8 times per prompt at temperature 1.0, on prompts drawn from the RL mix. We scored every sample with the reward of its own prompt. On instruction-following prompts (200 prompts, 1,600 samples), the push is -0.068 (95% CI -0.112 to -0.028). In 64% of the groups where the reward varies at all, the shorter samples receive the higher advantage. On math prompts with the answer-checking reward, the push is +0.007 (CI -0.046 to +0.051), and this interval includes zero. On general-chat prompts, we used a single-answer judge with the judge prompt of OLMo (79 prompts; Claude Sonnet 5 replaces Qwen3-32B as the judge model). There the push is -0.022 (CI -0.048 to +0.003), and this interval also includes zero. We measured three of the four rewards (not code). Of these three, the instruction-following verifier is the only one whose push toward shorter answers has a confidence interval that excludes zero. Its incentive also matches the offset in Table 13, which does not depend on the constraint.
  2. The OLMo authors observed the same pattern. The Olmo 3 report (Section 4.4, key findings) states that RL on instruction-following data alone raises IFEval and lowers AlpacaEval. It also states that adding the LLM-judge chat reward to the mix keeps AlpacaEval high. However, a chat judge that scores one answer at a time detects only part of an omission, as Section 2.1 shows. This is consistent with the mixed stage still adding the offset.
  3. The offset does not depend on the constraint (Table 13: mean -8.8 points, SD 3.2, slope 0.96 with CI 0.80 to 1.11). We expect this from a pressure that is the same whichever constraint is present. The reward of the verifier, the fraction of checks passed, predicts such a pressure. Pyatkin et al. (2025) also report that RL on verifiable instruction-following rewards lowers judged response quality ("the verifiable rewards teach the model to prioritize the constraints"). Concurrent work (Yoon et al., 2026, arXiv:2610.19525) reaches the same conclusion on the same checkpoints. In all three matched OLMo 3 DPO-RLVR pairs, RLVR raises constraint satisfaction and lowers response quality. The authors also write that the decline under instruction-following RL "is accompanied by a marked reduction in response length, suggesting that the policy satisfies c while omitting substantive content needed to answer x."

We add the measurement of which reward favors shorter answers as a new appendix (G.1), and Section 4 now cites the finding of the Olmo 3 report.

(2/2)

W2. Facts recovered by steering. Yes. We applied the factual-coverage measure of Section 2 to the rescued answers. We used the same claim lists as Table 7 and the 100 held-out benchmark questions of the new steering evaluation (Qwen3-4B-Instruct, the direction of Table 4, 1,100 pairs). The model is sampled, so a second unconstrained answer to the same question conveys only 53.0% of the first answer's claims. This is the level of an answer that loses nothing, and 100% is not reachable (the last row of Table 7 gives the same control on all 160 questions). The unsteered constrained answers convey 33.0% of the claims, and the rescued answers convey 44.7%. Measured against the 53.0% level, steering cuts the loss of facts from 37.7% to 15.7%, so it removes 58% of the loss. Compared pair by pair with the unsteered answer, the rescued answer conveys more claims in 773 pairs and fewer in 196. The gain holds both on the constraints used in the construction (from 32.3% to 45.2%) and on the five that were not used (from 33.9% to 44.1%). The claim count cannot be raised by writing more words (Section 2). The recovered facts include essential ones. Among the claims graded essential (grade 3, Appendix B), the rescued answers convey 65.4%, against 55.0% for the unsteered answers. On Llama-3.1-8B-Instruct (40 questions, the setting of Table 4), coverage rises from 52.4% to 55.1%, against 63.9% for a second unconstrained answer. We add the held-out coverage result to Section 3 and the full numbers to Appendix C.

We believe these changes address the reviewer's questions, and we would be glad to answer any further questions during the discussion.

Dear Reviewer epWv,

As the discussion period ends soon, we wanted to follow up on our response and the revised PDF. If they address the reviewer's concerns, we would be grateful if the reviewer would consider raising the score. We would be glad to answer any remaining questions before the deadline.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of our revisions and answers to common questions

We thank the three reviewers for their careful reviews and the detailed discussion, which have made the paper stronger. We have answered every question, ambiguity, and concern raised in the reviews and the discussion, and we have revised the paper throughout in response to the reviews. For the convenience of the reviewers and the AC, new and revised text in the PDF is shown in blue, so the changes can be checked directly without reading the whole paper again.

Contributions. Everyday constraints on the form of an answer make instruction-tuned models withhold much of the answer's content while still obeying the constraint. We show that form primers, phrases that refer to the form of the response rather than to the information the question asks for, are enough to trigger this collapse, even when they restrict nothing; that a single direction in the model's activations controls it; and we trace its origin to reward-based post-training. The reviewers found the question meaningful and practical (gvbQ, 19WH), the no-op finding an interesting extension of prior work (epWv) and useful evidence (gvbQ), the experiments extensive and not tied to a single metric (gvbQ), and the steering experiments promising (epWv), of empirical value (gvbQ), and interesting (19WH).

Overview of revisions

  • Section 2.1: the definition of a form primer made explicit, and Reviewer 19WH's alternative account (R5), tested in a pre-registered experiment (Appendix M).
  • Appendix N: for every constraint in the paper, whether it is a form primer, how often the unconstrained answer already obeys it, and its effect, plus the effect of asking for a better answer after a form primer.
  • Appendix A: every judge prompt verbatim, and five checks on the pairwise judge (A.1).
  • Section 3 and Appendix C.1: a held-out evaluation of the steering directions, the factual coverage of the steered answers, and the paired baselines of Table 4.
  • Section 4: measurements separated from the proposed mechanism, with two new analyses of what the DPO and RL rewards favor (Appendices D.4 and G.1).
  • Appendices L and H.1: how the 160 benchmark questions were chosen, with the full list, and compliance at every post-training stage. The code is in the supplementary material.

Common questions

C1. What supports the post-training account? (gvbQ, epWv) Section 4 now separates what we measure from the mechanism we propose. Two new measurements support the mechanism: the reward learned by DPO favors the elaborated form and not the facts it carries, and among the RL rewards we measured, only the instruction-following reward clearly pushes toward shorter answers.

C2. Does steering generalize, and what does it recover? (gvbQ, epWv) On 100 held-out questions, subtracting the direction recovers 70% of the loss on Qwen3-4B-Instruct, and about as much on five constraints never used to build it. On the same model and questions, factual coverage shows that steering also recovers facts: it cuts the loss of facts from 37.7% to 15.7%. Steering keeps the prompt fixed, so it tests cause and effect, which a change to the prompt cannot do (19WH).

C3. What triggers the collapse? (19WH) Form primers are enough to trigger it, which is a claim of sufficiency. Several form primers that nearly every unconstrained answer already obeys still collapse it, and a request for the best possible answer does not prevent the collapse (Appendix N). A pre-registered test compares form priming with Reviewer 19WH's alternative account (Appendix M).

C4. Is the pairwise judge reliable? (19WH) A count of facts that asks no judge which answer is better gives a loss of the same size (37.8% against 36.1%). The judge finds no loss when nothing is missing and picks the same winner in 394 of 395 pairs when the answer order is swapped, a length-discounted judge still finds most of the loss, and a judge from another model family picks the same winner in 207 of 220 pairs (Table 3 and Appendix A.1).

We thank the reviewers again for their time and engagement throughout the discussion.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.