acceptodds
Under review as a conference paper at ICLR 2027

One Token Away from Collapse: The Fragility of Instruction-Tuned Helpfulness

Abstract

Instruction-tuned large language models produce thorough responses when prompted normally. We show that this helpfulness is fragile. Adding a trivial constraint, such as a ban on a single punctuation mark or on the word “the”, costs **17-48%** of comprehensiveness on average in pairwise evaluation of seven models, and all three judges detect the loss. The loss is in content rather than presentation: content criteria degrade - more than surface criteria, and a blinded human evaluation reproduces the pattern. The trigger is the constraint's presence in the prompt, not the effort of obeying it. Four converging tests support this, including bans on words that essentially never occur, which still cost 10-21%, and decoder-level enforcement of the identical constraint with a clean prompt, which removes almost all of the loss (26.0% 4.5%). The response-strategy decision is made before generation, where a linear probe can detect it (-), and it is reversible at a single mid-network layer: steering along a difference-of-means direction with a compliance-preserving decoder mask recovers 64-82% on Qwen-2.5-7B at 100% constraint satisfaction, with no training, and at most 14% at the six other depths tested. Beyond the best steering dose, responses stay near full length while comprehensiveness recovery falls by 18 points, so the judge is not rewarding length. The collapse extends to deployment constraints (14-31%), independent scoring underestimates it by -, and LoRA fine-tuning on 80 examples recovers 65-86%.

Primary area:
foundation or frontier models, including LLMs
Keywords:
instruction following, instruction tuning, large language models, robustness, constrained generation, interpretability, alignment

Reviews

Among:
Shape:
1234this paper, Top 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.67 over 3 reviews, above 99% of the others.
Reviewer_PFKiOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2(+1)
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies a failure mode in instruction-tuned LLMs where adding a seemingly minor constraint, such as banning a punctuation mark or a common word, leads to a substantial loss in response comprehensiveness. Across seven models, the authors find that the loss primarily affects content rather than presentation. They argue that the constraint triggers a change in response planning rather than simply making generation harder.

Critical strengths
  • The paper identifies a practically relevant failure mode of LLMs (since a lot of the use of LLMs comes with specific constraints such as formatting, which I personally experience too), showing that minor instructions can substantially reduce the information provided by an LLM.

  • The evaluation breadth is extensive covering multiple model families along with LLM judges and human evaluations.

Critical weaknesses
  • The paper makes strong claims about a localized planning decision, but the most convincing intervention result is concentrated at a single layer of Qwen2.5-7B. It is unclear how far this mechanistic explanation generalizes to other models and constraint types.

  • The paper shows that steering and fine-tuning can recover much of the lost comprehensiveness, but it remains unclear how robust these interventions are across tasks and constraints beyond those evaluated. The steering result also depends on a decoder mask to maintain full constraint satisfaction, which limits how directly it translates into a general-purpose mitigation.

  • This brings me to my general view of the paper. The paper is fully diagnostic (as mentioned by the authors) and they acknowledge that solutions remain future work. While the initial steering and results are useful, the paper would be much stronger if a solution was developed that goes in the line with these diganostics.

  • Minor comment: The writing, especially in the introduction, could be improved. Some sections are difficult to read, and the contributions section of the intro feels like a list of claims rather than a coherent explanation of the paper's main contributions.

Questions for authors
  • How well does the proposed steering intervention generalize to other model families and sizes?
  • How sensitive are the results to the choice of prompts and constraints?
6 comments
AuthorsOpenReview ↗

Response to Reviewer PFKi (Part 1/5)

We thank the reviewer for the insightful comments. We address the concerns below. Unless mentioned otherwise, all new results use a pairwise GPT-4o-mini judge, sampling with T = 0.7 and top-p 0.9. Δ% is the relative change in judged comprehensiveness from the unconstrained answer to the constrained answer for the same question, and recovery is the share of this loss that an intervention removes. Table, section and appendix numbers (for example Table 24, §6.2 or App. H) refer to the submitted paper. The revised paper keeps these numbers and adds Appendices U–Z. Tables named R3T1, R3T2, ... are new results in this response. Each new table comes from its own run, with new samples or new judge calls. The same cell can therefore differ by a few points from the paper or from another new table.

Q1 / W1. How well does the steering intervention generalize to other model families and sizes, given that the most convincing result is concentrated at a single layer of Qwen2.5-7B?

The paper reports steering with the decoder mask on three model families. It recovers 64–82% on Qwen-2.5-7B and 36–46% on Llama-3.1-8B (Table 19). On Mistral-7B, whose collapse is the smallest of the three, it recovers about 25%, in line with the paper's finding of recovery following the collapse severity (Table 19). The paper also tests the direction on a ban it was not built from. We built it only from the bans on commas and on the word "the", and on Llama it recovers 46% of the loss under the semicolon ban.

The paper ran the layer sweep only on Qwen-2.5-7B. To test the generalization, we ran two new experiments:

  1. the full procedure on a fourth model family (OLMo-3-7B-Instruct) and model with a larger size (Qwen-2.5-14B-Instruct), checked on 80 MT-Bench questions not used to choose the settings,
  2. steering Qwen-2.5-7B only while it reads the prompt, to test whether the shift is set before generation.

1. A fourth family and a larger size. We used the method of the paper. The direction comes from the bans on commas and on the word "the". The dose f sets the length of the added steering vector to f times the average length of the hidden state at that layer (§6.3). We use the decoder mask. As a control, we also steer with a random direction of the same length, at the same layer and dose (the paper's placebo). We first ran a five-depth sweep on the comma ban and then ran the best layers at higher doses. Finally, we applied the same settings, unchanged, to the 80 MT-Bench questions (seed 0). We did not use these questions to build the direction or to choose the settings. Table R3T1 reports the result, using the six-criterion judge of Table 19.

R3T1

Model, banLayer (depth)Unsteered Δ%, paper's prompts / MT-BenchRecovery, paper's promptsRecovery, MT-BenchRandom direction
OLMo-3-7B, word "the"12 of 32 (38%)−38.2 / −42.479%79%2 to 7%
OLMo-3-7B, comma16 of 32 (50%)−47.7 / −51.534%47%−1 to 3%
Qwen-2.5-14B, word "the"24 of 48 (50%)−42.4 / −31.782%60%−12 to 6%
Qwen-2.5-14B, comma30 of 48 (63%)−73.0 / −64.260%55%−12 to −3%

As Table R3T1 shows, steering recovers 34–82% on the paper's prompts and 47–79% on MT-Bench, while the random direction recovers between −12% and 7%. In the sweep, recovery peaks in the middle of the network, at layers 12–16 of 32 in OLMo and at layers 24–30 of 48 in Qwen-2.5-14B. At the earliest and latest depths tested, it falls to at most 6% in OLMo and at most 27% in Qwen-2.5-14B.

The comma collapses are deeper (−47.7% and −73.0%), and steering recovers 34–60% of them. Without steering, Qwen-2.5-14B's answers under the comma ban drop the spaces between words, with 1.4 words per answer on the 40 prompts. The model does the same with the standard Hugging Face generation code and its own chat template, so this is its own behavior. Steering restores normal text.

We also checked that steering keeps the constraint satisfied and the answers fluent:

  1. Under steering, compliance is 90–100%. The mask alone recovers −3% to +2%, so the recovery comes from the direction.
  2. 5-gram repetition (the share of repeated five-word sequences) is at most 0.021 on OLMo, and 0.015 (ban on "the") and 0.056 (ban on commas) on Qwen-2.5-14B.

These checks are in Appendix W of the revised paper. So steering at a mid-network layer works in a fourth family and at a larger size, and on prompts not used to choose the settings.

AuthorsOpenReview ↗

Response to Reviewer PFKi (Part 2/5)

2. Steering only the prompt. The paper steers at every position from the last prompt token onward. We steered Qwen-2.5-7B at layer 15 only while the prompt is processed, and then generated with no intervention (20 prompts × 3 seeds). At dose 0.38, this recovers 51% of the loss under the comma ban and 48% under the ban on "the". A random direction recovers only 18% and 6%. Steering every token at the same dose recovers 59% and 58%, so most of the shift is set before generation (Table R2T3 in our response to Reviewer a4sT).

With the paper's results, steering at a mid-network layer works in Qwen-2.5-7B, Llama-3.1-8B, OLMo-3-7B and Qwen-2.5-14B, which cover three families and two sizes. The revised paper replaces "a single layer" with the layers that work in each model: layer 15 of 28 in Qwen-2.5-7B, layers 12–16 of 32 in OLMo-3-7B, layers 24–30 of 48 in Qwen-2.5-14B, and layer 16 of 32 in Llama (Appendix W). All new results are in Appendices U–Z of the revised paper.

W2. How robust are steering and fine-tuning across tasks and constraints beyond those evaluated, and does the dependence on a decoder mask limit steering as a general mitigation?

We tested this in three ways: 1. constraints that were not used to build the direction, including one with no mask, 2. tasks outside the 40 prompts, for both steering and fine-tuning, 3. whether the steering result depends on the mask.

1. Held-out constraints. On Qwen-2.5-7B at layer 15, we kept the direction unchanged (built only from the bans on commas and on the word "the") and applied it to four other constraints. One of them, no preamble, has no mask. Table R3T2 reports the result (20 prompts × 3 seeds).

R3T2

Held-out constraintUnsteered Δ%Recovery (dose 0.26 / 0.30)Random direction (0.30)
No colons−42.680 / 74%22%
No semicolons−40.080 / 76%22%
No discourse markers−35.275 / 67%12%
No preamble (no mask)−38.492 / 65%10%

As Table R3T2 shows, steering recovers 65–92% of the loss on all four held-out constraints, against 10–22% for a random direction. The direction therefore carries over to constraints it was not built from, including one that cannot be written as a mask.

2. Held-out tasks. We applied the same direction to the 80 MT-Bench questions, which cover writing, roleplay, reasoning, math, coding, extraction, STEM and humanities. It recovers 61% under the comma ban and 85% under the ban on "the", against 11% and 16% for a random direction. On the 30 math, reasoning and coding questions, the correctness loss under the comma ban falls from −36.6% to −13.8%.

Fine-tuning also transfers to prompts the adapters never saw. On MT-Bench, the retrained LoRA adapters of Table R2T9 (our response to Reviewer a4sT) close 42% of the comma-ban gap on both Llama and Qwen. This is measured against each adapter's own unconstrained answers, the stricter baseline. Both steering and fine-tuning therefore carry over to tasks outside the 40 prompts.

3. Dependence on the mask. The no-preamble constraint has no mask, yet steering recovers 65–92% (Table R3T2), with 100% of answers preamble-free. On the semicolon ban, the mask never changes any token, and Llama's outputs with and without it are identical (App. J.2). In both cases the recovery comes from the direction alone. In the paper, the mask holds compliance fixed during steering, so any recovery must come from the direction (§6.3). The mask alone recovers 0% to −3% (Table 2, line 1b), and −3% to +2% in the new models.

AuthorsOpenReview ↗

Response to Reviewer PFKi (Part 3/5)

W3. The paper is fully diagnostic, and it would be much stronger with a solution in line with the diagnostics.

The diagnosis says the clause changes the response strategy while the model can still write comprehensive constrained text. Any intervention that restores the strategy without removing the clause should therefore help. We measured four such mitigations: 1. LoRA fine-tuning, 2. drafting freely and then rewriting under the constraint, 3. steering with the mask, 4. prompt-level instructions.

1. LoRA fine-tuning (§7.3). We train a small adapter (rank 16) on only 80 examples of two lexical bans. It recovers 65–86% of the loss on lexical constraints it never saw in training, relative to the original model (Table 39). Against each adapter's own unconstrained answers, retrained adapters close 52% (Llama) and 67% (Qwen) of the gap (Table R2T9 in our response to Reviewer a4sT). So one short fine-tuning run on 67–80 examples teaches the model to keep its answers full under other constraints as well.

2. Drafting freely, then rewriting under the constraint. This is the §6.1 diagnostic used as a black-box procedure. It needs no training, no logit access and no token form of the constraint, so it applies to API models and to constraints such as hedging. Table R3T3 reports it with the GPT-4o judge, with compliance in brackets.

R3T3

ModelConstraintDirect Δ%Two-pass Δ%
GPT-4o-minicomma−42.3+4.6 (100%)
GPT-4o-minino preamble−25.1+4.1 (100%)
GPT-4o-minihedging−28.3−15.1 (100%)
Qwen-2.5-7Bno preamble−41.5−6.2 (100%)
Qwen-2.5-7Bhedging−30.3−11.9 (92.5%)
Llama-3.1-8Bno preamble−19.9−2.2 (100%)
Llama-3.1-8Bhedging−24.1−12.3 (95%)

As Table R3T3 shows, two-pass reduces the loss in every cell. It removes 85% or more of the loss on GPT-4o-mini's comma ban and on no preamble, and about half of it on hedging (47–61%). On GPT-4o-mini it brings the comma ban from −42.3% to +4.6%.

3. Steering with the mask (§6.3 and Part 1). Steering needs no training. It adds one vector at one layer, and we get this vector by comparing the model's internal state on the 40 prompts with and without two bans. It recovers 64–82% of the loss on Qwen-2.5-7B and 34–82% on the two new models. The same vector also recovers 65–92% on four constraints it was not built from (Table R3T2), and 61–85% on the 80 MT-Bench questions.

4. Prompt-level instructions. "Keep your answer equally thorough" removes between a third and a half of the loss (−27.2% → −14.3% and −24.8% → −15.7%, Table 29), and chain-of-thought prompting recovers 16–43% (Table 25). However, these instructions help only a user who already knows the problem exists. A user who meets the problem for the first time has no reason to add "keep your answer equally thorough" to a formatting request, so in normal use the loss remains.

The revised §7.3 names all four, and Appendices M, V and W give the new numbers above. The introduction no longer leaves solutions to future work and briefly names the four mitigations we tested.

Reviewer_a4sTOpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper examines whether the helpfulness acquired through instruction tuning remains robust to minor changes in expression requirements. Starting from a simple intervention—appending a constraint such as “do not use commas”—the authors observe that models produce substantially less comprehensive answers despite the underlying task and knowledge requirements remaining unchanged. They first establish that the degradation involves substantive information loss rather than merely changes in presentation. Through controlled comparisons involving decoder-level enforcement, rarely occurring banned words, and noncompliant responses, they then argue that the presence of a restriction in the prompt, rather than the difficulty of satisfying it alone, is the principal trigger. Further analyses investigate whether this behavior reflects a change in response strategy: two-stage rewriting suggests that some constrained-generation capability remains available, while linear probing and activation interventions indicate that representations predictive of the subsequent response emerge before generation and can be manipulated to recover content. The authors interpret these findings as evidence that instruction-tuned helpfulness may depend on fragile response templates, with additional constraints prompting a shift toward less informative answers. Finally, they extend the analysis to practical deployment requirements and show that independent response scoring can underestimate the resulting degradation. Overall, the paper argues for treating content preservation under constraints as an explicit objective in both training and evaluation.

Critical strengths
  1. The paper identifies a meaningful gap between instruction compliance and task quality, supported by evaluations across models, questions, and constraint types.

  2. Content coverage checks, human assessment, and format-normalized comparisons support genuine information loss beyond evaluator preferences for length or presentation.

Critical weaknesses
  1. The proposed mechanism is not uniquely established. Additional instructions may impose coordination or monitoring costs even when rarely relevant or ultimately violated. Human dual-task interference offers a useful analogy [1], without establishing a shared mechanism.

  2. Reduced comprehensiveness does not necessarily imply worse practical performance. The main questions favor detailed answers, whereas some tasks reward brevity or plain language. Necessary information loss should be distinguished from appropriate simplification.

  3. Recovery claims require stronger validation. Two-stage experiments primarily measure length recovery and involve additional computation, motivating content-sensitive, budget-matched comparisons. Fine-tuning gains should also be measured against each adapted model’s own unconstrained baseline to separate general quality improvements from improved constraint robustness.

[1] Wu, Y., Wang, Q. Dual-task interference: Bottleneck constraint or capacity sharing? Evidence from automatic and controlled processes. Atten Percept Psychophys 86, 815–827 (2024). https://doi.org/10.3758/s13414-024-02854-1

Questions for authors
  1. What evidence specifically distinguishes template disruption or a planning failure from general costs of integrating and monitoring additional instructions? Could the mechanistic claims be narrowed where these explanations remain observationally compatible?

  2. For the existing two-stage experiment, can you report substantive information coverage, comprehensiveness, correctness, and final constraint satisfaction alongside length retention? Do these measurements support the claimed distinction between a capability limitation and a strategy-selection failure?

  3. Under comparable total inference budgets, how does direct constrained generation compare with unrestricted drafting followed by constrained rewriting? Including an unrestricted drafting-and-revision control, and reporting latency or token costs, would help isolate the benefit of separating task solving from constraint handling.

9 comments
AuthorsOpenReview ↗

Response to Reviewer a4sT (Part 1/6)

We thank the reviewer for the thoughtful review and the specific requests. We ran each comparison the reviewer asked for and address them below. Unless mentioned otherwise, all new results use the pairwise GPT-4o-mini judge of Figure 2, the 40 prompts of the paper, and sampling with T = 0.7 and top-p 0.9. Δ% is the relative change in judged comprehensiveness from the unconstrained answer to the constrained answer for the same question. A negative Δ% is a loss of content, and we call the size of this loss the cost of the constraint. Table, section and appendix numbers (for example Table 24, §6.2 or App. H) refer to the submitted paper. The revised paper keeps these numbers and adds Appendices U–Z. Tables named R2T1, R2T2, ... are new results in this response. Each new table comes from its own run, with new samples or new judge calls. The same cell can therefore differ by a few points from the paper or from another new table.

Q1 / W1. What evidence distinguishes template disruption or a planning failure from general costs of integrating and monitoring additional instructions? Could the mechanistic claims be narrowed where these explanations remain compatible?

The monitoring account says that any added instruction costs content, because the model must keep the instruction in mind while it writes. This is the capacity-sharing account in the dual-task work the reviewer cites (Wu & Wang, 2024). We tested it with four new experiments:

  1. a rule that needs the same monitoring as a ban but removes no word the model would use,
  2. sentences that ban no word,
  3. steering the model only while it reads the prompt,
  4. steering the prompt in the opposite direction, with no constraint in the prompt.

After the four experiments, we list two further results that bear on this question, and the phrases we narrowed in the revised paper.

1. A rule that needs monitoring but removes no word. Every ban we had tested asks the model for two things: to track the rule and to leave something out. To separate the two, we added the rule "Write the word 'the' in capital letters (THE) every time you use it in your response." To follow it, the model must check every "the" it writes, as it must under a ban on "the", but it removes no word from the answer. Table R2T1 compares this rule with the ban on "the" and with the ban on "zephyr", a word that never occurs (40 prompts, one sample each). Each cell gives Δ%, with answer length as a percentage of the unconstrained answer in brackets.

R2T1

ModelBan "the"Write every "the" as THEBan "zephyr" (never occurs)
Llama-3.1-8B−24.7 (50%)−17.2 (75%)−13.3 (78%)
Qwen-2.5-7B−42.2 (15%)−34.1 (42%)−21.0 (62%)
Mistral-7B−18.7 (58%)−10.7 (85%)−11.7 (84%)

As Table R2T1 shows, the capitals rule costs 7.5, 8.1 and 8.0 points less than the ban in the three models (bootstrap 95% CIs over prompts exclude zero), and its answers are 25–27 points longer. Qwen writes 94% of its "the"s as THE and still shows the same gap. Llama writes 67% of them as THE, and Mistral 23%. Qwen also capitalizes unrelated words in 27 of 40 answers, which may make its capitals cost higher, not lower.

If the loss came from tracking the rule, the ban and the capitals rule would cost the same, because both need the same checking. They do not. The cost follows how much of the default answer the rule changes. Removing every "the" changes more than capitalizing it, and the zephyr ban, which requires no change to the answer, costs the least or about the same as capitals. This is the gradient the paper reports in §5.2 (Table 16).

AuthorsOpenReview ↗

Response to Reviewer a4sT (Part 2/6)

2. Sentences that ban no word. We added five sentences to the prompt, one at a time. The first permits a word: "You may use the word 'the' in your response." The other four say nothing about wording: "Make sure your answer is accurate.", "Base your answer on well-established information.", "Answer in English." and "Answer the question that was asked." Table R2T2 reports Δ% for each (40 prompts, one sample each).

R2T2

Sentence added to the promptLlamaQwenMistral
"You may use the word 'the' in your response."−12.9−27.3−16.4
"Make sure your answer is accurate."+4.2−3.8+4.0
"Base your answer on well-established information."+3.0+2.4+4.3
"Answer in English."+0.3−8.0−3.5
"Answer the question that was asked."−3.8−12.5−9.5

As Table R2T2 shows:

  1. The two instructions about accuracy cost at most 3.8% (−3.8% to +4.3%).
  2. The permission costs 12.9–27.3%, and its answers keep only 87 / 48 / 68% of the unconstrained length. They still use "the" at least as often as unconstrained answers do (5.2–6.9 vs 4.3–5.5 per 100 words).
  3. "Answer in English" and "Answer the question that was asked" cost up to 8.0% and 12.5%, which is less than the permission in every model.

Banning "the" (Table R2T1) costs 11.8 and 14.9 points more than permitting it on Llama and Qwen (bootstrap 95% CIs exclude zero), and 2.3 points more on Mistral, where the CI includes zero.

The permission gives the model no rule to track and no word to leave out, yet it costs content. The four other sentences are added instructions too, and each costs less than the permission in every model. So the loss is not a general cost of any added instruction, and it is not a cost of tracking a rule. A sentence about the wording of the answer is enough to cause it, even when it bans no word.

3. Steering only while the model reads the prompt. The paper steers at every position from the last prompt token onward. To test whether the change is set before generation, we steered Qwen-2.5-7B at layer 15 only while the prompt is processed, and then generated with no intervention (20 prompts × 3 seeds, decoder mask on, 100% compliance). This uses the six-criterion judge, which scores six criteria separately, including comprehensiveness and correctness. The dose f sets the length of the added steering vector to f times the average length of the hidden state at that layer, so dose 0.38 adds a vector 38% as long as the hidden state (§6.3). Table R2T3 compares this with steering every token in the same run. Each cell is the recovery, the share of the loss that steering removes.

R2T3

BanPrompt only, dose 0.30 / 0.38 / 0.50Every token, dose 0.30 / 0.38Random direction, prompt only, dose 0.38
Comma39 / 51 / 63%55 / 59%18%
Word "the"50 / 48 / 52%78 / 58%6%

As Table R2T3 shows, steering only the prompt removes 39–63% of the loss, against 6–18% for a random direction. At dose 0.38 it removes 51% and 48%, which is 83–86% of what steering every token removes at the same dose (59% and 58%). Most of the effect is therefore set while the model reads the prompt, before it writes any word of the answer.

AuthorsOpenReview ↗

Response to Reviewer a4sT (Part 3/6)

4. Steering in the opposite direction, with no constraint. We steered Qwen-2.5-7B along the same direction with the opposite sign, on prompts with no constraint, again only while the prompt is processed (20 prompts × 3 seeds, six-criterion judge). Table R2T4 reports Δ% (mean ± sd over seeds) and answer length.

R2T4

Steering of the promptΔ%Length
Dose 0.30−19.6 ± 1.669%
Dose 0.38−29.8 ± 2.454%
Dose 0.50−42.2 ± 3.228%
Random direction, dose 0.38−7.3 ± 2.992%

As Table R2T4 shows, the loss grows with the dose, and at dose 0.38 it is four times that of a random direction (−29.8% vs −7.3%). Correctness changes by at most 2%, and most answers are fluent prose with no added repetition. Here the prompt contains no rule, yet the answers lose content in the same way as under a constraint: they are shorter and fluent, and they stay correct. The direction alone, set while the prompt is read, is enough to cause the loss.

Two further results. Two other results also bear on the monitoring account.

  1. The model can write the full answer under the rule. Llama writes a draft under the comma ban and then revises it under the same ban, with an instruction to keep all its information and add what is missing. The loss moves from −26.2% to −8.2% at 95% compliance (Table R2T8, Part 6).
  2. The model's plan also shrinks where the rule does not apply. With thinking enabled, Qwen3-30B-A3B writes a reasoning trace before any constrained text. Under the ban on "the", that trace shrinks from 671 to 268 words and is shorter on 40 of 40 prompts. The trace still uses "the" at 5.8 vs 6.4 per 100 words, so the ban is not applied there.

In Experiments 2 and 4, the loss appears with no rule to track in the prompt. Experiment 2 also shows that instructions about accuracy cost at most 3.8%, and Experiment 1 shows that two rules with the same monitoring differ in cost by how much of the answer they change. A monitoring account alone predicts none of these three results. They match the paper's account, which has two parts: a cost from the presence of a sentence about the answer's wording, and a cost that grows with how much of the default answer the rule changes (§5.2, Table 16).

Changes to the paper. As the reviewer suggested, we narrowed two phrases. In the abstract, which we will revise upon acceptance, "The response-strategy decision is made before generation, where a linear probe can detect it" becomes "The constraint's effect on response length is set before generation: it is linearly readable from the prompt representation". We also removed "localized" from the introduction, the §6 title, §6.3 and §8. These now say that the effect is reversible by steering at a mid-network layer, and Appendix W lists the layers for Qwen-2.5-7B, Llama-3.1-8B, OLMo-3-7B and Qwen-2.5-14B (see our response to Reviewer PFKi). §6.2 is now titled "The Effect Is Set Before Generation". §5 now states that a sentence about the answer's wording costs content even when it bans no word, while two instructions about accuracy cost at most 3.8%. Appendices W and X report prompt-only steering in both directions, the capitals test, and the permission and neutral-instruction controls, with pointers from §5 and §6.3.

Reviewer_eYVFOpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper discloses an important issue of current LLMs: a simple token can destroy the instruction-tuned helpfulness. Besides, this paper has identify the triggers and presents some mechanisms and control.

Critical strengths
  1. This paper discloses a very important issue in terms of the helpfulness of LLMs, that a simple word can destroy the performance of LLMs. Besides, it explores what triggers the collapse by providing four converging tests.

  2. This paper provides a general explanation of the mechanism to study potential internal reasons, which are overall reasonable.

Critical weaknesses
  1. The abstract is too full of numbers, affecting the overall fluency. I think there is not necessary to present so many numbers in the abstract. You can put it in to Intro or Experiments.

  2. The motivation case in Figure 1 is good, but some terms lack of explanations, such as what " R^2 = 0.75 " means and what you means of 27% comprehensiveness?

  3. You have highlighted some general types of collapse in Figure 2, I would like to know if there are types or can you form up a large set of these collapse types?

Questions for authors

See above

4 comments
AuthorsOpenReview ↗

Response to Reviewer eYVF (Part 1/2)

We thank the reviewer for the positive assessment and the helpful suggestions. We address the three points below. Unless mentioned otherwise, Δ% is the relative change in judged comprehensiveness under the pairwise GPT-4o-mini judge of Figure 2, on the 40 prompts of the paper. Table, section and appendix numbers (for example Table 24, §6.2 or App. H) refer to the submitted paper. The revised paper keeps these numbers and adds Appendices U–Z. Table R1T1 is a new result in this response.

W1. The abstract has too many numbers, which affects its fluency. They could go in the introduction or the experiments.

We plan to revise the abstract as follows. It keeps only the numbers behind the main claims and moves the rest to the introduction (§1). We will make this change upon acceptance, so that the PDF and the abstract on OpenReview match during the discussion.

"Instruction-tuned large language models produce thorough responses when prompted normally. We show that this helpfulness is fragile. Adding a trivial constraint, such as a ban on one punctuation mark or on the word 'the', makes seven instruction-tuned models give answers that are 17–48% less comprehensive on average. Three LLM judges and a blinded human evaluation all detect the loss, and it falls on content rather than presentation. The trigger is the constraint's presence in the prompt, not the effort of obeying it: bans on words that never occur still cost 10–21%, while enforcing the same constraint at the decoder with a clean prompt removes almost all of the loss. The constraint's effect on response length is set before generation: it is linearly readable from the prompt representation. Steering at a mid-network layer, with no training, recovers 34–82% of the loss in four models while the decoder mask enforces the constraint, and length alone does not explain the recovered scores. The failure extends to realistic deployment constraints, standard independent scoring underestimates it by 2.0–6.8×, and LoRA fine-tuning on 80 examples recovers 65–86% on held-out lexical constraints."

W2. In Figure 1, what do "R² = 0.75" and "27% comprehensiveness" mean?

"R² = 0.75" is the cross-validated R² of a linear (Ridge) probe. The probe predicts response length from the hidden state of the last prompt token at layer 16 of Llama-3.1-8B-Instruct. It is computed before any token is generated, across unconstrained and constrained prompts (0.747 in Table 10; per-layer values in Table 33; method in §6.2). This means the probe explains 75% of the variance in response length in 5-fold cross-validation, and 71% when whole prompts are held out (App. I). "−27% comprehensiveness" is Δ%, the relative change in mean judged comprehensiveness from the unconstrained to the constrained response (defined in §3). For Llama under the comma ban, mean comprehensiveness on the 1–10 scale falls from 9.25 to 6.75, so Δ% = (6.75 − 9.25) / 9.25 = −27.0% (pairwise GPT-4o-mini judge, Table 5). The revised paper uses this caption:

"Constraint-induced response collapse. Adding 'Do not use any commas' to the same prompt makes Llama-3.1-8B-Instruct switch from structured answers to plain prose: averaged over the 40 prompts, responses shrink from 685 to 297 tokens, and mean judged comprehensiveness falls from 9.25 to 6.75 on a 1–10 scale, a 27% relative drop (Δ%, §3). R² = 0.75: a linear probe on the last prompt token's hidden state at layer 16 predicts response length before generation (§6.2). The task, the knowledge required, and the permitted output format are unchanged."

AuthorsOpenReview ↗

Response to Reviewer eYVF (Part 2/2)

W3. Figure 2 highlights some general types of collapse. Are there more types, and can they be formed into a larger set?

Table R1T1 sorts all 27 constraints in the paper into six types. The 27 constraints are:

  1. the eight constraints presented in the Figure 2 of the paper: bans on commas, colons, semicolons, the word "the", discourse markers and bullet lists, plus two combined bans (comma + colon, comma + bullets) (App. P),
  2. 15 bans on single words, from words that never appear in a normal answer (such as "zephyr") to words that appear in almost every sentence (such as "and") (§5.2, Table 16),
  3. four deployment constraints: professional tone, hedging, no preamble and plain language (§7.1, App. L).

We tested each constraint on three models: Llama, Qwen and Mistral. Each row of Table R1T1 is one type of constraint, and the "Cases" column counts the tests of that type. For example, the no-preamble row has 3 cases: the no-preamble constraint on Llama, on Qwen and on Mistral. To keep the table short, we combine the 15 single-word bans into four groups by how often the word appears. Each model then has 16 cases: the eight constraints of Figure 2, the four groups of word bans and the four deployment constraints. The three models together give 48. All Δ% values come from the paper, and the deployment rows use the GPT-4o judge of §7.1. The three length columns sort the cases with a loss of 5% or more by how long the answers are compared with the normal answer. The last column counts the cases with almost no loss (Δ% above −5%).

R1T1

What the constraint asks forCasesMean Δ% (range)Length < 50%50–80%≥ 80%Almost no loss
Bans an item the model uses often (≥ 1 per answer: comma, colon, the word "the", comma + colon, very frequent words)15−31.9 (−60.9 to −17.7)9600
Bans an item the model rarely uses (< 1 per answer: semicolon, discourse markers, words used sometimes, rarely or never)15−19.5 (−34.3 to −10.1)31110
Bans list structure (bullets, comma + bullets)6−14.4 (−38.8 to +4.9)1212
Sets tone or stance (professional tone, hedging)6−14.6 (−24.7 to −0.3)0231
Restricts only the opening (no preamble)3−20.9 (−34.3 to −13.8)1200
Asks for simpler content (plain language)3−34.2 (−45.4 to −27.1)1200
All48152553

Table R1T1 and one related result in the paper show four things:

  1. For bans, the loss grows with how often the model would use the banned item. Bans on items it uses at least once per answer cost 31.9% on average, and 9 of their 15 cases are below half length. Bans on items it uses less often cost 19.5%, and 11 of their 15 cases keep 50–80% of the length.
  2. A ban still costs content when obeying it needs no change to the answer. Bans on words that never occur, such as "zephyr", cost 10.1–21.2% (Table 16).
  3. Constraints that do not ban a word or symbol also cost content. Professional tone and hedging cost up to 24.7%. The no-preamble constraint costs 13.8–34.3%, although it applies only to the first sentence. Plain language costs the most of the deployment constraints (27.1–45.4%), but some simplification is intended there, so §7.1 reports it separately.
  4. Content can be lost while length is kept. Under the list ban, answers keep 89% of their length but only 58% of the baseline's factual claims (App. H).

Only 3 of the 48 cases have almost no loss, all on Mistral: the two list bans and professional tone (Δ% between −3.2 and +4.9; Table 7 and App. L).

The collapse therefore takes several forms, and for bans its size grows with how often the banned item appears in the default answer. Table R1T1 is now Appendix U, and §4.1 refers to it.

For this rebuttal we also ran new tests of the mechanism, reported in our responses to Reviewers a4sT and PFKi. On Qwen-2.5-7B, steering only while the model reads the prompt removes 39–63% of the loss, and steering the opposite way with no constraint causes a 19.6–42.2% loss (Tables R2T3 and R2T4). Steering also recovers 34–82% of the loss in OLMo-3-7B and Qwen-2.5-14B (Table R3T1).

We thank the reviewer again for these suggestions, which have made the revised paper clearer. If these changes and the new results address the reviewer's concerns, we would be grateful if the reviewer would consider raising the score, and we are happy to discuss any remaining point.

Thanks for the detailed responses. The authors have resolved my concerns. I would encourage the authors to incorporate these details into the next version. I am keen to raise my score and suggest a clear acceptance to ICLR.

AuthorsOpenReview ↗

A big thank you to Reviewer eYVF

We are truly grateful to the reviewer for their time, their careful reading of our responses, and their strong support of our paper. We sincerely appreciate the reviewer's decision to raise the score and recommend acceptance.

We will make sure that all of the reviewer's suggestions are included in the final camera-ready version upon acceptance.

Thank you again for the thoughtful review and kind words.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of our revisions and answers to common questions

We thank the reviewers for their careful reading and constructive feedback. We have answered every question, ambiguity and concern raised in the reviews, ran each experiment the reviewers asked for, and revised the paper wherever a reviewer asked for a change.

Our paper in brief. Trivial constraints, such as a ban on commas or on the word "the", make seven instruction-tuned models give 17–48% less comprehensive answers on average. The trigger is the constraint's presence in the prompt, not the effort of obeying it. The effect is set before generation and can be reversed. The reviewers found this failure mode important and practically relevant (eYVF, PFKi), with a broad evaluation across models, judges and human raters (PFKi, a4sT).

Blue text. For the convenience of the reviewers and the area chair, every revised or added passage in the PDF is shown in blue. The changes can be checked directly, without reading the whole paper again. Table, section and appendix numbers of the submitted version are unchanged, and the new material is in Appendices U–Z.

Abstract. Reviewer eYVF asked for an abstract with fewer numbers, and Reviewer a4sT asked us to narrow the mechanistic claims, including one phrase in the abstract. Our responses give the new abstract. To keep the PDF identical to the abstract on the OpenReview page during the discussion, we will make this change upon acceptance.

Main revisions

  • Figure 1 caption now defines R² and Δ% (eYVF).
  • The contributions are rewritten as one argument, and the introduction names the four mitigations we tested (PFKi).
  • The mechanism claims are narrowed as requested: "localized" is removed, and §6 and §6.2 are retitled (a4sT, PFKi).
  • Appendix U: a taxonomy of all 27 constraints over 48 model–constraint cases (eYVF).
  • Appendix V: full two-pass metrics (claims, correctness, compliance, tokens), a draft-then-revise comparison with equal calls, and two-pass as a black-box mitigation (a4sT, PFKi).
  • Appendix W: steering on OLMo-3-7B and Qwen-2.5-14B, on held-out constraints and MT-Bench, and steering of the prompt alone in both directions (PFKi, a4sT).
  • Appendix X: a capitals rule and single-sentence controls that separate the clause from the cost of monitoring an instruction (a4sT).
  • Appendix Y: fact coverage and a judge that scores only necessary information (a4sT).
  • Appendix Z: paraphrases of each ban and the ban moved to the system prompt (PFKi).
  • Appendix M: LoRA measured against each adapter's own unconstrained answers (a4sT).

Common questions

C1. Does the mechanism hold beyond one layer of one model? (PFKi, a4sT) Yes. We tested this in three ways.

  • Steering works at a mid-network layer in four models from three families, not only in Qwen-2.5-7B. It recovers 34–82% of the loss, and it also works on MT-Bench questions that were not used to set it up (Appendix W).
  • The effect is set before the answer starts. Steering only the prompt removes about half of the loss, and steering the prompt the opposite way causes the loss with no constraint in the prompt (Appendix W).
  • The loss is not a general cost of following an extra instruction. A rule that needs the same checking as a ban but removes no word costs less than the ban, and instructions about accuracy cost almost nothing (Appendix X).

C2. Can the loss be reduced, and is recovery measured on content? (PFKi, a4sT) Yes. The revised paper reports four mitigations: LoRA, drafting then rewriting, steering and prompt instructions.

  • LoRA closes more than half of the gap even when measured against each adapter's own answers (Appendix M).
  • Drafting freely and then rewriting needs only an API. On GPT-4o-mini it removes the whole loss under the comma ban (Appendix V).
  • Recovery is measured on content, not only length. The rewritten answers keep almost all the facts of the unconstrained answer, and with the same number of calls a free draft beats a constrained draft in every case (Appendix V).

C3. How broad is the effect across constraints and prompts? (eYVF, PFKi) Of 48 model–constraint cases across six types of constraint, only 3 show almost no loss. Three paraphrases of each ban, and the ban moved to the system prompt, keep the loss within 8.0 points of the original wording. The paper already shows the effect on 80 MT-Bench and 400 AlpacaEval prompts (App. E, Table 24).

We hope these revisions and results address all of the reviewers' concerns, and we are glad to answer any further question.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.