- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2(+1)
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies a failure mode in instruction-tuned LLMs where adding a seemingly minor constraint, such as banning a punctuation mark or a common word, leads to a substantial loss in response comprehensiveness. Across seven models, the authors find that the loss primarily affects content rather than presentation. They argue that the constraint triggers a change in response planning rather than simply making generation harder.
Critical strengths
-
The paper identifies a practically relevant failure mode of LLMs (since a lot of the use of LLMs comes with specific constraints such as formatting, which I personally experience too), showing that minor instructions can substantially reduce the information provided by an LLM.
-
The evaluation breadth is extensive covering multiple model families along with LLM judges and human evaluations.
Critical weaknesses
-
The paper makes strong claims about a localized planning decision, but the most convincing intervention result is concentrated at a single layer of Qwen2.5-7B. It is unclear how far this mechanistic explanation generalizes to other models and constraint types.
-
The paper shows that steering and fine-tuning can recover much of the lost comprehensiveness, but it remains unclear how robust these interventions are across tasks and constraints beyond those evaluated. The steering result also depends on a decoder mask to maintain full constraint satisfaction, which limits how directly it translates into a general-purpose mitigation.
-
This brings me to my general view of the paper. The paper is fully diagnostic (as mentioned by the authors) and they acknowledge that solutions remain future work. While the initial steering and results are useful, the paper would be much stronger if a solution was developed that goes in the line with these diganostics.
-
Minor comment: The writing, especially in the introduction, could be improved. Some sections are difficult to read, and the contributions section of the intro feels like a list of claims rather than a coherent explanation of the paper's main contributions.
Questions for authors
- How well does the proposed steering intervention generalize to other model families and sizes?
- How sensitive are the results to the choice of prompts and constraints?
6 comments
Response to Reviewer PFKi (Part 2/5)
2. Steering only the prompt. The paper steers at every position from the last prompt token onward. We steered Qwen-2.5-7B at layer 15 only while the prompt is processed, and then generated with no intervention (20 prompts × 3 seeds). At dose 0.38, this recovers 51% of the loss under the comma ban and 48% under the ban on "the". A random direction recovers only 18% and 6%. Steering every token at the same dose recovers 59% and 58%, so most of the shift is set before generation (Table R2T3 in our response to Reviewer a4sT).
With the paper's results, steering at a mid-network layer works in Qwen-2.5-7B, Llama-3.1-8B, OLMo-3-7B and Qwen-2.5-14B, which cover three families and two sizes. The revised paper replaces "a single layer" with the layers that work in each model: layer 15 of 28 in Qwen-2.5-7B, layers 12–16 of 32 in OLMo-3-7B, layers 24–30 of 48 in Qwen-2.5-14B, and layer 16 of 32 in Llama (Appendix W). All new results are in Appendices U–Z of the revised paper.
W2. How robust are steering and fine-tuning across tasks and constraints beyond those evaluated, and does the dependence on a decoder mask limit steering as a general mitigation?
We tested this in three ways: 1. constraints that were not used to build the direction, including one with no mask, 2. tasks outside the 40 prompts, for both steering and fine-tuning, 3. whether the steering result depends on the mask.
1. Held-out constraints. On Qwen-2.5-7B at layer 15, we kept the direction unchanged (built only from the bans on commas and on the word "the") and applied it to four other constraints. One of them, no preamble, has no mask. Table R3T2 reports the result (20 prompts × 3 seeds).
R3T2
| Held-out constraint | Unsteered Δ% | Recovery (dose 0.26 / 0.30) | Random direction (0.30) |
|---|---|---|---|
| No colons | −42.6 | 80 / 74% | 22% |
| No semicolons | −40.0 | 80 / 76% | 22% |
| No discourse markers | −35.2 | 75 / 67% | 12% |
| No preamble (no mask) | −38.4 | 92 / 65% | 10% |
As Table R3T2 shows, steering recovers 65–92% of the loss on all four held-out constraints, against 10–22% for a random direction. The direction therefore carries over to constraints it was not built from, including one that cannot be written as a mask.
2. Held-out tasks. We applied the same direction to the 80 MT-Bench questions, which cover writing, roleplay, reasoning, math, coding, extraction, STEM and humanities. It recovers 61% under the comma ban and 85% under the ban on "the", against 11% and 16% for a random direction. On the 30 math, reasoning and coding questions, the correctness loss under the comma ban falls from −36.6% to −13.8%.
Fine-tuning also transfers to prompts the adapters never saw. On MT-Bench, the retrained LoRA adapters of Table R2T9 (our response to Reviewer a4sT) close 42% of the comma-ban gap on both Llama and Qwen. This is measured against each adapter's own unconstrained answers, the stricter baseline. Both steering and fine-tuning therefore carry over to tasks outside the 40 prompts.
3. Dependence on the mask. The no-preamble constraint has no mask, yet steering recovers 65–92% (Table R3T2), with 100% of answers preamble-free. On the semicolon ban, the mask never changes any token, and Llama's outputs with and without it are identical (App. J.2). In both cases the recovery comes from the direction alone. In the paper, the mask holds compliance fixed during steering, so any recovery must come from the direction (§6.3). The mask alone recovers 0% to −3% (Table 2, line 1b), and −3% to +2% in the new models.
Response to Reviewer PFKi (Part 3/5)
W3. The paper is fully diagnostic, and it would be much stronger with a solution in line with the diagnostics.
The diagnosis says the clause changes the response strategy while the model can still write comprehensive constrained text. Any intervention that restores the strategy without removing the clause should therefore help. We measured four such mitigations: 1. LoRA fine-tuning, 2. drafting freely and then rewriting under the constraint, 3. steering with the mask, 4. prompt-level instructions.
1. LoRA fine-tuning (§7.3). We train a small adapter (rank 16) on only 80 examples of two lexical bans. It recovers 65–86% of the loss on lexical constraints it never saw in training, relative to the original model (Table 39). Against each adapter's own unconstrained answers, retrained adapters close 52% (Llama) and 67% (Qwen) of the gap (Table R2T9 in our response to Reviewer a4sT). So one short fine-tuning run on 67–80 examples teaches the model to keep its answers full under other constraints as well.
2. Drafting freely, then rewriting under the constraint. This is the §6.1 diagnostic used as a black-box procedure. It needs no training, no logit access and no token form of the constraint, so it applies to API models and to constraints such as hedging. Table R3T3 reports it with the GPT-4o judge, with compliance in brackets.
R3T3
| Model | Constraint | Direct Δ% | Two-pass Δ% |
|---|---|---|---|
| GPT-4o-mini | comma | −42.3 | +4.6 (100%) |
| GPT-4o-mini | no preamble | −25.1 | +4.1 (100%) |
| GPT-4o-mini | hedging | −28.3 | −15.1 (100%) |
| Qwen-2.5-7B | no preamble | −41.5 | −6.2 (100%) |
| Qwen-2.5-7B | hedging | −30.3 | −11.9 (92.5%) |
| Llama-3.1-8B | no preamble | −19.9 | −2.2 (100%) |
| Llama-3.1-8B | hedging | −24.1 | −12.3 (95%) |
As Table R3T3 shows, two-pass reduces the loss in every cell. It removes 85% or more of the loss on GPT-4o-mini's comma ban and on no preamble, and about half of it on hedging (47–61%). On GPT-4o-mini it brings the comma ban from −42.3% to +4.6%.
3. Steering with the mask (§6.3 and Part 1). Steering needs no training. It adds one vector at one layer, and we get this vector by comparing the model's internal state on the 40 prompts with and without two bans. It recovers 64–82% of the loss on Qwen-2.5-7B and 34–82% on the two new models. The same vector also recovers 65–92% on four constraints it was not built from (Table R3T2), and 61–85% on the 80 MT-Bench questions.
4. Prompt-level instructions. "Keep your answer equally thorough" removes between a third and a half of the loss (−27.2% → −14.3% and −24.8% → −15.7%, Table 29), and chain-of-thought prompting recovers 16–43% (Table 25). However, these instructions help only a user who already knows the problem exists. A user who meets the problem for the first time has no reason to add "keep your answer equally thorough" to a formatting request, so in normal use the loss remains.
The revised §7.3 names all four, and Appendices M, V and W give the new numbers above. The introduction no longer leaves solutions to future work and briefly names the four mitigations we tested.
Response to Reviewer PFKi (Part 1/5)
We thank the reviewer for the insightful comments. We address the concerns below. Unless mentioned otherwise, all new results use a pairwise GPT-4o-mini judge, sampling with T = 0.7 and top-p 0.9. Δ% is the relative change in judged comprehensiveness from the unconstrained answer to the constrained answer for the same question, and recovery is the share of this loss that an intervention removes. Table, section and appendix numbers (for example Table 24, §6.2 or App. H) refer to the submitted paper. The revised paper keeps these numbers and adds Appendices U–Z. Tables named R3T1, R3T2, ... are new results in this response. Each new table comes from its own run, with new samples or new judge calls. The same cell can therefore differ by a few points from the paper or from another new table.
Q1 / W1. How well does the steering intervention generalize to other model families and sizes, given that the most convincing result is concentrated at a single layer of Qwen2.5-7B?
The paper reports steering with the decoder mask on three model families. It recovers 64–82% on Qwen-2.5-7B and 36–46% on Llama-3.1-8B (Table 19). On Mistral-7B, whose collapse is the smallest of the three, it recovers about 25%, in line with the paper's finding of recovery following the collapse severity (Table 19). The paper also tests the direction on a ban it was not built from. We built it only from the bans on commas and on the word "the", and on Llama it recovers 46% of the loss under the semicolon ban.
The paper ran the layer sweep only on Qwen-2.5-7B. To test the generalization, we ran two new experiments:
1. A fourth family and a larger size. We used the method of the paper. The direction comes from the bans on commas and on the word "the". The dose f sets the length of the added steering vector to f times the average length of the hidden state at that layer (§6.3). We use the decoder mask. As a control, we also steer with a random direction of the same length, at the same layer and dose (the paper's placebo). We first ran a five-depth sweep on the comma ban and then ran the best layers at higher doses. Finally, we applied the same settings, unchanged, to the 80 MT-Bench questions (seed 0). We did not use these questions to build the direction or to choose the settings. Table R3T1 reports the result, using the six-criterion judge of Table 19.
R3T1
As Table R3T1 shows, steering recovers 34–82% on the paper's prompts and 47–79% on MT-Bench, while the random direction recovers between −12% and 7%. In the sweep, recovery peaks in the middle of the network, at layers 12–16 of 32 in OLMo and at layers 24–30 of 48 in Qwen-2.5-14B. At the earliest and latest depths tested, it falls to at most 6% in OLMo and at most 27% in Qwen-2.5-14B.
The comma collapses are deeper (−47.7% and −73.0%), and steering recovers 34–60% of them. Without steering, Qwen-2.5-14B's answers under the comma ban drop the spaces between words, with 1.4 words per answer on the 40 prompts. The model does the same with the standard Hugging Face generation code and its own chat template, so this is its own behavior. Steering restores normal text.
We also checked that steering keeps the constraint satisfied and the answers fluent:
These checks are in Appendix W of the revised paper. So steering at a mid-network layer works in a fourth family and at a larger size, and on prompts not used to choose the settings.