acceptodds
Under review as a conference paper at ICLR 2027

The Hidden Cost of Prompt Formatting: Measuring Representational Sensitivity in Language Models

Abstract

Prompt formatting is often treated as a superficial implementation detail, yet meaning-preserving format changes can alter model behavior. We ask whether such changes also leave a systematic footprint inside the model before decoding, and whether this internal movement is associated with output instability. We introduce an activation-level audit for prompt-format sensitivity using matched one-shot prompt pairs in which the task instruction, demonstration, input, and answer semantics are fixed while exactly one structural formatting property is perturbed. Across 53 Super-NaturalInstructions tasks and four open-weight 7B-scale models—Falcon, LLaMA-2, Mistral, and Qwen-2.5—we analyze 1,739,400 atomic interventions per model spanning separator, enumeration, spacing, and casing edits. We find a stable hierarchy of representational sensitivity: separator edits induce the largest hidden-state shifts, enumeration edits form a consistent second tier, and spacing and casing produce smaller effects. We further evaluate activation–output coupling over 5,573,004 eligible base–perturbation pairs spanning 168 task–model strata. Controlled activation-distance coefficients are positive in 158/168 strata, while 167/168 within-stratum Spearman correlations are positive, with median , showing that larger activation shifts are generally associated with larger changes in semantic-label output distributions. Finally, we introduce PFSS, a compact 25-term interpretable predictor of activation movement from prompt-format metadata alone, achieving task-holdout raw-scale scores of 0.831, 0.773, 0.852, and 0.821. These results establish prompt formatting as a measurable component of the representational input space and activation movement as an informative, though heterogeneous, intermediate signal of output instability.

Primary area:
interpretability and explainable AI
Keywords:
prompt sensitivity, prompt formatting, in-context learning, large language models, representation analysis, hidden states, activation analysis, interpretability, model robustness, prompt engineering, activation distance, feature attribution, elastic net, model auditing, evaluation methodology

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_EgnAOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The paper measures changes in the final-layer hidden state at the last prompt token under formatting perturbations. It reports that separator and enumeration edits produce larger activation distances, that activation distance correlates with changes in restricted next-token label distributions, and that formatting metadata predicts activation distance.

Critical strengths

The paper uses matched prompt pairs, multiple open-weight models, task-held-out evaluation, and several robustness analyses. The experimental pipeline is extensive and clearly documented.

Critical weaknesses
  1. The central activation measure is strongly confounded by tokenization and token identity. Formatting perturbations can change the final input token and its position, yet the paper compares final-layer states at that token. The reported separator hierarchy may therefore primarily reflect comparing representations of different terminal tokens rather than a meaningful prompt-level representational shift.

  2. The activation–output association is largely expected because the measured hidden state is directly projected into the next-token logits. The paper does not demonstrate that activation distance provides useful information beyond output-side quantities. Its incremental predictive value is very small: JSD MAE does not improve, while answer-flip AUC increases only from 0.8082 to 0.8128.

  3. PFSS generalizes across held-out tasks but not unseen formatting operations. Training and test sets share the same small perturbation vocabulary, making high R^2 compatible with memorizing transition-specific distance scales. The evaluation therefore does not establish general prediction of prompt sensitivity.

Questions for authors
  1. Do the results hold when base and perturbed prompts have identical terminal tokens, identical token counts, and aligned measurement positions?
  2. Does activation distance outperform logit distance, margin change, or output entropy for predicting answer instability?
  3. Can PFSS generalize to formatting transitions entirely excluded during training?
Reviewer_BSBjOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper asks whether small, meaning preserving formatting changes in a prompt also change the model's internal state, and if that internal change relates to changes in the output. The authors take 53 Super-NaturalInstructions tasks and four 7B open models (Falcon, LLaMA-2, Mistral, Qwen-2.5). They build pairs of one-shot prompts that differ in exactly one formatting choice: separator, enumeration style, spacing, or casing. For each pair they compute the Euclidean distance between the final-layer hidden states at the last input token. There are about 1.7M pairs per model. They report three findings: First, separator edits move the hidden state the most, enumeration edits come second, and spacing and casing move it the least. A larger hidden-state distance goes with a larger change in the label probabilities, measured by Jensen–Shannon divergence. The correlation is positive in 167 of 168 task–model groups, with a median Spearman ρ of 0.41. A 25-term linear model (PFSS), using only a description of the formatting edit, predicts the hidden-state distance with R² of about 0.77–0.85 on held-out tasks.

Critical strengths

1- The setup is clean and controlled. Each pair changes exactly one formatting property and keeps the instruction, demonstration, input and answer the same, so any difference can be traced to that one edit. The authors also handle multiple-choice labels carefully, so an enumeration edit doesn't change which answer is correct.

2- The study is large and the authors are careful about their limits. It covers four model families and millions of pairs, pools results across tasks with a random-effects analysis, and checks that the coupling isn't explained by changes in how the labels are tokenized. The authors also say clearly that hidden-state distance is a signal, not a causal explanation. The question itself matters for how we evaluate LLMs.

3- The authors release their analysis notebook, and the PFSS numbers in Table 10 match its outputs.

Critical weaknesses
  1. The main coupling result is close to true by construction. The hidden state the paper measures (final layer, last token) is exactly the vector the next-token logits are computed from, through the unembedding matrix. So if the label probabilities change, this vector must change. Finding that "a bigger hidden-state change goes with a bigger output change" is therefore expected and tells us little about the model. A correlation of only 0.41 mostly reflects how much of the movement lies in directions that don't affect the label logits. To learn something new, the paper would need, for example, intermediate layers, or a split of the movement into the part that affects the label logits and the part that doesn't. This matters because it is presented as the paper's main bridge between representations and outputs.

  2. The hierarchy result is confounded and not properly reported, and the PFSS analysis is weaker than described. Confound: Figure 1 shows the prompt ends with "Answer:", and a separator edit turns it into "Answer::". So for separator edits, the token whose hidden state is measured is itself the edited token. Casing and spacing edits change tokens earlier in the prompt. Separator edits also apply to every field, so they change many more tokens than casing does. The ordering "separator > enumeration > spacing/casing" could just reflect how close the edit is to the measured token and how many tokens it changes, not anything about structure. Not reported: No table or figure in the paper gives the per-family average distances defined in Eq. 5. I found them in the released notebook. The means do follow the claimed order, but separator distances are very skewed: the mean is much larger than the median in every model (e.g., Mistral 231 vs. 112, Qwen 171 vs. 101). For Mistral, the median enumeration distance (126) is higher than the median separator distance (112), so the ordering flips. The cosine distances are also split in two for LLaMA-2, Mistral and Qwen: the median is about 0.02, but the top 10% are 0.54–0.68. Both patterns fit the idea that a few separator edits that change the last input token drive the result. PFSS: The input features basically encode which edit was made, and every edit is applied to every example, so PFSS works like a lookup table of averages per edit type. Plain one-hot Ridge beats PFSS on task-holdout raw-scale R² for all four models (Table 10). The notebook also shows several things the paper doesn't state: PFSS and all baselines are fit on a random 50k-row sample per model, not the full 1.7M rows. The task-holdout result is a single random 80/20 split of tasks, with no repeats and no spread reported. The predictors include prompt hashes and the example index, even though the paper says identifiers are excluded. In the saved outputs, the refit has a condition number of about 10³³ and falls back to Ridge. This is because several of the "25 terms" are the same one-hot column under different names (e.g., "separator ': ' → '::'", "to value '::'", "perturbed separator '::'"), so the stable terms in Table 1 overlap. The suggested use of hidden-state distance as a "cheap pre-screen" before output testing also doesn't hold up. Computing the hidden state already gives you the logits, so it saves nothing. Adding the distance improves output prediction only slightly (R² 0.099 → 0.109, AUC 0.808 → 0.813).

Questions for authors

1- Please report per-family and per-parameter mean and median distances (Eq. 5) for each model, with confidence intervals, and the distribution of distances for each separator value. 2- For separator edits, the last input token, where you read the hidden state, is itself the edited token. Does the hierarchy still hold if you read the hidden state at a token that is identical in both prompts, or if you exclude edits that change the final "Answer" separator? Does it hold after controlling for the number of changed tokens? 3- The final-layer last-token state maps linearly to the logits. Can you show the coupling at intermediate layers, or separate the movement into the label-relevant subspace and its complement? Otherwise, what does the coupling result tell us beyond this direct link? 4- Which exact checkpoints did you use (base or instruct, and which Qwen-2.5 and Mistral versions)? Is the hidden state taken before or after the final normalization layer? This changes what the Euclidean distance means. 5- Please confirm the sample size and number of task-holdout splits behind Table 10. Please rerun with prompt hashes and the example index removed from the predictors, and report the spread over several task splits. 6- Why use PFSS rather than one-hot Ridge, which scores higher on task-holdout? Many selected terms are identical columns. How many distinct features does PFSS actually use? How does it do when whole formatting values are held out, instead of tasks? 7-The format grammar and task split come from Sclar et al. (2024). What is new relative to that work, including any analysis of internal representations it contains? 8- Where in the repository are the prompt-construction, hidden-state-extraction and activation–output coupling code? I could not find them in the analysis notebook.

Reviewer_rH1VOpenReview ↗
Rating
2
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies whether meaning-preserving prompt-format edits (separator, enumeration, spacing, casing) change LLMs’ internal representations. Using matched one-shot prompt pairs from 53 Super-NaturalInstructions tasks and four 7B-scale models (Falcon, LLaMA-2, Mistral, Qwen-2.5), it measures the Euclidean distance between final-layer, last-token hidden states. It reports three findings: a stable sensitivity hierarchy (separator > enumeration > spacing/casing), a positive association between activation distance and semantic-label output divergence, and a 25-term metadata-only predictor (PFSS) of activation distance.

Critical strengths

The study covers about 1.74M atomic interventions per model across 53 tasks, four model families, and four perturbation families. Every pair differs in exactly one formatting property, with task content held fixed, which is a well-controlled design.

Critical weaknesses
  1. All four models are 7B-scale, 2023–2024-era models (Falcon-7B, LLaMA-2-7B, Mistral-7B, Qwen-2.5-7B), evaluated in a one-shot label-selection setting. Current LLMs are much larger, are heavily instruction- and preference-tuned on diverse input formats, and are typically used through chat templates and with extended reasoning. The paper provides no evidence that the sensitivity hierarchy or the activation–output coupling holds for such models, and no analysis across model scale or post-training stage. Without at least one current model family evaluated at multiple scales, it is unclear whether the results say anything about the models the community actually uses. The practical significance of the problem as studied is therefore limited.

  2. The paper suggests using activation distance as a “cheap pre-screen” to prioritize which formatting variants deserve full output-level testing (Sections 3.4 and 4). However, the hidden state and the next-token logits come from the same forward pass, and the paper itself records both. The output-level quantities that activation distance is meant to flag (semantic-label JSD and answer flips) are therefore available at exactly the same cost. Moreover, the incremental predictive value of activation distance over formatting metadata is very small: task-held-out JSD R² rises only from 0.0989 to 0.1085 (MAE unchanged), and answer-flip AUC from 0.8082 to 0.8128. As a result, the paper does not demonstrate a setting in which the activation-level audit offers information or efficiency beyond directly measuring output changes.

Questions for authors

Do the hierarchy and the coupling results hold for current instruction-tuned and reasoning models? Is there any concrete scenario in which activation distance is available but the output distribution is not, or is more expensive to obtain?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.