acceptodds
Withdrawn from ICLR 2027

Latent Structure, Failed Selection: Representation–Use Gaps in Language Models

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Performance gaps across language models are commonly attributed to scale, data, and training, yet benchmark accuracy alone cannot distinguish what models represent from how effectively they use those representations. Across 18 base models from six model families, we examined internal structure and forced-choice recall for 5 relation types: ordinal relations, part–whole, category–property, abstract–concept, and attribute–object. Ordinal structure was highly recoverable across model sizes, whereas forced-choice recall increased with size. Erasing ordinal directions selectively impaired matching judgments for temperature, size, quality, and anger after comparison with overlap matched random controls; likelihood showed no detectable effect because baseline recall was near chance in most models. Mapped hidden-state transfer of six novel words spanning temperature, size, and quality outperformed sham injections in 76.8% of 306 ordered sender–receiver pairs. To understand why small models fail, we decomposed answer preferences into question-responsive evidence and question-invariant preference, whose magnitude we call stubbornness. In small models, stubbornness exceeded question-responsive evidence, while signed question-invariant preference was more strongly aligned with lexical priors. To test how question-responsive information reaches the answer, we blocked direct attention from the answer position to the candidate words. This reduced evidence, increased stubbornness, and strengthened its alignment with these baseline word preferences. Because stubbornness is invariant to question polarity, symmetrised scoring cancels this preference without training. Applying the same invariance principle to answer positions improved 126 of 162 model benchmark combinations across nine multiple-choice benchmarks, with gains of up to 23.7 percentage points. The four non-ordinal relation types also showed strong internal structure but only small gains from symmetrisation. Together, these results show that models can represent task-relevant relational structure that fails to control their answers, and that some performance gains with scale arise from using existing representations more effectively rather than from acquiring those representations alone.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Large Language Models, Internal Representations, Knowledge Recall, Zero-shot Debiasing

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_jHsoOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper studies 18 pretrained language models from six families to examine whether poor task performance reflects missing internal representations or failure to use representations that are already present. It argues that smaller models often contain useful relational structure but fail to translate it reliably into answers.

Critical strengths
  • The topic is interesting and timely, and the paper evaluates a relatively broad set of models rather than relying on one model family.
  • The attempt to combine representation analysis with causal interventions is also potentially valuable.
Critical weaknesses
  • The methodological presentation is overly complicated and difficult to follow. The paper combines several quite different analyses—ordinal probing, forced-choice recall, representation erasure, cross-model hidden-state transfer, attention blocking, lexical-prior analysis, “stubbornness”, and symmetrised scoring—within one narrative. As a result, the central scientific message becomes hard to see from the breadth of analyses.

  • The study does not convincingly identify what factors make one language model perform better than another. The paper repeatedly relates behaviour to model size, but model family, architecture, training procedure, and parameter count are also important. This substantially limits the broader conclusions regarding why some models make better use of their internal representations.

Questions for authors
  • Can the authors provide a clearer causal framework showing how the different analyses, e.g., probing, erasure, transfer, attention intervention, lexical priors, and symmetrisation, fit together and which experiments provide the primary evidence for the claimed representation gap?

  • Given that model size, architecture, family, and training are highly relevant to this topic, what evidence supports attributing the observed performance differences specifically to better “use” of representations?

1 comment

Thank you for the review.

  • Q1. How do the analyses fit together, and which provide the primary evidence?

Each analysis answers a question raised by the one before it.

  1. Do small models lack the knowledge? We measured ordinal structure and recall on the same items. Structure is 0.83–0.94 in all 18 models and does not change with size, while recall rises from 0.55 to 0.78. Small models have the structure but answer poorly.
  2. Is that structure actually used, or only decodable? We erased the ordinal direction. The matching judgment is damaged more than under matched random directions (17/18 models), so the structure contributes to the answer. Transfer between models is a second.
  3. Then why do small models fail? We split each answer into evidence that follows the question (E) and a preference that ignores it (S), whose magnitude we call stubbornness. In small models stubbornness exceeds the evidence. 94% of failures pick the same word whether the question asks for the higher or the lower end.
  4. Where does the preference come from, and how does the evidence arrive? The preference follows the model's own lexical prior for the candidate words. Blocking attention from the answer position to the candidates lowers the evidence and raises stubbornness, so the evidence travels through that attention.
  5. Can the preference be removed? It does not depend on the question, so scoring both polarities cancels it. Recall rises from 0.686 to 0.930 in all 18 models without training.
  6. How far does this extend? The same idea applied to answer positions improves multiple-choice benchmarks. The four non-ordinal relations show little gain and mark the limit.

Steps 1 to 3 are the primary evidence for the gap. Steps 4 and 5 explain and correct it. Step 6 tests its scope.

  • Q2. What supports attributing the differences to "use" rather than size, architecture, family or training?

Two results support it.

  • Within each model. Without changing the model, cancelling the question-invariant preference at scoring raises accuracy from 0.686 to 0.930 in all 18 models. The model already had what it needed to answer correctly, so the failure is in use, and this does not involve comparing different models.
  • Across models. With family held fixed, recall rises with size (+0.099 per tenfold increase in parameters, p = 0.0002) and ordinal structure does not (+0.002, p = 0.86). The same holds for dense models alone and for models with full attention in every layer.

We cannot say which training factor produces better use, because data, recipe and architecture cannot be varied independently across public checkpoints.

Reviewer_a742OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

The paper compares relational representations and forced-choice recall across 18 base models. Ordinal structure remains recoverable when recall is weak, and controlled erasure supports its task-level relevance. The authors decompose answer margins into question-responsive and invariant components, test attention interventions, and improve performance through symmetrised scoring on nine benchmarks. The evidence is extensive, but the explanation of how represented knowledge fails to determine particular answers remains incomplete.

Critical strengths
  1. Substantial intervention and behavioral evidence. Matched erasure controls support selective effects on four ordinal scales, and symmetrised scoring significantly improves 126 of 162 model–benchmark combinations (§§4.2, 4.5).
  2. Useful negative results. Attention amplification, ordinal-signal amplification and fixed bias-subspace removal do not consistently improve recall (§4.4; Appendix C). These findings constrain simple repair hypotheses.
Critical weaknesses
  1. The explanatory advance is limited. The distinction between represented knowledge and predictions has close precedents (Park et al.). Here, the margin decomposition describes answer preferences, while erasure and attention blocking establish task-level relevance. They do not yet explain the representation's role in the same failed judgments claimed in §2.2. Scoring gains support a correction procedure more directly than that mechanism.
  2. Key supporting results are missing. The paper describes selection-matched rank-shuffle and frame split-half checks but does not report their outcomes (Appendix A.4). Transfer results compare mapped with random injections, omitting the evaluated no-injection and verbal-definition conditions (Appendix A.7). These omissions limit assessment of representation stability and usable transfer; they do not invalidate the benchmark gains.
Questions for authors
  1. What are the numerical rank-shuffle and split-half results, including the best-layer selection used for the null?
  2. How does mapped transfer compare with no injection and verbal definitions, separately for each scale?
  3. Which existing analysis links the recovered representation to individual failed judgments, beyond aggregate task-level erasure effects?
1 comment

Thank you for the precise questions. We are withdrawing the submission, but we report the results you asked for briefly here.

  • Q1. Rank-shuffle and frame split-half results.

We shuffled the ranks of the words within each scale 1,000 times and recomputed the leave-one-out agreement each time, taking the maximum over layers exactly as for the observed value. The five-scale mean used in the paper is 0.893, against a null mean of 0.616 and a null 95th percentile of 0.770, and it exceeds the null in all 18 models (p ≤ 0.002). This result is carried by quality (significant in 17/18 models) and size (13/18). Temperature, anger and likelihood are above the null mean in all 18 models, but none is significant in any single model, and combining the three does not reach significance either.

In addition, we split the 32 frames into two random halves 30 times, fitted the direction on one half and scored the other. Mean agreement is 0.883, and it varies little across splits (std: 0.012–0.031). The measure does not depend on particular frames.

  • Q2. Mapped transfer against no injection and verbal definition, per scale.

Mean accuracy over the 306 ordered sender–receiver pairs:

scaleverbal definitionmappedno injectionshammapped − no injection (95% CI)
quality0.8600.5540.5130.517+0.041 (+0.014, +0.070)
size0.8580.5790.5460.494+0.032 (−0.006, +0.070)
temperature0.7770.5660.5290.462+0.037 (−0.008, +0.084)
pooled0.8310.5660.5290.491+0.037 (+0.015, +0.060)

No injection, which is the right baseline, mapped injection is higher in 72% of pairs, and the mean gain is positive for 17 of 18 senders (p = 0.00004) and 15 of 18 receivers (p = 0.002). The effect is small (+0.037).

The verbal definition is an upper bound, not a competing method: the experiment asks whether one model's hidden state carries ordinal information that another model can use, and the answer is yes, weakly.

  • Q3. Which analysis links the representation to individual failed judgments?

We ran the item-level analysis on all 18 models and 1,164 probes. For each probe we read the hidden states of the two candidate words inside the prompt and projected them onto an ordinal direction fitted without those two words.

On probes the model answers wrongly, this reader still orders the pair correctly in 70.4% of cases (above chance in 18/18 models, p = 4×10⁻⁶), against 75.1% on correctly answered probes.

94% of failed items are same-answer failures: the model picks the same word whether asked for the higher or the lower end. Independent errors would give 80%.

Item by item, the reader's margin correlates with question-responsive evidence E (Spearman +0.17, positive in 17/18 models) and not with stubbornness |S| (−0.04).

Reviewer_NLg2OpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The core contribution is a way to measure what a model represents and how well it uses that representation on the same stimuli, plus a decomposition of forced-choice answers into question-responsive evidence (E) and question-invariant preference (S) that explains why small models fail even when the relation is encoded. The study covers 18 base models from six families (1B to 35B), five ordinal scales (temperature, size, quality, likelihood, anger; 33 words, 97 pairs), and four non-ordinal relation types as a comparison.

The main claims are four.

  • First, ordinal structure is present at all sizes (leave-one-out agreement 0.833 to 0.943, uncorrelated with size) while forced-choice recall rises with size (0.545 to 0.784, r = +0.78), so some scaling gains are better use of representations that are already there.
  • Second, the ordinal directions causally matter: erasing them hurts matching-scale judgments relative to overlap-matched random directions for four of five scales, and mapped hidden states for six novel words transfer across models in 76.8% of 306 ordered pairs.
  • Third, in small models the magnitude of S ("stubbornness") exceeds E and S tracks lexical priors; blocking attention from the answer position to the candidate words lowers E and raises stubbornness, so direct attention to the candidates is what carries the question-responsive signal.
  • Fourth, scoring symmetrically over question polarity removes S without training and lifts ordinal recall from 0.665 to 0.891; applying the same idea to answer positions improves 126 of 162 model-by-benchmark cells on nine multiple-choice benchmarks, by up to 23.7 pp.

The conclusion is that accuracy reflects not only what models represent but how effectively those representations control output. The practical reading is that a wrong answer can be a selection failure rather than missing knowledge, and that this can be diagnosed and partly corrected at inference time.

Critical strengths

The intervention design is careful, and it is what makes the causal claim credible. Random controls are not just random: they are matched for projection onto the span of the gradable-word embeddings (Equations 15 and 16). The full 5-by-5 design lets the authors show diagonal versus off-diagonal specificity (+0.081 versus -0.011, in 17 of 18 models). The cross-model transfer (76.8% of pairs beating sham by more than 0.02) is a nice secondary piece of evidence.

The E/S decomposition and the attention-blocking result are the parts I would point other people to. They give a clean operational handle on the "knows but cannot say" problem and a mechanism for one common failure. For a practitioner, the E/S diagnostic can tell you whether a small model's failure is missing knowledge or a selection bug, and the symmetrized scoring is a free accuracy improvement for evaluation pipelines on small models.

The reporting is honest. The likelihood null (a floor effect), the lack of specificity in Gemma-4-31B, the three additional interventions that did not help, and the note that the analyses were "pre-specified, not preregistered" are all stated plainly.

Critical weaknesses

The most prominent result, the benchmark gains, is a re-application of known permutation debiasing. Cycling options through every position and summing label log-probabilities (Equation 31) is essentially the permutation averaging of Zheng et al. (2024), and contextual calibration (Zhao et al. 2021) addresses the same bias. The authors acknowledge in Section 5 and Appendix D.2 that this does not test the E/S decomposition. Without a comparison to those methods at matched forward-pass budget, I cannot tell what is new, and the abstract leads with this number.

There is also a format issue. HellaSwag, WinoGrande, and BoolQ were converted to lettered multiple choice, and the HellaSwag baseline of 0.459 is far below the usual 0.75 to 0.85. A good part of the +9.08 pp may be undoing a position bias that the conversion itself introduced.

The headline contrast may be exaggerated. Best-layer ordinal agreement is 0.83 to 0.94, near ceiling, and even the smallest model (OLMo2-1B) is at 0.867. Larger models have little room to show improvement. Best-layer selection is an upper-bound estimator, which biases the representation measure upward but not the recall measure, so comparing their size correlations with Steiger's test needs care. Size also covaries with family, architecture, and training data, as the authors say. Within-family analyses (OLMo-2, Falcon-3, Gemma-4, and Qwen3.5 each have four sizes) would address much of this.

The erasure effects are small and heterogeneous: +0.07 to +0.12 excess damage. Gemma-4 controls cause as much damage as targeted erasure, and Qwen3.5-9B shows broad rather than specific damage (Appendix B.1). The authors call it a group-level pattern, which is fair, but "selectively impaired" in the abstract is stronger than that. The mechanism rests on 33 words and 97 pairs, and the 2 to 3 pp gains on non-ordinal relations show how far it extends.

Questions for authors
  1. How does the multiple-choice symmetrization compare to Zheng et al. (2024) and contextual calibration at the same forward-pass budget, including performance versus number of permutations? Do the gains on HellaSwag, WinoGrande, and BoolQ hold under the standard continuation-likelihood format?

  2. Within each family (OLMo-2, Falcon-3, Gemma-4, Qwen3.5), what are the representation-size and recall-size correlations? Does the size independence of ordinal structure survive a harder representation measure (continuous regression R-squared, pairwise distance recovery) that is not near ceiling?

  3. Are the symmetrization gains larger on items where structure is well recovered but baseline recall is low? Does per-item position or lexical preference decrease where answers are recovered on the general benchmarks? Do you have a hypothesis for why erasure has no specific effect in Gemma-4, for example sliding-window attention interacting with erasure at all positions through 60% of depth?

1 comment

Thank you for a careful review.

  • Q1. Comparison at matched budget, and the continuation format.

All methods were computed from the same stored forward passes (18 models). Mean accuracy on the six natively lettered benchmarks:

methodforward passes per itemaccuracy
baseline10.672
contextual calibration (Zhao et al. 2021)~10.667
PriDe (Zheng et al. 2024)~1.1–1.40.679
two shifts20.704
symmetrisation (all shifts)~50.721

Single-pass debiasing barely helps on these models; contextual calibration improves 2 of 18. The reason is that position preference is mostly item-specific: only a median 29% is shared across a benchmark, so one estimated prior cannot remove it. Two shifts already give 65% of the full gain and beat PriDe in 18/18 models. While the computational cose is increased, there are performance benefits.

benchmarkcontinuationletteredlettered + symmetrised
HellaSwag0.7570.4590.550
WinoGrande0.7110.6110.619
BoolQ0.7880.7320.771

Symmetrisation recovers only part of what the lettered conversion lost. The six benchmarks are not affected.

  • Q2. Within-family correlations, and harder representation measures.
familyrepresentation–size rrecall–size r
Falcon-3+0.92+0.74
Gemma-4−0.69+0.83
OLMo-2−0.06+0.95
Qwen3.5+0.58+0.89

Recall rises with size in all 4 families. The representation measure has no consistent direction and moves by at most 0.06 within a family.

The size independence holds under harder measures:

measureranger with size
continuous regression R² (held-out)0.64–0.87−0.29
pairwise distance recovery0.43–0.64−0.09
fixed depth (no best-layer selection)–−0.23 to −0.46

In all three cases the difference from the recall–size correlation remains significant (Steiger p ≤ 0.002).

  • Q3. Item-level questions.

Larger gains where structure is recovered but recall is low?: The effect is in that direction but weak. In failed items, symmetrisation recovers 71% when the internal representation orders the pair correctly and 66% when it does not (p = 0.07).

Position and lexical preference on recovered benchmark items?: Recovered items are those with stronger position preference at baseline (17/18 models). Across thirds of position preference, the share of errors recovered rises from 23% to 31% to 39%. Because symmetrisation cancels this preference by construction, this is a consistency check, not independent evidence. We did not measure lexical preference on the benchmarks.

Gemma-4?: The sliding-window hypothesis was not supported. In Gemma-4, erasing any single random direction does as much damage as the targeted erasure (0.203, against 0.004 in the other models), and erasing only in a narrow band (25–45% of depth) does the same (0.202 vs 0.200 for the full window). We have not found the cause.

Thank you again. The paper's limits are clearer to us because of your review.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.