- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper investigates how large language models organize task-relevant information in their internal representations. It proposes a metric k*, to measure the minimum number of features required to achieve performance close to that of a dense linear probe. Through cross-model comparisons, instruction tuning, controlled synthetic experiments, and downstream applications, the authors analyze the consistency of readout sparsity across models, the effects of training on readout sparsity, and its relationship with label requirements and robustness to model pruning.
Critical strengths
- This paper distinguishes between linear readability and readout sparsity of task-relevant information. Compared with traditional dense linear probes that focus solely on predictive performance, k* provides an additional perspective for characterizing how task information is organized within language models.
- The experiments are relatively comprehensive. The authors conduct evaluations across models from multiple labs and a variety of tasks, and further investigate their findings through SAE feature matching, controlled synthetic experiments, and behavioral interventions.
Critical weaknesses
-
The authors use test-set's AUROC to determine whether different feature subsets meet the performance threshold and subsequently select the minimum number of features. Meanwhile, the reference performance of the dense probe is obtained by taking the maximum test AUROC across multiple layers and random seeds. This means that the test set is used not only for final evaluation but also for selecting k*, potentially introducing selection bias. Although the appendix provides statistical analyses, these mainly focus on the estimation error of the dense-probe ceiling and do not fully address how this selection process affects the final k* and the related conclusions.
-
The random rotation and paraphrase experiments show that k* can change substantially even when dense-probe performance remains relatively stable. Although the authors acknowledge this limitation, it raises concerns about the interpretation of k*: to what extent does it reflect the internal organization of task-relevant information, rather than sparse decodability under a particular feature basis and input distribution?
Questions for authors
- Could the authors use an independent validation set to select feature subsets and determine k*, and then evaluate their performance on a separate test set that was not involved in the selection process? Under this setting, would the main findings regarding cross-model task ranking correlations and changes in k* after instruction tuning still hold?
- Could the authors conduct a controlled fine-tuning experiment on pretrained language models using real-world semantic tasks? For example, supervision could be provided for only a subset of classification tasks, while related tasks serve as controls.
2 comments
Response to Q2
Q2 Could the authors conduct a controlled fine-tuning experiment on pretrained language models using real-world semantic tasks? For example, supervision could be provided for only a subset of classification tasks, while related tasks serve as controls.
Q2. Thank you for this insightful suggestion and for recognizing the comprehensiveness of our experiments! The answer is yes. Following your exact design, we ran controlled single-task fine-tuning on two pretrained models under our main five-seed protocol (three fine-tuning seeds per setting). In each run, we supervise the labels of one task and evaluate the supervised task, related unsupervised tasks, and unrelated tasks. On Gemma-2-9B, we fine-tune with LoRA on harmfulness or sentiment and evaluate 12 tasks. On Gemma-3-1B, we run full-parameter fine-tuning on each of six tasks in turn.
-
Supervised tasks concentrate. On Gemma-2-9B, harmfulness falls from 341 to 16 coordinates and sentiment falls from 72 to 16 (median reduction factor 8.4 across supervised runs), while dense-probe AUROC changes by less than 0.01. On Gemma-3-1B, harmfulness falls from 234 to 16, sentiment from 128 to 16, and jailbreak detection from 16 to 2, while tasks already at 1 to 4 coordinates stay unchanged (banking intent moves from 34 to 32).
-
Unrelated tasks stay unchanged. Unrelated unsupervised tasks change by a median factor of 1.0 on both Gemma-2-9B and Gemma-3-1B (across all 78 observations on 1B).
-
Related tasks split by shared concept versus shared text. Unsupervised tasks that share a similar semantic concept on different texts also concentrate (median factor 4.0 on Gemma-2-9B), as ToxicChat toxicity falls from 117 to 49 under harmfulness supervision and IMDB and Yelp sentiment fall from 16 and 8 to 2 under sentiment supervision. By contrast, an unsupervised task with a different label on the exact same prompts does not concentrate, as jailbreak detection under harmfulness supervision rises from 8 to 32 on Gemma-2-9B and from 16 to 32 on Gemma-3-1B.Readout concentration therefore tracks the supervised concept and does not come from exposure to the input texts alone, matching our synthetic control that shares half of a target's input factors yet still needs 60.5 coordinates.
Thank you again for proposing this thoughtful experiment! It strengthens Section 4 by directly connecting our synthetic and instruction-tuning findings, and we will include all of these results in the revised version.
Response to W1 and Q1
W2 The random rotation and paraphrase experiments show that k∗ can change substantially even when dense-probe performance remains relatively stable. Although the authors acknowledge this limitation, it raises concerns about the interpretation of k∗: to what extent does it reflect the internal organization of task-relevant information, rather than sparse decodability under a particular feature basis and input distribution?
W2. Thank you for raising this insightful point and for recognizing that k* adds a perspective beyond dense probes. A single k* value measures sparse decodability under a given coordinate basis and input text pool, yet comparisons of k* across models and training stages reflect internal model organization in both respects.
Feature basis. Applying a random orthogonal rotation preserves all linear information and leaves median dense-probe AUROC unchanged (+0.000), yet raises k* by a median factor of 1.79. Because rotation increases k*, task readouts align partially with the model's native coordinates, a property that rotation-invariant dense probes cannot detect. Moreover, our conclusions hold even when we remove dependence on the native basis. Remeasuring all ten base models under our five-seed protocol in random orthogonal bases and in a rotation-invariant PCA basis preserves cross-lab task-order agreement, with median rank correlations of 0.929 (native), 0.963 (rotated), and 0.947 (PCA). Supervision also concentrates readouts across all three bases, as instruction tuning on Gemma-2-9B lowers harmfulness k* by factors of 7.3 (native), 2.6 (rotated), and 8.0 (PCA).
Input distribution. Paraphrasing shifts absolute counts by factors of 1.5 to 10.8 while leaving the task order unchanged. More importantly, all of our model and training comparisons hold the text pool fixed, so input distribution alone does not explain the differences. On identical texts, fine-tuning Gemma-2-9B on harmfulness labels lowers k* from 341 to 16, and controlling for bag-of-words probe scores on the same texts leaves the cross-lab task-order correlation between 0.86 and 0.92.
Thank you again for this insightful point! And we will clarify this distinction between absolute counts and comparative structure in Section 2 and include the ten-model basis experiments in the revision.
Q1 Could the authors use an independent validation set to select feature subsets and determine k∗, and then evaluate their performance on a separate test set that was not involved in the selection process? Under this setting, would the main findings regarding cross-model task ranking correlations and changes in k∗ after instruction tuning still hold?
Q1 & W1. Thank you for this constructive question and for proposing this concrete experiment! The answer to both questions is yes. We ran the independent validation experiment across all 19 models and 14 tasks (266 cells). In our protocol, each seed fits probes on an 85% training subsample and leaves the remaining 15% completely untouched by standardization, feature ranking, regularization tuning, and fitting. Using this untouched 15% as an independent validation set, we select the dense-probe reference AUROC, threshold, confirming layers, and k*, and then evaluate the chosen sparse probe once on the original test set. Because the probe fits are identical to the submission, any difference comes purely from separating selection from evaluation.
Unbiased test-set recovery. On the untouched test set, the validation-selected probes recover a median of 96.9% of the dense probe's above-chance AUROC against the 95% target, with 94.0% of all instances (96.2% on readable cells with dense-probe AUROC >= 0.90) recovering at least 90%.
Stability of k*. Across finite cells, the ratio of validation-selected k* to the submitted k* has a median of 1.00 (interquartile range 0.94 to 1.00), and 84.4% of cells stay within one factor-of-2 grid step.
Both main findings hold. Median cross-lab task-order correlation is 0.924, compared with 0.929 in the submission. Instruction tuning lowers harmfulness k* in all 9 model pairs (for example, 344 to 64 on Gemma-2-9B vs. 341 to 47 in the submission). The median reduction factor across the 9 pairs is 2.8 vs. 4.6 in the submission, because the smaller 15% validation split increases seed-to-seed variance on some base models across the factor-of-2 search grid.
Thank you again for this constructive point and we will report this in the revised version!