acceptodds
Under review as a conference paper at ICLR 2027

How Language Models Organize Task Information

Abstract

Dense linear probes assess task readability from language model (LM) representations, but do not directly reveal how many features suffice for prediction. We measure readout sparsity with , the smallest feature count found by a fixed search protocol that recovers 95% of dense-probe AUROC above chance. We evaluate 19 LMs from five labs on 14 core tasks. Ten base models give the nine reliably readable tasks similar feature counts and nearly the same order. This agreement extends to the selected features themselves, which on three tasks show cross-model response correspondence beyond class-mean differences and a label-correlation-matched null. Instruction-tuned deployment selectively reduces , with both weights and input format contributing. In controlled synthetic experiments across 20 models, successfully learned targets admit single-feature readouts, whereas readable controls require tens of features. Reference-model is associated with label requirements across eight students, following an empirical scaling of . After continual learning, saved heads can fail while linear readability remains high, and recovery requires fewer labels than initial task learning under the tested procedures. Pruning reduces linear readability, with wider pre-pruning readouts generally associated with larger losses. These findings distinguish task readability from readout sparsity and establish their complementary value for predicting learning requirements and diagnosing LM changes: [k\*](https://anonymous.4open.science/r/taskprobe_ICLR_anonymous-8368/)

Primary area:
interpretability and explainable AI
Keywords:
interpretability, probing, large language models, representation sparsity, knowledge distillation, continual learning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_cJpeOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates how large language models organize task-relevant information in their internal representations. It proposes a metric k*, to measure the minimum number of features required to achieve performance close to that of a dense linear probe. Through cross-model comparisons, instruction tuning, controlled synthetic experiments, and downstream applications, the authors analyze the consistency of readout sparsity across models, the effects of training on readout sparsity, and its relationship with label requirements and robustness to model pruning.

Critical strengths
  1. This paper distinguishes between linear readability and readout sparsity of task-relevant information. Compared with traditional dense linear probes that focus solely on predictive performance, k* provides an additional perspective for characterizing how task information is organized within language models.
  2. The experiments are relatively comprehensive. The authors conduct evaluations across models from multiple labs and a variety of tasks, and further investigate their findings through SAE feature matching, controlled synthetic experiments, and behavioral interventions.
Critical weaknesses
  1. The authors use test-set's AUROC to determine whether different feature subsets meet the performance threshold and subsequently select the minimum number of features. Meanwhile, the reference performance of the dense probe is obtained by taking the maximum test AUROC across multiple layers and random seeds. This means that the test set is used not only for final evaluation but also for selecting k*, potentially introducing selection bias. Although the appendix provides statistical analyses, these mainly focus on the estimation error of the dense-probe ceiling and do not fully address how this selection process affects the final k* and the related conclusions.

  2. The random rotation and paraphrase experiments show that k* can change substantially even when dense-probe performance remains relatively stable. Although the authors acknowledge this limitation, it raises concerns about the interpretation of k*: to what extent does it reflect the internal organization of task-relevant information, rather than sparse decodability under a particular feature basis and input distribution?

Questions for authors
  1. Could the authors use an independent validation set to select feature subsets and determine k*, and then evaluate their performance on a separate test set that was not involved in the selection process? Under this setting, would the main findings regarding cross-model task ranking correlations and changes in k* after instruction tuning still hold?
  2. Could the authors conduct a controlled fine-tuning experiment on pretrained language models using real-world semantic tasks? For example, supervision could be provided for only a subset of classification tasks, while related tasks serve as controls.
2 comments
AuthorsOpenReview ↗

Response to W1 and Q1

W2 The random rotation and paraphrase experiments show that k∗k^{\ast} can change substantially even when dense-probe performance remains relatively stable. Although the authors acknowledge this limitation, it raises concerns about the interpretation of k∗k^{\ast}: to what extent does it reflect the internal organization of task-relevant information, rather than sparse decodability under a particular feature basis and input distribution?

W2. Thank you for raising this insightful point and for recognizing that k* adds a perspective beyond dense probes. A single k* value measures sparse decodability under a given coordinate basis and input text pool, yet comparisons of k* across models and training stages reflect internal model organization in both respects.

  1. Feature basis. Applying a random orthogonal rotation preserves all linear information and leaves median dense-probe AUROC unchanged (+0.000), yet raises k* by a median factor of 1.79. Because rotation increases k*, task readouts align partially with the model's native coordinates, a property that rotation-invariant dense probes cannot detect. Moreover, our conclusions hold even when we remove dependence on the native basis. Remeasuring all ten base models under our five-seed protocol in random orthogonal bases and in a rotation-invariant PCA basis preserves cross-lab task-order agreement, with median rank correlations of 0.929 (native), 0.963 (rotated), and 0.947 (PCA). Supervision also concentrates readouts across all three bases, as instruction tuning on Gemma-2-9B lowers harmfulness k* by factors of 7.3 (native), 2.6 (rotated), and 8.0 (PCA).

  2. Input distribution. Paraphrasing shifts absolute counts by factors of 1.5 to 10.8 while leaving the task order unchanged. More importantly, all of our model and training comparisons hold the text pool fixed, so input distribution alone does not explain the differences. On identical texts, fine-tuning Gemma-2-9B on harmfulness labels lowers k* from 341 to 16, and controlling for bag-of-words probe scores on the same texts leaves the cross-lab task-order correlation between 0.86 and 0.92.

Thank you again for this insightful point! And we will clarify this distinction between absolute counts and comparative structure in Section 2 and include the ten-model basis experiments in the revision.

Q1 Could the authors use an independent validation set to select feature subsets and determine k∗k^{\ast}, and then evaluate their performance on a separate test set that was not involved in the selection process? Under this setting, would the main findings regarding cross-model task ranking correlations and changes in k∗k^{\ast} after instruction tuning still hold?

Q1 & W1. Thank you for this constructive question and for proposing this concrete experiment! The answer to both questions is yes. We ran the independent validation experiment across all 19 models and 14 tasks (266 cells). In our protocol, each seed fits probes on an 85% training subsample and leaves the remaining 15% completely untouched by standardization, feature ranking, regularization tuning, and fitting. Using this untouched 15% as an independent validation set, we select the dense-probe reference AUROC, threshold, confirming layers, and k*, and then evaluate the chosen sparse probe once on the original test set. Because the probe fits are identical to the submission, any difference comes purely from separating selection from evaluation.

  1. Unbiased test-set recovery. On the untouched test set, the validation-selected probes recover a median of 96.9% of the dense probe's above-chance AUROC against the 95% target, with 94.0% of all instances (96.2% on readable cells with dense-probe AUROC >= 0.90) recovering at least 90%.

  2. Stability of k*. Across finite cells, the ratio of validation-selected k* to the submitted k* has a median of 1.00 (interquartile range 0.94 to 1.00), and 84.4% of cells stay within one factor-of-2 grid step.

  3. Both main findings hold. Median cross-lab task-order correlation is 0.924, compared with 0.929 in the submission. Instruction tuning lowers harmfulness k* in all 9 model pairs (for example, 344 to 64 on Gemma-2-9B vs. 341 to 47 in the submission). The median reduction factor across the 9 pairs is 2.8 vs. 4.6 in the submission, because the smaller 15% validation split increases seed-to-seed variance on some base models across the factor-of-2 search grid.

Thank you again for this constructive point and we will report this in the revised version!

AuthorsOpenReview ↗

Response to Q2

Q2 Could the authors conduct a controlled fine-tuning experiment on pretrained language models using real-world semantic tasks? For example, supervision could be provided for only a subset of classification tasks, while related tasks serve as controls.

Q2. Thank you for this insightful suggestion and for recognizing the comprehensiveness of our experiments! The answer is yes. Following your exact design, we ran controlled single-task fine-tuning on two pretrained models under our main five-seed protocol (three fine-tuning seeds per setting). In each run, we supervise the labels of one task and evaluate the supervised task, related unsupervised tasks, and unrelated tasks. On Gemma-2-9B, we fine-tune with LoRA on harmfulness or sentiment and evaluate 12 tasks. On Gemma-3-1B, we run full-parameter fine-tuning on each of six tasks in turn.

  1. Supervised tasks concentrate. On Gemma-2-9B, harmfulness falls from 341 to 16 coordinates and sentiment falls from 72 to 16 (median reduction factor 8.4 across supervised runs), while dense-probe AUROC changes by less than 0.01. On Gemma-3-1B, harmfulness falls from 234 to 16, sentiment from 128 to 16, and jailbreak detection from 16 to 2, while tasks already at 1 to 4 coordinates stay unchanged (banking intent moves from 34 to 32).

  2. Unrelated tasks stay unchanged. Unrelated unsupervised tasks change by a median factor of 1.0 on both Gemma-2-9B and Gemma-3-1B (across all 78 observations on 1B).

  3. Related tasks split by shared concept versus shared text. Unsupervised tasks that share a similar semantic concept on different texts also concentrate (median factor 4.0 on Gemma-2-9B), as ToxicChat toxicity falls from 117 to 49 under harmfulness supervision and IMDB and Yelp sentiment fall from 16 and 8 to 2 under sentiment supervision. By contrast, an unsupervised task with a different label on the exact same prompts does not concentrate, as jailbreak detection under harmfulness supervision rises from 8 to 32 on Gemma-2-9B and from 16 to 32 on Gemma-3-1B.Readout concentration therefore tracks the supervised concept and does not come from exposure to the input texts alone, matching our synthetic control that shares half of a target's input factors yet still needs 60.5 coordinates.

Thank you again for proposing this thoughtful experiment! It strengthens Section 4 by directly connecting our synthetic and instruction-tuning findings, and we will include all of these results in the revised version.

Reviewer_nYrcOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

The paper explains how to compute the smallest number of qualifying features to achieve a certain task performance threshold. They apply it to different models and find that they get similar feature counts. On a small number of models, the selected features also correspond to each other. By comparing the base models and their instruction-tuned counterparts, the authors also show that training alters the readout sparsity.

Critical strengths

Section 5.3 could have interesting implications, helping design a better-informed pruning algorithm.

Critical weaknesses
  • Clarity and presentation: The paper is hard to understand. It needs improvement in explaining the problem the authors are trying to solve, the challenges with existing methods, the proposed method, and the takeaways. The paper is too dense with details that are not important to conveying the message. There are variables that are introduced without being defined (e.g., N^* in the abstract). It’s currently presenting the work as a stream of things the authors tried without a statement of what the overarching goal is.
  • Significance: The paper lacks a “so what?” and fails to communicate it to the readers clearly.
Questions for authors
  • Could you explain concisely the importance of studying how task information is organized in LMs?
  • Building on Section 5.3, could you expand on its implications? For example, how could it inform a better-informed pruning algorithm?
  • Regarding Section 5.3, could you explain why protecting weight rows and columns associated with selected task feature coordinates does not outperform the random baseline? Isn’t this actually showing that the selected features are not really task-relevant?
4 comments
AuthorsOpenReview ↗

Response to W1 and Q1

W1 Clarity and presentation.

W1. Thank you for these constructive suggestions on presentation. In the revision, we will state the overarching goal and the four points you list more clearly.

  1. Problem and overarching goal. Do language models agree on how many representation coordinates a linear probe needs to read out each task, and how does supervision change this number?

  2. Challenges with existing methods. Dense probes use all coordinates, reach an AUROC above 0.90 on 9 of our 14 tasks across all ten base models, and barely move under supervision, so they do not separate tasks, models, or training stages. Sparse probing (Gurnee et al., 2023) reports a score at each fixed coordinate budget k, which cannot tell whether supervision added task information or concentrated an existing readout onto fewer coordinates.

  3. Proposed method. We measure k*, the smallest number of coordinates needed to recover 95% of the dense probe's above-chance AUROC under a fixed protocol, separating readout concentration from overall linear readability.

  4. Takeaways and paper structure. Sections 3, 4, and 5 follow this goal in three steps, showing what pretrained models share (Section 3), how supervision concentrates task readouts (Section 4), andwhat readout sparsity implies and bounds in downstream adaptation, continual learning, and pruning (Section 5), as summarized in Q1 below.

To remove distracting details from the main text, we will unify terminology throughout (using dense-probe AUROC and k* consistently instead of multiple synonyms), open each section with the question it answers and its main takeaway, and keep secondary protocol details in the appendix.

Finally, thank you for pointing out that N* was not defined in the abstract, and we apologize for this oversight. N* is the number of labels a student model needs to reach 95% of the above-chance AUROC gain attained at the largest tested label budget. We will state this in plain words in the abstract and define every symbol at its first appearance.

Thank you again for this constructive feedback! And we will state them more clearly in the revised version.

Q1 Could you explain concisely the importance of studying how task information is organized in LMs?

Q1 & W2. Thank you for this constructive question! It helps us to state our contribution more clearly.

Behavioral tests observe outputs, and dense probes test whether task information is linearly readable. Because two representations with the same dense-probe AUROC can differ in how they organize that information, our work can study the differences across tasks, models, and training stages. Measuring k* turns this internal organization into a coordinate count and yields four main conclusions.

  1. Pretrained models share a common organization. Ten base models from five labs agree on which tasks need few coordinates and which need many, with a median rank correlation of 0.929, and selected SAE features also correspond across models.

  2. Supervision concentrates task readouts. Supervision on a task lowers k* without changing dense-probe AUROC, from 341 coordinates to 47 after instruction tuning and 16 after label fine-tuning for harmfulness on Gemma-2-9B, while unrelated tasks stay unchanged.

  3. Readout sparsity informs downstream behavior. Measuring k* once on a frozen model estimates student label requirements before training, identifies which task a fine-tuning run supervised, and informs which tasks lose dense-probe AUROC earlier under pruning.

  4. Readout concentration differs from information storage. Later continual learning spreads a concentrated readout without erasing linear readability, and protecting only the selected coordinates' weights during pruning does not preserve the task, indicating that task information remains distributed across many coordinates.

Therefore, studying how task information is organized captures a static layout shared across pretrained models and a dynamic process in which supervision concentrates readouts while leaving distributed traces across the network. We will state this motivation at the start of the introduction.

AuthorsOpenReview ↗

Response to Q2 and Q3

Q2 Building on Section 5.3, could you expand on its implications? For example, how could it inform a better-informed pruning algorithm?

Q2. Thank you for seeing the potential of Section 5.3 and for this insightful question! In our opinion, tracking how task readouts respond to pruning suggests three design principles for pruning algorithms and one testable hypothesis.

  1. Set target sparsity by the widest task readout. Before pruning, the four tasks with small k* (up to 34) lose at most 0.015 in dense-probe AUROC even at 90% sparsity, whereas sentiment and harmfulness (k* of 72 or more) begin dropping between 50% and 70% sparsity. For a model serving multiple tasks, pre-pruning k* shows how much we can prune safely and which tasks post-pruning validation should check first.

  2. Track k* alongside dense probes and refit sparse heads after pruning. On the four small-k* tasks, k* at least doubles in 26 of the 32 pruning trajectories while dense-probe AUROC drops by less than 0.01. Pruning can therefore spread a concentrated readout even when linear readability appears intact, indicating that sparse heads fitted before pruning should be refitted and that k* provides a complementary check on pruning quality.

  3. Constrain readout activations in the pruning objective. Protecting the weight rows and columns of selected coordinates performs no better than random protection, indicating that task information remains distributed across many coordinates beyond the selected subset. Task-aware pruning should therefore penalize activation changes along the task readout direction.

  4. Hypothesis: concentrate before pruning. Reflecting on your question led us to this new hypothesis, and we thank you for prompting it. Supervision lowers harmfulness k* from 341 to 16 on Gemma-2-9B, and tasks with concentrated readouts withstand higher sparsity. Applying supervision on target tasks before pruning may therefore preserve them at higher sparsity. We will add these points to Section 5.3.

Thanks again for this insightful question!

Q3 Regarding Section 5.3, could you explain why protecting weight rows and columns associated with selected task feature coordinates does not outperform the random baseline? Isn’t this actually showing that the selected features are not really task-relevant?

Q3. Thank you for this insightful question! The answer is no. The selected coordinates are task-relevant, and the protection result shows that coordinates sufficient for readout are not the only ones sustaining the task.

Two results confirm task relevance. First, on Gemma-2-9B, 256 selected coordinates read harmfulness better than 256 random ones (0.962 vs. 0.945 to 0.949 AUROC). Second, ablating 1 to 4 selected language features collapses target-language replies (gap from 0.88 to 0.99 down to at most 0.015), while matched unselected ablations leave them intact.

Protecting the weights of selected coordinates fails to uotperform random protection for three reasons.

  1. Redundancy at the protection scale. At the union scale (~27% of 3,584 coordinates), 975 random coordinates already read harmfulness as well as the top 975 (0.969 to 0.973 vs. 0.973), allowing a refitted probe to rebuild the readout from other coordinates.

  2. Cross-layer computation. Earlier layers compute each coordinate's activation from all coordinates, so keeping weights attached to a coordinate leaves its upstream computation unprotected.

  3. Pruning quota shift. Keeping 27% of columns dense forces a 70% sparsity target to prune 96% of weights on the remaining columns, hurting overall performance (sentiment AUROC drops from 0.947 under ordinary Wanda to 0.891 and 0.888).

Together with our continual-learning and pinning results, these observations suggest a testable conjecture about training dynamics. Pretraining distributes task information across many coordinates, and supervision concentrates the model's readout onto a few coordinates (341 to 47 to 16) with residual information left in the rest. Consequently, fixed classification heads and model behavior depend on the concentrated coordinates, whereas refitted probes can fall back on the distributed residuals. This points to two concrete directions for future work, i) tracking readout concentration alongside residual erasure across fine-tuning checkpoints, and ii) protecting task-specific upstream circuits across layers instead of masking readout coordinates alone.

Thank you again for this insightful question! We will add this in the revised version.

Thank you for the detailed responses and additional experiments. However, the practical value of the analysis remains unclear. The tested approach does not improve on the baselines: protecting weights at selected coordinates performs no better than random protection and worse than standard Wanda. The proposed principles for designing pruning algorithms are interesting, but they have not yet been evaluated within a pruning method. I will therefore maintain my current rating. Evidence of practical benefit would help me reconsider. For example, showing that incorporating constraints on readout activations into the pruning objective preserves task performance better than standard Wanda at the same sparsity would strengthen the case.

Thank you very much for your time and reply, and thank you for acknowledging our responses and additional experiments! We agree that practical use can further strengthen the paper.

However, as an analysis paper, our work uses the pruning experiment more as an analytical tool than a proposed pruning method. Our intention is using the experiment to reveal phenomenon on how language model organize task information. And the finding is that protecting the selected coordinates does not protect task information, which shows that task information is distributed across many coordinates beyond the selected ones.

This phenomenon provides two insights.

  1. Reading a task and storing it are different properties. A small subset of coordinates suffices to read a task, yet the task stays readable from many others.

  2. Locating where a probe reads a task does not locate everywhere the model holds it. A dense probe uses all coordinates and cannot show this distinction, whereas k* and the comparison against random coordinates make it measurable.

These insights are our contribution to the interpretability community. And we will make the presentation clearer as you suggestted in the revised version. Also, in future work, we will continue to explore how they can more directly inform the design of better pruning algorithms.

Thank you again for your time and constructive feedback!

Reviewer_SbYgOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
3
Read review
Paper summary

The paper proposes k*, the smallest number of representation coordinates that a fixed search protocol finds sufficient to recover 95% of a dense linear probe's above-chance AUROC, as a measure of "readout sparsity" complementary to dense-probe readability. Using a carefully specified protocol (relative threshold, two search paths, neighbor-layer confirmation, five seeds), the authors evaluate 19 LMs from five labs on 14 tasks. They report that base models share similar per-task counts and task orderings (median cross-lab Spearman 0.93), and that selected SAE features show cross-model response correspondence on three tasks. Instruction-tuned deployment selectively reduces k* (e.g., harmfulness 341 → 47 on Gemma-2-9B), with both weights and chat format contributing. In 20 controlled synthetic models, supervised targets admit single-feature readouts while readable but unsupervised controls need tens of features. Finally, reference-model k* is associated with student label requirements (N* ∝ (k*)^0.65) and with pruning vulnerability, while k*-based label allocation and parameter protection do not outperform their baselines.

Critical strengths
  1. The measurement protocol is rigorous and its limits are reported honestly (Section 2, Appendices A.1–A.2). The relative threshold, neighbor-layer confirmation, shuffled-label control, rotation and paraphrase controls, and finite-sample analysis make k* a well-defined and reproducible quantity, and the authors report negative results (allocation and protection) rather than hiding them. This makes the reported numbers trustworthy even where their significance is debatable.

  2. The synthetic target–control experiment cleanly isolates the effect of supervision (Section 4.2, Table 38). Targets and controls have matched label distributions and identical input information, differing only in whether training rewards the aggregate; supervised targets read at k = 1 in all 20 models while controls need tens of features, including a control sharing half its factors with a target. This is the clearest evidence that k* captures something dense accuracy does not.

Critical weaknesses
  1. Novelty over sparse probing and the motivation for k are not established (Sections 1, 5, 6).* That task labels can be read from few coordinates was already shown by sparse probing (Gurnee et al., 2023), and the main additions here are a refined measurement protocol and broader comparisons. The introduction motivates k* only by noting that it differs from dense accuracy, which is not a reason why it matters. The practical value claimed in Section 5 is weak: label-requirement forecasts carry a held-out error factor of 5.3, k*-based allocation costs a median of 1.03× the uniform budget (while a simple 16-label pilot is cheaper in all settings), k*-based parameter protection is no better than random protection, and the saved-head-versus-readability distinction in continual learning was already shown by Davari et al. (2022). This matters because the paper's significance rests on k* being useful beyond existing probing measures, and the evidence for that is mostly negative.

  2. The paper covers many loosely connected experiments, and the main measurements are strongly basis- and text-dependent (Sections 3–5, Tables 14–16, Appendix A.2.3). Cross-model comparison, SAE correspondence, instruction tuning, synthetic training, label budgets, continual learning, and pruning are each examined, but no single question ties them together, and each is studied with its own estimator variant (Table 9). Moreover, k* rises by 1.6–2.0× (up to 20×) under random rotation and by 1.5–10.8× under paraphrasing, and dense readability correlates at 0.95 with a bag-of-words baseline. This limits the claim that k* reveals "how LMs organize task information" rather than properties of a particular basis and text pool.

Questions for authors
  1. Beyond refining the protocol of Gurnee et al. (2023), what is the main new finding about LMs that k* reveals and that dense-probe ceilings plus existing sparse probing could not? Is there any downstream decision (data budgeting, compression, auditing) where using k* actually improves over a simple baseline such as the 16-label pilot?
  2. The cross-model task ordering is measured in each model's native residual basis, while rotation changes counts by up to 20× and was tested on only two models and one seed. Does the cross-model agreement (median ρ = 0.93) hold if counts are measured in a basis-independent way (e.g., a rotated or PCA basis) across all ten base models?
  3. The synthetic setting produces counts of either 1 or tens, while pretrained models show intermediate counts of hundreds. How do you connect the synthetic finding to the instruction-tuning reductions in Section 4.1, given that the paper acknowledges the synthetic world does not explain these intermediate counts?
  4. The main text covers seven distinct experimental lines with an 80-page appendix. Focusing the paper on one or two central questions (e.g., what supervision does to readout sparsity) and moving the rest to the appendix would make the contribution much clearer.
4 comments
AuthorsOpenReview ↗

Response to W1.1 and Q1.1

Q1.1 & W1.1. Beyond refining the protocol of Gurnee et al. (2023), what is the main new finding about LMs that k∗k^* reveals and that dense-probe ceilings plus existing sparse probing could not?

Q1.1 & W1.1. Thank you for recognizing the rigor of our measurement protocol, and thank you for this constructive question! It helps us state our contribution more clearly.

Our contribution is: pretrained models agree on which tasks need few coordinates and which need many, with a median cross-lab rank correlation of 0.93, and supervision on a task concentrates its readout onto a few coordinates while its dense-probe AUROC stays the same.

Section 4 reports this concentration for instruction tuning and for the synthetic experiment that you highlight. Following your feedback, we ran controlled fine-tuning experiments that confirm this finding in pretrained models. Specifically, harmfulness on Gemma-2-9B needs 341 coordinates in the base model, 47 after instruction tuning, and 16 after fine-tuning. The dense-probe AUROC stays between 0.979 and 0.986, and the counts of unrelated tasks change by a median factor of 1.0.

Dense probing and existing sparse probing measure how well a task can be read from a given number of coordinates. Gurnee et al. (2023) use k coordinates, study models from one family at the end of pretraining, and list the change of neurons over training as a future direction. Our work explores this direction and measures how many coordinates a probe needs to recover 95% of the above-chance AUROC of the dense probe at the same layer.

Taken together, the two existing measures still give scores but our work gives a coordinate count. In the fine-tuning on harmfulness above, the dense-probe AUROC is 0.979 before and 0.982 after, while the AUROC of the sparse probe rises by 0.17 at k = 1 and by 0.01 at k = 1024. How much supervision changed the readout therefore depends on the chosen k, and a higher sparse-probe score alone does not separate a more concentrated readout from a higher dense-probe AUROC.

k* gives each task a specific number of coordinates relative to its own dense probe. With this number, we can compare tasks across models and compare the same task before and after supervision. Both parts of our finding come from k*, and they are also the reason k* matters.

AuthorsOpenReview ↗

Response to W1.2 and Q1.2

Q1.2 & W1.2. Is there any downstream decision (data budgeting, compression, auditing) where using k∗k^* actually improves over a simple baseline such as the 16-label pilot?

Q1.2 & W1.2. Thank you for this question and for noting that we report the allocation and protection results honestly. The answer is yes. Fine-tuning and pruning can change the readout of a task while the dense-probe AUROC stays the same, and k* traces this change.

  1. Auditing studies which tasks a fine-tuning run changed, where a student pilot does not apply. After we fine-tune Gemma-2-9B on the labels of one task, the dense-probe AUROC of every task changes by less than 0.01, so it does not show which task was supervised. k* falls by a median factor of 8.4 on the supervised task and changes by a median factor of 1.0 on unrelated tasks.

  2. Compression uses k* before and after pruning. Before pruning, k* predicts which tasks lose dense-probe AUROC first and shows how much we can prune safely. Tasks with k* up to 34 lose at most 0.015 even at 90% sparsity, whereas tasks with k* of 72 or more start dropping between 50% and 70% sparsity. After pruning, k* informs us when to refit a sparse head even if a dense probe shows no loss. On the four tasks with small k*, the count at least doubles in 26 of the 32 pruning trajectories while dense-probe AUROC drops by less than 0.01.

  3. Data budgeting estimates how many labels a student model needs. The 16-label pilot allocates labels by training each student on each task first. Before we train any student, k* measured once on a reference model gives the lowest prediction error. Its held-out error factor is 5.3, compared with 5.7 for a bag-of-words baseline, 6.0 for dense-probe AUROC, 8.4 for LogME, and 9.5 with no predictor.

  4. Parameter protection and continual learning test whether the selected coordinates are the only ones carrying a task. Although the coordinates selected by k* suffice to read a task, protecting their weights during pruning performs no better than random protection. In continual learning, Davari et al. (2022) show in vision models that a saved head fails while task information stays readable, and k* tracks what changes inside the representation. Across four language models, fine-tuning concentrates harmfulness onto a median of 4 to 8 coordinates, and later training on other tasks spreads it back to 64 to 214 while median dense-probe AUROC drops by less than 0.01. Together, these observations lead us to conjecture that supervision builds a concentrated readout on a few coordinates while task information remains distributed across many coordinates that the model already had.

As an analysis paper, our goal is to map both where k* guides decisions and where its boundary lies. This boundary points directly to future work, including measuring how many coordinates must be removed to erase readability, constraining readout activations during pruning, and combining k* with a small pilot to allocate labels.

AuthorsOpenReview ↗

Response to W2.2 and Q2, Q3

Q2 & W2.2. The cross-model task ordering is measured in each model's native residual basis, while rotation changes counts by up to 20× and was tested on only two models and one seed. Does the cross-model agreement (median ρ = 0.93) hold if counts are measured in a basis-independent way (e.g., a rotated or PCA basis) across all ten base models?

Q2 & W2.2. Thank you for this contructive point and for proposing this concrete test! The answer is yes. Changing the basis or the text shifts absolute counts, yet both of our main findings hold, namely the cross-model task order and the concentration under supervision.

  1. Basis. Following your suggestion, we remeasured all ten base models under our five-seed protocol in a random orthogonal basis (one random rotation per seed) and in a rotation-invariant PCA basis. Cross-lab agreement on the task order stays high, with a median rank correlation of 0.929 in the native basis, 0.963 in the rotated basis, and 0.947 in the PCA basis. Supervision also concentrates readouts in all three bases, as instruction tuning on Gemma-2-9B lowers harmfulness k* by factors of 7.3 (native), 2.6 (rotated), and 8.0 (PCA). Meanwhile, random rotation leaves median dense-probe AUROC unchanged (+0.000) while raising k* by a median factor of 1.79 across all cells (the 20x case you noted occurs in one of 20 intent classes on Gemma-3-4B, whose class median is 2.4x). This increase shows that readouts align partially with the model's native coordinates rather than an arbitrary basis.

  2. Text. Changing the text changes the value of k*, yet text alone cannot explain either the cross-model agreement or the drop under supervision. First, holding the texts fixed, fine-tuning Gemma-2-9B on harmfulness labels lowers k* from 341 to 16. Second, paraphrasing raises k* by factors of 1.5 to 10.8 while preserving the task order. Third, controlling for bag-of-words probe scores on the same texts leaves the cross-lab task-order correlation between 0.86 and 0.92. Finally, the 0.95 correlation you noted is between dense-probe AUROC and the bag-of-words baseline. Because raw text largely determines dense-probe AUROC, k* is needed to capture how the model itself organizes the readout.

Q3. The synthetic setting produces counts of either 1 or tens, while pretrained models show intermediate counts of hundreds. How do you connect the synthetic finding to the instruction-tuning reductions in Section 4.1, given that the paper acknowledges the synthetic world does not explain these intermediate counts?

Q3. Thank you for this constructive question and for recognizing that the synthetic experiment isolates the effect of supervision. Two things explain the difference in counts and connect the synthetic models to Section 4.1. i) Wider models need more coordinates on unsupervised tasks, and ii) more direct supervision concentrates a task onto fewer coordinates. Following your feedback, we ran controlled fine-tuning experiments inside pretrained models to test this bridge directly.

  1. Model width explains why unsupervised counts grow from tens to hundreds. The 20 synthetic models have widths of 64 to 256, so their unsupervised controls need 29 to 60.5 coordinates. Our pretrained models have widths of 640 to 5,376, so tasks without direct supervision start in the hundreds, such as harmfulness at 341 on Gemma-2-9B and 234 on Gemma-3-1B.

  2. Direct label fine-tuning reproduces the synthetic target and control split inside pretrained models. On Gemma-2-9B, fine-tuning on harmfulness labels lowers harmfulness from 341 to 16 coordinates. On Gemma-3-1B, single-task fine-tuning lowers harmfulness from 234 to 16, sentiment from 128 to 16, and jailbreak detection from 16 to 2, while tasks already at 1 to 4 stay unchanged and banking intent moves from 34 to 32. Unrelated tasks change by a median factor of 1.0 on Gemma-2-9B and across all 78 observations on Gemma-3-1B. Shared inputs alone do not concentrate a readout in either setting. Just as a synthetic control sharing half of a target's input factors still needs 60.5 coordinates, supervising harmfulness does not lower jailbreak detection on the same prompts, which instead rises from 8 to 32 on Gemma-2-9B and from 16 to 32 on Gemma-3-1B.

  3. Indirect supervision produces intermediate counts between these two extremes and connects the synthetic result to Section 4.1. Whereas synthetic models either supervise a target directly or give a control no supervision, instruction tuning in Section 4.1 supervises responses. On Gemma-2-9B, it places harmfulness at 47 coordinates, between 341 in the base model and 16 after direct label fine-tuning. A related task without direct supervision also lands in between, as fine-tuning Gemma-2-9B on harmfulness labels lowers ToxicChat toxicity from 117 to 49.

Thank you again for this constructive question! It helps us to connect the synthetic experiment to pretrained models directly.

AuthorsOpenReview ↗

Response to Q4 and W2.1

Q4 & W2.1. The main text covers seven distinct experimental lines with an 80-page appendix. Focusing the paper on one or two central questions (e.g., what supervision does to readout sparsity) and moving the rest to the appendix would make the contribution much clearer.

Q4 & W2.1. Thank you for this constructive suggestion! Our seven experimental lines form a single three-part argument around two central questions. i) Do models agree on how many coordinates each task needs, and, in your words, ii) what does supervision do to readout sparsity? The first four lines answer these two questions, and the last three test what readout sparsity implies downstream and map its boundaries.

  1. What pretrained models share (Section 3, cross-model comparison and SAE correspondence). Pretrained models agree on which tasks need few coordinates and which need many, and the selected SAE features also correspond across models.

  2. What supervision changes (Section 4, instruction tuning and synthetic models). Instruction tuning lowers k* selectively in deployed models, and the synthetic experiment isolates supervision as the cause. Our new fine-tuning experiments reproduce the synthetic design inside pretrained models and connect these two lines directly.

  3. Downstream implications and boundaries (Section 5, label budgets, continual learning, and pruning). Tasks with larger k* require more labels for student adaptation and lose dense-probe AUROC earlier under pruning, while later training on other tasks spreads a concentrated readout. Together, these lines also define the boundary reported in Q1.2, showing that the selected coordinates suffice for a readout without uniquely sustaining it. Regarding the estimator variants in Table 9, each variant matches where its experiment computes or trains features, such as the query position where synthetic models output answers and the final token where the SAEs were trained. Every comparison stays within a single row of Table 9, and task orders across the variants in Table 19 correlate with the main protocol at a median of 0.97 to 0.99. Moreover, all new fine-tuning and basis experiments use the main estimator.

Thank you again for this constructive feedback! It gives the paper a much clearer main line. In the revision, we will center the narrative on these two questions, adopt your phrasing of the second question in the introduction, and include the new experiments.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Rebuttal Summary

Dear Area Chair and Senior Area Chair,

Thank you for overseeing our submission. Reviewers SbYg and cJpe posted no follow-ups, while Reviewer nYrc acknowledged our responses and experiments and kept the score. We summarize our contributions, rebuttal outcomes, and case for acceptance.

1. Paper Summary and Core Contributions

Problem and metric. Dense probes exceed 0.90 AUROC on 9/14 tasks across ten base LMs, correlate at 0.95 with bag-of-words, and barely move under supervision, while sparse probing (Gurnee et al., 2023) scores fixed k in one family after pretraining and cannot separate readability from concentration. Asking if LMs share per-task coordinate counts and how supervision changes them, we measure k*, the fewest coordinates recovering 95% of same-layer above-chance dense AUROC.

Core findings. Ten base LMs share per-task counts and rankings (median cross-lab correlation 0.929), selected SAE features correspond across LMs, and ablating 1 to 4 language features removes target-language replies (Sec. 3). Supervision concentrates readouts at flat dense AUROC, scaling from instruction tuning (harmfulness 341 to 47 on Gemma-2-9B) to 20 synthetic models (k* = 1 on targets vs. 29 to 60.5 on controls, Sec. 4). Reference k* tracks student label needs and pruning loss, while continual learning and protection separate reading from storage (Sec. 5).

2. Key Issues Resolved in the Rebuttal

2.1. Held-out validation (cJpe). Across all 266 cells, selecting references, layers, and k* on the untouched 15% training split recovers 96.9% median dense AUROC on the untouched test set (95% target), matches submitted k* at 1.00 ratio (84.4% within one 2x step), keeps cross-lab order at 0.924 (vs. 0.929), and confirms IT harmfulness drops in 9/9 pairs.

2.2. Basis and text controls (SbYg, cJpe). Remeasuring all ten base LMs in random orthogonal and rotation-invariant PCA bases preserves cross-lab order (0.929 native, 0.963 rotated, 0.947 PCA) and 9B IT harmfulness drops (7.3x, 2.6x, 8.0x), while bag-of-words controls keep 0.86 to 0.92 correlation. Rotation leaves dense AUROC unchanged (+0.000) while raising k* 1.79x, showing partial native-coordinate alignment.

2.3. Controlled pretrained fine-tuning (cJpe, SbYg). Single-task fine-tuning on Gemma-2-9B (LoRA) and Gemma-3-1B (full, six tasks) bridges synthetic and IT results, with width (64 to 256 vs. 640 to 5,376) explaining hundreds-scale baselines. Supervised tasks drop sharply under 0.01 dense shift (9B median 8.4x, harmfulness 341 to 16, and 1B harmfulness 234 to 16, sentiment 128 to 16, jailbreak 16 to 2), while unrelated tasks stay at 1.0x (all 78 runs on 1B). Related concepts concentrate (9B median 4.0x, ToxicChat 117 to 49, IMDB/Yelp 16/8 to 2), whereas jailbreak on the same prompts widens (8 to 32 on 9B, 16 to 32 on 1B), proving concept specificity.

2.4. Narrative and notation (SbYg, nYrc). Our revision will center Secs. 3 to 5 on our two core questions, clarify within-row Table 9 comparisons (0.97 to 0.99 correlation with the main protocol, used in all new runs), and define N* and all terms clearly.

3. Why This Paper Merits Acceptance

3.1. Rigor and empirical closure. SbYg rated soundness at 3 (praising protocol rigor, synthetic isolation, and honest negatives), and our work answers the training question left open by Gurnee et al. (2023). Every test proposed by SbYg and cJpe is completed under our main protocol and confirms our claims.

3.2. Diagnostic and predictive value without pilots. Frozen-model k* audits supervised tasks at flat dense AUROC (8.4x vs. 1.0x), gives the lowest pre-training error for student label need, separates resilient tasks from vulnerable ones, and flags post-pruning spreading for head refitting.

3.3. Reviewer nYrc's concern and analysis scope. Reviewer nYrc called our pruning principles interesting, keeping the score only because weight protection did not beat Wanda and asking for activation-constrained pruning that beats Wanda. In our analysis paper, pruning and protection serve as analytical probes rather than a compression method. Because 975 random coordinates (~27% scale) already match the top 975 on harmfulness (0.969 to 0.973 vs. 0.973) and upstream weights stay unprotected, this result and continual-learning spreading show that reading a task and storing it are distinct properties, informing future activation-constrained pruning.

Finally, we sincerely thank all reviewers for their constructive and insightful comments, which helped us greatly improve our paper, and we will reflect them in our revisions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.