acceptodds
Under review as a conference paper at ICLR 2027

Hidden States as States: Discretizing Hidden-State Geometry in Language Model Computation

Abstract

While large language models (LLMs) operate on discrete tokens, their internal computation unfolds in a continuous, high-dimensional representation space. We introduce Hidden States as States (HSS), a training free and label free framework that discretizes hidden representations at a chosen granularity into a cross-layer vocabulary of geometric states. HSS clusters representations within each layer and aligns cluster centroids across adjacent layers, converting each forward pass into a layer-indexed discrete state sequence. The resulting maps are stable under clustering perturbations, expanding in intermediate layers and contracting later in model-specific patterns. Conditioning on behavioral labels further reveals distinct state occupancy patterns, with failures collapsing into a few persistent sink states that emerge in early layers. Simple count-based predictors over these discrete state sequences outperform closed-form continuous baselines, match trained continuous probes on capability and safety benchmarks, and support online failure warning during generation. Reconstruction and activation editing show that, despite their coarse granularity, these states retain functionally relevant structure for downstream computation. HSS thus converts continuous hidden dynamics into an actionable discrete structure.

Primary area:
interpretability and explainable AI
Keywords:
Model Interpretability, Representation Geometry, Latent Space Clustering, Hidden State Dynamics, Model Evaluation, Safety

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_E6EzOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes a method called Hidden States as States (HSS), which is put forward to describe the hidden representations of LLMs in terms of a sequence of discrete states. At each layer, the representations are clustered, and then the centers of the adjacent clusters are linked. In practice, every forward pass is transformed into a sequence of states across the layers in the target architecture. This approach describes the evolution of representations across layers and it is used to predict whether the model is correct and safe using a count-based classifier. Experiments are conducted using Llama and Qwen models (1B to 8B).

Critical strengths
  • The simplicity and practicality of the approach, which needs no training and no labels.
  • The experiments on activation editing are interestingly controlled, showing some actual evidence that the states identified are not just descriptive.
Critical weaknesses
  • I have some doubts about the evaluation protocol, which reduces the soundness of the experimental analysis on prediction. In particular, regarding the results shown in table 1: to the best of my understanding, they are achieved on a single split 40-60, most differences with the trained baselines are smaller than 3 points, and no standard deviations are reported. In addition, I could not find information on how some baseline hyperparameters (e.g., size and regularization of the MLP) were chosen. The MLP itself shows better performance than the proposed method in all the columns of table 1, and the other baselines are better on the safety columns. Based on these observations, the claim that the proposed method matches trained continuous probes on capability and safety benchmarks, given in the abstract, seems not supported by the given results.

  • A central part of the proposed method is the cross-layer alignment of states (section 3.3). However, the prediction results seem not to test this part of the method, as in fact the count-based predictor depends only on the clusters of each layer and not on the cross-layer alignment. Moreover, from what I could understand, the number of states per layer is sensitive to the number of samples, the selection tolerance and the clustering algorithm. Thereby, it risks not being really comparable across different settings.

Questions for authors
  • Please report mean and std over different random splits for the results in tables 1 and 2, possibly including a statistical test against the baselines (this would support the claim in the abstract about the fact that the proposed approach matches the trained continuous probes).
  • Please clarify how you chose the hyperparameters for the baselines and the settings for HSS.
  • Please clarify what is adding the cross-layer alignments in terms of the results given in section 5 (as it seems that the count-based predictor is not depending on that).
  • Given the sensitivity to the number samples, selection tolerance and clustering alg., please clarify how the number of states should be compared among the tested models and across the settings?
4 comments
AuthorsOpenReview ↗

Response to Reviewer E6Ez (1/2): W1, Q1-Q2

W1 / Q1

Question 1: Please report mean and std over different random splits for the results in tables 1 and 2, possibly including a statistical test against the baselines (this would support the claim in the abstract about the fact that the proposed approach matches the trained continuous probes).

Response:

We thank the reviewer for emphasizing statistical reliability. We repeated the prediction evaluation over ten stratified 40/60 train/test splits, refitting the state map on each training split. We also conducted a separate evaluation over five shared splits with hyperparameters selected using training data only.

Table A. Repeated-split prediction results.

MethodQwen2--MATHLlama-3--MATHLlama-3.2--MATHLlama-2--WildJailbreak
HSS-NB (τ=0,α=1\tau=0,\alpha=1)76.0 ±\pm 0.676.0 ±\pm 0.875.6 ±\pm 0.790.2 ±\pm 0.4
LDA62.9 ±\pm 2.461.4 ±\pm 0.952.6 ±\pm 1.189.6 ±\pm 0.5
Gaussian NB70.2 ±\pm 0.669.9 ±\pm 0.969.7 ±\pm 0.583.9 ±\pm 0.9
Linear SVM71.0 ±\pm 1.168.9 ±\pm 0.768.7 ±\pm 0.787.8 ±\pm 0.8
Logistic regression73.8 ±\pm 0.874.3 ±\pm 0.471.1 ±\pm 0.591.1 ±\pm 0.4
Logistic regression, best single layer73.4 ±\pm 0.773.6 ±\pm 0.570.8 ±\pm 0.890.4 ±\pm 0.3
MLP76.4 ±\pm 1.076.9 ±\pm 1.276.8 ±\pm 0.990.7 ±\pm 1.1

For Table 2, we evaluated five response-level 40/20/40 train/validation/test splits. Each method's threshold was calibrated on validation responses to target a false-alarm rate of at most 10%.

Table 2. Early warning on Qwen2--MATH.

MethodAUROC @50%AUROC @FinalDetected (%)FAR (%)Saved (%)
Prefix entropy †\dagger62.166.827.48.618.5
Prefix logprob †\dagger59.865.124.17.916.2
MLP78.9 ±\pm 0.680.7 ±\pm 0.443.3 ±\pm 3.210.2 ±\pm 1.437.3 ±\pm 2.2
HSS-NB75.4 ±\pm 1.079.5 ±\pm 0.943.9 ±\pm 1.610.4 ±\pm 1.537.8 ±\pm 1.5
Logistic regression75.8 ±\pm 0.479.0 ±\pm 0.239.4 ±\pm 7.010.1 ±\pm 2.733.5 ±\pm 5.8

These statistical tests further support our paper findings. HSS-NB as a simple label-free count-based model matches continuous probes. We respectfully point out that it is still encouraging that useful predictive information survives reducing each layer's activation from thousands of continuous dimensions to a discrete state defined without labels, and can be recovered through simple class-conditional counts. What's more, this also provides a more interpretable way to track and analyze internal representations in the quantized geometric regions. For a given example, we can inspect each visited state's class counts and contribution to the score, connecting the prediction to the regions being studied. Once the map is fixed, different behavioral labels can also be examined by estimating their counts over the same partitions.


W1 / Q2

Question 2: Please clarify how you chose the hyperparameters for the baselines and the settings for HSS.

Response:

We thank the reviewer for asking us to clarify the hyperparameter choices. For the baselines, we performed a grid search over different model capacities and regularization strengths to select the best hyperparameter settings. For the MLP, we tested single hidden layers with 64, 128, 256, and 512 units, as well as a two-layer architecture with 256 and 128 units, jointly varying L2 regularization over {10−4,10−3,10−2,10−1}\{10^{-4},10^{-3},10^{-2},10^{-1}\}. We selected the configuration with the highest mean AUROC in five-fold cross-validation within each training split. For the HSS, we select the number of clusters at each layer using fixed ICL on the training activations, for the prediction, we use τ=0\tau=0 for precise partition. This provides a consistent setup for assessing performance.

ComponentPredictionMonitoring
GMM candidate KK{2,…,40}\{2,\ldots,40\}{2,…,80}\{2,\ldots,80\}
ICL tolerance τ\tau00: minimum ICL00: minimum ICL
HSS-NBPer-layer smoothing α=1\alpha=1; empirical class priorsSame
All-layer logistic regression--C=10−3C=10^{-3}
Linear SVMC=1C=1, balanced class weights--
LDA / Gaussian NBSVD solver / default variance smoothing--
MLPOne 128-unit hidden layer; L2 =10−4=10^{-4}One 256-unit hidden layer; L2 =10−3=10^{-3}
Alarm threshold--Validation-calibrated to at most 10% FAR
AuthorsOpenReview ↗

Response to Reviewer E6Ez (2/2): W2, Q3-Q4

W2 / Q3

Question 3: Please clarify what is adding the cross-layer alignments in terms of the results given in section 5 (as it seems that the count-based predictor is not depending on that).

Response:

We thank the reviewer for pointing out this distinction. We agree that HSS-NB uses per-layer counts, so its predictions do not depend on cross-layer alignment. We did not attribute the prediction performance to alignment. Extracting additional predictive signals from the structure of the state map remains a promising next step.

At the same time, alignment gives states a shared identity across layers, which does help us understand and interpret the model. These allow us to distinguish a persistent failure-associated region from isolated low-accuracy clusters observed at individual layers. We can then examine where that region emerges, how long it persists, and which samples remain in it, enter it, or leave it. And we want to go further and understand why failures concentrate there. These are central questions in our analysis of hidden-state organization.

To further prove the necessity and legitimacy of alignment, we directly tested whether these geometric correspondences are behaviorally meaningful. For each cluster, we then measure the fraction of its samples whose final answers are correct. Across matched pairs of clusters in adjacent layers, we correlate these correctness rates on the log-odds scale, using random pairing within the same layer pairs as a control. The strong correlations show that low-accuracy regions tend to connect to low-accuracy regions, and high-accuracy regions to high-accuracy regions.

Model–datasetGeometric alignment: correctness log-odds correlation rrRandom pairing: mean rr
Qwen2–MATH0.930.01
Llama-3.2–MATH0.920.02
Qwen2–MMLU0.830.00
Phi-4–MATH0.910.01

These results show that alignment links clusters in a behaviorally consistent way, providing a meaningful description of how hidden-state geometry evolves across layers.


W2 / Q4

Question 4: Given the sensitivity to the number samples, selection tolerance and clustering alg., please clarify how the number of states should be compared among the tested models and across the settings?

Response:

We thank the reviewer for carefully examining the state counts. The number of clusters naturally depends on the sample size, parameters, and clustering algorithm. These choices determine the granularity at which we describe the hidden space, but our main findings in failure prediction, interpretation, and hidden-space organization still hold across the resolutions. Finer partitions can reveal more lexical and contextual distinctions, consistent with Shafran et al.’s observations.

Our experiments provide evidence for both. In prediction, increasing the ICL tolerance substantially reduces the number of states, while much of the predictive information remains.

Model–datasetMean states per layer, τ = 0% → 6%AUROC, τ = 0%AUROC, τ = 2%AUROC, τ = 6%
Qwen2–MATH20.4 → 4.776.074.873.7
Llama-3–MATH24.9 → 5.976.074.673.5
Llama-3.2–MATH22.6 → 3.175.673.872.4
Llama-2–WildJailbreak7.0 → 3.190.289.989.8

For interpretation, the behavioral consistency measured above remains strong at ICL tolerances of 2%, 3%, and 5% (r=0.93,0.93,0.95r=0.93, 0.93, 0.95, respectively). Middle-layer expansion also appears with both GMM and KMeans, despite their different partition sizes (Appendix B.4).

In general, absolute state counts should be compared using matched datasets and sample sizes, the same representation extraction procedure, and the same clustering and selection settings. Across models with different layers, we recommend comparing state-count on normalized layer depth, focusing on where expansion and compression occur and whether these patterns persist across reasonable changes in granularity. Following this approach, Figure 4(b) uses the same clustering settings and normalized depth to show that expansion and compression recur across the tested models, while their locations differ.

I thank the authors for the nice responses and for reporting the additional results. I appreciate the clarification on the hyper parameters of the baselines and the analysis about the cross-layer alignment.

The results over repeated splits, however, seem to refer to an exp. setting that is different from the one in Table 1. In facts, some models and dataset pairs seem to be different. Can you please clarify what changes wrt Table 1, and give mean / std over the splits for the same settings as in Table 1 (or, please update Table 1 in the revised manuscript)?

Moreover, in the new table, the MLP is still better in 5/7 columns (and no statistical test is given). I would suggest to reformulate the claim in the abstract regarding matching the trained continuous probes. I finally think that the manuscript should report clearly that the cross-layer alignment supports the interpretation of the results (and not the prediction).

AuthorsOpenReview ↗

Follow-up Response to Reviewer E6Ez: Updated Table 1 and Statistical Tests

We thank you for the quick and careful follow-up. Here we address your comments and have updated the manuscript accordingly (changes are marked in blue).

1. Repeated Splits for Table 1


For the split experiments, we reran the whole pipeline with the original model-dataset settings, from data collection to evaluation and prediction. The main conclusion holds the same: HSS-NB is above the closed-form baselines and approaches the continuous probe. Some variations are expected. One setting differs from the original submission: we now use ICL tolerance τ=0\tau=0 for the prediction maps, while the original version used τ=6%\tau=6\%. τ=0\tau=0 creates more states, and these states separate correctness better. Below are all eight Table 1 settings, as mean ± SD over ten shared stratified 40/60 train/test splits. Within a setting, all methods use the same examples, labels, and splits.

AUROC (%)

MethodQwen2–MATHQwen2–ThmQAQwen2–MMLUL3–MATHL3–ThmQAL3–MMLUL2–JBBL2–HarmBench
HSS-NB76.0 ± 0.674.8 ± 1.979.1 ± 0.576.0 ± 0.875.6 ± 2.678.9 ± 0.696.6 ± 1.695.2 ± 1.5
LDA62.9 ± 2.465.2 ± 2.269.2 ± 0.561.4 ± 0.967.3 ± 2.668.6 ± 0.593.5 ± 4.395.0 ± 4.3
Gaussian NB70.2 ± 0.671.1 ± 2.674.5 ± 0.369.9 ± 0.970.2 ± 0.972.3 ± 0.489.1 ± 3.187.2 ± 6.8
Linear SVM71.0 ± 1.171.0 ± 3.876.8 ± 0.968.9 ± 0.772.9 ± 2.773.1 ± 0.996.0 ± 1.894.9 ± 4.0
Logistic regression73.8 ± 0.871.3 ± 3.773.1 ± 0.874.3 ± 0.472.8 ± 3.273.3 ± 0.897.3 ± 1.896.9 ± 2.1
MLP76.4 ± 1.073.1 ± 4.879.6 ± 1.276.9 ± 1.275.3 ± 4.179.9 ± 1.296.3 ± 1.896.1 ± 2.2

Accuracy (%)

MethodQwen2–MATHQwen2–ThmQAQwen2–MMLUL3–MATHL3–ThmQAL3–MMLUL2–JBBL2–HarmBench
HSS-NB69.1 ± 0.771.8 ± 2.273.5 ± 0.570.4 ± 1.277.5 ± 3.373.1 ± 0.494.3 ± 1.995.8 ± 2.3
LDA60.0 ± 1.967.0 ± 2.967.4 ± 0.464.2 ± 1.276.5 ± 1.365.6 ± 0.594.2 ± 1.395.3 ± 1.1
Gaussian NB68.9 ± 0.572.7 ± 2.068.0 ± 0.472.1 ± 0.577.8 ± 1.966.5 ± 0.393.9 ± 0.694.0 ± 1.8
Linear SVM65.3 ± 0.969.6 ± 2.871.8 ± 0.371.5 ± 0.572.8 ± 2.670.7 ± 0.892.9 ± 1.689.9 ± 1.8
Logistic regression66.9 ± 0.672.5 ± 2.370.4 ± 0.373.3 ± 0.778.8 ± 2.371.6 ± 1.094.1 ± 1.495.3 ± 1.4
MLP70.4 ± 1.073.1 ± 1.874.2 ± 0.471.8 ± 1.079.5 ± 1.374.4 ± 0.993.7 ± 0.695.1 ± 1.3

HSS-NB: the number of clusters per layer is the ICL minimum (τ=0\tau=0) on training activations, with per-layer Laplace smoothing α=1\alpha=1. MLP: architecture (five candidates) and L2 penalty (four values) selected jointly by five-fold cross-validation inside each training split, then refit on that training split. All continuous baselines use the standardized concatenation of prompt-last vectors from all layers. The full parameter list is in Appendix A.5 of the revised manuscript.

We have replaced Table 1 with the ten-split means and report the standard deviations in Appendix Tables 5 and 6, so that the manuscript is consistent with these results.


2. Statistical Test

We ran a two-sided corrected resampled t-test (Nadeau & Bengio, 2003) on the paired AUROC differences between HSS-NB and each baseline, using the same ten shared splits. The p-values below are Holm-adjusted across all 40 comparisons; bold means adjusted p<0.05p<0.05.

Model–datasetvs LDAvs Gaussian NBvs SVMvs LRvs MLP
Qwen2–MATH0.2090.0200.6861.0001.000
Qwen2–ThmQA0.8671.0001.0001.0001.000
Qwen2–MMLU8.90×10−5\mathbf{8.90\times10^{-5}}0.0031.0000.0051.000
L3–MATH3.48×10−5\mathbf{3.48\times10^{-5}}0.0350.0211.0001.000
L3–ThmQA1.0001.0001.0001.0001.000
L3–MMLU1.61×10−4\mathbf{1.61\times10^{-4}}2.48×10−4\mathbf{2.48\times10^{-4}}0.2000.1621.000
L2–JBB1.0001.0001.0001.0001.000
L2–HarmBench1.0001.0001.0001.0001.000

These significant differences are in favor of HSS-NB: over Gaussian NB on the four MATH and MMLU settings, over LDA on three of them (Qwen2–MMLU, L3–MATH, L3–MMLU), over SVM on L3–MATH, and over LR on Qwen2–MMLU. Against the MLP there is no significant difference in any setting.


3. Paper Presentation

For the presentation, we updated the phrasing in the abstract and now state clearly in Section 5.1 the function of cross-layer alignment.

Reviewer_sgsyOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes HSS, a training-free and label-free way to turn a language model’s continuous hidden states into discrete states. The idea is simple: cluster the hidden states at each layer, then link similar clusters across neighboring layers, so each input becomes a sequence of state IDs over layers. The authors show that these state maps are fairly stable and look different across model families. They also find that correct and incorrect (or safe and unsafe) outputs tend to land in different states, with many failures ending up in a few persistent “sink” states. Using these states, a simple count-based predictor gets close to trained probes at predicting correctness and safety, and can also warn about failures during generation. Finally, editing experiments suggest the discovered regions point to directions that actually matter for the model’s computation.

Critical strengths
  • The core idea is simple and intuitive, and turning continuous hidden states into discrete layer-wise states offers a fresh and easy-to-interpret view of model internals.
  • The method is training-free and label-free, so it can be applied to any frozen model at low cost.
  • The paper is clearly written, and the figures, especially the state maps, make the framework easy to understand.
  • The authors check the stability of the state maps carefully, including different seeds, subsamples, search ranges, and clustering algorithms.
  • The paper goes beyond descriptive analysis by showing practical uses, such as predicting correctness and safety before generation and giving early failure warnings during generation.
Critical weaknesses

I should say up front that I am not an expert in this area, so please take the following points as suggestions rather than firm criticisms.

  1. The main parts of HSS, such as clustering each layer, matching clusters across layers, and using a naive Bayes predictor, seem fairly standard, and some related works look quite similar. For example, CLUE also uses hidden-state clustering to tell correct and incorrect reasoning apart, and Shafran et al. (2026) also split activations into local regions with their own directions, which feels close to Section 6.2. The paper mentions these works, but I was not fully sure what HSS adds on top of them. A clearer explanation, or a direct comparison where possible, would really help. Could the authors explain what HSS can do that these existing methods cannot?

  2. In Table 1, the continuous baselines take the hidden states from all layers stacked together, so the input is very high-dimensional, while only 40% of the data is used for training. In this setting, simple methods like LDA and Gaussian NB might struggle anyway, so I wonder if the comparison is a bit in favor of HSS-NB. It might be nice to also try simpler baselines, like a probe on the best single layer or logistic regression after PCA, and to add confidence intervals or results over several splits. In terms of AUROC, HSS-NB does well against logistic regression and SVM on the reasoning tasks, but it stays below the MLP everywhere and below all trained probes on the safety tasks. In which situations would the authors suggest using HSS-NB instead of a normal probe?

  3. The reconstruction and removal experiments use one model, one layer, and 64 problems, so I am not sure how broadly the findings generalize. The teacher-forcing checks across several layers are helpful, but the free-generation evaluation is still small and does not report a significance test for the accuracy improvement. These experiments provide useful initial evidence, though I would be cautious about drawing broader conclusions from them.

  4. The experiments use models from 1B to 8B parameters, all from the Llama and Qwen2 families, including the somewhat older Llama-2-7B, and the early-warning experiment uses only one model and one dataset. I am curious whether the same patterns, like the layer-wise expansion and the sink states, would also show up in larger models or newer reasoning models, which might organize their hidden states differently. Have the authors tried HSS on any bigger or reasoning-focused models?

Questions for authors

See the weaknesses.

3 comments
AuthorsOpenReview ↗

Response to Reviewer sgsy (1/2): W1-W2

W1

Question: Could the authors explain what HSS can do that these existing methods cannot?

Response:

We thank the reviewer for further interest in these close works. Using clustering as a method, CLUE, Shafran, and HSS target different objectives. CLUE uses labels to build two centroids for a classification task, whereas HSS constructs the state map without correctness labels, then studies how behavior is distributed over its geometry. Shafran et al. use regions and local subspaces as an alternative to SAE-style decomposition, focusing on static, local activation decomposition and within-region variation, while we focus on the dynamic, population-level description of how hidden states are organized and evolve across layers.

HSS provides a macroscopic description of how hidden-state geometry and a way to trace hidden dynamics. We start from a geometric perspective (like Shafran et al.) and attribute it to behavioral understanding and applications (like CLUE), offering a brand new angle on the dynamics of hidden representations during the forward pass.


W2

Question: In which situations would the authors suggest using HSS-NB instead of a normal probe?

Response:

We thank the reviewer for these suggestions. To check whether input dimensionality affects the comparison, we added logistic regression on a training-selected single layer and on PCA-reduced features. We used the same 40/60 train/test splits for all methods and repeated the comparison over ten splits. The table reports AUROC × 100, with mean ± standard deviation.

Table A. Dimensionality controls over ten splits.

MethodQwen2–MATHLlama-3–MATHLlama-3.2–MATHLlama-2–WildJailbreak
HSS-NB (τ=0, α=1\tau=0,\ \alpha=1)76.0 ± 0.676.0 ± 0.875.6 ± 0.790.2 ± 0.4
Logistic regression, best single layer73.4 ± 0.773.6 ± 0.570.8 ± 0.890.4 ± 0.3
PCA-256 + logistic regression, all layers75.6 ± 0.674.5 ± 0.773.6 ± 0.787.3 ± 1.0
MLP76.4 ± 1.076.9 ± 1.276.8 ± 0.990.7 ± 1.1

The single layer is selected by five-fold cross-validation on the training split. PCA and the HSS map are fitted using training activations only. Across these splits, HSS-NB performs close to or outperforms the MLP on the three MATH models and exceeds the single-layer logistic baseline with fixed C=1C=1 and PCA + logistic.

We respectfully point out that it is still encouraging that useful predictive information survives reducing each layer’s activation from thousands of dimensions to a discrete state defined without labels, and can be recovered through simple class-conditional counts. Further improving HSS-NB to outperform tuned continuous probes, while keeping the discrete states interpretable, is an interesting direction for future work. What’s more, HSS-NB is useful when prediction is part of studying the state map. For a given example, we can inspect each visited state’s class counts and contribution to the score, connecting the prediction to the regions being studied. Once the map is fixed, different behavioral labels can also be examined by estimating their counts over the same partitions.

AuthorsOpenReview ↗

Response to Reviewer sgsy (2/2): W3-W4

W3

Response:

We thank the reviewer for suggesting a larger evaluation. We expanded the reconstruction and removal experiments eightfold, from 64 to 512 GSM8K problems, keeping the state map and intervention setup fixed. The expanded evaluation supports the same findings: keeping the region’s own directions (Local8) preserves the next-token distribution better than keeping shared directions (Shared8) or another region’s directions (Wrong-local8). With equal edit size, removing Local8 also changes the next-token distribution more than the controls. All four comparisons have paired 95% confidence intervals excluding zero. Lower KL indicates better preservation; higher KL under removal indicates a larger output change.

Reconstruction and removal on 512 problems.

ExperimentComparison (first − second)Difference in next-token KL [95% CI]
ReconstructionLocal8 − Shared8−1.67 [−1.96, −1.39]
ReconstructionLocal8 − Wrong-local8−2.28 [−2.60, −1.97]
RemovalLocal8 − Random8+0.47 [0.34, 0.62]
RemovalLocal8 − complement control+0.38 [0.24, 0.53]

We also extended the original sink-clamp intervention to a larger set of 226 additional sink-selected problems. The larger evaluation supports the same qualitative finding as the original free-generation experiment: compared with no edit, the clamp increases the number of completed answers, pushing generation out of sink states.

Free-generation steering on 226 additional sink-selected problems.

OutcomeNo editSink clampOrthogonal control
Completed answers89140113
Correct answers203830
Responses remaining in the sink12551100

W4

Question: Have the authors tried HSS on any bigger or reasoning-focused models?

Response:

We thank the reviewer for this suggestion. We have now applied HSS to Phi-4 (14B) and DeepSeek-R1-Distill-Qwen-7B, extending the study to a larger model from another family and a reasoning-focused model. Both show the same qualitative findings as our original models: expansion followed by compression across depth, and persistent sink states with average accuracy well below overall answer accuracy. Below, we summarize the state count trend and compare sink-state accuracy with overall accuracy.

MeasurementPhi-4 (14B)DeepSeek-R1-Distill-Qwen-7B
MATH problems2,000500
State count trend across layers2 (L0) → 11 (L10) → 18 (L22) → 9 (L30) → 6 (L40)2 (L0) → 3 (L6) → 6 (L12) → 2 (L20) → 2 (L28)
Sink-state layer spanLayers 11–39Layers 4–13
Average sink-state answer accuracy20.6%27.8%
Overall answer accuracy73.7%79.8%

Sink-state accuracy is 53.1 and 52.0 percentage points below overall accuracy in Phi-4 and R1, respectively. These gaps show that failures concentrate in the identified sink states. Together with the recurring expansion–compression pattern, these extend our observation to bigger and different models. One of the other contributions of HSS is that it also reveals differences in the behavior associated with these structures instead of just AUROC. In Phi-4, failures in the low-accuracy region often involve completed but incorrect answers. In R1, long reasoning traces often forget to produce a final boxed answer. HSS makes these differences visible in the state map, connecting geometric regions to concrete behaviors such as answering incorrectly or failing to finish.

Thank you for addressing my concerns. I will maintain my original score.

Reviewer_G7Q8OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

HSS turns frozen language-model activations into discrete state maps. It clusters each layer separately, then matches clusters across adjacent layers to assign shared state identities. Map construction does not use behavioral labels; these are added later for analysis and prediction. Prediction uses prompt-last representations, while monitoring uses averages over generated prefixes.

The experiments examine clustering robustness across models and datasets, predict correctness and safety, and track failure-associated states. The maps also support warnings during generation.

Reconstruction and removal tests examine whether local regions retain directions that matter to model outputs. An exploratory steering study edits a direction associated with failed responses.

Critical strengths
  1. The layer-wise geometric patterns recur across seeds, sample sizes and clustering choices. Tests on several models and datasets also show where the patterns agree and where they differ.

  2. The count-based predictors perform competitively with trained probes. The maps are fitted on training data, and the prediction and monitoring splits keep responses separate.

  3. At the tested layer, eight local directions preserve outputs better than eight shared directions. Removing the local directions changes outputs more than matched random removals.

    The paired intervals support both results. The paper also makes clear that the local bases require more storage and that a larger shared basis preserves outputs better.

Critical weaknesses
  1. The paper states stability for alignment thresholds of 0.3–0.7, but gives little detail on state lifetimes or sink links across this range. The cross-layer persistence claim needs clearer support.

  2. The improvements in detection and saved tokens over the continuous probe are small, and the reported AUROC is lower. Paired intervals would clarify the evidence for a monitoring advantage.

  3. A minor clarification concerns the alignment-invariance claim. As written, global-vocabulary smoothing can affect default classification decisions when class counts differ. This does not affect AUROC or monitoring after threshold recalibration.

Questions for authors
  1. For the stated stable range of 0.3–0.7, can you show how state lifetimes and the low-accuracy sink's cross-layer links vary?

  2. Under the stated global-vocabulary smoothing, is the default classification decision also invariant to alignment? AUROC and recalibrated monitoring appear unaffected.

  3. Can you provide response-level paired intervals for detection and saved-token differences at the validation-calibrated thresholds?

1 comment
AuthorsOpenReview ↗

Response to Reviewer G7Q8

W1 / Q1

Question 1: For the stated stable range of 0.3–0.7, can you show how state lifetimes and the low-accuracy sink's cross-layer links vary?

Response:

We thank the reviewer for checking threshold details. State persistence is largely insensitive to the alignment threshold. This is because the cluster centroids are close by cosine similarity; for the Qwen2–MATH map, the threshold does not disconnect any matched pairs, thus not affecting any state lifetimes. It is worth mentioning that, despite the high cosine similarity between centroids, Hungarian matching still works very well, as evidenced by the self-transition rate. On average, 60% of the samples remain in clusters with the same ID across consecutive layers. However, the threshold will still play a role in avoiding matches between two clusters with very different centroids. Below, we sweep η on Qwen2–MATH with fixed clusters, reporting mean state persistent layer ranges, self-transition rate, and the span of the same sink chain across thresholds.

Alignment threshold ηMean state lifetime (layers)Self-transition rateSink layer span
0.309.3056.7%2–27
0.509.3056.7%2–27
0.709.3056.7%2–27
0.956.0155.0%2–27
0.991.022.0%16–18

W2 / Q3

Question 3: Can you provide response-level paired intervals for detection and saved-token differences at the validation-calibrated thresholds?

Response:

We thank the reviewer for emphasizing the statistical rigor and evaluated five response-level train/validation/test splits (40/20/40), calibrating thresholds to strict 10% validation FAR. Below, we report paired differences against the logistic probe with 95% CIs from 2000 response-level bootstrap resamples. These results do match the paper's takeaways for HSS. We find it encouraging that HSS performs surprisingly well despite its coarse discretization. The state map is constructed without labels; nevertheless, it retains behaviorally relevant information that supports useful failure prediction.

SplitΔ Detection (HSS-NB - Logistic Probe)Δ Saved token
1+8.96 [5.21, 12.69]+5.80 [+3.11, +8.45]
2+6.96 [3.29, 10.36]+7.60 [+4.89, +10.19]
3−6.48 [−9.76, −3.10]−6.90 [−9.36, −4.16]
4+4.00 [0.29, 7.58]+9.20 [+6.61, +11.75]
5+9.15 [5.69, 12.98]+5.80 [+3.10, +8.60]

W3 / Q2

Question 2: Under the stated global-vocabulary smoothing, is the default classification decision also invariant to alignment? AUROC and recalibrated monitoring appear unaffected.

Response:

We thank the reviewer for checking this imprecision in our paper presentation. We carefully checked the implementation used in our codebase and confirmed per-layer smoothing over KℓK_\ell states. In that case, alignment changes neither the counts nor the smoothing denominators, so both scores and default decisions remain unchanged. We will clarify this definition in §5.1 and Appendix A.5.

We also appreciate the discussion of global-vocabulary smoothing. Here, the reviewer raises a good point: when class counts differ, alignment-dependent changes in vocabulary size shift all samples’ log-odds by a constant. AUROC and monitoring after threshold recalibration remain unchanged, but default decisions might change. So we further compared both definitions using the same clusters and data splits. On Qwen2–MATH, AUROC was identical across all ten splits, while 4–15 out of 3,000 test predictions per split changed at the default threshold, confirming the reviewer’s observation.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response to All Reviewers and the Area Chair

We thank all reviewers for their thoughtful and constructive feedback. We are encouraged that reviewers found HSS simple, intuitive, and practical, found the state maps easy to interpret, and saw the editing experiments as evidence that the states are "not just descriptive". Below we restate our contributions and summarize how each main concern was addressed.


Summary of Contributions

  • A dynamic, population-level view of hidden states. Without labels or training, HSS provides a macroscopic account of how hidden states are organized and evolve across layers.
  • Consistent geometry linked to behavior. Across models and clustering choices, states expand in intermediate layers and compress in later layers in model-specific patterns, while failures concentrate in persistent sink states tied to concrete, interpretable behaviors.
  • Simple states that remain predictive and functional. HSS retains predictive information after reducing activation to a single discrete state: simple counts approach trained probes and support early warning. Editing tests further confirm that these states causally affect the model's output.

Addressing Major Concerns

1. Statistical reliability and generalizability (E6Ez, sgsy, G7Q8). The additional experiments suggested by the reviewers make the paper's evidence more convincing: the results hold under repeated splits and beyond the original models. Over ten shared splits on all eight Table 1 settings, HSS-NB stays above the closed-form baselines and approaches the trained continuous probes, with no significant difference from the tuned MLP. The same patterns appear on Phi-4 (14B) and DeepSeek-R1, and steering generation out of the failure sink state still works on a larger problem set.

2. Novelty (sgsy). CLUE uses labels to build two centroids for classification, and Shafran et al. focus on static, local activation decomposition. HSS is a simple yet effective pipeline, which we view as a strength, and it provides a new label-free abstraction of hidden-state dynamics.

3. Usefulness of alignment (E6Ez, G7Q8). HSS-NB serves as the simplest demonstration that the states are informative, and it does not require alignment. Alignment is what makes the map interpretable: state persistence, transitions and the sink states underlying our analyses and steering experiments are all defined through it. It is also behaviorally grounded, linking clusters that behave alike (r = 0.83–0.93, versus about 0 for random pairing), so that a failure region can be followed across layers.

All changes in the revised PDF are marked in blue. We hope this response, together with our individual replies, clarifies our contributions and resolves the concerns raised, and we remain available for further discussion.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.