- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposed chunk-level SAEs as an alternative to a known problem with token-level SAEs: reconstructing one token at a time pushed features toward surface characteristics instead of meaning. The authors used mean-pooled activations over different sizes of token chunks and trained 3 variants: Mean-Chunk (reconstruct the chunk mean), Cross-Chunk (predict a same-document neighbor’s mean) and Joint-Chunk (both). The authors evaluated fidelity, LLM-judged interpretability, retrieval, reasoning detection, classification and steering.
Critical strengths
-
The paper addressed a known limitation in a widely used technique.
-
The experimental design is sound: Mean- vs. Cross-Chunk separated the effect of pooling from the effect of the prediction target. All SAEs shared the same model, data, width and sparsity, which provided a good global control.
-
The evaluation included many controls: low word overlap and length-matched candidates for retrieval, frozen features and held-out cue-free/cue-only rewrites for reasoning, a matched-pooling audit, and an α sweep.
Critical weaknesses
-
The figures were really hard to read. Some examples (but there are many more): a) The two oranges in Fig. 1 look exactly the same under the red-shaded hexagon filter. b) The labels in Fig. 6 almost overlap, and the numbers were hard to read and rotated 90 degrees to the side. c) The traces in Fig. 4 are way too small to follow. d) Fig. 7A has no error bars, while its 0-1 scale made the small differences between methods hard to see. As presented, the results are difficult to verify.
-
Key hyperparameter and experimental design choices were not justified. Some examples (again, there are more): a) The paper is about changing the SAE's observation unit, but the unit itself was never studied. The chunk lengths {32, ..., 512} were justified only as "to cover different context scales" and mixed into one run, with no results by length. I urge the authors to explain why these lengths were chosen and to back the choice empirically. b) The same applies to the comparison: retrieval ranked by cosine, the dense baseline was raw hidden states with no standardization mentioned, and token codes were simply averaged. But the paper’s own Appendix showed that re-pooling moved BatchTopK’s backtracking results by 24-31 points. It was not clear how much of the gap came from the dictionaries rather than from mean-pooling plus cosine.
-
The evaluation was mostly built around the method. All evaluations are custom, and there are no standard benchmarks (e.g., SAEBench, which the paper cited). The training objective matched the test: Cross-Chunk was trained on same-document neighbors, then evaluated on same-document retrieval and persistence. The reasoning test was built around the baselines: the cue-only negatives kept the token SAEs’ anchor words. Interpretability rested on one LLM judge with no human check and no chance baseline. Random guessing would score roughly 43-50 on InterpScore, and the explainer prompt asked for topic-style descriptions, which favored chunk features.
-
All experiments used a single model and a single layer.
Questions for authors
Clarifications:
- Why were these chunk lengths chosen? How did the results change as a function of length?
- What similarity and normalization did the dense baseline use?
- How were the explainer and scoring passages chosen, and were the scoring passages held out?
- How many training seeds were used per SAE?
Experiments that could change the assessment: 5. Retrieval with standardized hidden states for the dense baseline, and with max-pooled or IDF-weighted codes for the token-level SAEs. 6. A chance-level and random-model InterpScore baseline, plus a small human check of the judge. 7. Standard benchmarks where applicable (relevant SAEBench metrics, AxBench as-is), or an explanation of why these benchmarks cannot be used.
If the gap survived standardized baselines and the chunk-length choice were given a rationale and empirical support, a higher score would be warranted.
8 comments
2, Public Tasks, Generalization and Evaluation Protocols(W3、W4、Q1、Q3)
W3: Validate on public tasks beyond the custom evaluations.
To test sparse semantic access without generated examples or an LLM judge, we use AG News's public topic labels (Character-level Convolutional Networks for Text Classification, NeurIPS 2015). Following SAEBench sparse probing, four balanced topic-versus-rest tasks use 4,000/1,000 official train/test documents. k denotes training-selected coordinates, and entries are mean accuracy.
| Representation | k=1 | k=2 | k=5 |
|---|---|---|---|
| Token | 74.00% | 76.90% | 85.15% |
| Temporal | 74.10% | 79.65% | 84.70% |
| Mean | 74.00% | 80.60% | 88.20% |
| Cross | 82.80% | 86.10% | 91.10% |
| Joint | 82.70% | 85.00% | 89.75% |
| Standardized dense | 80.75% | 84.10% | 88.15% |
At five coordinates, Cross exceeds standardized dense by 2.95pp after correction over 180 tests. To distinguish dictionary learning from pooling or random expansion, we also compare pooling-only Token (84.75%) and same-width random dictionaries (87.78±0.90%). Unrestricted dense classification reaches 93.60%.
To independently test relations in natural text, we use human-annotated DiscoGeM (DiscoGeM: A Crowdsourced Corpus of Genre-Mixed Implicit Discourse Relations, LREC 2022). With frozen dictionaries and document-disjoint splits, five-coordinate classifiers distinguish reason/result relations from other relations. AUROC measures positive–negative ranking, with 0.5 indicating chance.
| Method | Token | Temporal | Mean | Cross | Joint |
|---|---|---|---|---|---|
| DiscoGeM AUROC,k=5 | 0.5605 | 0.5585 | 0.6092 | 0.5942 | 0.5858 |
All three chunk methods exceed both token-level baselines.
Public classification and human-annotated discourse both demonstrate useful semantic information accessible through very few chunk-feature coordinates. We think these evaluations address dependence on custom tasks and support the paper's sparse-access conclusion.
W4: Replicate across model families and layers.
To test whether the findings depend on one model and layer, we trained all five methods for 1B at another Qwen layer and on Gemma-3-12B. We reuse AG News because the same five-coordinate topic task compares semantic accessibility across representations.
| Model/block | Token,k=5 | Temporal | Mean | Cross | Joint |
|---|---|---|---|---|---|
| Qwen-9B/21 | 85.15% | 84.70% | 88.20% | 91.10% | 89.75% |
| Qwen-9B/15 | 80.30% | 86.55% | 88.50% | 89.70% | 89.30% |
| Gemma-12B/31 | 82.20% | 86.95% | 88.45% | 89.65% | 89.10% |
All nine Chunk-versus-Token gains pass correction. To check Joint's dual-target benefit across these representations, we also compare neighbor reconstruction on common targets: Joint explains 2.96–4.21 percentage points more target variance than Cross.
Sparse-classification gains and Joint's retention of neighboring-passage information extend across layers and model families.
Q1: Why these chunk lengths, and how does performance vary with length?
The 32/64/128/256/512 doubling grid covers a 16-fold context range. To identify each length's contribution, we split retrieval results from the mixed-length checkpoints.
| Query tokens | Token R@5 | Cross R@5 |
|---|---|---|
| 32 | 50.00% | 73.23% |
| 64 | 62.09% | 77.23% |
| 128 | 66.60% | 78.86% |
| 256 | 72.30% | 81.44% |
| 512 | 75.64% | 86.22% |
Cross's gain has a positive confidence interval at every query length.
To isolate added context from candidate difficulty, we fixed the same 249 documents and all 249 candidates.
| Real tokens | Standardized dense R@5 | Cross cosine | Cross paper CL-SR |
|---|---|---|---|
| 32 | 58.43% | 64.06% | 68.07% |
| 64 | 70.48% | 76.51% | 77.11% |
| 128 | 82.53% | 81.73% | 85.94% |
Cross CL-SR gains 17.87pp from 32 to 128 tokens.
To test training-length effects, we retrained Mean/Cross for 100M tokens using fixed 512-token chunks. We use CSFCube to test fine-grained semantic matching between papers (CSFCube—A Test Collection of Computer Science Research Articles for Faceted Query by Example, NeurIPS Datasets and Benchmarks 2021). Compared with mixed-length training, top-ten ranking quality (nDCG@10) improves by approximately 16.3%/19.3%, respectively.
The length choice has empirical support: gains occur across the grid, and added context improves retrieval even with documents and candidates fixed.
Q3: How are explanation examples selected, and are scoring documents held out?
To rule out reuse of explanation evidence during scoring, we separate generation and scoring documents and verify their hashes globally and per feature. Prompt comparisons reuse the same examples, with method identities and activation labels hidden.
| Stage | Examples | Document separation |
|---|---|---|
| Explanation generation | 2 highest-activation examples and 2 activation-weighted samples | 4 development documents |
| Scoring | 4 active and 4 inactive contexts, each 128 tokens | Held-out documents, hash-verified |
| Coverage | 4,999 features and 9,998 judgments | 1 unsupported Joint feature marked NA |
Lexical explanations are allowed, so descriptions can match the feature's actual level of abstraction.
Explanation development and scoring use separate documents, with verified separation at both global and feature level.
3, Reproducibility, Judge Calibration and Standard Benchmarks(Q4、Q6、Q7)
Q4: How many training seeds were used per SAE?
Our original multi-seed evaluation used three seeds per SAE at 100M. To test reproducibility at the main 1B budget, we also report three full-budget seeds, 42/43/44, for Token, Mean and Cross, each with 31,250 updates and validation checkpoint selection.
We reuse AG News because its fixed labels and five-coordinate classifier make training variation directly comparable without judge variation.
| Method | Seed 42,k=5 | Seed 43 | Seed 44 | Mean±SD |
|---|---|---|---|---|
| Token | 85.15% | 83.95% | 84.65% | 84.58±0.60% |
| Mean | 88.20% | 90.00% | 89.55% | 89.25±0.94% |
| Cross | 91.10% | 90.90% | 89.25% | 90.42±1.02% |
Mean/Cross average 4.67/5.83pp above Token, with positive training-seed confidence intervals. Label-subset sampling seeds are counted separately from SAE training seeds.
The core Mean/Cross sparse-classification gains reproduce across independent full-budget training seeds. This provides the requested seed accounting and evidence of training reproducibility.
Q6: Calibrate InterpScore with chance/random baselines and a human check.
To assess the reliability of automated interpretation, we add chance and random-dictionary baselines and a small human check (100 features, use the same rating panel, and the final result is the average of the assessments made by the seven individuals.). InterpScore averages the per-feature harmonic mean of active-context recall and inactive-context rejection on a 0–100 scale.
For the random-dictionary controls, we evaluate 100 features per method, matching each SAE's activation sparsity using training data and reusing cached contexts. Both explanation generation and scoring use the same LLM as in the paper. The chance baseline is the exact expected score when independently guessing each context's activation with probability 0.5 on the four-active/four-inactive scoring panel.
| Evaluation | Token | Temporal | Mean | Cross | Joint |
|---|---|---|---|---|---|
| Chance-level InterpScore | 42.9 | 42.9 | 42.9 | 42.9 | 42.9 |
| Random-dictionary InterpScore | 37.8 | 35.8 | 52.5 | 55.8 | 53.8 |
| Paper InterpScore | 53.0 | 53.0 | 64.0 | 67.0 | 54.1 |
| Human InterpScore | 58.4 | 60.5 | 69.2 | 65.0 | 62.7 |
Mean and Cross exceed their matched random-dictionary controls by 11.5pp and 11.2pp. Human assessment also places both above Joint, consistent with the automated evaluation within the chunk-level family. We compare each method against its own controls, without treating scores across different observation units as an interchangeable measure of interpretability.
The human check corroborates the method-level conclusions of the automated evaluation, with Mean and Cross leading the chunk-level family under both assessments.
Q7: Which standard evaluations are appropriate for chunk SAEs?
-
Token-oriented metrics require care when the observation unit changes. We considered SAEBench's metrics, which were designed around token-level SAE activations. Applying them unchanged to chunk-level activations does not give a like-for-like comparison. For example, an explainer can readily infer “apple” from repeated highly activating tokens, while finding a common pattern across variable-length chunks is a substantially harder inference problem. An unchanged standard autointerp score therefore cannot directly rank the two levels. This is why our sparse-probing adaptation uses a common passage task and coordinate budget, while interpretation evaluates whether chunk features remain faithful tools.
-
Steering supports a common output-level comparison. Following your suggestion, we use AxBench because all methods can be assessed on the same generated outputs and concept goals (AxBench: Steering LLMs? Even Simple Baselines Outperform Sparse Autoencoders, ICML 2025). We adapt its public concepts and three official scoring templates to Qwen, using common prompts, scorers and scaling rules, with separate strength-selection and test prompts.
| Shared scaling factor 1 | Token | Temporal | Mean | Cross | Joint | Unsteered | CAA | Prompt |
|---|---|---|---|---|---|---|---|---|
| Held-out harmonic score,0–2 | 0.288 | 0.324 | 0.418 | 0.424 | 0.398 | 0.293 | 0.316 | 1.114 |
Factor 1 multiplies each concept's predefined intervention magnitude. CAA uses activation differences, while Prompt uses explicit text instructions. Cross achieves the highest score among the activation-intervention methods tested here. Prompt steering remains strongest, consistent with AxBench's finding. Mean/Cross/Joint also exceed the unsteered baseline.
Chunk SAEs remain interpretable tools and provide the strongest activation-steering result in this comparison. This supplementary experiment is consistent with the conclusions drawn from the experiments in our original paper.
Thank you again for your careful review and for reconsidering our paper!
1, Figure Clarity and Controlled Comparisons(W1、W2/Q2/Q5)
We are grateful for the reviewer’s constructive feedback, which helped us improve clarity! We have added targeted experiments that directly address each of your concerns and provide the evidence you requested.
To address each presentation issue, we revised all four figures. Please see the revised manuscript for the complete figures.
The revised figures make the comparisons and their uncertainty directly readable. We have made the revisions in the latest version of the paper as per your request.
To test dictionary quality at equal context, we froze all SAEs and reran the language model on exactly 32 real tokens on both sides of 1,736 documents, using identical 16×32 batches. Every query searches all 1,736 candidates, without variable padding or access to later tokens.
To address normalization and aggregation, dense averages are standardized using training-set means and standard deviations. Token/Temporal use mean or max pooling, optionally weighted by inverse document frequency, IDF, to downweight common features. Cross top128 cosine retains the 128 largest activations. R@5 is the fraction of correct paired passages found in the top five.
We additionally apply CL-SR because it tests the same dictionary with an established concept-based retrieval score (Decoding Dense Embeddings: Sparse Autoencoders for Interpreting and Discretizing Dense Retrieval, EMNLP 2025).
Cross exceeds standardized dense by 3.02pp with cosine and 6.65pp with paper CL-SR, both passing correction over 168 tests. Independent 1B training repeats average 2.82/6.24pp gains under the cached 32-token protocol.
To check the original mixed-length comparison under stronger baselines, we added training-fitted centering, per-dimension standardization and feature IDF. The original dense baseline used valid-token means and cosine, comparing L2-normalized vectors.
Cross cosine reaches 77.02%, exceeding Token/Temporal mean+IDF by 2.42/2.07pp. Applying paper/repository CL-SR to the same Cross dictionary gives 81.68%/81.42%, both above standardized dense after correction. Index statistics use no relevance labels.
To isolate target learning from mean pooling, we also compare Cross with Mean under identical pooling. Cross gains 10.71pp in linking R@5. Complete results retain all 35 scoring configurations and distinguish full pooled codes from historically storage-truncated codes.
Cross's learned dictionary improves retrieval under equal text length, a common full candidate set and strengthened dense/token baselines.