acceptodds
Under review as a conference paper at ICLR 2027

Different Features, Reproducible Responses: Sparse Autoencoders Across Model Changes

Abstract

Sparse autoencoders (SAEs) represent model activations using sparse combinations of learned features. Each learned feature has a latent activation; features are often compared across dictionaries by matching these latents. We ask whether this is sufficient to assess the reproducibility of an observed activation response. Across independently fitted sparse dictionaries before and after model adaptation, compact combinations recover responses that no affine single-latent predictor recovers to the same accuracy. The singleton comparator searches the entire target dictionary and optimizes directly on evaluation inputs; group memberships and coefficients are fixed beforehand. We compare three ways to select groups: unbalanced optimal transport (UOT), descriptor cost, and calibration-response correlation. Across TinyLlama and Qwen3, with five fits per endpoint, compact groups increase recovery across all target fits. Token responses and unchanged-model refits show the same qualitative separation, and simple cost-selected groups outperform UOT groups. These findings distinguish the recoverability of an observation from the reappearance of an individual latent. Cross-dictionary references can retain compact realizations and their measured accuracy, allowing response evidence to accumulate without a canonical feature vocabulary.

Primary area:
learning theory
Keywords:
sparse autoencoders, mechanistic interpretability, feature correspondence, reproducibility, model adaptation, unbalanced optimal transport

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_3GLLOpenReview ↗
Rating
1(−1)
Confidence
3
Informativeness
1
Presentation
1(−1)
Soundness
2
Read review
Paper summary

The paper investigates whether activation responses observed in one sparse autoencoder remain recoverable across independently fitted dictionaries, including before and after model adaptation. It compares predictors using small groups of target latents against an oracle that selects and affinely fits the best individual target latent directly on evaluation data. Experiments on TinyLlama and Qwen3 show that compact groups recover some responses beyond this singleton ceiling. The authors compare unbalanced optimal transport (UOT), descriptor-cost selection, and activation-correlation selection; simpler selectors achieve higher recovery than UOT.

Critical strengths

S1. The affine-singleton oracle provides a strong comparator. Because it searches the entire target dictionary and optimizes on evaluation inputs, successful group predictors demonstrate a limitation of affine single-latent prediction on those inputs that cannot be explained by an unsuccessful matching algorithm.

S2. Independent SAE fits, unchanged-model refits, and separately calibrated group predictors support the central empirical finding. The authors also transparently report that simpler selectors outperform UOT and carefully distinguish response recovery from semantic or causal equivalence.

Critical weaknesses

W1. Limited novelty. SAE feature matching across random seeds has already been studied extensively in [1], which compares several matching approaches, including optimal transport, and evaluates them through functional tests. Moreover, [2] already shows that unstable individual SAE features can span reproducible subspaces. The present work extends these ideas to fine-tuning and compact activation reconstruction, but the overall contribution appears incremental.

W2. Limited value of the proposed method. Despite the substantial development of unbalanced optimal transport, simple descriptor-cost and activation-correlation baselines consistently outperform it. The paper therefore provides little evidence that its more sophisticated matching procedure is necessary or beneficial.

W3. Limited functional validation. The main result shows that combining several target features improves activation reconstruction. However, improved reconstruction does not necessarily imply semantic or causal equivalence. Compared with the intervention-based evaluation in [1], the paper provides limited evidence that these recovered groups represent functionally corresponding features.

[1] Benchmarking cross-seed feature correspondence in sparse autoencoders. 2026. https://openreview.net/forum?id=5cy6WtSC8f

[2] Gerasimov et al. Unstable features, reproducible subspaces: Understanding seed dependence in sparse autoencoders. 2026. https://arxiv.org/abs/2606.12138

Questions for authors

Q1. How does the proposed approach advance beyond the matching methods already studied in [1] and the reproducible-subspace findings in [2]? Could the authors clarify what fundamentally new insight is established by compact response recovery?

Q2. Given that simple descriptor-cost and activation-correlation matching consistently outperform UOT, what practical advantage does the proposed transport formulation provide? Are there settings where UOT performs better?

Q3. How does the proposed method compare with simple greedy forward selection or sparse regression using the same four-feature budget? Could these baselines explain most of the observed improvement over singleton matching?

Q4. Do the recovered feature groups preserve causal effects under activation substitution or ablation, as evaluated in [1]? How can we establish that successful activation reconstruction corresponds to meaningful functional equivalence?

7 comments
AuthorsOpenReview ↗

Rebuttal to 3GLL — Part 1/5

Reply to 3GLL — Part 1/5

Thank you for highlighting the strength of our evaluation-fitted singleton comparator (S1) and the distinction between response recovery and causal equivalence (S2). The new capability is to reuse a specified activation measurement across existing, independently learned dictionaries even when no affine single-feature predictor can recover it adequately. Prior findings of feature instability and shared decoder subspaces leave open whether the same input-dependent response remains accessible through a few target activations on held-out inputs. Establishing that it does gives researchers a way to carry numerical observations across SAE fits and model adaptation without requiring a shared feature vocabulary or jointly retraining the dictionaries.

Across five base and five adapted SAEs per model, descriptor-cost predictors using at most four target features raise recovery in all five target dictionaries from 19.3% to 33.3% on TinyLlama and 15.0% to 25.2% on Qwen3 at NSE≤0.25\text{NSE} \le 0.25. Requiring an adequate individual feature therefore excludes a substantial set of measurements that remain recoverable across all five target dictionaries. For these measurements, reproducibility does not require the same individual-feature representation to recur. These predictors use existing dictionaries, are fitted on separate calibration data, and are fixed before evaluation; the singleton comparator is fitted on evaluation data. Among responses that both methods recover in a common target fit, the groups also nearly halve the failure rate in the other target fits.

W1 & Q1: Retaining reproducible response measurements beyond [1] and [2]

With the target transformer held fixed, an affine single-feature predictor can meet the error threshold in one SAE fit and fail in another. Calibrated groups recover additional responses across the target fits. Standard sparse regression reproduces this gain (OMP and Lasso, Q3). Fitting activation weights also improves recovery over the tested decoder-projection weights on the same four target features. [1] benchmarks pairwise matching through functional interventions and visualizes split/merge plans; [2] identifies reproducible decoder subspaces under basis ambiguity. We measure whether a specified source response remains accurately predictable across independently learned target dictionaries (Section 2, p. 2).

We test this on existing SAEs across mutually disjoint 1,998-document descriptor, 998-document calibration, and 7,600-document evaluation splits. Holding each base-SAE response and the fine-tuned model fixed across five target SAE seeds, the singleton comparator searches all 4,096 targets and fits signed scale and offset directly on evaluation text. On those evaluation responses, this oracle upper-bounds every 1-to-1 matcher restricted to predicting from a single target latent with an affine map, however it selects that partner. Where a calibration-frozen group (≤4\le 4 latents, fitted on the 998 calibration documents) passes NSE≤0.25\text{NSE} \le 0.25 and this oracle fails, at most four target coordinates recover the response where no affine singleton does (Section 3.1):

Document-response result at NSE≤0.25\text{NSE} \le 0.25TinyLlama-1.1BQwen3-1.7B
Singleton passes in at least one target fit and fails in another35.8% (7,332 / 20,474)28.4% (5,665 / 19,972)
Evaluation-fitted singleton passes in all five fits19.3%15.0%
Calibration-frozen descriptor-cost group (≤4\le 4) passes in all five fits33.3%25.2%
Conditional on row 1 (seed-dependent cases), cost groups pass in all five fits36.3% (2,660 / 7,332)35.1% (1,987 / 5,665)
Matched-anchor non-replication in Bs≠rB_s \ne r (both pass in BrB_r): singleton →\to cost group23.5% →\to 12.1%23.9% →\to 12.8%

Pooled across five base SAEs (each target fit uses a separately calibrated, frozen predictor). Rows 1–3 use all eligible records; row 4 conditions on row 1 (UOT: 32.2%, 30.9%). Row 5 shows that on the same 37,577 TinyLlama and 28,074 Qwen3 (source, BrB_r) positive anchors, four-feature recovery nearly halves failure across the four other seeds Bs≠rB_s \ne r (150,308 and 112,296 ordered comparisons; using each method's own BrB_r passes gives 23.6% →\to 15.8% and 24.1% →\to 16.9%). On the fixed set passing ≥1\ge 1 singleton seed, five-seed mixed outcomes fall from 65.0% to 41.9% and 65.4% to 43.4% (5/5 rising 35.0% →\to 57.7% and 34.6% →\to 55.7%, with only 0.5% and 1.0% falling to 0/5), while unconditional mixed rates are 35.8% →\to 32.4% and 28.4% →\to 27.1% as 0/5 singleton failures enter partial recovery.

AuthorsOpenReview ↗

Reply to 3GLL — Part 2/5

W1 & Q1 — continued

Two researchers testing whether a source activation response stays reproducible on the same fine-tuned checkpoint can reach opposite verdicts from an exhaustive 1-to-1 adequacy test solely because they trained different target SAE seeds. Among the responses rescued across all five target fits by four-feature cost groups where the singleton oracle fails in at least one seed (2,975 on TinyLlama, 14.5% vs. 0.5% in reverse; 2,203 on Qwen3, 11.0% vs. 0.8% in reverse), 89.4% and 90.2% pass as a single feature in at least one target seed yet fail the singleton oracle in another seed where a calibrated four-feature group succeeds.

Per-seed singleton pass rates are nearly flat across B0–B4B_0\text{–}B_4 (36.9%–37.4% on TinyLlama; 28.1%–29.2% on Qwen3), so no tested seed consistently eliminates this dictionary dependence. Where a singleton passes (as in B1B_1 in Figure 1), no group is needed; recording each dictionary's fitted predictor and held-out error (such as two latents in B3B_3) retains responses that a singleton-only criterion would miss, without a canonical vocabulary (Abstract; Section 6). Because calibration fits the source response, a successful predictor shows that a few target features can estimate that response on the tested distribution. It does not tell us whether native activation amplitudes changed or whether causal circuits were preserved.

Using the same four target features, fitting weights to activations improves held-out response recovery over the tested decoder-projection weighting rule (Reply to HrVG, Table R1). On the four nearest target decoder directions, static projection (calibrated scalar gain and offset) recovers 26.0% on TinyLlama and 15.9% on Qwen3 (barely above Qwen3's 15.0% exhaustive singleton oracle); calibrating those four coefficients on activations raises five-dictionary recovery to 32.8% and 24.0%. Decoder geometry identifies where to look; activation calibration sets the relative weights that carry to held-out text. On recovered source–target pairs, four-step nonnegative OMP places over 72% of its fitted weight inside each source's eight-nearest descriptor-cost set S8S_8; excluding S8S_8 drops OMP recovery to 4.4% and 3.6%, while prevalence-matched random groups and positively correlated near-orthogonal targets recover little.

Same-checkpoint, threshold, and split controls: Holding the checkpoint fixed and varying only the SAE seed (Section 5.3, p. 7; Appendices A.2–A.4, e.g., 25.7% singleton vs. 39.0%–42.2% four-feature recovery across B0→B1–B4B_0 \to B_1\text{–}B_4 documents) produces the same separation, isolating dictionary re-estimation from fine-tuning. Requiring singleton error ≤0.20\le 0.20 in one target fit and ≥0.30\ge 0.30 in another still flags 22.2% of TinyLlama and 16.0% of Qwen3 records (over 31% recovered across all five fits by frozen cost groups); evaluating descriptor-free predictors (decoder cosine, correlation, OMP, Lasso) on the unused 1,998 descriptor documents reproduces the separation alongside our token (Table 1) and 0.15–0.35 cutoff (Figure 2) checks.

W3 & Q4: Cohort-level causal interventions (activation substitution and ablation)

Although fitted only to activations rather than downstream logits, four-feature groups selected by decoder cosine or descriptor cost cut held-out substitution squared logit error by 36%–39% and weighted-ablation shift error by 30.5%–34.7% relative to matched singletons across all four target seeds on our class-selective cohort. To isolate dictionary correspondence from downstream fine-tuning, we compare reseeded SAEs on a fixed TinyLlama-1.1B checkpoint (B0→B1–B4B_0 \to B_1\text{–}B_4, 256 evaluation documents), following [1]'s fixed-model setting. Our primary cohort C30C_{30} comprises the 30 most class-selective B0B_0 features (tie-averaged calibration AUROC) among the 2,987 active in ≥20\ge 20 calibration documents whose best calibration singleton fails in at least one target seed (0.188 zero-ablation margin MAE).

AuthorsOpenReview ↗

Reply to 3GLL — Part 3/5

W3 & Q4 — continued

Candidate selection uses splits disjoint from the 256 evaluation documents (unit-decoder cosine for closest decoder, 1,998 descriptor docs for UOT/cost, 998 calibration docs for correlation); every k=1k = 1 and k=4k = 4 arm freezes 998-doc calibration weights before evaluation and never fits downstream logits. Document-calibrated slopes apply tokenwise without an intercept, so inactive targets contribute zero. Because the patched vector sums all 30 source-to-group terms, these tests evaluate cohort-level readout fidelity rather than per-feature causality. For Layer-11 substitution (Table R6), we subtract FB,0(x)F_{B,0}(x), add F^B,r(x)\widehat F_{B,r}(x), and run the remaining layers:

h←h−FB,0(x)+F^B,r(x),FB,0(x)=∑i∈CziB0(x) diB0,F^B,r(x)=∑i∈C∑j∈Jirwirj zjBr(x) djBr,h \leftarrow h - F_{B,0}(x) + \widehat F_{B,r}(x), \qquad F_{B,0}(x) = \sum_{i \in \mathcal{C}} z_i^{B_0}(x)\, d_i^{B_0}, \qquad \widehat F_{B,r}(x) = \sum_{i \in \mathcal{C}} \sum_{j \in J_{ir}} w_{irj}\, z_j^{B_r}(x)\, d_j^{B_r},

scoring source-normalized residual vector error, zero-ablation-normalized four-class logit error (= 1.000), and true-minus-best-other margin MAE:

Sq. Vector Error=∑x∥FB,0(x)−F^B,r(x)∥2∑x∥FB,0(x)∥2,Sq. Logit Error=∑d∥ℓsub(d)−ℓorig(d)∥2∑d∥ℓabl(d)−ℓorig(d)∥2.\text{Sq. Vector Error} = \frac{\sum_x \|F_{B,0}(x) - \widehat F_{B,r}(x)\|^2}{\sum_x \|F_{B,0}(x)\|^2}, \qquad \text{Sq. Logit Error} = \frac{\sum_d \|\ell_{\mathrm{sub}}(d) - \ell_{\mathrm{orig}}(d)\|^2}{\sum_d \|\ell_{\mathrm{abl}}(d) - \ell_{\mathrm{orig}}(d)\|^2}.

Table R6: Layer-11 activation substitution on class-selective cohort C30C_{30} (B0B_0 to B1B_1–B4B_4, mean ± SD across 4 target seeds)

Substitution armSq. Vector ErrorSq. Logit ErrorMargin MAE
Zero ablation (remove source cohort, insert 0; reference)1.000 ± 0.0001.000 ± 0.0000.188 ± 0.000
Positive calibration correlation top-1 (k=1k = 1, NNLS)0.982 ± 0.1440.429 ± 0.0790.110 ± 0.006
Descriptor-cost top-1 (k=1k = 1, NNLS)0.772 ± 0.0420.358 ± 0.0920.103 ± 0.009
Closest decoder direction top-1 (k=1k = 1, NNLS)0.768 ± 0.0390.339 ± 0.0680.098 ± 0.003
Closest decoder directions top-4 (k=4k = 4, NNLS)0.663 ± 0.0230.217 ± 0.0340.079 ± 0.003
Descriptor-cost top-4 (k=4k = 4, NNLS)0.661 ± 0.0260.218 ± 0.0190.081 ± 0.004
UOT top-4 (k=4k = 4, NNLS)0.713 ± 0.0220.252 ± 0.0200.086 ± 0.002
Positive calibration correlation top-4 (k=4k = 4, NNLS)0.768 ± 0.0410.294 ± 0.0520.092 ± 0.011
Orthogonal positively correlated top-4 (abs(cos) < 0.10, NNLS)1.586 ± 0.2130.491 ± 0.0370.122 ± 0.004

On C30C_{30}, decoder-selected singletons outperform correlation singletons on substitution, consistent with target decoder direction mattering in residual-stream interventions. Expanding from k=1k = 1 to k=4k = 4 within each local selector lowers squared logit error by 36% (closest decoder) and 39% (descriptor cost), improving all four seeds, whereas correlated near-orthogonal targets (∣cos⁡∣<0.10|\cos| < 0.10) perform worse than k=1k = 1.

Reviewer_HrVGOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper shows that matching individual SAE features across independently trained dictionaries underestimates response reproducibility: compact groups of up to four target features recover source activation responses that no single latent can match, even when the singleton comparator searches the entire target dictionary and optimises directly on evaluation data. Across TinyLlama and Qwen3 with five SAE fits per checkpoint, compact groups consistently improve recovery, with simple cost-selected groups outperforming more complex UOT-based selection. The same pattern holds for token-level responses and same-model refits, indicating the phenomenon is not specific to cross-model adaptation. The paper's key contribution is conceptual: it separates the recoverability of an observed response from the reappearance of any individual feature, arguing that cross-dictionary references can accumulate response evidence without requiring a canonical feature vocabulary.

Critical strengths
  • I like the way the authors cleanly separates two questions that the SAE literature has largely conflated: whether an individual latent feature reappears in another dictionary, and whether an observed activation response can be recovered. The paper makes an interesting empirical observation: sometimes you can recover an SAE activation response using a compact combination of latents, even when no single affine latent gets you there with comparable accuracy.
Critical weaknesses
  1. The paper's finding that 4-feature groups outperform single features may simply reflect the generic advantage of using multiple linear predictors, not any SAE-specific structure. If a target feature is a sum of two correlated latents, only their combination recovers it, without implying semantic "splitting" into dictionaries. This is especially problematic for SAEs, where features from different seeds are likely correlated or entangled. The paper should rule out this trivial explanation by comparing against sparse regression baselines: nonnegative OMP, Lasso (calibration data only), Ridge + Top-k, cosine decoder similarity, and random matched groups.

  2. The paper invests substantial effort in developing UOT; Yet the final recovery results show that simpler alternatives match or outperform it. Section 5.5 further reveals that balanced OT performs comparably to UOT, rejection transport outperforms it, and descriptor cost alone selects better candidates. Most tellingly, when the same UOT group memberships are kept but coefficients are properly calibrated rather than transport-derived, recovery jumps dramatically, suggesting the critical technical step is fitting a small linear predictor on a good candidate set, not the transport machinery itself.

Questions for authors
  • Have you evaluated general-purpose sparse feature selection methods, such as nonnegative OMP or Lasso (fit exclusively on calibration data), to determine whether your UOT and cost-based matching constructions recover structure that arbitrary sparse regression cannot?

  • If a target feature happens to be a linear combination of two correlated latents, a two-feature group will trivially outperform any singleton without implying meaningful semantic decomposition across dictionaries. How do you rule out this explanation, and do you have evidence that the recovered groups reflect SAE-specific representational structure rather than generic statistical predictability?

  • Your ablation in Section 5.5 shows that descriptor cost selection alone outperforms full UOT (33.3% vs 31.0% on TinyLlama), and that calibrated regression coefficients matter far more than transport-derived ones. What specific capability does the UOT machinery provide that cost-based candidate selection plus calibrated regression does not? If the answer is global correspondence structure, can you demonstrate a concrete downstream task or analysis where that global structure is essential and where simpler per-feature selection fails?

7 comments
AuthorsOpenReview ↗

Reply to HrVG — Part 1/5

Reply to HrVG — Part 1/5

Thank you for noting that a few features in an independently trained SAE can recover a source activation response that its best affine single-feature predictor cannot. The new capability is to reuse a specified activation measurement across existing, independently learned dictionaries even when no affine single-feature predictor meets the required accuracy. In the submitted study, groups of at most four features raise recovery across all five target dictionaries from 19.3% to 33.3% on TinyLlama and from 15.0% to 25.2% on Qwen3 at normalized squared error <= 0.25. Requiring an adequate individual feature therefore excludes a substantial set of measurements that remain recoverable across all five target dictionaries. For these measurements, reproducibility does not require the same individual-feature representation to recur. Researchers can therefore carry the same numerical question across SAE fits and model adaptation without a shared feature vocabulary or joint retraining. The requested OMP and Lasso controls test how broadly this capability holds across selection methods; W1/Q1 compares recovery rates and calibration-data needs.

The controls show where these predictors work. Removing nearby decoder directions sharply reduces OMP recovery, and restricting the search to nearby directions helps when calibration data are scarce (Tables R1–R2). The tested untrained dictionaries yield lower recovery rates than learned SAEs, although their poorer reconstruction limits the comparison (Table R4). Correlated local mixtures remain a possible explanation.

UOT lets us test whether coordinating matches across source features helps more than choosing each group locally. The submitted comparison favors local cost selection when prediction weights are freely fitted. New controls identify two settings where coupling helps relative to the uncoupled construction: retaining the matching rule's relative weights, and transferring a source classifier from stored summaries without paired response regression (W2 & Q3).

W1 & Q1: Sparse regression, decoder locality, and calibration sample size

Section, page, and appendix references below refer to the submitted paper; tables prefixed R report rebuttal results.

Using 998 calibration documents, the requested sparse-regression baselines reproduce the recovery advantage over the singleton oracle. Screened Lasso reaches 36.18% / 26.26% all-five recovery versus 33.32% / 25.23% for descriptor cost; unrestricted nonnegative OMP reaches 35.87% / 24.30% (TinyLlama / Qwen3; Table R1 below).

Allowing extra nonnegative predictors cannot worsen the best training fit, and fitting a response with 2–4 features does not establish semantic splitting into named sub-concepts (Section 6, p. 9). We test whether a few target features predict the source response accurately on held-out inputs (Section 3.1): where the evaluation-fitted singleton oracle fails (NSE>0.25\text{NSE} > 0.25 over all 4,096 targets, signed scale and offset), no affine singleton suffices on that corpus, whereas a predictor fitted on separate calibration data recovers it using ≤4\le 4 target coordinates (two frozen features already beat the singleton oracle on documents and tokens in Section 5.2, p. 7, and Table 6 of the submitted paper).

Table R1 evaluates all requested baseline families on document mean activations across all 25 base-to-fine-tuned SAE pairs per model (20,474 pooled eligible base features on TinyLlama-1.1B; 19,972 on Qwen3-1.7B). Except for the explicitly labeled singleton and static-projection rows, each method selects ≤4\le 4 targets per fit (hyperparameters fixed across pairs), fits NNLS weights plus intercept on 998 calibration docs, and freezes them for the 7,600 evaluation docs (Gap = per-pair calibration minus evaluation pass rate in pp).

AuthorsOpenReview ↗

Reply to HrVG — Part 2/5

W1 & Q1 — continued

Table R1: Sparse-regression, decoder-geometry, and locality controls (998 calibration docs, 7,600 evaluation docs)

Selection method (≤4\le 4 target features unless noted)TinyLlama Pair / All-5 %TL GapQwen3 Pair / All-5 %Q3 Gap
Best single feature, evaluation fit (1-feature bound, signed)37.12 / 19.28n/a28.79 / 15.02n/a
Best single feature, calibration fit (signed)35.74 / 17.64+1.6526.93 / 12.52+2.33
Prevalence-matched random 4-feature group0.74 / 0.02+0.040.77 / 0.02+0.09
UOT top-4 + calibration NNLS (submitted paper, Table 1)48.36 / 30.97+1.8537.41 / 23.30+2.27
Descriptor-cost top-4 + calibration NNLS (submitted paper, Table 1)50.15 / 33.32+2.0638.94 / 25.23+2.56
Calibration correlation top-4 + calibration NNLS (submitted paper, Table 1)48.55 / 32.37+4.1237.06 / 23.35+4.86
Closest decoder top-4 + calibration NNLS (no sel. activations)49.97 / 32.76+1.9738.35 / 23.97+2.32
Closest decoder top-4 + static projection (gain + offset)43.49 / 26.02+0.4130.40 / 15.90+0.66
Ridge (λ=10\lambda = 10 on unit-norm cal. activations) top-4 + NNLS49.13 / 32.72+4.5737.16 / 23.15+5.42
Screened nonnegative Lasso (α=0.2cmax⁡\alpha = 0.2 c_{\max}, top-32) top-4 + NNLS52.33 / 36.18+5.4839.87 / 26.26+6.41
Nonnegative OMP (4 greedy steps over active targets) + NNLS52.70 / 35.87+7.3139.02 / 24.30+9.18
Nonnegative OMP excluding 8 closest decoder neighbors + NNLS7.63 / 5.32+4.367.14 / 4.56+5.43
1-feature eval. ceiling excluding 8 closest decoder neighbors4.42 / 3.09n/a4.98 / 3.68n/a
Top-4 positively correlated features with abs(cos) < 0.10 + NNLS5.40 / 3.61+1.645.25 / 3.52+2.13

OMP recovery depends strongly on nearby features: on recovered source–target pairs, unrestricted 4-step OMP places 77.9% (TinyLlama) and 72.8% (Qwen3) of its fitted weight inside each source's directed eight-nearest descriptor-cost set S8S_8 (excluding S8S_8 drops OMP all-five recovery to 4.43% and 3.63%, while restricting OMP to S8S_8 reaches 35.96% and 27.04%). Excluding the 8 closest decoder neighbors, a separate control reported in Table R1, likewise collapses OMP near the outside-neighbor singleton bound, and correlated near-orthogonal targets (∣cos⁡∣<0.10|\cos| < 0.10) and prevalence-matched random groups recover few responses (adding three prevalence-matched random targets to the calibrated singleton adds ≤0.10\le 0.10 points). Neighboring local directions can correlate, so these controls localize useful predictors near the source decoder without proving semantic sub-specialization.

Ranking all 4,096 unit decoders without activation data (Closest decoder top-4 + calibration NNLS) nearly matches descriptor cost with under one-third of OMP's calibration-to-evaluation gap. On those identical four decoder directions, static projection (arg⁡min⁡w≥0∥diA−∑j∈JwjdjB∥22\arg\min_{w \ge 0}\|d_i^A - \sum_{j \in J} w_j d_j^B\|_2^2 with calibrated gain and offset) recovers 26.02% on TinyLlama and 15.90% on Qwen3 (barely above Qwen3's 15.02% singleton oracle), whereas calibrating those four coefficients on activations reaches 32.76% and 23.97% (+6.7 and +8.1 points across fine-tuning; +7.0 and +8.5 points across adapted-checkpoint reseeds Bi→B−iB_i \to B_{-i}). Evaluating descriptor-free predictors (closest decoder, correlation, OMP, Lasso) on the unused 1,998-doc descriptor split (eligibility, selection, and NNLS on the 998 calibration docs only) reproduces the separation (32.2%–36.2% vs. 19.8% singleton oracle on TinyLlama; 23.7%–26.4% vs. 15.5% on Qwen3).

Protocol: All-5 % requires passing (NSE≤0.25\text{NSE} \le 0.25) in every target dictionary with a separately calibrated, frozen predictor per target. Eligibility follows Section 4 (p. 6: ≥20\ge 20 descriptor and ≥10\ge 10 calibration active docs, cal. variance >10−12> 10^{-12}, labels unused; active targets: 4,096 TinyLlama, 4,015–4,045 Qwen3). Ridge (λ=10\lambda = 10 on centered unit-norm calibration activations) passes its 4 largest positive weights to NNLS; screened Lasso runs 12 nonnegative coordinate-descent sweeps at α=0.2cmax⁡\alpha = 0.2 c_{\max} on the 32 most positively correlated active targets (≤0.20\le 0.20 pp change vs. 150 sweeps on A0→B0A_0 \to B_0) and refits the top 4 with NNLS; prevalence-matched random draws 4 targets from the 64 closest in calibration firing rate.

AuthorsOpenReview ↗

Reply to HrVG — Part 3/5

W1 & Q1 — continued

For Q1, varying Ncal∈{64,128,256}N_{\text{cal}} \in \{64, 128, 256\} (5 uniform calibration subsamples scored on all 7,600 evaluation docs) shows where geometric restriction helps (Tables R2a–R2b). Unlike UOT and cost (which use the 1,998 descriptor docs), static top-4 decoder cosine uses zero selection activations and sees the exact same NcalN_{\text{cal}} docs as hyperparameter-free correlation and OMP (and Lasso under its fixed Ncal=998N_{\text{cal}} = 998 penalty rule): at Ncal=64N_{\text{cal}} = 64, dictionary-wide OMP and correlation overfit severely, whereas static decoder cosine (and S8S_8-restricted OMP at 12.76% and 8.04% all-five recovery) holds up until dictionary-wide search catches up between Ncal=256N_{\text{cal}} = 256 and 998998.

Table R2a: TinyLlama-1.1B sample-size sweep (Pair % / All-5 %, mean ± SD over 5 subsamples; Ncal=998N_{\text{cal}}=998 in Table R1)

MethodNcal=64N_{\text{cal}} = 64Ncal=128N_{\text{cal}} = 128Ncal=256N_{\text{cal}} = 256
Best single feature (cal.)20.72±0.54 / 6.71±0.2327.32±0.70 / 10.14±0.4432.07±0.31 / 13.76±0.46
Descriptor-cost top-431.76±1.03 / 14.37±0.6239.09±0.86 / 21.13±0.9145.09±0.22 / 27.54±0.40
Closest decoder top-4 (0-act.)31.70±1.00 / 14.22±0.5439.01±0.82 / 20.93±0.8644.92±0.26 / 27.04±0.38
UOT top-431.11±1.04 / 13.72±0.6538.05±0.84 / 19.96±0.8743.66±0.21 / 25.86±0.33
Pos. cal. correlation top-416.84±1.05 / 5.18±0.5328.12±1.08 / 12.28±1.0038.68±0.52 / 21.73±0.78
Screened Lasso (α=0.2cmax⁡\alpha = 0.2 c_{\max}) top-414.81±1.11 / 4.04±0.4927.22±1.28 / 11.41±0.9840.15±0.64 / 22.88±0.84
Nonnegative OMP (4 steps)10.59±1.00 / 1.37±0.2220.12±1.60 / 4.67±0.8835.27±0.67 / 16.00±0.80

Table R2b: Qwen3-1.7B sample-size sweep (Pair % / All-5 %, mean ± SD over 5 subsamples; Ncal=998N_{\text{cal}}=998 in Table R1)

MethodNcal=64N_{\text{cal}} = 64Ncal=128N_{\text{cal}} = 128Ncal=256N_{\text{cal}} = 256
Best single feature (cal.)12.82±0.62 / 3.77±0.0917.83±0.66 / 5.84±0.2222.57±0.35 / 8.54±0.30
Descriptor-cost top-421.54±1.16 / 9.19±0.6727.75±1.01 / 14.05±0.8333.43±0.22 / 19.27±0.42
Closest decoder top-4 (0-act.)21.41±1.01 / 8.85±0.4927.36±0.95 / 13.33±0.7132.91±0.21 / 18.21±0.37
UOT top-421.10±1.02 / 8.67±0.4626.93±0.94 / 12.94±0.6932.27±0.22 / 17.82±0.38
Pos. cal. correlation top-410.59±0.92 / 3.26±0.3218.24±0.90 / 7.22±0.3927.14±0.50 / 13.70±0.71
Screened Lasso (α=0.2cmax⁡\alpha = 0.2 c_{\max}) top-49.07±0.88 / 2.52±0.2317.24±0.99 / 6.46±0.4127.60±0.58 / 13.86±0.72
Nonnegative OMP (4 steps)5.70±0.62 / 0.66±0.1610.73±1.02 / 1.98±0.3821.42±0.63 / 6.93±0.39

Q2: Correlated local latents, seed-by-seed singleton switching, and non-SAE dictionary controls

Correlated local mixtures are compatible with our results and may explain why the predictors work. They do not, by themselves, establish how often a specified source response remains accurately recoverable from at most four features across independently trained target dictionaries, with coefficients fixed before evaluation. Our experiments measure that prevalence and repeatability. The following analyses examine consistency across SAE fits and compare learned with untrained dictionaries; they do not establish semantic decomposition (Section 6, p. 9).

First, a measurement can have an adequate single-feature predictor in one SAE fit but lack one in another. Holding the fine-tuned checkpoint fixed, this occurs for 7,332 / 20,474 (35.8%) TinyLlama records and 5,665 / 19,972 (28.4%) on Qwen3. Per-seed singleton pass rates remain similar (36.9%–37.4% and 28.1%–29.2% across B0–B4B_0\text{–}B_4), so no tested seed eliminates this dependence on the fitted dictionary.

Small groups recover some of these measurements across every tested fit. Frozen ≤4\le 4-feature cost groups recover 2,660 / 7,332 (36.3%) and 1,987 / 5,665 (35.1%) of these mixed-pass records across all five fits (32.2% and 30.9% under UOT; reverse cases, where the singleton passes in all five fits but the group does not, are 0.5%–1.3% of pooled eligible sources). Table R3 breaks down all responses recovered by groups across all five fits but not by singletons.

We also ask whether success in one reference fit carries over to other fits. On 37,577 TinyLlama and 28,074 Qwen3 (source, BrB_r) cases where both the singleton and cost group pass in reference fit BrB_r, failure across the four other seeds Bs≠rB_s \ne r falls from 23.5% to 12.1% and 23.9% to 12.8% when using groups. Conditioning instead on each method's own reference-fit passes gives 23.6% →\to 15.8% and 24.1% →\to 16.9%; these use different reference populations.

Reviewer_TqnvOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

I am not sure what this paper is about, it reads as AI generated science (I am familiar with what this looks like because my students have shown me many examples of the kind of jargon claude produces). If it's not AI, then it's still the case that the abstract and intro/contribution statements are incomprehensible to an expert researcher in the field, so should be rewritten entirely to be considered for publication.

Here are a list of jargony phrases that appear in the abstract alone as though they are common vernacular in this context, but which are not meaningful at all in the way they are appearing here: Compact combinations Affine single-latent predictor Singleton comparator Group memberships Unbalanced optimal transport Descriptor cost Calibration-response correlation Fits per endpoint Compact groups Unchanged-model refits Cost-selected groups Recoverability of an observation

Here is the statement of contribution in the intro. I hope any informed AC can recognize this is completely incomprehensible: “We test response recovery by comparing compact, separately calibrated predictors with an evaluation-optimized affine-singleton ceiling. Across independently fitted TinyLlama and Qwen3 dictionaries, we measure recovery beyond that ceiling under adaptation and unchanged-model refitting, accounting for all eligible source responses. We compare three group selectors under one calibration procedure. Transport and coefficient comparisons distinguish candidate selection, activity-mass constraints, and response weighting. The resulting reference records the source observation, its realizations in other dictionaries, and their evaluation errors.”

Given the many demands on my time, I am not going to read and carefully review/interpret AI science slop. The authors should go through this in very careful detail and explain it in human words for human readers to warrant a deep review.

Critical strengths

N/A

Critical weaknesses

I am not sure what this paper is about, it reads as AI generated science (I am familiar with what this looks like because my students have shown me many examples of the kind of jargon claude produces). If it's not AI, then it's still the case that the abstract and intro/contribution statements are incomprehensible to an expert researcher in the field, so should be rewritten entirely to be considered for publication.

Here are a list of jargony phrases that appear in the abstract alone as though they are common vernacular in this context, but which are not meaningful at all in the way they are appearing here: Compact combinations Affine single-latent predictor Singleton comparator Group memberships Unbalanced optimal transport Descriptor cost Calibration-response correlation Fits per endpoint Compact groups Unchanged-model refits Cost-selected groups Recoverability of an observation

Here is the statement of contribution in the intro. I hope any informed AC can recognize this is completely incomprehensible: “We test response recovery by comparing compact, separately calibrated predictors with an evaluation-optimized affine-singleton ceiling. Across independently fitted TinyLlama and Qwen3 dictionaries, we measure recovery beyond that ceiling under adaptation and unchanged-model refitting, accounting for all eligible source responses. We compare three group selectors under one calibration procedure. Transport and coefficient comparisons distinguish candidate selection, activity-mass constraints, and response weighting. The resulting reference records the source observation, its realizations in other dictionaries, and their evaluation errors.”

Given the many demands on my time, I am not going to read and carefully review/interpret AI science slop. The authors should go through this in very careful detail and explain it in human words for human readers to warrant a deep review.

Questions for authors

N/A

4 comments
AuthorsOpenReview ↗

Rebuttal Part 1/4 — Reviewer explicitly states they will not carefully review the paper, calls established mathematics ‘jargon'

On the authority of their students ("my students have shown me many examples..."), this reviewer labels established mathematics and elementary statistical/set-theoretic terms—including unbalanced optimal transport, affine predictors, set singletons, set memberships, optimal-transport cost, and correlation—"jargon" that is "not meaningful at all in the way they are appearing here," and declines to read the paper ("Given the many demands on my time, I am not going to read and carefully review/interpret"). Directly beneath that refusal, the review selects Presentation: 1 ("Difficult to follow despite careful reading"), enters N/A for both Critical Strengths and Questions, pastes the Paper Summary verbatim a second time as Critical Weaknesses, quotes nothing past page 2, line 068, discusses no equation, baseline, or table in Sections 3–6, and assigns Soundness: 1, Informativeness: 1, and Rating: 1 at Confidence: 4 ("substantial direct research experience on this topic"). We address three points below and and show these terms have precise technical roles in the paper:

  1. None of the twelve listed Abstract phrases (p. 1, lines 011–026, Sentences 4–9) is meaningless: One is a classical field of mathematics—unbalanced optimal transport (UOT) (Chizat et al., Math. Comp., 2018; Cuturi, 2013; Séjourné et al., 2019). Six more combine standard terms from linear algebra, optimal transport, set theory, and statistics: singleton comparator (1-element set {j}\{j\}) and group memberships (selected indices j∈Jj \in J) are explained in the very Abstract sentence from which they were copied (Sentence 5: "The singleton comparator searches the entire target dictionary and optimizes directly on evaluation inputs; group memberships and coefficients are fixed beforehand", repeated on pp. 1–2, lines 053–054); affine single-latent predictor (Sentence 4) applies a 1D affine map azj+ba z_j + b (Eq. 2, p. 3) to "latent activation" (defined in Sentence 2, line 012); descriptor cost and cost-selected groups (Sentences 6 & 8) pair UOT with its standard optimal-transport ground cost CijC_{ij} (Eq. 3, p. 4; Section 3.3, p. 5); and calibration-response correlation (Sentence 6) ranks targets by absolute sample correlation (Section 3.3, p. 5) with the "observed activation response" (Sentence 3) on the calibration split. Three more (fits per endpoint, unchanged-model refits, recoverability of an observation; Sentences 7–9) are ordinary descriptions of the experiments, explained on p. 1, lines 038–051 ("Same-checkpoint refits..."; "train five sparse dictionaries independently at each endpoint") right between the two passages the review quoted.
  2. Every clause of Contributions (p. 2, lines 063–068) refers to specific equations and ablations that the reviewers who read the paper seemed to understand: For example, the sentence on "candidate selection, activity-mass constraints, and response weighting" summarizes Section 5.5's balanced/rejection/UOT ablation (Table 2, p. 8, lines 404–423) and our comparison on the exact same UOT-selected features, where uniform weights recover 4.1%, transport weights recover 19.1%, and calibrated weights recover 31.0% (p. 9, lines 435–440). Both Reviewer HrVG and Reviewer 3GLL summarized these exact concepts accurately: 3GLL cited the "affine-singleton oracle" (singleton comparator) as Critical Strength S1, and HrVG cited separating "recoverability of an observed response" from single-feature reappearance as Critical Strength and analyzed Section 5.5 directly (W2, Q3).
  3. Making pages 1–2 clear on a first read: Of the twelve phrases, only the two using "compact" (compact combinations, compact groups; Sentences 4 & 7) use a word with a different mathematical meaning: in topology "compact" means closed and bounded in Rn\mathbb{R}^n, whereas we meant "small — at most four features" (written as "four-latent" on p. 2, line 055 and Eq. 1, p. 3). The Abstract also named baselines and splits before Sections 3–4 defined Eq. 3 and the 1,998 / 998 / 7,600 document splits, so readers had to look ahead. Consistent with Reviewer HrVG and Reviewer 3GLL scoring Presentation: 2, we replace "compact" with "at most four features" and define every baseline, metric, and data split directly on pages 1–2.

We recognize that this math- and theory-heavy paper spans SAE feature matching, sparse regression, optimal transport, statistical reproducibility, and mechanistic interpretability, making it demanding to evaluate across all these areas. We appreciate the feedback that the abstract and introduction could be plainer, and provide revisions here and in our responses to the other reviewers.

AuthorsOpenReview ↗

Rebuttal Part 2/4 — All 12 cited phrases were already defined in the submitted paper

1. Where the submitted paper already defines all 12 cited Abstract phrases (p. 1, lines 011–026) and Contributions

A. An established mathematical field and six combinations of standard terms (7 of 12 phrases)

Abstract phrase & sentenceMeaning & Page 1–2 context in submitted paperFormal definition in submitted paper (Sections 3–4)
Unbalanced optimal transport (UOT) (Sentence 6)Branch of optimal transport allowing the two marginals to differ by adding divergence penalties; briefly defined on p. 1, lines 048–049.Section 3.3, Eq. 4 (p. 5, lines 246–255), citing Cuturi (2013), Chizat et al. (Math. Comp., 2018), and Séjourné et al. (2019) on lines 253–254.
Affine single-latent predictor (Sentence 4)Standard 1D affine map azj+ba z_j + b (signed slope plus intercept) applied to one feature's "latent activation" (defined in Sentence 2, p. 1, line 012; repeated on pp. 1–2, lines 053–054).Section 3.1, Eq. 2 (p. 3, lines 142–145).
Singleton comparator (Sentence 5)Single-feature (singleton, {j}\{j\}) baseline, defined in Sentence 5 itself: "searches the entire target dictionary and optimizes directly on evaluation inputs" (3GLL S1).Section 3.1, Eq. 2 (p. 3, lines 139–150).
Group memberships (Sentence 5)Indices j∈Jirj \in J_{ir} specifying which target features belong to the group, contrasted in Sentence 5 itself with their weights: "group memberships and coefficients are fixed beforehand".Section 3.1 & Section 3.3 (p. 3, lines 130–135; p. 5, lines 225–229).
Descriptor cost & Cost-selected groups (2 phrases; Sentences 6 & 8)Pairwise feature dissimilarity CijC_{ij} serving as the ground cost in optimal transport; cost-selected groups pick the kk candidates with lowest CijC_{ij}, whereas "UOT groups" pick from the transport plan.Section 3.2, Eq. 3 (p. 4, lines 210–215) & Section 3.3 (p. 5, lines 216–226).
Calibration-response correlation (Sentence 6)Ranking target features by absolute sample correlation with the "observed activation response" (Sentence 3) on the calibration split (p. 1, line 051).Section 3.3 (p. 5, lines 225–227).

B. Three descriptions of the experiments on Page 1 & two uses of "compact" (5 of 12 phrases)

Abstract phrase & sentenceWhere already defined in submitted paperPlain wording in revised Abstract & Intro
Fits per endpoint (Sentence 7)Independent SAE training runs (fits) before and after adaptation (endpoints); stated on p. 1, lines 050–051 ("train five sparse dictionaries independently at each endpoint") and Section 4 (p. 6, lines 272–276).Five SAEs trained from independent random seeds before fine-tuning and five after
Unchanged-model refits (Sentence 8)Re-training SAEs from different random seeds on a fixed checkpoint; stated on p. 1, lines 049–050 ("Same-checkpoint refits") and Section 5.3 (p. 7, lines 370–377).Pairs of SAEs trained from different random seeds on each fixed checkpoint
Recoverability of an observation (Sentence 9)Asks whether a feature's activations on held-out text can still be predicted, rather than whether one matching feature reappears in the second SAE (p. 1, lines 038–042; Eqs. 1–2, p. 3; HrVG Critical Strength).Whether a feature's activations on held-out evaluation text can be predicted within NSE≤0.25\text{NSE} \le 0.25
Compact combinations / Compact groups (2 phrases; Sentences 4 & 7)Here "compact" meant only k≤4k \le 4 out of 4,096 features (written as "four-latent" on p. 2, line 055 and subset size at most 4 in Eq. 1, p. 3, lines 127–138); replaced because "compact" already has a standard meaning in topology.Nonnegative weighted sums of at most four features from the second SAE

Each clause of Contributions (p. 2, lines 063–068) refers to these sections: (i) comparing predictors that use at most four calibrated features against the best single feature fitted on evaluation (Section 3.1, Eqs. 1–2, p. 3); (ii) measuring recovery across all five seeds for 20,474 eligible TinyLlama features and 19,972 Qwen3 features under fine-tuning and same-checkpoint reseeds (Section 4, p. 6; Sections 5.1–5.3, Table 1, p. 7); (iii) comparing the three selectors under identical NNLS calibration (Section 3.3, p. 5; Section 5.5, p. 8); (iv) testing separately which candidates are selected, which marginal constraints are imposed (Table 2, p. 8), and whether selected features receive uniform, transport, or calibrated weights (p. 9, lines 435–440); and (v) recording the selected features, their weights, and the evaluation error for each target (Section 3.3, p. 5, lines 260–269).

(Continued in Part 3/4: Step-by-step Figure 1 example and revised Abstract.)

AuthorsOpenReview ↗

Rebuttal Part 3/4 — Revised Abstract

2. Revised Abstract

A sparse autoencoder (SAE) represents a language model’s hidden activations as a sparse weighted sum of learned features. Standard cross-SAE evaluations ask whether each feature in a first (source) SAE matches a single feature in a second (target) SAE trained from a different random seed or after fine-tuning. We ask a broader reproducibility question: when no single target feature suffices, can a small group of up to four existing target features still predict the source feature’s activation response on held-out text, without joint retraining or a shared feature vocabulary? Across TinyLlama-1.1B and Qwen3-1.7B, we train five SAEs from independent seeds before AG News LoRA fine-tuning and five after, and evaluate all 25 cross-checkpoint pairs and 20 same-checkpoint reseed pairs per model. Measuring document responses as mean activations over content tokens, we count a source feature as recovered only if a frozen predictor meets normalized squared error NSE≤0.25\mathrm{NSE} \le 0.25 across all five target SAEs. An evaluation-fitted single-feature oracle (an upper-bound search over all 4,096 target features, with slope and intercept fitted directly on the evaluation data) recovers 19.3% of active source features in TinyLlama and 15.0% in Qwen3. Frozen four-feature predictors, selected by descriptor cost (decoder cosine plus activation-summary distance), activation correlation, or unbalanced optimal transport (UOT) and fitted with nonnegative least squares on a separate calibration split, recover 31.0%–33.3% in TinyLlama and 23.3%–25.2% in Qwen3. Standard sparse regression (nonnegative Lasso and orthogonal matching pursuit) matches or slightly exceeds these selectors (35.9%–36.2% and 24.3%–26.3%), whereas excluding the eight nearest target decoder directions collapses both matching pursuit (5.3% and 4.6%) and the oracle (3.1% and 3.7%). Token-level responses and same-checkpoint reseeds show the same pattern. Among responses both methods recover in a given target SAE, four-feature predictors nearly halve failure across the remaining four target seeds (23.5% to 12.1% in TinyLlama; 23.9% to 12.8% in Qwen3); roughly nine in ten rescued responses pass with a single feature in some seeds yet require multiple features in others. Weighted four-feature groups also cut downstream logit errors when substituting or ablating 30 class-selective Layer-11 features on a fixed TinyLlama checkpoint, relative to calibrated singletons and without fitting logits. UOT target coupling improves fixed-weight recovery and regression-free classifier transfer, whereas local descriptor selection leads when nonnegative regression weights are freely calibrated. Single-feature matching thus misses substantial local, multi-feature response structure that remains reproducibly measurable across independently trained SAEs.

3. Revised Introduction — Opening (replacing Section 1; continued in Part 4/4)

Sparse autoencoders (SAEs) increasingly anchor interpretability workflows: dashboards label individual features (Lin, 2023; Templeton et al., 2024), model-diffing studies track features across fine-tuning (Lindsey et al., 2024), persistence analyses monitor feature survival, and causal studies ablate selected units (Marks et al., 2025). Those workflows depend on whether a feature measurement established in one dictionary remains recoverable across model adaptation and independent SAE training. Because SAE features permit arbitrary coordinate permutations, index identity cannot answer this question; moreover, independent training seeds can decompose the same activation space differently, leaving a single target feature unable to match the source feature’s activations even after an exhaustive search.

This paper separates two questions that standard evaluations conflate: (i) whether an individual feature reappears with comparable accuracy in every independently trained target SAE, and (ii) whether the source feature’s specified activation response remains predictably measurable across those SAEs, without joint retraining or a shared feature vocabulary. Our main contribution shows that across independent dictionaries and fine-tuning checkpoints, sparse nonnegative combinations of at most four existing target features recover thousands of source responses on held-out text even where the best affine single-feature match fails.

AuthorsOpenReview ↗

Rebuttal Part 4/4 — Revised Introduction

3. Revised Introduction (continued)

We test this in TinyLlama-1.1B and Qwen3-1.7B, training five 4,096-feature SAEs from independent random seeds before AG News fine-tuning and five after, alongside 20 same-checkpoint reseed pairs per model. Using mutually disjoint splits (1,998 descriptor documents for summary-based matching, 998 calibration documents for response-based selection and weight fitting, and 7,600 evaluation documents), we score all eligible active source features (99.97% in TinyLlama and 97.5% in Qwen3). For each source feature and target SAE, we fit a frozen nonnegative four-feature sum plus intercept, counting success only when normalized squared error (NSE) stays at most 0.25 across all five target SAEs. To rule out weak matching as the driver, we benchmark against an evaluation-fitted single-feature oracle (an upper-bound search over all 4,096 target features with unconstrained slope and intercept on the evaluation split). Frozen four-feature predictors ranked by descriptor cost (combining decoder cosine and activation-summary distance) raise all-five document recovery from 19.3 ± 0.4% to 33.3 ± 0.5% in TinyLlama (+14.0 ± 0.3 percentage points across the five source SAEs) and from 15.0 ± 0.2% to 25.2 ± 0.6% in Qwen3 (+10.2 ± 0.5 points).

A second contribution clarifies when joint transport coordination adds value over local feature selection. Unbalanced optimal transport (UOT) couples candidate assignments across all source features using global activity-mass constraints, whereas removing target coupling reduces transport ranking to independent per-source descriptor selection. Comparing both constructions isolates candidate selection from response weighting: local selection leads when prediction weights are freely calibrated on paired responses, whereas target coupling improves recovery under fixed matcher weights and regression-free linear-classifier transfer from stored summaries.

Contributions. Across 50 cross-checkpoint and 40 same-checkpoint SAE pairs in TinyLlama-1.1B and Qwen3-1.7B, we establish four findings:

  1. Multi-feature response reproducibility substantially exceeds single-feature reproducibility and halves cross-seed failures. Frozen four-feature predictors recover thousands of source responses across all five target SAEs where the evaluation-fitted single-feature oracle fails in at least one (31.0%–33.3% vs. 19.3% in TinyLlama; 23.3%–25.2% vs. 15.0% in Qwen3). The gain is corroborated by sparse regression (35.9%–36.2% and 24.3%–26.3%) and concentrated in local decoder neighborhoods (excluding the eight nearest decoder directions drops both matching pursuit and the oracle to 3.1%–5.3%). Among responses both methods recover in a given target SAE, four-feature groups cut failure across the remaining four target seeds from 23.5% to 12.1% in TinyLlama and from 23.9% to 12.8% in Qwen3.
  2. Whether one feature suffices depends on the target SAE seed. Holding the fine-tuned checkpoint fixed, the single-feature oracle reverses pass/fail verdicts across seeds on 35.8% of active TinyLlama features and 28.4% of Qwen3 features. Among responses that four-feature groups rescue across all five fits, 89.4%–92.0% succeed with a single feature in some seeds yet require multiple features in others.
  3. Weighted groups improve downstream intervention agreement without fitting logits. Across five independent SAEs on the fixed adapted TinyLlama checkpoint, replacing calibrated singletons with four-feature decoder or descriptor-cost groups reduces 30-feature Layer-11 substitution squared-logit error by 36.0% and 39.1% and weighted-ablation shift error by 30.5% and 34.7% across all four target seeds.
  4. Target coupling aids fixed-ratio transfer; free calibration favors local selection. Zeroing UOT’s target marginal penalty recovers the local descriptor-cost ranking. Local selection leads by 2 percentage points under freely fitted NNLS weights; coupling leads with matcher-derived relative weights, raising regression-free 40-feature classifier transfer accuracy by +0.62 points in TinyLlama (24/25 pairs) and +1.93 points in Qwen3 (25/25 pairs).

Comments on the paper

3 comments
AuthorsOpenReview ↗

What this paper establishes and why it should be accepted - Reproducible SAE measurements without a shared feature vocabulary

Cross-SAE evaluations usually ask whether each feature in one SAE matches a single counterpart in another. We separate two questions:

  • Can a single feature reproduce a specified activation response in each independently trained SAE?
  • When no single feature suffices, can at most four features still predict that response on held-out text?

Across existing, independently seeded SAEs, separately calibrated groups keep responses recoverable across all target fits even when at least one fit lacks an adequate affine single-feature predictor, including after fine-tuning and without joint retraining.

Submitted Figure 1 shows why single-feature verdicts depend on the seed. Each response is a source feature's mean activation over a document's content tokens, scored by normalized squared error (NSE, mean squared error over evaluation-set variance; threshold 0.25). Holding the source response and adapted model fixed while changing only the target SAE seed shifts the best single-feature NSE from 0.104 to 0.351; a frozen two-feature predictor reaches 0.172 in that fit, and calibrated groups pass in all five seeds.

Across our benchmarks, roughly nine in ten responses recovered in all five targets by four-feature cost groups, yet missed by the singleton in at least one, still pass as a single feature in another seed on the same checkpoint (HrVG rebuttal, Table R3; the rest lack one in all five). Different SAE seeds on the same model can thus yield opposite single-feature verdicts on a response measurable in every fit.

Prior work shows that structure persists without recurring individual features: Leask et al. evaluate group swaps by reconstruction and sparsity, Gerasimov et al. find shared decoder subspaces, and the cross-seed benchmark tests functional interchangeability. None measures how accurately a few target features predict a specified source response on new inputs, or how often this holds across independent fits.

In the submitted study, our singleton comparator searches the full target dictionary and fits signed scale and offset on evaluation text, upper-bounding every affine single-feature predictor on those inputs. For each target SAE, a descriptor-cost group selects at most four features by summary similarity on a descriptor split, fits nonnegative weights and an offset on calibration text, and freezes both before evaluation. Requiring NSE <= 0.25 in all five adapted SAEs (averaging five source-SAE rates over all eligible responses, unfiltered by matching success), these groups raise document recovery from 19.3% to 33.3% on TinyLlama and from 15.0% to 25.2% on Qwen3 (Table 1), with the same gap on tokens, same-checkpoint reseeds, and looser cutoffs (Tables 1, 3–6). On responses both methods recover in a shared reference fit, groups nearly halve failure in the other four fits: 23.5% to 12.1% on TinyLlama and 23.9% to 12.8% on Qwen3 (HrVG rebuttal, Q2).

Rebuttal controls show this gain is not selector-specific: calibration-fitted nonnegative OMP and screened Lasso reach 35.9%–36.2% on TinyLlama and 24.3%–26.3% on Qwen3 on the same target SAEs (HrVG rebuttal, Table R1). Excluding the eight nearest target features collapses OMP (ruling out non-local fitting), and calibrating activation weights on the four closest decoders beats static decoder-subspace projection (calibrated gain and offset) by 6.7 and 8.1 points, respectively. Correlated local mixtures remain compatible, and the evaluation point holds either way: single-feature tests still miss those responses in some seeds while frozen groups retain them. On a fixed TinyLlama checkpoint, those groups cut Layer-11 cohort substitution logit error by 36%–39% and weighted-ablation error by 30.5%–34.7% over matched singletons (3GLL rebuttal, Tables R6–R7; cohort-level readout fidelity, not per-feature causality).

Comparing unbalanced optimal transport (UOT) with its uncoupled limit (local cost ranking) clarifies where cross-feature coordination helps. With freely calibrated weights, local cost beats UOT (33.3% vs. 31.0% on TinyLlama; 25.2% vs. 23.3% on Qwen3, Table 1), so global transport is unnecessary for main recovery. Coupling helps without paired regression, lifting 40-feature classifier transfer from stored summaries (+0.62 and +1.93 points) and beating local selection under fixed matcher ratios, though fixed ratios trail free calibration (HrVG rebuttal, W2/Q3, Table R5; dense balanced OT is comparable, isolating target coupling from sparse-graph KL feasibility in Appendix C.3).

The primary study covers two small models, one layer and Top-64 SAE configuration each, one AG News adaptation per model, and an evaluation corpus reused from an earlier study (selections and weights locked before evaluation), without establishing semantic interpretations or per-feature causality. Within this scope, single-feature matching misses measurements that remain reproducible across independently trained SAEs.

AuthorsOpenReview ↗

Final note

Dear Area Chair and Reviewers,

Thank you for your time overseeing our submission, "Different Features, Reproducible Responses: Sparse Autoencoders Across Model Changes."

As you prepare your recommendation, we want to highlight three core misalignments between the reviews and our paper's stated scope and contributions:

  • The novelty and significance assessments substantially underestimate our central empirical contribution. Our paper demonstrates that specified activation measurements can remain recoverable across independently trained SAEs even when individual-feature matching fails. This is a distinct scientific finding, not simply an incremental extension of existing feature-correspondence methods.
  • We disagree with the assertion that our conclusions are already well known. The cited literature establishes feature instability, reproducible subspaces, and various forms of correspondence, but does not measure how accurately a few target features predict a specified activation response on held-out inputs, or how often this holds across independently trained dictionaries.
  • The success of standard sparse regression was treated as diminishing the significance of our findings, although our empirical contribution concerns what measurements remain recoverable, not the novelty of the algorithm used to recover them.

As noted in our public revision summary, we also clarified the writing across the uploaded manuscript for a general audience while keeping the underlying study and headline results.

We appreciate your time and consideration in evaluating our work.

Sincerely,

The Authors

AuthorsOpenReview ↗

Summary of Revised Manuscript

We have uploaded a revised manuscript addressing the presentation feedback and incorporating the rebuttal experiments.

The underlying study is unchanged. Specifically, the revision:

  • Clarifies the writing and section headings across the manuscript so it reads clearly for a broad ML audience, defining terms, baselines, and metrics in plain language at first use (including the revised Abstract and Introduction from our rebuttals) and expanding the prior-work comparison in Section 2.
  • Introduces Equation 4 alongside its uncoupled local-cost limit in Section 3.3.
  • Incorporates the rebuttal analyses requested by the reviewers in Sections 5.2–5.5 and Appendix C: the seed-by-seed singleton breakdown, sparse-regression (OMP and Lasso), decoder-locality, static decoder-projection, sample-size, and non-SAE dictionary controls, fixed-ratio and regression-free classifier transfer comparisons, and fixed-checkpoint Layer-11 cohort substitution and ablation tests.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.