acceptodds
Under review as a conference paper at ICLR 2027

Sparse Compositionality Identifies Features of Computation

Abstract

Does a reconstruction objective recover the features that participate in a network's computation, or a basis that merely reconstructs its activations? The distinction matters if, as a growing body of work suggests, learners generalize through compositional sparsity. The features that explain such a network are then the intermediate functions it composes, and nothing guarantees that a featurizer trained to reconstruct will single them out. In this work, we make the distinction measurable by planting a ground-truth feature DAG, whose intermediate features and compositional relations are known, inside a transformer through per-layer supervision. Reconstruction-based featurizers, including sparse autoencoders, do not reliably identify these intermediate features as individual latents, although the same architectures recover them with direct supervision. This suggests that reconstruction alone cannot identify the planted basis even when the featurizer has the capacity to represent it, isolating a critical identifiability failure. We address this by constraining the featurizer dictionary to represent features relevant to downstream computation by pressuring the features to align with the Jacobian. On the synthetic model, this featurizer identifies the ground-truth basis to reveal the planted compositional hierarchy where baselines could not. We then apply this approach to DINOv3, where no intermediate ground-truth is available. Without any supervision, the resulting attribution graphs yield part-to-whole object circuits: features for a rabbit’s eye, nose, snout, and chest funnel into a downstream rabbit feature. Intervening on targeted upstream features changes the responses of connected downstream features, supporting the functional relevance of these links. For instance, patching a stripe concept on the horse body shifts a frozen ImageNet readout toward the zebra. Together, our results show that grounding features in downstream gradients addresses shortcomings of optimizing reconstruction alone, providing a path from featurizing representations to featurizing computation.

Primary area:
interpretability and explainable AI
Keywords:
Interpretability, Compositionality

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_MhKpOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

The paper uses downstream-gradient coverage to learn sparse features hierarchically. It evaluates recovery of planted compositional variables in a synthetic model and part-to-object relationships in DINO 3. Main idea is that meaningful features have strong correlation with downstream gradients direction.

Critical strengths
  1. Elegant and well-motivated objective. I find the proposed loss elegant and intuitive, the backward training procedure makes sense.

 

  1. Supporting evidence and various tests: I appreciate the fact that the synthetic experiments include several baselines, supervised controls, and tests that remove gradient pressure during training. The DINO experiments include interventions across 668 linked feature pairs, supporting the functional relevance of the discovered connections.
Critical weaknesses
  1. The advantage over the closest methods remains unclear. Jacobian SAEs and gradient SAEs already incorporate downstream computation into dictionary learning. While both are mentioned, neither is compared. The authors chose not to use the benchmarks used by prior works and not to adapt prior works to their benchmarks. The overall idea is therefore not justified, because it might be that the existing similar approaches work as well or better. I am very open to raising my rating if comparisons done fairly establish a useful advantage.

 

  1. Minor: The synthetic experiments do not isolate the core coverage loss. The toy objective combines coverage with several additional gradient terms, and the removal experiment disables them together. A coverage-only ablation would clarify how much of the feature-recovery improvement comes from the central proposed mechanism. The DINOv3 experiment does use coverage alone, but this does not resolve its contribution to recovery in the synthetic benchmark.

 

Post rebuttal: My main concerns have been resolved and score is raised to 3.

Questions for authors
  1. See weakness 1: Other than the actual comparison which is needed, is there a structural advantage in this method over the other two?
4 comments
AuthorsOpenReview ↗

Response to Reviewer MhKp (1/2)

Thank you for the thoughtful review and the constructive suggestions. We appreciate your positive assessment of the objective and for acknowledging “proposed loss is elegant and intuitive”. Your comments highlighted two important additions: comparisons with the closest function-aware methods and an ablation of the local coverage loss. We have now carried out both experiments and address them below.


“The advantage over the closest methods remains unclear.”

&

“...is there a structural advantage in this method over the other two?”

 

We agree that these comparisons were missing. We have extended the comparison to four additional featurizer families: Cross-Layer Transcoders (Ameisen et al., 2025), End-to-end SAEs (Braun et al., 2024), Gradient SAEs (Olmo et al., 2025), and Jacobian SAEs (Farnik et al., 2025). We adapted these methods to our synthetic testbed, where the known intermediate features allow us to measure one-to-one recovery directly. All featurizers are evaluated on the same frozen transformer and evaluation inputs, using the same recovery metrics.These methods incorporate downstream computation in different ways. The results for these featurizers are shown in the table below:

 

Table. Reconstruction/prediction R^2 and one-to-one Hungarian F1 for the original baselines and additional function-aware featurizers, alongside our Jacobian coverage method.

FeaturizerBlock 2 R2R^2Block 2 F1F_1Block 3 R2R^2Block 3 F1F_1Block 4 R2R^2Block 4 F1F_1
TopK0.9400.5730.7940.3820.9400.294
Transcoder0.9020.5650.8830.3750.9760.429
CLT0.9060.5670.9600.3910.9760.537
End-to-end SAE (e2e+ds)0.8010.5040.4220.4030.5350.329
Gradient SAE (gradient at inference)0.8750.5490.7870.3650.8450.327
Jacobian SAE, input side0.9250.6690.7910.3900.9340.300
Jacobian SAE, output side——0.8390.4000.7140.412
Jacobian Coverage (ours)0.8290.7540.6950.5630.7240.596
  • Cross-Layer Transcoders (CLT) allow features from earlier blocks to contribute directly to predicting later activations. CLTs improve identifiability over the per-block transcoder, particularly at deeper blocks (block 4 F1=0.537 versus 0.429), with the same prediction R^2 ~ 0.976. One possible explanation is that earlier dictionaries already account for information carried forward in the residual stream, so later dictionaries do not need to represent all of that information again. Their latents can instead focus on what the earlier contributions leave unexplained, potentially allowing features at that depth to be represented more distinctly.
  • End-to-end (e2e) SAEs train a reconstruction to preserve the model’s downstream outputs. In our testbed, this variant still gives limited one-to-one feature recovery, most likely because the e2e objective can tolerate a planted feature being split across latents, provided their combined reconstruction preserves the outputs and activations being measured (for the e2e+ds variant).
  • Jacobian SAEs jointly train dictionaries before and after a transformer block, encouraging the Jacobian between their latents to be sparse. In our setup, the input-side dictionary improves feature recovery at earlier blocks while the output-side dictionary does better at deeper blocks. At block 4, the output-side dictionary reaches F1=0.412, compared with 0.596 for Jacobian coverage, with similar reconstruction R^2 (0.714 and 0.724, respectively). The objective constrains how the learned latents interact, without specifying which features they should each represent. Because both dictionaries are learned together, they may find a sparse mapping between their latents while still splitting or mixing the planted features. Sparsity of the interactions alone therefore need not imply recovery of the individual variables participating in the computation.
  • Gradient SAEs use downstream-loss gradients to influence which latents are selected in the forward pass while the training objective still remains reconstruction. One possible explanation for their lower Hungarian F1 in this testbed is that multiple latents are allocated to similar influential directions. In their paper, they report that latents cluster around influential directions, but whether this leads to “problematic instances of feature splitting” remains unclear (Olmo et al., 2025, Section 4.4).

The structural distinction is that our coverage objective holds the downstream features fixed and asks a selected upstream latent direction to align with each downstream feature’s gradient. We view this constraint as a plausible explanation for the higher one-to-one recovery in our testbed. We will add these comparisons and a dedicated appendix section in the revised paper. Thank you for encouraging us to explore these comparisons.

AuthorsOpenReview ↗

Response to Reviewer MhKp (2/2)


“A coverage-only ablation would clarify how much of the feature-recovery improvement comes from the central proposed mechanism.”

Fair point. To address this, we retrained the entire backchained sequence with the four auxiliary Jacobian terms removed, keeping the local coverage term (Equation 7) as the only Jacobian pressure. This is the pressure we emphasize in the main text and use in the scaled-up vision experiments. This simpler recipe retained much of the identifiability gains, with Hungarian F1 scores of 0.685, 0.542, and 0.562 at blocks 2, 3 and 4, compared with 0.509, 0.324, and 0.239 for the reconstruction-only control. In these runs, we used the same local coverage and reconstruction weights across all blocks and applied coverage to both the encoder and decoder directions at every block.

We also tested whether the improvement depended on the local coverage weight by training new backchained featurizers with weights of 5, 15, and 50 and found no major difference in the F1 scores (Hungarian F1 varied by at most 0.017 at any block). These results show that substantial feature-recovery gains can be obtained in this synthetic testbed with local coverage. We will include these ablation results and clarify the role of the auxiliary terms in the revised paper, which was to assess the most optimal recovery in this setup over a broad sweep (the same paradigm that we used for all the featurizers) for a fair comparison.


Thank you for the substantial additional experiments! The new comparisons and coverage-only ablation resolve my main concerns, and the gate-intervention tests further strengthen the paper. I am raising my rating to 3.

Could you also clarify the baselines’ dictionary sizes, sparsity settings and tuning budgets, whether results cover multiple seeds, and whether hyperparams selection and final evaluation used separate data?

AuthorsOpenReview ↗

Response to Reviewer MhKp

Thank you for your engagement and positive re-evaluation. We really appreciated the time you took to review our work, and we genuinely believe it helped make the paper stronger! We clarify your comments below.


"...the baselines' dictionary sizes, sparsity settings..."

These are given in Table 2 (Appendix B.2). Every baseline has twice as many latents as ground-truth features, which makes the dictionaries overcomplete relative to the 128-dimensional transformer state (expansion factors of 2× to 16× from block 2 to block 5). The sparsity setting is 1.5 times the mean number of active ground-truth features: TopK, BatchTopK, the archetypal SAE and the transcoder use it as their activity budget, matching pursuit uses it as its number of pursuit steps, and JumpReLU learns its thresholds instead. The four additional featurizers in our response (CLT, end-to-end SAE, gradient SAE and Jacobian SAE) use the same widths and budgets as the TopK baseline. The sizes and sparsity of the Jacobian coverage dictionaries and the matched reconstruction-only dictionaries are specified in Table 4 (Appendix C.5).


"...and tuning budgets"

For the baselines, we swept the dictionary width and the sparsity, with up to 15 configurations per method. Figures 12 and 13 show these sweeps, and Figure 14 shows the best configuration of each method at each block (all in Appendix B.4). Even at their best configuration within these sweeps, the baselines reach a Hungarian F1 lower than the Jacobian coverage featurizer at every block. For the four additional function-aware featurizers we kept the widths and budgets of Table 2 (Appendix B.2) and swept the one coefficient that is specific to each method, reporting its best value at each block. For our Jacobian coverage featurizer, the hyperparameters in Table 3 (Appendix C.4) were selected with broader sweeps spanning ranges for all Jacobian terms; nonetheless, the ablation analysis in our previous response shows that most of the gain in identifiability comes from the local jacobian coverage term.


"...whether results cover multiple seeds..."

To address this, we trained the Jacobian coverage and the reconstruction-only featurizers with five new seeds, keeping every setting of the paper the same (the tables in the paper report single runs). Because the Jacobian featurizer is backchained, each seed is a complete new sequence of block 4, block 3 and block 2 featurizers. Across the five seeds the variation is very small: the standard deviation of Hungarian F1 is at most 0.016 for the Jacobian coverage featurizer and at most 0.009 for the reconstruction-only featurizer, at every block. Across all seeds, the Jacobian coverage featurizer beats the reconstruction-only control on identifiability by a wide margin at every block.


"...whether hyperparams selection and final evaluation used separate data?"

All metrics in the paper were computed on 10,000 held-out inputs, generated independently of the training data. We used these inputs to select configurations by Hungarian F1, following the same rule for all the featurizers (Appendices B.4 and C.1). To test whether the final results hold on data that played no part in selection, we generated five fresh sets of 10,000 inputs and scored all featurizers again on each of these sets. Hungarian F1 changed by less than 0.003 in any cell.


Thank you again for these questions.

Reviewer_rj5cOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
2
Soundness
3(+1)
Read review
Paper summary

This paper examines the usefulness of featurizers in identifying the mechanism by which data is created. It introduces an intriguing toy model: a Boolean feed-forward network that a transformer trains to reproduce. In this way, it plants a ground truth in the transformer on which we can validate or falsify the effectiveness of the featurizer. Using the Hungarian F1 measure, they find that there is no one-to-one recovery of planted features and latents when using an SAE alone. The featurizer misses the true Boolean structure, providing a strong critique of the tool.

The authors provide a partial solution to this problem by introducing Jacobian pressure, which takes into account the effect of downstream data, and they find that the resulting SAE features substantially improve, but do not fully close, the identifiability gap with the Boolean features. They then apply the featurizers with Jacobian pressure beyond a toy model to DINOv3 and find that the SAE features allow them to meaningfully steer. By emphasizing the "stripes" feature, they reclassify a horse as a zebra. Moreover, they can trace the individual features of a rabbit into the rabbit classifier. Therefore, the paper not only highlights a gap in using featurizers as an interpretability tool, but also begins to fill that gap with Jacobian pressure.

Critical strengths
  1. The paper makes a widely suspected failure of featurizers measurable. While the reconstruction of the activations by SAE is very accurate, with the R^2 \approx .999, the Hungarian F1 is .05, meaning that the SAE feature is a very poor fit for the true Boolean features.

  2. Through the use of a planted Boolean network ground truth, the paper presents a useful tool for grading interpretability tools. I anticipate that this technique could become a standard practice for testing whether the featurizer strategy correctly identifies the mechanism of computation.

Critical weaknesses
  1. The paper gives the impression that it aims to discover the logical architecture of the planted Boolean circuit. This is what "meaningful components of the computation" (page 2) evokes. However, many possible circuits could lead to the same Boolean values throughout the DAG. When addressing the mechanism of a computation, one must go beyond the input and output values that are realized, and address the precise circuit elements that produce the computation. It is not clear that Jacobian pressure gives us access to this, because there are no numerical tests that validate it. I would suggest gate-type recovery tests to make this result stand out.

  2. The proposed model is trained with five Jacobian terms whose per-block weights were hand-set (Table 3). If these values needed to be tuned to get the algorithm to work, it seems like this is not a general technique for discovering the features, but rather a tool that needs human guidance.

Questions for authors
  1. What is the usefulness of knowing the values of the nodes in a Boolean circuit without knowing the precise computation that maps between those nodes? Given that the DAG architecture was introduced as the object of interest, I interpreted this as what we would like to learn from a "good" featurizer, but I'm willing to be convinced otherwise.

  2. Was it necessary to tune the five weights in Table 3 by hand to get a good fit?

4 comments
AuthorsOpenReview ↗

Response to Reviewer rj5c (1/2)

Thank you for the clear summary, the encouraging comments, and the targeted suggestions. We particularly appreciate your view that the testbed “could become a standard practice” for evaluating featurizers, and your recognition that Jacobian pressure helps address the feature-recovery gap. Your questions helped us think more deeply about our results, and we address your concerns below:


“The paper gives the impression that it aims to discover the logical architecture of the planted Boolean circuit. ….I would suggest gate-type recovery tests to make this result stand out.”

&

“What is the usefulness of knowing the values of the nodes in a Boolean circuit without knowing the precise computation that maps between those nodes?”

This is an important point, and we really appreciate you raising it. We agree that matching a latent to the values of a planted feature does not, by itself, establish how that feature is “computed’’ as the gate is not tested directly. Two observations help explain why improved feature recovery makes the computation easier to test:

  • First, the improvement in recovery holds for every gate type and is substantially larger for AND gates, whose outputs are new features and not mere copies of an input.
    • AND / OR / AND-NOT / COPY F1 scores are 0.116 / 0.346 / 0.221 / 0.300 for reconstruction only and 0.435 / 0.558 / 0.563 / 0.703 for Jacobian coverage at block 4.
    • AND / OR / AND-NOT / COPY F1 scores are 0.180 / 0.576 / 0.258 / 0.522 for reconstruction only and 0.386 / 0.659 / 0.466 / 0.831 for Jacobian coverage at block 3.
  • Second, of the 1,042 planted gates with two or more inputs, 585 have a latent with F1 of at least 0.5 for every input with Jacobian coverage, against 45 with reconstruction only. These observations show that many more gates have well-matched latents in the jacobian featurizer, but do not tell us if these gates are being used in the computation. We examine this directly with a new Switch Test.

Switch Test

For a planted gate we take inputs in a known state and switch off one of the gate's inputs inside the frozen transformer, by moving the block state along the direction of that input's latent (this is the matched latent’s encoder direction) until the latent reads off (typical “off” score over a held out set). We re-run one block and read the gate's output with the transformer's own probe. The planted gate predicts what must happen, and the prediction depends on the gate type: an AND output must go off, an OR output must stay on while another input is on and go off with its last input, and an AND-NOT output must come on when its negated input is removed. For AND gates this is 0.953 when we switch along the true feature directions, 0.825 with the Jacobian coverage latents, 0.166 with the reconstruction-only latents and 0.078 for a random direction. Across conditions requiring an output change, Jacobian coverage outperforms reconstruction-only and random directions, while unrelated features usually remain on.

 

MethodAND: goes offOR: stays onOR: goes offAND-NOT: comes onAND-NOT: goes offCOPY: goes offUnrelated feature stays on
Start: all inputs on, output on. Action: switch off one input. Prediction: output goes off.Start: two or more inputs on, output on. Action: switch off one of them. Prediction: output stays on.Start: exactly one input on, output on. Action: switch off that input. Prediction: output goes off.Start: plain inputs on, negated input on, output off. Action: switch off the negated input. Prediction: output comes on.Start: plain inputs on, negated input off, output on. Action: switch off one plain input. Prediction: output goes off.Start: its input on, output on. Action: switch off that input. Prediction: output goes off.Start: a feature that is on and does not depend on the input. Action: switch off the input. Prediction: the feature stays on.
True features0.9530.6850.9950.7480.9790.9930.889
Reconstruction only0.1660.9960.0650.0980.1140.0910.964
Jacobian coverage0.8250.8580.9190.6420.8490.8690.905
Random direction0.0780.9970.0720.0890.0880.0660.920

These results show why recovering individual features is useful: the matched latents provide directions through which we can test whether downstream responses follow the planted gate’s logic. The Jacobian coverage latents act as “computational” handles through which the transformer's gates can be tested. We will add this as a dedicated section in the appendix in the revised draft.

AuthorsOpenReview ↗

Response to Reviewer rj5c (2/2)


“The proposed model is trained with five Jacobian terms whose per-block weights were hand-set …a tool that needs human guidance.”

&

“Was it necessary to tune the five weights in Table 3 by hand to get a good fit?”

We selected the weights in Table 3 through broad hyperparameter sweeps, using Hungarian F1 on the evaluation inputs. As in our sweeps of the other featurizers, the aim was to assess the best feature recovery found among the tested settings. We will state this selection procedure explicitly in Appendix C.

To test whether separately tuning these terms at each block was necessary, we retrained the entire backchained sequence with the four auxiliary Jacobian terms removed, keeping the local coverage term (Equation 7) as the only Jacobian pressure. This is the pressure we emphasize in the main text and use in the scaled-up vision experiments. This simpler recipe retained much of the identifiability gains, with Hungarian F1 scores of 0.685, 0.542, and 0.562 at blocks 2, 3 and 4, compared with 0.509, 0.324, and 0.239 for the reconstruction-only control. In these runs, we used the same local coverage and reconstruction weights across all blocks and applied coverage to both the encoder and decoder directions at every block. We also tested whether the improvement depended on the local coverage weight by training new backchained featurizers with weights of 5, 15, and 50 and found no major difference in the F1 scores (Hungarian F1 varied by at most 0.017 at any block).

These results show that substantial feature-recovery gains can be obtained with local coverage. We will include these results and clarify the role of the auxiliary terms in the revised paper. Thank you for prompting this ablation analysis, which strengthens the evidence supporting our emphasis on local Jacobian coverage as the main pressure for feature identifiability.


Reviewer_rj5cOpenReview ↗

Response to authors

I would like to thank the authors for a thorough rebuttal and for running new experiments on both of my concerns. One note: the revised PDF I have looks identical to the original, so I'm assuming the new material will make it into the paper.

My concern about the honest hand-set weights is settled for the synthetic testbed. The new ablation shows that most of the benefit comes from the coverage term alone, with one shared weight whose value barely matters, so the method isn't succeeding because it was tuned against the ground truth. Whether this transfers to real models is still an open question.

My concern about values versus computation is half-answered. The Switch Test is close to what I asked for: an intervention whose predicted outcome depends on the type of gate, taking a step beyond matching feature values. The only problem is that it compares to known gates, and doesn't say anything about gates you don't know about. It doesn't show a method for discovering the wiring of gates autonomously.

Overall, I'm happy with the changes, and I'm changing the soundness to 3 while keeping the overall rating the same.

AuthorsOpenReview ↗

Response to Reviewer rj5c

Thank you so much for raising the soundness score! We’re glad the additional analyses helped address your concerns and we will include all the revisions in the camera-ready version. We clarify your remaining points below.


"...transfers to real models is still an open question."

We agree. In DINOv3 there is no ground truth, so feature identifiability cannot be measured directly. Our evidence there is functional. In the featurizer that was trained with the local coverage as the only Jacobian pressure, amplifying or suppressing a linked upstream feature increases or decreases the connected downstream feature accordingly. Control features, which are active on the same images with similar strength but are not connected or linked to that downstream feature, show no such effect (668 pairs, Appendix D.3).


"...method for discovering the wiring of gates autonomously."

We completely agree with your assessment. In DINOv3 we do trace links between features from gradients alone and test them by intervention, but yes, we do not currently have automated circuit discovery in the synthetic testbed.


Thank you again for your engagement and constructive feedback, it really made the paper stronger.

Reviewer_3xZROpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

In the paper, it shows that sparse reconstruction alone do not fully recover the intermediate variables of compositional computation. This is a hypothesis, not a proved identifiability result. Experiments contain synthetic Boolean feature labels and DINOv3.

Critical strengths

The posed question and hypothesis is nice. Distinguishing reconstruction quality, distributed decodability, and one-to-one feature recovery is valuable. Table 1 shows that these are different properties.

Critical weaknesses

The paper goes with no code submission.

This topic lies outside my expertise. While I understand some parts of the paper and found that the framework is interesting, I do not have enough experience to identify a weak aspect of the paper, or have a useful question to the authors.

Questions for authors

Please see Weaknesses.

3 comments
AuthorsOpenReview ↗

Response to Reviewer 3xZR

Thank you for your review! We appreciate your acknowledgement that the paper dissociates reconstruction and one-to-one feature recovery, and that you see this as a valuable contribution. We address your comments below and share some additional results that further support our findings.


“The paper goes with no code submission.”

We plan to release all code in the camera-ready version. This will include the toy compositional testbed (synthetic data-generating process and circuit planting procedure), training and evaluation code for the baseline featurizers, and the pipeline for training the Jacobian featurizers. We will also release the trained featurizer weights for DINOv3. For reproducibility, the appendices describe the experimental procedures, training objectives, hyperparameters, and evaluation methods.


“This is a hypothesis, not a proved identifiability result. “

We completely agree that our contribution is empirical. Our aim was to make this question measurable in a controlled setting where the intermediate feature targets are known. The experiments show that featurizers can reconstruct activations accurately while failing to recover these targets as individual latents. For example, TopK at block 5 achieves reconstruction R^2 = 0.927, but Hungarian F1=0.162. With direct supervision, the same architectures recover the features much more closely while retaining good sparse reconstruction (Fig. 4). This shows that the identifiability gap is not simply a limitation of what the architectures can represent or express. Also, Fig. 5B shows that removing Jacobian pressure during training causes the feature recovery to decline even as reconstruction improves.

We have now extended the comparison to four additional featurizer families: Cross-Layer Transcoders (Ameisen et al., 2025), End-to-end SAEs (Braun et al., 2024), Gradient SAEs (Olmo et al., 2025), and Jacobian SAEs (Farnik et al., 2025). These methods incorporate downstream computation in different ways. The results for these featurizers are shown in the table below. Cross-Layer Transcoders (CLTs) improve feature recovery over per-block transcoders, particularly at deeper blocks (Block 4 F1=0.537 versus 0.429), with the same prediction R^2=0.976. One possible explanation is that earlier dictionaries account for information carried forward in the residual stream, allowing later dictionaries to focus on what actually remains unexplained. End-to-end SAEs still give limited one-to-one recovery and this can happen because preserving downstream outputs allows the dictionary to tolerate a planted feature being split across its latents, provided their combined reconstruction preserves those outputs. Gradient SAEs use downstream-loss gradients to influence latent selection via topK, but they retain reconstruction as their training objective. Jacobian SAEs improve recovery at some blocks by encouraging sparse interactions between learned dictionaries. However, because both dictionaries are learned jointly, a sparse mapping between their latents can still be compatible with splitting or mixing the planted features. Our coverage objective holds the downstream features fixed and aligns selected upstream latent directions with each downstream feature’s gradient and hence achieves higher one-to-one recovery than the other function-aware featurizers that we tested here.

Table. Reconstruction/prediction R2 and one-to-one Hungarian F1 for the original baselines and additional function-aware featurizers, alongside our Jacobian coverage method.

FeaturizerBlock 2 R2R^2Block 2 F1F_1Block 3 R2R^2Block 3 F1F_1Block 4 R2R^2Block 4 F1F_1
TopK0.9400.5730.7940.3820.9400.294
Transcoder0.9020.5650.8830.3750.9760.429
CLT0.9060.5670.9600.3910.9760.537
End-to-end SAE (e2e+ds)0.8010.5040.4220.4030.5350.329
Gradient SAE (gradient at inference)0.8750.5490.7870.3650.8450.327
Jacobian SAE, input side0.9250.6690.7910.3900.9340.300
Jacobian SAE, output side——0.8390.4000.7140.412
Jacobian Coverage (ours)0.8290.7540.6950.5630.7240.596

These results support the empirical distinction between sparsely reconstructing representations and identifying their individual features. We will clarify this scope in the discussion section, without claiming a general identifiability theorem.

Thank you for your response. I appreciate the additional comparison and will maintain my original positive score.

AuthorsOpenReview ↗

Response to Reviewer 3xZR

Thank you for your time and positive assessment of our work!

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.