acceptodds
Under review as a conference paper at ICLR 2027

Let's interpret step-by-step

Abstract

Interpretability research often seeks to explain model computation using ordered sequences of discrete steps, particularly when describing latent reasoning pipelines or circuits. When selecting features for circuits, we should therefore prefer features that actually capture discrete steps in the model's forward pass. In practice, however, this criterion is rarely verified or explicitly targeted when building feature dictionaries. We propose a new metric, *punctuality*, to measure if a feature was computed in a step, rather than a gradual refinement process. As a case study, we analyze a popular sparse autoencoder (SAE) feature circuit and find that many features are punctual and represent a shared step in computation. Next, we assess residual stream SAE features and find them to be less punctual than neuron basis features—and furthermore, that more punctual SAE features are less causally important, limiting their value for interpreting latent reasoning mechanisms. Finally, we show that crosscoders, which are designed to merge features that persist across layers, outperform SAEs in finding features that are both punctual and causally important. By explicitly measuring punctuality along with other desiderata like causal importance and interpretability, we can assess which features can be used in circuits or explanations of latent reasoning.

Primary area:
interpretability and explainable AI
Keywords:
circuits, SAEs, transcoders, crosscoders, sparse dictionary learning, interpretability, evaluation, features, mechanistic interpretability

Reviews

Among:
Shape:
1234this paper, Bottom 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.25 over 4 reviews, above 49% of the others.
Reviewer_QwzMOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates whether the features used to construct step-by-step reasoning circuits with sparse autoencoders (SAEs) truly correspond to discrete steps in the model's forward computation. To this end, the authors propose the punctuality metric, which measures whether a feature suddenly becomes readable within a few layers or gradually forms across many layers. The authors analyze an existing SAE circuit on the Bias-in-Bios task and find feature clusters that can indeed be ordered by their time of emergence. They also find that the punctuality of SAE features is negatively correlated with causal importance: many highly punctual features disappear rapidly and have almost no effect on the model's output. Finally, the paper compares SAEs with Crosscoders. The results show that Crosscoders find more persistent features in earlier layers and more punctual features in later layers, and overall outperform SAEs jointly on punctuality and causal importance.

Critical strengths
  1. The definition of punctuality is clear. The authors construct punctuality from layer-wise feature availability and the participation ratio, giving the metric a clear intuitive interpretation. Punctuality is high when a small number of layers account for most of the progress in forming a feature, and low when many layers jointly and gradually advance its formation.
  2. The paper first uses the Bias-in-Bios circuit to show that punctual features can indeed exist. It then compares SAEs with neurons at the distributional level, analyzes the relationships among punctuality, persistence, and causal importance, and finally uses Crosscoders to demonstrate a possible direction for improvement.
  3. The authors compare multiple definitions of punctuality, measure availability in different ways using the original feature direction, linear regression, and logistic regression, and use multiple methods to control for sparsity differences between SAEs and neurons. When comparing SAEs with Crosscoders, the authors also conduct a stratified analysis based on activation sparsity. These supplementary experiments strengthen the credibility of the main conclusions to some extent.
Critical weaknesses
  1. The paper has not directly demonstrated that high punctuality produces better circuits. The authors propose punctuality as an important criterion for selecting circuit features, but they do not construct circuits separately using high-punctuality and low-punctuality features, nor do they compare these circuits in terms of faithfulness, sufficiency, minimality, human interpretability, or behavioral predictive ability.
  2. The experimental coverage is relatively limited. The main experiments focus on the Gemma model family, Gemma Scope SAEs and Crosscoders, Pile data, and one Bias-in-Bios circuit case study. Given that the paper discusses general questions of latent reasoning and circuit interpretation, it remains unclear whether the conclusions generalize to other model families, language models of different scales, other SAE architectures, and tasks involving mathematical reasoning, multi-hop question answering, or algorithms.
Questions for authors
  1. If a model reasons through continuous, gradual representation updates, could low-punctuality features still form faithful and meaningful circuits? Could the authors distinguish more clearly between being suitable for discrete-step explanations and being suitable for general mechanistic explanations?
  2. Does high punctuality actually improve circuit quality? Could the authors construct circuits separately using high-punctuality and low-punctuality features and compare their faithfulness, sufficiency, completeness, minimality, and intervention accuracy? Such an experiment would provide a more direct validation of the practical value of this metric.
  3. What is the main reason that Crosscoders outperform SAEs?
  4. Can the conclusions generalize to complex reasoning tasks? I suggest that the authors evaluate them on multi-hop question answering, mathematical computation, synthetic algorithmic tasks, or data with known causal structures, in order to determine whether punctuality can truly identify intermediate reasoning steps inside the model that have clear semantics and causal effects.
2 comments
AuthorsOpenReview ↗

Response to Reviewer QwzM

Thank you for your detailed review. We provide clarifications here and all additions will appear in the updated version of the paper.

Weaknesses

  1. This is a great question. We would like to point out that punctuality is a metric of circuit interpretability quality, separate from metrics like faithfulness and minimality. As such, the goal is not to directly produce better circuits using highly punctual features, but to evaluate whether a proposed circuit can be interpreted as a sequence or relationship of ordered steps. We can also evaluate dictionaries of features according to whether those features could be employed in a step-by-step circuit.
  2. We’ve repeated the experiments on several new models spanning different model families, scales, and SAE types, please see the global response to all reviewers for the results. We are also working on a math benchmark and hope to share those results soon.

Responses to questions

  1. If a model reasons entirely through continuous, gradual representation updates, then it would be hard to construct a meaningful circuit. The edges in circuits imply order; that if upstream feature A points to downstream feature B, then A is finished computing before B. We would be forced to either view the features in a continuous model as only “finished” computing at the final layer, in which case the circuit can only be 1 layer deep and is vacuous because it does not break down the model’s computation, or we could label each layer’s computation as a new feature, in which case the circuit is vacuous because it does not sufficiently abstract the model’s computation in a way useful for human interpretability. The difference in discrete-step explanations vs general mechanistic explanations is whether steps are ordered with respect to each other. For example in figure 1, model B which gradually computes Austin can be described mechanistically as “gradually upweight Texas concepts and capital concepts in parallel”, but this is not a discrete-step mechanism because it only has a step depth of 1.
  2. Please see response to weakness 1 above.
  3. We believe crosscoders outperform SAEs because their training objective (to find features that persist across multiple layers, while deduplicating features from early layers) incentivizes persistence in earlier layer features and punctuality in later layer features.
  4. Please see response to weakness 2 above.

Summary

You identified the lack of demonstration that punctual features produce better circuits, and limited experimental coverage, as the critical weaknesses of our work. If we have convinced you that punctuality is a metric of circuit interpretability quality and not circuit quality, and you find the additional experimental results compelling, would you consider raising your score?

Thank you for the response. Your response addresses most of my concerns, and I will maintain my positive score.

Reviewer_hzhHOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
2(+1)
Read review
Paper summary

The paper presents the idea of computations that happen more in discrete steps are easier to interpret in circuits. In order for steps to be discrete, the authors argue that the intermediate units of computation should become active within a few layers than across many layers. The authors define a metric called punctuality that measures how different intermediate units are computed continually vs over many layers. The authors look at a circuit found in previous work on the Bias-in-Bios task and show that the features in the circuit satisfy this property of punctuality. The authors also show that neurons and crosscoder features are more punctual than SAE features and argue that punctuality is an important metric to judge interpretability tools for unsupervised feature discovery.

Critical strengths
  • The motivation of the paper for neural network circuits satisfying the property of computation happening in discrete units is valuable and important topic to study for the field. Not much of the field has focused on this concern and the paper attempting to operationalize this using a metric is interesting.
  • The Bias-in-Bio case study is suggestive of the author asking the right questions.
Critical weaknesses
  • Punctuality, as defined, seems like an okay metric but doesn't really support the conclusions the authors claim in this work. The authors use punctuality to show that neurons and crosscoder are more punctual units of computation, but the appendix also show that random directions in activation space also score high on punctuality, so comparison based on punctuality alone is not sound. The authors show that punctuality is anti-correlated with causal importance, so it seems like a metric that combines the notion of punctuality and causal importance or persistence may be the right thing to look at.
Questions for authors
  • Neurons are the least causally important group at every layer in figure 12. Why does being less punctual than neurons make SAE features "poorly suited"?
  • Does the anticorrelation in figure 4 survive among persistent features only?
  • Can you define random direction properly in the paper and compare in the main text and add implications of the comparison to your conclusions? I feel like your conclusions are not well supported if you compare punctuality across everything, then it seems like a metric that could support arbitrary conclusions (and should definitely not be something that interpretability tools should optimize for).
  • Why do you normalize by L − 1? The same computation scores higher at deeper layers as a result of the normalization. Are comparisons across layers intended, and could you report effective layers (k) instead?
4 comments
AuthorsOpenReview ↗

Response to Reviewer hzhH

Thank you for recognizing that our work studies a valuable, important, and neglected topic, and for finding our work interesting. We provide clarifications here and all additions will appear in the updated version of the paper. Please see appendix A.4 for figures referenced here.

Weaknesses

We agree that punctuality is insufficient by itself as a metric of feature quality – indeed it is not intended to be a catch-all metric but rather to isolate one understudied aspect of what makes a feature interpretable, especially as part of a larger circuit. Section 5 explores the tension between two desirable feature properties, punctuality and causal importance, and in general we should consider punctuality among other desiderata. Maybe it is possible to construct a meta-metric that encapsulates multiple desiderata, but our goal here is to isolate the evaluation of whether features find a step in model computation. The tension between punctuality and causality would be hidden if we used a singular metric.

Response to questions:

  • You are completely correct that SAEs being less punctual than neurons does not make them poorly suited for circuits in an absolute sense. Rather, they are worse than neurons at finding steps, but they are better than neurons on other properties like causal importance. We will revise the paper to make this crucial point clear.
  • Thank you for this suggestion, this is an interesting experiment to run! We check correlations across persistence quantiles and find that the more persistent quantiles roughly are less anticorrelated, but this is not robust across all models. Please see Appendix A.4 for the figure.
  • The random direction baseline uses uniform random directions on the unit hypersphere as feature directions. Equivalently, one can think of it as rotating an SAE feature direction arbitrarily along the unit hypersphere. We control for sparsity by choosing the threshold at which the feature is considered active to match the proportion of tokens the feature fires on to SAEs. Thank you for pointing out this was unclear, we will clarify in the main text.
  • Can you clarify what you mean by “compare punctuality across everything”?
  • We normalize by L-1 rather than L because if one layer does all the work (k_EC=1) then we want to have a punctuality score of 1. It’s true that the same computation at a deeper layer scores higher on punctuality; this is a great point. Whether we normalize the effective layer count or not measures relative or absolute punctuality: normalized tells us # layers of work relative to how deep in the model, which bakes in the assumption that features in later layers tend to be more complicated. We think both relative and absolute versions are informative, and further investigation here would be a great direction for future work. Please see A.4 for a plot of effective computation layers.

Summary

You identified the critical weakness of our work as the fact that punctuality is insufficient alone as a metric of feature quality, and that it might be better to incorporate persistence and causal importance all together. If we have convinced you that we need not construct a single metric for all these desiderata, and that indeed the goal of punctuality is to isolate one specific property to study, would you consider raising your score? Please let us know if we can address any further questions.

compare punctuality across everything

The main text compares punctuality across SAE and neurons and shows that SAE features are less punctual than neurons. However, figure 22 shows that uniform random directions are also more punctual than SAE features. So I'm not really sure what to really conclude from such comparisons. What I'm confused about is how do the different comparisons all come together.

I agree that punctuality could be an independent measure for judging the quality of units of a circuit to claim something like, all else being equal, a circuit made up of punctual features is easier to interpret as ordered steps. The all else being equal looking at other more important metrics like causal importance, persistence, etc. But from Figure 22, I see that uniform random directions have higher causal importance than neurons which makes me think that the comparison of SAE and neurons on punctuality doesn't really tell anything, because SAEs have more causally important units than neurons.

My main point is that looking at punctuality in isolation is bad, because it's not a catch-all metric as you agree. So section 4 comparison doesn't really have a takeaway. In comparison, section 6 seems better since Figure 7 compares both punctuality and causal importance simultaneously, and shows that crosscoders make pareto improvement on both metrics, thus providing a takeaway of crosscoder is a better basis than SAE for finding circuits composed of discrete features that are causally important as well.

Thank you for the persistence-quartile analysis. It looks like the most persistent quartile shows roughly no anti-correlation at any layer for gemma-3-1b. I think Section 5 should say that the tension between punctuality and causal importance is concentrated in the less persistent features. I would like to see more of a joint analysis of these metrics since all these metrics are important and comparing them alone could give a wrong takeway.

The paper also seems to be conflating parallel vs sequential computation and gradual vs stepwise emergence. The paper and the rebuttal replies argue for stepwise emergence being necessary for ordering. However, two gradual features whose availability rises over different layer ranges can be ordered by when they emerge, and the case study orders its clusters this way. Appendix A.9 makes the same point that ordering depends on which layer a feature becomes available at, not only on how quickly it becomes available. So punctuality captures gradual vs stepwise, which doesn't seem that important as compared to parallel vs sequential for interpreting circuits.

Another issue I see with the metric is that I'm not sure what comparisons are fair to make given how the metric is defined. Are comparisons across layers valid? Are comparisons across models valid? I would like to see the tradeoffs in defining such a metric. I read appendix section A.9 but didn't get much out of it in terms of trade-offs and whether punctuality as defined holds more desirable properties of having such a metric compared to any other alternatives.

Happy to continue discussing.

Thank you for your response. We are happy to hear that you agree punctuality is important, because that argument and an operationalization of it is the main contribution we intend. We agree that we cannot use punctuality in isolation to evaluate features broadly and absolutely; the connection between the different desiderata and the difficulty of considering punctuality in isolation is a major result and theme in the paper.

It is true that neurons > uniform random features > SAE features on punctuality, while SAE > uniform random > neurons on causal importance. Treating uniform random features as the baseline, these results together tell us that SAEs and neurons have different strengths and weaknesses, scoring higher vs lower on the 2 desiderata examined here. We will revise Section 4 to emphasize the relative comparison of punctuality between SAEs, neurons, and random directions, rather than make absolute claims. We will also make it clearer in Section 5 that different feature types have different trade-offs and currently no feature type wins across the board on all the metrics.

I think Section 5 should say that the tension between punctuality and causal importance is concentrated in the less persistent features.

We agree and will discuss this finding in the paper. We think it is potentially promising for dictionary learning methods that much of the anticorrelation in gemma-3-1b seems to be concentrated in the less persistent features which would not have been good candidates for mechanistic interpretation anyway, but further investigation is needed to understand what happens in the other models.

punctuality captures gradual vs stepwise, which doesn't seem that important as compared to parallel vs sequential

This is correct, punctuality captures gradual vs stepwise. It is possible for gradual features to happen in sequence and stepwise features to happen in parallel, but more punctual features have more potential to occur in sequence. A feature that emerges gradually over layers 2-9 in a model with 10 layers has little room to be ordered before or after a feature. On the other hand a feature that emerges in a single layer is very easily ordered with respect to other punctual features. In the case study, e.g. the green cluster, though the features make progress through much of the model, there is a clear jump at layer 8 which would register them as more punctual.

Are comparisons across layers valid? Are comparisons across models valid?

The metric is intended to only be used to compare across features found in the same layer of the same model. The case study supports that this is a fairly reasonable requirement because we can find features in a later layer that emerged primarily at an earlier layer (green features after layer 8). We will emphasize this in the text, thank you for pointing this out.

Among the different variants we study in the appendix, all are intended to only be used to compare within features in the same layer of the same model. We suspect it may be difficult to design a metric that supports comparison more broadly, due to block structure that could potentially dominate availability differences at particular layers, and large differences/heterogeneity across models.

Reviewer_hzhHOpenReview ↗

Thanks

Thanks for the response! Most of my concerns have been addressed. The paper needs a bit of rewriting work to improve clarity and add more discussion around certain claims as the authors agree are worth doing. I have updated my score to a 3 for weak acceptance. The paper is interesting and a valuable contribution to the interpretability research community. The rewriting will help make it more legible.

Reviewer_Eg3AOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The authors address the problem of interpretability of features in a deep neural network by defining measurable characteristics that make features interpretable and understanding their causal importance across layers. The hypothesis of the authors is that interpretability is aided by features that become suddenly available in a discrete step rather than gradually

They propose two metrics - punctuality and persistence. Punctuality is the measure of how suddenly a feature emerges in the layer of a model, persistence is how it persists beyond a layer. They consider the "Bias-in-Bios" case study circuit from Marks et al. (205) and show that there are some features that are punctual and some that are not (gradual). Their main experimental setup consists of a gemma-3-1b-pt model with pre-computed Matryoshka SAEs. They use a small subset of the Pile dataset for analysis. One of the main insights the authors come up with is that punctuality is negatively correlated with causal importance (for next token prediction). Then next insight is that most punctual and least persistent features are usually noise.

They then move on to analyze crosscoders and show that later layer crosscoder features are more punctual and earlier layer crosscoder features are more persistent.

Critical strengths

One of the things that I really liked about the paper is the general focus on feature evolution. There have been many works that study SAE circuits but there is not much work on what makes features interpretable. The general idea presented here is interesting.

The insights make sense at a high level - in SAEs more punctual/less persistent features can be ephemeral, punctual features can be less causally important, and in cross-coders this tradeoff may get better since cross-coders learn features across layers.

Critical weaknesses

I do not understand the need for separating out punctuality and persistence, to me it appears that these are the two sides of the same coin only conditioned by where we are looking. For example, one could easily come up with a measure of how the features are locally concentrated around a single layer as a measure of how interpretable they are. Having two measures instead of one is confusing sometimes since it feels like we could get at the same insights by using just a single measure. Also why does this "discrete, step-wise" characteristic so important to make a feature more interpretable - a bit more justification here would be helpful.

The experimental validation seems quite limited - for the general insights the authors want to claim. Bias-in-Bios case study is nice, but the real experiments use a single model for demonstration.

Questions for authors

It would be nice if the authors can provide their thoughts on the weaknesses I mentioned to the extent possible. A few other clarifications would also be helpful for the reader to understand the paper.

  1. What is a verbatim feature direction (line 122)?
  2. There are a bunch of known problems with SAEs including the fact that not all features used by the network are discoverable and problems like feature absorption and splitting - how do those impact the punctuality and persistence measures?
  3. The computation of causal importance in lines 297-298 can be explained better.
  4. In Fig. 4, the correlation gets weaker as layers progress. What does this mean?
  5. In Fig. 7 why not use the same correlation measures used for SAEs?
  6. In Fig. 6, the median persistence of cross oder seems to be fairly high even in later layers. The punctuality claim seems to hold true slightly better.
  7. What does "incomparable" mean in Table 1, and how does this impact the insights obtained from this table?
5 comments
AuthorsOpenReview ↗

Response to Reviewer Eg3A (1/2)

Thank you for recognizing the interestingness and novelty of our work, and that punctuality is an understudied area. We provide clarifications here and all additions will appear in the updated version of the paper. Please see appendix A.3 for figures referenced here.

Weaknesses:

I do not understand the need for separating out punctuality and persistence, to me it appears that these are the two sides of the same coin only conditioned by where we are looking.

The reason we separate them out is because all features are guaranteed to be at their maximum availability at the layer they were found (the “home layer”), by definition. Therefore, we cannot treat their “home layer” as just another intermediate computation, because it anchors the definition of the feature. Therefore, almost any feature will slightly “peak” at its home layer, but punctuality measures how quickly they appear while persistence measures whether they stick around. Although persistence and punctuality are empirically correlated, they are independent, and a punctual feature might jump up at any previous layer, not just the most recent ones. We want to distinguish punctual features that are persistent or not. See A.3 for some example features that illustrate the distinction.

Also why does this "discrete, step-wise" characteristic so important to make a feature more interpretable

A major goal in interpretability research is to explain entire systems as pipelines or circuits, not just to look at individual features in isolation. For humans, a system is most interpretable if we can describe it as a sequence of steps rather than fuzzy gradual refinement. We speculate that more punctual features may lend themselves better to more symbolic explanations, which tend to be much easier to understand and have led to some successes in recent interpretability work, for example explaining attention heads with programs (Hayes et al 2026: Explaining Attention with Program Synthesis).

Even separate from whether steps are more interpretable or not in isolation, if researchers hope to describe latent reasoning as an algorithm or circuit, these objects involve sequences of ordered steps, and so it is important to explicitly check this presupposition.

the real experiments use a single model for demonstration

We’ve repeated the experiments on several new models (across model families and scales) and SAE types and found the results to hold broadly, please see the global response to all reviewers for the results.

AuthorsOpenReview ↗

Response to Reviewer Eg3A (2/2)

Responses to questions:

  1. Thank you for pointing out that “verbatim [feature direction]” is confusing, we will remove this in the paper. We mean the SAE feature direction, specifically the encoder direction since we measure feature availability (readability).
  2. These are interesting questions! Regarding SAEs not finding all useful features, we won’t be able to directly measure punctuality and persistence of these features but we can speculate that useful features are more persistent (the model has an incentive to “keep them around” if they are useful). We also speculate that feature splitting and absorption are orthogonal to persistence (if the feature is useful, whether it is split or not should not impact persistence), so our metrics here may not say much about these phenomena.
  3. Thank you for pointing out this was unclear! We have added details on the exact update applied to the residual stream in Appendix A.1.
  4. This is a great question. Features at later layers tend to be more causally important because they are closer to the unembedding decoding layer, so this may partially mitigate the tradeoff with punctuality. We also notice that filtering out features that make their largest gain at the final layer (where the features are found) – to cut out ephemeral noise – reduces the negative correlation, and in later layers there are fewer ephemeral features because the model is preparing to decode, so this may also partially explain it. A more detailed analysis would be a great direction for future work.
  5. We use a dominance analysis to compare crosscoders to SAEs because we would like to see if crosscoder features are jointly more punctual and causal than SAE features. If the distribution of crosscoder features was a strict translation up and to the right in figure 7, they would have the same punctuality vs causality correlation as SAE features while being Pareto dominant.
  6. It’s true that crosscoder features are more persistent even in later layers. We provide a reason to expect high persistence vs SAE features in early layers; it is possible there are other reasons to expect high persistence in later layers as well.
  7. Incomparable means neither feature is Pareto dominant (so the SAE feature wins on one axis and the crosscoder feature wins on the other). Thank you for pointing out that this was confusing terminology, we will change the wording to be “neither [dominates]”!

Summary

You identified separating out punctuality and persistence and narrow experimental scope as the critical weaknesses of our work. If we have convinced you that punctuality and persistence are distinct and the additional experimental results establish that our conclusions hold across models, would you consider raising your score? Please let us know if we can address any further questions.

AuthorsOpenReview ↗

Reminder to engage in the review process

Dear reviewer Eg3A,

Since today is the last day of the discussion period, we would greatly appreciate it if you could review and engage with our response. We believe our detailed response and additional experiments address your concerns, but please let us know if you have any further questions.

If we have successfully addressed your concerns, we kindly ask the reviewer to consider raising their score accordingly to positively support our paper.

Reviewer_FomMOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes punctuality to measure how concentrated gains in feature readability are across transformer layers. It compares residual stream SAEs, residual coordinates and crosscoders in Gemma models. The authors report that highly punctual SAE features often have low persistence and small intervention effects. Crosscoders show a better tradeoff between punctuality and output impact under the reported metrics.

Critical strengths

I like the question of whether dictionary features help locate steps of computation. Tracking availability across layers gives a diagnostic beyond reconstruction error. Looking at persistence and interventions alongside punctuality is also a strength. The examples show why abrupt feature emergence alone does not establish that a feature matters for the output.

Critical weaknesses
  1. The causal framing in Section 1 goes beyond the metric. Gradual accumulation can still feed a discrete downstream step. Low readability along one direction at the next layer also does not establish noise. The running maximum hides availability drops, while depth normalization assigns scores of 0.71 and 0.91 at layers 8 and 23 for the same effective count of 3 emergence layers (Definition 2.3). These properties need to be accounted for when interpreting the score and comparing depths. The general SAE claim also needs narrowing to the tested residual stream SAEs.

  2. The crosscoder comparison is hard to assess from Section 6. Source layers, directions, labels and intervention locations are unclear. These details determine whether the measured effects are comparable. The advantage weakens after sparsity stratification, with the 95% interval including 0 at layer 14 in the sparsest retained stratum (Figure 11). I think that the dominance claim should reflect this uncertainty. Closer frequency matching and retained feature counts would help establish how broadly the result actually holds.

Questions for authors
  1. Clarification: For crosscoders, how are the source layer, binary labels and projection direction defined? Does clamping modify one layer or several?

  2. Clarification: Is causal importance computed from on versus off, or unperturbed versus off? I believe that Section 5 describes both. Also, can you please give the exact update applied to the residual stream.

  3. Additional evidence: Does the crosscoder advantage remain with closer activation frequency matching and the same intervention scope? Please report retained feature counts and confidence intervals by layer and sparsity group.

  4. Clarification: Do Figures 3b, 4 and 5a exclude features whose largest gain occurs at the first transformer layer, as stated in Section 4.2?

2 comments
AuthorsOpenReview ↗

Response to Reviewer FomM 1/2

Thank you for the review and for recognizing the interestingness of the research question. We provide clarification and additional information below. We have added supporting figures to appendix A.2.

Weakness 1:

The causal framing in Section 1 goes beyond the metric

Punctuality is not intended to be a catch-all feature quality metric; rather, we intend it to isolate an understudied aspect of what makes a feature interpretable, especially as part of a larger circuit. Indeed there are many aspects of features that we desire, and sometimes these are in tension. We highlight that the current literature focuses on either causality or interpretability, and neglects punctuality—even when features are intended to be components in a circuit or pipeline.

Gradual accumulation can still feed a discrete downstream step.

This is true, but the more gradual it is, the harder it is to order a step downstream. Punctuality measures the potential for orderability, and admits degrees (is not binary).

Low readability along one direction at the next layer also does not establish noise.

True, which is why we treat “ephemerality” as a factor that relates a pattern of availability (sudden emergence followed by immediate disappearance) to causal impact. Therefore, we establish features as ephemeral noise using a combination of three properties: high punctuality, low persistence, and low causal effect.

The running maximum hides availability drops

Extreme availability drops are uncommon – it is unlikely that models “deliberately” make feature availability strongly non-monotonic – and thus we choose not to investigate it in this work by using the running maximum. Please see appendix A.2 for a plot of the ratio of total gain (including drops) over gain excluding drops. A feature whose availability increases monotonically would score 1 on this ratio, and a feature that exhibits drops would score higher. Most features have a ratio under 1.25 which demonstrates that availability is largely monotonic on individual features.

depth normalization assigns scores of 0.71 and 0.91 at layers 8 and 23 for the same effective count of 3 emergence layers (Definition 2.3).

It’s true that the same 3 layer computation at a deeper layer scores higher on punctuality; this is a great point, and we are careful to only compare features from the same layer when making general claims. Whether we normalize the effective layer count or not measures relative or absolute punctuality: normalized tells us # layers of work relative to how deep in the model, which bakes in the assumption that features in later layers tend to be more complicated. We think both relative and absolute versions are informative, and further investigation here would be a great direction for future work.

The general SAE claim also needs narrowing to the tested residual stream SAEs.

We specify residual stream in the abstract and the main text (the claim in the intro, section 4 setup), but we can absolutely make this clearer in the writing. We would also say that residual stream SAEs are the canonical choice due to the interpretation of the residual stream as “running notepad of what the model is thinking thus far”. For example, they are the site of choice for sparse feature circuits (Marks et al 2025).

AuthorsOpenReview ↗

Response to Reviewer FomM 2/2

Weakness 2:

The crosscoder comparison is hard to assess from Section 6. Source layers, directions, labels and intervention locations are unclear. These details determine whether the measured effects are comparable.

Thank you for pointing this out! We will add these details in the text. The Gemma Scope crosscoder is trained with one shared dictionary on the same 4 layers in the model as for their canonical residual stream SAEs: 8, 14, 18, and 23. It is a weakly causal crosscoder: “a latent’s encoder weights are restricted to a single layer and its decoder may reconstruct activations from that layer or any future layer. This ensures latents cannot use future-layer information to encode past activations.” (McDougall et al 2025). As with SAEs, the encoder and decoder directions (at each downstream layer of the set {8, 14, 18, 23} compared to the encoder layer) may be different in general. We measure availability using the encoder direction, as we do with SAEs. We intervene at the encoder layer on contexts where the feature is active at the ON 80th percentile by clamping the feature value to OFF 20th percentile only at the encoder layer. This is equivalent to the protocol for intervening on SAE features if we ignore the crosscoder feature’s future decoder layers. If we interpret the crosscoder as finding features persistently stored across layers, we are measuring its causal effect at the layer in which it was found (the encoder layer).

The advantage weakens after sparsity stratification, with the 95% interval including 0 at layer 14 in the sparsest retained stratum (Figure 11). I think that the dominance claim should reflect this uncertainty. Closer frequency matching and retained feature counts would help establish how broadly the result actually holds.

The layer 14 sparsest bucket’s 95% CI is [-0.00271, 0.0936]. It is true that this interval contains 0, but the dominance is significant across all other buckets at this layer and all the other layers we checked. Please see Appendix A.2 for an updated figure with bucket counts.

Responses to questions:

  1. Please see our response for “Weakness 2” above for the details on the crosscoder setup.
  2. Thank you for pointing this out. We are measuring causal importance by sampling contexts at which the feature is already active at the ON 80th percentile, and measuring symKL unperturbed there vs clamping to OFF 20th percentile. We have added details on the exact update applied to the residual stream in Appendix A.1.
  3. Can you clarify what you mean by “same intervention scope”? For feature counts per stratum, please see the figure in A.2.
  4. Figure 3b included first layer largest gain features, and figures 4 and 5a already filter them out. We have added a comparison for figure 3b with vs without the filtering in A.2. We find the results are similar between those conditions.

Summary

You identified the fact that punctuality is not a catch-all metric of feature quality and missing details in the crosscoder dominance claim as the critical weaknesses of our work. If we have convinced you that punctuality need not encapsulate all aspects of feature quality as it intends to isolate one specific aspect, and clarified that the crosscoder dominance result holds across layers and buckets, would you consider raising your score? Please let us know if we can address any other questions.

Comments on the paper

5 comments
AuthorsOpenReview ↗

Response to all reviewers: experiments on additional models and SAEs

We thank all reviewers for their helpful feedback and critical questions which have helped us improve our draft! We are glad that all the reviewers found the motivating research question compelling, important, and interesting. We are also excited to report results on additional models and SAEs which broadly support our conclusions.

We add the following models:

  • Gemma-3-1B; SAE dictionary width 65k, JumpReLU, L0 medium around 50
  • Llama-3.1-8B; SAE dictionary width 32k, topK (but converted to JumpReLU at inference), k=50
  • Qwen3-8B; SAE dictionary width 65k, topK, k=100
  • GPT-2 small; SAE dictionary width 32k, topK, k=32

Figures for the results below can be found in Appendix A.1:

  1. Punctuality distributions of SAE features vs neurons: broadly, neurons remain much more punctual than SAE features except for the smallest model we test, GPT-2 small, which has only 12 layers.
  2. Punctuality vs causal importance of SAE features: these properties remain negatively correlated except for Llama-3.1 8B, and the size of the negative correlation is generally lower in later layers, perhaps due to proximity to the unembedding decoding layer. We further find that filtering out features that make their largest gain at the last layer, in order to remove ephemeral features, generally reduces the size of the negative correlation to a modest extent.
  3. Punctuality vs persistence of SAE features: these properties also remain negatively correlated except for late layers in Llama-3.1-8B and gpt-2 small. The size of the negative correlation also generally is lower in later layers, perhaps due to proximity to the unembedding decoding layer.

Thanks for the extra model results and for clarifying the interventions with the crosscoder. That said, I still do not see enough evidence that a higher punctuality score helps explain the computation as a sequence of ordered steps. Furthermore, the current evidence and reply do not report the requested crosscoder comparison with more closely matched activation frequencies within each group. Overall, I will keep my score and encourage a resubmission that tests this link and revises the claims accordingly.

Thank you for your reply.

I still do not see enough evidence that a higher punctuality score helps explain the computation as a sequence of ordered steps

Figure 1 in the paper is an example where the model that solves the task in 2 hops (steps) uses features with high punctuality, whereas the model that does not use steps uses features that have lower punctuality. Model A (sequential punctual) actually uses 2 hops to compute the answer as intended, and its features for "Texas" and "capital" would score k_{EC}=1 and thus have a punctuality of 1. This means Model A's features are indeed steps, and therefore have more potential to be ordered in sequence. On the other hand, Model B (parallel gradual)'s features are computed spread throughout the entire model, and would have a high k_{EC} and therefore low punctuality. This means Model B's features do not capture steps and therefore are difficult to be ordered in sequence. We built a model with mock activations to verify that model A's features indeed have 1 effective computation layer and so a punctuality of 1, and model B's features have k_{EC}=5.39 and punctuality of 0.60: https://imgur.com/a/70ujFdl (the availability curves on the left are from measuring PR AUC empirically)

We regard the importance of computation over a small range of layers to be an intrinsic definition of what it would mean for a feature to be a “step”, but we are happy to collect empirical evidence beyond the construction above, if you can offer a definition of what a “sequence of ordered steps” might be without that notion. Do you have a specific experiment suggestion?

requested crosscoder comparison with more closely matched activation frequencies within each group

Can you clarify if you mean more bins than 20?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.