- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper investigates whether the features used to construct step-by-step reasoning circuits with sparse autoencoders (SAEs) truly correspond to discrete steps in the model's forward computation. To this end, the authors propose the punctuality metric, which measures whether a feature suddenly becomes readable within a few layers or gradually forms across many layers. The authors analyze an existing SAE circuit on the Bias-in-Bios task and find feature clusters that can indeed be ordered by their time of emergence. They also find that the punctuality of SAE features is negatively correlated with causal importance: many highly punctual features disappear rapidly and have almost no effect on the model's output. Finally, the paper compares SAEs with Crosscoders. The results show that Crosscoders find more persistent features in earlier layers and more punctual features in later layers, and overall outperform SAEs jointly on punctuality and causal importance.
Critical strengths
- The definition of punctuality is clear. The authors construct punctuality from layer-wise feature availability and the participation ratio, giving the metric a clear intuitive interpretation. Punctuality is high when a small number of layers account for most of the progress in forming a feature, and low when many layers jointly and gradually advance its formation.
- The paper first uses the Bias-in-Bios circuit to show that punctual features can indeed exist. It then compares SAEs with neurons at the distributional level, analyzes the relationships among punctuality, persistence, and causal importance, and finally uses Crosscoders to demonstrate a possible direction for improvement.
- The authors compare multiple definitions of punctuality, measure availability in different ways using the original feature direction, linear regression, and logistic regression, and use multiple methods to control for sparsity differences between SAEs and neurons. When comparing SAEs with Crosscoders, the authors also conduct a stratified analysis based on activation sparsity. These supplementary experiments strengthen the credibility of the main conclusions to some extent.
Critical weaknesses
- The paper has not directly demonstrated that high punctuality produces better circuits. The authors propose punctuality as an important criterion for selecting circuit features, but they do not construct circuits separately using high-punctuality and low-punctuality features, nor do they compare these circuits in terms of faithfulness, sufficiency, minimality, human interpretability, or behavioral predictive ability.
- The experimental coverage is relatively limited. The main experiments focus on the Gemma model family, Gemma Scope SAEs and Crosscoders, Pile data, and one Bias-in-Bios circuit case study. Given that the paper discusses general questions of latent reasoning and circuit interpretation, it remains unclear whether the conclusions generalize to other model families, language models of different scales, other SAE architectures, and tasks involving mathematical reasoning, multi-hop question answering, or algorithms.
Questions for authors
- If a model reasons through continuous, gradual representation updates, could low-punctuality features still form faithful and meaningful circuits? Could the authors distinguish more clearly between being suitable for discrete-step explanations and being suitable for general mechanistic explanations?
- Does high punctuality actually improve circuit quality? Could the authors construct circuits separately using high-punctuality and low-punctuality features and compare their faithfulness, sufficiency, completeness, minimality, and intervention accuracy? Such an experiment would provide a more direct validation of the practical value of this metric.
- What is the main reason that Crosscoders outperform SAEs?
- Can the conclusions generalize to complex reasoning tasks? I suggest that the authors evaluate them on multi-hop question answering, mathematical computation, synthetic algorithmic tasks, or data with known causal structures, in order to determine whether punctuality can truly identify intermediate reasoning steps inside the model that have clear semantics and causal effects.
2 comments
Thank you for the response. Your response addresses most of my concerns, and I will maintain my positive score.
Response to Reviewer QwzM
Thank you for your detailed review. We provide clarifications here and all additions will appear in the updated version of the paper.
Weaknesses
Responses to questions
Summary
You identified the lack of demonstration that punctual features produce better circuits, and limited experimental coverage, as the critical weaknesses of our work. If we have convinced you that punctuality is a metric of circuit interpretability quality and not circuit quality, and you find the additional experimental results compelling, would you consider raising your score?