acceptodds
Under review as a conference paper at ICLR 2027

What Carries Mathematical Computation? Tracing Functional Changes in Language Models

Abstract

Modern language models exhibit strong mathematical capabilities, yet fine-tuning on specific tasks can compromise performance on other mathematical problems. Can the changes supporting acquired mathematical skills be isolated within a compact update? We propose SAE-guided update selection, which connects changes in internal representations to the selection of parameter updates. A shared sparse autoencoder (SAE) identifies feature changes between the base and fine-tuned models. We then score rank-one terms of the LoRA update by their effects on recovering fine-tuned feature values and predicting correct solutions, retaining a subset under a fixed storage budget. Empirically, across four model pairs from the Qwen, Gemma, and Llama families, incorporating feature matching improves selection over prediction loss alone. The joint criterion surpasses the full adapter on addition, subtraction, and variable isolation on three pairs, while also preserving more GSM8K accuracy. Beyond selection performance, we examine whether the identified features contribute to the computation itself. Removing them impairs later computation and final answers even after supplying the correct next token. Moreover, with the selected update applied, restoring features associated with addition carry to their base values selectively disrupts carry-dependent solutions. Together, these findings connect the retention of mathematical skills in compact updates to internal features that support successive computation steps.

Primary area:
interpretability and explainable AI
Keywords:
Mechanistic Interpretability, Sparse Autoencoders, Mathematical Reasoning, Parameter-Efficient Fine-Tuning, LoRA, Model Editing, Feature Intervention, Parameter Update Selection

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_nANGOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies which parts of a LoRA update are needed to keep the mathematical skills learned during fine-tuning. The authors use a shared sparse autoencoder (SAE) to find internal features that change after training. They then split the LoRA update into rank-one components and score them using answer prediction loss and a loss that matches the fine-tuned model’s feature values. The highest-scoring components are kept under a fixed storage budget. Experiments cover addition, subtraction, and simple equations on four models. Using 25% of the adapter storage, the combined score outperforms full LoRA on three models and reduces their loss of GSM8K accuracy. The paper also intervenes on selected features to test their role in later computation, including whether deleting them still causes errors after the correct next token is supplied.

Critical strengths

(1) The main comparison shows a consistent benefit from feature-based scoring. Behavior, SAE-only, and Joint use the same candidate components, storage budget, and allocation rule. SAE-only outperforms Behavior on all four models, and the advantage holds across the tested budgets. This provides useful evidence that feature matching adds information beyond answer prediction loss.

(2) The intervention experiments include meaningful controls. Deleting selected features still harms later computation when the correct next token is supplied, so the damage cannot be explained solely by an incorrect immediate prediction. Restoring carry-related features also affects carry problems much more than matched no-carry problems. These results give causal evidence that some identified features contribute to computation.

Critical weaknesses

(1) The experiments do not establish what SAE specifically contributes. SAE-only outperforming Behavior shows that matching the fine-tuned model’s internal features helps, but it does not show that SAE features—or selecting those that change most—are important. A simpler hidden-state matching objective could potentially provide the same benefit. Without this comparison, the main gain could come from additional supervision from the fine-tuned model rather than the proposed feature analysis. This leaves the paper’s central technical contribution insufficiently supported.

(2) The practical value of the full pipeline remains unclear. The method requires a fully trained LoRA adapter, activation collection, SAE training, and gradient scoring before producing a smaller adapter. The reported storage reduction covers only the adapter. The experiments do not establish whether this process offers enough benefit over directly training a similarly sized adapter or using simpler compression methods to justify the additional work. Beating the full adapter on these tasks is interesting, but it does not by itself establish a compelling reason to use this pipeline.

Questions for authors

(1) The feature-matching objective looks quite similar to directly matching the fine-tuned model’s hidden states. Have the authors tried this simpler baseline? I would like to know whether SAE actually performs better, since it adds another model to train and another set of features to select.

(2) I understand that removing the carry-related features hurts Joint, but how does this explain why Joint works better than Behavior? Does Joint preserve these features better? If the same features are removed from both models, does Joint still have an advantage?

(3) For practical use, why train a rank-64 LoRA and then reduce it to the storage size of rank 16, rather than train rank 16 directly? Have the authors compared the two? I would like to see how much performance this extra process gains.

(4) The method scores components using local gradients, but then removes many components at once. Have the authors checked whether removing high-scoring components actually hurts performance more? Also, does averaging gradients along the path work better than just computing them at one point?

(5) GSM8K seems to be the only benchmark used to check whether existing capabilities are retained. I do not think results on this one benchmark are enough to draw a broader conclusion about reducing interference, even with several models. Have the authors checked whether the same benefit holds on other mathematical tasks?

1 comment

Dear Reviewer nANG,

We understand that this is a busy period and would like to gently follow up on our response. When you have a chance, we would be grateful to hear whether the additional experiments and clarifications have addressed your concerns, or whether any questions remain. We remain happy to provide further clarification.

Thank you again for your time and consideration!

Reviewer_czYAOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates which parameter updates carries the math capabilities acquired during finetuning, and whether the corresponding representation features actually support subsequent computation. They finetune a math LoRA and use SAE to build a shared feature space for the base model and the finetuned model, and then find the SAE features that varies the most after finetuning. LoRA SVD components are evaluated and selected from their effect on prediction loss and effect on SAE feature recovery. Extensive experiments span across 4 LLMs, 3 types of tasks, and baseline strategies with various configurations. They report that SAE feature guided strategies perform better than the behavior based. Mechanistic causal intervention is also introduced to examine SAE features' effect on computations in subsequent states and final answers

Critical strengths

I find the idea of connecting parameter-update selection with representation-level analysis well-motivated and interesting. The experiments are designed with careful controls and provide evidence that behavioral loss alone may not be the best metric for evaluating a compact fine-tuning update, which is useful and intuitive. The interventions also gives further insightful results which make their story and implementation very complete.

Critical weaknesses

About what the SAE-guided selection actually contributes

I think this paper might benefit from additional analyses on the experiment results. The main results in Figure 2 and 3 shows that SAE-based scoring is always more effective than the loss based, however it does not further confirm whether this advantage comes specifically from the discovered sparse features, from matching the fine-tuned model’s internal representations more generally, or from focusing the scoring objective on particular math-related computation positions, etc. The complete pipeline for SAE based method has introduced numerous intermediate processes, like sparse regularization, normalization, feature selection threshold, etc. which, if can be further carefully ablated, for example, running further experiment with matching the raw residual activations or random SAE feature selection, seems quite necessary and would potentially provide more helpful insights. The current random baseline in Fig 2 directly selects parameter components, and the current random SAE feature experiments only concern activation interventions. These current results do not yet satisfactorily ablate the additional value of the core components within the proposed SAE-specific design.

About interpretation of intervention experiments

I think the results of the intervention experiments should be analyzed and interpreted more carefully. The fixed-token continuation experiments support the claim that the intervention affects later outputs through something other than the identity of the immediately generated token, yet this does not distinguish disruption of a specific computation-related state from a more general disruption of the internal representations used by later next-token predictions. Therefore the carry-specific restoration experiments becomes really important. However, neither Section 3.5 nor Appendix D provide sufficient details how and where (which layers) the related features were identified, or how the no-carry and random controls were implemented.

Overall, the experiments provide evidence that the selected directions are causally relevant to performance. I am less certain that they sufficiently establish what computational information those directions carry, which is central to the paper’s interpretation. I believe the authors should document more results here.

Questions for authors

Please address the major concerns listed above in Critical Weaknesses. I also have found some minor points:

  1. Several design choices and configurations essential for the reproducibility are unspecified. This includes for SAE the sparsity levels, selected feature counts, etc., and for the training part the learning rates, batch sizes, schedulers, etc. I'm not sure if Table 1 and 6 also miss information like the specific model, feature counts, and evaluation sample sizes, etc.

  2. Since main results in Fig 2 and 3 reports results averaged over 3 seeds, they can be improved with uncertainty estimates visualizations, like error bars or confidence bands, to provide more informative results on how consistent and robust the improvements are.

  3. Section 3.5 line 414 "Feature 731 has mean activations..." does not report which layer this feature is from.

3 comments

Dear Reviewer czYA,

We understand that this is a busy period and would like to gently follow up on our response. When you have a chance, we would be grateful to hear whether the additional experiments and clarifications have addressed your concerns, or whether any questions remain. We remain happy to provide further clarification.

Thank you again for your time and consideration!

I appreciate the additional experiments and clarifications you provided. My main concerns have been well addressed. I look forward to seeing these new results reflected in an updated version.

I will keep my rating as 3 (weak accept), but I am slightly more positive about this submission.

Dear Reviewer czYA,

Thank you so much for your timely and encouraging reply! We are thrilled to hear that our additional experiments and clarifications have well addressed your concerns and made you more positive about our submission. Your thoughtful feedback played a vital role in helping us strengthen the rigor and clarity of this paper.

We deeply appreciate your time, effort, and support!

Reviewer_7NufOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates which fine-tuning updates preserve acquired mathematical capabilities and whether the feature changes induced by fine-tuning contribute to later mathematical steps beyond predicting the immediate next token. It proposes combining SAE-derived feature changes with prediction loss to select rank-one LoRA components under a fixed storage budget. Across four model pairs, it reports improvements over prediction-loss selection and gains over Full LoRA on both target tasks and GSM8K for three pairs. It further examines the computational role of the identified features through fixed-next-token and carry-related interventions.

Critical strengths
  1. A useful empirical signal under a controlled selection comparison. The paper proposes using SAE-derived feature changes to select compact parameter updates that preserve fine-tuned mathematical capabilities. Empirical results across four model pairs show improved target-task accuracy over prediction-loss selection, supporting representation changes as a useful signal for update selection.
  2. Improved task performance with smaller updates. The paper uses compact update selection to preserve learned mathematical capabilities while reducing performance loss on other tasks. Joint improves both target-task and GSM8K accuracy over Full LoRA on three of four model pairs, showing that a smaller update can improve task performance rather than merely save storage.
  3. Practical value of the selection method. The proposed method converts feature-based analysis into a compact update that can be used directly for generation. By keeping SAE-based processing in the offline selection stage, it allows the resulting model to benefit from representation analysis without requiring additional feature extraction or intervention during inference.
Critical weaknesses
  1. The contribution of SAE features is not isolated. The paper does not explain how annotated computational states are mapped to the token positions used for feature matching. Moreover, Behavior scores the complete solution, whereas feature matching uses these selected positions (Equation 2). The gains may therefore partly come from focusing the scoring objective on computation positions rather than from SAE features. The experiments do not isolate these effects through a comparison that evaluates prediction loss at the same positions.
  2. The computational cost of selection is not evaluated. The paper compares methods at matched factor-storage budgets but does not report the time or memory required to produce the selected updates. SAE-guided selection additionally requires activation collection, SAE training, and feature-based gradient scoring. Without a computational cost comparison against SVD and Behavior, it remains unclear whether the accuracy gains justify the additional offline processing.
  3. Limited evidence beyond structured mathematical tasks. The method is evaluated on synthetic arithmetic and equation-solving tasks with verified intermediate states and annotated computation positions. It remains unclear how the selection procedure would apply to less structured tasks where such annotations are not readily available. GSM8K measures performance retention rather than the effectiveness of applying the method to a new task, leaving its applicability to broader mathematical reasoning or code generation unestablished.
Questions for authors
  1. Experimental details: Please provide key SAE and fine-tuning settings, evaluation set sizes, and per-seed results. Does the stated 25% storage budget refer to the entire Full LoRA adapter?
  2. SAE contribution: How are computation-token positions identified? Does the advantage over Behavior persist when prediction loss is evaluated at those same positions?
  3. Computational cost: What are the total selection time and peak memory usage, including activation collection, SAE training, and scoring, compared with SVD and Behavior?
  4. Applicability: How would the method select scoring positions for tasks without annotated intermediate states? Has its effectiveness been evaluated in such a setting?
2 comments

Dear Reviewer 7Nuf,

We understand that this is a busy period and would like to gently follow up on our response. When you have a chance, we would be grateful to hear whether the additional experiments and clarifications have addressed your concerns, or whether any questions remain. We remain happy to provide further clarification.

Thank you again for your time and consideration!

Dear Reviewer 7Nuf,

We sincerely appreciate your time and understand that you may be busy. We would like to follow up once more, as we have provided additional experiments directly addressing your main concerns, including the matched-position control, computational overhead, and evaluation without position annotations.

We would be very grateful if you could briefly let us know whether these results have resolved your concerns. If so, we would sincerely appreciate your reconsideration of the rating. If any concerns remain, we would be happy to address them.

Thank you again for your time and consideration!

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.