acceptodds
Under review as a conference paper at ICLR 2027

TopoTuner: Topological Finetuning of Large Language Models

Abstract

Full fine-tuning makes all pretrained parameters trainable and can be computationally expensive. LoRA reduces the number of trainable parameters, but it does not directly answer which pretrained components should be trained and which can be frozen during adaptation. We introduce TopoTuner, a topology-guided fine-tuning framework for selective freezing of attention projection matrices. TopoTuner treats each projection matrix as a row cloud and uses Wasserstein distances between persistence diagrams to measure how its topology changes during fine-tuning. TopoTuner learns a reusable freezing profile from a source dataset and reuses the selected matrix indices to efficiently fine-tune models on unseen target datasets, both within and outside the source domain. We evaluate whether the same freezing decisions remain effective across question answering and sentiment analysis tasks. Across LLaMA-3.1-8B, Mistral-7B-v0.3, and Qwen3-8B-Base, TopoTuner is competitive with full fine-tuning while updating only 0.56%–4.85% of pretrained parameters and outperforms LoRA in 8 out of 9 model-dataset settings. Along with minimized updates, TopoTuner reduces training time by 16.5% relative to full fine-tuning and 5.3% relative to LoRA on average. TopoTuner opens a new direction for Efficient AI where matrix-freezing decisions learned on one dataset can be reused across multiple tasks.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Large Language Models, Topological Data Analysis, PEFT, Efficient Fine-Tuning

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_VzRLOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes TopoTuner, a way to decide which attention matrices of an LLM should be trained and which should be frozen during fine-tuning. The main idea is to look at the rows of a weight matrix as a point cloud and describe its shape with persistent homology, using H0 diagrams built from cosine distances between rows. The authors compare the diagram of a matrix before and after fine-tuning with a Wasserstein distance and get a score of how much its structure has changed. The authors compute these scores once, by fully fine-tuning the model on GSM8K, and use them to build a freezing profile. In practice only the two projection families that change the most are trained, and in each of them the three layers with the highest score are frozen anyway, as a kind of regularization. This profile is then applied as it is to other tasks. The same score is also used to stop training early, when it stops changing from one epoch to the next. The experiments are on LLaMA-3.1-8B, Mistral-7B and Qwen3-8B, where the method gets close to full fine-tuning and beats LoRA in most settings, while also being faster.

Critical strengths
  1. Interesting question and practical idea. The paper does not only try to reduce the number of trainable parameters, but asks which parts of a pretrained model really need to change during fine-tuning. I find this a nice point of view. I also find it very interesting to use topology to get a kind of map of the relations between the weights of a matrix. Looking at how the rows are arranged with respect to each other, and how this arrangement changes during fine-tuning, says something different from simply measuring how much each weight moves, and it could be useful also outside this paper. The reusable profile is cheap and simple to use: it is computed once on a source task and then it is just a list of matrix indices. The rankings are also quite stable across tasks (Table 3), which suggests that which matrices change most depends partly on the pretrained model and not only on the task.

2 Broad experiment coverage The profile is built on GSM8K and reused on three different target tasks (SST-2, IMDB and MMLU), so the transfer is tested from math reasoning to both sentiment classification and general knowledge QA. All of this is repeated on three 8B models from different families (LLaMA, Mistral and Qwen), where the ranking selects different projections (V and O for LLaMA and Mistral, Q and K for Qwen). This shows the method is not tied to one architecture. The baselines cover different strategies: low-rank adapters (LoRA), parameter selection (Spectrum), layer dropping (DropBP) and a magnitude-based ranking that keeps the same pipeline but changes only the score. Besides accuracy, the authors measure training time and forgetting on a broad set of retention tasks, which matters a lot for fine-tuning in practice. The early-stopping experiment also shows that the same score can cut 50–67% of the training epochs while keeping almost the same validation accuracy.

Critical weaknesses
  1. Not clear how much of the gain comes from the topology. From Algorithms 1–3, TopoTuner trains only the two projection families that change the most on average (e.g. V and O for LLaMA) and freezes everything else, MLPs included. The topological ranking is then used only to freeze 3 of the 32 layers in each family, so about 10% of the trained matrices. Most of the reduction in trainable parameters, and maybe most of the gain, could come from the choice of the two families and from freezing the MLPs, and not from the topology. The budget ablation (b ∈ {3, 6, 9, 12, 15}) never includes b = 0, so training the two families without freezing any layer is never tested. Recent work also suggests that in pretrained models it matters little which weights are trained: SliceFine (Kowsher et al., ICML 2026) gets almost the same accuracy when the trained slices are the most important, the least important or random ones. I think two baselines are needed: training the same two families with no frozen layers, and freezing 3 random layers per family instead of the ones chosen by the topology.

  2. The topological score loses information. The H0 diagram only keeps the distances at which rows merge, not WHICH rows merge. Then the Wasserstein distance compares these values without using the fact that row ii before fine-tuning is the same row ii after fine-tuning, even if this is known. So if rows move from one cluster to another but the merge distances stay similar, the score does not see it. Also, cosine distance does not look at the norm of the rows, so if a matrix is rescaled the score is zero, even if the output of the layer changes. The choice of cosine distance is not fully justified and not ablated. It would be useful to try other options, for example Euclidean distance. The paper compares the topological score with a spectral change and a pairwise row-distance change, but only in terms of ranking stability across tasks, not of final accuracy. It would be useful to show how the profile and the accuracy change with these alternatives.

Questions for authors
  1. At the family level the method trains the projections that change the most, but inside these families it freezes the layers that change the most. Why these two opposite choices? Only the second is discussed in the main text, as a form of regularization, while the family selection appears only in the pseudocode of Appendix B, even though it decides most of what is trained. Since neither freezing the most-changed nor the least-changed layers is always better, what motivates High-3 beyond the empirical results? And what happens if the families are fixed, for example V and O for every model?
  2. What accuracy do you get when you train the same two families without freezing any layer, and when you freeze 3 random layers per family instead of the ones chosen by the topology?
  3. Why cosine distance? Did you try Euclidean distance, distances on activations, or a score that uses the correspondence between rows before and after fine-tuning? How do the profiles and the accuracy change?
  4. The magnitude-based ranking gives results very close to the topological one. Does the topology choose matrices that a simpler measure of change would not choose?
4 comments

We thank the reviewer for carefully separating the different components of TopoTuner. Your comments mainly question how much of the gain comes from the topological ranking itself versus the broader selective fine-tuning design, and whether our family- and layer-level choices are sufficiently justified. Upon your review, we added no-freezing and random controls, alternative distance metrics, and simpler ranking criteria to directly address these points.

W1 & Q2. No-Freezing and Random-Freezing Controls

Not clear how much of the gain comes from the topology. From Algorithms 1–3, TopoTuner trains only the two projection families that change the most on average (e.g. V and O for LLaMA) and freezes everything else, MLPs included. The topological ranking is then used only to freeze 3 of the 32 layers in each family, so about 10% of the trained matrices. Most of the reduction in trainable parameters, and maybe most of the gain, could come from the choice of the two families and from freezing the MLPs, and not from the topology. The budget ablation (b ∈ {3, 6, 9, 12, 15}) never includes b = 0, so training the two families without freezing any layer is never tested. Recent work also suggests that in pretrained models it matters little which weights are trained: SliceFine (Kowsher et al., ICML 2026) gets almost the same accuracy when the trained slices are the most important, the least important or random ones. I think two baselines are needed: training the same two families with no frozen layers, and freezing 3 random layers per family instead of the ones chosen by the topology.

What accuracy do you get when you train the same two families without freezing any layer, and when you freeze 3 random layers per family instead of the ones chosen by the topology?

We appreciate this concern about separating the contribution of topology from the broader fine-tuning setup.

We agree that the contribution of topology should be separated from the broader selective fine-tuning design. Upon your review, we therefore added the no-freezing (train all) and train x only requested in your question. These baselines directly test whether the gain comes mainly from restricting training to V/O or from the topology-based matrix selection within those families. In this new experiment, we freeze all parameters except the highest-ranked one, two, or three matrices in each V and O projection family. These High1, High2, and High3 configurations train two, four, and six projection matrices in total, corresponding to approximately only 0.26%–0.78% of model parameters. Each selection rule receives the same trainable parameter budget within a model and configuration.

As in the earlier results, our new results in Table x1 below show that topology matters in selection. For Qwen, topological selection exceeds random selection by 0.31 and 0.35 percentage points at High1 and High2 and matches it at High3. For LLaMA, random selection is highly unstable, reaching only 51.95±47.93%, 46.22±44.10%, and 63.61±30.66% at High1, High2, and High3, while TDA remains consistently around 95–96%.

Table x1: SST-2: Qwen and LLaMA. The low values are underlined.

SettingMethodQwen Train.%Qwen Upd.%Qwen Acc.%LLaMA Train.%LLaMA Upd.%LLaMA Acc.%
Train all--9.221.0996.25±0.138.362.2696.06±0.37
Train1Random0.260.0895.60±0.240.260.1151.95±47.93‾\underline{51.95\pm47.93}
Train1TDA0.260.0795.91±0.070.260.1095.53±0.12
Train2Random0.510.1395.64±0.420.520.1846.22±44.10‾\underline{46.22\pm44.10}
Train2TDA0.510.1195.99±0.000.520.1996.25±0.35
Train3Random0.770.1895.87±0.120.780.2563.61±30.66‾\underline{63.61\pm30.66}
Train3TDA0.770.1595.87±0.000.780.2795.79±0.13

W2 & Q3. Distance Metric and Alternative Selection Criteria

The topological score loses information. The H0 diagram only keeps the distances at which rows merge, not WHICH rows merge. Then the Wasserstein distance compares these values without using the fact that row i before fine-tuning is the same row i after fine-tuning, even if this is known. So if rows move from one cluster to another but the merge distances stay similar, the score does not see it. Also, cosine distance does not look at the norm of the rows, so if a matrix is rescaled the score is zero, even if the output of the layer changes. The choice of cosine distance is not fully justified and not ablated. It would be useful to try other options, for example Euclidean distance. The paper compares the topological score with a spectral change and a pairwise row-distance change, but only in terms of ranking stability across tasks, not of final accuracy. It would be useful to show how the profile and the accuracy change with these alternatives.

Why cosine distance? Did you try Euclidean distance, distances on activations, or a score that uses the correspondence between rows before and after fine-tuning? How do the profiles and the accuracy change?

Thank you for the question. Yes, the H0 persistence diagram is intentionally a compact summary of the row geometry, so it does not preserve explicit row-to-row correspondence. Cosine distance measures changes in the relative geometry of the rows and is invariant to uniform rescaling. As you point out, this does not capture every possible change in the weight matrix. Upon your review, we conducted an additional LLaMA and Qwen SST-2 experiment with three training seeds per configuration, comparing topological, element-wise, spectral, and Euclidean-H0 selection. For Euclidean H0, we replace cosine row distance with Euclidean row distance while keeping the same H0-Wasserstein pipeline. In this experiment, we freeze all parameters except the highest-ranked one, two, or three matrices in each V and O projection family. Each selection rule therefore receives the same trainable parameter budget.

For Qwen, topological selection exceeds element-wise selection by 0.27, 0.31, and 0.23 points at High1, High2, and High3. For LLaMA at High2, TDA reaches 96.25±0.35%, compared with 95.95±0.17% for element-wise and 95.76±0.23% for spectral selection. With Euclidean H0, Qwen reaches 95.49±0.07%, 95.76±0.00%, and 95.68±0.07% at High1–High3, while LLaMA reaches 95.07±0.00%, 96.14±0.26%, and 95.83±0.18%. The original cosine-H0 selection is higher than Euclidean H0 at all three Qwen budgets and at High1 and High2 for LLaMA, while Euclidean H0 is slightly higher at LLaMA High3.

Table x2: SST-2 alternative selection criteria.

SettingMethodQwen Train.%Qwen Upd.%Qwen Acc.%LLaMA Train.%LLaMA Upd.%LLaMA Acc.%
Train1TDA0.260.0795.91±0.070.260.1095.53±0.12
Train1Euclidean H00.260.0795.49±0.070.260.1095.07±0.00
Train1Elementwise0.260.0795.64±0.120.260.1095.64±0.23
Train1Spectral0.260.0895.57±0.060.260.1395.11±0.06
Train2TDA0.510.1195.99±0.000.520.1996.25±0.35
Train2Euclidean H00.510.1395.76±0.000.520.1896.14±0.26
Train2Elementwise0.510.1395.68±0.070.520.1895.95±0.17
Train2Spectral0.510.1495.60±0.060.520.2495.76±0.23
Train3TDA0.770.1595.87±0.000.780.2795.79±0.13
Train3Euclidean H00.770.1795.68±0.070.780.2595.83±0.18
Train3Elementwise0.770.1795.64±0.120.780.2695.76±0.12
Train3Spectral0.770.1995.64±0.000.780.3296.03±0.18

These matched-budget results further support our original motivation that topology matters in selection: TDA is the best method across all three Qwen budgets and gives the strongest LLaMA result at High2, while remaining competitive at High1 and High3. The Euclidean-H0 results also show that the choice of row distance affects the selected profile and downstream accuracy, with cosine H0 performing better in five of the six model-budget comparisons. Moreover, as we show in Table 3, the advantage is not only in downstream accuracy: H0-Wasserstein also produces more stable rankings across tasks, with a median Spearman correlation of 0.894, compared with 0.628 for spectral change and 0.723 for row-distance change.

Q1. Projection-Family Selection and High-Drift Freezing

At the family level the method trains the projections that change the most, but inside these families it freezes the layers that change the most. Why these two opposite choices? Only the second is discussed in the main text, as a form of regularization, while the family selection appears only in the pseudocode of Appendix B, even though it decides most of what is trained. Since neither freezing the most-changed nor the least-changed layers is always better, what motivates High-3 beyond the empirical results? And what happens if the families are fixed, for example V and O for every model?

We appreciate this question. The family-level analysis and High3 serve different purposes in our method. The family-level analysis identifies where adaptation is concentrated, while High3 is used as an empirical regularization rule within those families. As we discuss in Section 5 and Appendix D, High3 freezes a small number of the largest departures from the pretrained model and allows the remaining matrices to absorb adaptation. Figure 1 shows this redistribution. We do not claim that High3 is universally optimal. Among the budgets b ∈ {3, 6, 9, 12, 15} studied in the paper, High3 is the smallest high-drift budget that already provides strong transfer performance. We therefore use it as a fixed reusable profile instead of selecting the freezing direction and budget separately for every target task.

Your fixed-family question is also directly addressed by the new experiments above, where V and O are fixed for both Qwen and LLaMA. As shown in Tables x2, TDA remains strong under this fixed-family setting, including the best Qwen results across all three budgets and the strongest LLaMA result at High2. We will make the distinction between the projection-family analysis and High3 regularization clearer in the revised manuscript.


Q4. Does topology select matrices that simpler change measures do not?

The magnitude-based ranking gives results very close to the topological one. Does the topology choose matrices that a simpler measure of change would not choose?

Thank you for asking whether the different criteria actually select different matrices.” Yes. Similar downstream accuracy does not mean that the methods select the same matrices. As we showed in Tables x1 and x2, the matched-budget experiments use the same V/O candidate families and trainable budgets, but the different selection criteria choose substantially different matrices.

For Qwen, TDA selects V: {0, 5, 1} / O: {15, 18, 17}, Euclidean H0 selects V: {1, 29, 25} / O: {33, 35, 16}, element-wise selects V: {26, 3, 30} / O: {25, 34, 31}, and spectral selects V: {35, 1, 29} / O: {35, 0, 33}. For LLaMA, TDA selects V: {0, 2, 8} / O: {13, 10, 11}, Euclidean H0 selects V: {17, 2, 10} / O: {11, 3, 9}, element-wise selects V: {0, 1, 25} / O: {9, 8, 11}, and spectral selects V: {0, 2, 5} / O: {0, 2, 13}.

This is also consistent with Table 3, where H0-Wasserstein produces the most stable cross-task rankings, with median Spearman correlation 0.894, compared with 0.628 for spectral change and 0.723 for row-distance change. Thus, the methods can reach similar accuracy while selecting meaningfully different matrices, and the topological ranking is more stable across tasks.

We would like to emphasize that our main contribution is identifying a reusable, model-specific fine-tuning profile that remains consistent across tasks. We believe our experimental results provide strong evidence for this finding. The additional experiments in this response also helped us better understand how much of the result comes from topology and how the profile changes under different selection choices. We still believe that the individual model results are secondary to the identification of this shared profile.

Your review helped us clarify the role of topology and the design choices behind the reusable profile.

Reviewer_jVvgOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes TopoTuner, a topology-guided framework for selective fine-tuning of LLMs. The authors measure the structural changes of attention projection matrices using Wasserstein distances between persistence diagrams, and use these distances to identify matrices to freeze during fine-tuning. The freezing profiles learned from a source task can be reused on unseen target tasks. The authors also propose a topology-based early stopping criterion. Experiments on LLaMA, Mistral, and Qwen show that TopoTuner achieves competitive performance with fewer parameter updates and shorter training time compared to full fine-tuning and LoRA.

Critical strengths
  • The paper evaluates three different LLM families and compares against several fine-tuning baselines. The reported results are promising in terms of both downstream performance and training efficiency.
  • The idea of learning reusable freezing profiles from one task and applying them to other tasks is interesting. The observed cross-task consistency in matrix rankings provides some useful insights into LLM fine-tuning.
Critical weaknesses
  • The novelty and necessity of the topological analysis are not sufficiently justified. Selective fine-tuning has been studied in prior works such as Spectrum (Hartford et al., 2024) and Surgical Fine-Tuning (Lee et al., 2023). While the authors introduce a different metric for selecting matrices, it is unclear whether the topological metric provides meaningful benefits over simpler alternatives. The paper includes magnitude-based selection experiments, but does not provide a direct comparison of reusable freezing profiles using these simpler metrics on the target tasks. This makes it difficult to attribute the improvements to the proposed topological analysis.
  • The experimental comparisons are not sufficiently controlled. LoRA uses a fixed rank of 16 without a reported rank sweep, while TopoTuner trains around 7.57–8.45% of parameters compared to only 0.17–0.19% for LoRA. Neither trainable parameter count nor compute is matched. Furthermore, the reported Std is computed over three evaluation subsets instead of independent training runs, which does not measure training variance. This is particularly concerning for SST-2, where the improvements are only around 0.1–0.3 points on 872 examples. Finally, the reported end-to-end training time seems to exclude the full fine-tuning required to construct the source freezing profile, which may reverse the claimed efficiency improvement.
Questions for authors
  • Can the authors compare the proposed freezing profiles against random selection and magnitude-based selection under the same trainable parameter budget on the target tasks? This would help clarify whether the topological analysis is really necessary.
  • Can the authors provide results with a better tuned LoRA baseline, including different ranks, and report variance across independent training runs rather than evaluation subsets?
  • Does the reported end-to-end training time include the initial full fine-tuning on GSM8K? Based on Tables 2, 6, and 7, including this cost appears to make TopoTuner slower than LoRA across the evaluated target tasks.
6 comments

We thank the reviewer for the detailed comments. Your review mainly raises two broader questions: whether the topological signal provides a meaningful selection criterion beyond simpler alternatives, and whether the baseline comparisons and efficiency claims are sufficiently controlled, particularly with respect to LoRA rank selection, parameter efficiency, training variability, and source-profile cost. We prepared additional experiments directly targeting these points, including alternative selection criteria, a LoRA rank sweep, and independent training runs.

W1 & Q1. Topological Selection vs. Simpler Alternatives

The novelty and necessity of the topological analysis are not sufficiently justified. Selective fine-tuning has been studied in prior works such as Spectrum (Hartford et al., 2024) and Surgical Fine-Tuning (Lee et al., 2023). While the authors introduce a different metric for selecting matrices, it is unclear whether the topological metric provides meaningful benefits over simpler alternatives. The paper includes magnitude-based selection experiments, but does not provide a direct comparison of reusable freezing profiles using these simpler metrics on the target tasks. This makes it difficult to attribute the improvements to the proposed topological analysis.

Can the authors compare the proposed freezing profiles against random selection and magnitude-based selection under the same trainable parameter budget on the target tasks? This would help clarify whether the topological analysis is really necessary.

We appreciate this feedback and the concern about whether topology provides a meaningful advantage over simpler selection rules.

We agree that the contribution of topology should be compared against simpler selection rules; in fact, we already reported element-wise weight-change selection in Tables 8, 9, and 10. Upon your review, we extended this comparison on SST-2 with three training seeds per configuration, adding spectral and random selection alongside topological and element-wise selection. To make this comparison controlled, we train only the highest-ranked one, two, or three matrices in each V and O projection family and freeze all remaining parameters, so every selection rule receives the same trainable parameter budget within each model and configuration.

These results in Table x1 further support our original motivation for using topology in selection. For Qwen, topological selection exceeds elementwise selection by 0.27, 0.31, and 0.23 percentage points at High1, High2, and High3, respectively. It also exceeds random selection by 0.31 and 0.35 points at High1 and High2 and matches it at High3. For LLaMA at High2, topological selection reaches 96.25 ± 0.35%, compared with 95.95 ± 0.17% for elementwise selection and 95.76 ± 0.23% for spectral selection, with only 0.52% of model parameters trainable.

Table x1: SST-2: Qwen and LLaMA. The low values are underlined.

SettingMethodQwen Train.%Qwen Upd.%Qwen Acc.%LLaMA Train.%LLaMA Upd.%LLaMA Acc.%
Baseline--9.221.0996.25±0.138.362.2696.06±0.37
High1Random0.260.0895.60±0.240.260.1151.95±47.93‾\underline{51.95\pm47.93}
High1TDA0.260.0795.91±0.070.260.1095.53±0.12
High1Elementwise0.260.0795.64±0.120.260.1095.64±0.23
High1Spectral0.260.0895.57±0.060.260.1395.11±0.06
High2Random0.510.1395.64±0.420.520.1846.22±44.10‾\underline{46.22\pm44.10}
High2TDA0.510.1195.99±0.000.520.1996.25±0.35
High2Elementwise0.510.1395.68±0.070.520.1895.95±0.17
High2Spectral0.510.1495.60±0.060.520.2495.76±0.23
High3Random0.770.1895.87±0.120.780.2563.61±30.66‾\underline{63.61\pm30.66}
High3TDA0.770.1595.87±0.000.780.2795.79±0.13
High3Elementwise0.770.1795.64±0.120.780.2695.76±0.12
High3Spectral0.770.1995.64±0.000.780.3296.03±0.18

The LLaMA results also show that random selection can be highly unstable under these restricted budgets, whereas the topology-based profile remains consistently near 96% accuracy across High1–High3.

W2 & Q2. LoRA Rank Selection and Parameter Efficiency

The experimental comparisons are not sufficiently controlled. LoRA uses a fixed rank of 16 without a reported rank sweep, while TopoTuner trains around 7.57–8.45% of parameters compared to only 0.17–0.19% for LoRA. Neither trainable parameter count nor compute is matched.

Can the authors provide results with a better tuned LoRA baseline, including different ranks, and report variance across independent training runs rather than evaluation subsets?

Thank you for this concern regarding the LoRA comparison and rank selection. We agree that LoRA trains far fewer parameters. However, our ‘updated’ and ‘trained’ metrics are correct. As we explain in the paper, Train.% and Upd.% measure different things: Train.% is the fraction of parameters optimized during fine-tuning, while Upd.% is the fraction of pretrained weights whose absolute change after adaptation exceeds 10−610^{-6}. In LoRA, two matrices A and B are learned and then they are used to update the entire pretrained weight matrix (W = W0 + BA ). Thus, LoRA has a much smaller Train.% (i.e., A and B), but its merged updates (i.e., BA) affect a substantially larger fraction of the pretrained weights (W0) than TopoTuner.

Upon your review, we also tested LoRA ranks 8, 10, 16, and 20 with three independent training seeds on SST-2. The sweep shows that rank 16 is not uniformly optimal: the best LoRA result is obtained with r=20r=20 for LLaMA, and with r=16r=16 for Qwen. We will include the full sweep in the revised manuscript.

Table x2: SST-2: LoRA rank sensitivity across three independent training runs.

MethodQwen Train.%Qwen Upd.%Qwen Acc.%Qwen Time (m)LLaMA Train.%LLaMA Upd.%LLaMA Acc.%LLaMA Time (m)
LoRA (r=8)0.0912.8195.99±0.1131.00.0812.8495.91±0.0726.8
LoRA (r=10)0.1213.1496.10±0.2331.30.1113.1595.87±0.1127.0
LoRA (r=16)0.1913.8796.18±0.2431.00.1713.7695.95±0.3526.7
LoRA (r=20)0.2314.1995.99±0.3031.30.2114.0396.18±0.4327.0

As Table x2 shows, increasing the LoRA rank does not consistently improve accuracy, while Upd.% steadily increases with rank.


W3 & Q2. Independent Training Runs and Statistical Reliability

Furthermore, the reported Std is computed over three evaluation subsets instead of independent training runs, which does not measure training variance. This is particularly concerning for SST-2, where the improvements are only around 0.1–0.3 points on 872 examples.

Can the authors provide results with a better tuned LoRA baseline, including different ranks, and report variance across independent training runs rather than evaluation subsets?

We appreciate you pointing out the issue. In our original experiments, the reported standard deviations were calculated across three non-overlapping evaluation subsets. Upon your review, we reran the SST-2 experiments with three independent training seeds (42, 123, and 456) and evaluated every run on the complete 872-example validation set.

The results in Tables x1 and x2 above now report the mean accuracy and standard deviation across independent training runs. In fact, the accuracy levels remain consistent with our original results. TopoTuner originally achieved 95.99% on Qwen and 95.98% on LLaMA, and the independently seeded results remain consistent with these accuracy levels. We will clarify this distinction in the revised manuscript.

W4 & Q3. Source Profile Construction Cost

Finally, the reported end-to-end training time seems to exclude the full fine-tuning required to construct the source freezing profile, which may reverse the claimed efficiency improvement.

Does the reported end-to-end training time include the initial full fine-tuning on GSM8K? Based on Tables 2, 6, and 7, including this cost appears to make TopoTuner slower than LoRA across the evaluated target tasks.

We agree that the source-freezing profile is needed before applying TopoTuner. However, we show that any given task’s profile can be used to learn the profile of an LLM, and this is our most important finding. We expect at least one fine-tuning of a target to be available before TopoTuner is used. Given that most open-weight LLMs have hundreds of such fine-tuned weights on HuggingFace, we believe that our assumption is practical.

In the current experiments, we run GSM8K once to obtain the source checkpoint and build a model-specific profile, then reuse it unchanged for SST-2, IMDB, and MMLU. In fact, Section F.1 already reports the additional offline TDA computation after the source checkpoint is available: 3.23 min for LLaMA, 3.59 min for Qwen, and 3.20 min for Mistral, including both persistence-diagram construction and Wasserstein computation. If GSM8K were fine-tuned only to construct a profile for a single target task, then its training time should also be counted. This is not the setting we study; our goal is to construct the profile once and reuse it across downstream tasks.

Reviewer_bqYmOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper proposes TopoTuner, a fine-tuning method that measures topological changes in attention projection matrices using Wasserstein distances between H0 persistence diagrams before and after full fine-tuning.

Based on a single GSM8K fine-tuning run, the method selects two projection families with the largest average changes and freezes the three most-changed layers within each family. This freezing profile is then reused across tasks.

The authors report outperforming LoRA in 8 of 9 settings across three models and three tasks while reducing training time. They also introduce topology-based early stopping and a theoretical condition for profile transferability (Theorem 5.1).

Critical strengths
  • Attention matrices exhibiting large changes appear consistent across tasks within the same model, with rank correlations around 0.9 for LLaMA and Mistral. Reusing these patterns as freezing profiles is a practical idea.
  • The experiments cover three models and 11 retention benchmarks, and the authors provide code.
Critical weaknesses
  • The GSM8K ablations show that topology-based freezing and simple weight-change-based freezing perform similarly within the reported standard deviations. Freezing the most-changed versus least-changed layers also makes little difference, and the authors acknowledge that neither signal consistently dominates. Crucially, random-freezing and no-freezing baselines are missing. Thus, the improvements may simply result from full-rank fine-tuning of selected attention components rather than the proposed topological analysis.

  • On IMDB, Qwen Full FT (82.83), Qwen LoRA (82.89), and LLaMA LoRA (83.36) perform 6–7 points below their respective pretrained base models (90.0 and 89.8), as reported in the same paper. Three of the claimed eight wins over LoRA come from these comparisons. The remaining three SST-2 comparisons show differences within the reported standard deviations. These results raise serious concerns about baseline reliability and the claimed improvements.

  • Most importantly, the reported standard deviations appear to come from splitting a single evaluation set into three subsets rather than independent training runs. The runtime analysis also refers to nine total runs in Table 2 (three models × three tasks). If each configuration was trained only once, the reported 1–2 point differences cannot establish reliable superiority.

  • H0 persistence diagrams essentially encode the distribution of minimum spanning tree edge lengths among matrix rows. Thus, the measurement is closer to geometric statistics than to distinctive higher-order topological structure. Notably, using H1 performs worse. Theorem 5.1 assumes gradient descent, whereas the experiments use AdamW with coordinate-wise adaptive updates. Consequently, the theorem's error analysis does not directly apply to the experimental setting. Furthermore, the theorem does not determine which layers should be frozen.

Questions for authors
  • How does TopoTuner compare against randomly freezing layers within the same two projection families, or training those families without freezing any layers?
  • Why do several Full FT and LoRA results on IMDB perform substantially worse than the corresponding pretrained base models?
  • Were seeds 42, 123, and 456 in Table 4 used for independent training runs? How many training runs were conducted per configuration?
4 comments

We thank the reviewer for carefully examining our experimental design and statistical reporting. Your comments focused on two important issues: isolating the effect of topology from the broader selective fine-tuning setup, and evaluating variability across independent training runs. We therefore added new selection metrics and report new results from three independent runs across the models on SST-2. We hope these additional experiments address the main concerns and clarify TopoTuner's contribution.

W1 & Q1. Baselines and Simpler Selection Rules

The GSM8K ablations show that topology-based freezing and simple weight-change-based freezing perform similarly within the reported standard deviations. Freezing the most-changed versus least-changed layers also makes little difference, and the authors acknowledge that neither signal consistently dominates. Crucially, random-freezing and no-freezing baselines are missing. Thus, the improvements may simply result from full-rank fine-tuning of selected attention components rather than the proposed topological analysis.

How does TopoTuner compare against randomly freezing layers within the same two projection families, or training those families without freezing any layers?

We agree that the contribution of topology should be compared against simpler selection rules; we had reported a simple norm-based selection (element-wise results) in tables such as 8, 9, and 10. Upon your review, we also conducted an additional Llama and Qwen SST-2 experiment with three training seeds per configuration, comparing topological, element-wise, spectral, and random selection. Spectral drift measures changes in the normalized singular-value energy distribution of a weight matrix between its pretrained and fine-tuned states using the Euclidean distance. In this new experiment, we freeze all parameters except the highest-ranked one, two, or three matrices in each V and O projection family. These High1, High2, and High3 configurations train two, four, and six projection matrices in total, corresponding to approximately only 0.26%–0.78% of model parameters. Each selection rule receives the same trainable parameter budget within a model and configuration.

As in the earlier results, our new results in Table x1 below show that topology matters in selection. For Qwen, topological selection exceeds elementwise selection by 0.27, 0.31, and 0.23 percentage points at High1, High2, and High3, respectively. It exceeds random selection by 0.31 and 0.35 points at High1 and High2 and matches it at High3. For LLaMA at High2, topological selection reaches 96.25 ± 0.35%, compared with 95.95 ± 0.17% for elementwise selection and 95.76 ± 0.23% for spectral selection, with 0.52% of model parameters trainable.

We would also like to point out the LLaMA random results. Random selection is highly unstable under these restricted budgets, reaching only 51.95±47.93, 46.22±44.10, and 63.61±30.66 at High1–High3, while TDA remains consistently around 95–96%. We also include the no-freezing baseline requested in your question.

Table x1: SST-2: Qwen and LLaMA. The low values are underlined.

SettingMethodQwen Train.%Qwen Upd.%Qwen Acc.%LLaMA Train.%LLaMA Upd.%LLaMA Acc.%
Baseline--9.221.0996.25±0.138.362.2696.06±0.37
High1Random0.260.0895.60±0.240.260.1151.95±47.93‾\underline{51.95\pm47.93}
High1TDA0.260.0795.91±0.070.260.1095.53±0.12
High1Elementwise0.260.0795.64±0.120.260.1095.64±0.23
High1Spectral0.260.0895.57±0.060.260.1395.11±0.06
High2Random0.510.1395.64±0.420.520.1846.22±44.10‾\underline{46.22\pm44.10}
High2TDA0.510.1195.99±0.000.520.1996.25±0.35
High2Elementwise0.510.1395.68±0.070.520.1895.95±0.17
High2Spectral0.510.1495.60±0.060.520.2495.76±0.23
High3Random0.770.1895.87±0.120.780.2563.61±30.66‾\underline{63.61\pm30.66}
High3TDA0.770.1595.87±0.000.780.2795.79±0.13
High3Elementwise0.770.1795.64±0.120.780.2695.76±0.12
High3Spectral0.770.1995.64±0.000.780.3296.03±0.18

These matched-budget results further support our original motivation that topology matters in selection: TDA is the best method across all three Qwen budgets and gives the strongest LLaMA result at High2, while remaining competitive at High1 and High3. Moreover, as we show in Table 3, the advantage is not only in downstream accuracy: H0-Wasserstein also produces more stable rankings across tasks, with a median Spearman correlation of 0.894, compared with 0.628 for spectral change and 0.723 for row-distance change.

W2 & Q2. IMDB Baseline Reliability and Pretrained Performance

On IMDB, Qwen Full FT (82.83), Qwen LoRA (82.89), and LLaMA LoRA (83.36) perform 6–7 points below their respective pretrained base models (90.0 and 89.8), as reported in the same paper. Three of the claimed eight wins over LoRA come from these comparisons. The remaining three SST-2 comparisons show differences within the reported standard deviations. These results raise serious concerns about baseline reliability and the claimed improvements.

Why do several Full FT and LoRA results on IMDB perform substantially worse than the corresponding pretrained base models?

We thank the reviewer for raising this concern. Our results are valid, and the discrepancy is due to the evaluation protocols. The pretrained IMDB accuracies of 90.0% for Qwen and 89.8% for LLaMA are reported in the catastrophic forgetting experiments (Appendix Tables 14 and 15), where few-shot examples are included in the evaluation prompts (we had to few-shot, otherwise the base model scored low accuracy for these tasks and we could not study the forgetting behavior). In contrast, the main fine-tuning experiments in Table 2 use zero-shot evaluation without few-shot demonstrations. Consequently, the pretrained accuracies from the forgetting experiments are not directly comparable to the results in Table 2. We will add the relevant description to the forgetting section in the appendix.


W3 Independent Training Runs and Statistical Reliability

Most importantly, the reported standard deviations appear to come from splitting a single evaluation set into three subsets rather than independent training runs. The runtime analysis also refers to nine total runs in Table 2 (three models × three tasks). If each configuration was trained only once, the reported 1–2 point differences cannot establish reliable superiority.

We agree with the concern and evaluated the new results of Table X1 with three seeds. We are happy to report that the accuracy values remain consistent with our original results for Qwen and LLaMA. In Table 2, TopoTuner achieves 95.99% on Qwen and 95.98% on LLaMA, while the new experiments achieve 95.87%–95.99% and 95.53%–96.25%, respectively, across the three selection budgets. These new results support the stability of accuracy levels for Qwen and LLaMA under highly restricted parameter adaptation. We will clarify the evaluation protocols and distinguish the new independently seeded experiments from the original freezing experiments in the revised manuscript.


W4. H0 Topology and Theoretical Scope under AdamW

H0 persistence diagrams essentially encode the distribution of minimum spanning tree edge lengths among matrix rows. Thus, the measurement is closer to geometric statistics than to distinctive higher-order topological structure. Notably, using H1 performs worse.

We agree with your observation. In fact, we explicitly state in Section 4 that finite H0 death times correspond to minimum-spanning-tree edge weights. However, we believe that our H1 experiments also provide useful negative results. We found that H1 features are present in the projection matrices, but they yield slightly lower accuracy at substantially higher computational cost than H0. We also tried H1 in our forgetting and overfitting experiments but found no additional benefit. Perhaps more interestingly, we observed that extracting H1 features from O projections takes substantially more time than from similarly sized V projections because of the much larger number of simplices. We found this surprising given the role of O projections in combining attention-head outputs. We believe that studying these structural differences could be useful, particularly in mechanistic interpretability and model watermarking. More importantly, our results show that topology produces more consistent matrix rankings across tasks. In Table 3, we report a median Spearman correlation of 0.894 for H0-Wasserstein, compared with 0.628 for spectral change and 0.723 for row-distance change. We believe these results provide evidence that H0-Wasserstein captures finetuning changes.

Theorem 5.1 assumes gradient descent, whereas the experiments use AdamW with coordinate-wise adaptive updates. Consequently, the theorem's error analysis does not directly apply to the experimental setting. Furthermore, the theorem does not determine which layers should be frozen.

Yes, we agree, but we also report this directly in the proof on page 16. We say that Theorem 5.1 establishes a conditional result about ranking preservation under gradient descent. Section 5 identifies the choice of which end of the ranking to freeze as empirical, and Appendix A explicitly acknowledges that we use the result to motivate ranking stability, not as a guarantee for the experimental optimizer.

Q3 Independent Training Runs

Were seeds 42, 123, and 456 in Table 4 used for independent training runs? How many training runs were conducted per configuration?

Thank you for raising this question. In our original experiments, the reported standard deviations were calculated across three nonoverlapping evaluation subsets. To address your concern, we ran new SST-2 experiments with three independent training runs using seeds 42, 123, and 456 for each configuration. We evaluated each run on the complete validation set and report the mean accuracy and standard deviation across the three runs in Table x1. We will clarify this distinction in the revised manuscript.

We would like to emphasize that our main contribution is identifying a reusable, model-specific fine-tuning profile that remains consistent across tasks. We believe our experimental results provide strong evidence for this finding. However, we also believe that model-specific results are secondary in importance to the identification of that shared profile. We believe this opens a promising direction for understanding and improving the efficiency of LLM fine-tuning.

Your review raised important concerns about baseline comparisons and statistical reliability, and given the time limit, we have addressed them through additional experiments and clarifications. We are committed to responding to your further questions.

Comments on the paper

1 comment

I thank the authors for the rebuttal and for the new experiments. However, some of my main points remain open, and the new experiments and results raise new concerns. My main concern is the change of setting: in the paper High-B means freezing the B matrices with the largest topological distance in each family and training the others. TopoTuner freezes 3 layers per family and trains the remaining 29, about 7.6% of the parameters for LLaMA. In the new experiments the setting is flipped: “High1/2/3” (renamed “Train1/2/3” in the answer to me) means training only the 1–3 highest-ranked matrices per family and freezing everything else. This change is not explained, and the section is titled “Random-Freezing Controls”, although the random baseline actually trains random matrices. As a result, my requests in W1/Q2 and W2/Q3 are only partially answered:

W1/Q2.

I asked for freezing 3 random layers per family and training the remaining 29, as TopoTuner does with the topological ones. The new results show whether the topological ranking finds good matrices to train, not whether it finds the right matrices to freeze, which is what TopoTuner does in the paper. The random-freezing control in the original setting is still missing. Could the authors explain why the setting was changed?

W2/Q3.

The comparison between cosine-H0, Euclidean-H0, element-wise and spectral selection is also done in the flipped setting, so it does not show how the choice of distance affects TopoTuner itself.

Q1.

The answer restates the role of the two levels but does not explain why the same largest change criterion leads to training at the family level and to freezing at the layer level. High3 is also the smallest freezing budget tested, and the new “Train all” results are at least as good as TopoTuner in the paper. This suggests that freezing fewer matrices is not worse, and that the topological freezing brings no clear benefit.

For these reasons, I keep my score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.