- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper proposes TopoTuner, a way to decide which attention matrices of an LLM should be trained and which should be frozen during fine-tuning. The main idea is to look at the rows of a weight matrix as a point cloud and describe its shape with persistent homology, using H0 diagrams built from cosine distances between rows. The authors compare the diagram of a matrix before and after fine-tuning with a Wasserstein distance and get a score of how much its structure has changed. The authors compute these scores once, by fully fine-tuning the model on GSM8K, and use them to build a freezing profile. In practice only the two projection families that change the most are trained, and in each of them the three layers with the highest score are frozen anyway, as a kind of regularization. This profile is then applied as it is to other tasks. The same score is also used to stop training early, when it stops changing from one epoch to the next. The experiments are on LLaMA-3.1-8B, Mistral-7B and Qwen3-8B, where the method gets close to full fine-tuning and beats LoRA in most settings, while also being faster.
Critical strengths
- Interesting question and practical idea. The paper does not only try to reduce the number of trainable parameters, but asks which parts of a pretrained model really need to change during fine-tuning. I find this a nice point of view. I also find it very interesting to use topology to get a kind of map of the relations between the weights of a matrix. Looking at how the rows are arranged with respect to each other, and how this arrangement changes during fine-tuning, says something different from simply measuring how much each weight moves, and it could be useful also outside this paper. The reusable profile is cheap and simple to use: it is computed once on a source task and then it is just a list of matrix indices. The rankings are also quite stable across tasks (Table 3), which suggests that which matrices change most depends partly on the pretrained model and not only on the task.
2 Broad experiment coverage The profile is built on GSM8K and reused on three different target tasks (SST-2, IMDB and MMLU), so the transfer is tested from math reasoning to both sentiment classification and general knowledge QA. All of this is repeated on three 8B models from different families (LLaMA, Mistral and Qwen), where the ranking selects different projections (V and O for LLaMA and Mistral, Q and K for Qwen). This shows the method is not tied to one architecture. The baselines cover different strategies: low-rank adapters (LoRA), parameter selection (Spectrum), layer dropping (DropBP) and a magnitude-based ranking that keeps the same pipeline but changes only the score. Besides accuracy, the authors measure training time and forgetting on a broad set of retention tasks, which matters a lot for fine-tuning in practice. The early-stopping experiment also shows that the same score can cut 50–67% of the training epochs while keeping almost the same validation accuracy.
Critical weaknesses
-
Not clear how much of the gain comes from the topology. From Algorithms 1–3, TopoTuner trains only the two projection families that change the most on average (e.g. V and O for LLaMA) and freezes everything else, MLPs included. The topological ranking is then used only to freeze 3 of the 32 layers in each family, so about 10% of the trained matrices. Most of the reduction in trainable parameters, and maybe most of the gain, could come from the choice of the two families and from freezing the MLPs, and not from the topology. The budget ablation (b ∈ {3, 6, 9, 12, 15}) never includes b = 0, so training the two families without freezing any layer is never tested. Recent work also suggests that in pretrained models it matters little which weights are trained: SliceFine (Kowsher et al., ICML 2026) gets almost the same accuracy when the trained slices are the most important, the least important or random ones. I think two baselines are needed: training the same two families with no frozen layers, and freezing 3 random layers per family instead of the ones chosen by the topology.
-
The topological score loses information. The H0 diagram only keeps the distances at which rows merge, not WHICH rows merge. Then the Wasserstein distance compares these values without using the fact that row before fine-tuning is the same row after fine-tuning, even if this is known. So if rows move from one cluster to another but the merge distances stay similar, the score does not see it. Also, cosine distance does not look at the norm of the rows, so if a matrix is rescaled the score is zero, even if the output of the layer changes. The choice of cosine distance is not fully justified and not ablated. It would be useful to try other options, for example Euclidean distance. The paper compares the topological score with a spectral change and a pairwise row-distance change, but only in terms of ranking stability across tasks, not of final accuracy. It would be useful to show how the profile and the accuracy change with these alternatives.
Questions for authors
- At the family level the method trains the projections that change the most, but inside these families it freezes the layers that change the most. Why these two opposite choices? Only the second is discussed in the main text, as a form of regularization, while the family selection appears only in the pseudocode of Appendix B, even though it decides most of what is trained. Since neither freezing the most-changed nor the least-changed layers is always better, what motivates High-3 beyond the empirical results? And what happens if the families are fixed, for example V and O for every model?
- What accuracy do you get when you train the same two families without freezing any layer, and when you freeze 3 random layers per family instead of the ones chosen by the topology?
- Why cosine distance? Did you try Euclidean distance, distances on activations, or a score that uses the correspondence between rows before and after fine-tuning? How do the profiles and the accuracy change?
- The magnitude-based ranking gives results very close to the topological one. Does the topology choose matrices that a simpler measure of change would not choose?
4 comments
W2 & Q3. Distance Metric and Alternative Selection Criteria
The topological score loses information. The H0 diagram only keeps the distances at which rows merge, not WHICH rows merge. Then the Wasserstein distance compares these values without using the fact that row i before fine-tuning is the same row i after fine-tuning, even if this is known. So if rows move from one cluster to another but the merge distances stay similar, the score does not see it. Also, cosine distance does not look at the norm of the rows, so if a matrix is rescaled the score is zero, even if the output of the layer changes. The choice of cosine distance is not fully justified and not ablated. It would be useful to try other options, for example Euclidean distance. The paper compares the topological score with a spectral change and a pairwise row-distance change, but only in terms of ranking stability across tasks, not of final accuracy. It would be useful to show how the profile and the accuracy change with these alternatives.
Why cosine distance? Did you try Euclidean distance, distances on activations, or a score that uses the correspondence between rows before and after fine-tuning? How do the profiles and the accuracy change?
Thank you for the question. Yes, the H0 persistence diagram is intentionally a compact summary of the row geometry, so it does not preserve explicit row-to-row correspondence. Cosine distance measures changes in the relative geometry of the rows and is invariant to uniform rescaling. As you point out, this does not capture every possible change in the weight matrix. Upon your review, we conducted an additional LLaMA and Qwen SST-2 experiment with three training seeds per configuration, comparing topological, element-wise, spectral, and Euclidean-H0 selection. For Euclidean H0, we replace cosine row distance with Euclidean row distance while keeping the same H0-Wasserstein pipeline. In this experiment, we freeze all parameters except the highest-ranked one, two, or three matrices in each V and O projection family. Each selection rule therefore receives the same trainable parameter budget.
For Qwen, topological selection exceeds element-wise selection by 0.27, 0.31, and 0.23 points at High1, High2, and High3. For LLaMA at High2, TDA reaches 96.25±0.35%, compared with 95.95±0.17% for element-wise and 95.76±0.23% for spectral selection. With Euclidean H0, Qwen reaches 95.49±0.07%, 95.76±0.00%, and 95.68±0.07% at High1–High3, while LLaMA reaches 95.07±0.00%, 96.14±0.26%, and 95.83±0.18%. The original cosine-H0 selection is higher than Euclidean H0 at all three Qwen budgets and at High1 and High2 for LLaMA, while Euclidean H0 is slightly higher at LLaMA High3.
Table x2: SST-2 alternative selection criteria.
| Setting | Method | Qwen Train.% | Qwen Upd.% | Qwen Acc.% | LLaMA Train.% | LLaMA Upd.% | LLaMA Acc.% |
|---|---|---|---|---|---|---|---|
| Train1 | TDA | 0.26 | 0.07 | 95.91±0.07 | 0.26 | 0.10 | 95.53±0.12 |
| Train1 | Euclidean H0 | 0.26 | 0.07 | 95.49±0.07 | 0.26 | 0.10 | 95.07±0.00 |
| Train1 | Elementwise | 0.26 | 0.07 | 95.64±0.12 | 0.26 | 0.10 | 95.64±0.23 |
| Train1 | Spectral | 0.26 | 0.08 | 95.57±0.06 | 0.26 | 0.13 | 95.11±0.06 |
| Train2 | TDA | 0.51 | 0.11 | 95.99±0.00 | 0.52 | 0.19 | 96.25±0.35 |
| Train2 | Euclidean H0 | 0.51 | 0.13 | 95.76±0.00 | 0.52 | 0.18 | 96.14±0.26 |
| Train2 | Elementwise | 0.51 | 0.13 | 95.68±0.07 | 0.52 | 0.18 | 95.95±0.17 |
| Train2 | Spectral | 0.51 | 0.14 | 95.60±0.06 | 0.52 | 0.24 | 95.76±0.23 |
| Train3 | TDA | 0.77 | 0.15 | 95.87±0.00 | 0.78 | 0.27 | 95.79±0.13 |
| Train3 | Euclidean H0 | 0.77 | 0.17 | 95.68±0.07 | 0.78 | 0.25 | 95.83±0.18 |
| Train3 | Elementwise | 0.77 | 0.17 | 95.64±0.12 | 0.78 | 0.26 | 95.76±0.12 |
| Train3 | Spectral | 0.77 | 0.19 | 95.64±0.00 | 0.78 | 0.32 | 96.03±0.18 |
These matched-budget results further support our original motivation that topology matters in selection: TDA is the best method across all three Qwen budgets and gives the strongest LLaMA result at High2, while remaining competitive at High1 and High3. The Euclidean-H0 results also show that the choice of row distance affects the selected profile and downstream accuracy, with cosine H0 performing better in five of the six model-budget comparisons. Moreover, as we show in Table 3, the advantage is not only in downstream accuracy: H0-Wasserstein also produces more stable rankings across tasks, with a median Spearman correlation of 0.894, compared with 0.628 for spectral change and 0.723 for row-distance change.
Q1. Projection-Family Selection and High-Drift Freezing
At the family level the method trains the projections that change the most, but inside these families it freezes the layers that change the most. Why these two opposite choices? Only the second is discussed in the main text, as a form of regularization, while the family selection appears only in the pseudocode of Appendix B, even though it decides most of what is trained. Since neither freezing the most-changed nor the least-changed layers is always better, what motivates High-3 beyond the empirical results? And what happens if the families are fixed, for example V and O for every model?
We appreciate this question. The family-level analysis and High3 serve different purposes in our method. The family-level analysis identifies where adaptation is concentrated, while High3 is used as an empirical regularization rule within those families. As we discuss in Section 5 and Appendix D, High3 freezes a small number of the largest departures from the pretrained model and allows the remaining matrices to absorb adaptation. Figure 1 shows this redistribution. We do not claim that High3 is universally optimal. Among the budgets b ∈ {3, 6, 9, 12, 15} studied in the paper, High3 is the smallest high-drift budget that already provides strong transfer performance. We therefore use it as a fixed reusable profile instead of selecting the freezing direction and budget separately for every target task.
Your fixed-family question is also directly addressed by the new experiments above, where V and O are fixed for both Qwen and LLaMA. As shown in Tables x2, TDA remains strong under this fixed-family setting, including the best Qwen results across all three budgets and the strongest LLaMA result at High2. We will make the distinction between the projection-family analysis and High3 regularization clearer in the revised manuscript.
Q4. Does topology select matrices that simpler change measures do not?
The magnitude-based ranking gives results very close to the topological one. Does the topology choose matrices that a simpler measure of change would not choose?
Thank you for asking whether the different criteria actually select different matrices.” Yes. Similar downstream accuracy does not mean that the methods select the same matrices. As we showed in Tables x1 and x2, the matched-budget experiments use the same V/O candidate families and trainable budgets, but the different selection criteria choose substantially different matrices.
For Qwen, TDA selects V: {0, 5, 1} / O: {15, 18, 17}, Euclidean H0 selects V: {1, 29, 25} / O: {33, 35, 16}, element-wise selects V: {26, 3, 30} / O: {25, 34, 31}, and spectral selects V: {35, 1, 29} / O: {35, 0, 33}. For LLaMA, TDA selects V: {0, 2, 8} / O: {13, 10, 11}, Euclidean H0 selects V: {17, 2, 10} / O: {11, 3, 9}, element-wise selects V: {0, 1, 25} / O: {9, 8, 11}, and spectral selects V: {0, 2, 5} / O: {0, 2, 13}.
This is also consistent with Table 3, where H0-Wasserstein produces the most stable cross-task rankings, with median Spearman correlation 0.894, compared with 0.628 for spectral change and 0.723 for row-distance change. Thus, the methods can reach similar accuracy while selecting meaningfully different matrices, and the topological ranking is more stable across tasks.
We would like to emphasize that our main contribution is identifying a reusable, model-specific fine-tuning profile that remains consistent across tasks. We believe our experimental results provide strong evidence for this finding. The additional experiments in this response also helped us better understand how much of the result comes from topology and how the profile changes under different selection choices. We still believe that the individual model results are secondary to the identification of this shared profile.
Your review helped us clarify the role of topology and the design choices behind the reusable profile.
We thank the reviewer for carefully separating the different components of TopoTuner. Your comments mainly question how much of the gain comes from the topological ranking itself versus the broader selective fine-tuning design, and whether our family- and layer-level choices are sufficiently justified. Upon your review, we added no-freezing and random controls, alternative distance metrics, and simpler ranking criteria to directly address these points.
W1 & Q2. No-Freezing and Random-Freezing Controls
We appreciate this concern about separating the contribution of topology from the broader fine-tuning setup.
We agree that the contribution of topology should be separated from the broader selective fine-tuning design. Upon your review, we therefore added the no-freezing (train all) and train x only requested in your question. These baselines directly test whether the gain comes mainly from restricting training to V/O or from the topology-based matrix selection within those families. In this new experiment, we freeze all parameters except the highest-ranked one, two, or three matrices in each V and O projection family. These High1, High2, and High3 configurations train two, four, and six projection matrices in total, corresponding to approximately only 0.26%–0.78% of model parameters. Each selection rule receives the same trainable parameter budget within a model and configuration.
As in the earlier results, our new results in Table x1 below show that topology matters in selection. For Qwen, topological selection exceeds random selection by 0.31 and 0.35 percentage points at High1 and High2 and matches it at High3. For LLaMA, random selection is highly unstable, reaching only 51.95±47.93%, 46.22±44.10%, and 63.61±30.66% at High1, High2, and High3, while TDA remains consistently around 95–96%.
Table x1: SST-2: Qwen and LLaMA. The low values are underlined.