- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper investigates training-free knowledge transfer between models of different sizes. The authors argue that directly truncating the frequency-domain representation of pretrained weights may discard useful information because the ordering of neurons can disperse that information across frequency components. To address this issue, they propose Spatial Permutation Alignment (SPA), which exploits permutation symmetries in neural networks to reorder hidden neurons and channels without changing the function of the source model. This reordering promotes spatial smoothness and spectral concentration in the weights. The reordered weights are then passed to FRONT for low-frequency DCT extraction, yielding SPA-FRONT. The experiments cover depth and width scaling of vision models, downstream classification tasks, and CNN architectures, as well as language experiments with BERT, RoBERTa, and GPT-2. Overall, SPA-FRONT consistently improves upon FRONT and, without auxiliary training, approaches or exceeds the performance of methods such as WAVE that require additional training.
Critical strengths
- Clear motivation and a novel combination of ideas. Before frequency-domain extraction, the method exploits functional equivalence in parameter space to improve the arrangement of weights, concentrating more energy in the retained low-frequency components. SPA thereby allows more weight information to be preserved in the low-frequency representation. This is a natural approach and provides an intuitive explanation for why SPA can retain more structural information under the same low-frequency coefficient budget.
- A practical, training-free approach. SPA rearranges parameters through procedures such as Hungarian matching and Anchored Joint Permutation, without gradient-based training, additional data, or auxiliary models. Compared with methods such as WAVE and TLEG that require auxiliary optimization, SPA-FRONT offers a clear advantage in transfer cost.
- Strong experimental results and convincing comparisons with competitive baselines. SPA-FRONT outperforms FRONT across the evaluated depth and width scaling settings for DeiT, and achieves consistent gains across seven downstream classification datasets. Compared with WAVE, a representative method in this area, SPA-FRONT approaches or exceeds its performance in multiple depth and width settings without auxiliary training, demonstrating a favorable trade-off between performance and efficiency.
Critical weaknesses
- The empirical evidence primarily supports SPA-FRONT. The appendix provides valuable analyses of potential extensions involving wavelets, TLEG, and WAVE, but these extensions remain conceptual. Thus, while the current experiments provide strong evidence that SPA improves FRONT, its effectiveness as a more general preprocessing method for knowledge extraction remains to be validated.
- The robustness of the AJP search strategy could be further characterized. AJP uses greedy initialization and 2-opt local search to optimize the ordering of weights. Table 7 supports the effectiveness of this design, but its sensitivity to the choice of starting anchor and search budget remains unclear. Moreover, path smoothness, low-frequency concentration, and downstream performance are not equivalent objectives. Additional analysis or experiments would help clarify the conditions under which this heuristic consistently provides benefits.
- Downstream evaluation in the language experiments remains relatively narrow. The paper demonstrates pretraining efficiency gains for BERT, RoBERTa, and GPT-2, but downstream evaluation is largely limited to GLUE results for BERT-S, and all scaling settings involve transfer from Base to Small models. These results provide initial evidence of effectiveness across modalities. Additional GLUE results for RoBERTa and GPT-2, or experiments with other scaling configurations, would further strengthen the evidence.
- Statistical robustness could be better documented. Reporting means, standard deviations, or confidence intervals across multiple random seeds for key settings would help establish the stability and effectiveness of SPA-FRONT.
Questions for authors
- Could the authors provide a more complete comparison of the overall efficiency and final converged performance of SPA-FRONT? It would be helpful to compare the end-to-end time required to reach a common performance target against other methods, including WAVE and TLEG, accounting for each method's initialization and auxiliary training costs. Please also clarify how the common target accuracy in Figure 3 is selected. In addition, Figure 3 appears to indicate differences in converged performance across methods. Could the authors more systematically assess whether SPA-FRONT also offers an advantage in final converged performance?
- How sensitive is the method to its hyperparameters and design choices? For example, do the low-frequency retention ratio, number of alignment rounds, anchor selection, 2-opt search scope, and permutation constraints on different dimensions substantially affect the results?
- Which components contribute most to the gains from SPA? It would be useful to disentangle the contributions of depth-wise alignment, width-wise alignment, and their alternating application, report results using each component alone, and examine whether the order of applying the two components affects performance.
- Could the authors add further language experiments? For example, evaluating FRONT and SPA-FRONT on GLUE tasks using RoBERTa-S or GPT-2 would help assess whether the downstream gains also hold for another language model. Alternatively, language experiments with additional scaling configurations would be informative.
11 comments
Response to Reviewer F66c (2/8)
W2.a) The robustness of the AJP search strategy could be further characterized. AJP uses greedy initialization and 2-opt local search to optimize the ordering of weights. Table 7 supports the effectiveness of this design, but its sensitivity to the choice of starting anchor and search budget remains unclear.
R2(a): Thank you for this valuable suggestion. We have added a systematic ablation study to examine AJP's sensitivity to the starting anchor, 2-opt search budget, and reversal scope. The complete experimental details and analyses are provided in Appendix E.9. All experiments use the same pretrained DeiT-S-L6 as the source model for permutation, followed by FRONT initialization of DeiT-Ti-L6 and 10 epochs of ImageNet-1K training. The results are summarized below.
| Ablation | Configuration | Top-1 Acc. (%) |
|---|---|---|
| Baseline | Scratch (He-Init) | 40.60 |
| Starting anchor | Max. norm (default) | 58.75 |
| Min. norm | 57.49 | |
| Median norm | 58.07 | |
| First index | 57.47 | |
| Nearest mean | 57.59 | |
| Random | 56.77 | |
| 2-opt budget | 0 (greedy only) | 56.83 |
| 1 pass | 58.31 | |
| 3 passes | 57.95 | |
| 10 passes | 58.26 | |
| 1,000 passes (default) | 58.75 | |
| Reversal scope | 16 | 57.36 |
| 64 | 57.18 | |
| 256 | 58.27 | |
| Full (default) | 58.75 |
First, starting-anchor selection affects the resulting ordering, but AJP does not depend critically on a specific anchor. The maximum-norm anchor achieves the highest accuracy (58.75%), outperforming the other strategies by 0.68–1.98 percentage points. Nevertheless, all anchor choices maintain substantial transfer benefits, indicating that AJP is relatively robust to its initialization.
Second, 2-opt refinement provides a clear improvement over greedy initialization, while most of its benefit is obtained with a small search budget. Greedy initialization alone achieves 56.83%, whereas a single 2-opt pass improves accuracy to 58.31% (+1.48 points), already approaching the default configuration. Additional passes yield smaller, non-monotonic accuracy changes, suggesting that further reductions in path length do not necessarily translate into proportional improvements in knowledge transfer.
Third, a broader reversal scope provides additional refinement opportunities, but restricting the search range has a limited impact on transfer performance. The full scope achieves the highest accuracy, while limiting the maximum reversal length to 256 still achieves 58.27%, only 0.48 points lower. Even smaller scopes retain substantial transfer benefits.
Overall, these results clarify the roles of AJP's components: the starting anchor establishes the initial ordering, greedy search constructs a viable permutation, and 2-opt refinement further improves transfer performance. Importantly, the effectiveness of AJP is largely preserved across different anchor choices and reduced search budgets, showing that its benefits do not rely on exhaustive search or limited configurations.
Response to Reviewer F66c (3/8)
W2.b) Moreover, path smoothness, low-frequency concentration, and downstream performance are not equivalent objectives. Additional analysis or experiments would help clarify the conditions under which this heuristic consistently provides benefits.
R2(b): Thank you for raising this important comment. We have added further analyses in Appendices E.5 to clarify the relationship among these objectives and the effectiveness of the AJP heuristic.
First, path smoothness serves as a theoretically motivated surrogate for spectral preconditioning, rather than a direct objective for downstream performance. AJP reduces local variation by minimizing the distances between adjacent joint descriptors of coupled weights. As established in Appendix C.1, lower structural variation favors low-frequency energy concentration in the DCT domain. Under a fixed coefficient budget, higher LF concentration further reduces weight reconstruction error (Appendix C.2). However, these relationships do not guarantee better knowledge transfer, since neural network weights encode computational functionality beyond their spectral structure. Different parameter organizations may retain different task-relevant information even when their spectral concentration is similar. This explains why improving path smoothness or LF concentration does not necessarily yield proportional improvements in downstream accuracy.
Second, our additional experiments provide empirical evidence for the effectiveness and limitations of this heuristic. In Appendix E.5, we use the same AJP-based permutation framework to generate six functionally equivalent parameterizations of pretrained DeiT-S-L6, with different LF concentrations. We then initialize DeiT-Ti-L6 using FRONT, keeping the extraction procedure, target architecture, and training settings identical.
| LF Level | LR(%) | Descendant Top-1 Accuracy (%) |
|---|---|---|
| Scratch | — | 40.62 |
| Level 1 | 10.22 | 53.58 |
| Level 2 | 20.48 | 54.71 |
| Level 3 | 30.73 | 57.05 |
| Level 4 | 40.98 | 57.34 |
| Level 5 | 51.22 | 57.42 |
| Level 6 | 61.46 | 58.75 |
Under this consistent extraction setting, increasing LF concentration from 10.22% to 61.46% is accompanied by an improvement in descendant accuracy from 53.58% to 58.75%. However, the gains are not proportional: increasing the LF ratio from 30.73% to 51.22% improves accuracy by only 0.37 percentage points. Overall, these findings support path smoothness as a practical surrogate for improving spectral extraction, particularly under a fixed low-frequency coefficient budget. AJP therefore provides an effective training-free heuristic for identifying favorable parameterizations, rather than a guarantee of optimal downstream performance.
Response to Reviewer F66c (1/8)
Dear Reviewer F66c:
Thank you for your careful review and helpful comments. We have addressed the points raised below and incorporated the relevant clarifications and experiments into the revised manuscript. We hope that our responses and revisions have satisfactorily addressed your questions and concerns.
R1: Thank you for this valuable comment. We have added three proof-of-concept experiments on DWT, TLEG, and WAVE to Appendix F.4. Specifically, we evaluate SPA-DWT, SPA-TLEG, and SPA-WAVE on BERT-S-L6, using the same 150,000-step target pretraining protocol as our main language experiments. The results are summarized below.
Table. FLOPs reductions are measured relative to scratch training and exclude auxiliary training costs.
These results demonstrate two complementary benefits. First, SPA improves an existing training-free extractor: SPA-DWT increases the FLOPs reduction from 35.5% to 45.7%, showing that function-preserving reparameterization also benefits wavelet-based extraction. Second, SPA offers training-free alternatives to auxiliary-training methods: SPA-TLEG and SPA-WAVE achieve 47.8% and 18.1% FLOPs reductions, respectively, without the 10,000 auxiliary training steps that their learned counterparts require. Although these alternatives do not fully match the convergence gains of the original TLEG and WAVE, they demonstrate the feasibility of effective knowledge transfer without expensive auxiliary optimization.
These results provide empirical support for our insight: functionally equivalent parameterizations can yield different knowledge extraction and transfer outcomes. SPA therefore offers a broader principle for improving existing extractors or developing computationally efficient alternatives.