acceptodds
Under review as a conference paper at ICLR 2027

Reparameterize Before You Extract: Training-Free Spectral Preconditioning for Knowledge Transfer

Abstract

Large-scale pre-trained models contain transferable knowledge, but their parameters are tied to the architecture and size of the source model, limiting direct reuse across model sizes. Existing learngene-based approaches seek compact, task-agnostic representations for variable-sized model initialization, but many require auxiliary training. FRONT shows that low-frequency components of pre-trained weights are informative for foundational and transferable knowledge. However, standard pre-training does not explicitly organize weights for subsequent knowledge extraction, and the resulting spectra can exhibit limited low-frequency concentration. Under a fixed coefficient budget, this constrains the amount of spectral energy that can be retained. We introduce Spatial Permutation Alignment (SPA), a training-free structural preconditioner for frequency-domain knowledge extraction. Our key observation is that functionally equivalent parameterizations within the same permutation-equivalence class can exhibit different spectral energy distributions. SPA exploits this reparameterization freedom through depth-wise bipartite matching and width-wise anchored joint permutation, reducing inter- and intra-layer structural variation. Integrating SPA with FRONT yields SPA-FRONT, which captures a larger fraction of spectral energy in the retained low-frequency coefficients under the same coefficient budget. Across vision and language models, SPA-FRONT substantially improves variable-sized model initialization, reducing vision-model training epochs to reach target accuracy by 30.0% on average and cutting language-model pre-training FLOPs by up to 58.7%. The code is available at https://anonymous.4open.science/r/SPA-FRONT.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
Model Initialization, Variable-Sized Models, Permutation Invariance

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_F66cOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates training-free knowledge transfer between models of different sizes. The authors argue that directly truncating the frequency-domain representation of pretrained weights may discard useful information because the ordering of neurons can disperse that information across frequency components. To address this issue, they propose Spatial Permutation Alignment (SPA), which exploits permutation symmetries in neural networks to reorder hidden neurons and channels without changing the function of the source model. This reordering promotes spatial smoothness and spectral concentration in the weights. The reordered weights are then passed to FRONT for low-frequency DCT extraction, yielding SPA-FRONT. The experiments cover depth and width scaling of vision models, downstream classification tasks, and CNN architectures, as well as language experiments with BERT, RoBERTa, and GPT-2. Overall, SPA-FRONT consistently improves upon FRONT and, without auxiliary training, approaches or exceeds the performance of methods such as WAVE that require additional training.

Critical strengths
  1. Clear motivation and a novel combination of ideas. Before frequency-domain extraction, the method exploits functional equivalence in parameter space to improve the arrangement of weights, concentrating more energy in the retained low-frequency components. SPA thereby allows more weight information to be preserved in the low-frequency representation. This is a natural approach and provides an intuitive explanation for why SPA can retain more structural information under the same low-frequency coefficient budget.
  2. A practical, training-free approach. SPA rearranges parameters through procedures such as Hungarian matching and Anchored Joint Permutation, without gradient-based training, additional data, or auxiliary models. Compared with methods such as WAVE and TLEG that require auxiliary optimization, SPA-FRONT offers a clear advantage in transfer cost.
  3. Strong experimental results and convincing comparisons with competitive baselines. SPA-FRONT outperforms FRONT across the evaluated depth and width scaling settings for DeiT, and achieves consistent gains across seven downstream classification datasets. Compared with WAVE, a representative method in this area, SPA-FRONT approaches or exceeds its performance in multiple depth and width settings without auxiliary training, demonstrating a favorable trade-off between performance and efficiency.
Critical weaknesses
  1. The empirical evidence primarily supports SPA-FRONT. The appendix provides valuable analyses of potential extensions involving wavelets, TLEG, and WAVE, but these extensions remain conceptual. Thus, while the current experiments provide strong evidence that SPA improves FRONT, its effectiveness as a more general preprocessing method for knowledge extraction remains to be validated.
  2. The robustness of the AJP search strategy could be further characterized. AJP uses greedy initialization and 2-opt local search to optimize the ordering of weights. Table 7 supports the effectiveness of this design, but its sensitivity to the choice of starting anchor and search budget remains unclear. Moreover, path smoothness, low-frequency concentration, and downstream performance are not equivalent objectives. Additional analysis or experiments would help clarify the conditions under which this heuristic consistently provides benefits.
  3. Downstream evaluation in the language experiments remains relatively narrow. The paper demonstrates pretraining efficiency gains for BERT, RoBERTa, and GPT-2, but downstream evaluation is largely limited to GLUE results for BERT-S, and all scaling settings involve transfer from Base to Small models. These results provide initial evidence of effectiveness across modalities. Additional GLUE results for RoBERTa and GPT-2, or experiments with other scaling configurations, would further strengthen the evidence.
  4. Statistical robustness could be better documented. Reporting means, standard deviations, or confidence intervals across multiple random seeds for key settings would help establish the stability and effectiveness of SPA-FRONT.
Questions for authors
  1. Could the authors provide a more complete comparison of the overall efficiency and final converged performance of SPA-FRONT? It would be helpful to compare the end-to-end time required to reach a common performance target against other methods, including WAVE and TLEG, accounting for each method's initialization and auxiliary training costs. Please also clarify how the common target accuracy in Figure 3 is selected. In addition, Figure 3 appears to indicate differences in converged performance across methods. Could the authors more systematically assess whether SPA-FRONT also offers an advantage in final converged performance?
  2. How sensitive is the method to its hyperparameters and design choices? For example, do the low-frequency retention ratio, number of alignment rounds, anchor selection, 2-opt search scope, and permutation constraints on different dimensions substantially affect the results?
  3. Which components contribute most to the gains from SPA? It would be useful to disentangle the contributions of depth-wise alignment, width-wise alignment, and their alternating application, report results using each component alone, and examine whether the order of applying the two components affects performance.
  4. Could the authors add further language experiments? For example, evaluating FRONT and SPA-FRONT on GLUE tasks using RoBERTa-S or GPT-2 would help assess whether the downstream gains also hold for another language model. Alternatively, language experiments with additional scaling configurations would be informative.
11 comments
AuthorsOpenReview ↗

Response to Reviewer F66c (1/8)

Dear Reviewer F66c:

Thank you for your careful review and helpful comments. We have addressed the points raised below and incorporated the relevant clarifications and experiments into the revised manuscript. We hope that our responses and revisions have satisfactorily addressed your questions and concerns.

W1. The empirical evidence primarily supports SPA-FRONT. The appendix provides valuable analyses of potential extensions involving wavelets, TLEG, and WAVE, but these extensions remain conceptual. Thus, while the current experiments provide strong evidence that SPA improves FRONT, its effectiveness as a more general preprocessing method for knowledge extraction remains to be validated.

R1: Thank you for this valuable comment. We have added three proof-of-concept experiments on DWT, TLEG, and WAVE to Appendix F.4. Specifically, we evaluate SPA-DWT, SPA-TLEG, and SPA-WAVE on BERT-S-L6, using the same 150,000-step target pretraining protocol as our main language experiments. The results are summarized below.

Table. FLOPs reductions are measured relative to scratch training and exclude auxiliary training costs.

InitializationAuxiliary Training StepsFLOPs Reduction (%)
Scratch00.0
DWT035.5
SPA-DWT045.7
TLEG10,00052.9
SPA-TLEG047.8
WAVE10,00036.2
SPA-WAVE018.1

These results demonstrate two complementary benefits. First, SPA improves an existing training-free extractor: SPA-DWT increases the FLOPs reduction from 35.5% to 45.7%, showing that function-preserving reparameterization also benefits wavelet-based extraction. Second, SPA offers training-free alternatives to auxiliary-training methods: SPA-TLEG and SPA-WAVE achieve 47.8% and 18.1% FLOPs reductions, respectively, without the 10,000 auxiliary training steps that their learned counterparts require. Although these alternatives do not fully match the convergence gains of the original TLEG and WAVE, they demonstrate the feasibility of effective knowledge transfer without expensive auxiliary optimization.

These results provide empirical support for our insight: functionally equivalent parameterizations can yield different knowledge extraction and transfer outcomes. SPA therefore offers a broader principle for improving existing extractors or developing computationally efficient alternatives.

AuthorsOpenReview ↗

Response to Reviewer F66c (2/8)

W2.a) The robustness of the AJP search strategy could be further characterized. AJP uses greedy initialization and 2-opt local search to optimize the ordering of weights. Table 7 supports the effectiveness of this design, but its sensitivity to the choice of starting anchor and search budget remains unclear.

R2(a): Thank you for this valuable suggestion. We have added a systematic ablation study to examine AJP's sensitivity to the starting anchor, 2-opt search budget, and reversal scope. The complete experimental details and analyses are provided in Appendix E.9. All experiments use the same pretrained DeiT-S-L6 as the source model for permutation, followed by FRONT initialization of DeiT-Ti-L6 and 10 epochs of ImageNet-1K training. The results are summarized below.

AblationConfigurationTop-1 Acc. (%)
BaselineScratch (He-Init)40.60
Starting anchorMax. norm (default)58.75
Min. norm57.49
Median norm58.07
First index57.47
Nearest mean57.59
Random56.77
2-opt budget0 (greedy only)56.83
1 pass58.31
3 passes57.95
10 passes58.26
1,000 passes (default)58.75
Reversal scope1657.36
6457.18
25658.27
Full (default)58.75

First, starting-anchor selection affects the resulting ordering, but AJP does not depend critically on a specific anchor. The maximum-norm anchor achieves the highest accuracy (58.75%), outperforming the other strategies by 0.68–1.98 percentage points. Nevertheless, all anchor choices maintain substantial transfer benefits, indicating that AJP is relatively robust to its initialization.

Second, 2-opt refinement provides a clear improvement over greedy initialization, while most of its benefit is obtained with a small search budget. Greedy initialization alone achieves 56.83%, whereas a single 2-opt pass improves accuracy to 58.31% (+1.48 points), already approaching the default configuration. Additional passes yield smaller, non-monotonic accuracy changes, suggesting that further reductions in path length do not necessarily translate into proportional improvements in knowledge transfer.

Third, a broader reversal scope provides additional refinement opportunities, but restricting the search range has a limited impact on transfer performance. The full scope achieves the highest accuracy, while limiting the maximum reversal length to 256 still achieves 58.27%, only 0.48 points lower. Even smaller scopes retain substantial transfer benefits.

Overall, these results clarify the roles of AJP's components: the starting anchor establishes the initial ordering, greedy search constructs a viable permutation, and 2-opt refinement further improves transfer performance. Importantly, the effectiveness of AJP is largely preserved across different anchor choices and reduced search budgets, showing that its benefits do not rely on exhaustive search or limited configurations.

AuthorsOpenReview ↗

Response to Reviewer F66c (3/8)

W2.b) Moreover, path smoothness, low-frequency concentration, and downstream performance are not equivalent objectives. Additional analysis or experiments would help clarify the conditions under which this heuristic consistently provides benefits.

R2(b): Thank you for raising this important comment. We have added further analyses in Appendices E.5 to clarify the relationship among these objectives and the effectiveness of the AJP heuristic.

First, path smoothness serves as a theoretically motivated surrogate for spectral preconditioning, rather than a direct objective for downstream performance. AJP reduces local variation by minimizing the distances between adjacent joint descriptors of coupled weights. As established in Appendix C.1, lower structural variation favors low-frequency energy concentration in the DCT domain. Under a fixed coefficient budget, higher LF concentration further reduces weight reconstruction error (Appendix C.2). However, these relationships do not guarantee better knowledge transfer, since neural network weights encode computational functionality beyond their spectral structure. Different parameter organizations may retain different task-relevant information even when their spectral concentration is similar. This explains why improving path smoothness or LF concentration does not necessarily yield proportional improvements in downstream accuracy.

Second, our additional experiments provide empirical evidence for the effectiveness and limitations of this heuristic. In Appendix E.5, we use the same AJP-based permutation framework to generate six functionally equivalent parameterizations of pretrained DeiT-S-L6, with different LF concentrations. We then initialize DeiT-Ti-L6 using FRONT, keeping the extraction procedure, target architecture, and training settings identical.

LF LevelΔ\DeltaLR(%)Descendant Top-1 Accuracy (%)
Scratch—40.62
Level 110.2253.58
Level 220.4854.71
Level 330.7357.05
Level 440.9857.34
Level 551.2257.42
Level 661.4658.75

Under this consistent extraction setting, increasing LF concentration from 10.22% to 61.46% is accompanied by an improvement in descendant accuracy from 53.58% to 58.75%. However, the gains are not proportional: increasing the LF ratio from 30.73% to 51.22% improves accuracy by only 0.37 percentage points. Overall, these findings support path smoothness as a practical surrogate for improving spectral extraction, particularly under a fixed low-frequency coefficient budget. AJP therefore provides an effective training-free heuristic for identifying favorable parameterizations, rather than a guarantee of optimal downstream performance.

Reviewer_zwDcOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Spatial Permutation Alignment (SPA), which rearranges the weights of a pretrained model using function-preserving permutations to increase low-frequency spectral energy without additional training. SPA-FRONT, which combines SPA with the existing FRONT, improves performance and training efficiency for initializing vision and language models of various sizes.

Critical strengths
  1. The training free pipeline is practical, and applying it to the existing baseline (Front) appears to have a meaningful effect on improving initialization performance.

  2. The author faithfully carried out experiments and ablation across various backbones.

Critical weaknesses
  1. Baseline comparison on language models experiments : the language model experiments seem to require comparison with more competitive initialization methods such as LiGO and WAVE.

  2. Computational Efficiency : Although AJP preprocessing alone requires 66.4 seconds of CPU time, the time and memory complexity as a function of model size, as well as the end-to-end cost, have not been thoroughly analyzed.

Questions for authors

see weakness

3 comments
AuthorsOpenReview ↗

Response to Reviewer zwDc (1/2)

Dear Reviewer zwDc:

We sincerely appreciate your thoughtful comments and valuable suggestions, as well as the time and effort you devoted to reviewing our manuscript. We have provided detailed responses below and made corresponding revisions to the manuscript. We hope that our responses and revisions have satisfactorily addressed your concerns.

W1. Baseline comparison on language models experiments : the language model experiments seem to require comparison with more competitive initialization methods such as LiGO and WAVE.

R1: Thank you for this valuable suggestion. Our original language model experiments focused on training-free initialization methods to evaluate knowledge transfer without additional data-driven optimization. In contrast, LiGO requires gradient-based alignment, while WAVE and TLEG rely on auxiliary training to learn transferable representations.

Following your suggestion, we have extended the comparison to include LiGO, WAVE, and TLEG, with results provided in Appendix E.4 (Fig. 7). All methods initialize the same BERT-S-L6 descendant under identical target-pretraining settings, as detailed in Table 1. LiGO uses 100 alignment steps, while WAVE and TLEG each require 10,000 auxiliary-training steps.

Table 1. FLOPs reductions are measured relative to scratch training and exclude initialization overhead.

MethodAuxiliary Optimization StepsFLOPs Reduction (%)
Scratch00.0
LiGO10012.6
WAVE10,00033.8
TLEG10,00038.8
FRONT047.5
SPA-FRONT (Ours)058.7

The additional baselines show that WAVE and TLEG achieve substantially greater FLOPs reductions than LiGO (33.8% and 38.8% vs. 12.6%), demonstrating the effectiveness of auxiliary training for learning transferable representations. Notably, SPA-FRONT achieves a 58.7% reduction without any auxiliary optimization, outperforming WAVE and TLEG by 24.9 and 19.9 percentage points, respectively. This advantage highlights the effectiveness of our reparameterization-based approach. Rather than relying on expensive auxiliary training to learn transferable representations, SPA-FRONT exploits function-preserving permutations to improve the spectral organization of pretrained weights, allowing more information to be retained during low-frequency extraction.

Overall, these results demonstrate that SPA-FRONT not only outperforms competitive learned initialization methods in target-pretraining efficiency, but also eliminates their auxiliary-training requirements, further supporting the effectiveness of training-free structural preconditioning for knowledge transfer.

AuthorsOpenReview ↗

Response to Reviewer zwDc (2/2)

W2. Computational Efficiency: Although AJP preprocessing alone requires 66.4 seconds of CPU time, the time and memory complexity as a function of model size, as well as the end-to-end cost, have not been thoroughly analyzed.

R2: Thank you for this valuable comment. We have added a comprehensive computational efficiency analysis in Appendix E.3, covering (1) the runtime and memory scalability of SPA across different model sizes and (2) end-to-end costs compared with auxiliary-training methods, including WAVE and TLEG. All measurements were conducted on the same hardware. These additional analyses demonstrate that SPA introduces a modest one-time preprocessing cost while providing a favorable end-to-end performance–efficiency trade-off.

(1) Computational scalability of SPA. We measure the preprocessing time of depth-wise alignment and AJP, together with peak CPU memory, across six DeiT configurations.

ModelLayersWidthParams (M)Depth (s)AJP (s)Total (s)Peak CPU RAM (MiB)
DeiT-Ti-L661923.100.092.692.78953.2
DeiT-Ti-L12121925.830.155.485.63976.0
DeiT-S-L6638411.460.8113.0813.891018.0
DeiT-S-L121238422.161.0129.3230.331034.9
DeiT-B-L6676844.107.1166.4073.511016.4
DeiT-B-L121276886.6810.34132.58142.921010.9

The results show that preprocessing time increases with model size, while AJP runtime approximately doubles as depth increases from 6 to 12 layers. Peak CPU memory remains around 0.93–1.01 GiB across the evaluated configurations. We have also provided a theoretical complexity analysis in Appendix E.3.

(2) End-to-end computational efficiency. We further compare SPA-FRONT with WAVE and TLEG, accounting for auxiliary training, preprocessing, initialization, and descendant training under identical target-training budgets.

Target SettingMethodAux. Time (s)SPA (s)Init. (s)Train (s)Total (s)Top-1 (%)
Depth (DeiT-Ti-L8)Scratch00012,600≈12,60043.7
TLEG196,0000<212,600≈208,60062.9
WAVE234,0000<212,600≈246,60065.4
FRONT00<112,600≈12,60064.4
SPA-FRONT0<1<112,600≈12,60066.6
Width (DeiT-S-L6)Scratch00017,300≈17,30049.4
WAVE357,0000<217,300≈374,30064.8
FRONT00<117,300≈17,30062.3
SPA-FRONT066.4<117,300≈17,36767.6

SPA-FRONT achieves the highest accuracy in both settings without auxiliary training. In depth transfer, it achieves 66.6% accuracy, compared with 65.4% for WAVE and 62.9% for TLEG. In width transfer, it outperforms WAVE by 2.8 percentage points while avoiding 357,000 seconds of auxiliary training. The reported results follow a single-target cost convention and compare performance under fixed descendant training budgets rather than time to a common accuracy threshold.

AuthorsOpenReview ↗

Follow-up on Our Rebuttal and Author Responses

Dear Reviewer zwDc,

We are sincerely grateful for your thoughtful review of our paper and the constructive suggestions you have provided. We truly appreciate the time and attention you have dedicated to evaluating our work.

We have carefully prepared detailed responses to the concerns raised in your review, together with additional experiments and analyses into the revised manuscript. With the discussion period nearing its conclusion, we would be very grateful if you could review our responses when your schedule permits and share your thoughts on the revisions. We would particularly value knowing whether there are any outstanding concerns or aspects that may benefit from further clarification.

We fully recognize the demands on your time and greatly appreciate your contribution to the review process. Thank you once again for your time, consideration, and valuable feedback. We wish you continued success in your research and all the very best.

Best regards,

The Authors of Paper #1904

Reviewer_y7ejOpenReview ↗
Rating
4(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes SPA-FRONT, a method for improving training-free knowledge transfer across models of different sizes. The key idea is to apply function-preserving neuron permutations to pretrained weights before performing spectral extraction using FRONT. By combining depth-wise and width-wise alignment, the proposed method aims to concentrate more spectral energy in the low-frequency components retained during extraction. Experimental results on vision and language models demonstrate improved initialization quality and training efficiency compared to the original FRONT.

Critical strengths
  1. The main idea is interesting and relatively novel. While permutation symmetry has been studied in other contexts, such as neural alignment and model merging, applying it to improve frequency-based knowledge extraction is a fresh perspective. I think this is a meaningful extension of the existing FRONT framework.

  2. The experimental evaluation is fairly extensive. The authors investigate several alignment strategies and provide detailed ablation studies to examine different design choices. The experiments also cover both vision and language models using different architectures.

  3. The proposed method consistently improves upon FRONT in the reported experiments without requiring additional training. And the corresponding overhead to the additional permutation seems to be light.

Critical weaknesses
  1. Although the paper provides extensive ablation studies on different alignment strategies, the individual contributions of depth-wise and width-wise alignment are not clearly demonstrated. In particular, I could not find a direct comparison of depth-wise alignment alone, width-wise alignment alone, and their combination. Without this comparison, it is difficult to understand how much each component contributes to the overall performance improvement and whether both are necessary.

  2. The technical contribution is somewhat incremental. The proposed method essentially extends FRONT by introducing permutation-based preprocessing, rather than proposing a fundamentally new knowledge transfer framework. Nevertheless, I acknowledge that applying permutation symmetry to spectral extraction is an interesting idea, and the empirical improvements are meaningful.

Questions for authors
  1. Could the authors provide an ablation study comparing depth-wise alignment only, width-wise alignment only, and their combination under the same experimental settings?
  2. The proposed method aims to increase low-frequency spectral energy concentration. However, the results in Table 7 suggest that higher energy concentration does not always lead to better downstream accuracy. Could the authors provide further insights into the relationship between spectral energy concentration and the transfer of task-relevant knowledge?
5 comments
AuthorsOpenReview ↗

Response to Reviewer y7ej (1/3)

Dear Reviewer y7ej:

Thank you for your insightful and constructive comments and for the time and effort you devoted to reviewing our paper. We have incorporated the corresponding results and clarifications into the revised manuscript. We hope that our responses and revisions have satisfactorily addressed your concerns.

W1. Although the paper provides extensive ablation studies on different alignment strategies, the individual contributions of depth-wise and width-wise alignment are not clearly demonstrated. In particular, I could not find a direct comparison of depth-wise alignment alone, width-wise alignment alone, and their combination. Without this comparison, it is difficult to understand how much each component contributes to the overall performance improvement and whether both are necessary.

Q1. Could the authors provide an ablation study comparing depth-wise alignment only, width-wise alignment only, and their combination under the same experimental settings?

R1: Thank you for this valuable suggestion. We have added an ablation study comparing depth-only, width-only, and their combined alignment under identical BERT pre-training settings. We have included the detailed convergence curves in Appendix E.2 (Fig. 6). The results are summarized below.

MethodFLOPs Reduction
Scratch0.0%
FRONT47.5%
Depth-only48.6%
Width-only52.8%
SPA-FRONT (Depth → Width)58.7%
SPA-FRONT-w (Width → Depth)58.7%

Both depth-only and width-only alignment improve convergence efficiency over FRONT, demonstrating that each component contributes independently. Combining both achieves the largest FLOPs reduction (58.7%), outperforming either component alone. Moreover, both alignment orders achieve comparable results, indicating limited sensitivity to execution order in this setting. These results confirm the complementary benefits of depth- and width-wise alignment.

AuthorsOpenReview ↗

Response to Reviewer y7ej (2/3)

W2. The technical contribution is somewhat incremental. The proposed method essentially extends FRONT by introducing permutation-based preprocessing, rather than proposing a fundamentally new knowledge transfer framework. Nevertheless, I acknowledge that applying permutation symmetry to spectral extraction is an interesting idea, and the empirical improvements are meaningful.

R2: Thank you for your thoughtful comment and for recognizing the interesting idea of applying permutation symmetry to knowledge extraction and the meaningful empirical improvements. We agree that SPA-FRONT builds upon FRONT's spectral extraction framework. However, our contribution lies in identifying a dimension previously not fully considered for existing knowledge transfer methods.

Our key insight is that functional equivalence does not imply equivalence for knowledge extraction. A pretrained model can have multiple functionally equivalent parameterizations, yet their different weight organizations can lead to different extraction outcomes and descendant initialization performance. This suggests a different perspective: the effectiveness of knowledge transfer depends not only on what the source model has learned, but also on how its parameters are organized for extraction. By exploiting function-preserving permutations, we can improve knowledge transfer without altering the source model's function or requiring auxiliary training.

Building on this insight, we develop SPA, a training-free reparameterization method while preserving the source-model function. When integrated with FRONT, SPA concentrates more spectral energy in the retained low-frequency coefficients, improving knowledge extraction under the same coefficient budget.

Importantly, this reparameterization-based perspective is not limited to FRONT. To investigate its applicability beyond FRONT, we have added proof-of-concept experiments with SPA-DWT, SPA-TLEG, and SPA-WAVE in Appendix F.4. SPA-DWT improves the target-pretraining FLOPs reduction from 35.5% to 45.7% relative to scratch. SPA-TLEG and SPA-WAVE achieve reductions of 47.8% and 18.1%, respectively, without the 10,000 auxiliary-training steps required by their learned counterparts. Although their trained counterparts achieve larger reductions of 52.9% and 36.2%, these results demonstrate the potential of function-preserving reparameterization as a training-free alternative. Notably, these extensions employ extractor-specific reparameterization strategies rather than directly reusing SPA's alignment procedure.

Therefore, our contribution goes beyond a FRONT-specific preprocessing step: we establish function-preserving reparameterization as a distinct design direction for knowledge extraction and provide empirical evidence of its applicability across different extraction mechanisms.

AuthorsOpenReview ↗

Response to Reviewer y7ej (3/3)

Q2. The proposed method aims to increase low-frequency spectral energy concentration. However, the results in Table 7 suggest that higher energy concentration does not always lead to better downstream accuracy. Could the authors provide further insights into the relationship between spectral energy concentration and the transfer of task-relevant knowledge?

R3: Thank you for this insightful question. To further clarify the relationship between spectral energy concentration and downstream performance, we have added a controlled ablation study in Appendix E.5.

First, higher LF concentration generally benefits downstream performance under a consistent permutation mechanism. In the added Appendix E.5, we use the same AJP-based permutation method to generate six functionally equivalent parameterizations of a pretrained DeiT-S-L6 model with different LF concentrations. We then initialize DeiT-Ti-L6 using FRONT, keeping the extraction procedure, target architecture, and training settings identical.

LF LevelΔ\DeltaLR(%)Top-1 Accuracy (%)
Scratch—40.62
Level 110.2253.58
Level 220.4854.71
Level 330.7357.05
Level 440.9857.34
Level 551.2257.42
Level 661.4658.75

Under the same AJP-based method, increasing the relative LF energy ratio from 10.22% to 61.46% is accompanied by a consistent improvement in descendant accuracy from 53.58% to 58.75%. Although the gains are not proportional to the increase in LF concentration, these results support its effectiveness as a practical objective for spectral preconditioning.

As might be expected, LF concentration alone does not fully determine the effectiveness of knowledge transfer. As shown in Table 7, STSP and AJP achieve nearly identical relative LF-ratio gains (81.98% vs. 81.82%), yet AJP consistently achieves higher accuracy across all three target widths. This is because different permutations, even with similar LF energy concentration, can produce distinct weight organizations across layers and coupled modules, as well as different parameter orientations relative to the spectral basis. Consequently, the same LF retention budget may preserve different weight structures, leading to different transfer outcomes. In addition, the underlying reason is that spectral compactness and functional transferability are not equivalent. DCT-based image compression exploits spatial smoothness, where greater LF concentration improves reconstruction fidelity. However, unlike image pixels, neural network weights define computations through structured interactions across layers and modules. As established in Appendix C.2, higher LF concentration guarantees lower weight reconstruction error under a fixed orthonormal spectral truncation, but does not necessarily imply better preservation of task-relevant knowledge. This distinction motivates SPA: by exploiting function-preserving permutations, we seek a parameter arrangement that facilitates spectral knowledge extraction without changing the source model's function. The effectiveness of the resulting transfer is then evaluated empirically.

Overall, LF concentration provides a useful measure of spectral compactness, but is not a complete predictor of knowledge transfer quality. Our findings highlight the importance of both the amount of spectral energy retained and the structural organization of the retained parameters, motivating function-preserving reparameterization as an effective approach to improving knowledge extraction.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

We sincerely thank all three reviewers for their thoughtful and constructive feedback. Their comments have helped us strengthen the experimental evaluation, clarify the methodological contributions, and improve the presentation of our work. We have conducted extensive additional experiments and incorporated the corresponding results and analyses into the revised manuscript, with all revisions marked in blue text.

Summary of the reviews

The reviewers described our approach as interesting and relatively novel, with clear motivation and a novel combination of ideas. They recognized its practical, training-free approach and clear advantage in transfer cost, noting that it consistently improves upon FRONT without additional training. They also highlighted the fairly extensive evaluation and strong experimental results and convincing comparisons with competitive baselines.

We have addressed the reviewers' concerns in five key areas:

  1. Generality beyond FRONT: Whether SPA can benefit other knowledge extraction methods beyond FRONT (y7ej, F66c).
  2. Alignment contributions: Whether depth-wise and width-wise alignment contribute independently and complement each other (y7ej, F66c).
  3. Spectral concentration and transferability: How path smoothness and low-frequency concentration relate to downstream performance (y7ej, F66c).
  4. Robustness and sensitivity: Whether the improvements remain stable across different AJP configurations, hyperparameters, and random seeds (F66c).
  5. Broader evaluation and efficiency: Whether SPA-FRONT remains competitive against stronger baselines in computational efficiency and generalizes across more language models, downstream tasks, and scaling configurations (zwDc, F66c).

We briefly summarize our responses below. Replies to the individual reviews follow under each review.

1. Generality beyond FRONT

We clarify our key insight that functionally equivalent parameterizations can yield different knowledge extraction outcomes. New experiments (Appendix F.4) extend this principle to DWT, TLEG, and WAVE. SPA-DWT improves FLOPs reduction from 35.5% to 45.7%, while SPA-TLEG and SPA-WAVE demonstrate training-free alternatives that avoid 10,000 auxiliary-training steps.

2. Alignment contributions

New ablations (Appendix E.2) show that depth-only and width-only alignment achieve 48.6% and 52.8% FLOPs reductions, respectively, compared with 47.5% for FRONT. Combining both achieves 58.7%, with either execution order, demonstrating their complementary benefits.

3. Spectral concentration and transferability

We clarify that higher low-frequency concentration reduces weight reconstruction error but does not guarantee better downstream performance. A controlled experiment (Appendix E.5) shows that increasing the relative low-frequency energy gain from 10.22% to 61.46% improves accuracy from 53.58% to 58.75%. This supports spectral concentration as a useful surrogate for knowledge extraction, rather than a direct predictor of transfer performance.

4. Robustness and sensitivity

New ablations (Appendix E.9) demonstrate consistent transfer benefits across AJP anchors, search budgets, and reversal scopes. A single 2-opt pass already achieves 58.31% accuracy, close to the default configuration (58.75%). Additional analyses examine low-frequency retention and alignment choices. Five-seed experiments on DeiT-Ti, DeiT-S, and BERT-S further confirm consistent improvements with low variability (Appendix E.8).

5. Broader evaluation and efficiency

We substantially expand the comparisons and evaluations:

  • Competitive baselines: SPA-FRONT achieves 58.7% FLOPs reduction without auxiliary training (Appendix E.4), outperforming LiGO (12.6%), WAVE (33.8%), and TLEG (38.8%).
  • Computational efficiency: We analyze preprocessing scalability across six DeiT models and end-to-end costs, including auxiliary training and descendant training. SPA-FRONT achieves higher accuracy with substantially lower total cost than auxiliary-training methods under matched descendant training budgets (Appendix E.3).
  • Language generalization: SPA-FRONT improves RoBERTa-S GLUE scores from 52.89 to 61.47 and GPT-S zero-shot accuracy from 37.97% to 41.39% over FRONT. Four additional BERT source configurations achieve 50.0%–63.7% reductions in training steps relative to scratch (Appendices E.6–E.7).

We sincerely appreciate the reviewers' constructive suggestions, which have led to substantial additional experiments and a more comprehensive evaluation of SPA-FRONT. We hope these new results and clarifications help address the remaining concerns and provide a clearer understanding of the method's contributions, effectiveness, and limitations.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.