acceptodds
Under review as a conference paper at ICLR 2027

Architecture-Driven Shift: A Lightweight Proxy for Logit-Shift Trends

Abstract

Continual Learning (CL) is a practical paradigm to utilize power of deep pre-trained neural networks, but which pre-trained model has a better ability to balance “Plasticity-Stability”, deserving to be chosen? The logit shift serves as a natural proxy, which represents the logit shift in CL. However, obtaining the logit shift requires huge computational cost, which hinders large-scale model selection. Existing theoretical analyses fail to offer an efficient alternative because of the assumption of uniform hidden layer widths, which ignores the structural heterogeneity (variable width and depth) of real-world architectures. This raises a critical question: what theoretical relationship can be identified between heterogeneous architecture and logit shift on prior tasks (that the model has been trained on)? To answer the question, we decouple logit shift into architecture dependency and data dependency to establish our framework, which reveals that the combination of two dependency, defined as Architecture-driven Shift (ADS), can capture the logit shift tendency well computable via a lightweight calibration using a few data samples and architectures. Specifically, for a well-optimized model on prior tasks, higher ADS is associated with a larger logit shift after training on the current task, which derived based on three mechanistic components: (1) spectral norm scaling of weight matrix gradients with layer width, (2) the optimization path length of the new task, and (3) the asymptotic task conflict in wide networks. Extensive empirical results across more than 175 diverse architectures demonstrate a strong monotonic correlation (the weakest Spearman's ) between ADS and logit shift on datasets scenarios. Practically, we demonstrate that ADS can serve as a lightweight proxy of the expected calibration error, which is a widely used metric for reliable model selection.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
Logit Shift, Continual Learning, Transfer Learning, Heterogeneous Architectures, Model Calibration

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_rzwtOpenReview ↗
Rating
3
Confidence
1
Informativeness
1
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces an interesting approach of predicting the continual learning capability of a particular model architecture, that is to use a proxy defined as the logit shift of the old task after the new task is trained over. To approximate this proxy, the authors proposed a theoretical framework that factors in several intuitive measures like gradient, the model complexity and task-gradient alignment, followed by an regression to make this form concrete based on real data. A good amount of model architectures and datasets are selected to perform the experiments that verify the effectiveness of the method.

Critical strengths

Overall the proposed method seems to demonstrate its superiority based on the extensive data points in the experiments. The proposed formulation is also easy to implement, despite being a bit adhoc.

Critical weaknesses

I do not find major flaws, but my below questions could be answered before we call the theory in this paper really sound.

Questions for authors
  1. Consider a scenario of multi-class classification with softmax prediction, and the logics for all classes shift by the same constant after the new task. In this case, the prediction distribution on the old task doesn't change, but ADS changed likely. Put it more generally, why the logit and its derived signal could be a good surrogate for transferability measurement, apart from the theoretical convenience?

  2. ADS afaik is an upperbound of the logit shift. But upperbound doesn't automatically guarantee the monotonity. For example it's possible that A < B, B_i > B_j but A_i < A_j. A further analysis of this, at least a guarantee even if not 100% accurate, will make the theory more justifiable.

2 comments

Point 1

  1. We thank the reviewer for this observation. We agree that an upper bound alone does not guarantee monotonicity between ADS and logit shift, and our current theoretical analysis does not establish such a guarantee.
  2. To address this limitation empirically, we evaluate the cross-architecture ranking consistency between ADS and measured logit shifts using correlation coefficients. The results in Tables 1 and 2 provide empirical evidence of ranking agreement across multiple scenarios and model families, while also revealing scenario-specific limitations.
  3. We acknowledge that establishing sufficient theoretical conditions for monotonicity remains an important direction for future work.

Point 2

  1. We thank the reviewer for this observation. We agree that an upper bound alone does not guarantee monotonicity between ADS and logit shift, and our current theoretical analysis does not establish such a guarantee.
  2. To address this limitation empirically, we evaluate the cross-architecture ranking consistency between ADS and measured logit shifts using correlation coefficients. The results in Tables 1 and 2 provide empirical evidence of ranking agreement across multiple scenarios and model families, while also revealing scenario-specific limitations.
  3. We acknowledge that establishing sufficient theoretical conditions for monotonicity remains an important direction for future work.

Reference

[1] Etash Kumar Guha and Vihan Lakshman. On the diminishing returns of width for continual learning. In International Conference on Machine Learning, pp. 16706–16730. PMLR, 2024.

[2] Joonkyu Kim, Yejin Kim, and Jy yong Sohn. Measuring representational shifts in continual learning: A linear transformation perspective. In Forty-second International Conference on Machine Learning, 2025. URL https://openreview.net/forum?id=Kz1zCJRr1r.

AuthorsOpenReview ↗

Looking forward to the response from Reviewer rzwt

Dear Reviewer rzwt,

Thanks for your valuable feedback. As the author–reviewer discussion period will end in approximately three days , we kindly ask you to review our rebuttal and let us know whether your concerns have been addressed.

If you have any remaining questions, we would be happy to provide further clarification.

Best regards, The Authors

Reviewer_bXXROpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper presents a new proxy to predict logit shift in continual learning. The central observation of the paper is that logit shift is often depends on architectural property (i.e. width of each layer). The paper constructs series of theory to characterize contribution of architecture on logit shift. Such correlation is empirically validated on image classification tasks.

Critical strengths

None

Critical weaknesses

The paper is incredibly unreadable. I believe the paper is largely written by LLMs and not polished enough by humans. Generally, the texts are largely focused on presenting theories and results, without explaining why they are needed or what motivates them.

  1. It is hard to understand the motivation of the paper.
  • The observation that large width neural network is robust to forgetting is already common. Therefore the observation that increasing width leads to less logit shift is not surprising. Even the prior works are based on CL metrics, those metrics are largely correlated to logit shift and I don't see any clear difference between them.
  1. It is hard to understand purpose of the theories.
  • In L70-80, the paper lists theoretical results, but does not explains why they are needed and how they are connected.
  • For example, why linking the width of hidden layers and the spectral norm of the weight gradients matters in context of the paper? Why the paper considers dynamic optimization level? Why decouple adaptation process into macro- and microscopic phenomena? What is "scaling of the optimization path", and why does it matter?
  • Except listed ones above, there are numerous things that is hard to understand
  • Section 4 simply lists theories without explaining why they are needed and what insight they gives
  1. It is hard to understand main claim of the paper.
  • It is hard to distinguish central hypothesis or research question that the paper is studies.
  • The paper largely focuses on contructing theoretical framework related to architecture (width of network) and logit shift but does not explains why it is needed, or why it is novel question. A connection between architecture (width of network) and forgetting has been already proven and simply switching metric to logit shift instead of accuracy or forgetting does not brings any novelty.
Questions for authors

See weakness

3 comments

Novelty Beyond Prior Work.

We read the comment about "lacking of novelty" as two claims: (i) the width–forgetting effect is already well established, and (ii) swapping accuracy/forgetting for logit shift therefore adds no novelty. However, both points are already discussed in the paper (Sec. 2.2; Sec. 3.2).

As for (i), the comment takes a one-sided view of one of our contributions. Our contribution is not limited to network width.

  • As discussed in Sec. 2.2 (pp. 3–4, lines 156–170), prior works primarily consider homogeneous architectures with uniform hidden-layer widths. However, for heterogeneous architectures, which layer's width should represent network width?
  • In contrast, we explicitly model heterogeneous architectures with variable layer-wise widths and depths (Secs. 4.2–4.5; Eq. (14)).

On (ii), the comment attributes to us a novelty claim we do not make. We do not claim novelty from introducing logit shift or simply replacing conventional CL metrics with it. Logit shift is an established research object adopted from prior work, and its motivation and distinction from conventional CL metrics are already discussed in Sec. 2.2 (lines 149–155) and Sec. 3.2 (lines 198–203).

One contribution is establishing the theoretical framework linking heterogeneous architectures (variable width and depth) to logit shift (Eq. (14)), which captures the tendency of logit shift across heterogeneous architectures without requiring full new-task training and direct logit-shift measurement for every candidate architecture (Sec. 3.2; Sec. 4.5).

To illustrate why width is insufficient for heterogeneous architectures, we consider the average width (the natural extension of the width metric for homogeneous architectures) as a simple metric (Table Y).

Table Y. Architecture-wise ADS and measured logit shift in MNIST →\to FashionMNIST.

Model: Hidden-layer widthsAvg. widthADSLogit shift
A: [2048, 2048, 512]15363.8711.23
B: [512, 2048, 2048]15364.2112.15
C: [4096, 512, 4096]≈\approx 29014.5813.49

As shown in Table Y, average width alone cannot distinguish Models A and B. Model C has a greater average width yet exhibits a larger logit shift, which generally indicates that it is less robust to forgetting. ADS correctly ranks all three, illustrating that layer-wise architectural modeling captures distinctions missed by average width alone.

Motivation and Central Research Question.

We respectfully clarify that the central research question is explicitly stated in the Abstract (p. 1, lines 19–21): what theoretical relationship can be identified between heterogeneous architectures and logit shift on prior tasks? Furthermore, Sec. 3.2 (pp. 4–5, lines 204–218) explains the computational motivation: exhaustive logit-shift measurement requires training every candidate architecture, whereas ADS aims to capture logit-shift trends across different architectures through lightweight calibration on subsets of architectures AcalA_{\text{cal}} and data DcalD_{\text{cal}}.

Theoretical Motivation and Connections

We respectfully clarify that the motivation, roles, and logical connections of these theoretical components are already explained in the manuscript, particularly in Sec. 3.3, Secs. 4.1–4.5, and the derivation roadmap in Appendix Fig. 4.

Q1. Why link hidden-layer widths to the spectral norm of weight gradients?

As explained in Sec. 4.1 (Eq. (5)), logit shift is decomposed into layer-wise contributions involving gradient norms and gradient alignment. Since Eq. (5) identifies gradient norms as a factor governing logit shift, characterizing their width-dependent scaling (Proposition 4.1) establishes the static architectural component required for ADS.

Q2. Why consider dynamic optimization?

As stated in Sec. 3.2, logit shift measures changes in model outputs after learning a new task. Since these changes arise through optimization, static architectural properties alone cannot characterize them. Therefore, Secs. 4.3–4.4 analyze how architecture influences the optimization trajectory and cross-task gradient interactions.

Q3. Why decouple adaptation into macroscopic and microscopic phenomena?

As explained in the Introduction (lines 75–81), these analyses capture two complementary aspects of optimization. We distinguish them by their levels of analysis:

  • Macroscopically: we characterize the cumulative optimization path length by integrating gradient norms along the training trajectory (Lemma 4.3);

  • Microscopically: we examine layer-wise interactions between prior-task output sensitivity and new-task update directions, separating systematic task interference from high-dimensional stochastic noise (Proposition 4.5).

Both characterize distinct factors in the layer-wise decomposition of logit shift (Eq. (5)).

Q4. What is "scaling of the optimization path," and why does it matter?

As defined in Sec. 4.3 (Lemma 4.3), it characterizes how the cumulative optimization path length scales with layer widths (Eq. (9)). As explained in our response to Q3, this provides the macroscopic dynamic component required to derive ADS (Eqs. (13)–(14)).

Overall theoretical organization.

Thus, Sec. 4 does not merely list theoretical results: Eq. (5) identifies the factors contributing to an upper bound on logit shift. Secs. 4.2–4.4 characterize their architectural dependencies, and Sec. 4.5 integrates these components into ADS (Eq. (14)). This logical progression is explicitly illustrated in Appendix Fig. 4.

Request for Specific Evidence.

We respectfully ask the reviewer to identify the specific technical concerns underlying the Soundness score of 1, as the current review does not specify any concrete technical flaws in the assumptions, derivations, or experimental results. We would also appreciate specific examples of passages considered difficult to follow, so that we can address these concerns directly.

AuthorsOpenReview ↗

Looking forward to the response from Reviewer bXXR

Dear Reviewer bXXR,

Thanks for your valuable feedback. As the author–reviewer discussion period will end in approximately three days, we kindly ask you to review our rebuttal and let us know whether your concerns have been addressed.

If you have any remaining questions, we would be happy to provide further clarification.

Best regards, The Authors

Reviewer_4okeOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a lightweight proxy for predicting logit-shift trends in continual learning, Architecture-driven Shift (ADS). It incorporates layer widths, depth, and task-dependent calibration parameters. Experiments are conducted using fully connected neural networks, CNNs, and ViTs on vision tasks, such as FashionMNIST, MNIST, and CIFAR-10. The results show a monotonic correlation between ADS and measured logit shift under different model architectures.

Critical strengths
  • The paper is well-written.
  • The information obtained from the model architecture could be an interesting idea.
  • The experiments cover multiple model architectures, from regular neural networks to transformer-based models.
Critical weaknesses
  • The computational cost for calibration on new tasks should be included and reported. Without it, it's difficult to fairly evaluate the claimed computational benefit.
  • The abstract and conclusion both suggest a strong correlation between the proposed selector and logit shift, but Table 2 contains several low correlations across different calibration ratios.
Questions for authors

None

2 comments

Point 1

We measured the wall-clock costs of exhaustive direct measurement and ADS calibration-based scoring across ten scenarios on an RTX 4090 D. Under the nominal 5% architecture calibration ratio in Table 2, ADS selects 10 of 175 candidate FNNs via stratified sampling and measures their logit shifts using 30% of the old-task data. In contrast, exhaustive direct measurement adapts all 175 candidates and evaluates their logit shifts on the full old-task dataset.

Table 3. Wall-Clock Cost Comparison Between Exhaustive Direct Measurement and ADS Calibration-Based Scoring. Reported times include Task-1 and subsequent-task training, probing, and associated overhead, including ADS fitting and scoring. Relative Time (%) denotes the ADS runtime as a percentage of exhaustive direct measurement runtime. For simplicity, M and F denote MNIST and FashionMNIST, respectively.

ScenarioDirect measurementADS calibration + scoringSpeedupRelative Time
M →\to F250.07 min13.72 min18.22$\times$5.49%
F →\to M251.74 min13.22 min19.04$\times$5.25%
Split M (0–4 →\to 5–9)180.67 min9.99 min18.09$\times$5.53%
Split M (5-9 →\to 0–4)179.39 min10.05 min17.85$\times$5.60%
Split F (0–4 →\to 5–9)178.18 min10.00 min17.82$\times$5.61%
Split F (5-9 →\to 0–4)177.27 min10.03 min17.68$\times$5.66%
Split M (5 tasks)118.36 min6.53 min18.14$\times$5.51%
Split F (5 tasks)116.89 min6.41 min18.23$\times$5.49%
Rotated M (5 tasks)502.99 min28.20 min17.84$\times$5.61%
Rotated F (5 tasks)502.73 min27.37 min18.37$\times$5.45%

As shown in Table 3, ADS achieves 17.68×–19.04× speedups across the ten scenarios, requiring only 5.25%–5.66% of the runtime of exhaustive direct measurement. These results demonstrate that ADS retains a substantial computational advantage even when training and calibration overheads are included, including in 2-task and 5-task sequence learning.

We have incorporated this comparison and its analysis in the Sec. 5.1 (Table 3) of the revised manuscript to support the calibration efficiency of ADS.

Point 2

We thank the reviewer for this observation. We clarify that the strong correlation reported in Table 1 concerns full-pool verification, whereas Table 2 evaluates held-out calibration performance and reveals scenario-specific limitations.

  1. Scope of the correlation claim. The strong correlation result reported in the abstract is based on Table 1, which evaluates 175 FNN architectures per scenario across 3 datasets and 8 scenarios, with the weakest Spearman coefficient being (rs=0.731r_s=0.731). We acknowledge that the original wording did not sufficiently distinguish this setting from the held-out calibration protocol in Table 2. We have clarified the scope of this claim in the abstract and conclusion.
  2. Corrections to Table 2. We have corrected the numerical errors in Table 2, without affecting the other reported results or the main conclusions. We acknowledge that ADS exhibits weak correlation in the C →\to M scenario (rs=0.207r_s=0.207), indicating that strong held-out correlations are not guaranteed across all scenarios.
  3. Failure-case analysis. Appendix A.3.2 of the original manuscript has already analysed this failure case, identifying the discrepancy between nominal and gradient-path effective depth as a possible contributor.
  4. Practical utility. As shown in Table 6 of the original manuscript (Table 8 of the revised manuscript), the experiment objective is to retain the architecture with the smallest logit shift. Despite weak global ranking correlation, ADS-based coarse-grained selection retains the lowest-shift architecture in 91.0% of 200 splits while pruning 40% of candidate architectures, compared with 60.0% under random selection, illustrating that ADS has practical utility for coarse-grained candidate selection in this challenging scenario.
AuthorsOpenReview ↗

Looking forward to the response from Reviewer 4oke

Dear Reviewer 4oke,

Thanks for your valuable feedback. As the author–reviewer discussion period will end in approximately three days, we kindly ask you to review our rebuttal and let us know whether your concerns have been addressed.

If you have any remaining questions, we would be happy to provide further clarification.

Best regards, The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.