acceptodds
Under review as a conference paper at ICLR 2027

Parameter-Efficient Temporal Domain Generalization

Abstract

Temporal domain generalization (TDG) learns from a sequence of observed domains to predict models for unseen future domains. Existing parameter-dynamics methods become prohibitively expensive at modern backbone scales because their temporal encoders and generators operate on the entire adapted parameter state. Replacing full-model updates with LoRA does not remove this failure mode: the dynamics model must still encode and reconstruct every adapter parameter, making the auxiliary model itself prohibitively large for modern backbones. We propose Low-rank Time-varying Core (LoTiC), a parameter-efficient TDG method that decomposes temporal updates into shared low-rank bases and a compact core. The bases define a stable update subspace, while only the core evolves. LoTiC therefore learns temporal dynamics in an r* r space instead of tracking the full adapter. We justify this factorization through the geometry of parameter increments and support continuous, recurrent, and time-conditioned core models. Experiments span six controlled and real-world benchmarks, three data modalities, classification and generation, and backbones from 66M to 8B parameters. LoTiC ranks first in 14 of 15 settings, including near-perfect future-domain accuracy with an 8B backbone on Rotating 2-Moons. It adds only 0.13–0.51% trainable parameters beyond LoRA and trains 1.8–7.1 faster than parameter-dynamics baselines. These results establish that temporal parameter dynamics can remain both learnable and efficient as pretrained backbones scale. Code is available at https://github.com/dfwsds/LoTiC.

Primary area:
learning theory
Keywords:
Temporal domain generalization, parameter dynamics, model manifold, pretrained models

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_uyneOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

The paper proposes LoTiC, a parameter-efficient approach to temporal domain generalization. It assumes that optimal parameters across time lie on a common low-dimensional embedded manifold and shows that parameter increments relative to a fixed pretrained model preserve this structure. Unlike independently trained domain-specific LoRA adapters with factors BtB_t and AtA_t, LoTiC uses shared, time-invariant bases BB and AA, together with a time-varying core FtF_t, to represent updates as ΔWt=BFtA\Delta W_t=BF_tA. This factorization is exact when the shared bases span the relevant column and row subspaces across domains; in practice, the method learns compact shared bases. Experiments demonstrate its performance and parameter efficiency across multiple benchmarks and backbone scales, with ablations supporting the main design choices.

Critical strengths
  1. The method offers a compelling way to reduce the cost of temporal parameter modeling by concentrating temporal dynamics in a compact core. The experiments provide substantial evidence of its effectiveness and efficiency.

  2. The paper is clearly written and well organized, and the derivations are easy to follow

Critical weaknesses
  1. Trade-off between efficiency and representational capacity. The exact factorization requires shared bases that span the relevant subspaces across time, whereas the practical method uses a compact shared rank. A rank that is too small may exclude important update directions that cannot be recovered by changing the temporal core alone; increasing the rank raises the parameter and computational costs. Although Appendix A.4 reports rank sensitivity on Rotating 2-Moons, it remains unclear how this trade-off behaves on real-world datasets. Further discussion of the practical rank-selection procedure and its applicability across datasets would strengthen the paper.
Questions for authors
  1. How was the shared rank of AA and BB selected for the real-world benchmarks? Beyond the controlled rank sweep in Appendix A.4, is there evidence or practical guidance on how rank affects predictive performance and efficiency on these datasets?

  2. There appears to be an extra blank space at the end of the first line in Section 5.5.

3 comments

Thank you for your thoughtful review and for recognizing our work. We address your concerns below.

W1 The exact factorization requires shared bases that span the relevant subspaces across time, whereas the practical method uses a compact shared rank. A rank that is too small may exclude important update directions that cannot be recovered by changing the temporal core alone; increasing the rank raises the parameter and computational costs. Although Appendix A.4 reports rank sensitivity on Rotating 2-Moons, it remains unclear how this trade-off behaves on real-world datasets. Further discussion of the practical rank-selection procedure and its applicability across datasets would strengthen the paper.

Q1 How was the shared rank of AA and BB selected for the real-world benchmarks? Beyond the controlled rank sweep in Appendix A.4, is there evidence or practical guidance on how rank affects predictive performance and efficiency on these datasets?

Existing studies have shown that the configured LoRA rank need not be fully used by single-domain adaptation. The LoRA paper’s empirical analysis shows that ranks as low as 1–2 can achieve competitive performance on selected GPT-3 tasks, supporting the low effective rank of adaptation updates.

Let r1r_1 denote the effective rank needed for single-domain adaptation. Temporal distribution shifts may also have low intrinsic complexity, requiring only a small number r2r_2 of additional update directions. Intuitively, the combined requirement can be viewed as r1+r2r_1+r_2. Since r1r_1 may be smaller than the configured rank rr, the remaining capacity may already accommodate temporal variation without increasing rr.

Figure 4 shows strong performance on Moons already at r=4r=4, with stable results across a broad rank range. We additionally evaluate Yearbook and arXiv to examine this behavior on real-world visual and textual tasks.

DatasetMethodRank 4Rank 8Rank 16Rank 32
YearbookStatic LoRA15.0213.2714.7113.77
LoTiC9.539.429.839.42
arXivStatic LoRA55.1254.6554.6855.03
LoTiC52.9051.8851.6451.77

LoTiC outperforms static LoRA at every tested rank. And increasing the rank beyond r=8r=8 provides little additional benefit on either dataset, suggesting that a compact subspace is sufficient for these tasks.

In our main experiments, we use r=8r=8 for both LoTiC and the baselines, ensuring that the gains of LoTiC are not attributable to a higher adapter rank. For real-world tasks with greater complexity, increasing the rank slightly may yield better results.

Q2 There appears to be an extra blank space at the end of the first line in Section 5.5.

Thank you for pointing this out. We have traced the formatting issue to Table 4 and will correct it in the final version.

We hope these clarifications address your concerns.

Thank you again for your positive assessment and thoughtful feedback. In our rebuttal, we have provided additional evidence and analysis to further support our contributions. As the discussion period is nearing its end, we would greatly appreciate your consideration of these additional results and analyses.

Thank you for your experiments and clarification, which well address my concerns.

Reviewer_4giYOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces LoTiC, a parameter-efficient approach to temporal domain generalization that represents temporal parameter updates using shared low-rank bases and a compact time-varying core. Instead of predicting the complete adapter parameters at each timestamp, the method models temporal evolution through (\Delta W_t = BF_tA), where only (F_t) changes over time. The authors provide a theoretical discussion of parameter-increment manifolds and shared-subspace stability, and investigate continuous, recurrent, and time-conditioned core dynamics.

Experiments across six benchmarks, multiple modalities, and backbone scales ranging from 66M to 8B parameters demonstrate strong future-domain performance. The method reports substantially lower auxiliary parameter overhead and training costs than the evaluated parameter-dynamics baselines.

Critical strengths

An elegant and practically significant parameterization. The decomposition into shared bases and a time-varying core directly addresses the scalability limitations of existing parameter-dynamics TDG approaches. The formulation is simple, compatible with parameter-efficient fine-tuning, and allows different temporal dynamics models to operate on a compact state.

Strong empirical results across architectures and tasks. The evaluation covers controlled and real-world temporal shifts, classification and generation, and substantially different backbone scales. The reported performance is generally convincing, while the parameter-efficiency results suggest that the approach could be practical for large pretrained models.

Critical weaknesses

The theoretical justification for the shared-subspace assumption is limited. Lemmas 1–2 establish that translating an embedded manifold preserves its dimension and structure. However, this result does not imply that the corresponding parameter updates can be represented using fixed low-rank bases. A low-dimensional nonlinear manifold can still traverse different row and column subspaces.

The subsequent stability analysis provides sufficient conditions involving bounded gradient leakage and dissipativity, but the paper does not demonstrate that these conditions hold in the evaluated settings. Consequently, the theoretical results provide only limited support for the general applicability of the central factorization assumption.

Direct measurements of subspace overlap or reconstruction errors across independently adapted domains would strengthen this aspect of the paper.

The scalability comparisons are potentially unfavorable to the competing methods. Table 2 shows that the evaluated LoRA-adapted DRAIN, Koodos, and FreKoo baselines introduce auxiliary models with hundreds or thousands of times more trainable parameters than standard LoRA. While these implementations preserve the original temporal mechanisms, they may not represent competitive parameter-efficient alternatives.

As a result, the reported efficiency advantage could partly reflect the choice of baseline implementations rather than the fundamental superiority of the proposed factorization. Comparisons with parameter-matched compact temporal generators or dimensionality-reduced dynamics models would provide stronger evidence.

Questions for authors

Validation of the shared-subspace assumption: Can the authors quantify the similarity of domain-specific LoRA update subspaces and evaluate the reconstruction error when projecting independently optimized updates onto the learned shared bases? This would help establish whether the proposed factorization captures the intrinsic temporal structure or primarily acts as a strong regularizer.

Fairness of efficiency comparisons: Can the authors compare LoTiC against parameter-matched versions of the parameter-dynamics baselines, for example using compressed latent states or compact temporal encoders? Such comparisons would help determine whether the reported gains arise from the specific shared-core construction rather than a substantial difference in auxiliary model capacity.

2 comments

Thank you for your thoughtful review and for recognizing our work! We address your concerns below.

W1.1 Lemmas 1–2 establish that translating an embedded manifold preserves its dimension and structure. However, this result does not imply that the corresponding parameter updates can be represented using fixed low-rank bases. A low-dimensional nonlinear manifold can still traverse different row and column subspaces.

We clarify that Lemmas 1–2 do not derive the shared low-rank subspace on their own. Instead, they identify parameter increments as the modeling target for TDG with pretrained models. Prior parameter-dynamics TDG studies full weights, leaving the structure of pretrained-model updates unspecified. To bridge the gap, Lemmas 1–2 establish that the temporal-manifold formulation carries over to parameter increments.

For these increments, established research on low-rank adaptation makes LoRA a natural choice. Building on this low-rank representation, Section 4.2.2 derives the subspace-shared time-varying parameterization.

W1.2 The subsequent stability analysis provides sufficient conditions involving bounded gradient leakage and dissipativity, but the paper does not demonstrate that these conditions hold in the evaluated settings. Consequently, the theoretical results provide only limited support for the general applicability of the central factorization assumption.

Q1 Can the authors quantify the similarity of domain-specific LoRA update subspaces and evaluate the reconstruction error when projecting independently optimized updates onto the learned shared bases? This would help establish whether the proposed factorization captures the intrinsic temporal structure or primarily acts as a strong regularizer.

Theoretically, TDG assumes gradual distribution shifts, suggesting similar adaptation requirements across neighboring domains.
Empirically, we added an analysis on Rotating 2-Moons and Yearbook to assess cross-domain subspace stability. From a common checkpoint, we independently fine-tune LoRA adapters for each domain with the backbone frozen. For each adapted matrix, we measure how much of one domain’s update is captured by another domain’s rank-8 row and column subspaces. This is quantified by the fraction of update energy retained after projection. Results are averaged over adapted matrices and shown in the table, using all distinct domain pairs (All domain pairs) or only consecutive pairs (Adjacent domain pairs).

DatasetAll domain pairsAdjacent domain pairs
Rotating 2-Moons75%85%
Yearbook72%73%

Most update energy is retained across domains. These results provide empirical evidence of substantial cross-domain subspace overlap, supporting the compact shared-subspace modeling choice.

W2 While these implementations preserve the original temporal mechanisms, they may not represent competitive parameter-efficient alternatives. [...] Comparisons with parameter-matched compact temporal generators or dimensionality-reduced dynamics models would provide stronger evidence.

Q2 Can the authors compare LoTiC against parameter-matched versions of the parameter-dynamics baselines, for example using compressed latent states or compact temporal encoders?

We clarify that Table 2 already evaluates parameter-efficient adaptations of these methods. Their dominant overhead comes from neural encoders and decoders that compress predictor parameters and reconstruct them after temporal evolution. For pp predictor parameters and hidden width hh, the encoder’s input layer and decoder’s output layer alone require 2ph2ph parameters. Following the original architectures, Koodos and FreKoo use h=1024h=1024, making these mappings particularly costly. Even reducing the width offers little improvement.

The table compares full-parameter extensions with our LoRA adaptations on Moons/DistilBERT. Even on DistilBERT 66M, their original version storage alone exceeds our GPU budget.

MethodOriginal Version-LoRA VariantReduction achieved by our implementation
DRAIN34.05B115.64M294×
Koodos136.98B465.65M294×
FreKoo204.94B696.67M294×

We therefore freeze the backbone and model LoRA states while retaining each method’s original temporal mechanism. This adaptation already reduces their trainable parameters by approximately 294×. Nevertheless, substantial overhead remains because they still require encoding and decoding complete adapter states. Eliminating this overhead requires redesigning the mappings. This structural change is a core contribution of LoTiC.

We hope these clarifications and additional results address your concerns.

Thank you again for your positive assessment and thoughtful feedback. In our rebuttal, we have provided additional evidence and analysis to further support our contributions. As the discussion period is nearing its end, we would greatly appreciate your consideration of these additional results and analyses.

Reviewer_QYFcOpenReview ↗
Rating
4(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
3(+1)
Read review
Paper summary

This paper studies temporal domain generalization (TDG) with large pretrained backbones. It argues that existing parameter-dynamics methods such as DRAIN, Koodos and FreKoo become costly at scale because they model the full parameter or LoRA state over time. The authors proposed LoTiC, which writes each domain update as ΔW(t)=BFtA\Delta W(t) = B F_t A with shared bases BB and AA, meanwhile, only a small r×rr \times r core FtF_t changes over time. The core is modeled by a matrix-exponential flow, a recurrent model, or a time-conditioned network. The analysis shows that subtracting pretrained weights preserves the dimension of an assumed manifold of optima, and gradient descent keeps LoRA factors close to fixed reference subspaces. Experiments are conducted on six benchmarks, eight baselines and backbones range from 66M to 8B. Overall, LoTiC ranks first in 14 of 15 settings, with only 0.13 to 0.51% additional parameters than LoRA.

Critical strengths
  • Moving the temporal state from the full adapter to an r×rr \times r core per matrix makes parameter-dynamics TDG feasible on 8B backbone. And there’s clear evidence of parameter savings with less than 0.6% additional trainable parameters relative to LoRA (tables 2&3). Figure 3 also shows memory use close to ERM, while Koodos-LoRA runs out of memory much earlier.
  • Overall the evaluation is broad, which convers three modalities, classification and generation, and several backbone families. The gains are also substantial on the controlled benchmarks, Yearbook and WMT. On Rotating 2-Moons with DistilBERT, the sensitivity and convergence results suggest that the gains are not tied to a narrow setting or an isolated checkpoint.
Critical weaknesses
  • The theory offers limited support for the specific design. Lemmas 1-2 follow directly from translation being a diffeomorphism and do not establish a shared low-rank subspace. Theorem 4 relies on Assumptions 2 and 3, which impose strong conditions on gradients outside the reference subspaces, also they are not checked empirically. The relation between the gradient-step index and the domain index is also unclear. The analysis describes sequential gradient descent across domains, while LoTiC learns the shared bases jointly, so the link to the actual method seems somewhat loose.
  • It is unclear how much of the improvement comes from sharing BB and AA, and how much from the temporal core model. Table 4 does compare several core models, but lacks the control such as shared bases and independently learned per-domain cores with simple extrapolation.
Questions for authors
  • Could you clarify whether LoTiC jointly optimizes the shared bases and core model over all observed domains, with a loss summed over tt? Also, how is tt encoded or scaled for the flow core, and how sensitive are the results to this choice?
  • In Table 4, BAtBA_t and BtAtB_tA_t both reach 51.5% error on the binary Yearbook task. How were these variants extrapolated to future domains, and were they tuned under the same protocol as BFtABF_tA?
  • Could you report a variant with shared BB and AA and independently learned per-domain cores FtF_t, using the last core or a linear fit for future domains? This would help isolate what the temporal model adds beyond the shared subspace.
  • It seems that several ablation entries in table 4 lack standard deviations, could you clarify if these were also run with three seeds as in the main results, and report their standard deviations?
3 comments

Thank you for your thoughtful review and for recognizing our work. We address your concerns point by point below.

W1.1 Lemmas 1-2 follow directly from translation being a diffeomorphism and do not establish a shared low-rank subspace.

We clarify that Lemmas 1–2 do not derive the shared low-rank subspace on their own. Instead, they identify parameter increments as the modeling target for TDG with pretrained models. Prior parameter-dynamics TDG studies full weights, leaving the structure of pretrained-model updates unspecified. To bridge the gap, Lemmas 1–2 establish that the temporal-manifold formulation carries over to parameter increments.

For these increments, established research on low-rank adaptation makes LoRA a natural choice. Building on this low-rank representation, Section 4.2.2 derives the subspace-shared time-varying parameterization.

W1.2 Theorem 4 relies on Assumptions 2 and 3, which impose strong conditions on gradients outside the reference subspaces, also they are not checked empirically.

Theoretically, TDG assumes gradual distribution shifts, suggesting similar adaptation requirements across neighboring domains.
Empirically, we added an analysis on Rotating 2-Moons and Yearbook to assess cross-domain subspace stability. From a common checkpoint, we fine-tune LoRA adapters for each domain with the backbone frozen. For each adapted matrix, we measure how much of one domain’s update is captured by another domain’s rank-8 row and column subspaces. This is quantified by the fraction of update energy retained after projection. Results are averaged over adapted matrices and shown in the table, using all distinct domain pairs (All domain pairs) or only consecutive pairs (Adjacent domain pairs).

DatasetAll domain pairsAdjacent domain pairs
Rotating 2-Moons75%85%
Yearbook72%73%

Most update energy is retained across domains. These results provide empirical evidence of substantial cross-domain subspace overlap, supporting the compact shared-subspace modeling choice.

W1.3 The relation between the gradient-step index and the domain index is also unclear.

Thank you for pointing this out. We will revise the notation to remove this ambiguity, using kk for gradient steps and tt for domain index.

W1.4 The analysis describes sequential gradient descent across domains, while LoTiC learns the shared bases jointly, so the link to the actual method seems somewhat loose.

Q1 (first part) Could you clarify whether LoTiC jointly optimizes the shared bases and core model over all observed domains, with a loss summed over tt?

The analysis provides a conditional justification for the shared-subspace design. Evaluating all domains simultaneously on a large pretrained backbone would substantially increase VRAM. For efficient training, we therefore update the shared bases and core parameters using one domain’s loss at a time, and then revisiting all observed domains each epoch. Every domain repeatedly contributes gradients to the same shared parameters.

In summary for whole W1, we hope the above clarifications address the ambiguities. We will clarify Sections 4.1 and B.1 and add training and inference pseudocode in the final manuscript.

W2 It is unclear how much of the improvement comes from sharing BB and AA, and how much from the temporal core model.

Q3 Could you report a variant with shared BB and AA and independently learned per-domain cores FtF_t, using the last core or a linear fit for future domains?

We have added both requested controls on Rotating 2-Moons with DistilBERT and Yearbook with DINOv2-B. With the backbone frozen, we learn shared B,AB,A and a free 8×88\times8 core CtC_t for each observed domain. Future cores use either the last observed core or element-wise least-squares linear extrapolation fitted to all observed cores. Other settings follow the main experiments. We report error and std over three seeds. T+1T+1 denotes the first future domain, and Avg. averages all future domains.

VariantMoons T+1Moons Avg.Yearbook T+1Yearbook Avg.
Static BABA15.3 (0.3)18.5 (1.0)3.9 (0.5)13.8 (0.9)
BCtABC_tA + last core7.0 (3.1)27.3 (4.6)3.4 (0.7)12.6 (0.9)
BCtABC_tA + linear fit14.2 (6.0)32.5 (1.8)4.6 (0.8)15.2 (1.2)
BFtABF_tA (LoTiC)0.2 (0.3)0.6 (0.3)2.9 (0.1)9.3 (0.4)

Even with shared B,AB,A, both free-core controls underperform LoTiC. This is because the free cores are trained to fit individual domains without being constrained to follow a temporal principle. Learning them independently does not guarantee that they are suitable for post-hoc extrapolation. Together with Static BABA, these controls show that the gains of LoTiC are not simply from sharing a subspace or adding domain-specific parameters.

Q1 (second part) Also, how is tt encoded or scaled for the flow core, and how sensitive are the results to this choice?

Time is supplied as a scalar without additional embeddings. On Rotating 2-Moons, we use the domain index t=0,…,19t=0,\ldots,19 in Ft=exp⁡(tW)F_t=\exp(tW).

Scaling time to t/st/s requires a generator sWsW to maintain the same flow. This changes the required parameter scale and motivates adjusting the FtF_t's learning rate under a same training budget.

We tested normalization by the training range (t/14t/14) and the full timestamp range (t/19t/19) on Moons/DistilBERT. Keeping the generator rate unchanged yielded high error, but adjusting only its learning rate (x20) restored performance. The table reports future-domain error and standard deviation over three seeds.

Time inputGenerator LRError (%, ↓)
tt (main experiment)10−310^{-3}0.6 (0.3)
t/14t/142×10−22\times10^{-2}0.3 (0.1)
t/19t/192×10−22\times10^{-2}0.4 (0.1)

Thus, time normalization preserves strong performance when paired with an appropriate learning rate.

Q2 In Table 4, BAtBA_t and BtAtB_tA_t both reach 51.5% error on the binary Yearbook task. How were these variants extrapolated to future domains, and were they tuned under the same protocol as BFtABF_tA?

This ablation tests whether isolating temporal variation in the compact core is necessary. We apply flows as Bt=exp⁡(tUV⊤)BB_t=\exp(tUV^\top)B and At=Aexp⁡(tU′V′⊤)A_t=A\exp(tU'V'^\top), and extrapolate by evaluating them at future block indices. All variants use the same backbone, split, adapter rank/targets, optimizer settings, and epoch budget. The table reports future-domain error for each run.

VariantRun1Run2Run3
BtAB_tA51.5†15.214.4
BAtBA_t51.5†51.5†51.5†
BtAtB_tA_t51.5†51.5†51.5†
BFtABF_tA (LoTiC)8.89.39.7

† These runs initially fit the observed blocks but later diverged to NaN. Evaluation then returned a single class, with the 51.5% error determined by the label proportions across future blocks.

We can see introducing dynamics into B or A leads to substantial training instability. It makes sense as modeling high-dimensional dynamics is a well-known challenge. Also, even the fortunate converged case BtAB_tA runs underperform static LoRA.

These results support isolating temporal variation in the compact core for stable training and stronger future-domain generalization.

Q4 It seems that several ablation entries in table 4 lack standard deviations, could you clarify if these were also run with three seeds as in the main results, and report their standard deviations?

Thank you for pointing this out. The updated Table 4 below reports the mean and standard deviation over three seeds for all variants.

VariantError (%, ↓)
(a) Factorization
Static BABA13.8 (0.9)
BtAB_tA27.0 (21.2)
BAtBA_t51.5 (0.0)
BtAtB_tA_t51.5 (0.0)
BFtABF_tA9.3 (0.4)
(b) Core model
Flow9.3 (0.4)
Recurrent13.1 (0.3)
Time-to-Core11.5 (0.3)

Thank you again for your positive assessment and thoughtful feedback. In our rebuttal, we have provided additional evidence and analysis to further support our contributions. As the discussion period is nearing its end, we would greatly appreciate your consideration of these additional results and analyses.

Comments on the paper

1 comment

Dear reviewers and authors,

As the discussion period is limited, please follow up on the latest responses and comments in a timely manner to address any remaining concerns. Thank you!

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.