acceptodds
Under review as a conference paper at ICLR 2027

CLAIR: Consistent LLM Adaptation via Index-free Reprogramming

Abstract

Large language models (LLMs) have been adapted to time-series forecasting through patch-based reprogramming and task-specific conditioning. However, aligning contextualized patch representations requires choosing which patches to compare, while representation agreement alone does not ensure forecast agreement. We propose CLAIR, an asymmetric teacher-guided framework for Consistent LLM Adaptation via Index-free Reprogramming that couples patch-level transport alignment (PTA) with forecast-level consistency (FC). A clean-view exponential moving average (EMA) teacher provides a shared reference for a student processing a mildly perturbed view of the same input. PTA uses entropically regularized optimal transport to establish soft correspondences between post-backbone student and teacher patch representations, while FC encourages agreement between their predictions. Together with structure-aware input conditioning, these objectives complement the supervised forecasting loss without reordering the patch states used for prediction. Experiments across long-term, short-term, and few-shot forecasting demonstrate competitive performance against representative LLM-based and non-LLM forecasters, with mean relative reductions in horizon-averaged MSE of 3.1% and 3.8% across six benchmarks under the 10% and 5% few-shot settings, respectively, relative to the best evaluated LLM baseline on each dataset. Controlled comparisons on ETTh1 and ETTm1 show that PTA and index-wise alignment perform comparably without FC. When both are combined with FC, PTA reduces horizon-averaged MSE by 2.1% and 2.6%, respectively, relative to index-wise alignment.

Primary area:
learning on time series and dynamical systems
Keywords:
Time-Series Forecasting, Teacher–Student Learning, Optimal Transport

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_yRFGOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes CLAIR, a teacher–student framework for adapting frozen LLM backbones to time-series forecasting. CLAIR combines two complementary consistency objectives, i.e. patch-level transport alignment (PTA) via entropic optimal transport between clean-view EMA teacher and perturbed student patch representations, and forecast-level consistency between their normalized predictions. Experiments demonstrate the effectiveness of the proposed methods.

Critical strengths
  • The motivation is clear. The authors analysis index-free correspondence through token-sensitivity diagnostics and the student–teacher transport visualization.

  • The training-time-only overhead with no inference-time cost increase is attractive for deployment.

  • The paper is well written and easy to follow.

Critical weaknesses
  • The OT alignment introduces O(N^2) per-sequence complexity in training, which may limit scalability to longer inputs.

  • Some reported gains over LLM baselines are modest and may fall within expected variance. It is suggested to give statistical significance testing or confidence intervals.

  • The backbone LLM choices could bias comparisons: e.g., SE-LLM’s backbone was replaced with GPT-2 small to reduce differences, but it is unclear whether similar harmonization was applied to Time-LLM and other methods that typically use larger LLMs.

  • The overall overhead such as parameter counts and FLOPs are not reported.

Questions for authors

NA

1 comment

W1

We would like to clarify the computational cost of PTA. Here, N denotes the number of patches rather than the input length. With an input length of 512, patch length 16, stride 8, and end padding, N=64, resulting in a 64×64 cost matrix. Since the backbone's self-attention already scales quadratically with sequence length, PTA does not change the asymptotic training complexity. As reported in Appendix H, Sinkhorn computation takes approximately 8.7 ms per iteration (<1% of a full training step). The full teacher-guided training step requires 1.44× time and 1.24× peak memory relative to the student-only baseline, while inference latency remains unchanged.

W2

We agree that modest performance differences should be evaluated against run-to-run variability. We therefore analyzed three-run results for CLAIR and the strongest LLM baseline in terms of horizon-averaged MSE on Weather, ETTh2, ETTm1, and ETTm2. Table 7 reports mean ± SD and approximate 95% confidence intervals for the MSE differences using Welch's method.

Table 7. Run-to-run variability and 95% confidence intervals for MSE differences

DatasetBaselineHCLAIR MSE ± SDBaseline MSE ± SDΔMSE [95% CI]
WeatherFSCA960.1498 ± 0.001210.1492 ± 0.00090−0.0006 [−0.0031, 0.0019]
WeatherFSCA1920.1923 ± 0.000440.1935 ± 0.000890.0012 [−0.0007, 0.0031]
WeatherFSCA3360.2469 ± 0.000500.2474 ± 0.001880.0005 [−0.0038, 0.0048]
WeatherFSCA7200.3152 ± 0.000580.3221 ± 0.000810.0069 [0.0052, 0.0086]
ETTh2FSCA960.2840 ± 0.000550.2886 ± 0.001680.0046 [0.0009, 0.0083]
ETTh2FSCA1920.3530 ± 0.000600.3571 ± 0.000100.0041 [0.0027, 0.0055]
ETTh2FSCA3360.3715 ± 0.000390.3797 ± 0.000870.0082 [0.0064, 0.0100]
ETTh2FSCA7200.4171 ± 0.000510.4134 ± 0.00289−0.0037 [−0.0106, 0.0032]
ETTm1S²IP-LLM960.2902 ± 0.000210.2928 ± 0.002130.0026 [−0.0026, 0.0078]
ETTm1S²IP-LLM1920.3330 ± 0.002020.3324 ± 0.00237−0.0006 [−0.0056, 0.0044]
ETTm1S²IP-LLM3360.3616 ± 0.002540.3625 ± 0.002000.0009 [−0.0044, 0.0062]
ETTm1S²IP-LLM7200.4125 ± 0.002620.4160 ± 0.003050.0035 [−0.0030, 0.0100]
ETTm2FSCA960.1671 ± 0.000270.1675 ± 0.001440.0004 [−0.0030, 0.0038]
ETTm2FSCA1920.2230 ± 0.000500.2301 ± 0.002790.0071 [0.0005, 0.0137]
ETTm2FSCA3360.2740 ± 0.000590.2752 ± 0.002770.0012 [−0.0053, 0.0077]
ETTm2FSCA7200.3609 ± 0.000370.3594 ± 0.00355−0.0015 [−0.0102, 0.0072]

CLAIR achieves lower horizon-averaged MSE across all four datasets and lower mean MSE in 12 of the 16 horizon-wise comparisons. Notably, five comparisons have 95% confidence intervals entirely above zero, providing statistical evidence of improvements beyond run-to-run variability in these settings. The remaining intervals include zero, indicating that the strength of evidence varies across settings. We will incorporate these uncertainty estimates into the revised paper.

W3

We apologize for the insufficient clarification of the experimental settings in Appendix A.1. To reduce backbone-related differences, all LLM-based methods, including Time-LLM, OFA, CALF, S²IP-LLM, FSCA, and SE-LLM, used the first six layers of pretrained GPT-2 small. For FSCA and SE-LLM, we additionally adjusted their input configurations to ensure consistency with the common evaluation protocol, as described in Appendix A.1. These adjustments were made to reduce differences in experimental conditions while preserving each method's adaptation mechanisms, allowing a more direct comparison of their methodological contributions. We will clarify these settings in Appendix A.1.

W4

We would like to clarify that the computational costs are reported in Appendix H. CLAIR uses 19.18–19.24M trainable parameters (Table 14), comparable to CALF (18.53M) and fewer than Time-LLM (55.41M), S²IP-LLM (57.04M), FSCA (31.75M), and SE-LLM (97.33M), although OFA uses fewer (17.33M). At inference, only the student is retained, with 88.55M stored parameters and approximately 42–43 GFLOPs per forward pass at H=96 on ETTm1 (Appendix H.2–H.3). Its inference latency is 4.39 ms/sample, matching the student-only baseline (Table 16). Full training requires 1.44× step time and 1.24× peak GPU memory compared with student-only training (Table 15). Most of this overhead comes from the teacher forward pass and additional backward computation, while Sinkhorn PTA takes approximately 8.7 ms per iteration, less than 1% of a full training step (Appendix H.4).

Reviewer_q7VkOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a framework for adapting LLMs to time-series forecasting using a teacher–student approach. The method uses a clean-view EMA teacher and a student that receives perturbed inputs. It introduces patch-level transport alignment (PTA), which uses optimal transport to align student and teacher patch representations without relying on their original temporal positions. This is combined with forecast-level consistency (FC) to encourage agreement between their predictions. The authors evaluate the method on long-term, short-term, and few-shot forecasting benchmarks and report competitive results against existing LLM-based and non-LLM forecasting methods.

Critical strengths
  1. The paper proposes a framework that jointly considers patch-level representation alignment and forecast-level consistency. Although the individual components are established techniques, their combination is relevant to improving LLM adaptation for time-series forecasting.

  2. CLAIR demonstrates competitive forecasting results across long-term, short-term, and few-shot settings. The evaluation includes several LLM-based methods as well as non-LLM baselines, such as DLinear, iTransformer, and PatchTST.

Critical weaknesses
  1. Limited novelty: The proposed method mainly combines existing techniques, such as EMA teacher–student learning, consistency regularization, optimal transport, and patch-based LLM adaptation for time-series forecasting. Previous methods, including CALF and TS-HTFA, have already explored feature alignment together with output-level supervision, with TS-HTFA also using optimal transport. The main difference in CLAIR is that it applies optimal transport at the patch representation level and combines it with forecast-level consistency. However, it is not entirely clear how much this change adds beyond existing approaches. The patch projection mechanism is also fairly standard, as mapping time-series patches into the embedding space of an LLM has been explored in earlier forecasting methods. While CLAIR introduces a different combination of these techniques, its contribution seems relatively incremental. The paper would benefit from a clearer explanation of what makes this combination particularly useful compared with existing methods.

  2. Insufficient motivation for index-free patch correspondence: My main concern is that the paper does not clearly explain why index-free patch correspondence is necessary. Unlike cross-tokenizer distillation (Section 2.1), where teacher and student tokens may not have a direct one-to-one correspondence, CLAIR uses Gaussian noise and elementwise dropout to create two views of the same time series. These perturbations do not change the original patch positions, so each student patch already has a corresponding teacher patch. Although self-attention adds contextual information to patch representations, this does not necessarily mean that their positional correspondence is lost. Furthermore, the EMA teacher–student framework (Tarvainen & Valpola, 2017) naturally encourages consistency between clean and perturbed representations at corresponding positions to improve robustness. However, optimal transport allows patches at different positions to be matched, which could reduce the alignment loss without necessarily improving robustness at the corresponding positions. Although the results in Section 4.3 show some improvements over index-wise alignment, the experiments are limited to two closely related ETT datasets, and the gains over Index + FC are relatively modest. I think the paper needs a stronger explanation of why relaxing the known positional correspondence is beneficial, along with broader experiments and results across multiple random seeds to support this design choice.

  3. Interaction between PTA and FC: The results in Section 4.3 suggest that PTA works better when combined with forecast-level consistency (FC). However, it is unclear whether FC actually helps PTA learn more meaningful patch correspondences or whether the two losses simply provide complementary regularization. Do the learned transport plans change systematically when FC is introduced? For example, could the authors compare the transport plans learned with PTA alone and PTA + FC, and examine whether these changes are associated with improvements in forecasting performance? The analysis in Section 4.5 does not seem to directly address this question.

Questions for authors
  1. Given that prior work (TS-HTFA) already combines intermediate-feature alignment with OT-based output alignment, what specific benefit comes from moving OT to patch-level alignment in CLAIR, particularly when teacher and student patch positions are already aligned?
  2. In Section 4.3, PTA and index-wise alignment perform comparably without FC (slightly outperforming it on ETTh1 but underperforming it on ETTm1), but PTA performs better when combined with FC. Could the authors explain why forecast-level consistency benefits PTA more than index-wise alignment? Does FC lead to more meaningful transport correspondences, or do the two objectives simply provide complementary regularization?
  3. Could the authors compare PTA with simpler correspondence strategies, such as nearest-neighbor matching, under the same FC objective? A controlled comparison of index-wise alignment, nearest-neighbor matching (or another simple similarity-based approach), PTA, and no feature alignment across more diverse datasets would help establish whether optimal transport provides a meaningful advantage over simpler alternatives.
  4. In Table 6, setting the EMA momentum to zero leads to the largest performance drop among the reported ablations, suggesting that teacher stabilization plays an important role in CLAIR. Could the authors provide a controlled comparison of index-wise alignment and PTA under both EMA and non-EMA teacher configurations (i.e., with temporal weight averaging and with the teacher directly synchronized to the student)? This would help clarify whether the improvements from PTA are mainly due to the proposed transport-based correspondence or depend heavily on EMA teacher stabilization.
  5. The student patches and structural anchors are perturbed independently, potentially introducing inconsistencies between representations of the same time series. Could the authors explain why this is beneficial?
2 comments

W1 & Q1

We agree that EMA-based consistency learning and entropic OT are established techniques, and we do not claim novelty in either component individually. Related alignment frameworks also build on established methods to address specific representation problems. For instance, CALF combines cross-attention with intermediate feature regularization and output consistency, while TS-HTFA incorporates contrastive feature alignment and Sinkhorn-based OT for output alignment. These approaches illustrate how the role of an alignment mechanism depends on the representations being aligned and the supervision it provides. In this context, CLAIR focuses on a different question: how to define correspondence between contextualized patches of two views of the same time series while preserving temporal order for forecasting. Unlike the output-level OT used in TS-HTFA, PTA defines correspondence between contextualized patch states, while FC separately supervises forecasts at corresponding future time steps.

Although the two views retain the same patch positions, CLAIR uses PTA to establish flexible correspondence between contextualized student and teacher states without changing the original patch order used for forecasting. Since representation agreement under this flexible matching does not guarantee forecast agreement, FC provides a complementary constraint on predictions at corresponding future time steps. The controlled comparison in Section 4.3 (Table 5) supports this design. Without FC, PTA and index-wise alignment achieve similar horizon-averaged MSE (0.4307 vs. 0.4325 on ETTh1; 0.3660 vs. 0.3652 on ETTm1). Adding FC reduces MSE by 0.0213 and 0.0167 for PTA, compared with 0.0142 and 0.0067 for index-wise alignment, resulting in relative improvements of 2.1% and 2.6% for PTA + FC over Index + FC. These results support considering patch correspondence and forecast consistency jointly. We will clarify this design distinction in Sections 1 and 2.2.

W3 & Q2

We appreciate the reviewer's question regarding whether FC changes the correspondences learned by PTA or primarily provides complementary regularization. The distinction lies in the constraints imposed by the two objectives. PTA allows flexible correspondence between contextualized student and teacher patch states, but representation agreement under this matching does not necessarily ensure agreement between the resulting forecasts (Section 3.3). FC addresses this remaining discrepancy by constraining predictions at corresponding future time steps. In contrast, index-wise alignment imposes position-specific representation constraints that may overlap more with those induced by forecast consistency. The controlled comparison in Section 4.3 (Table 5) provides evidence for this interaction. Adding FC to index-wise alignment reduces horizon-averaged MSE by 0.0142 on ETTh1 and 0.0067 on ETTm1, whereas adding FC to PTA yields larger reductions of 0.0213 and 0.0167, respectively. Appendix F (Figure 6b) further shows that removing FC increases the student–teacher PTA cost during training, suggesting an interaction at the representation-alignment level.

To examine this interaction from an optimization perspective, we analyzed the cosine similarity between the alignment and FC gradients at the trained CLAIR parameters (Table 8).

Table 8. Cosine similarity between loss gradients at the trained CLAIR parameters

Gradient pairETTh1ETTh2ETTm1Weather
Index vs FC0.8960.7840.3210.123
PTA vs FC0.9280.8670.6390.490
Uniform plan vs FC0.9390.9120.9100.670
PTA vs Uniform plan0.9900.9550.6940.684
PTA vs Index0.9890.9680.7250.747
Uniform plan vs Index0.9670.8900.3530.284

The PTA gradient is more aligned with the FC gradient than the index-wise gradient on all four datasets (ETTh1 0.928 vs. 0.896, ETTh2 0.867 vs. 0.784, ETTm1 0.639 vs. 0.321, and Weather 0.490 vs. 0.123). PTA also produces fewer batches whose gradient opposes the FC gradient (3% vs. 20% on ETTm1 and 6% vs. 27% on Weather), and within the GPT-2 backbone parameters this holds in all 16 checkpoints. These observations are consistent with PTA and FC providing more compatible optimization signals. Although the uniform plan exhibits higher gradient alignment with FC than PTA, gradient similarity alone does not determine forecasting performance. Under the same FC objective on Weather, PTA + FC achieves lower mean MSE than Uniform-OT + FC at both H = 96 (0.1498 vs. 0.1507) and H = 720 (0.3152 vs. 0.3193), as reported in Table 1 of our response to Reviewer jKmm.

W2 & Q3

We agree that each student patch has a same-index teacher counterpart, and contextualization does not invalidate this correspondence. However, restricting alignment to the same index may be overly restrictive. In CLAIR, the student and EMA teacher differ due to stochastic perturbations and parameter updates. The EMA parameter gap and student dropout explain most of the same-index discrepancy, while context propagation contributes less (Tables 5–6, response to Reviewer jKmm). PTA therefore keeps the patch order for forecasting but lets multiple contextualized teacher states supervise each student state.

Matched comparisons

With identical architectures, perturbations, EMA teachers, and supervised objectives, PTA and index-wise alignment perform similarly without FC (Section 4.3, Table 5). With FC, PTA reduces MSE relative to Index + FC by 2.1% (ETTh1) and 2.6% (ETTm1). On Weather, PTA + FC reduces MSE from 0.1508 to 0.1498 (H = 96) and from 0.3197 to 0.3152 (H = 720) (Table 1, response to Reviewer jKmm).

Learned correspondences

The learned transport plans exhibit non-uniform, input-dependent matching patterns, including dataset-dependent locality and temporal-lag structure (Tables 2–4, response to Reviewer jKmm). These diagnostics characterize how PTA distributes correspondence across patches rather than restricting alignment to identical positions.

Simpler alignment baselines

Under the same FC objective on Weather, PTA + FC achieves lower mean MSE than Uniform-OT (0.1498 vs. 0.1507 at H = 96; 0.3152 vs. 0.3193 at H = 720) and Local-OT (0.1498 vs. 0.1503; 0.3152 vs. 0.3184).

Nearest-neighbor comparison

Table 9. Comparison of Patch Alignment Strategies on Weather.

Weather (mean ± SD)H=96 MSEH=96 MAEH=720 MSEH=720 MAE
FC only0.1517 ± 0.000590.2057 ± 0.001370.3216 ± 0.002040.3383 ± 0.00179
Index + FC0.1508 ± 0.000040.2042 ± 0.001840.3197 ± 0.002780.3356 ± 0.00040
NN + FC0.1509 ± 0.000250.2046 ± 0.003110.3204 ± 0.001470.3355 ± 0.00202
PTA + FC0.1498 ± 0.001210.2001 ± 0.001900.3152 ± 0.000580.3301 ± 0.00163

As requested, we compare PTA + FC with hard nearest-neighbor matching (NN + FC), index-wise alignment, and FC only (no feature alignment) under matched architecture, perturbations, EMA teacher, FC objective, and training settings. PTA + FC achieves lower mean MSE and MAE than these alternatives at both horizons, whereas NN + FC performs on par with Index + FC, suggesting that hard similarity-based matching alone does not reproduce the gains of PTA.

Q4

To examine how teacher stabilization affects patch correspondence, we compare PTA + FC and Index + FC under two teacher configurations: an EMA teacher (μ = 0.9999) and a directly synchronized teacher (μ = 0). The architecture, perturbations, FC objective, and training schedule are held fixed.

Table 10. EMA × alignment comparison on ETTh1 and ETTm1 (averaged over four forecasting horizons)

Teacher updateAlignmentETTh1 MSEETTh1 MAEETTm1 MSEETTm1 MAE
EMA (μ = 0.9999)PTA + FC0.40940.42870.34930.3820
EMA (μ = 0.9999)Index + FC0.41830.43570.35850.3876
Direct (μ = 0)PTA + FC0.43570.44570.36810.3923
Direct (μ = 0)Index + FC0.43710.44630.36920.3947

Under the same EMA teacher, PTA + FC reduces MSE relative to Index + FC by 2.1% on ETTh1 and 2.6% on ETTm1. With direct teacher synchronization (μ = 0), PTA + FC still achieves slightly lower MSE, but the relative reductions narrow to approximately 0.3% on both datasets. EMA stabilization improves both alignment strategies, with larger reductions for PTA + FC than Index + FC (6.0% vs. 4.3% on ETTh1 and 5.1% vs. 2.9% on ETTm1). Consistent with these results, Appendix F (Figure 6a) shows lower epoch-to-epoch reference drift with EMA teacher stabilization.

Q5

We thank the reviewer for raising this point. The patch and structural-anchor windows are extracted from the same time series but perturbed independently only in the student branch, while the EMA teacher processes their unperturbed counterparts (Section 3.3, Eq. 8). This design exposes the student to mild variations in local patches and multi-scale structural summaries while preserving their common temporal origin. Rather than relying on a shared perturbation pattern, the student is encouraged to maintain representation and forecast consistency with the clean-view EMA teacher through PTA and FC. The perturbations are mild, with Gaussian noise of standard deviation 0.01 and elementwise dropout of 0.05 (Appendix A.5). The ablation in Section 4.4 (Table 6) further shows that removing Gaussian noise increases horizon-averaged MSE from 0.4094 to 0.4272 on ETTh1 and from 0.3493 to 0.3598 on ETTm1, supporting the role of perturbation-based regularization.

Reviewer_jKmmOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents CLAIR for LLM-based time-series forecasting. A clean-view ponential moving average teacher supervises a perturbed student through patch-level transport alignment and forecast-level consistency. The alignment uses entropically regularized optimal transport to match contextualized patch representations across positions. Soft prompts and a multiscale structural anchor provide additional input conditioning. Experiments cover long-term, short-term, and few-shot forecasting, with controlled comparisons between transport-based and index-wise alignment.

Critical strengths

S1. The paper evaluates multiple forecasting settings and includes controlled comparisons, ablations, and computational profiling. Such experiments provide useful evidence for the framework.

S2. The teacher and transport computation are removed after training, allowing deployment using only the student for efficient forecasting.

Critical weaknesses

W1. The methodological contribution is limited. EMA-based prediction consistency follows Mean Teacher, while joint feature regularization and output consistency already appear in CALF. PTA applies standard entropic optimal transport to student–teacher patch representations. The main distinction is the choice of patch correspondence within an existing consistency-learning framework, and the reported improvements alone do not establish a broader conceptual contribution.

W2. The necessity and predictive value of cross-position matching remain insufficiently established. Both views retain the same temporal grid, and Gaussian noise and dropout do not change patch positions. Contextualization therefore does not by itself invalidate index-wise correspondence. Table 5 provides useful evidence, but PTA performs similarly to index-wise alignment without FC, and its additional benefit with FC is demonstrated on only two datasets. Transport visualizations characterize matching patterns without establishing why off-diagonal matches improve forecasting. Thus, the evidence does not distinguish the proposed matching mechanism from a more general regularization effect.

Questions for authors

Q1. What specific methodological advance distinguishes CLAIR from combining an EMA consistency framework with standard Sinkhorn feature alignment, given that feature- and output-level supervision already exist in prior forecasting methods?

Q2. Since both views preserve patch positions, what evidence establishes the benefit of cross-position correspondence over index-wise constrained matching? Would the advantage persist when these alternatives are trained under matched settings across additional datasets?

Q3. What is the variability across runs for the PTA + FC versus Index + FC comparison? Are the reported 2.1% and 2.6% MSE reductions consistent across seeds and forecasting horizons?

2 comments

W1 & Q1

We agree that EMA-based consistency learning and entropic OT are established techniques, and we do not claim novelty in either component individually. Related alignment frameworks also build on established methods to address specific representation problems. For instance, CALF combines cross-attention with intermediate feature regularization and output consistency, while TS-HTFA incorporates contrastive feature alignment and Sinkhorn-based OT for output alignment. These approaches illustrate how the role of an alignment mechanism depends on the representations being aligned and the supervision it provides. In this context, CLAIR focuses on a different question: how to define correspondence between contextualized patches of two views of the same time series while preserving temporal order for forecasting.

Although the two views retain the same patch positions, CLAIR uses PTA to establish flexible correspondence between contextualized student and teacher states without changing the original patch order used for forecasting. Since representation agreement under this flexible matching does not guarantee forecast agreement, FC provides a complementary constraint on predictions at corresponding future time steps. The controlled comparison in Section 4.3 (Table 5) supports this design. Without FC, PTA and index-wise alignment achieve similar horizon-averaged MSE (0.4307 vs. 0.4325 on ETTh1; 0.3660 vs. 0.3652 on ETTm1). Adding FC reduces MSE by 0.0213 and 0.0167 for PTA, compared with 0.0142 and 0.0067 for index-wise alignment, resulting in relative improvements of 2.1% and 2.6% for PTA + FC over Index + FC. These results support considering patch correspondence and forecast consistency jointly. We will clarify this design distinction in Sections 1 and 2.2.

Q3

To examine the consistency of the reported improvements across runs and forecasting horizons, we evaluated PTA + FC and Index + FC over three runs at each horizon on ETTh1 and ETTm1 (Table 7).

Table 7. Run-to-run variability (MSE, mean ± SD).

DatasetHPTA + FCIndex + FCReduction
ETTh1960.3695 ± 0.0003900.3786 ± 0.0001682.40%
ETTh11920.4018 ± 0.0000730.4091 ± 0.0007561.78%
ETTh13360.4167 ± 0.0005820.4258 ± 0.0004172.14%
ETTh17200.4497 ± 0.0006110.4595 ± 0.0009782.13%
ETTm1960.2902 ± 0.0002140.2960 ± 0.0008571.96%
ETTm11920.3330 ± 0.0020200.3441 ± 0.0025643.23%
ETTm13360.3616 ± 0.0025360.3705 ± 0.0014062.40%
ETTm17200.4125 ± 0.0026230.4233 ± 0.0014482.55%

PTA + FC achieves lower mean MSE than Index + FC across all eight dataset–horizon settings, with reductions of 1.78–2.40% on ETTh1 and 1.96–3.23% on ETTm1. These results show that the reported horizon-averaged improvements of 2.1% and 2.6% are consistent across the evaluated forecasting horizons rather than driven by a single setting.

W2 & Q2

We agree that patch positions remain unchanged and index-wise correspondence remains valid. However, matching each student state only to its same-index EMA teacher state may be restrictive. To assess predictive benefits beyond general regularization, we compared matching strategies on Weather and analyzed transport plans and student–teacher discrepancies on ETTh2, ETTm1, and Weather.

1. Predictive value under matched settings

Table 1. Weather forecasting

Model variantH=96 MSEH=96 MAEH=720 MSEH=720 MAE
Supervised only0.1528 ± 0.002500.2063 ± 0.003890.3231 ± 0.001620.3394 ± 0.00067
FC only0.1517 ± 0.000590.2057 ± 0.001370.3216 ± 0.002040.3383 ± 0.00179
Index only0.1512 ± 0.001770.2053 ± 0.002380.3207 ± 0.002380.3371 ± 0.00127
PTA only0.1511 ± 0.000900.2046 ± 0.003420.3196 ± 0.001730.3360 ± 0.00066
Index + FC0.1508 ± 0.000040.2042 ± 0.001840.3197 ± 0.002780.3356 ± 0.00040
NN + FC0.1509 ± 0.000250.2046 ± 0.003110.3204 ± 0.001470.3355 ± 0.00202
Uniform-OT + FC0.1507 ± 0.001900.2042 ± 0.002400.3193 ± 0.003710.3353 ± 0.00330
Local-OT + FC0.1503 ± 0.001370.2038 ± 0.000590.3184 ± 0.002620.3348 ± 0.00140
PTA + FC (CLAIR)0.1498 ± 0.001210.2001 ± 0.001900.3152 ± 0.000580.3301 ± 0.00163

On Weather, PTA + FC achieves lower mean MSE than Index + FC, NN + FC, Uniform-OT + FC, and Local-OT + FC at both horizons. PTA and Index perform similarly without FC. These matched comparisons support the benefit of learned correspondence combined with FC.

2. Learned correspondence: structure and input dependence

To examine whether PTA learns correspondence rather than merely relaxing index-wise alignment, we present three transport-plan diagnostics.

Table 2. Statistics of PTA transport plans

MetricETTh2ETTm1Weather
Normalized entropy0.9780.9830.988
Diagonal mass0.0430.0320.028
TV distance from uniform0.2000.1870.139
Near-diagonal mass (within ±1, × uniform)1.801.671.60
Transport / same-index cost1.771.361.62

Table 3. Daily-lag enrichment and peak nonzero-lag enrichment of PTA transport mass

Dataset24-h lag (patches)24-h mass (× uniform)Peak nonzero lag (patches)Peak mass (× uniform)
ETTh231.4531.45
ETTm1121.3511.47
Weather181.0711.50

Table 4. Input dependence of the transport plan

MetricETTh2ETTm1Weather
Corr(plan, input-patch distance)0.450.450.49
Corr(plan, patch-shape similarity)0.250.090.03
Mismatched-window control (both metrics)0.000.000.00
Same-phase (24-h) mass (× uniform)1.301.371.06
Corr(daily autocorr., same-phase mass; within channel)−0.040.580.45

The learned plans remain soft but favor nearby patches over uniform matching (Table 2), with dataset-dependent temporal preferences, including elevated daily-lag mass on ETTh2 and ETTm1 (Table 3). Moreover, transport weights correlate with input-patch distance (r=0.45–0.49) after removing the mean lag pattern, whereas mismatched-window controls yield correlations near zero (Table 4). These results indicate that PTA learns structured, input-dependent correspondences rather than merely following a fixed temporal matching pattern.

3. Sources of same-index discrepancy

We further examined how contextualization and teacher–student differences affect representations at corresponding patch positions.

Table 5. Context propagation in the alignment space

DiagnosticNoise σETTh2ETTm1Weather
Off-position change share (one patch)0.0142%44%48%
Next-patch / perturbed-patch change0.17.6%5.6%7.6%
Propagation share of same-index gap (all patches)0.018.6%5.1%8.2%
Propagation share of same-index gap (all patches)0.114.9%6.5%14.7%
Structural-anchor path share0.010.05%1.7%1.1%

Table 6. Sources of same-index student–teacher discrepancy

SourceETTh2ETTm1Weather
EMA parameter gap50%84%63%
Student GPT-2 dropout46%14%34%
Input noise + dropout5.1%2.3%4.2%
EMA share of transport cost81%93%85%

Although perturbations propagate across positions, their contribution to same-index discrepancy is limited (5.1–8.6% under evaluation-mode noise at σ=0.01; Table 5). Under training conditions, the EMA parameter gap and student-side dropout account for most of the discrepancy (Table 6). These findings suggest that relaxing fixed same-index supervision may be useful, rather than implying that contextualization invalidates positional correspondence.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.