acceptodds
Under review as a conference paper at ICLR 2027

Matched Endpoints, Different Futures: Forecasting under Order-Dependent Non-Stationarity

Abstract

Forecasting models can adapt to distribution shift and encode long histories, but these capabilities do not isolate whether temporal order remains predictive once the observable endpoint and the experienced transformations are fixed. We formalize this phenomenon as order-dependent predictive non-stationarity (ODN): matched-endpoint histories containing the same transformation instances in different orders induce different conditional futures. Under squared loss and balanced paired orders, collapsing a reversed-order pair incurs minimum excess risk equal to one quarter of the per-target squared separation between its conditional means. We introduce ORBIT (Ordered Residual Bias for Interacting Transitions), a compact forecasting adapter that converts local predictive changes into bounded low-rank corrections and composes them in temporal order. We analyze when this interaction separates reversed paths and how approximation error propagates through the composition. On the primary context-held-out benchmark, ORBIT reduces unseen-order NMSE from for Mamba to , a 21.5% reduction under matched training budgets and similar reported parameter scales. Removing or perturbing ordered composition increases paired error. An independent rotation diagnostic shows larger gain point estimates across increasing future-gap bins, and a no-auxiliary-loss variant retains a positive paired gain over Mamba. Across five semi-synthetic datasets, unseen-order error decreases by . Together, these results support ordered composition as a useful inductive bias for forecasting under path-dependent shift.

Primary area:
learning on time series and dynamical systems
Keywords:
time-series forecasting, order-dependent predictive non-stationarity, path dependence, compositional generalization, low-rank adaptation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_fP2qOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates order-dependent predictive non-stationarity, where historical sequences with matched observable endpoints but different transformation orders may lead to different future distributions. The authors propose ORBIT, a forecasting framework that models local predictive changes through bounded low-rank corrections and their temporal composition. The paper theoretically analyzes the predictive consequences of discarding order information and the properties of ordered composition, and evaluates the effectiveness and generalization of the proposed method through controlled benchmarks, semi-synthetic datasets, standard time series forecasting tasks, and ablation studies.

Critical strengths
  1. Formalization of order-dependent predictive non-stationarity. The paper defines a forecasting setting in which different transformation orders may lead to different future distributions despite matched observable endpoints. The associated risk analysis clarifies the predictive consequences of discarding historical order information, providing a focused theoretical motivation for order-sensitive forecasting.
  2. Targeted evaluation of order-dependent generalization. The controlled experiments explicitly distinguish seen and unseen transformation orders, while the structural ablations examine the effects of ordered composition and residual corrections. This evaluation directly addresses the proposed research problem and provides evidence for the effectiveness of ORBIT's order-sensitive modeling mechanism.
Critical weaknesses
  1. Limited evaluation of generalization under increasingly complex ordered compositions. Although the experiments investigate reversed transformation orders and permutation coverage, they do not systematically examine how forecasting performance changes as the number of transformations increases under a controlled setting. The theoretical perturbation bounds also leave the empirical sensitivity of ordered composition to increasing path length unresolved. Consequently, the current evidence is insufficient to establish whether ORBIT maintains its forecasting advantage under more complex and longer transformation sequences.
  2. Insufficient justification for the specific architectural design. The theoretical analysis characterizes the predictive consequences of discarding order information and the order sensitivity of matrix composition, but does not establish the necessity of bounded low-rank corrections. Although the paper includes comparisons with order-sensitive forecasting models and structural ablations, these experiments do not fully isolate the advantages of the proposed adaptation mechanism over alternative order-sensitive designs under a shared backbone and comparable capacity. This limits the evidence supporting the distinctive contribution of ORBIT's architectural choices.
Questions for authors
  1. Can ORBIT maintain its forecasting advantage under longer transformation chains and varying numbers of transformations?
  2. How does the empirical sensitivity to approximation errors change as the ordered composition becomes longer, and how does this relate to the theoretical perturbation bounds?
  3. What justifies the use of bounded low-rank corrections rather than other order-sensitive modeling mechanisms?
  4. Can the authors provide controlled comparisons with alternative order-sensitive adaptation methods under comparable model capacity and computational costs?
  5. Figure 1 appears to exhibit some AI-generated design characteristics. Could the authors provide a clearer explanation of the overall model architecture and workflow?
1 comment
AuthorsOpenReview ↗

Longer chains, perturbation sensitivity, capacity-matched adapters, and Figure 1

Thank you for asking how chain length and capacity affect the comparison. We extended the permutation experiment, compared two adapters at matched parameter counts, and revised Figure 1. We distinguish the number of transformations in the data (KK) from the number of updates inside ORBIT (LL). Q and W number your questions and weaknesses in their original order.

Q1 / W1 · Transformation count. Near 20% permutation coverage, ORBIT versus Delta Transformer unseen NMSE is .126 vs .102 at K=3K=3, .151 vs .137 at K=4K=4, and .200 vs .207 at K=5K=5. Table 11 (p. 22) now includes K=6K=6, with 144/720 admitted orders, 28,800/5,760/28,800 train/validation/test histories, and the original training recipe. Unseen NMSE is .221 for ORBIT, .238 for Delta, .246 for Raw-History Transformer, and .259 for Path Development. The paired Delta-minus-ORBIT difference is +.017 [.004, .030].

The short-chain results favor Delta. The ranking changes as both chain length and the number of admitted orders increase (1/5/24/144), with 200 training histories per admitted order at K=6K=6. The fixed-K=5K=5 sweep helps interpret exposure: ORBIT trails Delta at 0% coverage (.255 vs .252), then leads at every tested level from 20% to 80%. Its normalized AUPC advantage is .013 [.003, .023] (Section 6.3, p. 9; Tables 9–10).

Q2 / W1 · Perturbations along longer paths. Eq. (6) ensures ∥Rℓ∥2≤.2\|R_\ell\|_2\leq.2 for the normalized model. A state perturbation introduced after the first correction, with subsequent factors frozen, therefore has norm ratio at most 1.2k−11.2^{k-1} after factor kk. At k=2/5/10k=2/5/10 the envelope is 1.200/2.074/5.160. Eq. (19) bounds the product difference for two aligned sequences of bounded correction matrices.

Figure 3a (p. 10) and Table 24 (p. 27) show the measured state-norm ratios on the Appendix E benchmark (L=10L=10, five seeds, 2,000 held-out BA contexts per seed). ORBIT's means at factors 2/5/10 are 1.04/1.09/1.16 (95th percentiles 1.11/1.24/1.41). Without factor normalization they are 1.18/1.73/3.02 (95th percentiles 1.42/2.61/5.48). The envelope applies only to normalized ORBIT. This test isolates propagation through fixed factors. Section 7 identifies the additional empirical question of how errors accumulate when a disturbed history changes both the initial state and the estimated corrections.

Figure 3b uses independently trained models on the primary benchmark. At T=96T=96, the w=2sw=2s grids yield L=4/10/22L=4/10/22. ORBIT unseen NMSE is .141/.124/.122, versus .149/.157/.181 without normalization. Full ORBIT uses .48/.52/.58 GFLOPs per sample and 13.2/14.1/15.4 ms per A100 batch. Moving from 10 to 22 factors changes ORBIT NMSE by −.002 [−.009, .005]. The added cost is measurable, while the interval includes zero for the accuracy change. Stride, window width, and auxiliary span all vary in this sweep (Appendix F.5, Table 25, p. 27).

Q3 / W2 · Bounded low rank. Even rank-one factors can have a nonzero commutator. Low rank lets ORBIT express ordered interactions at O(dr)O(dr) transport cost per update (Section 4.3). On the primary benchmark, replacing the product by addition raises NMSE by .051 [.028, .074], and shuffling factor order raises it by .111 [.088, .134] (Table 1). The full-rank variant gives .133 at 1.81M parameters versus .124 at 1.21M for ORBIT (Table 16).

Q4 / W2 · Matched capacity. To compare order-sensitive adapters on the same backbone, we widened only the Delta adapter from Table 15. Table 26 (p. 27) reports parameters matched at the displayed precision:

BackboneAdded params eachNMSE, Delta / ORBITGFLOPs/sample, Delta / ORBITPaired Delta − ORBIT [95% CI]
PatchTST+.31M.394 / .377.56 / .57+.017 [.004, .031]
TimeMixer+.36M.399 / .385.60 / .62+.014 [.002, .027]

Widening lowered Delta's NMSE from .397 to .394 on PatchTST and from .403 to .399 on TimeMixer. ORBIT retained lower error on both backbones at slightly higher compute. This supports the design choice among these alternatives. The Mamba contrast (.158 vs .124) still compares complete configurations with different fusion paths.

Q5 · Figure 1 workflow. We revised Figure 1 and its caption (p. 4). To address the concern about the visual style, we removed the decorative check mark and redundant annotations and enlarged key text. One history (1) feeds the backbone (2) and local encoder (3). Window changes determine the direction and strength of bounded factors (4). Starting from z0=hz_0=h, ORBIT applies zℓ=(I+Rℓ)zℓ−1z_\ell=(I+R_\ell)z_{\ell-1} in temporal order (5), then fuses zLz_L with hh before forecasting (6). The bottom panel shows the evaluator's AB and BA paths reaching the same observable endpoint before their conditional mean futures diverge. Each forecast uses one history. The AI Use Statement (p. 11) describes assistance with figure and table code and the authors’ verification.

Reviewer_y3raOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces order-dependent predictive non-stationarity (ODN), where histories involving identical transformations and matched endpoints can yield different future outcomes. To address this challenge, the authors propose ORBIT, which sequentially composes bounded low-rank corrections derived from local predictive changes. Experimental results demonstrate improvements over Mamba in unseen-order forecasting, supported by structural ablations and semi-synthetic evaluations.

Critical strengths
  1. Interesting problem formulation. The paper introduces a meaningful perspective on forecasting under path-dependent distribution shifts, highlighting the importance of transformation order.
  2. Promising empirical results. ORBIT achieves improvements in NMSE over Mamba, while structural ablations provide evidence supporting the effectiveness of ordered composition.
Critical weaknesses
  1. Insufficient validation of the proposed mechanism. The theoretical analysis assumes fixed noncommutative operators, whereas ORBIT employs learned, context-dependent corrections. The connection between these corrections and the underlying transformations remains insufficiently established.
  2. Limited characterization of local dynamics. ORBIT relies on differences between adjacent window descriptors, which may not adequately distinguish meaningful changes in predictive dynamics from transient fluctuations or noise.
  3. Potential instability under long compositions. Although individual corrections are bounded, their sequential composition may accumulate approximation errors or amplify perturbations, raising concerns about stability and scalability over longer sequences.
  4. Incomplete experimental validation. The primary benchmark lacks direct diagnostics of ODN, while the mixed results on permutation generalization raise concerns about the robustness and consistency of the reported improvements.
Questions for authors

Could the authors clarify how the learned low-rank corrections correspond to the noncommutative transformations assumed in the theoretical analysis? Additionally, can they provide direct evidence of ODN in the real-world benchmarks and further investigate the stability of sequential composition over longer horizons? A more detailed explanation of the inconsistent permutation generalization results would also strengthen the empirical conclusions.

1 comment
AuthorsOpenReview ↗

Learned corrections, direct ODN evidence, and sensitivity to perturbations

Thank you for asking how the operator analysis applies to ORBIT's learned corrections. We tested whether their order changes both the forecast and its accuracy. W numbers the weaknesses in your review.

W1 · Learned corrections and the theory. Theorem 1 gives the excess risk of discarding order on balanced matched pairs. Eq. (8) concerns two fixed corrections. For context-dependent factors, Proposition 1 bounds the error of using aligned reference factors. Corollary 1 gives a sufficient separation condition under a common base state and linear readout (Section 5, pp. 6–7).

On 10,000 held-out BA contexts in Appendix E (five seeds), we reversed only the order of each checkpoint's learned factors, keeping the history, backbone state, gate, head, and target fixed. Across the four future-gap strata, mean forecast separation was .007/.018/.041/.061. The projected adjacent-factor commutator averaged .004/.012/.029/.052 in the same strata (Table 22, p. 26), using

Cgrid(H)=1L−1∑ℓ=1L−1∥W(Rℓ+1Rℓ−RℓRℓ+1)h∥mse.C_{\rm grid}(H)=\frac{1}{L-1}\sum_{\ell=1}^{L-1}\|W(R_{\ell+1}R_\ell-R_\ell R_{\ell+1})h\|_{\rm mse}.

Did that order improve accuracy? Reversal raised NMSE from .126 to .137 (paired difference +.011 [.004, .018]). At D=0D=0 the change was +.001 [−.004, .006] and at 0<D≤.120<D\leq.12 it was +.006 [−.001, .013]. For .12<D≤.28.12<D\leq.28 and .28<D≤.4.28<D\leq.4 it was +.018 [.006, .030] and +.026 [.008, .044], respectively (Section 6.4, p. 9; Appendix F.7, Table 27, p. 27). Chronological order improves prediction on this benchmark. Identifying the learned factors with generator primitives would require an independent alignment, and testing Corollary 1 would also require bounds on the approximation errors.

W2 · Local dynamics versus noise. The descriptor summarizes overlapping 16-step windows, and its auxiliary task predicts the next eight observations within the history. The magnitude gate scales a correction by ρ(m)=.2m/(m+1)\rho(m)=.2m/(m+1). To test its response to a transient, we applied a single positive pulse of three input standard deviations to a fixed channel at t=48t=48 after normalization, with unchanged BA targets. Clean-to-perturbed NMSE was .124 to .129 for ORBIT (paired increase .005 [.001, .010]), .146 to .169 without the gate (.023 [.012, .036]), and .158 to .166 for Mamba (.008 [.003, .014]). The gate attenuates sensitivity to this pulse (Section 6.4, p. 9; Table 23, p. 26). No-auxiliary-loss ORBIT also retains a .028 [.009, .047] gain over Mamba on Appendix E (Table 20c).

W3 · Stability of longer compositions. Figure 3a (p. 10) and Table 24 (p. 27) track a state perturbation injected after the first factor with later matrices fixed. At factors 2/5/10 the theoretical envelope is 1.200/2.074/5.160; observed mean ratios are 1.04/1.09/1.16 for ORBIT and 1.18/1.73/3.02 without normalization. Eq. (19) bounds the product difference between two aligned sequences of bounded corrections. Section 7 distinguishes this bound from the joint state and factor errors that can arise when corrections are re-estimated from a disturbed history. In the separately trained primary models (Fig. 3b), 4/10/22 grid updates give ORBIT NMSE .141/.124/.122 versus .149/.157/.181 without normalization. The paired 22-minus-10 difference is −.002 [−.009, .005]. Window, stride, and auxiliary span vary together (Appendix F.5, Table 25, p. 27).

W4 · Direct evidence of ODN. The direct evidence comes from the analytic constructions in Appendices B and E. All 10,000 Appendix E test contexts have identical terminal states across orders. Of these, 2,500 have zero future gap and 7,500 have a positive gap. Appendix B divides each target coordinate by its validation standard deviation before computing DD. Weighting the displayed Table 20(b) strata gives mean D2≈.03370D^2\approx.03370. Applying Theorem 1 in these normalized coordinates gives a balanced order-blind excess NMSE of .03370/4≈.0084.03370/4\approx.0084. The .035 BA gain compares two trained predictors on those contexts. In the zero-gap stratum that contrast is −.001 [−.017, .015] (p. 25).

For the primary benchmark, Table 1 reports held-out prediction error without a separate analytic gap distribution. On real base windows the rotation study verifies endpoint matching (maximum .1498, tolerance .15; Table 12). On unmodified series the macro NMSE improves from .3203 to .3032 (Table 2). Neither real-series evaluation identifies both order-specific conditional means (Section 7, p. 10).

W4 · Mixed permutation results. Near 20% coverage, Delta-minus-ORBIT NMSE is −.024/−.014/+.007/+.017 at K=3/4/5/6K=3/4/5/6, with the six-primitive gain at +.017 [.004, .030]. Admitted orders increase from 1 to 144. At fixed K=5K=5, ORBIT trails Delta at 0% coverage (.255 vs .252) and leads at each tested level from 20% to 80% (Section 6.3, p. 9; Tables 10–11, pp. 21–22).

Reviewer_ubWBOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces a forecasting setting called ODN, where two histories contain the same transformation instances and matched observable endpoints, but their conditional futures differ due to the ordering of transformations. The authors first characterize the excess squared prediction risk incurred by collapsing reversed-order histories. Then the authors propose ORBIT, a bounded low-rank adapter that composes local predictive changes in temporal order. Specifically, ORBIT extracts local descriptor changes, maps them into bounded low-rank residual transformations, and composes these transformations in temporal order. The resulting adapted representation is combined with the original backbone representation for forecasting. Experiments on constructed benchmarks show that ORBIT improves unseen-order NMSE over multiple sequence modeling baselines, such as GRU, Transformer, Mamba, Signature, and Path Development models. Additional analysis and ablation study are also provided.

Critical strengths
  • This paper introduces an interesting problem. The paper provides a formulation for studying whether temporal order carries predictive information beyond the observable endpoint and the collection of transformations. This offers a useful perspective on forecasting under history-dependent distribution shifts.
  • The use of bounded low-rank corrections and temporally ordered matrix composition is intuitive and well-motivated.
  • The emprical results significantly improve the performance compared to multiple baseline models. The matched training budgets and parameter scales make this result promising.
  • The ablation study is useful.
Critical weaknesses
  • My major concern is that the evaluation relies heavily on synthetic or semi-synthetic constructions aligned with the proposed inductive bias. I would like to see stronger evidence that the method generalizes beyond the specific settings of rotational interventions.
  • The primary are strong, but the more general permutation-coverage experiments reveal several limitations. For example, at 20% permutation coverage as shown in Table11, ORBIT performs worse than the baselines when the number of primitives is three and four. Also as shown in Table 13, PGG of ORBIT is larger than that of the comparators across all datasets. These observations may suggest that the benefits of the proposed compositional inductive bias depend on the task structure and coverage regime. The paper would benefit from a more explicit discussion of when and why ordered composition helps.
Questions for authors

Please address the weaknesses.

2 comments
AuthorsOpenReview ↗

Beyond rotations: when does ordered composition improve unseen-order forecasts?

Thank you for pointing out the short-chain results and the larger permutation generalization gaps (PGGs). We agree that both qualify the main finding. Section 6.3 (p. 9) now discusses them alongside a second intervention family. W numbers the weaknesses in your review.

W1 · Beyond rotations. We replaced Givens rotations with shears on the latent dynamics parameters. For example, A=I+αe1e2⊤A=I+\alpha e_1e_2^\top and B=I+βe2e3⊤B=I+\beta e_2e_3^\top give AB−BA=αβe1e3⊤AB-BA=\alpha\beta e_1e_3^\top. Shears on disjoint supports provide a commuting control. We retained the split-local real base windows, stability map, AB-only training and validation, held-out BA testing, and five-seed paired evaluation. Repeating pair selection for shears gave a maximum endpoint discrepancy below the prespecified .15 tolerance (Appendix F.1, p. 25). Table 21 (p. 26) reports held-out BA forecast errors, reproduced below.

Real base seriesComparator BA NMSEORBIT BA NMSEComparator − ORBIT
ETTh1.451.406+.045
ETTm1.425.387+.038
Weather.318.270+.048
Electricity.252.213+.039
Traffic.488.453+.035
Equal-weight mean.3868.3458+.0410

The paired 95% CI for the mean gain under noncommuting shears is [.014, .067]. Under commuting shears the contrast is −.001 [−.009, .008]. Rotations give a similar pattern, with contrasts of +.049 [.021, .077] and −.002 [−.011, .006], respectively (Table 13, p. 22). Thus the gain extends to a nonorthogonal intervention family on all five real base series. On unmodified ETTh1, ETTm1, and Weather, PatchTST+ORBIT also achieves a four-horizon macro NMSE of .3032 versus .3203 for the best displayed baseline macro (Table 2). These ordinary forecasts supply one observed continuation per history, so the two order-specific conditional means remain unidentified there (Section 7, p. 10).

W2 · Permutation coverage and chain length. At roughly 20% coverage, you are right that Delta Transformer has lower unseen NMSE for three and four primitives. ORBIT versus Delta is .126 vs .102 at K=3K=3, .151 vs .137 at K=4K=4, and .200 vs .207 at K=5K=5. We extended Table 11 (p. 22) to K=6K=6 using the frozen recipe and the same 28,800/5,760/28,800 train/validation/test histories. ORBIT achieves .221 versus .238 for Delta, a paired gain of +.017 [.004, .030]. The number of admitted orders rises from 1 to 5, 24, and 144 across the four rows. This makes it difficult to attribute the change in ranking to chain length alone.

The fixed-K=5K=5 sweep addresses exposure more directly. With only single-primitive training, ORBIT trails Delta (.255 vs .252). At 20/40/60/80% coverage, ORBIT gives .200/.155/.125/.105 versus .207/.170/.145/.125 for Delta. Its normalized area-under-the-curve advantage is .013 [.003, .023], although no model comes within 5% of its own all-order reference at any tested coverage through 80% (Tables 9–10, p. 21).

W2 · Larger permutation generalization gaps. On ETTh1 under noncommuting rotations, the comparator's seen and unseen NMSE are .413 and .460. ORBIT lowers them to .353 and .407. The improvement is larger on seen orders (.060) than on unseen orders (.053), so the within-model gap grows from .047 to .054 even as held-out error falls. ORBIT's PGG is numerically larger on all five datasets. The macro comparator-minus-ORBIT PGG difference is −.008 [−.031, .015], leaving its direction uncertain (Section 6.3, p. 9; Table 13, p. 22). On the primary benchmark PGG falls from .061 for Mamba to .024 for ORBIT (Table 1). We therefore report the gain in held-out accuracy and the change in PGG separately for each protocol.

Thank you for the response.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.