acceptodds
Under review as a conference paper at ICLR 2027

FlowTSFM: Test-Time Scalable Time Series Forecasting with Recurrent Quantile Flows

Abstract

Encoder-based time series foundation models (TSFMs) stack independently parameterized Transformer layers whose intermediate states carry no predictive meaning. We introduce FlowTSFM, which recasts encoder depth as transport over predictive quantiles. A single weight-shared Transformer block is integrated as a depth-conditioned residual update, and a shared decoder reads out a full quantile forecast at every step. A quantile-flow objective, inspired by flow matching, combines pinball supervision with a path term that moves the decoded quantiles along a straight line from an analytic Gaussian source to the terminal forecast. We prove that this position-matching term has the same zero-loss trajectories as regressing decoded velocities onto the constant flow-matching velocity. With 38.8M parameters, FlowTSFM is competitive with state-of-the-art TSFMs on GIFT-Eval and TIME and comes within 3–5% of the 119.5M-parameter Chronos-2. Its decoded updates consistently point toward the final forecast (CosMean 0.92, versus 0.35 for layer-wise probes of Chronos-2), and in our setup training without the path term diverged. A complete training/inference depth study shows that fixed-depth training specializes the recurrence to its budget. Training with randomized depth removes this specialization: a single checkpoint improves with inference depth and matches or exceeds depth-specialized models across budgets.

Primary area:
learning on time series and dynamical systems
Keywords:
Time Series Foundational Models, Flow Matching, Test Time Scaling

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_HdanOpenReview ↗
Rating
3(+1)
Confidence
2(−1)
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes FlowTSFM, a 38.8M-parameter encoder-based time series foundation model that applies a single weight-shared Transformer block K times as an Euler-style residual update and decodes a 99-quantile forecast at every intermediate state. It is trained with a quantile-flow objective combining pinball losses at final and interior exits, a soft anchor to a Gaussian source, and a straight-line path term toward the detached final forecast, alongside a randomized-depth variant sampling K from 3 to 15. On GIFT-Eval and TIME, the model trails Chronos-2 by 3% to 5% and TiRex by 1% to 2%, while a 5x5 train-by-inference depth grid shows fixed-depth models specialize to their training budget whereas randomized depth improves up to 9 steps.

Critical strengths

The paper provides a complete and transparent depth study across 100 grid cells and a randomized-depth checkpoint, showing clearly that fixed-depth models degrade by 10% to 89% in CRPS off their training budget while randomized depth reduces error by 26% to 29% at 6 steps compared to a shortened 12-step model, and candidly reporting degradation beyond the training range at 18 steps.

The design is clean and honestly scoped, reaching within 3% to 5% of Chronos-2 using a single shared block plus one pre-recurrent layer, while clearly stating the theoretical boundaries of Proposition 1 and experimental limitations.

Critical weaknesses

The efficiency claim confuses parameter count with compute, as the 38.8M model applies a width-1024 block 12 times over 512 patches, costing roughly as much compute as a 12-layer untied encoder with 150M non-embedding parameters which exceeds Chronos-2, while reporting no FLOPs, latency, or memory and losing on all four metrics to the similarly sized 35.3M TiRex.

The test-time compute knob has a small dynamic range and incomplete evaluation, since increasing compute by 3x from 3 to 9 steps improves error by only 3% to 5% before plateauing and the 4-step specialist is only 2.3% to 3.0% behind the 12-step one; moreover, the randomized checkpoint is compared with specialists at only 2 of 5 budgets, namely 6 and 12 steps while omitting 4, 8, and 10, performs 0.4% to 2.0% worse on all four scores at the base 12-step budget, and relies on a straw-man shortened 12-step baseline that feeds unseen step sizes instead of evaluating true early exit.

Questions for authors

Can you report the accuracy of intermediate exits across steps k alongside a true early-exit baseline, add geometry ablations comparing non-straight paths or simple consistency losses, and test whether divergence without the path term persists under lower peak learning rate, longer warm-up, QK-norm, or gating?

Can you report FLOPs, inference latency, and memory for K from 3 to 15 against Chronos-2, TiRex, and Moirai-2 to substantiate the efficiency claims under actual compute?

Why was the randomized-depth model evaluated at steps 3, 9, 15, and 18 instead of matching the specialists at 4, 8, and 10, and can you report its performance at 4, 8, and 10 steps as well as K=1 and K=2?

2 comments

Thank you for a very concrete review. Each question pointed to a clear experiment, and we ran each of them. On compute you were right, and we have corrected the claim.

In short

  • Sharing saves parameters and memory, not FLOPs at K = 12. Section 5.1 and the Limitations now say so.
  • One randomized checkpoint stays within 2% of the specialist at every budget where one exists, now from K = 1 to 18.
  • A true early exit from one 12-step run works when extrapolated along the path, but the randomized model remains better at small budgets.
  • Without the path term, every stabilizer we tried costs 2 to 7%, and gating does not stabilize at all.

W1 and Q2 · Parameters versus compute

One H200, batch 1. Appendix J.2 lists every K from 1 to 15.

FlowTSFM, K = 12Chronos-2Moirai-2TiRex
GFLOPs18932 (130 at 8k)39
Latency8.1 ms1.6×0.85×3.2× *
Peak memory265 MB264 MB80 MB173 MB

* TiRex needs its own CUDA kernel, so it was timed in a separate environment, with FlowTSFM and Chronos-2 as controls (42.5 vs 13.1 ms). The Chronos-2 ratio is the same in both sessions.

Each step costs about 0.55 ms and 14 GFLOPs. TiRex has the fewest FLOPs but is the slowest, because it recurs over time, token by token; it also reads only the last 2,048 points, where we use all 8,192.


W2 and Q3 · The randomized model at every budget

We had picked 3, 9 and 15 as the ends and middle of the training range, which left out the budgets where specialists exist. Table 3 now covers K = 1 to 18. GIFT-Eval / TIME MASE:

K124681012
Latency (ms)2.63.14.25.16.47.28.1
Randomized.907 / .861.802 / .765.756 / .719.748 / .707.746 / .702.746 / .701.746 / .701
Specialist––.761 / .719.754 / .711.749 / .716.757 / .707.743 / .698

At K = 4, 8 and 10 the single model beats the specialist on 9 of 12 scores (MASE and CRPS) and is never more than 1% worse. At K = 12 it is 0.4% behind on MASE, within seed noise, and 2.0% on GIFT-Eval CRPS: the price of one model for all budgets.

We agree the useful range starts near K = 4. Our claim is one checkpoint for every budget, not that depth buys much beyond six steps.


W2 and Q1 · True early exit

We now stop one 12-step run after k steps, with the training step size (Appendix J.4, Figure 1). The raw read-out is poor at small k. Extrapolating along the path, q^=q0+(qk−q0) K/k\hat q = q^0 + (q^k - q^0)\,K/k, where q0q^0 is the forecast decoded before the first step, needs one extra decoder read and no extra block. GIFT-Eval MASE:

Steps used346812
12-step run, raw read-out1.3711.2561.0440.8690.743
12-step run, extrapolated0.9590.9260.8750.8360.743
Path-only run, extrapolated0.8550.8260.8010.7870.734
Randomized model at K = k0.7680.7560.7480.7460.746

Extrapolation removes 26–30% of the error at k = 3–4, and 41–42% for the path-only model. For a small budget, though, running the randomized model at K = k remains the better choice, and we say so.


Q1 · Divergence under lower LR, longer warm-up, QK-norm or gating

Lower LR with a longer warm-up, a latent stabilizer and QK-norm all make the no-path run stable, but each costs accuracy and loses the trajectory:

No path term, plusGIFT-EvalTIMECosMean
½ LR, 20k warm-up+4.1%+4.7%0.37
Latent stabilizer+3.3%+2.1%0.40
QK-norm+5.5%+6.7%0.26
Learned residual gatediverges at step 3,000

The gate is per-channel on every update (g=2σ(a)g = 2\sigma(a), starting at 1). Among the options we tested, the path term is the only one that is both stable and accurate.


Q1 · Geometry ablations

Endpoint distillation keeps the final GIFT-Eval accuracy but loses the trajectory (4-step exit 2.85). A consistency loss between exits is 0.9 / 1.9% worse, again with no usable early exit (2.82). A curved path matches the final accuracy and, extrapolated along its own curve, exits better than the straight one (0.85 vs 0.96). So the experiments do not show that the path must be straight, only that prescribing one helps; the paper now discusses the choice of path.


If these results address your concerns, we would be grateful if you would consider raising your score.

Thank you for the detailed new compute and depth experiments, which address my main concerns. I am updating my rating from 2 to 3.

Reviewer_K4BEOpenReview ↗
Rating
2
Confidence
4(+1)
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper proposes FlowTSFM, a time-series foundation model that reformulates encoder depth as a progressive refinement process for probabilistic forecasting. Specifically, FlowTSFM introduces (i) a Weight-Shared Transformer that iteratively updates latent representations, (ii) a Quantile-Flow Objective that guides intermediate forecasts along a straight path from a Gaussian source to the final prediction, and (iii) Randomized-Depth Training that enables flexible inference computation with a single model. Experiments on GIFT-Eval and TIME demonstrate competitive forecasting performance, improved training stability, and flexible test-time scaling.

Critical strengths
  • The paper shows that recurrent depth can reduce the parameter requirements of time-series foundation models while maintaining competitive forecasting accuracy. The ability to adjust inference depth without retraining also offers a practical advantage.

  • Unlike flow-matching models such as Sundial [1] that generate forecasts through iterative sampling, FlowTSFM directly refines predictive quantiles from an analytic Gaussian source, avoiding stochastic sampling while providing interpretable intermediate forecasts.

Critical weaknesses
  • The practical benefit of the proposed design remains unclear. Despite emphasizing parameter efficiency, FlowTSFM has more parameters and lower forecasting accuracy than TiRex [2]. Moreover, using fewer recurrent steps further reduces accuracy, raising questions about the trade-off between computational efficiency and forecasting performance. The paper also does not sufficiently justify why intermediate forecasts must follow a consistent trajectory, beyond enabling flexible inference.

  • The significance of CosMean as an evaluation metric is not well justified. Although FlowTSFM achieves substantially higher CosMean than Chronos-2 [3], it produces less accurate final forecasts. This raises questions about the practical value of enforcing consistent intermediate predictions, particularly when their relationship to forecasting accuracy has not been established.

Questions for authors
  • Could you provide an ablation comparing the proposed quantile-flow objective with simpler alternatives, such as distillation-based supervision, to demonstrate the benefit of straight-path supervision?

  • Could you discuss whether weight sharing and straight-path supervision limit the diversity of intermediate representations? Prior work shows that different Transformer layers capture distinct levels of information [4]. I wonder whether these constraints could negatively affect final forecasting accuracy.

[1] Liu et al., "Sundial: A Family of Highly Capable Time Series Foundation Models", ICML 2025.
[2] Auer et al., "TiRex: Zero-Shot Forecasting Across Long and Short Horizons with Enhanced In-Context Learning", NeurIPS 2025.
[3] Ansari et al., "Chronos-2: From Univariate to Universal Forecasting", arXiv 2025.
[4] Park et al., "What Do Self-Supervised Vision Transformers Learn?", ICLR 2023.

4 comments

Thank you. Your review pushed us to test the claim that matters most, that the path is useful, rather than take it for granted. We tested it directly (Appendix J.1), and one result changed how we state it.

In short

  • In our experiments, the path term does two things: it keeps the shared loop trainable, and it makes early exits from a single run usable.
  • Distillation and a consistency loss keep the final accuracy but lose the trajectory. A prescribed path keeps both.
  • That path need not be straight: a curved one works as well, and the paper now says so.
  • CosMean is not a measure of accuracy; we now present it only as a diagnostic of early exits.

W1 · Why follow a consistent trajectory?

Beyond flexible inference, we found two reasons.

Trainability. Without the path term the shared loop diverges. Every other way we found to stabilize it (lower learning rate with a longer warm-up, a latent stabilizer, QK-norm) is 2 to 7% less accurate.

Early stopping. With the path, stopping a 12-step run after 4 steps costs 1.1 to 1.3× its final error. Without it, 3.3 to 3.7×.


W2 · Is CosMean linked to accuracy?

You are right that it is not. Across our eight main variants, its correlation with final MASE is weak (−0.58). What it does track closely is early-exit quality: −0.95 at 4 of 12 steps, −0.89 at 8.

Eight variants are a small, structured sample, so we report this as an association in our setting, and Section 5.2 now uses CosMean only as a diagnostic of early exits. The Chronos-2 comparison stays as a probe of how depth is organized, since Chronos-2 was not trained for it.


Q1 · Distillation-based supervision

Same budget; every exit regressed onto the detached final forecast. We added two more alternatives next to it.

GIFT-EvalTIMECosMeanExit 4/12
Ours0.7650.7340.920.96
Endpoint distillation0.7650.7450.112.85
Consistency loss between exits0.7720.7480.182.82
Curved path0.7700.7370.730.85 *

* Extrapolated along its own curve.

Distillation recovers the final accuracy, but its intermediate forecasts are unusable, and a consistency loss behaves the same way. A path fixed in advance gives both, and a curved one even exits better than the straight one.

So, in these experiments, prescribing a trajectory works better than pulling exits toward the final forecast or toward the next exit, and its shape is secondary. We have rewritten the abstract and Section 5.2 accordingly. The straight line remains the simplest choice, with a closed-form exit and the link to flow matching of Proposition 1.


Q2 · Diversity of intermediate representations

Thank you for the pointer to [4], which we now cite. We tested the concern in two ways.

Accuracy. At equal FLOPs, untying the block into 2, 4 or 12 copies changes TIME by at most 0.7%, within seed noise, and GIFT-Eval by at most 2.1%.

What each step computes. We compared the updates hk+1−hkh_{k+1} - h_k across depth (linear CKA, 256 windows, Appendix J.6). The first and last updates of the shared block are as dissimilar as those of a standard untied encoder (CKA 0.30 for both).

The shared block is conditioned on its position and on K, and it does produce distinct, depth-dependent updates. This does not show that the full representation spaces are equivalent, and the paper says so.


W1 · Accuracy and parameters versus TiRex

TiRex recurs across time, FlowTSFM across depth, so the two ideas are complementary. Appendix J.5 reports a longer-trained configuration of the same 38.8M architecture and objective:

ParamsGIFT-Eval MASEGIFT-Eval CRPSTIME MASE
FlowTSFM, longer training †38.8M0.6850.4710.674
TiRex35.3M0.7160.4880.683
Chronos-2119.5M0.6970.4850.662

† 190k steps, 24 recurrent steps, input re-injection, NorMuon, more real series in training; symmetric averaging on GIFT-Eval only. About 4× the inference compute of our K = 12 model.

Since several things change at once, this is a scaling result, not an ablation of training length or a like-for-like comparison. What it does show is that the gap in Table 2 is not a ceiling of the architecture.


W1 · Fewer steps reduce accuracy

True, but over most of the range the cost is small. K = 6 is within 0.3% of K = 12 at 63% of the latency, and K = 4 within 1.3% at about half.


We hope these answers resolve your concerns. If they do, we would be grateful if you would consider raising your score.

I appreciate your efforts to address my concerns. However, I believe my main concerns remain unresolved, as discussed below.


W1 · Why follow a consistent trajectory? With the path, stopping a 12-step run after 4 steps costs 1.1 to 1.3× its final error. Without it, 3.3 to 3.7×.

W2 · Is CosMean linked to accuracy? You are right that it is not. Across our eight main variants, its correlation with final MASE is weak (−0.58). What it does track closely is early-exit quality: −0.95 at 4 of 12 steps, −0.89 at 8. Eight variants are a small, structured sample, so we report this as an association in our setting, and Section 5.2 now uses CosMean only as a diagnostic of early exits. The Chronos-2 comparison stays as a probe of how depth is organized, since Chronos-2 was not trained for it.

[Response] I am still unclear about the practical need for early stopping. Since randomized-depth training already allows inference with fewer steps and achieves better accuracy than early stopping, what additional benefit does early stopping provide? This also raises questions about the practical significance of CosMean, given that its primary value is now presented as a diagnostic of early-exit quality.


W1. Accuracy and parameters versus TiRex. Appendix J.5 reports a longer-trained configuration of the same 38.8M architecture and objective.

[Response] The additional results are difficult to interpret as evidence of efficiency. The improved accuracy over TiRex requires longer training, multiple changes to the experimental setup, and approximately four times the inference compute. I am therefore still unconvinced of the practical advantage of FlowTSFM over TiRex in terms of the trade-off between forecasting accuracy and computational cost.


Overall, I have decided to maintain my original rating. The necessity of the proposed design is not sufficiently justified, and its practical benefits over existing approaches remain unclear. Given the emphasis on efficiency and flexible inference, I believe stronger evidence is needed to establish the significance of the proposed framework.

We thank the reviewer for engaging further. We believe two of the remaining concerns stem from a misunderstanding of our design, and we have run an additional experiment to address the third.


Concern 1 — Why early exit, if randomized-depth training is better?

Response. Randomized-depth training is what enables early exit; it is not an alternative to it.

There is an important distinction between the training mechanism and the inference capability. Randomized-depth training is what allows FlowTSFM to operate at different inference depths with a single checkpoint. Early exit is simply one way to exploit this capability when the inference budget is limited.

Model4 steps8 steps12 stepsCost of 4 vs 12 steps
FlowTSFM, randomized depth (one checkpoint)0.7560.7460.746+1.3% MASE, 2.5× fewer FLOPs
Fixed-depth model, truncated———+11–25% MASE

GIFT-Eval MASE (lower is better).

Using 4 rather than 12 steps therefore costs about 2.5× fewer FLOPs for a 1.3% increase in MASE, without retraining or maintaining separate models. The fixed-depth experiment is a controlled ablation showing why randomized-depth training matters; it is not the inference strategy we recommend. The distinction is particularly relevant when inference budgets vary across deployment settings, and we will make it explicit in the introduction.


Concern 2 — What is the significance of CosMean?

Response. CosMean captures a property of the refinement trajectory that final accuracy does not reveal.

CosMean measures the consistency of refinement updates across steps. We studied its relationship with early-exit quality across 10 trained variants that differ in training objective, parameter sharing and stabilization choices.

Quantity correlated with CosMean (10 variants)Spearman ρ\rhopp
Error increase when stopping at 4 instead of 12 steps−0.972×10⁻⁶
Full-depth MASE−0.530.12 (n.s.)
  • Variants with CosMean > 0.85 incur only 13–31% additional error at 4 steps.
  • Variants with CosMean < 0.4 incur 236–273%.
  • Yet all 10 variants reach similar full-depth MASE (0.74–0.80).

Final-depth accuracy alone therefore does not reveal whether a refinement trajectory supports early exit; CosMean does. We will include this analysis as a figure and present CosMean as a diagnostic of early-exit quality, not as a general predictor of forecasting accuracy.


Concern 3 — Efficiency versus TiRex (longer training, setup changes, ~4× inference compute)

Response. FlowTSFM's accuracy advantage over TiRex does not require additional inference passes.

We re-evaluated the J.5 checkpoint in a single pass, without test-time augmentation:

ModelParamsGIFT-Eval MASE ↓Latency (H200, ctx 2048, batch 1)
TiRex35.3M0.71642 ms
FlowTSFM J.5 (24 steps, single pass, no TTA)38.8M0.693≈ 22 ms
FlowTSFM, main budget (12 steps)38.8M0.734–0.74313 ms

The accuracy advantage remains without the ~4 inference passes used in the original J.5 evaluation. At similar parameter counts and without test-time augmentation, FlowTSFM achieves better accuracy at lower measured latency, despite its iterative refinement. TiRex was evaluated without its optional custom CUDA kernel, which failed to build in our environment; we will state this explicitly in the comparison table.

We acknowledge that J.5 uses a different, more computationally intensive training configuration. We report it as an additional scaling experiment, not as evidence of an inference-cost advantage, and we keep the main-budget results (0.734–0.743) alongside it.


Summary

ConcernWhat we add
Early exit vs randomized depthOne checkpoint serves 4–12 steps: +1.3% MASE for 2.5× fewer FLOPs
Significance of CosMeanPredicts early-exit loss (ρ=−0.97\rho = -0.97); final accuracy does not (ρ=−0.53\rho = -0.53, n.s.)
Efficiency vs TiRexSingle pass, no TTA: 0.693 vs 0.716, at ≈ 22 ms vs 42 ms

We hope these additional experiments clarify both the role of randomized-depth training in enabling flexible inference and FlowTSFM's accuracy advantage without additional inference passes. We would appreciate your reassessment in light of this evidence and would be happy to clarify any remaining concerns.

I understand that early exit can reduce inference costs, and the additional experiments support this benefit. However, I remain unconvinced of the practical value of making intermediate forecasts interpretable. Given that the proposed model does not achieve superior forecasting accuracy over existing approaches under the main evaluation setting, I find the motivation and significance of this design insufficiently justified.

I feel that my remaining concerns relate more to the fundamental motivation and direction of the paper, which may be difficult to fully address through further discussion. I appreciate the authors' efforts and additional clarifications, but I will maintain my original rating.

Reviewer_zyP7OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

FlowTSFM repeatedly applies a shared Transformer block and decodes forecast quantiles at each depth. A straight-path objective combines deep supervision with a Gaussian source anchor, while randomized-depth training enables selectable inference budgets.

Critical strengths
  1. The complete training/inference-depth grid is informative: fixed-depth checkpoints transfer poorly across budgets, while randomized training supports improvements within its training range. The reported degradation at 18 steps helps establish the limits of this flexibility.

  2. The finite-difference result in Proposition 1 is appropriately limited to fixed-budget trajectories. The appendix is explicit about what it does not establish, including endpoint agreement across budgets.

Critical weaknesses

The architectural and objective contributions are not sufficiently isolated. Table 2 uses external pretrained baselines rather than models trained under a common data and compute budget. The paper also lacks an untied encoder with the same objective and a stable recurrent control without the path loss. Two divergent loss-off runs support stabilization in the tested configuration, but do not establish the advantage of the prescribed geometry over deep supervision, endpoint distillation, or other regularizers.

The practical compute–accuracy tradeoff remains under-quantified. Increasing recurrent calls from 3 to 9 improves CRPS by about 2.6% on GIFT-Eval and 3.9% on TIME, after which performance largely plateaus and degrades beyond the training range. Equal optimizer steps do not match training compute across depth configurations. Without measured latency and peak memory, the results establish flexibility across trained budgets, but leave the practical savings uncertain.

Questions for authors

Could you compare an untied encoder and a stable recurrent alternative to the path objective under matched data and training compute? Retaining the existing intermediate supervision, residual scaling, and clipping would help isolate the contributions of weight sharing and path geometry.

Could you report accuracy–latency and peak-memory curves across inference budgets, particularly around the nine-step plateau? How does the randomized-depth model compare with depth-specific models when cumulative training compute is matched?

1 comment

Thank you for a precise review. Your two questions shaped most of the new Appendix J, and we answer them in order below.

In short

  • At matched data and FLOPs, removing the path term costs 2 to 7%, however training is stabilized.
  • The same objective also improves an untied encoder, and our 38.8M shared model matches the standard 214M encoder.
  • On training compute, the paper's K = 12 comparison was tilted against the randomized model. At K = 8, where we can match it exactly, the randomized model is better.

W1 and Q1 · Untied encoder and a stable control, at matched budget

We kept deep supervision, residual scaling and clipping fixed and changed one thing at a time. Every run sees the same data and schedule and applies the block 12 times per forecast, so FLOPs are equal. The table gives the change in MASE relative to our model (positive = worse) and the error when the 12-step model is stopped after 4 steps (ours: 0.96).

VariantParamsGIFT-EvalTIMEExit 4/12
12 untied blocks, our objective214M−2.1%+0.1%0.98
4 untied blocks, our objective87M+0.2%−0.5%–
2 untied blocks, our objective55M−1.4%−0.7%–
No path, ½ LR + 20k warm-up38.8M+4.1%+4.7%2.67
No path, latent stabilizer38.8M+3.3%+2.1%2.72
No path, QK-norm38.8M+5.5%+6.7%2.63
No path, learned residual gate38.8Mdiverges
Path term only38.8M+0.5%−0.7%0.87
Endpoint distillation38.8M0.0%+1.6%2.85
Consistency loss38.8M+0.9%+1.9%2.82
Curved path38.8M+0.6%+0.4%0.85 *
Standard encoder, final loss only214M+0.8%+1.7%1.19

* Extrapolated along its own curve.

Four things come out of it.

  1. Sharing is a small, fair trade. Untying all 12 blocks gains 2.1% on GIFT-Eval and nothing on TIME, for 5.5× the parameters. With 2 or 4 blocks the gaps are within noise or just at its edge.
  2. A stable control exists, but it is worse. Lower learning rate, a latent stabilizer and QK-norm all make the no-path run train, and all lose 2 to 7%. A residual gate does not even prevent the divergence.
  3. A prescribed path helps; a straight one is not required. Deep supervision alone diverges, while the path term alone matches the full objective. Distillation and a consistency loss keep the final accuracy but give no usable early exit. A curved path does as well as the straight one, and exits better at 4 steps.
  4. A common-budget baseline for Table 2. Trained on the final loss only, the standard 214M encoder is level with our 38.8M model on GIFT-Eval and 1.7% behind on TIME. With our objective, the same encoder gains 3.0% and 1.7%.

W2 and Q2 · Accuracy, latency and memory around the plateau

One randomized-depth checkpoint, H200, batch 1:

K36912
Latency3.6 ms5.1 ms7.1 ms8.1 ms
GIFT-Eval MASE0.7680.7480.7460.746
TIME MASE0.7350.7070.7010.701

Peak memory stays at 265 MB for every K, and Appendix J.2 lists every K from 1 to 15. K = 6 keeps 99.7% of the K = 12 accuracy at 63% of the latency and 55% of the FLOPs. Beyond about eight steps, more steps buy nothing, and the paper now says so.


Q2 · Matched training compute

We count block applications during training, the part of the cost that depends on K. The randomized model averages 9 per step, the K = 12 specialist 12, so at equal steps it had 25% fewer and still lands within 0.4% of the specialist. At K = 4 and 6 the imbalance goes the other way, and the paper now calls those equal-step comparisons, not equal-compute ones.

K = 8 can be matched exactly: the randomized checkpoint at 90k steps has as many block applications as the specialist at 100k (9 × 90k ≈ 8 × 100k).

K = 8, equal computeGIFT-Eval MASEGIFT-Eval CRPSTIME MASETIME CRPS
Randomized, 90k steps0.7440.5130.7040.587
Specialist, 100k steps0.7490.5110.7160.594

If these controls answer your concerns, we would be grateful if you would consider raising your score. And if another control would help, we will gladly run it in the time left.

Comments on the paper

2 comments
AuthorsOpenReview ↗

New controlled experiments

We thank the three reviewers for careful and constructive reviews. Read together, they ask three fair questions: does the objective matter once architecture and budget are fixed, what does a forecast really cost, and why should the intermediate forecasts follow a path at all? We answer each with new controlled experiments (Appendix J; main-text changes in blue).

In short

  • Without the path term, the shared loop diverges. Other fixes make it train, but cost 2–7% and give poor early exits.
  • The objective is not tied to weight sharing: it also improves an untied encoder, by 3.0% and 1.7%.
  • Sharing saves parameters and memory, not FLOPs. Reviewer Hdan was right, and the paper now says so.
  • The path need not be straight: a curved one works as well. The alternatives that do not prescribe a path lose the early exits.

All runs use the paper's code and the setting of the K = 12 row of Table 6, which our pipeline reproduces (0.743 / 0.506 on GIFT-Eval, 0.698 / 0.582 on TIME).


1 · The objective, isolated

Same data and schedule, 30k steps, and 12 block applications per forecast in every row, so the same FLOPs (untied rows spread them over 2, 4 or 12 distinct blocks). Normalized MASE, lower is better. The last column stops the 12-step model after 4 steps.

VariantParamsGIFT-EvalTIMECosMeanExit 4/12
Ours, shared block (2 seeds)38.8M0.7650.7340.920.96
Path term only38.8M0.7690.7290.920.87
No path term38.8Mdiverges
No path, ½ LR + 20k warm-up38.8M0.7960.7690.372.67
No path, latent stabilizer38.8M0.7900.7490.402.72
No path, QK-norm38.8M0.8070.7830.262.63
Endpoint distillation38.8M0.7650.7450.112.85
Consistency loss38.8M0.7720.7480.182.82
Curved path38.8M0.7700.7370.730.85 ¹
12 untied blocks, our objective214M0.7490.7340.880.98
12 untied blocks, final loss only214M0.7710.7470.611.19

¹ Stopped after 4 steps and extended to the end of its curved path, the same way the straight-path rows are extended along a straight line (1.31 if extended along a straight line instead).

Without the path term, the shared loop diverges. A lower learning rate, a latent stabilizer or QK-norm make it train, but each costs 2 to 7%, and their 4-step exits are about three times worse than ours.

Our 38.8M shared model matches the standard 214M encoder trained the usual way. Untying all 12 blocks under our objective gains 2.1% on GIFT-Eval and nothing on TIME, for 5.5× the parameters.

On noise: the reference seeds differ by 0.7% (GIFT-Eval) and 1.6% (TIME), and the other rows have one seed. Appendix J.1 gives paired intervals over evaluation configurations, which do not capture seed variance, so we treat gaps below about 1.5% as unresolved.


2 · What a forecast costs

One H200, batch 1, median of 20 calls. FlowTSFM rows use the randomized-depth checkpoint of Table 3 (the base model of Table 2, trained with a larger batch, scores 0.729 / 0.694 at K = 12).

LatencyGFLOPsGIFT-EvalTIME
FlowTSFM (randomized), K = 44.2 ms760.7560.719
FlowTSFM (randomized), K = 65.1 ms1040.7480.707
FlowTSFM (randomized), K = 128.1 ms1890.7460.701
Chronos-2 (119.5M)13.0 ms32–1300.6970.662
Moirai-2 (11.4M)6.9 ms3–140.7280.703
TiRex (35.3M)3.2× ours ²90.7160.683

² Relative to FlowTSFM at K = 12. TiRex needs its own CUDA kernel, so it was timed in a separate session; Appendix J.2 gives the source of every number.

At K = 12, sharing weights does not save FLOPs. It saves parameters (3× fewer than Chronos-2), keeps memory at 265 MB for every K, and leaves the choice of K to deployment: K = 6 keeps 99.7% of the accuracy at 63% of the latency.


3 · Why a path

Stopping a 12-step run after 4 steps costs 1.1–1.3× its final error with the path term, and 3.3–3.7× without it. Across our eight main variants, CosMean tracks this closely (Pearson −0.95) but not final accuracy (−0.58), so we now present it as a diagnostic of early exits.

A curved path does as well at the end and better at 4 steps, so the experiments do not show that the path must be straight. They do show that the alternatives without a prescribed path (distillation, a consistency loss between exits) lose the early exits.


Why this may interest the community

In our experiments, supervising decoded outputs along a path trains a looped transformer stably at a normal learning rate, with no change to the architecture. A single checkpoint serves every budget, within 2% of a dedicated model at each, and every step is a forecast one can read. Code, checkpoints and forecasts for all runs in Appendix J will be released.

AuthorsOpenReview ↗

Changes to the paper

Here is where each change sits in the revised PDF, and which review it answers. Everything new is in blue.

In the main text

WhereWhat changedAsked by
Abstract, IntroductionWhat else stabilizes training and what it costs; a curved path works too, so the path need not be straightall
ContributionsThe third one now rests on the controlled comparisons of Appendix J.1all
Section 5.1Parameters are not compute: sharing saves parameters and memory, not FLOPs at K = 12Hdan
Section 5.2Stabilized no-path runs, residual gate, distillation, consistency loss, curved path; CosMean as a diagnostic of early exitsall
Section 5.3, Table 3The randomized model at K = 1 to 18, next to every specialistHdan
Figure 1The true early exit of a single 12-step runHdan
LimitationsFLOPs, the cost of untying, seeds, and what our intervals measureall

New Appendix J

ContentsAsked by
J.1Matched ablations, with paired intervalsall
J.2FLOPs, latency and memory for K = 1 to 15 against three baselines; training computezyP7, Hdan
J.3One checkpoint at every budgetHdan
J.4True early exit, the extrapolation rule, and CosMeanK4BE, Hdan
J.5A longer-trained configuration of the same architectureK4BE, Hdan
J.6CKA between depth stepsK4BE

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.