acceptodds
Under review as a conference paper at ICLR 2027

What Can Time-Series Foundation Models Actually Do? Towards Capability-Level Evaluation

Abstract

Time-series foundation models (TSFMs) achieve strong zero-shot forecasting performance, but why do they work, and what abilities support their predictions? Existing evaluations compare forecasting accuracy across datasets or under selected forecasting conditions, leaving unclear which abilities models possess and when those abilities fail. We investigate these questions through six forecasting abilities: Structure Induction, History Retrieval, State Tracking, Uncertainty Modeling, Channel Modeling, and Exogenous Modeling. For each ability, we construct controlled experiments that vary the information available to a model or the difficulty of using that information while holding other conditions fixed. We evaluate 21 frozen TSFMs through their supported interfaces, examining whether they can infer temporal structure, retrieve historical information, track changing states, represent uncertainty, combine information across channels, and respond to external conditions. The results reveal ability differences that aggregate forecasting rankings do not capture. Separating periodic structure from residual variation improves forecasting, whereas supplying additional informative channels does not consistently reduce error. Stochastic generation also does not ensure that predictive distributions express the uncertainty present in the target. These findings connect forecasting performance to how models represent, combine, and use predictive information. They motivate evaluating TSFMs through explicit ability profiles and examining whether model designs translate available information into accurate predictions and calibrated uncertainty. All code is available at https://anonymous.4open.science/r/sortinghat-32B3/.

Primary area:
learning on time series and dynamical systems
Keywords:
Time-Series Foundation Models, Time-Series Forecasting, Benchmarking

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_gjR3OpenReview ↗
Rating
1
Confidence
4(+1)
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The paper proposes a different benchmarking view, assesing model performance by a set of capabilities instead of forecast error. It defines six tasks to represent these capabilities and tests several TSFMs on those.

Critical strengths

Benchmarking models against a set of skills the models should have, could be an interesting way of assessing the models performance without realying solely on forecasting error as it happens nowadays.

Critical weaknesses

The authors do not establish why or how these capabilites defined should be important or even correlated to the task of forecasting, which is the ultimate goal. Authors claim that there is not correlation between the performance in their tasks vs typical forecasting benchmarks (i.e. GiftEval). This claim should be supported by why, in their opinion, they should be correlated to begin with.

The authors do not explain how they evaluate these tasks they define in these models, being the core of the paper. Its not posible to understand, either replicate, these tasks' based on the paper body. The authors use the Appendix as an extension of the paper when it should be just complementary material and not in any way crucial to assess the soundness of the work.

The main contribution (the six main tasks) gets diluted between other things that do not add extra value. The dicussion on the results does not give any meaningful information since the previous part itself is not correctly and sufficienltly explained.

Too many formatting issues such as the reading flow not being one directional, there are figures without a caption (i.e. figure on page 6), demands the reader to go back and fourth between the main body and appendix to understand the fundamental parts of the work (section 3), important variables are not defined (i.e. C).

Questions for authors

Why and how are these tasks important for forecasting? Do they cover all, or most, of the capabilities a TSFM would need to do correct forecasting? i.e. are they sufficient?

Based on which rule do you decide the input length of the models? Is it related to the tasks? Why does not every model use their default context lenght?

Which parts of the paper main body could be dropped to ensure clarity and transparency in the most important part, the definition of the tasks?

Based on which decisions are the TSFMs in this work chosen? Given the wide amount of models tested, is strange to see very recent models like TimesFM3 but no Tirex2 or Chronos2 (base or small). Also given its relevance in the current literature, not adding Moirai[1,2] in favor of other models which have not gone through a reviewing process seems inconsistent (Tirex2 also applies here).

Do you plan to release this repository in a benchmarking style package such as GiftEval?

  1. Woo, G., et al. (2024). Unified Training of Universal Time Series Forecasting Transformers. Proceedings of the 41st International Conference on Machine Learning (ICML)

  2. Liu, X., et al. (2025). Moirai-MoE: Empowering Time Series Foundation Models with Sparse Mixture of Experts. Proceedings of the 42nd International Conference on Machine Learning (ICML)

10 comments
AuthorsOpenReview ↗

Response to Reviewer gjR3 [1/9]

We thank the reviewer for recognizing the value of evaluating models through the capabilities required for forecasting. We take the comments on presentation and organization seriously: the original manuscript did not sufficiently explain the logic behind the capability taxonomy, and too many key task details appeared only in the appendix. We will rewrite the relevant sections so that the main text provides a self-contained account of the motivation, task definitions, and evaluation procedures.

W1: The authors do not establish why or how these capabilites ... should be important or even correlated to the task of forecasting ... typical forecasting benchmarks (i.e. GiftEval) ...

Response: We thank the reviewer for raising the relationship between capability evaluation and forecasting performance. We may not have made the evaluation perspective sufficiently clear. Explaining this perspective requires distinguishing three levels: datasets, patterns, and capabilities.

(1) Our central contribution advances the level of abstraction in evaluation. We organize this progression as:

Dataset-level→Pattern-level→Capability-level\boxed{ \text{Dataset-level} \rightarrow \text{Pattern-level} \rightarrow \text{Capability-level} }

Evaluation levelBasis for organizing evaluationQuestion addressed
Dataset-levelDatasets, domains, and forecasting tasksWhere does a model forecast well?
Pattern-levelTrends, periodicity, and other temporal structuresWhich patterns does a model handle well?
Capability-levelInformation-processing operations required for forecastingWhat can a model do, and under what conditions are these operations reliable?

GIFT-Eval compares forecasting performance across datasets, domains, frequencies, and horizons. TIME advances to pattern-level evaluation by relating performance to structural time-series features. These approaches provide the context for our conceptual progression. GIFT-Eval, TIME

SortingHat makes the forecasting capability a model exhibits the unit of analysis: whether it can infer temporal structure, retrieve historical information, track changing states, combine information across channels, respond to external conditions, and represent uncertainty.

The key distinction is that patterns describe the structure present in a sequence, while capabilities describe the operations a model can perform.

Capabilities thus provide a level of abstraction for organizing forecasting behavior across datasets and patterns. Patterns serve as test settings for capabilities, while capabilities provide the basis for explaining forecasting behavior and organizing diagnostics.

(2) Pattern-level performance leaves capability-level questions unresolved. Consider two models that both accurately forecast a periodic sequence. This shows that they perform well on that periodic task, but leaves several questions open:

  • Do they rely on common periodic priors acquired during pretraining?
  • Do they primarily use corresponding observations from the previous cycle in the current history?
  • Do they infer the period, phase, and their evolution from the current input?
  • Can they still use these structures when multiple predictable components are combined?
  • Can they update their forecasts promptly when the period changes?

These sources of success and forecasting requirements may produce similar errors on ordinary tasks, yet lead to different responses when task conditions change.

For example:

  • Increasing the distance to relevant historical observations tests how far back the model can use evidence.
  • Changing periodic parameters tests adaptation to the current structure.
  • Changing the combination of individually predictable components tests performance under composition.
  • Introducing a state transition into the history tests whether an outdated relationship interferes with the current forecast.

“Handling periodic patterns well” therefore needs to be examined in terms of which information the model uses, which operations it performs, and under what conditions it fails.

These are sources of success that can be investigated through behavioral diagnostics. Claims about a particular internal mechanism require corresponding evidence.

AuthorsOpenReview ↗

Response to Reviewer gjR3 [2/9]

(3) Patterns and capabilities have a many-to-many relationship. This relationship directly explains why capabilities provide a higher level of abstraction.

One pattern can demand different capabilities. Periodic or seasonal sequences, for example, can be used to examine:

  • Structure Induction: identifying the period and its parameters.
  • History Retrieval: using relevant observations from earlier cycles.
  • State Tracking: updating forecasts after changes in period, amplitude, or phase.
  • Channel Modeling: using periodic information from other sequences.
  • Exogenous Conditioning: responding to known periodic future inputs.
  • Uncertainty Modeling: representing conditional distributions across states or forecast horizons.

A capability can also be tested through different patterns. State Tracking, for example, can be examined through changes in mean, autoregressive relationships, growth-to-plateau transitions, or frequency. These sequences have different shapes, but all require the model to update its assessment of the active dynamics from new observations.

Capability evaluation therefore organizes tasks with different shapes and origins around shared forecasting requirements. This is also why capability profiles can help explain behavior across tasks.

(4) The comparison with GIFT-Eval examines the relationship between these evaluation perspectives. We compare with GIFT-Eval to study how diagnostics of forecasting operations relate to conventional forecasting performance.

Both evaluations share a forecasting objective, so positive correlations are plausible. A model with strong overall forecasting performance may also be better at inferring structure, using historical evidence, or producing probabilistic forecasts.

Correlation asks:

Do the two measurements vary together across models?

It does not directly answer:

Does an aggregate score contain all the diagnostic information provided by the other measurement?

Aggregate forecasting performance combines tasks, conditions, and capability requirements. A model's strengths under some conditions can offset weaknesses under others. Models with similar aggregate forecasting performance can therefore exhibit different capability profiles and failure conditions.

Many capability questions also require specific controls:

  • Low forecasting error does not directly establish whether distant history was used.
  • Low error with joint inputs does not directly establish how much other channels contributed.
  • A forecast that changes with an external input does not establish whether the direction, magnitude, and timing of that response are correct.
  • Accurate point forecasts do not directly establish whether conditional intervals are calibrated.

These questions concern different evaluation targets, beyond alternative ways of ranking the same models.

(5) Correlation results support the argument after the evaluation perspectives have been explained. The correlations are best presented after the conceptual distinction, as evidence about the empirical relationship between the levels of evaluation.

We first explain the progression from datasets and patterns to forecasting operations. We then explain why these operations, which serve forecasting, should relate to aggregate forecasting performance. Finally, we show how differences between capabilities and failures under controlled conditions reveal information that aggregate rankings do not directly describe.

This organization lets the correlations support and explain the argument while keeping the conceptual contribution clear.

Our central question is therefore:

Given a model's overall forecasting performance, we still need to understand how it uses predictive information and under which requirements that use breaks down.

The value of capability evaluation does not depend on observing zero correlation or ranking reversals. It comes from the specificity of the questions being measured and the further analyses these diagnostics support.

AuthorsOpenReview ↗

Response to Reviewer gjR3 [3/9]

(6) Aggregate error, gains from additional information, and responses to conditions are distinct quantities. Our discussion of Channel Modeling and GIFT-Eval illustrates this distinction.

One evaluation may measure absolute forecasting error with joint inputs. Another may measure the gain from adding auxiliary channels while holding the target history and future fixed.

Both concern forecasting, but their measurement targets differ:

  • The first measures final forecasting performance.
  • The second measures how much additional information is converted into forecasting gains.

Similarly, Exogenous Conditioning can test whether a model responds correctly to different future conditions with the history fixed. It requires recovering the effect of the changed conditions, beyond achieving low average error.

The relationship between these scores and aggregate rankings should therefore be interpreted in light of their specific measurement targets. Capability diagnostics extend “how well does the model forecast?” to “which information contributes to the forecast, and which available information is not used effectively?”

We will revise the motivation as follows:

“We may not have made the distinction between evaluation perspectives sufficiently clear. Our contribution advances evaluation from the dataset level, through the pattern level, to the capability level. Dataset-level evaluation asks where a model forecasts well; pattern-level evaluation asks which temporal structures it handles well; capability-level evaluation asks which predictive operations it can perform and under what conditions those operations remain reliable.

This distinction matters because a pattern does not uniquely specify the operation supporting successful prediction. Accurate forecasts of seasonal sequences may rely on familiar priors, relevant historical observations, or relationships inferred from the current context. Conversely, the same capability, such as state tracking, can be examined through changes in means, trends, frequencies, or autoregressive dynamics. Capabilities therefore provide an abstraction for organizing forecasting behavior across patterns and datasets.

The comparison with GIFT-Eval examines the relationship between these perspectives. Positive correlations are consistent with their shared forecasting objective. Aggregate rankings, however, do not directly establish whether relevant history is used, additional channels provide a benefit, future conditions elicit the correct response, or predictive intervals are calibrated. Our controlled diagnostics make these questions explicit. We will revise the motivation to foreground this conceptual progression and present the correlations as evidence about its relationship to aggregate forecasting performance.”

Reviewer_EnEWOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces a capability-level evaluation framework for time series foundation models (TSFMs). The main idea is to construct controlled forecasting probes that vary the information available to the model or the task difficulty to examine whether TSFMs can infer temporal structure, retrieve historical information, track changing states, represent uncertainty, combine information across channels, and respond to external conditions. The authors also provide rank-based capability profiles to show differences between models beyond aggregate forecasting performance. Experiments cover 21 frozen TSFM versions, demonstrating that similar aggregate forecasting performance can coexist with different capability profiles and that additional informative channels do not consistently improve predictions.

Critical strengths

Strengths:

  • [S1] The idea of capability-level evaluation is clear and well-motivated. Evaluating specific forecasting capabilities beyond aggregate accuracy can help identify model weaknesses and guide future model development, and Fig. 2 gives concrete examples of differences in capability profiles that aggregate forecasting rankings do not capture.
  • [S2] The random-walk probe is thoughtfully designed. It uses a known conditional distribution as a reference, allowing a direct check of whether predictive uncertainty grows appropriately with the forecast horizon.
  • [S3] The experiments are comprehensive, covering a wide range of TSFMs and controlled settings. The additional real-data diagnostics and sensitivity analysis of the capability profiles are also well appreciated.
Critical weaknesses

Weaknesses:

  • [W1] The technical nuance and analytical depth of the paper are somewhat limited to me, which is my main concern. The work is mainly an evaluation framework and provides a broad diagnosis of existing TSFMs, but findings such as additional informative channels not consistently reducing error and stochastic generation not ensuring calibration remain fairly general. The explanations for these observations are largely based on cross-model comparisons, which lack deeper insights. Further evidence showing how these diagnoses could guide future model improvements would strengthen the paper's contribution.
  • [W2] In Sec. 4.3, the authors increase the context length to examine whether additional historical repetitions improve forecasting. However, as the authors acknowledge, this also changes the forecast origin and template phase, so the models are predicting different future targets. This makes it difficult to isolate the benefit of additional repetitions. It would be better to compare histories of different lengths while keeping the forecast origin and target future fixed.
  • [W3] It is unclear how much the capability rankings depend on the particular histories used for evaluation. The original uncertainty, channel, and conditioning suites use only three histories/seeds (Table 2), while Table 6 reports the resulting capability scores without showing how the rankings vary across histories. Although several supplementary experiments include confidence intervals, a focused analysis of whether these model orderings hold across different histories would make the comparisons more convincing.
Questions for authors

See weaknesses.

6 comments
AuthorsOpenReview ↗

Response to Reviewer EnEW [1/4]

We thank the reviewer for describing our capability-level evaluation as “clear and well-motivated” and the random-walk probes as “thoughtfully designed,” and for recognizing the experimental coverage, real-data diagnostics, and sensitivity analyses. We agree that the paper should make clearer how the diagnostics yield specific insights and guide model improvement.

The reviewer's Questions for Authors state “See weaknesses” and are addressed in W1–W3 below.

W1: The technical nuance and analytical depth of the paper are somewhat limited to me ... how these diagnoses could guide future model improvements ...

Response: Our work takes a further step in abstraction through Dataset-level evaluation → Pattern-level evaluation → Capability-level evaluation. GIFT-Eval compares forecasting performance across datasets and forecasting conditions. TIME advances to pattern-level evaluation by relating performance to structural time-series features. Our work takes the next step: even after identifying the patterns on which a model forecasts well, we still need to understand which operations support that performance. Moving from patterns to capabilities introduces a level of evaluation centered on the operations required for forecasting. It decomposes “performing well on seasonal sequences” into more specific questions:

Can the model identify the period? Can it retrieve distant observations at the same phase? Can it recognize that the period has changed? Can it handle periodicity and trend together? Can it represent the uncertainty that remains in the future?

From this perspective, we distinguish capabilities, patterns, and stress conditions. We use controlled forecasting probes, paired information interventions, and explicit references to turn these capability questions into measurable forecasting behavior.

These diagnostics characterize capability profiles, information use, and failure conditions beyond aggregate forecasting error. The existing periodic decomposition experiment further shows how capability diagnosis can guide model development without retraining. The channel and random-walk experiments identify specific gaps in how models combine information across channels and represent conditional uncertainty.

The framework also provides a common tool for studying model design. Training tasks, history representations, variable interactions, output mechanisms, and training data composition can each motivate concrete hypotheses about capability differences. The same diagnostics can then test which capabilities an improvement strengthens, where the gains occur, and whether new weaknesses emerge.

(1) Structural diagnostics have already led to an actionable forecasting improvement. Our composition experiments show that directly forecasting a composite sequence can produce substantially different errors from summing forecasts obtained separately from the true component histories. This observation motivated a further question: can we estimate a periodic template from history, subtract it from the input, forecast the residual with the frozen TSFM, and add the template back to the forecast?

When the period length is given, this procedure reduces the mean MSE of all five evaluated models without retraining: by 4.63%–12.37% at C=128C=128 and 2.06%–7.06% at C=512C=512. The gains depend on the residual structure. AR mixtures improve, whereas white-noise mixtures can deteriorate under short histories. The diagnosis therefore identifies both the difficulty of forecasting composite structures and the conditions under which explicit periodic decomposition helps or introduces additional error.

(2) The channel experiments reveal a gap between having more information and using it effectively. We hold the target history, forecast origin, and future target fixed while adding auxiliary channels. In the distributed-parent task at C=512C=512, increasing the total number of channels from 384 to 512 reduces the reference conditional variance by 41.37%, yet increases Toto-1's first-step NMSE by 22.04%.

This comparison separates the predictive information available in the input from the information used by the model. Additional channels reduce the remaining uncertainty about the target, but the model fails to combine their contributions into a more accurate forecast. It identifies information selection and cross-channel fusion as concrete directions for model development.

AuthorsOpenReview ↗

Response to Reviewer EnEW [2/4]

(3) The random-walk diagnostics identify where probabilistic forecasts fail. The conditional distribution of a random walk is known, allowing us to distinguish the forecast center, interval width, and coverage. At C=512,H=96C=512,H=96, Sundial's nominal 80% interval achieves only 40.1% coverage and has approximately 48.6% of the reference interval's width. These narrow intervals and low coverage indicate that the generated forecasts underrepresent conditional uncertainty. This gives a directly testable improvement target: predictive distributions should represent how uncertainty changes with noise scale and forecast horizon. Stochastic generation alone does not ensure calibrated forecasts.

Abrupt changes in forecast residuals at output-block boundaries, even when the underlying sequences are smooth, also provide a concrete starting point for examining consistency across output blocks. Together, these findings show that supported interfaces, effective information use, forecast continuity, and uncertainty calibration are forecasting requirements that need separate evaluation.

We will make these links between diagnostics, findings, and directions for improvement more explicit in the revision.

Further research questions

(1) CPM may strengthen global structure modeling, while NTP may strengthen the application of local rules. For a periodic sequence, multiple prediction positions within a contiguous gap share a period, phase, and amplitude. Identifying these shared factors can improve predictions at several positions simultaneously.

CPM may therefore encourage a more global representation of structure, beyond local correlations between adjacent positions. This could help models:

  • Maintain periodicity and phase over longer forecast horizons.
  • Recover regularities when local observations are missing.
  • Integrate common structure across multiple historical segments.
  • Reduce reliance on continuation based only on the most recent observations.

NTP also has a potential advantage worth examining. If a process follows a simple local recurrence, such as

xt+1=axt+b,x_{t+1}=ax_t+b,

learning the correct local update rule and applying it repeatedly may support reliable extrapolation over long horizons. A model that directly predicts a fixed future window may instead rely more heavily on global shapes and forecast horizons commonly encountered during training.

(2) Encoder and decoder differences may arise from how they organize historical evidence. A bidirectional encoder allows representations at different historical positions to inform one another. This organization may support global matching: repeated segments can be compared, observations from different cycles can jointly determine phase, and distant clues can reinforce one another across positions.

We therefore hypothesize that bidirectional history encoding may favor global pattern alignment, historical segment matching, and integration of evidence from multiple locations.

A causal decoder organizes representations in temporal order, with recent positions aggregating earlier information. This structure may more naturally form a representation of the current state and favor continuation based on the latest evidence.

We therefore hypothesize that causal decoders may favor local dynamic continuation and recent state updates.

The underlying research question concerns how historical representations are formed. The final position in a causal model can also access the full history. The distinction lies in whether earlier representations can be reinterpreted using later historical observations, and in the paths through which information reaches the forecast.

Encoder and decoder labels are therefore still coarse descriptions. The construction of forecast queries, interactions across layers, and training tasks may all affect the resulting capability strengths.

Our work establishes a basis for asking these questions, distinguishing failure causes, and testing whether proposed improvements are effective.

AuthorsOpenReview ↗

Response to Reviewer EnEW [3/4]

W2: ... It would be better to compare histories of different lengths while keeping the forecast origin and target future fixed.

Response: We thank the reviewer for this targeted suggestion. Following the suggestion, we have added a fixed-origin experiment to test whether longer histories containing more repetitions improve forecasting when the future target is held fixed.

For each random seed, we take suffixes of different lengths from the same sequence, keeping the forecast origin and future target unchanged while varying the available history length. The comparisons below use noise-free lag-copy sequences with a fixed forecast origin of 8449 and dependency distance d=129d=129. We request H=96H=96 forecast steps and report mean MSE over the first 8 steps, averaged across three random seeds.

ModelC=512C=512C=1024C=1024C=2048C=2048C=4096C=4096C=8192C=8192
Toto-20.0600800.0093630.0024590.0016340.001115
TimesFM-30.5432630.1567930.0265870.0064480.001897

Across the context lengths shown, both models' mean MSE decreases as history length increases. Increasing the context from C=512C=512 to C=8192C=8192 reduces mean MSE by 98.14% for Toto-2 and 99.65% for TimesFM-3.

These results support our original finding that longer histories containing more repetitions can improve forecasting, with the gains persisting when the forecast origin and future target are held fixed. The additional experiment directly addresses the reviewer's concern about changing prediction targets. We will include this comparison and the corresponding curves in the revision.

Reviewer_nUV5OpenReview ↗
Rating
3
Confidence
4
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

In this paper, the authors systematically evaluate a total of 21 Time Series Foundational Models to test six capabilities: Structure Induction, History Retrieval, State Tracking, Uncertainty Modeling, Channel Modeling, and Exogenous Conditioning. With controlled input changes and synthesized functions with known properties, the authors critically test them to uncover the strengths and weaknesses across these axes. The overall results show interesting differences in different model behaviors, that aggregate forecasting scores like known leaderboards in Time Series often fail to take into account.

Critical strengths
  1. This paper provides a clear framework for tests of TSFMs across six granular Time Series properties. Due to controlled input changes in known functions, it is easier to identify which information a model can use.

  2. Model capability variation across different models is quite interesting. The poor uncertainty calibration in random walk test is no doubt quite interesting, uncovering the flaws in learning styles of TSFMs. In fact, I believe these findings can pave the way for the next generation of TSFMs that can specifically target to handle all those capabilities to further boost the performance of TSFMs.

Critical weaknesses
  1. While the coverage of 21 model is quite good, channel modeling and exogenous capabilities evaluation only cover 4 and 5 models respectively. Models like Moirai 1.1, Moirai-MoE could potentially be good candidates to add in the pool. Additionally TabPFN-TS inclusion could increase the coverage of exogenous conditioning.

  2. Figure 2 does not have any caption. Please fix this.

  3. Please double check the radar chart in Figure 1, looks like it is contradictory with Table 6.

Questions for authors

Please see above. Additionally, for Section 4.3, it notes that the forecast origin changes with context length. Does the reported benefit of longer history also transfer if inputs end at the same time and predict same future window??

4 comments
AuthorsOpenReview ↗

Response to Reviewer nUV5 [1/2]

We thank the reviewer for recognizing the clarity of our six-capability framework and the value of the random-walk diagnostics for understanding uncertainty modeling in TSFMs.

W1: While the coverage of 21 model is quite good, channel modeling and exogenous capabilities evaluation only cover 4 and 5 models respectively. ...

Response: We thank the reviewer for their positive assessment of the framework and their specific suggestions on model coverage. Following these suggestions, we have evaluated Moirai-1.1-R-base, Moirai-MoE-1.0-R-base, TabPFN-TS, TiRex-2, standard Chronos-2, and Chronos-2 Small. These additions expand the model pool from 21 to 27 versions. We construct the expanded capability rankings using the original main probes, metrics, and scoring rules.

The number of models with complete results on the main probes for each capability is as follows:

CapabilityOriginal model countExpanded model count
Structure Induction2127
History Retrieval2127
State Tracking2127
Uncertainty Modeling2026
Channel Modeling510
Exogenous Conditioning49

Channel Modeling and Exogenous Conditioning, the two capabilities highlighted by the reviewer, each gain 5 comparable models. Their results are included in the corresponding expanded rankings, extending the comparisons to more model families and designs.

The added models' ranks within each expanded capability pool are reported below. Lower ranks indicate better relative performance; tied ranks are averaged.

Added modelStructure Induction / 27History Retrieval / 27State Tracking / 27Uncertainty Modeling / 26Channel Modeling / 10Exogenous Conditioning / 9
TiRex-24111216.55
Chronos-2935109.52
Chronos-2 Small104.547.553
Moirai-1.1-R-base252724226.56.5
Moirai-MoE-1.0-R-base14.52214179.59
TabPFN-TS5101713N/AN/A

We also evaluated TabPFN-TS on Exogenous Conditioning as suggested. It produced valid results on three main probes: one-compartment pharmacokinetics, two-compartment pharmacokinetics, and car-following dynamics. The remaining main probes require multiple target variables, which its current native interface does not support. We therefore retain the available probe results and mark its overall Exogenous Conditioning rank as N/A. Channel Modeling is likewise recorded as unsupported by its native interface.

We will update the model list, capability coverage, and rankings in the revision, with explicit model versions and supported tasks so that readers can directly check the scope of the expanded comparison.

W2: Figure 2 does not have any caption. Please fix this.

Response: Thank you for pointing out this formatting oversight. We will add a complete caption explaining the scores on each axis, model coverage, and the connection to the full results table:

“Figure 2: Capability profiles of five selected TSFMs. Each axis reports a rank-based score within the eligible model pool, with higher values indicating better relative performance. Unsupported capabilities remain unscored. The complete profiles and numerical scores are provided in Table 6.”

AuthorsOpenReview ↗

Response to Reviewer nUV5 [2/2]

W3: Please double check the radar chart in Figure 1, looks like it is contradictory with Table 6.

Response: Thank you for identifying this issue. We will regenerate the radar chart in Figure 1 directly from the values in Table 6, using consistent model names, axis order, and score mappings. Unsupported capabilities will remain explicitly missing. The caption will also distinguish empirical results from conceptual illustrations so that readers can directly check consistency between the figure and table.

Q1: ... Does the reported benefit of longer history also transfer if inputs end at the same time and predict same future window??

Response: We thank the reviewer for suggesting this important control. We have added experiments that fix the forecast origin and future window. For each random seed, we take histories of different lengths from the same sequence, all ending at the same time. The models predict exactly the same future target, with only the available history length changing.

The additional results show that longer histories can still improve forecasts when the target is fixed, with the benefit depending on the model and dependency distance. For example, in the noise-free setting with dependency distance d=129d=129 and a fixed forecast origin of 8449, we request H=96H=96 forecast steps and evaluate MSE over the first 8 steps. Increasing history length from C=512C=512 to 81928192 gives the following results, averaged over three random seeds:

  • Toto-2: MSE decreases from 0.0600800.060080 to 0.0011150.001115, a 98.14% reduction.
  • TimesFM-3: MSE decreases from 0.5432630.543263 to 0.0018970.001897, a 99.65% reduction.

These results directly show that models can benefit substantially from additional historical evidence while the forecast origin, future target, and target phase remain fixed.

The benefit is not monotonic under every condition. For example, in the same noise-free setting with d=25d=25, Toto-2's mean MSE increases from 0.0002850.000285 at C=2048C=2048 to 0.0009180.000918 at C=4096C=4096. The additional experiments therefore clarify our conclusion: more history can improve forecasting, while the ability to continue benefiting from that information depends on the task requirements and the model's use of history.

We will include the fixed-origin curves and paired results in the revision to show both the benefits of longer histories and the conditions under which they occur, using comparisons against the same future target.


Thanks for the added experiments. I'll hold my original positive score.

AuthorsOpenReview ↗

To Reviewer nUV5

Thank you for your thoughtful review and for your positive assessment of our work. We sincerely appreciate your support.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.