acceptodds
Under review as a conference paper at ICLR 2027

Can Time-Series Foundation Models Serve as Simulators for Process Control?

12 views

Abstract

Can a time-series foundation model (TSFM) predict what happens when we change an input, and help choose actions that keep an output near its target? We introduce a benchmark covering five synthetic process families and twelve evaluation factors for prescribed-input prediction, with executed control tests on selected conditions from all five families. For each proposed action, known dynamics generate two trajectories from the same state: one with the action and one with the input held unchanged. Their difference reveals the action response. We assess that response, then use each predictor to select inputs in an executed control test. We compare three frozen TSFMs with four locally fitted models. On clean first-order systems with input delays, TSFM predictions include transient reversed responses, timing errors, and amplitude errors. On integrating processes, their response-skill point estimates exceed the tested finite-impulse-response and multilayer-perceptron models, and all three also achieve lower tracking error. A locally fitted autoregressive model nevertheless has the lowest family-equal core tracking error among learned models. Better prediction does not guarantee better control: among local models with identical fitted artifacts reused across stages, orderings reverse in 10.2% of weighted within-task comparisons on clean first-order tasks, retaining ties in the denominator. The benchmark evaluates physical response fidelity together with executed control, exposing capabilities and failures that forecasting scores alone do not capture.

Primary area:
datasets and benchmarks
Keywords:
time series foundation model, process control, deep learning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_N4L3OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

Authors introduce ProSimCtrlBench, a synthetic benchmark evaluating whether frozen tsfms can accurately predict incremental physical responses to input changes and support closed-loop mpc. Over 5 process families and various evaluation factors, the authors compare these zero-shot tsfm models against locally fitted baseline models. They show that tsfm models suffer from physical fidelity flaws like transient reversed responses and delay errors.

Critical strengths
  • Testing prediction and closed-loop decision-making in a controlled way. Moreover, the paper simulates the active input plan and held-input trajectory from the exact same physical state and prevents shared model biases from cancelling out. 

  • Evaluating predictors statistically on queries and dynamically inside a shared MPC loop, show when prediction rankings reverse during execution is a smart and interesting design.
  • Covering physical behaviors (FOPDT, nonlinear mixing tank, nonlinear outflow tank, oscillatory SOPDT, and integrating processes) and varying factors eg dead time, gain, historical controller aggressiveness, and observation corruption makes the benchmark reasonably useful.
Critical weaknesses
  • One major limitation is that all of the 5 considered physical processes do not generalize to high dimensional and complex real world settings. In addition, all of them are siso which limits the benchmark if one wants to study and evaluate multi input multi output coupling and interaction. In these cases, it might be likely that neural network models perform better than simple linear systems.
  • Evaluation is limited to 1024 steps for history length and forecast horizon of 96 only. Having a sensitivity analysis for various values would be valuable.
Questions for authors

N/A

3 comments
AuthorsOpenReview ↗

Thank you for the careful assessment. (Part 1)

Thank you for recognizing the value of testing action responses and control. We added coupled MIMO tests and sensitivity analyses to clarify the benchmark's scope and how results depend on the settings. Part 1 addresses W1; Part 2 addresses W2 and presentation.

W1: Added MIMO tests extend coverage to coupled prediction and control.

We added a coupled four-tank system with two inputs and two outputs, testing both prediction and closed-loop control. The original SISO families study physical factors separately. This extension tests whether the same-state action/hold and shared MPC protocols also work with coupled inputs and outputs.

The system has two pump inputs, two observed lower-tank levels, and two hidden upper-tank states. Each input can affect both outputs. Two physical configurations (minimum-phase and nonminimum-phase), three operating points, and two histories per setting give 12 instances. All learned models receive input/output histories and candidate future inputs, but no hidden states or future outputs. ARX and the state-based MLP are selected and fitted only on the supplied history. The MLP uses observed output history and its own predictions for later steps, not hidden states. TSFMs jointly predict both outputs through their native interfaces with frozen weights. Model parameters remain fixed during control.

ModelEffect nRMSE ↓Tracking nIAE ↓
ARX0.00780.1402
State-based MLP0.21091.0107
TiRex-20.33270.3574
Chronos-20.29880.2842
TimesFM-30.09820.1777

The action effect is the output difference between changing and holding the input. Results are 12-instance means, normalized by each output's initial-history P95−P5 range (95th minus 5th percentile). All models share a 128-step MPC horizon, 256 executed steps, replanning every four steps, two-second sampling, and 32 candidates per round for two CEM search rounds.

Nonlinear dynamics change the model gaps, but do not guarantee a neural-model advantage. Your suggestion agrees with some original results: MLP predicts action responses more accurately than ARX on nonlinear mixing and outflow tanks (Sec. 5.1, Table 13). We further tested how nonlinear dynamics affect model gaps in the coupled system.

The first MIMO experiment used small input changes, with about 1.40% mean linearization response error. We therefore compared nonlinear and linearized four-tank systems using six new histories per configuration, all five models, and input-change amplitudes of 0.2, 0.6, and 1.1 V. The two versions share input histories and future plans but generate their own output histories and final states. Within each version, action/hold starts from the same full physical state; histories and starting states stay fixed across amplitudes. Each tested pair of input values has appeared in the history.

In a separate check starting both dynamics from the nonlinear system's final state, the relative RMS difference in action effects rises from about 2% to 11%–12% as amplitude increases. Normalization uses the corresponding nonlinear history's output-wise P95−P5, fixed across both versions and all amplitudes. These new histories and input ranges differ from the first experiment, so the two tables are not directly comparable.

Action-effect nRMSE at 1.1 V (lower is better):

ConfigurationVersionARXTimesFM-3Gap to ARX
Minimum-phaseLinearized0.000004630.07370.0737
Minimum-phaseNonlinear0.05500.09140.0364
Nonminimum-phaseLinearized0.000001080.09130.0913
Nonminimum-phaseNonlinear0.05240.11510.0626
  • At a fixed amplitude: neural models have smaller error gaps to ARX in the nonlinear version than in the linearized version. In the TimesFM-3 example shown here, its error increases, but ARX error increases more.
  • Within the nonlinear version: increasing amplitude makes these gaps larger in our tests.

ARX has the lowest mean effect and output-trajectory errors across all tested conditions in this paired experiment.

These controlled comparisons let researchers test how model gaps depend on process conditions, rather than rely on one overall ranking. Together with the original results where MLP performs better, they do not imply that linear models are always better.

The first extension tests coupled prediction and control; the second tests nonlinear prediction. Both cover two low-dimensional simulated configurations, not high-dimensional industrial systems. Seen inputs also do not guarantee future outputs within historical ranges. We will include all model/amplitude results and these limits in the paper.

AuthorsOpenReview ↗

Thank you for the careful assessment. (Part 2)

W2: Mean rankings are unchanged, but errors depend on history length.

In the tested FOPDT settings, longer histories improve some frozen TSFMs, while ARX already has very low error with shorter histories. We tested 12 new first-order-plus-dead-time (FOPDT) parameter groups, each with delays of 2 and 36 steps. From each 2048-step history, we take the final 256, 512, 1024, or 2048 steps. The final physical state, future input plans, and ground truth stay fixed, with a 96-step forecast horizon. ARX is fitted and selected only on the supplied history; TSFM weights stay frozen.

Long-delay action-effect RMSE, in original output units:

ModelHistory 256History 512History 1024History 2048
ARX0.017712.79×10⁻⁶2.93×10⁻⁶2.59×10⁻⁶
TiRex-23.99513.40281.94521.3392
Chronos-24.57774.43033.26772.5758
TimesFM-33.04991.20750.86670.9117

From 1024 to 2048 steps, TiRex-2 and Chronos-2 errors fall by about 31% and 21%. Exploratory 95% paired-bootstrap intervals over the 12 parameter groups for “error at 2048 minus error at 1024” are below zero for both models. TimesFM-3 improves from 256 to 1024, but its 1024-to-2048 interval includes zero. ARX has very low error at 512 steps. With the short delay, ARX already has very low error at 256 steps, and all three TSFMs improve from 256 to 1024.

These results show why model comparisons should specify the history budget: stable rankings can coexist with substantial changes in accuracy. The evidence is limited to noiseless linear FOPDT systems, whose structure matches ARX. Longer histories also contain more input changes; this experiment does not separate these effects or test their closed-loop benefits.

A longer forecast request does not remove the long-delay response errors. On a separate set of 24 FOPDT instances, we fix history length at 1024 and make separate 96-step and 192-step predictions. We compare both the same first 96 steps and each full window. Long-delay action-effect RMSE is:

ModelRequest 96: fullRequest 192: first 96Request 192: full
ARX2.02×10⁻⁶2.02×10⁻⁶1.43×10⁻⁶
TiRex-21.98552.06862.3385
Chronos-23.17863.33483.9314
TimesFM-31.04371.05281.2229

For the same first 96 steps, all three TSFMs' paired error-difference intervals include zero, and long-delay response errors remain. This does not prove that the forecast settings are equivalent. ARX reuses the same fitted model and gives identical predictions for these steps.

Over the full 192-step window, all three TSFMs have higher mean errors than over their full 96-step windows in the long-delay condition. Short-delay results are mixed: full-window 96/192-step errors are 0.4204/0.4304 for TiRex-2, 1.0221/0.9909 for Chronos-2, and 0.2197/0.3044 for TimesFM-3.

Full windows measure different time ranges; the same-first-96-step comparison tests whether the requested length changes predictions for those steps. Full-window differences cannot be attributed to request length alone, and the intervals for the first 96 steps do not apply to them. We did not test changes to the MPC planning horizon.

The benchmark can therefore test both context requirements and forecast-length effects, beyond a single default setting. We will include the full results and paired intervals, and clarify the shared 1024/96 setting.

Presentation

We will use the existing tank example throughout to connect two tasks: predicting how input changes affect tank levels, and choosing inputs to keep levels near a target. We will:

  1. Simplify Figure 1 to separate same-state action/hold branches from the control loop: candidate inputs → predictions → input selection and execution → new observations → replanning.
  2. Add a settings table for the original SISO protocol, MIMO extension, and sensitivity experiments, listing history length, forecast/planning horizon, execution length, and replanning interval.
  3. Put findings first in each results paragraph, followed by metrics and limits. Define PV, MV, and DV at first use and link them to the tank example. Move model settings, search budgets, and detailed statistics to the settings table or appendix.

Thank you for the additional experiments and clarifications. The MIMO extension and sensitivity analyses partially address my concerns. However, the evaluation remains limited to low-dimensional simulated systems, and the sensitivity analysis covers only noiseless FOPDT settings without varying the MPC planning horizon. These additions strengthen the paper, but do not sufficiently resolve my concerns about the benchmark's scope. I therefore maintain my original score.

Reviewer_GTHDOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces a benchmark to evaluate time series forecasting models when used as part of a control system. The benchmark is used to assess the performance of time series foundation models for the task of process control, and is used to compare them to specialized models. In particular, the benchmark consists of synthetic data, where samples obtained under a specific control action are paired to samples obtained under a hold-control reference on otherwise identical conditions. The paper evaluates both predictive performance, i.e., how well a model predicts the system evolution under paired control inputs, and control performance, i.e., how well the controlled system behavior aligns with the target one. The empirical results suggest that time series foundation models perform worse, in general, compared to specialized models.

Critical strengths
  • Assessing whether pre-trained time series foundation models can support applications that are different from their forecasting pre-training objective is an interesting research direction. This work provides some evidence that suggests time series foundation models may struggle with faithful modeling of covariate response, e.g., in presence of high response delay, compared to specialized models. However, it does not provide generalizable insights into why this happens, and the considered process classes appear too narrow to draw definitive conclusions.
Critical weaknesses
  • It is unclear what specific shortcoming of existing evaluation frameworks the proposed benchmark addresses. The authors aknowledge some existing benchmarks, and state that their paired action/hold design complements them. However it is unclear why evaluating on action/hold samples is relevant, or why it provides a useful perspective compared to simply evaluating on a collection of control objectives or covariate/response pairs from a target system. Furthermore, the proposed benchmark considers five synthetic SISO process classes, hence its overall applicability to evaluate models for real use cases remains unclear.
  • The paper doesn't seem to provide sufficiently generalizable insights that deepen the understanding of the shortcomings or strengths of time series foundation models when used for process control. The overall results show that these models perform worse than specialized models, which is somewhat expected. The exceptions, where foundation models outperform local models, regards cases that are potentially confounded by difference in input information, i.e., the integration case where only FIR, MLP, and RBF-SVR, are outperformed by foundation models. Ultimately, the relevance of the reported results, and how they can advance scientific understanding, is unclear.

Overall the paper sets itself an interesting goal, yet, when evaluated from a benchmarking perspective, the scope of the evaluation appears too narrow, while, when evaluated from the perspective of a study on foundation models, the paper does not seem to provide a contribution that sufficiently advances scientific understanding of the current gaps of time series foundation models when applied to process control.
The paper answers the following questions: (i) Do TSFM predictions preserve physical response properties? (ii) How effectively do they support closed-loop control? (iii) Do better predictions translate into better control? However, the benchmark lacks sufficienct breadth and the paper does not clarify what level of fidelity corresponds to preserving physical response properties. Additionally, it is unclear why or whether foundation models are expected to succeed in this task, hence if the results constitute a surprising failure. For the second question, the conclusion that foundation models can support model predictive control (MPC) cannot be falsified by the experimental setup, due to the lack of an operational definition of what "being able to support MPC" entails. It is true that a foundation model can be technically integrated in an MPC pipeline, yet they do not seem useful for such task, given results show that a simple ARX model is a better choice in most evaluated scenarios. Finally, for the third question, the diversity between predictive accuracy and control accuracy is already generally understood, and the paper does not seem to establish novel insights on what causes the mismatch in the considered settings.

Questions for authors
  • Could the authors clarify what the proposed benchmark allows evaluating that existing benchmarks could not, and why this is relevant for evaluating predictive control?
  • Could the authors explain the rationale for testing foundation models against models that do not use the same input information (i.e., FIR, MLP, RBF-SVR)?
  • Could the authors explain why (when DV is not measured) they chose to feed the foundation models a constant input for future realizations of the load variable, instead of omitting or masking it where the foundation model allows missing values?
  • Could the authors explain the choice of metrics used in Eq. 4 and Eq. 5? Are those standard? How is λTV\lambda_{TV} chosen in Eq. 5? What is the rationale for using a ratio of averages instead of an average of ratios in Eq. 4? Are the results robust to such choices?
3 comments
AuthorsOpenReview ↗

Thank you for the careful assessment. (Part 1)

Thank you for the careful assessment. Our benchmark provides controlled tests of action responses and their effects on control decisions. The added analyses examine background errors, cross-channel responses, and output-state information. Part 2 addresses the protocol questions.

1. Pairing separates background errors from action-response errors.

Section 2 and Appendix A.2 define da=ya−y0d_a=y_a-y_0, with both branches sharing the complete initial physical state and load sequence. Holding the input does not freeze the output: if the last output is 50, hold reaches 55, and action reaches 63, the action contributes 8, while the total output change is 13.

Each evaluation unit provides a history, candidate inputs, a same-state hold reference, and branch ground truth. An observed covariate/response pair alone lacks the unexecuted alternative's response at that state. We evaluate both absolute trajectories and incremental responses because shared prediction errors can cancel in the difference.

GIFT-Eval and fev-bench evaluate forecasting, including covariates; BOPTEST studies also compare prediction and MPC. Our paired references add controlled tests of background and action-effect errors at fixed states and candidate plans. Other simulators could implement this diagnostic; our contribution is the organized paired data and shared protocol. The source framework described in Section 3 and Appendix G supports configurable generation, predictor adapters, control execution, and inspection of responses and failures. Its reference workflow supports new evaluations with different interfaces and settings from the paper experiments; their complete numerical archives are not bundled.

2. Correct action effects alone do not guarantee correct action selection.

An overpredicted background tank level can make a controller correct a deviation that does not exist, even if action increments are correct. From six tasks, we analyze saved states from the original TiRex controller and from a controller using true action effects. At each fixed state, we use the same 69 plans and target and replace only the action effect: y~a=y^0+(ya−y0)\tilde y_a=\hat y_0+(y_a-y_0). Selection loss is the chosen plan's true cost minus the pool's minimum true cost.

Source of later statesOriginal predictionsWith true action effects
Original controller0.2620.221
Controller using true effects0.3000.538

Using truth for both components gives zero selection loss. Remaining hold errors can change costs or feasibility; correcting action effects has benefits that depend on the states and remaining errors. This shows why improvements should be checked against both absolute trajectories and action responses.

These are fixed-state tests, not closed-loop improvements. The original 10.2% reversals concern local models on clean FOPDT; this TiRex test examines a relevant factor, not those reversals case by case. Official scores still use each model's own action and hold predictions (Appendix A.2).

3. MIMO tests reveal errors in specific input-output channels.

The five SISO families and twelve factors organize controlled tests of response mechanisms and input/observation conditions. We extended the diagnostic to a coupled four-tank system: two pump inputs, two observed lower-tank levels, and hidden upper-tank states. Two parameter configurations (minimum/nonminimum phase), three operating points, and two histories give 12 instances from two physical configurations. All learned models receive both output/input histories and both candidate future inputs; TSFM weights remain frozen. The protocol uses a 128-step horizon, 2-second sampling, and 256-step execution with replanning every four steps.

For each pump, we separate effects on its corresponding output (direct) and the other output (cross). Tail-direction agreement is:

ModelDirectCross
TiRex48/4834/48
Chronos-247/4848/48
TimesFM-348/4848/48

Each denominator is 12 instances x 2 channels x 2 action signs; direction uses the last 16 steps. TiRex's incorrect cross-channel signs accompany near-zero predictions. Across four configuration-by-cross-channel cells, mean effect nRMSE ranges from 0.058 to 0.098 for TimesFM-3 and 0.270 to 0.454 for TiRex, scaled by each output's historical P95-P5 range. These tests locate weak cross-channel responses that an overall score can hide. Tail agreement is distinct from the transient FOPDT reversals in the paper. These probes extend coupling coverage in a locally near-linear regime (mean relative RMS linearization discrepancy 1.40%); strong-nonlinearity and industrial generalization remain open.

AuthorsOpenReview ↗

Thank you for the careful assessment. (Part 2)

4. Output-state information affects the comparison.

Input-only FIR/MLP/RBF-SVR test finite input-history-to-output mappings; ARX provides an output-feedback baseline. MLP and RBF-SVR can also use output history: our versions are specific representations, not the strongest forms of these model classes. PV as a fitting target differs from PV as a prediction input. These comparisons do not isolate architecture or pretraining effects; Section 5.2 accordingly limits the integration advantage to the tested representations.

To examine this difference, we refitted MLPs with and without one recursive PV lag on 12 preselected existing integrating test groups. Mean nIAE improves from 0.593 to 0.404. Input variation increases, but the penalized objective improves (0.614 to 0.429). This supports the value of PV state information on this subset. The new MIMO tests supply output/input histories to all learned models, retaining their native interfaces and postprocessing.

TSFMs' future-covariate interfaces enable a testable use: predicting candidate-input responses without fitting weights to each process. Forecasting pretraining provides a motivation to investigate this transfer, while faithful action responses require evaluation. The benchmark supports locating these capability boundaries and testing improvements. Deployment-cost savings remain unverified.

5. We measure response fidelity and control performance, not integration alone.

We will replace the broad "support MPC" claim in Section 7: under the tested protocol, TSFMs generate executable plans, but aggregate tracking is worse than ARX and some outflow tasks terminate. Figure 8 reports the weighted fraction of all 2,228 core tasks completing within each nIAE threshold; terminations fail every finite threshold. The integrating-process advantage over tested FIR/MLP remains representation-specific. Response fidelity is measured through direction, onset, amplitude, and trajectory errors. These measurements characterize performance without imposing a universal pass threshold or equating technical integration with effective control.

6. Hidden DV is omitted; measured DV without future preview is held constant.

Hold-last supplies a common nominal load scenario for candidate inputs without future information or another forecaster. Section 2 distinguishes hidden DV (no channel) from measured DV without future preview. Only the latter is held across the horizon and updated with new observations. Masking delegates the unknown future to each model's missing-data mechanism. We have not shown hold-last to be superior; results concern the stated convention.

7. Metric choices are explicit; robustness is partial, not universal.

PV skill compares output RMSE with persistence; MV skill compares incremental-response RMSE with zero action effect. Both are needed because common errors can cancel in the difference. nIAE measures tracking and nTV input variation; J=nIAE+λTVnTVJ=\mathrm{nIAE}+\lambda_{TV}\mathrm{nTV} expresses their trade-off. RMS error, integrated absolute error, and total variation are established quantities; our references, scales, aggregation, and weight are protocol choices. nTV is not a calibrated wear cost.

The value λTV=0.05\lambda_{TV}=0.05 was a shared empirical development setting used before the full experiments, not an optimized value. We reran MPC on eight preselected tasks with ARX, MLP, and TiRex at 0.01, 0.05, and 0.2. ARX retains the lowest mean nIAE and objective at all three values, but MLP and TiRex exchange order at 0.2. This rerun tests behavior as well as scoring; rescoring fixed trajectories alone does not.

The ratio of averages limits the influence of near-zero reference denominators. For positive reference errors BiB_i and aggregation weights wiw_i, it averages individual skills with weights proportional to wiBiw_iB_i, giving larger-reference cases more weight. Query-level mean-of-ratios and within-task ratio-of-means preserve the seven-model ordering on common valid clean-FOPDT support; this does not establish invariance across families.

Even the true-model controller is affected by the search budget. On four preselected MIMO instances, CEM increases from 2 to 8 iterations with 32 candidates and 8 elites per iteration. nIAE averages the two outputs after scaling each by its historical P95-P5 range:

Model2 iterations8 iterations
ARX0.1500.054
MLP1.2061.340
TiRex0.3710.228
Chronos-20.2050.130
TimesFM-30.1730.134
True-model reference0.1510.055

Predictors and initial predictions are fixed; later search and states can differ. Improvement with the true model shows that finite search can limit control even with accurate dynamics. The small TSFM order change is descriptive. Together, these tests locate response errors and examine how they and the planning budget affect decisions.

I thank the authors for their answers and for revising the manuscript. While this positions the contribution more clearly, and extends the coverage of the benchmark, I still believe my fundamental concerns remain. In particular:

  • even though the distinction with existing benchmarks is clearer, it remains unclear if the proposed benchmark design addresses specific issues of existing benchmarks and the relevance of the considered evaluation factors.
  • from a benchmarking contribution perspective, the proposed benchmark appears still narrow, as it is limited to a few synthetic settings and a few TSFM.
  • from a TSFM evaluation perspective, the conclusions do not seem to clearly advance scientific understanding of the shortcomings of TSFMs in predictive control scenarios. The overall result that they are lacking compared to simpler task-specific baselines is not particularly unexpected, and the work does not clearly establish deeper generalizable insights beyond a report of the empirical observations.
  • the specific connection between the benchmark and TSFM evaluation is also unclear. In particular, why should TSFM be evaluated in this specific way rather than on established control benchmarks.

On a technical note, regarding answer point (6): I see the authors decided to adopt the hold-last setting as an experimental choice for measured DV. Though, from my understanding, TSFMs assume that an unmasked signal is a true signal, hence I believe the choice is not a principled way of evaluating these models, as the provided input may be misleading for the model, with respect to its intended use.

Overall, given the above motivations, I am inclined to keep my score. My suggestion to the authors is to commit more decisively to one of the two core aspects of the work (i.e., the benchmark or the assessment of TSFMs for predictive control), in order to build a clearer and more substantial contribution around it. In the current form, I believe the work, while potentially interesting, does not establish a clear and substantial enough contribution to result in an acceptance recommendation.

Reviewer_Au2NOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces a benchmark for evaluating whether frozen time-series foundation models (TSFMs) can serve as process simulators for predictive control. The benchmark covers five single-input single-output (SISO) process families and incorporates several commonly used process-control metrics. The problem is interesting, although the practical motivation, benchmark scope, and interpretation of the findings would benefit from further development.

Critical strengths

The paper investigates an interesting question concerning the applicability of frozen TSFMs to predictive control.

The evaluation considers both prediction accuracy and control performance, providing a useful perspective beyond forecasting alone.

Comparisons with conventional modeling approaches help contextualize the performance of TSFMs.

Critical weaknesses
  1. The practical motivation for using frozen TSFMs needs a stronger justification.

When sufficiently rich historical data are available, it remains unclear why a frozen, general-purpose TSFM should be preferred over a model fitted specifically to the process. The reported results are also linked to this point, as the fitted ARX, FIR, MLP, and RBF-SVR models substantially outperform the evaluated TSFMs. Moreover, the long historical sequences required by TSFMs result in substantially larger model inputs. A clearer discussion of the intended use cases, benefits, and computational trade-offs would strengthen the motivation for the benchmark.

  1. The benchmark scope and methodological contribution are somewhat limited.

All five process families are SISO systems. It is not that clear whether the conclusions extend to more general process-control applications involving strongly coupled multi-input multi-output (MIMO) dynamics. While evaluation metrics used are appropriate, the benchmark’s novelty can be limited in terms of both the process scenarios covered and the evaluation methodology.

  1. The reported observations would benefit from deeper analysis.
Questions for authors

Under what practical conditions would a frozen TSFM offer an advantage over a process-specific model fitted using the same historical data? How does the computational cost compare when these models are used within MPC?

To what extent are the conclusions expected to carry over to strongly coupled MIMO processes, and what distinguishes the proposed benchmark from existing learning-based control evaluation practices?

Can additional ablation studies or mechanistic analyses explain the observed mismatch between prediction accuracy and closed-loop control performance?

2 comments
AuthorsOpenReview ↗

Thank you for the careful assessment. (Part 1)

Thank you for recognizing the research question, the value of evaluating prediction and control together, and the role of conventional baselines. We added MPC timing measurements, coupled MIMO tests, and analyses of action selection. Part 1 addresses the intended use and cost; Part 2 addresses scope, contribution, and the prediction/control mismatch.

Q1. Intended use cases, benefits, and computational trade-offs

We study whether one shared, frozen predictor can use process histories and candidate future inputs to predict action responses across processes, without fitting weights separately to each process. The intended use is to reuse the same model across processes, updating its history rather than its weights. When the same historical data are available for local fitting, our results do not show a general advantage in accuracy or total cost for the tested TSFMs. Locally fitted models, especially ARX, remain strong baselines. Frozen models need no weight fitting for each process; savings in model management have not been measured. The benchmark tests the response and control capabilities needed for this use, identifies gaps, and provides conditions for evaluating improvements.

We also examined the benefit of long histories. On noiseless first-order-plus-dead-time (FOPDT) tasks with long delays, increasing history from 1024 to 2048 steps reduces action-effect RMSE by about 31% for TiRex-2 and 21% for Chronos-2. TimesFM-3 shows no equally clear further improvement, while ARX already has very low error with 512 steps. More context can help some frozen models, but the benefit differs across models. This experiment tests prediction, not the closed-loop benefit of longer histories.

TSFMs require more online computation, but their measured planning times are well below the update interval in the tested setting. On two added four-tank instances, we measured complete MPC replanning after model loading and warm-up, using the same H800 GPU for all five models. The setting uses a 1024-step history, a 128-step planning horizon, and two cross-entropy method (CEM) search rounds with 32 candidates per round. Timing includes candidate generation, input processing and transfer, prediction, cost evaluation, and action selection.

ModelMedian complete replanning time (ms)
ARX43.20
State-based MLP49.39
Chronos-2151.80
TiRex-2189.50
TimesFM-3455.89

All three TSFM medians are below 6% of this experiment's 8-second update interval. For context, Darby and Nikolaou report typical process-industry MPC execution periods of 30-120 seconds (MPC: Current practice and challenges, 2012). On the tested hardware and problem size, planning times after warm-up fit within these slower process-control update intervals. These timings apply to the added setting; they do not verify field deployment or a worst-case time bound.

We report local model selection and fitting, loading and first calls, and steady-state planning separately. Over three repetitions per instance on a single CPU thread, full selection plus fitting takes about 2.85-2.88 seconds for ARX and 23.13-24.04 seconds for MLP. Fitting with an already selected configuration is much faster. These fitting costs are paid when preparing a model, not at every MPC update. They do not remove TSFMs' higher cost per update. Loading and first-call costs are also excluded from the steady-state table and reported separately. The comparison measures the computational trade-off; it does not show that frozen TSFMs are cheaper overall.

AuthorsOpenReview ↗

Thank you for the careful assessment. (Part 2)

Q2. MIMO scope and the benchmark's contribution

The benchmark provides controlled process conditions, paired action/hold ground truth, and a shared protocol for prediction and executed control. The five SISO families examine mechanisms such as delay, inertia, damping, integration, and nonlinearity; twelve factors organize variations in histories, candidate inputs, observations, and loads. Action and hold branches start from the same complete physical state, allowing us to measure background evolution and action effects separately. Holding the input does not freeze the output.

We added a coupled four-tank system with two pump inputs, two observed lower-tank levels, and two hidden upper-tank states. Two physical configurations (minimum/nonminimum phase), three operating points, and two histories give 12 instances. All learned models receive input/output histories and candidate future inputs, but no hidden states or future outputs. Process-specific models are selected and fitted only on the supplied history; TSFM weights remain frozen.

ModelEffect nRMSETracking nIAE
ARX0.00780.1402
State-based MLP0.21091.0107
TiRex-20.33270.3574
Chronos-20.29880.2842
TimesFM-30.09820.1777

Lower is better. These are 12-instance means, normalized by each output's initial-history P95-P5 range. All models share a 128-step planning horizon, 256-step execution, two-second sampling, and replanning every four steps, with 32 candidates per round and two CEM rounds.

The added tests also identify errors in specific input-output channels. TiRex predicts the correct response direction in 48/48 direct-channel cases and 34/48 cross-channel cases. Its incorrect directions occur with near-zero predicted effects, showing weak cross-channel responses with unreliable direction. Each count covers 12 instances x 2 channels x 2 action signs, using the last 16 steps. Correct direction alone does not imply accurate response timing or amplitude.

This extends the protocol to low-dimensional MIMO with cross-channel effects; rankings and other SISO findings may differ in strongly coupled, high-dimensional processes. Existing learning-based control studies also evaluate prediction and MPC together. Our benchmark adds organized paired ground truth and controlled tests of response errors. The open-source generator framework supports process configuration, predictor adapters, control execution, and response inspection, so researchers can extend the evaluation.

Q3. Mismatch between prediction accuracy and closed-loop control performance

We test how background prediction errors and a limited search budget affect control decisions. The paper's approximately 10.2% ranking reversals concern local models on clean FOPDT tasks. The added tests use different models and tasks; they do not explain each reversal in that statistic.

Even with correct action effects, predicting too high a tank level under hold can make a controller correct an error that does not exist. A shared error that cancels in the action/hold difference can therefore still affect candidate costs.

We analyze saved states from TiRex controllers on six tasks. At each fixed state, we use the same 69 plans and target, replacing predicted action effects with true effects while keeping the predicted hold trajectory. Selection loss is the chosen plan's true cost minus the lowest true cost in that pool.

Source of later statesOriginal predictionsWith true action effects
Original controller trajectories0.2620.221
Trajectories with true action effects0.3000.538

Using true hold trajectories and action effects gives zero selection loss. Remaining hold errors can change candidate costs or whether a plan satisfies constraints. The benefit of correcting action effects depends on the states and remaining errors. These tests compare decisions at fixed states, not the two controllers' closed-loop performance.

On four preselected MIMO instances, we also fixed the predictors and increased CEM from two to eight rounds, keeping 32 candidates per round. Mean nIAE falls from 0.150 to 0.054 for ARX and from 0.151 to 0.055 for the true-model reference; Chronos-2 changes from 0.205 to 0.130 and TimesFM-3 from 0.173 to 0.134. Improvement with the true model shows that finite search can limit control even when the process model is accurate.

These analyses support examining absolute trajectories, action responses, and planning budgets together. Paired ground truth and a shared control protocol allow these factors to be tested separately, beyond a single prediction score. We will include these results and distinguish the original SISO, MIMO, and supplementary settings.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Updated manuscript and thanks

Dear Reviewers,

Thank you all for your time and constructive feedback. Your comments have helped us clarify the benchmark’s contribution, strengthen the experimental analysis, and better explain the scope of our findings.

We have uploaded a revised manuscript, with revised and added text highlighted in blue. The main updates are:

  1. Clearer benchmark motivation and contribution. We clarify the intended use of frozen TSFMs and explain how paired action/hold trajectories help distinguish background prediction errors from errors in the response to an input change.

  2. Additional experiments and analysis. We add coupled four-tank evaluations, comparisons of linear and nonlinear dynamics, and history- and forecast-length studies. We also examine how background prediction errors and the search budget affect action selection and control performance.

  3. Computational cost and robustness. We report inference and MPC replanning times on the same GPU, with model loading and local-model fitting costs reported separately. Additional analyses examine output-history inputs, the input-variation weight, and metric aggregation.

Key findings are summarized in the main text, with detailed experiments and analyses in Appendix H. Existing section, figure, and table numbers have been preserved to make the revision easier to follow.

Thank you again for helping us improve the paper. We welcome further questions or comments during the discussion period.

Best regards,
The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.