- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
Authors introduce ProSimCtrlBench, a synthetic benchmark evaluating whether frozen tsfms can accurately predict incremental physical responses to input changes and support closed-loop mpc. Over 5 process families and various evaluation factors, the authors compare these zero-shot tsfm models against locally fitted baseline models. They show that tsfm models suffer from physical fidelity flaws like transient reversed responses and delay errors.
Critical strengths
- Testing prediction and closed-loop decision-making in a controlled way. Moreover, the paper simulates the active input plan and held-input trajectory from the exact same physical state and prevents shared model biases from cancelling out.
- Evaluating predictors statistically on queries and dynamically inside a shared MPC loop, show when prediction rankings reverse during execution is a smart and interesting design.
- Covering physical behaviors (FOPDT, nonlinear mixing tank, nonlinear outflow tank, oscillatory SOPDT, and integrating processes) and varying factors eg dead time, gain, historical controller aggressiveness, and observation corruption makes the benchmark reasonably useful.
Critical weaknesses
- One major limitation is that all of the 5 considered physical processes do not generalize to high dimensional and complex real world settings. In addition, all of them are siso which limits the benchmark if one wants to study and evaluate multi input multi output coupling and interaction. In these cases, it might be likely that neural network models perform better than simple linear systems.
- Evaluation is limited to 1024 steps for history length and forecast horizon of 96 only. Having a sensitivity analysis for various values would be valuable.
Questions for authors
N/A
3 comments
Thank you for the careful assessment. (Part 2)
W2: Mean rankings are unchanged, but errors depend on history length.
In the tested FOPDT settings, longer histories improve some frozen TSFMs, while ARX already has very low error with shorter histories. We tested 12 new first-order-plus-dead-time (FOPDT) parameter groups, each with delays of 2 and 36 steps. From each 2048-step history, we take the final 256, 512, 1024, or 2048 steps. The final physical state, future input plans, and ground truth stay fixed, with a 96-step forecast horizon. ARX is fitted and selected only on the supplied history; TSFM weights stay frozen.
Long-delay action-effect RMSE, in original output units:
| Model | History 256 | History 512 | History 1024 | History 2048 |
|---|---|---|---|---|
| ARX | 0.01771 | 2.79×10⁻⁶ | 2.93×10⁻⁶ | 2.59×10⁻⁶ |
| TiRex-2 | 3.9951 | 3.4028 | 1.9452 | 1.3392 |
| Chronos-2 | 4.5777 | 4.4303 | 3.2677 | 2.5758 |
| TimesFM-3 | 3.0499 | 1.2075 | 0.8667 | 0.9117 |
From 1024 to 2048 steps, TiRex-2 and Chronos-2 errors fall by about 31% and 21%. Exploratory 95% paired-bootstrap intervals over the 12 parameter groups for “error at 2048 minus error at 1024” are below zero for both models. TimesFM-3 improves from 256 to 1024, but its 1024-to-2048 interval includes zero. ARX has very low error at 512 steps. With the short delay, ARX already has very low error at 256 steps, and all three TSFMs improve from 256 to 1024.
These results show why model comparisons should specify the history budget: stable rankings can coexist with substantial changes in accuracy. The evidence is limited to noiseless linear FOPDT systems, whose structure matches ARX. Longer histories also contain more input changes; this experiment does not separate these effects or test their closed-loop benefits.
A longer forecast request does not remove the long-delay response errors. On a separate set of 24 FOPDT instances, we fix history length at 1024 and make separate 96-step and 192-step predictions. We compare both the same first 96 steps and each full window. Long-delay action-effect RMSE is:
| Model | Request 96: full | Request 192: first 96 | Request 192: full |
|---|---|---|---|
| ARX | 2.02×10⁻⁶ | 2.02×10⁻⁶ | 1.43×10⁻⁶ |
| TiRex-2 | 1.9855 | 2.0686 | 2.3385 |
| Chronos-2 | 3.1786 | 3.3348 | 3.9314 |
| TimesFM-3 | 1.0437 | 1.0528 | 1.2229 |
For the same first 96 steps, all three TSFMs' paired error-difference intervals include zero, and long-delay response errors remain. This does not prove that the forecast settings are equivalent. ARX reuses the same fitted model and gives identical predictions for these steps.
Over the full 192-step window, all three TSFMs have higher mean errors than over their full 96-step windows in the long-delay condition. Short-delay results are mixed: full-window 96/192-step errors are 0.4204/0.4304 for TiRex-2, 1.0221/0.9909 for Chronos-2, and 0.2197/0.3044 for TimesFM-3.
Full windows measure different time ranges; the same-first-96-step comparison tests whether the requested length changes predictions for those steps. Full-window differences cannot be attributed to request length alone, and the intervals for the first 96 steps do not apply to them. We did not test changes to the MPC planning horizon.
The benchmark can therefore test both context requirements and forecast-length effects, beyond a single default setting. We will include the full results and paired intervals, and clarify the shared 1024/96 setting.
Presentation
We will use the existing tank example throughout to connect two tasks: predicting how input changes affect tank levels, and choosing inputs to keep levels near a target. We will:
- Simplify Figure 1 to separate same-state action/hold branches from the control loop: candidate inputs → predictions → input selection and execution → new observations → replanning.
- Add a settings table for the original SISO protocol, MIMO extension, and sensitivity experiments, listing history length, forecast/planning horizon, execution length, and replanning interval.
- Put findings first in each results paragraph, followed by metrics and limits. Define PV, MV, and DV at first use and link them to the tank example. Move model settings, search budgets, and detailed statistics to the settings table or appendix.
Thank you for the additional experiments and clarifications. The MIMO extension and sensitivity analyses partially address my concerns. However, the evaluation remains limited to low-dimensional simulated systems, and the sensitivity analysis covers only noiseless FOPDT settings without varying the MPC planning horizon. These additions strengthen the paper, but do not sufficiently resolve my concerns about the benchmark's scope. I therefore maintain my original score.
Thank you for the careful assessment. (Part 1)
Thank you for recognizing the value of testing action responses and control. We added coupled MIMO tests and sensitivity analyses to clarify the benchmark's scope and how results depend on the settings. Part 1 addresses W1; Part 2 addresses W2 and presentation.
W1: Added MIMO tests extend coverage to coupled prediction and control.
We added a coupled four-tank system with two inputs and two outputs, testing both prediction and closed-loop control. The original SISO families study physical factors separately. This extension tests whether the same-state action/hold and shared MPC protocols also work with coupled inputs and outputs.
The system has two pump inputs, two observed lower-tank levels, and two hidden upper-tank states. Each input can affect both outputs. Two physical configurations (minimum-phase and nonminimum-phase), three operating points, and two histories per setting give 12 instances. All learned models receive input/output histories and candidate future inputs, but no hidden states or future outputs. ARX and the state-based MLP are selected and fitted only on the supplied history. The MLP uses observed output history and its own predictions for later steps, not hidden states. TSFMs jointly predict both outputs through their native interfaces with frozen weights. Model parameters remain fixed during control.
The action effect is the output difference between changing and holding the input. Results are 12-instance means, normalized by each output's initial-history P95−P5 range (95th minus 5th percentile). All models share a 128-step MPC horizon, 256 executed steps, replanning every four steps, two-second sampling, and 32 candidates per round for two CEM search rounds.
Nonlinear dynamics change the model gaps, but do not guarantee a neural-model advantage. Your suggestion agrees with some original results: MLP predicts action responses more accurately than ARX on nonlinear mixing and outflow tanks (Sec. 5.1, Table 13). We further tested how nonlinear dynamics affect model gaps in the coupled system.
The first MIMO experiment used small input changes, with about 1.40% mean linearization response error. We therefore compared nonlinear and linearized four-tank systems using six new histories per configuration, all five models, and input-change amplitudes of 0.2, 0.6, and 1.1 V. The two versions share input histories and future plans but generate their own output histories and final states. Within each version, action/hold starts from the same full physical state; histories and starting states stay fixed across amplitudes. Each tested pair of input values has appeared in the history.
In a separate check starting both dynamics from the nonlinear system's final state, the relative RMS difference in action effects rises from about 2% to 11%–12% as amplitude increases. Normalization uses the corresponding nonlinear history's output-wise P95−P5, fixed across both versions and all amplitudes. These new histories and input ranges differ from the first experiment, so the two tables are not directly comparable.
Action-effect nRMSE at 1.1 V (lower is better):
ARX has the lowest mean effect and output-trajectory errors across all tested conditions in this paired experiment.
These controlled comparisons let researchers test how model gaps depend on process conditions, rather than rely on one overall ranking. Together with the original results where MLP performs better, they do not imply that linear models are always better.
The first extension tests coupled prediction and control; the second tests nonlinear prediction. Both cover two low-dimensional simulated configurations, not high-dimensional industrial systems. Seen inputs also do not guarantee future outputs within historical ranges. We will include all model/amplitude results and these limits in the paper.