acceptodds
Under review as a conference paper at ICLR 2027

TEMPO-PDE: Trajectory-Aware Post-Training for PDE Foundation Models

Abstract

Partial differential equation foundation models (PDE-FMs) are neural predictors pretrained across physical systems to learn reusable representations of PDE dynamics. However, accurate early predictions do not ensure that an adapted model remains accurate over the full forecast horizon. Reference trajectories provide ground-truth fields at multiple future times, but the corresponding loss gradients can differ in magnitude or conflict with one another. To address this problem, we propose TEMPO-PDE, a post-training framework with two complementary components. (i) Trajectory-aware supervision scores future predictions using field-value errors, spectral discrepancies, and local spatial differences, and organizes these measures into primary and auxiliary objectives. (ii) Primary-guided gradient composition removes opposing auxiliary components and bounds the remaining auxiliary norm relative to the primary gradient. Both components update the existing predictor without adding an inference module. Within the registered main comparison, TEMPO achieves the lowest observed time-averaged and final-step errors on the five main PDE benchmarks.

Primary area:
applications to physical sciences (physics, chemistry, biology, etc.)
Keywords:
Machine Learning, Partial Differential Equations, Neural Operators, Temporal Objectives

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_CM5xOpenReview ↗
Rating
3(+1)
Confidence
2(−1)
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes TEMPO-PDE, a post-training framework for pretrained PDE predictors. It combines field-value, spectral, and spatial-difference errors and organizes trajectory supervision into primary and auxiliary objectives. For direct prediction, paired future lead times define the primary objective and the window endpoint provides auxiliary supervision. For recurrent prediction, rollout losses form the primary objective, supplemented by direct forecasts from the same observed state. The update clips the primary gradient, removes conflicting auxiliary components, and caps the auxiliary gradient norm relative to the primary gradient.

Experiments primarily use Poseidon-B and include matched training controls, component ablations, and extended-horizon forecasts. Additional studies examine score transfer to Flower-Tiny and an adapted TEMPO-AR formulation on pretrained DPOT-L. The reported results show improvements in the main Poseidon-B comparisons, while evidence for the additional benefit of the full combination is less consistent on other backbones.

Critical strengths
  1. The article presents a clear post-training method. This method utilizes the existing prediction interface and introduces additional scoring and gradient processing during the training process without the need for an inference module. The main goals and auxiliary goals, as well as their processing sequence, are sufficiently clear, making the core suggestions easy to implement. This is of practical value for adjusting existing partial differential equation predictors when the target system data is limited.

  2. The Poseidon-B comparison, component ablation, and long-term sequence evaluation indicate that the impact of training selection on the overall prediction quality may differ from the accuracy of the early steps. Additionally, these additional experiments also reveal limitations, such as in some cases, the configuration based on simple scoring performs better than the complete transfer version. Although these findings do not establish a complete solution that is always superior to other methods, they still provide valuable empirical information.

Critical weaknesses
  1. The numerical report and evaluation scope of the article are not clear. Appendix A reports unexplainable TEMPO results (24.04/42.44, while other configurations are 0.976/1.348), and increased Flower-Tiny prediction errors without verification against the five-task list. Moreover, this appendix summarizes FinalTest144 as eight victories in twelve task-indicator cells, but does not provide the complete results; while the main comparison uses Fresh128 tested during the development process. These omissions make it difficult to judge the evaluation scope and the independent support for the reported improvement effect.

  2. The methodological differences in the article are limited, and there is a lack of sufficient evidence to support a complete combination. The proposed update is closely related to MetaBalance and PEGrad, while Poseidon has achieved the reuse of time-point snapshots. The conditional projection and clipping selection of TEMPO are different, but their added value has not been fully verified. In Table 17, Score-H4 shows lower average errors than TEMPO-AR on all five DPOT-L tasks, although TEMPO-AR improved the final error on two tasks. The detailed Flower-Tiny study only tested the effect of score transfer. Overall, the evidence supports the scoring function more clearly than the innovation and added benefits of the complete combination.

Questions for authors
  1. Which tasks, configurations and evaluation schemes do the TEMPO and Flower-Tiny values mentioned in the opening paragraph of Appendix A correspond to? Please compare these data with Table 4-5 and the Flower-Tiny results in Appendix E.3. If there are tasks not included in the displayed subsets, please explain the relationship between this subset and the original evaluation scope.

  2. Can the author provide the complete FinalTest144 results for tasks and indicators, including the relevant comparison groups, and explain the relationship between the twelve task-indicator units and the five-task main evaluation?

  3. Compared to the similar primary-secondary alternatives such as PEGrad or target-referenced gradient rescaling, what specific advantages do the proposed gradient rules offer? This request aims to confirm the current contribution rather than extending the work to a broader future benchmark.

13 comments

Part 1 of 5. Question 1 and Weakness 1: the numbers in Appendix A and the evaluation scope

Thank you for the careful reading of the appendices. We answer Question 1 and Weakness 1 here, Question 2 in parts 2 and 3, and Question 3 with Weakness 2 in parts 4 and 5.

Q1. All three sets of values in Appendix A belong to one task, Allen-Cahn, which we abbreviate as ACE. ACE is the sixth task of the original registry, and its entry in the TEMPO-PDE row is direct fine-tuning. Tables 4 and 5 display the other five of the six registered tasks, and the revision will show all six. The table below identifies each set of values. All error values in our five parts are multiplied by 1000.

ValuesModel, input interface and evaluation set
Mean 24.04, Final 42.44Poseidon-B with direct fine-tuning on ACE, on Fresh128 with two seeds. This is the ACE entry of the TEMPO-PDE row [r57], for which the registered selection retained direct fine-tuning.
Mean 0.976, Final 1.348The original Flowers implementation with its four-frame input history, on Fresh128 with two seeds. This is the row "Flowers, 4-frame TEMPO target" [r15] of Table 4, whose ACE column is not displayed there. It has the lowest ACE Mean error of the 52 registered configurations evaluated on ACE. Its Mean and Final error are 3.6% and 26.4% below those of the same model with its native objective [r14], which has 1.012 and 1.831. It is 9.3% and 21.4% below its MSE-H4 control [r16], which has 1.076 and 1.714.
Mean from 3.977 to 4.013, Final from 4.998 to 5.351Single-state Flower-Tiny when its native loss is replaced by the TEMPO score, on Gate96 with three seeds. This is the ACE row of the study in Appendix E.3 and Table 3a, and this row is not displayed there. On the five displayed tasks the same study lowers the Mean error with the TEMPO score in all five.

With Poseidon-B, the direct TEMPO recipe lowers the error on ACE. We trained the submitted direct TEMPO recipe on ACE with two seeds and the same 1,280 updates. The recipe lowers the error of the fine-tuning entry in both seeds and on every trajectory, and this holds on all three evaluation sets of the table below. The table gives two-seed averages of the Mean and Final error. Its last row is the test set HeldOut128, which we have added. There the two direct TEMPO models are evaluated on the same 128 trajectories as the fine-tuning reference.

ACE with Poseidon-BDirect fine-tuningDirect TEMPOReduction
Fresh12824.043 / 42.44317.744 / 25.73126.2% / 39.4%
Gate9623.904 / 42.74517.324 / 25.13627.5% / 41.2%
HeldOut12824.838 / 44.13218.201 / 26.54226.7% / 39.9%

In relation to Tables 4 and 5 and Appendix E.3, the three sets of values are the ACE entries of three different rows. The first is the TEMPO-PDE row [r57], the second is the row [r15], and the third is the Flower-Tiny study of Appendix E.3. The four-frame Flowers model and the single-state Flower-Tiny differ in input interface and training budget, so 0.976 and 3.977 belong to two different models. The Flower-Tiny row among the external architectures in Table 1 and in part 3 is the standardized external baseline with its own training runs. It is again a different set of models from the Appendix E.3 study. The revision will rewrite the opening paragraph of Appendix A so that each value is given with its task, model, evaluation set and number of seeds, and it will add the six-task tables to the appendix.

W1. Two test sets support the main comparison independently of Fresh128.

The first is HeldOut128, which is new. It contains 128 trajectories per task that were not used during development, all from the official test files. We fixed the trajectory lists, checkpoints and analysis before inference, and we evaluated 204 models with three seeds for every matched configuration. On this set TEMPO is 5.2% to 42.9% below the stronger fine-tuning reference of each task in Mean error and 6.8% to 49.2% below it in Final error. It is also below all seven alternative training rules in all 42 recurrent comparisons of the general response.

The second is FinalTest144, whose complete results are in parts 2 and 3. On this one-shot test set TEMPO has the lowest Mean and Final error on all five main tasks among all methods of the comparison pool that was frozen before the evaluation.

In addition, results obtained on development data carry over to unused test data. All 72 comparisons of the new three-seed study that were registered on the development set Gate96 keep their sign when the same models are evaluated on HeldOut128.

Part 2 of 5. Question 2: the complete FinalTest144 results

Within the comparison pool that was frozen before the test, TEMPO has the lowest error in all ten cells of the five-task main evaluation. The twelve cells of Appendix A are these ten cells plus two cells for the sixth task, ACE. The count of 8 of 12 arises when comparators that were appended later are included.

In detail, the twelve task and metric cells are the six registered tasks times the two metrics, Mean and Final. The six tasks are the five main tasks and ACE, so the five-task main evaluation corresponds to the first ten cells. All cells are computed on FinalTest144, which contains 144 test trajectories per task. Each model was evaluated on it once, and every row reported here uses two training seeds.

The comparators come from two stages. The first stage is the original pool, which was frozen before any FinalTest144 value was accessed. It contains the 13 external architectures, the residual-head control [r52] and the Poseidon-B controls registered at that time, which are [r20], [r21], [r36], [r38] and [r51]. The table gives the TEMPO entry and the strongest comparator of this pool on the five main tasks, each as Mean and Final error. Here [r38] is the per-tensor TEMPO variant, [r51] is the recurrent fine-tuning reference, and [r36] is the matched direct fine-tuning for CE-RPUI.

TaskTEMPO entryStrongest comparator of the original pool
NS-PwC9.121 / 12.385[r38] 9.255 / 12.723
NS-SL17.373 / 28.083[r51] 17.947 / 28.761
NS-SVS20.181 / 34.668[r51] 20.850 / 35.921
FNS-KF9.573 / 13.638[r38] 9.890 / 14.242
CE-RPUI136.024 / 206.903[r36] 137.598 / 208.858

The TEMPO entry has the lowest error in both metrics on all five main tasks, which means in all ten cells of the five-task main evaluation. This pool includes the 13 external architectures, and the best of them has 1.6 to 5.5 times the Mean error of TEMPO on the five main tasks.

The second stage appended the baseline portfolio [r13] and the DPOT-L rows. The portfolio [r13] has 9.378 / 12.689 on NS-PwC, 18.546 / 30.106 on NS-SL, 22.456 / 38.263 on NS-SVS, 9.806 / 13.995 on FNS-KF and 134.090 / 197.861 on CE-RPUI. With these comparators included, TEMPO leads in 8 of 12 cells, which is the count quoted in Appendix A. These cells are both metrics on NS-PwC, NS-SL, NS-SVS and FNS-KF, each with a Holm-adjusted p of at most 0.0093 over the twelve hypotheses.

For CE-RPUI, the submitted recipe leads on a separate confirmation cohort, and the entry on FinalTest144 is older than this recipe. The entry was produced before the four-variable rerun noted in App. B.3, and the submitted Table 1 uses the recipe of this rerun. We compared the submitted recipe with the same comparators on the confirmation cohort. This cohort has 128 trajectories per task and was set aside from unused data before the recipe was fixed. It is separate from the sets named in the general response. The comparators include the appended portfolio [r13], which on the direct tasks is the three-query supervision of TEMPO with independent sampling and ordinary summation. On this cohort the submitted recipe has a Mean and Final error of 114.585 and 166.523 against 117.018 and 169.581 for the portfolio, with a Holm-adjusted p of 0.0002 for both metrics, and the TEMPO entries lead in all ten main-task cells. On HeldOut128 the submitted recipe reaches 114.441 and 167.101, which is 5.2% and 8.0% below the fine-tuning reference at 120.759 and 181.553.

For ACE, the entry is the direct fine-tuning retained by the selection rule, as part 1 explains, and it has 23.953 / 42.498. This entry therefore compares Poseidon-B fine-tuning with other models and does not involve TEMPO post-training of Poseidon-B. Its strongest comparator in the original pool is [r52], a frozen Flower-Tiny expert with a residual head, which has 8.132 / 13.216. The strongest appended comparator is TEMPO-AR on DPOT-L with 13.146 / 20.879. The direct TEMPO recipe lowers the error of the ACE entry by 26.2% and 39.4% on Fresh128 and by 26.7% and 39.9% on HeldOut128, as part 1 shows. Within DPOT-L, TEMPO-AR is the best of the three configurations on ACE on FinalTest144.

On DPOT-L, TEMPO-AR is below the native fine-tuning in all twelve FinalTest144 cells, by 3.4% to 14.2% in Mean error and by 2.4% to 14.1% in Final error.

Part 3 lists all rows of the comparison groups. The revision will include these tables and the comparator identities in the appendix, and every table caption will state the evaluation set, the number of seeds and the configuration identity.

Part 3 of 5. Question 2, continued: all comparison groups on FinalTest144

This part lists the rows of every comparison group on FinalTest144. Every cell gives the Mean error and then the Final error, with two seeds per row. A cell that reads "n/a" means that the row is not defined for that task, as in Tables 4 and 5 of the paper.

The first table gives the TEMPO entry and the same-backbone controls. On the five main tasks the TEMPO entry is below every control of the original pool, which are [r38] and the two fine-tuning columns. [r38] is the TEMPO variant that applies the projection per parameter tensor. Direct fine-tuning is the row [r20], and for CE-RPUI the row [r36]. Recurrent fine-tuning is the row [r21], and for NS-SL and NS-SVS the row [r51]. The last column is the appended baseline portfolio [r13] described in part 2. For ACE the TEMPO entry itself is direct fine-tuning, as part 1 explains.

TaskTEMPO entry[r38]Direct fine-tuningRecurrent fine-tuning[r13], appended
NS-PwC9.121 / 12.3859.255 / 12.72314.322 / 23.39817.608 / 27.7559.378 / 12.689
NS-SL17.373 / 28.083n/a29.642 / 59.39217.947 / 28.76118.546 / 30.106
NS-SVS20.181 / 34.668n/a30.518 / 67.90820.850 / 35.92122.456 / 38.263
FNS-KF9.573 / 13.6389.890 / 14.24217.216 / 27.89520.615 / 33.0749.806 / 13.995
CE-RPUI136.024 / 206.903n/a137.598 / 208.858152.613 / 235.021134.090 / 197.861
ACE23.953 / 42.498n/an/a31.783 / 53.840n/a

The second table gives the three DPOT-L configurations.

TaskNative fine-tuningScore-H4TEMPO-AR
NS-PwC52.007 / 80.51644.258 / 70.23544.629 / 69.131
NS-SL50.893 / 78.68241.320 / 65.27645.277 / 68.666
NS-SVS30.368 / 49.15426.419 / 45.22229.334 / 47.346
FNS-KF52.064 / 80.38744.077 / 69.57244.742 / 69.107
CE-RPUI314.159 / 400.839272.004 / 355.193280.004 / 357.860
ACE13.961 / 21.39113.598 / 22.17613.146 / 20.879

On the five main tasks every external architecture is above TEMPO in both metrics. The third table gives the 13 external architectures, and its last row is the strongest ACE comparator of part 2.

External architectureNS-PwCNS-SLNS-SVSFNS-KFCE-RPUIACE
FNO70.569 / 115.65982.723 / 142.96963.217 / 117.49973.514 / 121.103300.237 / 406.11470.557 / 107.090
F-FNO97.769 / 161.752120.152 / 193.866102.833 / 178.41697.794 / 161.443363.259 / 482.08771.308 / 114.007
U-NO227.084 / 366.666155.669 / 238.43280.078 / 141.623238.489 / 377.606676.307 / 793.335283.536 / 396.394
LSM261.878 / 420.592137.856 / 222.49353.791 / 96.419254.680 / 409.117629.200 / 767.757134.281 / 203.996
FactFormer484.147 / 711.294424.772 / 635.899320.859 / 483.947481.645 / 705.819598.883 / 651.48939.365 / 61.616
Transolver479.129 / 741.561440.147 / 664.103214.870 / 378.085464.018 / 741.148610.784 / 660.49373.084 / 109.511
UPT493.847 / 730.336456.948 / 670.325556.155 / 740.537492.956 / 727.289639.906 / 684.11119.851 / 34.680
DRIFT-Net567.465 / 715.621518.629 / 633.049461.515 / 575.420579.053 / 721.921658.596 / 710.066329.909 / 397.549
SGNO204.414 / 332.049257.379 / 409.923224.690 / 387.989220.866 / 351.835500.024 / 608.70558.680 / 105.194
Flower-Tiny38.677 / 64.34363.425 / 101.68840.592 / 71.73053.840 / 89.488283.252 / 391.77910.159 / 17.510
DPOT-M52.517 / 81.32048.024 / 75.26231.645 / 50.89652.970 / 82.308282.057 / 367.56516.079 / 24.563
CNO-FM160.917 / 259.258139.781 / 219.46274.052 / 129.108150.296 / 241.250542.671 / 700.652112.047 / 170.832
PDE-Transformer192.264 / 323.264251.062 / 401.525261.177 / 422.975209.680 / 357.267465.603 / 606.95761.589 / 101.599
Flower-Tiny expert with residual head [r52]n/an/an/an/an/a8.132 / 13.216
Reviewer_XAG2OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2(+1)
Soundness
2(+1)
Read review
Paper summary

This paper addresses the adaptation of pretrained PDE foundation models to a new target system for multi-step forecasting. Motivated by the distinction between short-term and full-horizon accuracy, the authors propose TEMPO-PDE, a post-training approach that supervises forecasts at multiple future time steps and combines the resulting losses through a primary-guided gradient composition. The authors distinguish two post-training settings, direct queries and recurrent rollout, and define a primary and an auxiliary loss for each. To avoid conflicts between the two gradients, they propose to remove the auxiliary component that opposes the primary direction whenever the two gradients conflict, and then to cap the auxiliary norm relative to the clipped primary gradient. Experiments on five PDE benchmarks with the Poseidon-B backbone report the lowest errors compared to external architectures, and mixed results compared to same-backbone fine-tuning controls.

Critical strengths

The framework is flexible and not tailored to a specific foundation model or downstream task. TEMPO-PDE can be instantiated for both direct-query and recurrent interfaces without architectural changes. Moreover, the primary and auxiliary objectives can be redefined for each interface while the composition rule stays the same, which makes the approach easy to extend when a system requires a specific physical metric. Additionally, the proposed per-step scoring function transfers to other backbones, Flower-Tiny (trained from scratch) and DPOT-L.

Critical weaknesses

The evaluation does not establish that the reported gains are robust or clearly attributable to the proposed method. The improvement over the strongest Poseidon-B controls is small (between 1% and 4%), and the follow-up results suggest that it is sensitive to the specific trained models. In the three-seed results reported in Appendix C.3, the full method is worse than the best reported control on NS-PwC, which makes the main-table advantage on that task dependent on the specific run reported. The same follow-up also shows that TEMPO-PDE achieves the lowest Mean error in only two of the six settings, while ablated variants achieve a lower Mean error in the others. In addition, the comparison grid (Tables 4 and 5) is incomplete: many controls are not clearly defined or are not evaluated on all tasks. For instance, the repaired Poseidon-B reference [r51] improves over the original baseline on NS-SL and NS-SVS, but it is not reported on the other tasks, and the nature of this repair is not explained. Taken together, these results show that the gains over same-backbone training are small, sensitive to the specific runs, and not consistently attributable to each proposed component. The benefits of TEMPO-PDE are therefore not yet fully supported by the current experiments.

Insufficient comparison with standard multi-step training alternatives. The main comparison does not include established trajectory-supervision approaches applied to the same backbone, such as the pushforward trick [1] or multi-step (unrolled) training losses with different weights for each rollout time step [2]. These approaches are natural alternatives to TEMPO-PDE's supervision, since they also train the model on several future time steps or on its own predictions. Related controls appear only in limited settings in the appendix. Notably, on DPOT-L, a plain multi-step loss using TEMPO-PDE's scores outperforms the full method in Mean error on all five tasks. Without these baselines in the main comparison on Poseidon-B, it is difficult to assess what TEMPO-PDE adds beyond standard multi-step training. Approaches that use different per-step weights, such as [2], should also be discussed in the Related Work section, which currently does not position TEMPO-PDE with respect to them.

[1] Brandstetter, J., Worrall, D., & Welling, M. (2022). Message passing neural PDE solvers. ICLR.

[2] Wu, T., Maruyama, T., & Leskovec, J. (2022). Learning to accelerate partial differential equations via latent global evolution. NeurIPS.

Questions for authors
  1. Can the authors provide a comparison over three fixed seeds (mean and standard deviation) on a test set for all main-text experiments?

  2. Using the same Poseidon-B backbone, how does the method compare on the recurrent tasks against the pushforward and multi-step baselines with different weights for each time step?

  3. In lines 308--309, the authors state: ``NS-PwC, FNS-KF, and CE-RPUI use direct queries, while NS-SL and NS-SVS use recurrent prediction.'' However, in Table 2 and Appendix C.3, NS-PwC is used in both the direct and recurrent settings. Can the authors report both settings for NS-PwC and explain how each setting is paired with each system?

  4. What is the repaired Poseidon-B reference [r51], and why is it reported only for NS-SL and NS-SVS? Could it be evaluated on all tasks?

11 comments

Part 1 of 4. Question 1: three fixed seeds with mean and standard deviation on a test set

Thank you for the detailed and constructive review. We answer Question 1 here, Weakness 1 in part 2, Weakness 2 and Question 2 in part 3, and Questions 3 and 4 in part 4. Part 4 also answers your remark on the comparison grid.

We now report all configurations of Table 2 with three fixed seeds on a new test set, and TEMPO has a lower error than the fine-tuning reference on every task. The set is HeldOut128, which contains 128 test trajectories per task that were not used during development. The general response describes its protocol. All error values in the four parts of this response are multiplied by 1000.

The two tables below give all configurations of Table 2 on HeldOut128 with seeds 9841, 9842 and 9843. Each cell gives mean ± standard deviation for the Mean error and then for the Final error.

Direct query

ConfigurationNS-PwCFNS-KFCE-RPUI
Full TEMPO9.182 ± 0.081 / 12.463 ± 0.1519.611 ± 0.101 / 13.756 ± 0.221114.441 ± 0.434 / 167.101 ± 0.455
Without auxiliary9.953 ± 0.798 / 14.765 ± 1.37010.156 ± 0.068 / 16.097 ± 0.247114.625 ± 2.041 / 169.227 ± 2.915
Without structure terms9.516 ± 0.126 / 12.793 ± 0.2279.897 ± 0.276 / 13.981 ± 0.477112.350 ± 0.616 / 164.838 ± 0.087
Ordinary sum9.178 ± 0.066 / 12.481 ± 0.1039.579 ± 0.222 / 13.514 ± 0.427115.664 ± 0.039 / 168.594 ± 0.218
Field-only primary10.492 ± 0.990 / 15.449 ± 1.61610.664 ± 0.062 / 16.564 ± 0.369112.678 ± 0.500 / 166.915 ± 0.735

Recurrent rollout

ConfigurationNS-PwCNS-SLNS-SVS
Full TEMPO16.038 ± 0.117 / 25.090 ± 0.20916.822 ± 0.299 / 26.868 ± 0.58219.450 ± 0.139 / 33.852 ± 0.402
Without auxiliary17.354 ± 0.703 / 27.848 ± 1.30317.743 ± 0.405 / 28.518 ± 0.77820.873 ± 0.362 / 36.636 ± 0.371
Without structure terms16.090 ± 0.148 / 24.500 ± 0.47817.360 ± 0.139 / 27.636 ± 0.33020.402 ± 0.411 / 35.489 ± 0.884
Ordinary sum16.862 ± 0.606 / 26.849 ± 1.39217.180 ± 0.039 / 27.487 ± 0.10820.863 ± 0.689 / 36.878 ± 1.266
Field-only primary17.953 ± 0.539 / 28.126 ± 0.94818.166 ± 0.300 / 29.000 ± 0.51221.310 ± 0.178 / 37.186 ± 0.391

TEMPO has a lower error than the fine-tuning reference of its deployed path on every task. In the table, TEMPO has three seeds and the reference has two seeds, and both are evaluated on the same HeldOut128 trajectories. The references are the rows [r20], [r36] and [r51] of Tables 4 and 5. The best controls of Table 1 on NS-PwC and FNS-KF, which are [r38] and [r13], are compared with TEMPO on the one-shot test set in part 2.

Task and pathFine-tuning referenceTEMPOReduction
NS-PwC, direct14.027 ± 0.219 / 22.672 ± 0.6979.182 ± 0.081 / 12.463 ± 0.15134.5% / 45.0%
FNS-KF, direct16.842 ± 0.923 / 27.094 ± 1.6719.611 ± 0.101 / 13.756 ± 0.22142.9% / 49.2%
CE-RPUI, direct120.759 ± 5.538 / 181.553 ± 9.221114.441 ± 0.434 / 167.101 ± 0.4555.2% / 8.0%
NS-SL, recurrent18.048 ± 0.524 / 28.834 ± 0.95616.822 ± 0.299 / 26.868 ± 0.5826.8% / 6.8%
NS-SVS, recurrent21.036 ± 1.059 / 36.719 ± 2.71419.450 ± 0.139 / 33.852 ± 0.4027.5% / 7.8%

The remaining main-text experiments also have seed statistics, and we list them here with their evaluation sets.

  1. Figure 1 uses seeds 9841, 9842 and 9843 on the long-horizon cohort of 96 trajectories with 247-step references. TEMPO has 332.294 ± 16.272 in Mean error and 737.796 ± 36.051 in Final error, while ordinary summation has 357.170 ± 10.307 and 799.659 ± 21.401. Averaged over the paired seeds, summation is 7.7% and 8.7% higher.
  2. Table 3a on Flower-Tiny uses three seeds on Gate96, and Table 15 gives the mean and standard deviation. The DPOT-L score study now also has three seeds on Fresh128. Across all six comparisons, the TEMPO score lowers the error by 9.2% to 15.7%, and every seed shows a reduction. On NS-PwC, for example, it lowers the Mean error from 49.663 ± 0.064 to 41.869 ± 0.362.
  3. The external architectures of Table 1 have two seeds and were evaluated on the one-shot test set FinalTest144. TEMPO is below all 13 of them on all five main tasks and in both metrics. For the best external architecture of each task, the Mean error is 38.677 ± 0.292 against 9.121 ± 0.029 for TEMPO on NS-PwC, 48.024 ± 0.194 against 17.373 ± 0.262 on NS-SL, 31.645 ± 0.013 against 20.181 ± 0.179 on NS-SVS, 52.970 ± 0.119 against 9.573 ± 0.126 on FNS-KF, and 282.057 ± 3.183 against 136.024 ± 5.767 on CE-RPUI. The ratio is between 1.6 and 5.5, which is far beyond the variation between seeds.

Part 2 of 4. Weakness 1: robustness of the gains and their attribution to the components

The gains are robust across seeds and across evaluation data, and each component contributes to them. Against standard alternatives on test data, the margins are larger than the 1% to 4% of Table 1.

The 1% to 4% in Table 1 is the margin over the best entry of the whole registered Poseidon-B control pool for each task, and for three of the five tasks this entry is a close relative of TEMPO. For NS-PwC it is [r38], a TEMPO variant that differs only in applying the projection per parameter tensor. For FNS-KF and CE-RPUI it is the baseline portfolio [r13], which on these tasks uses the same three-query supervision as TEMPO under independent sampling and ordinary summation. This supervision uses two sampled leads plus the endpoint, with the structure scores. For NS-SL and NS-SVS it is recurrent fine-tuning [r51]. Against standard alternatives on the same backbone and on test trajectories, the differences are larger. The Mean error of TEMPO is 5.2% to 42.9% below the fine-tuning references of part 1. In Mean error, pushforward, uniform multistep and linear multistep are 11.3% to 15.6%, 4.6% to 6.0% and 8.0% to 11.9% above TEMPO, as part 3 shows.

You raise the concern that the results are sensitive to the specific trained models. On HeldOut128 the gains are consistent across seeds, and two results show this. First, the seed variation is small. In the tables of part 1, the standard deviation of Full TEMPO over three seeds is 0.3% to 2.2% of its error in all six settings and for both metrics, which is well below the margins over the standard alternatives. Every individual TEMPO seed is also below every seed of the fine-tuning reference in all six settings and both metrics. Second, in the three recurrent settings TEMPO has the lower error in all 42 comparisons with the seven alternative training rules of item 1 of the general response, which are pushforward, three multistep weightings, PEGrad, MetaBalance and ordinary summation. Against pushforward, uniform multistep, linear multistep and PEGrad, every one of the three seeds favors TEMPO in all 24 cells.

For the three-seed result of Appendix C.3 on NS-PwC, the one-shot test set separates TEMPO from its best control. The best control is [r38], so this comparison is between two TEMPO variants that differ only in projection granularity. In Appendix C, Full TEMPO has 9.127 ± 0.090 with three seeds in Table 6, and [r38] has 9.075 with two seeds in Table 4. The two values differ by 0.6%, which is less than one seed standard deviation, so Appendix C does not separate the two granularities. On the one-shot test set FinalTest144, with two seeds per model, TEMPO has 9.121 against 9.255 in Mean error with a Holm-adjusted p of 0.0002, and 12.385 against 12.723 in Final error. The error of TEMPO is also stable across evaluation data. On the new test set HeldOut128, the three-seed Full TEMPO models give 9.182 ± 0.081, against 14.027 ± 0.219 for direct fine-tuning.

On FinalTest144, TEMPO is also below the best control [r13] on FNS-KF. It has 9.573 and 13.638 in Mean and Final error against 9.806 and 13.995, with two seeds per model and a Holm-adjusted p of at most 0.0093.

As for the lowest Mean error in two of six settings, on the new test set Full TEMPO has the lowest Mean error of the five Table 2 configurations in all three recurrent settings. On NS-SL and NS-SVS, the two tasks deployed with the recurrent path, it also has the lowest Final error. In the direct settings of NS-PwC and FNS-KF it is within 0.03% and 0.34% of the lowest Mean error as a paired-seed average, and both gaps are less than one seed standard deviation. On CE-RPUI, a shock-dominated task, Full TEMPO is 5.2% below fine-tuning in Mean error and 8.0% in Final error.

Finally, each component contributes to the gains, as removing or replacing each one on test data with three seeds shows. Removing the auxiliary objective raises the Mean and the Final error in all six settings, by up to 8.4% and 18.5%. Removing the structure terms raises the Mean error by 3.0% to 4.9% on NS-PwC and FNS-KF with direct queries and on NS-SL and NS-SVS with recurrent rollout. Replacing the primary-guided composition by ordinary summation raises the Mean error by 2.1% to 7.3% and the Final error by 2.3% to 9.0% in the three recurrent settings. On CE-RPUI the composition is also significantly better in both metrics.

Part 3 of 4. Weakness 2 and Question 2: comparison with standard multi-step training

TEMPO is more accurate than pushforward and than multi-step training with uniform, linear or harmonic step weights. This holds on the same Poseidon-B backbone and on test data, where TEMPO has the lowest Mean and Final error in all three recurrent settings.

We trained both kinds of baseline on Poseidon-B in the three recurrent settings, which are NS-PwC, NS-SL and NS-SVS. They use the same backbone, seeds, data, 1,280 updates and checkpoint rule as TEMPO. Pushforward [1] samples a rollout depth, advances without gradients, and scores the next step. The multi-step losses unroll the model and weight the field error of step k in proportion to 1, to k, or to 1/k, and we call these the uniform, linear and harmonic weights. The linear weighting is the Field-only primary configuration of Table 2. The harmonic weighting emphasizes early steps, in the spirit of the per-step weights of [2]. The tables give mean ± standard deviation over three seeds on HeldOut128.

Mean error

MethodNS-PwCNS-SLNS-SVS
TEMPO16.038 ± 0.11716.822 ± 0.29919.450 ± 0.139
Pushforward17.852 ± 0.08319.305 ± 0.30322.481 ± 0.874
Multistep, uniform16.993 ± 0.32117.640 ± 0.36120.340 ± 0.492
Multistep, linear17.953 ± 0.53918.166 ± 0.30021.310 ± 0.178
Multistep, harmonic16.581 ± 0.63916.974 ± 0.30019.588 ± 0.358

Final error

MethodNS-PwCNS-SLNS-SVS
TEMPO25.090 ± 0.20926.868 ± 0.58233.852 ± 0.402
Pushforward28.766 ± 0.57331.579 ± 0.79639.922 ± 1.446
Multistep, uniform26.384 ± 0.35528.358 ± 0.63035.315 ± 0.825
Multistep, linear28.126 ± 0.94829.000 ± 0.51237.186 ± 0.391
Multistep, harmonic25.509 ± 1.10327.495 ± 0.55634.220 ± 0.597

Relative to TEMPO, pushforward is 11.3% to 15.6% higher in Mean error and 14.6% to 17.9% higher in Final error. In Mean and Final error, uniform multistep is 4.6% to 6.0% and 4.3% to 5.5% higher, and linear multistep is 8.0% to 11.9% and 7.9% to 12.1% higher. For pushforward, uniform and linear weighting, all three seeds favor TEMPO in all 18 cells. For pushforward and linear weighting, all 12 cells are significant after Holm correction. The revision will add these baselines to the main comparison table.

Beyond multi-step training, the same data show that each component of TEMPO adds accuracy. Linear multistep training with the plain field loss is 2.1% to 3.5% above the same training with TEMPO's score in Mean error. The training with TEMPO's score is the row named Without auxiliary in part 1. That training is in turn 5.5% to 8.2% above full TEMPO in Mean error and 6.2% to 11.0% above it in Final error. This second gap comes from the direct-query auxiliary and the primary-guided composition, which full TEMPO adds.

The DPOT-L result in Table 17 is evidence for the method, and the difference between Score-H4 and TEMPO-AR has a specific cause. Score-H4 is the first component of TEMPO, trajectory-aware scoring, used on its own. With three seeds it lowers the error of DPOT-L by 9.2% to 15.7% in all six comparisons. TEMPO-AR is the full method adapted to DPOT-L. Both variants improve on the native multi-step fine-tuning of DPOT-L on all five main tasks and in both metrics, and on the one-shot test set FinalTest144 TEMPO-AR is below native fine-tuning in every task and metric. The cause is the DPOT interface, which determines the auxiliary that TEMPO-AR can use. DPOT has no lead-time-conditioned query, so TEMPO-AR had to take its auxiliary from the endpoint of the same rollout that already forms the primary objective. We tested this kind of auxiliary on Poseidon-B. With a same-rollout endpoint auxiliary, the Mean error is 5.3% to 14.5% higher and the Final error is 5.7% to 15.2% higher than with the direct-query auxiliary of TEMPO, and the difference is significant in all six cells. This supports the design of Eq. 8, in which the auxiliary supervises forecasts that the primary path does not produce.

For the related work, the revision will add the following text to Section 2. "Multi-step training unrolls the predictor and penalizes several future steps. The pushforward trick [1] detaches the earlier steps, and LE-PDE [2] assigns a separate weight to each rollout step. TEMPO differs in three respects. It pairs the rollout with direct forecasts from the same observed state, it scores fields with spectral and difference terms, and it combines the two gradients with the primary as reference instead of fixed scalar weights. We compare with pushforward and with uniform, linear and harmonic per-step weightings on the same backbone."

[1] Brandstetter et al., 2022.

[2] Wu et al., 2022.

Reviewer_VuEpOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes TEMPO-PDE, a training method that improves how pretrained PDE foundation models predict physical systems over time. It uses information from multiple future time steps, spatial features, and gradient control to reduce prediction errors and improve long-term forecasting. TEMPO perform better in the main comparison across five PDE benchmarks, but additional experiments show that simpler training methods sometimes perform better.

Critical strengths
  • TEMPO achieves the lowest average and final prediction errors across all five benchmarks in Table 1. For example, on NS-SVS, it reduces mean error from 20.47 to 19.77, showing improvements across different physical systems.
  • Instead of optimizing only the next prediction step, TEMPO learns from multiple future states. For example, when predicting fluid movement, the method considers how accurately the model predicts later flow patterns, which is useful because small early errors can grow over time.
  • TEMPO modifies the training process without changing the underlying model architecture. Once training is complete, the adapted model makes predictions normally, without requiring extra gradient calculations or additional prediction modules.
Critical weaknesses
  • TEMPO improves prediction accuracy by only 1–3.4% compared to the baselines in the main experiments. For example, on NS-PwC, the error decreases from 9.075 to 8.974, which is a relatively small improvement considering the additional training complexity.

  • The proposed gradient combination method does not consistently outperform ordinary loss summation. For example, on direct-query NS-PwC and FNS-KF, simply adding the losses produces slightly better results. This raises questions about whether the more complicated gradient control is always necessary.

  • The paper introduces spectral and spatial-difference losses to better capture physical structures. However, removing these losses improves mean prediction accuracy by about 1.57% on CE-RPUI. This suggests that the proposed spatial losses may not benefit all PDE problems.

  • The paper shows improved predictions over 247 time steps, but the advantage changes across experimental settings. For example, one comparison reports approximately 7.4% improvement, while another sampling realization reduces the improvement to around 1%. This makes it difficult to determine how consistently TEMPO improves long-term forecasts.

Questions for authors
  • The improvements over the strongest baselines are relatively small, around 1–3.4%. Can authors justify the additional training complexity and computational cost of TEMPO?
  • Ordinary loss summation performs slightly better than TEMPO on some direct-query tasks, such as NS-PwC and FNS-KF. Why is the proposed gradient combination method necessary if a simpler loss combination can sometimes achieve better results?
  • Removing spatial losses improves prediction accuracy on CE-RPUI. Why do the proposed spectral and spatial-difference losses help some PDE problems but reduce accuracy on others? Could the authors explain when these losses should be used?
  • Long-term prediction improvements vary across experimental settings, and most experiments focus on Poseidon-B. Can the authors provide more experiments across different foundation models, random seeds, and longer prediction horizons to demonstrate that TEMPO's improvements are consistent and reliable?
9 comments

Part 1 of 4. Weakness 1 and Question 1: the size of the gains and the training cost

Thank you for the review and the clear questions. Each of the four weaknesses you list matches one of your four questions, so we answer each pair together in one part. All error values in our four parts are multiplied by 1000.

The gain over fine-tuning, pushforward training and multistep training is larger than the 1% to 3.4% of Table 1, and TEMPO adds no cost at inference and only a few GPU hours of training per task. Against fine-tuning of the same backbone, TEMPO lowers the Mean error on test data by 5% to 43%.

The 1% to 3.4% of Table 1 is the margin over the best of a pool of controls that contains close relatives of TEMPO. The label "best registered Poseidon-B control" in that table means the lowest value in the whole pool of registered same-backbone controls for each task. In your NS-PwC example, the value 9.075 belongs to the control [r38], which is a TEMPO variant that differs only in projection granularity. A comparison with fine-tuning instead measures the effect of the whole method, and Table 4 of the submission already contains it for the two direct fluid tasks. Direct fine-tuning [r20] has 13.97 on NS-PwC and 17.21 on FNS-KF, while TEMPO has 8.974 and 9.507, which is 36% and 45% lower.

On held-out test data TEMPO has a lower error than every alternative in the table below. We measured the gains on 128 held-out test trajectories per task, with three seeds for TEMPO and for every matched training rule. The table shows how much lower TEMPO's error is, as a fraction of the error of each alternative. For the three training rules, the general response gives the same comparisons the other way round, as the amount by which the error of each alternative exceeds TEMPO's error.

Alternative on the same backboneMean errorFinal error
Fine-tuning, the stronger of direct and recurrent, two seeds34.5% NS-PwC, 42.9% FNS-KF, 6.8% NS-SL, 7.5% NS-SVS, 5.2% CE-RPUI45.0%, 49.2%, 6.8%, 7.8%, 8.0%
Pushforward training10.2% to 13.4%12.8% to 15.2%
Multistep training, linear weights7.4% to 10.6%7.4% to 10.7%
Multistep training, uniform weights4.3% to 5.6%4.1% to 5.2%

The last three rows cover the three recurrent tasks, and TEMPO is better in every seed in all 18 cells.

TEMPO adds no cost at inference, because it changes only training. The deployed model is the same network with the same inference path. The measured latency of an 11-step forecast on NS-PwC is 729.4 ms with TEMPO and 729.4 ms with fine-tuning, as Table 18 reports. A deployed surrogate spends its compute at inference, so the accuracy gain adds nothing to the cost of using the model.

In training, TEMPO needs a few GPU hours per task, and this cost is paid once. The table gives the training time on one RTX 5090 with 1,280 updates on NS-PwC.

Prediction pathField loss onlyTEMPO
Direct query0.35 h0.72 h
Recurrent rollout1.21 h2.16 h

In return for this training time, the Mean error on test trajectories is 5% to 43% below fine-tuning and 4% to 13% below pushforward and uniform or linear multistep training, with no added inference module, parameter or latency.

As for complexity, the update rule adds only one backward pass and three vector operations per step, which are the clip, the conditional projection and the cap. It adds no architecture component and keeps no state across updates, and the same code serves both prediction interfaces.

Part 2 of 4. Weakness 2 and Question 2: why the gradient composition is needed when ordinary summation is sometimes as good

The composition is needed for recurrent training, where it is more accurate than ordinary summation in every setting. On the two direct tasks that you point to, the two rules are statistically equal on the 11-step window. Over 247 steps on NS-PwC, however, TEMPO has the lower error in the three-seed study of the paper. The composition adds little cost.

We repeated the comparison between TEMPO and ordinary summation on held-out test trajectories with three seeds per method. Ordinary summation keeps TEMPO's targets and scores and replaces only the update rule, so the comparison isolates the composition. The table gives the Mean and Final error on HeldOut128, and its last row uses the long-horizon cohort of Figure 1. The last column shows by how much the error of summation exceeds the error of TEMPO, averaged over the paired seeds.

SettingTEMPOOrdinary summationSummation relative to TEMPO
NS-PwC, recurrent16.038 / 25.09016.862 / 26.8495.16% / 7.04% higher
NS-SL, recurrent16.822 / 26.86817.180 / 27.4872.15% / 2.33% higher
NS-SVS, recurrent19.450 / 33.85220.863 / 36.8787.28% / 8.97% higher
CE-RPUI, direct114.441 / 167.101115.664 / 168.5941.07% / 0.89% higher
NS-PwC, direct, 11 steps9.182 / 12.4639.178 / 12.481equal within 0.15%
NS-PwC, direct, 247 steps332.294 / 737.796357.170 / 799.6597.71% / 8.65% higher

The composition has its clearest effect in recurrent training and over the long forecast of Figure 1. In recurrent training TEMPO is better in all six cells. The difference from summation is significant after Holm correction for both metrics on recurrent NS-SVS and on direct CE-RPUI. Direct NS-PwC is the case you point to, and there the two rules are equal on the 11-step window. Over 247 steps, however, summation is on average 7.7% worse in Mean error and 8.7% worse in Final error in the three-seed study of Table 14. This is the comparison that Figure 1 shows.

FNS-KF is the second task you name, and on its 11-step window the two rules are statistically indistinguishable. They are within 0.34% of each other in Mean error and within 1.78% in Final error, and neither difference is significant after correction.

Summation is not sufficient because it lets the raw gradient norms decide how much the auxiliary objective contributes. The gradient of the summed loss, which is the primary loss plus the weighted auxiliary loss, is clipped as a whole, and the raw norms change with the rollout length and over training. The contribution of the auxiliary objective therefore changes with them. TEMPO instead fixes the scale by the primary gradient. It clips the primary gradient first, bounds the auxiliary gradient relative to it, and removes an auxiliary component that opposes the primary gradient. As a result, in all logged updates the auxiliary part of TEMPO does not reduce the first-order decrease of the primary objective in gradient space, up to floating-point precision.

Overall, ordinary summation is 2% to 9% worse than TEMPO in recurrent post-training and about 1% worse on CE-RPUI. On the 11-step direct window of NS-PwC and FNS-KF the two rules are statistically equal, but over the 247-step forecast of Table 14 on NS-PwC summation is 8% to 9% worse. The composition adds about 20 minutes of training per task relative to ordinary summation.

Part 3 of 4. Weakness 3 and Question 3: when the spectral and spatial-difference losses should be used

The two losses should be used for smooth, vortex-dominated flows, because they lower the error on all four fluid tasks. For a shock-dominated task such as CE-RPUI, a validation step decides whether to use them, and the test data confirm its decision.

The spectral and spatial-difference losses, which we call the two structure terms, help on all four fluid tasks. On HeldOut128 with three seeds, removing them raises the Mean and Final error by 3.6% and 2.6% on NS-PwC and by 3.0% and 1.6% on FNS-KF, both with direct queries. With recurrent rollout, removing them raises the two errors by 3.2% and 2.9% on NS-SL and by 4.9% and 4.9% on NS-SVS.

The choice of whether to use the two terms is made per task on validation data, so the user does not need to decide it in advance. The 16 validation trajectories of the training split decide whether the structure terms are kept. The recipes with and without structure terms are both among the predeclared candidates, and we fixed the choice before any HeldOut128 inference. For CE-RPUI this step selected the recipe without structure terms, and the test set confirms the choice.

CE-RPUI on HeldOut128MeanFinal
TEMPO with structure terms114.441 ± 0.434167.101 ± 0.455
TEMPO without structure terms, selected on validation112.350 ± 0.616164.838 ± 0.087
Fine-tuning reference120.759181.553

The recipe selected on validation has the lower error in all three seeds. Both recipes are also below the fine-tuning reference, by 5.2% to 7.0% in Mean error and by 8.0% to 9.2% in Final error. For the four fluid tasks the same step keeps the structure terms. Our answer to your question of when to use the losses is therefore that validation switches them on or off per task. The decision made on 16 validation trajectories was the right one on 128 unseen test trajectories.

As to why CE-RPUI differs, the residual spectra show the direction of the effect on each task, and the direction on CE-RPUI is consistent with the shocks in its solutions. Removing the structure terms raises the residual power at wavenumbers 33 to 64 by 59% on NS-PwC and by 48% on NS-SL. On these incompressible flows the terms recover fine-scale content that the field loss alone fits poorly. On CE-RPUI the same change lowers the residual power in this band by 11%. CE-RPUI is a compressible Riemann problem with shocks and contact discontinuities, and for such fields the two structure terms are dominated by the discontinuities. At a discontinuity, a small displacement of a front produces a large penalty, and this penalty competes with the field error that the metric measures. This leads to the same guideline as the validation step, which is to use the structure terms for smooth, vortex-dominated flows and to let validation decide for shock-dominated flows.

Comments on the paper

1 comment

We thank the three reviewers for their careful and constructive reviews. We ran new experiments for the three requests made in your comments, and every comparison below favors TEMPO or its per-step score. The first is evidence that the gains hold across random seeds and on test data. The second is a comparison with standard trajectory training and gradient balancing on the same backbone. The third is evidence beyond Poseidon-B and beyond the 11-step window. In all our replies, TEMPO means the full method of Algorithm 1 unless stated otherwise, and error values that are not percentages are multiplied by 1000.

We use five evaluation sets, and we name them here so that every table is clear. HeldOut128 is new and contains 128 previously unused test trajectories per task. FinalTest144 is the one-shot test set that we evaluated before submission. It has 144 trajectories per task and shares no trajectory with HeldOut128. Gate96 and Fresh128 are the evaluation sets of the paper. The long-horizon cohort is the set of 96 trajectories with 247-step references used in Figure 1.

HeldOut128 gives an independent test of the paper's conclusions. For each task we drew from the official test files 128 trajectories that had not been used during development. We fixed the trajectory lists, the model checkpoints and the statistical analysis before the first inference, and we selected nothing on this set. We evaluated 204 Poseidon-B models on it. These are the 90 models of Table 2, 90 newly trained matched models, and 24 fine-tuning references that cover both prediction paths of the six registered tasks. Matched models use seeds 9841, 9842 and 9843, and they share initialization, data, 1,280 updates, optimizer and checkpoint rule. We report Δ, the relative excess error of an alternative over TEMPO averaged over paired seeds, so a positive value means that TEMPO is more accurate. We call a result significant when it passes a seed-by-trajectory bootstrap with Holm correction over all registered comparisons.

1. In the three recurrent settings, TEMPO is more accurate than standard trajectory training and gradient balancing on the same backbone. Each cell gives Δ in percent as Mean error / Final error.

AlternativeNS-PwCNS-SLNS-SVS
Pushforward+11.32 / +14.64+14.78 / +17.58+15.57 / +17.91
Multistep, uniform weights+5.96 / +5.17+4.86 / +5.55+4.58 / +4.34
Multistep, linear weights, the Field-only primary row of Table 2+11.94 / +12.12+7.99 / +7.94+9.57 / +9.85
Multistep, harmonic weights+3.38 / +1.67+0.91 / +2.35+0.72 / +1.10
PEGrad+3.66 / +4.72+3.64 / +3.99+4.34 / +5.28
MetaBalance+3.34 / +3.07+1.53 / +2.05+1.83 / +1.66
Ordinary summation+5.16 / +7.04+2.15 / +2.33+7.28 / +8.97

TEMPO has the lower error in all 42 cells. Against pushforward, uniform multistep, linear multistep and PEGrad, every one of the three seeds also favors TEMPO in every cell, and against pushforward, linear multistep and PEGrad every cell is significant.

2. TEMPO is more accurate than fine-tuning on every task. On HeldOut128, TEMPO has a lower error than the stronger of the direct and recurrent fine-tuning references of each task. Its Mean and Final errors are lower by 34.5% and 45.0% on NS-PwC, 42.9% and 49.2% on FNS-KF, 6.8% and 6.8% on NS-SL, 7.5% and 7.8% on NS-SVS, and 5.2% and 8.0% on CE-RPUI.

3. On HeldOut128, the results vary little across seeds. The standard deviation of TEMPO's error across the three seeds is between 0.3% and 2.2% of the error itself. This holds for all six pairs of task and prediction path and for both metrics.

4. Over 247 steps on NS-PwC, TEMPO has a lower error than ordinary summation. In the 247-step study of the paper with three seeds and 96 trajectories, ordinary summation has 7.7% higher Mean error and 8.7% higher Final error than TEMPO. We also ran a new check of the reference solution of this study. When we recompute the 247-step reference on finer grids and with smaller time steps for all 96 trajectories, it changes by only 0.008% of TEMPO's Mean error and 0.010% of TEMPO's Final error.

5. TEMPO's per-step score also improves another pretrained backbone. On DPOT-L with three seeds, replacing the native fine-tuning loss by TEMPO's per-step score lowers the Mean and Final error by 15.7% and 14.0% on NS-PwC, 13.3% and 9.2% on NS-SVS, and 13.9% and 12.1% on CE-RPUI. Every seed improves in all six comparisons.

6. The one-shot test set also favors TEMPO. On FinalTest144, which we evaluated once before submission, TEMPO has a lower error than all 13 external architectures of Table 1 and every fine-tuning reference. This holds on all five main tasks and in both metrics.

Our replies to each reviewer answer every weakness and question and give the complete tables.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.