- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper introduces FOCO, a post-hoc method that trains a small U-Net to iteratively correct the predictions of a frozen surrogate (FNO, TFNO or Transolver), following a relaxed Picard iteration whose number of steps sets the test-time budget. The main contribution is the training procedure: the corrector learns from its own detached intermediate states, which requires only three corrector evaluations per example and avoids backpropagating through the refinement trajectory. The authors derive error bounds under an assumed contraction and provide empirical results on ERA5 super-resolution and four 2D/3D dynamical systems.
Critical strengths
-
Simple and efficient training scheme. The objective has no spectral, fixed-point or PDE-residual terms, and the staged backward pass keeps memory independent of the refinement depth. The efficiency comparison with IRNO is carefully controlled.
-
Broad evaluation. Five benchmarks, including a 3D MHD problem and global weather super-resolution,
-
This paper has a clear value: it presents a method capable of improving PDE surrogate solutions at a low cost, with a simple yet unexplored method.
-
The experimental section provides a clear proof that this method is able to improve predictions for 2D or 3D benchmarks. The rollout analysis is specially interesting.
Critical weaknesses
-
I find the methods section slightly unclear. Specially on the definition of and , that are not explicitly presented. For instance, it is confusing that section 3.1 only mentions and then section 3.2 starts by saying that is frozen, which did not appear during section 3.1.
-
It would be better if the authors proposed a method to automatically decide the hyper-parameter K (number of refinement steps) rather than hard coding it, since there is clearly an optimal value of K according to their experiments. The “test-time scaling” claim is not well supported.
-
Missing ablations and related work on the training objective. Training on detached model-generated states is closely related to the push-forward trick of, which unrolls a model without gradients and trains on the resulting states to reduce distribution shift in autoregressive solvers. This should be discussed.
-
The evidence suggests the corrector may act as a second predictor rather than a refiner of the surrogate. The corrector receives the full conditioning input x in addition to the current estimate, so it can in principle predict y from x and largely ignore u. The corrector has some times more parameters than the base model.
Questions for authors
-
What does the corrector achieve when trained and evaluated from a trivial initialization instead of the surrogate prediction? How sensitive is its output to u, for example measured by perturbing u_0?
-
Can you do ablations on K?
-
How does FOCO compare with the pushforward trick and with a standard two-step unroll with full backpropagation?
-
What are the results of ablating each of the three loss branches?
4 comments
Official Comment by Authors Part II
Q1. What does the corrector achieve when trained and evaluated from a trivial initialization instead of the surrogate prediction? How sensitive is its output to , for example measured by perturbing ?
We added a conditioning-only U-Net control and direct perturbation tests on ERA5 and TR-2D. These experiments demonstrate better primary benchmark accuracy than standalone prediction, improved autoregressive performance, and learned state dependence beyond the identity skip. In the tables, Trivial Init. denotes a standalone U-Net receiving without an FNO prediction and matching the corrector’s width, depth, and normalization. It uses the standalone-surrogate training recipe, while Standard FOCO retains its correction-training recipe. This compares the two trained systems rather than isolating initialization under an identical objective.
Table 1. One-step prediction: Standard FOCO () versus Trivial Init.
| Method | ERA5 RFNE ↓ | ERA5 KE ACC ↑ | TR-2D VRMSE ↓ |
|---|---|---|---|
| Standard FOCO | 0.147572 | 0.928167 | 0.223098 |
| Trivial Init. | 0.155683 | 0.919094 | 0.241256 |
Standard FOCO reduces ERA5 RFNE by 5.21% and TR-2D VRMSE by 7.53% relative to Trivial Init. For autoregressive prediction, both systems start from the same four ground-truth frames and subsequently feed back their own predictions. FOCO applies one correction after each FNO forecast. FOCO achieves lower VRMSE at every tested horizon, with a 17.60% reduction at seven steps. This supports the practical benefit of the surrogate-plus-corrector system beyond one-step prediction.
Table 2. TR-2D autoregressive VRMSE ↓ by physical forecast horizon.
| Method / Horizon | 1 | 2 | 3 | 4 | 5 | 6 | 7 |
|---|---|---|---|---|---|---|---|
| Standard FOCO | 0.223098 | 0.296164 | 0.354021 | 0.405948 | 0.452254 | 0.496439 | 0.535835 |
| Trivial Init. | 0.241256 | 0.332835 | 0.411453 | 0.482496 | 0.544866 | 0.602814 | 0.650302 |
Transfer across different initial estimates. Our existing zero-shot cross-operator experiments provide complementary evidence (§4.6; Figure 7). A corrector trained around one frozen backbone is applied directly to another without retraining. Although FNO, TFNO, and Transolver have different initial accuracies and error structures, transferred correctors improve all 48 off-diagonal dataset–metric comparisons across ERA5, TR-2D, Active Matter, and MHD. Appendix G.8, Tables 33–36, reports the initial errors and corrected performance. These results demonstrate useful correction across naturally different surrogate estimates within the same physical task.
Perturbation robustness. We additionally hold fixed and perturb only the frozen surrogate prediction, without retraining:
Noise is applied in training-standardized coordinates: 10% means a noise standard deviation equal to 0.1 times each field’s training standard deviation. We evaluate one or two full corrections using the same noise directions across amplitudes and correction counts.
Table 3. ERA5 initialization perturbations.
| Noise | Noisy FNO RFNE ↓ | FOCO RFNE ↓ | FOCO RFNE ↓ | FOCO KE ACC ↑ | FOCO KE ACC ↑ |
|---|---|---|---|---|---|
| 0% | 0.2584 | 0.147572 | 0.146911 | 0.928167 | 0.928700 |
| 5% | 0.264763 | 0.157568 | 0.155311 | 0.925196 | 0.926673 |
| 10% | 0.282294 | 0.180588 | 0.175500 | 0.918001 | 0.921336 |
| 20% | 0.338784 | 0.243246 | 0.230900 | 0.893030 | 0.902251 |
Table 4. TR-2D teacher-forced initialization perturbations.
| Noise | Noisy FNO VRMSE ↓ | FOCO VRMSE ↓ | FOCO VRMSE ↓ | FOCO NMSE ↓ | FOCO NMSE ↓ |
|---|---|---|---|---|---|
| 0% | 0.298898 | 0.223098 | 0.219124 | 0.065921 | 0.065245 |
| 5% | 0.367708 | 0.282683 | 0.270397 | 0.067281 | 0.066268 |
| 10% | 0.494960 | 0.385594 | 0.359914 | 0.071433 | 0.069471 |
| 20% | 0.793656 | 0.625712 | 0.569298 | 0.088920 | 0.082899 |
Both correction counts improve the corresponding noisy FNO prediction at every reported noise level. At 10% noise, two corrections reduce ERA5 RFNE by 37.83% and TR-2D VRMSE by 27.28%. Errors increase with corruption, but correction remains useful throughout the tested range.
Official Comment by Authors Part III
We measure the change in the corrected endpoint relative to the injected perturbation:
Here denotes full corrections at fixed conditioning.
Table 5. Endpoint sensitivity.
| Noise | ERA5 | ERA5 | TR-2D | TR-2D |
|---|---|---|---|---|
| 5% | 0.9961 | 0.8732 | 0.7427 | 0.6647 |
| 10% | 0.9611 | 0.8646 | 0.7477 | 0.6658 |
| 20% | 0.9199 | 0.8343 | 0.7632 | 0.6723 |
The corrected outputs respond to initialization changes, while a second correction consistently lowers endpoint sensitivity. All reported sensitivity averages remain below one, indicating attenuation of the tested perturbations on average.
We additionally examine the net learned update , excluding the identity contribution. At 10% noise, the learned-update change normalized by the injected-noise norm is 0.6116 on ERA5 and 0.5197 on TR-2D. These nonzero responses demonstrate that the learned update itself depends on the current estimate, beyond the explicit skip connection.
Q2. Can you do ablations on K?
Section 4.5 and Figure 6 evaluate across four dynamical benchmarks using the same trained corrector without retraining. Tables 2–3 additionally compare one- and two-step refinement. Appendix G provides detailed trajectories, and Appendix G.9 reports frequency-band errors along a fixed trajectory. We will make these existing experiments more prominent and clarify that the budget-aware schedule changes the discretization when changes.
Thank you for the new experiments. Looking at the trivial init results, it seems most of the improvement over the FNO comes from the U-Net alone (0.156 vs 0.148 on ERA5), so I think this should be shown in the main paper. The pushforward comparison and the loss ablations are still missing.
I will keep my score.
Official Comment by Authors Part I
We thank the reviewer for recognizing FOCO’s practical value, efficient training procedure, controlled comparisons, and broad evaluation. We address the methodological clarifications and requested controls below.
Equation (1) defines Gθ(x,u) as the residual corrector and Dθ(x,u)=u+Gθ(x,u) as the corrected-solution map. Section 3.1 uses Dθ to express endpoint supervision; its trainable parameters are those of Gθ. The pretrained surrogate Bϕ remains frozen throughout corrector training. After training, Gθ is also frozen for inference. We will explicitly restate these relationships at the beginning of Sec3.1–3.2 and label the maps consistently in Figure 1.
Most practical gains are obtained in one or two evaluations under the budget-independent schedule. The budget-aware experiments demonstrate effective correction along longer trajectories, with limited additional terminal-accuracy gains across budgets. We will therefore use “test-time refinement” throughout the title and manuscript, making the empirical scope explicit.
The budget-independent schedule supports anytime stopping along a shared trajectory, allowing the inference budget K to be selected according to available computation without retraining. Automatically selecting the accuracy-optimal stopping point for each example remains future work, as noted in the conclusion. We will make the distinction between budget flexibility and adaptive stopping explicit.
Thanks for the great suggestion. We will explicitly discuss and cite the pushforward trick (Brandstetter et al., ICLR 2022). Both approaches train on detached model-generated states to reduce deployment-state mismatch.
FOCO applies this principle to post-hoc correction of an unchanged frozen surrogate. The original pushforward formulation trains an autoregressive solver using generated histories and subsequent physical-time targets. FOCO instead holds the conditioning–target pair (x,y) fixed and supervises initial, interpolated, and successor correction states toward the same solution. Its contribution is this specific correction-state construction and efficient endpoint-supervised training procedure, rather than detachment alone.
The corrector can be substantially smaller than the frozen surrogate. For the FNO-based experiments, Tables 7 and 10 report the following real-scalar parameter counts:
On 3D MHD, this corrector reduces the frozen FNO’s NMSE by 78.59%. Substantial improvements are therefore achieved with a correction network containing only a small fraction of the backbone’s parameters. The corrector is smaller than the FNO/TFNO backbones, although it can exceed the compact Transolver variants. Our controlled IRNO–FOCO comparisons hold corrector architecture and capacity fixed, so their performance differences cannot be attributed to a larger FOCO corrector.
Appendix F.3 further tests the residual-learning choice: with the same U-Net architecture and (x,u) inputs, direct full-solution prediction performs worse than residual correction on TR-2D and can slightly underperform the frozen surrogate. This supports residual parameterization; directly quantifying dependence on u requires the initialization and sensitivity controls discussed in Q1.