acceptodds
Under review as a conference paper at ICLR 2027

When Does Diversity Help? Realizable Complementarity in Neural Operator Ensembles

Abstract

Neural operators are commonly developed and evaluated as individual predictors, with improvements largely pursued through architectural advances. Yet our experiments reveal a surprising result: a simple average of independently trained models can deliver substantial accuracy gains. Across six PDE tasks, simple equal-weight averaging reduces error by up to , and the effect persists across different model families and controlled settings. However, the benefit varies sharply across expert pools, revealing that predictive diversity alone does not imply useful aggregation. We show that what matters is whether differences among models produce complementary errors that actually improve the combined prediction. This motivates the notion of useful complementarity. We further uncover a second gap: useful structure may exist within an expert pool without being reliably exploitable from finite development data. Increasing the flexibility of the combination rule does not necessarily translate into larger held-out gains, indicating a distinction between complementarity that is available and complementarity that can actually be realized. We call the latter realizable complementarity. Finally, our results show that information useful for constructing a strong aggregate need not remain equally useful for refining it afterwards. Together, these findings provide a framework for understanding when diversity among neural operators translates into reliable accuracy gains.

Primary area:
applications to physical sciences (physics, chemistry, biology, etc.)
Keywords:
PDE, Neural operator

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_EQ4BOpenReview ↗
Rating
3
Confidence
2(−1)
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

This paper primarily investigates the question: under what conditions can differences between models be transformed into actual improvements in predictive performance? By exploring various ensemble methods, the paper focuses on distinguishing between potential combinatorial gains existing in the pool of experts and actual gains that can be achieved on unseen cases with limited development data. Experiments show that having diverse models does not always mean they work well together, and simply adding more combinations does not always improve how well the models perform on new cases.

Critical strengths
  1. The paper clearly states its research problem and presents a systematic experimental analysis. It verifies the effectiveness of equal-weight ensembles and also examines the sources and limitations of ensemble benefits. The study uses computational control, compares different model architectures, analyzes error-space distributions, and tests across cases, creating a thorough experimental analysis framework.
  2. It is methodologically valuable to distinguish between potential gains and gains that can actually be achieved. The paper uses an oracle as an ideal reference to show the possible combinatorial benefits within a fixed group of experts. At the same time, it shows through limited data learning and independent testing that these gains may not be possible, which helps prevent confusing model differences or training set improvements with real generalization gains.
Critical weaknesses
  1. The main contribution focuses on empirical analysis, but there is still no combination method that works well in all cases. The article clearly explains when current combination strategies work and where they fall short. However, the experiments depend on specific expert groups and task settings, and using more complex selection strategies does not always lead to better results than simpler ones. As a result, the findings do not show that the conclusions can be applied to other PDE tasks, and they do not offer a general algorithm that can reliably deliver the possible benefits of combining methods.
Questions for authors
  1. The paper compares equal-weight averaging, fixed convex combination, learnable combination with different spatial granularities, and risk-based expert selection strategies. However, it does not fully explain how these methods are better than other learnable ensemble methods. Could the authors add comparative experiments with methods such as stacking and input-conditioned gating, and evaluate them on the same dataset and with the same computational budget?
  2. The point-wise oracle for goal perception in the paper shows much higher gains than learnable combination methods on the Elasticity and Pipe tasks. However, since the oracle depends on evaluation goals, it cannot be used directly for real prediction. Could the authors discuss the main reasons why ensemble gains are hard to achieve, such as limited development data, not enough expressive power in the combination rules, or weak generalization across cases? This would help clarify the practical meaning and limits of the achievable complementarity concept in the paper and also help test the methods in more varied scenarios to strengthen the experiments, such as by evaluating performance on cross-resolution or geometric generalization tasks. Please also consider whether changes in grid resolution and discretization (like mesh and point cloud) affect error complementarity between experts and the gains from equal-weight averaging and learnable combination.
5 comments
AuthorsOpenReview ↗

Response to Reviewer EQ4B, Weakness 1: Scope and Generality of the Findings

We thank the reviewer for recognizing our systematic analysis and the distinction between available and realizable complementarity. Below, we provide the response point by point to address the reviewer's concern.

Response to Weakness 1: Scope and Generality of the Findings

Our study demonstrates substantial recoverable accuracy in competitive neural operators and systematically investigates when further gains can be realized. Simple averaging achieves these improvements without architectural modifications or learned aggregation weights.

Equal averaging improves over the strongest constituent in all six fixed LinearNO pools, including reductions of 18.46% on Elasticity and 17.80% on Navier–Stokes (Table 1). On Elasticity, seven-model LRSA averaging reduces error by 23.80% relative to the strongest constituent (Sec. 3.2).

Additional analyses of member-disjoint triples support robustness to expert-pool composition. Two member-disjoint LinearNO triples achieve gains of 18.46% and 21.64%, while two member-disjoint LRSA triples achieve 16.63% and 18.30%. Training configurations remain fixed within each architecture, and all comparisons use the same Elasticity evaluation cases.

Our compute-controlled experiments also demonstrate improvements beyond the tested single-model alternatives. The supplementary Elasticity controls show reductions of 4.79% relative to long training and 5.06% relative to SWA, at the same counted training-FLOP budget.

Beyond these accuracy gains, our analyses investigate how additional combination complexity and expert information contribute to realized performance. Fig 4 compares learned combinations against strong fixed references, while Table 5 separates residual correction from the incremental value of individual expert predictions. In the heterogeneous Pipe study, case-level advantage selection further improves over a fitted fixed convex combination by 6.46%, with a paired-case 95% interval of [1.95%, 11.72%] (Tab 4).

These findings provide empirical insights into both the potential and the practical realization of neural operator complementarity. They motivate strong fixed references, matched centre-only controls, and explicit evaluation of whether additional combination flexibility provides reliable gains. We will highlight these broader implications and supplementary robustness results in the revision.

AuthorsOpenReview ↗

Response to Reviewer EQ4B, Question 1: Comparisons with Stacking and Input-Conditioned Gating

We thank the reviewer for this suggestion. Our study evaluates constrained linear stacking, input-conditioned gating, and nonlinear prediction-level correction. We further conducted controlled experiments to strengthen these comparisons.

Stacking and input-conditioned gating. Our fixed convex combination serves as a constrained linear stacker, learning shared weights across cases. The Global combiner predicts input-dependent weights from coordinates and expert predictions, while regional and pointwise variants allow spatially varying weights. These gating methods use the same frozen experts, fitting data, model capacity, and optimization budget within each task. We also investigate nonlinear prediction-level correction using a larger residual MLP (Sec. 5.3, Tab 5), with matched centre-only and expert-aware controls to evaluate the additional value of individual expert predictions.

Further evaluation of learned combination. We evaluated gating models with widths of 32, 128, and 256 using the same expert pools and data splits. At 100 fitting cases, the gains over equal averaging are 2.77%, 2.83%, and 2.85% on Elasticity, and 3.35%, 3.71%, and 4.40% on Pipe, respectively. We further used validation data to select the best-performing combination strategy, including the option of retaining the fixed convex combination. On Elasticity, validation-selected gating further reduces error by 0.62% compared with the fitted fixed convex combination, with a paired-case 95% interval of [0.47%, 0.78%]. This shows that learned weighting can provide additional accuracy even beyond a strong fixed combination.

AuthorsOpenReview ↗

Response to Reviewer EQ4B, Question 2: Factors Affecting Realizable Gains and Discretization

We thank the reviewer for highlighting these factors. We further investigated how fitting data, learner capacity, the starting aggregate, and generalization influence the realization of neural operator complementarity.

(a) Fitting-data availability and learner capacity.

We evaluated gate widths of 32, 128, and 256 with 25, 50, and 100 fitting cases, using the same frozen experts and data splits. At 100 fitting cases, the mean relative-L2L_2 improvements over equal averaging are:

Gate widthElasticity: PointPipe: 2×2
32+2.77%+3.35%
128+2.83%+3.71%
256+2.85%+4.40%

Additional fitting data also improves realized gains. With width 32, increasing the fitting cases from 25 to 100 improves Elasticity from 2.48% to 2.77%, while increasing from 50 to 100 cases improves Pipe from 0.64% to 3.35%. These results show how data availability and learner capacity can influence the accuracy recovered through learned combination.

(b) Starting aggregate and the value of expert information.

We extended the nonlinear correction experiments in Tab 5 by interpolating between equal averaging and the fitted convex combination:

\qquad \lambda\in\{0,0.25,0.5,0.75,1\}.$$ Using matched centre-only and expert-aware correction models, **the additional error reductions from individual expert information are 3.00%, 1.93%, 1.10%, 0.70%, and 0.66%**, respectively. **These results reveal that the incremental value of individual expert predictions depends on the starting aggregate.** Their additional contribution decreases along the tested path from equal averaging to the fitted convex combination, offering further insight into how expert complementarity can be exploited beyond an existing aggregate. **(c) Cross-case generalization.** Our five-fold transfer experiments (Tab 3) further evaluate whether learned combinations retain their gains on unseen cases. On Elasticity, fitted convex combinations achieve **2.83% improvement under random holdout and 2.82% under input-statistic holdout**, demonstrating successful cross-case transfer in these settings. **(d) Resolution and discretization.** We additionally evaluated three frozen Darcy models on the same 200 physical cases. Equal averaging achieves **8.87% improvement** over the best fixed constituent at the native $85\times85$ resolution and retains **8.40% improvement** when predictions are interpolated to a $421\times421$ evaluation grid. **These results demonstrate that the ensemble benefit is largely preserved under the tested output-resampling operation**, providing evidence beyond evaluation at a single scoring resolution. **Together, these experiments provide concrete insights into how available complementarity translates into realized accuracy**, highlighting the roles of fitting data, learner capacity, the starting aggregate, and generalization.
Reviewer_o6WJOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
3
Soundness
3
Read review
Paper summary

The paper studies how combining independently trained neural operators can improve PDE prediction accuracy, with simple averaging reducing errors by up to 18%. It then validates on standard 2D PDE examples.

Critical strengths

Extensive experiments across six PDE benchmarks and multiple neural-operator architectures. Interesting empirical results showing substantial accuracy gains from simple averaging.

Critical weaknesses

Limited novelty: Most conclusions are already established in ensemble learning, with similar work in the 1990s and 2000s (albeit not about neural PDEs specifically)

Experiments would need more analysis in particular with more robustness (independently trained model pools) and also a justification on why this additional complexity is needed. The comparison in table 2 is good for training, but what about inference?

Questions for authors

Can the authors demonstrate that ensembling offers a better accuracy–compute trade-off than either larger, longer-trained, or distilled single neural operators (for similar computationnal budgets). I would extend the results shown for Darcy to the other cases.

3 comments
AuthorsOpenReview ↗

Response to Reviewer o6WJ, Weakness 1

We thank the reviewer for recognizing the extensive experiments and substantial accuracy gains. Our contribution combines the finding of substantial recoverable accuracy in competitive neural operators with a systematic investigation of when predictive diversity yields useful and realizable gains. Below, we give the response to the reviewer's concern point by point.

Response to Weakness 1: Novelty Beyond Ensembling

We thank the reviewer for this comment. A simple yet effective strategy for improving competitive neural operators has practical significance. Our study demonstrates substantial, non-trivial accuracy gains through averaging, without architectural modifications or learned aggregation weights.

Equal averaging improves over the strongest constituent in all six fixed LinearNO pools, including reductions of 18.46% on Elasticity and 17.80% on Navier–Stokes (Tab 1). On Elasticity, averaging seven independently trained LRSA models reduces error from 0.003079 to 0.002346, a 23.80% reduction.

Controlled comparisons further support the significance of these improvements. Among the tested alternatives, the gains cannot be explained merely by a larger training-compute budget or model size. In particular, the ensemble outperforms a wider single model with approximately matched training FLOPs, total parameters, and forward inference FLOPs (Tab 2).

We also contribute systematic analyses of the sources and conditions of these gains. Spatial and case-level measurements identify localized, shared errors that dominate the remaining risk and limit aggregation gains (Fig. 3). Learned-combination experiments examine whether additional spatial flexibility improves performance beyond strong fixed references (Fig. 4). Matched correction controls separately assess the additional value of individual expert predictions beyond the aggregate (Tab 5).

The study also demonstrates realized gains beyond fixed aggregation. In a separate heterogeneous Pipe experiment, case-level advantage selection reduces error by 6.46% relative to a fitted fixed convex combination, with a paired-case 95% interval of [1.95%, 11.72%] (Tab 4).

Our contribution therefore extends beyond applying an existing ensemble operation to PDE benchmarks. It combines effective accuracy recovery, controlled evidence supporting its value, and insights into the error structures and learning conditions governing further gains.

AuthorsOpenReview ↗

Response to Reviewer o6WJ, Weakness 2: Robustness Across Expert Pools and Inference Compute

We thank the review for these comments.

Reponse to Weakness 2: Robustness Across Expert Pools and Inference Compute

Robustness to expert-pool composition. We evaluated member-disjoint three-model pools formed from independently trained LinearNO and LRSA models on Elasticity. Training configurations remain fixed within each architecture, and all comparisons use the same 200 evaluation cases. Errors below are mean relative L2L_2, with gains relative to the strongest fixed constituent within each triple.

ArchitectureBest constituentEqual averageGain
LinearNO5.24454.276518.46%
LinearNO5.17494.055321.64%
LRSA3.10062.584916.63%
LRSA3.11852.547818.30%

Substantial gains recur across member-disjoint triples in both backbones. We also exhaustively evaluated all three-model combinations: all 56 LinearNO triples and all 35 LRSA triples improve over their respective strongest constituents. Median gains are 17.10% and 17.65%, with ranges 5.62–25.27% and 16.12–19.05%, respectively.

Within each architecture, the two example pools share no members. The exhaustive combinations overlap and share evaluation cases, so these results quantify sensitivity to member composition rather than represent independent replications for every combination.

Inference compute. We further compare the ensemble with larger single LinearNO models on Darcy to examine whether its accuracy gains can be explained by increased computational resources. All strategies use approximately 131 training PFLOPs, and we report the total forward FLOPs across all ensemble members. Errors are mean relative L2L_2.

StrategyParameters (M)Forward GFLOPsError
Longer-trained single model1.7729.135.5850
Wider single model5.3784.235.7736
Larger-latent single model1.8855.655.1779
Three-model equal average5.3087.404.4282

At similar parameter counts and forward FLOPs, the ensemble reduces error by 23.30% relative to the wider single model. This shows that its accuracy advantage cannot be explained simply by a larger model or more forward computation in this comparison.

Response to Question 1: Comparisons with Larger, Longer-Trained Single Operators

Our experiments show that the tested single-model alternatives do not match the ensemble's accuracy under comparable training budgets. On Darcy, three-model averaging reduces error by 20.71%, 23.30%, and 14.48% relative to the longer-trained, wider, and larger-latent single models, respectively (Tab 2). The wider model also has comparable total parameters and forward FLOPs, as discussed above.

We further extended the compute-controlled comparison to Elasticity, adding long-training and SWA baselines. All strategies use the same LinearNO architecture, training data, and evaluation cases, with approximately 7.31 counted training PFLOPs. The single-model baselines are trained for 1,500 epochs, while the ensemble consists of three independently trained 500-epoch models.

PredictorTraining epochsMean relative L2L_2 (×10−3\times10^{-3})
Long training1,5004.49159
SWA1,5004.50471
Three-model output averaging3×5003\times5004.27655

SWA averages 30 checkpoints and improves over its own final iterate by 1.44%. Output averaging further reduces error by 5.06% relative to SWA and 4.79% relative to long training.

Together, the Darcy and Elasticity experiments demonstrate that independent-model averaging can recover additional accuracy beyond the tested single-model scaling and weight-averaging strategies at comparable training compute. A compute-matched distillation comparison is not included, and we make no claim about distilled operators.

AuthorsOpenReview ↗

Response to Reviewer o6WJ, Question 1: Comparisons with Larger, Longer-Trained, and Distilled Single Operators

We thank the reviewer for this question. Our compute-controlled experiments show that the tested larger and longer-trained single operators do not match the ensemble's accuracy at comparable training compute. We also evaluated single-model distillation in a separate MoNo experiment.

Larger and longer-trained single operators. As shown in Tab. 2, the three-model ensemble outperforms all tested single-model alternatives at approximately matched training FLOPs, reducing error by 20.71%, 23.30%, and 14.48% relative to the longer-trained, wider, and larger-latent models, respectively. Notably, the ensemble also outperforms the wider model by 23.30%, despite their comparable total parameter counts and forward FLOPs (as detailed in response to weakness 2).

Additional long-training and SWA controls. We extended the compute-controlled comparison to Elasticity. All strategies use the same LinearNO architecture, training data, batch size of one, and FP32 precision. Each strategy uses 1.5 million total optimizer updates and approximately 7.31 counted training PFLOPs, under the same accounting convention.

PredictorTraining epochsMean relative L2L_2 (×10−3\times10^{-3})
Long training1,5004.49159
SWA1,5004.50471
Three-model output averaging3×5003\times5004.27655

SWA averages 30 late-training checkpoints and improves over its own final iterate by 1.44%. Output averaging further reduces error by 5.06% relative to SWA and 4.79% relative to long training.

Single-model distillation. On Elasticity, we distilled a two-model MoNo ensemble into a single MoNo student. The student uses the same architecture as the original model and receives 50 additional epochs of supervision from both ground truth and teacher predictions. At inference, only the student is required.

PredictorMean relative L2L_2 (×10−3\times10^{-3})Error reduction
Original MoNo3.28598—
Distilled student3.250881.07%
Teacher ensemble2.8390413.60%

The distilled student recovers only 7.85% of the teacher ensemble's error reduction, showing that the tested distillation strategy does not fully preserve the ensemble's accuracy gains. Together, these experiments demonstrate that the tested single-model scaling, longer-training, weight-averaging, and distillation approaches do not match the accuracy of output averaging.The LinearNO comparisons use matched training compute, while the MoNo distillation experiment provides a separate assessment of transferring ensemble gains to a single model.

Reviewer_vCpKOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper shows that equal-weight averaging of three independently trained LinearNO models cuts error by 0.5–18.5% across six PDE benchmarks, and that the ensemble beats single-model scaling at matched training FLOPs on Darcy. It then defines "useful complementarity" (diversity that actually reduces error) and "realizable complementarity" (the part a learned combiner can recover from finite data). Supporting analyses cover the V − Q decomposition, risk concentration on Pipe, spatially adaptive combiners against a pointwise oracle, risk-based switching, and correctors that do or don't see the individual experts.

Critical strengths
  • The Pipe analysis is the most informative part. The ensemble improves 177 of 200 cases, yet the hardest 10% carry 99% of the risk, and the outlet errors are shared across LinearNO, LRSA and MoNo. That is a concrete diagnosis of why averaging fails there.
  • Comparing learned combiners against a strong fixed convex combination (Fig. 4c), not just the equal mean, is the right control, and the paper reports the negative result honestly.
Critical weaknesses
  1. Deep ensembles beating their best member is well-known (Lakshminarayanan et al. 2017), and unfortunately "neural operators too" is not an ICLR-level finding.

  2. Eq. 1 undercuts the paper's central claim. For squared risk, R_avg − R_eq is exactly the ambiguity term (Krogh & Vedelsby 1995): the mean squared spread of the members around their average. Per-case normalization by ‖u_i‖² doesn't change this. So V is disagreement, and it is computable without labels.

    • The paper's statement that the diagnostic "uses observed targets" is wrong for V. Only Q needs labels.
    • "Disagreement alone is not sufficient" therefore reduces to "ambiguity must exceed the gap between the average member and the best member." That is the standard bias–variance–diversity decomposition, not a new notion.
    • The paper calls Eq. 1 "a standard identity" but cites neither Krogh–Vedelsby nor Wood et al. (JMLR 2023, A Unified Theory of Diversity in Ensemble Learning). The latter asks exactly "when does diversity help" and answers it with this decomposition.
    • The mismatch between squared risk in Sec. 4 and mean relative L2 in Table 1 is acknowledged but leaves the diagnostic only loosely connected to the headline numbers.
  3. A target-aware pointwise convex oracle measures how often the target falls inside the expert interval, not structure a learner could exploit. Any pool with nontrivial spread produces a large "potential," so 64.6% vs 2.8% says nothing about learnability. A meaningful upper reference would be a combiner with access to features that could be observed at deployment.

  4. Ninety-nine percent of risk is in a small number (20) of cases and at the outlet, shared across architectures. That pattern suggests a data or boundary-condition issue (outlet treatment, out-of-distribution geometries) rather than a property of ensembles. Without investigating it, Pipe works as a counterexample to "averaging helps," not as evidence for a complementarity principle.

The averaging gains are real and Pipe tail analysis is useful, but the conceptual contributions repackage the ambiguity decomposition and stacking generalization without citing them. The empirical base also varies too much across pools and protocols to support general mechanism claimS.

Questions for authors
  • Since V equals mean ambiguity and needs no labels, how well does V/R_avg alone predict the averaging gain across tasks and pools?
  • How do model soups or SWA compare against the 3-model ensemble at matched training compute and single-model inference cost?
  • Is the Navier–Stokes ensemble applied per step or post hoc?
6 comments
AuthorsOpenReview ↗

Response to Reviewer vCpK, Weakness 1: Novelty Beyond Ensembling

We thank the reviewer for the careful assessment. We would like to say that our contribution combines the finding of substantial recoverable accuracy in competitive neural operators with a systematic investigation of when predictive diversity yields useful and realizable gains.

We characterize how error cancellation, member quality, and the concentration of shared errors shape the observed gains. We distinguish available combination potential from the gains learned using finite development data. We further show that predicting errors does not necessarily identify beneficial combination decisions, and that the incremental value of individual expert predictions changes after forming a stronger aggregate.

Together, these findings provide empirical insights into the opportunities and limitations of neural operator ensembles. Below, we respond each concern one by one.

Response to weakness 1: Novelty Beyond Ensembling

We thank the reviewer for this comment. A simple yet effective strategy for improving competitive neural operators has practical significance. Our study demonstrates substantial accuracy recovery through averaging, without architectural modifications or learned aggregation weights.

Equal averaging improves over the strongest constituent in all six fixed LinearNO pools, including reductions of 18.46% on Elasticity and 17.80% on Navier–Stokes (Tab 1). On Elasticity, averaging seven independently trained LRSA models also achieves a 23.80% reduction.

Controlled comparisons further support the value of these gains. Among the tested alternatives, the compute-controlled results show that the improvements cannot be explained merely by a larger training-compute budget or model size (Tab 2). In additional Elasticity experiments conducted for this rebuttal, three-model equal averaging reduces error by 4.79% relative to long training and by 5.06% relative to a fixed-configuration SWA baseline, under the same counted training-FLOP budget.

We then systematically investigate the sources and conditions of these gains. Our spatial and case-level analyses identify localized, shared errors that dominate the remaining risk and limit aggregation gains (Fig. 3). These analyses explain why the spatial distribution of expert errors matters beyond the overall magnitude of prediction disagreement.

We further investigate realizable complementarity through learners using deployment-observable information and finite development data. Greater spatial flexibility does not consistently improve evaluation performance (Fig. 4). In a separate heterogeneous Pipe study, case-level advantage selection reduces error by 6.46% relative to a fitted fixed convex combination, with a paired-case 95% interval of [1.95%, 11.72%] (Tab 4). This provides a positive example of learned gains beyond strong fixed aggregation.

Our contribution therefore combines effective accuracy recovery, controlled evidence supporting its value, and systematic insights into its sources and limitations. These results inform both the use of simple averaging and the evaluation of more elaborate neural operator ensembles.

AuthorsOpenReview ↗

Response to weakness 2: Eq. (1) Uses Existing Theory

We thank the reviewer for highlighting the connection to classical ensemble theory and suggesting these references. As stated in Sec. 4.1, we use Eq. (1) as a standard diagnostic identity providing a principled foundation for our empirical analysis. We will cite Krogh and Vedelsby (1995) [1] and Wood et al. (2023) [2] directly alongside the equation. Our investigation examines how cancellation and member quality manifest across PDE tasks, and which additional factors enable or limit realized gains.

Comment (a): The paper's statement that the diagnostic ‘uses observed targets’ is wrong for V. Only Q needs labels.

The statement in Sec. 4.1 refers to the complete normalized risk comparison (V−Q)/Ravg(V-Q)/R_{\mathrm{avg}}, discussed immediately beforehand. Evaluating QQ and RavgR_{\mathrm{avg}} requires observed prediction errors. The statement that this diagnostic uses observed targets is therefore correct in context.

The raw per-case ambiguity AiA_i is computable from predictions alone. To make the distinction more explicit, we'd like to replace “the diagnostic” with “the complete normalized risk comparison.” This clarification will not influence the formulation and empirical results.

Comment (b): Disagreement alone is not sufficient” follows from established theory.

Our study goes beyond a straightforward application of conventional ensembling to neural operator benchmarks. We identify substantial recoverable accuracy in competitive operators through a simple yet effective aggregation strategy. Our compute-controlled comparisons further show that, among the tested alternatives, the gains cannot be explained merely by a larger training-compute budget or model size.

We also develop a systematic empirical account of how these gains arise and can be realized. The analyses locate shared errors across cases and spatial regions, examine why broader prediction coverage can leave dominant errors unresolved, and test whether additional combination flexibility and expert information improve predictions beyond strong fixed references.

Classical ensemble theory provides a principled foundation for interpreting these findings. Eq. (1) organizes the relationship between cancellation and member quality, while our controlled experiments investigate how that relationship manifests in actual neural operator predictions and what improvements can be learned from finite development data.

The contribution combines practically significant accuracy findings, controlled evidence supporting their value, and insights into the error structures and learning conditions governing further gains.

(c) Missing references to Krogh–Vedelsby and Wood et al.

We will cite Krogh and Vedelsby (1995)[1] and Wood et al. (2023) [2] directly alongside Eq. (1). These references establish the theoretical foundation for our measurements of cancellation and member quality.

(d) Connection between squared risk and the headline relative-L2L_2 results.

Our manuscript uses these metrics for complementary purposes. The primary metrics measure the reported accuracy gains, while normalized squared risk supports the analysis of cancellation and concentrated errors. Sec. 4.1 explicitly distinguishes their improvement percentages.

We additionally recomputed squared risk on the exact six three-model pools in Table 1. The following gains are relative to the strongest fixed constituent under each metric. Squared-risk gain is 100(V−Q)/Rbest100(V-Q)/R_{\mathrm{best}}.

TaskPrimary-metric gainSquared-risk gain
Elasticity18.46%32.22%
Navier–Stokes17.80%29.42%
Darcy8.53%14.51%
Plasticity7.04%13.44%
Airfoil6.69%5.84%
Pipe0.49%2.43%

Averaging improves both metrics in all six pools, and their relative-gain rankings coincide, with descriptive Spearman ρ=1.0\rho=1.0. The strongest constituent also coincides under both metrics in each pool. This establishes consistency in the direction and cross-task ordering of the gains for the examined pools, while keeping their magnitudes distinct.

The original Fig. 3a additionally analyzes case-wise error changes directly in the primary evaluation metric, complementing the squared-risk diagnostic.

[1] Krogh A, Vedelsby J. Neural network ensembles, cross validation, and active learning[J]. Advances in neural information processing systems, 1994, 7.

[2] Wood D, Mu T, Webb A M, et al. A unified theory of diversity in ensemble learning[J]. Journal of machine learning research, 2023, 24(359): 1-49.

AuthorsOpenReview ↗

Response to Reviewer vCpK, Weakness 3: Target-Aware Oracle and Learnability

We thank the reviewer for this comment. The oracle is used to analyze the combination potential available within a fixed expert pool. It measures how much error could be reduced by choosing the best pointwise convex combination of the existing predictions with knowledge of the true solution. We separately evaluate how much improvement can be learned using deployment-observable information.

Fig. 4a directly evaluates such learners. The Global, 2×22\times2, 4×44\times4, and Point gates use spatial coordinates, expert prediction means, member deviations, and prediction spread. A width-32 MLP produces softmax weights. Separate calibration cases determine a step toward the gate, preserving the convex-combination constraint. Inference uses coordinates and frozen predictions without access to evaluation targets.

Within each task, the learned combinations and oracle share the same frozen expert pool, evaluation cases, and equal-averaging reference. This allows their results to distinguish available combination potential from the gains achieved by the tested learners.

We further examine how fitting-data availability affects realized gains and report the results below. The frozen experts, features, calibration cases, and evaluation cases remain fixed, with 1,000 optimizer updates at each fitting budget. Gains in mean case-relative L2L_2 over equal averaging are calculated from mean errors across three fitting runs, without ensembling their predictions. Fitting counts exclude the separate calibration data.

Fitting casesElasticity: PointPipe: 2×22\times2
25+2.48%−0.29%
50+2.70%+0.64%
100+2.77%+3.35%

These measurements demonstrate actual recovery from observable features. They also show how the recovery changes with fitting-data availability: Pipe exhibits a larger change across the tested budgets, while Elasticity shows smaller improvements.

We additionally examine learner capacity using widths 32, 128, and 256 with matched features, fitting subsets, calibration/evaluation cases, and update schedules. At 100 fitting cases, the gains over equal averaging are:

WidthElasticityPipe
32+2.77%+3.35%
128+2.83%+3.71%
256+2.85%+4.40%

These are point estimates from mean errors across three fitting runs. The additional Pipe reduction of width 256 relative to width 32 is 1.08%, with a conditional paired-case 95% interval of [−1.14%, 4.93%]. Thus, the higher point estimate does not establish a reliable capacity advantage. Wider learners require more FLOPs, so these controls match optimizer updates rather than training compute.

Together, these experiments connect the analytical potential to actual learning results. The oracle quantifies an opportunity represented by existing predictions. The observable-feature gates show that part of this opportunity can be recovered on evaluation cases, while the fitting-budget and capacity controls directly examine the learning conditions. Our interpretation therefore rests on these empirical measurements rather than on the oracle–learner gap alone.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.