acceptodds
Under review as a conference paper at ICLR 2027

Energy-Decrease Training for Learned Conjugate Directions

Abstract

Learned iterative solvers are often trained without solution labels by minimizing the residual left after a correction. For a symmetric positive-definite system this under-rewards removing low-eigenvalue error: squared residual weights each error eigenmode by while energy weights it by , so such error can be large in the solution yet small in the residual. We show that, in a learned conjugate-direction solver, changing only the training objective substantially changes how much of this error is removed. Energy-DCDM (E-DCDM) maximizes the exact energy decrease of each step, divided by the error energy visible on a small coarse space (32 sine modes); it needs no solution labels and leaves the deployed solver unchanged. When both losses train on the same solver states, E-DCDM lowers relative solution error after eight operator products by at least on every paired seed on public Darcy and screened-Poisson problems, both under a shared optimizer recipe ( to ) and when each loss's learning rate and clipping are tuned separately ( to on screened Poisson). Numerically verified Darcy eigenvectors indicate that energy training removes more low-eigenvalue error. The gain needs a suitable state weighting: in 2D, energy decrease weighted by is worse than the residual loss, and a generic coarse-free normalizer does not reproduce the 2D gain. E-DCDM can have higher residual error, the method and main comparison were designed after earlier methods were scored on the same evaluation data, and on 47 held-out 3D operators only a coarse-free variant was tested ( lower one-step solution error). The study isolates the training objective rather than proposing a competitive solver: with stencil access, algebraic multigrid reaches far higher accuracy on Darcy within seven V-cycles.

Primary area:
applications to physical sciences (physics, chemistry, biology, etc.)
Keywords:
learned iterative solvers, neural operators, conjugate directions, energy minimization

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_He21OpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

The paper proposes a training objective, Energy-DCDM, as an alternative to Residual DCDM. Unlike Residual-DCDM, E-DCDM weights spectral coefficients by eigenvalues. The paper compares the two methods on the Darcy2D and Screened Poisson2D setups.

Critical strengths
  1. The proposed change is conceptually simple and easy to insert into any learned conjugate-direction solver.
  2. The proposed training objective is well motivated mathematically. The authors clearly explain why minimizing residual error can underemphasize low-eigenvalue components of the solution error, and why maximizing energy decrease better aligns with their solver's exact line step.
  3. The authors carefully perform paired comparisons, which makes it easier to attribute any performance gains to the change in loss.
Critical weaknesses
  1. The experiments section is difficult to follow. The presentation feels disorganized. The main result, normalized ablation, 3D PDE experiments, resolution, and longer-run results are spread across the main text and appendix without any clear progression. A clearer explanation of the experimental setup and a consolidated results table would make the evidence easier to assess.
  2. Generalization across PDEs is limited. The main 2D results cover only 2 PDEs. A third PDE family would make a convincing case for the generalizability of the proposed method
  3. While the controlled comparison with Residual-DCDM is valuable, the practical benefit would be clearer with rigorous benchmarking against classical solvers. The appendix includes CG, Jacobi-PCG, and AMG, but the wall-clock comparison is limited to one Darcy batch and does not measure the full pipeline, including warm-start inference. A cost-to-accuracy comparison across more problems, tolerances, and suitable numerical baselines would better establish when the learned method is competitive. I would place these results in the main body.
Questions for authors
  1. In Fig.4, what does table and toilet mean?
2 comments

In Fig.4, what does table and toilet mean?

Response. "Table" and "toilet" are held-out ModelNet10 object categories. Each mesh defines a 3D conductivity operator, not a boundary condition. We will label the panels "ModelNet10 table / toilet conductivity operators" and say this in the caption.

The experiments section is difficult to follow. The presentation feels disorganized. The main result, normalized ablation, 3D PDE experiments, resolution, and longer-run results are spread across the main text and appendix without any clear progression. A clearer explanation of the experimental setup and a consolidated results table would make the evidence easier to assess.

Response. We agree. In the revision, Section 5 opens with one setup paragraph (problems, grids, budgets, seeds, splits) and the table below. Then 5.2 objective and normalizer controls, 5.3 transfer limits (resolution, 3D, elasticity) and 5.4 cost against classical solvers (next reply). The table gives relative errors after eight operator products for validation-selected checkpoints on the full test sets, averaged equally over four warm-start groups (FNO or U-Net, crossed with two warm-start training sets for Darcy or with in/out-of-distribution forcing for screened Poisson).

ProblemLossFieldEnergyTrue residual
DarcyResidual6.43×10−66.43\times10^{-6}1.84×10−51.84\times10^{-5}1.40×10−41.40\times10^{-4}
DarcyEnergy6.57×10−76.57\times10^{-7}2.69×10−62.69\times10^{-6}4.24×10−54.24\times10^{-5}
Screened PoissonResidual6.43×10−96.43\times10^{-9}2.71×10−92.71\times10^{-9}1.73×10−91.73\times10^{-9}
Screened PoissonEnergy6.88×10−106.88\times10^{-10}4.01×10−104.01\times10^{-10}5.23×10−105.23\times10^{-10}

These test sets also informed method development. As a check, we fixed checkpoints and analysis, then generated 256 new Darcy masks with our Python reimplementation of the public field generator (not byte-identical to the original). E-DCDM's (E's) field error is 0.120, 0.098 and 0.098 times Residual-DCDM's for the three paired seeds, and all 16 warm-start networks favor E for every seed. The same check on screened Poisson with newly generated forcings gives 0.0960 [0.0916, 0.1005] for the validation-selected pair (same-seed pairs: 0.11 to 0.14).

Generalization across PDEs is limited. The main 2D results cover only 2 PDEs. A third PDE family would make a convincing case for the generalizability of the proposed method

Response. We added static SPD linear elasticity, built from meshes of the public SEAT2D-GNN dataset. It uses a graph network and a 12-function polynomial coarse normalizer. Results are on 174 held-out cases in 29 geometry groups, after seven corrections (fixed checkpoints, intervals over groups).

MethodWorkField error
E-DCDM7 corrections0.0855
Residual-DCDM7 corrections1.010
CG / Jacobi-PCG7 products0.997 / 0.997
CG / Jacobi-PCGto reach 0.0855363 / 331 products
AMG, rigid-body modes7 V-cycles0.103
AMG, rigid-body modesto reach 0.08558 V-cycles

E's field error is 0.085 [0.079, 0.091] times Residual-DCDM's; each seed favors E in all 29 groups. Residual-DCDM's field error stays at its starting level (1.010; 1.007 for the retuned optimizer below; CG: 0.997 after seven products), although both Residual models lower the true residual (0.373 and 0.405): they reduce what they are trained on and leave the low-eigenvalue error that the energy objective targets. A Residual learning-rate/clipping sweep (selected on development data) leaves the ratio at 0.085. Training both 5x longer gives 0.043 [0.040, 0.045], or 0.048 when both objectives use learning rate 10−310^{-3}; Residual-DCDM's field error stays at 0.96 to 0.98; true residuals tie (0.99).

Limits: E does not lower the true residual on elasticity (1.13 of its start after seven corrections, versus 0.37 to 0.41 for the 30-epoch Residual models), the expected trade when weighting low-eigenvalue error; CG and Jacobi-PCG reach 1.23 and 1.19 after seven products. So a residual tolerance cannot stop E on elasticity, but a label-free rule can: the exact energy decrease of each step gives a delayed lower bound on energy error (4 extra corrections). Stopping when it reaches 0.1 (picked post hoc from 0.3/0.2/0.1, delay from 2/4) took a median 14 products (90th percentile 24) and left field error at most 0.171 (2x the 0.0855 above) in 97% of 522 case-seed runs (median 0.068; true energy error about 0.2). On new 45-degree loads for the same 174 held-out cases, E transfers only partly: its field error after seven corrections is 0.58, still below Residual-DCDM's 0.99 (ratio 0.58 [0.53, 0.65]), but its true residual rises to 2.6 times its starting value (Residual-DCDM: 1.37), and coarse weighting gives no gain over uniform weighting there (1.005 [0.997, 1.013]). These are post-hoc tests on one family.

While the controlled comparison with Residual-DCDM is valuable, the practical benefit would be clearer with rigorous benchmarking against classical solvers. The appendix includes CG, Jacobi-PCG, and AMG, but the wall-clock comparison is limited to one Darcy batch and does not measure the full pipeline, including warm-start inference. A cost-to-accuracy comparison across more problems, tolerances, and suitable numerical baselines would better establish when the learned method is competitive. I would place these results in the main body.

Response. We extended the comparison to three settings (Darcy 64264^2, Darcy 1282128^2, screened Poisson 64264^2; the same 32 cases each), targets on both field error and residual, batch sizes 32 and 1, and six methods: E-DCDM (E), Residual-DCDM, and CG and Jacobi-PCG, each from a zero and a neural warm start. Warm-start inference and the initial operator product are inside the timed region. We add CPU AMG (matrix access) as a stronger reference, and the elasticity family. In the revision this goes into the main body.

When E-DCDM is competitive. Against matrix-free solvers (operator products and diagonal only), E reaches every field-error and residual target we measured in 8 to 23 times fewer products than the faster of CG and Jacobi-PCG (15 to 23 times on every field-error target). To residual targets it is faster per case at batch 1 on all three settings and at batch 32 on both 64264^2 settings (a tie on Darcy 1282128^2). At an equal product budget it is far more accurate but 2.2 to 5.3 times slower. The margin shrinks with grid size: from 64264^2 to 1282128^2, E's products to field error 10−410^{-4} grow from 6 to 11, about as fast as Jacobi-PCG's (118 to 244), while eight E products go from 2.2 to 5.3 times the time of eight warm-start Jacobi-PCG products. So at batch 32 the residual-target speedup falls from 1.9x to a tie. We have not measured larger grids, and this trend does not support assuming a wall-clock gain there. With the assembled matrix, AMG is far more accurate than E on Darcy 1282128^2 (mean field error 2.53×10−52.53\times10^{-5} after eight V-cycles versus 1.40×10−31.40\times10^{-3} after eight E products, when half of E's cases are already below 10−310^{-3}) and reaches E's elasticity error in eight V-cycles. A V-cycle costs several products, so we do not rank the two by work; when the assembled matrix is available, our data show no advantage for E over AMG. Without it, geometric multigrid is the natural baseline on our structured grids; we have not run it. We did not compare with sparse direct solvers.

Operator products until 16 of 32 cases reach relative field error 10−310^{-3} (all 32 cases reach every target for every method)

ProblemEResidualbest CG / Jacobi-PCGCPU AMG (V-cycles)
Darcy 64264^25690not run
Darcy 1282128^28131835 (all 32 by 6)
Screened 64264^233504

At 10−210^{-2}: 3/6/2 products for E versus 59/127/30 for the best Krylov method. We did not time field targets directly. E's batch-32 time at its field-10−310^{-3} budget was measured (1.00, 4.10 and 0.49 ms per case); linearly extrapolating the fastest classical method's timings from its two nearest measured budgets to 90/183/50 products gives about 1.7, 5.6 and 0.9 ms, so we estimate E is 1.4 to 1.8 times faster to field error 10−310^{-3} at batch 32.

ms per case to relative residual 10−310^{-3} (E / Residual / fastest classical)

ProblemBatch 32, fixed budgetBatch 1, per-case stop
Darcy 64264^21.272 / 1.262 / 2.37212.358 / 12.433 / 76.570
Darcy 1282128^27.788 / 8.287 / 8.14327.617 / 29.650 / 165.723
Screened 64264^20.492 / 0.485 / 0.7726.348 / 6.327 / 22.027
  • Residual target (Darcy 1282128^2 uses E and Residual checkpoints trained at 1282128^2): E needs 7/15/3 products versus 124/268/56 for the fastest-timed classical method. At batch 32, E is faster on both 64264^2 settings; the 1282128^2 gap (4 to 5%) is a tie. The 3.5 to 6.2x batch-1 gap is single-case latency with a stopping check after every product, which favors methods that need few products.
  • AMG: screened Poisson reaches residual 10−310^{-3} in 4 cycles. We do not rank CPU V-cycles against GPU products by time.

Caveats: "fastest classical" is picked after measurement (zero-start Jacobi-PCG, except warm-start CG for screened Poisson at batch 1 and in the field-target estimate). Batch-32 times use the first fixed budget at which 16/32 cases reach the target. The field-target counts were measured after the timing study on the same cases, checkpoints and solver code. Upload, checkpoint loading and screened input normalization are not timed. We claim no general wall-clock advantage. The revision's supplement will add per-case times, every solver's settings and the timing scripts.

Reviewer_TYA3OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies training objectives for learned conjugate-direction solvers for symmetric positive-definite (SPD) linear systems. The proposed Energy-DCDM (E-DCDM) replaces the residual-based training loss with an exact one-step energy-decrease objective normalized by the error energy visible on a coarse space. The deployed solver retains the same A-orthogonalization, exact line search, and recurrent update, and training requires no solution labels. Through comparisons with matched network architectures, training states, and operator-product budgets, the authors investigate how the training objective affects solution accuracy. The primary experiments on 2D Darcy and screened-Poisson problems show lower field error (relative solution error). Spectral analyses, normalizer ablations, and supplementary 3D experiments further examine the gains and their scope.

Critical strengths

A well-defined training-objective improvement supported by targeted experiments. The authors introduce a coarse-normalized energy-decrease objective within an existing learned conjugate-direction framework. It requires no solution labels and improves field accuracy without changing the deployed update rule. Comparisons with matched networks, recurrent states, and operator-product budgets, together with spectral analyses and ablations, allow a reasonably clear assessment of the complete training objective. The study offers a concrete way to improve the training of learned iterative solvers and useful evidence about the relationship between training objectives and final solution accuracy.

Critical weaknesses

1. Coarse-space selection remains empirical, without a validated selection rule. Coarse normalization is an important component of the primary 2D gains, but the basis family, coarse-space dimension, and training-state distribution all affect the results. The paper compares different numbers of sine modes and a local-hat basis, yet does not establish a selection procedure that uses only development data and can be applied to new problems. The main 3D experiments use the generic coarse-free normalizer, and the exploratory 3D local-hat results do not establish a transferable selection rule for the primary normalization scheme. It therefore remains unclear how to configure the method reliably for a new operator family or error distribution. This limits the evidence for its transferability and practical applicability.

2. The primary gains lack confirmatory validation independent of method development. The authors disclose that the main method and primary comparison were designed after earlier methods had been scored on the same evaluation data. Selecting checkpoints using a separate validation set does not fully remove the potential selection effects arising from the use of evaluation results during method development. The 3D and resolution-transfer experiments provide supplementary evidence, but do not fully substitute for a fresh test-set evaluation of the primary 2D method. This weakens the primary results as independent evidence of generalization. Confirmation on new evaluation data that did not inform method development would substantially strengthen my confidence in the main conclusions.

Questions for authors

Q1 The paper already compares 8, 32, and 64 sine modes, as well as 32 local hats. For a new operator family and training-state distribution, can the authors provide an actionable procedure for choosing the basis family, coarse-space dimension, and normalizer using only development data? What evidence supports the stability of this procedure across problems or resolutions?

Q2 Can the authors verify the gains of the primary coarse-normalized loss over residual training on a fresh 2D test set that did not inform method development, with the models and analysis protocol fixed before evaluation?

Q3 The paper already reports training runtimes and time to residual tolerance on a fixed Darcy batch. Please clarify whether the training timings include training-state generation, coarse-space construction, normalizer precomputation, and validation, and report the full training cost relative to Residual-DCDM. At deployment, could the authors supplement the existing results with wall-clock comparisons against Residual-DCDM, CG, and Jacobi-PCG across different problems, problem sizes, and batches?

4 comments

The paper already compares 8, 32, and 64 sine modes, as well as 32 local hats. For a new operator family and training-state distribution, can the authors provide an actionable procedure for choosing the basis family, coarse-space dimension, and normalizer using only development data? What evidence supports the stability of this procedure across problems or resolutions?

Response. Partly. The procedure below uses only development data (solved development operators, no test data). Within one PDE family it passed one frozen-rule test at a new grid (Darcy 1282128^2), but only after being rerun there: a choice made at one grid did not carry to another, and we have no label-free or cross-family rule.

On 2D Darcy, having a coarse space matters more than which one. On 256 fresh Darcy masks (sweep set after the main fresh result), every coarse-space candidate (seven: sine 8/32/64 or local-hat 32, raw or mean-matched; sine-32 raw only) has field error between 5.1×10−75.1\times10^{-7} and 7.7×10−77.7\times10^{-7} after eight products, versus about 6.3×10−66.3\times10^{-6} for Residual-DCDM. Only the coarse-free steepest-descent normalizer (1.11.1 to 1.2×10−51.2\times10^{-5}) is worse than Residual. So coarse weighting gives 8 to 12x over Residual-DCDM; the choice of coarse space, at most 1.5x. A post hoc sweep on these fresh tests (3 seeds) found 4 modes enough, 1 not: relative to sine-32, the 4 lowest sine modes give 0.95 (Darcy) and 0.67 (screened Poisson), one mode 2.3 and 4.2, coarse-free 18 and 6.6. This need not carry over to other families. On elasticity (our reply to Reviewer He21), coarse weighting beat uniform-weight energy training by 4% on the original loads and not at all on new loads (1.005 [0.997, 1.013]). Our 3D runs use the coarse-free normalizer; an exploratory 64-hat coarse one changed field error by -19% to +7% over 1 to 16 corrections.

Procedure (written down after the candidates were trained but before any of its scores were computed; needs solved development cases; 27 trainings: 9 candidates × 3 seeds):

  1. Train each candidate (basis family, size, and scale) with three seeds on the same training states. "Raw" uses 1/Z1/Z as is; "mean-matched" rescales it to the training-split mean of the default normalizer.
  2. On solved development operators at the target grid, score ∥e7∥2/∥e0∥2\|e_7\|_2/\|e_0\|_2, averaged within and then across warm-start architectures; take the median over seeds.
  3. Pick the lowest score. Candidates within 1% count as tied; no tie occurred.

Results.

  • Chosen: sine-8/mean-matched (Darcy 64264^2) and local-hat-32/mean-matched (screened Poisson). Resampling half the development operators reproduced the choice in 773/1,000 (Darcy) and 804/1,000 (screened) subsets; every alternative pick was a scale change or sine-8 (Darcy: sine-8/raw 227; screened: sine-8/raw 167, local-hat-32/raw 29). These subsets reuse development data, so they are not independent validation.
  • On new masks (screened Poisson: unused public masks, reused forcings), the selected normalizer's field error was 0.830 (Darcy) and 0.682 (screened) times sine-32/raw's. Specified after the choice.
  • Failure across resolution. On the same 128 masks, the 64264^2 choice has 0.793 times sine-32/raw's field error at 64264^2 but 1.021 times at 1282128^2, where it ranks sixth of nine.
  • Recalibration at the new grid. Rerunning the same procedure at 1282128^2 (rescoring the nine existing 64264^2-trained candidates on solved 1282128^2 development operators; no retraining), with the rule and data frozen before 256 new 1282128^2 masks were prepared or scored, chose sine-64/raw. On those masks it scored 0.918 [0.899, 0.937] against sine-32/raw and ranked first of nine, with no other candidate within 1%; the 64264^2 choice again ranked sixth. So the procedure has passed one test on masks prepared after its choice was frozen. The rule was written after the 1282128^2 failure above, whose masks became its development data, so we treat this study as secondary; it is one test at one grid, after recalibration.
  • A label-free rule we tried (seven-step energy decrease) did not separate candidates.
  • Without solved cases, use the sine-32/raw default (fixed before all our new tests, though chosen during development on the original evaluation data): on every new 2D test set we ran it beat Residual-DCDM on field error (its field error was 0.098 to 0.120 times Residual-DCDM's across seeds on fresh Darcy 64264^2, 0.096 for the selected pair on fresh screened Poisson). This holds in these two families only; elasticity and 3D use other normalizers.
  • The objective gain survived the selection failure in our one test: at 1282128^2, the original sine-32/raw E-DCDM still has lower field error: 0.390 [0.369, 0.412] times Residual-DCDM's (one seed pair, 16/16 warm-start networks; energy error 0.437 times). True residual favors Residual-DCDM there (E's is 1.15 times). This check is post hoc; its interval excludes training-seed variation.

Can the authors verify the gains of the primary coarse-normalized loss over residual training on a fresh 2D test set that did not inform method development, with the models and analysis protocol fixed before evaluation?

Response. Yes, for both 2D families. Both favor E-DCDM (E) on every seed and every warm-start network.

Darcy. We fixed the original E-DCDM (sine-32/raw) and Residual-DCDM checkpoints, the solver, the metrics and the analysis, and only then generated 256 new Darcy masks. After eight operator products, E's mean field error is 0.120, 0.098 and 0.098 times Residual-DCDM's for the three paired training seeds (geometric mean over 16 warm-start networks). For every seed, all 16 networks favor E. E's energy-norm error is 0.14 to 0.17 times Residual-DCDM's. For the validation-selected seed (0.098), the 95% interval is [0.093, 0.104]; it resamples masks with checkpoints fixed and does not include training-seed variation, which the three seeds show directly.

Caveats:

  • The masks come from our Python reimplementation of the public Gaussian-field recipe. It matches the public data on descriptive checks but is not byte-identical to the original MATLAB generator. All 256 masks are unique, and none exactly matches a mask in the public training set (5,000), the public resolution-test set (1,000) or our earlier evaluation set (1,000).
  • One procedural amendment: for one warm-start network, a record-keeping step failed after its metrics were computed. We fixed only that step, discarded the partial output without inspecting it, and rescored. Masks, checkpoints, solver, metrics and analysis did not change.
  • This tests the already-developed method on new data. It cannot undo the earlier use of evaluation results during development.

Screened Poisson. Before scoring any model, we fixed the validation-selected checkpoints of both objectives and the analysis, then generated new forcings from the same distribution for 128 public masks not used in any earlier screened-Poisson follow-up. E's field error is 0.0960 [0.0916, 0.1005] times Residual-DCDM's (geometric mean over four warm-start groups: FNO or U-Net warm start × in- or out-of-distribution forcing). All four groups, all 512 paired inputs and all three same-seed comparisons favor E. Absolute E field errors are already 1.6×10−101.6\times10^{-10} to 2.4×10−92.4\times10^{-9} (group means), so this is a ranking check rather than a practical accuracy claim.

Records. For each test, a manifest with SHA-256 hashes of the checkpoints, written protocol, scripts and analysis code was saved before any model was scored (Darcy: before the masks were generated). The revision's supplement will include these manifests, protocols and per-case results.

The paper already reports training runtimes and time to residual tolerance on a fixed Darcy batch. Please clarify whether the training timings include training-state generation, coarse-space construction, normalizer precomputation, and validation, and report the full training cost relative to Residual-DCDM.

Response. The submitted timers start after the shared training states are loaded. They include correction training, validation and checkpoint saving. They exclude state generation, coarse-space construction, warm-start training and setup.

To answer the question directly, we re-ran the paired correction pipeline on one Quadro RTX 8000: one shared build and three E-DCDM (E) and three Residual-DCDM trainings per PDE. The shared build generates the recurrent training states, the coarse basis and the normalizer ZZ.

PDEShared build3 E trainings3 Residual trainingsE time ÷ Residual time
Darcy18.9 s5902.2 s5966.7 s0.989
Screened Poisson6.8 s1403.1 s1400.3 s1.002

Which costs are objective-specific. Only the energy loss reads the normalizer; the Residual loss does not use ZZ, the coarse basis or the Gram matrices. For screened Poisson the coarse Gram matrices V⊤AVV^\top A V are formed inside the timed shared build; for Darcy the build reads them from disk. We re-timed the Darcy construction from scratch: 21.7 s on one CPU thread for all 1,600 training and validation rows (matching the stored matrices to 5×10−155\times10^{-15}). As a conservative bound, we charge the entire shared build plus the Gram construction to E, although Residual also needs the recurrent states. This adds 40.6 s (Darcy) and 6.8 s (screened) per E model, which is 2.1% and 1.5% of one E training (1967 s and 468 s). The remaining unmeasured steps (training the warm-start and other pretrained networks, computing the initial warm-start predictions, data download and environment setup) are identical for both objectives, because both losses train on the same warm starts and recurrent states.

Answer. With all normalizer work charged to E, training one E model costs 1.01 times Residual-DCDM on Darcy and 1.02 times on screened Poisson. Shared upstream costs move these ratios toward one. Timing ran on a shared host, so differences of about 1% are within timing noise. If the normalizer is chosen by the calibration procedure in our reply to the first question, training is multiplied by the number of candidates (27 trainings instead of three); with the default sine-32 normalizer no search is needed. Recalibrating at a new grid reuses the trained candidates (no retraining) but needs direct solves of the development operators, which we did not time.

At deployment, could the authors supplement the existing results with wall-clock comparisons against Residual-DCDM, CG, and Jacobi-PCG across different problems, problem sizes, and batches?

Response. Yes. We timed E-DCDM (E), Residual-DCDM, CG and Jacobi-PCG (each with zero and neural warm starts) on GPU for three settings (Darcy 64264^2, Darcy 1282128^2, screened Poisson 64264^2), at batch 32 and at batch 1, on the same 32 cases per setting. Neural warm-start inference is inside the timed region.

Summary.

  • Equal products: after eight products, E is far more accurate than warm-start Jacobi-PCG but 2.2 to 5.3 times slower per case, because each learned product includes a network evaluation.
  • Equal residual target (10−310^{-3}): E needs 7/15/3 products versus 124/268/56 for the fastest classical method. At batch 32, E takes 1.272 vs 2.372 ms per case on Darcy 64264^2 and 0.492 vs 0.772 on screened Poisson; on Darcy 1282128^2 the 4 to 5% gap (7.788 vs 8.143) is a tie. Darcy 1282128^2 uses E and Residual checkpoints trained at 1282128^2 (the 0.390 resolution check in another reply instead applies 64264^2-trained checkpoints without retraining). At batch 1 with per-case stopping the gap widens to 3.5 to 6.2x; this is a single-case latency regime with a stopping check after every product, which favors methods that need few products.
  • Equal field-error target (10−310^{-3}): on the same cases, E needs 5/8/3 products (first budget at which 16 of 32 cases reach it) versus 90/183/50 for the best of CG and Jacobi-PCG; all methods reach the target on all 32 cases. Residual-DCDM needs 6/13/3. We did not time field targets directly; extrapolating the fastest classical method's measured times to 90/183/50 products, we estimate E is 1.7/1.4/1.8 times faster to field error 10−310^{-3} at batch 32 (1.00/4.10/0.49 versus about 1.7/5.6/0.9 ms).
  • E versus Residual: nearly identical time per product, so the accuracy gain is not paid for at deployment.
  • AMG (CPU, with matrix access) is stronger: on the same 32 Darcy 1282128^2 cases, every case reaches field error 10−310^{-3} within six V-cycles and 27 reach residual 10−310^{-3} within seven; after eight V-cycles mean field error is 2.53×10−52.53\times10^{-5} versus 1.40×10−31.40\times10^{-3} for E after eight products. Screened Poisson reaches residual 10−310^{-3} in four V-cycles. We claim no general wall-clock advantage over matrix-aware solvers.

ms per case to relative residual 10−310^{-3} (E / Residual / fastest classical)

ProblemBatch 32, fixed budgetBatch 1, per-case stop
Darcy 64264^21.272 / 1.262 / 2.37212.358 / 12.433 / 76.570
Darcy 1282128^27.788 / 8.287 / 8.14327.617 / 29.650 / 165.723
Screened 64264^20.492 / 0.485 / 0.7726.348 / 6.327 / 22.027

After eight products on the 32 timing cases (mean field error / ms per case, batch 32)

ProblemEResidual-DCDMWarm-start Jacobi-PCG
Darcy 64264^26.67×10−76.67\times10^{-7} / 1.4127.35×10−67.35\times10^{-6} / 1.3930.0958 / 0.655
Darcy 1282128^21.40×10−31.40\times10^{-3} / 4.0982.92×10−22.92\times10^{-2} / 4.0970.1225 / 0.775
Screened 64264^21.38×10−101.38\times10^{-10} / 1.1421.43×10−91.43\times10^{-9} / 1.1580.0933 / 0.334

Caveats:

  • "Fastest classical" is chosen after measurement: zero-start Jacobi-PCG, except warm-start CG for screened Poisson at batch 1 and in the field-target estimate.
  • Batch-32 times are at the first fixed budget where 16/32 cases reach the target, not per-case stopping. Every method reached residual 10−210^{-2} and 10−310^{-3} on all 32 cases at batch 1.
  • Problem size varies only on Darcy (64264^2 and 1282128^2).
  • Data upload, checkpoint loading and screened input normalization are not timed.

The revision will add the 10−210^{-2} results and all methods.

Reviewer_8z3sOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes E-DCDM, replacing the residual-based loss in DCDM with an energy-decrease loss. The authors argue that a smaller residual does not necessarily mean a smaller solution error. The proposed loss uses the energy decrease at each correction step, normalized by a coarse-space estimate. Experiments on Darcy and screened Poisson problems show lower solution errors than residual-based DCDM. The paper also examines which error components are reduced and compares different normalization methods.

Critical strengths

Strengths 1: The DCDM architecture and update procedure remain unchanged, and both losses are trained on the same states. This allows a direct comparison of the training objectives.

Strengths 2: The paper includes an analysis of error reduction across different eigenvalues and ablation studies on the normalization methods.

Critical weaknesses

Weaknesses 1: The choice of coarse-space normalization needs further justification. The energy-based loss performs worse without it in the 2D experiments, and using 8 basis modes gives better results than the default 32. The paper does not establish a clear basis for choosing these settings.

Weaknesses 2: The proposed 2D normalization scheme needs further validation. The main 2D comparisons were designed after earlier evaluations on the same test data, while the 3D experiments use a different normalization method.

Questions for authors

Questions 1: Why does coarse-space normalization improve the energy-based loss so substantially, when the loss without it performs worse than the residual baseline?

Questions 2: How should the number of coarse basis modes be chosen for a new problem, given that 8 modes performed better than 32?

2 comments

The proposed 2D normalization scheme needs further validation. The main 2D comparisons were designed after earlier evaluations on the same test data, while the 3D experiments use a different normalization method.

Response. We agree this was the main gap, and we ran fresh tests in which nothing could be tuned on the evaluation data. Both favor E-DCDM (E) with the submitted 2D normalizer on every seed and every warm-start network.

Fresh Darcy test. We fixed the original E-DCDM (sine-32) and Residual-DCDM checkpoints, the solver, the metrics and the analysis, and only then generated 256 new Darcy masks. After eight operator products, E's mean field error (distance to the exact solution) is 0.120, 0.098 and 0.098 times Residual-DCDM's for the three paired training seeds (geometric mean over 16 warm-start networks), i.e. 8 to 10x lower error, and all 16 networks favor E for every seed. For the validation-selected seed (0.098) the 95% interval is [0.093, 0.104] (masks resampled, checkpoints fixed). None of the new masks exactly matches a mask in the public training set, the public resolution-test set or our original evaluation set. The masks come from our Python reimplementation of the public Gaussian-field recipe, which matches the public data on descriptive checks but is not byte-identical to the original generator.

Fresh screened Poisson test. Before scoring any model, we fixed the validation-selected checkpoints of both objectives and the analysis, then generated new forcings from the same distribution for 128 public masks not used in any earlier screened-Poisson follow-up. E's field error is 0.0960 [0.0916, 0.1005] times Residual-DCDM's, about 10x lower; all four warm-start groups, all 512 paired inputs and all three same-seed comparisons favor E. Absolute E field errors are already 1.6×10−101.6\times10^{-10} to 2.4×10−92.4\times10^{-9} (group means), so this is a ranking check rather than a practical accuracy claim.

Resolution (secondary; specified after the fresh 64264^2 result). On 128 new masks at 1282128^2, the same sine-32 E-DCDM and Residual-DCDM checkpoints, trained at 64264^2 and applied without retraining, still favor E: its field error is 0.390 [0.369, 0.412] times Residual-DCDM's (about 2.6x lower), one seed pair, all 16 warm-start networks favoring E.

3D. You are right: the main 3D runs use a coarse-free (steepest-descent) normalizer, so they test the energy objective with a different weighting, not the 2D sine-32 scheme; the fresh 2D tests above are the direct validation of that scheme. Sec. 5.3 also reports an exploratory follow-up with a 64-local-hat coarse normalizer, specified after the main 3D results and reported under a report-if-favorable rule that the section states; on the paper's fixed one-step test states (not the rollouts below, where it is 39% at one correction) it lowers field error 48% versus Residual-DCDM and 7% versus coarse-free. Post hoc, from the existing evaluation traces (rollouts from a zero start; 47 held-out ModelNet10 operators, five seeds), field error relative to Residual-DCDM:

Corrections1471016
Coarse-free0.650.210.240.280.35
64 hats, raw (1/Z1/Z)0.610.170.220.300.34
64 hats, mean-matched0.610.190.210.260.28

Mean-matched rescales to the default normalizer's mean. Coarse-free wins on 46/47 operators at 1 correction, 47/47 after; mean-matched hats 47/47 always. So in 3D the energy objective beats Residual-DCDM at every budget with either normalizer; relative to coarse-free, 64 hats range from 16% lower to 7% higher field error (raw) or 5 to 19% lower (mean-matched), far less than in 2D, where coarse weighting decides whether energy training beats Residual-DCDM at all. The training-state weights fit this partly. Grouping states by correction step and starting-guess source, the coarse denominator's group medians span 112 to 275x in 2D but only 5.6x for the 3D hats, close to the coarse-free denominator's 5.5x (2.1 to 2.3x in 2D); within groups the hats still spread 3.5 to 6.0x versus 1.1 to 2.0x, so in 3D the two weightings differ, but far less than in 2D. Elasticity does not fit: its polynomial coarse denominator spans 4,780x across groups (largest between the two starting-guess sources), yet in post-hoc controls coarse weighting helps only 4% on the original loads and not at all on new loads. Weight spread alone is therefore not a rule for when coarse weighting is needed, and we do not claim one. The revision will say this in the 3D section and will not present 3D as support for the 2D normalizer.

These tests check the already-developed method on new data; they cannot undo the earlier use of evaluation results during development. The revision's appendix will give each test's protocol, per-seed results and scripts.

The choice of coarse-space normalization needs further justification. The energy-based loss performs worse without it in the 2D experiments, and using 8 basis modes gives better results than the default 32. The paper does not establish a clear basis for choosing these settings.

Response. We agree the 32-mode default was inherited, not derived. On our 2D families, whether to use a coarse normalizer matters a lot; which coarse space matters much less. On 256 fresh Darcy masks (sweep set after the main fresh test), all seven coarse-space candidates we trained (sine 8/32/64 or 32 hats, raw or mean-matched; sine-32 raw only) have field error between 5.1×10−75.1\times10^{-7} and 7.7×10−77.7\times10^{-7} after eight products, vs about 6.3×10−66.3\times10^{-6} for Residual-DCDM. Only coarse-free (1.11.1 to 1.2×10−51.2\times10^{-5}) is worse. On linear elasticity, post-hoc controls show coarse weighting beat uniform by only 4% on the original held-out loads (0.956 [0.947, 0.965]; 1 to 9% by seed) and not at all on new loads (1.005 [0.997, 1.013]), so we do not claim it is always needed.

Why does coarse-space normalization improve the energy-based loss so substantially, when the loss without it performs worse than the residual baseline?

Response. For coarse modes VV, let G=V⊤AVG=V^\top AV and g=V⊤r~g=V^\top\tilde r, where r~=r/rms⁡(r)\tilde r=r/\operatorname{rms}(r) is the normalized residual of a training state. Then Z=g⊤G−1gZ=g^\top G^{-1}g is proportional to the energy decrease the best coarse correction would achieve on that state. ZZ does not depend on the network's output, so dividing by ZZ changes only how much each training state counts. On Darcy training states, the median ZZ per correction step and starting-guess source ranges from about 0.011 to 3.0 (275x); within a group, 5th to 95th percentile spans 3.9 to 15x. Coarse-free weighting normalizes each state, but relative to coarse ZZ it misweights whole groups by up to 126x on Darcy; dividing by ZZ asks for a fixed fraction of the coarse optimum on every state.

A new control tests this. We trained the energy loss with the coarse-free weight times one constant per (correction step, starting-guess source) group, matching the coarse ZZ group medians on training states (three seeds per PDE). Median field error:

WeightingDarcy (256 fresh)Screened (128 new, reused forcings)
Coarse-free1.1×10−51.1\times10^{-5}5.9×10−95.9\times10^{-9}
Group-reweighted1.6×10−61.6\times10^{-6}1.6×10−91.6\times10^{-9}
Sine-32 (per state)6.2×10−76.2\times10^{-7}8.9×10−108.9\times10^{-10}

Group reweighting alone beats Residual-DCDM on Darcy (6.3×10−66.3\times10^{-6}) and recovers about two thirds of the log gap, but per-state coarse weighting is still 2.6x (Darcy) and 1.8x (screened) better and wins 93% and 86% of paired cases. The weighting suits the energy loss: in the paper's factorial, 1/Z1/Z on the residual loss helps Darcy but hurts screened Poisson; energy with 1/Z1/Z is best on both.

How should the number of coarse basis modes be chosen for a new problem, given that 8 modes performed better than 32?

Response. Few smooth modes suffice. A post-hoc sweep (3 seeds per PDE, same fresh tests) tried 1 and 4 sine modes. Median field error relative to sine-32:

Coarse spaceDarcyScreened
4 sine modes0.950.67
1 sine mode2.34.2
Coarse-free186.6

4 modes win 61 to 70% (Darcy) and 66 to 87% (screened) of paired cases per seed; with 1 mode one screened seed trails coarse-free. A sine-32 retrain moved 1.33x: smaller gaps are noise. To tune further with solved development cases at the target grid: train nine candidates (the seven above, two coarse-free) with three seeds each, score field error after eight products on development operators, and pick the lowest median. The rule was fixed before any of its scores existed. It chose sine-8 (mean-matched) for Darcy 64264^2 and 32 local hats (mean-matched) for screened Poisson; on new masks (screened: reused forcings), checked after the choice, these reduced field error to 0.830 and 0.682 of sine-32.

A choice need not transfer across grids: on 128 further new masks, the 64264^2 choice has 0.793 times sine-32's field error at 64264^2 but 1.021 times at 1282128^2. Rerun at 1282128^2 with the rule fixed before new data were prepared, it chose sine-64 (raw; so 8 beating 32 is grid-specific), at 0.918 of sine-32's error on 256 new 1282128^2 masks ([0.899, 0.937]). So, for these two 2D families: use a coarse normalizer with 4 or more smooth modes (two families only; default sine-32 beat Residual-DCDM on every new 2D test); with solved cases at the target grid, the procedure gave a further 8% reduction in its one test with the rule fixed before the data existed (1282128^2; secondary, since its development data came from the failed transfer) and 17 to 32% in checks specified after the choice (64264^2). The revision will add the sweep and the group control, per seed.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.