- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes learning the tangent operator of a PDE directly, from cheap and noisy directional labels produced by Walk on Spheres [1]. The CALTO architecture separates a nonlinear encoding of the PDE context from a linear pathway for the perturbation , so the model is exactly linear in . On four 2D elliptic problems and a 3D problem on CAD geometries, the error is 51-75% lower than that of the best adapted baseline. The paper also demonstrates an application to coefficient identification.
Critical strengths
-
- The inductive bias is clear and well motivated: the tangent operator is linear in , and the model is exactly linear in . Labels generated with shared random realizations preserve the same linear relations, so the model structure and the supervision are consistent with each other. A cheap full Jacobian and VJPs follow directly (Table 4 and Table 14).
-
- The applied experiment in Table 5b is well chosen. Correcting the local derivative of SC-FNO [2] and DE-DeepONet surrogates with CALTO reduces the final residual by about 99%. The analysis of the supervision budget is also useful: at fixed , covering more contexts matters more than adding directions per context.
Critical weaknesses
-
- The practical motivation is not yet supported quantitatively. The main argument is that the labels are cheap, but the paper does not compare the cost of generating them (a coarse quadratic (P2) finite-element base state plus stochastic walks) with solving the tangent equation directly by finite elements. For 2D elliptic problems on a fixed geometry, that is one linear solve with an already factorized matrix. In the inverse-problem experiment, finite elements are used for the forward evaluations anyway, and exact tangents give a result two orders of magnitude better . Moreover, checkpoints are selected using high-accuracy validation targets, which partly weakens the claim of learning from weak labels alone.
-
- The contribution relative to Neural Green's Operators [3] seems incremental: the same idea of nonlinear conditioning with a linear action is applied to the tangent equation, and for SP-D/SP-DN with data perturbations the tangent operator essentially coincides with the Green's operator. At the same time, the gap between the linear and nonlinear CALTO variants is small (3.03 vs. 3.17; 3.49 vs. 3.97), so linearity is unlikely to be the main source of the gain over the baselines. It would help to describe the baseline adaptation in more detail. SC-FNO [2], DINO [4] and DE-DeepONet are trained on JVPs only, without solution values, which is unusual for these models, and the information on parameter counts and hyperparameter tuning is limited. Generalization to new geometries remains a weak point (21% on SP-D, 42% on SL-RD).
References:
-
- B.Miller, R.Sawhney, K.Crane, I.Gkioulekas. Differential Walk on Spheres. // ACM Transactions on Graphics, 43(6), 2024.
-
- A.Behroozi, C.Shen, D.Kifer. *Sensitivity-Constrained Fourier Neural Operators for Forward and Inverse Problems in Parametric Differential Equations.} // International Conference on Learning Representations (ICLR), 2025.
-
- H.Melchers, J.Prins, M.Abdelmalik. Neural Green's Operators for Parametric Partial Differential Equations. // Computer Methods in Applied Mechanics and Engineering, 455:118893, 2026.
-
- T.O'Leary-Roseberry, P.Chen, U.Villa, O.Ghattas. Derivative-Informed Neural Operator: An Efficient Framework for High-Dimensional Parametric Derivative Learning. // Journal of Computational Physics, 496:112555, 2024.
Questions for authors
-
- Could you report the wall-clock time for generating the label set, compared with computing tangents by finite elements, and indicate the number of queries at which CALTO pays off?
-
- How do the results change if checkpoints are selected using only weak validation labels?
-
- What are the parameter counts of the baselines, and how were their hyperparameters tuned in the JVP-only training setting?
-
- How does the method scale with the parameter dimension and how sensitive are the SL-RD results to the quality of the coarse state ?
6 comments
Reply to W2 (rest: source of the gain and new geometries)
Source of the gain. Paper Table 3 isolates exactness inside an architecture that already conditions the operator on the context alone. When directional observations are few (R ≤ 8 with four-walk labels), the linear model is lower by 0.15–0.20 points (three seeds), with more directions or cleaner labels the two variants agree. In that case, the value of exactness lies elsewhere: on directions scaled 16×, CALTO keeps 3.19% mean error while the nonlinear variant reaches 11.60% (Appendix E.1). The larger gap to the baselines comes mainly from how the context is used: with exact linearity and parameter count held fixed (Table 13), removing the joint query–source dependence raises error by 33–51% and removing physical information from the gates by 8–22%. The gap persists when every model is trained on strong FEM tangents (ID error 52–76% lower in all four families). The gain thus has several sources rather than a single one.
New geometries. CALTO has the lowest geometry-OOD error in every family (e.g., 17.73% vs 20.52% on SP-DN, 30.72% vs 48.02% on VC-DR). Absolute errors are high because training domains have one outer boundary, whereas geometry OOD adds holes, each an interior Dirichlet boundary absent from training. This is a coverage limit, not a label-precision limit: retraining on FEM tangents cuts ID error by 37–58% but leaves geometry-OOD error nearly unchanged, and higher on VC-DR () and SL-RD (). Adding just 32 weakly labelled new contexts nearly halves the SP-DN perforated-domain error (), so cheap labels extend coverage efficiently.
Reply to W2 (baseline part: adaptation, parameters, tuning) and Q3
Adaptation. The baselines were initially built to predict solutions or fields, so each needs a minimal adaptation to predict the tangent at the queries from weak JVP labels. SC-FNO, DE-DeepONet and DINO keep their upstream backbones, and we differentiate their predicted solution at the queries along the normalized physical direction (geometry and queries fixed); GINO, Transolver and Transolver++ take the context and perturbation as pointwise inputs and predict the tangent directly. All methods share the inputs, W4 labels, optimizer, batch size, update budget and validation split, so the comparison isolates the model, not the supervision. JVP-only training is the favourable setting for the state surrogates: adding primal supervision raises their tangent error on all four benchmarks (e.g., DE-DeepONet on SP-D: ).
Parameters. CALTO has about 142k trainable parameters: more than SC-FNO (109k) and DINO (31k), fewer than DE-DeepONet (250k–773k), and fewer than the strongest baseline of each family (Transolver 757k–758k on SP-D and VC-DR, Transolver++ 644k on SP-DN, GINO 1.38M on SL-RD). Capacity does not explain the gap. Paper Table 10 gives more detailed information about parameters.
Tuning. We only tune the learning rate. We keep each baseline's official architecture configuration (e.g., SC-FNO: four Fourier layers of width 20, eight modes per dimension; DINO: DIPNet 100-50 projection network, 50-dimensional output; GINO: latent modes), adapting only inputs and outputs. All methods share AdamW (weight decay ), cosine decay and update count (12,500 on SP-D/SP-DN, 3,200 on VC-DR/SL-RD). In SP-D, we trained each model for 200 updates on 320 subset training contexts at 0.001, 0.005 and 0.01 and selected checkpoints by strong-validation error. For example, DINO, the best validation L2RE was 31.50%, 29.54% and 53.05%; 0.005 was also best for DE-DeepONet and Transolver(++), and for SC-FNO the best value, 0.01, was only 0.17 points below 0.005. For CALTO it was 11.44%, 8.21% and 7.95%, so the shared learning rate 0.005 is not CALTO's best either (0.01 is 0.26 points lower): we did not tune it in CALTO's favour. Appendix E.4 describes the tangent route of each baseline.
Reply to Q4
Parameter dimension. The response branch takes perturbation fields at the support points as input, so model size and the cost of one JVP do not depend on . Training needs only observed directions per context: with on SP-DN (), CALTO recovers the full Jacobian with 3.49% error. The full Jacobian costs evaluations sharing one context encoding (2.11–2.27 ms at , vs 9.86–16.86 ms for SC-FNO), and each VJP is a matrix–vector product. We tested up to ; for field-valued parameters a direction is itself a field, so cost grows only with the number of columns requested.
Coarse state in SL-RD. We froze the model and replaced its coarse state at inference by P2 solutions at , 0.2 and 0.8, updating all derived inputs. Mean error changes by at most 0.03 points on every split (Table 1). Even at the state is within about 1% of the solution, and the tangent coefficient and forcing change by about 0.002% and 0.02%. The model does respond to the state: a discretization-shaped perturbation of 5–50% of the state norm raises ID error from 8.30% to 17.8–19.6% at 50% (Table 2). SL-RD results are thus insensitive to the coarse state over the accuracy coarse P2 solves deliver.
Table 1: frozen SL-RD model with genuine P2 coarse states at inference, L2RE (%)
| Split (contexts) | 0.08 (training) | 0.2 | 0.8 | |
|---|---|---|---|---|
| ID (256) | 8.2978 | 8.2982 | 8.2992 | 8.3044 |
| Geometry OOD (128) | 41.7430 | 41.7414 | 41.7406 | 41.7560 |
| Nonlinearity OOD (128) | 11.5915 | 11.5912 | 11.5682 | 11.5653 |
| State error vs , ID / geo. OOD (%) | 0 | 0.07 / 0.10 | 0.68 / 0.61 | 1.19 / 0.87 |
Table 2: same model, state perturbed along its discretization error, L2RE (%), positive / negative sign
| Perturbation (% of state norm) | 0 | 5 | 10 | 20 | 50 |
|---|---|---|---|---|---|
| ID | 8.30 | 8.58 / 8.65 | 9.30 / 9.49 | 11.17 / 11.83 | 17.83 / 19.59 |
| Geometry OOD | 41.74 | 41.91 / 41.85 | 42.33 / 42.22 | 43.56 / 43.48 | 47.71 / 48.69 |
Thank you for asking for precise cost accounting. We answer with direct measurements on SP-DN and VC-DR, the family of the inverse problem. A query below is one new-context request returning eight directions at 1,024 points.
Labels. Each context changes the operator, so FEM must mesh, assemble and factorize a new system for it. On one Intel Core i7-12700 core, W4 including its coarse P2 base state costs 0.80 s (SP-DN) and 0.26 s (VC-DR) per context, against 2.28 s and 2.31 s for FEM including meshing. For whole training sets both generators run on eight parallel workers, one per core: 14.3 vs 70.3 min on SP-DN (4,000 contexts) and 1.1 vs 12.1 min on VC-DR (1,024 contexts).
Factorized FEM. Reusing the factorization of a fixed problem, FEM still needs 0.85 s and 0.74 s for the same request; CALTO answers in 1.56 ms and 1.53 ms on a GPU.
Inverse problem. Both Jacobian sources use the same FEM forward model, so only the tangent differs. Until the sensor residual is halved, the cumulative six-direction Jacobian costs 1.246 s with FEM (one core, forward factorization reused) and 5.62 ms with CALTO; total time including forward solves is 4.64 s vs 3.48 s. The 1.7×10−4 belongs to CALTO-Hybrid, and FEM's lower final residual (2.6×10−6) reflects that the final Gauss–Newton contraction is limited by tangent accuracy. Without FEM tangents, CALTO reduces the final residual of learned primal surrogates by 98.9–99.5% (paper Table 5b).
Break-even. CALTO's one-time cost is label generation plus 100-epoch training on one GPU: 14.3 + 51.8 min ≈ 1.1 h on SP-DN and 1.1 + 29.7 min ≈ 31 min on VC-DR (include data loading). A new context then costs about 1.5 ms, against 2.3 s for FEM on one core, so the one-time cost equals FEM's cost for 1,740 (SP-DN) and 800 (VC-DR) new contexts, or 4,670 and 2,490 if FEM reuses a factorization (0.85 s and 0.74 s per request). After break-even, CALTO answers each new context about 1,500× faster than FEM, and about 500× faster than FEM with a reused factorization.
Weak validation only (W1, Q2). For CALTO and each family's strongest baseline we applied two selection rules to the same 100 epoch-wise checkpoints, W4 validation observations only vs strong validation responses, and scored both on the same strong test references. CALTO is unchanged: on SP-D and SL-RD both rules pick the same checkpoint, and over all ID splits test error moves by at most 0.013 points (SP-DN 3.0276). For Transolver(++) the change is at most 0.05 points; GINO on SL-RD is 0.46 points better with W4 selection. CALTO is lower on every split under either rule. W4 observations are unbiased estimates of the same tangent, so the weak validation loss differs from the strong one by a noise term essentially independent of the checkpoint, averaged over the responses. Training and selection need only weak labels.
NGO (W2). We credit NGO for one principle, nonlinear conditioning with a linear action, and CALTO follows it no further. For pure data perturbations of SP-D and SP-DN the reviewer is right that the tangent is a Green's action on the perturbed data, δu=G[θ]!(δf,δg). The learning problem still differs from NGO's: the domain and, in SP-DN, the Dirichlet/Neumann partition change with every context (NGO lists this as future work), and labels are mesh-free four-path estimates, not FEM solutions. NGO expands G in a fixed basis on a fixed domain, whereas CALTO is a point-set operator with no basis, mesh or system matrix. Its network follows the tangent equation Lcδu=Bcv: perturbation features at queries, interior supports and boundary supports; attention weights computed jointly from query and source contexts, independent of v; and context-generated gates on both sides of each linear kernel. The gates matter beyond data perturbations. A screening direction gives data scaled pointwise by the base state, −u0δλ, and a shape direction gives −Vn∂nu0; u0 is not an input, so this scaling must come from the context. For coefficient directions the tangent is the derivative of the Green's operator with respect to its own coefficients, a composition G∘B∘G rather than one Green's action, and in SL-RD the linearized operator itself depends on the state. Each design choice is tested with exact linearity and parameter count fixed (paper Table 13): removing the joint query–source dependence raises error by 33–51%, and removing physical information from the gates by 8–22%. The size of the gain over the baselines and the results on new geometries are addressed in the next reply.