acceptodds
Under review as a conference paper at ICLR 2027

Learning PDE Tangent Operators from Stochastic Walks

Abstract

Gradient-based optimization for PDE-constrained inverse problems and inverse design requires accurate solution sensitivities. These sensitivities can be approximated by differentiating a learned solution surrogate, but accurate solution predictions do not guarantee accurate derivatives. Learning sensitivities directly offers an alternative, although obtaining high-accuracy derivative supervision can be costly. We introduce the Context-Aware Linearity-preserving Tangent Operator (CALTO), which learns reusable tangent operators from sparse, noisy directional observations generated with few stochastic paths. Its nonlinear context encoder modulates both sides of linear kernels, with a bias-free readout. This enforces superposition consistent with shared-path observations, jointly constraining one linear operator per base problem. Across four 2D elliptic benchmarks and one 3D CAD diffusion–reaction benchmark, CALTO achieves 0.71–8.88% context-averaged relative error on held-out in-distribution instances, reducing error by 50.7–74.8% against the strongest learned baseline per benchmark. Comparisons with a nonlinear variant support directional linearity under sparse, noisy supervision; controlled coefficient identification further shows that CALTO tangents improve parameter recovery and reduce final residual ratios by 98.9–99.5% when correcting learned primal surrogates.

Primary area:
applications to physical sciences (physics, chemistry, biology, etc.)
Keywords:
Neural Operators, Scientific Machine Learning, Tangent Operator Learning, PDE Sensitivity Analysis, Weak Supervision

Reviews

Among:
Shape:
1234this paper, Top 17%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.75 over 4 reviews, above 83% of the others.
Reviewer_EPUZOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes learning the tangent operator (ϑ,v)↦Jϑv(\vartheta, v)\mapsto J_\vartheta v of a PDE directly, from cheap and noisy directional labels produced by Walk on Spheres [1]. The CALTO architecture separates a nonlinear encoding of the PDE context from a linear pathway for the perturbation vv, so the model is exactly linear in vv. On four 2D elliptic problems and a 3D problem on CAD geometries, the error is 51-75% lower than that of the best adapted baseline. The paper also demonstrates an application to coefficient identification.

Critical strengths
    1. The inductive bias is clear and well motivated: the tangent operator is linear in vv, and the model is exactly linear in vv. Labels generated with shared random realizations preserve the same linear relations, so the model structure and the supervision are consistent with each other. A cheap full Jacobian and VJPs follow directly (Table 4 and Table 14).
    1. The applied experiment in Table 5b is well chosen. Correcting the local derivative of SC-FNO [2] and DE-DeepONet surrogates with CALTO reduces the final residual by about 99%. The analysis of the supervision budget is also useful: at fixed N⋅RN\cdot R, covering more contexts matters more than adding directions per context.
Critical weaknesses
    1. The practical motivation is not yet supported quantitatively. The main argument is that the labels are cheap, but the paper does not compare the cost of generating them (a coarse quadratic (P2) finite-element base state plus stochastic walks) with solving the tangent equation directly by finite elements. For 2D elliptic problems on a fixed geometry, that is one linear solve with an already factorized matrix. In the inverse-problem experiment, finite elements are used for the forward evaluations anyway, and exact tangents give a result two orders of magnitude better (2.6⋅10−6vs.1.7⋅10−4)(2.6\cdot10^{-6} \quad vs. \quad 1.7\cdot10^{-4}). Moreover, checkpoints are selected using high-accuracy validation targets, which partly weakens the claim of learning from weak labels alone.
    1. The contribution relative to Neural Green's Operators [3] seems incremental: the same idea of nonlinear conditioning with a linear action is applied to the tangent equation, and for SP-D/SP-DN with data perturbations the tangent operator essentially coincides with the Green's operator. At the same time, the gap between the linear and nonlinear CALTO variants is small (3.03 vs. 3.17; 3.49 vs. 3.97), so linearity is unlikely to be the main source of the gain over the baselines. It would help to describe the baseline adaptation in more detail. SC-FNO [2], DINO [4] and DE-DeepONet are trained on JVPs only, without solution values, which is unusual for these models, and the information on parameter counts and hyperparameter tuning is limited. Generalization to new geometries remains a weak point (21% on SP-D, 42% on SL-RD).

References:

    1. B.Miller, R.Sawhney, K.Crane, I.Gkioulekas. Differential Walk on Spheres. // ACM Transactions on Graphics, 43(6), 2024.
    1. A.Behroozi, C.Shen, D.Kifer. *Sensitivity-Constrained Fourier Neural Operators for Forward and Inverse Problems in Parametric Differential Equations.} // International Conference on Learning Representations (ICLR), 2025.
    1. H.Melchers, J.Prins, M.Abdelmalik. Neural Green's Operators for Parametric Partial Differential Equations. // Computer Methods in Applied Mechanics and Engineering, 455:118893, 2026.
    1. T.O'Leary-Roseberry, P.Chen, U.Villa, O.Ghattas. Derivative-Informed Neural Operator: An Efficient Framework for High-Dimensional Parametric Derivative Learning. // Journal of Computational Physics, 496:112555, 2024.
Questions for authors
    1. Could you report the wall-clock time for generating the label set, compared with computing tangents by finite elements, and indicate the number of queries at which CALTO pays off?
    1. How do the results change if checkpoints are selected using only weak validation labels?
    1. What are the parameter counts of the baselines, and how were their hyperparameters tuned in the JVP-only training setting?
    1. How does the method scale with the parameter dimension dcd_c and how sensitive are the SL-RD results to the quality of the coarse state u~0\tilde u_0?
6 comments

Reply to W1 (cost, factorized FEM, inverse problem) and Q1

Thank you for asking for precise cost accounting. We answer with direct measurements on SP-DN and VC-DR, the family of the inverse problem. A query below is one new-context request returning eight directions at 1,024 points.

Labels. Each context changes the operator, so FEM must mesh, assemble and factorize a new system for it. On one Intel Core i7-12700 core, W4 including its coarse P2 base state costs 0.80 s (SP-DN) and 0.26 s (VC-DR) per context, against 2.28 s and 2.31 s for FEM including meshing. For whole training sets both generators run on eight parallel workers, one per core: 14.3 vs 70.3 min on SP-DN (4,000 contexts) and 1.1 vs 12.1 min on VC-DR (1,024 contexts).

Factorized FEM. Reusing the factorization of a fixed problem, FEM still needs 0.85 s and 0.74 s for the same request; CALTO answers in 1.56 ms and 1.53 ms on a GPU.

Inverse problem. Both Jacobian sources use the same FEM forward model, so only the tangent differs. Until the sensor residual is halved, the cumulative six-direction Jacobian costs 1.246 s with FEM (one core, forward factorization reused) and 5.62 ms with CALTO; total time including forward solves is 4.64 s vs 3.48 s. The 1.7×10−41.7\times 10^{-4} belongs to CALTO-Hybrid, and FEM's lower final residual (2.6×10−62.6\times 10^{-6}) reflects that the final Gauss–Newton contraction is limited by tangent accuracy. Without FEM tangents, CALTO reduces the final residual of learned primal surrogates by 98.9–99.5% (paper Table 5b).

Break-even. CALTO's one-time cost is label generation plus 100-epoch training on one GPU: 14.3 + 51.8 min ≈\approx 1.1 h on SP-DN and 1.1 + 29.7 min ≈\approx 31 min on VC-DR (include data loading). A new context then costs about 1.5 ms, against 2.3 s for FEM on one core, so the one-time cost equals FEM's cost for 1,740 (SP-DN) and 800 (VC-DR) new contexts, or 4,670 and 2,490 if FEM reuses a factorization (0.85 s and 0.74 s per request). After break-even, CALTO answers each new context about 1,500× faster than FEM, and about 500× faster than FEM with a reused factorization.

Reply to W1 (weak labels only), Q2 and W2 (NGO part)

Weak validation only (W1, Q2). For CALTO and each family's strongest baseline we applied two selection rules to the same 100 epoch-wise checkpoints, W4 validation observations only vs strong validation responses, and scored both on the same strong test references. CALTO is unchanged: on SP-D and SL-RD both rules pick the same checkpoint, and over all ID splits test error moves by at most 0.013 points (SP-DN 3.02763.0276\\%\to 3.0405\\%). For Transolver(++) the change is at most 0.05 points; GINO on SL-RD is 0.46 points better with W4 selection. CALTO is lower on every split under either rule. W4 observations are unbiased estimates of the same tangent, so the weak validation loss differs from the strong one by a noise term essentially independent of the checkpoint, averaged over the responses. Training and selection need only weak labels.

NGO (W2). We credit NGO for one principle, nonlinear conditioning with a linear action, and CALTO follows it no further. For pure data perturbations of SP-D and SP-DN the reviewer is right that the tangent is a Green's action on the perturbed data, δu=G[θ]!(δf,δg)\delta u=G\bigl[\theta\bigr]\\!(\delta f,\delta g). The learning problem still differs from NGO's: the domain and, in SP-DN, the Dirichlet/Neumann partition change with every context (NGO lists this as future work), and labels are mesh-free four-path estimates, not FEM solutions. NGO expands GG in a fixed basis on a fixed domain, whereas CALTO is a point-set operator with no basis, mesh or system matrix. Its network follows the tangent equation Lcδu=BcvL_{c}\delta u=B_{c}v: perturbation features at queries, interior supports and boundary supports; attention weights computed jointly from query and source contexts, independent of vv; and context-generated gates on both sides of each linear kernel. The gates matter beyond data perturbations. A screening direction gives data scaled pointwise by the base state, −u0δλ-u_{0}\delta\lambda, and a shape direction gives −Vn∂nu0-V_{n}\partial_{n}u_{0}; u0u_{0} is not an input, so this scaling must come from the context. For coefficient directions the tangent is the derivative of the Green's operator with respect to its own coefficients, a composition G∘B∘GG\circ B\circ G rather than one Green's action, and in SL-RD the linearized operator itself depends on the state. Each design choice is tested with exact linearity and parameter count fixed (paper Table 13): removing the joint query–source dependence raises error by 33–51%, and removing physical information from the gates by 8–22%. The size of the gain over the baselines and the results on new geometries are addressed in the next reply.

Reply to W2 (rest: source of the gain and new geometries)

Source of the gain. Paper Table 3 isolates exactness inside an architecture that already conditions the operator on the context alone. When directional observations are few (R ≤ 8 with four-walk labels), the linear model is lower by 0.15–0.20 points (three seeds), with more directions or cleaner labels the two variants agree. In that case, the value of exactness lies elsewhere: on directions scaled 16×, CALTO keeps 3.19% mean error while the nonlinear variant reaches 11.60% (Appendix E.1). The larger gap to the baselines comes mainly from how the context is used: with exact linearity and parameter count held fixed (Table 13), removing the joint query–source dependence raises error by 33–51% and removing physical information from the gates by 8–22%. The gap persists when every model is trained on strong FEM tangents (ID error 52–76% lower in all four families). The gain thus has several sources rather than a single one.

New geometries. CALTO has the lowest geometry-OOD error in every family (e.g., 17.73% vs 20.52% on SP-DN, 30.72% vs 48.02% on VC-DR). Absolute errors are high because training domains have one outer boundary, whereas geometry OOD adds holes, each an interior Dirichlet boundary absent from training. This is a coverage limit, not a label-precision limit: retraining on FEM tangents cuts ID error by 37–58% but leaves geometry-OOD error nearly unchanged, and higher on VC-DR (30.7230.72\\%\to 34.62\\%) and SL-RD (41.7441.74\\%\to 42.65\\%). Adding just 32 weakly labelled new contexts nearly halves the SP-DN perforated-domain error (29.6629.66\\%\to 15.51\\%), so cheap labels extend coverage efficiently.

Reply to W2 (baseline part: adaptation, parameters, tuning) and Q3

Adaptation. The baselines were initially built to predict solutions or fields, so each needs a minimal adaptation to predict the tangent at the queries from weak JVP labels. SC-FNO, DE-DeepONet and DINO keep their upstream backbones, and we differentiate their predicted solution at the queries along the normalized physical direction (geometry and queries fixed); GINO, Transolver and Transolver++ take the context and perturbation as pointwise inputs and predict the tangent directly. All methods share the inputs, W4 labels, optimizer, batch size, update budget and validation split, so the comparison isolates the model, not the supervision. JVP-only training is the favourable setting for the state surrogates: adding primal supervision raises their tangent error on all four benchmarks (e.g., DE-DeepONet on SP-D: 2.032.03\\%\to 4.46\\%).

Parameters. CALTO has about 142k trainable parameters: more than SC-FNO (109k) and DINO (31k), fewer than DE-DeepONet (250k–773k), and 4.5–9.7×4.5\text{–}9.7\times fewer than the strongest baseline of each family (Transolver 757k–758k on SP-D and VC-DR, Transolver++ 644k on SP-DN, GINO 1.38M on SL-RD). Capacity does not explain the gap. Paper Table 10 gives more detailed information about parameters.

Tuning. We only tune the learning rate. We keep each baseline's official architecture configuration (e.g., SC-FNO: four Fourier layers of width 20, eight modes per dimension; DINO: DIPNet 100-50 projection network, 50-dimensional output; GINO: 16×1616\times 16 latent modes), adapting only inputs and outputs. All methods share AdamW (weight decay 10−410^{-4}), cosine decay and update count (12,500 on SP-D/SP-DN, 3,200 on VC-DR/SL-RD). In SP-D, we trained each model for 200 updates on 320 subset training contexts at 0.001, 0.005 and 0.01 and selected checkpoints by strong-validation error. For example, DINO, the best validation L2RE was 31.50%, 29.54% and 53.05%; 0.005 was also best for DE-DeepONet and Transolver(++), and for SC-FNO the best value, 0.01, was only 0.17 points below 0.005. For CALTO it was 11.44%, 8.21% and 7.95%, so the shared learning rate 0.005 is not CALTO's best either (0.01 is 0.26 points lower): we did not tune it in CALTO's favour. Appendix E.4 describes the tangent route of each baseline.

Reply to Q4

Parameter dimension. The response branch takes perturbation fields at the support points as input, so model size and the cost of one JVP do not depend on dcd_{c}. Training needs only RR observed directions per context: with R=2R=2 on SP-DN (dc=28d_{c}=28), CALTO recovers the full Jacobian with 3.49% error. The full Jacobian costs dcd_{c} evaluations sharing one context encoding (2.11–2.27 ms at dc=15–28d_{c}=15\text{–}28, vs 9.86–16.86 ms for SC-FNO), and each VJP is a matrix–vector product. We tested up to dc=28d_{c}=28; for field-valued parameters a direction is itself a field, so cost grows only with the number of columns requested.

Coarse state in SL-RD. We froze the model and replaced its h/D=0.08h/D=0.08 coarse state at inference by P2 solutions at h/D=0.02h/D=0.02, 0.2 and 0.8, updating all derived inputs. Mean error changes by at most 0.03 points on every split (Table 1). Even at h/D=0.8h/D=0.8 the state is within about 1% of the h/D=0.02h/D=0.02 solution, and the tangent coefficient λ+3γu2\lambda+3\gamma u^{2} and forcing change by about 0.002% and 0.02%. The model does respond to the state: a discretization-shaped perturbation of 5–50% of the state norm raises ID error from 8.30% to 17.8–19.6% at 50% (Table 2). SL-RD results are thus insensitive to the coarse state over the accuracy coarse P2 solves deliver.

Table 1: frozen SL-RD model with genuine P2 coarse states at inference, L2RE (%)

Split (contexts)h/D=0.02h/D=0.020.08 (training)0.20.8
ID (256)8.29788.29828.29928.3044
Geometry OOD (128)41.743041.741441.740641.7560
Nonlinearity OOD (128)11.591511.591211.568211.5653
State error vs h/D=0.02h/D=0.02, ID / geo. OOD (%)00.07 / 0.100.68 / 0.611.19 / 0.87

Table 2: same model, h/D=0.08h/D=0.08 state perturbed along its discretization error, L2RE (%), positive / negative sign

Perturbation (% of state norm)05102050
ID8.308.58 / 8.659.30 / 9.4911.17 / 11.8317.83 / 19.59
Geometry OOD41.7441.91 / 41.8542.33 / 42.2243.56 / 43.4847.71 / 48.69
Reviewer_og5ZOpenReview ↗
Rating
3
Confidence
1
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper proposes CALTO, a model for directly learning PDE tangent responses from sparse and noisy directional observations generated by stochastic walks. The architecture separates nonlinear conditioning on the base PDE from the linear perturbation direction. Then the author design the training framework contains context-dependent attention and multiplicative gates determine how perturbations are propagated to query points. The paper evaluates tangent prediction on four 2D elliptic PDE families and a 3D diffusion–reaction benchmark, together with architectural ablations, Jacobian/VJP recovery, and inverse-problem applications.

Critical strengths

1. The central motivation is well motivated. Separating nonlinear PDE conditioning from linear dependence on perturbations reflects the mathematical structure of the tangent operator. Shared-path supervision preserves compatible directional relations even at finite sampling budgets. This provides a aligned connection between the target, the supervision procedure, and the architecture.

2. The reported empirical improvement are promising and supported by several ablation studies. CALTO achieves the lowest reported in-distribution errors across the five benchmarks. The experiments investigate supervision density and noise, linear versus nonlinear response models, and parameter-matched architectural variants. In particular, the structural ablations provide useful evidence for global aggregation, query–source-dependent weighting, successive response layers, and physical conditioning.

Critical weaknesses

1. The main source of the performance gains is unclear.
Table 3 shows a small gain from directional linearity: 3.03% versus 3.17% error under W4/R8. Some baselines already produce JVPs that are linear in the direction. It is therefore unclear how much of the overall gain comes from direct tangent learning, linearity, or the network architecture. Results across multiple seeds would help assess the smaller differences.

2. The main comparison only uses JVP supervision.
SC-FNO, DE-DeepONet, and DINO are trained without primal-value labels in the main experiments. This keeps the supervision the same, but leaves open whether CALTO still performs better when both primal and JVP labels are used. Section 5.4 tests joint supervision for some baselines, but not across all four 2D benchmarks.

3. The practical speedup is unclear.
Section 5.4 still uses FEM forward solves to compute residuals and accept steps. Table 5 compares the number of forward evaluations, not total runtime, and FEM tangents reach lower final residuals. A runtime comparison at matched accuracy is needed to show the practical benefit over numerical tangent solvers.

4. The paper writing makes the method difficult to follow. Sections 3–4 introduce the PDE context, physical perturbation features, stochastic observations, and network components in rapid succession without a concrete end-to-end example connecting them. Figure 1 describes the architecture but does not show the supervision pipeline, while important input and implementation details are deferred to the appendices. This makes the draft difficult to follow.

Questions for authors
  1. Can the authors separate the gains from direct tangent learning, directional linearity, and the network architecture?

  2. Can the authors report means and standard deviations across multiple training seeds for Tables 3–4?

  3. Does CALTO still perform better when the baselines use both primal and JVP labels across all four 2D benchmarks?

  4. Can the authors compare total runtime at matched inverse-problem accuracy against FEM tangents, and report data-generation and training costs separately?

5 comments

Reply to W1, Q1 and Q2

Thank you for recognizing the alignment between target, supervision and architecture, and we agree that one seed is not enough. We repeated every setting of paper Tables 3 and 4 over three independent training seeds (schedule and selection unchanged); Tables 3 and 4 now report means ±\pm s.d. (Appendix E.1). The multi-seed results do not change our core claims.

Exact linearity (Q2, paper Table 3). When directional observations are few (R=2, 4 and 8 with four-walk labels), CALTO-Linear is lower than CALTO-Nonlinear by 0.15–0.20 points on ID directions and by 0.11–0.27 on held-out directions. With all 28 directions, or 8–16 walks, the variants agree; with strong labels the linear model is slightly lower. This structural prior supplies the constraint cheap labels cannot, at no measurable cost.

Reusability. On 1,000 SP-DN ID contexts and three seeds, we applied the W4/R8 models to random combinations of the eight test directions, scaled from 0.25×0.25\times to 16×16\times. CALTO-Linear stays at 3.19% at every scale; the nonlinear variant rises from 3.32% to 11.60% at 16×16\times. Exactness also recovers the 28-column SP-DN Jacobian from two observed directions (paper Table 4: 3.49±0.103.49\pm 0.10\\% vs 3.93±0.053.93\pm 0.05\\%).

Direct tangent learning (Q1). In one CALTO-Hybrid checkpoint, using the predicted JVP instead of the AD derivative of its own primal output lowers the final inverse residual from 1.33×10−31.33\times 10^{-3} to 1.72×10−41.72\times 10^{-4} (−87-87\\%), with the same network, data and solver.

Architecture and baselines (W1, Q1). CALTO-Nonlinear keeps CALTO's context branch, so paper Table 3 isolates exactness alone, the smaller gain. The larger gain comes from what both variants share: the context generates the attention weights and gates for perturbation propagation. GINO and Transolver(++) mix context and perturbation, and AD-JVP baselines differentiate a state surrogate; on the same labels they are at least 2.7×2.7\times less accurate (every baseline at least 2.0×2.0\times; paper Table 1). Changing only the kernel raises SP-DN error from 3.03% to 4.04% or 15.94% (paper Table 13). With every model trained on strong FEM tangents, CALTO's ID error is still 52–76% below the strongest baseline in all four families.

Table 1: SP-DN ID L2RE (%), mean ±\pm s.d., three seeds

SupervisionCALTO-LinearCALTO-Nonlinear
W4, R24.10±0.124.10\pm 0.124.30±0.074.30\pm 0.07
W4, R83.07±0.043.07\pm 0.043.22±0.083.22\pm 0.08
Strong, R81.92±0.031.92\pm 0.031.97±0.021.97\pm 0.02

Reply to W2 and Q3

Yes, CALTO still performs better. Thank you for the suggestion; we completed the comparison on all four 2D benchmarks. SC-FNO, DE-DeepONet and DINO, the three baselines with a primal output, were trained with the same weak primal and JVP labels as CALTO-Hybrid under one protocol: eight W4 JVP directions at 1,024 points, equal weights on primal and JVP relative L2L_{2} losses, selection by their validation sum, and the same 100-epoch budget. CALTO-Hybrid has the lowest tangent error on every benchmark, 61–67% below the best jointly trained baseline, DE-DeepONet (Table 2). DE-DeepONet fits the solution better than CALTO-Hybrid on SP-D and SP-DN (primal error 6.10% vs 8.74%, 9.91% vs 12.10%), yet its tangent error there is three times higher. Primal labels in fact raise every state surrogate's tangent error relative to JVP-only training (e.g., DE-DeepONet on SP-D: 2.032.03\\%\to 4.46\\%), because a derivative taken from a state model inherits the state-fit error. JVP-only training is thus the more favourable setting for these baselines, and CALTO leads under both.

Table 2: tangent L2RE (%), full test-ID sets, all models trained on weak primal + W4 JVP labels

ModelSP-DSP-DNVC-DRSL-RD
CALTO-Hybrid1.513.6610.8210.39
DE-DeepONet4.4610.9530.9726.33
SC-FNO6.8520.3332.6329.27

Reply to W3 and Q4

Thank you for asking for this measurement. In the inverse problem of paper Table 5, the sensor observations are simulated with FEM, and both methods use the same FEM forward model to evaluate residuals and accept updates, so only the tangent source differs. FEM tangents reuse the LU factorization of each forward solve, the most favourable setting for the numerical solver. We report three costs separately.

Online, at matched accuracy. We solve the 16 VC-DR inverse problems of Table 5, each from two initial guesses (32 runs); all runs reach both targets with both Jacobian sources (FEM on one Intel Core i7-12700 core, CALTO on one GPU). A six-direction Jacobian costs 1.24 s with FEM and 3.1 ms with CALTO per call. Halving the sensor residual takes 3.48 s with CALTO tangents vs 4.64 s with FEM tangents; a tenfold reduction takes 5.55 s vs 6.05 s (Table 3). FEM forward solves account for 90% and 94% of CALTO's total time at these targets. Near convergence the Gauss–Newton contraction is limited by tangent accuracy, so FEM tangents reach the lower final residual (paper Table 5a); without any FEM tangent, CALTO corrects the derivatives of a learned primal surrogate and reduces its final residual by 98.9–99.5% (Table 5b).

Offline data generation. With eight parallel workers on the same CPU (eight directions, 1,024 points), labels for the whole training set take 1.1 min with weak labels vs 12.1 min with FEM including meshing on VC-DR (1,024 contexts), and 14.3 vs 70.3 min on SP-DN (4,000 contexts). Across the 2D families FEM costs 4.9–11×4.9\text{–}11\times more.

Offline training. In our original runs, 100 epochs took 29.7 min (VC-DR) and 51.8 min (SP-DN) on one GPU, including data loading, initialization and validation.

Table 3: VC-DR inverse problem, same FEM forward protocol, 32 runs

QuantityFEM tangents (LU reused)CALTO tangents
Jacobian per call, 6 directions1.24 s3.1 ms
Jacobian to residual ≤0.5\le 0.51.246 s5.62 ms
Total time to residual ≤0.5\le 0.54.64 s3.48 s
Jacobian to residual ≤0.1\le 0.11.870 s9.58 ms
Total time to residual ≤0.1\le 0.16.05 s5.55 s

Reply to Weakness 4

Thank you for this concrete feedback. §3 now opens with one running SP-DN example, and Figure 1 now includes the supervision pipeline.

The context cc is a domain with Dirichlet and Neumann parts, the equation −Δu+λu=f-\Delta u+\lambda u=f, and data gDg_{D}, gNg_{N}.

A direction vv perturbs these data, e.g., (δf,δλ,δgD,δgN)(\delta f,\delta\lambda,\delta g_{D},\delta g_{N}); the tangent δu\delta u solves the same operator with forcing δf−u0δλ\delta f-u_{0}\delta\lambda, and a perturbation compiler turns vv into these fields at the support points.

Labels: four random walks per query are run once and reused for all eight directions, so labels are noisy but satisfy superposition exactly.

Prediction: the context encoder reads only cc and produces attention weights and gates; the response branch combines the compiled perturbation fields through them, with no nonlinearity or bias on the perturbation, so doubling vv doubles the output.

Updated Tables 3-4 with three seeds

Updated Table 3: SP-DN direction prediction, linear / nonlinear variant. L2RE (%), mean ± s.d. over three seeds.

SupervisionIDDirection OOD
W4, R24.10 ± 0.12 / 4.30 ± 0.074.31 ± 0.15 / 4.58 ± 0.07
W4, R43.47 ± 0.07 / 3.66 ± 0.033.70 ± 0.10 / 3.90 ± 0.06
W4, R83.07 ± 0.04 / 3.22 ± 0.083.19 ± 0.07 / 3.30 ± 0.11
W4, Full dir.2.76 ± 0.05 / 2.79 ± 0.17– / –
W8, R82.65 ± 0.05 / 2.64 ± 0.092.78 ± 0.05 / 2.73 ± 0.09
W16, R82.38 ± 0.01 / 2.37 ± 0.072.48 ± 0.01 / 2.46 ± 0.10
Strong, R81.92 ± 0.03 / 1.97 ± 0.021.97 ± 0.00 / 2.05 ± 0.02

Updated Table 4: Jacobian / VJP error (%) under W4R2 supervision, mean ± s.d. over three seeds.

FamilyLinearNonlinearRaw W4
SP-D0.91 ± 0.19 / 0.87 ± 0.211.07 ± 0.14 / 1.05 ± 0.1513.18 / 9.89
SP-DN3.49 ± 0.10 / 3.26 ± 0.113.93 ± 0.05 / 3.79 ± 0.0840.35 / 29.22
VC-DR11.75 ± 0.41 / 12.01 ± 0.3412.42 ± 0.94 / 12.49 ± 0.9083.31 / 60.25
SL-RD9.78 ± 0.04 / 10.36 ± 0.2310.08 ± 0.07 / 10.63 ± 0.0473.32 / 58.65
Reviewer_CzrROpenReview ↗
Rating
2
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

CALTO (Context-Aware Linearity-preserving Tangent Operator) learns tangent operators from directional observations with few stochastic paths and enforces directional linearity in the perturbation direction. The contributions are: (1) the CALTO method, which combines nonlinear PDE conditioning and a linear perturbation pathway, (2) learning the tangent responses from few-path stochastic observations, and (3) shared-path labels to preserve directionality relations. On the 2D elliptic families CALTO reaches 0.71 to 8.8% L2 error on test-ID predictions, which is a 50.7 to 74.8% error reduction against the strongest learned baselines.

Critical strengths

The problem formulation is well-motivated and the solution is well conditioned. This paper combines sensitivity approximation and state reconstruction to learn from few-path observations is a useful framing for PDE-constrained inverse problems. Additionally, the evaluation is rigorous, as we see the four 2D elliptic families, 3D CAD benchmark, ID, and OOD splits, controlled studies of walk count, training contexts, and directions per context. The addition of the linear vs. nonlinear ablation and downstream coefficient identification is additionally interesting.

Critical weaknesses

This paper is motivated by the idea that FEM directional supervision is expensive but few-path stochastic estimates are inexpensive, but there is no comparison for label generation, so the reader cannot identify whether stochastic supervision has any advantage over computing the FEM Jacobians. Additionally, the novelty is limited relative to prior works as the tangent equation is a linear BPE, so CALTO is very similar to the Neural Green's Operators applied to the linearized problem. So, the contribution is primarily the combination of known pieces.

Questions for authors

What is the cost of computation and accuracy for generating the shared-path labels compared to the FEM tangent solves?

3 comments

Reply to W (label-cost part) and Q

Thank you for the question. We timed the full label pipelines from scratch for every training context of each benchmark (8 directions, 1,024 points), using eight parallel workers on an Intel Core i7-12700 (one per core), and report wall-clock time for the whole training set. FEM is the P2 generator (h/D=0.02h/D=0.02) of our references: boundary-fitted meshing, assembly, one factorization, and primal and tangent solves (refining to h/D=0.01h/D=0.01 changes its tangents by less than 0.04%).

Cost. On the 2D families, labels take 1–14 minutes with weak labels and 10–70 minutes with FEM, so FEM costs 4.9–11×4.9\text{–}11\times more (Table 1). Meshing is only 1–5% of FEM time there. The ratio varies because walk cost is set by the PDE, whereas FEM cost is set by the mesh: reaction terms and all-Dirichlet boundaries end a VC-DR walk after 3.8 steps on average, while Neumann reflection (walk on stars [1]) gives SP-DN walks 9.6 costlier steps.

CAD-DR (3D). From the same STEP files, walks need only a surface triangulation and closest-point queries, while FEM must first build a tetrahedral volume mesh of each geometry; both are counted once per geometry. Walk labels take 1.2 min (15 s triangulation, 60 s walks) and FEM labels 3.2 min (109 s volume meshing, 86 s solves): 2.6× more, with volume meshing 56% of FEM time. FEM on harder geometry also needs local refinement near re-entrant edges and corners, where the solution is singular [4], whereas walks need no volume mesh at any resolution [2, 3].

Accuracy. A single W4 label has 15.2–87.7% error on the full test-ID sets, but CALTO trained on such labels reaches 0.71–8.88%, 9.2–21.4×9.2\text{–}21.4\times more accurate, and 4.6–10.7×4.6\text{–}10.7\times more accurate than averaging 16 paths per point. On CAD-DR, CALTO-3D has 2.02% error, about four times below the labels at the same points, and then answers queries anywhere in milliseconds. Learning across contexts, rather than more sampling per context, turns cheap observations into accurate tangents.

These measurements and this discussion are in the setup of §5 and Appendix E.5.

Table 1: end-to-end label generation time (wall-clock) for the whole training set, eight parallel workers; both generators include all preprocessing

FamilyContextsWeak (min)FEM (min)FEM / weakFEM mesh share
SP-D4,0008.357.96.9×6.9\times1%
SP-DN4,00014.370.34.9×4.9\times5%
VC-DR1,0241.112.111×11\times1%
SL-RD1,0241.010.29.9×9.9\times1%
CAD-DR1,0001.23.22.6×2.6\times56%

References

[1] Sawhney, R., Miller, B., Gkioulekas, I. and Crane, K. 2023. Walk on Stars: A Grid-Free Monte Carlo Method for PDEs with Neumann Boundary Conditions. ACM Transactions on Graphics 42(4), 80:1–80:20.

[2] Sawhney, R. and Crane, K. 2020. Monte Carlo Geometry Processing: A Grid-Free Approach to PDE-Based Methods on Volumetric Domains. ACM Transactions on Graphics 39(4).

[3] Zhou, Z., Sawhney, R., d'Eon, E. and Jarosz, W. 2026. Grid-Free Monte Carlo for Time-Dependent Diffusion. arXiv:2609.12306.

[4] Grisvard, P. 1985. Elliptic Problems in Nonsmooth Domains. Monographs and Studies in Mathematics 24. Pitman, Boston.

Reply to W (novelty part: relation to Neural Green's Operators)

Thank you for the careful summary. We cite NGO [1] for one principle, nonlinear conditioning with a linear action, and CALTO follows it no further: the learned operator, the network that represents it, and the learning signal all differ.

A different operator. NGO learns the solution operator of a linear PDE, u=G[θ]!(f,g)u=G\bigl[\theta\bigr]\\!(f,g). CALTO learns the derivative of the solution map, Jcv=PXLc−1BcvJ_{c}v=P_{X}L_{c}^{-1}B_{c}v, whose linear argument is a perturbation direction and whose conditioning is the whole base problem. For a coefficient direction, δu=G[θ]!(∇!⋅!(δκ,∇u0))\delta u=G\bigl[\theta\bigr]\\!\left(\nabla\\!\cdot\\!\left(\delta\kappa\\,\nabla u_{0}\right)\right) with u0=G[θ]!(f,g)u_{0}=G\bigl[\theta\bigr]\\!(f,g): the derivative of the Green's operator with respect to its own coefficients, a composition G∘B∘GG\circ B\circ G rather than one Green's action. An NGO-type model would need a base solve per context, or differentiation through its conditioning input, which is the route of our AD-JVP baselines. Within one CALTO-Hybrid checkpoint, replacing the AD derivative of its own primal output by the directly predicted tangent lowers the mean inverse residual after 16 forward evaluations from 1.33×10−31.33\times10^{-3} to 1.72×10−41.72\times10^{-4} (an 87.1% reduction, paper Table 5a), so linearity alone is not sufficient.

A different network, derived from the structure of the tangent. The tangent solves Lcδu=BcvL_{c}\delta u=B_{c}v, with data in the interior and on the boundary, for example δf−u0δλ\delta f-u_{0}\delta\lambda in Ω\Omega and δgD−Vn∂nu0\delta g_{D}-V_{n}\partial_{n}u_{0} on a Dirichlet boundary under shape perturbations. CALTO therefore takes perturbation features at queries, interior supports and boundary supports. Since Lc−1L_{c}^{-1} is a linear operator fixed by cc alone, whose response at a query depends jointly on the query and every source, attention weights are computed jointly from query and source contexts and never depend on vv. The base state u0u_{0} is not an input, so the pointwise scaling of the data, such as u0δλu_{0}\delta\lambda, must come from the context; context-generated gates on both sides of each linear kernel supply it. The perturbation pathway uses only bias-free linear maps and weighted sums, so Fθ(c,v)=Aθ(c;P),Cc,PvF_{\theta}(c,v)=\mathcal{A}_{\theta}(c;\mathcal{P})\\,\mathcal{C}_{c,\mathcal{P}}v is linear in vv by construction, and the context-derived weights are computed once and reused for every direction. With exact linearity and parameter count held fixed, removing the joint query–source dependence raises error by 33–51% and replacing global attention by 32-neighbour aggregation raises SP-DN error from 3.03% to 15.94% (paper Table 13).

A different setting. NGO expands GG in a fixed basis on a fixed domain and lists varying geometry and boundary-condition types as future work. CALTO uses no basis, mesh or system matrix; it is a point-set operator, so the domain and, in SP-DN, the Dirichlet/Neumann partition can change with every context. On unseen geometries, the model has the lowest error in every family (17.73% vs 20.52% for Transolver++ on SP-DN; paper Table 11). NGO also trains on FEM or manufactured solutions, whereas CALTO trains on mesh-free four-path estimates with 15–88% error. The walks are shared across directions, so the labels are exact linear images of one random operator (Eq. 6), which an exactly linear model fits at any path budget.

Advantage on our task. To our knowledge, learning a reusable tangent operator from few-path stochastic directional observations has not been done before, and NGO cannot be applied directly, since it needs a fixed basis and system matrix that mesh-free labels on varying geometries cannot provide. On the same labels, every baseline is at least 2.0×2.0\times less accurate, and the AD-JVP baselines, which are also exactly linear in vv, at least 2.7×2.7\times (paper Table 1). Even with strong FEM tangents, the strongest baseline stays behind CALTO trained on four-path labels (Table 2), so the gap does not come from label quality.

Table 2: test-ID L2RE (%), strongest baseline of each family

FamilyCALTO, W4Baseline, strong FEM
SP-D0.711.25
SP-DN3.036.14
VC-DR8.8822.28
SL-RD8.309.85

References

[1] Melchers, H., Prins, J. and Abdelmalik, M. 2026. Neural Green's Operators for Parametric Partial Differential Equations. Computer Methods in Applied Mechanics and Engineering 455, 118893.

Dear Reviewer CzrR

Thank you again for your constructive feedback. Since the discussion period closes in less than 24 hours, we wanted to gently follow up to see if our responses and new results have addressed your concerns, or if there is anything else we can clarify before the deadline.

Best,

Authors

Reviewer_xjRGOpenReview ↗
Rating
3
Confidence
4
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper presents a weak supervision framework for training neural operators to model parameter sensitivities and tangent operators from cheap Monte-Carlo estimates derived from Walk-on-Spheres. This framework thus avoids expensive autograd computations required for computing higher order gradients. Evaluated across elliptic PDE families, CALTO achieves 0.71-8.88% in-distribution relative L2L^2 error, reducing errors by 50.7-74.8% against the strongest derivative-informed baselines.

Critical strengths

Building on the recent derivative-free and data-free learning strategies, this framework provides an intuitive formulation for directly learning tangent operators for inverse design problems, thus avoiding numerical and computational instabilities associated with computing gradients.

The reviewers provide thorough empirical studies, they choose good representative problem sets and CAD models to showcase their performance.

Critical weaknesses

A weakness of walk-on-spheres is it's limitations in modeling PDE families, and as such the framework is currently tested only on elliptic families. However, this is still an important problem to solve, and speaks to the limitation of walk-on-spheres rather than their approach, but it would be particularly useful if the authors can provide a discussion of how it can be extended to other PDE families, whether there are other stochastic approaches (unbiased, high-variance) for modeling derivatives that can be used for problems that WoS cannot model.

OOD degradations still remain despite strong performance. I'm curious whether the authors tested any zero-shot or few-shot capabilities of the model.

The paper also does not discuss computational speedups clearly. It would be helpful to provide gains on equal-time, equal-compute settings.

The authors do not provide discussions on scaling. Can the model scale to million-point geometries?

Questions for authors

Please refer to the weakness section

5 comments

Reply to W1

Thank you for the positive assessment. We agree that the restriction comes from what WoS can model, not from what CALTO needs. The question is where an unbiased derivative estimator can come from, and we see two cases. For each context, observations y=Aωvy=A_\omega v with E[Aω]=Jc\mathbb E[A_\omega]=J_c and ω\omega shared by all directions (Eq. 6). WoS appears only in the label generator.

PDEs with a probabilistic representation. Here the estimator comes from a stochastic process, and WoS-type methods have steadily widened their reach: volumetric Poisson problems [1], spatially varying coefficients and screening by delta tracking [2], mixed Dirichlet–Neumann conditions by walk on stars [3], time-dependent diffusion [4,5], and parameter and shape derivatives by differential WoS [6]. Our benchmarks already use delta tracking (VC-DR, SL-RD), walk on stars (SP-DN) and differential WoS (shape directions). SL-RD shows that nonlinearity is no obstacle: its tangent is linear in vv, so only the linearized operator, here −Δ+λ+3γu~02-\Delta+\lambda+3\gamma\widetilde u_0^2, needs an estimator. Parabolic problems are the natural next step. For ∂tu=κΔu\partial_t u=\kappa\Delta u, Feynman–Kac [7] gives δu(x,t)=E[δg(x+2κ,Bt),1τx>t]\delta u(x,t)=\mathbb E\bigl[\delta g(x+\sqrt{2\kappa}\\,B_t)\\,\mathbf 1\\{\tau_x>t\\}\bigr] for an initial-data perturbation δg\delta g, and source or boundary perturbations follow through Duhamel terms. The grid-free estimator of [4] realizes this with a finite time budget and an exit time sampled at each spatial step, at the price of a very small sampling bias; as long as the sampling does not depend on the perturbation, one walk reused across directions gives observations that are exactly linear in δg\delta g, as in §4.3.

PDEs without a probabilistic representation. For hyperbolic and other problems with no convenient Feynman–Kac-type representation, the randomness has to come from the solver instead of the PDE. Discretize the tangent equation of a smooth base solution, Ac,δu=b(v)A_c\\,\delta u=b(v), and, with a splitting or preconditioner PP, write it as the fixed point δu=Hδu+d(v)\delta u=H\delta u+d(v) with H=I−P−1AcH=I-P^{-1}A_c and d(v)=P−1b(v)d(v)=P^{-1}b(v), so that the Neumann series ∑kHkd\sum_k H^k d converges. A Monte Carlo linear solver [8] estimates one entry δui\delta u_i by a random walk over the indices: from ii it moves to mm with probability pimp_{im}, multiplies a weight by Him/pimH_{im}/p_{im} at each step, and accumulates the weighted data dikd_{i_k} along the path. Randomized truncation [9] keeps the path length almost surely finite, with finite expected length, without bias. The path and its weights WkW_k depend on AcA_c and PP only, not on vv, so one path from ii applied to the data of every direction gives yj=∑kWk,d(vj)iky_j=\sum_k W_k\\,d(v_j)_{i_k}, which is linear in vjv_j for fixed randomness and has mean (Jcvj)i(J_c v_j)_i for the discretized operator. This is the shared-randomness observation CALTO needs. It is unbiased, but convergence of the series does not make the variance finite: that requires ρ(H(2))<1\rho(H^{(2)})<1 with Him(2)=Him2/pimH^{(2)}_{im}=H_{im}^2/p_{im}, so the variance can be high. The price is the mesh-free property, since the linearized system of each base problem must be available, although only the entries a walk visits are needed; no factorization or adjoint solve is needed.

This discussion is summarized in §6 and Appendix E.11.

References

[1] Sawhney, R. and Crane, K. 2020. Monte Carlo Geometry Processing: A Grid-Free Approach to PDE-Based Methods on Volumetric Domains. ACM Transactions on Graphics 39(4).

[2] Sawhney, R., Seyb, D., Jarosz, W. and Crane, K. 2022. Grid-Free Monte Carlo for PDEs with Spatially Varying Coefficients. ACM Transactions on Graphics 41(4).

[3] Sawhney, R., Miller, B., Gkioulekas, I. and Crane, K. 2023. Walk on Stars: A Grid-Free Monte Carlo Method for PDEs with Neumann Boundary Conditions. ACM Transactions on Graphics 42(4), 80:1–80:20.

[4] Zhou, Z., Sawhney, R., d'Eon, E. and Jarosz, W. 2026. Grid-Free Monte Carlo for Time-Dependent Diffusion. arXiv:2609.12306.

[5] Bao, A., Shen, E., Zhang, Y., Liu, Z. and Wang, J. 2026. Walking on Heat Stars for Parabolic Heat Equations with Neumann Boundary Conditions. arXiv:2606.16578.

[6] Miller, B., Sawhney, R., Crane, K. and Gkioulekas, I. 2024. Differential Walk on Spheres. ACM Transactions on Graphics 43(6), Article 174.

[7] Øksendal, B. 2003. Stochastic Differential Equations: An Introduction with Applications, 6th ed. Springer, Berlin.

[8] Forsythe, G. E. and Leibler, R. A. 1950. Matrix Inversion by a Monte Carlo Method. Mathematical Tables and Other Aids to Computation 4(31), 127–129.

[9] Rhee, C.-H. and Glynn, P. W. 2015. Unbiased Estimation with Square Root Convergence for SDE Models. Operations Research 63(5), 1026–1043.

Reply to W2

Thank you for the question. Zero-shot. Every OOD result in the paper is zero-shot: the trained model is frozen and evaluated on unseen coefficients, boundary partitions, directions and geometries (Figure 2, paper Table 11). CALTO has the lowest geometry-OOD error in all four 2D families (e.g., 17.73% vs 20.52% for Transolver++ on SP-DN, 30.72% vs 48.02% for Transolver on VC-DR) and on every CAD-DR OOD split (2.36–4.50%, 57–77% below the strongest baseline).

Where the degradation comes from. It is concentrated on new topology: training domains have one outer boundary, whereas geometry OOD adds toothed boundaries and one or two holes, each an interior Dirichlet boundary that no training context contains (SP-DN perforated domains: 29.66%). This is a coverage gap, and walk-based labels are well suited to closing it, since labelling a new geometry needs only its boundary representation, not a mesh.

Few-shot adaptation. We used SP-DN geometry OOD, which introduces both kinds of new geometry while changing only the geometry; Transolver++ is its strongest baseline. We fine-tuned copies of both trained models on 8 or 32 new contexts with the same four-path weak labels and 100 updates, kept the final checkpoint, and evaluated on 500 held-out contexts never used for adaptation or selection, over three draws of the adaptation set. With 32 contexts CALTO falls from 17.73% to 13.92% (−21.5-21.5\\%), against 17.26% for Transolver++, and CALTO adapted on 8 contexts (15.75%) already beats Transolver++ adapted on 32. On perforated domains CALTO's error nearly halves (29.6629.66\\%\to 15.51\\%). Labelling the 32 contexts takes about 28 s on one CPU core. This experiment is in Appendix E.8.

Table 2: SP-DN geometry OOD after weak-label adaptation, L2RE (%), mean ±\pm s.d. over three draws

New contexts0 (zero-shot)832
CALTO17.7315.75±0.7315.75\pm 0.7313.92±0.4713.92\pm 0.47
Transolver++20.5219.86±0.8419.86\pm 0.8417.26±0.6717.26\pm 0.67

Reply to W3

Thank you for the suggestion. "Equal compute" has three meanings, and we report all three.

Equal training time. On SP-DN and VC-DR, CALTO and its strongest baseline (Transolver++, Transolver) received the same one-GPU training and validation budget (32 and 8 minutes, calibrated to CALTO's 100-epoch time; data loading, initialization, label generation and final testing excluded), over three seeds. CALTO reaches 3.22±0.113.22\pm 0.11\\% vs 19.44±0.2819.44\pm 0.28\\% on SP-DN and 8.95±1.318.95\pm 1.31\\% vs 31.18±0.2931.18\pm 0.29\\% on VC-DR, and is better for every seed at all four budgets (Table 3). At 32 minutes, CALTO's 3.22% is also below the 6.41% Transolver++ reaches after its full schedule.

Equal deployment compute. One request (8 directions, 1,024 points) takes 1.56 ms (SP-DN) and 1.53 ms (VC-DR) with CALTO on a GPU, vs 6.81 ms and 4.71 ms for the baselines: 4.4×4.4\times and 3.1×3.1\times faster and more accurate (2.60% vs 6.27%; 9.88% vs 31.23%).

Against the numerical solver. For a new context, FEM must mesh, assemble, factorize and solve the whole field before evaluating any point: 2.3 s on one Intel Core i7-12700 core. Reusing the factorization of a fixed problem, FEM needs 0.85 s (SP-DN) and 0.74 s (VC-DR), about 500×500\times CALTO. In VC-DR coefficient identification (paper Table 5), both Jacobian sources use the same FEM forward model, so only the tangent differs. Until the residual is halved, six-direction Jacobians cost 1.246 s per run with FEM (one core, factorization reused) and 5.62 ms with CALTO; total time with forward solves is 4.64 s vs 3.48 s (tenfold reduction: 6.05 s vs 5.55 s).

We have added these results to Appendix E.6; the inverse-solve cost is also in §5.4.

Table 3: L2RE (%), three seeds (baseline: Transolver++ / Transolver)

BenchmarkTime (s)CALTOBaseline
SP-DN1925.83±0.825.83\pm 0.8284.61±20.3784.61\pm 20.37
SP-DN4804.72±0.284.72\pm 0.2833.13±1.3233.13\pm 1.32
SP-DN9603.67±0.133.67\pm 0.1324.79±0.4724.79\pm 0.47
SP-DN19203.22±0.113.22\pm 0.1119.44±0.2819.44\pm 0.28
VC-DR4825.07±2.3625.07\pm 2.36100.44±0.75100.44\pm 0.75
VC-DR12015.02±0.2115.02\pm 0.2162.33±12.0762.33\pm 12.07
VC-DR24011.59±0.7011.59\pm 0.7038.30±0.3838.30\pm 0.38
VC-DR4808.95±1.318.95\pm 1.3131.18±0.2931.18\pm 0.29

Reply to Weakness 4

Thank you for raising the question regarding scalability. A "million-point geometry" can mean a million-point geometric input or a million output points, and CALTO handles both. We tested the frozen 3D model, without retraining, on 16 held-out CAD-DR contexts (8 ID, 8 geometry OOD).

Million-point input. For each CAD surface we sampled 1,048,576 candidate points and drew the boundary supports from them. With 256 supports per set, CALTO reaches 2.21% (ID) and 1.93% (OOD), against 10.75% and 9.97% for GINO, the strongest baseline on CAD-DR (paper Table 2). Accuracy improves with denser geometry: from 128 to 1,024 supports per set, ID error falls from 3.04% to 1.68% and OOD from 2.40% to 1.65%.

Million-point output. With 1,048,576 queries per context, CALTO’s error is 2.23% (ID) and 1.98% (OOD), comparable to the standard evaluation with 4,096 queries per context; GINO’s error is 10.91% and 10.63%.

Time and memory. Both models run the same groups of 256 queries on one GPU. From 4,096 to 1,048,576 queries per context (256×), CALTO's time grows linearly from 0.040 s to 8.2 s and its peak GPU memory stays at 12.1 MiB; GINO needs 72.4 s and 1.11 GiB, so CALTO is 8.8× faster with 94× less memory. This follows from the architecture: queries attend to a fixed set of supports and to each other only within groups of 256, and neither the training labels nor inference need a volume mesh at any resolution (Appendix E.9).

Comments on the paper

1 comment
AuthorsOpenReview ↗

Common Response to Area Chair and All Reviewers

We sincerely thank the Area Chair and all four reviewers (xjRG, CzrR, og5Z, EPUZ) for their time and constructive feedback. Your questions led us to add quantitative evidence the paper lacked and to make the method easier to follow, and the paper is stronger for it. All new experiments are in the individual replies and in the revised paper, where changes are marked in blue.

Common concerns and our responses

1. Practical cost and speedup (xjRG W3; CzrR W, Q; og5Z W3, Q4; EPUZ W1, Q1, Q2).

With eight parallel workers, weak labels for a whole training set take 1–14 min on the 2D families against 10–70 min for P2 FEM (4.9–11× more), and 1.2 against 3.2 min on CAD-DR. Counting label generation and training, CALTO's one-time cost equals FEM's for 800–1,740 new-context requests; a request then takes 1.5 ms against 2.3 s. At equal training time, error at the full budget is 71–83% lower than the strongest baseline. In the inverse problem, with FEM forward solves kept, CALTO tangents halve the sensor residual in 3.48 s against 4.64 s and cut it tenfold in 5.55 s against 6.05 s; FEM tangents still reach lower final residuals, and we state this. Selecting checkpoints with weak validation labels only changes CALTO's error by at most 0.013 points.

2. Relation to Neural Green's Operators (NGO) and source of the gain (CzrR W; og5Z W1, Q1; EPUZ W2).

NGO contributes the principle of nonlinear conditioning with a linear action. CALTO differs in the operator it learns (the derivative of the solution map, which composes Green's operators with the perturbation), in its mesh-free point-set representation on varying geometries, and in learning from four-path labels with 15–88% error. With the strongest baselines retrained on strong FEM labels, CALTO's ID error is still 52–76% lower, and CALTO trained on W4 labels remains more accurate than they are. With three training seeds, exact linearity gives lower mean error at W4 for every direction budget, and lower Jacobian and VJP error in all four families.

3. Fairness and completeness of the comparison (og5Z W2, Q2, Q3; EPUZ W2, Q3).

When every model also receives weak primal labels, CALTO-Hybrid is still best on all four 2D benchmarks, 61–67% below the strongest jointly trained baseline (DE-DeepONet). We describe how each baseline is adapted, and report parameter counts and tuning. Tables 3 and 4 now report means and standard deviations over three training seeds.

4. Generalization, scaling and scope (xjRG W1, W2, Weakness 4; EPUZ Q4).

All OOD results are zero-shot. With 32 weakly labelled new contexts, SP-DN geometry-OOD error falls from 17.73% to 13.92%, and CALTO adapted on 8 contexts beats Transolver++ adapted on 32. The frozen 3D model reaches 1.93% geometry-OOD error with supports drawn from a million-point CAD surface and 1.98% at a million output queries, in 8.2 s and 12.1 MiB against 72.4 s and 1.11 GiB for GINO. Model size does not depend on the parameter dimension, and SL-RD results change by at most 0.03 points when the coarse state is replaced by P2 solutions of different resolution. For other PDE classes, CALTO needs unbiased observations that are linear in the direction and share randomness. WoS-type estimators supply them where a probabilistic representation exists, including parabolic problems, and a Monte Carlo linear solver on the discretized tangent equation supplies them where it does not.

5. Clarity (og5Z Weakness 4).

Section 3 now opens with a running SP-DN example, and the supervision pipeline is added to Figure 1.

Where the revised paper changed (marked in blue)

LocationChange
§2 Related WorkTime-dependent grid-free Monte Carlo; relation to NGO
§3 Problem FormulationRunning SP-DN example: model input, prediction, label generation
§4, Figure 1Supervision step: shared walks give a weak label per direction
§5 SetupLabel-generation time against FEM (2D families and CAD-DR)
§5.3, Tables 3–4Means ± s.d. over three seeds; reuse under scaled direction combinations
§5.4Inverse-solve cost: Jacobian time and total time at matched accuracy
§6 ConclusionLimitation and future work for other PDE classes
Appendix E (new)Extended experiments during rebuttal

Once again, we thank the Area Chair and the reviewers for their effort and for helping us improve the paper.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.