acceptodds
Under review as a conference paper at ICLR 2027

Residual-Aware Output Representations for Scientific Surrogates

17 views

Abstract

Scientific surrogates are commonly trained to predict a complete physical response, whereas designs care about performance metrics derived from that response. This creates a distinct failure mode when a design-relevant metric is controlled by a small physical residual recovered through cancellation. We propose the residual-aware output representation method, where the surrogate exposes the small residual as an explicit output coordinate, reconstructs the complete response through an invertible transformation, and retains the original response-space training objective. In radio-frequency passives, exposing dissipation through impedance-like coordinates reduces relative error by on differential inductors and on six-port transformers with minimal change in scattering-response error; it also reduces circuit-level prediction violations. We further validate the proposed method on nanophotonic metasurfaces, reducing relative absorptance error. These results establish residual-aware output representation as a practical approach to cancellation-sensitive scientific problems.

Primary area:
applications to physical sciences (physics, chemistry, biology, etc.)
Keywords:
Scientific machine learning, Surrogate modeling, Physics-aware machine learning, Inverse design, Electromagnetic modeling, RFIC

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_GzFWOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper addresses a failure mode in scientific surrogates: small errors in a complete physical response can produce large errors in downstream metrics governed by cancellation-sensitive residuals. The authors propose residual-aware output representations that explicitly encode the relevant residual, reconstruct the complete response through an invertible transformation, and retain the original response-space training objective. The accompanying analysis characterizes the directional amplification of prediction errors near physical boundaries. For RF passives, the method exposes dissipation through impedance-based coordinates and reduces relative quality-factor errors by approximately 3.3–6.8× while maintaining comparable scattering-response accuracy. A nanophotonic experiment demonstrates an approximately twofold reduction in relative absorptance error. These results support output parameterization as a practical way to improve derived-metric accuracy without replacing complete-response prediction with task-specific regression.

Critical strengths
  1. Clear and practically relevant problem. The directional error analysis goes beyond observing a mismatch between response and task accuracy. The measured reduction in radial prediction error provides useful empirical support for the proposed mechanism.
  2. Simple, physically motivated formulation. The invertible representation preserves the complete response and its consistency with derived metrics, while avoiding additional task supervision in the main experiments.
  3. Meaningful downstream validation. Selected circuit designs are reevaluated using solver responses, and improvements remain after adding passivity filtering to the baseline. This strengthens the practical relevance of the prediction gains.
Critical weaknesses

The link between sensitivity analysis and learning remains incomplete. Equations (8) and (10) use different coordinate-error metrics. Their comparison does not establish reduced sensitivity under identical physical perturbations, nor explain why the unchanged response-space loss produces more favorable learned error directions. The main source of improvement deserves clearer emphasis. The direct (R,X) ablation nearly matches (log R,X), suggesting that explicit dissipation coordinates contribute more than the logarithmic parameterization. This distinction is important given the emphasis on logarithmic coordinates in the general formulation.

Questions for authors

How does the theoretical coordinate-error metric relate to the error distribution actually learned under the unchanged response-space objective? Does the strong performance of direct (R,X) prediction extend to the other RF templates and downstream circuit tasks? How does a direct-S baseline with comparable scalar contractivity constraints perform?

4 comments

Thank you for the careful reading and informative feedback, and we answer your three questions below, two of them with new experiments, and we will revise the manuscript accordingly.

1. Sensitivity analysis and error. Whichever representation does not change the physical sensitivity of QQ to SS; it changes the direction of the network's own error. Under an identical perturbation of SS, ΔQ\Delta Q is the same for every representation, because QQ is computed from SS mathematically. Eqs. 8 and 10 compare a different quantity: the QQ error produced by an error of a given size in the quantities the network outputs. When the network outputs SS, its error is set against the small dissipative margin ϵ\epsilon, and a radial error KK gives a relative QQ error of about 2K/ϵ2K/\epsilon (Eq. 7). When it outputs log⁡R\log R, an error of the same size is a relative error in RR, and the relative QQ error has that same size (Eq. 10).

To compare the two models (same backbone) with one ruler, we measure their learned errors in the physical SS plane (Sec. 4.2, Fig. 2b). The mean radial error falls from 8.3×10−38.3\times10^{-3} to 1.5×10−31.5\times10^{-3}, which is a 5.5×5.5\times reduction, while the mean tangential error stays at the same scale (4.1×10−34.1\times10^{-3} and 5.9×10−35.9\times10^{-3}). As a result, K/TK/T moves from 2.082.08 to 0.260.26. The training loss (MAE of SS) is the same in both cases.

2. Source of the improvement; (R,X)(R,X) on the other templates and circuit tasks. Direct (R,X)(R,X) is an instance of the proposed representation. R=Zd ϵ/∣1−s∣2R=Z_{\mathrm d}\,\epsilon/|1-s|^2 vanishes with the dissipative margin, so RR is the residual, and (R,X)(R,X) gives it its own output coordinate with ρ(m)=m\rho(m)=m in place of ρ(m)=log⁡m\rho(m)=\log m. Its performance therefore supports the main claim: the gain comes from exposing the residual as an output coordinate. We trained (R,X)(R,X) on the other two templates and re-ran both circuit tasks (based on the protocol in Sec. 4.4, pooled over three seeds).

SS(R,X)(R,X)(log⁡R,X)(\log R,X)
Transformer rel. QpQ_{\mathrm p} / QsQ_{\mathrm s} error0.725 / 0.7380.103 / 0.1030.109 / 0.114
Spiral rel. QQ error0.8040.1610.119
Spiral non-passive test points1.17%0.09%0
Matching: IL violation, median / p95 (dB)4.58 / 11.150.23 / 1.220.11 / 0.26
Matching: optimality gap, median / p95 (dB)3.24 / 5.830.06 / 0.920.10 / 0.20
Matching: selected designs with σmax⁡>1\sigma_{\max}>1100%1.0%0
LC tank: PN violation, median / p95 (dB)2.35 / 4.790.14 / 1.350.23 / 1.27

Both residual-exposing representations reduce QQ error by 55–7×7\times on both templates at similar MAE of (S)(S).

The logarithm achieves clearly better result in circuit synthesis. It guarantees R>0R>0 for any network output, leading to a passive reconstructed response (Proposition 1). In transformer matching-network selection, where the design is chosen by minimizing the predicted insertion loss, (log⁡R,X)(\log R,X) halves the median insertion-loss violation (0.110.11 against 0.230.23 dB), lowers its 95th percentile from 1.221.22 to 0.260.26 dB and that of the optimality gap from 0.920.92 to 0.200.20 dB, and every selected design is passive (while (R,X)(R,X) coordinate has 1% non-passivity).

3. Direct SS with a scalar contractivity constraint. We trained two constrained componentwise-SS baselines on the differential inductor. The training protocols and loss definition here match those of the componentwise-SS row of Table 1. The only change is a map applied to the predicted differential-mode reflection s~d\tilde s_{\mathrm d}, the mode from which QQ is extracted, before the complete response is reconstructed.

  • (a) Radial projection: The network predicts sds_{\mathrm d} as in the baseline, and a prediction outside the unit circle is moved back onto that circle along the radial direction, while a prediction inside it is left untouched. This is the baseline plus the constraint ∣sd∣≤1.0|s_{\mathrm d}|\le1.0.
  • (b) Smooth radial map: The network's raw output is not used as sds_{\mathrm d} directly. It is first squashed: its angle is kept, and its magnitude rr is replaced by tanh⁡(r)\tanh(r), which is below 11 for any rr. Every prediction therefore lies inside the unit circle.
MAE$(S)$rel. QQ error
componentwise SS0.00620.172
SS + radial projection (a)0.00600.184
SS + tanh⁡\tanh radius (b)0.00710.071
(log⁡R,X)(\log R,X)0.00650.053

The radial projection (a) does not change QQ error, so we can conclude that enforcing contractivity is not the mechanism. The smooth radial map (b) is changing coordinates: near the boundary the network's radial output equals −12log⁡(1−∣sd∣)-\tfrac12\log(1-|s_{\mathrm d}|) up to a constant, which is already a residual-exposing coordinate, and as a result it recovers most of the gain in QQ accuracy. Both results support the paper's claim that the improvement comes from residual-aware output coordinates.

Thanks for your reply. Basically, I think the motivation is critical and practical. Many scenes, such as Battery Life Prediction, Machine Learning Interatomic Potential, face the problem demonstrated in this paper. The model's performance is always extremely high in metrics, while can only provide limited screening or ranking ability, since the residual error is inherently small. Despite this, since I am not familiar with RF passive components, I can't further increase my score to four. That's why I give a weak accept with confidence 2. Best luck to the authors, I hope the paper can be accepted. By the way, I would appreciate it if the author could extend this paper to other scientific applications.

Thank you for your time and for your supportive comments. We agree that the same structure appears well beyond RF, including in battery-life prediction and interatomic potentials: a model can be accurate on the complete response and still unreliable for the small residual quantity which determines the quality of designs. The atmospheric heating-rate case added in this rebuttal is one of our first steps in that direction.

Reviewer_pdvXOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

I could not evaluate the work. See Critical Weaknesses.

Critical strengths

I could not evaluate the work. See Critical Weaknesses.

Critical weaknesses
  1. I regret that I could not review the paper due to lack of clarity. Specifically, Line 87 in the introduction presents an equation that seems critical to the paper's motivation, but several quantities are undefined: What are the functions \rho and u? These are never defined. Somewhat less crucially, the dimensionality and/or set membership of all quantities in the introduction are left unclear (e.g., y, z, x). The authors should carefully proofread their paper for clarity before resubmission.

There are other examples of poor writing quality: e.g., Line 40 "because only the response-error component normal to the physical boundary is amplified to the first order". What physical boundary are we talking about? This is unclear.

Questions for authors

I have no questions that can be effectively addressed in the rebuttal.

2 comments

Thank you for your comments. Below we demonstrate where each quantity is defined in the original manuscript. In the revised version, we will move these definitions into the introduction.

ρ\rho and uu (Line 87). The equation at L87 previews the construction defined in Sec. 2.2 (Eq. 3, Line 132-134). There, m(y)>0m(y)>0 is the small physical residual (Line 89), ρ(m)=log⁡m\rho(m)=\log m is its logarithmic form, and u(y)u(y) is the remaining coordinates needed to reconstruct yy, chosen so that Φ\Phi is invertible. For RF passives, the residual is the resistance RR and the representation is z=(log⁡R,X)z=(\log R, X).

Quantities in the introduction. xx denotes the design parameters and yy the complete physical response (Line 29-30), with y∈Y⊂Rdy\in\mathcal{Y}\subset\mathbb{R}^d (Line 112); for RF passives, yy is the scattering response S(f)S(f), and z=Φ(y)z=\Phi(y) is the residual-aware representation of yy.

Physical boundary (Line 39). The boundary is m(y)=0m(y)=0, where the residual vanishes. It could be a conservation, passivity, or feasibility limit (Line 113). For RF passives it is the lossless unit circle ∣s∣=1|s|=1. Dissipation is carried by the narrow radial margin 1−∣s∣21-|s|^2 between the response and the unit circle (Line 50-52, Fig. 1b).

We hope these pointers make the paper's motivation and method clear, and we would be grateful if you would reconsider your assessment.

This paper is definitely worth a higher score. Even for the readers unfamiliar with the application, I think the motivation and method are already clear and easy to follow.

Reviewer_HTvZOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

A surrogate model can predict a physical response accurately overall and still get a design metric badly wrong when that metric depends on a small residual left after near-cancellation. The paper calls this "directional error amplification" (Eq. 2). It proposes predicting the log of the residual plus complementary coordinates, rebuilding the response through an invertible map, and keeping the original loss on the response. For RF passive components it uses (log R, X). Relative Q error falls 3.3× for differential inductors, 6.7×/6.5× for transformers and 6.8× for spiral inductors, with similar MAE(S) (Table 1). The paper also shows gains at circuit level (Table 3), a 2.01× improvement on metasurfaces (Sec. 4.5), and a replication on an open process design kit (Table 7).

Critical strengths

The controls are strong. Q-supervision, a Jacobian-weighted loss, an impedance-space loss and a separate Q head (Sec. 4.3, App. E) separate the effect of the representation from changes to the objective. Auxiliary Q supervision reaches relQ 0.044 only at 20.6× the baseline MAE(S) (Table 2), whereas the proposed representation improves Q without that cost.

Critical weaknesses

The ML novelty and generality are limited. Each domain needs a hand-built invertible map, and impedance coordinates are already standard in RF. App. B shows that plain (R, X) without the log already gets relQ 0.0576 vs 0.0525, so most of the gain comes from a known reparameterization. There is no general recipe for choosing the coordinates, and the only non-RF result is a modest 2.01× with an MLP on 8-dimensional outputs.

Questions for authors

Could you apply (log R, X) to the flat ResNet in Table 4, and ideally one more backbone? Outside cases with an obvious closed-form residual, how would someone build the coordinates and the invertible map? Could they be learned? Please add a stronger non-RF case, such as a field-level output or the formation-energy or drag settings cited in Sec. 1.

3 comments

Thank you for the detailed review and the constructive questions. We ran the experiments you asked for, and we will revise the manuscript accordingly. We divide the reply into three parts: (1) novelty, and how to construct the coordinates, (2) a non-RF case supporting the claim of the manuscript, (3) other backbones and the role of (R,X)(R,X).

1. Novelty.

What is known. The mathematical equation between SS and impedance is textbook knowledge: engineers use it to read LL and QQ from a SS response they already have. Transforming a regression target is also known, so we did not conclude the contribution in these domains.

What is new. We investigated each prior approach in ML venue, EDA venue and Circuit venue (discussed in Sec. 5 and tested for ablations in Sec. 4.3). We can conclude that each family of prior approaches gives up either the accuracy of the response or the accuracy of the metric read from it. The table below lists the comparison of approaches, with the values measured on the differential inductor.

approachMAE$(S)$rel. QQ errorwhat it sacrifices
componentwise response surrogate (SS)0.00620.00620.1720.172metric accuracy
direct prediction of QQ (separate head)--0.1550.155-0.2530.253the predicted QQ is not the QQ of the response
equivalent-circuit----restricts the response to a template (losing information)
auxiliary QQ supervision0.1290.129 (20.6×20.6\times)0.0440.044response accuracy
loss reweighted toward impedance--0.1740.174metric accuracy
target transformation (loss on (log⁡R,X)(\log R,X))0.300.30--response accuracy
passivity constraint on SS (new experiment)0.00600.00600.1840.184metric accuracy
proposed: residual coordinate, response-space loss0.0065\mathbf{0.0065}0.053\mathbf{0.053}--

The proposed method separates two roles which target transformation ties together: the residual becomes an output coordinate, while the objective stays on the complete response. The output remains a complete SS that circuit calculations reuse (Table 3).

Why it is not specific to RF. Equation 2 states the cause in general form: for a metric controlled by a small residual m(y)m(y), only the component of the response error along ∇m\nabla m is amplified, by ∥∇m∥/m\lVert\nabla m\rVert/m. The representation gives that component its own coordinate. Measured in the SS plane, the radial error of the learned model falls by 5.5×5.5\times while the tangential error stays at the same scale (Sec. 4.2). Nothing in this argument relies on impedance: (log⁡R,X)(\log R,X) is its RF instance, and the metasurface (Sec. 4.5) and atmospheric (Part 2) cases are instances whose coordinates are not impedance.

2. Constructing the coordinates.

The residual is given by the task. A design metric is computed from the response by a formula, and m(y)m(y) is the small quantity in that formula: the dissipated power behind QQ, the absorbed power behind an absorptance, the flux difference behind a heating rate. It is available in closed form from the definition of the metric.

Method. (i) Take m(y)m(y) from the definition of the metric. (ii) Use ρ=log⁡m\rho=\log m when m>0m>0 is guaranteed by a conservation or passivity bound. (iii) Complete the coordinates with a u(y)u(y) that makes the map invertible, and keep the loss on the reconstructed response.

domainresidual m(y)m(y)representation
RF passivesR∝1−∣s∣2R\propto 1-\lvert s\rvert^2(log⁡R,  X)(\log R,\;X)
metasurfaceA=1−∥v∥2A=1-\lVert v\rVert^2(log⁡A,  direction of v)(\log A,\;\text{direction of }v)
atmospheric columnflux difference across a layer(top flux, layer increments)

Can the coordinates be learned? In principle, yes: the map is invertible and differentiable, so it can be parameterized by an invertible network and trained with the definition of the metric as the signal; the response loss alone does not distinguish between invertible maps. In this paper the residual coordinate is taken from the definition of the metric, which is available in closed form in the cases we study, and the remaining coordinates use the simplest invertible completion. Learning the representation map is considered future work.

3. A field-level non-RF case: atmospheric heating rates. We added the heating-rate problem cited in Sec. 1 (Ukkonen, 2022). Its output is a vertical profile, and it has no connection to impedance.

Background. In every grid column, a weather or climate model computes how infrared radiation is absorbed and emitted by the atmosphere. The radiation solver returns the upward and downward energy fluxes F↑,F↓F^{\uparrow},F^{\downarrow} (W/m$^2$) at each vertical level. What the model uses is the heating rate of each layer, i.e. how fast radiation warms or cools it, which is set by the difference of the net flux Fnet=F↑−F↓F^{\mathrm{net}}=F^{\uparrow}-F^{\downarrow} between the two levels that bound the layer: Hi=gcp Fi+1net−Finetpi+1−pi.H_i=\frac{g}{c_p}\,\frac{F^{\mathrm{net}}_{i+1}-F^{\mathrm{net}}_{i}}{p_{i+1}-p_i}. The fluxes are hundreds of W/m$^2$, while their difference across one layer is about 1%1\% of them (median 0.8%0.8\% in our data). A flux error of 0.1%0.1\% is therefore of the order of 10%10\% of that difference, which is the cancellation structure of the paper with the flux difference as the residual.

Data. We implemented a gray (single-band) two-stream longwave solver with 6060 isothermal layers between 11 and 10001000 hPa (6161 levels) and a diffusivity factor of 1.661.66. The surface emits as a black body and no radiation enters at the top. A column is defined by six parameters, sampled independently: surface temperature (260260-310310 K), tropospheric lapse rate (44-8.58.5 K/km), tropopause height (88-1717 km), stratospheric temperature (190190-230230 K), absorber scale height (1.51.5-44 km), and total optical depth (0.50.5-1010, log-uniform). The solver reproduces the analytic limits: zero heating in a transparent column, and the closed-form surface net flux of an isothermal column to machine precision. We use 8,0008{,}000 columns for training and 2,0002{,}000 for testing.

Surrogate and training. The surrogate maps the six parameters to the complete response, F↑F^{\uparrow} and F↓F^{\downarrow} at the 6161 levels (122122 outputs). Both representations use the same backbone (MLP with four hidden layers of width 512512, GELU) and the same training: Adam, learning rate 10−310^{-3} with cosine decay, batch size 256256, 1,0001{,}000 epochs. Both minimize the same loss, the MSE on the standardized fluxes. The heating rate is not explicitly supervised.

Representations. Componentwise: the 122122 fluxes. Proposed: for each stream, the flux at the top level and the 6060 level-to-level increments; the fluxes are recovered by a cumulative sum before the loss. The increments are the residual. A heating rate can have either sign, so the residual coordinate is linear, ρ(m)=m\rho(m)=m, in place of the logarithm.

Metric. For each test column, the mean absolute heating-rate error over the 6060 layers, in K/day and also normalized by the root-mean-square heating rate of that column; we report the median over test columns. The typical heating-rate magnitude is 1.31.3 K/day.

flux MAE (W/m$^2$)heating-rate error (K/day)normalized heating-rate error
componentwise fluxes0.0440.02260.0088
proposed0.0410.01010.0039

The heating-rate error falls by 2.2×2.2\times (median of three seeds), and the flux error is slightly lower. Therefore, it proves effective of the residual-aware output representation and supports the claim of the manuscript.

4. Other backbones. We applied (log⁡R,X)(\log R,X) to two further backbones on the differential inductor. Both are residual networks built from the blocks of the flat network of Table 4.

  • Residual network with frequency input. Geometry and frequency are concatenated and passed through 66 residual blocks of width 512512 (3.43.4 M parameters); the network predicts SS one frequency at a time.
  • Flat residual network with a frequency-conditioned head. A residual trunk of the same width as the flat network of Table 4, whose per-frequency output layer is replaced by one 44-layer head shared across frequencies and fed with the Fourier frequency features of the main model. It has 19.019.0 M parameters, comparable to the 19.119.1 M of the flat network in Table 4.

Both are trained for 1,0001{,}000 epochs with the same training protocol.

backboneparametersrel. QQ error, SSrel. QQ error, (log⁡R,X)(\log R,X)reduction
attention operator (paper)6.7 M0.1720.0533.3×3.3\times
residual network, frequency input3.4 M0.2360.1521.55×1.55\times
flat residual network, frequency head19.0 M0.1110.0851.31×1.31\times

The proposed representation lowers the QQ error on each of the three backbones and in every training seed.

5. The role of (R,X)(R,X). Direct (R,X)(R,X) is an instance of the proposed representation. R=Zd ϵ/∣1−s∣2R=Z_{\mathrm d}\,\epsilon/|1-s|^2 vanishes with the dissipative margin ϵ=1−∣s∣2\epsilon=1-|s|^2, so RR is the residual, and (R,X)(R,X) gives it its own output coordinate with ρ(m)=m\rho(m)=m in place of ρ(m)=log⁡m\rho(m)=\log m. Its performance therefore supports the main claim: the gain comes from exposing the residual as an output coordinate. We trained (R,X)(R,X) on the other two templates and re-ran the circuit tasks.

SS(R,X)(R,X)(log⁡R,X)(\log R,X)
Transformer rel. QpQ_{\mathrm p} error0.7250.1030.109
Spiral rel. QQ error0.8040.1610.119
Matching: IL violation, median / p95 (dB)4.58 / 11.150.23 / 1.220.11 / 0.26
Matching: selected designs with σmax⁡>1\sigma_{\max}>1100%1.0%0

Both residual-exposing representations reduce the QQ error by 55–7×7\times. The logarithm is clearly better in circuit synthesis: it guarantees R>0R>0 for any network output, hence a passive reconstructed response, and in matching-network selection it halves the median insertion-loss violation and every selected design is passive.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Revisions and New Results

We thank the reviewers for their feedback. In our rebuttal, several main results are provided and summarized as follows:

  1. Source of the QQ accuracy improvement. The gain comes from exposing the residual as an output coordinate. Direct (R,X)(R, X), which also exposes it, reduces relative QQ error by 5-7× on the transformer and spiral templates, while a contractivity constraint alone does not help. The logarithm additionally guarantees a passive response: in matching-network selection, it halves the insertion-loss violation, and every selected design is passive.

  2. Non-RF field-level case. On atmospheric heating rates, a vertical-profile output with no impedance involved, the residual-aware representation reduces heating-rate error by 2.2× with slightly lower flux error.

  3. Other backbones. The improvement holds on all three backbones and every training seed.

We are happy to clarify further, and we hope these results address the reviewers’ concerns. We would be grateful if they could be reflected in the updated evaluations.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.