acceptodds
Under review as a conference paper at ICLR 2027

Recursive Transformer with State Injection and Per-Slice Supervision for Thermo-Mechanical Surrogate Modeling

Abstract

Transformer-based surrogate models are increasingly replacing expensive first-principles simulation in engineering design, but conventional architectures are often over-parameterized for the small, low-dimensional datasets typical of this setting, where large simulation data is expensive to generate. Under these conditions, excess parameter capacity leads to overfitting and unnecessary memory/compute overhead — this motivates architectures that trade additional compute for additional parameters, an idea explored by weight-shared recursive architectures such as the Universal Transformer (UT). We adapt this recursive paradigm to surrogate thermo-mechanical analysis of semiconductor advanced packages, where each sample consists of a fixed set of Finite Element Analysis (FEA) material and design parameters paired with a sequence of stress contours across the package's layers, and simulation data is far too scarce to support a large-parameter model. However, UT's recursion is "unguided": every recursion step performs the same undifferentiated refinement, with no inherent meaning attached to being at loop 3 versus loop 7— and although each frame in the sequence shares the same FEA parameters, UT treats each frame as an independent sample rather than part of a continuous progression through the sequence. We introduce DEPTH, an RNN-like recursive transformer, that instead gives each recursion step genuine meaning: its single transformer block is applied recursively as a sequence, consuming the static design parameters once and, through recursion produces a matching stress contours at each z-axis (which is the sequence), using exactly one forward pass per layer. An ablation study shows DEPTH's two key ingredients — state injection and per-step supervision are synergistic, not additive— removing either collapses retrieval to near-chance and only their combination recovers strong performance. Since UT must loop per-frame to be competitive, we sweep its internal recursion from 1x to 9x (till it collapse) to quantify the compute overhead needed to match DEPTH: it requires several times DEPTH's per-frame cost to reach parity, then collapses sharply beyond that, while DEPTH matches or exceeds this peak at a fraction of the compute. Gating experiments show carry-gating modestly helps while injection-gating hurts. We benchmark DEPTH against a Tiny Recursive Model and UT (which instead loops internally within each frame) on predictive performance, parameter count, and FLOPs on actual Ansys simulated Stress and Warpage datasets. We validate generalization on a third task, a Laplace PDE capacitance-field solver, confirming recursive weight-sharing transformers as an effective, parameter-efficient choice for small-data engineering surrogate modeling.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
Recursive Transformer, Surrogate modeling, Finite element analysis (FEA), per slice supervision, state injection

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_FqeoOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes DEPTH, a weight-shared recursive transformer for predicting spatially indexed PDE solution fields. The model encodes the input conditions once, then propagates a hidden state sequentially along a physical coordinate using a shared transformer block, with predictions and supervision at each spatial slice. Experiments on thermo-mechanical stress, warpage, and electrostatic field datasets show competitive accuracy and computational efficiency compared with several recursive transformer and recurrent baselines.

Critical strengths
  1. The proposed architecture is simple and computationally efficient. By sharing the transformer block and carrying hidden states across spatial slices, it avoids some redundant computations compared with predicting each slice independently.
  2. The paper provides reasonable ablation studies on coordinate injection, per-slice supervision, recursion depth, and gating mechanisms, which help explain the effects of different design choices.
Critical weaknesses
  1. I find the methodological novelty somewhat limited. The proposed approach looks quite similar to a coordinate-conditioned recurrent latent-state model, with a transformer block used as the recurrent update. Weight sharing, hidden-state propagation, and per-step supervision are all well-established ideas. It is not entirely clear to me what the main technical novelty is compared with existing recurrent and latent-state prediction methods.

My major concern is the motivation and justification for sequentially predicting spatial slices. The considered thermo-mechanical and electrostatic problems are governed by globally coupled, largely elliptic equations. These problems do not naturally have a one-directional information propagation process along a spatial coordinate. Therefore, predicting the solution through a single sequential sweep in a fixed direction seems to impose an artificial ordering that may not properly capture global interactions and dependencies.

I am not convinced that the paper sufficiently addresses this issue. For such globally coupled problems, the solution at one location can depend on conditions throughout the entire domain, rather than just information propagated from previous slices. The paper provides little analysis of how the proposed approach handles these nonlocal dependencies or maintains global physical consistency. It would be important to examine the effects of prediction direction, reverse/bidirectional propagation, or some form of global/iterative refinement.

  1. I also find the experimental validation rather limited. The tested problems involve relatively simple, structured, low-resolution fields (13 × 13 grids with only 15–16 slices) and very few input parameters. It is unclear whether the method would remain effective for much higher resolutions, more complex geometries, heterogeneous problems, or stronger global coupling.

The baseline comparisons are also not fully convincing. Most comparisons are against recursive transformer variants, while the basic RNN baseline has much lower representational capacity. I would expect comparisons with stronger and more relevant methods, such as coordinate-conditioned models, modern local-global neural operators, and latent/compressed transformer-based models.

In addition, the evaluation mainly focuses on latent-space retrieval metrics, which do not directly reflect the accuracy or physical consistency of the predicted PDE solutions. More detailed field-wise errors, physics-related evaluations, and actual computational timings would be helpful to better support the claimed accuracy and efficiency.

Questions for authors
  1. What is the physical motivation for predicting spatial slices sequentially in one direction, especially for globally coupled PDE problems? How does the model capture nonlocal dependencies and global effects of boundary conditions? Have the authors considered reverse/bidirectional propagation or iterative refinement?

  2. How does DEPTH perform on higher-resolution fields, longer sequences, nonuniform spatial discretizations, and more complex geometries? In particular, can a model trained on 16 slices generalize to substantially more slices without retraining?

  3. Could the authors compare against stronger, capacity-matched baselines, including modern coordinate-conditioned models, neural operators, and latent/compressed transformer approaches? How would the results change with a more faithful TRM implementation?

  4. Could the authors report more comprehensive field-wise prediction errors, physical-consistency measures, and actual training/inference times, rather than mainly retrieval scores and estimated FLOPs? Also, could the authors clarify the test-based run selection in Appendix B.3 and report results consistently across all independent runs?

14 comments

Q3: Stronger, capacity-matched baselines. (Could the authors compare against stronger, capacity-matched baselines, including modern coordinate-conditioned models, neural operators, and latent/compressed transformer approaches? How would the results change with a more faithful TRM implementation?)

We thank the reviewer for this suggestion. All new models are within 2% of DEPTH-1b's parameter count, use the same per-slice loss and optimiser, and predict the same frozen-autoencoder latent.

ConcernResponseLocation
"compare against stronger, capacity-matched baselines"All baselines are now parameter-matched to DEPTH-1b (9,336): DeepONet (9,245), RNN-emb (9,319) and an MLP-Mixer variant of DEPTH (9,432), with attention replaced by token mixing and everything else unchanged. DEPTH-1b leads on every metric: MAP 0.617 ± 0.068 against 0.540 (RNN-emb), 0.529 (DeepONet D) and 0.436 (MLP-Mixer, at 2.4× DEPTH's FLOPs).Appendix F, Table 10; Sec. 3.3; Sec. 5
"modern coordinate-conditioned models" and "neural operators"DeepONet D is the standard operator-learning model for parameter-to-field maps, and it is also coordinate-conditioned: it predicts each slice independently from the design and z, with no state carried between slices. It reaches MAP 0.529 ± 0.049 against 0.617 ± 0.068 for DEPTH-1b. DEPTH forms the interactions between design parameters from the raw inputs through attention over parameter tokens, and carries state between slices. DeepONet D uses about 20× fewer FLOPs but trails DEPTH-1b on every metric.Appendix F, Tables 10–11; Sec. 5
"latent/compressed transformer approaches"All models already operate in the same compressed latent space: each predicts the 48-dim latent of a frozen autoencoder, decoded identically, so the comparison is among latent-space predictors and isolates the predictor architecture.Appendix F, "Other baseline families"; Appendix C.1
"How would the results change with a more faithful TRM implementation?"Our simplifications are no gradient through the latent updates, no deep supervision and no adaptive halting. The simplified version already needs 27–34× DEPTH's FLOPs to approach it (trm-zy-1b: MAP 0.580). The full procedure adds repeated supervision steps and halting, which increase compute further, so the efficiency gap would widen.Appendix E; Table 2

 

For transparency, Appendix A.3 provides complete forward and backward flowcharts for DEPTH and each ablation (No-carry, No-coord, No-attn), showing exactly which component each variant removes. Likewise Appendix A.4 for UT.

 

Table 10: Capacity-matched comparison on Stress10K

MethodParamsFLOPsR@1 ↑MAP ↑RMSE ↓SSIM ↑Peak ↓
DeepONet D9,2450.11M0.386 ± 0.0500.529 ± 0.0490.0150 ± 0.00170.958 ± 0.0100.067 ± 0.007
RNN-emb9,3190.86M0.394 ± 0.0320.540 ± 0.0280.0156 ± 0.00050.946 ± 0.0020.065 ± 0.002
No-attn DEPTH9,4325.47M0.292 ± 0.0820.436 ± 0.0830.0204 ± 0.00720.919 ± 0.0530.082 ± 0.017
DEPTH-1b9,3362.29M0.476 ± 0.0770.617 ± 0.0680.0132 ± 0.00180.963 ± 0.0070.057 ± 0.009

 

Table 11: How each model in the capacity-matched comparison

MethodDesign inputDesign interactions formed bySlice coordinate zzCarry
DeepONet A4 raw parametersMLP on raw parameterstrunk MLP, per sliceno
DeepONet D70 polynomials (degree 4)explicit polynomial featurestrunk MLP, per sliceno
RNN-emb4 raw parameterslinear embedding + tanh updateS(z)S^{(z)} input at each stepyes
No-attn DEPTH4 parameter tokensMLP token mixingS(z)WsS^{(z)} W_s injected into stateyes
DEPTH-1b4 parameter tokensattention over tokensS(z)WsS^{(z)} W_s injected into stateyes

Weakness 1 (major concern) and Q1

(What is the physical motivation for predicting spatial slices sequentially in one direction, especially for globally coupled PDE problems? How does the model capture nonlocal dependencies and global effects of boundary conditions? Have the authors considered reverse/bidirectional propagation or iterative refinement?)

ConcernResponseLocation
"What is the physical motivation for predicting spatial slices sequentially?"All slices of a design come from a single global FEA solve (Table 6), so no slice is physically computed before another. DEPTH reflects this: it encodes the full design once as the initial state H(0), so every step starts with global information about the whole domain. Predicting slices in sequence is therefore an inductive bias, not a model of information travelling through the material. It encodes two physical properties. First, neighbouring slices are strongly correlated, so carrying a state from one slice to the next lets the model predict each slice as a change from the previous one rather than from scratch. Second, in the package, the stress in each layer is set by its CTE mismatch with the layers above and below it, so visiting slices in physical order lets the state follow this layer-to-layer structure. The No-carry ablation, which predicts each slice independently, supports this: removing the carry lowers MAP from 0.617 to 0.11 (Table 1).Sec. 3.1, Eq. 1; Sec. 6
"governed by globally coupled, largely elliptic equations"The two benchmarks pose different problems. Package stress and warpage are governed by the static equilibrium equations of linear thermoelasticity over a heterogeneous 3D domain: a stack of materials (die, moulding compound, underfill, redistribution layer) whose stiffness and CTE jump at each interface. The CTE mismatch drives stress concentrations at these interfaces. The equations are elliptic, but the field changes sharply from layer to layer, so the slices are physically distinct. The PINN benchmark instead solves Laplace’s equation on a homogeneous domain. Each slice is an independent 2D solve at a different plate width d, so the slices are not spatially coupled; moving from one slice to the next follows a smooth change in a geometric parameter. DEPTH gives the best or near-best MAP on both.Sec. 6; Appendix C.2–C.3; Tables 2–3
"maintains global physical consistency"For PINN, we test the governing equation directly on every predicted slice with the 5-point residual r^i,j=V^i+1,j+V^i−1,j+V^i,j+1+V^i,j−1−4V^i,j\hat r_{i,j} = \hat V_{i+1,j} + \hat V_{i-1,j} + \hat V_{i,j+1} + \hat V_{i,j-1} - 4\hat V_{i,j} at each interior node, averaged as ∣r^∣‾\overline{\lvert\hat r\rvert}. The autoencoder-reconstructed ground truth gives the reference floor (5.90×10−35.90\times10^{-3}). DEPTH-1b tracks it at every slice (ratio 0.985). UT3x-1b matches it only on average, through errors that cancel between shallow and deep slices (1.004). RNN-emb exceeds it at every slice, with the gap growing with depth from about 10% to 17% (1.114).Sec. 6, Eq. (X); Fig. 4

 

Table 6: State update of DEPTH and its three ablated variants

VariantUpdate
DEPTHH(z)=RECUR(H(z−1)+S(z)Ws)H^{(z)} = \mathrm{RECUR}\big(H^{(z-1)} + S^{(z)} W_s\big)
No-attnH(z)=RECURmix(H(z−1)+S(z)Ws)H^{(z)} = \mathrm{RECUR}_{\mathrm{mix}}\big(H^{(z-1)} + S^{(z)} W_s\big)
No-coordH(z)=RECUR(H(z−1))H^{(z)} = \mathrm{RECUR}\big(H^{(z-1)}\big)
No-carryH(z)=RECUR(E+S(z)Ws)H^{(z)} = \mathrm{RECUR}\big(E + S^{(z)} W_s\big)

 

Figure 4: Laplace residual on PINN

Model|r| (×10⁻³)Ratio
GT5.901.000
RNN-emb6.581.114
UT3x-1b5.931.004
DEPTH-1b5.820.985

 

Table 1: Component-removal ablations of DEPTH on Stress10K

R@1 ↑R@5 ↑MAP ↑RMSE ↓SSIM ↑Peak ↓
RNN-emb0.40 ± 0.0310.73 ± 0.0210.55 ± 0.0270.0157 ± 0.00050.95 ± 0.0030.065 ± 0.003
DEPTH (full)0.48 ± 0.0770.80 ± 0.0630.62 ± 0.0680.0132 ± 0.00180.96 ± 0.00740.057 ± 0.0086
No-attn0.29 ± 0.0820.61 ± 0.0880.44 ± 0.0830.0204 ± 0.00720.92 ± 0.0530.082 ± 0.017
No-coord0.35 ± 0.0290.69 ± 0.00320.50 ± 0.0200.0189 ± 0.00250.94 ± 0.00190.074 ± 0.0045
No-carry0.05 ± 0.0200.15 ± 0.0410.11 ± 0.0310.0597 ± 0.0210.82 ± 0.0320.157 ± 0.024

Weakness 1 (major concern) and Q1

(What is the physical motivation for predicting spatial slices sequentially in one direction, especially for globally coupled PDE problems? How does the model capture nonlocal dependencies and global effects of boundary conditions? Have the authors considered reverse/bidirectional propagation or iterative refinement?)

ConcernResponseLocation
"seems to impose an artificial ordering"We tested the ordering directly. With no propagation at all (No-carry DEPTH, each slice predicted independently), MAP falls from 0.62 to 0.11; with a random slice order it is 0.51. Propagation along neighbouring slices is therefore essential, and removing the ordering hurts rather than helps.Sec. 5; Sec. 6; Table 1
"Have the authors considered reverse ... propagation?"Yes. Bottom-up and top-down sweeps perform the same: MAP 0.62 vs. 0.61, R@1 0.48 vs. 0.47, with RMSE and SSIM equal within one std. The result does not depend on direction, consistent with load transfer having no preferred direction.Sec. 6; Table 1
"How does the model capture nonlocal dependencies and global effects of boundary conditions?"Global conditions enter through H(0) and are available at every step. In PINN, the boundary-defining parameters (a, b) are encoded in H(0), while the plate width d, which moves the boundary junction, is the slice coordinate. Because each target slice is a full-domain solution, the per-slice loss trains every state on the global field. If one-directional propagation lost global information, errors would build up towards the end of the sweep; instead, DEPTH matches the ground-truth physics at every slice, including the last (next row).Sec. 3.1; Sec. 6; Fig. 4
"bidirectional propagation or iterative refinement?"We added an iterative refinement pass after the sweep. The standard sweep gives H(z)=RECUR(H(z−1)+S(z)Ws)H^{(z)} = \mathrm{RECUR}(H^{(z-1)} + S^{(z)}W_s) for z=1,…,Zz = 1,\dots,Z. A second pass then recomputes each slice from both of its neighbours, using the same block: H~(z)=RECUR(wl H(z−1)+(1−wl) H(z+1)+S(z)Ws)\tilde H^{(z)} = \mathrm{RECUR}\big(w_l\, H^{(z-1)} + (1-w_l)\, H^{(z+1)} + S^{(z)}W_s\big), with wl=0.6w_l = 0.6 weighting the slice below and 1−wl=0.41 - w_l = 0.4 the slice above (the top slice uses its lower neighbour only). This is a learned Jacobi relaxation step: every slice is updated from the first-pass states of its neighbours, and H~(z)\tilde H^{(z)} gives the prediction. In a preliminary run it reaches MAP 0.55, within the range of a single sweep (0.617 ± 0.068); we will evaluate it over multiple seeds for the final version. Together with the direction test, this suggests that a single sweep from the global initial state already captures most of the coupling at this scale.Sec. 6; new experiment (rebuttal)

   

Table 1: Sweep order along zz for DEPTH on Stress10K

R@1 ↑R@5 ↑MAP ↑RMSE ↓SSIM ↑Peak ↓
Bottom-up0.48 ± 0.0770.80 ± 0.0630.62 ± 0.0680.0132 ± 0.00180.96 ± 0.00740.057 ± 0.0086
Top-down0.47 ± 0.0260.79 ± 0.0130.61 ± 0.0200.0124 ± 0.00060.97 ± 0.00300.050 ± 0.0027
Random0.36 ± 0.100.68 ± 0.120.51 ± 0.100.0164 ± 0.00320.95 ± 0.0140.072 ± 0.010
Reviewer_c5UXOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

DEPTH uses a weight-shared transformer to predict physically indexed output sequences from static design conditions. A tokenized state is initialized from the condition and propagated across coordinates, producing a supervised prediction at each step. Experiments on stress, warpage, and electrostatic fields support favorable accuracy–computation trade-offs against the evaluated transformer baselines. The main concerns involve the distinct methodological contribution, comparisons with alternative recurrent predictors, and the alignment between experimental evidence and explanatory claims.

Critical strengths
  • Useful computational-sharing perspective. Propagating state across related outputs assigns each recurrent step a distinct prediction target and provides a reasonable inductive bias for the studied tasks.

  • Supported accuracy–computation trade-offs. On Stress10K, Figure 4 and Table 10 report comparable MAP and reconstruction quality for DEPTH and UT-5x, using 16 versus 80 block applications per design. Other datasets and backbone sizes provide additional evidence of favorable trade-offs.

  • Substantive empirical study. Three datasets, two backbone sizes, repeated runs, and several ablations provide useful empirical coverage. Neutral and negative findings concerning injection locations and gating also help characterize the design choices.

Critical weaknesses
  • The methodological contribution needs sharper delineation. Static-condition initialization, shared state transitions, and per-output supervision fit the conditional recurrent modeling framework discussed in §1.1. DEPTH provides a useful architectural adaptation, but the distinct methodological insight contributed by physical-coordinate alignment and the tokenized attention transition remains less clearly established than its empirical effectiveness.

  • Comparisons with alternative cross-output recurrent predictors remain limited. The principal transformer comparisons contrast cross-slice state propagation with independent-slice prediction. Although RNN baselines are included, the raw-input version is substantially capacity-limited, while the polynomial version uses a different initial representation. One or two comparably budgeted conditional recurrent decoders—such as a GRU decoder or a shared MLP transition—with learned initialization, per-step coordinate inputs, and per-slice supervision would provide more informative comparisons. These would help assess DEPTH’s architectural advantage over other models that also share computation across outputs. (§3.4, Table 3)

  • Some mechanism claims exceed the ablation evidence. Both injection variants receive coordinates at every step, so Table 1 does not establish that coordinates must be supplied at every step. Likewise, the final-slice supervision ablation removes both intermediate prediction targets and their local gradient contributions. Its performance collapse therefore does not independently establish the proposed gradient-stability explanation. These experiments support the tested configurations more directly than the claimed necessity or causal mechanisms. (§3.2, §3.5, Appendix B.1)

  • Result interpretation and dataset descriptions contain inconsistencies. Section 4.1 states that additional recursion worsens two-block UT performance on PINN, whereas Table 4 reports higher mean values on all three retrieval metrics, including MAP from 0.766 to 0.782; the subsequent discussion also acknowledges this increase. Separately, Appendix A.2.1 reports 625 designs and 2,000 test slices, while the main text and Table 7 specify 500 training and 124 test designs. These discrepancies require correction or explanation, although they do not by themselves invalidate the overall results.

Questions for authors

Please refer to weaknesses.

4 comments
ConcernResponseLocation in revised PDF
W1. What new idea the physical coordinate and the attention transition add is less clear than the strong results (§1.1)DEPTH shares three features with conditional recurrent decoders such as Show and Tell (CVPR 2015) and LE-PDE (NeurIPS 2022) (closer to no-coord. DEPTH), which encode the parameters once and evolve a single global latent vector with an MLP. DEPTH differs in two ways. First, it recurses along the physical depth coordinate, not an abstract sequence index. Second, its transition is an attention block over parameter tokens, not an update of a single hidden vector. The global design parameters are kept as initial static tokens, and one shared attention block re-mixes them at every physical slice, conditioned on that slice’s live token (changes every recursion). This lets a ~9k-parameter block learn parameter interactions that vary with depth, directly from raw inputs, and the same shared weights produce a slice-specific prediction at every depth. At the same parameter budget, a single-vector conditional recurrence (RNN-emb) and a feature-engineered conditional operator (DeepONet D) fall short of DEPTH’s accuracy.Sec. 1.1
W1 (1). Each step is a physical sliceIn both models, each step's output is carried into the next step. In Show and Tell, a step is the next word. In DEPTH, a step is the next physical slice: the carried output H(z−1)H^{(z-1)} is combined with that slice's real zz-coordinate, H(z−1)+S(z)WsH^{(z-1)} + S^{(z)} W_s, so each recursion step is tied to a physical location in the package. On PINN, the Laplace residual of DEPTH's predictions tracks the ground-truth floor at every slice (ratio 0.985) and shows no drift along the sweep.Sec. 3.1, Eq. (1); Sec. 5, Table 1; Sec. 6, "Physical consistency check"
W1 (2). The state is a set of design tokens, not one vectorIn Show and Tell, the hidden state is the output of an LSTM cell, a single vector. In DEPTH, the hidden state H(z)H^{(z)} is the output of the RECUR transformer block: 4 static but carried tokens (EMC modulus, EMC CTE, die size, die gap). Keeping the parameters as separate tokens lets attention learn how they interact directly from the raw inputs.Sec. 3.1; Appendix F Table 10
W1 (3). The transition re-mixes the parameters at every sliceThe LSTM cell mixes the condition once, inside one vector. DEPTH's RECUR (self-attention followed by an MLP) re-mixes the 4 carried tokens at every slice, with a live depth signal, S(z)S^{(z)} , state injected each recursion. The interaction between static and live tokens can therefore change with attention. This matters for multi-material stacks, where different parameters dominate at the die, the moulding compound and the interfaces.Sec. 3.1, Eq. (1); Table 6
W1 (summary). Evidence that (2) and (3) drive performanceThe ablations in Sec. 5 show that (2) and (3) are what drive performance. Removing the carry (No-carry DEPTH) drops MAP from 0.62 to 0.11. Replacing attention with a shared-MLP transition (No-attn DEPTH) drops it to 0.44. A parameter-matched RNN-emb reaches 0.55. Against recursive transformers, the design is also efficient: the parameter count stays fixed and cost grows linearly with the number of slices. The only baseline that matches DEPTH-1b's MAP, UT5x-1b, is 4.3× slower.Sec. 5, Table 1; Appendix G, Tables 12–13
W1 (extended). The physical step admits physically meaningful extensionsBecause each step is a physical slice, the recursion can be smoothed along zz, which has no direct counterpart when a step is the next word. We give two extensions, each with a fixed β\beta and no extra parameters. Leaky DEPTH smooths the carried state across neighbouring slices, EMA DEPTH smooths the depth signal instead. Both are evaluated in the smoothing group of the ablation table.Sec. 5, Table 1 (smoothing); Appendix B

 

Table 6: State update of DEPTH, its three ablated variants and two smoothing

VariantUpdate
DEPTHH(z)=RECUR(H(z−1)+S(z)Ws)H^{(z)} = \mathrm{RECUR}\big(H^{(z-1)} + S^{(z)} W_s\big)
No-attnH(z)=RECURmix(H(z−1)+S(z)Ws)H^{(z)} = \mathrm{RECUR}_{\mathrm{mix}}\big(H^{(z-1)} + S^{(z)} W_s\big)
No-carryH(z)=RECUR(E+S(z)Ws)H^{(z)} = \mathrm{RECUR}\big(E + S^{(z)} W_s\big)
No-coordH(z)=RECUR(H(z−1))H^{(z)} = \mathrm{RECUR}\big(H^{(z-1)}\big)
Leaky DEPTHH(z)=βH(z−1)+(1−β) RECUR(H(z−1)+S(z)Ws)H^{(z)} = \beta H^{(z-1)} + (1-\beta)\,\mathrm{RECUR}\big(H^{(z-1)} + S^{(z)} W_s\big)
EMA DEPTHv(z)=βv(z−1)+(1−β)S(z)Ws,H(z)=RECUR(H(z−1)+v(z))v^{(z)} = \beta v^{(z-1)} + (1-\beta) S^{(z)} W_s,\quad H^{(z)} = \mathrm{RECUR}\big(H^{(z-1)} + v^{(z)}\big)
ConcernResponseLocation in revised PDF
W2. The old RNN baselines were too small or used different inputs. Compare with one or two recurrent predictors of the same size (e.g. GRU or shared-MLP transition) that use learned initialisation, a coordinate input at each step and per-slice supervision (§3.4, Table 3)We replaced the old RNN baselines with RNN-emb, a recurrent predictor of comparable size (9,319 vs. 9,336 parameters) that uses a learned embedding, as DEPTH does. RNN-emb reaches MAP 0.546, against 0.617 for DEPTH-1b. We also added the shared-MLP transition (No-attn DEPTH): DEPTH with self-attention replaced by a token-mixing MLP, essentially an MLP-Mixer block (Tolstikhin et al., 2021), of comparable size (9,432 parameters, including LayerNorm). DEPTH updates H(z)=RECUR(H(z−1)+S(z)Ws)H^{(z)} = \mathrm{RECUR}(H^{(z-1)} + S^{(z)}W_s), while No-attn DEPTH updates H(z)=RECURmix(H(z−1)+S(z)Ws)H^{(z)} = \mathrm{RECUR}_{\mathrm{mix}}(H^{(z-1)} + S^{(z)}W_s). Everything else is the same as DEPTH, including per-slice supervision and a coordinate input at each step. It reaches MAP 0.44, against 0.62 for DEPTH. Both baselines score below DEPTH at the same budget. The shared-MLP transition (No-attn DEPTH) also scores below RNN-emb, but well above UT1x-1b (MAP 0.249), which has attention but no carried state.Sec. 3.3, Table 2; Sec. 5, Tables 1 and 7
W2 (physical consistency). Beyond accuracy, how do the recurrent predictors compare?We also compare the recurrent predictors on physical consistency on PINN, where every slice must satisfy the Laplace equation. We compute the 5-point finite-difference residual on each decoded slice, relative to the residual of the autoencoder-reconstructed ground truth, which sets the achievable floor. DEPTH-1b tracks the ground truth at every slice (ratio 0.985). RNN-emb exceeds it at every slice (ratio 1.114), and the gap grows from about 10% to 17% with depth.Sec. 6, "Physical consistency check"
W2 (GRU baseline). How does DEPTH compare with a gated recurrence such as a GRU?Leaky DEPTH already beats plain DEPTH on five of six metrics, which suggests that controlling how much of the live token is carried across slices matters. A natural next step is a gated DEPTH, where a learned gate replaces the fixed leak rate. We plan to benchmark both variants against a Gated Recurrent Unit (GRU) based recurrence at matched parameter budgets. This will test whether attention over parameter tokens still pays off once the baseline also has gated memory.Appendix B

 

Table 1 and 10 (combined): Component-removal ablations of DEPTH on Stress10K

R@1 ↑R@5 ↑MAP ↑RMSE ↓SSIM ↑Peak ↓
DeepONet D0.39 ± 0.0500.70 ± 0.0490.53 ± 0.0490.0150 ± 0.00170.96 ± 0.0100.067 ± 0.007
RNN-emb0.40 ± 0.0310.73 ± 0.0210.55 ± 0.0270.0157 ± 0.00050.95 ± 0.0030.065 ± 0.003
DEPTH0.48 ± 0.0770.80 ± 0.0630.62 ± 0.0680.0132 ± 0.00180.96 ± 0.00740.057 ± 0.0086
No-attn0.29 ± 0.0820.61 ± 0.0880.44 ± 0.0830.0204 ± 0.00720.92 ± 0.0530.082 ± 0.017
No-carry0.05 ± 0.0200.15 ± 0.0410.11 ± 0.0310.0597 ± 0.0210.82 ± 0.0320.157 ± 0.024
EMA DEPTH0.45 ± 0.0470.77 ± 0.0530.59 ± 0.0480.0143 ± 0.00120.96 ± 0.00500.061 ± 0.0036
Leaky DEPTH0.52 ± 0.0280.83 ± 0.0220.66 ± 0.0240.0135 ± 0.000700.97 ± 0.00300.056 ± 0.0026

 

Table 7: Parameter counts of DEPTH, its three ablated variants and UT1x-1b

ComponentWeightsDEPTH / No-carryNo-coordNo-attnUT1x-1b
EmbeddingWe1–We4 (24 each; UT adds We5)969696120
Depth injectionWsW_s (24)24–24–
AttentionWQ,WK,WV,WoW_Q, W_K, W_V, W_o (384 each)1,5361,536–1,536
Token mixingWa,WbW_a, W_b (4 × 192, 192 × 4)––1,536–
Skip 1none0000
LayerNorm 1γ1,β1\gamma_1, \beta_1 (24 each)––48–
MLPW1,W2W_1, W_2 (24 × 64, 64 × 24)3,0723,0723,0723,072
Skip 2none0000
LayerNorm 2γ2,β2\gamma_2, \beta_2 (24 each)––48–
Final layerWfinalW_{\mathrm{final}} (96 × 48; UT 120 × 48)4,6084,6084,6085,760
Total9,3369,3129,43210,488

 

Figure 4: Laplace residual on PINN

Model|r| (×10⁻³)Ratio
GT5.901.000
RNN-emb6.581.114
UT3x-1b5.931.004
DEPTH-1b5.820.985
ConcernResponseLocation in revised PDF
W3. Both injection methods give the coordinate at every step, so Table 1 cannot show the coordinate must be given at every step (§3.2)We agree that the previous Table 1 alone cannot show this; the per-step coordinate is tested in Sec. 5. DEPTH supplies the coordinate at every slice, H(z)=RECUR(H(z−1)+S(z)Ws)H^{(z)} = \mathrm{RECUR}(H^{(z-1)} + S^{(z)}W_s), while No-coord DEPTH never supplies it, H(z)=RECUR(H(z−1))H^{(z)} = \mathrm{RECUR}(H^{(z-1)}). Since DEPTH always sweeps the slices in the same order, a coordinate given only once would tell the model nothing about which slice it is predicting, so supplying it once is equivalent to No-coord DEPTH. Dropping the per-step coordinate lowers MAP from 0.62 to 0.50, with the carry, embeddings and per-slice supervision unchanged. No-coord DEPTH does not collapse because the carried state can track depth implicitly, by counting how many times RECUR has been applied. However, the Stress10K field alternates between low and high from slice to slice as the slices cross different material layers. The coordinate tells each step which layer it is predicting, so the model can produce these abrupt changes directly rather than relying on counting alone. In Sec. 6, we also test the order in which the slices are visited: bottom-up (MAP 0.62) and top-down (0.61) perform about the same, while a random order is clearly worse (0.51). The coordinate tells each step which slice it is on, but the carried state works best when neighbouring slices are visited in physical order.Sec. 3.2; Sec. 5, Tables 1 and 6; Sec. 6, Table 1; Appendix C.3
W3 (per-slice supervision). The final-slice ablation removes both the intermediate targets and their local gradients, so does its collapse establish the gradient-stability explanation?We agree that the Final-only ablation does not by itself establish the gradient explanation. We have revised §3.5 and App. B to separate what the ablation shows from how we interpret it. Eqn. 2 (below) splits the gradient at each slice into two parts, which give the comb-shaped gradient (Fig. 7). The first is the tooth: the local gradient that the slice's own loss delivers directly to its step. The second is the spine: the carry gradient from all deeper slices, flowing back through the carried state. The two ablations each remove one part of the comb: Final-only removes the teeth of every slice except the last, and No-carry removes the carry between slices, and with it the spine. Both collapse (MAP 0.62 → 0.0085 and 0.62 → 0.11), which is consistent with DEPTH relying on both routes. Since each ablation also changes the forward pass, we now present this as an interpretation supported by the ablations rather than an established mechanism.Sec. 3.5, App. B.1, Fig. 7

∂L∂H(z)=∂Lz∂H(z)+∂L>z∂H(z),L>z=∑k=z+1ZLk\frac{\partial L}{\partial H^{(z)}} = \frac{\partial L_{z}}{\partial H^{(z)}} + \frac{\partial L_{>z}}{\partial H^{(z)}}, \quad L_{>z} = \sum_{k=z+1}^{Z} L_{k} The first term is the local gradient from slice z's own loss (the tooth); the second is the carry gradient from all deeper slices (the spine).

 

Teeth (local)Spine (carry)R@1 ↑R@5 ↑MAP ↑RMSE ↓SSIM ↑Peak ↓
DEPTH✓✓0.48 ± 0.0770.80 ± 0.0630.62 ± 0.0680.0132 ± 0.00180.96 ± 0.00740.057 ± 0.0086
Final-only✗✓0.0020 ± 0.000710.0073 ± 0.00180.0085 ± 0.00160.35 ± 0.0920.14 ± 0.0520.62 ± 0.0060
No-carry✓✗0.05 ± 0.0200.15 ± 0.0410.11 ± 0.0310.0597 ± 0.0210.82 ± 0.0320.157 ± 0.024

 

Table 1: Component-removal ablations of DEPTH on Stress10K

R@1 ↑R@5 ↑MAP ↑RMSE ↓SSIM ↑Peak ↓
DEPTH (full)0.48 ± 0.0770.80 ± 0.0630.62 ± 0.0680.0132 ± 0.00180.96 ± 0.00740.057 ± 0.0086
No-coord0.35 ± 0.0290.69 ± 0.00320.50 ± 0.0200.0189 ± 0.00250.94 ± 0.00190.074 ± 0.0045
Bottom-up0.48 ± 0.0770.80 ± 0.0630.62 ± 0.0680.0132 ± 0.00180.96 ± 0.00740.057 ± 0.0086
Top-down0.47 ± 0.0260.79 ± 0.0130.61 ± 0.0200.0124 ± 0.00060.97 ± 0.00300.050 ± 0.0027
Random0.36 ± 0.100.68 ± 0.120.51 ± 0.100.0164 ± 0.00320.95 ± 0.0140.072 ± 0.010

For the revision, we reran all experiments in Tables 2–4 under a single, consistent protocol: [held-out test designs for every model; retrieval on the predicted latents; RMSE, SSIM and Peak error on the decoded fields for Stress10K and PINN], [All three datasets are split by case design: every slice of a test design is held out, so the model is evaluated on whole sequences it never saw during training (Stress10K and Warpage10K: 500 training and 124 test designs; PINN: 384 and 100)]. All numbers therefore differ sightly from the original submission. The conclusions are unchanged: DEPTH has the on-par or better MAP in five of six dataset–tier combinations, with the fewest parameters and lowest FLOPs among the transformer models.

ConcernResponseLocation in revised PDF
W4. The text says more recursion hurts 2-block UT on PINN, but the table shows MAP going up from 0.766 to 0.782 (§4.1, Table 4)We reran all experiments in Tables 2–4 for the revision with one common setup (see general response), so all numbers differ from the original submission. The text now matches the table: for 2-block UT on PINN, MAP rises from 0.530 at 1× to 0.622 at 3×, then drops to 0.414 at 5×.Sec. 7, Table 2-4
W4. The appendix says 625 designs and 2,000 test slices, but the main text says 500 training and 124 test designs (App. A.2.1)One design had corrupted simulation output, so we removed it. This leaves 624 designs: 500 for training and 124 for testing (1,984 test slices). The appendix now matches the main text.Appendix C.2.1, Table 5

 

Table 2: Retrieval and field-reconstruction performance on Stress10K (Z = 16)

MethodParamsBlocks/caseFLOPs/caseRecall@1Recall@5MAPRMSESSIMPeak error
1-block
RNN-emb9,31916856,7280.400 ± 0.0310.732 ± 0.0210.546 ± 0.0270.0157 ± 0.00050.947 ± 0.0030.065 ± 0.003
UT1x-1b10,488162,892,3840.133 ± 0.0890.364 ± 0.1740.249 ± 0.1200.0462 ± 0.01660.879 ± 0.0450.121 ± 0.027
UT3x-1b10,488487,527,7440.390 ± 0.0920.740 ± 0.0640.550 ± 0.0790.0182 ± 0.00330.962 ± 0.0130.063 ± 0.013
UT5x-1b10,4888012,163,1040.481 ± 0.0430.806 ± 0.0480.623 ± 0.0460.0151 ± 0.00530.972 ± 0.0040.051 ± 0.006
trm-z-1b11,64025617,601,0240.294 ± 0.0150.632 ± 0.0140.447 ± 0.0120.0246 ± 0.00390.941 ± 0.0110.081 ± 0.004
trm-zy-1b17,49649662,133,0240.424 ± 0.0560.780 ± 0.0460.580 ± 0.0510.0175 ± 0.00320.961 ± 0.0170.068 ± 0.009
DEPTH-1b9,336162,293,0560.476 ± 0.0770.796 ± 0.0630.617 ± 0.0680.0132 ± 0.00180.963 ± 0.0070.057 ± 0.009

 

Table 3: Retrieval and field-reconstruction performance on the PINN dataset (Z = 15).

MethodParamsBlocks/caseFLOPs/caseRecall@1Recall@5MAPRMSESSIMPeak error
1-block
RNN-emb9,17315802,4160.375 ± 0.1260.862 ± 0.0970.574 ± 0.1120.0040 ± 0.00140.9985 ± 0.00170.013 ± 0.003
UT1x-1b8,136151,606,0950.387 ± 0.0670.884 ± 0.0320.594 ± 0.0540.0047 ± 0.00120.9983 ± 0.00150.022 ± 0.006
UT3x-1b8,136454,178,9250.453 ± 0.0620.921 ± 0.0710.646 ± 0.0590.0034 ± 0.00030.9988 ± 0.00120.015 ± 0.002
UT5x-1b8,136756,751,7550.424 ± 0.0700.888 ± 0.0750.615 ± 0.0680.0050 ± 0.00130.9974 ± 0.00220.020 ± 0.002
trm-z-1b9,28824010,851,8400.438 ± 0.0760.937 ± 0.0320.641 ± 0.0620.0037 ± 0.00060.9991 ± 0.00030.016 ± 0.002
trm-zy-1b12,84046536,098,1900.449 ± 0.0290.918 ± 0.0180.643 ± 0.0180.0045 ± 0.00030.9970 ± 0.00020.019 ± 0.002
DEPTH-1b6,984151,074,7080.462 ± 0.0880.932 ± 0.0620.659 ± 0.0750.0034 ± 0.00080.9990 ± 0.00050.013 ± 0.003

 

Table 4: Retrieval and reconstruction error on Warpage10K (Z = 16, 1-block).

MethodParamsBlocks/caseFLOPs/caseRecall@1Recall@5MAPRMSEPeak
RNN-emb9,31916856,7280.382 ± 0.0780.811 ± 0.1140.566 ± 0.0850.0121 ± 0.00190.035 ± 0.004
UT1x-1b10,488162,892,3840.429 ± 0.0950.797 ± 0.1120.589 ± 0.0970.0201 ± 0.00410.069 ± 0.032
...
...
DEPTH-1b9,336162,293,0560.596 ± 0.1550.909 ± 0.1270.732 ± 0.1370.0135 ± 0.00700.063 ± 0.056
Reviewer_po3UOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

DEPTH uses a weight-shared transformer to predict an ordered sequence of physical field slices. It encodes the design condition once, carries a hidden state between slices, supplies the physical coordinate at each step, and supervises each output. The method is tested on stress and warpage fields and a Laplace-equation benchmark, with comparisons against recursive transformers and recurrent models.

Critical strengths
  1. Using one recursion step per physical slice is a sensible way to reuse computation. On Stress10K, the UT requires five block applications per slice to match DEPTH's retrieval performance (Figure 4). Tables 3-4 also show competitive retrieval accuracy at much lower training FLOPs than the strongest recursive baseline. The efficiency result is the main reason I favor acceptance.

  2. The supervision ablations help explain the design. Final-slice-only supervision performs poorly, whereas supervising every slice preserves useful intermediate predictions. The comparison between coordinate-injection schemes also helps separate essential design choices from optional ones.

Critical weaknesses
  • The attribution of the gain to state carry is not fully isolated. The transformer baselines also differ in how the coordinate enters the model, while Table 1 compares injection schemes only with carry enabled. The RNN comparison has a similar limitation: the raw-input RNN has a very small state, whereas DEPTH learns token embeddings. These comparisons support the combined design, but are less decisive about which component causes the gain.

  • The main comparisons rely on retrieval metrics. I understand the authors' reason for using them, but comparable retrieval scores do not settle the accuracy of the reconstructed physical fields. In the one-block setting, DEPTH's MAP advantage over trm-zy is small relative to run-to-run variation. The evidence is stronger for comparable accuracy at lower cost than for consistently better field prediction.

Questions for authors
  1. Could the authors compare against a no-carry variant with the same coordinate injection and learned embeddings? A similarly sized RNN with learned input embeddings would also help clarify whether the gain specifically requires the attention-based carried state.

  2. Do decoded-field RMSE and peak error support the same comparison between DEPTH and the strongest recursive baseline in Tables 3-4? Please clarify whether the intended conclusion is better physical-field accuracy or similar accuracy at substantially lower cost.

3 comments

We thank the reviewer for the careful and positive assessment. We address each point below. Section, table and equation numbers refer to the revised PDF.

ConcernResponseLocation in revised PDF
W1 / Q1. The gain is not clearly due to the carried state: the transformer baselines also differ in how the coordinate enters, and Table 1 compares injection methods only with the carry on. Please compare a no-carry variant with the same coordinate injection and learned embeddings, and a similarly sized RNN with learned input embeddingsWe added both. No-carry DEPTH changes only one thing: every slice starts from the design embedding EE instead of the previous slice's state. DEPTH updates H(z)=RECUR(H(z−1)+S(z)Ws)H^{(z)} = \mathrm{RECUR}(H^{(z-1)} + S^{(z)}W_s), while No-carry DEPTH updates H(z)=RECUR(E+S(z)Ws)H^{(z)} = \mathrm{RECUR}(E + S^{(z)}W_s). Everything else is the same: the learned embeddings, the coordinate injection, per-slice supervision and the 9,336 parameters. Removing the carry drops MAP from 0.62 to 0.11. RNN-emb is a recurrent predictor of the same size (9,319 vs. 9,336 parameters) with learned input embeddings, the same coordinate input and per-slice supervision. It reaches MAP 0.546, against 0.617 for DEPTH-1b. So the carry is what makes DEPTH work, and a carried state alone is not enough: it also needs the attention transition. No-attn DEPTH keeps the carry but replaces self-attention with a token-mixing MLP, H(z)=RECURmix(H(z−1)+S(z)Ws)H^{(z)} = \mathrm{RECUR}_{\mathrm{mix}}(H^{(z-1)} + S^{(z)}W_s), at a comparable size (9,432 parameters), and reaches MAP 0.44. No-attn DEPTH also scores below RNN-emb: splitting the state into tokens helps only when attention mixes them according to their content. With fixed token-mixing weights, a single recurrent vector is the better use of the same budget.Sec. 5, Tables 1 and 6; Sec. 3.3, Table 2; Appendix F, Tables 10

 

Table 1: Component-removal ablations of DEPTH on Stress10K

R@1 ↑R@5 ↑MAP ↑RMSE ↓SSIM ↑Peak ↓
RNN-emb0.40 ± 0.0310.73 ± 0.0210.55 ± 0.0270.0157 ± 0.00050.95 ± 0.0030.065 ± 0.003
DEPTH (full)0.48 ± 0.0770.80 ± 0.0630.62 ± 0.0680.0132 ± 0.00180.96 ± 0.00740.057 ± 0.0086
No-attn0.29 ± 0.0820.61 ± 0.0880.44 ± 0.0830.0204 ± 0.00720.92 ± 0.0530.082 ± 0.017
No-coord0.35 ± 0.0290.69 ± 0.00320.50 ± 0.0200.0189 ± 0.00250.94 ± 0.00190.074 ± 0.0045
No-carry0.05 ± 0.0200.15 ± 0.0410.11 ± 0.0310.0597 ± 0.0210.82 ± 0.0320.157 ± 0.024

 

Table 6: State update of DEPTH and its three ablated variants

VariantUpdate
DEPTHH(z)=RECUR(H(z−1)+S(z)Ws)H^{(z)} = \mathrm{RECUR}\big(H^{(z-1)} + S^{(z)} W_s\big)
No-attnH(z)=RECURmix(H(z−1)+S(z)Ws)H^{(z)} = \mathrm{RECUR}_{\mathrm{mix}}\big(H^{(z-1)} + S^{(z)} W_s\big)
No-coordH(z)=RECUR(H(z−1))H^{(z)} = \mathrm{RECUR}\big(H^{(z-1)}\big)
No-carryH(z)=RECUR(E+S(z)Ws)H^{(z)} = \mathrm{RECUR}\big(E + S^{(z)} W_s\big)

 

Table 10: Capacity-matched comparison on Stress10K

MethodParamsFLOPsR@1 ↑MAP ↑RMSE ↓SSIM ↑Peak ↓
DeepONet D9,2450.11M0.386 ± 0.0500.529 ± 0.0490.0150 ± 0.00170.958 ± 0.0100.067 ± 0.007
RNN-emb9,3190.86M0.394 ± 0.0320.540 ± 0.0280.0156 ± 0.00050.946 ± 0.0020.065 ± 0.002
No-attn DEPTH9,4325.47M0.292 ± 0.0820.436 ± 0.0830.0204 ± 0.00720.919 ± 0.0530.082 ± 0.017
DEPTH-1b9,3362.29M0.476 ± 0.0770.617 ± 0.0680.0132 ± 0.00180.963 ± 0.0070.057 ± 0.009
ConcernResponseLocation in revised PDF
W2 / Q2. The main comparisons rely on retrieval. Do decoded-field RMSE and peak error support the comparison with the strongest recursive baseline? Is the claim better field accuracy, or similar accuracy at much lower cost?Stress10K and PINN now report decoded-field RMSE, SSIM and peak error. Warpage10K reports RMSE and peak error on the latents only. DEPTH has the lowest or tied-lowest RMSE in all four Stress10K and PINN tiers, while SSIM and peak error are within the range of the strongest baselines rather than always ahead. For example, on Stress10K at one block, UT5x-1b has slightly higher SSIM (0.972 vs. 0.963) and lower peak error (0.051 vs. 0.057). This comparison is not like for like: UT5x-1b refines each slice by applying the block 5 times, while DEPTH applies it once per slice and does no refinement, so UT5x-1b uses 5.3× DEPTH's FLOPs. At matched compute, UT1x-1b is far behind DEPTH (RMSE 0.0462 vs. 0.0132, SSIM 0.879 vs. 0.963, peak error 0.121 vs. 0.057). Our claim is therefore the second one: DEPTH has similar field accuracy at much lower cost, with a retrieval advantage in most tiers. We now state this explicitly in the paper.Sec. 7, Tables 2–4 ("Field prediction and cost")
 

Table 2: Retrieval and field-reconstruction performance on Stress10K (Z = 16)

MethodParamsBlocks/caseFLOPs/caseRecall@1Recall@5MAPRMSESSIMPeak error
1-block
RNN-emb9,31916856,7280.400 ± 0.0310.732 ± 0.0210.546 ± 0.0270.0157 ± 0.00050.947 ± 0.0030.065 ± 0.003
UT1x-1b10,488162,892,3840.133 ± 0.0890.364 ± 0.1740.249 ± 0.1200.0462 ± 0.01660.879 ± 0.0450.121 ± 0.027
UT3x-1b10,488487,527,7440.390 ± 0.0920.740 ± 0.0640.550 ± 0.0790.0182 ± 0.00330.962 ± 0.0130.063 ± 0.013
UT5x-1b10,4888012,163,1040.481 ± 0.0430.806 ± 0.0480.623 ± 0.0460.0151 ± 0.00530.972 ± 0.0040.051 ± 0.006
trm-z-1b11,64025617,601,0240.294 ± 0.0150.632 ± 0.0140.447 ± 0.0120.0246 ± 0.00390.941 ± 0.0110.081 ± 0.004
trm-zy-1b17,49649662,133,0240.424 ± 0.0560.780 ± 0.0460.580 ± 0.0510.0175 ± 0.00320.961 ± 0.0170.068 ± 0.009
DEPTH-1b9,336162,293,0560.476 ± 0.0770.796 ± 0.0630.617 ± 0.0680.0132 ± 0.00180.963 ± 0.0070.057 ± 0.009

 

Table 3: Retrieval and field-reconstruction performance on the PINN dataset (Z = 15).

MethodParamsBlocks/caseFLOPs/caseRecall@1Recall@5MAPRMSESSIMPeak error
1-block
RNN-emb9,17315802,4160.375 ± 0.1260.862 ± 0.0970.574 ± 0.1120.0040 ± 0.00140.9985 ± 0.00170.013 ± 0.003
UT1x-1b8,136151,606,0950.387 ± 0.0670.884 ± 0.0320.594 ± 0.0540.0047 ± 0.00120.9983 ± 0.00150.022 ± 0.006
UT3x-1b8,136454,178,9250.453 ± 0.0620.921 ± 0.0710.646 ± 0.0590.0034 ± 0.00030.9988 ± 0.00120.015 ± 0.002
UT5x-1b8,136756,751,7550.424 ± 0.0700.888 ± 0.0750.615 ± 0.0680.0050 ± 0.00130.9974 ± 0.00220.020 ± 0.002
trm-z-1b9,28824010,851,8400.438 ± 0.0760.937 ± 0.0320.641 ± 0.0620.0037 ± 0.00060.9991 ± 0.00030.016 ± 0.002
trm-zy-1b12,84046536,098,1900.449 ± 0.0290.918 ± 0.0180.643 ± 0.0180.0045 ± 0.00030.9970 ± 0.00020.019 ± 0.002
DEPTH-1b6,984151,074,7080.462 ± 0.0880.932 ± 0.0620.659 ± 0.0750.0034 ± 0.00080.9990 ± 0.00050.013 ± 0.003

 

Table 4: Retrieval and reconstruction error on Warpage10K (Z = 16).

MethodParamsBlocks/caseFLOPs/caseRecall@1Recall@5MAPRMSEPeak
1-block
RNN-emb9,31916856,7280.382 ± 0.0780.811 ± 0.1140.566 ± 0.0850.0121 ± 0.00190.035 ± 0.004
UT1x-1b10,488162,892,3840.429 ± 0.0950.797 ± 0.1120.589 ± 0.0970.0201 ± 0.00410.069 ± 0.032
UT3x-1b10,488487,527,7440.617 ± 0.1160.862 ± 0.0800.727 ± 0.0970.0190 ± 0.00480.082 ± 0.018
UT5x-1b10,4888012,163,1040.592 ± 0.1380.847 ± 0.0820.707 ± 0.1120.0210 ± 0.00550.099 ± 0.023
trm-z-1b11,64025617,601,0240.480 ± 0.1370.806 ± 0.1130.624 ± 0.1230.0300 ± 0.02600.085 ± 0.054
trm-zy-1b17,49649662,133,0240.554 ± 0.1150.863 ± 0.1020.691 ± 0.1070.0180 ± 0.00700.088 ± 0.035
DEPTH-1b9,336162,293,0560.596 ± 0.1550.909 ± 0.1270.732 ± 0.1370.0135 ± 0.00700.063 ± 0.056

I thank the authors for their careful answers to my questions. The ablations show that the gain comes mainly from the carry, with a further benefit from attention. Also, the added metrics support the claim of similar accuracy at much lower cost. I confirm my positive assessment of 3 (weak accept).

Comments on the paper

2 comments

Summary of the rebuttal and revisions

We thank all reviewers for their feedback. Below is a brief summary of how the revised PDF addresses the main concerns.

Reviewer po3U. After the rebuttal, the reviewer confirmed weak accept. They noted that the new ablations show the gain comes mainly from the carry, with a further benefit from attention, and that the added field metrics support similar accuracy at much lower cost.

Reviewer c5UX

  • The methodological contribution needs sharper delineation: we isolated each component with a single-change ablation on Stress10K. No-carry DEPTH lowers MAP from 0.62 to 0.11, No-coord DEPTH lowers it to 0.50, and No-attn DEPTH lowers it to 0.44. Because each step is a physical slice, the recursion can also be smoothed along z, which has no counterpart when a step is the next word. Leaky DEPTH smooths the carried state (MAP 0.66) and EMA DEPTH smooths the depth signal (0.59), each with a fixed β and no extra parameters. (Sec. 5, Table 1; Appendix A.5, Table 6; Appendix B)
  • Comparably budgeted recurrent baselines: we added RNN-emb (MAP 0.55, with learned embeddings, a per-step coordinate and per-slice supervision) and the shared-MLP transition the reviewer suggested (No-attn DEPTH, 0.44), both parameter-matched to DEPTH (0.62). (Sec. 5, Table 1; Appendix F, Table 10)
  • Claims beyond the ablation evidence: No-coord DEPTH now tests the per-step coordinate directly (MAP 0.62 → 0.50). For per-slice supervision, Eq. (2) splits the gradient at each slice into a local term from that slice's own loss and a carry term from all deeper slices. The two ablations each remove one term: Final-only removes the local terms at every slice except the last (MAP 0.62 → 0.0085), and No-carry DEPTH removes the carry term (0.62 → 0.11). Since each ablation also changes the forward pass, we now present this as an interpretation supported by the ablations. (Sec. 3.5, Eq. 2; Appendix B, Fig. 7)
  • Inconsistencies: we reran all experiments on all three datasets (Stress10K, Warpage10K and PINN) under one consistent protocol, and the text now matches the updated tables throughout. (Sec. 7, Tables 2–4; Appendix C)

Reviewer Fqeo

  • Globally coupled, elliptic fields: all slices of a design come from one global solve, and DEPTH encodes the full design once as the initial state, so every step starts with global information. The sweep is an inductive bias, not a model of information travelling through the material. For Stress10K and Warpage10K the groundtruth fields are simulated by solving the elliptic static-equilibrium equations of linear thermoelasticity. The stack of materials changes stiffness and CTE at each interface, so the field changes sharply from layer to layer, and each layer's stress is set by its mismatch with its neighbours. In PINN, each slice is an independent 2D Laplace solve at a different plate width, so the slices are not spatially coupled. For PINN, we added a new test that checks the governing equation directly: the 5-point finite-difference Laplace residual on every predicted slice, compared with the residual of the autoencoder-reconstructed ground truth. DEPTH tracks the ground truth at every slice (ratio 0.985), with no drift along the sweep. (Sec. 3.1, Eq. 1; Sec. 6, Fig. 4; Appendix C)
  • Sequential sweep in a fixed direction: bottom-up and top-down sweeps perform the same (MAP 0.62 vs. 0.61). No-carry DEPTH (0.11) and a random slice order (0.51) both hurt. (Sec. 6, Table 1)
  • Stronger capacity-matched baselines: DeepONet (0.529), RNN-emb (0.540) and an MLP-Mixer variant of DEPTH (0.436), all within 2% of DEPTH-1b's parameter count (0.617). (Appendix F, Tables 10–11)
  • Field-wise accuracy, timing and run selection: every results table now reports RMSE, SSIM and peak error. We added measured wall-clock time: the only model that matches DEPTH-1b's MAP, UT5x-1b, is 4.3× slower. (Sec. 7, Tables 2–4; Appendix G, Tables 12–13)

We hope this summary is helpful for the discussion.

Thank you to the authors for the summary, and thank you to all reviewers for your reviews and engagement so far.

I would like to draw the reviewers' attention to the authors' rebuttal, the summary note mapping each concern to the changes in the revised PDF, and the revised manuscript itself. Reviewers c5UX and Fqeo, if you get a chance to look them over, your thoughts would be very welcome, including whether the revisions address your concerns or you would like to update your assessment.

Thank you again for your time on this paper.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.