acceptodds
Under review as a conference paper at ICLR 2027

LITE: Layer-Wise Intermediate Token-Dropping for Efficient Structured-Grid PDE Emulation

Abstract

Transformer-based neural surrogates for partial differential equations achieve strong predictive accuracy on structured grids, but their training cost grows rapidly with resolution, and in three dimensions it becomes the binding constraint on what can be trained at all. We introduce LITE (Layer-wise Intermediate Token-dropping), which makes sparse–dense token processing work in hierarchical PDE transformers. Rather than routing tokens around the entire network, LITE applies a drop–process–reconstruct cycle independently at each encoder and decoder stage while keeping the latent stage dense. Before each stage an -norm importance score retains one token per spatial block, concentrating computation on physically active regions. The stage's transformer blocks operate on that subset alone. A depthwise spread convolution then propagates the result to the dropped positions, and a learned linear fusion recombines it with the dense features carried past the stage untouched. Because full resolution is restored between stages, multi-scale feature flow and skip connections remain intact and no full-token fine-tuning phase is needed, so peak memory never returns to that of the dense model. In 2D, LITE halves wall-clock training time while reducing normalized RMSE by on a 16-equation benchmark. Whole-network dropping at the same keep rate degrades sharply over a ten-step rollout, ending up worse than the dense model it aims to accelerate. In 3D at , dropping cuts per-sample latency and peak memory by – without increasing terminal rollout error, so a 167M-parameter model trains with lower latency and less memory than a 42M-parameter dense backbone. On full volumes, three times the linear extent used in prior work on this data, the dense baselines exhaust memory, while LITE trains and improves rollout accuracy. At these resolutions token dropping is thus a precondition for training rather than an optimization.

Primary area:
applications to physical sciences (physics, chemistry, biology, etc.)
Keywords:
PDE Surrogates, Transformer-Based Neural Operators, Token Dropping

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_NHPiOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper uses SPRINT-style sparse-dense token processing to hierarchical (U-Net-shaped) PDE transformers. LITE runs a drop-process-reconstruct cycle at every encoder and decoder stage while keeping the latent stage dense. In each stage, it does three things:

  • Selection: one token per 2×2 (or 2×2×2) block is kept, chosen by the L2 norm of its embedding (plus some noise during training).
  • Processing: the stage's transformer blocks run only on the kept tokens.
  • Reconstruction: kept tokens are scattered back with a learned mask token, a depth-wise "spread" convolution passes information to dropped positions, and a linear fusion combines the result with the stored dense input. Full resolution is restored after each stage. The paper shows some promising improvement for speed, memory and accuracy. The validation looks comprehensive.
Critical strengths
  1. The method is simple, practical and effective. The results are good, latency and peak memory drop by 2.0-2.8× with no loss in terminal rollout error. This lets a 167M-parameter model train in 10.2 GB, and lets models train on full 384 cubic KdV volumes where the dense counterparts run out of memory on a 48 GB card. For 3D scientific surrogates, memory is often what limits what can be trained, so a drop-in per-stage wrapper that removes this barrier is useful to the community。

  2. The central argument is well-motivated and well-supported by the results. Specifically, the SPRINT-style whole-network dropping baseline, at the same keep rate and on the same backbone, does reasonably well on one-step error but 48% worse than dense on 10-step rollout, which supports the paper's central argument that where tokens are dropped in a hierarchical model matters more than whether they are dropped.

Critical weaknesses
  1. The claim that "token dropping is a precondition for training" at 384 is only compared against dense models without any memory-saving technique. Activation (gradient) checkpointing, or checkpointing per stage, would very likely let P3D-Patch S/B fit at 384 at a compute cost. A comparison with checkpointing would quantify the time-memory trade-off honestly.
Questions for authors

Q1. Can the authors report the dense PDE-Transformer S trained with exactly LITE's recipe (gated MLP, GELU, same LR/schedule/batch) at 256/512 resolution? Does LITE still beat it in nRMSE1/nRMSE10?

Q2. Can P3D-Patch S/B be trained at 384 resolution with activation checkpointing on the same 48 GB card? If so, what are its latency and accuracy compared with LITE-3D?

3 comments

We thank the reviewer for recognizing LITE’s practical value for 3D PDE surrogates and the evidence that per-stage dropping improves rollout efficiency. We also appreciate the reviewer’s observation that the drop-once control supports the importance of where sparsification is applied.

Q1. Recipe-matched 2D comparison

At 2562256^2, our matched-GPU-time control uses the same shared architecture and training recipe. LITE achieves 0.3211 ten-step nRMSE, versus 0.353 for dense PDE-Transformer—a 9.0% lower rollout error. At 5122512^2, the paper reports a better one-step nRMSE for LITE (0.0289 vs. 0.0389) but a slightly worse 10-step nRMSE (0.3383 vs. 0.3296). Since training is one-step, the one-step gain does not necessarily carry over to autoregressive rollout. The learning rate was tuned at 2562256^2 and reused at 5122512^2, so it may be suboptimal there; we therefore do not claim a 5122512^2 rollout-error win.

W + Q2. Checkpointing at 3843384^3

Checkpointed forward/backward probes show that P3D-Patch S and B fit within 48 GiB at 3843384^3; the B result is a probe, not a completed full-volume training run. Checkpointing also combines with LITE. The key capacity result is at matched large-model scale: with GC, the 164.9M-parameter P3D-Patch-L still needs 34.88 GiB and cannot fit on a 24-GiB GPU, while the slightly larger 166.6M-parameter LITE-L needs only 20.23 GiB and fits. Checkpointing recomputes activations; LITE reduces the tokens—and therefore the activations—to be stored and recomputed. The same single-GPU probes give:

ModelLatency (ms/sample), no GC → GCPeak memory (GiB), no GC → GC
P3D-Patch S454 → 54526.6 → 9.1
LITE-S195 → 20610.6 → 7.8
P3D-Patch BOOM → 1,104OOM → 17.6
LITE-B414 → 44017.4 → 11.8
P3D-Patch LOOM → 2,484OOM → 34.88
LITE-L1,008 → 1,07431.45 → 20.23

Across S/B/L, LITE+GC is 2.3–2.6× faster and uses 14–42% less memory. GC raises S-model latency by 20% for P3D-Patch but only 6% for LITE; LITE-B/L also incur only about 6–7% overhead.

For full-volume 3843384^3 KdV, the rollout errors are:

ModelMean nRMSE (t=1–29)nRMSE (t=29)
P3D-Patch S + GC0.28970.5060
LITE-3D S0.27640.4823
LITE-3D B0.25680.4369

LITE-S improves on the checkpointed P3D-Patch-S result, and LITE-B further lowers both errors. We will clarify that checkpointing enables dense S/B at 3843384^3 on 48-GiB hardware, while combining checkpointing with LITE reduces per-sample cost and enables larger models on smaller GPUs.

Thank you for the detailed response. The rebuttal addresses my main concern about checkpointing and clarifies some limitations of the method. I'll maintain my original score (weak acceptance).

The added experiments show that dense models can fit on the GPU with gradient checkpointing, while LITE still offers speed and memory benefits. The comparison at 384³ also supports the paper’s main argument, with LITE-S achieving lower rollout errors than the checkpointed dense S baseline.

Regarding Q1, the authors acknowledge that LITE does not consistently improve rollout accuracy, particularly at 512^2. I appreciate the honest clarification, and these limitations should be reflected in the paper’s claims.

We thank the reviewer for their thoughtful response and will ensure our claims reflect the observed limitations in rollout accuracy. We emphasize that, while dense models with gradient checkpointing fit on a 48 GB GPU, they may not fit on more accessible hardware. In contrast, LITE-3D B, L, and S fit within 16 GB, 24 GB, and 8 GB, respectively, where their dense counterparts do not. This makes training on large (3843)(384^3) grids more accessible without expensive GPU clusters. Finally, the lack of rollout improvement was observed only a specific 2D case under unequal GPU-time budgets, with the dense model trained for twice as long (and where no lr search was conducted). Our approach’s primary benefits lie in 3D settings.

Reviewer_EjDSOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This work proposes LITE (Layer-wise Intermediate Token-dropping), a method for information exchange between layers that aims to train neural PDE models on structured grids more efficiently while maintaining broadly comparable accuracy. At each encoder–decoder stage, LITE by default selects tokens within each partitioned patch according to their L2 norms and processes the retained tokens through the Transformer. It then combines the dense input information with the Transformer output at the same stage to reconstruct the hidden states, while keeping the latent stage dense. Compared with processing all tokens through the Transformer, this reduces computation and GPU memory usage. Experiments in two and three dimensions report accuracy–efficiency trade-offs at high resolutions and demonstrate the ability to train on volumes of size 3843384^3.

Critical strengths
  1. LITE offers computational and GPU memory advantages on high-resolution structured-grid problems. Compared with the dense baselines evaluated in the paper, such as PDE-Transformer, it substantially reduces computational cost and memory usage, enabling larger models and larger spatiotemporal domains under hardware constraints.
  2. The experiments have broad coverage, including comparisons on multiple two-dimensional datasets and extensions to three-dimensional datasets.
Critical weaknesses
  1. In the two-dimensional APEBench experiments, Appendix B.2 states that LITE and the dense PDE-Transformer differ in block variants, activations, and tuning configurations. In particular, the authors disclose that LITE is tuned at 2562256^2, whereas the dense baseline uses its original configuration. LITE achieves better accuracy at 2562256^2, but slightly underperforms the dense baseline in nRMSE10\mathrm{nRMSE}_{10} at 5122512^2. It is therefore unclear how much of the accuracy gain comes from LITE and how much comes from other architectural differences and tuning.
  2. In the standard benchmark experiments in Appendix C, Table 10 shows that LITE has higher errors than the dense PDE-Transformer on Navier–Stokes, Airfoil, and Darcy, and matches it on Plasticity and Pipe. On Airfoil and Darcy, LITE even has the highest error among LITE, PDE-Transformer, Transolver, and MSPT. This indicates that efficiency improvements come at an accuracy cost on some tasks, whereas Appendix C.3 claims that the method transfers "without loss of accuracy." The discussion of this trade-off therefore needs to be strengthened.
Questions for authors
  1. Given the architectural and tuning differences reported in Appendix B.2, could you provide a comparison with matched MLP blocks, activations, and training configurations to clarify whether the accuracy gains come from LITE?
  2. On benchmarks such as Airfoil and Darcy, LITE has higher errors than the dense baseline. How should we assess the trade-off between resource savings and accuracy loss on these tasks? Could you compare final errors under equal training-time or GPU memory budgets?
  3. The three-dimensional comparisons currently focus on the P3D family. Could you add at least one suitable baseline with a different architecture and compare accuracy, training cost, and GPU memory usage under the same data and evaluation protocol?
5 comments

We thank the reviewer for recognizing LITE’s computational and GPU-memory advantages on high-resolution structured-grid problems and the breadth of our 2D and 3D evaluation. Our new controlled comparisons directly address the concerns about recipe, compute budget, and architecture coverage.

Matched 2D comparison

Using the same backbone, MLP blocks, activations, and training recipe at matched GPU time, LITE achieves 0.3211 ten-step nRMSE on 2D APEBench, versus 0.353 for the dense PDE-Transformer—a 9.0% lower error. This isolates the benefit of LITE’s stage-wise drop-and-reconstruct design.

Standard benchmark comparison

We retrained the standard-benchmark baselines with GPU-time budgets matched to LITE, rather than matching epoch counts. LITE has the lowest relative ℓ2\ell_2 error on four of five tasks against Transolver and MSPT; Transolver is better on Darcy. Values are scaled by 10−210^{-2}, lower is better. The Transolver/MSPT values are from the new time-matched runs; LITE values are from Appendix C, Table 10.

2D benchmark (resolution)TransolverMSPTLITE
Navier–Stokes (64×64)42.888.276.30
Plasticity (101×31)0.210.400.03
Pipe (129×129)0.550.590.32
Airfoil (221×51)1.913.090.74
Darcy (85×85)0.521.890.72

LITE therefore achieves the best error on four tasks at comparable training time, while using 19–51% less peak memory than the dense PDE-Transformer. We will clarify this favorable accuracy–resource trade-off without implying that LITE wins on every task.

Additional 3D architecture baseline

We trained AFNO on the same 14-family APEBench 3D task at 2563256^3, with the same 100-epoch training and 15-step evaluation protocol. Latency is forward-plus-backward time per sample. AFNO memory (13.78 GB) is converted to GiB.

ModelParametersLatency (ms/sample)Peak memory (GiB)Mean nRMSE (t=1–15)nRMSE (t=15)
LITE-3D S11.2M623.20.39100.7248
LITE-3D B42.7M1065.30.36270.7077
AFNO64.0M330.112.830.50340.9809

Despite having more parameters, AFNO is 3.1× slower than LITE-B and 5.3× slower than LITE-S, uses 2.4× and 4.0× more memory, and has higher error on both rollout metrics. LITE-B achieves these gains with 33% fewer parameters. This confirms that LITE’s efficiency and accuracy benefits extend beyond the P3D family.

Thank you for the detailed rebuttal and additional experiments. The addition of a comparison between LITE and PDE-Transformer for the 2D case, LITE and Transovler for the same GPU time,MSPT, and the addition of AFNO to the 3D baseline basically addressed my main concerns. But since there was no response to choosing tokens according to the L_2 norm, I kept the score at 3.

We thank the reviewer for taking the time to review our work and for acknowledging the additions made in the rebuttal. We attempted to locate the remaining question or concern regarding the ℓ2\ell_2 norm but were unable to identify it from the initial review. To further investigate this point, we conducted an ablation study comparing ℓ2\ell_2-based dropping with random dropping and trained two variants on the main benchmarks.

In matched 3D multi-PDE runs, ℓ2\ell_2-based selection reduces nRMSE relative to random selection by 57.9% at t=1t=1, 13.7% averaged over t=1t=1–15, and 1.6% at t=15t=15. The gains are concentrated in nonsmooth, spatially localized Gray–Scott regimes: for α\alpha, δ\delta, ι\iota, and θ\theta, the mean rollout nRMSE reductions are 30.9%, 78.9%, 41.1%, and 44.1%, respectively. The benefits diminish over the rollout, consistent with accumulated errors in autoregressive states making the importance scores less informative at later steps. We will clarify that the ℓ2\ell_2 criterion is most beneficial for localized dynamics, rather than claiming a uniform advantage across all PDEs.

If this response does not fully address your concern, we would be grateful if you could clarify the specific question or issue you had in mind, as we were unable to identify it in the initial review. Thank you very much for taking the time to review our work and provide feedback.

Reviewer_1VtFOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
3(+1)
Read review
Paper summary

This paper proposes LITE, a token sparsification method for efficient training of transformer-based neural PDE surrogates on structured grids. Applied to a hierarchical, multi-scale backbone PDE transformer, the method applies token dropping at each stage while maintaining a residual representation, then a depthwise convolution reconstructs the sparse features and fuses them back with the dense features. Empirically, evaluated across 16 PDEs in 2D, LITE reduces training time by roughly 2x while slightly improving one-step prediction accuracy. In 3D, LITE substantially reduces memory usage, allowing the training of larger models with higher-resolution grids, up to 384^3, which would otherwise be infeasible, while maintaining or improving rollout accuracy.

Critical strengths
  • The problem is well-motivated and the proposed approach is practical. Reducing the computational and memory costs of neural PDE surrogates, especially in higher dimensional and higher resolution settings, is an important problem. The proposed approach is intuitive and slots in naturally with existing methods.
  • The empirical results are promising and the empirical studies are generally comprehensive. The authors evaluate the method across multiple PDE families, model sizes, and resolutions in both 2D and 3D. The ablation studies are generally clear and help characterize the relative contributions of each proposed architectural component. The reported efficiency improvements are substantial, especially in 3D.
Critical weaknesses
  • Limited conceptual novelty. Token sparsification has been extensively studied in efficient transformer architectures. In particular, the proposed method shares substantial similarities with SPRINT, including the use of a sparse processing path and dense residual features for reconstruction. The main innovations (applying token dropping independently at different stages of the hierarchical architecture, selecting tokens based on embedding norms, and depthwise convolutions) seem to be relatively straightforward applications of existing techniques.
Questions for authors
  • Appendix B.2 notes that the LITE experiments use several training and architectural choices that differ from the reproduced dense PDE-Transformer baseline, including gated MLP blocks and GELU activations. These modifications are reported to improve accuracy even for dense models. Could the authors provide a comparison where the dense backbone and LITE use identical training recipes and architectural components, differing only in the token-dropping mechanism? This would help clarify how much of the reported 23% improvement in one-step nRMSE can be attributed to LITE alone.
  • The benefit of the proposed embedding norm important scoring appears limited compared to random token selection. For example, in Appendix D.3, the L2-based criterion outperforms random selection at only three of seven keep ratios, and the improvements on the 3D KdV task are relatively small (1-2%). Furthermore, the authors mention that several of the standard PDE benchmarks use random rather than adaptive selection. Could the authors further clarify the importance of the proposed selection criterion, for example through a matched comparison against random token selection on the main 2D and 3D benchmarks?
2 comments

We thank the reviewer for recognizing the importance of reducing the computational and memory costs of neural PDE surrogates, the practical fit of our approach, and the breadth of our evaluation across PDE families, model sizes, and resolutions. We also appreciate the positive assessment of our ablations and the substantial efficiency gains in 3D. We address the novelty concern and both questions below.

Novelty and relation to SPRINT

LITE differs from SPRINT in both its computation path and token routing. During sparse pre-training, SPRINT still processes all full-resolution tokens through its dense encoder and decoder. It selects a subset once for the sparse middle path and reuses that subset through those blocks, then fine-tunes the middle on full tokens. LITE has no corresponding full-resolution dense Transformer path: before each high-resolution encoder/decoder stage, it selects tokens anew, runs that stage’s Transformer blocks on the subset, and reconstructs the full map after the stage. Only the already-downsampled latent bottleneck remains dense, and LITE requires no full-token fine-tuning. Thus, the next stage can select a different subset based on updated features, while full-grid reconstruction preserves the multiscale flow. Taken together, these differences constitute important architectural changes to how sparsification is integrated into the hierarchy.

Our SPRINT-style drop-once control on the same PDE-Transformer backbone illustrates why this design matters: at the same 25% keep rate, it reaches 0.5333 ten-step nRMSE, compared with 0.3211 for LITE. The benefit also persists at inference: at 3843384^3, peak inference memory for the small dense P3D-Patch model is 3.94 GB, compared with 2.01 GB for LITE-S. For the same S model, Figure 5 shows peak memory of 3.2, 10.6, and 25.0 GiB for LITE at 2563256^3, 3843384^3, and 5123512^3, respectively, compared with 7.9 and 26.6 GiB for P3D-Patch S at the first two resolutions. The estimated 5123512^3 memory for the dense model is approximately 63 GiB, above the 48-GiB limit. These results show that LITE’s stage-wise sparse processing enables higher-resolution operation while reducing memory during both training and inference.

Question 1: Matched 2D comparison

Since wall-clock time, rather than memory, is the main constraint in our 2D setting, we trained the dense PDE-Transformer with the same shared architecture and training components as LITE under a matched GPU-time budget. The dense model obtains 0.353 ten-step nRMSE, compared with 0.3211 for LITE. This compute-matched result shows that LITE’s rollout advantage persists when the recipe is controlled and training time is held fixed.

Question 2: ℓ2\ell_2 selection versus random selection

In matched 3D multi-PDE runs, ℓ2\ell_2-based selection reduces nRMSE relative to random selection by 57.9% at t=1t=1, 13.7% averaged over t=1t=1–15, and 1.6% at t=15t=15. The gains are concentrated in nonsmooth, spatially localized Gray–Scott regimes: for α\alpha, δ\delta, ι\iota, and θ\theta, the mean-rollout nRMSE reductions are 30.9%, 78.9%, 41.1%, and 44.1%, respectively. The benefit narrows over rollout, consistent with accumulated errors in autoregressive states making the importance scores less informative at later steps. We will clarify that the ℓ2\ell_2 criterion is most beneficial for localized dynamics, rather than claiming a uniform advantage across all PDEs.

Thank you to the authors for the detailed response and additional experiments. I am convinced that the experiments are thorough and the architecture differences between LITE and SPRINT are important for the performance gains, so I have raised my soundness rating accordingly. I am still not fully convinced that the overall work represents a significantly novel contribution, as the differences from SPRINT seem relatively standard in the literature, so I am inclined to maintain my overall recommendation. However, if the other reviewers feel strongly about the work and would like to advocate for it, I am happy to see it accepted.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.