- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper uses SPRINT-style sparse-dense token processing to hierarchical (U-Net-shaped) PDE transformers. LITE runs a drop-process-reconstruct cycle at every encoder and decoder stage while keeping the latent stage dense. In each stage, it does three things:
- Selection: one token per 2×2 (or 2×2×2) block is kept, chosen by the L2 norm of its embedding (plus some noise during training).
- Processing: the stage's transformer blocks run only on the kept tokens.
- Reconstruction: kept tokens are scattered back with a learned mask token, a depth-wise "spread" convolution passes information to dropped positions, and a linear fusion combines the result with the stored dense input. Full resolution is restored after each stage. The paper shows some promising improvement for speed, memory and accuracy. The validation looks comprehensive.
Critical strengths
-
The method is simple, practical and effective. The results are good, latency and peak memory drop by 2.0-2.8× with no loss in terminal rollout error. This lets a 167M-parameter model train in 10.2 GB, and lets models train on full 384 cubic KdV volumes where the dense counterparts run out of memory on a 48 GB card. For 3D scientific surrogates, memory is often what limits what can be trained, so a drop-in per-stage wrapper that removes this barrier is useful to the community。
-
The central argument is well-motivated and well-supported by the results. Specifically, the SPRINT-style whole-network dropping baseline, at the same keep rate and on the same backbone, does reasonably well on one-step error but 48% worse than dense on 10-step rollout, which supports the paper's central argument that where tokens are dropped in a hierarchical model matters more than whether they are dropped.
Critical weaknesses
- The claim that "token dropping is a precondition for training" at 384 is only compared against dense models without any memory-saving technique. Activation (gradient) checkpointing, or checkpointing per stage, would very likely let P3D-Patch S/B fit at 384 at a compute cost. A comparison with checkpointing would quantify the time-memory trade-off honestly.
Questions for authors
Q1. Can the authors report the dense PDE-Transformer S trained with exactly LITE's recipe (gated MLP, GELU, same LR/schedule/batch) at 256/512 resolution? Does LITE still beat it in nRMSE1/nRMSE10?
Q2. Can P3D-Patch S/B be trained at 384 resolution with activation checkpointing on the same 48 GB card? If so, what are its latency and accuracy compared with LITE-3D?
3 comments
Thank you for the detailed response. The rebuttal addresses my main concern about checkpointing and clarifies some limitations of the method. I'll maintain my original score (weak acceptance).
The added experiments show that dense models can fit on the GPU with gradient checkpointing, while LITE still offers speed and memory benefits. The comparison at 384³ also supports the paper’s main argument, with LITE-S achieving lower rollout errors than the checkpointed dense S baseline.
Regarding Q1, the authors acknowledge that LITE does not consistently improve rollout accuracy, particularly at 512^2. I appreciate the honest clarification, and these limitations should be reflected in the paper’s claims.
We thank the reviewer for their thoughtful response and will ensure our claims reflect the observed limitations in rollout accuracy. We emphasize that, while dense models with gradient checkpointing fit on a 48 GB GPU, they may not fit on more accessible hardware. In contrast, LITE-3D B, L, and S fit within 16 GB, 24 GB, and 8 GB, respectively, where their dense counterparts do not. This makes training on large grids more accessible without expensive GPU clusters. Finally, the lack of rollout improvement was observed only a specific 2D case under unequal GPU-time budgets, with the dense model trained for twice as long (and where no lr search was conducted). Our approach’s primary benefits lie in 3D settings.
We thank the reviewer for recognizing LITE’s practical value for 3D PDE surrogates and the evidence that per-stage dropping improves rollout efficiency. We also appreciate the reviewer’s observation that the drop-once control supports the importance of where sparsification is applied.
Q1. Recipe-matched 2D comparison
At 2562, our matched-GPU-time control uses the same shared architecture and training recipe. LITE achieves 0.3211 ten-step nRMSE, versus 0.353 for dense PDE-Transformer—a 9.0% lower rollout error. At 5122, the paper reports a better one-step nRMSE for LITE (0.0289 vs. 0.0389) but a slightly worse 10-step nRMSE (0.3383 vs. 0.3296). Since training is one-step, the one-step gain does not necessarily carry over to autoregressive rollout. The learning rate was tuned at 2562 and reused at 5122, so it may be suboptimal there; we therefore do not claim a 5122 rollout-error win.
W + Q2. Checkpointing at 3843
Checkpointed forward/backward probes show that P3D-Patch S and B fit within 48 GiB at 3843; the B result is a probe, not a completed full-volume training run. Checkpointing also combines with LITE. The key capacity result is at matched large-model scale: with GC, the 164.9M-parameter P3D-Patch-L still needs 34.88 GiB and cannot fit on a 24-GiB GPU, while the slightly larger 166.6M-parameter LITE-L needs only 20.23 GiB and fits. Checkpointing recomputes activations; LITE reduces the tokens—and therefore the activations—to be stored and recomputed. The same single-GPU probes give:
Across S/B/L, LITE+GC is 2.3–2.6× faster and uses 14–42% less memory. GC raises S-model latency by 20% for P3D-Patch but only 6% for LITE; LITE-B/L also incur only about 6–7% overhead.
For full-volume 3843 KdV, the rollout errors are:
LITE-S improves on the checkpointed P3D-Patch-S result, and LITE-B further lowers both errors. We will clarify that checkpointing enables dense S/B at 3843 on 48-GiB hardware, while combining checkpointing with LITE reduces per-sample cost and enables larger models on smaller GPUs.