- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper is a diagnostic study of cross-Reynolds generalization failure in neural PDE surrogates, explicitly positioned as a measurement contribution rather than a new method. Working on zero-shot transfer from ν=10⁻³ (Re≈1,000) to ν=10⁻⁴ (Re≈10,000) in forced 2D Navier–Stokes (64×64 vorticity fields, 500 target trajectories, 10-step rollouts), the authors adopt a retrieval-based (analog/RIDE-style) predictor as a decomposable instrument: its encoder, database, selection rule, and autoregressive loop permit nested oracle substitutions that localize the deployment error (38.30%) into an encode–decode reference (16.20%), a +5.66 pp gap to a refreshed-base direction oracle, a dominant +14.01 pp state-refresh (compounding drift) gap, and a +2.44 pp selector gap. The central finding is a decoupling between reconstruction fidelity and deployment ordering: at equal latent width and a common output route, PCA-64 reconstructs better than ConvAE (10.99% vs 16.20%) yet transfers significantly worse (41.93% vs 38.30%); widening PCA from 64 to 512 improves reconstruction 3.16-fold without improving deployment; a random-projection control and an opposite-direction JEPA intervention (11.92 pp worse despite passing collapse checks) further rule out dimensionality, capacity, and non-reconstruction objectives as sufficient explanations. A fidelity-preserving reparameterization intervention shows that invertible linear changes to the retrieval coordinate frame leave reconstruction unchanged to numerical tolerance while nonorthogonal frames degrade deployment by up to 4.75 pp, indicating that retrieval geometry matters even at fixed fidelity. An inventory of eight candidate explanations (mismatch growth, temporal context, database coverage, similarity metric, curvature, phase coordinates, dynamics objective, step-1 anomaly) yields mixed evidence without identifying a responsible mechanism, and a paired spectral-norm audit shows metric weighting only mildly affects method ordering. The authors emphasize scope limits (one benchmark, one tenfold shift, exploratory design with paired bootstrap intervals), retain dual records and pre-registered expectations, and state plainly that no tested variant beats its canonical baseline.
Critical strengths
S1. A genuinely novel diagnostic methodology with one cleanly controlled causal-style demonstration. Using a retrieval predictor not as a competitor but as a decomposable instrument — with nested substitutions (reconstruction reference → refreshed-oracle → free-running-oracle → deployment) that localize the transfer deficit into conditional components — is, to my knowledge, a new experimental design in neural-PDE evaluation. Within it, the fidelity-preserving reparameterization experiment is the standout: orthogonal transforms leave deployment unchanged to ~10⁻⁶ pp (5,000/5,000 index agreement, as cosine invariance analytically requires), while nonorthogonal frames degrade deployment monotonically with index-agreement loss (up to +4.75 pp under whitening) — all with reconstruction held fixed to numerical tolerance. This is as close to an interventional identification of "retrieval coordinate frame matters" as one can get without retraining, and the equal-width PCA–ConvAE reversal (10.99% vs 16.20% reconstruction, yet +3.63 pp worse deployment, paired CI excluding zero) is a well-controlled demonstration that reconstruction fidelity cannot order deployment. This strength is consequential to my assessment because it is the paper's real claim to a lasting contribution: a measurement methodology and a cautionary result that should change how the community screens representations for cross-regime deployment. S2. Audit discipline of a standard rarely seen in this literature. Pre-registered expectations are confronted with outcomes (four of five expectations falsified, including the whitening reversal, and reported as such); a pre-specified Stage-3 gate was not met and the stage was not run, explicitly flagged as "not a null result"; paired bootstrap intervals with pinned seeds throughout; numerical validation against analytic targets (affine PCA decoder checks within 1.9×10⁻⁶); dual records preserved rather than silently reconciled. This is consequential because the paper's genre — an exploratory diagnostic study — lives or dies on the credibility of its negative and reversal claims, and this rigor is what makes the central reversal believable rather than anecdotal. It also gives the paper reusable value (retained arrays, protocols, stopping rules) independent of its conclusions.
Critical weaknesses
W1. The headline claim is demonstrated only on a knowingly non-competitive retrieval instrument, and its portability to learned predictors — the systems readers will assume it concerns — is entirely untested. Within the retrieval pipeline, "transfer" means borrowing source-regime increments: deployment error never passes through any learned dynamics, so the reconstruction–transfer decoupling is, mechanistically, a statement about retrieval geometry and neighbour selection under distribution shift. The title and abstract invite the broader reading (neural PDE surrogates generally — FNO, U-Net, ConvAE-GRU), yet no learned-transition pathway is subjected to the same reconstruction-vs-deployment ordering test. This is consequential because it caps the paper's significance: the finding may be specific to analog-style prediction rather than a property of representation learning for PDE dynamics. Combined with the single benchmark, single grid (64×64), single forcing mode, and single tenfold viscosity shift (with the more discriminating boundary regime disqualified by the resolution audit), the evidence supports a narrower claim than the framing asserts. W2. The study fails on its own stated goal: no mechanism is identified, and the key quantitative pillars rest on fragile foundations. The paper sets out to probe what property of representation geometry drives transfer, but the eight-candidate inventory ends in mixed or inconclusive evidence; the coordinate-frame intervention establishes that the frame matters while explicitly not identifying which property is responsible; and the most informative anomaly (whitening's unexpected +4.75 pp degradation, against a pre-registered expectation of improvement) is not pursued to a mechanistic follow-up. Meanwhile the load-bearing numbers are weaker than the presentation suggests: the oracle ladder — the paper's error-localization backbone — exists only for seed 42, while seed-to-seed spread in the reconstruction reference (13.89%–16.20%) implies the ladder's absolute magnitudes may vary materially; the selection×refresh factorial is incomplete; the JEPA contrast uses a seed-42 physical baseline rather than a three-versus-three control; and several headline-adjacent records remain unreconciled (U-Net 34.72% vs 34.668555%; deployment 38.300117% vs 38.300047%). This is consequential because a diagnostic paper whose diagnostics are single-seed and whose mechanism hunt comes home empty delivers less than its framing promises — this, jointly with W1, is what places the paper below the acceptance line for me.
Questions for authors
Clarification requests (answers could materially change my rating): Q1. Can you run the reconstruction-vs-deployment ordering test on at least one learned-transition pathway — e.g., the same ConvAE encoder paired with (a) retrieval transitions and (b) the trained GRU transition module — and report whether the PCA–ConvAE reversal (or its absence) reproduces? A positive result would directly address W1 and raise my score; a clearly scoped negative result, with claims re-titled to retrieval-based deployment, would partially address it. Q2. Can you replicate the nested oracle ladder on the other two training seeds (123, 7) and report seed-level spread of the three gaps (5.66 / 14.01 / 2.44 pp)? The state-refresh dominance is the paper's most actionable attribution; its robustness across seeds materially affects how much weight it can bear. Q3. Regarding the whitening reversal (+4.75 pp, against your prior expectation): did you examine whether whitening suppresses precisely the low-variance directions that align with the enstrophy-containing spectral band? A dose–response analysis of deployment change against κ(A) or index agreement across your seven frames, plus this one spectral-alignment check, would convert your most intriguing anomaly into a falsifiable mechanistic hypothesis — which would substantively address W2. Q4. Please reconcile or adjudicate the dual records that touch headline comparisons (U-Net 34.72% vs 34.668555%; deployment 38.300117% vs 38.300047%), stating which is authoritative for each claim and what the provenance difference turned out to be. The paper's credibility model is "all records retained"; it needs one designated authoritative value per claim to be interpretable. Suggestions for future work (not required for acceptance; not part of my rating): Q5. Extending the instrument to a second shift magnitude (e.g., fivefold) and a second forcing scale to map how the reversal strength scales with distribution distance; nonlinear intrinsic-dimension estimators (you cite but do not run TWO-NN/MLE) to test whether the curvature-proxy null is estimator-specific; and completing the phase-aligned control. These would strengthen a follow-up but exceed what I consider necessary for this submission.
3 comments
Response to Reviewer MwS2 (2/3): Q2, factorial, Q4 and JEPA
Q2. Is state-refresh dominance robust across seeds? Yes.
We parameterised the ladder and reran it for seeds 123 and 7 (no retraining), after reproducing seed 42 to within 10⁻³ pp with identical oracle indices.
| Seed | Ref. | Refreshed | Free-running | Deployed | Ref.→refreshed | State refresh | Selector |
|---|---|---|---|---|---|---|---|
| 42 | 16.20 | 21.85 | 35.86 | 38.30 | 5.66 [5.56, 5.76] | 14.01 [13.91, 14.11] | 2.44 [2.27, 2.60] |
| 123 | 13.89 | 21.10 | 35.97 | 38.29 | 7.21 [7.08, 7.35] | 14.87 [14.75, 14.99] | 2.31 [2.13, 2.49] |
| 7 | 14.05 | 21.22 | 36.11 | 38.44 | 7.17 [7.04, 7.30] | 14.89 [14.78, 15.01] | 2.33 [2.14, 2.51] |
Levels (%) are rounded for display; gaps (pp) and paired 95% CIs are computed from per-trajectory arrays. State refresh is the largest gap in every seed. As the reviewer anticipated, magnitudes move with the seed: the reference-to-refreshed gap is about 7.2 pp for seeds 123 and 7, so the seed-42 value understated it. Deployment is stable (38.29–38.44%).
W2. The selection × refresh factorial. We ran the missing cell, canonical selection with true-state refresh (N_RIDE = 3 unchanged). For seed 42: canonical 38.30 (generated state) / 21.69 (refresh); direction oracle 35.86 / 21.85. Under refresh, canonical is slightly better than the oracle in every seed: −0.16 [−0.19, −0.13], −0.17 [−0.21, −0.13], −0.22 [−0.26, −0.19] pp. The oracle's advantage (2.31–2.44 pp) therefore appears only after the state has drifted; the ladder's selector gap is conditional on drift, not an independent component. The direction oracle is thus not an upper bound on selection, which the revision now states (Table 2).
Q4. Dual records, adjudicated.
- Deployment (38.300117% vs 38.300047%). The 5,000 selections are identical; the difference is float32 versus float64 aggregation. Three independent code paths agree with 38.300047% to about 10⁻⁸ pp, which we designate authoritative (38.30% in the main text).
- U-Net (34.72% vs 34.668555%). These were two different metrics, not a provenance discrepancy: 34.668555% came from the spectral-weighting evaluation, which excludes the k = 0 mode, while Eq. (1) gives 34.7156%. Re-running all three checkpoints reproduces the cached fields elementwise. The revision removes the "unreconciled" statements, labels the spectral evaluation explicitly, and recomputes the stepwise U-Net errors under Eq. (1). We now report the sample SD (0.74); the earlier ±0.60 is the population SD of the same three seeds.
W2. JEPA, three versus three. Per-trajectory JEPA arrays were not retained, so no paired interval is available. At the seed level, JEPA gives 50.77 / 50.48 / 51.29% and the physical-route ConvAE baseline 38.93 / 39.07 / 39.11%. The ranges do not overlap; the most conservative difference is 11.37 pp (vs 11.92 pp in the original seed-42 pairing). We report this as an unpaired n = 3 comparison.
Response to Reviewer MwS2 (3/3): Q3, Q5 and a question
Q3. Is the whitening degradation due to its direction, or only its conditioning? Its direction.
Dose–response across the seven frames. Deployment error rises with κ(A) and falling index agreement, and the curve is strongly convex: at κ = 2, about 38% of selections change, yet deployment moves by only +0.01 pp (CI includes zero), consistent with many near-equivalent neighbours.
Fixed dose, varied direction. This experiment was designed in direct response to the reviewer's question. Cosine similarity is invariant to a left orthogonal factor, so A = UΣVᵀ affects retrieval only through Σ (the dose) and V (the direction); dropping U reproduces all 5,000 whitening selections exactly. We held whitening's spectrum fixed (κ = 56.35) and changed only V:
| Frame (same Σ) | Δ deployment vs identity (pp) |
|---|---|
| Source whitening | +4.75 [4.30, 5.21] |
| Random orientations (5 draws) | +0.92 to +1.30 |
| Reversed whitening | −0.36 [−0.49, −0.22] |
Whitening is worse, and reversed whitening better, than every random orientation, with all pairwise CIs excluding zero (pre-registered: direction-specific in both directions). Generic distortion at this conditioning costs about 1 pp; the remaining ~3.5 pp is direction-specific.
The spectral hypothesis. One correction: whitening up-weights, rather than suppresses, low-variance directions. In the seed-42 ConvAE latent, low source-variance directions are both the most shifted between regimes (ρ = −0.85 with the target/source variance ratio; permutation p ≈ 10⁻⁴) and decode to higher wavenumbers (ρ = −0.44; p ≈ 10⁻⁴). Whitening raises the distance-weight share of the most-shifted quartile from 0.0247 to 0.2500; reversed whitening lowers it to 0.0040. This is correlational; the causal evidence is the fixed-dose test.
Scope. This uses one encoder seed, and a pre-registered check shows the account does not extend across encoders: PCA-64 places less weight on its most-shifted directions than ConvAE (0.0026 vs 0.0244–0.0268), yet transfers worse. Shift weighting explains frame effects within an encoder, not the PCA–ConvAE reversal; the revision keeps the two separate. Reversed whitening uses source statistics only, so it is a valid zero-shot frame, and it means one diagnostic variant now beats its baseline. We have corrected that statement (Sec. 8) and report the frame as a direction test, not a method (one seed, 0.36 pp).
Q5. We ran TWO-NN and Levina–Bickel MLE, validated on a synthetic 5-dimensional manifold in 64 dimensions. For both representations, target-regime intrinsic dimension exceeds the source (ConvAE about 4.6–4.9 vs 6.1–6.3; PCA-64 4.4–4.8 vs 6.1–6.6). This is descriptive and does not test the curvature–error association. A second shift magnitude and the phase-aligned control need new simulation data and remain future work.
A note on process. Our first reproduction gate for Q1 caught a genuine bug in the adapted training script (a different random-number call order). After the fix, the adapted script matched the original to 0.0002 pp, within the paper's pre-existing 0.01 pp tolerance. Numbers produced before the fix were discarded, and all deviations are logged in our experiment records.
We are genuinely grateful for this review. Its questions could be answered by experiment, and answering them changed the paper: the reversal now holds beyond retrieval, the ladder is robust across seeds, and the whitening anomaly is now a direction-specific result with a testable account. We will respond promptly to any follow-up.
One question for our follow-up work: would the reviewer consider a second shift magnitude on this benchmark (e.g., fivefold) or a second system with a parameter-driven regime change (e.g., Burgers with vanishing viscosity, as Reviewer 4Ffu suggests) more informative?
Response to Reviewer MwS2 (1/3): summary and Q1
We thank the reviewer for an exceptionally careful review, and we apologise for replying so close to the end of the discussion period: we chose to answer the clarification requests by running them, and the experiments took most of the available time.
The reviewer's two central criticisms were fair. Before this round, our evidence could not show whether the reconstruction–transfer reversal extends beyond retrieval (W1), and our error attribution rested on a single seed (W2). Both now have experimental answers. Every new pipeline was first required to reproduce an existing number, and every classification was pre-registered before the corresponding run. The revised PDF has been uploaded. Because of the length limit, our response is split into three comments.
Summary
Q1 / W1. Does the reversal reproduce under a learned transition? Yes, and it is transfer-specific.
We paired the same encoders (ConvAE seeds 42/123/7; PCA-64 as in the paper) with the GRU transition module, using an identical architecture and training protocol for both, trained on the source regime only. To separate a transfer effect from a difference in fit quality, we evaluated on the target regime and on held-out source trajectories (4500–4999, unseen by both encoders and GRUs). Pre-registered criterion: the reversal counts as transfer-specific only if PCA is worse on the target and the target gap exceeds the source gap, with CIs excluding zero in every seed.
The ordering flips between regimes: on held-out source data PCA-64 is better in every seed, consistent with its better reconstruction; on the target it is worse in every seed. The criterion is met. The result holds with 3× training (seed 42, 30 epochs: Δ target +2.84 [2.57, 3.11], DiD +2.40 [2.11, 2.70]). Per-dimension standardisation of the PCA latent performs worse still (50.55 / 50.43 / 50.40%).
So, within this benchmark, reconstruction fidelity fails to order deployment for a learned transition as well as for retrieval, by a similar margin (3.63 pp for retrieval). We have revised the framing accordingly while keeping the scope explicit: one learned transition architecture, one benchmark, one tenfold shift.