- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The authors present PALM, which builds on Mamba and uses memory timescales of multiple scales from the frequency spectrum of the inputs. The intent is to correct for an ambiguity between and A (which the authors term "memory-timescale misalignment"), particularly for scientific modeling. This appaers as a kind of inductive bias, where the input modality is assumed to have distinct regimes in the frequency spectrum and some heads are allocated to each regime. Specifically, the regime-specific heads have their fade rates set appropriately. The paper evaluates the model for fMRI brain imaging and weather forecasting, with mild improvements over included baselines for the former and stronger ones for the latter.
Critical strengths
- The paper presents an interesting inductive bias that - to my knowledge - is novel and quite unique, appears a good candidate for the appropriate domains. As such, it appears to potentially open a new design space, although this is conditional on the utility of the research direction (see below).
Critical weaknesses
- The evaluations - in particular for fMRI - are oddly chosen. In brain imaging, distribution shifts are common and a serious problem in need of better solutions. These can concern shifts between sites, scanners/hardware, and age (spectral shape changes across lifespan!). So the obvious choice to me would have been to do here do also leave-one-site-out, cross-dataset with different TRs, and age-transfer; they're common and have issues w.r.t. 'acquisition units'. Instead, the authors use mostly contrived examples of which I do not see practical relevance, except the rest/task comparison. This means the primary evidence comes from weather, which seems to be missing relevant baselines.
Questions for authors
- Would the authors be able to explain why they chose these fMRI evaluations, and what their practical relevance is? Why were the more 'obvious' options relying on site and hardware not chosen?
- Would it be possible to add standard forecasting baselines (the weather-5k paper itself seems to include them..)?
- I'm uncertain what the Gauge-theory contributes to the paper. Would it be possible for the authors to concretely tie the identified 'mislaignment' to failure behaviour of SSMs in practice?
10 comments
W1 & Q1 & Q2
We thank the reviewer for the perspective from brain imaging; the suggestion to test shifts that change acquisition units led to the paper's most direct test. The inductive bias the reviewer found novel, memory timescales tied to physical units, is exactly what that test isolates: only models whose A is anchored in acquisition units keep a memory and can use a new TR (Q3).
W1 / Q1. Why these fMRI evaluations; why not site, scanner, cross-dataset with different TRs, age? In short: we added both acquisition shifts the reviewer suggests, a change of TR and a change of site and scanner; age transfer is left to future work.
- Why the original axes. Each one changes a single, controllable factor while subjects, labels, and acquisition stay fixed, so any gain can be attributed to how the model handles time; site, scanner, and age shifts change many factors at once. Each also comes from a practical need: (i) decoding from the first few TRs after onset is the calibration problem of fMRI brain–computer interfaces and real-time fMRI, where only a few volumes are available; (ii) fewer labeled subjects is the usual situation in fMRI, where labeled data are scarce; (iii) resting-state to task transfer is what a brain foundation model is meant to do, pretraining on abundant resting-state data and serving task decoding (Section 3).
- Change of TR: zero-shot cross-TR (Section 6.4). One encoder trained at TR = 0.72 s, applied without retraining to the same HCP runs resampled to 1.44 s and 2.16 s (whole blocks; a different protocol from the 3-TR early-window accuracy, where separate models are trained on the first 3 TRs). Over ten seeds at 2.16 s, among the models we compared, only those with an anchored A use the new TR (PALM 0.943 → 0.973, log grid 0.845 → 0.963); the unanchored ones barely change (learnable Δ 0.851 → 0.851, random Δ 0.958 → 0.962). Given the TR, PALM is the most robust SSM (0.973 vs. log grid 0.963, random Δ 0.962, Mamba2 0.887, learnable Δ 0.851). On Weather-5K, after 3 h resampling, PALM has the lowest error of all variants in both splits (0.510 vs. 0.541–0.559; 0.719 vs. 0.755–0.767).
- Change of site and scanner: held-out ABCD sites (Appendix). 22 sites, each scanned on its own GE, Siemens, or Philips scanners (TR = 0.8 s); train on 17 sites, test on the other 5. By device serial number (ABCD release 5.1), the test sites used seven scanners, none of which scanned a training subject; the same scanner models do occur in training. Three seeds, all models at the main-text size: PALM 0.728, Mamba2 0.700, 1D-CNN 0.597, Transformer 0.548. Every PALM-family Δ variant is also above Mamba2 (log-spaced 0.733, random 0.731, learnable 0.726).
Q2. Standard forecasting baselines. In short: PatchTST, which the Weather-5K paper benchmarks, is added with RevIN at a matched budget (17.99–18.31M parameters), together with the three zero-shot forecasting foundation models. PALM is within 3–5% of PatchTST at 168 h and 6–7% at 24 h (split 1 / split 2), and below every foundation model at both horizons in both splits. The gap is largest at 24 h, where the most recent values matter most: PALM uses the same head as for fMRI classification (temporal mean pooling, then a linear map), whereas PatchTST keeps every patch token and maps their concatenation to the forecast. Results:
| Horizon | Model (Weather-5K) | Test MSE, split 1 / split 2 |
|---|---|---|
| 24 h | PatchTST | 0.2893 / 0.3535 |
| PALM | 0.3104 / 0.3757 | |
| Chronos-Bolt (zero-shot) | 0.3417 / 0.4175 | |
| TimesFM-2.5 (zero-shot) | 0.3435 / 0.4167 | |
| Moirai-1.1 (zero-shot) | 0.3629 / 0.4417 | |
| 168 h | PatchTST | 0.4456 / 0.6251 |
| PALM | 0.4590 / 0.6556 | |
| TimesFM-2.5 (zero-shot) | 0.5455 / 0.7559 | |
| Chronos-Bolt (zero-shot) | 0.5607 / 0.8021 | |
| Moirai-1.1 (zero-shot) | 0.5706 / 0.8138 |
Q3
Q3. What the gauge theory contributes; tie misalignment to failure in practice. In short: the theory turns a change of sampling interval into a known transformation of a single parameter, and it predicts exactly when a selective SSM loses its memory. Both predictions are confirmed on two data sets.
- The symmetry. A selective SSM discretizes h_t = exp(Δ_t A) h_{t−1} + Δ_t B_t x_t, so its function is unchanged under (Δ, A, B) → (cΔ, A/c, B/c): the data identify only the products ΔA and ΔB. The physical timescale of memory, τ = 1/|A|, is therefore a gauge freedom, not something training can determine (credited to Gu et al., 2022).
- Why it matters for a selective SSM. Δ_t is recomputed from each input while A is shared, so the gauge decides what a given Δ means physically. Once the gauge is fixed in acquisition units (Δ in seconds, A in 1/s), a change of sampling interval T → rT acts on the model as Δ → rΔ with A, B, and every learned weight unchanged; this is the exact discretization of the same continuous-time system at the new step, so the decay per second, |A|Δ/T, is preserved. Without units on Δ, there is no principled way to know which parameters to rescale.
- Prediction 1: an inconsistent gauge erases memory. The state keeps a fraction e^{−|A|Δ} per step. With Δ in seconds but A at Mamba's unit-free scale, that is e^{−1.67} ≈ 19% per step on fMRI (one sample = 0.72 s) but e^{−10⁴} ≈ 0 on hourly weather data (one sample = 3,600 s): the same mismatch is harmless in one acquisition unit and fatal in another. Measured on Weather-5K, the unanchored models indeed keep no memory: switching their memory off changes the error by 0% in both station splits, while it costs the anchored models 15–34%.
- Why Mamba's defaults give this. A large Δ resets the state (Gu & Dao, 2023, §3.5), and Mamba2 initializes |A| ∈ [1, 16] and Δ ∈ [0.001, 0.1] without units (Dao & Gu, 2024), assuming Δ is a small per-step quantity; the Δ range sets the timescales the model can capture (Gu et al., 2022). Δ in seconds breaks that assumption on hourly data. Anchoring A in the same units restores memory with the same learnable Δ (median 30–35 h; Table 5; see below), so the memory loss is a unit mismatch, not a limit of the architecture.
- Prediction 2: only an anchored model can use a new interval. Given the new TR, the anchored models improve (HCP, whole blocks resampled to TR 2.16 s, ten seeds: PALM 0.943 → 0.973, log grid 0.845 → 0.963; weather: PALM's degradation at r = 2 drops from +10.6% to +5.7%), whereas the unanchored ones barely change (weather: by less than 10⁻⁴; HCP: learnable Δ 0.851 → 0.851, random Δ 0.958 → 0.962), since their Δ carries no unit or there is no memory for the rescaling to act on. Switching the memory off shows that it is what uses the TR (PALM 0.973 vs. 0.956 without memory).
- Where it pays off. Accuracy gains are expected only where the task depends on that memory. On Weather-5K, under 3 h resampling or 12 h truncation, every anchored variant degrades less than every unanchored one in both splits, and after 3 h resampling PALM has the lowest error of all variants, although it is not the most accurate on unchanged inputs.
Table 5
| Variant | A anchored | Test MSE | Median memory | Error increase, memory off | 3 h resampling | 12 h truncation |
|---|---|---|---|---|---|---|
| PALM | ✓ | 0.4590 / 0.6556 | 6.0 / 7.4 h | +18% / +17% | +11.8% / +9.7% | +27.1% / +22.1% |
| Log-spaced Δ | ✓ | 0.4670 / 0.6815 | 31 / 50 h | +26% / +34% | +13.8% / +11.6% | +25.9% / +19.0% |
| No coupling penalties | ✓ (init only) | 0.4702 / 0.6579 | 3.8 / 4.7 h | +15% / +32% | +9.2% / +10.6% | +21.5% / +22.9% |
| Learnable Δ, A anchored | ✓ | 0.4510 / 0.6450 | 35 / 30 h | +15% / +20% | +13.5% / +13.6% | +28.3% / +27.4% |
| Spectral Δ, no A anchor | – | 0.4447 / 0.6296 | 0 / 0 | 0% / 0% | +21.2% / +20.8% | +38.1% / +35.0% |
| Learnable Δ (Mamba2's Δ and A, Δ in seconds) | – | 0.4441 / 0.6270 | 0 / 0 | 0% / 0% | +22.6% / +21.5% | +41.9% / +37.9% |
| Random Δ | – | 0.4479 / 0.6325 | 0 / 0 | 0% / 0% | +24.0% / +19.8% | +42.3% / +35.7% |
General response (to all reviewers)
We thank the reviewers for careful and constructive reviews.