- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper investigates representations learned by a masked autoencoder for near-infrared spectroscopy. The authors pretrain on approximately 7.2 million food and feed spectral scans from 1.3 million measurements. The model divides spectra into contiguous patches, uses positional encodings based on absolute wavenumber, and trains a Transformer through reconstruction of independently normalized masked patches.
Critical strengths
-
Practical relevance and valuable transfer evaluation with limited labels. Chemical-property labels require costly laboratory measurements, making unlabeled pretraining valuable for reducing annotation requirements. Transfer from food and feed to soil measured with a different instrument provides meaningful evidence under both material and instrument shifts, extending beyond evaluation within the pretraining domain.
-
A concrete, testable approach to representation analysis. Savitzky–Golay (SG) filters provide a mathematically defined reference for interpreting learned operations. Beyond visualizing weights, the authors examine their findings using untrained encoders, phase-randomized controls with matched smoothness, polynomial subspace analysis, and layerwise probes.
Critical weaknesses
The link from filter similarity to chemometric functionality and transfer gains remains insufficiently established.
The analyses support the emergence of first-layer weights resembling SG templates and a low-rank polynomial structure. However, these observations do not establish that the filters perform the corresponding derivative or baseline-removal operations within the network. High template correlation does not guarantee the expected response to constant or linear baselines, depending on derivative order. If Pearson correlation is used, adding a constant to every filter coefficient leaves the correlation unchanged while altering the response to the local spectral mean. Structural similarity therefore needs to be distinguished from functional equivalence.
The connection to downstream gains also remains observational. The 90% masking model has a substantially higher template-match rate than the 75% model, yet performs no better in few-label prediction; the distributions of matched derivative orders also differ. This does not rule out the value of derivative features, but indicates that match rate alone cannot explain representation quality. Targeted interventions or controlled filter replacements are needed to establish whether the identified structures contribute to the reported transfer gains.
Because the architecture and training objective largely follow the standard MAE framework, the strength of this mechanistic explanation is central to my assessment of the paper’s substantive contribution.
Questions for authors
-
Does the correlation measure center the learned filter weights? Beyond high correlation, do matched filters exhibit the expected responses to constant, linear, and other low-degree polynomial inputs for their identified derivative orders? Direct comparisons of learned-filter and SG outputs would help assess functional correspondence beyond weight similarity.
-
Does “rediscovering chemometric preprocessing” refer to similarity between local weights and SG templates, derivative-like behavior of local mappings, or suppression of physical interference while preserving chemical information in the overall representation? Given that Appendix E explicitly states that the encoder does not remove scatter by itself, which interpretation does the current evidence support? This distinction materially affects my assessment of the main analytical contribution.
-
What invariances and information losses do per-spectrum input normalization and per-patch target normalization introduce, respectively? In particular, how does the emphasis on local shape induced by target normalization relate to the observed polynomial and derivative-like filters? How do the authors distinguish representational preferences induced by the task design from preprocessing mechanisms learned from spectral data?
3 comments
Response to Reviewer pqiS (Q3, Q4)
Q3: Because the architecture and training objective largely follow the standard MAE framework, the strength of this mechanistic explanation is central to my assessment of the paper’s substantive contribution.
A3: The mechanistic explanation now rests on 3 levels, each tested against a control. The filters resemble SG templates, 2 to 39 times more often than filters of the same smoothness, and per derivative order far more than random filters in the same subspace (Tables 1 and 11). The encoder performs the derivative operation: from the first block on, it removes level and slope, while an untrained encoder responds to them at chance (A1, Table 10). These operations contribute to transfer: frozen SG filters transfer as well as learned ones on every seed, while random filters lose transfer on 3 of 5 seeds. Every one of these encoders that transfers removes level and slope, including the 2 random ones, and the 3 that drop do not (A2, Figures 4 and 10).
Q4: Does the correlation measure center the learned filter weights? Beyond high correlation, do matched filters exhibit the expected responses to constant, linear, and other low-degree polynomial inputs for their identified derivative orders? Direct comparisons of learned-filter and SG outputs would help assess functional correspondence beyond weight similarity.
A4: Yes. The correlation is computed after removing the mean of both the filter and the template (L843-844). We therefore tested each matched filter directly: an SG derivative of order k gives no response to polynomials below degree k, and a linear filter's response to each polynomial is its weight on it. The matched filters put 80 to 99% of their response on their own degree and at most 13% on the lower ones, against 17 to 84% for random filters in the same subspace (L930-940, Table 11). Per seed, only 1 to 9 of 29 to 73 first-derivative filters still respond more to a constant, which the encoder removes from the first block on (Table 10). The functional correspondence thus holds for the matched filters, beyond weight similarity.
Response to Reviewer pqiS (Q5, Q6)
Q5: Does “rediscovering chemometric preprocessing” refer to similarity between local weights and SG templates, derivative-like behavior of local mappings, or suppression of physical interference while preserving chemical information in the overall representation? Given that Appendix E explicitly states that the encoder does not remove scatter by itself, which interpretation does the current evidence support? This distinction materially affects my assessment of the main analytical contribution.
A5: We apologise for the ambiguity. Chemometric preprocessing comprises 2 families of methods, scatter corrections and spectral derivatives (Rinnan et al., 2009). Scatter corrections such as SNV standardise each spectrum, to reduce the multiplicative and additive effects of particle size and packing. Spectral derivatives such as SG act locally: they remove the offset and slope of the baseline within a window and sharpen overlapping bands. Pipelines usually apply both, SNV followed by a derivative, and part of the scatter remains, which also carries information such as particle size (L1101-1115). Our tuned pipelines apply SNV beforehand, and cross-validation then adds an SG derivative in 87% of cases (L1115-1125). The encoder is set up comparably: it uses the patching and input normalisation that PatchTST (Nie et al., 2022) popularised for transformers on time series, which subtracts the mean of each spectrum and divides by its standard deviation (L139-140, L152-154). This coincides with SNV. It thus applies the standard pipeline, SNV followed by a derivative, and learns the derivative itself, as derivative-like behaviour of local mappings (A1 and A4, Tables 10 and 11). In this sense, it rediscovers chemometric preprocessing. We added these points to Section 2 and Appendix E.
Q6: What invariances and information losses do per-spectrum input normalization and per-patch target normalization introduce, respectively? In particular, how does the emphasis on local shape induced by target normalization relate to the observed polynomial and derivative-like filters? How do the authors distinguish representational preferences induced by the task design from preprocessing mechanisms learned from spectral data?
A6: Per-spectrum input normalisation: the encoder is invariant to the offset and scale of each spectrum, as with SNV. No embedding changes by more than under either (L1083-1089). It loses only these 2 numbers per spectrum. Extended multiplicative scatter correction (EMSC) still moves the embedding by about half the distance between samples. The scatter it removes therefore stays in the embedding (Table 15).
Per-patch target normalisation: as in He et al. (2022), the mean and scale of each hidden patch are removed from its reconstruction target (L752-757). The loss is therefore invariant to the level and scale of each hidden patch, while the encoder still sees each visible patch in full. This emphasis on local shape could at most explain why the encoder ignores the level. Suppose it did. Then every model with the same normalisations would ignore the level. In our study (Tables 10 and 12), models with the same normalisations, trained on less data, still give the level 1.5 to 8%, against less than 1% for our encoder. Ignoring the level therefore cannot be explained by the target normalisation alone. For the slope, no assumption is needed: the target normalisation leaves it in the target, yet our encoder removes it, and responds mostly to curvature and cubic shape, 81 to 84% from the first block on (Table 10).
Task design or learned mechanism: to distinguish representational preferences induced by the task design from preprocessing mechanisms learned from spectral data, we pretrained the same model, with the same objective and both normalisations, on less data: LUCAS and food and feed spectra of the same size. With the same task design, these models respond mostly to slope and curvature, 87 to 95% from the first block on, and are far from untrained, with 3 to 8% on other directions against 92% (Table 12). The task design thus induces a preference for local shape, but not the derivative operators: the higher-order operators form with masked pretraining on the full food and feed corpus, as a preprocessing mechanism learned from spectral data. We added these comparisons to Section 6 and Appendices C and I (L949-957, Tables 12 and 23, L459-467).
Response to Reviewer pqiS (Q1, Q2)
We appreciate that pqiS finds the transfer evaluation with limited labels practically relevant and valuable, and the approach to representation analysis concrete and testable.
A1: We ran the requested tests to distinguish structural similarity from functional equivalence, and added them to Section 3 and Appendix C. Light scattering from particle size and packing adds offsets and slopes to a spectrum, and derivatives remove a constant offset and a linear baseline locally (Rinnan et al., 2009). A derivative operation should therefore respond little to the level and slope of a patch, and strongly to its curvature and cubic shape, which follow the overtone and combination bands (L031-042). We added each of these 4 shapes to one patch at a time and measured how much its token changes, at every layer. From the first block on, level and slope cause less than 6% of the change, and curvature and cubic shape 81 to 84%, over 6 seeds with 75% masking and closely with 90%. An untrained encoder gives each of the four about 2%, the chance share among the 48 directions we test (L232-239, Table 10). The pretrained encoder thus performs the derivative and baseline-removal operations, as the Savitzky-Golay (SG) derivatives of chemometrics do. Because the Pearson correlation ignores a constant added to a filter, we also report the level alone. The first layer gives it 3 to 5%, slightly above chance, and from the first block on the encoder gives it less than 1% (Table 10): the network removes the level, as a derivative does.
A2: We followed the request for targeted interventions and controlled filter replacements in a study of 32 pretrainings, added to Section 3 and Appendix C (L240-247, L969-977). We froze the first layer to SG filters on 5 seeds and to random filters on 5 seeds, pretrained the rest, and compared each with the learned encoder of the same seed (Figure 4a). With SG filters, transfer stays at the level of the learned encoder on every seed, losing at most 0.013 in R2 (median over all settings). With random filters, transfer drops on 3 of 5 seeds. The response test separates them: all 13 encoders that transfer, the 6 learned, the 5 SG and the 2 remaining random ones, give level and slope at most 4% of their output change, and the 3 that drop give them 13 to 20% (Figure 10). With the derivative operations built in, transfer is reliable. Without them, it depends on whether the trainable layers rebuild them.
Following Yosinski et al. (2014), we then also kept the blocks after the first layer at their random weights, up to 12 random layers in total, on 2 seeds (L270-278, Figure 4b). Up to 8 random layers, transfer stays within 0.10 of the learned encoder. Beyond 8, it falls on both seeds, by 0.25 with 12 random layers. Yet these encoders still solve the pretraining task: with 12 random layers, the validation loss of the reconstruction is 0.029, against 0.028 with 6. Solving the pretraining task therefore does not guarantee transfer.
This also explains why the 2 mask ratios transfer alike despite different match rates and derivative orders: both respond little to level and slope and mostly to curvature and cubic shape (Tables 8 and 10). These results suggest that the derivative operations, not the match rate, contribute to the transfer gains.