- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
The paper studies how invertible transformations of input features affect neural networks’ finite-sample generalization, even when they preserve all predictive information. Equivalent representations can create different learning difficulties by changing the input distribution and the complexity of the target function in the transformed coordinates. The paper investigated this using synthetic tabular regression experiments with independent Gaussian features and monotonic transformations.
The experiments consider four settings depending on whether the observed features and the response-generating representation are transformed. In matched settings, the target is linear in the observed features; in mismatched settings, the learner must approximate a forward transformation or its inverse.
The results show that equal nonlinearity does not imply equal learning difficulty. NN exhibits different learning curves, and the relative difficulty of transformations can reverse between forward and inverse settings. To explain these differences, the paper decomposes nonlinear targets into orthogonal polynomial components. Separating quadratic energy from higher-order nonlinear energy improves predictions of representation-induced performance penalties on a certain transformation family, particularly in the forward setting. Preprocessing is also not universally beneficial. SPecifically, transforming features to Gaussian marginals can help when the target is linear in the original coordinates but hurt when it is linear in the transformed coordinates.
Finally, representation sensitivity varies across learning algorithms. Standard multilayer perceptrons and a ResNet-style model show performance degradation under certain transformations, while TabICLv2 is less sensitive and XGBoost is nearly invariant. The paper then concluded that representation sensitivity reflects interactions among target structure, input distribution, and the learner’s inductive bias, rather than information loss. The experiment mainly uses synthetic data.
Critical strengths
Novel findings: The paper establishes a tightly controlled paired tabular regression framework that matches alternative invertible feature representations by their population-level departure from linearity. This rigorous design uncovers the surprising and consequential finding that equal aggregate nonlinearity does not guarantee equal NN learning difficulty, and that performance orderings can completely reverse between forward and inverse maps. This cleanly decouples information preservation from finite-sample learning behavior. It shatters the naive assumption that invertible data transformations are benign, providing clear, undeniable proof that neural networks are deeply sensitive to how nonlinear target energy is distributed across feature coordinates.
Critical weaknesses
Relying on a fully Synthetic Setting: The paper’s conclusions are derived entirely from a controlled synthetic environment featuring independent Gaussian latent features, an additive response, and purely coordinate-wise monotone transformations. This is a very simple setting. Because real-world tabular datasets typically involve correlated predictors, complex feature interactions, and heterogeneous noise, the paper leaves open the question of whether these representation sensitivity phenomena frequently persist in actual practice. Without validation on real tabular benchmarks, it is difficult to determine whether the observed performance penalties and ranking reversals are broad properties of tabular learning or artifacts of a strictly controlled environment.
Architectural Scope is also limited: Most of the primary learning curves and evaluations rely heavily on a single standard MLP pipeline, while alternative neural architectures (e.g. ResNet, TabICLv2) are only spot-checked at isolated sample sizes.
Questions for authors
Q1. Impact of Feature Correlation: The current synthetic setup assumes independent latent features. In real-world tabular data, predictors are frequently correlated. How does introducing a non-diagonal covariance matrix affect the orthogonal polynomial projections and the resulting representation penalties? Does feature correlation largely affect the sensitivity differences observed between the forward and inverse maps?
Q2. Non-Additive Target Functions: The data-generating process strictly models the response as an additive sum of active features. How do multi-feature interaction terms (e.g., multiplicative or cross-feature nonlinearities) alter the degree-wise energy allocation and the predictive capacity of the spectral summary models?
One suggestion is to do validation on Real Tabular Benchmarks. To address the limitation of relying entirely on a synthetic setup, future extensions should test whether these representation sensitivity phenomena and penalty orderings consistently manifest on real tabular benchmark suites (e.g., TabZilla or OpenML datasets) where the true data-generating process and latent representations are unknown.
4 comments
Part 2
1. Real data: spectral characterization and interpretation
Table 2. Quadratic () and higher-order () shares of nonlinear coordinate-map energy, in percent, for eight datasets. Each cell reports ; F and I denote forward and inverse maps. Training sizes match Table 1.
| Dataset | Transform | F () | I () | F () | I () |
|---|---|---|---|---|---|
| Abalone | Cubic | 54.3 / 45.7 | 70.4 / 29.6 | 53.5 / 46.5 | 55.9 / 44.1 |
| Abalone | Quintic | 53.2 / 46.8 | 83.7 / 16.3 | 52.3 / 47.7 | 74.3 / 25.7 |
| Abalone | Exponential | 95.9 / 4.1 | 81.2 / 18.8 | 87.0 / 13.0 | 59.3 / 40.7 |
| Abalone | Quadratic–cubic | 98.4 / 1.6 | 73.5 / 26.5 | 89.1 / 10.9 | 46.4 / 53.6 |
| Airfoil | Cubic | 83.3 / 16.7 | 90.3 / 9.7 | 83.3 / 16.7 | 63.9 / 36.1 |
| Airfoil | Quintic | 71.7 / 28.3 | 96.3 / 3.7 | 71.7 / 28.3 | 83.0 / 17.0 |
| Airfoil | Exponential | 96.7 / 3.3 | 85.2 / 14.8 | 89.2 / 10.8 | 64.4 / 35.6 |
| Airfoil | Quadratic–cubic | 99.1 / 0.9 | 78.5 / 21.5 | 95.8 / 4.2 | 47.9 / 52.1 |
| Auto MPG | Cubic | 52.2 / 47.8 | 68.4 / 31.6 | 53.0 / 47.0 | 52.6 / 47.4 |
| Auto MPG | Quintic | 49.3 / 50.7 | 81.4 / 18.6 | 49.5 / 50.5 | 70.5 / 29.5 |
| Auto MPG | Exponential | 95.3 / 4.7 | 83.8 / 16.2 | 85.9 / 14.1 | 65.2 / 34.8 |
| Auto MPG | Quadratic–cubic | 98.6 / 1.4 | 75.5 / 24.5 | 93.2 / 6.8 | 49.4 / 50.6 |
| Bike sharing | Cubic | 18.8 / 81.2 | 29.0 / 71.0 | 34.4 / 65.6 | 39.3 / 60.7 |
| Bike sharing | Quintic | 21.5 / 78.5 | 43.2 / 56.8 | 37.3 / 62.7 | 59.7 / 40.3 |
| Bike sharing | Exponential | 93.4 / 6.6 | 79.5 / 20.5 | 82.1 / 17.9 | 55.3 / 44.7 |
| Bike sharing | Quadratic–cubic | 97.5 / 2.5 | 69.9 / 30.1 | 86.6 / 13.4 | 43.8 / 56.2 |
| California housing | Cubic | 62.7 / 37.3 | 57.9 / 42.1 | 80.9 / 19.1 | 59.2 / 40.8 |
| California housing | Quintic | 62.3 / 37.7 | 65.9 / 34.1 | 78.8 / 21.2 | 72.1 / 27.9 |
| California housing | Exponential | 94.2 / 5.8 | 74.7 / 25.3 | 92.9 / 7.1 | 58.9 / 41.1 |
| California housing | Quadratic–cubic | 95.5 / 4.5 | 68.3 / 31.7 | 95.3 / 4.7 | 48.4 / 51.6 |
| Concrete | Cubic | 53.3 / 46.7 | 56.5 / 43.5 | 72.1 / 27.9 | 57.7 / 42.3 |
| Concrete | Quintic | 47.9 / 52.1 | 61.2 / 38.8 | 63.9 / 36.1 | 71.6 / 28.4 |
| Concrete | Exponential | 96.9 / 3.1 | 87.1 / 12.9 | 91.3 / 8.7 | 68.1 / 31.9 |
| Concrete | Quadratic–cubic | 99.2 / 0.8 | 81.2 / 18.8 | 96.5 / 3.5 | 55.8 / 44.2 |
| Power plant | Cubic | 18.0 / 82.0 | 25.9 / 74.1 | 23.1 / 76.9 | 26.4 / 73.6 |
| Power plant | Quintic | 21.5 / 78.5 | 42.8 / 57.2 | 27.4 / 72.6 | 48.3 / 51.7 |
| Power plant | Exponential | 94.1 / 5.9 | 81.2 / 18.8 | 83.1 / 16.9 | 58.5 / 41.5 |
| Power plant | Quadratic–cubic | 98.0 / 2.0 | 71.6 / 28.4 | 78.2 / 21.8 | 43.4 / 56.6 |
| Red wine | Cubic | 65.5 / 34.5 | 69.0 / 31.0 | 64.0 / 36.0 | 50.0 / 50.0 |
| Red wine | Quintic | 62.6 / 37.4 | 78.3 / 21.7 | 60.9 / 39.1 | 64.7 / 35.3 |
| Red wine | Exponential | 96.9 / 3.1 | 78.6 / 21.4 | 91.3 / 8.7 | 58.2 / 41.8 |
| Red wine | Quadratic–cubic | 98.4 / 1.6 | 71.9 / 28.1 | 95.9 / 4.1 | 45.0 / 55.0 |
and .
Higher-order energy includes the truncation residual. Shares describe coordinate maps, not the response function. Values average features within each repetition, then repetitions. Identity shares are undefined. Transformed feature sets may change across , limiting cross-level comparisons.
The effect of information-preserving transformations on MLP performance varies by dataset and family. Stronger transformations often hurt performance, but exponential and quadratic–cubic transformations improve Bike Sharing performance at every tested strength. Thus, preserving predictive information need not preserve finite-sample performance. Improvements may reflect an easier predictive relationship, although we do not directly identify that mechanism.
The table describes coordinate-map spectra under empirical feature distributions, not the regression function itself. In synthetic experiments, the response is an additive combination of known coordinate maps, directly relating their spectra to the target. In real data, if , an invertible map gives . Its complexity depends on both the original predictive relationship (including interactions) and the transformation. Therefore, marginal map spectra alone need not predict performance changes. Real data supports representation sensitivity; the synthetic setting isolates its connection to spectral structure.
End Part 2
Part 3
2. More neural models
We are currently running experiments using the ResNet and TabICLv2 models on a wider set of sample sizes. We will provide an additional comment here when those experiments have finished.
Questions
3. Feature correlation:
Table 3. ANN test-RMSE improvement (%) relative to the corresponding matched representation under correlated predictors. Positive values indicate improvement; negative values indicate degradation. Entries are mean SD across three repetitions, after averaging the four transformation families within each repetition.
| Setup | Forward BT vs. TT, | Forward BT vs. TT, | Inverse TB vs. BB, | Inverse TB vs. BB, | ||
|---|---|---|---|---|---|---|
| 500 | Active--active | 0.3 | ||||
| 0.6 | ||||||
| 0.9 | ||||||
| Proxies unchanged | 0.3 | |||||
| 0.6 | ||||||
| 0.9 | ||||||
| Proxies transformed | 0.3 | |||||
| 0.6 | ||||||
| 0.9 | ||||||
| 2000 | Active--active | 0.3 | ||||
| 0.6 | ||||||
| 0.9 | ||||||
| Proxies unchanged | 0.3 | |||||
| 0.6 | ||||||
| 0.9 | ||||||
| Proxies transformed | 0.3 | |||||
| 0.6 | ||||||
| 0.9 |
is total sample size before the 80/20 train/test split. B and T denote base and transformed coordinates; the first letter specifies model inputs and the second the response-generating signal. Active--active correlates the ten active predictors. The proxy setups pair each active predictor with a correlated zero-coefficient proxy; active predictors are transformed in both setups. Signal variance is normalized on the training sample. Each comparison holds the response fixed. These preliminary results use the first three complete repetitions; SD is descriptive and is not a confidence interval.
The correlated-feature experiments show that representation sensitivity persists when predictors are dependent. We consider correlation among active predictors and correlation between active predictors and additional predictors that do not directly enter the response equation. The MLP continues to exhibit mismatch penalties across the evaluated correlation levels.
The clearest distinction appears when these additional correlated predictors are left untransformed. At and , the mean inverse-mismatch RMSE increase is 34.7% at both sample sizes when the proxies remain untransformed, compared with 61.6% and 55.5% when they are also transformed. These correspond to negative improvements in the table. This pattern is consistent with the MLP exploiting an untransformed correlated predictor as an easier alternative representation of the signal. The benefit is less pronounced at weaker correlations.
These results suggest that the effect of dependence depends on how correlated predictors are represented relative to the response, rather than correlation being uniformly beneficial or harmful. The table averages the four transformation families within each repetition and reports the mean and standard deviation across three repetitions. We therefore treat these findings as preliminary evidence extending representation sensitivity beyond independent inputs.
I thank the authors for their rebuttal and especially the additional experimental results. The authors have addressed some of my concerns about weakness. However, it is seems the paper requires material update to incorporate all the new experimental result on feature sets and the broad family of model architectures. I will keep my current rating for now and re-consider it after reviewer discussion.
We thank the reviewer for their comments and acknowledgment of our strengths. We address the concerns in three comments.
1. Real data: performance
We have extended our experiments to real datasets, where we also observe representation sensitivity.
Table 1. ANN performance across eight real-world regression datasets and four transformation families, averaged over 25 paired repetitions. The Identity R2 column reports the mean and standard deviation of test R2 for the original, untransformed dataset, providing a baseline for comparison. The remaining columns report the mean percentage improvement in test RMSE relative to the identity baseline at each transformation strength η. Positive values indicate lower RMSE (better performance), while negative values indicate higher RMSE (worse performance). Training size is 500 except for Auto MPG (250). Across these conditions, XGBoost mean RMSE changes were negligible (less than 0.10% in absolute magnitude).
California housing caveat. Its paired RMSE changes vary substantially across splits (quadratic–cubic at η=.20: mean improvement −90.89%, SD 170.68 percentage points). Diagnostics from an earlier partial run showed extreme standardized test-feature values and, in some splits, nearly all squared error concentrated in the worst 1% of test observations. This suggests sensitivity to extreme observations and extrapolation; it does not establish the mechanism for every completed run.
End Part 1