acceptodds
Under review as a conference paper at ICLR 2027

Same Information, Different Generalization: Representation Sensitivity in Neural Network Regression

Abstract

Invertible feature transformations preserve predictive information, yet can affect finite-sample learning performance. We study this representation sensitivity in controlled synthetic tabular regression. Our paired experiments compare transformations with the same normalized population error under optimal affine approximation, thereby matching the total amount of induced nonlinearity while varying its structure. Despite this matching, neural networks can exhibit different learning curves, and the ordering of representation penalties can reverse depending on whether the transformed coordinates define the inputs or the target relationship. In contrast, XGBoost is nearly unaffected by the transformations. We characterize these differences by decomposing nonlinear target energy across polynomial degrees using projections adapted to the observed input distribution. Our results show that neural-network representation sensitivity depends on the target's nonlinear structure in the observed coordinates, the input distribution, and the learner.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
tabular regression, representation learning, generalization, neural networks, feature transformations

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_i8GfOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The paper studies how invertible transformations of input features affect neural networks’ finite-sample generalization, even when they preserve all predictive information. Equivalent representations can create different learning difficulties by changing the input distribution and the complexity of the target function in the transformed coordinates. The paper investigated this using synthetic tabular regression experiments with independent Gaussian features and monotonic transformations.

The experiments consider four settings depending on whether the observed features and the response-generating representation are transformed. In matched settings, the target is linear in the observed features; in mismatched settings, the learner must approximate a forward transformation or its inverse.

The results show that equal nonlinearity does not imply equal learning difficulty. NN exhibits different learning curves, and the relative difficulty of transformations can reverse between forward and inverse settings. To explain these differences, the paper decomposes nonlinear targets into orthogonal polynomial components. Separating quadratic energy from higher-order nonlinear energy improves predictions of representation-induced performance penalties on a certain transformation family, particularly in the forward setting. Preprocessing is also not universally beneficial. SPecifically, transforming features to Gaussian marginals can help when the target is linear in the original coordinates but hurt when it is linear in the transformed coordinates.

Finally, representation sensitivity varies across learning algorithms. Standard multilayer perceptrons and a ResNet-style model show performance degradation under certain transformations, while TabICLv2 is less sensitive and XGBoost is nearly invariant. The paper then concluded that representation sensitivity reflects interactions among target structure, input distribution, and the learner’s inductive bias, rather than information loss. The experiment mainly uses synthetic data.

Critical strengths

Novel findings: The paper establishes a tightly controlled paired tabular regression framework that matches alternative invertible feature representations by their population-level departure from linearity. This rigorous design uncovers the surprising and consequential finding that equal aggregate nonlinearity does not guarantee equal NN learning difficulty, and that performance orderings can completely reverse between forward and inverse maps. This cleanly decouples information preservation from finite-sample learning behavior. It shatters the naive assumption that invertible data transformations are benign, providing clear, undeniable proof that neural networks are deeply sensitive to how nonlinear target energy is distributed across feature coordinates.

Critical weaknesses

Relying on a fully Synthetic Setting: The paper’s conclusions are derived entirely from a controlled synthetic environment featuring independent Gaussian latent features, an additive response, and purely coordinate-wise monotone transformations. This is a very simple setting. Because real-world tabular datasets typically involve correlated predictors, complex feature interactions, and heterogeneous noise, the paper leaves open the question of whether these representation sensitivity phenomena frequently persist in actual practice. Without validation on real tabular benchmarks, it is difficult to determine whether the observed performance penalties and ranking reversals are broad properties of tabular learning or artifacts of a strictly controlled environment.

Architectural Scope is also limited: Most of the primary learning curves and evaluations rely heavily on a single standard MLP pipeline, while alternative neural architectures (e.g. ResNet, TabICLv2) are only spot-checked at isolated sample sizes.

Questions for authors

Q1. Impact of Feature Correlation: The current synthetic setup assumes independent latent features. In real-world tabular data, predictors are frequently correlated. How does introducing a non-diagonal covariance matrix affect the orthogonal polynomial projections and the resulting representation penalties? Does feature correlation largely affect the sensitivity differences observed between the forward and inverse maps?

Q2. Non-Additive Target Functions: The data-generating process strictly models the response as an additive sum of active features. How do multi-feature interaction terms (e.g., multiplicative or cross-feature nonlinearities) alter the degree-wise energy allocation and the predictive capacity of the spectral summary models?

One suggestion is to do validation on Real Tabular Benchmarks. To address the limitation of relying entirely on a synthetic setup, future extensions should test whether these representation sensitivity phenomena and penalty orderings consistently manifest on real tabular benchmark suites (e.g., TabZilla or OpenML datasets) where the true data-generating process and latent representations are unknown.

4 comments

We thank the reviewer for their comments and acknowledgment of our strengths. We address the concerns in three comments.

1. Real data: performance

We have extended our experiments to real datasets, where we also observe representation sensitivity.

Table 1. ANN performance across eight real-world regression datasets and four transformation families, averaged over 25 paired repetitions. The Identity R2R^2 column reports the mean and standard deviation of test R2R^2 for the original, untransformed dataset, providing a baseline for comparison. The remaining columns report the mean percentage improvement in test RMSE relative to the identity baseline at each transformation strength η\eta. Positive values indicate lower RMSE (better performance), while negative values indicate higher RMSE (worse performance). Training size is 500 except for Auto MPG (250). Across these conditions, XGBoost mean RMSE changes were negligible (less than 0.10% in absolute magnitude).

DatasetTransformIdentity R2R^2 (mean ±\pm SD)η=.05\eta=.05η=.10\eta=.10η=.15\eta=.15η=.20\eta=.20η=.30\eta=.30
AbaloneCubic0.514±0.05210.514 \pm 0.0521-3.42-6.28-5.64-7.19-8.73
AbaloneQuintic-3.23-4.20-5.55-7.48-11.49
AbaloneExponential-2.82-3.50-5.12-5.46-9.04
AbaloneQuadratic–cubic-2.15-3.12-4.78-6.02-9.70
AirfoilCubic0.827±0.06590.827 \pm 0.0659-6.41-10.81-12.45-18.00-29.40
AirfoilQuintic-10.54-10.26-15.31-17.45-22.41
AirfoilExponential-15.44-20.40-24.67-30.88-41.51
AirfoilQuadratic–cubic-13.17-23.41-32.62-40.47-64.21
Auto MPGCubic0.828±0.04590.828 \pm 0.0459-0.10-3.83-6.68-7.97-6.29
Auto MPGQuintic-1.11-1.15-4.47-4.90-3.83
Auto MPGExponential-10.36-10.29-15.09-15.03-6.11
Auto MPGQuadratic–cubic-8.48-12.91-16.46-19.47-5.89
Bike sharingCubic0.816±0.03290.816 \pm 0.0329-2.36-3.56-4.17-6.11-0.50
Bike sharingQuintic-0.80-3.37-3.82-4.891.35
Bike sharingExponential4.113.954.001.922.39
Bike sharingQuadratic–cubic3.423.473.031.092.82
California housingCubic−0.114±1.51-0.114 \pm 1.51-42.11-58.29-36.14-66.62-70.36
California housingQuintic-50.73-54.44-47.13-62.44-53.26
California housingExponential-34.32-41.58-47.73-66.97-66.65
California housingQuadratic–cubic-24.92-53.78-63.04-90.89-40.15
ConcreteCubic0.838±0.03650.838 \pm 0.0365-1.01-6.40-11.71-20.49-40.56
ConcreteQuintic0.46-3.81-8.39-10.04-16.27
ConcreteExponential-7.88-17.45-22.55-27.75-38.96
ConcreteQuadratic–cubic-9.31-20.11-34.44-55.24-59.86
Power plantCubic0.935±0.004860.935 \pm 0.00486-3.37-4.28-5.09-10.62-0.72
Power plantQuintic-2.82-3.96-4.85-5.460.09
Power plantExponential-1.57-2.82-4.74-8.25-0.06
Power plantQuadratic–cubic-2.52-6.56-15.47-30.64-0.68
Red wineCubic0.311±0.05790.311 \pm 0.0579-2.60-3.26-4.81-4.86-7.03
Red wineQuintic-3.40-4.51-5.42-6.04-7.70
Red wineExponential-2.04-3.10-3.76-5.26-7.34
Red wineQuadratic–cubic-2.42-2.99-3.79-5.01-6.55

California housing caveat. Its paired RMSE changes vary substantially across splits (quadratic–cubic at η=.20\eta=.20: mean improvement −90.89%-90.89\%, SD 170.68 percentage points). Diagnostics from an earlier partial run showed extreme standardized test-feature values and, in some splits, nearly all squared error concentrated in the worst 1% of test observations. This suggests sensitivity to extreme observations and extrapolation; it does not establish the mechanism for every completed run.

End Part 1

Part 2

1. Real data: spectral characterization and interpretation

Table 2. Quadratic (QQ) and higher-order (HH) shares of nonlinear coordinate-map energy, in percent, for eight datasets. Each cell reports Q/HQ/H; F and I denote forward and inverse maps. Training sizes match Table 1.

DatasetTransformη=.10\eta=.10 F (Q/HQ/H)η=.10\eta=.10 I (Q/HQ/H)η=.30\eta=.30 F (Q/HQ/H)η=.30\eta=.30 I (Q/HQ/H)
AbaloneCubic54.3 / 45.770.4 / 29.653.5 / 46.555.9 / 44.1
AbaloneQuintic53.2 / 46.883.7 / 16.352.3 / 47.774.3 / 25.7
AbaloneExponential95.9 / 4.181.2 / 18.887.0 / 13.059.3 / 40.7
AbaloneQuadratic–cubic98.4 / 1.673.5 / 26.589.1 / 10.946.4 / 53.6
AirfoilCubic83.3 / 16.790.3 / 9.783.3 / 16.763.9 / 36.1
AirfoilQuintic71.7 / 28.396.3 / 3.771.7 / 28.383.0 / 17.0
AirfoilExponential96.7 / 3.385.2 / 14.889.2 / 10.864.4 / 35.6
AirfoilQuadratic–cubic99.1 / 0.978.5 / 21.595.8 / 4.247.9 / 52.1
Auto MPGCubic52.2 / 47.868.4 / 31.653.0 / 47.052.6 / 47.4
Auto MPGQuintic49.3 / 50.781.4 / 18.649.5 / 50.570.5 / 29.5
Auto MPGExponential95.3 / 4.783.8 / 16.285.9 / 14.165.2 / 34.8
Auto MPGQuadratic–cubic98.6 / 1.475.5 / 24.593.2 / 6.849.4 / 50.6
Bike sharingCubic18.8 / 81.229.0 / 71.034.4 / 65.639.3 / 60.7
Bike sharingQuintic21.5 / 78.543.2 / 56.837.3 / 62.759.7 / 40.3
Bike sharingExponential93.4 / 6.679.5 / 20.582.1 / 17.955.3 / 44.7
Bike sharingQuadratic–cubic97.5 / 2.569.9 / 30.186.6 / 13.443.8 / 56.2
California housingCubic62.7 / 37.357.9 / 42.180.9 / 19.159.2 / 40.8
California housingQuintic62.3 / 37.765.9 / 34.178.8 / 21.272.1 / 27.9
California housingExponential94.2 / 5.874.7 / 25.392.9 / 7.158.9 / 41.1
California housingQuadratic–cubic95.5 / 4.568.3 / 31.795.3 / 4.748.4 / 51.6
ConcreteCubic53.3 / 46.756.5 / 43.572.1 / 27.957.7 / 42.3
ConcreteQuintic47.9 / 52.161.2 / 38.863.9 / 36.171.6 / 28.4
ConcreteExponential96.9 / 3.187.1 / 12.991.3 / 8.768.1 / 31.9
ConcreteQuadratic–cubic99.2 / 0.881.2 / 18.896.5 / 3.555.8 / 44.2
Power plantCubic18.0 / 82.025.9 / 74.123.1 / 76.926.4 / 73.6
Power plantQuintic21.5 / 78.542.8 / 57.227.4 / 72.648.3 / 51.7
Power plantExponential94.1 / 5.981.2 / 18.883.1 / 16.958.5 / 41.5
Power plantQuadratic–cubic98.0 / 2.071.6 / 28.478.2 / 21.843.4 / 56.6
Red wineCubic65.5 / 34.569.0 / 31.064.0 / 36.050.0 / 50.0
Red wineQuintic62.6 / 37.478.3 / 21.760.9 / 39.164.7 / 35.3
Red wineExponential96.9 / 3.178.6 / 21.491.3 / 8.758.2 / 41.8
Red wineQuadratic–cubic98.4 / 1.671.9 / 28.195.9 / 4.145.0 / 55.0

Q=100E2/(E2+E3:8+Eres)Q=100E_2/(E_2+E_{3:8}+E_{\mathrm{res}}) and H=100(E3:8+Eres)/(E2+E3:8+Eres)H=100(E_{3:8}+E_{\mathrm{res}})/(E_2+E_{3:8}+E_{\mathrm{res}}).

Higher-order energy includes the truncation residual. Shares describe coordinate maps, not the response function. Values average features within each repetition, then repetitions. Identity shares are undefined. Transformed feature sets may change across η\eta, limiting cross-level comparisons.

The effect of information-preserving transformations on MLP performance varies by dataset and family. Stronger transformations often hurt performance, but exponential and quadratic–cubic transformations improve Bike Sharing performance at every tested strength. Thus, preserving predictive information need not preserve finite-sample performance. Improvements may reflect an easier predictive relationship, although we do not directly identify that mechanism.

The table describes coordinate-map spectra under empirical feature distributions, not the regression function itself. In synthetic experiments, the response is an additive combination of known coordinate maps, directly relating their spectra to the target. In real data, if m(x)=E[Y∣X=x]m(x)=\mathbb{E}[Y\mid X=x], an invertible map U=T(X)U=T(X) gives m(T−1(u))m(T^{-1}(u)). Its complexity depends on both the original predictive relationship (including interactions) and the transformation. Therefore, marginal map spectra alone need not predict performance changes. Real data supports representation sensitivity; the synthetic setting isolates its connection to spectral structure.

End Part 2

Part 3

2. More neural models

We are currently running experiments using the ResNet and TabICLv2 models on a wider set of sample sizes. We will provide an additional comment here when those experiments have finished.

Questions

3. Feature correlation:

Table 3. ANN test-RMSE improvement (%) relative to the corresponding matched representation under correlated predictors. Positive values indicate improvement; negative values indicate degradation. Entries are mean ±\pm SD across three repetitions, after averaging the four transformation families within each repetition.

NNSetupρ\rhoForward BT vs. TT, η=.10\eta=.10Forward BT vs. TT, η=.30\eta=.30Inverse TB vs. BB, η=.10\eta=.10Inverse TB vs. BB, η=.30\eta=.30
500Active--active0.3−8.7±5.0-8.7 \pm 5.0−34.7±8.8-34.7 \pm 8.8−11.2±4.6-11.2 \pm 4.6−33.4±2.2-33.4 \pm 2.2
0.6−19.5±14.2-19.5 \pm 14.2−45.1±25.4-45.1 \pm 25.4−10.7±2.3-10.7 \pm 2.3−36.2±8.5-36.2 \pm 8.5
0.9−29.3±25.3-29.3 \pm 25.3−85.5±59.0-85.5 \pm 59.0−14.9±9.2-14.9 \pm 9.2−48.0±26.6-48.0 \pm 26.6
Proxies unchanged0.3−25.3±6.5-25.3 \pm 6.5−62.2±15.3-62.2 \pm 15.3−27.1±7.6-27.1 \pm 7.6−71.0±13.2-71.0 \pm 13.2
0.6−24.9±7.1-24.9 \pm 7.1−65.4±17.3-65.4 \pm 17.3−23.4±6.2-23.4 \pm 6.2−56.6±4.6-56.6 \pm 4.6
0.9−22.3±9.0-22.3 \pm 9.0−62.0±14.7-62.0 \pm 14.7−14.0±1.5-14.0 \pm 1.5−34.7±5.2-34.7 \pm 5.2
Proxies transformed0.3−23.8±8.0-23.8 \pm 8.0−63.6±20.2-63.6 \pm 20.2−28.4±9.7-28.4 \pm 9.7−70.5±16.2-70.5 \pm 16.2
0.6−25.4±9.3-25.4 \pm 9.3−63.7±18.5-63.7 \pm 18.5−22.2±6.6-22.2 \pm 6.6−61.5±11.6-61.5 \pm 11.6
0.9−23.1±10.0-23.1 \pm 10.0−60.4±19.1-60.4 \pm 19.1−26.5±5.8-26.5 \pm 5.8−61.6±5.2-61.6 \pm 5.2
2000Active--active0.3−12.5±3.6-12.5 \pm 3.6−36.2±7.7-36.2 \pm 7.7−10.1±1.1-10.1 \pm 1.1−26.2±1.9-26.2 \pm 1.9
0.6−11.3±4.1-11.3 \pm 4.1−29.0±9.2-29.0 \pm 9.2−8.5±0.5-8.5 \pm 0.5−21.9±1.2-21.9 \pm 1.2
0.9−12.9±7.3-12.9 \pm 7.3−31.7±12.9-31.7 \pm 12.9−9.2±0.7-9.2 \pm 0.7−21.6±2.0-21.6 \pm 2.0
Proxies unchanged0.3−30.8±3.4-30.8 \pm 3.4−71.3±5.2-71.3 \pm 5.2−26.3±1.9-26.3 \pm 1.9−64.8±2.9-64.8 \pm 2.9
0.6−30.3±3.2-30.3 \pm 3.2−71.3±5.0-71.3 \pm 5.0−25.8±1.9-25.8 \pm 1.9−63.8±1.2-63.8 \pm 1.2
0.9−28.1±3.5-28.1 \pm 3.5−64.4±5.5-64.4 \pm 5.5−17.3±1.9-17.3 \pm 1.9−34.7±1.9-34.7 \pm 1.9
Proxies transformed0.3−30.2±3.7-30.2 \pm 3.7−70.3±7.3-70.3 \pm 7.3−26.4±2.3-26.4 \pm 2.3−65.9±3.9-65.9 \pm 3.9
0.6−29.9±3.4-29.9 \pm 3.4−70.7±5.3-70.7 \pm 5.3−27.4±2.5-27.4 \pm 2.5−65.4±3.1-65.4 \pm 3.1
0.9−29.0±2.9-29.0 \pm 2.9−66.5±6.1-66.5 \pm 6.1−23.0±3.1-23.0 \pm 3.1−55.5±5.1-55.5 \pm 5.1

NN is total sample size before the 80/20 train/test split. B and T denote base and transformed coordinates; the first letter specifies model inputs and the second the response-generating signal. Active--active correlates the ten active predictors. The proxy setups pair each active predictor with a correlated zero-coefficient proxy; active predictors are transformed in both setups. Signal variance is normalized on the training sample. Each comparison holds the response fixed. These preliminary results use the first three complete repetitions; SD is descriptive and is not a confidence interval.

The correlated-feature experiments show that representation sensitivity persists when predictors are dependent. We consider correlation among active predictors and correlation between active predictors and additional predictors that do not directly enter the response equation. The MLP continues to exhibit mismatch penalties across the evaluated correlation levels.

The clearest distinction appears when these additional correlated predictors are left untransformed. At ρ=0.9\rho=0.9 and η=0.30\eta=0.30, the mean inverse-mismatch RMSE increase is 34.7% at both sample sizes when the proxies remain untransformed, compared with 61.6% and 55.5% when they are also transformed. These correspond to negative improvements in the table. This pattern is consistent with the MLP exploiting an untransformed correlated predictor as an easier alternative representation of the signal. The benefit is less pronounced at weaker correlations.

These results suggest that the effect of dependence depends on how correlated predictors are represented relative to the response, rather than correlation being uniformly beneficial or harmful. The table averages the four transformation families within each repetition and reports the mean and standard deviation across three repetitions. We therefore treat these findings as preliminary evidence extending representation sensitivity beyond independent inputs.

I thank the authors for their rebuttal and especially the additional experimental results. The authors have addressed some of my concerns about weakness. However, it is seems the paper requires material update to incorporate all the new experimental result on feature sets and the broad family of model architectures. I will keep my current rating for now and re-consider it after reviewer discussion.

Reviewer_esQGOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper investigates whether neural networks exhibit different finite-sample generalization performance when two feature representations are related by an invertible transformation and therefore preserve exactly the same predictive information. The authors generate independent Gaussian features in synthetic tabular regression tasks and construct new feature representations using monotonic invertible transformations, including cubic, quintic, exponential, and quadratic-cubic transformations. By matching the aggregate nonlinearity of different transformations η(T)=1−Corr⁡(Z,T(Z))2\eta(T)=1-\operatorname{Corr}(Z,T(Z))^2, they compare transformations with the same aggregate nonlinearity but different structures. The paper argues that although invertible transformations preserve predictive information, they change the structure of the target function in the observed coordinates and the input distribution. Consequently, finite-sample learning difficulty also depends on the specific representation and the learner's inductive bias.

Critical strengths
  1. The paper finds that the forward and inverse maps of the same invertible transformation have the same aggregate nonlinear residual but may have substantially different polynomial energy distributions. The difficulty ranking of different transformations may also reverse between the two directions.
  2. Rather than merely reporting performance differences, the paper attempts to explain them using an orthogonal polynomial decomposition adapted to the input distribution.
  3. The paper includes fairly extensive robustness experiments. The authors examine the number of training epochs, model capacity, active and inactive features, where errors occur in the distribution, and different learners.
Critical weaknesses
  1. The experimental scope depends heavily on synthetic data All of the main conclusions are based on independent Gaussian features, an additive response, homoscedastic Gaussian noise, and coordinate-wise monotonic transformations. Therefore, it remains unclear how prevalent the phenomena reported in the paper are in real-world tasks and whether their impact is sufficiently substantial. This is the main limitation of the paper.

  2. The novelty is limited. The idea that invertible representations preserve information but may change finite-sample learning difficulty is not conceptually surprising. The paper needs to explain more clearly what previously unknown conclusion it establishes relative to prior work.

  3. The conclusions need to be validated on more neural models. The main learning curves focus on a single default MLP. ResNet is compared only around n=500n=500, while TabICLv2 is compared mainly at n=2000n=2000. Therefore, the paper has not yet demonstrated sufficiently that different neural network architectures exhibit consistent patterns across the full range of sample sizes.

Questions for authors
  1. Could the authors validate representation sensitivity on real tabular data or semi-synthetic data? For example, invertible transformations could be applied to real-valued numerical features while keeping the labels unchanged, and the resulting performance changes of MLP, ResNet, and TabICLv2 could be examined.
  2. Why is only one Yeo-Johnson transformation family used for held-out validation? Could the authors include multiple unseen transformation families?
  3. In the forward condition, adding spectral energy increases the predictive R2R^2 from 0.826 to 0.985, whereas in the inverse condition it increases only from 0.983 to 0.987. How do the authors explain this pronounced asymmetry? Why is η\eta already sufficiently effective in the inverse task?
4 comments

We thank the reviewer for their comments and acknowledgment of our strengths. We address the concerns in three comments.

1. Real data: performance

We have extended our experiments to real datasets, where we also observe representation sensitivity.

Table 1. ANN performance across eight real-world regression datasets and four transformation families, averaged over 25 paired repetitions. The Identity R2R^2 column reports the mean and standard deviation of test R2R^2 for the original, untransformed dataset, providing a baseline for comparison. The remaining columns report the mean percentage improvement in test RMSE relative to the identity baseline at each transformation strength η\eta. Positive values indicate lower RMSE (better performance), while negative values indicate higher RMSE (worse performance). Training size is 500 except for Auto MPG (250). Across these conditions, XGBoost mean RMSE changes were negligible (less than 0.10% in absolute magnitude).

DatasetTransformIdentity R2R^2 (mean ±\pm SD)η=.05\eta=.05η=.10\eta=.10η=.15\eta=.15η=.20\eta=.20η=.30\eta=.30
AbaloneCubic0.514±0.05210.514 \pm 0.0521-3.42-6.28-5.64-7.19-8.73
AbaloneQuintic-3.23-4.20-5.55-7.48-11.49
AbaloneExponential-2.82-3.50-5.12-5.46-9.04
AbaloneQuadratic–cubic-2.15-3.12-4.78-6.02-9.70
AirfoilCubic0.827±0.06590.827 \pm 0.0659-6.41-10.81-12.45-18.00-29.40
AirfoilQuintic-10.54-10.26-15.31-17.45-22.41
AirfoilExponential-15.44-20.40-24.67-30.88-41.51
AirfoilQuadratic–cubic-13.17-23.41-32.62-40.47-64.21
Auto MPGCubic0.828±0.04590.828 \pm 0.0459-0.10-3.83-6.68-7.97-6.29
Auto MPGQuintic-1.11-1.15-4.47-4.90-3.83
Auto MPGExponential-10.36-10.29-15.09-15.03-6.11
Auto MPGQuadratic–cubic-8.48-12.91-16.46-19.47-5.89
Bike sharingCubic0.816±0.03290.816 \pm 0.0329-2.36-3.56-4.17-6.11-0.50
Bike sharingQuintic-0.80-3.37-3.82-4.891.35
Bike sharingExponential4.113.954.001.922.39
Bike sharingQuadratic–cubic3.423.473.031.092.82
California housingCubic−0.114±1.51-0.114 \pm 1.51-42.11-58.29-36.14-66.62-70.36
California housingQuintic-50.73-54.44-47.13-62.44-53.26
California housingExponential-34.32-41.58-47.73-66.97-66.65
California housingQuadratic–cubic-24.92-53.78-63.04-90.89-40.15
ConcreteCubic0.838±0.03650.838 \pm 0.0365-1.01-6.40-11.71-20.49-40.56
ConcreteQuintic0.46-3.81-8.39-10.04-16.27
ConcreteExponential-7.88-17.45-22.55-27.75-38.96
ConcreteQuadratic–cubic-9.31-20.11-34.44-55.24-59.86
Power plantCubic0.935±0.004860.935 \pm 0.00486-3.37-4.28-5.09-10.62-0.72
Power plantQuintic-2.82-3.96-4.85-5.460.09
Power plantExponential-1.57-2.82-4.74-8.25-0.06
Power plantQuadratic–cubic-2.52-6.56-15.47-30.64-0.68
Red wineCubic0.311±0.05790.311 \pm 0.0579-2.60-3.26-4.81-4.86-7.03
Red wineQuintic-3.40-4.51-5.42-6.04-7.70
Red wineExponential-2.04-3.10-3.76-5.26-7.34
Red wineQuadratic–cubic-2.42-2.99-3.79-5.01-6.55

California housing caveat. Its paired RMSE changes vary substantially across splits (quadratic–cubic at η=.20\eta=.20: mean improvement −90.89%-90.89\%, SD 170.68 percentage points). Diagnostics from an earlier partial run showed extreme standardized test-feature values and, in some splits, nearly all squared error concentrated in the worst 1% of test observations. This suggests sensitivity to extreme observations and extrapolation; it does not establish the mechanism for every completed run.

End Part 1

Part 2

1. Real data: spectral characterization and interpretation

Table 2. Quadratic (QQ) and higher-order (HH) shares of nonlinear coordinate-map energy, in percent, for eight datasets. Each cell reports Q/HQ/H; F and I denote forward and inverse maps. Training sizes match Table 1.

DatasetTransformη=.10\eta=.10 F (Q/HQ/H)η=.10\eta=.10 I (Q/HQ/H)η=.30\eta=.30 F (Q/HQ/H)η=.30\eta=.30 I (Q/HQ/H)
AbaloneCubic54.3 / 45.770.4 / 29.653.5 / 46.555.9 / 44.1
AbaloneQuintic53.2 / 46.883.7 / 16.352.3 / 47.774.3 / 25.7
AbaloneExponential95.9 / 4.181.2 / 18.887.0 / 13.059.3 / 40.7
AbaloneQuadratic–cubic98.4 / 1.673.5 / 26.589.1 / 10.946.4 / 53.6
AirfoilCubic83.3 / 16.790.3 / 9.783.3 / 16.763.9 / 36.1
AirfoilQuintic71.7 / 28.396.3 / 3.771.7 / 28.383.0 / 17.0
AirfoilExponential96.7 / 3.385.2 / 14.889.2 / 10.864.4 / 35.6
AirfoilQuadratic–cubic99.1 / 0.978.5 / 21.595.8 / 4.247.9 / 52.1
Auto MPGCubic52.2 / 47.868.4 / 31.653.0 / 47.052.6 / 47.4
Auto MPGQuintic49.3 / 50.781.4 / 18.649.5 / 50.570.5 / 29.5
Auto MPGExponential95.3 / 4.783.8 / 16.285.9 / 14.165.2 / 34.8
Auto MPGQuadratic–cubic98.6 / 1.475.5 / 24.593.2 / 6.849.4 / 50.6
Bike sharingCubic18.8 / 81.229.0 / 71.034.4 / 65.639.3 / 60.7
Bike sharingQuintic21.5 / 78.543.2 / 56.837.3 / 62.759.7 / 40.3
Bike sharingExponential93.4 / 6.679.5 / 20.582.1 / 17.955.3 / 44.7
Bike sharingQuadratic–cubic97.5 / 2.569.9 / 30.186.6 / 13.443.8 / 56.2
California housingCubic62.7 / 37.357.9 / 42.180.9 / 19.159.2 / 40.8
California housingQuintic62.3 / 37.765.9 / 34.178.8 / 21.272.1 / 27.9
California housingExponential94.2 / 5.874.7 / 25.392.9 / 7.158.9 / 41.1
California housingQuadratic–cubic95.5 / 4.568.3 / 31.795.3 / 4.748.4 / 51.6
ConcreteCubic53.3 / 46.756.5 / 43.572.1 / 27.957.7 / 42.3
ConcreteQuintic47.9 / 52.161.2 / 38.863.9 / 36.171.6 / 28.4
ConcreteExponential96.9 / 3.187.1 / 12.991.3 / 8.768.1 / 31.9
ConcreteQuadratic–cubic99.2 / 0.881.2 / 18.896.5 / 3.555.8 / 44.2
Power plantCubic18.0 / 82.025.9 / 74.123.1 / 76.926.4 / 73.6
Power plantQuintic21.5 / 78.542.8 / 57.227.4 / 72.648.3 / 51.7
Power plantExponential94.1 / 5.981.2 / 18.883.1 / 16.958.5 / 41.5
Power plantQuadratic–cubic98.0 / 2.071.6 / 28.478.2 / 21.843.4 / 56.6
Red wineCubic65.5 / 34.569.0 / 31.064.0 / 36.050.0 / 50.0
Red wineQuintic62.6 / 37.478.3 / 21.760.9 / 39.164.7 / 35.3
Red wineExponential96.9 / 3.178.6 / 21.491.3 / 8.758.2 / 41.8
Red wineQuadratic–cubic98.4 / 1.671.9 / 28.195.9 / 4.145.0 / 55.0

Q=100E2/(E2+E3:8+Eres)Q=100E_2/(E_2+E_{3:8}+E_{\mathrm{res}}) and H=100(E3:8+Eres)/(E2+E3:8+Eres)H=100(E_{3:8}+E_{\mathrm{res}})/(E_2+E_{3:8}+E_{\mathrm{res}}).

Higher-order energy includes the truncation residual. Shares describe coordinate maps, not the response function. Values average features within each repetition, then repetitions. Identity shares are undefined. Transformed feature sets may change across η\eta, limiting cross-level comparisons.

The effect of information-preserving transformations on MLP performance varies by dataset and family. Stronger transformations often hurt performance, but exponential and quadratic–cubic transformations improve Bike Sharing performance at every tested strength. Thus, preserving predictive information need not preserve finite-sample performance. Improvements may reflect an easier predictive relationship, although we do not directly identify that mechanism.

The table describes coordinate-map spectra under empirical feature distributions, not the regression function itself. In synthetic experiments, the response is an additive combination of known coordinate maps, directly relating their spectra to the target. In real data, if m(x)=E[Y∣X=x]m(x)=\mathbb{E}[Y\mid X=x], an invertible map U=T(X)U=T(X) gives m(T−1(u))m(T^{-1}(u)). Its complexity depends on both the original predictive relationship (including interactions) and the transformation. Therefore, marginal map spectra alone need not predict performance changes. Real data supports representation sensitivity; the synthetic setting isolates its connection to spectral structure.

End Part 2

Part 3

2. Novelty

Why neural networks perform well on some tabular datasets and poorly on others remains only partly understood. Previous work has associated their performance with feature-distribution characteristics such as skewness, heavy tails, and related measures of irregularity (McElfresh et al., 2023). Other studies have identified difficulties learning irregular target functions and connected these difficulties to neural networks' spectral biases (Grinsztajn et al., 2022; Beyazit et al., 2023). Building on these findings, we ask a more specific question: how does the structure of the target relationship affect finite-sample neural generalization when predictive information and aggregate departure from linearity are held fixed?

We address this question through controlled synthetic regression experiments comparing invertibly related feature representations. Responses remain fixed within each paired representation comparison, and transformation families are calibrated to have the same population error under optimal affine prediction. Despite these controls, neural networks exhibit substantially different learning curves across transformation families. In the forward setting, these differences arise even though the input distributions are identically Gaussian. Moreover, the relative difficulty of transformation families can reverse between forward and inverse tasks. These results isolate a dependence on the structure of the target relationship that aggregate nonlinearity and input-distribution characteristics alone do not capture.

We further characterize this dependence using polynomial decompositions adapted to the observed input distribution. Separating quadratic energy from the remaining nonlinear residual improves prediction of representation penalties on a held-out transformation family, particularly in the forward direction. The experiments also demonstrate a practical consequence: Gaussianizing inputs can improve or degrade performance depending on how the target relates to the observed coordinates.

We will clarify this novelty further in the revised paper.

3. More neural models

We are currently running experiments using ResNet and TabICLv2 on a wider set of sample sizes. We will provide an additional comment when those experiments have finished.

Responses to questions

Q1 (real data). Please see our response to weakness 1 in comments 1 and 2.

Q2 (held-out transformations). The Yeo–Johnson evaluation serves as an initial test of transfer beyond the four transformation families used to fit the penalty-prediction models. The fitted coefficients were held fixed when evaluating this unseen family across transformation strengths and sample sizes. We agree that testing additional held-out families would provide stronger evidence of generality. Our current results establish transfer to Yeo–Johnson, particularly in the forward direction; predictive performance across a broader collection of transformation families remains an open question. We will make this scope explicit in the contribution statement and discussion.

Q3 (forward/inverse asymmetry). Although η\eta is identical between directions, it measures total nonlinear energy, whose allocation across higher-order polynomial degrees changes after inversion. Our enhanced model adds the quadratic-versus-remaining-energy distinction, so its incremental benefit need not be symmetric. For the held-out Yeo–Johnson family, η\eta, sample size, and their interaction already predict inverse penalties very well. Moreover, using only quadratic and cubic energy reduces inverse prediction R2R^2 to 0.866, supporting the importance of retaining the full nonlinear residual. We therefore interpret the small improvement from 0.983 to 0.987 as specific to this evaluation; it does not establish that spectral structure is generally unnecessary for predicting inverse difficulty.

Reference: Beyazit et al., "An Inductive Bias for Tabular Deep Learning," NeurIPS, 2023.

Thank you for the detailed response. The authors have addressed some of my concerns, but several issues remain unresolved. For example, the proposed method has not been evaluated on additional held-out transformation families, so its broader generalizability remains unclear. I will keep my current score for now and revisit it after considering any further author responses and the broader discussion among the reviewers.

Reviewer_qJAKOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

In supervised learning problems, an invertible transformation of the input features preserves the information available for prediction, but may not preserve the difficulty of learning the resulting input-output relationship from finite data. This issue is particularly relevant for tabular learning, where transformations of numerical features are common. The sensibility to non-linearity is measured by a function of the correlation. The authors show that neural-network representation sensitivity depends on the target’s non-linear structure in the observed coordinates, the input distribution, and the learner.

The paper has no significant contributions because the framework is too simple (a unique random variable with usual nonlinear transforms). The complexity of neural networks is not really taken into account. The sensitivity is just studied from an experimental point of view. The validation is convincing but limited.

Critical strengths

The authors compare several cases of non-linearity, which is interesting and relevant. By this way, they show that equal sensitivity does not imply equal neural-network learning difficulty.

The spectral decomposition is interesting. It helps to distinguish quadratic from higher-order nonlinear energy.

Critical weaknesses

The value \eta(Z) is well known: it measures departure from linear predictability, not all forms of nonlinear behavior.

The analysis is based on models with a single unidimensional random variable. A vector model would be more relevant.

The nonlinear transformations that are studied are rather standard.

Neural networks are limited to MLP and ResNet. Transformers could be also compared.

The decomposition on a polynomial basis is well known. There is only a slight novelty to exploit this kind of decomposition.

Questions for authors

Clarification requests.

What is the rationale for fitting the model as a function of \log n?

What is the rationale for using only a correlation-based sensitivity measure to quantify nonlinearity?

6 comments

We thank the reviewer for their comments, which we address below.

Significance of the contribution:

Understanding why neural networks perform well on some tabular datasets and poorly on others remains incomplete. Previous work has identified feature-distribution irregularity and target-function complexity as contributing factors (McElfresh et al., 2023; Grinsztajn et al., 2022; Beyazit et al., 2023). Our contribution is a controlled empirical characterization of how the observed representation affects finite-sample learning difficulty.

We use η\eta specifically to match the normalized population error of optimal affine prediction across transformation families. The synthetic design provides known signal-generating coordinates and preserves the response within each paired representation comparison. These controls reveal three major findings:

  1. Forward and inverse mismatches have the same η\eta, yet the ordering of families can reverse between them. For example, at η=0.30\eta=0.30 and larger nn, the cubic and quintic maps are the hardest in the forward condition, while the quintic map is among the easiest in the inverse condition (Figure~1).
  2. Gaussianizing preprocessing can improve or degrade performance depending on the target relationship. For example, at η=0.30\eta=0.30, n=2000n=2000, averaged over families, a normal-quantile transform raises test R2R^2 by 0.204 when the signal is linear in the base coordinates and lowers it by 0.232 when it is linear in the transformed ones (Figure~3).
  3. Splitting η\eta into quadratic and remaining energy predicts penalties on a held-out Yeo-Johnson transformation family. In the forward direction the R2R^2 for predicting the penalty rises from 0.826 with the baseline using η\eta, sample size, and their interaction, to 0.985; in the inverse direction the baseline already predicts well and the gain is small (Table~2).

We also note that we are working on additional experiments that will address this and other concerns below. We will post those results as they finish.

Weaknesses 1, 3 and 5: η\boldsymbol{\eta}, the families and the polynomial decomposition are well known

We agree and do not claim them as new. As you note, η\eta measures departure from affine predictability and not all nonlinear behavior; that gap is what the paper studies. η\eta is only the matching criterion, and the families are standard on purpose because they are easy to interpret and calibrate. Hermite expansions are also standard, including in theory such as the information exponent (Ben Arous et al., JMLR 2021).

Two things are specific to our use. In the inverse condition the observed input is not Gaussian, so we build polynomials orthonormal under its distribution and keep the projection residual (Section 2.3, Appendix A.2). This shows the same η\eta is allocated very differently in the two directions: at η=0.30\eta=0.30 the fraction of nonlinear variation unexplained by projection through degree three is 0.167 for the forward quintic map and 0.927 for its inverse (Table 4). At each matched η\eta, the standardized tasks share the same degree-one energy, c12=1−ηc_1^2=1-\eta, while the allocation of the remaining nonlinear variation can differ substantially.

Weakness 2: single unidimensional random variable

The prediction experiments are multivariate (Section 2.1): p=50p=50 independent Gaussian base features, observed in either their original or transformed coordinates. a=10a=10 of them are active, and

Y=∑j=110Sj+ε,Y=\sum_{j=1}^{10}S_j+\varepsilon,

where SjS_j is the active coordinate in the base or transformed representation. The transformations act coordinatewise, and the scalar analysis describes what each does to one coordinate. Appendix B.3 varies how many active features are mismatched: test R2R^2 falls almost linearly in that number, and transforming inactive features has a negligible effect. We will make the setup clearer in the revised paper. The additive response and the absence of coordinate mixing are real restrictions, listed in Section 4. We chose them to keep the comparison controlled, and extending to interactions and correlated features is future work.

Weakness 4: only MLP and ResNet, no Transformers

The paper already includes two further learners (Section 3.3) including TabICLv2, a pretrained transformer-based tabular model. At n=2000n=2000 it is much more robust than the MLP, especially in the inverse condition, but small differences across families and directions remain (Appendix D.2, Figure 12). XGBoost is also included and is nearly invariant, with ∣ΔR2∣|\Delta R^2| below about 4×10−44\times10^{-4} (Appendix C.3). So the comparison already goes beyond the MLP and ResNet, and it shows that the size of the effect depends on the learner. We will make these results more prominent in the main text. We are also currently working on experiments with different sample sizes for this model in response to the other reviewers' comments.

End of Part 1.

Part 2

Question 1: why log⁡n\boldsymbol{\log n}?

It is an empirical choice used only in the penalty-prediction regressions of Section 2.5; Figure 1 does not depend on it. The sample sizes run from 250 to 10,000 in roughly geometric steps, so log⁡n\log n spaces them approximately evenly, and it described the penalty curves better than the alternatives we tried, including a fit linear in nn. We make no claim outside that range and will report the comparison.

Question 2: why a correlation-based measure?

For Z∼N(0,1)Z\sim\mathcal{N}(0,1), η(T)=1−Corr⁡(Z,T(Z))2=min⁡b0,b1E[(T(Z)−b0−b1Z)2]Var⁡(T(Z)),\eta(T) = 1-\operatorname{Corr}(Z,T(Z))^2 = \frac{\min_{b_0,b_1}\mathbb{E}\left[(T(Z)-b_0-b_1Z)^2\right]}{\operatorname{Var}(T(Z))}, the fraction of the variance of T(Z)T(Z) that the best affine function of ZZ fails to explain. It is invariant to affine rescaling and takes the same value for the forward and inverse problems, which is what lets us compare the two directions. It is not the only summary we consider: a curvature summary added almost nothing beyond η\eta on the held-out family, while the spectral split did in the forward direction (Table 2). We will consistently describe η\eta as the aggregate departure from affine predictability and distinguish it from the allocation of nonlinear variation across polynomial degrees.

Network complexity

We have already performed experiments to demonstrate that the representation penalties persist across the network widths and training budgets tested, despite near-perfect training fit (Appendix C.2). At n=500n=500 without early stopping, the penalties persist from 250 to 2000 epochs (Figure 9). Increasing the parameter count by a factor of 8.6 reduces the R2R^2 degradation penalty by only 6 to 14% (Table 6), and every fit reaches training R2≥0.99R^2\ge0.99, so the penalty is not a failure to fit the training data. A ResNet with a fixed configuration and no hyperparameter search shows the same qualitative penalties (Appendix D.1). TABICLv2 also shows degradation in the forward direction, though the overall penalities are reduced in comparison to the previously mentioned learners.

Beyazit et al., "An inductive bias for tabular deep learning," NeurIPS, 2023.

Reviewer_qJAKOpenReview ↗

Response to Authors

Thank you for your detailed response.

Significance of the contribution

I believe that you should provide a theoretical justification of the extent to which the correlation-based measure is a good indicator of non-linearity. Generally speaking, correlation measures the strength and direction of a linear relationship, rather than those of a non-linear relationship. Using η\eta as an arbitrary measure of non-linearity seems rather limited. The polynomial decomposition provides some insights into the spectral allocation of non-linearity, but this aspect should also be investigated in greater depth.

Weaknesses 1, 3 and 5

I understand that the same η\eta is allocated differently between the forward and inverse transformations. However, it remains unclear to me how this allocation can actually help improve the exploitation or design of a neural network.

Weakness 2: Single unidimensional random variable

I appreciate your answer, especially the sentence: "The additive response and the absence of coordinate mixing are real restrictions, listed in Section 4", which perfectly captures the limitation identified in Weakness 2.

Weakness 4: Only MLPs and ResNets, no Transformers

The title of the paper focuses on neural networks. Therefore, I would have expected a study specifically addressing transformers, rather than including them within a more general architectural framework. I do not consider XGBoost to be a neural network.

Question 1: Why log n?

Thank you for your answer.

Question 2: Why a correlation-based measure?

Thank you for your answer, but I am still not convinced that this measure is truly informative when studying neural networks for tabular data processing. The same criticism could also apply to R2R^2 as a measure of goodness of fit for studying the nonlinearity impact. This aspect should also be investigated in greater depth.

Network complexity

Indeed, your experiments show that the representation penalties persist across the network widths and training budgets considered. However, I still fail to see how this result can be used to adapt a neural network architecture to the degree of non-linearity present in the input tabular dataset.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.