acceptodds
Under review as a conference paper at ICLR 2027

What Determines the Optimal Target Transform? Resolution versus Retransformation Bias

Abstract

Regression models are often trained on a transformed target, such as its logarithm, and their predictions are then mapped back. We study which transform gives the smallest error on the original scale and show that the answer is set by two competing effects. The first is a retransformation bias: mapping predictions back shifts them away from the conditional mean by an amount that grows with the noise in the target, so this effect favors transforms close to the identity. The second is resolution: the transform decides where along the target range the model spends its limited precision, so this effect favors stretching the ranges where most targets lie. We derive an expression that contains these two effects and prove that no rule based on the target distribution alone can pick the best transform for every dataset. In controlled experiments with a fixed target distribution, the best transform moves with the predictability of the target as the analysis predicts, and removing the bias moves it toward stronger compression. Trained models keep their error roughly constant in the units in which they are trained, so standardizing the transformed target matters, but their error is far from uniform across the target range. On 41 tabular datasets, a simple statistic computed from the training targets predicts where the choice of transform matters, the uniform quantile transform is worse than a cube-root transform on every dataset with a neural network, and choosing the transform on a validation set works best in practice.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
Regression, Target Transformation, Target Preprocessing, Retransformation Bias

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_y8mrOpenReview ↗
Rating
3
Confidence
1
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies how to choose a regression-target transformation when predictions are evaluated on the original scale. It identifies two competing effects: retransformation bias, which arises when inverting the transformed conditional mean, and resolution, which describes how a transformation allocates the learner’s accuracy across the target range. The authors derive a small-noise risk expansion combining these effects and prove, under an idealized learner model, that the target distribution alone cannot universally determine the optimal transformation. Controlled experiments investigate these mechanisms, while experiments on 41 tabular datasets compare transformations across several learners. The main practical conclusion is to standardize transformed targets and select the transformation using validation performance.

Critical strengths
  1. The paper addresses an important and practical problem. Target transformations are widely used, yet their effects on original-scale prediction error are often poorly understood. Explaining why the same transformation helps one regression problem but hurts another is valuable.

  2. The theoretical framework provides an interesting synthesis. Combining classical quantization results with retransformation bias offers a coherent explanation of competing effects. The impossibility theorem provides a precise illustration of why information about the relationship between features and targets matters.

  3. The controlled experiments are thoughtfully designed. Holding the target distribution fixed while varying predictability helps distinguish marginal-distribution effects from conditional-distribution effects. The bias-correction experiments provide additional evidence that retransformation bias influences the preferred transformation.

Critical weaknesses
  1. There remains a substantial gap between the theoretical learner and practical models. Definition 1 assumes a centered learning error independent of the input, with a transform-independent shape. However, Table 1 shows strong error heterogeneity: the median maximum-to-minimum decile error ratio is 11.8 for the standardized MLP and 23.6 for HGB. Even after variance normalization, errors vary substantially across transformations. Proposition 1 accommodates more general errors, but their dependence on the transformation remains unspecified. Consequently, the framework explains some qualitative trends without reliably predicting practical optima.

  2. The guidance for selecting target transformations is still unclear. While the paper offers valuable theoretical insights into target transformations, translating these insights into actionable selection guidelines remains challenging. The practical recommendation still largely relies on validation-based selection. It would strengthen the paper to clarify when the proposed diagnostics can simplify this process or offer benefits beyond standard validation-based tuning.

While my expertise in this area is limited, my overall assessment of the paper is positive, and I recommend weak acceptance.

Questions for authors

Please see the weaknesses.

4 comments
AuthorsOpenReview ↗

Response to Reviewer y8mr (1/2)

We sincerely thank the reviewer for the careful review, positive evaluation, and insightful comments, which helped us further improve the paper. To address the reviewer's concerns, we added two new analyses and made our statements more precise. All changes in the revised PDF are marked in blue. We provide our responses below.


W1. There is a substantial gap between the theoretical learner and practical models. Definition 1 assumes a centered, input-independent error with a transform-independent shape, while Table 1 shows strong error heterogeneity (median decile ratio 11.8 for the standardized MLP and 23.6 for HGB). Proposition 1 accommodates more general errors, but their dependence on the transformation remains unspecified. Consequently, the framework explains some qualitative trends without reliably predicting practical optima.

A1. We agree with this characterization, and the revision states it explicitly. In the introduction, the answer to the third research question now reads: "It captures qualitative trends; a controlled synthetic diagnostic shows how error location contributes to discrepancies." To measure that contribution, new Appendix T (summarized in Section 5) reports a controlled synthetic diagnostic.

Design.

  • We use the noiseless setting of Theorem 2 with an MLP and HGB.
  • For each fitted model, we take its own transformed-space errors δ=s^−h(μ)\delta=\hat s-h(\mu) and reassign them to random test points.
  • This keeps the error distribution fixed and changes only where the errors fall.

Results.

  • For h′∝y−ch'\propto y^{-c} with c∈c\in{1/4,1/3,1/2}, the permuted risk is 2.2–3.7× lower than the actual risk.
  • Both learners place larger errors where h−1h^{-1} is steep: Cov(δ2,h′(μ)−2)>0\mathrm{Cov}(\delta^2,h'(\mu)^{-2})>0 in every seed.
  • After permutation, the optimum moves toward c∈c\in{1/3,1/2}. For the tested c∈c\in{1/4,1/3,1/2}, the scalar resolution term of Eq. (1) is 0.87–0.99 times the permuted risk.
  • Using the measured error profile, the term of Proposition 1, E[δ2/h′(μ)2]\mathbb E[\delta^2/h'(\mu)^2], is 0.94–1.02 times the MLP's actual risk for the tested values c∈c\in{0,1/4,1/3,1/2}.

Limits.

  • The profile term is 1.24 times HGB's risk at c=1/2c=1/2 and fails for the logarithm.
  • For raw targets the permutation effect is a clipping effect.
  • The intervention keeps transform-specific error sizes, so it does not test A1 or A2.
  • It uses measured errors, so it explains rather than predicts.

In this controlled setting, the diagnostic shows that where a learner makes errors can materially change original-scale risk even when the transformed-space error distribution is held fixed. This identifies error location as one contributor to the gap between scalar idealizations and trained learners. How a learner's error profile depends on the transform remains open, as stated in the Discussion (Section 8).

AuthorsOpenReview ↗

Response to Reviewer y8mr (2/2)

W2. The guidance for selecting target transformations is still unclear, and the practical recommendation still largely relies on validation-based selection. Please clarify when the proposed diagnostics can simplify this process or offer benefits beyond standard validation-based tuning.

A2. We agree that validation remains the final selector, and the revised abstract reports only that validation achieved lower aggregate selection regret than the target-only selectors we evaluated. Theorem 3 gives a formal reason within the idealized model: no fixed rule that uses only the target marginal is asymptotically optimal for every joint law. The practical role of KK is to prioritize candidates using training targets before fitting; validation then selects among the fitted candidates. The analysis provides the following guidance:

  1. Which candidates to fit when fits are limited (new Appendix Y). At each fit budget, the KK-based, fixed-CRC, and random shortlists use the same number of model fits, with final selection by validation RMSE. All ratios below are test RMSE relative to Std3, the validation-selected choice among raw targets, log1p and Box–Cox.
    • For the MLP, adding the three family members with the smallest KK to these three standard transforms (six fits) gives 0.986.
    • Adding the three CRC variants to the same standard set gives 0.992, while adding three random family members gives 0.999.
    • With nine fits, the mean paired ratio to validation within the 63-configuration pool is 1.020.
    • The gain is modest, does not appear for HGB (whose stored pool is narrower), and does not consistently reduce the 90th percentile of regret. The p-values are exploratory.
  2. Which candidates to treat cautiously. The uniform quantile transform is worse than CRC on 38–41 of 41 datasets across the four nonlinear learners tested, and on both new frozen-embedding tasks. The larger penalties of the neural learners coincide with more predictions near the steep upper end of the inverse map (Appendix U).
  3. How to train. Standardizing the transformed target changes how the learner's error scales with the transform and with it the ranking (Section 5), so every candidate should be standardized the same way.
  4. How to read LL. LL tracks how much transforms that place resolution differently can differ (Spearman 0.76 with the CRC-log/CRC contrast; Appendix S). We did not detect an association between LL and the gain from selection itself, so we no longer use an LL threshold to decide whether transformation selection is worthwhile. Instead we recommend always including raw targets. For the MLP, validation selection over the evaluated candidate pool gave a mean per-seed test RMSE ratio over 20 seeds (selected/raw) below 0.98 on 9 of 22 low-leverage datasets and above 1.02 on none.

We thank the reviewer again for the constructive review.

I thank the authors for their detailed response. The revisions strengthen the paper and address most of my concerns. Thus, I maintain my positive score of 3.

AuthorsOpenReview ↗

Thank You for Your Feedback

We sincerely appreciate the reviewer’s acknowledgment that our revisions have strengthened the paper and addressed most of their concerns, as well as their continued positive evaluation of our work. The reviewer’s insightful feedback prompted us to further investigate the gap between theory and practice and provide clearer practical guidance for selecting target transformations.

Reviewer_oyevOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper studies target transformation in regression, namely which transform gives the smallest error on the original scale. The paper derives an expression that captures two competing effects, retransformation bias and resolution. Then, the paper proves that no rule based on the target distribution alone can pick the best transform for every dataset, and shows on 41 tabular datasets that selecting the transform on a validation set works best in practice.

Critical strengths
  1. The paper gives a detailed analysis of the mechanism behind the target transformation problem. The paper's theory is solid and the experiments are extensive. This proof helps understand why the same transform can help on one dataset but hurt on another.

  2. The paper also offers some insightful practical recommendations.

Critical weaknesses
  1. On the real datasets, global smearing is the only explicit bias correction used. Please explain this choice and why other approaches were not considered.

  2. The study is restricted to tabular regression with relatively small models,, which may limit the breadth of its applicability.

Questions for authors
  1. On the real datasets, why is global smearing the only bias correction used, and why were other approaches not considered?

  2. Can the study be extended to non-tabular regression settings?

3 comments
AuthorsOpenReview ↗

Response to Reviewer oyev

We sincerely thank the reviewer for the positive evaluation and the two concrete questions, which helped us further strengthen the paper. To address the reviewer's concerns, we conducted new experiments for both questions. All changes in the revised PDF are marked in blue. We provide our responses below.


W1. On the real datasets, global smearing is the only explicit bias correction used. Please explain this choice and why other approaches were not considered.

Q1. On the real datasets, why is global smearing the only bias correction used, and why were other approaches not considered?

A1. We used global smearing (Duan, 1983) as a common nonparametric robustness check across the transforms considered. Because it does not generally remove conditional retransformation bias under heteroscedasticity (Manning, 1998), we used the controlled setting with a known conditional law to test the retransformation-offset mechanism directly. There, an offset correction removes the exact retransformation offset of the transformed conditional mean (Section 5, Appendix E), and a learned correction is also evaluated (Appendix P).

To address the reviewer's concern directly, new Appendix W applies the following three corrections to every transform on all 41 datasets and four learners:

  • Global smearing, as in the submission, which pools validation residuals over all predictions and so assumes homoscedastic errors.
  • Binned smearing, which pools them within deciles of the prediction and so allows heteroscedasticity.
  • A learned residual correction: a gradient-boosted model fitted on validation rows to y−h−1(s^)y-h^{-1}(\hat s) from (x,s^)(x,\hat s), in the spirit of TranSUN but not a reimplementation. Its validation scores come from two-fold cross-fitting; the test-time corrector is then fitted on all validation rows.

We apply the same corrections to raw targets as a control, so that a gain is not attributed to a transform when it comes from correcting the base model. CRC and CRC-log keep the configuration selected under plain validation, while the selected standard transform is reselected using each correction's validation score.

Table 37 shows the results:

  • In all-dataset median ratios, CRC stays within about 1% of raw targets under each correction, and the selected standard transform within 3% of CRC.
  • The learned correction also improves the untransformed model. The learned correction improves raw-target RMSE by about 3% for the MLP and ResNet. Thus, gains from this correction cannot be attributed solely to removing retransformation bias, since it also reduces errors of the untransformed model. The controlled experiment therefore remains our direct test of the retransformation-offset mechanism.
  • The high-leverage CRC advantage depends on the learner. Under the learned correction, CRC's advantage over raw targets disappears for the MLP (0.975 → 1.000) and the ResNet (0.993 → 1.003), but remains for XGBoost (0.892 → 0.905). We report this in Section 6.
  • The uniform quantile transform. It gains most from the learned correction (MLP: 0.91 of its plain RMSE, against 0.97 for raw targets). Its median ratio to raw targets still stays above 1 under every correction.

W2. The study is restricted to tabular regression with relatively small models, which may limit the breadth of its applicability.

Q2. Can the study be extended to non-tabular regression settings?

A2. The formal analysis does not require tabular inputs. Beyond the MLP, HGB and ridge regression, the submission already included XGBoost and a tabular ResNet on all 41 datasets (Appendix O).

Following your suggestion, new Appendix X extends the evaluation to two regression tasks using frozen pretrained text and image representations, with the same four learners:

TaskDataEmbeddingTargetLL
Wine98,166 wine reviewsall-mpnet-base-v2 text embeddingsprice (USD 4–3,300)7.6
IMDB-WIKI213,499 face imagesOpenCLIP ViT-B/16 (LAION-2B) image embeddingsage (1–100)1.11

The uniform-quantile penalty is also observed on both tasks (Table 38):

  • Wine. The uniform quantile transform again has a much higher RMSE than CRC, and the penalty is again largest for the neural learners: MLP 2.80, ResNet 2.26, HGB 1.27, XGBoost 1.22. Raw targets, log1p and the selected standard transform are within 2% of CRC.
  • IMDB-WIKI. Raw targets are slightly best by mean RMSE, and all transforms except the uniform quantile transform and equal-mass are within 1.2% of CRC.

Splits are random over rows, so these results do not address unseen identities. These experiments extend the evaluation to regression on frozen text and image representations; they do not evaluate end-to-end training or fine-tuning of the representation. The paper continues to describe the study as mainly tabular (Section 8).

We thank the reviewer again for these helpful questions.

Reviewer_oyevOpenReview ↗

Response to Rebuttal

Thank you for your rebuttal, which addressed most of my concerns. I will maintain my score.

AuthorsOpenReview ↗

Thank You for Your Feedback

We are pleased that our response has addressed most of the reviewer’s concerns, and we sincerely appreciate their continued positive evaluation of our work. The reviewer’s constructive feedback prompted us to evaluate additional correction methods and extend our experiments to regression on frozen text and image representations.

Reviewer_2dWzOpenReview ↗
Rating
2(+1)
Confidence
2
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

The paper investigates how target transformations affect regression performance, focusing on the trade-off between resolution and retransformation bias. The authors develop a theoretical framework to explain why the optimal transformation depends on the data distribution and learning procedure, supported by controlled experiments and evaluations on 41 tabular datasets with different learning algorithms. The paper also examines several transformation strategies and criteria for selecting them. Although several theoretical and empirical findings are presented, I found it difficult to identify a clear overarching message regarding what new understanding or practical guidance comes from these results. In particular, past work already evaluates such transformation and gives practical guidance for researchers. So, in the end, I struggled to see any major contribution from this work. More details in the weakness section.

Critical strengths

(+) In my opinion, the topic addressed by the paper is quite relevant. The impact of data transformations on model performance is often overlooked in machine learning research, despite being an important aspect on model performance. A better understanding of when and why these transformations help could have broad implications for the field.

(+) The experimental evaluation covers a substantial collection of 41 tabular regression datasets from OpenML-CTR23 as well as other benchmarks. So I believe a good diversity of datasets with different target distributions and characteristics was used to draw the main conclusions/studying the impact of target transformations. I also like that the authors start with a more controlled environment to validate some parts/assumption before studying real datasets. Also, with some extra/more careful analyses like different target quantiles presented in the appendix (although I do believe such analysis should be extended to consider a bigger range of datasets).

Critical weaknesses
  • ) I had considerable difficulty reading and following several parts of the manuscript due to how condensed the text and overall presentation is. In particular, the related work often reads as a collection of references with little development of how they connect to each other or to the problem being investigated. The first subsection (Section 2) is a good example, where different works on target transformations, forecasting, and preprocessing selection are introduced within just a few lines. I found it difficult to understand which approaches actually address transformation selection, which investigate different aspects of the problem and, more importantly, the limitations remain based on past literature. In addition, I also miss a more critical analysis of the literature that clearly positions the paper and motivates its novel contributions. At the end I was not conviced this paper has significant novel insights in the field compared to past analysis /works like (Nuyts and Davis (2025)). Another point of confusion is that it also discusses methods in forecasting in this very short and condensed text, which can give the impression the work will also touch on such types of applications, but that is apparently out of scope. Similar problems appear in the experimental sections, where some findings receive very little explanation or discussion, sometimes being summarized in a single sentence referring to the appendix (e.g., Section 6, lines 394–395).

  • I also have concerns about how clearly the paper establishes the significance of its contributions relative to existing work. Previous studies, such as Nuyts and Davis (2025), have already investigated when target transformations are beneficial and how their effects depend on datasets and learning algorithms. Moreover. using a validation set to choose the best target transformation is a standard procedure in pretty much any ML field. So, after reading it, I am not convinced there are any substantial new findings that justify an ICLR publication. I would like the authors to properly address these in the rebuttal period.

  • (Major) Although the paper presents extensive theoretical and empirical analyses, I found that the different experiments and findings are not always connected through a clear narrative. The manuscript often moves between several observations without sufficiently developing their implications or explaining how they contribute to answering the main research problem. For instance, the abstract highlights the poor performance of uniform quantile transformations with neural networks, while the experiments later reveal that the magnitude of this effect depends considerably on the learning algorithm. This is an interesting finding that deserves more discussion as one should take that into account too. I think establishing a clearer set of research questions or central claims from the introduction and/or the beginning of the experiments section would help organize the analyses and distinguish the main contributions from supporting results. I also miss a simple illustrative example or figure showing how the same transformation can help in one setting and hurt in another, making the central phenomenon easier to understand at the start of the manuscript.

  • (Major) I also have some concerns regarding the motivation and practical interpretation of the leverage statistic (L). The thresholds used to categorize datasets (L<1.3, (1.3>L<2, L>2) also seem somewhat arbitrary, without a clear analysis of their sensitivity or generalization. Moreover, there are some counterexamples that raise questions about its interpretation. For instance, WaveEnergy has (L=1), yet CRC seems to perform worse than raw targets, while BrazilHouses has a high (L) but CRC still performs poorly compared to standard transformations. So are these values really well justified? Moreover, the same cutoff of 1.3 is subsequently used after concluding the experiments. I found this somewhat problematic without an analysis of how sensitive the conclusions are to these choices.

  • The impact of preprocessing is often dependent on both the dataset and the learning algorithm (which are also shown by the results here demonstrating substantial differences in the magnitude of the effects). Based on the results, it seems the uniform quantile transformation performs considerably worse relative to CRC for neural networks than for XGBoost. I would appreciate a more developed discussion of why these differences occur and what they imply for the practical recommendations. In particular, it remains unclear to me which conclusions can be expected to generalize across learning algorithms and which are specific to the models and training protocols considered.

  • I think the connection to imbalanced regression could provide additional insights into the practical implications of the proposed analysis. Previous work, such as Ribeiro and Moniz (2020) and Yang et al. (2021), has shown the importance of evaluating regression performance beyond aggregate metrics, particularly when some target regions are underrepresented. Since the proposed framework explicitly studies how transformations redistribute resolution across the target range, I would have appreciated an evaluation using metrics from the imbalanced regression literature, such as SERA, or an analysis of performance across target regions with different frequencies. Appendix N already provides some initial evidence of this behavior, but with very few datasets and without considering appropriate metrics for imbalanced regression.

Questions for authors

Overall, my main problem with the paper was with the presentation, lack of proper positioning/critical analysis of the existing literature, and a clear message with relevant insights for the research community as pointed out in the weakness section. Here are some questions regarding the weaknesses. I would like these to be promptly answered during the rebuttal period.

  1. How were the thresholds for (L) determined, and how sensitive are the conclusions to these choices? In particular, how should we interpret the practical recommendation at (L=1.3), considering that Appendix L reports substantial improvements from transformation selection even for some low-(L) datasets?

2)Given the experiments showing results may differ fro different learning algorithms, which findings can be expected to generalize across learning algorithms? The results show considerable differences in the magnitude of transformation effects between neural networks and tree-based models. What explains these difference? More importantly, how should they influence the practical recommendations?

  1. What does the proposed theoretical framework and research paper contribution allow us to understand about target transformations that was not already established empirically by past works?

  2. Given that some theoretical predictions do not match the observed optima in Section 5, which conclusions can reliably be drawn from the framework beyond the idealized setting?

5 comments
AuthorsOpenReview ↗

Response to Reviewer 2dWz (1/3)

We sincerely thank the reviewer for the detailed feedback, thoughtful questions, and valuable suggestions to improve our work. To address the reviewer's concerns, we:

  • added research questions and an illustrative figure to frame the introduction;
  • analyzed the sensitivity of LL and corrected our own recommendation at L=1.3L=1.3;
  • analyzed how the effects depend on the learner;
  • evaluated the transforms with imbalanced-regression metrics.

Because these points concern how the results should be read, we also added a paragraph on them to the Discussion (Section 8, "Empirical guidance, learners and metrics"). All changes in the revised PDF are marked in blue. We provide our responses below.


W2 / Q3. Contributions beyond prior empirical guidance and standard validation

A1. We agree that validation-based selection is standard and do not claim it as a contribution. The revised abstract reports its performance only as an empirical comparison with the target-only selectors evaluated. What the paper adds:

  1. A risk decomposition. For an idealized learner, Eq. (1) separates retransformation bias from resolution, the learning error magnified by the inverse. The quantization optimum and the econometric bias are its two limits, and their balance depends on predictability. This yields qualitative predictions that we examine by holding the target marginal fixed while varying predictability (Figure 1b, Figure 3b, Section 5).
  2. A limit on target-only rules. Theorem 3 shows that, under the idealized learner, no fixed rule using only the target marginal is asymptotically optimal for every joint law. The choice must therefore depend on the conditional law. Consistently, the target-only selectors in Table 3 do not reach validation: argmin KK exceeds validation's regret by 0.042 [0.021, 0.070].
  3. Measurements of the learner. Standardization changes how the learning error scales with the transform (Section 5, Table 1). A synthetic diagnostic shows how the location of the learner's error contributes to the gap from the idealization (Appendix T).
  4. A practical use. When fits are limited, the variance-normalized objective KK ranks candidates for a validated shortlist and gives a modest gain for the MLP (Section 7, Appendix Y).

In addition, we reran the ridge protocol of Nuyts & Davis from their released code. Their evaluation code divides by ∑(yˉ−y^i)2\sum(\bar y-\hat y_i)^2 instead of the ∑(yi−yˉ)2\sum(y_i-\bar y)^2 of their Eq. 8. Within this replication and for the same predictions, changing only the formula turns the apparent gain of the Gaussian quantile transform (−13.3% RSE) into a loss (+6.0%) (Section 6, Appendix R). We restrict this finding to that replication.

W1. Positioning and presentation (condensed related work, forecasting references, little explanation of some findings)

A2. To address the reviewer's concern regarding positioning, we rewrote the first paragraph of Section 2 to separate the lines of work and to state how each relates to our problem:

  • Nuyts & Davis (2025) give rules of thumb for subjective targets, contextual dependencies and unsuitable target distributions, and compare transforms empirically for the last case. We study monotone transforms of a given target. We add an original-scale risk expression, and our impossibility result concerns only rules that use the target marginal.
  • Tabular benchmarks treat the transform as one binary hyperparameter and do not ask which transform to use.
  • Forecasting work is cited only for two results we use: the bias of back-transformed Box–Cox forecasts and original-scale selection criteria. The text now states that forecasting itself is out of scope.
  • Imbalanced regression: a new paragraph (see A6).

The two findings at lines 394–395 of the original submission are now explained as robustness checks. The low-leverage comparisons persist with the full candidate pool, and the uniform-quantile penalty persists under the three training protocols tested (Section 6):

  • Fitting all 60 family members on the 22 low-leverage datasets leaves the median family/Std ratio at 0.995 (Appendix L).
  • Under three alternative training protocols, the uniform quantile transform stays worse than CRC on all 19 high- and mid-leverage datasets (Appendix M, Table 22).
AuthorsOpenReview ↗

Response to Reviewer 2dWz (2/3)

W3. Narrative, research questions and an illustrative example

A3. Following the reviewer's suggestion, we added research questions and an illustrative example to the introduction. We also narrowed the uniform quantile statement in the abstract:

  • Research questions. The introduction now ends with four research questions (labeled Q1–Q4 in the paper), each with a one-line answer and its section:

    • What determines the optimal transform? (Section 3)
    • Can it be read off the target distribution? (Section 4)
    • Does the idealization describe trained learners? (Section 5)
    • When does the choice matter, and how should it be made? (Sections 6–7)

    Theory answers the first two and experiments the last two; the appendices only support these answers. Section 5 opens by naming its question. Sections 6–7 open by naming theirs and by separating the main results (resolution contrasts against LL, the uniform-quantile penalty, the selector comparison) from robustness checks.

  • Illustrative figure. New Figure 1, at the start of the paper:

    • (a) On 41 datasets, log1p lowers an MLP's test RMSE relative to raw targets on 14 datasets and raises it on 27.
    • (b) With the target distribution held fixed, the best member of a compression sweep changes with predictability ρ\rho: the most compressing member is near-best at ρ=0.5\rho=0.5 and worst at ρ=0.99\rho=0.99.
  • Uniform quantile result. The abstract states this result only for neural networks, where it holds on every dataset. Section 6 and new Appendix U discuss the learner dependence (see A5).

W4 / Q1. Interpretation and sensitivity of LL (thresholds, WaveEnergy and BrazilHouses, the 1.3 recommendation)

A4. Meaning and cutoffs. L=J[CRC-log]/J[CRC]L=J[\text{CRC-log}]/J[\text{CRC}] characterizes the relative resolution cost of CRC-log versus CRC under A1, computed from the training targets (Section 3). We chose the cutoffs before running the benchmark, from the distribution of LL across datasets: 2 to mark clearly wide-range targets, and 1.3 to limit computation on narrow-range targets (Appendix H). Neither is a theoretically derived boundary. New Appendix S tests the sensitivity:

  • Across all 16 tested pairs of (low, high) cutoffs, {1.1, 1.2, 1.3, 1.5} × {1.75, 2, 2.5, 3}, the high-group median of CRC-log/CRC stays within 1.09–1.18 and that of QU/CRC within 2.87–2.92 (Table 28).
  • Treating LL continuously, the Spearman correlation of CRC-log/CRC with LL is 0.76 (95% CI [0.55, 0.90]), and 0.75–0.81 when any one dataset is left out (Table 29).

The recommendation at 1.3. We agree with the reviewer and have removed this recommendation. Our earlier sentence "below about 1.3, the identity did as well as any transform" contradicted Appendix L.

  • For the MLP, the mean test RMSE ratio over 20 seeds (selected/raw) was below 0.98 on 9 of the 22 low-leverage datasets and above 1.02 on none (Table 30).
  • Across datasets we did not detect an association between LL and the gain from selection (ρ=0.09\rho=0.09, 95% CI [−0.26, 0.44]).
  • The 1.3 gate is now described as a computational shortcut only (Appendix H).
  • The Practical reading (Section 7) recommends including raw targets and selecting by validation RMSE under the same standardization. The Discussion (Section 8) states that LL describes a resolution contrast rather than the overall benefit of selection.

WaveEnergy and BrazilHouses. LL does not say that CRC is the best transform. In both cases the selected standard transform beats CRC (Std/CRC RMSE ratio, MLP, mean over 20 seeds; Table 14):

  • WaveEnergy: Std/CRC 0.23; raw-target R2=1.000R^2=1.000.
  • BrazilHouses: Std/CRC 0.85; ridge R2=0.95R^2=0.95.

Both targets are nearly deterministic, so the bias term, which vanishes as v→0v\to0, cannot explain these losses. One possible explanation is the A2 resolution term KK, which penalizes CRC's compression of the sparse extremes (Section 6, Appendix N). We present this as a conjecture.

AuthorsOpenReview ↗

Response to Reviewer 2dWz (3/3)

W5 / Q2. Learner dependence and practical implications

A5. To address the reviewer's concern regarding learner dependence, new Appendix U reports all comparisons for the MLP, ResNet, XGBoost and HGB (Table 33). Below we separate the findings shared across these learners from the learner-specific ones and then state the practical implications.

  • Shared across the four learners tested. The uniform quantile transform is worse than CRC on 41/41/41/38 datasets (MLP/ResNet/HGB/XGBoost), and on the two new frozen-embedding tasks (Appendix X). CRC-log/CRC grows with LL.
  • Learner-specific. The high-leverage median QU/CRC is 2.87 (MLP), 2.00 (ResNet), 1.31 (XGBoost) and 1.09 (HGB). Small effects such as Std/CRC vary in sign by group and learner.
  • What we measured. The inverse of the uniform quantile map is extremely steep in the sparse upper tail, and the networks sometimes predict bulk targets at the top of the quantile range. For example, on InsuranceSeverity the MLP predicts 774,000 for a target of 2,400.
    • These predictions are about 0.05% of rows but carry a median 11% of the uniform-quantile MLP's own squared error at high leverage (Table 34).
    • They account for a median 18% of the MLP's pooled QU–CRC squared-error gap (Table 35).
    • On the 11 high-leverage datasets, capping lowered the MLP uniform-quantile transform's own RMSE by a median 31%, while CRC's median change was negligible.
    • The boosted trees produced fewer such predictions; we do not claim a mechanism for this difference.
  • Implications. Include raw targets, use the same standardization for every candidate, compare on validation, and treat the uniform quantile transform cautiously. Effect sizes and some transform rankings depend on the learner, so the comparisons should be made for the learner and evaluation metric actually used. The Discussion (Section 8) relates the neural learners' larger penalties to their predictions near the steep end of the inverse map.

W6. Imbalanced-regression evaluation

A6. Following the reviewer's suggestion, we added a paragraph on imbalanced regression to Section 2 and evaluated the transforms with imbalanced-regression metrics. New Appendix V evaluates all 41 datasets with all four learners using SERA with a density-based relevance (Steininger et al. 2021), many-, medium- and few-shot RMSE, and RMSE on density terciles. Band-specific metrics exclude datasets where the corresponding band is absent. Results (Table 36):

  • Raw targets have a lower SERA than CRC (0.956–0.967) and a lower few-shot error (0.973–0.987), but a higher error in dense regions. CRC shows the opposite pattern.
  • The uniform quantile transform has a higher error than CRC on every reported aggregate metric.
  • The RMSE-best transform is also SERA-best on only 44–63% of datasets, so the preferred transform also depends on the metric.

Q4. Conclusions that hold beyond the idealized setting

A7. To clarify which conclusions extend beyond the idealized setting, we separate those supported by experiments with trained learners from those that remain specific to the idealized learner.

Supported with trained learners:

  • At a fixed marginal, the optimum moves with predictability (Figure 3b).
  • Removing the exact retransformation offset left the optimum at least as compressed in 22 of 24 controlled cells, with two HGB exceptions (Section 5).
  • With standardized targets, the learning error varies less across transforms in normalized units (Table 1), and KK ranks transforms better than JJ for the MLP on 34 of 41 datasets.
  • The uniform quantile transform underperforms CRC across learners.

Not confirmed: the closed-form optimum λ=3/4\lambda=3/4 for trained learners.

New Appendix T is a diagnostic in one noiseless synthetic setting. It uses test errors measured after training and the known conditional mean, and reassigns each fitted model's own errors to random test points.

  • For c∈c\in{1/4,1/3,1/2}, the risk falls 2.2–3.7×, because both learners place larger errors where h−1h^{-1} is steep, and the optimum moves toward c∈c\in{1/3,1/2}.
  • With the measured error profile, the term of Proposition 1 is 0.94–1.02 times the MLP's risk for the tested values c∈c\in{0,1/4,1/3,1/2}.
  • It is 1.24 times HGB's risk at c=1/2c=1/2, it fails for the logarithm, and the diagnostic is not a direct test of A1 or A2.

The framework provides a mechanistic account of the qualitative trends observed in these controlled experiments, while its ability to predict a trained learner's optimum remains limited.

We thank the reviewer again. Addressing these comments made the paper clearer, and the points on the relation to empirical guidance, the interpretation of LL, learner dependence and evaluation metrics are now also discussed in the Discussion (Section 8).

Comments on the paper

2 comments
AuthorsOpenReview ↗

General Response

In this paper, we study which target transformation gives the smallest squared error on the original scale when a regression model is trained on transformed targets and its predictions are mapped back. Our main contributions are:

  1. An original-scale risk decomposition. For an idealized learner, we separate retransformation bias from resolution, the learning error magnified by the inverse transform. The quantization optimum and the econometric retransformation bias are its two limits, and their balance depends on predictability.
  2. A limit on target-only rules. We prove that, under these assumptions, no fixed rule based on the target distribution alone is asymptotically optimal for every joint law. The choice must therefore depend on the conditional law.
  3. Measurements of trained learners. In controlled experiments with the target distribution fixed, the preferred transform moves with predictability, and measurements of the learning error show where trained learners follow and depart from the idealization.
  4. Empirical guidance on 41 tabular datasets with four learners. The uniform quantile transform underperforms cube-root companding by a learner-dependent amount. The variance-normalized objective KK can shortlist candidates for validation when fits are limited.

We express our sincere gratitude to all reviewers for dedicating their time and effort to evaluating our manuscript thoroughly. The constructive feedback has greatly improved the clarity and scope of the paper. To address the reviewers' concerns, we provide point-by-point responses to all questions and concerns raised by each reviewer. Below is a summary of the key updates to the paper:

  • Research questions and their answers at the end of the Introduction (Section 1) — 2dWz
  • An illustrative example of the same transform helping in one setting and hurting in another (Figure 1) — 2dWz
  • Rewritten Related Work, including the positioning relative to Nuyts & Davis (2025) and a new paragraph on imbalanced regression (Section 2) — 2dWz
  • Revised practical reading: the earlier L=1.3L=1.3 recommendation is removed (Section 7) — 2dWz, y8mr
  • New discussion of empirical guidance, learner dependence and evaluation metrics (Section 8) — 2dWz
  • Sensitivity of the leverage-based comparisons to the cutoffs, and LL treated continuously (Appendix S) — 2dWz
  • Where the learning error falls: a controlled synthetic diagnostic of the gap between the idealized and trained learners (Appendix T) — y8mr, 2dWz
  • Learner dependence of the uniform-quantile penalty, with a decomposition of the QU–CRC gap and a capping analysis (Appendix U) — 2dWz
  • Imbalanced-regression metrics: SERA and target-region errors on all 41 datasets and four learners (Appendix V) — 2dWz
  • Global smearing, binned smearing and a learned residual correction applied to every transform, with raw targets as a control (Appendix W) — oyev
  • Regression on frozen pretrained text and image representations (Appendix X) — oyev
  • Candidate shortlists under a fixed number of model fits (Appendix Y) — y8mr, 2dWz

For the revised and updated parts of the manuscript, we have marked the changes in blue. We hope that the responses provided below address all the reviewers’ concerns, and please let us know if the raised concerns are addressed. We are happy to answer further questions and to provide additional clarifications.

AuthorsOpenReview ↗

Follow-up on Our Responses to Reviewers

Dear Reviewers and Area Chair,

As the discussion period approaches its conclusion, we would like to provide a brief follow-up. We sincerely thank all the reviewers and the Area Chair for their time and effort in carefully evaluating our manuscript. Your constructive feedback has greatly helped us strengthen our work and improve its clarity. To address the reviewers' concerns, we have provided point-by-point responses to all the questions and concerns raised in the initial reviews.

We would be grateful if you could let us know whether our responses have adequately addressed your concerns. We would be happy to answer any further questions or provide additional clarifications.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.