- Rating
- 3
- Confidence
- 3(+1)
- Informativeness
- 3(+1)
- Presentation
- 3
- Soundness
- 3(+1)
Read review
Paper summary
This paper studies interpolation bias induced by log transformations in deep regression. The authors argue that when the underlying target function is locally additive in the original space, while DNNs approximately interpolate linearly in log space, the inverse-transformed predictions may exhibit systematic underestimation. The authors then propose Transient Variance Guidance (TVG), which uses the predictive variance learned through Gaussian NLL to construct auxiliary supervision in the original target space, with stop-gradient controlling gradient propagation. Various experiments validated the proposed method.
Critical strengths
-
The paper analyzes prediction bias induced by log transformations from the perspective of interpolation geometry, providing an interesting perspective on this problem. The design of TVG is well motivated and tailored to the proposed issue.
-
The empirical evaluation is relatively comprehensive. The proposed method demonstrates improvements across multiple real-world datasets and network architectures, while ablation studies support the effectiveness of its main design choices.
-
Appendix A explicitly investigates the applicability and limitations of the method. Through synthetic experiments with additive and multiplicative data-generating mechanisms, the authors demonstrate how TVG behaves under different target function structures, including cases where it may substantially underperform the baseline under strongly multiplicative relationships.
Critical weaknesses
My main concerns relate to the applicability of the proposed method and the conditions under which it is expected to be effective.
-
The proposed method mainly targets the mismatch between locally additive structures in the original target space and approximately linear interpolation in log space. To what extent is TVG limited to regression problems exhibiting this type of local structure? Could the underlying idea be extended to other commonly used nonlinear target transformations, such as the logit transformation?
-
The effectiveness of the method is primarily supported by empirical evidence, and the synthetic experiments also show that TVG can substantially underperform the baseline in certain settings. Are there simple empirical criteria or diagnostic procedures that could help practitioners determine in advance whether TVG is appropriate for a given task? Clearer guidance on its applicability would improve the practical usefulness of the method.
Questions for authors
Questions For Authors:
-
In the existing 1D synthetic experiments, could the authors plot the learned predictions for both the baseline and TVG, together with the ground truth and the prediction obtained by linearly interpolating between training points? This would help determine whether the baseline exhibits the interpolation bias described in the paper and whether TVG reduces it.
-
Could the authors examine how the learned variance relates to the squared fitting error during training? The paper motivates TVG using the optimal variance under Gaussian NLL, but it remains unclear whether the learned tracks this error in practice.
-
The synthetic experiments in Appendix A use only five training points. Could the authors vary the number of training points and examine how the performance gap between TVG and the baseline changes under and ? This would help clarify whether the observed effects depend on the sparsity of the training data.
3 comments
I thank the authors for their detailed response and for addressing all of my concerns. I will maintain my original positive assessment of this work and have accordingly increased my individual scores and confidence rating.
We thank the reviewer for carefully reconsidering our work and for confirming that the response and revisions address the concerns.
Author Response to Reviewer WjZt
We thank the reviewer for recognizing the method design, experimental results, and applicability analysis. Following the reviewer’s suggestions, we added the following experiments and discussion.
1. Actual prediction curves and bias reduction
Our work concerns the potential mismatch between the inductive bias induced by target transformation and the original-space target structure. Equations (1)–(2) use two points to illustrate how this mismatch manifests as trajectory deviation; the difference between the arithmetic and inverse-transformed geometric paths is not an assumption about the full generalization behavior of an actual network. For networks jointly constrained by multiple training points, the deviation depends on the architecture and optimization and must be examined through actual predictions.
We therefore added the ground-truth, Log-MSE, and TVG prediction curves for the additive task (b=10,50,100) alongside 500-seed signed-error statistics (Appendix A.3). The learned predictions exhibit both local overestimation and underestimation, but the baseline shows an overall negative bias that TVG reduces while lowering MAE: for b=10, 50, and 100, mean signed errors improve from −0.0272, −0.2374, and −0.7107 for Log-MSE to −0.0122, −0.1553, and −0.4703 for TVG. For visual clarity, deviations from the ground truth are magnified 10× and labeled in the prediction figure, while the signed-error plot and all metrics remain on their original scales. Thus, under multi-point training, TVG effectively alleviates the overall negative bias and reduces original-space error.
2. Relationship between predicted variance and fitting lag
We recorded the evolution of the predicted variance σt2(x) and squared fitting lag (E[Z∣x]−μt(x))2 under the joint loss in Equation (6) over 500 random seeds. In early training (Epochs 50 and 100), they show clear positive correlations of (0.8243,0.5584) for b=10, (0.9119,0.9656) for b=50, and (0.9090,0.9929) for b=100. By Epoch 2000, both means decrease substantially (Appendix A.4), showing that predicted variance retains information associated with fitting lag under joint training.
3. Effect of the number of training points
We varied the number of training points from 5 to 300 in both additive and multiplicative settings (Appendix A.5). Representative endpoint results (mean MAE ± std) are:
In the additive setting, TVG achieves lower average MAE and RMSE across all tested training-set sizes, showing its benefit is not restricted to the sparse five-point setting. In the multiplicative setting, TVG’s average disadvantage decreases substantially as training points increase—from −18.48 to −0.32—but remains present throughout the tested range, clarifying the applicability boundary.
4. Applicability criteria and other target transformations
TVG targets settings where the original-space response is locally additive, while logarithmic preprocessing is introduced to compress dynamic range or improve optimization. In practice, domain knowledge and mechanistic models provide prior evidence for applicability, while task-specific benefits should be confirmed on the validation set. The multiplicative experiments (Y=exp(a+bu) in Point 3 and Appendices A.2 and A.5) explicitly demonstrate the failure mode under predominantly multiplicative relationships. Thus, a locally additive original-space mechanism is the central criterion for applying TVG.
Conceptually, this perspective may be extended to other nonlinear target transformations such as the logit transformation. Because the present work focuses on logarithmic transformation, we leave extensions and evaluations under other nonlinear transformations to future work.