- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper studies the training of neural networks with SQR, an extension of ordinary QR that takes the quantile level as an additional model input. The authors attribute a particular prediction interval tail coverage collapse phenomenon in SQR to (i) affine dependence on the quantile level at model initialization and (ii) gradient updates that are biased toward high-density regions of the data. To correct this behavior, they suggest that perturbing closes the calibration gap between QR and SQR at extreme quantiles. The paper tests this explanation empirically by proposing several non-linear -scaling methods and benchmarking them against existing IQN and AQR approaches. Experiments across seven datasets show that the proposed non-linear scalings improve both average calibration error and Winkler score relative to vanilla SQR and match QR.
Critical strengths
Clear empirical motivation and supporting ablations. The authors provide empirical examples showing that the quantile function learned by SQR remains approximately affine in throughout training, which motivates their proposed non-linear correction. This finding establishes connections with prior approaches, notably IQN and AQR, and provides an explanation for why these models perform well in practice. The experiments also rule out several alternative explanations for the tail collapse by testing model capacity, optimization regimes, activation functions, and calibration losses.
Critical weaknesses
- Insufficient theoretical grounding. Much of the motivation behind the collapsing tails in Section 2 is informal. For example, the “neural network activations” paragraph only considers a single hidden-layer MLP and heuristically looks at the ratio of the variance contributions of and . The “loss derivative” and “impact of data-sample density on learning” paragraphs argue that the accumulated non-linearity is biased toward areas of high data density, but this claim is not rigorously supported. Instead, I would like to see stronger theoretical motivation behind the affine quantile function finding, through some combination of training dynamics analysis, gradient updates in relation to data density, or a small-scale theoretical case study that would better ground the empirical motivation.
- Incomplete or incremental novelty. The main novelty of the paper appears to be the explanation behind the tail coverage collapse phenomenon in SQR. While this is an interesting direction, the motivation is mostly informal (as noted previously) and lacks theoretical grounding. As a result, it feels somewhat underdeveloped. The remaining contributions, such as the proposed -scaling methods, benchmarking the coverage of existing SQR variants against QR, and analysis of post-hoc CQR, are mostly incremental.
- Tension with AQR results. The authors argue that non-linear scaling of closes the coverage gap between SQR and QR. However, using affine embeddings like in AQR performs comparably. I understand that the authors acknowledge this and it may be partly explained by the magnitude analysis of . Nonetheless, this seems to weaken the core message and suggests that some combination of the magnitude and non-linearity of the effect of may be the key factor.
- Evaluation scope. The evaluation benchmark uses seven datasets, but all the datasets are tabular and, as the authors note, do not have “highly non-normal distributions” except for two. I believe that the evaluation scope is thus somewhat narrow. In particular, my main concern is whether the proposed scalings are suitable for datasets where the conditional CDFs are not S-shaped or non-Gaussian, in which case the proposed non-linearities may not suit the data.
Questions for authors
n/a
3 comments
While I think the manuscript has some potential, even with this rebuttal, I am not yet persuaded that the contribution is at the level I would expect for a publication in ICLR. This assessment seems consistent with the other reviews. I am maintaining an open mind for the reviewer discussion.
Thank you for your comments! The experiments are done, and changes to the paper have been finalized. In this post, we will describe changes related to your previous comments. We also compiled more specific scope- and communication-related changes in the comment that is not a reply to any reviewer (for us, that is at the very top of the comment list).
We can add some more points to our initial response now:
- There was not enough time for a more thorough theoretical analysis. However, we did some more empirical evaluations that support our claims: We take a look at the properties of the curvature displayed by each approach, what differences there are in the pre-ReLU sign switching, and how gradient updates are coupled in tail quantiles vs. quantiles in the middle of the distribution. (These are at the end of the Results section in Figure 4 and Appendix J.1)
- Regarding novelty, we updated our phrasing in the Abstract and, especially, the Contributions section to communicate more precisely what novelty and contribution we offer to the community.
- We are more precise with our claims now. It is not only non-linearity in that can produce a non-linear quantile coverage distribution. However, we also observed that AQR did not perform as well as IQN or the SQR$_e/s$ approaches in non-Gaussian distributions. AQR appears to be just nonlinear enough to cover well in the original datasets. AQR also has improved performance through simply embedding with logit, like in SQR$_s$. Appendix J.1, K, and the Discussion in Appendix L).
- As promised, we added experiments with synthetic heterogeneous, skewed, heavy-tailed, and bimodal data. IQN, AQR, and the SQR$_e/s$ stay mostly close to QR performance. In our bimodal condition, all approaches were worse than QR, and AQR was worse than all others in the heavy-tailed condition. (These results were added in Appendix J.1) In conclusion, the new approaches seem to transfer to more data shapes apart from bimodal.
Thank you for all of your comments. We understand and share a number of your concerns and will strive to address them in an updated version of our manuscript.
Theoretical grounding: Indeed, our hypothesis could benefit from a more rigorous theoretical examination. The hypothesis on what causes the gap is, however, not disproven by our empirical analysis, suggesting the validity of the idea. We argue that the manuscript provides a substantial contribution nonetheless (see our explanation in Novelty). We do, however, agree that your suggestions on more analyses of training dynamics would add more value to the manuscript.
Novelty: We would like to disagree that the main novelty of this paper is the explanation behind the collapse and offer our perspective: Given our literature review, the coverage gap between SQR and QR has never received any formal attention. Further, to the best of our knowledge, there are no reports on the extent of the effect. According to their respective papers, the inventors of IQN or AQR were not aware of their methodology being more effective than simple SQR: neither were IQN and AQR empirically compared against SQR, nor did either paper include an empirical coverage analysis in their work.
As far as we understand, we thus 1.) are the first to observe the prevalence and significance of the effect and investigate it systematically, 2.) offer a hypothesis for why it happens, and 3.) identify direct solutions to it of our own but also from the literature, and 4.) document other relevant solutions that did not work. In addition, our solutions need fewer parameters than AQR or IQN. We hope this is a clearer explanation and will go through relevant sections in the paper again to evaluate if we can communicate this more clearly there as well.
AQR: We agree that AQR is not entirely explained by our hypothesis. With respect to AQR, this paper aims to offer a hypothesis on why there is a gap and how to fix it. While a non-linear transformation of tau is sufficient, it can appear not to be necessary. Thus, we assume that there may exist many more solutions to the problem. In the next version of the manuscript, we plan to include an ablation on AQR, as it can be split into shift, scale, and sampler components.
Evaluation scope: We agree that our current analysis can not conclude whether any of the analysed approaches can handle non-Gaussian or S-shaped data. Reviewer gsSj had a similar concern and suggested an additional analysis with synthetic data in heteroscedastic, skewed, and multi-modal conditions, which we will implement and report on. This will also include IQN and AQR and should provide further insights into how these approaches work.
We will post another comment explaining our changes and added experiments as soon as they are done.