- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
SOGN aims to address the negative impact of unreliable samples on regression training by dividing the target into ordered intervals and learning the probability of each sample belonging to each interval.
The proposed method estimates sample reliability based on the concentration and coherence of the learned ordinal probability distribution and uses these scores for stop gradient based weighted regression to reduce the influence of unreliable samples.
During inference, SOGN uses these reliability scores to select samples for prediction aiming to improve selective regression performance.
Critical strengths
- The problem addressed in this paper is clearly defined, and the proposed method is presented in a clear and accessible manner, making it easy to follow and understand.
Critical weaknesses
-
The paper should provide stronger theoretical or empirical evidence that the ordinal head, trained through a shared encoder, accurately estimates the probability of the target belonging to each interval. In particular, further analysis of probability calibration and estimation errors is needed, as they directly affect the reliability assessment and sample selection performance.
-
Although the types of scores used for each baseline are specified, the manuscript lacks sufficient details on how these scores are computed and how they are combined with the corresponding predictions to evaluate () at different coverage levels.
-
Regarding the presentation, The quality of Figure 1, and Figure 2 is poor.
Questions for authors
-
How can the authors ensure that the ordinal head, which shares an encoder with the regression model, accurately estimates the probability of the target belonging to each interval?
-
Is there a particular reason for the frequent use of em dashes throughout the manuscript? Could the authors consider revising some of these sentences to improve readability?
-
Regarding the results in Table 2, shouldn't the full SOGN model outperform the LS-only and HC-only variants, even at full coverage ((c=1.0))? Could the authors explain why this is not the case?
-
Could the authors consider including NGBoost in the experimental comparisons? I believe NGBoost is directly relevant to the problem addressed in this paper and would serve as a valuable baseline.
[1] Duan, Tony, et al. "Ngboost: Natural gradient boosting for probabilistic prediction." International conference on machine learning. PMLR, 2020.
2 comments
I sincerely thank you to authors for their additional experiments and response, especially for NGBoost as additional benchmark. But, given that substantial portions of the manuscript have changed between the original submission and the revised version, I remain cautious about placing full confidence in this paper. Therefore, I have decided to maintain my original score. Thank you.
Thank you for your review. I will respond to the issues you raised below.
[W1 and Q1] Although the ordinal head shares an encoder with the regression head, it receives direct supervision from target-interval labels. To examine the predicted interval probabilities, we added per-bin calibration analyses on five datasets to the revised manuscript. The six-bin macro-averaged classwise ECE, overall NLL, and Brier score all decreased in 45/45 runs; among the 30 dataset--bin results, 28 improved. The calibration error for the contiguous three-bin event used to assess ordered adjacency decreased on average across the five datasets and improved in 38 of 45 runs (Appendix B.2, Table 10). We also directly examined the relationship between the gating score and prediction error: on all five datasets, the mean RMSE of the top score decile is lower than that of the bottom score decile (Appendix B.2, Table 11).
[W2] In Appendix B.2 of the revised manuscript, we list each baseline's point prediction, selection score, and how that score is computed, and explain that each method selects samples at the specified coverage using its own score and is evaluated on the samples selected by that method .
[W3 and Q2] We are very sorry that the paper's writing was not sufficiently clear and that the figure quality caused difficulty. We carefully reviewed the paper, revised it as a whole according to your suggestions, and redrew the figures.
[Q3]
(1) At c=1.0, all test samples are evaluated, and inference-time ranking no longer affects the result. However, the full model, LS-only, and HC-only use different sample weights during training and therefore produce different regressors. The full model aims to improve selective prediction performance; it is not required to outperform single-criterion variants on every dataset at full coverage. SOGN's training objective is to obtain effective and stable selective rankings across different coverage levels, rather than to optimize the full-coverage result alone.
(2) We believe the poor full-coverage ablation result may be due to the SoftMin fusion weight in the full model being allowed to be negative, whereas the single-criterion gating weights are positive; this training difference may affect the full model's full-coverage performance. We also initially tested an alternative design that makes the SoftMin weights strictly nonnegative and places greater emphasis on training on all samples. However, the mean results over five seeds showed that this design had lower R2 than the current design at all ten coverage levels, especially at 10% coverage, where the result was −0.547 versus 0.409. Detailed values are below. Therefore, we retain the current SoftMin form to obtain better selective R2 performance across coverage levels, consistent with the purpose of selective prediction methods.
(3) In addition, we noticed that the previous ablation experiments were limited in scale, so we expanded them. The results are shown in Table 2.
[Q4] Thank you for suggesting a directly relevant baseline. We have added a comparison with NGBoost to the main experiments, and have also substantially expanded the scale of the main experiments (Table 1). This method has been helpful for comparison in our paper.