acceptodds
Under review as a conference paper at ICLR 2027

Selective Ordinal Gating Network: Structure-Gated Training via the External Reference Principle

Abstract

Existing methods for selective regression often estimate reliability from model predictions or learned representations. We propose an alternative approach that uses a learned ordinal structure of the target to identify samples with concentrated and coherent conditional target distributions. Our Selective Ordinal Gating Network (SOGN) combines an ordinal auxiliary head with a differentiable concentration–coherence gate and a stop-gradient weighted regression objective. We theoretically analyze how coherent ordinal probabilities can imply a narrow region in target space, and characterize the assumptions under which this implication holds. Across multiple benchmark datasets, SOGN consistently improves selective-regression performance at low coverage compared with uncertainty-based and rejection-based baselines. These results show that ordinal target structure can provide an effective signal for selective regression.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Selective Regression, Ordinal Gating, External Reference Principle, Anti-Contamination Training

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_RPpaOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

SOGN aims to address the negative impact of unreliable samples on regression training by dividing the target into ordered intervals and learning the probability of each sample belonging to each interval.

The proposed method estimates sample reliability based on the concentration and coherence of the learned ordinal probability distribution and uses these scores for stop gradient based weighted regression to reduce the influence of unreliable samples.

During inference, SOGN uses these reliability scores to select samples for prediction aiming to improve selective regression performance.

Critical strengths
  1. The problem addressed in this paper is clearly defined, and the proposed method is presented in a clear and accessible manner, making it easy to follow and understand.
Critical weaknesses
  1. The paper should provide stronger theoretical or empirical evidence that the ordinal head, trained through a shared encoder, accurately estimates the probability of the target belonging to each interval. In particular, further analysis of probability calibration and estimation errors is needed, as they directly affect the reliability assessment and sample selection performance.

  2. Although the types of scores used for each baseline are specified, the manuscript lacks sufficient details on how these scores are computed and how they are combined with the corresponding predictions to evaluate (R2R^2) at different coverage levels.

  3. Regarding the presentation, The quality of Figure 1, and Figure 2 is poor.

Questions for authors
  1. How can the authors ensure that the ordinal head, which shares an encoder with the regression model, accurately estimates the probability of the target belonging to each interval?

  2. Is there a particular reason for the frequent use of em dashes throughout the manuscript? Could the authors consider revising some of these sentences to improve readability?

  3. Regarding the results in Table 2, shouldn't the full SOGN model outperform the LS-only and HC-only variants, even at full coverage ((c=1.0))? Could the authors explain why this is not the case?

  4. Could the authors consider including NGBoost in the experimental comparisons? I believe NGBoost is directly relevant to the problem addressed in this paper and would serve as a valuable baseline.

[1] Duan, Tony, et al. "Ngboost: Natural gradient boosting for probabilistic prediction." International conference on machine learning. PMLR, 2020.

2 comments

Thank you for your review. I will respond to the issues you raised below.

[W1 and Q1] Although the ordinal head shares an encoder with the regression head, it receives direct supervision from target-interval labels. To examine the predicted interval probabilities, we added per-bin calibration analyses on five datasets to the revised manuscript. The six-bin macro-averaged classwise ECE, overall NLL, and Brier score all decreased in 45/45 runs; among the 30 dataset--bin results, 28 improved. The calibration error for the contiguous three-bin event used to assess ordered adjacency decreased on average across the five datasets and improved in 38 of 45 runs (Appendix B.2, Table 10). We also directly examined the relationship between the gating score and prediction error: on all five datasets, the mean RMSE of the top score decile is lower than that of the bottom score decile (Appendix B.2, Table 11).

[W2] In Appendix B.2 of the revised manuscript, we list each baseline's point prediction, selection score, and how that score is computed, and explain that each method selects samples at the specified coverage using its own score and is evaluated on the samples selected by that method .

[W3 and Q2] We are very sorry that the paper's writing was not sufficiently clear and that the figure quality caused difficulty. We carefully reviewed the paper, revised it as a whole according to your suggestions, and redrew the figures.

[Q3]

(1) At c=1.0c=1.0, all test samples are evaluated, and inference-time ranking no longer affects the result. However, the full model, LS-only, and HC-only use different sample weights during training and therefore produce different regressors. The full model aims to improve selective prediction performance; it is not required to outperform single-criterion variants on every dataset at full coverage. SOGN's training objective is to obtain effective and stable selective rankings across different coverage levels, rather than to optimize the full-coverage result alone.

(2) We believe the poor full-coverage ablation result may be due to the SoftMin fusion weight in the full model being allowed to be negative, whereas the single-criterion gating weights are positive; this training difference may affect the full model's full-coverage performance. We also initially tested an alternative design that makes the SoftMin weights strictly nonnegative and places greater emphasis on training on all samples. However, the mean results over five seeds showed that this design had lower R2R^2 than the current design at all ten coverage levels, especially at 10% coverage, where the result was −0.547-0.547 versus 0.4090.409. Detailed values are below. Therefore, we retain the current SoftMin form to obtain better selective R2R^2 performance across coverage levels, consistent with the purpose of selective prediction methods.

(3) In addition, we noticed that the previous ablation experiments were limited in scale, so we expanded them. The results are shown in Table 2.

CoverageStrictly nonnegative R2R^2Our method R2R^2
100%0.4280.498
90%0.4480.513
80%0.4920.532
70%0.4980.546
60%0.5240.554
50%0.5390.556
40%0.5350.556
30%0.5000.530
20%0.4770.519
10%-0.5470.409

[Q4] Thank you for suggesting a directly relevant baseline. We have added a comparison with NGBoost to the main experiments, and have also substantially expanded the scale of the main experiments (Table 1). This method has been helpful for comparison in our paper.

I sincerely thank you to authors for their additional experiments and response, especially for NGBoost as additional benchmark. But, given that substantial portions of the manuscript have changed between the original submission and the revised version, I remain cautious about placing full confidence in this paper. Therefore, I have decided to maintain my original score. Thank you.

Reviewer_YnWoOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes SOGN, a selective regression framework that uses ordinal target distributions to assess sample reliability. It combines concentration–coherence gating with stop-gradient weighted training to improve predictive performance at low coverage.

Critical strengths
  1. Explores an interesting direction of leveraging ordinal target structure for selective regression.
  2. Provides theoretical analysis connecting ordinal probability concentration to conditional variance.
  3. Includes comparisons with uncertainty-based baselines and component ablation studies.
Critical weaknesses
  1. The method primarily combines ordinal classification, confidence-based gating, and weighted regression. The proposed External Reference Principle is not fundamentally distinct from existing auxiliary-task-based confidence estimation, especially since both heads share the same encoder.
  2. The reported improvements mainly rely on R² at low coverage, while RMSE and MAE do not consistently improve. Since R² depends on the variance of the selected subset, higher R² does not necessarily indicate more reliable predictions.
  3. The evaluation covers only four main datasets with a single data split and three seeds. Moreover, the theoretical variance bound relies on strong calibration assumptions and does not establish that the proposed gate reliably identifies low-error samples.
Questions for authors

see weakness

2 comments

Thank you for your review. I will respond to the issues you raised below.

[W1]

(1) The contribution of this paper is not a simple combination of ordinal classification, confidence gating, and weighted regression. We have added two differentiable gating criteria based on the concentration of interval probabilities and ordered adjacency, and use stop-gradient weighted regression to apply the gate during training. To examine the role of stop-gradient weighted training, we added an ablation without stop-gradient to the revised manuscript (Table 2).

(2) ``External reference'' refers to the source of supervision and the way the gating signal is computed; it does not require the ordinal head and regression head to use independent encoders. The two heads share an encoder, but the ordinal head is supervised by target-interval labels. The gating score is computed from interval probabilities and their ordered structure; it is not taken directly from regression predictions, residuals, or predictive variance. To examine the role of ordered adjacency information, we added a control experiment comparing true and shuffled bin adjacency to the revised manuscript. In the tested Concrete setting, true adjacency achieves higher R2R^2 and lower RMSE (Table 12).

[W2]

(1) R2R^2 and RMSE/MAE measure different aspects of selective prediction: the latter two measure absolute errors on the selected samples, while R2R^2 measures the model's ability to explain variation in the targets of the selected samples. The target variance of the selected subset does affect R2R^2, so we now clearly distinguish these two types of conclusions in the revised manuscript.

(2) More importantly, we expanded the main experiments: across seven datasets, at every coverage level from 10% to 70%, SOGN achieves the highest mean R2R^2 on 5--6 datasets (Table 1; the complete results for all ten coverage levels are in Appendix B.2). This consistent advantage across datasets and coverage levels is our main empirical result regarding selective ranking.

(3) After further expanding the main experiments, Figure 3 was updated. This figure separately presents RMSE and MAE, making absolute-error performance directly comparable as well. Although our method is not the best among the many methods on RMSE and MAE, it has a positive effect on both metrics.

[W3]

(1) Regarding the experiments, thank you for pointing out their scale. We have now substantially expanded them, using three data-split seeds, three training seeds, and seven real-world datasets, and SOGN achieves good results (Table 1; the complete results for all ten coverage levels are in Appendix B.2).

(2) The theoretical variance bound relies on a strong calibration assumption and does not prove that the proposed gate can reliably identify low-error samples. This was a problem in our work. We re-examined the proof and found that the theorem does not require such a strong assumption; it only requires a one-sided event-probability condition for the contiguous three-bin window actually selected by the model. We provide a detailed proof (the revised condition and theorem are in Assumption 3.3 and Theorem 3.1; the full proof is in Appendix A.2).

(3) To directly examine the relationship between the gating score and actual prediction error, we added high-score versus low-score group experiments on five datasets. On all five datasets, the mean RMSE of the top score decile is lower than that of the bottom score decile. The results are in Appendix B.2, Table 11.

Additional comment: We are also very sorry that the previous paper's complex presentation caused difficulty in reading. We have updated the main text and improved its readability.

Reviewer_YnWoOpenReview ↗

Response to Author Rebuttal

Thank you for your detailed response and the additional experiments. I appreciate your efforts to address my concerns. After considering the rebuttal, I have decided to maintain my original scores.

Reviewer_A8yAOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes the Selective Ordinal Gating Network (SOGN) for selective regression. The method uses an auxiliary ordinal prediction head to capture target distribution structure and combines concentration and coherence criteria to guide sample weighting during training and sample selection during inference. The authors also provide a theoretical analysis of conditional variance compression. Experiments on several regression benchmarks demonstrate promising performance, particularly at low coverage.

Critical strengths
  1. The paper addresses an important problem in selective regression by exploring target structure as an alternative source of reliability information, rather than relying solely on predictive uncertainty.

  2. SOGN achieves competitive R^2 performance, particularly at low coverage, across four regression benchmarks.

  3. The authors provide a theoretical analysis connecting ordinal probability concentration with conditional variance. They also discuss the assumptions and limitations of the analysis and investigate the applicability of the method to circular targets.

Critical weaknesses
  1. The evaluation relies heavily on R^2, which may not fairly reflect selective regression performance because different methods select subsets with different target variances. The authors should better justify the choice of R^2 as the primary metric and provide a more balanced interpretation of the error metrics.

  2. The proposed framework mainly combines an auxiliary ordinal prediction head, confidence-based gating, and weighted regression. Moreover, the ordinal and regression heads share the same encoder, which makes the external reference principle less independent than its name suggests.

  3. The SoftMin function in Eq. (2) does not guarantee nonnegative outputs, yet its output is directly used as a weight in the regression loss. Negative weights could lead to an unbounded objective and unstable optimization.

  4. The variance compression bound relies on a strong assumption about the accuracy of ordinal probability estimates, which is not sufficiently validated experimentally. More importantly, the theoretical analysis does not directly establish that the proposed gating mechanism improves regression performance.

  5. Training combines concentration and coherence through a SoftMin gate, whereas inference uses a bonus score that prioritizes coherence. The authors should explain this difference and investigate whether using the same gating criterion during training and inference would yield better results.

  6. On Diabetes, SOGN achieves R^2 = 0.690 at 10% coverage in Table 5, compared with 0.778 in Table 2. Similarly, the improvement over Standard on Bike Sharing at 70% coverage differs between Tables 4 and 8.

Questions for authors
  1. The evaluation relies heavily on R^2, which may not fairly reflect selective regression performance because different methods select subsets with different target variances. The authors should better justify the choice of R^2 as the primary metric and provide a more balanced interpretation of the error metrics.

  2. The proposed framework mainly combines an auxiliary ordinal prediction head, confidence-based gating, and weighted regression. Moreover, the ordinal and regression heads share the same encoder, which makes the external reference principle less independent than its name suggests.

  3. The SoftMin function in Eq. (2) does not guarantee nonnegative outputs, yet its output is directly used as a weight in the regression loss. Negative weights could lead to an unbounded objective and unstable optimization.

  4. The variance compression bound relies on a strong assumption about the accuracy of ordinal probability estimates, which is not sufficiently validated experimentally. More importantly, the theoretical analysis does not directly establish that the proposed gating mechanism improves regression performance.

  5. Training combines concentration and coherence through a SoftMin gate, whereas inference uses a bonus score that prioritizes coherence. The authors should explain this difference and investigate whether using the same gating criterion during training and inference would yield better results.

  6. On Diabetes, SOGN achieves R^2 = 0.690 at 10% coverage in Table 5, compared with 0.778 in Table 2. Similarly, the improvement over Standard on Bike Sharing at 70% coverage differs between Tables 4 and 8.

1 comment

[W1 and Q1] Thank you for pointing out the effect of selected-subset target variance on R2R^2. We use R2R^2 to measure how well the model explains target variation among selected samples, while distinguishing it from RMSE and MAE, which measure absolute prediction error. We reran and expanded the main experiment (Table 1; complete results at ten coverage levels are in Appendix B.2). Figure 3 separately presents RMSE and MAE comparisons. Our conclusion concerns the selective R2R^2 advantage across datasets and coverage levels; we do not infer that higher R2R^2 necessarily means lower absolute error.

[W2 and Q2] The contribution of this paper is not simply a combination of ordinal classification, gating, and weighted regression. We propose differentiable criteria based on the concentration of interval probabilities and ordered bin adjacency, and use them in training through stop-gradient weighted regression. “External reference” describes the supervision source and ordered structure used to compute the gating signal; it does not imply that the ordinal and regression heads have independent encoders. The two heads share an encoder. In the revision, we added a no-stop-gradient ablation (Table 2) and a true-versus-permuted bin-adjacency comparison (Table 12) to examine the roles of the training design and ordered structure, respectively.

[W3 and Q3] We confirm that the original SoftMin can produce negative weights at low gate values. Retaining this form is a deliberate training-design choice; we have not overlooked the sign of its output. Stop-gradient prevents regression residuals from directly updating the ordinal gate through the weights. We also tested a strictly nonnegative SoftMin alternative. The results are shown below.

CoverageStrictly nonnegativeOur method
100%0.4280.498
90%0.4480.513
80%0.4920.532
70%0.4980.546
60%0.5240.554
50%0.5390.556
40%0.5350.556
30%0.5000.530
20%0.4770.519
10%-0.5470.409

In the evaluated setting, the current form achieves higher R2R^2 at all ten coverage levels. We therefore retain it for better selective performance.

[W4 and Q4] We revisited and revised the theoretical condition. The revised Assumption 3.3 does not require a strong pointwise error bound for every bin probability. Instead, it requires a one-sided event-probability condition only for the consecutive three-bin window selected by the model. Theorem 3.1 and Appendix A.2 provide the corresponding proof. We added per-bin and window-event calibration analyses on five datasets (Appendix B.2, Table 10). We also directly examined the relationship between the gate score and prediction error by comparing the RMSE of high- and low-score groups: the high-score group has lower mean RMSE on all five datasets (Table 11). The variance bound provides a conditional analysis of target structure; the latter experiment provides evidence about actual prediction error.

[W5 and Q5] The two scores serve different purposes. During training, SoftMin provides a continuous, differentiable weight that combines the effects of HC and LS on regression gradients. At inference, the bonus score prioritizes ordered adjacency while providing a complete ranking of test samples at any requested coverage. Section 2.3 of the revised manuscript explains this design and analyzes the properties of the ranking rule.

[W6 and Q6] Thank you for identifying the difference in numerical results between the earlier tables. We reran the main experiment and updated the corresponding tables. The revised main results are means over nine runs per dataset, comprising three data splits and three training seeds per split.

Comments on the paper

2 comments
AuthorsOpenReview ↗

General comment to AC and reviewers

Dear Area Chair and Reviewers,

Thank you for your thoughtful feedback. We have posted point-by-point responses and uploaded a revised manuscript. The revision expands the main evaluation to seven datasets, adds NGBoost and new mechanism and calibration analyses, clarifies the interpretation of selective R², and revises the variance result and its proof.

We would welcome any follow-up questions or remaining concerns while the discussion is open. Thank you again for taking the time to review our work.

AuthorsOpenReview ↗

Follow-up on Rebuttal and Remaining Concerns

Dear Reviewers,

As the discussion phase will end in around 24 hours, we would sincerely appreciate it if you could let us know whether our response has adequately addressed your concerns. We're happy to clarify anything further!

Thank you so much for your contributions!

Best, Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.