acceptodds
Withdrawn from ICLR 2027

Orthogonal Ensembles in Prediction Space: Diversity, Uncertainty, and Calibration

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Explicit diversity objectives shape the relationships among ensemble members, but their connection to predictive reliability depends on the representation being regularized. We study prediction-space coupling through a Gram penalty on centered, normalized member logits, optimized jointly with the supervised task loss. Our analysis relates the penalty to effective dimension and derives the rank constraint imposed by ensemble centering. We also construct members that attain the centered lower bound while producing identical class probabilities, establishing a limit of logit-space diversity. Across four classification datasets and five backbones, 40 coupled and uncoupled runs separate the gains from averaging predictions from the incremental effects of coupling. Both conditions exceed their average member's accuracy in every setting. Coupling improves accuracy in 6 of 20 settings, with a mean difference of accuracy points and different effects on proper scores and calibration. Additional experiments examine coupling strength, predictive entropy, and aggregation rules; a brain-age case study illustrates uncertainty alongside shared prediction bias. These findings clarify what the objective controls and show why predictive reliability must be evaluated alongside representation geometry.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Deep ensembles, Orthogonalization, ensemble diversity, uncertainty quantification, probabilistic calibration.

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_kqD2OpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper investigates diversification of ensembles and how it relates the effective dimension of the ensemble and the logit geometry. The diversity is achieved through coupled training of the ensemble, which regularizes the Gram matrix of the centered and normalized logits of the ensemble. The regularized training is evaluated empirically, comparing it to standard ensembles. The paper shows that orthogonality in logits does not necessarily imply prediction diversity.

Critical strengths

Proposition 2 shows that the ensemble members can change their raw logits to ensure orthogonality, while not changing the class probabilities. This gives a clear negative result regarding for penalties that rely on orthogonality of logits and is backed up by the experiments.

Critical weaknesses

The paper does not make its contribution and findings very clear. Proposition 1 and 2 are presented without any remarks about what they show, why they are interesting or how they are linked to the regularization and experiments. Section 3.1 and 3.2 present the regularization objective and ensembling, without stating which parts are novel contributions and why this normalization and regularization was chosen.

In the experiments the single seeds make the differences between the coupled and uncoupled ensemble very hard to interpret. The only significant difference between coupled and uncoupled ensembles shows that coupled training might not converge (Table 2, VGG16). It is not immediately clear what the experiments are designed to show and whether the very limited differences between coupled and uncoupled ensembles are due to the specific training setup or the shown limitations of orthogonal ensembles.

Formatting: The appendix is only in the supplementary material, making it harder to read the paper, as the references to the appendix point nowhere.

Questions for authors
  • Could you clarify the contribution and goal of this paper and explain the impact of proposition 1 and 2?
  • Does Figure 4 show the proposed regularization or just normal ensembles?
  • Could you clarify what Section 5.7 shows?
  • How would the results change if the penalty was applied to the post-softmax probabilities?
Reviewer_QtFLOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies whether training the members of an ensemble to have more orthogonal outputs helps in making the ensemble more accurate and reliable. Concretely, they train ensembles with five members and a penalty that pushes the (centered and normalized) logits to be orthogonal, and compare them with ensembles trained without the penalty across a number of datasets and different backbones. They find that in all settings averaging across members helps, but the additional orthogonality penalty only improves accuracy in a handful of them.

Critical strengths

The comparison between the standard ensembles and the ensembles with the penalty is set up in a fair way (i.e., every ensemble trained with the penalty also has a corresponding ensemble without it), and the paper reports all results, including the negative ones.

Critical weaknesses

A major empirical finding of the paper seems to be that the additional penalty does not reliably help. To me, a negative result usually has a much higher bar for being useful, in the sense that it should be investigated much more deeply. The paper takes some steps in this direction (e.g., the construction which shows that the penalty can be satisfied while all members make identical predictions), but to me these do not go far enough to make the negative result useful on its own.

Questions for authors

If you wanted to understand more deeply why the orthogonality penalty does not seem to give a positive result, which investigations would you do?

Reviewer_GC5hOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper is on the topic of training ensembles of neural networks. The propose a joint mechanism of training these ensembles.

Critical strengths

Interesting topic.

Critical weaknesses
  • Paper is missing its appendix and therefore should have been desk rejected. All references are broken, so I'm unsure how this was missed on submission.

I read the paper and note some major issues:

  • Paper presents as proposing a method, but seems to actually present a null results since the training method doesn't work? Very unclear.
  • The actual proposed penalty term has not clear motivation and appears to have some fundamental flaws. For example, as the paper notes: similarly scaled member deviations already would have pairwise cosines of about −1/(K−1)−1/(K−1) so J≈K/(K−1)=1.25J \approx K/(K−1) = 1.25. Independently trained ensembles are likely already at this lower bound without any coupling, so the reported coupled values near 1.25 are not evidence that the penalty does anything.
  • Results are hard to interpret and don't provide any meaningful conclusions.
  • Paper misses practically all relevant work on the topic, which is a well studied area. These are not tangental, some are directly relevant to the claims of the paper.

[1] Jeffares, Alan, et al. “Joint training of deep ensembles fails due to learner collusion.” Advances in Neural Information Processing Systems 36 (2023).

[2] Webb, Andrew M., et al. “To ensemble or not ensemble: When does end-to-end training fail?” Joint European Conference on Machine Learning and Knowledge Discovery in Databases (ECML-PKDD). Springer, 2020.

[3] Abe, Taiga, et al. “Pathologies of predictive diversity in deep ensembles.” Transactions on Machine Learning Research (2024).

[4] Ortega, Luis A., Rafael Cabañas, and Andrés Masegosa. “Diversity and generalization in neural network ensembles.” International Conference on Artificial Intelligence and Statistics. PMLR, 2022.

[5] Wood, Danny, et al. “A unified theory of diversity in ensemble learning.” Journal of Machine Learning Research 24.359 (2023): 1-49.

Questions for authors

None

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.