Orthogonal Ensembles in Prediction Space: Diversity, Uncertainty, and Calibration

Explicit diversity objectives shape the relationships among ensemble members, but their connection to predictive reliability depends on the representation being regularized. We study prediction-space coupling through a Gram penalty on centered, normalized member logits, optimized jointly with the supervised task loss. Our analysis relates the penalty to effective dimension and derives the rank constraint imposed by ensemble centering. We also construct members that attain the centered lower bound while producing identical class probabilities, establishing a limit of logit-space diversity. Across four classification datasets and five backbones, 40 coupled and uncoupled runs separate the gains from averaging predictions from the incremental effects of coupling. Both conditions exceed their average member's accuracy in every setting. Coupling improves accuracy in 6 of 20 settings, with a mean difference of $-0.0972$ accuracy points and different effects on proper scores and calibration. Additional experiments examine coupling strength, predictive entropy, and aggregation rules; a brain-age case study illustrates uncertainty alongside shared prediction bias. These findings clarify what the objective controls and show why predictive reliability must be evaluated alongside representation geometry.