acceptodds
Under review as a conference paper at ICLR 2027

Architectural Priors for Ordinal Regression via Local Heads and Ordered Features

Abstract

Ordinal prediction problems—age estimation, disease grading, historical image dating—involve labels with intrinsic order, yet deep ordinal methods act at the loss or the output-side posterior, leaving the head’s mapping from features to logits agnostic to label order. We argue that the head itself is the third and right locus of inductive bias, and develop a theoretical framework around it. We introduce a family of label-axis structured heads in which each feature dimension affects only a local neighborhood of ordered labels, and prove that such heads operate through two complementary mechanisms: an ordered classifier mapping that localizes predictions on the label axis, and an implicit bias that drives the backbone toward ordered feature geometry through label-frequency filtering of the gradient field. Across 23 dataset–backbone cells spanning age estimation, histology and diabetic-retinopathy grading, and historical image dating (K=4 to 70), and eight ordinal baselines, our heads win or tie 177–182 of 184 Holm-corrected pairwise comparisons on MAE (128–145 of 161 on the ranked probability score) and are the only methods within 3% median gap of the per-cell best in every label-axis regime, while every baseline fails in some regime by 22–233% at worst. Mechanism-level analysis confirms the two mechanisms are dissociable: the Poisson-Unimodal baseline realizes only the second—it stays within 3–10% at large K—the best of any baseline in that regime—but collapses at small K and transfers a weaker representation under head swapping.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
ordinal regression, ordinal classification, structured prediction, supervised representation learning, structured classifier heads, inductive bias, ordered representations

Reviews

Among:
Shape:
1234this paper, Bottom 25%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.75 over 4 reviews, above 25% of the others.
Reviewer_TcS8OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes GaussianFC and GaussianFC-PDF, which parameterize each feature’s weight profile along the label axis with a Gaussian-shaped function to exploit label order. The authors analyze their effects through theoretical analysis and head-swap experiments, and compare them against eight baselines across 23 experimental configurations covering four types of tasks (Sections 4 to 6).

Critical strengths

The method is simple, the experiments cover diverse settings, and hyperparameters are selected on a nested validation split. The head-swap experiments also support that some representation benefits carry over to other readouts. On FG-NET with ViT-Tiny, training a linear head on a frozen GFC-trained backbone achieves an MAE of 4.87, compared with 7.17 for end-to-end Linear+CE training. Similar improvements appear on both ResNet backbones (Figs. 2 and 5).

Critical weaknesses

W1: The additional value of the Gaussian parameterization is not sufficiently established.

coGOL (Lu et al., SDM 2022) already incorporates label order by penalizing weight differences between adjacent threshold classifiers. The semicircular cosine variant of Hard-CPL (Wang et al., AAAI 2023) constructs class proxies in label order and uses scaled cosine similarity to compute logits. These approaches differ from the proposed per-feature Gaussian parameterization, but show that the abstract’s characterization of prior work is too broad.

The experiments lack direct comparisons with heads that regularize classifier-weight smoothness or impose ordered prototype structures. The current evidence therefore does not sufficiently establish the additional benefit of choosing a Gaussian profile over other ways of imposing ordinal structure on the head.

W2: The interpretation of ordered representations goes beyond the formal theoretical results.

Proposition 3 bounds the component of a one-step update outside a particular feature subspace; it does not guarantee that class centers follow label order. Eq. (21) also contains a term reflecting how class-conditional mean predictions vary along the label axis. The current analysis therefore does not establish that head smoothness alone guarantees smoothness of the complete gradient field.

The authors acknowledge relevant limitations in Appendix A.5 and Section 7. However, the abstract presents an implicit bias that “drives the backbone toward ordered feature geometry” as a proven mechanism. Appendix B first bounds the target-driven term −W, but it is unclear whether its subsequent claim of smoothness “at every step, for every residual field” is restricted to that term. The scope of these claims should be made explicit and consistent with the formal results.

Questions for authors

[Clarification] Which existing results support an additional benefit of Gaussian parameterization over other ways of imposing ordinal structure on classifier heads?

[Clarification] Is the smoothness claim in Appendix B restricted to the target-driven term, or does it concern the complete gradient field? How do Proposition 3 and Eq. (21) support the mechanism claimed in the abstract, and which parts currently rely only on empirical evidence?

Reviewer_BAkwOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

this work tackles ordinal classification task (OC). authors argue that prior works focus only on introducing class orders via loss or model output. ordering features was left out, something that authors are pursing here. in particular, they considered re-parameterizing the last linear weights via a vector of gaussians. each gaussian component is 'aligned' with a class position (index) over labels axis. 2 variants of the gaussian are provided. results are reported over 5 datasets. a form of comparison to some OC works is presented.

Critical strengths

the paper tackles an important task - OC. some results and comparisons are provided.

Critical weaknesses

method:

  • writing: extremely poor writing, organization of the paper and major lack of justifications and discussions. poor introduction (1 general parag. followed by presenting the method in 1 parag, and then 4 parag for contributions). then, jump to preliminaries??? then, related work section. then, method section (4 tiny parag where things are given as facts.), then worse: sec 5 - theoretical framework: a series of propositions without a single discussion or explanations.
  • extreme lack of justification of the direction: abstract (line 12: "yet deep ordinal methods act at the loss or the output-side posterior, leaving the head’s mapping from features to logits agnostic to label order."), related work (line 140: "All (ie prior OC works) act at the loss or the output-side posterior, leaving the head’s mapping from features to logits agnostic to label order."): a- this is not totally true. there are few works who aimed to introduce a notion of order into feature space. Ordinal Deep Feature Learning for Facial Age Estimation, Lui et al. 2017. A Constrained Deep Neural Network for Ordinal Regression, Goh et al. 2018. b- yes, most dominant works in OC focus on loss/logits order. ordering features were left out as many other things. what is the issue here? why only mention features? there is no justification provided in the paper that supports: 1- ordering features will lead to ordering output. 2- no evidence to support: ordering via loss/output DOES or DOES NOT order features. 3- why it is important to order features. c- the paper keeps talking about ordering features. 1- it is super-vague. in what sense to order features. 2- what has been proposed here in this paper DOES NOT order features. after training, the final gaussians in phi vector will be attracted to each class (ordered at label axis), giving a notion of order to WEIGHTS.
  • reparameterization issues: what has been done here is change the standard W (dxK where d is the last feature layer dimension and K is the number of classes) weight matrix at the output by a single vector (dx1) where each component is a 1 dimension gaussian gi with parameters (ai, mui, sigmai) evaluated NOT AT FEATURE zi but at CLASS INDEX j. a- there is no justification or explanation why this change and what is its relation to ordering output logits. b- this way, each phi_i 'weight' (gaussian) REUSED (in computations) by each class, contrary to standard W parametrization where each class j has its own column vector W:,j. what is the impact of sharing parameters here (no discussion)?. c- this way, at convergence, and generally speaking, each gaussian will be 'attracted' to an index class j, to maximize its logit (eq.1, + ignore the translation bj). typically, the gaussian needs to 'peak' at index j. meaning, the mean mui needs to be equal to tj (the index used in the paper). this allows the gaussian to act locally and prevent it from influencing other classes (it still does but with low magnitude). this has 3 implications: 1- each class will learn to select a subset of weights phi that will be attracted around it. 2- these gaussians around the same class j will be redundant (roughly speaking). they will converge to the same mean tj (sigma could be different). 3- the value of the 'weights' (gaussian means mui) will depend on the label index tj - which is absurd. if K=1000, the gaussian mean will be in that range. d- so now 'weights' are 'groups' and ordered. what this has to do with ordering logits? how this drastic change of weights magnitudes will affect features z? the last class K-1 will have strong 'attraction' which can steer the logits always in its favor. this can largely change depending on the step between classes in label axis: tj+1 and tj, which can make tj go to higher magnitude, and consequently, mui. e- why introduce a new parameters ai in sec.4 - line 178? why not use sigma in gaussian?
  • why the 2 variants?
  • fig.1. it leads to confusion to present the size of phi vector (dim=d) to be the same as the logits vector (dim=K).

experiments:

  • no details about the experiments were provided.
  • why this form of evaluation: number of win/tie/loss???? the work must provide for each dataset a detailed numerical comparison to previous works on that dataset, using standard metrics, and publicly reported results, under identical experimental protocol, in comparison to extensive methods (with focus on recent works and SOTA). as it is, the results are not useful. -what is E1, ...E6??

overall the direction and method presented here are unjustified. the reported results are under a poor form.

please stop using ai.

Questions for authors

please see weaknesses.

Reviewer_qM7uOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes GaussianFC and GaussianFC-PDF, two ordinal classification heads that parameterize each feature’s weights over ordered labels using a Gaussian profile. The intended benefits are a structured mapping and an inductive bias toward ordinal representations. The theoretical analysis establishes a bound on off-target gradient contributions, bounds on head roughness, and a conditional one-step bound on class-center updates. The reported results suggest competitive MAE across different label cardinalities and improved linear readout from representations trained with the proposed heads.

Critical strengths

Parameterizing feature-to-label weight profiles by centers and bandwidths provides a compact way to incorporate ordinal structure while retaining a standard classification objective. The substantial reduction in head parameters and compatibility with different backbones make this a potentially useful practical design.The experiments include multiple domains, CNN and transformer backbones, separately tuned learning rates, and subject-exclusive splits for the facial-age datasets.

Critical weaknesses

The central mechanism claims exceed what the theory and diagnostics establish. Proposition 3 bounds one update outside a particular feature-space subspace under unverified kernel and residual assumptions. This does not establish that class centers become ordered or smooth along the label axis: the coefficients within that subspace still depend on the residual matrix and class priors.

The baseline failure cases underlying the robustness claim require further validation. Appendix C.1 states that DLDL searches Gaussian target widths down to σ=0.1\sigma=0.1. Under the standard normalized Gaussian construction on integer-spaced labels, this produces essentially one-hot targets, so its training objective should closely approximate ordinary cross-entropy when the remaining implementation choices match. This follows from the formulation in the original DLDL paper. Nevertheless, Table C.1 reports substantial failures: DLDL versus CE obtains 0.6585 versus 0.3295 MAE on BACH/R50@128, and 0.7537 versus 0.2362 on APTOS/ViT@224. The latter also has validation MAE 0.7505 versus 0.2050. These discrepancies do not establish an implementation error, but they warrant an explanation of target construction, loss scaling, decoding, and configuration selection before being interpreted as inherent baseline failure regimes.

Questions for authors

How should Appendix B’s claim about smoothness for every residual field be reconciled with the uncontrolled prediction-dependent term in Eq. (21)? Which properties of ordered feature geometry are formally established, and which remain empirical hypotheses?

Can you provide a controlled comparison that preserves parameter count and comparable weight scale while disrupting label order, such as consistently permuting the Gaussian head’s label columns?

Can you report a matched DLDL σ=0.1\sigma=0.1 versus CE sanity check for the failure cases above, including the exact target construction, loss reduction, prediction rule, and validation-selection results?

Do the principal conclusions hold under an analysis accounting for dependence among seeds sharing a test fold?

Reviewer_qPyuOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes GaussianFC and GaussianFC-PDF for ordinal prediction. Each feature’s classifier-weight profile is parameterized as a Gaussian over ordered labels, imposing locality and smoothness on the feature-to-logit mapping. The paper connects these constraints to gradient-tail bounds, head spectral energy, and conditional one-step class-center dynamics. Experiments across 23 dataset–backbone–resolution settings show competitive MAE, while head-swap and representation diagnostics investigate whether the benefits extend beyond the final classifier.

Critical strengths
  1. The architectural intervention is simple and practically useful.
  2. The head-swap results provide meaningful evidence of a representation-level benefit.
Critical weaknesses
  1. Proposition 3 bounds the component of a single update outside a feature-space subspace; it does not bound the ordering or label-axis curvature of the class centers. More concretely, the full class-conditional gradient field is �0��0�. Bounding the curvature of the target term �1��1� does not control the additional term �2��2�, as Eq. (21) explicitly acknowledges through its dependence on �3��3�. Nevertheless, Appendix B, p.16, states that head smoothness makes the transmitted field smooth “for every residual field.” This does not follow. Appendix A.5 also explicitly states that Proposition 3 does not imply inheritance of label order. These limitations materially weaken the central theoretical explanation, beyond merely restricting it to one optimization step. The claims should be narrowed or supported by additional conditions controlling the prediction field and the resulting class-center dynamics.
  2. E5(b), Figure 4, and the discussion interpret improvement with additional principal components as evidence that ordinal information occupies several low-frequency label-Laplacian modes, contrasting this with PU’s purported one-dimensional collapse. However, PCA directions are feature-variance directions, not label-Laplacian eigenvectors. Moreover, a single predictive direction can be a linear combination of several PCs, so this probe behavior does not establish the claimed distinction. Appendix A.5 acknowledges both issues, but the stronger interpretation remains in the main discussion and Appendix B. Since this experiment is used to substantiate a central mechanistic distinction, the paper needs a direct label-axis spectral analysis or a substantially narrower interpretation.
Questions for authors
  1. Can the authors provide conditions or measurements controlling the WPˉLW\bar P L term in Eq. (21), and clarify which claims about ordered geometry remain justified without this control? In particular, how should the unconditional smoothness statement in Appendix B be reconciled with Appendix A.3–A.5?
  2. Can the authors directly analyze the label-Laplacian spectrum of the learned class-center matrix, with suitable scale normalization, to test the claimed multi-mode structure? How do the existing PCA results distinguish that claim from a single ordinal readout direction distributed across several PCs?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.