acceptodds
Withdrawn from ICLR 2027

Beyond Likelihood: Robust Multiclass Classification with Transformers

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

We propose a robust multiclass classification approach based on the rho-estimation framework. An oracle inequality for the resulting estimator controls estimation error when the model is well specified and provides robustness guarantees under model misspecification and data contamination. For i.i.d. data, we further derive an upper bound on the excess misclassification error of the corresponding plug-in classifier. We then explore its implementation with Transformers and establish an approximation result for Hölder functions defined on neighborhoods of low-dimensional manifolds. The construction uses standard self-attention to implement a stable projection and separates network approximation error from the effect of neighborhood thickness. Building on this result, we derive explicit convergence rates for softmax Transformer logit models, with exponents governed by the projection dimension rather than the ambient dimension. Experiments on synthetic and real-world datasets support the theoretical findings.

Primary area:
learning theory
Keywords:
Robust multiclass classification, Transformers, Conditional distribution estimation, Low-dimensional manifolds

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_dBoNOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a robust multiclass classification approach based on the ρ\rho-estimation framework. The authors derive an oracle inequality for the resulting estimator that provides estimation error bounds under correct model specification and robustness guarantees under model misspecification and data contamination. The authors also conduct experiments on both synthetic and real-world data to validate the proposed method.

Critical strengths
  1. The paper addresses an important topic, as developing theoretical guarantees for robust multiclass classification methods is valuable.

  2. The theoretical analysis is comprehensive and appears to be sound.

Critical weaknesses
  1. It is unclear whether the bounds established in the theorems are tight or optimal, or whether corresponding lower bounds can be derived.

  2. It is unclear how the proof techniques differ from those in existing work (e.g., Jiao et al., 2023) and what the main technical challenges are in deriving the theoretical results.

  3. It would be helpful to include a more quantitative analysis of the synthetic data experiments to examine whether the empirical results are consistent with the theoretical bounds and convergence rates.

Questions for authors

See Weaknesses.

Reviewer_mcgdOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes robust multiclass classification via ρ-estimation. Instead of minimizing an unbounded negative log-likelihood, it maintains a current classifier and searches for a challenger that beats it: each observation contributes a bounded comparison of the two models' probabilities for the observed label, transformed by a bounded, non-decreasing, odd function, and summed over the sample. The ideal estimator is one that no challenger can beat by a sufficient margin, which controls contamination influence rather than merely downweighting large losses.

Critical strengths

The statistical framework covers both misspecification and contamination.

The manifold result is honestly scoped.

Experimental reporting is above average for a theory paper.

Critical weaknesses

The theory–algorithm gap is not closed. The theory requires controlling the supremum over the whole model class of the pairwise comparison; the implementation runs a finite local search from few starts and stops when no better challenger is found. Failing to find a challenger does not establish that none exists, and the local maximum is a lower bound on the supremum, not a certificate for Eq. 1. The returned model therefore cannot inherit the stated risk bounds without an additional result bounding the effect of approximate optimization.

Algorithm 1 stops without replacing the incumbent when the contrast falls below threshold; B.2 describes updating to the challenger and then stopping, and the returned parameter is not re-challenged. Which parameter satisfies the stopping test is therefore ambiguous, which matters for both reproducibility and the theoretical link.

Synthetic outliers are of magnitude ~20 and assigned the least likely class; real-data contamination replaces features with ±50 AND relabels. The main evidence is joint feature-and-label corruption. Random label noise with clean inputs, class-dependent mislabeling, correlated mislabeling, and near-distribution outliers are not tested, and small same-domain transformers do not directly support large-scale image or text claims.

Questions for authors

See weakness

Reviewer_VgUHOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper utilizes \rho-estimation to study multiclass classification, as well as implementation with softmax transformer.

Critical strengths
  • It is a well-writen paer. I enjoy reading it.
  • The proof is sound, based on my judgement.
Critical weaknesses
  • The transition from general multiclass clasification to a specific Transformer architecture somewhat abrupt to me. It is like two independent issues and it is unclear why Transformers are the focus here or what additional insight this choice brings.

  • Another concerns is the technical contribution beyond prior work is not clear enough to me. The paper appears to combine existing ρ-estimation and Transformer approximation results, but I would like a clearer explanation of what new ideas are needed to make this combination work.

  • I feel a little confused about the position of this paper. For a paper on such a theoretically well-studied problem, the lack of a meaningful comparison with existing robust guarantees is disappointing. Empirical gains alone do not clarify what the new bounds actually add.

Questions for authors
  • What are the main new technical ideas beyond combining existing ρ-estimation and Transformer approximation results?
  • The empirical results look promising (on very simple datasets). I would be appreciated if the authors can compare bounds with existing robust multicalss methrods or provide a theorem clarifying when the presented method improves on existing approaches.
  • Given the paper’s focus on Transformers, why are the experiments limited to small, low-dimensional datasets? Can the method scale to more demanding classification tasks?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.