acceptodds
Under review as a conference paper at ICLR 2027

OrdinalICL: Making Tabular Foundation Models Ordinal-Native

Abstract

Ordinal targets arise in ratings, disease stages, and educational attainment: their categories are ordered, but the distances between them are unknown. General tabular foundation models usually ignore this structure and predict nominal classes. We ask a controlled question: if the backbone, trainable capacity, data, initialization, optimizer, and compute are held fixed, does giving the model an ordinal target interface help? We introduce OrdinalICL, which represents a declared variable-cardinality scale by ordered thresholds and returns a coherent class distribution with exact reversal symmetry. A TabICLv2 ordinal adapter and a compact model trained from scratch first establish that the interface learns and uses order. The original survey identifies TabDPT-Turbo v1.2 as the best-performing surveyed backbone that supports an exactly matched training experiment. We keep this backbone unchanged and attach equally sized prediction-correction modules that differ only in target geometry: the control uses class identities; OrdinalICL uses rank coordinates and ordered boundary decisions. Two independently prespecified evaluations on unseen synthetic mechanisms favor the ordinal module for every seed, class count, mechanism, and coverage stratum. Across a unified 40-family real benchmark, the final model also improves both its matched nominal control and released TabDPT, with family-level intervals excluding zero. The synthetic experiments provide the architecture test, while the real-data analysis measures transfer beyond the synthetic generator. Under matched training conditions, explicitly modeling target order improves TabDPT on ordinal problems.

Primary area:
foundation or frontier models, including LLMs
Keywords:
ordinal classification, ordinal regression, tabular foundation models, in-context learning, structured prediction, few-shot learning

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_5TjgOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

OrdinalICL investigates whether explicitly modeling target order improves tabular foundation models under matched training conditions. The method proposes a shared ordinal interface for variable cardinalities, generating valid class probabilities via ordered thresholds and ensuring exact reversal symmetry through predictive averaging. A TabICLv2 adapter and a compact PFN trained from scratch validate the interface's ability to learn order. Next, using TabDPT-Turbo v1.2 with fixed backbone, initialization, data, optimizer, and compute, the authors compare equal-sized ordinal and nominal prediction-correction modules. Two evaluations on unseen synthetic mechanisms favor the ordinal module. Across a 40-family real benchmark, the final model outperforms both its matched control and published TabDPT, with confidence intervals excluding zero. The study supports ordinal adaptation in TabDPT while acknowledging limitations in context scale, metrics, and pretraining data overlap.

Critical strengths
  1. Ordinal prediction has important practical applications, making the research question meaningful.
  2. The title, abstract, and experimental narrative are well aligned, clearly conveying the motivation and overall approach.
  3. The AI Use Statement clearly discloses the scope of AI assistance, supporting transparency in the research process, thus authors don't overclaim their contribution.
Critical weaknesses

1.The experimental analysis lacks depth. Despite matched controls and reported subgroup trends, the main text primarily presents aggregate scores, with limited analysis and visualization of the effects of feature count, context size, and class count. 2. The four requirements in Section 2.1 are insightful. Although order scrambling experiments and reversal error checks are provided, the evaluation lacks systematic tests corresponding to all four requirements, particularly order-preserving recoding and unobserved categories. 3. Substantial space is devoted to initial implementations and backbone selection, leaving limited room for deeper analysis of the final method. As an adaptation of an existing backbone, the method needs stronger validation across different backbones and task settings. 4. The ordinal baselines remain incomplete. Table 4 includes proportional odds, LightGBM, and CORAL, but lacks sufficient classical methods and simple ordinal heads on the same backbone. These comparisons would clarify the method’s distinctive value, even without state-of-the-art performance.

Questions for authors
  1. Which components account for the gains? The ordinal module introduces rank encoding, risk-set processing, and ordinal decoding together. Could ablations separate their contributions and evaluate the performance gains and computational overhead of reversal averaging?
  2. Why is this interface needed instead of a simple ordinal prediction head? How would a standard cumulative-threshold head perform under the same TabDPT backbone, parameter budget, and training conditions? The nominal control supports the benefit of the complete ordinal module, but does not establish the necessity of its specific design.
1 comment
AuthorsOpenReview ↗

Answer to Ws and Qs

Thank you for the detailed questions. We have run the reversal tests, checked the label-handling code, and broken down the real-data results. Here is what they show.

Why this design rather than a simple ordinal head? (W4, Q1–Q2)

We agree. Our comparison with an equally sized nominal module tests the complete change in label representation and prediction rule. It does not show that each part is needed, or that our design beats a standard cumulative-threshold head on the same backbone. We have not completed that head comparison or the separate training ablations. The existing proportional-odds and CORAL baselines do not answer this question.

Since both arms already use an order-sensitive loss, our experiment asks whether adding order to the architecture helps further.

What does reversal averaging add, and what does it cost? (Q1)

We tested both arms, all three seeds, and all 40 families. The averaged scores reproduce the original results exactly.

Prediction ruleNominal nRPSOrdinalICL nRPS
Forward order0.1477820.146846
Reversed order, mapped back0.1474280.147929
Average of both0.1470590.146894

Lower is better. For OrdinalICL, averaging changes nRPS by +0.000049 compared with forward-only prediction; the 95% family-bootstrap interval is [−0.000308, +0.000399]. We therefore cannot claim an improvement over forward-only prediction. Averaging ensures the same answer whichever direction the scale is presented. We report both directions because the ordinal-minus-nominal difference is −0.000936 forward but +0.000501 reversed. Both arms use averaging in the main comparison.

Tests of the four requirements (W2)

We tested the declared-label API on ten real contexts using three model seeds and four feature permutations. Replacing labels with unevenly spaced numbers or strings changed probabilities by zero. Reversing the declared order and mapping outputs back gave a maximum difference of 2.22×10⁻¹⁶.

We also kept one interior category absent in each context: seven required row replacements and three already lacked it. All ten returned valid distributions over all K categories, including positive probability for the absent category. This verifies that the category is retained; it does not show that predictions for it are better than the nominal model's.

The submitted order-scrambling tests show that the earlier TabICL/PFN models use label order.

Where do the gains occur? (W1)

The real-data differences, ordinal minus nominal, vary with class count:

ClassesFamilies representednRPS difference
2–35+0.000245
4–526−0.000197
6–72+0.000291
8–108−0.000407

For 5–7, 8–10, and 11–12 selected features, the differences are −0.000464, −0.000157, and −0.000123 (4, 5, and 32 families). Contexts missing a rank average −0.000238, versus −0.000099 with all ranks present. These analyses were added after review. Families can appear in multiple groups, and some groups are small. The results are mixed; they do not establish how changing feature count or removing categories would affect performance. Larger-context tests are not completed.

Other backbones and presentation (W3)

The earlier TabICLv2/PFN studies show that the interface can be implemented on other models. The final matched design has been tested on TabDPT v1.2 and v1.3. For v1.3, the difference from its nominal control is −0.000429 [−0.000612, −0.000247], with 30/40 family wins. This repeats the result on another release of TabDPT, not another architecture.

We agree that the paper spends too much space on earlier implementations and backbone selection. The independent synthetic tests support the controlled comparison. The pooled real benchmark was examined during backbone and compute selection, so its results need that qualification. Our evidence supports a small benefit in the tested setting, without showing that every component is needed or that the method helps on every task.

Reviewer_JvqQOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes OrdinalICL, which integrates ordinal label encoding and monotonic probability decoding into tabular foundation models. Experiments on synthetic tasks and 40 real-world dataset families show improvements over nominal classification baselines.

Critical strengths
  1. Addresses a practical limitation of tabular foundation models in handling ordinal targets.
  2. Provides well-controlled experiments with matched model capacity and training budgets.
  3. Supports varying numbers of ordinal classes with theoretically guaranteed monotonicity and reversal equivariance.
Critical weaknesses
  1. The core techniques, including threshold encoding and cumulative probability modeling, are well established in ordinal regression. Their integration into existing tabular foundation models offers limited methodological innovation.
  2. The improvement over the capacity-matched nominal baseline is only about 0.11% on real-world datasets, raising concerns about practical significance.
  3. The paper lacks fine-grained ablations of key components and comprehensive evaluations across different in-context sample sizes.
Questions for authors

see weakness

1 comment
AuthorsOpenReview ↗

Answer to Ws

Thank you for recognizing the motivation and the controlled comparison. We agree that the paper needs to explain what is new and how small the real-data gain is.

What is the contribution?

Threshold encoding, cumulative probabilities, ordinal losses, and reversal averaging are established ideas. We do not claim to have invented them. We bring them into a model that learns across tables with different numbers of ordered categories, and test whether this helps when the backbone, number of trainable parameters, data, initialization, and training budget are held fixed.

Both arms already use the same order-sensitive RPS loss and KL regularization. The question is therefore whether putting order into the architecture helps beyond that shared loss.

The two independent synthetic tests give ordinal-minus-nominal nRPS differences of −0.000454 [−0.000624, −0.000293] and −0.000575 [−0.000736, −0.000419]. The direction is favorable across the reported mechanisms, class counts, seeds, and category-coverage groups. Repeating the training on TabDPT v1.3 also favors the ordinal model. These results support the controlled finding, but do not show that it extends to every backbone.

Ablations and context size

The same-backbone cumulative-threshold head and separate encoding/decoder training ablations are not completed. Our current comparison cannot identify which component causes the gain or show that the specific design is necessary. All primary real-data evaluations use 32 context rows, so larger contexts remain untested.

The reversal ablation is complete for both arms, all 40 families, and three seeds. OrdinalICL scores 0.146846 forward-only and 0.146894 after averaging both directions: +0.000049 [−0.000308, +0.000399]. Averaging ensures reversal symmetry, but does not show an nRPS gain over forward-only prediction here.

The claim we can support is a small benefit under the tested training conditions, repeated on independent synthetic tasks and another TabDPT release. We do not claim a large practical improvement or a general advantage over other ways of handling ordinal targets.

Reviewer_osgcOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes OrdinalICL, an ordinal target interface for tabular foundation models (TFMs). Context labels are encoded by threshold indicators, which outputs monotone cumulative probabilities, and are converted into a coherent class distribution. The model is trained with the ranked probability score (RPS), and two-orientation averaging guarantees exact equivariance under scale reversal. The authors first verify the interface as a TabICLv2 adapter and as a small PFN trained from scratch. They then select TabDPT-Turbo v1.2 through a survey on a new 40-family ordinal benchmark. In the main experiment, they attach equal-capacity ordinal and nominal correction modules to the frozen TabDPT backbone and compare them under matched training conditions. The ordinal module yields small but consistent improvements over the nominal control on held-out synthetic mechanisms and on the 40 real-data families, using 32-row contexts.

Critical strengths
  1. The problem formulation is clear. The authors distinguish target types via meaning-preserving relabelings and translate the requirements for ordinal handling into empirically testable criteria.
  2. The authors construct a new real-world benchmark with thorough statistical analysis.
Critical weaknesses
  1. The section organization is non-standard and may confuse readers. Section 3 contains content that is not part of the methodology (Sections 3.3 and 3.5), and the experimental designs and results are scattered across sections. Reorganizing the paper, e.g., Section 3 for the methodology, Section 4.1 for the design and key results of the feasibility experiments, and Section 4.2 for the design and key results of the main experiments, would improve readability and make the content easier to follow.
  2. The goal of the paper is to add an adapter that enables TFMs to natively interpret ordinal targets. However, the experiments apply the final design only to the best-performing TFM. Applying it to other backbones and showing consistent improvements would better support the claim that the adapter is model-agnostic.
  3. The authors adopt nRPS, a less commonly used metric, without reporting other metrics widely used in ordinal classification studies. This makes it difficult for readers unfamiliar with nRPS to assess how impactful the results are.
  4. The proposed method only handles ordinal targets and does not address ordinal input features.
  5. There is no comparison with the alternative of treating the ordinal target as a continuous variable, e.g., using the TabPFN regressor instead of the classifier. It is unclear what advantage the proposed method has over this simple alternative.
Questions for authors
  1. Could the authors describe in more detail how the synthetic ordinal tasks are generated?
  2. The proposed method provides an inductive bias that allows the model to treat the target column as ordinal. How large is the impact of this compared with simply treating the ordinal target as continuous? In other words, how much does the performance drop when a table with an ordinal target is handled by the TabPFN regressor instead of a classifier, and how does it compare with OrdinalICL–TabDPT?
2 comments
AuthorsOpenReview ↗

Answer to Ws and Qs

Thank you for the concrete suggestions. We have completed the regression comparison and added familiar metrics. We answer those questions first.

Treating the target as continuous (W5, Q2)

We tested the TabPFN v3 regressor and native classifier on the same 40 families, 46 views, frozen 32-row contexts, and queries. Both use their respective v3 checkpoints, eight estimators, and averaging over both label directions. All 368 new evaluations succeeded.

The regressor fits ranks 0,…,K−1. We convert its predicted distribution into category probabilities by evaluating the CDF at half-integer boundaries. This keeps its uncertainty rather than rounding a point prediction.

MethodMean nRPS ↓
Existing TabPFN v3 cumulative classifier0.150097
TabPFN v3 rank regressor0.148066
TabPFN v3 native classifier0.148054
Fixed OrdinalICL–TabDPT0.146894
MetricMatched nominalOrdinalICLDifference, 95% interval
Log loss ↓1.4119161.409317−0.002599 [−0.003965, −0.001355]
Rank MAE ↓0.9366840.935689−0.000994 [−0.003199, +0.001274]
Quadratic weighted kappa ↑0.1394530.141121+0.001668 [−0.000259, +0.003748]

MAE and kappa use posterior-median predictions.

Synthetic data generation (Q1)

Each final-model training table has 32 context and 32 query rows, 1–12 features, and 3–10 classes. Training balances 20 latent mechanisms and eight class counts. Correlated mixed-type features feed a nonlinear function with varying noise. Ordered cutpoints then generate categories; some cutpoints depend on features. Simulated raters vary in severity, response style, and adjacent-category errors. Their threshold frequencies provide soft training targets. Feature missingness and shifts between context and query rows also vary; some categories can be absent from the context.

Both arms receive identical tables. Each 7,680-task confirmation test covers six mechanisms absent from training and eight class counts; the second uses six further mechanisms. The earlier interface study also included K=2.

Backbones, input features, and organization (W1–W4)

The earlier TabICLv2/PFN models show feasibility. Repeating the unchanged recipe on TabDPT v1.3 gives −0.000429 [−0.000612, −0.000247] against its nominal control, with 30/40 wins.

Our method concerns the target. It does not give ordinal input columns a dedicated representation or recoding guarantee.

We agree with separating the method from the experiments, putting each experiment's setup next to its results, and shortening the backbone-selection account. Those manuscript edits are still to be integrated.

Thank you for the additional experiments. However, according to the new metrics, OrdinalICL shows only a marginal improvement over its matched nominal control, with no significant gains in rank MAE or quadratic weighted kappa. Therefore, I will keep my score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.