acceptodds
Under review as a conference paper at ICLR 2027

One Table, Many Queries: Benchmarking Generalist Tabular Models across Generation, Imputation, and Prediction

Abstract

Tabular generation, missing-value imputation, and supervised prediction are usually developed and benchmarked as separate problems, even though each can be written as a conditional query to the same table distribution. This fragmentation leaves a basic question unresolved: does competence on one query carry to others, or does a multi-interface model merely expose several APIs? We introduce OneTableBench, to our knowledge the first benchmark designed to turn this shared probabilistic view into a unified empirical study. Rather than merging three leaderboards, OneTableBench preserves task-specific data, information, and evaluation contracts while aligning 56 datasets, 46 registered methods, and 20 semantic capability axes. Its evaluation forms a four-level evidence chain: auditing metric redundancy, reliability, and construct validity; measuring cross-task rank alignment on matched support; estimating transfer with matched interventions over all seven non-empty combinations of generation, imputation, and prediction objectives; and comparing fixed model identities with deployable specialist portfolios through Pareto and leave-one-dataset-out analyses. The resulting picture is neither universal transfer nor complete task independence. Metrics are compressible but remain multidimensional, and method rankings agree only along particular capability axes. Observational alignment does not produce a robust off-diagonal transfer effect under our prespecified gate; several joint-objective interactions are instead sub-additive, and classification and regression induce different objective frontiers. Deployment conclusions are similarly profile-dependent: evaluated generalists can surpass selected specialist portfolios on downstream utility, whereas the portfolios retain a clear advantage on categorical fidelity. OneTableBench therefore reframes tabular generality from binary task support or a single leaderboard rank into a capability profile over queries, contracts, and deployment priorities.

Primary area:
datasets and benchmarks
Keywords:
Tabular Learning, Benchmarking, Generalist Models, Data Generation, Missing-Value Imputation

Reviews

Among:
Shape:
1234this paper, Top 17%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.75 over 4 reviews, above 83% of the others.
Reviewer_ExtsOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces OneTableBench, which uniformly evaluates the general-purpose capabilities of tabular models across data generation, missing value imputation, and prediction tasks. Through comparisons of cross-task consistency, capability transfer, and specialized models, it reveals the performance trade-offs and transfer limitations of general-purpose models across different tasks.

Critical strengths

1.The research questions addressed by the benchmark does have some value, and the authors have conducted a systematic set of experiments. 2.The paper is well written and the benchmark systematically organize the different kinds of datasets from unified perspectives .

Critical weaknesses

1.The construct validity is insufficient. Although the authors construct the benchmark from the statistical view, the model did not represent the general tabula datasets. 2.The author explicitly limits the scope of the study to these three tasks. This scope seems not sufficient to support the paper’s broader discussion of generalist tabular models.

Questions for authors
  1. Why is it necessary to unify the generation, imputation and prediction in one benchmark? What is the application for this unified benchmark?
  2. Compared with individual benchmarks like generation, what is the benefit of using the proposed OneTableBench?
  3. What does the emphasized conditional query bring benefits for users?
  4. The authors mention “This perspective requires four levels of evidence: measurement validity, cross-query rank alignment, objective-level transfer or interference, and deployment against fixed specialist portfolios.“, why are the four levels of evidence important? What can be improved for present benchmarks?
2 comments

We thank the reviewer for raising questions about the value of evaluating the three tasks together.

Weakness:

1. Does the benchmark represent real-world tabular datasets?

We started from 492 OpenML datasets reported in 24 studies. We clustered the 377 eligible datasets by size, feature types, missingness, and dependence structure, then selected one per cluster before evaluating any method. Of the 50 selected datasets, 40 are real-world or derived from real-world data. Six additional datasets with known causal structures provide a separate test of dependence recovery (Appendix A). The findings are grounded in the single-table settings covered by this selection. Independent datasets would help test how well they generalize.

2. Can three tasks support claims about generalist models?

We use generality to mean that one fixed system can generate rows, fill missing features, and predict a designated target. This supports claims about those three uses of a single table. Broader claims would require tests on workloads such as forecasting or reasoning across related tables.

Questions:

1. Why evaluate the three tasks together, and who benefits?

PFNs showed that a pretrained model can predict on new tables using examples in context. This led us to ask whether pretraining could also support imputation and generation. TabFORGE and TabGift show progress on generation [1, 3]. TabMosaic handles generation, imputation, and class-conditional generation with one pretrained model [2].

The three tasks query the same table distribution for complete rows, missing features, or a target. Evaluating them together lets us test whether shared training helps across tasks. A team may need to fill missing entries, generate synthetic data to test a pipeline, and predict outcomes on new records. OneTableBench tests when one system can serve all three needs and when task specialists perform better.

TabMosaic and TabGift appeared after the submission deadline. TabFORGE has public code, and the authors' model repository currently has no downloadable pretrained checkpoint. Our experiments cover systems such as LimiX-2, TabICLv2, and TabPFN-3 through their available inference procedures.

2. What does OneTableBench add to a generation-only benchmark?

OneTableBench compares the same methods on shared datasets and held-out splits while preserving task-specific metrics and information access. For example, methods that retain more predictive utility in synthetic data tend to rank higher on direct prediction. Some generation fidelity metrics show the opposite relationship with classification performance. This helps users see which strengths tend to co-occur across tasks.

3. How do conditional queries benefit users?

A query specifies which values users have and which they need. If a user wants to fill in missing floor area before a home is sold, the imputation test should hide the sale price. For historical records with known prices, the test can make the price available. Users can choose results that match their workflow, and developers can test both settings through one interface.

4. Why do the four levels of evidence matter, and how can benchmarks improve?

  1. Measurement validity. Report fidelity, dependence, and utility separately. Check that each metric responds as expected. For example, shuffling each column independently preserves its distribution while disrupting dependencies.
  2. Cross-query rank alignment. Compare ranks on shared splits and matching method–dataset pairs to see which capabilities co-occur. Changes caused by adding an objective require a controlled comparison.
  3. Objective transfer and interference. Compare objective combinations on matched backbones, datasets, and seeds to test whether adding one objective helps or hurts another task. Our seven combinations share a fixed total budget within each backbone. Matching exposure to each objective in follow-up experiments would help isolate the effect of reallocating that budget.
  4. Fixed specialist portfolios. Compare one fixed generalist with one preselected specialist per task on the same datasets. Report each task and the weakest task. In our study, specialists led on fidelity, while some generalists led on downstream utility.

References

[1] Jiang, Xiangjian, et al. "Tabular Foundation Model for Generative Modelling." arXiv preprint arXiv:2605.09424 (2026).

[2] Kim, Jayoung, Sanghyun Hong, and Noseong Park. "TabMosaic: Unified In-Context Diffusion for Tabular Generative Tasks." arXiv preprint arXiv:2610.15699 (2026).

[3] Li, Yu-Sheng, Jun-Peng Jiang, and Han-Jia Ye. "TabGift: A Context-Efficient Foundation Model for Tabular Data Generation." arXiv preprint arXiv:2610.16990 (2026).

Thank you for the detailed response. However, the motivation of “whether pretraining could also support the imputation and generation” does not sufficiently justify why these three tasks should be evaluated together to assess generality. Although the authors clarify the scope of their definition, my concern regarding the justification for task selection remains unresolved. Therefore, I will maintain my original score.

Reviewer_6TohOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents ONETABLEBENCH, a unified benchmark for assessing tabular models on data generation, missing value imputation, and supervised prediction. The tasks are framed as conditional queries over a common tabular distribution. It examines whether capabilities and performance rankings transfer among tasks.

The benchmark consists of 56 datasets, 46 registered methods, and 20 retained capability axes. The evaluation includes four components: metric redundancy analysis, cross-task rank correlation, controlled training objective interventions, and a comparison of generalist models with a fixed set of specialist portfolios.

The results show that cross-task ranking agreement depends on capability and generally does not produce robust gains from joint objective training. Several observations are sub-additive under the proposed rank-based evaluator; rank-based list models can outperform selected specialist portfolios on downstream utility, while specialists still do better on categorical fidelity.

Critical strengths
  • Unified, multi-dimensional evaluation: it brings together three separate tabular learning tasks traditionally evaluated in isolation and maintains task-specific evaluation contracts. The capability axis analysis and matched support comparisons offer a useful alternative to isolated leaderboards or universal aggregate scores.
  • Distinguishing observational agreement from transfer: the combination of cross-task correlations, matched objective interventions, and specialist portfolio comparisons gives a much finer-grained understanding of tabular generalisation. The negative and capability-dependent findings are potentially valuable for research and application.
Critical weaknesses
  • Rank-based interaction estimates make causal interpretation difficult. Section 5.3 computes objective interactions from normalized seven-arm ranks rather than underlying metric outcomes. As ranks are non-linear, ordinal transformations can lead to negative interactions even when early performance improvements are additive. The sub-additivity we report may partly reflect the evaluation scale rather than actual objective interference. The main interaction outcomes need validation by appropriately normalised performance outcomes.
  • Limited scope of intervention and limited comparability of optimisation: The benchmark covers 56 datasets but controlled objective transfer experiments are restricted to just 5 evaluation datasets and 5 model families. The main missingness setting is MCAR at 0.3. Furthermore, matching on architecture, dataset, and seed does not necessarily equate to matching on objective weighting, effective training exposure, or optimisation difficulty. This limits the generalisability and causal interpretation of the transfer results.
Questions for authors
  1. How do the negative objective interactions in Table 4 change if we compute them using the raw performance measures (or the appropriately normalised performance measures) rather than the percentile ranks? Do the conclusions remain the same across the different scales of evaluation?
  2. How are training budgets, loss weights, optimisation schedules and data exposure administered across the seven objective combinations? Could differences in multitask optimisation partly explain some of the negative or null transfer we've seen?
  3. How representative are the five intervention datasets in relation to the much larger 56-dataset benchmark? Do the key transfer conclusions remain valid under different missingness mechanisms, different aggregations of the capability axis, and uncertainty estimates that take into account the dependence at the dataset level?
2 comments

We thank the reviewer for the insightful comments on rank-based interaction, intervention scope, optimisation comparability, and dataset representativeness.

1. Rank-based interactions and scale sensitivity

Ranking is nonlinear and can yield a nonzero interaction even for an additive metric. We use within-block midranks to give heterogeneous metrics a common interpretation. Each score is the fraction of the other six prespecified arms outperformed, with half credit for ties. The interaction compares the gain in this fraction from adding an objective in two training contexts, following the logic of nonparametric relative effects [1].

We recomputed the four-arm contrasts on the same matched blocks using three continuous scales. Each applies the same positive affine transformation across arms, preserving additivity in the oriented metric.

Table 1. G×I∣PG\times I\mid P interaction estimates [95% crossed-bootstrap CI]

Evaluation axisRank (Table 4)Block min–maxDataset SDSeed-noise SD
Generation C2ST−0.320 [−0.527, −0.117]−0.474 [−0.738, −0.175]−1.14 [−2.16, −0.17]−3.28 [−6.42, −0.54]
Generation fidelity−0.356 [−0.589, −0.103]−0.616 [−0.924, −0.249]−1.71 [−3.03, −0.53]−6.43 [−11.27, −1.89]
Generation global utility−0.119 [−0.224, −0.035]−0.254 [−0.327, −0.135]−0.61 [−0.99, −0.27]−2.07 [−3.70, −0.75]
Imputation association−0.449 [−0.636, −0.223]−0.706 [−0.937, −0.372]−1.66 [−2.68, −0.60]−4.03 [−7.14, −1.39]
Imputation numerical point accuracy−0.526 [−0.703, −0.230]−0.761 [−0.971, −0.376]−2.02 [−3.25, −0.67]−5.59 [−9.84, −1.64]

All five interactions stayed negative in every backbone and passed the robustness criteria on all three continuous scales (Holm-adjusted p<.004p<.004). In native units, 12 of 13 metric members had intervals below zero. Authenticity had a positive interaction.

Adding I to P improved the five axes by 2.15–6.54 seed-noise SDs. Adding I to GP yielded +0.08 to +0.14 SDs, with all intervals spanning zero. The negative interactions reflect smaller incremental gains under the fixed budget.

2. Intervention scope and evaluation robustness

The study crosses seven objective arms, five backbones, and five datasets. Imputation covers MCAR, MAR, MNAR, and mixed missingness at rates .1, .3, and .5, with three draws per setting and the target hidden. We form matched contrasts per setting before equal-weight averaging. Both headline imputation G×I∣PG\times I\mid P estimates are negative in all 12 settings on all four scales.

The crossed bootstrap resamples datasets and backbones, then seeds within selected blocks, keeping arms paired. We aggregate imputation draws and settings within seed blocks. No pooled off-diagonal effect met the full robustness criteria, and neither G→I nor P→I met them in any individual setting.

3. Optimisation comparability and objective exposure

Within each backbone–dataset–seed block, the arms share the training split, preprocessing, architecture, optimiser, batch size, learning-rate schedule, loss-component weighting rules, and configured update and row-exposure budget (Appendix H.2).

Uniform query sampling gives each of kk enabled objectives an expected 1/k1/k of exposures. G’s share falls from 1/21/2 in GP to 1/31/3 in GIP. The contrasts estimate the effect of adding an objective under a fixed budget. Changes in exposure, gradient balance, or optimisation difficulty could contribute to negative or null effects. Isolating exposure dilution requires exposure-matched controls.

4. Representativeness of the five datasets

Our experiments involve five models, seven arms, 12 imputation settings, and bootstrap resampling, making the overall cost substantial. We therefore aimed to use a small but representative set of datasets. The five intervention datasets were selected from 50 OpenML datasets representing different meta-feature clusters. These five datasets cover 303–30,000 rows and 9–23 features. For 16 methods with at least 80% coverage, rankings on these five datasets closely matched those on the remaining 45, with Spearman correlations of .941, .936, and .909 for G, I, and P. These correlations were above the median of 5,000 random subsets under each sampling scheme. Based on these results and the coverage of sample sizes and feature counts, we chose these five datasets to balance coverage and representativeness.

References

[1] Brunner, E., Konietschke, F., Pauly, M., & Puri, M. L. (2017). Rank-based procedures in factorial designs: Hypotheses about non-parametric treatment effects. Journal of the Royal Statistical Society: Series B (Statistical Methodology), 79(5), 1463–1485. Article

Thank you for the detailed rebuttal and additional analyses. While I still have some reservations regarding objective exposure under the fixed training budget, the responses have addressed several of my concerns. I have updated my score accordingly.

Reviewer_JGKpOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

The paper presents a unified evaluation protocol for tabular data and considers the tasks of generation, imputation and prediction. Based on the assumption that the combination of these three tasks yield a better account of model performance across tasks and data sets.

The idea of combining several tasks, such as data cleaning and prediction, is interesting. Yet the manuscript remains a bit unclear as to why the existing separation of tasks should be overcome. Many models that are very important for tabular tasks are not generative, to name just one good reason for separating the tasks in their evaluation. Table 8 nicely shows this non-overlap between models and tasks.

Also, there are many benchmarks that do account for both imputation and downstream predictive performance, covering most methods that can be applied to those two tasks.

At the same time there are many other dimensions related to tabular prediction tasks or data quality, that are important to the comprehensive evaluation of tabular models, which are not considered in the proposed benchmark, including but not limited to: Fairness, robustness (to more errors than just missingness), efficiency, and others.

Overall the paper is an interesting contribution but needs to be improved with respect to clarity in its motivation and better embedding in the literature. It seems that there is a lot of complexity lumped together and the authors make the point that there is a good reason to integrate that complexity in one joint evaluation, but it doesn’t become really clear to me, why many of these things should be analysed jointly (like: why comparing generative and discrimination models when there are good reasons to analyse them separately).

Critical strengths

Interesting approach combining different modelling tasks for tabular setting

Comprehensive evaluation of multiple tasks

Combination of many state of the art methods and evaluation protocols

Critical weaknesses

Unclarity / lack of realistic missingness patterns

Unclarity / lack of realistic errors for robustness evaluation

Unclarity concerning many details of the protocol

Unclarity / lack of motivation as to why so many tasks / models need to be combined, when they cannot be combined.

Many sentences appear like they deserve a bit of (human expert / non-expert) proof-reading. For instance this one here in the abstract is really hard to parse without more context - and maybe the abstract is not the best place for unfolding all that context: "Observational alignment does not produce a robust off-diagonal transfer effect under our prespecified gate“ (gate, diagonal of what matrix, alignment are all not defined here)

Minor: The definition in eq 1 suggests that this could be a prank

Questions for authors

which missingness patters were used?

which error types and distributions were used for the robustness analysis (it would be great to have not only the references but the exact errors types and distributions, sorry in case I missed that ...)

3 comments

We thank the reviewer for the careful reading and for asking about the realism of the missingness and error settings, the evaluation protocol, and the motivation for joint evaluation. We also appreciate the comments on the abstract and notation.

1. What missingness patterns were used?

We use missing completely at random (MCAR), missing at random (MAR), missing not at random (MNAR), and mixed missingness (MIXED) at nominal rates of 10%, 30%, and 50% in selected feature columns. MCAR samples observed cells uniformly without replacement, independently of their values. For MAR, we condition each masked feature on the retained feature with the highest absolute Spearman correlation and sufficient training support. We sample cells in rows where that feature is at or above its training median if numeric or equals its training mode if categorical. MNAR applies the threshold or category rule to the masked feature's own value, trying other prespecified quantiles for numeric features if needed. MIXED takes the union of three submasks, each generated at one third of the target rate, and adjusts the total to the target where feasible.

These patterns model value-independent loss, selective collection based on observed information, and selective reporting based on the value itself. MIXED lets these processes coexist in one table. Methods share each mask and are scored against the original values. Thresholds, rates, and weights are controlled settings without dataset-specific calibration to natural missingness.

2. What error types and distributions were used in the robustness analysis?

We check imputation degradation as missingness increases, cross-task rank sensitivity to method removal, and the stability of transfer effects. Separate synthetic tests examine metric responses to specific errors.

Feature masking under the four mechanisms above is the input perturbation for imputation. With the model, dataset, mechanism, and seeds fixed, we compare each higher rate with the lowest available rate (usually 10%). Degradation is the larger of the available NRMSE increase for numeric features and macro-F1 decrease for categorical features (Appendices B.2.2 and C.2, Table 10).

For rank alignment, we remove one method at a time and recompute common support, axis ranks, and correlations to check sign stability (Appendix D). For transfer, we use three training seeds, normalize metric ranks across seven objective combinations to [0,1][0,1] within each backbone, dataset, seed, and evaluation setting, and compute paired rank differences. We form 95% CIs from 2,000 crossed bootstrap draws over backbones and datasets, resampling seeds within blocks and keeping arms paired. The robustness criteria also account for multiple testing, effect size, and sign consistency across backbones (Appendices E.1-E.2 and H.2).

In the generation stress tests (Appendix C.3), we shuffle values independently within columns, shift two numeric columns by +3.0+3.0 and −2.5-2.5, replace column values with their median or mode, and replay training rows. For point imputation, we apply Bernoulli(0.35)\mathrm{Bernoulli}(0.35) masks to cells in one numeric and one categorical feature. Biased fills add 2.02.0 to numeric imputations and use a fixed category.

For distributional imputation, we mask X1X_1 with Bernoulli(0.60)\mathrm{Bernoulli}(0.60) and compare the reference X1∣X2∼N(1.8X2,0.652)X_1\mid X_2\sim\mathcal N(1.8X_2,0.65^2) with a biased, underdispersed N(1.8X2+0.55,0.062)\mathcal N(1.8X_2+0.55,0.06^2) and an overdispersed N(1.8X2,2.22)\mathcal N(1.8X_2,2.2^2). For binary prediction, we multiply the logit of pp by 3.03.0 or 0.300.30 to change calibration while preserving ranking, and separately perturb ranking with 0.72p+0.28U0.72p+0.28U, where U∼Uniform(0,1)U\sim\mathrm{Uniform}(0,1).

3. Why evaluate the three tasks together?

Many tabular models target one task. Recent foundation models have begun to support multiple tasks. TabFORGE's official implementation supports generation, imputation, and prediction[1]. It was released on October 8, 2026, after our September submission, so it was not included.

Can one model match specialists across all three tasks? Generation tests joint distribution fidelity. Imputation and prediction test recovery of missing features and inference of targets from observed information, respectively. We use shared splits and fixed task-specific protocols. Specialists provide task-specific references, and fixed specialist portfolios serve as baselines for generalists. The resulting gaps and strengths can guide unified model design and training objectives.

4. Abstract and notation

We agree that the abstract relies too heavily on methodological context. We will state the main findings and their scope more directly in the abstract and define the relevant terms and notation in the main text.

References:

[1] Jiang X, Liu M, Simidjievski N, et al. Tabular Foundation Model for Generative Modelling[J]. arXiv preprint arXiv:2605.09424, 2026.

Reviewer_JGKpOpenReview ↗

systematic error protocols and motivation could strengthen the contribution

Thanks a lot to all authors for the careful and comprehensive answer to the reviewers’ comments. I think some concerns were addressed.

Overall the answers highlight the need for a bit more clarity in the description of the experimental details around the missingness and error injection. Given the claim of the paper for broad applicability and comprehensiveness I’d suggest to aim for a more systematic and broader examination of data errors in their protocol.

I guess that it’s true that there are some other papers that investigate all these tasks together, but that in itself is probably not the kind of explanation and motivation readers might find helpful. In order to make sense out of the rather complex suite of results, it would be important to have a clear motivation for why these tasks should be considered together, as well as an easy to follow argumentation and illustration of the insights that can be gained from investigating the results together.

AuthorsOpenReview ↗

Clarifying Missingness Protocols and Guidance for Multitask Tabular Modeling

Thank you for the constructive follow-up. Our missingness settings draw on Still More Shades of Null [1], covering MCAR, MAR, MNAR, and their mixture. In the revision, we will specify feature selection, conditioning and sampling rules, mixed-mask construction, and perturbation parameters more explicitly. We agree that broader examination of data errors would strengthen the evaluation. Our current robustness evidence concerns controlled missingness and metric responses to known output failures. We will clarify this scope and discuss additional input errors and application-specific missingness patterns as extensions.

The motivation for evaluating generation, imputation, and prediction together is to identify capabilities that can support a shared model and understand how to train and evaluate such a model. Our analysis connects numerical recovery and dependence preservation with prediction, and connects generation to both through synthetic-data predictive utility. These relationships motivate learning dependencies that support multiple conditional queries, while evaluating distributional fidelity explicitly. For readers generating data to train predictors, this also motivates using TSTR—training on synthetic records and testing on held-out real records—alongside fidelity measures.

The joint-training results provide evidence that this direction is feasible. Training with all three objectives yields a configuration on the joint three-task Pareto frontier, offering a competitive trade-off among the evaluated alternatives. This supports further development of unified models and makes objective selection a concrete design question governed by the application’s priorities and training budget.

For future multitask models, the resulting guidance is to develop shared conditional modeling capabilities, retain direct supervision for required tasks, and evaluate how objective combinations affect the complete capability profile. Evaluation should preserve task-specific metrics and examine joint Pareto trade-offs, making both shared strengths and remaining bottlenecks visible. This helps researchers identify improvements that benefit several uses of a table and determine where additional training or modeling effort is needed. We will make these training and evaluation implications more explicit in the revision.

[1] Khan et al. Still More Shades of Null: An Evaluation Suite for Responsible Missing Value Imputation. arXiv:2409.07510.

Reviewer_w8yPOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces ONETABLEBENCH, a benchmark for testing whether tabular models actually transfer capabilities across generation, imputation, and prediction, instead of just offering different task interfaces. The benchmark includes 56 datasets, 46 methods, and 20 capability axes, while keeping the evaluation rules for each task separate. It studies the structure of the metrics, agreement between task rankings, transfer across all combinations of the three objectives, and the practical tradeoff between generalist models and specialist portfolios. The main conclusion is nuanced: there is some alignment across tasks, but no clear universal transfer effect. The better choice between generalist and specialist models depends on the capability and the deployment goal.

Critical strengths

The question is important and well motivated. Supporting generation, imputation, and prediction in one model does not show that useful capability actually transfers across the tasks. I like the benchmark design. Instead of forcing all three tasks into one score, the paper keeps their individual evaluation rules and studies how the resulting capabilities relate to each other.

Critical weaknesses

The main concern I would want resolved is whether the transfer experiments clearly separate real cross-task transfer from differences in compute, data exposure, model capacity, and hyperparameter tuning between the single-task and joint-objective settings. A second concern is the definition of the 20 capability axes. The paper should clearly explain how these axes were created and validated, and whether any part of their definition depends on the same metrics later used to support the conclusions.

Questions for authors

How are the 20 capability axes defined, and what checks were used to make sure they represent different constructs rather than mostly correlated groups of metrics? In the transfer experiments, how are compute budget, model capacity, training data exposure, and hyperparameter search controlled between single-task and multi-objective models? For the generalist-versus-specialist comparison, what resource budget is kept fixed? In particular, how do total parameters, training cost, and tuning effort compare between one generalist and the specialist portfolio?

1 comment

We thank the reviewer for the careful reading and for asking about axis construction and validation, controls in the transfer experiments, and resource comparability between generalists and specialist portfolios. These questions help clarify the basis for our metrics, the conditions held fixed, and the scope of our conclusions.

1. How were the 20 capability axes defined and validated?

We started from common metrics for tabular generation, imputation, and prediction and included diagnostics for structural fidelity and privacy risk. Each axis answers a specific evaluation question and has a fixed set of one or more metrics. In imputation, for example, recovering masked values, recovering column distributions, and preserving inter-column dependencies are distinct goals. The grouping rules in Section 3.2 yield 20 axes with 42 member metrics, listed in Table 3.

For axis cc, let Kc\mathcal K_c be its metric set. Within evaluation stratum ss, we align metric directions and compute percentile ranks rs,m,kr_{s,m,k} among methods with valid results for all members. The score for method mm is the equal-weight mean of these ranks (Section 3.1, Eq. 1).

Ss,m,c=1∣Kc∣∑k∈Kcrs,m,k.S_{s,m,c}=\frac{1}{|\mathcal K_c|}\sum_{k\in\mathcal K_c}r_{s,m,k}.

Grouping uses both metric definitions and empirical rank correlations. For each metric pair, we compute percentile ranks on common support within each dataset and protocol, then compute the Pearson correlation of the pooled paired ranks. We use 500 bootstrap resamples of whole datasets. A pair is a redundancy candidate if ρ≥0.85\rho\ge 0.85 and the lower bound of its 95% CI is at least 0.70 (Appendix C.1). Pooling also requires that the metrics measure the same quantity. NMAE and NRMSE both measure masked numerical recovery and agree closely (ρ=0.947\rho=0.947, 95% CI [0.931,0.959][0.931,0.959]), so they share the numerical point accuracy axis (Fig. 10(a), Appendix C.2). MAE and RMSE also correlate (ρ≈0.88\rho\approx 0.88). They weight large errors differently, so we retain separate axes (Fig. 10(c)). Coverage and novelty exhibit a trade-off (ρ=−0.973\rho=-0.973), and both components remain individually reportable (Fig. 9).

We used controlled perturbations to check how selected metrics respond to specific changes (Appendix C.3). Independently shuffling values within each column preserves marginal distributions while breaking inter-column dependencies. Marginal-only imputation can recover column distributions while recovering individual values and joint structure poorly. Monotone rescaling of probabilities preserves rankings while changing calibration. ECE also remains separate after correcting metric correlations for test-retest reliability estimated from three seeds (Appendix C.4, Fig. 11).

Figure 12 shows that the 20 core generation metrics and 10 core imputation metrics require eight and five components, respectively, to reach 90% of the spectrum mass. The spectra show substantial shared variation, with several directions still contributing. Equation (11) evaluates the actual equal-weight axes. They retain 79.65%, 86.21%, 96.40%, and 99.99% of the nonnegative spectrum mass for generation, imputation, classification, and regression, respectively. The generation projection includes only metrics with a defined direction (Appendix C.5).

2. How were training conditions controlled in the transfer experiments?

For a given backbone, dataset, and seed, single- and multi-objective arms share the architecture and capacity settings, training splits and preprocessing, preset training budget, and hyperparameters. We run no arm-specific hyperparameter search. The intervention changes which training objectives are enabled. We sample uniformly among enabled objectives, so adding an objective reallocates training opportunities within the same budget. Table 15 lists the datasets and splits, and Appendix H.2, Table 16 gives the configurations and budgets for each backbone.

3. What was fixed in the generalist vs. specialist portfolio comparison?

We hold the evaluation conditions fixed, using common datasets, splits, task contracts, and metrics. We leave out one dataset at a time, select the specialist portfolio on the remaining datasets, and compare it with each generalist on the held-out dataset (Section 6, Appendix F.3). Resource settings are method-specific. We did not match total parameters, training cost, or tuning effort, and we do not report aggregate comparisons of these quantities.

Each method uses its default configuration with documented benchmark overrides. We did not run a shared external hyperparameter search. We considered matching parameter counts, but rescaling different architectures could alter their performance. We therefore retained their reference configurations (Appendix H.1, Table 14). Figure 39 separately reports measured inference time under a balanced three-task workload.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.