acceptodds
Under review as a conference paper at ICLR 2027

ProxyBench: A Tabular Classification Benchmark that Mirrors Enterprise Data

Abstract

Enterprises are increasingly deploying machine-learning models for tabular data. However, they face a structural evaluation gap: Very few proprietary enterprise datasets are publicly released, and public benchmarks often fail to capture enterprise deployment realities. Similarly, as model development within the scientific community relies on openly accessible benchmarks, it cannot directly address enterprise challenges and needs. To fill this gap, we introduce ProxyBench, an open-source enterprise-grade classification benchmark containing 134 tasks on 90 different tables. To ensure the similarity of ProxyBench with enterprise data without disclosing sensitive information, we first generate tasks from a large pool of public datasets. Then, we use a privacy-preserving data fingerprint to automatically identify the ones having the most similar characteristics to the tasks of an internal, proprietary benchmark. By comparing the resulting aggregated statistics, we show that ProxyBench is much more statistically similar to enterprise-grade data than existing open-source benchmarks. Furthermore, our experimental evaluation of eleven baseline models shows that ProxyBench offers a model ranking with a superior Spearman's correlation with the internal benchmark than open-source alternatives. ProxyBench enables reproducible and accessible evaluation of tabular models in an enterprise-grade context. It provides the research community with a tool to develop tabular learning methods that more closely reflect real-world enterprise data.

Primary area:
datasets and benchmarks
Keywords:
benchmarking, enterprise data, model evaluation, table discovery, tabular foundation models

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_4CafOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces ProxyBench, an open-source tabular classification benchmark containing 134 tasks designed to mirror real-world enterprise data. Because actual enterprise data cannot be released due to privacy and confidentiality constraints, the authors developed a privacy-preserving statistical fingerprint. They used this fingerprint to search a large pool of 270,000 public tables (RawTabs) to extract tasks that most closely match the statistical properties of a proprietary internal benchmark, EGI-Bench. Experimental results show ProxyBench produces model rankings significantly more aligned with enterprise data than existing open-source alternatives.

Critical strengths
  1. Innovative Privacy-Preserving Approach: The use of a statistical data fingerprint to bridge the gap between confidential enterprise data and open-source tables is a creative and highly practical solution to a major structural evaluation problem in the field.
  2. Strong Empirical Motivation: The authors convincingly demonstrate that current open-source benchmarks yield vastly different model rankings compared to actual enterprise data (Spearman's correlation of 0.555 vs 0.873), strongly justifying the need for ProxyBench.
Critical weaknesses
  1. Semantic and Relational Limitations: The similarity between ProxyBench and enterprise data is purely statistical (e.g., data types, entropy, cardinality), completely ignoring the semantic meaning, temporal dynamics, and linked relational structures that define complex enterprise databases.
  2. Lack of Temporal Splits: The benchmark relies on standard stratified random splits (I.I.D.) rather than time-based splits. This fails to capture the covariate and concept drift that is a primary challenge in real-world enterprise machine learning deployments.
Questions for authors
  1. Your analysis shows that ranking alignment drops significantly for task subsets with high feature entropy. Is the Zstandard compression method an insufficient proxy for capturing the complexity of these features, and how can this entropy estimation be improved?
  2. The proprietary reference dataset, EGI-Bench, is extracted from the internal data of a single large software vendor, EntCorp. How can you ensure that ProxyBench's targeted statistical characteristics generalize to enterprise data in completely different domains?
  3. The capability distribution for target class imbalance in ProxyBench is noted to be less aligned with EGI-Bench than the much smaller OS-Industry benchmark. What adjustments to the task selection and pruning pipeline are needed to better capture true enterprise class imbalance?
4 comments
AuthorsOpenReview ↗

Response to Reviewer 4Caf – Part 1/2: Scope and temporal splits

We thank you for highlighting the relevance and empirical motivation/evaluation of our contribution, as well as important limitations. We address the concerns in the following:

Semantic and relational limitations

We agree that statistical matching alone does not establish semantic similarity between tasks. As a first step towards bridging the gap between confidential enterprise data and public benchmarks, we focused on statistical rather than semantic properties. Incorporating semantic descriptors into the matching procedure is a promising direction for future research that will require substantial work, as well as a suitable task-level semantic representation and an estimation of its impact on the overall ranking alignment with the reference benchmark.

Regarding relational structure, EGI-Bench consists of tabular tasks, not relational ones; as a consequence, both EGI-Bench and ProxyBench are evaluated as single-table classification benchmarks. Our matching procedure and model evaluation do not explicitly account for relationships across tables; that is out of the scope of our current contribution. The reported alignment therefore concerns this single-table setting. Nevertheless, extending the approach to relational prediction tasks remains an important direction for future work.

We will expand the discussion of these limitations in the revised paper, clarifying the scope of our results and the challenges involved in incorporating semantic and relational information.

Lack of temporal splits

Indeed, our current split procedure does not account for the temporal dimension, partly because suitable temporal information is limited in RawTabs. While temporal dynamics are important in enterprise applications, we do not consider them the primary challenge in the setting studied here: our evaluation highlights substantial differences from many public benchmarks in other characteristics, such as target cardinality, class imbalance, and entropy, and drift in our reference EGI-Bench is mostly low (see Figure 1).

We will make this limitation more explicit in the revised paper and discuss the need to extend both our internal reference collection and RawTabs with tasks involving temporal dynamics and distribution shifts, providing a basis for future extensions of ProxyBench.

AuthorsOpenReview ↗

Response to Reviewer 4Caf – Part 2/2: Entropy, generalizability, and class imbalance

Feature entropy estimation with Zstandard compression

Indeed, ranking alignment is lowest for the high-feature-entropy subset, which contains 32 tasks. The cause of this discrepancy remains unclear. It may partly reflect sampling variability due to the limited subset size, the greedy optimization and selection procedure, or bias in the underlying RawTabs collection.

Applying the same proxy across all tasks ensures a consistent measurement procedure, although it does not rule out limitations of the proxy itself. We chose this compression-based approximation because it accommodates heterogeneous data types while remaining computationally feasible at the scale of our collection. These practical considerations motivate its use.

Nevertheless, accurate and computationally efficient entropy estimation across heterogeneous data types remains an important direction for future work.

EGI-Bench generalizability

Indeed, ProxyBench is designed to mirror EntCorp’s internal EGI-Bench, which contains tasks spanning diverse lines of business. This diversity does not establish generalization to other organizations or domains. As also noted in our response to Reviewer BMGW, our empirical claims are therefore limited to alignment with EGI-Bench.

The construction procedure could, in principle, be applied to other enterprise reference collections. Validating its effectiveness across independently curated enterprise benchmarks is an important direction for future work.

Target class imbalance

Indeed, OS-Ind contains 11 classification tasks drawn from SALT and TabRed and achieves closer alignment with EGI-Bench on the imbalance capability than ProxyBench, with Total Variation Distances (TVDs) of 0.031 and 0.150, respectively. However, the 95% bootstrap confidence interval for OS-Ind is wide [0.025, 0.316], indicating substantial uncertainty in this estimate (see our response to Reviewer BMGW, “Choice of binning thresholds”, for the bootstrap analysis used to quantify uncertainty in the TVD estimates).

Furthermore, SALT and TabRed have TVDs of 0.253 and 0.747 for this capability. The low TVD of OS-Ind may reflect complementary capability distributions of SALT and TabRed, which produce a closer aggregate match when combined. Although OS-Ind obtains the better point estimate, ProxyBench remains substantially closer to EGI-Bench than the broader OS collection on this capability, with a TVD of 0.150 compared with 0.661.

A possible extension would be to introduce capability-specific weights into the fingerprint similarity measure, allowing prioritization of capabilities such as class imbalance. Whether this improves imbalance alignment without compromising alignment on the other capabilities would need to be explored in future work.

Thank you for the detailed responses and clarifications. Most of my concerns are acknowledged rather than resolved or supported by additional empirical evidence. However, I understand that updating the dataset is challenging given the nature and scope of this work.

Therefore, I will maintain my original score.

Thank you for your swift response! We agree to have acknowledged your valid concerns but also reasoned about why we believe them to be out of scope (e.g. EGI-Bench not being composed of relational problems) or currently infeasible (e.g. due to the lack of sufficiently enough public data with temporal structure). We thank you for raising these concerns and have clarified the scope of our contribution accordingly!

Reviewer_BMGWOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

ProxyBench addresses the evaluation gap where public benchmarks fail to represent real enterprise data. It proposes a benchmark construction method based on privacy preserving statistical fingerprints. The authors curate 166 classification tasks from internal enterprise data as a private reference, then match 134 statistically similar tasks from 270 thousand public tables through a three stage pipeline to form an open benchmark. Experiments confirm that ProxyBench significantly outperforms existing public benchmarks in both statistical distribution alignment and model ranking correlation.

Critical strengths
  1. The constraint of enterprise data inaccessibility is cleverly transformed into a table discovery problem, where coarse grained discrete fingerprints naturally achieve privacy protection and quantify task similarity without leaking any raw data.
  2. Cross validation from multiple angles including statistical distribution, t-SNE visualization, model ranking correlation, and Elo scoring ensures high credibility of the conclusions.
Critical weaknesses
  1. The reliance on data characteristics from a single enterprise means generalization to other industries or organizations remains unverified.
  2. EGI Bench is entirely undisclosed, preventing third parties from independently reproducing or verifying experimental results, which limits transparency.
Questions for authors
  1. Could the authors justify the specific binning thresholds used for the 10 capabilities and provide a sensitivity analysis showing whether the alignment results remain stable under alternative thresholds or continuous feature representations?
  2. Could the authors discuss whether the ranking alignment conclusion holds when evaluated with more recent tabular foundation models such as TabFM, EXAONE, and TabDPT, which are omitted from the current evaluation?
  3. Could the authors provide statistical evidence, such as confidence intervals or permutation tests, to demonstrate that the observed ranking alignment between ProxyBench and EGI-Bench is robust rather than an artifact of the binary match rate over 10 categorical dimensions?
5 comments
AuthorsOpenReview ↗

Response to Reviewer BMGW – Part 1/3: Scope and baseline models

We thank the reviewer for the positive feedback on the relevance of our contribution and the thoroughness of our evaluation, while raising open questions that we shall address in the following:

Generalization to other industries or organizations

Thank you for raising this concern. We believe this to be a chicken-and-egg problem that our contribution is trying to overcome: indeed, we cannot verify if/how ProxyBench generalizes to other corporations’ internal benchmarks or other industries, as there are no publicly available benchmarks of meaningful scope. For example, note that TabRed contains only 3 classification tasks, and SALT contains 8 targets but just from a single joined dataset from 4 raw tables. ProxyBench is built to mirror the broad selection of tasks of EntCorp’s internal EGI-Bench, containing internal data spanning diverse business domains and tasks. Thus, within the scope of our diverse data in terms of business domains, our evaluation shows alignment with this diverse reference collection. However, we do not claim generalization of ProxyBench to other domains beyond what we studied.

As also mentioned in our answer to Reviewer 4Caf, the construction procedure could, in principle, be applied to other enterprise reference collections. Validating its effectiveness across independently curated enterprise benchmarks is an important direction for future work.

Non-disclosure of EGI-Bench

We agree that for full scientific reproducibility, EGI-Bench would need to be public. However, by construction of our problem, this is not possible (nor legal, due to data privacy laws), and is precisely the problem our contribution aims to overcome. Hence, we believe this issue highlights the relevance of our work, as opposed to being an intrinsic weakness.

Additional baseline model evaluations

At the time of submission/literature cutoff, the investigated models were considered state-of-the-art (and in large part still are, cf. TabArena). One key requirement to evaluate a model on ProxyBench is the support for high-cardinality classification targets. Note that both EXAONE and TabFM only support a maximum of 10 classes, which is infeasible in the case of ProxyBench, whereas TabPFN and TabICL support high-cardinality targets natively via error-correcting output codes.

Nevertheless, since submission, the field has evolved, and we have added results for Kumo-Tabular as well as TabPFN-3.5 to the Appendix.

AuthorsOpenReview ↗

Response to Reviewer BMGW – Part 2/3: Binning thresholds and sensitivity analysis

Choice of binning thresholds

We agree that the choice of binning thresholds requires further justification, as the thresholds have been chosen by convention rather than by statistics. For example, the notion of Small/Medium/Large tables coincides with those of BeyondArena, naturally following a log-linear thresholding. Other bins were chosen in a similar heuristic fashion. However, we note the following two points:

First, the first step of the pipeline (candidate selection) is performed on the actual raw statistical values, not the binned ones, and is hence independent of the binning thresholds.

Second, while we agree that quantifying the sensitivity of the benchmark ranking depending on these thresholds would be a valuable investigation, we argue that this is infeasible in the full end-to-end setting: the benchmark curation takes around 1 day in total using a large compute cluster with 166 multi-core nodes in parallel. Running a sensitivity analysis that requires the creation of at least dozens of benchmark candidates (created using different binning thresholds) is computationally, financially, and environmentally extremely costly.

Nevertheless, we agree that investigating the influence of binning boundaries on our pipeline is a valuable experiment.

To this end, without re-generating ProxyBench from scratch, we (1) analyzed the impact of a 10% increase or decrease of the capability thresholds on the Total Variation Distance (using the same curated tasks but recomputing the associated capabilities) and (2) built variants of Figure 2a and Figure 4 that include 95% bootstrap confidence intervals computed from 1,000 resamples.

CapabilityVariationProxyBenchOSOS-Ind.BA-Lite IIDBA-Lite non-IID
Default—0.1540.4220.2350.4030.426
Width+10%0.1580.4230.2350.4030.426
Width-10%0.1530.4250.2350.4070.433
Height+10%0.1510.4190.2370.4030.428
Height-10%0.1550.4230.2350.4010.422
Cardinality+10%0.1520.4200.2380.4010.428
Cardinality-10%0.1540.4170.2310.4060.430
Covariate Drift+10%0.1500.4170.2290.3980.427
Covariate Drift-10%0.1600.4280.2400.4070.427
Feature Entropy+10%0.1410.4180.2350.4010.406
Feature Entropy-10%0.1660.4230.2390.4020.427
Rare Classes+10%0.1530.4220.2350.4010.427
Rare Classes-10%0.1560.4230.2340.4050.426
Imbalance+10%0.1570.4210.2430.4050.425
Imbalance-10%0.1530.4240.2420.4040.424
Label Drift+10%0.1500.4180.2310.3990.422
Label Drift-10%0.1600.4280.2400.4090.423
Target Entropy+10%0.1540.4230.2350.4050.426
Target Entropy-10%0.1540.4210.2350.4040.427
All+10%0.1320.4030.2380.3900.406
All-10%0.1790.4340.2530.4180.428
min TVD—0.1320.4030.2290.3900.406
max TVD—0.1790.4340.2530.4180.433
average delta—0.0050.0040.0040.0030.004
average TVD—0.1540.4220.2370.4030.425

The sensitivity analysis suggests that the capability alignment of ProxyBench with EGI-Bench is robust against variations in the binning thresholds. Across all perturbations, ProxyBench's Total Variation Distance is in the range [0.132, 0.179], and the average TVD over all perturbed configurations (0.154) is identical to the default value. While the average per-perturbation delta (0.005) is larger than those of the competing benchmarks, it is on the same order of magnitude as them (0.003–0.004). Finally, even under the most pessimistic configuration (All −10%, TVD = 0.179), ProxyBench shows a lower TVD than the best competitor configuration: OS-Ind (best-case TVD = 0.229).

The bootstrap confidence intervals are reported in Part 3/3.

AuthorsOpenReview ↗

Response to Reviewer BMGW – Part 3/3: Bootstrap confidence intervals and ranking alignment

CapabilityProxyBenchOSOS-Ind.BA-Lite IIDBA-Lite non-IID
Width0.120 [0.050, 0.195]0.315 [0.224, 0.400]0.273 [0.091, 0.545]0.374 [0.254, 0.487]0.484 [0.284, 0.714]
Height0.478 [0.361, 0.594]0.289 [0.196, 0.389]0.807 [0.747, 0.867]0.294 [0.194, 0.421]0.375 [0.181, 0.581]
Cardinality0.080 [0.016, 0.190]0.365 [0.287, 0.439]0.336 [0.105, 0.584]0.378 [0.299, 0.452]0.292 [0.131, 0.428]
Covariate Drift0.143 [0.076, 0.207]0.174 [0.114, 0.233]0.181 [0.126, 0.241]0.167 [0.102, 0.229]0.107 [0.027, 0.215]
Feature Entropy0.239 [0.172, 0.313]0.539 [0.461, 0.612]0.000 [0.000, 0.000]0.387 [0.267, 0.493]0.500 [0.300, 0.700]
Rare Classes0.096 [0.029, 0.177]0.791 [0.722, 0.855]0.116 [0.018, 0.388]0.736 [0.637, 0.821]0.560 [0.334, 0.772]
Imbalance0.150 [0.048, 0.256]0.661 [0.577, 0.740]0.031 [0.025, 0.316]0.654 [0.558, 0.742]0.647 [0.481, 0.777]
Label Drift0.193 [0.133, 0.253]0.193 [0.133, 0.253]0.193 [0.133, 0.253]0.193 [0.133, 0.253]0.093 [0.036, 0.223]
Data type0.015 [0.000, 0.037]0.671 [0.592, 0.750]0.273 [0.091, 0.545]0.760 [0.653, 0.840]0.900 [0.750, 1.000]
Target Entropy0.027 [0.002, 0.065]0.221 [0.144, 0.297]0.140 [0.024, 0.412]0.091 [0.012, 0.176]0.308 [0.114, 0.540]
Average0.154 [0.124, 0.187]0.422 [0.393, 0.450]0.235 [0.209, 0.324]0.403 [0.369, 0.437]0.426 [0.368, 0.492]

The bootstrapping results further validate this analysis, suggesting that even in the most pessimistic cases, the Total Variation Distance of ProxyBench from EGI-Bench is lower than that of the most optimistic configuration of the competitor benchmarks. All of these new results will be added to the Appendix.

Statistical confidence of ranking alignment

To complement the threshold sensitivity analysis, we quantified uncertainty in model ranking alignment using 1,000 bootstrap resamples. The table below reports average model ranks, rank differences, and Spearman correlations, with 95% bootstrap confidence intervals in square brackets. Rank differences are computed as the average rank on EGI-Bench minus the average rank on the comparison benchmark.

ModelEGI-BenchProxyBenchOS-Full
SAP-RPT-OSS2.1 [1.8, 2.3]-0.4 [-0.9, +0.0]-2.1 [-2.5, -1.6]
RealMLP2.3 [2.0, 2.6]-0.1 [-0.6, +0.3]-1.2 [-1.6, -0.7]
TabICL-Skrub2.5 [2.2, 2.9]+0.1 [-0.3, +0.6]+0.3 [-0.1, +0.7]
TabPFN-Skrub2.6 [2.3, 2.9]-0.6 [-1.0, -0.1]-0.2 [-0.6, +0.2]
Random forest3.7 [3.4, 4.0]-1.6 [-2.0, -1.1]-2.1 [-2.6, -1.6]
XGBoost4.8 [4.4, 5.2]-0.0 [-0.6, +0.5]-1.0 [-1.5, -0.5]
TabPFN5.5 [5.2, 5.9]+1.4 [+0.9, +2.0]+2.9 [+2.4, +3.4]
TabICL5.8 [5.4, 6.1]+1.6 [+1.1, +2.2]+3.0 [+2.6, +3.5]
Linear7.9 [7.7, 8.0]-0.6 [-0.9, -0.3]+0.4 [+0.1, +0.7]
KNN8.5 [8.2, 8.7]+0.4 [-0.0, +0.7]+0.3 [-0.0, +0.7]
Naive9.3 [9.2, 9.5]-0.2 [-0.4, -0.1]+0.1 [-0.2, +0.4]
Spearman0.873 [0.836, 0.909]0.555 [0.518, 0.655]

ProxyBench achieves a Spearman correlation of 0.873 with EGI-Bench, with a 95% confidence interval of [0.836, 0.909], compared with 0.555 [0.518, 0.655] for OS-Full. These results support the robustness of the observed ranking alignment.

Reviewer_tdc1OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes an interesting idea of constructing a proxy benchmark that demonstrates similar model ranking to internal benchmarks. It proposes to use a non privacy-leaking statistical "fingerprints" of existing datasets and proposes a procedure to filter a large set of publicly available datasets to most closely follow private dataset properties.

Critical strengths

I find the core idea of the paper fresh, interesting and potentially useful for future work. The problem of most industrial and "economically interesting" tabular data being proprietary is timely and I believe should be addressed. I find the approach proposed by authors interestng and having potential for future improvements.

Critical weaknesses

I think the potential of the "idea" (making an as close to real as possible benchmark, without publishing data) in the paper is much broader then it seems from the current positioning.

One thought that comes to mind is how a benchmark composition affects implicit modeling decisions - e.g. if the field is currently focusing on tabarena ELO and collectively optimizes it, it could harm enterprise use-cases underrepresented in the current benchmarks, how could we ensure a more diverse benchmarking landscape with such privacy preserving benchmarks, what is needed at this point? Do we need deeper statistical "fingerprints" for this to be more useful, or more enterprise data collections?

I don't have immediate suggestions for the experiments, more of a feeling that utilizing synthetic benchmarks to drive research into a more practical "basin" is interesting and has potential for further impact.

Another weakness is I think the procedure that utilizes "scraped" OS tabular datasets could be limiting as they have a certain bias. Looking into utilizing synthetic data (eitther sampled based on private datasets, or via SCMs from TFM literature) for the source of publishable data could be a useful addition.

Questions for authors
  • Have you thought about follow up's and potential for further research, beyond just a single static benchmark (maybe how to increase the coverage of practical tasks, beyond matching a single enterprise setting, or bridging with the synthetic data generation in tabular ML / differential privacy)?
  • It would be interesting how the benchmark correlates with BeoyndArena subsets
  • Would a filtering over a TabICLv2 prior work better or worse then the proposed OS datasets filtering procedure?
4 comments
AuthorsOpenReview ↗

Response to Reviewer tdc1 – Part 1/2: Future directions and comparison with BeyondArena

Thank you for the positive feedback and constructive input. We appreciate that you highlighted the relevance and scope of our contribution and potential directions going beyond our primary focus. In the following, we happily address your open questions:

Potential Follow-Ups

Thank you for highlighting the general importance of our idea and benchmark contribution. We agree that the community needs a broader selection of public benchmarks, including those mirroring characteristics of enterprise-grade ones, to further ensure and enable progress in the field. We believe that our contribution is a first step in this direction. Still, as you highlight, there is a need for continued work on this topic. One line of research revolves around data generation (see our answer for details below); another involves focusing on semantics and time components (cf. our answer to Rev. 4Caf) – both are valuable and important future directions, albeit out of the scope of our current work.

Statistical comparison with BeyondArena’s subsets

We compared the capability distribution of ProxyBench against BeyondArena (IID and non-IID separately) and will add the full capability comparison to the Appendix. To quantify uncertainty in the reported estimates, we computed 95% bootstrap confidence intervals using 1,000 bootstrap resamples. Each entry reports the point estimate followed by its confidence interval in square brackets.

CapabilityProxyBenchBA-Lite IIDBA-Lite non-IID
Width0.120 [0.050, 0.195]0.374 [0.254, 0.487]0.484 [0.284, 0.714]
Height0.478 [0.361, 0.594]0.294 [0.194, 0.421]0.375 [0.181, 0.581]
Cardinality0.080 [0.016, 0.190]0.378 [0.299, 0.452]0.292 [0.131, 0.428]
Covariate Drift0.143 [0.076, 0.207]0.167 [0.102, 0.229]0.107 [0.027, 0.215]
Feature Entropy0.239 [0.172, 0.313]0.387 [0.267, 0.493]0.500 [0.300, 0.700]
Rare Classes0.096 [0.029, 0.177]0.736 [0.637, 0.821]0.560 [0.334, 0.772]
Imbalance0.150 [0.048, 0.256]0.654 [0.558, 0.742]0.647 [0.481, 0.777]
Label Drift0.193 [0.133, 0.253]0.193 [0.133, 0.253]0.093 [0.036, 0.223]
Data type0.015 [0.000, 0.037]0.760 [0.653, 0.840]0.900 [0.750, 1.000]
Target Entropy0.027 [0.002, 0.065]0.091 [0.012, 0.176]0.308 [0.114, 0.540]
Average0.154 [0.124, 0.187]0.403 [0.369, 0.437]0.426 [0.368, 0.492]

Our analysis suggests that, while the overall Total Variation Distance (TVD) of ProxyBench is considerably smaller (0.154 against 0.403 and 0.426), the presence of non-IID data helps achieve better values on the covariate drift and label drift capabilities, suggesting that using non-IID split approaches could improve alignment on the drift-related capabilities of ProxyBench with respect to EGI-Bench and is relevant, non-trivial future work, as also discussed in our answer to Rev. 4Caf.

AuthorsOpenReview ↗

Response to Reviewer tdc1 – Part 2/2: Synthetic data and references

Synthetic priors as a source for benchmark datasets

There are, in our opinion, two main arguments against using synthetic data in our scenario:

First, our focus is to mirror the internal EGI-Bench at EntCorp. As we mention in the paper, EGI-Bench contains predominantly text/categorical features for the majority of its tasks. Synthesizing text-like/semantic categorical data is an open research question also in the context of pretraining text/semantic-native tabular foundation models (cf. [ConTextTab, TabSTAR]).

Second, using the same or similar priors for both benchmarking and training, e.g., adapting the public TabICL priors, introduces a bias towards models that are trained on them. Our selection process may still steer the overall benchmark into being similar to EGI-Bench, yet its semantic and structural characteristics would be driven by the data generation process. As we used open-source tables scraped from a large collection spanning multiple public heterogeneous sources, we believe the risk of implicit bias is much lower than when using synthetically generated data.

One potentially interesting future direction is training data generators, e.g., diffusion models, on real enterprise data, and then controlling the generation process, e.g., using techniques from differential privacy. However, the generation of semantic categorical / text fields is an open research question, and training individual generators on each task would be costly, requiring more research on efficient cross-table data generation (cf. [CTSyn, CDMD, TabGift]).

References

[ConTextTab] Spinaci, Marco, et al. "ConTextTab: A semantics-aware tabular in-context learner." Advances in Neural Information Processing Systems 38 (2025): 147005-147040.

[TabSTAR] Arazi, Alan, Eilam Shapira, and Roi Reichart. "TabSTAR: A tabular foundation model for tabular data with text fields." Advances in Neural Information Processing Systems 38 (2025): 172108-172161.

[CTSyn] Lin, Xiaofeng, et al. "CTSyn: A foundation model for cross tabular data generation." International Conference on Learning Representations. Vol. 2025. 2025.

[CDMD] Ketata, Mohamed Amine, Maximilian Schambach, and Stephan Günnemann. "CDMD: A Cross-Dataset Mixed-Type Diffusion Model for Tabular Data." arXiv preprint arXiv:2609.39124 (2026).

[TabGift] Li, Yu-Sheng, Jun-Peng Jiang, and Han-Jia Ye. "TabGift: A Context-Efficient Foundation Model for Tabular Data Generation." arXiv preprint arXiv:2610.16990 (2026).

As the rebuttal period is coming towards an end, we would kindly ask if we were able to resolve your questions or whether you had any further points we could clarify?

Thank you for following up! The comparison with the IID and non-IID BeyondArena subsets answered my main empirical question, I find the "Covariate Drift" difference particularly interesting and insightful.

I believe the paper is a valuable contribution in general, and keeping my score at 3

Comments on the paper

1 comment

We again thank the reviewers for their thoughtful feedback and the constructive discussion during the rebuttal! We appreciate their recognition of the novelty and practical value of constructing public benchmarks from statistical fingerprints, addressing the gap between public benchmarks and confidential enterprise data, and the broad empirical evaluation supporting ProxyBench.

Based on the reviewers' feedback, we believe we have further strengthened the evaluation and summarize our changes included in our updated paper (changes highlighted in blue) as follows:

  1. Statistical confidence in ranking alignment. We now provide 95% bootstrap confidence intervals using 1000 resamples. Results and tables have been updated accordingly.

  2. Sensitivity to fingerprint thresholds. We evaluated ±10% threshold perturbations on the curated tasks. ProxyBench’s average capability TVD remains consistently below all comparison benchmarks across the evaluated configurations. We also provided bootstrap confidence intervals for capability alignment.

  3. Comparison with BeyondArena. We provided separate comparisons with its IID and non-IID subsets. ProxyBench achieves substantially closer overall capability alignment with EGI-Bench, while the non-IID comparison identifies opportunities to improve drift-related capabilities in the future.

  4. Updated model coverage. We added evaluations of Kumo-Tabular and TabPFN-3.5 to the Appendix and explained the high-cardinality classification requirements affecting baseline applicability.

  5. Clearer scope and future directions. We clarified that our empirical claims concern alignment with EGI-Bench in the single-table classification setting. We highlight directions for future extensions of ProxyBench.

We believe these additions strengthen the evidence for ProxyBench as a useful public benchmark for the enterprise setting studied here and provide a clearer foundation for extending the construction approach to additional reference collections.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.