- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper introduces ProxyBench, an open-source tabular classification benchmark containing 134 tasks designed to mirror real-world enterprise data. Because actual enterprise data cannot be released due to privacy and confidentiality constraints, the authors developed a privacy-preserving statistical fingerprint. They used this fingerprint to search a large pool of 270,000 public tables (RawTabs) to extract tasks that most closely match the statistical properties of a proprietary internal benchmark, EGI-Bench. Experimental results show ProxyBench produces model rankings significantly more aligned with enterprise data than existing open-source alternatives.
Critical strengths
- Innovative Privacy-Preserving Approach: The use of a statistical data fingerprint to bridge the gap between confidential enterprise data and open-source tables is a creative and highly practical solution to a major structural evaluation problem in the field.
- Strong Empirical Motivation: The authors convincingly demonstrate that current open-source benchmarks yield vastly different model rankings compared to actual enterprise data (Spearman's correlation of 0.555 vs 0.873), strongly justifying the need for ProxyBench.
Critical weaknesses
- Semantic and Relational Limitations: The similarity between ProxyBench and enterprise data is purely statistical (e.g., data types, entropy, cardinality), completely ignoring the semantic meaning, temporal dynamics, and linked relational structures that define complex enterprise databases.
- Lack of Temporal Splits: The benchmark relies on standard stratified random splits (I.I.D.) rather than time-based splits. This fails to capture the covariate and concept drift that is a primary challenge in real-world enterprise machine learning deployments.
Questions for authors
- Your analysis shows that ranking alignment drops significantly for task subsets with high feature entropy. Is the Zstandard compression method an insufficient proxy for capturing the complexity of these features, and how can this entropy estimation be improved?
- The proprietary reference dataset, EGI-Bench, is extracted from the internal data of a single large software vendor, EntCorp. How can you ensure that ProxyBench's targeted statistical characteristics generalize to enterprise data in completely different domains?
- The capability distribution for target class imbalance in ProxyBench is noted to be less aligned with EGI-Bench than the much smaller OS-Industry benchmark. What adjustments to the task selection and pruning pipeline are needed to better capture true enterprise class imbalance?
4 comments
Response to Reviewer 4Caf – Part 2/2: Entropy, generalizability, and class imbalance
Feature entropy estimation with Zstandard compression
Indeed, ranking alignment is lowest for the high-feature-entropy subset, which contains 32 tasks. The cause of this discrepancy remains unclear. It may partly reflect sampling variability due to the limited subset size, the greedy optimization and selection procedure, or bias in the underlying RawTabs collection.
Applying the same proxy across all tasks ensures a consistent measurement procedure, although it does not rule out limitations of the proxy itself. We chose this compression-based approximation because it accommodates heterogeneous data types while remaining computationally feasible at the scale of our collection. These practical considerations motivate its use.
Nevertheless, accurate and computationally efficient entropy estimation across heterogeneous data types remains an important direction for future work.
EGI-Bench generalizability
Indeed, ProxyBench is designed to mirror EntCorp’s internal EGI-Bench, which contains tasks spanning diverse lines of business. This diversity does not establish generalization to other organizations or domains. As also noted in our response to Reviewer BMGW, our empirical claims are therefore limited to alignment with EGI-Bench.
The construction procedure could, in principle, be applied to other enterprise reference collections. Validating its effectiveness across independently curated enterprise benchmarks is an important direction for future work.
Target class imbalance
Indeed, OS-Ind contains 11 classification tasks drawn from SALT and TabRed and achieves closer alignment with EGI-Bench on the imbalance capability than ProxyBench, with Total Variation Distances (TVDs) of 0.031 and 0.150, respectively. However, the 95% bootstrap confidence interval for OS-Ind is wide [0.025, 0.316], indicating substantial uncertainty in this estimate (see our response to Reviewer BMGW, “Choice of binning thresholds”, for the bootstrap analysis used to quantify uncertainty in the TVD estimates).
Furthermore, SALT and TabRed have TVDs of 0.253 and 0.747 for this capability. The low TVD of OS-Ind may reflect complementary capability distributions of SALT and TabRed, which produce a closer aggregate match when combined. Although OS-Ind obtains the better point estimate, ProxyBench remains substantially closer to EGI-Bench than the broader OS collection on this capability, with a TVD of 0.150 compared with 0.661.
A possible extension would be to introduce capability-specific weights into the fingerprint similarity measure, allowing prioritization of capabilities such as class imbalance. Whether this improves imbalance alignment without compromising alignment on the other capabilities would need to be explored in future work.
Thank you for the detailed responses and clarifications. Most of my concerns are acknowledged rather than resolved or supported by additional empirical evidence. However, I understand that updating the dataset is challenging given the nature and scope of this work.
Therefore, I will maintain my original score.
Thank you for your swift response! We agree to have acknowledged your valid concerns but also reasoned about why we believe them to be out of scope (e.g. EGI-Bench not being composed of relational problems) or currently infeasible (e.g. due to the lack of sufficiently enough public data with temporal structure). We thank you for raising these concerns and have clarified the scope of our contribution accordingly!
Response to Reviewer 4Caf – Part 1/2: Scope and temporal splits
We thank you for highlighting the relevance and empirical motivation/evaluation of our contribution, as well as important limitations. We address the concerns in the following:
Semantic and relational limitations
We agree that statistical matching alone does not establish semantic similarity between tasks. As a first step towards bridging the gap between confidential enterprise data and public benchmarks, we focused on statistical rather than semantic properties. Incorporating semantic descriptors into the matching procedure is a promising direction for future research that will require substantial work, as well as a suitable task-level semantic representation and an estimation of its impact on the overall ranking alignment with the reference benchmark.
Regarding relational structure, EGI-Bench consists of tabular tasks, not relational ones; as a consequence, both EGI-Bench and ProxyBench are evaluated as single-table classification benchmarks. Our matching procedure and model evaluation do not explicitly account for relationships across tables; that is out of the scope of our current contribution. The reported alignment therefore concerns this single-table setting. Nevertheless, extending the approach to relational prediction tasks remains an important direction for future work.
We will expand the discussion of these limitations in the revised paper, clarifying the scope of our results and the challenges involved in incorporating semantic and relational information.
Lack of temporal splits
Indeed, our current split procedure does not account for the temporal dimension, partly because suitable temporal information is limited in RawTabs. While temporal dynamics are important in enterprise applications, we do not consider them the primary challenge in the setting studied here: our evaluation highlights substantial differences from many public benchmarks in other characteristics, such as target cardinality, class imbalance, and entropy, and drift in our reference EGI-Bench is mostly low (see Figure 1).
We will make this limitation more explicit in the revised paper and discuss the need to extend both our internal reference collection and RawTabs with tasks involving temporal dynamics and distribution shifts, providing a basis for future extensions of ProxyBench.