- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes a framework for reusing heterogeneous tabular learnwares when user queries lack features required by available models. It aligns statistical specifications across models and uses TabPFN for missing-feature completion, followed by model retrieval, uncertainty-aware aggregation, and support-based abstention. Experiments on 41 tasks show improvements over LLM-based prediction and model-reuse baselines, with additional analyses of cross-model statistical relationships.
Critical strengths
- The paper addresses a meaningful problem of reusing heterogeneous tabular models without access to their original training data. Exploiting statistical relationships across model specifications to infer missing inputs is an interesting contribution with practical relevance.
- The core idea is supported by careful controlled experiments. Adding other models' specifications improves four of five task-group metrics with the predictor fixed. The relationship-shuffling and bridge-chain experiments further demonstrate the value of cross-model statistical information for feature completion.
Critical weaknesses
- Limited validation of the open-world setting. The model libraries are constructed from different row and feature subsets of the same datasets, using LightGBM throughout. Missing features are simulated mainly through random masking, while feature and target alignment are assumed. This setting does not fully capture independently developed models with different data distributions and realistic missingness patterns.
- Insufficient baselines and evaluation of key components. The main classification comparisons lack majority-class references and imbalance-sensitive metrics, making some results difficult to interpret, particularly on Amazon Access and DECOMP. Stronger alternatives appear mainly in ablations: mean imputation matches TabPFN completion on Financial tasks, while direct TabPFN-ICL slightly outperforms Ours. Moreover, the uncertainty-filtering rule is not fully specified, and Table 4 reports the best fixed retention size per task rather than directly validating query-dependent selection.
Questions for authors
- How does the framework perform with models trained on independently collected datasets or under non-random missingness?
- Can the authors clarify the uncertainty-filtering rule and compare it with fixed-size aggregation under identical settings?
- How is the abstention threshold selected without labeled user examples, and why does abstention provide limited benefits on some tasks?
- Can the authors report majority-class baselines and balanced accuracy for classification, and provide more direct comparisons with TabPFN-ICL and standard imputation methods?
5 comments
Response to Reviewer mLbm (Part 2/3)
3. "The main classification comparisons lack majority-class references and imbalance-sensitive metrics, making some results difficult to interpret, particularly on Amazon Access and DECOMP."
"Can the authors report majority-class baselines and balanced accuracy for classification, …"
We agree that accuracy alone is insufficient on imbalanced tasks. Below, the majority baseline always predicts the training-set majority class; it and Ours are evaluated on the same 200 queries. Balanced accuracy (BA) gives equal weight to each class's recall.
| Task | Majority accuracy (%) | Ours accuracy (%) | Majority BA (%) | Ours BA (%) |
|---|---|---|---|---|
| Amazon Access | 94.00 | 94.17 | 50.00 | 53.99 |
| DECOMP | 98.50 | 98.50 | 50.00 | 50.00 |
| IHM | 88.50 | 82.67 | 50.00 | 70.03 |
DECOMP's high accuracy reflects majority-class prediction, while the gain on Amazon Access is modest. IHM illustrates why BA matters: Ours has lower accuracy than the majority baseline but substantially better recall averaged across classes. Across all 26 classification tasks, Ours exceeds majority-class BA on 25 tasks. We have added class proportions and majority-class baselines, together with accuracy, BA, and score-based AUROC/AUPRC comparisons, to Appendix E.2.1 (Tables 13–15) of the revision.
4. "Stronger alternatives appear mainly in ablations: mean imputation matches TabPFN completion on Financial tasks, while direct TabPFN-ICL slightly outperforms Ours."
"…and provide more direct comparisons with TabPFN-ICL and standard imputation methods?"
Standard imputation. Revised Table 4 (submitted Table 3) already reports the Financial tie, alongside gains from TabPFN completion in the other four group metrics. We have extended this comparison with KNN and Bayesian-ridge iterative imputation, keeping retrieval, aggregation, queries, and masks fixed. General classification accuracy is 80.948% with TabPFN, versus 80.667% with KNN and 79.062% with Bayesian ridge. TabPFN has higher mean accuracy in all three classification groups and lower clinical regression RMSE than these two methods. We have added these comparisons to Section 5.4 (revised Table 4) and Appendix F.2 (Tables 23–24) of the revision.
Direct TabPFN-ICL. Revised Table 25 (submitted Table 13) compares direct target prediction from specification pseudo labels with our completion followed by model reuse; Ours is better on four of five group metrics. The Financial accuracy difference is 0.31 percentage points in favor of TabPFN-ICL. Our added classwise analysis shows a different balance: Ours has higher Financial BA (70.65% versus 62.58%) and minority-class recall (52.90% versus 30.97%), while TabPFN-ICL has higher majority-class recall. Across the benchmark, Ours has better accuracy/RMSE on 28 of 41 tasks, with two ties, and higher BA on 24 of 26 classification tasks. This supports retaining trained predictors after completion, while explaining the Financial accuracy exception. We have expanded this direct comparison in Appendix F.3 (revised Table 25 and Fig. 5) of the revision.
Response to Reviewer mLbm (Part 3/3)
5. "Moreover, the uncertainty-filtering rule is not fully specified, and Table 4 reports the best fixed retention size per task rather than directly validating query-dependent selection."
"Can the authors clarify the uncertainty-filtering rule and compare it with fixed-size aggregation under identical settings?"
Thank you for pointing this out. Eq. (6) defines as the mean normalized entropy of the missing inputs required by model . Query-wise selection retains models satisfying , where is the lowest retrieved-model uncertainty. For task-wise selection, let be the th-lowest retrieved uncertainty and compute over a batch of unlabeled user queries from the same task. We choose the largest shared satisfying ; each query then retains its own least-uncertain models.
The main benchmark uses this task-wise rule, with selected without query labels; both selection rules use . For each fixed , we retain the least-uncertain models for each query. All rules use identical queries, masks, completed inputs, and eight retrieved predictors. The table summarizes the ten illustrative tasks in revised Table 5. We report equal-weight means separately for the five classification and five regression tasks, with regression summarized by RMSE.
| Aggregation | Accuracy, 5 classification tasks (%) | RMSE, 5 regression tasks |
|---|---|---|
| All 8 models | 72.27 | 0.4957 |
| Fixed | 70.50 | 0.4997 |
| Fixed | 72.33 | 0.4938 |
| Fixed | 72.57 | 0.4946 |
| Task-wise selection | 73.17 | 0.4920 |
| Query-wise selection | 72.27 | 0.4935 |
On these illustrative tasks, task-wise selection gives the best mean performance among the compared rules. Query-wise selection has lower mean RMSE than the listed fixed-size controls (0.4935 versus 0.4938–0.4997), while its mean classification accuracy remains below fixed and . These results support the task-level rule used in the main benchmark and directly characterize the query-wise alternative. We have clarified both rules in Section 4.3 and expanded the comparison in Section 5.4 (Table 5) and Appendix E.3.1 of the revision.
6. "How is the abstention threshold selected without labeled user examples, and why does abstention provide limited benefits on some tasks?"
For a given feature set and preprocessing, the threshold specifies the low-support operating point. Without user labels, we inspect the candidate-context support-score distribution, place the cutoff at the lower-tail quantile that leaves the desired retained coverage (e.g., 90%) of scores above it, and fix that threshold before processing user queries. This provides a simple, label-free heuristic for selecting .
Support-based abstention targets errors associated with unfamiliar feature combinations. Errors that occur in familiar regions can retain high support and therefore pass the threshold, which makes the benefit task-dependent. This distinction may explain why abstention benefits some tasks more than others, consistent with positive mean AURC gains on 25 of 41 tasks (Section 5.4; Appendix E.3.2). We have clarified the label-free threshold choice in Section 4.4 of the revision.
We hope these clarifications and revisions address your concerns.
References
[1] Kaggle. Predict Future Sales, 2018. URL https://www.kaggle.com/competitions/competitive-data-science-predict-future-sales.
[2] Kaggle. M5 Forecasting - Accuracy, 2020. URL https://www.kaggle.com/competitions/m5-forecasting-accuracy.
[3] Kaggle. Corporación Favorita Grocery Sales Forecasting, 2017. URL https://www.kaggle.com/competitions/favorita-grocery-sales-forecasting.
[4] Zhi-Hao Tan, Jian-Dong Liu, Xiao-Dong Bi, Peng Tan, Qin-Cheng Zheng, Hai-Tian Liu, Yi Xie, Xiao-Chuan Zou, Yang Yu, and Zhi-Hua Zhou. Beimingwu: A learnware dock system. In Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pages 5773–5782, 2024.
[5] Jian-Dong Liu, Zhi-Hao Tan, and Zhi-Hua Zhou. Identifying and reusing learnwares across different label spaces. In Proceedings of the 34th International Joint Conference on Artificial Intelligence, pages 5734–5742, 2025.
[6] Lan-Zhe Guo, Zhi Zhou, Yu-Feng Li, and Zhi-Hua Zhou. Identifying useful learnwares for heterogeneous label spaces. In Proceedings of the 40th International Conference on Machine Learning, pages 12122–12131, 2023.
Response to Reviewer mLbm (Part 1/3)
Thank you for recognizing the practical value of our work and for finding the paper clear and informative. We address your concerns below.
Independently collected sources. Beyond the original controlled benchmark (Section 5.1), we use three independently collected sales-forecasting datasets, evaluating prediction on Predict Future Sales (PFS) [1] with models trained on M5 [2] and Corporación Favorita [3]. PFS contributes neither training data nor specifications to the model library. The library contains LightGBM, random-forest, and Ridge predictors from each training source. The table compares Ours with the strongest classical-completion baseline using the same six predictors; RMSE is computed on the common log-sales target. We have added this evaluation to Section 5.3 (revised Table 3), with the protocol and store-level results in Appendix E.2.3 (Table 21).
Ours reduces RMSE by 7.3% and 5.9%, respectively, with improvements at every target store. These results show that the framework can reuse models trained on independent sources despite differing input requirements.
Non-random missingness. Beyond the original random masking (Appendix D.3), we evaluate grouped MAR: the probability that a feature group is missing depends on an input that remains observed. We test weaker and stronger dependence on this observed input, keeping expected overall missingness at 30%. Specifications, queries, masks, and predictors are fixed across methods. The table reports the relative RMSE reduction from mean imputation to TabPFN, computed from means over three seeds for each task and averaged equally across four regression datasets.
Individual task-level RMSE reductions range from 3.8% to 25.9%. The completion benefits persist on all four datasets when missingness depends on observed features. We have included these experiments in Appendix F.6 (Fig. 7) of the revision.
The framework accesses tabular models through their prediction interfaces and statistical specifications, so it does not require a particular predictor architecture (Sections 3–4). To assess this generality, we additionally tested random-forest and Logistic/Ridge predictors across all 41 tasks, using the same feature-block completion setting and holding each predictor fixed across completion methods. Relative to mean imputation, TabPFN raises average classification accuracy by 2.32 and 3.65 percentage points for random forests and logistic regression, respectively; balanced accuracy and regression rank also improve for both families. These results support the framework's applicability beyond LightGBM, alongside the mixed-family library evaluated above. We have added this comparison to Appendix F.4 (Table 26) of the revision.
As formulated in Section 3, feature and target matching precede our query-time procedure. Existing methods support semantic feature matching [4] and matching or adapting models across different label spaces [5, 6]. They can provide upstream alignment when the required task information is available. Our focus is the subsequent challenge: completing missing inputs for individual queries through cross-model statistical relationships, without developer training data or additional user examples. We have clarified this interface in Section 3 and discussed the related work in Appendix A.2 of the revision.