acceptodds
Withdrawn from ICLR 2027

Zero- and Few-Shot Arbitrary Conditioning with Semantic and Predictive Context

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Generalist statistical inference requires answering conditional queries whose targets and observed features may change across previously unseen schemas. We study a zero- and few-shot arbitrary conditioning problem over semantically described tabular datasets, where observed feature–value pairs alone may leave the target distribution ambiguous. We introduce the Arbitrary Set-based Permutation-Invariant Reasoning Engine (ASPIRE), which draws on feature and dataset semantics, optional support examples, and external predictive context. ASPIRE represents observed values and external forecast probabilities as semantically grounded atoms, contextualizes co-occurring atoms with a shared permutation-equivariant module, and combines distinct sources of evidence through role-aware late fusion. Target-native likelihood heads return categorical probability mass functions or continuous densities; the same conditional interface supports chain-rule table generation. Trained on over 1,400 real-world datasets, ASPIRE leads the evaluated zero-shot comparisons. With five support examples, it achieves 12.7% higher F1 and 5.9% lower normalized RMSE than the strongest competing methods. External predictive context yields further gains, while the standalone conditional model also achieves the strongest five-shot table-generation fidelity among the evaluated methods.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
arbitrary conditioning, zero-shot learning, few-shot learning, conditional density estimation, in-context learning, permutation invariance

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_coKwOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper aims to solve the problem of estimating the distribution of target features from an arbitrary set of observed features in a zero-shot or few-shot setting for tabular data with unseen schemas. Previous research has proposed foundation models based on fine-tuning using labeled support data or in-context learning. However, these models struggled to operate in situations where no support data was available. This paper proposes a conditional distribution estimation framework called ASPIRE. ASPIRE consists of (i) an atom representation that semantically embeds features, (ii) an intra-instance set contextualizer that models feature relationships within the same sample, and (iii) role-aware late fusion that integrates information from different sources. For classification and regression, ASPIRE outputs categorical distributions via inner products with language embeddings and continuous distributions via a Gaussian mixture model, respectively. Furthermore, predictions from external models can be utilized at both the input and output levels. ASPIRE is trained by minimizing the negative log-likelihood via episodic training. In the experiments, prediction accuracy was evaluated using OpenTabs as a baseline, and the paper reports performance that surpasses existing baselines as well as the impact of each component.

Critical strengths
  • The paper proposes a model capable of zero-shot prediction of the feature distribution in tabular data with unseen schemas.
  • The paper designs an architecture that utilizes text features to create semantic embeddings of features and learns dependencies among features across heterogeneous schemas.
  • The paper experimentally demonstrates that ASPIRE can achieve performance that surpasses that of previous foundation models.
Critical weaknesses
  • Arbitrary conditioning has not been directly evaluated. While the central research question of the paper is the estimation of conditional distributions for arbitrary target variables and arbitrary sets of observed variables, the experimental evaluation was conducted only on OpenTabs' fixed table prediction problem. For example, it would be desirable to evaluate the proposed method when the target variable is changed, even for the same dataset, and to assess its robustness with respect to the number and combinations of observed variables.
  • It is unclear whether the zero-shot setting is truly "zero-shot." ASPIRE uses category candidate values and the domains of features as input metadata. If these are based on the actual dataset, data leakage occurs, and this setting cannot be called zero-shot.
  • There is a lack of evaluation of conditional probability distributions. The evaluation in the paper primarily measures the performance of downstream tasks and does not assess the validity of the estimated distributions. At the very least, evaluation using negative log-likelihood is necessary.
  • Late fusion and early fusion have not been compared. The paper argues that modeling that preserves the structure of each information source through role-aware late fusion is effective. This claim could be clearly evaluated by actually comparing it with early fusion, but as of now, the paper does not provide such a comparative experiment.
Questions for authors

See the critical weaknesses and address the concerns.

Reviewer_taJ7OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces ASPIRE, a framework for zero- and few-shot arbitrary conditioning on unseen tabular schemas. To resolve target ambiguity when labeled samples are scarce, ASPIRE represents semantic descriptions, support examples, and external forecasts as atom sets, processing them via a permutation-equivariant module, role-aware late fusion, and target-native likelihood heads. Pretrained on over 1,400 datasets, it outperforms baselines in zero-/5-shot classification, regression, and synthetic table generation, though extending beyond tabular data and improving interpretability remain future challenges.

Critical strengths

S1. Unified Arbitrary Conditioning Interface Across Unseen Schemas: Unlike traditional arbitrary conditioning methods restricted to a fixed schema known during training, ASPIRE extends conditional inference to previously unseen schemas by grounding feature-value pairs and external forecasts as semantic atoms. By combining a permutation-equivariant intra-instance module, role-aware late fusion, and target-native likelihood heads, a single model can seamlessly handle classification, regression, and autoregressive synthetic table generation.

S2. Strong Empirical Performance in Zero-/Few-Shot and Generative Tasks: Pretrained on over 1,400 real-world datasets, ASPIRE consistently outperforms existing tabular baselines across both zero-shot and 5-shot classification (F1) and regression (RMSE) benchmarks, while also achieving superior distributional fidelity (WNMIS and C2ST AUC) in 5-shot synthetic table generation.

Critical weaknesses

W1. Limited Evaluation of Role-Aware Late Fusion. While the paper presents several ablation studies, it does not directly compare role-aware late fusion with simpler fusion methods. Such comparisons would help clarify the contribution of this key design choice.

W2. Limited Robustness Analysis. Since semantic descriptions are important for adapting to unseen schemas, performance may degrade when descriptions are inaccurate or unclear. In addition, the paper does not sufficiently examine how poorly performing or biased external predictions affect the dynamic mixing mechanism.

W3. Lack of Computational Efficiency Analysis. Because the model uses multiple transformer layers, role-aware late fusion, and dynamic mixing with an external predictive model, its computational cost and memory consumption may be higher than those of simpler baselines. However, inference latency, throughput, and memory usage are not compared.

W4. Limited Analysis of Performance Consistency. The main results are reported as averages without standard deviations or confidence intervals. Per-dataset results would help determine whether the improvements are consistent across different tasks.

W5. Unclear Baseline Evaluation Settings. The paper compares models in a five-shot setting, but Appendix A.6 reports that some baselines use the full training set. It is unclear whether all models use the same five support examples, which raises concerns about the fairness of the comparison.

Questions for authors

[Clarification Requests]

Q1. Beyond empirical ablation studies, it would be beneficial to include a comparison with simpler fusion methods to better demonstrate the contribution of the role-aware late-fusion architecture.

Q2. In Table 3, combining hints from an external model significantly improves performance. However, the authors should clarify how robustly ASPIRE can defend against poorly performing or biased external predictions. Including stress-test results for degraded external predictions would strengthen the paper.

Q3. Given the complexity of the architecture and the integration of external models, adding a comparison table of actual inference latency, throughput, and memory usage against competing baselines is recommended.

Q4. Could the authors provide standard deviations or confidence intervals, along with per-dataset results, to show whether the performance improvements are consistent?

Q5. Appendix A.6 describes some baselines that use the full training set. Could the authors clarify whether all models used the same five support examples in the five-shot experiments?

[Minor Comments]

  • Consider placing Related Works before the Method section and adjusting the placement of Table 1 accordingly to improve the flow of the paper.

  • In Table 1, using a triangle symbol instead of an asterisk to indicate partial support would be more intuitive for readers.

  • Please add a dedicated Conclusion section to properly summarize the paper.

Reviewer_MHu8OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper aims to solve zero-shot and few-shot arbitrary conditioning on unseen tabular datasets. The authors propose ASPIRE, which combines semantic feature representations, permutation-invariant set modeling, support examples, and external predictions to perform classification, regression, and table generation.

Critical strengths

The works target a question that is critical and provide significant performance gain compared to previous works on different settings.

Critical weaknesses

Overall, I find the paper difficult to follow, and I am not fully convinced of its novelty and significance. The authors introduce many components, but it is hard to understand what the core contribution is and why the proposed design is necessary compared to previous works.

  1. The novelty and positioning are not clear enough. Many components, including semantic embeddings, Set Transformers, permutation invariance, and probabilistic prediction heads, are already well-established techniques. The authors discuss many related works, but I do not think they clearly explain what ASPIRE brings beyond combining these existing ideas. What is the actual new insight we learn from this paper?
  2. The writing is difficult to follow. There's too many terminology that assume readers should get it without providing enough explanation. Figure 2 is also not easy to understand by its own. What does the authors want to convey?

Overall, the paper feels overly complicated for what appears to be a combination of existing methods. Although the results are promising, I do not think the authors have sufficiently demonstrated what genuinely new understanding or capability their work provides. It would be my main criteria of raising the score. Please clarify this properly during the rebuttal.

Questions for authors
  1. What is the main contribution of ASPIRE beyond combining existing techniques? Could the authors more clearly distinguish their approach from the closest previous works?
  2. Could the authors provide experiments that directly evaluate arbitrary conditioning, especially when varying target variables and observed feature subsets?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.