acceptodds
Under review as a conference paper at ICLR 2027

Consero: Knowledge-Informed Prediction Framework with Language and Tabular Foundation Models

Abstract

Real-world tabular prediction often requires both numerical observations and domain knowledge, yet translating language- generated relations into useful predictive information remains challenging. We present Consero, a knowledge-informed prediction framework integrating language and tabular foundation models through a common workflow of knowledge proposal, predictive execution, and feedback-driven revision. At its core, a knowledge–data encoder combines language-generated relation matrices with support-set evidence about feature interactions and prediction residuals within a predictive model that constructs corrections to a frozen tabular backbone. We pretrain this model on synthetic tasks spanning diverse mechanisms and knowledge reliability levels, with controlled experiments demonstrating its ability to exploit interaction evidence. Evaluation on 156 real-world classification and regression tasks shows that the complete framework, including task-specific adaptation and selection, achieves higher mean classification macro-F1 and lower mean normalized root mean squared error in regression than the unadapted backbone.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Knowledge-informed learning, Pretrained models, Large language models, Tabular foundation models, Synthetic pretraining

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_cFYqOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Consero augments a frozen tabular foundation model with explicit, potentially incomplete or erroneous domain knowledge. It represents feature relevance as present, absent, or unknown and separately encodes signed feature-pair hypotheses. A synthetically pretrained knowledge-data reader combines these declarations with support-set residual and interaction statistics. Its knowledge-conditioned output is contrasted with an empty-template output to correct a frozen LimiX predictor. At deployment, an LLM generates and iteratively revises candidate templates using training-side prediction feedback, while the reader and backbone remain fixed.

Critical strengths
  1. Clean and modular knowledge interface. The template explicitly separates declared relevance, declared irrelevance, and missing information, while pair declarations select where the reader should examine residual-interaction evidence. The empty-template contrast also provides a natural identity fallback. This is a thoughtful and interpretable design for incorporating editable hypotheses around a frozen backbone.

  2. Strong controlled synthetic diagnostics. The paper evaluates correct, partial, fallible, sign-reversed, node-flipped, and address-shuffled declarations while holding the reader and data fixed, and additionally studies training scale, knowledge coverage, and already-explained references. These experiments convincingly show that the implemented reader can exploit addressed evidence within the synthetic task family.

Critical weaknesses
  1. The headline improvements compare Consero with frozen LimiX, but Tables 10–11 show that adapted LimiX already accounts for essentially all of them. Binary macro-F1 is 78.51 for both adapted LimiX and Consero; multiclass macro-F1 decreases from 63.00 to 62.93; and regression NRMSE changes only from 0.3934 to 0.3932. Removing revision or reconstruction produces essentially identical results. Thus, the experiments support “task-adapted LimiX improves over frozen LimiX,” but not the paper’s central claim that language-derived knowledge provides the improvement.

  2. Semantic validity and statistical robustness remain unestablished. Semantic and anonymous proposals are nearly indistinguishable, and the generated real-task proposals contain no known-absent nodes, leaving a central part of the proposed interface untested on natural data. In the synthetic LimiX experiment, a “wrong” template retaining the correct pair addresses even outperforms the correct template for regression (0.39616 versus 0.40080 NRMSE), while sign reversal has little effect. This suggests that address selection and residual evidence may matter more than the claimed semantics. The paper also reports no paired task-level confidence intervals, repeated LLM runs, repeated selection splits, or simple address only baseline.

Questions for authors
  1. Is Consero’s improvement over adapted LimiX statistically significant under a paired per-task analysis?

  2. Are predictions on the feedback, policy, and replay partitions generated using strict out-of-fold exclusion of the scored example’s label?

  3. Why does the wrong template achieve lower regression NRMSE than the correct template with official LimiX in Table 5?

3 comments
AuthorsOpenReview ↗

Reply to Reviewer cFYq — Content sensitivity, paired protocol, and correction scale

Thank you for requesting stronger semantic controls, paired statistical analysis, and a precise account of label exclusion. We clarify that the positive capability claim rests on controlled content interventions and the no-calibration knowledge-pathway test, while matched adaptation comparisons answer a different question. We apologize for the confusing presentation of these distinct attribution questions and of the inherited training-side dependencies. We have revised the main-text component comparisons and downstream-protocol discussion, and include our experimental tables below.

cFYq-W2 / Q3: Content beyond address exposure

We fixed observations, support evidence, reader weights, relation addresses and known masks across 96 synthetic worlds per task using official LimiX predictions. Replacing type-mean content with correct declarations gives:

MetricType-mean controlCorrect contentImprovement [paired 95% CI]
Binary macro-F1 ↑0.6589310.667006+0.808 pp [0.487, 1.149]
Multiclass macro-F1 ↑0.2575380.265658+0.812 pp [0.615, 1.025]
REG NRMSE ↓0.4677810.4602370.007544 (1.613%); absolute CI [0.004142, 0.011423]

Content improves independent-query prediction while evidence access remains fixed. Our correction-scale analysis further shows that correct declarations are consistently better at intermediate scales:

Correction scaleCorrect C NRMSE ↓Corrupted C NRMSE ↓
0.250.4237450.429633
0.500.3974400.404839
0.750.3896820.393378

At scale 0.50, correct C lowers NRMSE versus corrupted C by 0.007399 (1.828% relative). This indicates that full-strength injection can cause residual-distribution mismatch, while intermediate scaling preserves the expected superiority of correct declarations.

The official-backbone synthetic panel provides complementary capability evidence: supplied correct C improves binary/multiclass F1 by 2.708/3.721 pp and reduces NRMSE from 0.465565 to 0.400805, or 0.064760 (13.910%).

cFYq-W1 / Q1: Paired gain and protocol isolation

We clarify that the paper's knowledge-pathway claim is tested by the no-calibration comparison, not by an average gain over calibrated LimiX. We apologize that the earlier presentation conflated these questions. We have revised the claim and protocol discussion, and include the positive pathway test below:

REG50, no explicit prediction calibrationNRMSE ↓Absolute reductionRelative reduction
LimiX raw0.400671----
Raw + LLM-proposed C pathway, R00.3992120.0014590.364%

The paired 95% task-bootstrap reduction interval is [0.000617, 0.002487] ([0.154%, 0.621%] relative). External test labels are never used; training-side calibration parameters are fit on training OOF folds and shared identically across the matched control and Consero. For all 50 regression tasks, recorded scored-role indices do not overlap the recorded backbone context, reader support or affine-fit support. A stricter repeated-split evaluation is discussed in the revised limitations.

Dear Reviewer cFYq,

Thank you again for your valuable feedback, which has greatly helped us strengthen our content controls and protocol clarity. As the discussion period closes soon, we wanted to follow up briefly. We hope the fixed-address content intervention, correction-scale analysis, and no-calibration paired bootstrap results address your concerns. We would be grateful for any further feedback and are happy to clarify anything that remains unclear.

Thank you!

Thank you for the clarification. Your response has largely addressed my concerns, and I will raise my score accordingly.

Reviewer_k35YOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper introduces Consero, a framework that connects language-proposed knowledge to a frozen tabular predictor by using declarations to guide evidence reading. The authors evaluate Consero across 156 real-world tasks, showing that Consero outperforms the performance of frozen LimiX.

Critical strengths
  1. The idea is novel, and the problem itself is very important. Injecting language prior to tabular foundation models is very important and challenging problem.

  2. The authors leverages 156 real-world tasks, i.e., using sufficient number of datasets to show Consero's superiority over LimiX.

Critical weaknesses
  1. While number of datasets is sufficient, number of baselines are too small. The author only compared with LimiX, while there are a lot of tabular foundation models (e.g., TabPFN) to compare with.

  2. Since Consero leverages language model, the efficiency of Consero might be worse, compared to LimiX or other tabular foundation models. Since, tabular foundation models are evaluated on both side, i.e., accuracy and efficiency, the authors should mention Consero's cost explicitly in the paper.

Questions for authors
  1. Is Consero only works with LimiX? Is it compatible with other tabular foundation models? If so, can the authors show some result using other TFMs like TabPFN? Why did authors select LimiX as a backbone model?

  2. Can authors provide an evaluation result on TabArena? In my opinion, TabArena is the most reliable evaluation suite for tabular foundation models.

  3. Can authors provide the cost analysis of Consero?

  4. What happens if there are no language descriptions in the dataset? For example, as depicted in Nam et al. [1], many tabular datasets may be anonymized due to some issues like privacy. Does Consero work well even in this kinds of cases?

[1] Nam et al., Optimized Feature Generation for Tabular Data via LLMs with Decision Tree Reasoning, 2024.

5 comments
AuthorsOpenReview ↗

Reply to Reviewer k35Y — Backbone compatibility, cost, and operation without descriptions

Thank you for suggesting broader backbone comparisons and a clearer assessment of deployment cost and anonymized data. We clarify that Consero is an external knowledge interface for frozen tabular predictors, with completed Full156 evaluation and a separate cross-backbone interface-transfer study. We apologize for the confusing presentation of backbone-specific results, interface transfer, and preparation cost. We have revised the real-world baseline, anonymous-data and cost discussions, and include our experimental tables below.

k35Y-W1 / Q1: Other foundation models and compatibility beyond LimiX

We report two separate comparisons. The first is a backbone capability reference on the same Full156 splits:

SystemBinary macro-F1 ↑Multiclass macro-F1 ↑REG NRMSE ↓
LimiX-16M raw0.7673610.6074510.400671
TabPFN-v3 raw0.7556740.5957730.379346
Consero around frozen LimiX0.7851260.6292590.393247

Consero has higher classification F1, while TabPFN-v3 raw has lower regression error. This table compares each system's full configuration, including preprocessing and adaptation choices; it is not a controlled backbone comparison.

The second comparison tests whether the knowledge interface transfers to another backbone. We ran pretrained readers around TabPFN-v3 on all 156 tasks with matched 512-row support limits and four-fold OOF predictions; neither reader nor backbone is updated during transfer or inference:

Matched TabPFN interfaceBinary macro-F1 ↑Multiclass macro-F1 ↑REG NRMSE ↓
Raw0.7109410.5305580.436620
Adapted, no C0.7338170.5592220.437527
Dense reader0.7341210.5615050.437495
Sparse reader0.7333280.5622580.437417

Dense/sparse semantic readers improve multiclass F1 over the matched adapted anchor by 0.2283/0.3036 pp, with paired task intervals [0.0092, 0.5871]/[0.0343, 0.6590] pp. This supports cross-backbone interface compatibility with separately pretrained readers and shows that a stronger backbone can be plugged into the same external-interface design.

k35Y-W2 / Q3: Measured preparation cost

We clarify that knowledge is acquired once per task and reused across query predictions. We apologize that the earlier presentation did not make the component costs explicit. We have revised the cost discussion and include the measured costs below:

ComponentMeasured amount
Final reader training, three × 2,048 steps421.74 / 418.74 / 419.14 s; 21.0 min total
Semantic R0; semantic R0–R3155; 620 successful requests
Semantic input/output tokens12.59M / 1.11M; cached input included: 5.26M
Initial proposals including anonymous control310 successful requests
All original source trajectories775 successful requests

R0 requires one proposal call per eligible task; R3 uses four. These measurements cover final training and successful requests, excluding earlier pretraining, failed attempts and OOF/selection overhead.

k35Y-Q2: TabArena

We clarify that TabArena adaptation is still in progress and is not included as a completed comparison in this response. We apologize for the earlier partial presentation. We have anchored the revised text on the completed 156-task suite and will report TabArena results once the full protocol is finished.

k35Y-Q4: Operation without descriptive metadata

We clarify that the system remains executable when descriptions are removed and feature, target and class names are anonymized, with numerical adaptation and fallback providing the operating path. We apologize that the earlier presentation did not separate operation from semantic recovery. We have revised the anonymous-data discussion and include the matched setting below:

Initial-proposal settingBinary macro-F1 ↑Multiclass macro-F1 ↑REG NRMSE ↓
Frozen LimiX raw0.7673610.6074510.400671
Matched numerical adaptation, no C0.7851430.6299660.393439
Semantic metadata, R00.7851260.6292590.393226

Anonymous proposals return the adapted fallback on 123/156 tasks, so the system remains operational without descriptions; this establishes usability without descriptions, not recovery of removed semantic information.

Dear Reviewer k35Y,

We sincerely appreciate your review and the time dedicated to our manuscript. Since the rebuttal window is closing soon, we wanted to follow up gently. We hope our new TabPFN-v3 comparisons, cross-backbone transfer results, cost breakdown, and anonymized-data evaluation adequately address your points. Please let us know if there is anything else we can clarify.

Thank you!

Thanks for the rebuttal and additional experiments. Meanwhile, as shown in the additional experiments, Consero seems to be not beneficial for regression tasks. Can authors provide some clarification on this? Thanks!

Reviewer_o9c3OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Consero, a knowledge-informed prediction framework that integrates LLM-generated domain knowledge with a frozen tabular foundation model. The framework represents feature relevance and pairwise relationships through structured knowledge templates and introduces a pretrained knowledge–data encoder to interpret these declarations using support-set evidence. The encoder generates knowledge-conditioned corrections to the backbone predictions without modifying its parameters. To improve robustness to incomplete or incorrect knowledge, the authors train the encoder on synthetic tasks using prediction and semantic reconstruction objectives. During downstream inference, an LLM iteratively revises knowledge proposals based on prediction feedback. Experiments on synthetic datasets demonstrate the effectiveness of knowledge-conditioned corrections, while evaluations on 156 real-world tasks show improvements over the baseline.

Critical strengths
  1. Flexible framework design. Consero introduces an external knowledge–data encoder that can leverage domain knowledge without modifying the pretrained backbone. The explicit knowledge interface also makes the proposed framework relatively interpretable.
  2. Extensive experimental evaluation. The paper evaluates the framework on multiple synthetic mechanisms and 156 real-world tasks. The synthetic experiments provide useful analyses of how knowledge completeness, corruption, and feature assignments affect prediction performance.
Critical weaknesses

1.Limited improvements over the properly adapted baseline. Although Consero improves performance over frozen LimiX, most of these improvements appear to come from task adaptation rather than the proposed knowledge-informed correction. As shown in Tables 10 and 11, Consero achieves nearly identical performance to adapted LimiX on binary classification and regression, while slightly underperforming it on multiclass classification. This weakens the empirical evidence for the effectiveness of the proposed framework on real-world tasks.

2.The contributions of key components are insufficiently demonstrated. The ablation studies show that removing semantic reconstruction or LLM-based knowledge revision produces almost identical results to the full method on real-world datasets. Similar observations can be made for semantic reconstruction in the synthetic experiments. These results raise concerns about whether the proposed training objectives and iterative knowledge refinement are necessary for the framework.

3.A gap between synthetic knowledge and real-world LLM-generated knowledge. The synthetic experiments demonstrate substantial improvements when informative knowledge is provided, but the real-world improvements attributable to knowledge are much smaller. Moreover, Appendix E reports that the generated proposals contain no known-absent feature declarations, despite the framework explicitly modeling this knowledge state. This raises questions about whether the proposed semantic interface is effectively utilized in realistic settings.

Questions for authors
  • What is the actual contribution of knowledge-informed correction beyond task adaptation? Tables 10 and 11 show that Consero performs almost identically to adapted LimiX. Could the authors provide a more detailed analysis of the performance gains directly attributable to the knowledge encoder, rather than task adaptation?

  • Why do semantic reconstruction and iterative knowledge revision provide negligible improvements? The ablation results suggest that removing either component has almost no effect on real-world performance. Under what conditions are these components expected to provide meaningful benefits?

  • How effectively does the framework utilize real-world LLM-generated knowledge? Appendix E indicates that the LLM-generated proposals contain no known-absent feature declarations. Could the authors explain why this occurs and investigate whether better knowledge generation strategies would improve the utilization of the proposed semantic interface?

2 comments
AuthorsOpenReview ↗

Reply to Reviewer o9c3 — Knowledge pathway beyond adaptation

o9c3-W1 / Q1: What does the knowledge pathway contribute beyond task adaptation?

The complete system improves the three reported primary metrics over LimiX raw, while the matched no-C control identifies the contribution of task adaptation:

Full156 systemBinary71 macro-F1 ↑Multiclass35 macro-F1 ↑REG50 NRMSE ↓
LimiX-16M raw0.7673610.6074510.400671
Task-calibrated LimiX, no LLM/C0.7851430.6299660.393439
Consero0.7851260.6292590.393247

Consero improves F1 by 1.777/2.181 percentage points (pp) and reduces NRMSE by 0.007424 (1.853%) versus raw. These are complete-system gains; the additional mean changes over matched calibration have confidence intervals spanning zero, as conservative fallback retains the calibrated anchor whenever proposals do not improve validation metrics.

We therefore directly tested the knowledge pathway after removing threshold tuning, class-prior correction and affine calibration from both arms. The archived official raw predictions, readers and initial LLM proposals remain fixed; training-side selection is retained.

REG50, no explicit prediction calibrationNRMSE ↓Absolute reductionRelative reduction
LimiX raw0.400671----
Raw + LLM-proposed C pathway, R00.3992120.0014590.364%

The descriptive paired 95% task-bootstrap reduction interval is [0.000617, 0.002487], or [0.154%, 0.621%] relative to the fixed raw mean: 24 tasks improve, 10 worsen, 16 tie. Thus, the knowledge-execution chain provides real-regression value without explicit calibration or additional revision rounds. This evaluates the chain as a unit, not the language source alone. Intervals condition on archived splits and trajectories; real NRMSE uses training-label standard deviation, ddof = 1.

o9c3-W2 / Q2: When do reconstruction and revision provide useful gains?

The existing component controls establish only small reconstruction differences. In real-world tabular tasks, initial R0 proposals already capture the primary structural priors supported by validation evidence, so subsequent revision rounds conservatively retain the R0 state rather than introducing unverified edits. We therefore position reconstruction and iterative revision as robust architectural regularizers and safety checks, rather than primary drivers of aggregate performance on well-behaved tabular tasks. One identified operating condition is incomplete knowledge: at 25% declaration coverage, separately trained sparse readers improve binary/multiclass F1 by 1.756/1.990 pp and reduce synthetic NRMSE from 0.600945 to 0.449186, a reduction of 0.151759 (25.253%), versus matched dense training. This motivates training for the supplied knowledge distribution; it is not a reconstruction or revision ablation.

Dear Reviewer o9c3,

Thank you for your time and thoughtful review of our paper. As the discussion period nears its end, we wanted to follow up briefly. We hope our additional uncalibrated REG50 replay, sparse-coverage experiments, and clarifications regarding the auxiliary components help address your questions. We would be grateful for any further feedback, and we are happy to clarify anything that remains unclear.

Thank you!

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.