acceptodds
Under review as a conference paper at ICLR 2027

Beyond Correctness: Evaluating Semantic Knowledge in Cross-Table Transfer

Abstract

Semantic knowledge is increasingly used to bridge heterogeneous schemas in tabular learning, but how much does that knowledge actually improve prediction? Studies in tabular learning commonly answer this question through semantic ablations that modify or suppress the supplied semantic knowledge. We show that these ablations can lead to misleading conclusions about predictive benefit: poor performance under altered semantics may be taken as evidence that the intended knowledge is beneficial. Across real and controlled experiments, altering semantic content can produce large performance differences even when the model gains little predictive benefit from having that semantic knowledge in the first place. To separate these effects, we distinguish two quantities. Content sensitivity measures the change in performance when semantic content is altered, whereas predictive utility measures the benefit of the intended semantic knowledge relative to a suitable reference without that knowledge. This distinction motivates an evaluation framework in which the control is chosen according to the question being asked: altered controls assess sensitivity to semantic content, whereas claims that semantic knowledge improves prediction require a suitable reference. Even then, predictive utility is not fixed; it varies across suitable references and decreases when the reference can more easily recover the tested knowledge from other inputs or labeled examples. In a bounded audit of 25 semantic-ablation comparisons across nine studies, only one of 18 explicit predictive-utility claims is paired with a control that clearly isolates the tested semantic contribution. Together, these findings motivate a simple evaluation principle: semantic-ablation controls should be chosen and interpreted according to the question they are intended to answer.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
Cross-table transfer, Tabular learning, Semantic knowledge, Transfer learning, Model evaluation, Predictive utility

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_L6VMOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper examines whether commonly used semantic ablations reliably quantify the predictive contribution of semantic knowledge in cross-table transfer. Its central argument is that comparing intended semantic knowledge against altered semantic content does not necessarily establish that the intended knowledge improves prediction. An altered control may perform poorly because the intervention introduces misleading information, rather than because the intended knowledge provides a positive benefit. To formalize this intervention, the authors introduce predictive utility and content sensitivity and study this.

Critical strengths
  1. The paper clearly shows why comparing correct semantic knowledge with incorrect or shuffled knowledge can lead to misleading conclusions. If a model performs worse with incorrect knowledge, it does not necessarily mean that the correct knowledge improves prediction. The authors demonstrate this using well designed experiments. I consider this an important contribution for semantically grounded cross tabular models.

  2. Another strength is that the authors evaluate their findings across different models, datasets, and experimental settings rather than relying on a single example. They conduct experiments using TabLLM on nine datasets, cross-table transfer experiments on clinical datasets such as NHANES, KNHANES, and HRS, and hydrological datasets such as CAMELS-US and CAMELS-GB. They also use a controlled benchmark where the correct semantic relationships are known. This makes the findings more convincing because the same evaluation problem appears in different settings.

Critical weaknesses
  1. One important weakness is that although the paper argues that predictive utility should be measured by comparing correct semantic knowledge with a suitable reference, it does not clearly establish how researchers should choose that reference when multiple valid options exist. In Section 4.3, the authors evaluate 24 different references that replace feature names with anonymous identifiers. They find that the measured predictive utility changes depending on which reference is used, and for four of the nine datasets, the range of estimates crosses zero, although only one dataset shows a statistically confirmed sign change. This raises an important concern because two researchers could follow the proposed framework but reach different conclusions about how useful the same semantic knowledge is.

  2. Another limitation is that the experiments primarily evaluate semantic knowledge by masking, permuting, or replacing feature names and modifying predefined semantic relationships. While these experiments are useful for demonstrating the difference between content sensitivity and predictive utility, they do not fully explore how the wording and quality of semantic descriptions affect model performance. Semantic descriptions can be improved through text optimization methods, such as automatic prompt optimization, iterative description refinement, or searching over semantically equivalent descriptions.

Questions for authors
  1. How should researchers select an appropriate reference when different references produce different conclusions? In Section 4.3, the authors evaluate 24 different references and show that the measured predictive utility can vary significantly depending on the anonymous identifiers used. For four of the nine datasets, the range of utility estimates crosses zero, although only one shows a statistically confirmed sign change. If multiple references satisfy the proposed criteria but produce different results, how should researchers decide which reference to use? Is there a principled way to select or combine references rather than relying on an arbitrary choice? Clarifying this would strengthen my assessment because reference construction is central to the proposed framework.

  2. How can we be sure that the performance differences are caused by semantic knowledge rather than how it is presented to the model? The experiments show that modifying or removing semantic information can change predictive performance. However, especially for language-model-based approaches such as TabLLM, the wording of feature names and descriptions may also influence predictions. Could some of the observed differences be caused by the model responding differently to particular text representations rather than the semantic information itself?

3 comments
AuthorsOpenReview ↗

Response to Reviewer L6VM

Thank you for highlighting the importance of reference choice and the distinction between semantic content and its textual realization.

W1/Q1 — Choosing and combining references. In the revision to be uploaded, Section 3.2 outlines a five-step procedure (with full protocol details, a 14-item verification checklist, and role-specific reference templates in the supplement): (i) specify the tested semantic component and preserved non-tested information; (ii) construct candidate removal references; (iii) verify Removal, Preservation, and Setup directly on the model's actual inputs; (iv) select a primary reference using a rule, fixed before seeing results, that prioritizes preservation of non-tested information; and (v) report the primary estimate alongside the range across candidate references fixed in advance.

To make disagreements across admissible references actionable, we apply a three-case decision rule: (1) if the 95% confidence interval against the pre-specified primary reference excludes zero, we state a claim conditioned explicitly on that primary baseline rather than claiming baseline invariance; (2) making a broad, reference-invariant claim requires consistent statistical support in the same direction across all evaluated candidate references; and (3) when different admissible references yield statistically significant effects in opposite directions, we report both bounds and flag the finding as reference-sensitive—never cherry-picking the most favorable baseline or averaging across baselines unless a weighting distribution over references is independently justified.

Section 4.3 illustrates this reporting format on TabLLM. In our original TabLLM experiment, the canonical primary reference was fixed before scoring in field order, and a subsequent 23-permutation sensitivity analysis evaluated additional dataset-seeded assignments. Across this 24-reference family, 4 of 9 datasets have point-estimate ranges crossing zero. Under simultaneous 95% CIs across all 216 dataset–reference comparisons, only Jungle exhibits statistically significant utility in opposite directions (+.025+.025 against the primary vs. −.020-.020 against another evaluated reference), whereas all 24 references support positive utility in Bank, Car, and Income.

W2/Q2 — Semantic content versus wording. We fully agree with your point on wording quality. Because we have not tested same-meaning paraphrases or optimized semantic descriptions, our conclusions apply specifically to the tested name and identity interventions rather than all possible textual wordings—and the same claim-matched framework could guide evaluations of prompt optimization, requiring controls that isolate optimized wording from formatting shifts.

Importantly, how column names affect predictions depends on how each learner ingests them. In XGBoost, column names are used solely during preprocessing to align matching source and target columns into the same feature positions; once aligned, the model sees only numerical feature matrices, so original names and shared anonymous IDs produce identical inputs by design. Even in models that directly embed column-name text, replacing shared anonymous IDs with original names yielded no statistically significant gain: In TransTab, none of the six evaluated settings showed a statistically significant positive gain from original names over shared anonymous identifiers; three instead showed a statistically significant negative contrast. In pretrained CARTE (ntarget=256n_{\mathrm{target}}=256), the difference was +.0002+.0002 AUROC (95% CI [−.0036,+.0038][-.0036, +.0038]). Together, these results suggest that, in our tested settings, the lexical meaning of the names added little beyond identifying which columns correspond across tables. In TabLLM (Section 4.3), an audit of 2,768,075 prompts confirmed exact preservation of feature values and task questions without truncation. In addition, a post hoc Jungle diagnostic in the supplement tested whether permutation sensitivity was merely an artifact of token-length differences in the original anonymous IDs. Even when we switched to fixed-width identifiers (slot_a–slot_f) that preserve exact token positions, changing which column received which ID still shifted performance by +.061+.061 AUROC (simultaneous 95% CI [+.051,+.070][+.051, +.070]), with a significant mapping-by-vocabulary interaction (+.017+.017, [+.007,+.026][+.007, +.026]). Thus, in TabLLM, part of any name contrast reflects the specific text realization of the reference, which is why we report the full reference family rather than a single anonymous baseline.

Thanks for addressing my questions. I was wondering if there are any experiments on searching for discrete text using gradient-based optimization. I suppose that would be interesting and a good addition. In any case, I keep my positive score.

Thank you for the helpful suggestion. Following your comment, we conducted additional exploratory experiments on gradient-based discrete feature-name optimization using several datasets from our TabLLM reanalysis.

We found that this approach can produce promising results in some settings. For example, in one Heart setting, AUROC improved from 0.688 with fixed anonymous names to 0.812 with gradient-optimized names, while in one Blood setting, held-out NLL decreased from 0.605 to 0.567.

However, the observed improvements varied across datasets, optimization objectives, and search settings. These findings suggest that gradient-guided discrete text optimization is a promising direction for further investigation. Further investigation will help clarify when gradient-guided text optimization improves predictive performance and how these improvements relate to feature-name semantics.

We will include a brief summary of these exploratory experiments in Appendix B.1 (Exploratory Gradient-Guided Feature-Name Optimization) and discuss the remaining questions in the supplementary Limitations and Future Work section. We would appreciate it if you could refer to these additions in the updated supplementary material.

We sincerely appreciate your insightful suggestion and the valuable research direction it has highlighted.

Reviewer_QveSOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates an important methodological issue in evaluating semantic knowledge for cross-table transfer: whether performance differences caused by altering semantic inputs can be interpreted as evidence that the original semantic knowledge improves prediction. The authors distinguish between content sensitivity, defined as the performance difference between intended and altered semantic conditions, and predictive utility, defined as the improvement of the intended condition over a suitable reference that removes the evaluated semantic component while preserving the remaining information and experimental setup. To operationalize this distinction, the paper proposes claim-matched evaluation criteria based on Removal, Preservation, and Setup. The empirical evaluation includes a reanalysis of TabLLM, real-world clinical and hydrological transfer tasks, controlled experiments with known semantic structures, and a retrospective audit of published semantic-ablation studies. The results show that altering semantic information can lead to substantial performance degradation even when the intended semantic knowledge provides little additional predictive benefit over an appropriate reference. The authors also demonstrate that estimated predictive utility can vary across seemingly suitable references and generally decreases when the evaluated information becomes easier to recover from other inputs or labeled target examples. Finally, a bounded audit of 25 semantic-ablation comparisons from nine studies identifies only one clearly supported predictive-utility claim among 18 explicit claims, according to the proposed criteria. Overall, the paper provides a timely and empirically supported methodological perspective. However, important questions remain regarding the operational validity of suitable references, the incremental methodological contribution beyond existing control-based evaluation principles, and the reliability and representativeness of the retrospective literature audit.

Critical strengths

S1. A clear and practically important distinction between sensitivity to semantic content and the predictive benefit of semantic knowledge. The paper addresses a methodological problem that can affect the interpretation of semantic-ablation experiments. The distinction between intended–altered and intended–reference comparisons is conceptually sound. More importantly, the authors demonstrate that this distinction has practical consequences rather than treating it solely as a mathematical observation. For example, Table 3 shows that in the NHANES–KNHANES relation experiment, content sensitivity under reversed relations is +0.183, whereas predictive utility relative to the unconstrained reference is approximately −0.002. Replacing reversed relations with random relations reduces content sensitivity to approximately −0.001 without changing predictive utility. These findings provide convincing evidence that the measured effect of an altered semantic condition can largely reflect degradation induced by the control rather than an improvement attributable to the intended knowledge. The proposed claim-matched evaluation perspective is therefore useful for designing and interpreting future semantic-ablation studies. S2. A careful empirical investigation supported by robustness analyses and transparent reporting. A major strength of the paper is its effort to examine the proposed distinction across several complementary experimental settings. The authors combine released-code TabLLM reanalysis, clinical and hydrological cross-table transfer, and controlled experiments with explicitly defined semantic structures and outcome mechanisms. The experiments use paired evaluation conditions and report uncertainty estimates. The paper further examines 24 anonymous-name references, demonstrating that reference choice affects estimated predictive utility even when the tested feature-name semantics have been removed. Importantly, the authors distinguish point-estimate sign changes from statistically resolved changes. Section 4.3 reports that four datasets exhibit point-estimate ranges crossing zero, but only one has a resolved sign change under simultaneous confidence intervals. The supplementary material also documents analysis provenance, post-result experiments, model-specific limitations, and the distinction between findings supported by controlled experiments and mechanisms not established in real-data settings. This transparency substantially strengthens the credibility of the empirical analysis.

Critical weaknesses

W1. The framework clarifies the interpretation of performance contrasts but provides limited guidance for identifying a sufficiently informative and defensible reference in practice. Evidence: Sections 2.2, 3.2, and 4.3; Equations (1)–(3); Appendix C. My primary methodological concern is the gap between defining a suitable reference and establishing that a particular reference yields a sufficiently meaningful estimate of the semantic contribution under investigation. The paper defines predictive utility as U_R = Q_intended − Q_R, where R is a reference satisfying Removal, Preservation, and Setup. The decomposition in Equation (3) is an algebraic identity; its validity does not depend on the particular semantic-learning problem. The substantive methodological challenge is therefore the construction and validation of R, rather than the decomposition itself. The three proposed criteria provide useful necessary checks. However, they do not determine a unique reference, nor do they guarantee that different implementations satisfying these criteria will have comparable effects on the learner. The authors themselves demonstrate this limitation in Section 4.3 and Appendix C. Across 24 references that replace feature names with stable anonymous identifiers, estimated predictive utility varies substantially at the dataset level. For example, estimated utility ranges from −0.021 to +0.143 for Heart and from −0.040 to +0.045 for Credit-g. The paper appropriately acknowledges that only one of the four observed sign-crossing cases is statistically resolved. Nevertheless, this variation indicates that satisfying the proposed criteria does not eliminate important model- and representation-dependent effects. When several admissible references yield different utility estimates, it is unclear which estimate should substantiate a claim that semantic knowledge contributes to prediction. The recommendation to report reference sensitivity is valuable, but the framework does not yet provide a sufficiently developed procedure for selecting, validating, or aggregating references across different forms of semantic knowledge. This issue is particularly important beyond feature-name experiments, where removing correspondence or relation knowledge can change how a learner shares information across domains or imposes structural constraints. Furthermore, related literature on control tasks, probing, and information removal has already established that attribution claims depend on appropriate controls. The paper would benefit from a sharper explanation of what methodological capability its framework adds beyond applying these established principles to cross-table transfer. I encourage more operational guidance, including minimum reference-validation tests, reporting conventions when multiple references are admissible, and diagnostic experiments in settings where the intended semantic contribution can be independently characterized. Without such guidance, the framework is best understood as an important evaluation principle rather than a general procedure that uniquely identifies the predictive contribution of semantic knowledge. W2. The retrospective literature audit has limited reliability and representativeness, making its strongest numerical conclusion sensitive to the coding protocol and sampling unit. Evidence: Section 4.5; Table 4; Appendix F.1–F.4, especially Tables A19 and A20. The retrospective audit is interesting, but its evidential strength requires more careful qualification. It examines 25 comparisons from nine studies selected from a bounded candidate pool of 12 studies. The authors explicitly state that this is not a systematic literature review, which is appropriate. However, the headline observation that only one of 18 explicit predictive-utility claims is supported depends on the interpretation of the claims and the reference-suitability criteria. Appendix F.2 reports that initial agreement on Preservation was 15 out of 25 comparisons, with Cohen's κ = 0.265. After introducing six explicit checks, the coders agreed on 105 of 150 individual judgments (70.0%). Their independently derived support labels agreed for 15 of the 18 explicit predictive-utility claims (83.3%, κ = 0.693). These improvements are meaningful, and the authors should be credited for disclosing and addressing the initial disagreement. Nevertheless, the remaining differences indicate that the coding procedure is not entirely objective or trivial to reproduce. The study-level structure raises an additional concern: several comparisons originate from the same paper and may share similar experimental designs or interpretation patterns. For example, multiple unsupported predictive-utility comparisons are associated with LIFT and ConTextTab. Treating all 18 comparisons as equally informative observations may give disproportionate influence to studies contributing many related ablations. The statement that only one comparison is fully supported is accurate under the authors' final coding rules. However, it should not be interpreted as a statistically independent assessment of 18 research studies or as an estimate of evaluation quality across the broader literature. I recommend complementary study-level analysis, sensitivity to reasonable alternative coding decisions, and assessment of whether the result depends heavily on a few papers. Additional independent coding by reviewers not involved in developing the evaluation framework would further strengthen this finding.

Questions for authors

Q1. How should predictive utility be reported when several suitable references produce materially different estimates? Section 4.3 demonstrates substantial variation across 24 anonymous-name references, including one statistically resolved change in utility direction. Does the proposed framework recommend a primary reference selected according to an independent criterion, a range over admissible references, or another summary? Could the authors specify a reporting protocol that prevents reference selection from being influenced by resulting performance estimates? Q2. What additional methodological capability does the proposed framework provide beyond existing control-based evaluation principles? Equation (3) is an algebraic decomposition, and the need for carefully matched controls has been discussed in earlier work on probing and attribution. Could the authors clarify which aspects of their framework are new methodological contributions, rather than application and empirical validation of established control-design principles in cross-table transfer? A more explicit comparison with prior approaches would strengthen the novelty claim. Q3. Are the real-data relation experiments representative of well-supported semantic knowledge? Appendix A.5 reports that evidence supporting some directional relations is limited or context-dependent. For example, precipitation–discharge evidence largely concerns longer-term relationships, whereas prediction uses daily observations; the negative relationship between potential evapotranspiration and discharge is acknowledged as weak and not uniformly valid. Could the authors distinguish strongly supported relations from uncertain directional assumptions and report results separately? Would conclusions remain similar with independently validated relations and plausible semantic errors rather than deliberately reversed constraints? Q4. How stable is the literature-audit conclusion under alternative coding and aggregation procedures? Given the initial Preservation agreement of κ = 0.265 and the subsequent 70.0% agreement across six detailed checks, can the authors report how supported, unsupported, and indeterminate counts vary under reasonable alternative coding decisions? Could they provide study-level instead of comparison-level summaries and, if feasible, validation by independent annotators not involved in developing the framework? Q5. To what extent do the findings generalize across different learner architectures and forms of semantic input? XGBoost is the primary learner for cross-table transfer, while supplementary TransTab and CARTE analyses report heterogeneous results. In particular, CARTE uses a derived-value representation rather than directly replicating the directional constraints used with tree-based models. Could the authors clarify which empirical conclusions are expected to generalize across learner classes and which are specific to particular semantic representations and training procedures? An additional evaluation with a semantic-aware pretrained tabular model under matched intervention settings would help establish breadth.

3 comments
AuthorsOpenReview ↗

Response to Reviewer QveS (1/3)

Thank you for the careful discussion of reference validity and the paper's relation to control-based evaluation. This first reply addresses W1/Q1 and Q2; parts 2/3 and 3/3 address the literature audit, relation evidence, and representation learners.

W1/Q1 — Practical reference construction, validity, and reporting. We agree that our decomposition is an algebraic identity and that the substantive challenge lies in building a baseline that removes the tested semantic component without altering other predictive information or training configurations. In the revision to be uploaded, Section 3.2 details our five-step operational procedure (supplemented by a 14-item verification checklist and role-specific reference templates in the supplement): (i) specify the tested semantic component and preserved non-tested information; (ii) construct candidate removal references; (iii) verify Removal, Preservation, and Setup directly on the model's actual inputs; (iv) select a primary reference using a rule, fixed before seeing results, that prioritizes preservation of non-tested information (with a reproducible tie-breaking convention); and (v) report its point estimate alongside the range across candidate references fixed in advance.

Checking the actual model inputs is essential, especially for learners that embed column names. For instance, in TransTab, superficially distinct source and target column names still share WordPiece token fragments that reveal corresponding columns; replacing them with independently hashed IDs reduces counterpart recoverability from 100% to 0–43%, without establishing complete removal (General response 2). In TabLLM, Section 4.3 documents an audit of 2,768,075 prompts verifying preserved values and zero truncation, and our synthetic validation pilot in the supplement shows that omitting the value-preservation check admits a corrupted candidate in all 20 simulated datasets and, in four of them, produces a positive-utility claim that contradicts a nonpositive oracle utility, whereas the full checker rejects it in all 20.

When multiple candidate references pass these checks, we interpret the results using a three-case decision rule:

  1. Primary-conditional claim: If the 95% confidence interval against the pre-specified primary reference excludes zero, we report a claim conditioned explicitly on that baseline (e.g., that semantic names improve performance relative to the canonical field-order anonymous baseline), rather than asserting baseline invariance.
  2. Reference-invariant (family-wide) claim: Making a broad claim that holds regardless of baseline choice requires consistent statistical support in the same direction across all evaluated candidate references (as in Bank, Car, and Income, where all 24 anonymous permutations yield statistically significant positive gains).
  3. Conflicting directions: When admissible references yield statistically significant effects in opposite directions (as in Jungle, where original names score +.025+.025 AUROC against the primary reference but −.020-.020 AUROC against another valid permutation under simultaneous 95% CIs), we report both bounds and flag the conclusion as reference-sensitive—never cherry-picking whichever baseline favors a desired conclusion, nor averaging across baselines unless a weighting distribution over references is independently justified.

Q2 — Novelty relative to control tasks and attribution. In the revision to be uploaded, we will expand Section 2.2 (Related Work) to acknowledge foundational precedents in probing, information removal, and control tasks while contrasting them with cross-table interface failures. What we add beyond general control-task principles is: (1) identifying cross-table specific failure modes—where removing a semantic component disables a specific implementation (name tokens, recoding rules, column alignment, or constraints) and inadvertently alters training-row eligibility, sample weights, shared columns, or input width; (2) concrete verification checks and outcome-independent reporting rules tailored to measurement, correspondence, and relation roles; (3) empirical evidence showing that standard ablations both overstate and understate utility (TabLLM Credit-g vs. Bank) and that reference choice can flip supported directions (Jungle); and (4) evidence that utility depends on what the reference can recover, rising when the tested information is harder to infer and falling as labeled target rows accumulate (Section 4.4).

AuthorsOpenReview ↗

Response to Reviewer QveS (2/3)

W2/Q4 — Reliability and unit of the literature audit. In the revision to be uploaded, Section 4.5 frames our audit as a targeted assessment of claim–control alignment rather than a population-level prevalence estimate, and places the headline emphasis on the subset of claims for which both original coders separately identified the same concrete failures. After establishing six fixed checks to address the initially low Preservation agreement (κ=.265\kappa=.265), both original coders separately flagged the exact same 10 of 18 explicit utility claims as Unsupported and agreed on at least one failed criterion for each. These 10 claims span five of the eight studies making explicit utility claims.

Notably, five of these 10 fail solely on preserving the original representation of non-tested inputs. This requirement is critical: for example, omitting feature names converts a key–value input into a values-only input, penalizing the baseline for formatting rather than knowledge loss (Figure 2). While the total count of Supported claims varies from 1 to 7 depending on how indeterminate cases—studies that omitted implementation details needed to verify a check—are resolved (reflecting coding-rule sensitivity rather than seven independently verified controls), the 10 jointly identified failures do not depend on how indeterminate cases are resolved. We also explicitly note that the previously reported 70% check-level agreement and the lack of third-party recoding remain limitations.

Q3 — Evidence supporting intended relations. The fixed sign directions encode general clinical and hydrological associations. As explained in our reply to N3bg (W4), the authors specified them from general domain knowledge with a language-model coding agent and committed them in code before the reported runs; the literature documenting them was compiled afterward. To assess how strongly that documentation supports each applied direction, two researchers separately graded the 44 applied sign edges and channels (not 44 independent relation types) and resolved differences by discussion. None met a deliberately strict criterion of direct support in the evaluated conditioning context: all 29 CAMELS channels had conditional or scale-limited support, whereas 6 of 7 NH–KN and 4 of 8 HRS clinical edges lacked sufficient support in the examined documentation. These grades are a documentary review, not correctness judgments, and the archived runs cannot isolate a strong-evidence-only subset.

For the implemented constraints, Table 3 separates utility relative to the unconstrained reference from degradation under altered constraints. In NH–KN, the reversed constraints substantially degrade performance relative to the unconstrained reference (+.183+.183), while the intended constraints provide little additional benefit (−.002-.002). This is consistent with the possibility that relevant directional information is already recoverable by the unconstrained learner (Section 4.4), without establishing that the intended signs are correct in these cohorts. Replacing extreme reversals with random relations on non-documented variables shrinks the intended-versus-altered gap from +.183+.183 to −.001-.001 (on the same 60 endpoint–seed pairs as the utility estimate), demonstrating dependence on altered-control construction without establishing the typical impact of natural semantic errors. The closest evidence on relations known to be correct comes from our controlled benchmark, where intended relations are valid by construction: the reversal gap is large for additive outcomes across all four target weightings and matched tuning, whereas for pairwise and sparse outcomes, reversing the relation barely hurts performance below the unconstrained baseline, so most of the intended-versus-reversed gap reflects genuine predictive utility rather than an altered-control penalty.

AuthorsOpenReview ↗

Response to Reviewer QveS (3/3)

Q5 — Learners and types of semantic input. General response 2 details our new matched evaluation with pretrained CARTE alongside our controlled TransTab reanalysis. In the revision to be uploaded, Section 4.2 summarizes the main empirical implications of these experiments, while full model-specific setups and uncertainty analyses are provided in the supplement.

In pretrained CARTE on the matched NHANES–KNHANES (NH–KN) diabetes-history task, the contrast between intended names and domain-distinct anonymous names is unresolved at ntarget=256n_{\mathrm{target}}=256 (+.0036+.0036 AUROC), while altered names underperform the anonymous baseline (+.0054+.0054 AUROC, exploratory), with all name contrasts remaining small (<.01<.01 AUROC). These results show that intended–altered and intended–anonymous contrasts can differ in name-based representation learners. However, the anonymous contrast should not be interpreted as isolated predictive utility without further validation of semantic removal—for example, TransTab's domain-distinct identifiers still retain recoverable pairing cues via shared WordPiece token fragments.

The evaluation principle itself is learner-agnostic, whereas utility magnitudes, complete semantic removal, and relative rankings across architectures do not transfer automatically. In particular, because CARTE's relation interface incorporates derived values and text rather than XGBoost's directional constraints, matched directional-constraint experiments were not performed for CARTE.

Reviewer_N3bgOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper examines how to evaluate the contribution of semantic knowledge in cross-table transfer, distinguishing content sensitivity (performance changes when semantic content is altered) from predictive utility (the benefit of the intended knowledge relative to a suitable reference without it). Through TabLLM reanalysis, real-data and synthetic experiments, and an audit of existing studies, the authors show that altered controls can overstate the apparent benefit of semantic knowledge. They also find that estimated utility depends on reference choice and available target data. The paper proposes claim-matched evaluation and recommends reporting sensitivity to reference choice.

Critical strengths
  • S1. The distinction between content sensitivity and predictive utility is clear and practically relevant.

  • S2. The audit of published ablations is a valuable addition and shows that the issue arises in practice.

  • S3. Reporting is quite careful and detailed, although much of it is in the appendix (see W1).

Critical weaknesses
  • W1. I understand that page limits are a hard constraint, especially with so many experimental comparisons, and I appreciate the detailed appendix. Still, I find the main paper difficult to follow and not sufficiently self-contained. Details that are essential for understanding the results and assessing the soundness of the protocol are deferred to the appendix. These include how the semantic knowledge is constructed and applied, the model settings, the design of the synthetic benchmark, the confidence intervals for the main results, and the experiments with TransTab and CARTE (see W2).

  • W2. The limited coverage of TransTab and CARTE in the main text appears to me quite problematic, since these are the models that best fit the paper’s cross-table motivation. In the main real-data experiments, semantic knowledge is applied outside the model, by recoding and aligning the data or by constraining training, after which an XGBoost model is trained on the stacked tables. As described in the main text, this seems closer to harmonization followed by pooled training than to the semantic cross-table models discussed in the introduction and related work. TransTab and CARTE receive only a few sentences in the main text and are evaluated only in a synthetic experiment in the appendix, using reduced configurations. CARTE’s results also partly differ from the main patterns, and this is not adequately discussed in the main text. In general, the paper seems to address, at least in the main text, a somewhat different (and easier) problem than the one it claims to tackle.

  • W3. Model settings are not adequately reported or justified in the main text. Hyperparameters are fixed and no tuning is reported, and target rows receive different weights across settings, with no clear justification. This matters because utility appears to be clearly model-dependent (App. E.2.1).

  • W4. The construction of the semantic knowledge is under-reported and only partially validated. The main text explicitly attributes only correspondence to an LLM, but Appendix A.6 reveals that, for NH-KN, a single run of one proprietary LLM proposed both the correspondence groups and the relation candidates. The candidates were accepted based only on mechanical checks, without human semantic judgment, and no repeated runs or alternative models are reported for this step. The audits in Appendix A.6 are described as separate from this candidate-generation process and are not reported as validating the specific candidates used in the experiments. I also could not find information on who extracted the measurement codes, or whether the relations for HRS and CAMELS were also generated using the LLM. I kindly ask the authors to clarify this.

  • W5. The finding that utility estimates depend strongly on the choice of reference is valuable (even if not unexpected). However, beyond recommending that sensitivity to reference choice be reported, the paper offers little guidance on how to select a primary reference in advance, summarize utility estimates when suitable references yield different values, or interpret cases where they disagree on the direction of the effect. I recognize that these questions may fall partly outside the scope of the work, but more actionable recommendations would have strengthened the contribution. I would expect a full paper to address these practical implications more directly.

  • W6. If I understand the experimental design correctly, the largest gaps in Table 3, which provide the most striking evidence for the main claim, come from reversing relation directions, that is an altered condition constructed by the authors. In this setting, near-zero utility is expected, since the relations may already be learned from the source data, while forcing the model to learn the opposite of a real effect is likely to hurt performance. I therefore see these results as illustrating the proposed mechanism, rather than showing how much it matters in practice.

Questions for authors

Questions:

  • Q1. Could you provide more detail on how the semantic knowledge was constructed? (see the points raised in W4)

  • Q2. How were the hyperparameters and target-row weights chosen, and why does the weighting differ across settings? (W3)

  • Q3. Given that the main cross-table experiments use XGBoost, and that the supplementary CARTE results differ from some of the main findings, which conclusions do you consider supported for models that integrate semantic information into learned representations? (W2) Why not insert them in the main text?

Suggestions for future work/revision

  • Move the details needed to assess the protocol into the main text (W1).

  • Test how sensitive the results are to the knowledge construction, using several LLMs or repeated runs and an expert-validated set (W4).

  • Beyond the analysis in App. E.2.1, tune hyperparameters with the same procedure for each condition, and run ablations on the weights (W3).

  • Provide more actionable guidance on how to choose or build a suitable reference, especially for models that embed column names, as also noted in App. G (W5).

7 comments
AuthorsOpenReview ↗

Response to Reviewer N3bg (1/4)

Thank you for the detailed comments. We address the experimental setup, robustness analyses, and knowledge provenance across the following replies.

W1/W2/Q3 — Main-text coverage, CIs, and representation learners. To address W1, we moved key methodological and empirical details into the main text: knowledge-construction pathways (Section 3.3), target-row weighting conventions (Section 4.1), and weighting/tuning robustness plus a cross-learner comparison (Section 4.2). In addition, we will add 95% confidence intervals for predictive utility directly to Table 3 in the main text, allowing readers to assess the uncertainty around the near-zero point estimates, and we will add brief sentences in Section 4.1 explaining how additive, pairwise, and sparse functions test different relation structures.

Regarding W2/Q3, XGBoost serves as a testbed where Removal and Preservation can be checked exactly for shared-anonymous name comparisons because feature names only align columns (so intended and shared-anonymous inputs are identical). To evaluate models that embed column names, Section 4.2 adds a controlled TransTab comparison and a matched real-data evaluation with pretrained CARTE. In CARTE at ntarget=256n_{\mathrm{target}}=256, all name contrasts are small (<.01<.01 AUROC), yet altered names still underperform the anonymous baseline (+.0054+.0054, exploratory), directionally consistent with our evaluation principle. Regarding the original synthetic CARTE results that partly differed from the main pattern: at ntarget=32n_{\mathrm{target}}=32, CARTE's intended-name nMSE exceeded 1.01.0 (worse than predicting the mean) across all three outcome families, limiting interpretability of unresolved contrasts in that low-shot regime. Different input interfaces (TransTab sub-tokens, CARTE fastText vectors, XGBoost input width 14→2214 \to 22) show why step (iii) must check removal on each learner's actual inputs.

Informativeness — What the revised paper shows. Our framework separates robust effects from reference-conditional ones across three contributions:

  1. Diagnosis: Altered-control gaps diverge from predictive utility in both directions (TabLLM original ablation 7/9, relation reversal, audit 10/18).
  2. When knowledge helps: Section 4.4 shows that utility depends on what the reference can recover—increasing as information becomes harder to recover from observed inputs (up to +.26+.26 in negative nMSE) and decreasing as labeled target rows increase (TabLLM +.088→+.007+.088 \to +.007 AUROC, 0→5120 \to 512 adaptation examples).
  3. Practice: Our verifiable reference procedure separates solid conclusions (Bank, Car, Income) from conditional ones (Jungle) and catches cross-table interface violations such as input-width changes.
AuthorsOpenReview ↗

Response to Reviewer N3bg (2/4)

W3/Q2 — Hyperparameters and target-row weights. Across all evaluated XGBoost weighting and tuning configurations in NH–KN, NH/KN–HRS, and the controlled additive benchmark (9,120 target-weighting refits and 19,890 matched-tuning refits; Section 4.2), most of the intended-versus-altered gap still comes from poor altered-control performance (accounting for over 90% of reversed-control sensitivity). Thus, within these tested settings, the large altered-control penalty is not an artifact of a single target-row weight or fixed tree configuration.

In the revision to be uploaded, Section 4.1 documents our original settings: the fixed XGBoost hyperparameters were values set once in our pipeline without tuning (listed in the supplement), and labeled target rows received weight nsource/ntargetn_{\mathrm{source}}/n_{\mathrm{target}} in NH–KN (giving equal total weight to source and labeled target rows), a fixed weight of 1010 in HRS and CAMELS-120 (an untuned pipeline convention), and unit weight (11) in the controlled benchmark, with identical weights across all arms within each comparison. Our new sensitivity analyses evaluate four weighting schemes (11, 1010, 50%50\% balanced target share, and 80%80\% target-dominant share). Matched hyperparameter tuning searched the exact same tree-depth and learning-rate candidate grid via 3-fold cross-validation on the labeled target support independently for each condition, without using query rows for model selection. (CAMELS-120 was excluded from these refits within the rebuttal period).

Utility estimates themselves are more sensitive to learner settings. In the controlled additive family, matched tuning lowers utility from +.024+.024 to an unresolved +.006+.006 at ntarget=32n_{\mathrm{target}}=32 primarily because tuning enables the unconstrained reference baseline to catch up—consistent with the recoverability findings in Section 4.4. Because the pairwise and sparse outcome families do not exhibit this large reversed-control penalty, Section 4.2 restricts our primary controlled-reversal conclusion to additive outcomes and reports these bounds without generalizing them to all task structures.

AuthorsOpenReview ↗

Response to Reviewer N3bg (3/4)

W4/Q1 — How knowledge was built and checked. In the revision to be uploaded, Section 3.3 separates the knowledge-construction pathways used in our experiments (with full provenance logs in the supplement):

  • Correspondence proposals. A single stored LLM generation (GPT-5.6 Sol, following one empty attempt) proposed four NH–KN variable groups and one pulse-pressure arithmetic relation (used only in earlier automated-relation variants in the supplement, not Table 3). The authors checked the nine selected pairs against official codebooks for measurement identity and units; this author-led verification is separate from the automated schema checks and does not constitute external expert validation of the full proposal bank. There were no repeated successful generations or alternative generator models, our separate seven-model extraction audit does not validate this proposal bank, and no independent domain-expert validation was conducted.
  • Fixed sign rules used in Table 3. The clinical and hydrological sign directions were specified by the authors using domain knowledge with assistance from a language-model coding agent that implemented the rules, committed before the reported runs (after exploratory runs on the same cohorts). Supporting literature compiled afterward was graded by two researchers (none had direct evidence from the same clinical or hydrological setting) and does not verify whether the assumed directions hold conditionally in the study cohorts.
  • Measurement mappings and AI Use Statement. The authors compiled the NH–KN measurement mappings from official codebooks with coding-agent assistance. Separate LLM panels proposed alternative label mappings (including one post hoc variant) and alternative CAMELS sign directions (documented in the AI Use Statement and supplement, not used in Table 3).

W5 — How to choose and report references. Section 3.2 introduces a five-step procedure for reference construction and reporting (with role-specific reference templates and a 14-item verification checklist in the supplement, including checks that anonymous IDs share no token fragments and do not place a column nearest to its cross-domain counterpart in embedding space, diagnosing residual pairing cues). When multiple admissible references pass these checks, we apply a three-case decision rule: (1) if the 95% CI against the pre-specified primary reference excludes zero, we state a claim conditioned explicitly on that primary baseline rather than asserting baseline invariance; (2) claiming a broad, reference-invariant effect requires consistent statistical support in the same direction across all evaluated candidate references (as in Bank, Car, and Income, where all 24 permutations are significantly positive); and (3) when admissible references yield statistically significant effects in opposite directions (as in Jungle: +.025+.025 vs. −.020-.020 AUROC under simultaneous 95% CIs in Section 4.3), we report both directions and label the result reference-sensitive rather than cherry-picking a favorable baseline or averaging across baselines without an explicit weighting rationale.

Comments on the paper

4 comments
AuthorsOpenReview ↗

General response 1: Summary of responses and planned revision

We thank all three reviewers for their thoughtful and constructive feedback. Our revised paper preserves the central empirical distinction between content sensitivity and predictive utility while clarifying how reference baselines are constructed, validated, and reported.

Revised PDF. We will upload an anonymous revised PDF incorporating the changes below on November 17, 2026 (AoE), with all revisions marked in blue, and will post a brief notification once it is available. Section numbers in all our replies refer to this revised PDF.

1. Diagnosis: Altered-control gaps diverge from predictive utility. In TabLLM's original permuted-name ablation, permuted names perform significantly differently from the anonymous reference in 7 of 9 datasets, causing standard ablations to either overstate reference-based utility (Credit-g: sensitivity +.066+.066 vs. utility −.003-.003) or understate it (Bank: −.032-.032 vs. +.089+.089). In our relation experiments, reversed constraints induce a large gap even when unconstrained utility is near zero. Furthermore, in our bounded audit, both original coders separately identified the same 10 of 18 explicit predictive-utility claims as failing at least one reference check.

2. When knowledge helps: Utility depends on reference recoverability. Utility is larger when the reference has fewer ways to recover the tested information. Section 4.4 shows that utility increases as information becomes harder to recover from observed inputs (up to +.26+.26 in negative nMSE) and decreases as labeled target rows allow the reference to catch up (e.g., TabLLM +.088→+.007+.088 \to +.007 AUROC, 0→5120 \to 512 adaptation examples). Matched tuning similarly reduces utility in the controlled additive setting by strengthening the reference baseline. Moreover, whether a baseline genuinely removes semantic cues depends on how each learner processes inputs (e.g., column alignment in XGBoost versus sub-token or fastText embeddings in TransTab and CARTE).

3. Practice: A verifiable reference procedure. In the revision to be uploaded, Section 3.2 provides a five-step operational procedure: (i) specify the tested semantic component and preserved non-tested information; (ii) construct candidate removal references; (iii) verify Removal, Preservation, and Setup directly on the model's actual inputs; (iv) select a primary reference using a rule, fixed before seeing results, that prioritizes preservation of non-tested information (with a reproducible tie-breaking convention); and (v) report that primary estimate alongside the range across candidate references fixed in advance. Section 4.3 illustrates this reporting format on TabLLM, and the supplement (synthetic validation pilot) checks fault detection across 20 simulated datasets. This procedure separates robust conclusions (Bank, Car, Income) from reference-conditional ones (Jungle) and catches cross-table interface violations such as input-width changes and sub-token leakage.

Supporting validations and clarified scope. The revision to be uploaded adds a TransTab/CARTE comparison (Section 4.2, General response 2), target-weighting and matched-tuning refits (Section 4.2), and explicit knowledge provenance and audit sensitivity (Sections 3.3 and 4.5). We also clarify in Section 4.4 that the +.197+.197 AUROC clinical gain is a whole-pipeline contrast (which also changes training-row eligibility and effective weights) rather than an isolated label-polarity effect, report a width-matched check for derived-value utility in Section 4.4, and include corrected MLP and shared- versus distinct-anonymous TransTab descriptions in the supplement.

AuthorsOpenReview ↗

General response 2: Learners that embed column names (QveS Q5; N3bg W1/W2/Q3)

To address QveS Q5 and N3bg W1/W2/Q3 together, we summarize how our evaluation framework and empirical findings extend to learners that natively embed column names. Our main real-data XGBoost experiments incorporate semantic knowledge through schema alignment and directional constraints prior to pooled training, whereas TransTab and CARTE process column-name representations natively. In the revision to be uploaded, Section 4.2 summarizes these findings, with full setups and uncertainty analyses in the supplement. Throughout, we distinguish two anonymous baselines: a shared-anonymous reference replaces matched source and target columns with the same anonymous ID (preserving schema pairing), whereas a domain-distinct anonymous reference assigns separate anonymous IDs across domains (removing pairing).

Matched real-data CARTE. We evaluated pretrained CARTE on the NHANES–KNHANES (NH–KN) diabetes-history transfer task using the exact same intended, shared-anonymous, domain-distinct anonymous, and altered name sets, target-support draws (at ntarget=64n_{\mathrm{target}}=64 and 256256), and 9,200 query rows used in XGBoost. At ntarget=256n_{\mathrm{target}}=256, the results are directionally consistent with our distinction, though all name contrasts are small (<.01<.01 AUROC). The contrast between intended names and domain-distinct anonymous names is unresolved (indistinguishable from zero: +.0036+.0036 AUROC, 95% CI [−.0004,+.0074][-.0004, +.0074], Holm p=.152p=.152), whereas altered names perform worse than the anonymous baseline (+.0054+.0054 AUROC, pointwise 95% CI [+.0021,+.0091][+.0021, +.0091], exploratory). Thus, performance degradation below an anonymous baseline does not by itself establish positive utility for original names.

Controlled TransTab. At ntarget=32n_{\mathrm{target}}=32, the intended-versus-altered gap (+.49+.49 to +.58+.58 negative nMSE) exceeds the intended-versus-domain-distinct contrast (+.10+.10 to +.13+.13) by a wide margin. However, the domain-distinct IDs still share WordPiece token fragments that reveal corresponding columns; replacing them with independently generated hash names shifts the ntarget=512n_{\mathrm{target}}=512 contrast from unresolved across all three outcome families to positive (+.014+.014 to +.017+.017) relative to that hash-name baseline. Because hashing also changes identifier formatting, we cannot attribute the resulting difference solely to shared sub-tokens. Note that TransTab's learnable weights were trained from scratch, and the previously reported gap between shared-anonymous (SS) and domain-distinct (DD) IDs measures the effect of cross-domain column pairing (S−DS-D) rather than the utility of original names (I−DI-D, where II denotes intended names).

How each learner receives the reference matters. In XGBoost, domain-distinct IDs prevent column sharing, increasing input width from 14 to 22 columns. In CARTE, independently hashed names alter pretrained fastText vectors so that for none of the 8 renamed columns was its cross-domain counterpart the nearest fastText neighbor (a diagnostic check rather than proof of complete semantic erasure). A post hoc matched comparison on the same query rows at ntarget=256n_{\mathrm{target}}=256 shows that XGBoost's intended-versus-anonymous contrast exceeds CARTE's by +.0101+.0101 AUROC (95% CI [+.0045,+.0157][+.0045, +.0157], Holm p=.0012p=.0012), driven by the anonymous condition (−.0085-.0085) rather than intended names (+.0016+.0016). This represents an implementation-specific contrast rather than an architecture-wide ranking. Finally, because CARTE's relation interface incorporates derived values and text rather than XGBoost's directional constraints, matched directional-constraint tests were not performed for CARTE.

AuthorsOpenReview ↗

Revised Manuscript Uploaded

We have uploaded a revised manuscript with changes highlighted in blue for ease of review.

We will upload a clean version without revision highlighting, along with the updated supplementary material, before the November 18 (AoE) deadline.

AuthorsOpenReview ↗

Final submission

We have uploaded a revised clean version of the manuscript with supplementary files.

We appreciate all reviewers taking time to review our paper

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.