- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper examines whether commonly used semantic ablations reliably quantify the predictive contribution of semantic knowledge in cross-table transfer. Its central argument is that comparing intended semantic knowledge against altered semantic content does not necessarily establish that the intended knowledge improves prediction. An altered control may perform poorly because the intervention introduces misleading information, rather than because the intended knowledge provides a positive benefit. To formalize this intervention, the authors introduce predictive utility and content sensitivity and study this.
Critical strengths
-
The paper clearly shows why comparing correct semantic knowledge with incorrect or shuffled knowledge can lead to misleading conclusions. If a model performs worse with incorrect knowledge, it does not necessarily mean that the correct knowledge improves prediction. The authors demonstrate this using well designed experiments. I consider this an important contribution for semantically grounded cross tabular models.
-
Another strength is that the authors evaluate their findings across different models, datasets, and experimental settings rather than relying on a single example. They conduct experiments using TabLLM on nine datasets, cross-table transfer experiments on clinical datasets such as NHANES, KNHANES, and HRS, and hydrological datasets such as CAMELS-US and CAMELS-GB. They also use a controlled benchmark where the correct semantic relationships are known. This makes the findings more convincing because the same evaluation problem appears in different settings.
Critical weaknesses
-
One important weakness is that although the paper argues that predictive utility should be measured by comparing correct semantic knowledge with a suitable reference, it does not clearly establish how researchers should choose that reference when multiple valid options exist. In Section 4.3, the authors evaluate 24 different references that replace feature names with anonymous identifiers. They find that the measured predictive utility changes depending on which reference is used, and for four of the nine datasets, the range of estimates crosses zero, although only one dataset shows a statistically confirmed sign change. This raises an important concern because two researchers could follow the proposed framework but reach different conclusions about how useful the same semantic knowledge is.
-
Another limitation is that the experiments primarily evaluate semantic knowledge by masking, permuting, or replacing feature names and modifying predefined semantic relationships. While these experiments are useful for demonstrating the difference between content sensitivity and predictive utility, they do not fully explore how the wording and quality of semantic descriptions affect model performance. Semantic descriptions can be improved through text optimization methods, such as automatic prompt optimization, iterative description refinement, or searching over semantically equivalent descriptions.
Questions for authors
-
How should researchers select an appropriate reference when different references produce different conclusions? In Section 4.3, the authors evaluate 24 different references and show that the measured predictive utility can vary significantly depending on the anonymous identifiers used. For four of the nine datasets, the range of utility estimates crosses zero, although only one shows a statistically confirmed sign change. If multiple references satisfy the proposed criteria but produce different results, how should researchers decide which reference to use? Is there a principled way to select or combine references rather than relying on an arbitrary choice? Clarifying this would strengthen my assessment because reference construction is central to the proposed framework.
-
How can we be sure that the performance differences are caused by semantic knowledge rather than how it is presented to the model? The experiments show that modifying or removing semantic information can change predictive performance. However, especially for language-model-based approaches such as TabLLM, the wording of feature names and descriptions may also influence predictions. Could some of the observed differences be caused by the model responding differently to particular text representations rather than the semantic information itself?
3 comments
Thanks for addressing my questions. I was wondering if there are any experiments on searching for discrete text using gradient-based optimization. I suppose that would be interesting and a good addition. In any case, I keep my positive score.
Thank you for the helpful suggestion. Following your comment, we conducted additional exploratory experiments on gradient-based discrete feature-name optimization using several datasets from our TabLLM reanalysis.
We found that this approach can produce promising results in some settings. For example, in one Heart setting, AUROC improved from 0.688 with fixed anonymous names to 0.812 with gradient-optimized names, while in one Blood setting, held-out NLL decreased from 0.605 to 0.567.
However, the observed improvements varied across datasets, optimization objectives, and search settings. These findings suggest that gradient-guided discrete text optimization is a promising direction for further investigation. Further investigation will help clarify when gradient-guided text optimization improves predictive performance and how these improvements relate to feature-name semantics.
We will include a brief summary of these exploratory experiments in Appendix B.1 (Exploratory Gradient-Guided Feature-Name Optimization) and discuss the remaining questions in the supplementary Limitations and Future Work section. We would appreciate it if you could refer to these additions in the updated supplementary material.
We sincerely appreciate your insightful suggestion and the valuable research direction it has highlighted.
Response to Reviewer L6VM
Thank you for highlighting the importance of reference choice and the distinction between semantic content and its textual realization.
W1/Q1 — Choosing and combining references. In the revision to be uploaded, Section 3.2 outlines a five-step procedure (with full protocol details, a 14-item verification checklist, and role-specific reference templates in the supplement): (i) specify the tested semantic component and preserved non-tested information; (ii) construct candidate removal references; (iii) verify Removal, Preservation, and Setup directly on the model's actual inputs; (iv) select a primary reference using a rule, fixed before seeing results, that prioritizes preservation of non-tested information; and (v) report the primary estimate alongside the range across candidate references fixed in advance.
To make disagreements across admissible references actionable, we apply a three-case decision rule: (1) if the 95% confidence interval against the pre-specified primary reference excludes zero, we state a claim conditioned explicitly on that primary baseline rather than claiming baseline invariance; (2) making a broad, reference-invariant claim requires consistent statistical support in the same direction across all evaluated candidate references; and (3) when different admissible references yield statistically significant effects in opposite directions, we report both bounds and flag the finding as reference-sensitive—never cherry-picking the most favorable baseline or averaging across baselines unless a weighting distribution over references is independently justified.
Section 4.3 illustrates this reporting format on TabLLM. In our original TabLLM experiment, the canonical primary reference was fixed before scoring in field order, and a subsequent 23-permutation sensitivity analysis evaluated additional dataset-seeded assignments. Across this 24-reference family, 4 of 9 datasets have point-estimate ranges crossing zero. Under simultaneous 95% CIs across all 216 dataset–reference comparisons, only Jungle exhibits statistically significant utility in opposite directions (+.025 against the primary vs. −.020 against another evaluated reference), whereas all 24 references support positive utility in Bank, Car, and Income.
W2/Q2 — Semantic content versus wording. We fully agree with your point on wording quality. Because we have not tested same-meaning paraphrases or optimized semantic descriptions, our conclusions apply specifically to the tested name and identity interventions rather than all possible textual wordings—and the same claim-matched framework could guide evaluations of prompt optimization, requiring controls that isolate optimized wording from formatting shifts.
Importantly, how column names affect predictions depends on how each learner ingests them. In XGBoost, column names are used solely during preprocessing to align matching source and target columns into the same feature positions; once aligned, the model sees only numerical feature matrices, so original names and shared anonymous IDs produce identical inputs by design. Even in models that directly embed column-name text, replacing shared anonymous IDs with original names yielded no statistically significant gain: In TransTab, none of the six evaluated settings showed a statistically significant positive gain from original names over shared anonymous identifiers; three instead showed a statistically significant negative contrast. In pretrained CARTE (ntarget=256), the difference was +.0002 AUROC (95% CI [−.0036,+.0038]). Together, these results suggest that, in our tested settings, the lexical meaning of the names added little beyond identifying which columns correspond across tables. In TabLLM (Section 4.3), an audit of 2,768,075 prompts confirmed exact preservation of feature values and task questions without truncation. In addition, a post hoc Jungle diagnostic in the supplement tested whether permutation sensitivity was merely an artifact of token-length differences in the original anonymous IDs. Even when we switched to fixed-width identifiers (slot_a–slot_f) that preserve exact token positions, changing which column received which ID still shifted performance by +.061 AUROC (simultaneous 95% CI [+.051,+.070]), with a significant mapping-by-vocabulary interaction (+.017, [+.007,+.026]). Thus, in TabLLM, part of any name contrast reflects the specific text realization of the reference, which is why we report the full reference family rather than a single anonymous baseline.