acceptodds
Under review as a conference paper at ICLR 2027

DiReCT: Diagnosing and Repairing Context Vulnerabilities in Tabular Foundation Models

Abstract

Robustness evaluation is essential for tabular foundation models (TFMs) because real-world tabular data commonly contain imperfections. Unlike marginal metrics, which can overlook heterogeneous vulnerabilities across queries, query-conditional evaluation can reveal fine-grained failure patterns. For in-context TFMs, these vulnerabilities can arise from the context itself: large contexts can contain irrelevant, corrupted, or distributionally mismatched examples. Also, robustness studies should be interpretable for better diagnosis and treatment. However, current evaluations largely measure marginal degradation, often focus on query-side perturbations, and rarely combine interpretable diagnosis and actionable repair. We address these limitations with a training-free framework, DiReCT. We characterize queries using interpretable descriptors, measure their heterogeneous sensitivity to context perturbations, and statistically confirm the reliability of descriptor–perturbation vulnerabilities. We map confirmed vulnerabilities to deterministic context repairs using available operators, moving beyond diagnosis alone. Across diverse tasks and two state-of-the-art TFMs, we uncover recurring query-conditioned defects, and the repair results show that confirmed defects can translate into useful interventions.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Tabular foundation models, context robustness, in-context learning, context perturbations, interpretable diagnosis

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_yUDLOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

In-context predictions of tabular foundation models (TFMs) can be sensitive to the quality of the provided context. In this work, the authors propose DiReCT, a training-free framework for diagnosing and repairing context vulnerabilities in TFMs. The method uses predefined statistical descriptors and artificial perturbations to identify vulnerable query groups, followed by statistical filtering and rule-based context repairs. The authors evaluate DiReCT on 19 RelBench tasks using TabICL and TabPFN, reporting improvements over the original contexts and several ablated policies.

Critical strengths
  • The framework is training-free and provides an interpretable approach to diagnosing and repairing context vulnerabilities.
  • The experiments include several ablations to evaluate the contributions of different components.
Critical weaknesses
  • Insufficient experimental evaluation. The method is only evaluated on two TFMs and one benchmark, with a fixed context size of 50. It is unclear whether the results generalize to larger contexts. More importantly, all baselines are internal ablations, without comparisons against existing context selection or retrieval methods. The reported gains are modest, and several tasks show performance degradation. Only relative loss changes are reported, making it difficult to assess the practical significance of the improvements.

  • Limited validation of the diagnosis-to-repair connection. The framework identifies vulnerabilities using artificial perturbations but applies repairs to naturally sampled contexts. While the ablation studies support the effectiveness of the diagnosis-guided policy, it remains unclear how reliably the identified sensitivities predict the benefits of specific repairs across different contexts. Moreover, several diagnosed perturbations have no corresponding repair operators.

  • Limited technical novelty. The method mainly combines existing techniques, including predefined perturbations, subgroup comparisons, statistical tests, and hand-designed context edits. While the combination is reasonable, the paper provides limited new methodological insights. Without stronger baselines, it is difficult to establish the additional value of this framework.

  • Presentation issues. The paper contains broken equation references, inconsistent terminology, and undefined symbols. Several important experimental settings, including perturbation strengths and repair budgets, are insufficiently specified. The conclusion also emphasizes recurring interactions across models under feature permutation, although the results show different interaction directions and limited recurrence.

Questions for authors
  1. Why was the context size fixed at 50? Since the proposed vulnerability descriptors depend on the context distribution, I wonder whether the identified vulnerabilities would change significantly with larger contexts.

  2. How does DiReCT compare with existing context selection or retrieval methods? The paper argues that existing approaches can be computationally expensive, but does not provide direct comparisons.

  3. How reliably do the diagnosed vulnerabilities predict the effectiveness of subsequent repairs? The ablation studies demonstrate the benefits of the overall diagnosis-guided policy, but I wonder whether stronger diagnosed vulnerabilities consistently correspond to larger improvements after repair.

  4. Could the authors report the absolute prediction performance for each task? The current results mainly focus on relative loss changes, making it difficult to understand the practical significance of the improvements. In addition, why does the repair policy lead to performance degradation on some tasks? Are there any common characteristics among these failures?

  5. How sensitive are the results to the experimental configurations?

5 comments

We thank the reviewer for the comments. They are very helpful.

W1: Limited models, benchmarks, and context sizes; missing external baselines, modest improvements, and insufficient absolute performance reporting.

Response

Thank you! To address these concerns, we have substantially expanded our experiments:

  • Added a third state-of-the-art TFM, Kumo Tabular (medium).
  • Expanded the evaluation to 24 tasks across eight databases, including Rel-Event.
  • Evaluated contexts containing up to 10,000 examples, capped by the available training samples.
  • Repeated diagnosis at M=50 for TabICL on 20 common tasks.
  • Added three budgeted external repair baselines: Facility Global, Facility Subgroup, both inspired by [1], and BS-kNN, the latter inspired by LoCalPFN [2].

Due to rebuttal-time constraints, the new large-context diagnosis covers 24 tasks on all three TFMs, while the repair comparisons cover 16 completed tasks on TabICL and Kumo.

To address the scalability limitations of the original submission, we introduce a shared-context formulation that eliminates the need to construct a separate context for every query. During diagnosis, validation queries share the same reference and perturbed contexts, enabling batched TFM evaluations. During repair, queries are grouped by their binary descriptor profiles, requiring at most 2J=162^J=16 distinct repaired contexts instead of one per query. Importantly, the mathematical formulation of the vulnerability contrast, statistical confirmation gates, and repair decision rules remains unchanged; only the context-sampling and implementation strategy, along with the corresponding notation, are revised. This substantially reduces redundant context processing while preserving DiReCT's core methodological principles. The following results all use the shared-context implementation.

Diagnosis results

We introduce three aggregate metrics to assess the framework's methodological properties:

  • MMR: Fraction of confirmed query-conditional interactions whose marginal effects fail the same practicality threshold.
  • DAR: Agreement between small- and large-context diagnoses, considering both-unconfirmed and both-confirmed interactions with matching directions.
  • DTR: Fraction of small-context confirmed interactions that remain confirmed with the same direction at large context sizes.

Table 1. Marginal Masking Rate (MMR).

ModelContext sizeMMR
TabICL5029.96%
TabICLLarge14.59%
TabPFNLarge31.65%
Kumo mediumLarge22.16%

The results demonstrate the importance of query-conditional evaluation: ~15–32% of confirmed interactions at large context sizes would be overlooked under the same marginal practicality criterion. Thus, conventional marginal evaluations cannot fully characterize heterogeneous context vulnerabilities.

Table 2. Context-size diagnostic agreement and transferability (TabICL).

Metric
DAR (M=50 vs. 10000)62.87%
DTR (M=50 vs. 10000)40.85%
Sign agreement among interactions confirmed at both sizes73.11%

The results show moderate overall diagnostic agreement across context sizes. Although not all small-context confirmations persist at larger sizes, 73.11% of interactions confirmed at both sizes preserve their direction. This indicates that some query-conditional vulnerabilities are consistent across substantially different context scales.

Table 3: recurring and model- or context-dependent vulnerabilities (full table in the revised paper).

InteractionTabICLTabPFNKumo-context-size comparisonM=50Large M
D3–P8 (tail-conditional noise)+5.08%+7.34%+4.76%-D2–P1 effect+15.17%+39.68%
D2–P2 (tail decrease)−10.55%+13.92%−3.80%-P9 confirmations3211

These results demonstrate how DiReCT captures both recurring and model- or context-dependent vulnerabilities.

Repair results

We compare DiReCT with three external baselines under comparable context-editing budgets.

Table 4. All-query primary loss change relative to reference (%; lower is better).

ModelConfirmed (main)UnconfirmedFacility GlobalFacility SubgroupBS-kNN
TabICL−1.50+3.91−0.19−1.19+24.43
Kumo medium−3.79−3.65+1.19−1.51+16.63

Confirmed achieves the best reported average loss change on both models. Moreover, confirmation reduces activation while improving or maintaining average performance. The BS-kNN results use a separate matched reference, so these comparisons are descriptive.

These results support our main objective: the diagnosed vulnerabilities provide actionable information that can guide beneficial, selective context interventions, without requiring full context optimization.

[1] Wei et al., Submodularity in Data Subset Selection and Active Learning, ICML 2015.

[2] Thomas et al., Retrieval & Fine-Tuning for In-Context Tabular Models, NeurIPS 2024.

W2: Insufficient evidence that diagnosed vulnerabilities predict repair benefits; some perturbations lack repair operators.

Response

As explained in Section 4.1 (line 209), we intentionally evaluate repair on naturally sampled, unperturbed contexts to reflect actual deployment, where the presence and type of context defects are unknown. Evaluating only artificially corrupted contexts would not establish whether the diagnosed vulnerabilities can guide beneficial interventions under normal inference conditions. Particularly, our confirmation gates are therefore important for deciding when intervention is warranted.

Regarding repair design, Appendix A.6 explains our two categories of operators: out-of-place sample replacement and in-place feature replacement. These simple, interpretable operators are designed to counteract or imitate the corresponding diagnostic perturbations. The diagnosis-to-repair connection is evaluated empirically rather than guaranteed mathematically.

For diagnosis-only perturbations, constructing reliable repairs may require identifying or reconstructing unknown corruptions, potentially introducing additional learning or assumptions. We intentionally avoid these mechanisms to keep the framework training-free and interpretable. Importantly, DiReCT is primarily a vulnerability diagnosis framework; repair is one demonstration of the actionability of its findings, not a requirement that every diagnosed perturbation admit a repair operator.

W3: Combines existing techniques without sufficiently establishing the value of the complete framework.

Response

We thank the reviewer for this comment. As discussed in the paper (line 64), to the best of our knowledge, DiReCT is the first training-free, query-conditional framework for diagnosing and repairing context vulnerabilities in frozen TFMs. Prior work largely relies on marginal, query-centered evaluations that overlook interpretable, query-conditional vulnerabilities, while existing mitigation strategies often require expensive context selection or extensive training.

Our novelty lies in formulating context robustness as an interpretable, statistically confirmed query-conditional diagnosis problem and connecting the resulting findings to actionable interventions. The individual components are deliberately simple to preserve interpretability and isolate their methodological contributions.

The original submission already includes five component ablations to give methodological insights, while the three newly added external budgeted baselines provide additional evidence that the diagnostic information supports useful interventions beyond generic context modifications, as shown in W1.

W4: Broken references, inconsistent notation, insufficient experimental details, and overstated cross-model consistency.

Response

We thank the reviewer for identifying these issues. We will carefully revise the equation references, terminology, and undefined symbols throughout the manuscript.

The appendix already contains dedicated sections on descriptor design, perturbation definitions, confirmation gates, and repair mechanisms. However, we agree that important experimental configurations should be more accessible. We will provide a compact summary of perturbation strengths, repair budgets, and task-specific settings in the revised manuscript, with complete implementation details available in our released code.

We also agree that recurring interactions should not be confused with consistent interaction directions. As our new experiments further demonstrate, some vulnerabilities recur across models with similar directions, while others exhibit model-specific responses. We will revise the conclusion to distinguish these cases and avoid overstating cross-model consistency.

Q1: Do diagnosed vulnerabilities remain consistent with larger contexts?

Response

We initially used M=50 to control the computational cost of query-specific context construction. To address this concern, we introduced shared-context diagnosis and evaluated contexts containing up to 10,000 examples. Our results, as responded to W1, show some vulnerabilities persist across context sizes, while others change. These findings reinforce the importance of evaluating context robustness under different context configurations.

Please see the next comment for the continuation of our response.

Q2: How does DiReCT compare with existing context selection and retrieval methods?

Response

We have added three external budgeted baselines: Facility Global, Facility Subgroup, BS-kNN. As shown in Table 3 of W1, Confirmed achieves the best reported average loss change among the evaluated policies on both TabICL and Kumo.

Importantly, context repair should not be mistaken for the main contribution of DiReCT. Its purpose is to demonstrate that diagnostic findings are actionable, rather than to develop a state-of-the-art context optimizer. Unlike unrestricted retrieval methods that construct contexts from scratch, our setting starts with an existing reference context and permits only budget-constrained modifications. This preserves the original context-selection policy and supports shared-context inference.

Therefore, the added baselines follow the same budgeted repair objective, allowing us to assess whether diagnosis-guided interventions provide practical benefits beyond generic context modifications.

Q3: Do stronger diagnosed vulnerabilities lead to larger repair improvements?

Response

We thank the reviewer for this question. We clarify that the diagnosed interaction VjpV_{jp} measures the relative perturbation sensitivity between query subgroups, rather than the expected magnitude of improvement from a repair. A larger interaction (e.g. stronger diagnosed vulnerabilities) therefore does not necessarily imply a larger repair benefit. Moreover, diagnostic reliability depends on the choice of perturbations, validation sample size, and context-sampling repetitions; increasing perturbation strength instead measures robustness to different corruption levels. Our new diagnosis experiments use 20,000 validation queries and 12 context-sampling repetitions to support reliable estimation. Importantly, DiReCT uses statistically confirmed interactions to guide whether and how to intervene, rather than predict the magnitude of repair gains. As shown in W1, the repair results demonstrate the actionability of these diagnoses, although they do not establish a monotonic relationship between vulnerability strength and repair improvement.

Q4: What are the absolute prediction results, and why does repair degrade performance on some tasks?

Response

We thank the reviewer for this suggestion. In addition to relative loss changes and 95% confidence intervals, we provide the following aggregate absolute performance results using the setting and implementations used in the original submitted draft.

Table 4. Classification performance.

ModelMetricReferenceConfirmedUnconfirmed
TabICLAccuracy (%)66.8767.9167.07
TabICLPredictive entropy0.74930.71440.7172
TabPFNAccuracy (%)68.1068.7667.53
TabPFNPredictive entropy0.75700.73710.7292

Confirmed improves mean classification accuracy on both models, although aggregation across heterogenous tasks might have led to small overall improvements. These results complement the task-relative loss changes and we will add them to the paper. Moreover, the original Figure 3 reports 95% task-bootstrap confidence intervals, with Confirmed's aggregate intervals entirely below zero in both evaluation populations for the two original TFMs. This supports beneficial average repair outcomes, although it does not guarantee improvement on every individual task or query.

Q5: How sensitive are the results to the experimental configurations?

Response

We have extended the sensitivity analysis to context size, comparing shared-context diagnosis at M=50 and up to M=10,000, as reported in W1. We acknowledge that additional sensitivity analyses of descriptor cutoffs, confirmation parameters, and repair budgets would further strengthen the evaluation. Due to the computational cost of these experiments, these additional ablations have not yet been completed, and we will investigate them in the revised work.

Reviewer_zVWLOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

DiReCT proposes a query-conditional evaluation protocol that compares matched loss changes under controlled context perturbations across interpretable query subgroups. It filters the resulting vulnerability matrix through statistical and quality checks, then maps eligible interactions to deterministic context repairs while keeping the model frozen.

Critical strengths
  1. The matched, query-conditional design makes the diagnosis informative. Comparing reference and perturbed contexts for the same query helps separate context sensitivity from baseline query difficulty and context-sampling variation. Grouping these loss changes by interpretable query properties reveals vulnerabilities that aggregate metrics can obscure, providing a clearer account of which queries are affected and under what conditions.

  2. The validation tests whether controlled diagnosis has practical value. The protocol screens interactions for statistical and practical significance, then evaluates the resulting repairs on held-out queries with naturally sampled contexts.

Critical weaknesses
  1. The repair rule would benefit from a clearer justification. In Equation (5), when a perturbation benefits both groups, the negative branch applies it only to the group that benefited less during diagnosis.

  2. The comparisons leave the value of the full pipeline unclear. The baselines mainly modify components of DiReCT. Including simple alternatives, such as nearest-neighbor context selection under comparable budgets, would help establish whether the gains require the proposed diagnosis or can be achieved through more direct context improvements.

  3. Figure 3 does not clearly establish the advantage over other policies. From my understanding, with the huge error bars at Figure 3 and overlapping CI95 results, it would make the evidences and pattern worse.

  4. Scalability is a issue here in this paper. All experiments use 50-example contexts and a fixed top-20% descriptor cutoff.

Questions for authors

Please fix the issues in weaknesses section, especially about scability and error bars ambiguous pattern.

3 comments

We thank the reviewer for the comments. They are very helpful.

W1: The negative repair branch targets the subgroup that benefits less from a perturbation, which requires clearer justification.

Response

Thank the reviewer! This behavior is intentional and follows from our query-conditional diagnosis objective. Estimand Vjp=A(1)jp−A(0)jpV_{jp}=A^{(1)}{jp}-A^{(0)}{jp} measures relative subgroup sensitivity, rather than absolute perturbation benefits. When Vjp<0V_{jp}<0 and both effects are negative, A(1)jp<A(0)jp<0A^{(1)}{jp}<A^{(0)}{jp}<0, both groups benefit, but the upper-tail group benefits more. The negative branch targets the complementary group, while Ajp(0)<−εA^{(0)}_{jp}<-\varepsilon ensures that the native perturbation is still beneficial for this group.

Selecting the subgroup with the largest absolute benefit would pursue a different objective, closer to our population-level πmrg\pi_{\mathrm{mrg}} baseline, for which Confirmed (main repair method) showed more reliable interventions with lower activation rates in the original experiments. Our repair policy is intended to show the actionability of query-conditional diagnosis, not to maximize context optimization performance. We will clarify that Eq. (5) defines a deterministic, diagnosis-aligned deployment rule rather than an optimal intervention policy.

W2: Comparisons mainly rely on internal ablations, leaving the advantage over simpler context-selection methods unclear.

Response

We thank the reviewer for this suggestion. To address your concern directly, we have substantially expanded our experiments:

  • Added a third state-of-the-art TFM, Kumo Tabular (medium).
  • Expanded the evaluation to 24 tasks across eight databases, including Rel-Event.
  • Evaluated contexts containing up to 10,000 examples, capped by the available training samples.
  • Repeated diagnosis at M=50 for TabICL on 20 common tasks.
  • Added three budgeted external repair baselines: Facility Global, Facility Subgroup, both inspired by [1], and BS-kNN, the latter inspired by LoCalPFN [2].

The new large-context diagnosis covers 24 tasks on all three TFMs, while the repair comparisons cover 16 completed tasks on TabICL and Kumo.

To address the scalability limitations of the original submission, we introduce a shared-context formulation that eliminates the need to construct a separate context for every query. During diagnosis, validation queries share the same reference and perturbed contexts, enabling batched TFM evaluations. During repair, queries are grouped by their binary descriptor profiles, requiring at most 2J=162^J=16 distinct repaired contexts instead of one per query. Importantly, the mathematical formulation of the vulnerability contrast, statistical confirmation gates, and repair decision rules remains unchanged; only the context-sampling and implementation strategy, along with the corresponding notation, are revised. This substantially reduces redundant context processing while preserving DiReCT's core methodological principles. The following results all use the shared-context implementation.

Diagnosis results

We introduce three aggregate metrics to assess the framework's methodological properties:

  • MMR: Fraction of confirmed query-conditional interactions whose marginal effects fail the same practicality threshold.
  • DAR: Agreement between small- and large-context diagnoses, considering both-unconfirmed and both-confirmed interactions with matching directions.
  • DTR: Fraction of small-context confirmed interactions that remain confirmed with the same direction at large context sizes.

Table 1. Marginal Masking Rate (MMR).

ModelContext sizeMMR
TabICL5029.96%
TabICLLarge14.59%
TabPFNLarge31.65%
Kumo mediumLarge22.16%

These results directly demonstrate the importance of query-conditional evaluation: ~15–32% of confirmed interactions at large context sizes would be overlooked by marginal evaluation under the same practicality criterion. This shows that the framework identifies meaningful heterogeneous effects that marginal robustness measurements can miss.

Please see the next comment for the continuation of our response.

Table 2. Context-size diagnostic agreement and transferability (TabICL).

Metric
DAR (M=50 vs. 10000)62.87%
DTR (M=50 vs. 10000)40.85%
Sign agreement among interactions confirmed at both sizes73.11%

The results show moderate overall diagnostic agreement across context sizes. Although not all small-context confirmations persist at larger sizes, 73.11% of interactions confirmed at both sizes preserve their direction. This indicates that some query-conditional vulnerabilities are consistent across substantially different context scales.

Table 3: recurring and model- or context-dependent vulnerabilities (full table in the revised paper).

InteractionTabICLTabPFNKumo-context-size comparisonM=50Large M
D3–P8 (tail-conditional noise)+5.08%+7.34%+4.76%-D2–P1 effect+15.17%+39.68%
D2–P2 (tail decrease)−10.55%+13.92%−3.80%-P9 confirmations3211

These results demonstrate how DiReCT captures both recurring and model- or context-dependent vulnerabilities.

Repair results

Unlike unrestricted kNN selection, new baselines modify an existing reference context and support shared-context evaluation, ensuring that the comparison addresses budgeted repair rather than full context reconstruction.

Table 4. All-query primary loss change relative to reference (%; lower is better).

ModelConfirmed (main)UnconfirmedFacility GlobalFacility SubgroupBS-kNN
TabICL−1.50+3.91−0.19−1.19+24.43
Kumo medium−3.79−3.65+1.19−1.51+16.63

Confirmed (main method) achieves the best reported average loss change on both models. Moreover, confirmation reduces activation while improving or maintaining average performance. The BS-kNN results use a separate matched reference, so these comparisons are descriptive.

These results support our main objective: the diagnosed vulnerabilities provide actionable information that can guide beneficial, selective context interventions, without requiring full context optimization.

[1] Wei et al., Submodularity in Data Subset Selection and Active Learning, ICML 2015.

[2] Thomas et al., Retrieval & Fine-Tuning for In-Context Tabular Models, NeurIPS 2024.

W3: Large, overlapping confidence intervals in Figure 3 make the superiority and reliability of the proposed repair policy difficult to establish.

Response

We thank the reviewer for this comment. We believe the uncertainty intervals in Figure 3 deserve a more detailed interpretation. The figure evaluates both population-level repair efficacy and efficacy on fired queries, where negative loss changes indicate improvement. In the original experiments, Confirmed is the only policy whose 95% task-bootstrap confidence intervals remain entirely below zero for both TFMs in both evaluations. This provides evidence of beneficial average repairs, even when interventions are applied to naturally sampled, unperturbed contexts.

Furthermore, the operator-level analysis in Table 1 supports this finding: 4 of the 5 Confirmed repair-operator outcomes have 95% confidence intervals entirely below zero, compared with only 1 of 5 for Marginal (using absolute effect without confirmation), while Confirmed also maintains a lower activation rate.

The new experiments in response to W2 further support the effectiveness and selectivity of statistical confirmation of DiReCT.

W4: Experiments use only 50-example contexts and a fixed top-20% descriptor cutoff, without evaluating larger contexts or parameter sensitivity.

Response

We thank the reviewer for raising this concern. As detailed in W2, we have extended diagnosis to contexts containing up to 10,000 examples, and also revised for batched TFM evaluation by sharing contexts across queries.

Regarding the top-20% descriptor cutoff, this is a configurable design choice rather than a requirement of DiReCT. We acknowledge that its sensitivity has not yet been evaluated and will clarify this limitation and investigate alternative cutoffs in the revised work.

Q1: Please fix the issues in weaknesses section

Response

We thank the reviewer for these helpful suggestions. We have addressed scalability through the new shared-context formulation and large-context experiments, and clarified the interpretation of Figure 3's confidence intervals in W3. We will incorporate these explanations and improve the corresponding presentation in the revised manuscript.

Dear reviewer zVWL,

Thank you for your constructive review. As the discussion period nears its end, we would appreciate your feedback on our rebuttal and additional experiments. We hope they have addressed your concerns and welcome any further questions.

Best regards,

Authors

Reviewer_pUYvOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper discusses the problem of the robustness of tabular foundation models (TFMs) with imperfect real-world data. The authors consider a wide range of heterogeneous vulnerabilities of in-context TFMs and design a corresponding evaluation framework, DiReCT, to reveal the failure patterns. The framework includes characterization of query descriptors, vulnerability matrix, confirmation gates, and rule-based context repair. Experiments on 19 tasks from RelBench on TabICL and TabPFN show that the proposed framework can provide recurring query-conditioned defects and potentially be converted into useful interventions.

Critical strengths
  1. This paper is well-written and easy to follow, with a broad related work discussion.

  2. The story of the paper is clear and complete: how to evaluate, what the context vulnerability is, and how to repair.

  3. The motivation for incorporating statistical checks into vulnerability diagnosis is reasonable, considering subgroup sample sizes, effect magnitudes, and statistical uncertainty.

Critical weaknesses
  1. The framework (Sections) consists of a common design of the components with limited methodological novelty. The effectiveness of the design is not really ablated; e.g., there are no experiments on query descriptor design vs. other unused descriptors; rule-based context repair vs. LLM-based repair; nearest-neighbor context selection with different similarity metrics.

  2. The experiments lack enough baseline models and data. The evaluation includes only two TFMs and a selection of 19 tasks from seven databases within a single benchmark, RelBench. All experiments also use a context size of 50, which is small given the paper’s motivation concerning the quality of larger contexts. While the tasks cover both classification and regression, the current evaluation leaves it unclear whether the findings generalize to other model families, benchmarks, data settings, and context sizes.

  3. The takeaways for developing better tabular foundation models are limited. The paper diagnoses context vulnerabilities, but uses these findings primarily to construct repair rules for frozen models. It would be more valuable to investigate whether the identified vulnerabilities can improve training to close the loop.

Questions for authors
  1. Why is training free a selling point for an evaluation framework, as most benchmarks serve frozen models?

  2. The selected perturbations (Section 2.2) are listed in Table 3 in the appendix, which is important and can be good to present in the main context. Besides, adding more rationales or statistics to justify the selection can be good.

  3. Figure 1 is of poor quality.

3 comments

We thank the reviewer for the comments. They are very helpful.

W1: Limited methodological novelty and insufficient ablation of the framework's components, descriptors, and repair strategies.

Response

Thank you! To the best of our knowledge, DiReCT is the first training-free, query-conditional framework for diagnosing and repairing context vulnerabilities in frozen TFMs (line 64). Unlike prior marginal, query-centered evaluations and computationally expensive mitigation methods, DiReCT identifies interpretable subgroup vulnerabilities, statistically confirms them, and translates the findings into actionable interventions.

Regarding descriptor selection, we chose four interpretable characteristics commonly observed in tabular data. Importantly, the framework can use any observable query characteristics with sufficient subgroup support.

We respectfully note that the original submission already includes five ablations specifically designed to assess the framework's components: Ungated ablates query-conditional deployment, Marginal ablates query-conditional diagnosis, Unconfirmed ablates statistical confirmation, Interaction ablates subgroup-effect directionality, and Random ablates diagnosis-guided intervention selection. Their design and motivation are further discussed in Section 4.1.2.

Furthermore, context repair should not be mistaken for the main contribution of DiReCT. Its purpose is to demonstrate that the diagnostic findings are actionable, rather than to develop a state-of-the-art context optimizer. Unlike methods that construct optimal contexts from scratch, we evaluate simple, interpretable interventions under a limited editing budget. For this reason, comparisons with unrestricted context optimization or LLM-based repair methods would not directly isolate the value of the diagnosis. Nevertheless, we have added three external, budget-matched kNN-based repair baselines, as discussed in W2.

W2: Limited model and dataset coverage, fixed context size of 50, and insufficient evaluation against external methods.

Response

To address these concerns, we have substantially expanded our experiments:

  • Added a third state-of-the-art TFM, Kumo Tabular (medium).
  • Expanded the evaluation to 24 tasks across eight databases, including Rel-Event.
  • Evaluated contexts containing up to 10,000 examples, capped by the available training samples.
  • Repeated diagnosis at M=50 for TabICL on 20 common tasks.
  • Added three budgeted external repair baselines: Facility Global, Facility Subgroup, both inspired by [1], and BS-kNN, the latter inspired by LoCalPFN [2].

Due to rebuttal-time constraints, the new large-context diagnosis covers all 24 tasks for three TFMs, while the new repair comparisons cover 16 completed tasks for TabICL and Kumo.

To address the scalability limitations of the original submission, we introduce a shared-context formulation that eliminates the need to construct a separate context for every query. During diagnosis, validation queries share the same reference and perturbed contexts, enabling batched TFM evaluations. During repair, queries are grouped by their binary descriptor profiles, requiring at most 2J=162^J=16 distinct repaired contexts instead of one per query. Importantly, the mathematical formulation of the vulnerability contrast, statistical confirmation gates, and repair decision rules remains unchanged; only the context-sampling and implementation strategy, along with the corresponding notation, are revised. This substantially reduces redundant context processing while preserving DiReCT's core methodological principles. The following results all use the shared-context implementation.

Diagnosis results

We report three aggregate metrics to evaluate the methodological properties of DiReCT:

  • Marginal Masking Rate (MMR): The fraction of confirmed query-conditional interactions whose marginal effects fail the same practicality threshold.
  • Diagnostic Agreement Rate (DAR): Agreement between small- and large-context diagnoses, counting both-unconfirmed and both-confirmed interactions with matching effect directions.
  • Diagnostic Transfer Rate (DTR): The proportion of interactions confirmed at M=50 that remain confirmed with the same direction at large context sizes.

Table 1. Marginal Masking Rate (MMR).

ModelContext sizeMMR
TabICL5029.96%
TabICL1000014.59%
TabPFN1000031.65%
Kumo medium1000022.16%

These results directly demonstrate the importance of query-conditional evaluation: ~15–32% of confirmed interactions at large context sizes would be overlooked by marginal evaluation under the same practicality criterion. This shows that the framework identifies meaningful heterogeneous effects that marginal robustness measurements can miss.

Please see the next comment for the continuation of our response.

Table 2. Context-size diagnostic agreement and transferability (TabICL).

Metric
DAR (M=50 vs. 10000)62.87%
DTR (M=50 vs. 10000)40.85%
Sign agreement among interactions confirmed at both sizes73.11%

The results show moderate overall diagnostic agreement across context sizes. Although not all small-context confirmations persist at larger sizes, 73.11% of interactions confirmed at both sizes preserve their direction. This indicates that some query-conditional vulnerabilities are consistent across substantially different context scales.

Table 3: recurring and model- or context-dependent vulnerabilities (full table in the revised paper).

InteractionTabICLTabPFNKumo-context-size comparisonM=50Large M
D3–P8 (tail-conditional noise)+5.08%+7.34%+4.76%-D2–P1 effect+15.17%+39.68%
D2–P2 (tail decrease)−10.55%+13.92%−3.80%-P9 confirmations3211

These results demonstrate how DiReCT captures both recurring and model- or context-dependent vulnerabilities.

Repair results

We also evaluate whether the diagnosed vulnerabilities lead to actionable improvements.

Table 4. All-query primary loss change relative to reference (%; lower is better).

ModelConfirmed (main)UnconfirmedFacility GlobalFacility SubgroupBS-kNN
TabICL−1.50+3.91−0.19−1.19+24.43
Kumo medium−3.79−3.65+1.19−1.51+16.63

Confirmed achieves the best reported average loss change on both models. Moreover, confirmation reduces activation while improving or maintaining average performance. The BS-kNN results use a separate matched reference, so these comparisons are descriptive.

These results show how DiReCT can guide actionable downstream interventions based on diagnosed vulnerabilities.

[1] Wei, K., et. al., "Submodularity in data subset selection and active learning." ICML 2015.

[2] Thomas, V., et. al., "Retrieval & fine-tuning for in-context tabular models." NeurIPS 2024.

W3: Limited takeaways for developing better TFMs; the diagnosed vulnerabilities could be used to improve model training.

Response

We thank the reviewer for this suggestion. We intentionally keep DiReCT training-free to make vulnerability diagnosis computationally manageable and its findings directly interpretable. The proposed context repair is one demonstration of how these findings can be used for performance improvement without modifying TFM parameters. The identified vulnerabilities could also inform training-time improvements, such as targeted data augmentation or robustness-aware training. However, retraining large TFMs would introduce a separate optimization problem beyond the scope of this work. We consider this a promising direction for future research.

Q1: Why is training free a selling point for an evaluation framework, as most benchmarks serve frozen models?

Response

We agree that evaluating frozen models is not novel by itself. Our training-free design extends beyond freezing the TFM: neither vulnerability identification nor the subsequent repair policy requires learning additional parameters. Unlike approaches that learn query representations, vulnerability predictors, or adversarial perturbations, DiReCT uses explicit, interpretable query descriptors and controlled perturbations, followed by statistical confirmation. This reduces additional computational overhead and makes the diagnosed vulnerabilities understandable and actionable without relying on learned, potentially opaque subgroup representations.

Q2: Move the perturbation definitions to the main paper and provide stronger justification for their selection.

Response

We thank the reviewer for this suggestion. We will move the perturbation descriptions into the main paper and clarify their selection rationale. The nine perturbations cover common context-quality issues, including covariate mismatch, feature corruption, missingness, target noise, target-tail imbalance, and redundancy. They are intentionally chosen to provide interpretable and controlled changes in context properties, allowing the framework to associate query characteristics with specific sources of vulnerability. The framework is also extensible to additional perturbations.

Q3: Figure 1 is of poor quality.

Response

We will improve Figure 1 with a clearer visualization of the diagnosis, confirmation, and repair stages, making the overall framework and its contributions easier to follow.

Dear reviewer pUYv,

Thank you for your constructive review. As the discussion period nears its end, we would appreciate your feedback on our rebuttal and additional experiments. We hope they have addressed your concerns and welcome any further questions.

Best regards, Authors

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of the rebuttal

We thank all reviewers for their helpful comments and suggestions. We have carefully addressed their concerns. In the following, we present a summary of the changes made in this rebuttal:

  • We expanded empirical results by adding 1 new state-of-the-art TFM, expanding the evaluation to 24 tasks across 8 databases, and adding 3 external baselines for budgeted context repair comparisons. We reran experiments at both the diagnosis and repair steps with large context sizes (up to 10000 samples, capped by the available training set size). We also introduced new metrics to evaluate the value of query-conditional diagnosis and the stability and transferability of vulnerability diagnostics across small and large contexts.
  • We enhanced scalability by introducing the shared-context implementation, enabling batched diagnosis and reuse of repaired contexts across queries with the same descriptor profile.
  • We clarified the methodological contributions of the work and enhanced discussions on DiReCT's novelty, the design principles, and contributions of the framework's components (e.g., confirmation gates and context repair heuristic). We also highlighted vulnerability diagnosis as the main contribution and repair as a direct demonstration of its applicability and actionability.
  • We added absolute performance measurements for classification datasets and reported prediction entropy from the original evaluation setting to complement our relative-loss evaluations.
  • We revised the paper accordingly to incorporate the changes discussed in the rebuttal.

Again, we greatly appreciate the reviewers' time and feedback, and hope the clarifications and new results address their concerns. We welcome any further questions or discussion.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.