- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
In-context predictions of tabular foundation models (TFMs) can be sensitive to the quality of the provided context. In this work, the authors propose DiReCT, a training-free framework for diagnosing and repairing context vulnerabilities in TFMs. The method uses predefined statistical descriptors and artificial perturbations to identify vulnerable query groups, followed by statistical filtering and rule-based context repairs. The authors evaluate DiReCT on 19 RelBench tasks using TabICL and TabPFN, reporting improvements over the original contexts and several ablated policies.
Critical strengths
- The framework is training-free and provides an interpretable approach to diagnosing and repairing context vulnerabilities.
- The experiments include several ablations to evaluate the contributions of different components.
Critical weaknesses
-
Insufficient experimental evaluation. The method is only evaluated on two TFMs and one benchmark, with a fixed context size of 50. It is unclear whether the results generalize to larger contexts. More importantly, all baselines are internal ablations, without comparisons against existing context selection or retrieval methods. The reported gains are modest, and several tasks show performance degradation. Only relative loss changes are reported, making it difficult to assess the practical significance of the improvements.
-
Limited validation of the diagnosis-to-repair connection. The framework identifies vulnerabilities using artificial perturbations but applies repairs to naturally sampled contexts. While the ablation studies support the effectiveness of the diagnosis-guided policy, it remains unclear how reliably the identified sensitivities predict the benefits of specific repairs across different contexts. Moreover, several diagnosed perturbations have no corresponding repair operators.
-
Limited technical novelty. The method mainly combines existing techniques, including predefined perturbations, subgroup comparisons, statistical tests, and hand-designed context edits. While the combination is reasonable, the paper provides limited new methodological insights. Without stronger baselines, it is difficult to establish the additional value of this framework.
-
Presentation issues. The paper contains broken equation references, inconsistent terminology, and undefined symbols. Several important experimental settings, including perturbation strengths and repair budgets, are insufficiently specified. The conclusion also emphasizes recurring interactions across models under feature permutation, although the results show different interaction directions and limited recurrence.
Questions for authors
-
Why was the context size fixed at 50? Since the proposed vulnerability descriptors depend on the context distribution, I wonder whether the identified vulnerabilities would change significantly with larger contexts.
-
How does DiReCT compare with existing context selection or retrieval methods? The paper argues that existing approaches can be computationally expensive, but does not provide direct comparisons.
-
How reliably do the diagnosed vulnerabilities predict the effectiveness of subsequent repairs? The ablation studies demonstrate the benefits of the overall diagnosis-guided policy, but I wonder whether stronger diagnosed vulnerabilities consistently correspond to larger improvements after repair.
-
Could the authors report the absolute prediction performance for each task? The current results mainly focus on relative loss changes, making it difficult to understand the practical significance of the improvements. In addition, why does the repair policy lead to performance degradation on some tasks? Are there any common characteristics among these failures?
-
How sensitive are the results to the experimental configurations?
5 comments
Part II
W2: Insufficient evidence that diagnosed vulnerabilities predict repair benefits; some perturbations lack repair operators.
Response
As explained in Section 4.1 (line 209), we intentionally evaluate repair on naturally sampled, unperturbed contexts to reflect actual deployment, where the presence and type of context defects are unknown. Evaluating only artificially corrupted contexts would not establish whether the diagnosed vulnerabilities can guide beneficial interventions under normal inference conditions. Particularly, our confirmation gates are therefore important for deciding when intervention is warranted.
Regarding repair design, Appendix A.6 explains our two categories of operators: out-of-place sample replacement and in-place feature replacement. These simple, interpretable operators are designed to counteract or imitate the corresponding diagnostic perturbations. The diagnosis-to-repair connection is evaluated empirically rather than guaranteed mathematically.
For diagnosis-only perturbations, constructing reliable repairs may require identifying or reconstructing unknown corruptions, potentially introducing additional learning or assumptions. We intentionally avoid these mechanisms to keep the framework training-free and interpretable. Importantly, DiReCT is primarily a vulnerability diagnosis framework; repair is one demonstration of the actionability of its findings, not a requirement that every diagnosed perturbation admit a repair operator.
W3: Combines existing techniques without sufficiently establishing the value of the complete framework.
Response
We thank the reviewer for this comment. As discussed in the paper (line 64), to the best of our knowledge, DiReCT is the first training-free, query-conditional framework for diagnosing and repairing context vulnerabilities in frozen TFMs. Prior work largely relies on marginal, query-centered evaluations that overlook interpretable, query-conditional vulnerabilities, while existing mitigation strategies often require expensive context selection or extensive training.
Our novelty lies in formulating context robustness as an interpretable, statistically confirmed query-conditional diagnosis problem and connecting the resulting findings to actionable interventions. The individual components are deliberately simple to preserve interpretability and isolate their methodological contributions.
The original submission already includes five component ablations to give methodological insights, while the three newly added external budgeted baselines provide additional evidence that the diagnostic information supports useful interventions beyond generic context modifications, as shown in W1.
W4: Broken references, inconsistent notation, insufficient experimental details, and overstated cross-model consistency.
Response
We thank the reviewer for identifying these issues. We will carefully revise the equation references, terminology, and undefined symbols throughout the manuscript.
The appendix already contains dedicated sections on descriptor design, perturbation definitions, confirmation gates, and repair mechanisms. However, we agree that important experimental configurations should be more accessible. We will provide a compact summary of perturbation strengths, repair budgets, and task-specific settings in the revised manuscript, with complete implementation details available in our released code.
We also agree that recurring interactions should not be confused with consistent interaction directions. As our new experiments further demonstrate, some vulnerabilities recur across models with similar directions, while others exhibit model-specific responses. We will revise the conclusion to distinguish these cases and avoid overstating cross-model consistency.
Q1: Do diagnosed vulnerabilities remain consistent with larger contexts?
Response
We initially used M=50 to control the computational cost of query-specific context construction. To address this concern, we introduced shared-context diagnosis and evaluated contexts containing up to 10,000 examples. Our results, as responded to W1, show some vulnerabilities persist across context sizes, while others change. These findings reinforce the importance of evaluating context robustness under different context configurations.
Please see the next comment for the continuation of our response.
Part III
Q2: How does DiReCT compare with existing context selection and retrieval methods?
Response
We have added three external budgeted baselines: Facility Global, Facility Subgroup, BS-kNN. As shown in Table 3 of W1, Confirmed achieves the best reported average loss change among the evaluated policies on both TabICL and Kumo.
Importantly, context repair should not be mistaken for the main contribution of DiReCT. Its purpose is to demonstrate that diagnostic findings are actionable, rather than to develop a state-of-the-art context optimizer. Unlike unrestricted retrieval methods that construct contexts from scratch, our setting starts with an existing reference context and permits only budget-constrained modifications. This preserves the original context-selection policy and supports shared-context inference.
Therefore, the added baselines follow the same budgeted repair objective, allowing us to assess whether diagnosis-guided interventions provide practical benefits beyond generic context modifications.
Q3: Do stronger diagnosed vulnerabilities lead to larger repair improvements?
Response
We thank the reviewer for this question. We clarify that the diagnosed interaction measures the relative perturbation sensitivity between query subgroups, rather than the expected magnitude of improvement from a repair. A larger interaction (e.g. stronger diagnosed vulnerabilities) therefore does not necessarily imply a larger repair benefit. Moreover, diagnostic reliability depends on the choice of perturbations, validation sample size, and context-sampling repetitions; increasing perturbation strength instead measures robustness to different corruption levels. Our new diagnosis experiments use 20,000 validation queries and 12 context-sampling repetitions to support reliable estimation. Importantly, DiReCT uses statistically confirmed interactions to guide whether and how to intervene, rather than predict the magnitude of repair gains. As shown in W1, the repair results demonstrate the actionability of these diagnoses, although they do not establish a monotonic relationship between vulnerability strength and repair improvement.
Q4: What are the absolute prediction results, and why does repair degrade performance on some tasks?
Response
We thank the reviewer for this suggestion. In addition to relative loss changes and 95% confidence intervals, we provide the following aggregate absolute performance results using the setting and implementations used in the original submitted draft.
Table 4. Classification performance.
| Model | Metric | Reference | Confirmed | Unconfirmed |
|---|---|---|---|---|
| TabICL | Accuracy (%) | 66.87 | 67.91 | 67.07 |
| TabICL | Predictive entropy | 0.7493 | 0.7144 | 0.7172 |
| TabPFN | Accuracy (%) | 68.10 | 68.76 | 67.53 |
| TabPFN | Predictive entropy | 0.7570 | 0.7371 | 0.7292 |
Confirmed improves mean classification accuracy on both models, although aggregation across heterogenous tasks might have led to small overall improvements. These results complement the task-relative loss changes and we will add them to the paper. Moreover, the original Figure 3 reports 95% task-bootstrap confidence intervals, with Confirmed's aggregate intervals entirely below zero in both evaluation populations for the two original TFMs. This supports beneficial average repair outcomes, although it does not guarantee improvement on every individual task or query.
Q5: How sensitive are the results to the experimental configurations?
Response
We have extended the sensitivity analysis to context size, comparing shared-context diagnosis at M=50 and up to M=10,000, as reported in W1. We acknowledge that additional sensitivity analyses of descriptor cutoffs, confirmation parameters, and repair budgets would further strengthen the evaluation. Due to the computational cost of these experiments, these additional ablations have not yet been completed, and we will investigate them in the revised work.
Part I
We thank the reviewer for the comments. They are very helpful.
Response
Thank you! To address these concerns, we have substantially expanded our experiments:
Due to rebuttal-time constraints, the new large-context diagnosis covers 24 tasks on all three TFMs, while the repair comparisons cover 16 completed tasks on TabICL and Kumo.
To address the scalability limitations of the original submission, we introduce a shared-context formulation that eliminates the need to construct a separate context for every query. During diagnosis, validation queries share the same reference and perturbed contexts, enabling batched TFM evaluations. During repair, queries are grouped by their binary descriptor profiles, requiring at most 2J=16 distinct repaired contexts instead of one per query. Importantly, the mathematical formulation of the vulnerability contrast, statistical confirmation gates, and repair decision rules remains unchanged; only the context-sampling and implementation strategy, along with the corresponding notation, are revised. This substantially reduces redundant context processing while preserving DiReCT's core methodological principles. The following results all use the shared-context implementation.
Diagnosis results
We introduce three aggregate metrics to assess the framework's methodological properties:
Table 1. Marginal Masking Rate (MMR).
The results demonstrate the importance of query-conditional evaluation: ~15–32% of confirmed interactions at large context sizes would be overlooked under the same marginal practicality criterion. Thus, conventional marginal evaluations cannot fully characterize heterogeneous context vulnerabilities.
Table 2. Context-size diagnostic agreement and transferability (TabICL).
The results show moderate overall diagnostic agreement across context sizes. Although not all small-context confirmations persist at larger sizes, 73.11% of interactions confirmed at both sizes preserve their direction. This indicates that some query-conditional vulnerabilities are consistent across substantially different context scales.
Table 3: recurring and model- or context-dependent vulnerabilities (full table in the revised paper).
These results demonstrate how DiReCT captures both recurring and model- or context-dependent vulnerabilities.
Repair results
We compare DiReCT with three external baselines under comparable context-editing budgets.
Table 4. All-query primary loss change relative to reference (%; lower is better).
Confirmed achieves the best reported average loss change on both models. Moreover, confirmation reduces activation while improving or maintaining average performance. The BS-kNN results use a separate matched reference, so these comparisons are descriptive.
These results support our main objective: the diagnosed vulnerabilities provide actionable information that can guide beneficial, selective context interventions, without requiring full context optimization.
[1] Wei et al., Submodularity in Data Subset Selection and Active Learning, ICML 2015.
[2] Thomas et al., Retrieval & Fine-Tuning for In-Context Tabular Models, NeurIPS 2024.