acceptodds
Under review as a conference paper at ICLR 2027

Vulnerable Sets Need Not Yield Reliable Pointwise Rankings: Wrong-Label Causal Vulnerability in Two Tabular Foundation-Model Readouts

Abstract

Mechanistic studies of tabular foundation models can identify context examples that strongly influence prediction and use these signals to construct targeted poisoning attacks. Attack effectiveness, however, does not answer two stronger questions: whether high-influence positions are more vulnerable to wrong labels than matched-random positions under the same corruption budget, and whether the same importance score can rank this damage pointwise. We test both questions across two mechanistically characterized readouts, using deletion of the same rows to separate the harm of retaining wrong supervision from removing the samples themselves. Retaining wrong labels is more harmful than deletion for the high-influence sets selected by both readouts. For TabPFNv2’s attention-vote readout, the combined 20-dataset analysis shows a 2.62 accuracy-point excess [0.78, 4.65] for high-influence over class-matched random corruption, with clear cohort heterogeneity. In a separate 9-dataset pointwise analysis, the same pre-corruption importance shows weak correspondence with single-point damage: macro Spearman correlation is 0.069 for NLL damage and −0.011 for accuracy damage. TabICLv2’s prototype readout establishes neither stable set-level risk enrichment nor reliable pointwise ranking. Mechanism-derived importance can therefore identify more vulnerable sets without guaranteeing reliable pointwise risk ranking; the two properties require separate validation.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Tabular foundation models, label noise, data poisoning, prediction importance, causal vulnerability, In-context learning

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_TQ1aOpenReview ↗
Rating
2
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates whether mechanism-derived importance scores in two tabular foundation models can establish reliable pointwise damage rankings and set-level risk enrichment under wrong-label corruption. On a 20-dataset-5-split set-level study and a 9-dataset-3-split pointwise study, the paper claims that 1) retaining wrong labels is more harmful on average than deleting the same rows, and 2) set-level enrichment does not by itself establish reliable pointwise risk ranking.

Critical strengths
  1. The counterfactual design is strong. Wrong-label retention is compared with deletion of the same rows, and high-influence sets are compared with class-matched random sets under a fixed corruption budget. The distinction between attack effectiveness, excess vulnerability over random positions, and harm relative to deletion is useful. The authors also correctly avoid interpreting the flip-delete contrast as a pure label effect.
  2. This paper offers interesting insights into the three problems: whether retaining corrupted labels is more harmful than deleting the same examples, whether selected sets suffer more damage than class-matched random sets, and whether importance scores rank individual corruption damage.
Critical weaknesses
  1. Limited accessibility and no theoretical characterization. For readers outside the investigated area, the paper is dense and the practical meaning of the quantities is not immediately clear. There is also no formal statement of when set-level enrichment can coexist with poor pointwise ranking, so the conceptual contribution rests entirely on the empirical pattern. This limits both significance and clarity.
  2. The paper’s framing is broader than the evidence supports. The distinction between set-level vulnerability and pointwise ranking is valuable. However, the strongest set-level result depends on the dataset cohort and is driven by the expansion datasets. Set-level enrichment remains inconclusive on the same nine datasets and three splits used to evaluate pointwise ranking. This matters because the paper’s central contribution concerns the relationship between these two properties. Without establishing set-level enrichment on the shared evaluation scope, the claimed separation between set-level enrichment and pointwise ranking needs to be stated more narrowly.
Questions for authors
  1. What supports the claimed separation on a common evaluation scope? Set-level enrichment is inconclusive on the nine datasets and three splits used for pointwise ranking. Can the author evaluate both properties on the full 20-dataset cohort? Otherwise, the author should narrow the central claim to reflect this evidence gap and the expansion-driven set-level result.

  2. The three-state criterion is custom (1pp threshold, ROPE ±0.5pp, ρ≥0.20). Please clarify: are these thresholds pre-registered, what happens to the conclusions if they are relaxed or tightened on the combined cohort? The current threshold-sensitivity check is only on the original cohort.

  3. The 15% corruption budget is fixed. Please report at least a small budget ablation (e.g., 5/10/20%) to show the set-level vs. pointwise separation is not an artifact of one budget.

5 comments
AuthorsOpenReview ↗

Response to Reviewer TQ1a

Response to Reviewer TQ1a

We thank TQ1a for the three questions: scope, thresholds and budget.

W1. No formal statement.

Appendix G.11 now states the relation formally. The set-level quantity contrasts the damage of corrupting the top-k block at once with that of a class-matched random block; the pointwise quantity is the rank correlation between the score and single-row damages. The two are measured on different objects, a joint corruption against individual corruptions, and answer different questions: which rows enter the block, and how the damages are ordered. The inference fails even if the joint damage were additive: with classes (0,1,0,1), s = (1,2,3,4) and single-row damages (3,1,4,2), the top two rows average 3.0 against 2.0 for their class-matched, disjoint control while ρ = 0. In the data the joint damage is 1.18 and 1.45 times the summed single-point damages (Appendix G.9), so neither quantity can be reconstructed from the other.

W2 / Q1. Scope.

Both properties are now measured on the same twenty datasets (Table 1). The high-influence block loses 2.62 pp more than a matched random block for TabPFNv2 (14/20 datasets agreeing) and 1.96 pp more for TabICLv2 (15/20). The pointwise rank correlation of the same score is −0.008 (9/20) and +0.052 (14/20), far below 0.20. The excess is larger on the expansion datasets and does not hold on the classic eight, and it is smaller or reversed on other metrics, while the block's own damage is positive on every metric (Table 19). The paper separates these two statements wherever the claim appears.

Q2. Thresholds.

Yes, and the repository record shows it: the model configuration, the datasets and splits, the corruption rules, the thresholds (MCID 1 pp, ROPE ±0.5 pp, ρ ≥ 0.20) and the verdict criteria were frozen before the main runs, and the commit that carries the frozen configuration (22:57) precedes the first main-run commit (01:01 the same night) by about two hours. Appendix A lists the frozen files with their SHA-256 hashes and the commits that carry them. The commit history records the freeze; the double-blind rules prevent us from pointing you to the repository itself. The twelve-combination scan on the combined cohort leaves every verdict unchanged (Appendix G.1), and the pointwise bar is unmet at 0.10, 0.15, 0.20 and 0.25.

Q3. Budget ablation.

Seven tiers from 3% to 21% (1,400 cells). J is strictly monotone for both readouts; Γ_flip rises overall and is strictly monotone for TabICLv2, so the reference tier is conservative for it (Figure 3, Table 27).

Paper changes. Twenty-dataset pointwise study; two-layer claim; Appendix G.8–G.11; nine-dataset scope check removed.

Dear Reviewer,

Thank you for your thoughtful review and contribution to the discussion.

As the discussion period approaches its conclusion, I would like to kindly remind you to review the authors' rebuttal and engage in the discussion if you have not already done so.

In particular, it would be helpful if you could indicate whether the authors have adequately addressed your concerns and, if appropriate, reconsider your assessment and score.

Thank you again for your time and effort!

AuthorsOpenReview ↗

Follow-up on the Revised Manuscript

We thank the reviewers for their time and consideration. We have uploaded a revised manuscript and provided detailed responses addressing the main concerns, including the expanded 20-dataset pointwise evaluation and additional analyses.

If any substantive concerns remain, we would welcome further discussion or clarification before the discussion period closes.

Thank you again for your consideration.

Reviewer_n4HJOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies the relationship between the importance of samples in the tabular base model and the causal damage caused by incorrect labels. It focuses on whether high-importance samples are more susceptible to label contamination and whether the sample importance score can accurately predict the actual damage of a single sample. For the TabPFNv2 and TabICLv2 models, through counterfactual experiments such as label flipping, sample deletion, and random matching contamination, analyses are conducted at the set level and the single-sample level respectively. The core contribution of this paper lies in revealing that the risk enrichment at the set level does not imply accurate risk prediction at the single-sample level. It emphasizes that the importance of model prediction and the causal vulnerability caused by incorrect labels are two different concepts, and they need to be verified separately.

Critical strengths
  1. This paper provides a thorough and systematic experimental verification. The author conducted experiments on two table-based models with different internal prediction mechanisms, TabPFNv2 and TabICLv2. Additionally, multiple datasets, different data partitions, and various evaluation metrics were used to test the research conclusions. Furthermore, the paper examined the causal damage caused by incorrect labels from multiple perspectives through counterfactual comparisons of label flipping and sample deletion, comparisons of important samples and randomly sampled samples matching by category, as well as independent analyses at the set-level and single-sample-level. The author employed Bootstrap confidence intervals and clear statistical determination criteria, and reported inconsistent results under different experimental scopes, enhancing the credibility and transparency of the experimental conclusions.

  2. This paper clearly distinguishes between Set-level Risk Enrichment and Pointwise Risk Ranking, stating that the set of high-importance samples identified by the model's internal mechanism, even if it shows a higher risk of label contamination, does not mean that the same importance score can reliably predict the actual damage of a single sample. This distinction breaks through the previous research perspective that mainly assesses the importance of samples through attack success rates, and provides targeted experimental evidence through counterfactual intervention.

Critical weaknesses
  1. The evidence supporting the core conclusions of the paper is insufficient. The article emphasizes that the risk enrichment at the aggregate level does not imply a reliable single-sample risk ranking, but the main experiments supporting these two conclusions used different data ranges: the aggregate-level analysis was based on 20 datasets, each divided 5 times, while the single-sample analysis only used 9 datasets, each divided 3 times. More importantly, when the authors limited the aggregate-level analysis to the same 9×3 experimental range, the results of risk enrichment no longer had sufficient statistical support. Additionally, the two analyses respectively used Accuracy and NLL as the main damage indicators, further weakening the fairness of the comparison and the reliability of the conclusion.

  2. The paper measures the causal damage of a single sample by modifying its labels and calculates the correlation between this damage and the pre-estimated importance score. For multi-class tasks, each sample only considers a randomly selected incorrect target category once, without fully considering the differences in damage that may arise from different incorrect labels. Since the target category of the incorrect label may significantly affect the prediction result, the damage value obtained from a single label flip may not be able to stably reflect the overall vulnerability of the sample.

Questions for authors
  1. The paper found that TabPFNv2 exhibited risk enrichment at the collective level within a wide range of experiments, but its importance score could not reliably predict the damage of a single sample. As an article that verifies experiments, the following questions should be studied: Are there interactions among highly important samples, such that the damage caused when multiple samples are simultaneously contaminated cannot be explained by the independent damage of a single sample?

  2. Same as weakness.

3 comments
AuthorsOpenReview ↗

Response to Reviewer n4HJ

Response to Reviewer n4HJ

We thank n4HJ for the careful reading. Both weaknesses are addressed by changing the design.

W1. Data ranges and main metrics.

The pointwise study now covers the same twenty datasets (Table 1, Table 8). NLL remains the pointwise primary and the accuracy-scale check is reported next to it: ρ = −0.008 (9/20) for TabPFNv2 and +0.024 (13/20) for TabICLv2, both far below 0.20, as is ρ_NLL. The set-level claim is split into block damage, positive on every metric (Table 19), and the excess over matched random, an accuracy result.

W2. One wrong target per multiclass point.

Removed: every point is flipped to all K−1 wrong classes and the damage averaged; the recomputation covers ten multiclass datasets (60 cells, 36,450 points, 165,222 flips) and took about two days of compute. The change matters at the point level (the across-target standard deviation exceeds the point's own mean damage on 33.0% and 41.4% of points) and leaves the aggregate reading unchanged: ρ_NLL = −0.008 and +0.052 (Table 1b), and the binary subset gives −0.024 and +0.049 (Table 25).

Q1. Interactions.

Measured on both scales (Table 28; this also answers Reviewer 74VP's Q3): on NLL the joint damage is 1.18 and 1.45 times the summed single-point damages, with residuals +0.027 [−0.046,+0.101] and +0.054 [−0.002,+0.111] nats; on accuracy it is +4.92 and +4.90 pp against −0.09 and +3.69 pp summed (residuals +5.01 and +1.20 pp). Both NLL intervals include zero, so the direction is consistent but the interaction is not yet statistically reliable; the block-level and point-level quantities are measured separately in the paper.

Paper changes. Twenty-dataset pointwise study with all K−1 targets averaged; explicit metric asymmetry; Appendix G.8–G.11 added; nine-dataset scope check removed.

Dear Reviewer,

Thank you for your thoughtful review and contribution to the discussion.

As the discussion period approaches its conclusion, I would like to kindly remind you to review the authors' rebuttal and engage in the discussion if you have not already done so.

In particular, it would be helpful if you could indicate whether the authors have adequately addressed your concerns and, if appropriate, reconsider your assessment and score.

Thank you again for your time and effort!

I have read the author's rebuttal and have basically resolved my issue. I will keep my score.

Reviewer_74VPOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper investigates whether mechanism-derived importance scores identify context examples whose incorrect labels cause greater damage in TabPFNv2 and TabICLv2. It compares label corruption with deletion, corruption of selected groups with matched-random groups, and importance scores with damage from individual label flips. Incorrect labels are consistently more harmful than deleting the same examples.

Critical strengths
  1. The matched-random and deletion controls help distinguish the effects of selecting particular examples from the general harm of label corruption.
  2. Explicitly reported uncertainty, inconclusive results, and limitations rather than presenting uniformly positive findings.
  3. Interesting topic
Critical weaknesses

Clarity & Readability:

The writing is very compressed, and the authors frequently use technical expressions without sufficiently explaining their meaning. For example, the statement "Tail selection does not recover useful pointwise enrichment" is difficult to interpret without consulting the appendix. More generally, expressions such as "set-level risk enrichment," "pointwise correspondence," and "wrong-label causal vulnerability" are used extensively, making the narrative unnecessarily complicated.

Lack of justification for initial dataset selection:

  • The paper distinguishes between eight "original" datasets and twelve expansion datasets. Appendix A explains that the original eight were included in an earlier recorded analysis plan, but I could not find a clear explanation of why these particular eight datasets were initially selected or how representative they are of the intended evaluation setting.
  • While the selection criteria for the four expansion datasets included in the pointwise experiments are discussed, the rationale for selecting only five of the original eight datasets for the pointwise analysis is less clear.
  • I would appreciate a more explicit justification of these choices, including whether they were made independently of any preliminary experimental observations.

Different experimental scopes weak the central comparison:

  • A major concern is that the set-level and pointwise analyses use substantially different experimental scopes. The set-level analysis considers 20 datasets with five splits, whereas the pointwise analysis considers only nine datasets with three splits. This complicates the interpretation of the paper's central claim, since differences between the two analyses may partly reflect the datasets, experimental interventions, or evaluation metrics rather than the distinction between set-level and individual vulnerability.
  • Appendix F.1 reports that when the set-level analysis is restricted to the same nine datasets and three splits used for the pointwise analysis, the high-influence versus matched-random advantage becomes inconclusive. So, the paper does not establish both the positive set-level result and the failure of pointwise ranking on an identical experimental scope. I believe this limitation should receive greater emphasis in the main discussion and conclusions.
  • There is a substantial difference in the results from the 8 datasets and then 12 datasets. I would like to see a more detailed investigation of which dataset characteristics might explain this behavior and how broadly the main conclusion can be expected to generalize.

Dependence on the evaluation metric:

  • The positive set-level result is specific to accuracy.
  • The additional experiments in Appendix G show that other metrics, including balanced accuracy, macro-F1, negative log-likelihood, and true-class probability, do not consistently support the same conclusion. In particular, the combined-cohort true-class probability comparison has a statistically negative excess.
  • Although the authors acknowledge this limitation, it raises questions about the general interpretation of the selected examples as more vulnerable.
  • The conclusion should be stated explicitly in terms of the particular damage metric for which supporting evidence was obtained.
Questions for authors
  1. What motivated the initial choice of eight datasets? Why were only five of these included in the main pointwise study?
  2. Can the authors explain the substantial difference between the original and expansion cohorts? Are there identifiable dataset characteristics associated with this difference?
  3. Could changing several labels together cause damage that is not captured by changing one label at a time? Would testing different numbers of corrupted examples help explain the difference between the two experiments?
  4. Why does corrupting the selected examples hurt accuracy more than random corruption, while the other metrics give mixed or opposite results?
4 comments
AuthorsOpenReview ↗

Response to Reviewer 74VP

Response to Reviewer 74VP

We thank Reviewer 74VP for the clarity comments. Two changes bear on them: the pointwise study now covers the same twenty datasets as the set-level study, with every multiclass point flipped to all K−1 wrong classes and the damage averaged; and the set-level claim is stated in two layers, with the block damage separated from the excess over matched-random corruption.

W1. Compressed writing.

We accept this. The three questions are stated in words at the end of §1 (wrong supervision, set enrichment, pointwise ranking) and the quantities are defined in §3.4. The quoted sentence is the negative half of the pointwise question: restricting to the top of the score distribution does not restore a usable ranking (Table 16). Why the two properties can differ is now stated formally in Appendix G.11.

W2. Why these eight datasets, and why only five for pointwise?

The five-of-eight gap is gone: pointwise now covers all twenty datasets. The eight are classic small-to-medium benchmarks fixed before the main runs; the twelve expansion datasets were selected from OpenML-CC18 under eligibility criteria fixed before any model outcome (§3, Appendix A).

W3. Different scopes.

Both properties are now measured on the same twenty datasets (Table 1). For TabPFNv2 the excess is +2.62 pp (14/20, Support) and pointwise ρ is −0.008 (9/20); for TabICLv2 they are +1.96 pp (15/20, Inconclusive) and +0.052 (14/20). The block is damaged on every metric while the excess is an accuracy result (Table 19), and the paper says so wherever the excess is claimed; per-cohort pointwise values are in Table 8.

W4. Why do the cohorts differ?

The two cohorts differ first in how they were chosen. The eight classic datasets are small-to-medium benchmarks fixed before the main runs; the twelve expansion datasets were selected later from OpenML-CC18 under eligibility criteria fixed before any model outcome. They also differ in their statistics: the classic eight average 637 rows, 13.3 features and 2.6 classes; the expansion twelve 1,329 rows, 16.1 features and 4.1 classes. The excess over matched random is +0.49 pp on the classic eight and +4.03 pp on the expansion twelve, 8.15 times larger, while J and the block's own damage differ by only 1.74 and 1.68 times (Table 1a). The class count is the strongest and most consistent correlate (ρ = −0.37, −0.36, −0.33), and the number of features is next (+0.19, +0.31, +0.24); Table 29 lists all five features we tested.

W5/Q4. Metric dependence.

The divergence is real: each metric reads the same per-query changes differently. Accuracy reads the argmax, balanced accuracy and macro-F1 read the same events under equal class weights, the true-class probability is linear in pyp_y, and NLL is convex. The two arms spread their damage differently: matched random lowers the true-class probability on 87.3% of queries against 82.7%, while the high block's most-damaged decile carries 40.6% of its total probability damage against 26.8% (Table 24, Appendix G.6). Class reweighting removes most of the accuracy excess (balanced accuracy +0.28 pp, macro-F1 +1.10 pp, against +2.62 pp), and the linear probability measure sums the random arm's broader but shallower decrease, which is why its excess is negative. J, retaining wrong labels versus deleting the rows, is positive on every metric with a clean baseline (Table 19). This describes the damage distribution, not the model's internal mechanism.

Q3. Several labels at once, and different numbers of corrupted rows.

The block costs more than the sum of its parts on both scales (Table 28). On NLL, joint damage is 1.18 and 1.45 times the summed single-point damages (residuals +0.027 and +0.054 nats). On accuracy, joint damage is +4.92 and +4.90 pp against summed single-point damages of −0.09 and +3.69 pp (residuals +5.01 and +1.20 pp); the summed arm is noisy because a single flip leaves query accuracy unchanged on 47.2% and 55.5% of records. Both NLL residual intervals still include zero, so the direction is consistent but the interaction is not yet statistically reliable.

The number of corrupted rows is varied in a seven-tier ladder from 3% to 21% of the context rows, on one shared wrong-label table (Appendix G.8, Figure 3, Table 27). Both set-level quantities grow with the budget: J goes from +0.43 to +7.25 pp (TabPFNv2) and from +0.36 to +7.19 pp (TabICLv2), strictly monotone in both (Spearman +1.000); Γ_flip goes from +0.32 to +2.85 pp and from +0.13 to +4.66 pp, strictly monotone for TabICLv2. The 15% reference tier is not the peak of either curve, and the excess has the same sign at every tier, so it is not an artefact of one budget.

Paper changes. Twenty-dataset pointwise study with the all-target measure; two-layer set-level claim; tables and Figure 1C redrawn; query-level spread analysis (Appendix G.6, Table 24); Appendix G.8–G.11 added.

Dear Reviewer,

Thank you for your thoughtful review and contribution to the discussion.

As the discussion period approaches its conclusion, I would like to kindly remind you to review the authors' rebuttal and engage in the discussion if you have not already done so.

In particular, it would be helpful if you could indicate whether the authors have adequately addressed your concerns and, if appropriate, reconsider your assessment and score.

Thank you again for your time and effort!

AuthorsOpenReview ↗

Follow-up on the Revised Manuscript

We thank the reviewers for their time and consideration. We have uploaded a revised manuscript and provided detailed responses addressing the main concerns, including the expanded 20-dataset pointwise evaluation and additional analyses.

If any substantive concerns remain, we would welcome further discussion or clarification before the discussion period closes.

Thank you again for your consideration.

I thank the authors for their response, the authors addressed the main questions, so I raise my score.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Revisions in Response to the Reviews

We thank all three reviewers for their constructive and detailed feedback. We have uploaded a substantially revised manuscript and provided individual responses to each review. Below we summarize the main changes made during the rebuttal period.

1. Expanded and redesigned the pointwise evaluation.
In response to a central concern shared by all three reviewers, we expanded the pointwise evaluation from 9 to all 20 datasets, aligning its dataset coverage with the set-level analysis. We also revised the single-point corruption protocol: instead of sampling one incorrect target label for each context point, we now evaluate every one of the K−1K-1 incorrect target labels and average the resulting damage. The expanded evaluation uses three splits per dataset, while the set-level analysis retains five splits. These changes required substantial additional computation and led us to recompute and update the affected pointwise results, correlations, uncertainty intervals, tables, figures, and corresponding conclusions throughout the manuscript.

2. Added a seven-level corruption-budget study.
We evaluated corruption budgets of 3%, 6%, 9%, 12%, 15%, 18%, and 21% of context examples, covering both models and all 20 datasets with five splits each. This analysis examines how the set-level quantities change with the number of corrupted examples, rather than relying solely on the original 15% budget.

3. Added joint-versus-single-point damage analysis.
Following suggestions from two reviewers, we compared the damage caused by corrupting a set of examples jointly with the sum of the damages caused by corrupting those examples individually. The revised manuscript reports these comparisons and their uncertainty, providing a direct examination of whether joint corruption effects can be approximated by summed single-point effects.

4. Improved the manuscript's clarity and presentation.
We revised the writing throughout the manuscript to make the research questions, experimental comparisons, metric definitions, and interpretation of results easier to follow. We also improved the connections between the main text, figures, tables, and supporting analyses.

5. Addressed additional reviewer-specific concerns.
We made further revisions in response to the individual reviews, including additional analyses, clarifications, and formal arguments. The corresponding changes and supporting evidence are described in our detailed responses under each review.

We appreciate the reviewers' time and consideration of the revised manuscript.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.