- Rating
- 2
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper investigates whether mechanism-derived importance scores in two tabular foundation models can establish reliable pointwise damage rankings and set-level risk enrichment under wrong-label corruption. On a 20-dataset-5-split set-level study and a 9-dataset-3-split pointwise study, the paper claims that 1) retaining wrong labels is more harmful on average than deleting the same rows, and 2) set-level enrichment does not by itself establish reliable pointwise risk ranking.
Critical strengths
- The counterfactual design is strong. Wrong-label retention is compared with deletion of the same rows, and high-influence sets are compared with class-matched random sets under a fixed corruption budget. The distinction between attack effectiveness, excess vulnerability over random positions, and harm relative to deletion is useful. The authors also correctly avoid interpreting the flip-delete contrast as a pure label effect.
- This paper offers interesting insights into the three problems: whether retaining corrupted labels is more harmful than deleting the same examples, whether selected sets suffer more damage than class-matched random sets, and whether importance scores rank individual corruption damage.
Critical weaknesses
- Limited accessibility and no theoretical characterization. For readers outside the investigated area, the paper is dense and the practical meaning of the quantities is not immediately clear. There is also no formal statement of when set-level enrichment can coexist with poor pointwise ranking, so the conceptual contribution rests entirely on the empirical pattern. This limits both significance and clarity.
- The paper’s framing is broader than the evidence supports. The distinction between set-level vulnerability and pointwise ranking is valuable. However, the strongest set-level result depends on the dataset cohort and is driven by the expansion datasets. Set-level enrichment remains inconclusive on the same nine datasets and three splits used to evaluate pointwise ranking. This matters because the paper’s central contribution concerns the relationship between these two properties. Without establishing set-level enrichment on the shared evaluation scope, the claimed separation between set-level enrichment and pointwise ranking needs to be stated more narrowly.
Questions for authors
-
What supports the claimed separation on a common evaluation scope? Set-level enrichment is inconclusive on the nine datasets and three splits used for pointwise ranking. Can the author evaluate both properties on the full 20-dataset cohort? Otherwise, the author should narrow the central claim to reflect this evidence gap and the expansion-driven set-level result.
-
The three-state criterion is custom (1pp threshold, ROPE ±0.5pp, ρ≥0.20). Please clarify: are these thresholds pre-registered, what happens to the conclusions if they are relaxed or tightened on the combined cohort? The current threshold-sensitivity check is only on the original cohort.
-
The 15% corruption budget is fixed. Please report at least a small budget ablation (e.g., 5/10/20%) to show the set-level vs. pointwise separation is not an artifact of one budget.
5 comments
Dear Reviewer,
Thank you for your thoughtful review and contribution to the discussion.
As the discussion period approaches its conclusion, I would like to kindly remind you to review the authors' rebuttal and engage in the discussion if you have not already done so.
In particular, it would be helpful if you could indicate whether the authors have adequately addressed your concerns and, if appropriate, reconsider your assessment and score.
Thank you again for your time and effort!
Follow-up on the Revised Manuscript
We thank the reviewers for their time and consideration. We have uploaded a revised manuscript and provided detailed responses addressing the main concerns, including the expanded 20-dataset pointwise evaluation and additional analyses.
If any substantive concerns remain, we would welcome further discussion or clarification before the discussion period closes.
Thank you again for your consideration.
Response to Reviewer TQ1a
Response to Reviewer TQ1a
We thank TQ1a for the three questions: scope, thresholds and budget.
W1. No formal statement.
Appendix G.11 now states the relation formally. The set-level quantity contrasts the damage of corrupting the top-k block at once with that of a class-matched random block; the pointwise quantity is the rank correlation between the score and single-row damages. The two are measured on different objects, a joint corruption against individual corruptions, and answer different questions: which rows enter the block, and how the damages are ordered. The inference fails even if the joint damage were additive: with classes (0,1,0,1), s = (1,2,3,4) and single-row damages (3,1,4,2), the top two rows average 3.0 against 2.0 for their class-matched, disjoint control while ρ = 0. In the data the joint damage is 1.18 and 1.45 times the summed single-point damages (Appendix G.9), so neither quantity can be reconstructed from the other.
W2 / Q1. Scope.
Both properties are now measured on the same twenty datasets (Table 1). The high-influence block loses 2.62 pp more than a matched random block for TabPFNv2 (14/20 datasets agreeing) and 1.96 pp more for TabICLv2 (15/20). The pointwise rank correlation of the same score is −0.008 (9/20) and +0.052 (14/20), far below 0.20. The excess is larger on the expansion datasets and does not hold on the classic eight, and it is smaller or reversed on other metrics, while the block's own damage is positive on every metric (Table 19). The paper separates these two statements wherever the claim appears.
Q2. Thresholds.
Yes, and the repository record shows it: the model configuration, the datasets and splits, the corruption rules, the thresholds (MCID 1 pp, ROPE ±0.5 pp, ρ ≥ 0.20) and the verdict criteria were frozen before the main runs, and the commit that carries the frozen configuration (22:57) precedes the first main-run commit (01:01 the same night) by about two hours. Appendix A lists the frozen files with their SHA-256 hashes and the commits that carry them. The commit history records the freeze; the double-blind rules prevent us from pointing you to the repository itself. The twelve-combination scan on the combined cohort leaves every verdict unchanged (Appendix G.1), and the pointwise bar is unmet at 0.10, 0.15, 0.20 and 0.25.
Q3. Budget ablation.
Seven tiers from 3% to 21% (1,400 cells). J is strictly monotone for both readouts; Γ_flip rises overall and is strictly monotone for TabICLv2, so the reference tier is conservative for it (Figure 3, Table 27).
Paper changes. Twenty-dataset pointwise study; two-layer claim; Appendix G.8–G.11; nine-dataset scope check removed.