acceptodds
Under review as a conference paper at ICLR 2027

Counterfactual Fragility Certificates for Tabular Classifications: Guarantees and Limits

Abstract

A high-confidence tabular prediction need not survive unreliable input fields. We study protocol-relative fragility: loss of support for the original prediction under a declared finite collection of field failures. Redundant fields can mask one-step effects, while partial degradation can differ from deletion. Counterfactual Fragility Certificates (CFC) combine a sequential removal path, recomputed after every step, with graded tests of every group. An original-class margin records collapse even after the predicted class changes. A reconstructable record exposes the tests, decisions, and witnesses behind a fixed fragility dominance score (FDS). We distinguish record verification from worst-case robustness, causal recourse, and prediction correctness. A conditional greedy bound and counterexamples delineate generic search limits; a model-specific affine solver certifies minimum removal depth. We evaluate a released implementation on seven OpenML datasets, eight classifier families, and three seeds, with paired row-level outputs and reference checks. On the in-family endpoint, CFC-FDS achieves 0.7743 AUROC versus 0.7741 for prespecified, query-matched random search with the same three-term score; the paired difference is +0.0001 (95% interval [-0.0054, 0.0063]). On withheld donor/quantization mechanisms, the difference is -0.0040 ([-0.0089, 0.0004]). These comparisons do not establish a consistent advantage for adaptive removal over query-matched random search across both endpoints. A separate five-dataset restoration study measures 2.36 net errors averted per 100 cases at 10% review using a validation-trained utility ranker, versus 2.61 for raw confidence.

Primary area:
interpretability and explainable AI
Keywords:
tabular data, robustness certificates, counterfactual audit, model fragility, selective prediction, trustworthy machine learning

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_DRcTOpenReview ↗
Rating
2
Confidence
1
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This paper focuses on the task of counterfactual fragility certificates. Specifically, for a given fixed setting, it first bulids stress-test sets, and then make the evaluation. The advantage is that this is reproducible. I find this paper hard to read. Specifically, (1) the main contribution of this paper is still not clear to me. It seems that the authors propose a standard procedure (as Fig1). However, I did not find significant contribution here. (2) there is a held-out evaluation, which means the authors evaluate the dataset in some patterns, and then use some held-out evaluation to re-evaluate. I understand this is a OOD ability. However, different patterns may perform pretty differently. I cannot see its significance. Overall, I do not think this paper meet the standard of ICLR.

Critical strengths

This paper focuses on an important question: high-confidence tabular prediction set. The authors propose a procedure that returns verifiable report. That is to say, one can guarantee an inspectable finite audit. I am not an expert in this field, but I do think this is an important task.

Critical weaknesses

Firstly, the writing is somehow confusing. I do think a polish would contribute much to this paper. For example, page 1, the second paragraph, "we address these questions with a fixed classifier and field-level stress tests". I do not understand why this could answer the second question "would checking the record imporve the decision?". This question is answered very late. Overall, there are many confuses in this paper which stops me from understanding the whole paper.

There are a few questions:

  1. I still cannot understand how this framework distinguish from existing ones. Could the authors provide more evidence on the contributions of this paper, technically?
  2. It seems that the key point of the method is verifiable. I am not sure why other existing reports are not verifiable. In my view, as long as all the operations are fixed and reported honestly, the report itself is verifiable. So could the authors distinguish how this method is more verifiable?
  3. The held-out evaluation. I am not sure this is an useful metric. See above.
Questions for authors

See above.

Reviewer_eXq5OpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
3
Read review
Paper summary

Summary. The paper proposes Counterfactual Fragility Certificates (CFC) for auditing individual predictions of a tabular classifier. Given an input x, CFC builds a set of modified inputs Ω_Π(x) (Eq. 4): a greedy path that replaces fields one at a time with training means, recomputing which field to remove after each step, plus graded attenuation, dropout, and noise tests on every field. It records the model’s responses in a reconstructable record and summarizes them in a fixed score FDS(x) (Eq. 7), which combines how early the original class’s margin collapses, the largest collapse, and the cheapest tested flip. The method is evaluated on seven OpenML datasets with eight model families, plus a separate restoration study on five datasets.

Critical strengths

The authors report null and negative results transparently, prespecify their primary comparison, and provide extensive reproducibility material.

Critical weaknesses

The empirical results do not support the method. The authors acknowledge this in the conclusion: “neither fixed FDS nor CFC-U establishes consistent superiority over matched baselines.”

The paper is also very difficult to read. The text is extremely terse, and the flow is repeatedly interrupted by “Why:” asides. Several sentences are unexplained, e.g., “Deployed floating-point execution is distinct” and “No scientific thresholds changed.” In Section 3.5, the symbol g is reused for the signed restoration benefit after denoting a feature group throughout the paper, which makes the definition hard to follow.

The paper has neither strong results nor a clear presentation, and I do not think it is ready for acceptance.

Questions for authors

Why is this method useful if it does not out-perform baseline?

Reviewer_T4ABOpenReview ↗
Rating
2
Confidence
1
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

In this paper, the authors address tabular prediction fragility auditing problem. This paper proposes Counterfactual Fragility Certificates, a protocol-relative, verifiable fragility auditing framework. CFC introduces sequential removal-path module and graded all-group degradation module.

Critical strengths
  1. This paper addresses an important issue, verifiable fragility auditing for high-stakes tabular classification..
  2. The removal-path module searches for redundancy-masked fragility, while the graded-degradation module covers every group under partial failure.
  3. Ablation studies systematically analyze the contributions of the adaptive removal path (against a fixed-order variant) and the graded test branch (against path-only FDS), showing each component’s effect.
  4. the experiments are comprehensive, covering diverse models and benchmarks, and the query-matched MC-FDS comparator is built so that it can falsify the method.
  5. the paper is well-structured and easy to follow.
  6. the CFC-U study separates whether a prediction is fragile from whether reviewing it is worth the cost, and keeps harmful reviews in the ledger rather than reporting corrections alone.
Critical weaknesses
  1. The graded-degradation construction in the stress-test module is a simple grid of three fixed corruption templates over five severity levels applied independently to each feature group, with no interaction or search.
  2. The ablations do not just leave the design unsupported, they contradict it. CFC-RCMA alone (0.7758) exceeds the full CFC-FDS (0.7743), and CFC-D (0.6086) is worse than MC-D (0.6453) at [-0.0774, -0.0022] — the only significant paired contrast in the paper, and it favors random search.
  3. The method introduces several sensitive hyperparameters, e.g., the equal FDS weights, the removal budget K, and the mean replacement baseline. Sensitivity is reported only for label-definition parameters (§S6.5), never for a protocol parameter. K is demonstrably too tight on multiclass: §S6.3 reports greedy missing 6 Covertype flips entirely within the budget.
  4. Missing comparison with established attribution and counterfactual-explanation methods — SHAP, Anchors, and sufficient input subsets — which the paper cites as related work but never evaluates as fragility rankers.
  5. The primary endpoint shares its failure family with the construction protocol, and performance collapses where it does not: 0.6166 on withheld mechanisms, and 0.5605 on observed errors against 0.6903 for confidence deficit.
  6. this phenomenon is only tested on the eight fixed scikit-learn/GBDT families (LR, RF, ET, HGB, MLP, XGBoost, LightGBM, CatBoost), with no modern tabular architectures — TabNet, FT-Transformer, or TabPFN — despite all three being cited in the related-work section on tabular robustness.
  7. No threat model is stated for “certificate”: Proposition S1 says only that deterministic recomputation reproduces itself.
Questions for authors

Given CFC-RCMA (0.7758) > CFC-FDS (0.7743) and CFC-D < MC-D with interval excluding zero, what justifies retaining the equal-weight three-term head as the primary estimand? Please state the threat model that certificate verification excludes, and its increment over rerunning the released pipeline. Please report the in-family contrast restricted to units with prevalence in [10%, 90%]. Are the single-coordinate probes behind the dropout ordering (§S3) inside the charged d + 2d_num term, or among the auxiliary probes? If the latter, is the auxiliary count also matched for MC-FDS?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.