- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The paper studies actionable counterfactual recourse that remains valid after a classifier update. It proposes PACER, combining projected gradient optimisation, a user-set logit-margin target, a categorical softmax relaxation, and an SPN-derived plausibility surrogate, while scoring plausibility with the original SPN. The authors position this as an alternative to mixed-integer recourse and diagnose the fragility of LiCE’s default validity threshold. Experiments on GMSC, Adult and German Credit use five folds and 100 factuals per fold, with comparisons, margin sweeps and ablations; a smaller German Credit experiment varies LiCE’s threshold. Reported retention improvements concern same-data, same-architecture retraining with a different seed; they do not establish robustness to distribution shift or certify the requested margin.
Critical strengths
- The robustness/cost-frontier experiments are interesting: they examine how retention after retraining changes with the recourse cost as the margin varies (Lines 367–378). This is an interesting empirical component of the paper.
Critical weaknesses
- The presentation is difficult to follow. “Exact” and the identity of the “leading exact method” remain unclear in the opening; notation appears before it is explained, and terms such as coverage, confidence level, robustness and adaptive penalty are introduced without enough immediate context (Lines 020–027, 046–069, 078–079). The repeated forward references and vague statements such as “raising it repairs both” make the contribution difficult to assess (Lines 049, 057).
- Some framing and empirical choices need clearer justification. The claim that exact satisfaction of the constraints has required integer programming seems too strong: could a gradient-based approach with rejection sampling also do this (Lines 014–015, 160–161)? The abstract’s numerical claims seem sensitive to the data, sampling and classifier (Lines 016–019), and Table 3 reports two vastly different thresholds without making their selection or the intermediate behaviour clear (Lines 378–387). These concerns leave the scope of the claims and the interpretation of the comparison unclear to me.
Questions for authors
- Clarification — Lines 014–015, 046–054 and 160–161: What precisely does “exact” mean here, and which method is the “leading exact method”? Why would exact constraint satisfaction require integer programming rather than, for example, a gradient-based approach with rejection sampling?
- Clarification — Lines 061–064 and 162–168: How should the confidence margin be interpreted relative to the scale of the space, and what is the significance of automatically deriving the rule base from metadata rather than specifying the constraints by hand?
- Clarification of experimental choice — Lines 378–387 and 395–405: How were the two reported τ values selected, and what happens at intermediate thresholds? The Table 3 caption refers to intermediate thresholds in the text, but I could not locate their numerical results.
2 comments
Thank you again for your review. In case it helps before the discussion closes: the revised PDF addresses each of your points. "Exact" and the leading exact method (LiCE) are now defined at first use, and the introduction is rewritten in short sentences. Table 3 now includes the intermediate thresholds ( and ), and Table 23 extends the sweep to all three datasets. The integer-programming claim is softened (§3.1). The abstract now states its results without point estimates, and Table 22 reports their sensitivity to retraining and model family. We would be glad to clarify anything that remains.
Thank you for the careful reading, and for highlighting the robustness/cost frontier; Figure 1 now also shows FACE, ElliCE and the exact method's threshold sweep. We answer in your order.
Scope (from your summary). We agree on both points and the revised paper says so. Retention is now also tested under LiCE's training recipe and two other model families, a wider MLP and a logistic regression (Table 22), but distribution shift remains out of scope (§7). The margin is a target, not a guarantee (abstract, §7), and Table 24 shows where it is lost.
Weaknesses
Questions