- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The authors of this paper motivate that counterfactuals generated for a model now may be invalid after the training distribution shifts, inducing changes in the model through retraining. In this paper, the authors propose GRACE, a post-hoc refinement that aims to provide counterfactuals to a model that improve future validity. They consider that population-level optimisation could achieve comparable robustness to individual worst-case optimisation, while being less conservative and thereby preserving greater proximity to the original inputs. The authors evaluate GRACE against robust CF baselines on four tabular datasets, alongside image and a text experiments. They assess GRACE and its comparisons via future validity and proximity, concluding that GRACE improves future validity with relatively small increases in proximity, while matching or exceeding the robustness of existing methods on tabular datasets.
Critical strengths
-
Placing GRACE as a post-hoc refinement rather than a brand new CF generator. This makes GRACE much easier to integrate with existing techniques. The evaluation with Wachter and DiCE provide evidence that the method also isn't dependent on a specific base generator.
-
The tabular results for GRACE show great improvements in terms of Future Validity, often approaching 100% while maintaining competitively low proximity. The paper then also extends to image benchmarks and a small text one, too.
Critical weaknesses
-
I think my biggest criticism of this paper is that I cannot find any direct ablation showing the proposed benefit of population-level optimisation. The central argument within the motivation for the methodology is that "each CF may use a different worst-case model and may guard against scenarios that never jointly arise, potentially moving CFs further than necessary", meaning I would have expected to see a demonstration of this point through an individual-level version of GRACE. Therefore, it is difficult to say whether the gains seen in the results arise from population-level optimisation or from the other components of GRACE.
-
Due to the proposed methodology focusing on population-level optimisation, it is possible that there could be some individuals receiving more costly counterfactuals, such as a few with large proximity, hidden by the reported averaged proximity. I assume the standard deviations in the results table represents standard deviation across individual counterfactuals, which would address this concern somewhat. However, this is not explicitly stated.
Questions for authors
-
(Relating to W1): Could the authors provide an ablation that compares GRACE against an individual-level version of GRACE, keeping every other component as similar as possible? This is my main concern with the paper, and a convincing ablation could change my recommendation.
-
(Relating to W2): Could the authors first clarify what the standard deviations are calculated across and add this to the table captions? Could the authors also comment on the effect of population-level optimisation on individual counterfactuals, particularly whether some individuals could experience substantially larger increases in proximity despite the relatively low average? Maybe reporting distributions of individual proximity changes before and after refinement would help clarify this. Also, a side note: I noticed that Table 7 and Table 1 have identical standard deviations, despite showing different mean results and using different classifiers. Could the authors explain why this is the case? Also, can the authors clarify why ROAR and PROPLACE cannot be applied to German Credit and FICO in Table 7?
-
Tables 1 and 7 report FV without CV, but both of these metrics are reported for the image experiments in Table 10. Could the authors also report CV for the tabular datasets?