- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
Generating counterfactual explanations / recourse can be understood as a MCTS (i believe there is previous work along these lines). This paper proposes using LLMs in the expansion phase of a typical MCTS recourse search. The authors claim that this results in more counterfactuals being found than via other approaches.
The authors conduct a small experiment testing 4 hyperparameter settings of their method on 4 available datasets. They compare with 3 baselines.
Critical strengths
It was difficult to find aspects of this work that were particularly strong. Generating counterfactuals with LLMs does not seem to be novel or unique, and the provided evaluation does not convince me that it confers meaningful advantages in terms of cost (computational cost of search or recourse implementation costs).
Critical weaknesses
The abstract states (line 14/15) that a natural way to generate counterfactuals is by prompting LLMs, but this claim does not seem to have a related citation or evidence. Further, the selected motivation/evaluation metric of "number of valid counterfactuals produced" seems arbitrary and suspect. I am not aware of prior literature that focuses on this aspect, and none of the baselines evaluated against optimise for this goal. Diversity in generated counterfactuals and returning a multitude of options to users as candidate recourses have some precedent (eg. the DICE baseline used in this paper), but this is not the same as simply generating more counterfactuals.
In particular, I suspect a naive baseline, an integer programming approach such as those already cited (Russel, 2019, or Ustun et al, 2019), or even a MCTS with a "random" expansion where the search does not terminate after reaching it tradiotnal goals (of finding a minimum edit counterfactual or a multitude of diverse counterfactuals) would generate more / better counterfactuals than prompting an LLM.
I would encourage the authors to either provide a thorough evaluation across hyperparameters and methods (appropriately modifying the methods for their objectives -- using a technique optimised for objective A and then evaluating it using objective B does not provide clear evidence) that demonstrates tangible benefits in using Reco-MCTS; or (more likely) to prove theoretically, empirically, through user studies, or other methods that the counterfactuals generated via LLM are "nicer" than those generated through other means.
Questions for authors
Claims that should be cited / proved instead of being asserted:
- line 14-15
- line 129-30
- A comparison of total flops/compute used (since the paper seems to claim the limited LLM call budget being satisfaction as a unique Reco-MCTS ability) by Reco-MCTS vs DICE and other baselines. Does the method really confer any advantage in compute-per-counterfactual produced than other "vanilla" search strategies?
Typos:
- line 137 missing space
Unclear:
- Line 160 mentions a quality-quantity trade-off, but I don't see this in the paper.
- Line 349-350 mention "same 100 evaluation prompts" and "every condition" but it is not immediately clear what these numbers are referring to
- Is
Lbeing overloaded in line 359? Does it refer to edits from parent to child as well as edits wrt the initial datapoint?