- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes DASH, a batched directional search method for finding nearby counterfactuals that satisfy target predictions and specified actionability constraints in ReLU networks. It combines single-feature perturbations, local gradient steps, factual-origin boundary candidates, beam search, and final bisection refinement. Results demonstrate strong validity, proximity, and runtime performance on mixed-feature tabular tasks. However, baseline fairness, component contributions, and scalability require further validation.
Critical strengths
Clear and practically relevant objective. DASH considers target predictions, feature bounds, immutable features, permitted change directions, and valid categorical assignments, rather than treating a prediction change alone as sufficient.
Strong tabular results. Across 9,000 query–norm cases, DASH returns a valid counterfactual within 5% of the best heuristic-observed distance in 94.6% of cases, with a median search time of 0.061 seconds. Comparisons against certified optima provide additional evidence of proximity quality.
Clearly stated limitations and detailed reporting. The authors acknowledge that DASH does not guarantee global optimality or certify infeasibility. They distinguish failure types and provide implementation details and an ablation of factual-origin anchors.
Critical weaknesses
Inconsistent constraint handling weakens baseline fairness. Appendix B.5 states that FAB handles feature ranges but not other actionability constraints, while DeepFool is unconstrained. Several other baselines repair categorical assignments only after search. DASH enforces constraints throughout search, so validity differences may partly reflect constraint handling rather than search quality. Constraint-aware adaptations of FAB and PGD would provide stronger comparisons.
Novelty and component contributions are insufficiently isolated. Local affine boundary estimates and candidates originating from both the current point and the factual input are closely related to existing minimum-norm attack methods. The factual-origin ablation is useful, but the contributions of Lipschitz-based scales, directional exploration, beam ranking, and batching remain unclear. In particular, the weight-product Lipschitz bounds may become loose with depth; their advantage over simpler step-size schemes should be demonstrated.
Runtime evaluation does not fully measure time to a valid output. PBMC3k outputs undergo an additional marker-constraint completion procedure that may enumerate 28 gene pairs and reevaluate the model, but this cost is excluded from reported runtime. Although shared across methods, its frequency and cost may differ. End-to-end latency should include completion and verification, with precomputation costs reported separately.
Scalability claims need broader support. PGD-bisect achieves 100% valid-within-5% coverage under both and on PBMC3k, exceeding DASH. PBMC3k uses only 50 input features, and all evaluated predictors are fully connected ReLU binary classifiers. Broader applicability to high-dimensional inputs, multiclass tasks, and other architectures remains unestablished.
Specified actionability does not establish real-world feasibility. Feature bounds, directional constraints, and valid one-hot assignments do not ensure that counterfactuals respect feature relationships or feasible interventions. The authors appropriately acknowledge this for PBMC3k, but additional plausibility evaluation or richer joint constraints would strengthen the practical interpretation of the generated explanations.
Questions for authors
see weakness
Thank you for the response. I appreciate the extra experiments evaluating the additional properties including plausibility and robustness (against model retraining), which help to compare DASH against existing methods. However, my main concerns about novelty and relevance (little addition to the current CE literature) remain unaddressed. They are more about the problem studied in the paper and the proposed method itself, rather than the evaluation metrics included. I will keep my current score.