- Rating
- 2(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The work proposes a new subgroup discovery method based on signed error score. The authors tested the method across a bunch of benchmarks. The proposed method outperforms some baseline subgroup discovery methods across a number of baselines.
Critical strengths
The subgroup discovery problem, where the subgroup corresponds to the erroneous group, is an interesting one. The proposed idea of subgroup discovery based on signed error and neighborhood graph construction is quite interesting.
Critical weaknesses
The main weakness of the work is the experiment set ups. The benchmarks are primarily synthetically induced shortcuts, where the proposed method fails to outperform at some benchmarks and the ablation studies are inadequate. The variance of their performance is quite high.
Questions for authors
-
The error direction advantage is not clear. The advantage is mostly discussed in experiments in section 5.1 but how it helps the ranking process (intuitively/ mathematically) in Eq (3) is not clear. Moreover, no controlled ablation study is shown with unsigned error.
-
The authors should appropriately define the subgroup. Does the subgroup correspond to any semantically meaningful attribute, such as race or sex? Do the evaluations assume only 1 target subgroup? How does the method recognize the target group(s) if there are multiple of such failing subgroups or subgroups at the insertion of two or more attributes.
-
The authors should explain the high standard deviations across the folds, when they claim no run-to-run variance. The authors should also explain the performance drop in celebA.
-
The advantages of the proposed methods are shown primarily with the synthetically induced shortcuts in ISIC datasets. “High classifier accuracy may limit the information available for error-guided subgroup discovery.” Further realistic benchmarks, choice of model architectures, or controlling errors through controlled model training are required to prove the claim and the generalizability of the method.
-
No systematic ablation studies are shown varying T, the with/ without PCA, varying dimensions of PCA. Local Linear Embedding (LLE) seems to be a suitable alternative dimensionality reduction method for the framework.
-
Why only pleural effusion on the CheXpert dataset? There are other labels within the dataset.
-
How does the method extend to a multi-class classification problem?
-
The rationale behind the confidence formula is not quite clear. Why not the direct probability from model output is used?
-
The authors should report the number of test samples.
3 comments
Follow-Up to Our Rebuttal
We thank the reviewer for their time and helpful feedback. We have addressed your concerns and revised the manuscript accordingly. We welcome further discussion and would be grateful if you would take our revisions into account when reconsidering your score and overall assessment.
I feel that the authors addressed most of my concerns. I have increased my score.
We thank the reviewer for their detailed feedback. We have uploaded the revised manuscript (with additions in blue and new tables in grey). Below, we provide point by point responses to the reviewer concerns: