- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes FIND, Feature necessity under INexact supervision via Deletion—to study feature necessity over the family of near-optimal predictors. Feature selection is an important topic, and the idea proposed in this paper is interesting. The authors made a series of theoretical research and some empirical experiment results presented.
Critical strengths
--- important topic, and idea is interesting. --- good theoretical induction and analysis
Critical weaknesses
The paper does not make enough empirical experiments. Compared with the theoretical side, the experiments and results are weak.
The current results are insufficient to demonstrate the effectiveness of the proposed method. It would be better if you can provide more empirical evidence to demonstrate the usefulness of the proposed method.
Questions for authors
Could you provide more direct and explicit experiments and results to show clearly the evidence of your proposed method is better?
Could you please use two datasets as examples to clearly show which feature sets are really critical? what consequences were generated by removing them created ?
3 comments
Many thanks for your providing responses to my comments. While I can see your great effort, you did not clearly mention the details of the four comparison methods. I would like to see who developed those comparison methods, in which year, which journals and/or conferences published them, and the authors.
I would also like to see how many features and what features those methods automatically selected, and the running time of the those methods.
Without those details, I cannot judge whether FIND can actually perform better than the most recent or state-of-the-art methods. So for now, I keep my scores unchanged. I can reconsider this if you provide those details.
Thank you for your reply. All details of the comparison methods are included in Appendix B, Experimental Details. Where can you get the details you want.
1. Predictive performance across benchmark settings
We ranked FIND and the eight baselines within each of the 18 benchmark settings using the budget-averaged scores reported in the paper. Ties at the reported precision received average ranks. FIND achieved the best mean rank on all four metrics.
The Friedman tests assess overall differences among the nine methods. Because some settings share an underlying dataset, we treat the setting-level p-values as exploratory. The average ranks and first-place counts describe the consistency of FIND's predictive performance across the evaluated settings.
2. Controlled ablation of necessity-first ranking
We held the fitted selector, splits, budgets, and downstream classifier fixed and compared FIND's necessity-first ranking with pure row-norm ranking (Norm). Prioritizing certified necessary features changed the selected top-k set in 323 of 1,440 fold–budget pairs (22.4%). The table includes all three settings with changes in at least 75% of pairs. Scores average all 80 pairs per setting.
In the three settings where necessity-first and pure row-norm ranking differed most, our method demonstrates advantages in both AP and Micro-F1. These results show that prioritizing certified necessary features can improve downstream prediction when it substantially changes the selected feature set, with gains varying by setting and metric. In the seven settings where both rules selected identical sets at every evaluated budget, row-norm ranking from the fitted ambiguity-aware objective reproduced FIND's predictive performance. The same objective accommodates candidate-label ambiguity and defines the near-optimal family used for deletion certification. In these seven settings, the predictive results are therefore attributable to the shared base selector.
3. Two examples of certified features and deletion outcomes
We illustrate individual necessity on Music emotion and collective non-removability on Mediamill (c=3), using evaluation fold 2 in both cases. Feature indices are one-based. Starting from FIND's top-k subset, we remove the certified features, refill the subset from the remaining ranked features to keep k fixed, and retrain the same downstream classifier.
On Music emotion, FIND certified each of the five features as individually necessary. Replacing the entire core increased held-out Ranking Loss and reduced Micro-F1 by 0.91 percentage points at the displayed budget.
On Mediamill, each of the three features was certified individually removable. Their joint deletion had a certified objective increase of at least 4.6815×10−4, above the tolerance upper bound of 3.1307×10−4. Thus, every near-optimal predictor must retain at least one member of this group. At the displayed budget, replacing the group increased held-out Ranking Loss and reduced Micro-F1 by 1.92 percentage points. The certificate establishes collective non-removability under FIND's learning objective, while the test scores show the predictive consequence at this budget.