- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The paper proposes to probe and evaluate whether feature synergy helps in a given active feature acquisition setting by evaluating both single and pairs of features at each step. The authors propose a method called SynAFA, which evaluates all single and pairwise features at a given step. The experiments attempt to answer why certain benchmark datasets may benefit from synergy and non-myopic acquisition, while others do not benefit. The results show that at certain budgets, pairwise proposals help due to synergistic information.
Critical strengths
- Understanding the properties of datasets such as the presence of synergistic information is useful and important for addressing how information should be acquired.
- Synthetic experiments demonstrate that the benefits of synergy is only obtained once the acquisition budget is high enough for acquisition to be feasible.
Critical weaknesses
-
To help readers better parse the proposed methods and key insights, a few aspects of the text could benefit from further elaboration:
- In Section 3.1, how are the quantities computed? For example, is a generative model fit using the training data and the quantities are approximated using a Monte Carlo approach?
- For evaluating single feature proposals, why is used instead of the CMI conditioned on all of ? This could explain why with larger budgets, the acquisitions do not become optimal as may be very different from .
- And it is also unclear why the specific experiment designs were chosen to disentangle whether synergy is important. For example, what is the rationale for comparing against specific baselines (such as CAE and AACO) rather than assessing how often pair proposals are selected over single proposals within SynAFA? It remains unclear why performance gains over these baselines can be uniquely attributed to feature synergy rather than other confounding factors.
-
The paper's structure and evaluation could be improved to better support its core claims. The SynAFA approach is not sufficiently evaluated (for example on synthetic data with varying degrees of synergy) to claim that the method can diagnose and detect datasets where synergy is useful versus not. Demonstrating that SynAFA correctly prioritizes pairwise proposals over baselines on these datasets (rather than evaluating performance solely through final accuracy) would significantly strengthen this claim.
Questions for authors
-
See weaknesses above.
-
Several terms are also introduced without sufficient detail for readers to fully grasp their meaning. For instance, what do "Syn" and "V" represent, and how should their ratio be interpreted? And in the conclusion, what does it mean for a population-level score to dominate?