acceptodds
Under review as a conference paper at ICLR 2027

When Does Synergy Help Active Feature Acquisition? A PID-Based Study

Abstract

Active feature acquisition (AFA) sequentially selects informative features under budget constraints. However, existing policies rarely distinguish whether information contributed by interacting features is redundant, unique, or synergistic. We introduce SynAFA, a state-dependent AFA policy that combines pairwise joint information and conditional information, with Partial Information Decomposition (PID) characterizing its information structure. Across five tabular datasets and MNIST-loop, performance is heterogeneous. SynAFA shows its strongest gains at low budgets on PhysioNet, where synergistic and redundant feature pairs are supported by permutation tests, but its advantage diminishes as budgets increase and does not depend on pair proposals. SynAFA performs significantly worse than nearly all baselines on MiniBooNE, and than CAE on MNIST-loop. Controlled synthetic experiments further show that, within budget, SynAFA's advantage rises as joint information becomes more synergy-dominated, including when total joint information is held approximately constant. Further analyses show that the budget-dependent erosion is not resolved by non-greedy local search, which improves a diagnostic set-level objective but leaves predictive performance no better, often significantly worse, and that improvements in this objective are weakly aligned with the fixed classifier's predictive utility. These findings characterize when pairwise synergy can benefit AFA while exposing a persistent challenge in translating local information into effective acquisition objectives.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
Active Feature Acquisition, Partial Information Decomposition, Information Synergy, Mutual Information, Budgeted Feature Acquisition

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_62ncOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The paper proposes to probe and evaluate whether feature synergy helps in a given active feature acquisition setting by evaluating both single and pairs of features at each step. The authors propose a method called SynAFA, which evaluates all single and pairwise features at a given step. The experiments attempt to answer why certain benchmark datasets may benefit from synergy and non-myopic acquisition, while others do not benefit. The results show that at certain budgets, pairwise proposals help due to synergistic information.

Critical strengths
  • Understanding the properties of datasets such as the presence of synergistic information is useful and important for addressing how information should be acquired.
  • Synthetic experiments demonstrate that the benefits of synergy is only obtained once the acquisition budget is high enough for acquisition to be feasible.
Critical weaknesses
  • To help readers better parse the proposed methods and key insights, a few aspects of the text could benefit from further elaboration:

    • In Section 3.1, how are the quantities computed? For example, is a generative model fit using the training data and the quantities are approximated using a Monte Carlo approach?
    • For evaluating single feature proposals, why is maxkCj∣xkmax_k C_{j \mid x_k} used instead of the CMI conditioned on all of xsx_s? This could explain why with larger budgets, the acquisitions do not become optimal as maxkCj∣xkmax_k C_{j \mid x_k} may be very different from I(Xj;Y∣Xs)I(X_j ; Y \mid X_s).
    • And it is also unclear why the specific experiment designs were chosen to disentangle whether synergy is important. For example, what is the rationale for comparing against specific baselines (such as CAE and AACO) rather than assessing how often pair proposals are selected over single proposals within SynAFA? It remains unclear why performance gains over these baselines can be uniquely attributed to feature synergy rather than other confounding factors.
  • The paper's structure and evaluation could be improved to better support its core claims. The SynAFA approach is not sufficiently evaluated (for example on synthetic data with varying degrees of synergy) to claim that the method can diagnose and detect datasets where synergy is useful versus not. Demonstrating that SynAFA correctly prioritizes pairwise proposals over baselines on these datasets (rather than evaluating performance solely through final accuracy) would significantly strengthen this claim.

Questions for authors
  • See weaknesses above.

  • Several terms are also introduced without sufficient detail for readers to fully grasp their meaning. For instance, what do "Syn" and "V" represent, and how should their ratio be interpreted? And in the conclusion, what does it mean for a population-level score to dominate?

Reviewer_CK62OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies active feature acquisition. A model buys features for each test one at a time, as in ordering medicine tests for a patient under some budget. Most of the previous works in the literature score each feature one at a time. This can lead to missing important features that are useless alone but together with another feature can be informative, that is called, synergistic pairs (for instance xor).

This work proposes SynAFA, which at each step compares best single feature with best unobserved pairs score by their joint info. After this, it buys one feature from the winner. Once the first feature is observed, the latter partner one can be attractive. PID (Partial Information Decomposition) is used to characterise the redundant, unique, and synergistic parts of each pair.

The authors evaluate SynAFA on tabular dataset, MNIST, and feature selection baselines. Empirical evaluation shows that results are mixed: some gains on physioNet (tho these advantages fade when budget grows), losing on MinibooNE, and ties on CKD.

Critical strengths

The controlled experiments are well-designed, includes a fixed information control in the synthetic sweeps, and ablation. Together they isolate when pair proposals help. Negative results are also reported well and honest which is appreciated, including MiniBooNE losses.

Critical weaknesses

I appreciate the study but I believe there are two major concerns here:

  • From the results (mainly Table 24 in the appendix), removing the pair branch changes only very slightly on real data such as PhysioNet, MiniBooNE (the results got better without it). So it is a bit unclear if the synergy mechanism is supported on real data.

  • Although it is a study paper, the novelty and takeaways are somewhat limited, as scoring features by pairwise joint mutual information is classic in information-theoretic and feature selection. Using PID for feature selection has been done before (Wollstadt et al., 2023). Adding clear takeaways from this study (before or in the Conclusion section) would improve this work.

Questions for authors

Suggestion: could you report balanced accuracy or AUROC and main comparisons under a policy-aligned classifier?

Reviewer_bnEMOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The paper studies the role of pairwise synergy in active feature acquisition using the partial information decomposition (PID) framework. The proposed method selects features by comparing marginal and conditional single-feature scores with joint information scores for feature pairs. Experiments on synthetic and real-world binary classification datasets show mixed results, with benefits depending on the dataset and acquisition budget.

Critical strengths
  1. The paper addresses an important problem and explores a promising direction, but its experimental results and scope are limited.
Critical weaknesses
  1. The paper is difficult to read. The authors should substantially revise it to improve readability.

  2. The experimental evaluation is limited to binary classification datasets, and the results are generally mixed. Although the authors do not aim to achieve state-of-the-art performance, demonstrating competitive performance remains important.

Questions for authors
  1. Please provide AUROC and AUPR scores alongside accuracy, as all tasks involve binary classification and some datasets are imbalanced. These additional metrics would provide a more informative evaluation.

  2. The AFA problem is sequential by nature, so scores conditioned on the currently observed features seem more natural. Your results are also consistent with this concern, as your method performs relatively better at lower budgets in several settings, where using unconditional scores may have less negative impact. Could you explain why the pair scores are not conditioned on the currently observed features?

  3. You could achieve sequential selection while accounting for pairwise relationships by calculating pair scores online, as in AACO, at the cost of greater computation during inference. AACO has a variant called AACO-greedy, which evaluates each feature’s conditional contribution to reducing prediction loss. Could you consider a version that evaluates pairs, following the original AACO approach but restricting candidate subsets to at most two features?

  4. You binarize the features to calculate the information tables, but do you also use these binarized features as classifier inputs, or is binarization used only for the information calculations?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.