- Rating
- 3(+1)
- Confidence
- 1
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper proposes AULC, a feature-selection criterion that integrates local response variation across kernel complexity levels. Combined with backward elimination, it selects feature subsets without prescribing their size. Experiments cover synthetic benchmarks, scientific discovery tasks, and medical datasets.
Critical strengths
- The method offers an intuitive criterion for comparing feature representations across matched complexity levels rather than relying on a single predictor configuration.
- The experiments cover diverse dependency structures and directly evaluate feature recovery, distinguishing false selections from missed informative variables.
Critical weaknesses
Numerical optimization may affect AULC rankings, without sufficient stability validation.
For a fixed bandwidth, let denote the linear correction, the solver’s objective, and its true Hessian. The implementation uses and stops when . Although this update can converge to the same minimum, for a positive-definite quadratic objective, Thus, a small gradient need not imply a sufficiently small objective error for ill-conditioned feature sets.
On one fixed standardized correlated-feature dataset, the provided code ranked two subsets in the opposite order to direct weighted least squares using the same bandwidth grid and quadrature, consistently across 20 random initializations. For a fixed initialization, correcting the Hessian or tightening the tolerance restored the reference ranking. The impact on the reported experiments remains unverified; the authors should establish their stability under more accurate optimization.
Questions for authors
- Could the authors clarify whether the subset preferences established for the population-level criterion in Equation (16) also hold for the empirically optimized criterion used in Algorithm 1?
- The SIS reference feature set is motivated by approximate epidemic thresholds, whereas labels come from finite-time stochastic simulations. How is its sufficiency for the actual labels established? Is recovery robust to changes in sampling, simulation horizon, and activity threshold?
7 comments
Thank you for the detailed rebuttal and revised manuscript. The Feynman rerun with the exact Hessian and tighter tolerance alleviates my numerical stability concern for the setting tested, and the revision clarifies the scope of the theoretical analysis. My main remaining concern is the interpretation of the SIS results. Could the authors test whether other candidate features provide additional information about the finite-time simulation labels after conditioning on the reference feature set, particularly near the transition region omitted by the current sampling design? This would help establish whether the reference feature set is sufficient for the labels used in the benchmark.
Further Response to Reviewer Eiw1
Thank you for this insightful question. We think that the main question here is whether additional features can sometimes better describe the SIS outcomes, given that the theoretical control condition is derived under relatively idealized assumptions.
From a data-driven perspective, we think the answer is potentially yes (see Table 1). In five independent experiments on ER networks sampled near the transition region () to obtain relatively balanced labels, we found that adding individual features generally did not reduce AULC. However, in one realization (RUN1), including degree heterogeneity led to a slight reduction in AULC.
Table 1: Evaluation of additional candidate features conditioned on the reference feature set for finite-time SIS simulation labels on ER networks. Results are reported across five independent realizations.
| Run | Sample Number | Average_d, beta, mu | Add var_degree | Add max_degree | Add degree_heterogeneity | Add mean_clustering | Add lambda_max | Add avg_shortest_path | Add rho0 |
|---|---|---|---|---|---|---|---|---|---|
| RUN1 | 1000 | 0.2643 | 0.2768 | 0.2971 | 0.2632 | 0.3073 | 0.2647 | 0.2761 | 0.3279 |
| RUN2 | 1000 | 0.2531 | 0.2711 | 0.2751 | 0.2544 | 0.2929 | 0.2542 | 0.2617 | 0.3190 |
| RUN3 | 1000 | 0.2408 | 0.2624 | 0.2663 | 0.2464 | 0.2885 | 0.2417 | 0.2596 | 0.3054 |
| RUN4 | 1000 | 0.2533 | 0.2664 | 0.2814 | 0.2555 | 0.2999 | 0.2542 | 0.2734 | 0.3133 |
| RUN5 | 1000 | 0.2509 | 0.2687 | 0.2769 | 0.2565 | 0.2975 | 0.2519 | 0.2718 | 0.3162 |
Further response to Reviewer Eiw1
Numerical Implementation We thank the reviewer for highlighting the numerical optimization issue. We acknowledge that the implemented Hessian differs from the exact Hessian. However, since the exact Hessian is positive semidefinite, the additional D/2 term in the implemented Hessian acts as a damping term, similar to damped Newton methods, and generally improves numerical stability (Table 2).
We agree that the stopping tolerance may be too loose for certain datasets, particularly those with small sample sizes or highly correlated features. More fundamentally, feature selection can become ambiguous when the available information is insufficient or features are strongly correlated. The ill-conditioning of the Hessian reflects (Table. 1) this ambiguity from an optimization perspective. We will clarify these limitations and the choice of the original stopping tolerance, ∣df∣<0.001, in the revised manuscript.
To directly assess the impact on our reported results, we repeated the Feynman experiments with 20 samples using the exact Hessian and a tighter tolerance of ∣df∣<10−7. We find that the rankings remain highly consistent. In the only dataset exhibiting a ranking change, two adjacent features exchange positions, but both are identified as informative. These results suggest that the reported feature selection results are robust to the improved optimization accuracy.
Table 1. Logarithm of the Hessian condition number under different sample sizes and noise variances (averaged over 50 realizations).
Table 2. Relative differences in condition numbers between the implemented and exact Hessians under different sample sizes and noise variances (averaged over 50 realizations).
Question 1 Please refer to our response to “Connection between theory and implementation” concerns.
Question 2 Thank you for your question. The data generation process for SIS is deliberately designed to facilitate a comparison between the proposed AULC-BE method and relevant baseline methods on a common benchmark with a known ground-truth relationship, within the context of scientific discovery.
We would also like to reiterate that data-driven scientific discovery fundamentally depends on the quality and informativeness of the available data. This observation is one of the key motivations behind our work, in which we aim to quantify data learnability using kernel-based techniques.
It is important to emphasize that AULC-BE methods alone cannot identify informative features unless the data contain sufficient information to support their discovery. We have also experimented with randomly generated data (e.g., randomly generate μ, β and Λ without balancing the samples). However, this often leads to the discovery of spurious equations that, while possibly capturing patterns present in the data, deviate from the ground-truth relationship.