acceptodds
Under review as a conference paper at ICLR 2027

Discovering Feature Subspace through Data Learnability: A Kernel-learning Realization

Abstract

Modern machine learning models can fit high-dimensional data despite irrelevant or redundant features. However, discovering a minimal feature subspace that preserves the underlying predictive structure remains challenging. A key difficulty is that feature utility is typically assessed through a particular model, and is therefore entangled with model-specific inductive biases. Here, we revisit the feature subspace discovery problem from a different perspective: the representation-induced learnability of the observed data. Our key idea is that an informative feature subspace should organize data into suitable structures, which benefit model performance across a range of complexities. To quantify this property, we construct kernel-based estimators and track empirical fitting performance as a function of model complexity, measured by the trace of the normalized Gram matrix. This defines a learning curve, and the Area Under the Learning Curve (AULC) provides a scalar measure of data learnability. AULC provides a principled criterion for comparing feature subspaces. Experiments across synthetic and real-world datasets demonstrate its effectiveness in feature subspace discovery and related scientific variable discovery tasks.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Feature Subspace Discovery, Data Learnability, Kernel Learning, Scientific Discovery

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_Eiw1OpenReview ↗
Rating
3(+1)
Confidence
1
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes AULC, a feature-selection criterion that integrates local response variation across kernel complexity levels. Combined with backward elimination, it selects feature subsets without prescribing their size. Experiments cover synthetic benchmarks, scientific discovery tasks, and medical datasets.

Critical strengths
  1. The method offers an intuitive criterion for comparing feature representations across matched complexity levels rather than relying on a single predictor configuration.
  2. The experiments cover diverse dependency structures and directly evaluate feature recovery, distinguishing false selections from missed informative variables.
Critical weaknesses

Numerical optimization may affect AULC rankings, without sufficient stability validation.

For a fixed bandwidth, let bb denote the linear correction, J(b)=2I^(b)J(b)=2\widehat I(b) the solver’s objective, and H=∇2J(b)H=\nabla^2J(b) its true Hessian. The implementation uses M=H+diag⁡(diag⁡H)2M=\frac{H+\operatorname{diag}(\operatorname{diag}H)}{2} and stops when ∥∇J(b)∥2⩽10−3\|\nabla J(b)\|_2\leqslant 10^{-3}. Although this update can converge to the same minimum, for a positive-definite quadratic objective, I^(b)−I^(b⋆)=14∇J(b)⊤H−1∇J(b).\widehat I(b)-\widehat I(b^\star)=\frac14\nabla J(b)^\top H^{-1}\nabla J(b). Thus, a small gradient need not imply a sufficiently small objective error for ill-conditioned feature sets.

On one fixed standardized correlated-feature dataset, the provided code ranked two subsets in the opposite order to direct weighted least squares using the same bandwidth grid and quadrature, consistently across 20 random initializations. For a fixed initialization, correcting the Hessian or tightening the tolerance restored the reference ranking. The impact on the reported experiments remains unverified; the authors should establish their stability under more accurate optimization.

Questions for authors
  1. Could the authors clarify whether the subset preferences established for the population-level criterion in Equation (16) also hold for the empirically optimized criterion used in Algorithm 1?
  2. The SIS reference feature set is motivated by approximate epidemic thresholds, whereas labels come from finite-time stochastic simulations. How is its sufficiency for the actual labels established? Is recovery robust to changes in sampling, simulation horizon, and activity threshold?
7 comments
AuthorsOpenReview ↗

Further response to Reviewer Eiw1

Numerical Implementation We thank the reviewer for highlighting the numerical optimization issue. We acknowledge that the implemented Hessian differs from the exact Hessian. However, since the exact Hessian is positive semidefinite, the additional D/2D/2 term in the implemented Hessian acts as a damping term, similar to damped Newton methods, and generally improves numerical stability (Table 2).

We agree that the stopping tolerance may be too loose for certain datasets, particularly those with small sample sizes or highly correlated features. More fundamentally, feature selection can become ambiguous when the available information is insufficient or features are strongly correlated. The ill-conditioning of the Hessian reflects (Table. 1) this ambiguity from an optimization perspective. We will clarify these limitations and the choice of the original stopping tolerance, ∣df∣<0.001|df|<0.001, in the revised manuscript.

To directly assess the impact on our reported results, we repeated the Feynman experiments with 20 samples using the exact Hessian and a tighter tolerance of ∣df∣<10−7|df|<10^{-7}. We find that the rankings remain highly consistent. In the only dataset exhibiting a ranking change, two adjacent features exchange positions, but both are identified as informative. These results suggest that the reported feature selection results are robust to the improved optimization accuracy.

Table 1. Logarithm of the Hessian condition number under different sample sizes and noise variances (averaged over 50 realizations).

Noise Variance σ2\sigma^22060100200300400
10−710^{-7}21.3616.5414.2511.1014.7114.27
10−510^{-5}21.5016.8415.066.4510.119.69
10−310^{-3}22.2018.0011.253.875.515.10
10−110^{-1}24.6119.6718.721.521.531.31
1124.7621.4815.011.491.290.90

Table 2. Relative differences in condition numbers between the implemented and exact Hessians under different sample sizes and noise variances (averaged over 50 realizations).

Noise Variance σ2\sigma^22060100200300400
10−710^{-7}-0.0006-0.0396-0.1463-0.9208-0.9987-0.9981
10−510^{-5}0.0000-0.0003-0.0006-0.9348-0.9984-0.9983
10−310^{-3}-0.07460.0000-0.0002-0.1071-0.9731-0.9793
10−110^{-1}-0.98100.00000.0000-0.0068-0.3483-0.3529
11-0.00410.00000.0000-0.0057-0.0157-0.0222

Question 1 Please refer to our response to “Connection between theory and implementation” concerns.

Question 2 Thank you for your question. The data generation process for SIS is deliberately designed to facilitate a comparison between the proposed AULC-BE method and relevant baseline methods on a common benchmark with a known ground-truth relationship, within the context of scientific discovery.

We would also like to reiterate that data-driven scientific discovery fundamentally depends on the quality and informativeness of the available data. This observation is one of the key motivations behind our work, in which we aim to quantify data learnability using kernel-based techniques.

It is important to emphasize that AULC-BE methods alone cannot identify informative features unless the data contain sufficient information to support their discovery. We have also experimented with randomly generated data (e.g., randomly generate μ\mu, β\beta and Λ\Lambda without balancing the samples). However, this often leads to the discovery of spurious equations that, while possibly capturing patterns present in the data, deviate from the ground-truth relationship.

Thank you for the detailed rebuttal and revised manuscript. The Feynman rerun with the exact Hessian and tighter tolerance alleviates my numerical stability concern for the setting tested, and the revision clarifies the scope of the theoretical analysis. My main remaining concern is the interpretation of the SIS results. Could the authors test whether other candidate features provide additional information about the finite-time simulation labels after conditioning on the reference feature set, particularly near the transition region omitted by the current sampling design? This would help establish whether the reference feature set is sufficient for the labels used in the benchmark.

AuthorsOpenReview ↗

Further Response to Reviewer Eiw1

Thank you for this insightful question. We think that the main question here is whether additional features can sometimes better describe the SIS outcomes, given that the theoretical control condition is derived under relatively idealized assumptions.

From a data-driven perspective, we think the answer is potentially yes (see Table 1). In five independent experiments on ER networks sampled near the transition region (q∈[0.9,1.6]q \in [0.9, 1.6]) to obtain relatively balanced labels, we found that adding individual features generally did not reduce AULC. However, in one realization (RUN1), including degree heterogeneity led to a slight reduction in AULC.

Table 1: Evaluation of additional candidate features conditioned on the reference feature set {⟨k⟩,β,μ}\{\langle k\rangle, \beta, \mu\} for finite-time SIS simulation labels on ER networks. Results are reported across five independent realizations.

RunSample NumberAverage_d, beta, muAdd var_degreeAdd max_degreeAdd degree_heterogeneityAdd mean_clusteringAdd lambda_maxAdd avg_shortest_pathAdd rho0
RUN110000.26430.27680.29710.26320.30730.26470.27610.3279
RUN210000.25310.27110.27510.25440.29290.25420.26170.3190
RUN310000.24080.26240.26630.24640.28850.24170.25960.3054
RUN410000.25330.26640.28140.25550.29990.25420.27340.3133
RUN510000.25090.26870.27690.25650.29750.25190.27180.3162
Reviewer_14ZUOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes a model-agnostic criterion for selecting a feature subset. The authors suggest the property of "how learnable" a selected subset makes the data. That is, fit a Nadaraya-Watson style Gaussian kernel smoother to (X_S, Y), measure the local residual variation I, and sweep the bandwidth h. Complexity is measured as C = tr(W)/n, where W is the row-normalised Gram matrix. Plotting I against C gives a learning curve, and the area under it (AULC) is the score. Subsets are then compared by AULC at matched complexity rather than matched bandwidth.

Critical strengths
  1. The actual idea is nice and, as far as I can tell, not in the feature selection literature in this form: compare subsets at equal realised smoother complexity (trace of the hat matrix) and integrate over the complexity axis so that you do not have to pick one bandwidth. This removes the usual "which model do I wrap" question, and the ablation with KRR-BECV (same kernel, same greedy search, CV error instead of AULC) is the right comparison to include.

  2. The results on the SIS threshold task and the Madelon-like task are convincing (near perfect support recovery, all baselines clearly worse), and the piecewise example makes the point that the criterion does not need a global functional form. The SIS experiment, where the method picks k for ER and lambda_max for BA, is a genuinely nice scientific-discovery demonstration.

Critical weaknesses
  1. The mechanism that rejects over-selection is purely dimensional, and I think this is the central soundness issue. Lemma 1 says the bandwidth at fixed C scales like n^{-2/d}, where d is the ambient dimension of the candidate subset, and Prop. 3 needs X_R independent of X^* (plus regular density on a compact d^+-dimensional support). In other words, what is really being penalised is the intrinsic dimension of the sample in the selected coordinates, not the number of columns. If the redundant features are (noisy) functions of X^, the data stay close to a d^-dimensional set, the bandwidth ratio in (22) does not diverge, and the argument collapses. But this is precisely the setting used in the experiments: Feynman adds "noisy copies of informative variables", Madelon adds linear transforms of the effective variables. So the theory does not cover the experiments that matter, and the experiments where the theory does apply (independent noise columns) are the easy ones. Conversely, in a problem with many genuinely independent but weak informative variables, Lemma 1 works against you, which Fig. 4 shows rather clearly: EMR drops to zero for d^* >= 6-7 no matter how many samples you have. The paper calls this "conservative"--> One would call it a curse-of-dimensionality failure mode of the criterion and it should be discussed as such.

  2. Empirically the case is mixed, and the writing oversells it. On Feynman (the largest benchmark, 120 systems) AULC-BE is worse than SR, KRR-BECV and ANN-BECV in EMR and recall at every n (Fig. 3); recall plateaus around 0.85. The direct ablation KRR-BECV, i.e. the same kernel and the same search with a CV criterion, beats the proposed criterion here. That undercuts the main story that "predictive performance is confounded by model complexity and AULC fixes this". The real-data section has no baselines and no downstream check at all; "4 of 8 features overlap with Tsanas et al." is not evidence either way. And the complexity measure tr(W)/n is the effective degrees of freedom of a linear smoother (Hastie & Tibshirani 1990, Buja et al. 1989). The paper does not mention this, although the whole framework is "compare at matched d.o.f.", which puts it closer to GCV-type arguments than the related-work section admits.

Questions for authors
  • Does Prop. 3 survive if X_R = g(X^*) + small noise (the Feynman/Madelon setting)? If not, what is the argument for why AULC-BE works there empirically. Is it the finite-sample noise level in the copies, i.e. would it break for exact duplicates?
  • Fig. 4: is the EMR collapse for d^* >= 6 a sample-size issue that goes away at n = 10^4, or a property of the criterion? One run with larger n on a few high-d^* Feynman equations would settle this.
  • Why does KRR-BECV beat AULC-BE on Feynman? Same kernel, same search. If the answer is "CV picks a single bandwidth per subset while AULC averages over all of them", then that is an argument against integrating over C.
  • How is tr(W)/n related, in your view, to the effective degrees of freedom of the smoother, and why not simply say so?
1 comment
AuthorsOpenReview ↗

Further Response to Reviewer 14ZU

We thank the reviewer for the insightful and professional comments. We would like to provide further clarification on the following points.

Critical Weakness 1 and Question 1. According to Eq. (17), the regression function is defined through the conditional expectation mX(x)=E[Y∣X=x]m^X(x)=\mathbb{E}[Y\mid X=x]. The residual decomposition in Line 930 therefore relies on the conditional mean remaining unchanged after introducing redundant features, namely E[Y∣X∗,XR]=E[Y∣X∗]E[Y | X^* , X^R ] = E [ Y | X ^ {*} ].

This condition is guaranteed by $Y \perp X^R | X^* $, which is weaker than the marginal independence assumption $X^R \perp X^* $. In particular, it can hold when XR=g(X∗)+εX^R=g(X^*)+\varepsilon, as in our noisy-copy experiments.

Consequently, the residual and gradient decompositions in Lines 930–936 remain valid for correlated redundant features. However, extending the full conclusion of Proposition 3 to noisy conditional-response models additionally requires accounting for the conditional variance term in Eq. (19), as well as the local regularity assumptions underlying the bandwidth scaling.

Critical Weakness 1 and Question 2. Our additional experiments show that increasing the sample size can help the algorithm recover the correct feature subspace in systems where it previously failed. Here, we report how the AULC values change with increasing sample size for Feynman Equation 33:

Y=12ϵcEf2⋅8πr23⋅ω4(ω2−ω02)2.Y=\frac{1}{2}\epsilon c E_f^2\cdot\frac{8\pi r^2}{3}\cdot\frac{\omega^4}{(\omega^2-\omega_0^2)^2}.

At n=1200n=1200, the algorithm under-selects the informative features EfE_f, ω\omega, ω0\omega_0, and rr. Table 1 reports the AULC values for the ground-truth feature subspace X∗X^*, the subspace without cc (denoted by X−X^-), and the subspace selected at n=1200n=1200 (denoted by X1200X_{1200}).

Table 1. Comparison of AULC values across different feature subspaces and sample sizes. In experiments, the backward elimination sequence keeps stable after n=1200n=1200.

Sample SizeX∗X^*X−X^-X1200X_{1200}
90000.2790.2850.289
80000.2830.2880.295
70000.2890.2940.297
60000.2970.2980.300
50000.3010.3030.306
40000.3070.3090.308
30000.3200.3180.315
20000.3160.3140.313
12000.3440.3310.329

We agree that this phenomenon can be better understood from the perspective of the curse of dimensionality, which we will further discuss in the revised manuscript.

Question 3 (Difference between KRR-BECV and AULC-BE). We believe this difference arises because comparing data variation at matched model complexity CC makes AULC more conservative toward high-dimensional feature subspaces, whereas KRR-BECV evaluates predictive performance through cross-validation.

We also acknowledge that integrating over CC can sometimes introduce undesirable effects. Fundamentally, this reflects a trade-off between a more sensitive feature selection criterion and a more conservative one. Please refer to our response to concerns related to the design of integrating over complexity levels for further discussion.

Finally, we would like to clarify that these two methods differ in their architectures and underlying mechanisms, which may lead to different behaviors. KRR relies on the representer theorem to find an optimal function in an RKHS, whereas our method employs kernel density estimation (KDE) to approximate the joint distribution.

Question 4. Please refer to our response to comments on the definition of model complexity and the ablation of the linear correction ΣXY\Sigma_{XY}. And this definition can be better understood in light of Eq. (4) in the main text, as it essentially characterizes self-influence.

Reviewer_q38fOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies feature subset discovery through a kernel-based measure of data learnability. For each candidate subset, a Gaussian kernel defines sample neighborhoods, and a learned linear correction removes part of the input–output trend. The remaining local variation is evaluated across bandwidths and plotted against the normalized trace of the kernel weight matrix. The resulting area under the learning curve (AULC) serves as the objective for backward elimination, allowing the number of retained features to be selected without being specified in advance. The paper analyzes population-level preferences under restricted noiseless assumptions and evaluates feature recovery on synthetic and simulated scientific systems, followed by exploratory applications to two medical datasets.

Critical strengths

The multiscale view provides a useful and interpretable feature-selection criterion. The method asks whether a representation makes the response locally coherent across a range of neighborhood scales. This is a meaningful way to assess feature subsets beyond their performance with a single tuned predictor. The variation decomposition in Appendix A.1 supports this intuition by separating residual signal variation from conditional uncertainty. Together, the formulation and analysis provide a clear rationale for why omitting informative variables can be unfavorable.

The controlled experiments provide encouraging evidence for recovering interacting variables. Evaluating exact support recovery, precision, and recall is well aligned with the paper's objective. The results on the piecewise system, SIS simulations, and constructed Madelon-like task are particularly promising (Figs. 5–7 and 13), and are supported by repeated realizations. These examples cover both regime-dependent relationships and higher-order interactions. The paper also reports weaker results on the Feynman benchmark and examines the effect of informative dimension, which helps delineate where the approach is effective.

Critical weaknesses

Matching the proposed complexity coordinate does not establish matching complexity of the fitted estimator. Equation (8) accounts for the kernel weights but not the linear correction learned through Eq. (7). In the uniform-weight limit, the fitted estimator reduces to ordinary linear regression: with a full-rank centered design containing d features, its fitted-value map has trace d+1, whereas Eq. (8) gives C=1/n regardless of d. The distinction can matter substantially in finite samples. If the centered candidate design has rank n−1, the linear correction can interpolate arbitrary centered responses, making the optimized variation zero at every bandwidth and hence AULC=0. This does not contradict the paper's fixed-dimensional asymptotic results, but it exposes an operating regime that the method does not address. Moreover, Eq. (16) analyzes the infimum of expected variation, rather than the empirically minimized quantity used by the algorithm. A justification connecting these quantities, and a more precise interpretation of C, are needed to support the central claim of comparison at matched empirical model complexity.

The validation leaves the specific contribution of curve integration and the real-data usefulness unresolved. Appendix B.1 observes that a single complexity level can suffice under the theoretical assumptions, but no controlled comparison establishes the benefit of integrating the full curve. An ablation of the learned linear correction is also missing. These comparisons are needed to identify which components explain the reported recovery gains. The medical experiments provide AULC trajectories and qualitative agreement with previously reported variables, but no baseline comparison, resampling stability, or held-out evaluation (Sec. 4.6, Fig. 8). Because the true subsets are unknown, these results alone do not establish that the selected features preserve the relevant information. This limits the evidence for the method's distinctive contribution and practical value.

Questions for authors

Complexity interpretation: How should C be interpreted once the linear correction is fitted to the responses? Can you reconcile Eq. (8) with the dimension-dependent degrees of freedom in the linear-regression limit, and clarify the intended operating regime and treatment of interpolating candidate designs?

Connection between theory and implementation: What connects the population-profiled criterion in Eq. (16) to the empirical minimization in Algorithm 1? Is there an estimation or concentration argument showing that the population preference is preserved after fitting the coefficients and comparing candidate subsets?

Evidence for the central design choice: Under the same backward-elimination search, how does full AULC compare with several fixed-complexity scores and with AULC using the linear correction fixed to zero? Do the gains on the piecewise, SIS, and Madelon-like tasks persist in these comparisons?

Evidence for the real-data conclusions: How stable are the selected medical features under resampling, and how do their held-out results compare with the full feature set and the baseline-selected subsets? Such an evaluation should perform feature selection within the training data and account for repeated measurements by subject in the Parkinson's dataset.

1 comment
AuthorsOpenReview ↗

Response to Reviewer q38f

We thank the reviewer for the insightful and constructive comments. We address the main concerns point by point below, with references to our detailed responses and additional experiments.

1. Integration over complexity levels and the linear correction. Please refer to our responses titled “Response to Concerns Regarding Integration Over Complexity Levels” and “Response to Concerns Regarding Model Complexity and the Linear Correction,” where we clarify the motivations behind these design choices and present additional ablation experiments to evaluate their contributions.

2. Evaluation on medical datasets. Please refer to our “Response to Concerns Regarding the Medical Dataset Experiments,” where we discuss the challenges of held-out evaluation and provide additional resampling experiments to assess the stability of feature rankings.

3. Connection between theory and implementation. Please refer to our “Response to Concerns Regarding the Connection Between Theory and Implementation,” where we clarify the distinction between the empirical and theoretical criteria and discuss their relationship.

Comments on the paper

5 comments
AuthorsOpenReview ↗

Response to “Connection between theory and implementation” concerns

We thank the reviewers for raising this important concern regarding the connection between the empirical algorithm and the theoretical analysis. We agree that this distinction deserves further clarification in the revised manuscript.

Distinction between the empirical and theoretical criteria. We first clarify the distinction between two related quantities:

  • Empirical criterion, inf⁡ΣXYI^nX(h(c),Sn)\inf_{\Sigma_{XY}}\hat{I}_n^X(h(c),S_n): The minimum empirical data variation achieved by optimizing ΣXY\Sigma_{XY} on a given dataset SnS_n. This is the quantity directly computed in Algorithm 1.

  • Theoretical criterion, inf⁡ΣXYESn∼PXYI^nX(h(c),Sn)\inf_{\Sigma_{XY}}E_{S_n\sim P_{XY}}\hat{I}_n^X(h(c),S_n): The minimum expected empirical variation, obtained by optimizing a single fixed ΣXY\Sigma_{XY} over the sampling distribution. This is the population-profiled criterion defined in Eq. (16) and analyzed in our theoretical results.

Motivation for the theoretical formulation. Our choice of Eq. (16) is motivated by the central concept of this work: data learnability. Although our method aims to identify informative feature subspaces, successful recovery depends not only on the evaluation criterion but also on whether the observed data contain sufficient information to reveal the underlying input-output structure.

By taking the expectation over datasets sampled from the underlying distribution, Eq. (16) allows us to investigate data learnability through the statistical properties of the data-generating distribution, rather than relying on a particular sample realization, which may be subject to sampling bias or random fluctuations. Therefore, our theoretical results characterize subspace preferences under idealized distributional conditions, rather than guaranteeing the same ordering for every empirically optimized dataset.

Connection to finite-sample experiments. Our repeated experiments complement the theoretical analysis by directly evaluating feature recovery from individual datasets. By examining performance across independent realizations and increasing sample sizes, we investigate how reliably the proposed empirical algorithm identifies the informative feature subspace from finite observations.

AuthorsOpenReview ↗

Response to comments on the definition of model complexity and the ablation of the linear correction

We thank the reviewers for raising these important concerns regarding the definition of model complexity and its relation to existing concepts. We agree that these aspects require further clarification in the revised manuscript.

Definition and interpretation. Both the implemented indicator and the model complexity CC characterize the relationships beyond those captured by the linear correction. Specifically, our learner f^\hat{f} incorporates a learned parameter ΣXY\Sigma_{XY} to remove the global linear dependence between XX and YY. This design has several implications:

  • In Eqs. (4) and (5), we estimate ΣXY\Sigma_{XY} from data rather than fixing it to zero. The conventional Nadaraya–Watson estimator can be recovered as a special case when ΣXY=0\Sigma_{XY}=0.

  • From the perspective of feature selection, we seek a feature subspace in which samples exhibit smooth local relationships and small local variations.

  • The parameter ΣXY\Sigma_{XY} is not included when quantifying the model complexity CC, which specifically characterizes the complexity of the kernel-smoothing component.

Motivation for the definition. This distinction is not imposed arbitrarily but follows from the construction of our estimator and complexity measure. Both originate from kernel density estimation of the joint distribution PXYP_{XY}. Allowing more parameters in the Gaussian kernel to be optimized increases model flexibility but also introduces substantial computational costs. Although optimizing all kernel parameters would provide greater flexibility, doing so is computationally expensive (see the discussion and Table 1 in the Appendix). Our formulation therefore represents a compromise between model flexibility and computational tractability.

This construction leads to two important properties:

  1. For linear systems, the proposed indicator yields an AULC close to zero when the linear correction ΣXY\Sigma_{XY} is included (Figure 1). In contrast, fixing ΣXY=0\Sigma_{XY}=0 results in a substantially larger AULC, even though the underlying relationship is purely linear.

  2. The proposed complexity measure CC is not equivalent to the effective degrees of freedom of the resulting linear smoother, although the two quantities share certain structural similarities. We will clarify their relationship and distinctions in the revised manuscript.

Ablation experiments. The ablation results reveal a trade-off introduced by the linear correction.

On the one hand, estimating ΣXY\Sigma_{XY} improves the sensitivity of the framework to subtle dependencies. This is illustrated by the Feynman experiments (Table 1), where the full method achieves higher exact match rates (EMR) than the ablated version when sufficient samples are available (n>80n>80). However, the ablated version achieves higher PPV, indicating that the increased sensitivity of the full method can also make feature selection less conservative.

On the other hand, estimating ΣXY\Sigma_{XY} can introduce additional estimation errors when the available data are insufficient. This is particularly evident in the Madelon-like experiments, where fixing ΣXY=0\Sigma_{XY}=0 increases EMR to 100% from n=100n=100. Further investigation suggests that the optimal linear correction for these systems is approximately zero (Figure 3), which can also be understood by noting that Y∼sign⁡(∏i=15Xi)Y\sim\operatorname{sign}\left(\prod_{i=1}^{5}X_i\right). In this setting, estimating an unnecessary correction can introduce errors rather than improve the representation of the underlying relationships.

Overall, these results indicate that the linear correction can improve feature selection by capturing global linear dependencies, particularly when sufficient data support its accurate estimation. However, it also introduces additional estimation uncertainty, which can reduce robustness when the true correction is negligible or the sample size is limited. We will clarify this trade-off and the corresponding limitations in the revised manuscript.

Table 1. Ablation experiments (ΣXY\Sigma_{XY}) on Feynman datasets. These experiments are performed and calculated on the first realization of Feynman physics experiments.

Sample Size204080150300600
EMR (Original)0.030.210.370.510.600.62
EMR (Ablation)0.060.160.340.420.480.52
PPV (Original)0.310.620.830.900.960.98
PPV (Ablation)0.650.830.890.940.970.98
Recall (Original)0.910.830.780.790.810.85
Recall (Ablation)0.370.540.620.690.740.78
AuthorsOpenReview ↗

Response to concerns related to the design of integrating over complexity levels

We thank the reviewer for raising concerns regarding the design choice of integrating over complexity levels. We provide further clarification of its motivation and report additional ablation experiments to evaluate its effect on feature selection performance. We will incorporate these discussions and results into the revised manuscript.

Interpretation of complexity levels. We first clarify the interpretation of the learning curves (data variation versus model complexity). In our framework, the complexity level CC controls the effective size of the neighbourhood used to evaluate whether samples exhibit a smooth and representative structure, as measured by data variation. A smaller CC corresponds to a larger neighbourhood. Consequently, for a fixed CC, increasing the sample size reduces the corresponding neighbourhood radius in the input space (Lemma 1), allowing the estimator to progressively resolve finer local structures of the underlying system.

Motivation for integrating over complexity levels. Suppose that samples are independently drawn from a distribution PXYP_{XY}, and that the underlying conditional relationship is sufficiently regular (e.g., continuous). As the sample size increases, the additional information about the underlying structure may first become detectable at higher complexity levels, corresponding to smaller neighbourhoods, while remaining unresolved at lower complexity levels.

By integrating data variation across complexity levels, AULC assesses improvements across a range of neighbourhood scales rather than relying on a single complexity level. In this sense, the integration provides a conservative criterion for evaluating the evidence supporting a candidate feature subspace. Furthermore, we observe that learning curves corresponding to different feature subsets can intersect (e.g., Figure 1 in the original manuscript), meaning that their relative performance may depend on the chosen complexity level. Integrating over complexity levels therefore provides a more general evaluation criterion that avoids the need to select an additional complexity-level hyperparameter.

Implications for feature selection.

The same principle applies when evaluating the inclusion of additional features. A newly introduced feature is favoured when it produces a sufficiently consistent reduction in data variation across complexity levels. This discourages feature inclusion based solely on improvements observed at particular neighbourhood scales.

The resulting conservativeness is also related to the curse of dimensionality: in higher-dimensional feature spaces, more samples are generally required to reliably characterize local structures. Thus, integrating over complexity levels helps balance the evidence for improved data representation against the additional dimensionality introduced by new features.

Ablation experiments on Feynman and Madelon.

To further examine this design choice, we conduct ablation experiments on the Feynman and Madelon benchmarks, reporting EMR, PPV, and recall across different sample sizes (Figures 4 and 5). Note that the ablation experiments use a fixed complexity level CC at 0.5 instead of the full AULC.

On Feynman, the original and ablated methods achieve broadly comparable performance across all three metrics, suggesting that the integration strategy does not substantially affect feature selection when the relevant structures can be identified reliably by both methods.

On Madelon, however, the original method shows a clearer advantage. In particular, at sample sizes of 500 and 600, the original method achieves an EMR of 1.0, while the ablated method achieves approximately 0.9 and 0.8, respectively. The original method also generally exhibits higher recall at intermediate sample sizes, while maintaining comparable or better PPV.

These results suggest that integrating over complexity levels can improve the reliability of feature selection in settings with more challenging feature structures, while maintaining comparable performance in the Feynman benchmark. They also support our motivation for evaluating candidate feature subspaces across multiple complexity levels rather than relying on evidence obtained at a single scale.

Table 1. Ablation study on the Feynman benchmark.

Sample Size40150300600
EMR0.210.510.600.62
EMR (Ablation)0.230.510.610.64
PPV0.620.900.960.98
PPV (Ablation)0.710.890.960.98
Recall0.830.790.810.85
Recall (Ablation)0.780.790.830.86

Table 2. On the Madelon-like benchmark.

Sample Size200400500600
EMR (Original)0.160.901.001.00
EMR0.200.900.900.80
PPV (Original)0.390.931.001.00
PPV0.360.960.980.97
Recal l (Original)0.630.931.001.00
Recall0.340.981.001.00

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.