- Rating
- 2(+1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
This paper proposes CLASP, an unsupervised feature-selection method for high-dimensional, small-sample biomedical data.
Critical strengths
-
The paper studies an important problem.
-
The proposed method is intuitive and relatively easy to implement.
-
The paper is well-written, organized, and easy to follow.
-
To the best of my knowledge, this is a novel approach.
Critical weaknesses
Since I work on theory, I have found some theoretical issues in the paper.
-
The paper introduces data subsamples, but Eq. (1) only computes the silhouette score for a single free index . It does not contain an average such as . Therefore, the claimed cross-subsample consensus mechanism is not mathematically specified in the current formulation.
-
The sampling rule uses , whereas the theorem and its assumption use and the signed row sums of . Non-uniform signed row sums do not imply non-uniform row sums for . Thus, Theorem 1 is not valid for the stated algorithm under its current assumptions.
-
The monotonicity of the elite threshold is asserted but not established for the changing sampling distribution. Full support alone does not imply that the upper quantile eventually reaches the global optimum. Moreover, the assumptions state , whereas the proof directly substitutes after a finite iteration. The experiments instead use a fixed . The proof also treats as a fixed deterministic function, although it is estimated using random data subsamples in the actual algorithm. Consequently, the theorem does not cover the implemented setting with finite , fixed smoothing, stochastic objectives, and early stopping.
Questions for authors
-
How exactly are the losses from the data subsamples aggregated in Eq. (1)? Are subsamples regenerated for every mask and iteration?
-
Why does Theorem 1 analyze while the algorithm uses ?
-
Can the authors provide a genuine sample-complexity result relating the mask budget to the probability of sampling near-optimal masks?
-
What guarantees that the elite thresholds are monotonic under the changing correlation-augmented sampling distribution?
-
How does Theorem 2 apply to the practical setting with , , a stochastic objective, and early stopping?
-
Can the authors report complete ablations for the correlation prior, data subsampling, and standard CEM?
7 comments
Response to Reviewer zwVL (1/4)
W1/Q1: The paper introduces data subsamples, but Eq. (1) only computes the silhouette score for a single free index . The claimed cross-subsample consensus mechanism is not mathematically specified in the current formulation.
We thank R4 for bringing up this point, which was also brought up by R1. The noted error is a typo. The implemented objective is , with . We have corrected this typo in the manuscript. The subsamples are redrawn independently for every mask.
W2/Q2: The sampling rule uses , whereas the theorem and its assumption use and the signed row sums of .
We thank R4 for pointing this out. We have updated Theorem 1 to use and modified assumptions. This theorem was also extended with a condition under which the optimizer moves to a good mask.
W3/Q3: The monotonicity of the elite threshold is asserted but not established for the changing sampling distribution. Full support alone does not imply that the upper quantile eventually reaches the global optimum. Moreover, the assumptions state approaches 1, whereas the proof directly substitutes after a finite iteration. The experiments instead use a fixed . The proof also treats the loss as a fixed deterministic function, although it is estimated using random data subsamples in the actual algorithm. Consequently, the theorem does not cover the implemented setting.
We thank R4 for highlighting this point, also brought up by R1. Accordingly, we have updated the theory section of the Appendix to reflect more realistic scenarios. Specifically, Theorem 1 has been expanded beyond arguing a search-space reduction to now including the conditions under which we approach a better mask, and the required sampling budget in terms of a failure probability . Theorem 2 has been updated to account for the finite-sample scenario. We derive almost-sure convergence and a geometric attraction to strict local optima. We recognize that the updated theorems have tempered the claims in our original manuscript, and we have modified the manuscript text to reflect this change.
Regarding the requested sample-complexity result (Q3), Theorem 1(c) addresses this in terms of a failure probability and the probability of sampling a near-optimal mask, and shows that the correlation prior can reduce the required number of masks. This bound applies to a single iteration, not to the whole run.
A.1 Setup
Masks are . The inclusion probabilities have entries in that sum to , starting from . Each iteration does three things:
- It draws masks. Each mask is formed by draws without replacement (equivalently, Gumbel top- sampling), picking feature with probability proportional to . Here is the correlation reward of Eq. (2) and . denotes the law of one mask.
- It scores each mask by , the silhouette averaged over random subsamples.
- It updates .
For a set of features, the margin is . For a mask , the outside mass is .
Response to Reviewer zwVL (2/4)
A.2 Theorem 1
Let be a set of features.
- (a) At initialization, if the rewards are not all equal, the prior () strictly lowers the entropy of the sampler compared with , which is uniform over masks.
- (b) If , then at every draw, given the features drawn so far, the odds that the next feature lies in are at least times their value at . For every , the probability that the mask contains at least features of is at least as large as at . If every feature outside has a reward at least instead, both statements hold in reverse. In other words, the prior amplifies whichever group of features has the highest reward, including a nuisance module.
- (c) masks suffice to sample, with probability at least , at least one mask from a set of masks of probability . For with and , the prior multiplies this budget by at most the factor , where . This is a statement about one iteration, not a budget for the whole run.
Proof sketch.
- (a) At and every mask is equally likely, so the entropy equals minus the KL divergence from the uniform law. An exchange argument on the Gumbel noise shows that a feature with a larger reward is included strictly more often. So the sampler is not uniform, and the divergence is positive.
- (b) The weights in are multiplied by at least and those outside by at most , so the odds ratio is at least . Coupling both samplers on the same noise, the prior raises every score in relative to every score outside it, which can only increase .
- (c) We have . For , exactly when every Gumbel score in beats every score outside , so can only drop if we lower the weights in to and raise those outside to (after rescaling). This sampler gives every ordering of the same weights as at , with less weight outside , so we can compare the two order by order and bound each of the factors.
ICLR Rebuttal: Please Submit Your Response Soon
This is a reminder that the ICLR rebuttal period is ending soon. I would appreciate it if the authors could submit their response as soon as possible so that I can finalize my review.