- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The manuscript studies classification problems where the input data contain "weak features" with strong predictive power and "cues" that are easy to learn and significantly reduce the training loss but do not necessarily have strong predictive power (e.g., background in birds classification dataset). The main finding of this work is that the poor learning of the weak features is not necessarily due to gradient starvation (the poor learning of the weak features is due to competition with the cue) but could result from the intrinsic difficulty of learning the weak features. To show that this phenomenon occurs, the authors propose a metric measuring how the model learns the weak feature. Then, they compare this metric (or the accumulated version) between different learning scenarios (informative/uninformative cue and no cue). When (informative cue run) and (cue free run) are both small, they conclude that the weak features are hard to learn. Conversely, if and (uninformative cue run) are both relatively large while remains small, they consider that as evidence of gradient starvation. Finally, they illustrate their tool on several datasets.
Critical strengths
The interaction between easy-to-learn cues and weaker predictive features, and its consequences for generalization, is an interesting problem.
Critical weaknesses
-
The proposed metric criterion is used to determine whether we are in a gradient starvation situation or if the weak features are intrinsically hard to learn, is arbitrary, could change depending on the dataset, and requires identifying the weak features (or generating them, as in the fashion MNIST experience). But in practice, the weak features and the cues are not known. This considerably limits the applicability of this diagnostic tool.
-
The proposed solution essentially amounts to a matched controlled experiment in which the cue is varied, while the other parameters are held fixed. I don't see particular conceptual novelty here.
-
The advertised mathematical contribution seems also very limited, none of the mentionned theorem is stated in the main text.
Questions for authors
The proposed method distinguishes gradient starvation from hard weak-feature learnability only a posteriori. Are there identifiable structural properties of the data/model that predict which regime occurs?
2 comments
I thank the authors for their rebuttal and for improving the presentation of the theoretical results. However, their response regarding novelty essentially confirms my original concern: the proposed approach consists of controlled experiments combined with heuristic criteria for measuring "learning deficits". While this may be useful in some situations, I remain unconvinced that this constitutes a substantial conceptual contribution, and therefore maintain my negative assessment.
Response to Reviewer j9KJ
We thank the reviewer for the thoughtful assessment. We have uploaded the revised version. B, N and W denote informative-cue, label-independent-cue and cue-free runs.
Addressing Weakness 1: choices and applicability
The response, target, budget and thresholds are fixed before confirmation (App. D). Each family's α* is the smallest grid value at which at most 5% of calibration controls meet the minimum-deficit condition. This prevents tuning to confirmation outcomes, but does not make the choices unique or guarantee a 5% rate on new runs. Three families miss that target. We describe a controlled operational test without established diagnostic error rates.
Threshold stability is limited. Post hoc increases from α=0.25 to 0.40 in F-nl and 0.05 to 0.15 in F-e2r preserve 318/319 and 96/98 informative-run decisions; CIFAR retains 8/8 through α=0.50. F-lin falls from 192/288 to 1/288 at α=1.00 versus 0.75. These conditional frequencies use W-target-reaching pools, additionally restricted to ρ≤4, lag≤4 for F-lin. They do not estimate sensitivity or independently validate new thresholds (Tables 4–5).
New datasets require justified measurements and controls. The method tests a specified cue hypothesis under an available intervention; it does not discover unknown cues (Sections 1 and 6). In CIFAR the weak feature is natural image content; in Waterbirds both bird and background are natural. With known group labels, a reweighted reference agrees with the pixel intervention in 8/8 Waterbirds pairwise decisions, all negative. This does not establish general equivalence.
Responding to Weakness 2: novelty
Cue-present/cue-free comparisons are established, including the shortcut-versus-XOR analysis of LaBonte and Muthukumar [1]. Our contribution combines reference progress, a relative deficit at first target crossing and a signed accumulated deficit against both cue-free and label-independent references, with analysis of these comparisons.
N separates cue-label information from the optimization effect of cue exposure. Under Corollary 1's assumptions, sign-symmetric initialization and a nonzero initial rate give a negative initial gap rate in half the initializations even for an uninformative cue. N excludes 3/617 positive B-versus-W cases in the reported eligible pool, but 29/32 at ρ=16, lag=4, outside F-lin's subset. That cell has 32/32 positive B-versus-W and N-versus-W comparisons but only 3/32 B-versus-N comparisons.
Reference progress prevents attributing failure to the cue when W also fails. At injected amplitude 0.001, W misses its target in all 64 records at tested budgets through H=5,000. The comparison is inconclusive, not evidence of cue-induced suppression or intrinsic unlearnability.
On Weakness 3: theory
Section 4 (pp. 4–6) now states the principal results and assumptions; App. B contains the proofs. Theorem 2(a) considers a noiseless, zero-lag two-layer linear model under unweighted, unregularized logistic gradient flow, weak-feature strength one, a perfectly reliable cue of strength ρ, and nonzero growing modes in both runs. It proves
MwB(t)/MwW(t)→1/(1+ρ2) as t→∞.
Both responses grow while their relative deficit persists. Other results give an exact response-drift decomposition, an initial-rate formula and sign symmetry, and an output-cue analysis. Validated integration certifies finite-time B-versus-W bounds and W's target attainment for 41–51% of 1,000 sampled initializations per configuration across nine small-network configurations. These results do not establish the full B/N/W criterion.
Answering the Question: structural predictors
Cue strength and reliability yield model-specific predictions: Theorem 2(a) gives the limiting relative deficit ρ²/(1+ρ²). In the same model, with an imperfect cue, agreeing and disagreeing examples, and loss below log 2 at some time, Theorem 2(b) proves directional convergence to the weak coordinate; convergence of the B/W response ratio to one remains a conjecture. Weak-signal strength and budget matter through reference progress. Under Proposition 3's assumptions, an output cue can slow progress along the same weak-learning path.
We do not provide a general predictor of the full criterion from initialization. In C1's linear RNN, −d₀ gives AUROC 0.617 [0.604, 0.630] for the B-versus-W condition at H=5, α=0.10. The separate A2-P2 test for any negative N-versus-W gap fails its endpoint: AUROC 0.306 [0.157, 0.471], with frozen score orientation. These are 95% intervals. The operational test uses observed trajectories (App. E.4; Table 13).
We thank the reviewer once again for their time and thoughtful suggestions, which have helped us improve the clarity and precision of the paper.
Reference
[1] LaBonte and Muthukumar. SGD Provably Prioritizes a Shortcut Spurious Feature in the XOR Model. NeurIPS 2026. https://arxiv.org/abs/2606.30444