acceptodds
Withdrawn from ICLR 2027

The Retention Gap: Why Optimal Features Need Not Remain Stationary

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Implicit-bias analyses often explain selection by asking which features training captures. A separate question is whether captured optimal features remain dynamically stable as the residual evolves. This paper isolates that distinction in an explicit two-dimensional regression problem whose minimum representation norm and complete optimal contact geometry are known exactly. An occupied optimal dual contact can lose stationarity as training proceeds; this dynamical failure is termed the retention gap. For a compact family of states already resting on the optimal contacts, a computer-assisted proof certifies a uniform transversal first exit with positive charge, and an exact local calculation shows immediate motion into positive dual slack. An intrinsic canonical-measure identity then connects this local event to function selection: at interpolation, excess representation norm equals canonical mass weighted by signed dual slack. Local loss of retention and persistence to the endpoint are therefore distinct obligations. Complementary full-width experiments lie outside the certified entrance family and do not test its event ordering. Instead, inspected trajectories exhibit the same qualitative sequence of approach, margin loss, and departure at the other contact. Across the robustness sweep, training reaches near-machine-precision loss with a positive endpoint excess diagnostic and a consistent off-contact signature, while splitting the exact contact tie changes the observed scale dependence. Overall, these results show that reaching an optimal feature geometry does not by itself make that geometry dynamically stable, identifying retention as a distinct requirement in feature-capture accounts of implicit bias.

Primary area:
learning theory
Keywords:
Feature learning, optimization dynamics, representation learning

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_r5WZOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper studies implicit bias in two-layer ReLU neural networks and investigates whether neurons will stay on optimal features or leave them later in training. The authors find the phenomenon of leaving optimal features both theoretically and empirically.

Critical strengths

The paper provides both theoretical and empirical evidence.

Critical weaknesses

The example with specific numbers in Section 3 is a good motivating example. However, it would be helpful to provide results beyond this example under some reasonable assumptions that do not restrict the claim to specific numerical values.

Questions for authors

If the authors think their example with specific numbers in Section 3 can be directly extended beyond those specific values, an explanation would be helpful.

1 comment

Thank you to the authors for the effort of extending the result. However, I think the authors' result is still far from a generic characterization of the condition. While some parts became better using the variable q, other parts (e.g., y=(1,0,2,1), (1,0), (0,1) in X_q) still remain as specific values. Furthermore, the interval [0.95, 1.05] for q is quite short. Therefore, I will keep my score.

Reviewer_eLPtOpenReview ↗
Rating
1
Confidence
1
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper studies whether 2-layer ReLU networks which reach an optimal minimum norm solution, remain at that optimum or not. Through complicated arguments it concludes that -- on a specific case with 4 data-points -- the training dynamics lead to an escape from the optimal solution. This theoretical result is also confirmed in a larger-scale experiment where the sequence of events (achieving optimal configuration and then leaving it) is slightly different.

Critical strengths

I'm not familiar enough with the literature but the goal of studying whether training dynamics is stuck in a global optimum, or whether it may free itself from it seems like an important one to tackle.

Critical weaknesses

I found the paper extremely hard to follow. The use of non-defined and non-standard terms (contact, charges, signed cohorts) right from the introduction makes it completely impenetrable. I did not manage to understand the setting well (other than 2-layer relu nets), and also the significance and the relationship with state of the art. I concede the authors the benefit of the doubt since I am definitely not an expert in the field of NN training dynamics and am therefore curious about the other reviewers thoughts.

Questions for authors

Could you try to write the paper using simpler definitions at the start which can be understood using natural language, and later make them more precise with formal notation. This will make the contributions and significance much clearer. I have not been able to get to the point of checking the math so I do not have any further questions.

Reviewer_YRyXOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper studies how optimal features already captured by the model evolve during training and whether the dynamics is stable. The authors introduce the notion of “retention gap” to characterize the potential failure mode and theoretically analyze a two-layer ReLU network on a synthetic toy problem. The results show that at “optimal contacts”, one could still observe loss of stationarity and motion into positive dual slack. It suggests that optimal features need not stabilize for later training. Empirically,

Critical strengths

Central message is of practical importance: The takeaway in this paper suggests that landing on optimal features does not always imply retention over the course of training. This observation follows from the analysis of implicit bias and somewhat challenges existing intuition about the minimum-norm interpolating solution.

Critical weaknesses

Gaps in Theory and Experiments: I identify some key gaps between the contents and the stated novelty of this paper:

  • The main theorem only implies the loss of stationarity and the neuron moving away from the optimal contact, but the introduction also discusses the minimum-norm solution and its implicit bias. This connection is not drawn later on and appears incomplete. If a theoretical explanation is difficult, can you include experiments that help visualize failure mode of the min-norm solutions more explicitly?
  • The experiments essentially replicate the results on the four-point dataset, which is a highly artificial toy setting, despite having varied model configurations. Additional datasets (stylized or real) that represent different structures and geometry will help strengthen the argument and demonstrate how universal the retention gap is.
  • The experiments only imply the loss of stationarity, but as the authors point out in lines 84-86, they do not “validate the theorem's event ordering”. While this is explicitly acknowledged, I find the discrepancy to be a substantial limitation. The claim would be significantly stronger if the authors can further reconcile this contradiction.

Issues in Presentation: Per the other reviewer’s previous public comment, I also find this paper extremely difficult to follow and the central goal is not clear from the beginning for lacking proper motivations. I would regard this as a substantial weakness rather than minor writing issues.

  • The theory is conveyed in a highly fragmented manner, without highlighted definitions and clear organizations. This is especially confusing in the Appendix, where the proof is written in text chunks without proper line breaks. Please consider breaking them into subsections and enhance the overall flow.
  • The authors do not have plots that help visualize the central claim in this paper. Figure 1 and Figure 2a are both conceptual illustrations, and only Figure 2b provides limited evidence. Please consider including better visual cues that directly showcase the failure modes and the “retention gap”.

For now, given the lack of clarity and some limitations in the results, I am leaning towards rejection and will recommend “weak rejection” as of now and sync with other reviewers.

Questions for authors
  1. Do these observations generalize to larger-scale models and datasets beyond the carefully crafted synthetic setting? This can be confirmed from heuristic experiments.
  2. Currently, there appears to be two perturbation values applied to the coordinates. Can you broaden the set and extent of perturbations and further ablate the robustness?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.