- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
- This paper proposes Dual Memory Retrieval (DMR), a unified perspective on self-supervised representation learning with and without negative pairs. The central idea is to interpret negative samples, learned predictors, and prototypes as different forms of memory that encode the representation distribution, and to formulate the learning gradient as the difference between two retrieval operations at different temperatures or spectral exponents. The proposed framework distinguishes retrieval agreement, which encourages consistency between positive views, from retrieval specificity, which aims to prevent representation collapse and recover weak or empty directions. The authors instantiate this framework using softmax and spectral retrieval families, establishing connections to existing methods such as InfoNCE, spectral contrastive learning, DirectPred, DINO, and SimSiam. The theoretical analysis examines the stability of degenerate representations under different retrieval levels and memory update schemes. Experiments on CIFAR-10, STL-10, ImageNet-100, and ImageNet-1k evaluate the proposed framework using kNN and linear-probe accuracy, along with diagnostic measures including effective rank, directional recovery, positive-pair similarity, and shared variance.
Critical strengths
- The paper presents an interesting and reasonably well-motivated perspective on contrastive and non-contrastive self-supervised learning. Treating negative samples, predictors, and prototypes as different forms of memory provides a useful common language for understanding seemingly different learning objectives. The decomposition into positive-guided and self-guided terms is particularly intuitive and helps clarify the distinction between representation alignment and collapse avoidance.
- The theoretical analysis is fairly detailed, and the experiments are designed to examine the proposed framework rather than simply pursue state-of-the-art accuracy. The analysis of directional stability, together with the effective-rank, recovery, and shared-fraction diagnostics, provides useful evidence about how different retrieval levels influence representation learning. The paper is also relatively transparent about its assumptions and limitations.
Critical weaknesses
-
While the unified retrieval perspective is interesting, its connection to several existing methods is somewhat more limited than the overall framing suggests. Much of the analysis assumes a shared memory state, auto-associative retrieval, and representations treated as free points on the unit sphere. However, the actual implementations of DINO and SwAV involve different memory states or additional mechanisms, while BYOL and SimSiam use nonlinear predictors and normalization that fall outside the analyzed retrieval construction. The authors acknowledge these differences, but they weaken the extent to which the theoretical results directly explain the behavior of these existing methods.
-
I also have concerns about the consistency between the theoretical formulation and the provided implementation. In the supplementary code, SpectralInBatch uses the average of the two views as its stop-gradient alignment target, rather than the positive view alone as in Eq. (9). In addition, second_moment() appears to omit the normalization factor in Eq. (22). These differences may affect the actual gradient dynamics and the interpretation of the experiments. The empirical evidence is also somewhat sensitive to the additional potential weight , while the unweighted spectral experiments are limited to one seed on STL-10. A clearer connection between the analyzed objective, its implementation, and the reported results would strengthen my confidence in the conclusions.
Questions for authors
- In the supplementary implementation, SpectralInBatch constructs the alignment target by averaging both views, whereas Eq. (9) uses the positive view directly. Moreover, second_moment() appears to use the unnormalized second-moment matrix, unlike Eq. (22). Could the authors clarify whether these are intentional implementation choices and which formulation was used to produce Tables 1 and 2? There also appear to be smaller configuration discrepancies, such as the CIFAR-10 softmax configuration using k=1 for kNN evaluation, while Appendix F states k=20. Resolving these issues would materially improve my assessment of the experimental soundness.
- The spectral results appear quite sensitive to the potential weight . For example, on STL-10, the (1, 2) configuration changes from an effective rank of 1 and 28.2% kNN accuracy at to an effective rank of 30 and 80.4% accuracy at =20. How much of the observed learning behavior should be attributed to the retrieval-level selection itself versus this additional weighting? A more systematic comparison across values and multiple seeds, especially for the unweighted configurations, would help isolate the central mechanism.
- The theoretical analysis primarily considers idealized representation dynamics and simplified memory updates, whereas existing methods such as DINO, BYOL, and SimSiam introduce additional mechanisms that can substantially affect collapse avoidance. Do the authors have empirical evidence that the predicted relationships between retrieval levels and directional recovery also hold for these methods in their standard implementations? Such evidence would help establish how broadly the proposed framework applies beyond the analyzed variants.
3 comments
Thank you for the detailed response and the additional experiments. The corrected implementation and new results largely address my questions regarding experimental consistency and sensitivity to . However, my main remaining reservation is the framework's applicability to standard SSL methods such as DINO, BYOL, and SimSiam, as several key mechanisms fall outside the scope of the current analysis. Overall, the rebuttal resolves most of my technical issues, though I remain somewhat hesitant about the broader significance of the proposed framework.
Thank you for the follow-up. Below we restate how DMR relates to existing methods, and we give two future directions that show what the framework enables.
Positioning: a shared perspective. DMR does not aim to reproduce each existing implementation. The standard versions of DINO, BYOL and SimSiam keep mechanisms outside the present analysis (App. B.3, G.1). What DMR provides instead is a common description. Methods with and without negatives are written in terms of a memory, a gate, a pair of levels and a retrieval. Each method can then be asked the same two questions: whether the positives agree, and whether empty directions grow back. The value of such a perspective lies in what it allows one to build. Two features of the construction lead to the directions below.
Direction 1: new families from a replaceable gate. A retrieval consists of matching, addressing and response, and the gate acts only in the addressing step. Replacing the gate therefore leaves the rest of the construction in place. For instance, the exponential kernel of the softmax gate can be replaced by the q-exponential of Tsallis statistics (Tsallis, 1988). The resulting gates can have sparse addressing weights. Whenever the new gate meets the gate condition (Prop. 6), as the Tsallis-entropy gate does (Blondel et al., 2020), the retrieval is still the gradient of a potential (Prop. 7). The loss at a pair of levels is then defined as before. The stability question that Cor. 1 answers for the spectral family, whether an empty direction grows, can also be posed for the new family before any training. Each admissible gate thus defines a new family of methods.
Direction 2: from paired views to ordered sequences. The loss of Eq. (6) is conditional. The positive enters only through its retrieval , which is the alignment target of . Nothing requires to be another view of the same sample. CPC (van den Oord et al., 2018) already applied contrastive learning to sequences, contrasting a future representation with negatives given a context of the past. DMR turns this order into a property of the loss itself. Setting and gives the ordered loss . This loss describes how a representation changes along a sequence, rather than how similar two samples are. Attention in Transformers performs the same softmax retrieval over queries, keys and values (Vaswani et al., 2017), which Ramsauer et al. (2021) relate to modern Hopfield networks. When the memory holds the past of a sequence, attention and the ordered loss can retrieve from the same memory. With CPC as the bridge, DMR thus connects contrastive learning to the Transformer architecture.
A conceptual foundation. This paper establishes a conceptual framework. It defines memory, gate, level and retrieval, states the two requirements of retrieval agreement and retrieval specificity, and gives a stability criterion for empty directions. The directions above build on this framework, but they cannot all fit in one paper. We hope this work can serve as their foundation.
References not in the paper
- C. Tsallis. Possible generalization of Boltzmann-Gibbs statistics. Journal of Statistical Physics, 52:479–487, 1988.
- A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser and I. Polosukhin. Attention is all you need. In Advances in Neural Information Processing Systems (NeurIPS), 2017.
Response to Reviewer Ldbh
Thank you for reading the code. We reran every spectral entry with Eq. (9) and (22) exactly (Q1, tables in our reply to XyzG). We re-evaluated the CIFAR-10 softmax checkpoints with k = 20 (Q1, Table R5). Q2 uses the same runs, and Q3 is answered in text.
Q1: implementation. Table 2 and the CIFAR-10, STL-10 and IN-100 spectral rows of Table 1 were produced with the released code, which departs from the paper's equations in two places. Neither changes the stability statements (Cor. 1, Prop. 29), but at β = 1 the second one changes which endpoint a run reaches. Every spectral entry of the revision now follows Eq. (9) and (22) exactly. The reruns confirm Cor. 1 on all four datasets and seeds (reply to XyzG, Table R1), and the code will be released with them.
The theory therefore predicts both endpoints. The numbers differ, which is why we reran every entry.
Q1: CIFAR-10 softmax and k. These cells used k = 1 and a longer linear probe than App. F states. We re-evaluated the checkpoints exactly as App. F states, with k = 20 and the App. F linear probe. Table R5 replaces these cells of Table 1 (submitted values in parentheses):
The order of the rows is unchanged, and so is the claim of Sec. 4.2 that the smallest τ of each family is best or within one point of the best.
Q2: level pair vs. β. They play different roles, and the added runs separate them (reply to XyzG, Tables R1–R2).
Your example, (1,2) going from ER 1 / 28.2 kNN at β = 1 to ER 30 / 80.4 at β = 20, is this boundary moving from about 1 direction to about 24, since the released code scaled β by 1.2 (item 2 above). At an equal boundary, (1/2,1) ≥ (1,2) on all three datasets, consistent with Prop. 1 (the pairs also differ in Δτ).
Q3: standard DINO, BYOL, SimSiam. We do not claim the level–recovery predictions for these implementations. Their decisive elements change the objective rather than add to it.
The two mechanisms that do enter DMR as additions, an EMA memory and a momentum target, are measured in Table R4 (reply to XyzG).
W1: scope. DMR places methods with and without negatives at the level pair at which each retrieves, and each departs from the loss at that pair by the mechanisms the paper names (General Response). The introduction and conclusion will say so in place of "one comparable axis", and the abstract will say that methods with and without negatives can be analyzed in the same terms.