acceptodds
Under review as a conference paper at ICLR 2027

Dual Memory Retrieval: Bridging Contrastive Learning With and Without Negative Pairs

Abstract

Contrastive learning builds self-supervision by comparing positive and negative samples. Methods such as SimSiam, DirectPred and DINO remove the negatives and resist collapse with a predictor or prototypes. Their contrast is carried jointly by the objective and these modules, rather than by the objective alone. We treat the negatives, the predictor and the prototypes alike as a memory of the representation distribution and recast contrast as retrieval from this memory, placing methods with and without negatives on one comparable axis. Revisiting contrastive learning from this perspective, we propose Dual Memory Retrieval (DMR). Its gradient is the difference between the retrievals of the two positive views at two temperatures in a softmax family or two exponents in a spectral family. We examine the resulting learning behavior on four image datasets. With a memory derived directly from the batch, the gap between the two temperatures or exponents is essential for training. With a memory learned by gradient steps, training fails when the positive enters the loss directly. Retrieving the positive through the memory lets the spectral family train with a learnable matrix, but not the softmax family without further mechanisms.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Self-supervised Learning, Contrastive Learning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_LdbhOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary
  • This paper proposes Dual Memory Retrieval (DMR), a unified perspective on self-supervised representation learning with and without negative pairs. The central idea is to interpret negative samples, learned predictors, and prototypes as different forms of memory that encode the representation distribution, and to formulate the learning gradient as the difference between two retrieval operations at different temperatures or spectral exponents. The proposed framework distinguishes retrieval agreement, which encourages consistency between positive views, from retrieval specificity, which aims to prevent representation collapse and recover weak or empty directions. The authors instantiate this framework using softmax and spectral retrieval families, establishing connections to existing methods such as InfoNCE, spectral contrastive learning, DirectPred, DINO, and SimSiam. The theoretical analysis examines the stability of degenerate representations under different retrieval levels and memory update schemes. Experiments on CIFAR-10, STL-10, ImageNet-100, and ImageNet-1k evaluate the proposed framework using kNN and linear-probe accuracy, along with diagnostic measures including effective rank, directional recovery, positive-pair similarity, and shared variance.
Critical strengths
  • The paper presents an interesting and reasonably well-motivated perspective on contrastive and non-contrastive self-supervised learning. Treating negative samples, predictors, and prototypes as different forms of memory provides a useful common language for understanding seemingly different learning objectives. The decomposition into positive-guided and self-guided terms is particularly intuitive and helps clarify the distinction between representation alignment and collapse avoidance.
  • The theoretical analysis is fairly detailed, and the experiments are designed to examine the proposed framework rather than simply pursue state-of-the-art accuracy. The analysis of directional stability, together with the effective-rank, recovery, and shared-fraction diagnostics, provides useful evidence about how different retrieval levels influence representation learning. The paper is also relatively transparent about its assumptions and limitations.
Critical weaknesses
  • While the unified retrieval perspective is interesting, its connection to several existing methods is somewhat more limited than the overall framing suggests. Much of the analysis assumes a shared memory state, auto-associative retrieval, and representations treated as free points on the unit sphere. However, the actual implementations of DINO and SwAV involve different memory states or additional mechanisms, while BYOL and SimSiam use nonlinear predictors and normalization that fall outside the analyzed retrieval construction. The authors acknowledge these differences, but they weaken the extent to which the theoretical results directly explain the behavior of these existing methods.

  • I also have concerns about the consistency between the theoretical formulation and the provided implementation. In the supplementary code, SpectralInBatch uses the average of the two views as its stop-gradient alignment target, rather than the positive view alone as in Eq. (9). In addition, second_moment() appears to omit the normalization factor in Eq. (22). These differences may affect the actual gradient dynamics and the interpretation of the experiments. The empirical evidence is also somewhat sensitive to the additional potential weight β\beta, while the unweighted spectral experiments are limited to one seed on STL-10. A clearer connection between the analyzed objective, its implementation, and the reported results would strengthen my confidence in the conclusions.

Questions for authors
  • In the supplementary implementation, SpectralInBatch constructs the alignment target by averaging both views, whereas Eq. (9) uses the positive view directly. Moreover, second_moment() appears to use the unnormalized second-moment matrix, unlike Eq. (22). Could the authors clarify whether these are intentional implementation choices and which formulation was used to produce Tables 1 and 2? There also appear to be smaller configuration discrepancies, such as the CIFAR-10 softmax configuration using k=1 for kNN evaluation, while Appendix F states k=20. Resolving these issues would materially improve my assessment of the experimental soundness.
  • The spectral results appear quite sensitive to the potential weight β\beta. For example, on STL-10, the (1, 2) configuration changes from an effective rank of 1 and 28.2% kNN accuracy at β=1\beta=1 to an effective rank of 30 and 80.4% accuracy at β\beta=20. How much of the observed learning behavior should be attributed to the retrieval-level selection itself versus this additional weighting? A more systematic comparison across β\beta values and multiple seeds, especially for the unweighted configurations, would help isolate the central mechanism.
  • The theoretical analysis primarily considers idealized representation dynamics and simplified memory updates, whereas existing methods such as DINO, BYOL, and SimSiam introduce additional mechanisms that can substantially affect collapse avoidance. Do the authors have empirical evidence that the predicted relationships between retrieval levels and directional recovery also hold for these methods in their standard implementations? Such evidence would help establish how broadly the proposed framework applies beyond the analyzed variants.
3 comments
AuthorsOpenReview ↗

Response to Reviewer Ldbh

Thank you for reading the code. We reran every spectral entry with Eq. (9) and (22) exactly (Q1, tables in our reply to XyzG). We re-evaluated the CIFAR-10 softmax checkpoints with k = 20 (Q1, Table R5). Q2 uses the same runs, and Q3 is answered in text.

Q1: implementation. Table 2 and the CIFAR-10, STL-10 and IN-100 spectral rows of Table 1 were produced with the released code, which departs from the paper's equations in two places. Neither changes the stability statements (Cor. 1, Prop. 29), but at β = 1 the second one changes which endpoint a run reaches. Every spectral entry of the revision now follows Eq. (9) and (22) exactly. The reruns confirm Cor. 1 on all four datasets and seeds (reply to XyzG, Table R1), and the code will be released with them.

  1. Averaged target. The code uses sg[(z+z+)/2]\mathrm{sg}[(z+z^+)/2] in place of z+z^+. It comes from a multi-crop variant of the implementation, which averages the views of each sample into one target, and the two-view runs kept it. The gradient becomes ΔR(z)+12χτ(z∣z+)\Delta R(z)+\tfrac12\chi_\tau(z\mid z^+). The self-guided term, which carries Cor. 1 and Prop. 29, is unchanged, and the positive-guided term is halved. Cor. 1 and Prop. 29 assume coinciding views, and there the two targets give the same gradient.
  2. Normalization of the memory state. The code omits the factor 1/(1+dϵ)1/(1+d\epsilon) of Eq. (22). Without it, FtF^t scales by (1+dϵ)t(1+d\epsilon)^t, so the loss becomes (1+dϵ)τLβ′(1+d\epsilon)^\tau\mathcal L_{\beta'} with β′=(1+dϵ)Δτβ\beta'=(1+d\epsilon)^{\Delta\tau}\beta. At d = 2048 and ε = 10⁻⁴, this rescales β by 1.2 (Δτ = 1) or 1.1 (Δτ = 1/2). At β = 1 the rescaling matters. It moves the boundary of Prop. 29 from one direction to 1 + dε ≈ 1.2, so the rank-one configuration becomes unstable at every pair with τ > 0 and Δτ > 0. The instability is fastest at (1/2,1), where the submitted run ended at two directions (kNN 67.1). With Eq. (22), all three seeds end at one (kNN 24.5, Table R1).

The theory therefore predicts both endpoints. The numbers differ, which is why we reran every entry.

Q1: CIFAR-10 softmax and k. These cells used k = 1 and a longer linear probe than App. F states. We re-evaluated the checkpoints exactly as App. F states, with k = 20 and the App. F linear probe. Table R5 replaces these cells of Table 1 (submitted values in parentheses):

kNNLP
(0.01T, T)90.9 (89.7)90.7 (91.1)
(0.1T, T)83.8 (80.3)80.7 (84.2)
(T, T)10.0 (10.0)10.0 (10.0)

The order of the rows is unchanged, and so is the claim of Sec. 4.2 that the smallest τ of each family is best or within one point of the best.

Q2: level pair vs. β. They play different roles, and the added runs separate them (reply to XyzG, Tables R1–R2).

  • The pair decides whether empty directions grow back. At β = 1 the 10 runs at τ = 0 train, and the 30 runs at τ > 0 collapse (Cor. 1, and Sec. 4.2 for (1,1)).
  • For τ > 0 and Δτ > 0, β sets a minimum number of directions (Prop. 29, App. G). Below β1/Δτ\beta^{1/\Delta\tau} equally occupied directions, an empty direction is unstable. In the sweep, every run ends above this boundary, and the effective rank grows monotonically with it.

Your example, (1,2) going from ER 1 / 28.2 kNN at β = 1 to ER 30 / 80.4 at β = 20, is this boundary moving from about 1 direction to about 24, since the released code scaled β by 1.2 (item 2 above). At an equal boundary, (1/2,1) ≥ (1,2) on all three datasets, consistent with Prop. 1 (the pairs also differ in Δτ).

Q3: standard DINO, BYOL, SimSiam. We do not claim the level–recovery predictions for these implementations. Their decisive elements change the objective rather than add to it.

  • DINO. Centering with a momentum teacher balances the load of the prototypes, as Sinkhorn does in SwAV. It acts on the prototypes' usage and each query's addressing weights together, outside the retrieval (App. B.3).
  • BYOL, SimSiam. The decisive element is where the ℓ2 normalization sits. After the predictor, it multiplies the retrieval at τ + Δτ by a gain ω(z, z⁺) that depends on both views (Eq. 41), so the loss differs from loss (6) at every pair. Halvagal et al. (2023) show the same split. The cosine loss couples the eigenmodes of a linear predictor, while the Euclidean loss, the form our spectral loss takes, decouples them. App. G.1 will cite them at Eq. (41).

The two mechanisms that do enter DMR as additions, an EMA memory and a momentum target, are measured in Table R4 (reply to XyzG).

W1: scope. DMR places methods with and without negatives at the level pair at which each retrieves, and each departs from the loss at that pair by the mechanisms the paper names (General Response). The introduction and conclusion will say so in place of "one comparable axis", and the abstract will say that methods with and without negatives can be analyzed in the same terms.

Thank you for the detailed response and the additional experiments. The corrected implementation and new results largely address my questions regarding experimental consistency and sensitivity to β\beta. However, my main remaining reservation is the framework's applicability to standard SSL methods such as DINO, BYOL, and SimSiam, as several key mechanisms fall outside the scope of the current analysis. Overall, the rebuttal resolves most of my technical issues, though I remain somewhat hesitant about the broader significance of the proposed framework.

Thank you for the follow-up. Below we restate how DMR relates to existing methods, and we give two future directions that show what the framework enables.

Positioning: a shared perspective. DMR does not aim to reproduce each existing implementation. The standard versions of DINO, BYOL and SimSiam keep mechanisms outside the present analysis (App. B.3, G.1). What DMR provides instead is a common description. Methods with and without negatives are written in terms of a memory, a gate, a pair of levels and a retrieval. Each method can then be asked the same two questions: whether the positives agree, and whether empty directions grow back. The value of such a perspective lies in what it allows one to build. Two features of the construction lead to the directions below.

Direction 1: new families from a replaceable gate. A retrieval consists of matching, addressing and response, and the gate acts only in the addressing step. Replacing the gate therefore leaves the rest of the construction in place. For instance, the exponential kernel of the softmax gate can be replaced by the q-exponential of Tsallis statistics (Tsallis, 1988). The resulting gates can have sparse addressing weights. Whenever the new gate meets the gate condition (Prop. 6), as the Tsallis-entropy gate does (Blondel et al., 2020), the retrieval is still the gradient of a potential (Prop. 7). The loss at a pair of levels is then defined as before. The stability question that Cor. 1 answers for the spectral family, whether an empty direction grows, can also be posed for the new family before any training. Each admissible gate thus defines a new family of methods.

Direction 2: from paired views to ordered sequences. The loss LΦ(z∣z+)\mathcal L_\Phi(z\mid z^+) of Eq. (6) is conditional. The positive z+z^+ enters only through its retrieval Rτ(z+;M)R_\tau(z^+;M), which is the alignment target of zz. Nothing requires z+z^+ to be another view of the same sample. CPC (van den Oord et al., 2018) already applied contrastive learning to sequences, contrasting a future representation with negatives given a context of the past. DMR turns this order into a property of the loss itself. Setting z=ztz=z_t and z+=zt−1z^+=z_{t-1} gives the ordered loss LΦ(zt∣zt−1)\mathcal L_\Phi(z_t\mid z_{t-1}). This loss describes how a representation changes along a sequence, rather than how similar two samples are. Attention in Transformers performs the same softmax retrieval over queries, keys and values (Vaswani et al., 2017), which Ramsauer et al. (2021) relate to modern Hopfield networks. When the memory holds the past of a sequence, attention and the ordered loss can retrieve from the same memory. With CPC as the bridge, DMR thus connects contrastive learning to the Transformer architecture.

A conceptual foundation. This paper establishes a conceptual framework. It defines memory, gate, level and retrieval, states the two requirements of retrieval agreement and retrieval specificity, and gives a stability criterion for empty directions. The directions above build on this framework, but they cannot all fit in one paper. We hope this work can serve as their foundation.

References not in the paper

  • C. Tsallis. Possible generalization of Boltzmann-Gibbs statistics. Journal of Statistical Physics, 52:479–487, 1988.
  • A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser and I. Polosukhin. Attention is all you need. In Advances in Neural Information Processing Systems (NeurIPS), 2017.
Reviewer_XyzGOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper treats negatives, predictors and prototypes as a memory of the representation distribution and recasts contrast as retrieval from it. In Dual Memory Retrieval (DMR) the gradient is the difference between retrievals of the two views at two levels, temperatures in a softmax family or exponents of the batch second moment F in a spectral family, which places SimCLR, DirectPred and DINO at level pairs. Corollary 1 predicts which spectral pairs keep empty directions alive. On CIFAR-10, STL-10, IN-100 and IN-1k (Table 1) a zero interval fails and the smallest τ is best or close, with STL-10 diagnostics in Table 2 and learned memories in Sec. 4.3.

Critical strengths

I think the identity g on p.6 is the real payoff. In one line it shows why τ = 0 keeps weak directions alive while τ > 0 shrinks them by λ^τ, and it turns SCL, DirectPred and DirectCopy into points with falsifiable predictions (Cor. 1, Prop. 29). The β = 1 ranks in Table 2 (ER 57 at (0,1), 2 and 1 at (1/2,1) and (1,2)) fit Cor. 1(i)-(ii), and every number I rechecked holds up.

Critical weaknesses

My main concern is that the spectral rows of Table 1 aren't DMR as defined. They use L_β with β = 20 or 40 (p.7), which App. E.1 (p.41) says "falls outside Definition 3", and the main text never says so. DMR proper (β = 1) only shows up in Table 2, one STL-10 seed per pair, and there every τ > 0 pair collapses: (1/2,1) ends at ER 2 and 67.1 kNN, (1,2) at ER 1 and 28.2. So β alone takes (1,2) from 28.2 to 80.4, and the p.8 claim that (1,2) beats (1/2,1) "on every dataset" flips at β = 1. β is never swept, and by your own Prop. 29 it isn't neutral across pairs (boundary 20 for (1,2) vs 400 for (1/2,1) at β = 20). Honestly, I think Table 1 tells me more about β than about the level pair.

I'm also not convinced the "one comparable axis" (abstract, p.2, conclusion) says much about practice. Published DirectPred and DirectCopy without the floor reach 76.8 and 76.6 STL-10 linear top-1 at (1/2,1) and (1,2), right where DMR collapses. You list their EMA F and momentum target as departures, but then either the pair doesn't decide collapse or those methods aren't on the axis, and the appendix already puts DINO and deployed BYOL/SimSiam off it (App. B.3.1, B.5). And correct me if I'm wrong, but Sec. 4.3 mostly reproduces known results: learned (1,2) is BYOL/SimSiam with a linear predictor (Tian et al., 2021), and softmax (0.04, 0.1) is DINO without centering or a momentum teacher.

Questions for authors

Here are some of my questions:

  1. Could you run β = 1 with three seeds on all four datasets, plus a β sweep, and say how β was picked? If τ > 0 pairs do train at β = 1 elsewhere, most of my main concern goes away.
  2. I'm curious whether the β = 1 collapse comes from the pair or from d ≫ B (2048 vs a batch of 256). Learned (1,2) reaches about 67 kNN on STL-10 (Fig. 2) at d = 128 and batch 512. Does a derived run at d = 128 still collapse?
  3. Your best spectral row has UniGrad's loss form (Tao et al., CVPR 2022), cited only in App. B. Could the main body credit it, and RDM (Zhuo et al., ICLR 2023)?
3 comments
AuthorsOpenReview ↗

Response to Reviewer XyzG (1/2)

Response to Reviewer XyzG (1/2)

Thank you for the close check. We add experiments for W1 and Q1 (Tables R1–R2), Q2 (Table R3) and W2 (Table R4), and answer Sec. 4.3 and Q3 in text. This comment covers W1 and Q1, and the next one covers the rest.

W1, Q1: β. Table 1's spectral rows use Lβ\mathcal L_\beta of Sec. 3.2.3 with β = 20/40 (Sec. 4.1, L367–369), and Prop. 29 gives the effect of β. β = 20/40 was tuned for a multi-crop variant of our implementation, and the two-view runs kept it. The revision states in Sec. 3.2.3 that β ≠ 1 lies outside Definition 3. Tables 1–2 now use β = 1, and a new Table 3 sets β from Prop. 29. All new runs use the target z+z^+ (Eq. 9), FF from Eq. (22) and the settings of App. F. kNN / LP are top-1 (%), mean ± half-range over three seeds.

Table R1. β = 1. Replaces the spectral rows of Tables 1–2.

pairCIFAR-10STL-10IN-100IN-1k†
(0,1)89.9±0.2 / 89.6±0.282.3±0.1 / 86.0±0.257.7±0.2 / 66.3±0.447.8 / 55.0
(1/2,1)29.0±0.8 / 13.3±2.224.5±2.0 / 16.1±4.26.5±1.4 / 2.1±0.82.6 / 0.7
(1,1)30.1±1.0 / 17.7±3.826.0±1.1‡ / 19.2±3.3‡7.2±0.5 / 6.5±0.41.1 / 1.6
(1,2)29.4±1.8 / 12.1±2.226.0±0.1 / 12.1±1.65.2±0.8 / 2.6±1.12.1 / 0.3

† One seed and 100 epochs. The IN-1k runs at τ > 0 and the runs marked ‡ were stopped early once they had collapsed.

Only τ = 0 trains, and all 40 runs agree with the paper's prediction. The effective rank (ER) of (0,1) ends at 15.8 / 27.7 / 31.7 / 46.1 on the four datasets. All 30 runs at τ > 0 end at ER 1.00, a single direction. On STL-10 (new Table 2), recovery is 0.004 at (0,1) and below 0.001 at τ > 0. The submitted single-seed (1/2,1) run ended at two directions and the new runs at one. The released code omitted the factor of Eq. (22), which raises the effective β above 1, and by Prop. 29 rank one is then unstable at this pair. With Eq. (22), rank one is stable at ε = 10⁻⁴ (App. E.1). The theory thus predicts both endpoints (reply to Ldbh, Q1).

Table R2. β set by Prop. 29. Your point that one β is not neutral across pairs is Prop. 29. A weight β moves the stability boundary of an empty direction to β1/Δτ\beta^{1/\Delta\tau} occupied directions. So β = 20 puts it at 20 directions for (1,2) and at 400 for (1/2,1). At β = 1 the boundary is one direction, which is Cor. 1(ii). Below the boundary, an empty direction of an equally occupied configuration grows, so a run should not end there. It may end above. We chose β for each pair so that the boundary is 4, 16, 64 and 256 directions (STL-10):

pairβboundary β1/Δτ\beta^{1/\Delta\tau}ERkNNLP
(1/2,1)248.080.284.1
41639.082.986.5
86425382.886.5
16256169780.883.4
(1,2)445.078.680.9
161620.681.485.2
646411581.985.2
25625648064.464.5

No run ends below the boundary, and ER grows monotonically with it on every seed. We then fixed the boundary at 16 by a rule set in advance: the smallest value whose mean kNN over the two pairs is within 0.5 of the best. With this boundary we ran all four pairs on three datasets (new Table 3, kNN / LP, half-ranges ≤ 0.6 except (1,1)):

pair (β)CIFAR-10STL-10IN-100
(0,1) (16)88.9 / 88.383.2 / 87.062.7 / 68.5
(1/2,1) (4)89.5 / 89.682.9 / 86.559.5 / 66.7
(1,1) (16)37.1 / 33.127.6‡ / 23.4‡1.6 / 3.5
(1,2) (16)89.4 / 88.881.4 / 85.255.9 / 64.8

‡ One of the three seeds was stopped early. The pairs (0,1) and (1,1) have no boundary (Prop. 29) and use β = 16, like (1,2). At (0,1) the weight changes accuracy, not whether directions return.

  • ER is nearly the same on the three datasets: 39.5 / 39.0 / 40.4 for (1/2,1) and 22.9 / 20.6 / 21.9 for (1,2) on CIFAR-10 / STL-10 / IN-100.
  • At the same boundary, (1/2,1) ≥ (1,2) on every dataset, consistent with Prop. 1. The pairs also differ in Δτ and in β. This replaces the p.8 sentence, which compared them at one β. Your example, (1,2) going from 28.2 to 80.4 kNN, is its boundary moving from about 1 direction to about 24, since the released code scaled β by 1.2 (reply to Ldbh).
  • With a zero interval, (1,1) keeps a high ER while its positive-pair cosine stays at 0.004–0.051, as Sec. 4.2 states.

Both predictions you single out, Cor. 1 and Prop. 29, thus hold in the new runs.

AuthorsOpenReview ↗

Response to Reviewer XyzG (2/2)

Response to Reviewer XyzG (2/2)

Q2: the pair or d ≫ B? d ≫ B is not the cause, and Cor. 1(ii) does not assume it. At d = 128 no direction is structurally empty (Table R3, STL-10, (1,2), β = 1):

d2BkNNLPER
204851226.0±0.112.1±1.61.00
12851227.7±1.415.9±1.41.00

All three seeds still end at one direction. This is the rank-one configuration, which App. E.1 finds stable at (1,2) for any d ≥ 3 as ε → 0. So d ≫ B does not explain why the derived run collapses while the learned (1,2) run of Fig. 2, also at d = 128, trains. App. G ("Batch estimates") will report Table R3.

W2: DirectPred and DirectCopy. Within DMR the pair decides collapse (Cor. 1, Table R1). DirectPred and DirectCopy start from DMR at (1/2,1) and (1,2) and add what App. B.4 and G.1 name. Two of these are additions, an EMA second moment and a momentum target. The third changes the objective, normalization after the predictor (Eq. 41). We added both to DMR, singly and together (Table R4, STL-10, β = 1, three seeds, kNN / LP / ER):

memorypositive(1/2,1)(1,2)
batchonline24.5 / 16.1 / 1.0026.0 / 12.1 / 1.00
batchmomentum67.6 / 65.1* / 1.9925.7 / 14.8 / 1.00
EMAonline19.6 / 10.2 / 1.0724.4 / 16.0 / 1.00
EMAmomentum68.1 / 72.1 / 1.9924.4 / 16.7 / 1.00

* LP varies across seeds (53.4–72.6), while kNN does not (67.4–67.8).

  • (1/2,1), DirectPred's pair. The momentum target lifts kNN from 24.5 to 67.6–68.1, while the projection keeps about two directions (ER 1.99). The downstream performance holds, but the ER is very low. An EMA memory alone leaves ER at 1.07.
  • (1,2), DirectCopy's pair. Neither addition changes ER 1.00. Your numbers are for the variants without the floor, so the remaining differences include normalization after the predictor, which App. G.1 places outside the analysis.

The asymmetry matches the rank bound of App. E.1 at β = 1, which allows up to three balanced directions at (1/2,1) and only one at (1,2). A mechanism that changes the path of training can thus leave (1/2,1) at two directions but not (1,2).

A published method therefore departs from the loss at its pair by the mechanisms the paper names, and those added to the loss can be measured one at a time. Table R4 goes into App. B.4.

Sec. 4.3. Yes. L462–463 already describe the softmax run as DINO without the momentum teacher and centering. Sec. 4.3 will add that learned (1,2) is SimSiam with a linear predictor, which Tian et al. (2021) also train and analyze. Within the objectives compared here, Sec. 4.3 tests a condition for a learned memory: the positive must be retrieved through it. The runs that violate it fail in both families (spectral (0,1) and softmax (0⁺, 0.1) in Fig. 2). The two that meet it reproduce the known cases, one training and one not, so the condition is necessary but not sufficient.

Q3: credit. Yes. UniGrad's loss is Lβ\mathcal L_\beta at (0,1), and its running second moment is a derived memory, so DMR places it with the spectral contrastive loss rather than with the learned predictors. Sec. 2 will credit it, and Sec. 3.3.2 will state this, together with its momentum target. Sec. 2 will also cite RDM among the explanations of collapse avoidance without negatives.

Thank you for the response and the new runs. So I checked other reviewer's concerns and check your rebuttal and paper again. The d = 128 check and the added citations address those points, but my main concern remains. At β = 1 only (0,1) trains, so the τ > 0 pairs rely on a β outside Definition 3, and Table R4 shows DirectCopy's pair still collapses. With Tables 1 and 2 replaced and a new Table 3, I think the revision is too large to assess here. I am maintaining my original score at this moment.

Reviewer_E3jwOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

DMR puts contrastive and non-contrastive SSL into a common memory-retrieval picture. It separates matching two views from preserving their shared information, and studies how temperature and spectral weighting affect that balance.

Critical strengths

In the simplified spectral model, aligning the views doesn't guarantee that a disappearing feature direction will come back. How the positive branch treats weak directions matters, not just the gap between branches. That's actual explanatory content, not just a change of notation.

Critical weaknesses
  1. The problem is the gap between that result and the proposed unification of actual SSL methods. The clean stability analysis lets representation points move freely and derives memory from their covariance. DINO has distinct teacher/student memories and centering; BYOL and SimSiam have nonlinear predictors and output normalization. How those pieces co-evolve with the encoder is outside the central analysis, as the authors acknowledge. Those aren't minor details if the goal is to explain why these methods work. (and they're all works several years ago)
  2. At that narrower scope, I see a useful conditional stability result. I don't yet see it resolve a major uncertainty about the real learning systems, or give a new design principle that carries over to them. The retrieval language makes methods comparable, but comparison isn't yet explanation.
Questions for authors
  1. What concrete change to a working SSL recipe follows from the weak-direction stability result, beyond what's already suggested by earlier spectral analyses?
  2. Which conclusions survive when memory and encoder evolve together, including the actual teacher and predictor mechanisms rather than simplified proxies?
3 comments
AuthorsOpenReview ↗

Response to Reviewer E3jw

Thank you. We share your reading of the core result. Below, the new runs address the scope concern for memories derived from the batch (tables in our reply to XyzG). We answer Q1 with three concrete changes to a training recipe and Q2 by locating what the teacher mechanisms do. We close with what the result says about published methods.

Scope, and comparison vs. explanation. The stability results are stated for a memory derived from the batch, and a published method departs from the loss at its level pair by its own mechanisms (General Response). The theory treats representations as free points (App. G, "Encoder parameters"), while the runs train real encoders. Within this scope the theory predicts the outcome before training, and the new runs confirm it (tables in our reply to XyzG). The predicted split between the pairs holds in all 40 runs at β = 1 on four datasets (Cor. 1, Table R1). No run of (1/2,1) or (1,2) ends below the Prop. 29 boundary β1/Δτ\beta^{1/\Delta\tau} (Table R2). These are predictions made before the runs, not comparisons after them.

Q1: concrete changes to a recipe.

  1. Retrieve the positive at τ = 0, and no weight is needed against collapse. By Cor. 1(i), every empty direction then grows back, for every n and ε. Accordingly, (0,1) trains on all four datasets at β = 1 (Table R1). Earlier analyses explain why a predictor aligns with the eigenspaces of F (Tian et al., 2021) and why asymmetric branches avoid collapse (Zhuo et al., 2023). Cor. 1 adds that the level of the positive decides, not the gap. The pairs (0,1) and (1,2) share Δτ = 1 and end oppositely.
  2. With τ > 0, set β from the number of directions to keep. By Prop. 29, an empty direction grows back while fewer than β1/Δτ\beta^{1/\Delta\tau} directions are occupied, so the weight converts between pairs. For example, β = 4 at (1/2,1) and β = 16 at (1,2) both keep at least 16. We chose 16 on STL-10 by a rule set in advance and reused it. It gave nearly the same effective rank on CIFAR-10 and IN-100, about 40 for (1/2,1) and about 22 for (1,2) (Table R2).
  3. Check both requirements, not one number. With the weight, (1,1) reaches ER 839–1953, while COS stays at 0.004–0.051 and accuracy is near chance on IN-100 (Table R2). This is specificity without agreement. Sec. 4.1 assigns COS to agreement and ER, REC and SF to specificity.

Q2: memory and encoder evolving together. The analysis takes the memory as given (App. G.1). In the runs, a derived memory is recomputed from the encoder at every step, so the two evolve together, and the predictions above hold there. With a learned memory, DMR locates what the actual teacher mechanisms do. Sinkhorn's optimal transport (SwAV, DINOv2) and DINO's centering with a momentum teacher both act outside the retrieval. Both balance the load in the sense of App. B.3: they constrain the usage of the prototypes and each query's addressing weights together. The first uses a correction computed from the current batch, and the second a correction carried across steps. A constraint on usage alone would not suffice, because merged prototypes already use the memory uniformly (App. B.3). Predictor methods keep a spectral memory, which has no usage to balance. For them, the decisive element is the normalization after the predictor (reply to Ldbh, Q3).

Significance. You call one result explanatory content: how the positive is retrieved decides, not the gap alone. This result also locates where published methods get their stability. At β = 1, the level pairs of DirectPred and DirectCopy collapse to one direction without those methods' additions (Table R1). Their stability therefore comes from what they add, and the DMR loss at each pair is the baseline against which each addition can be tested. The momentum target lifts DirectPred's pair to 67–68 kNN, while neither moving average lifts DirectCopy's (Table R4). The principle that carries over comes from the theory. At τ = 0 the objective keeps every direction by itself. At τ > 0 a weight β sets β1/Δτ\beta^{1/\Delta\tau} as the least number of directions. For learned memories, the additions of DINO and SwAV balance the load of the prototypes (Q2).

Reviewer_E3jwOpenReview ↗

final rating: 2 - Weak rejection

The corrected experiments strengthen the spectral stability result, and the β calibration provides useful guidance within that setting. However, my main concern remains: mechanisms central to standard DINO, BYOL, and SimSiam remain outside the analysis. Naming these departures clarifies the framework’s limits; it does not explain their behavior. The proposed gate and sequence extensions are future directions, not evidence addressing this gap.

I therefore maintain 2 (Weak rejection). The rebuttal improves technical credibility, but does not sufficiently establish the significance of the broader unification.

We thank the reviewer for the follow-up and for acknowledging the corrected experiments and the β\beta calibration. We agree that DINO, BYOL and SimSiam cannot be written exactly in the DMR form, but each of them corresponds to a trainable mode of DMR. The detailed analysis is in Appendix B.3.1 for DINO and in Appendix B.5 for BYOL and SimSiam.

Take sharpening as an example. DINO presents sharpening, together with centering, as a mechanism against collapse. SwAV uses a similar mechanism: its Sinkhorn targets are sharper than its predictions. DMR makes this mechanism explicit: sharpening lifts the alignment term to a positive level below that of the potential term. DMR also gives the general counterpart of this mechanism in the spectral family, which BYOL and SimSiam realize through their predictor.

On significance, adding components to reach state-of-the-art performance is a contribution. In our view, removing them to study the minimal framework that still trains is also a contribution. Components added for performance make methods look unrelated. DMR describes them in one standardized retrieval language and asks only whether a configuration trains, so they become comparable and can be classified by their trainable modes. We will state this scope in the introduction and point to Appendices B.3.1 and B.5 from the main text.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

We thank the reviewers.

What DMR offers. DMR reads the gradient of a contrastive pair as the difference of two retrievals from one memory at two levels. This one reading covers methods with negatives, with a predictor and with prototypes, and it turns their main design choice into one variable, the level pair. DMR explains collapse rather than describing it. On the spectral family the gradient is g=Fτ(FΔτz−z+)g=F^\tau(F^{\Delta\tau}z-z^+), and the factor FτF^\tau shrinks every empty direction at τ > 0 and leaves it untouched at τ = 0. So τ, the level at which the positive is retrieved, decides whether an empty direction returns, not the gap Δτ: pairs with the same gap end oppositely in the new runs. DMR also gives the weight on the potential term a meaning: β sets a lower bound, β1/Δτ\beta^{1/\Delta\tau}, on the number of directions kept. Finally, the bridge in the title: DMR places methods with and without negatives at the level pair at which each retrieves. Each departs from the loss at its pair by mechanisms that the paper names. Those added to the loss can be measured one at a time, as experiment 4 does for two of them, and others, such as normalization after the predictor, change the objective.

Key new experiments

  1. Spectral runs exactly as defined. We reran every spectral configuration with β = 1 (Definition 3) instead of 20/40, the positive itself as the target (Eq. 9) instead of the two-view average of the released code, and F as in Eq. (22). We use three seeds on CIFAR-10, STL-10 and ImageNet-100 and one seed of 100 epochs on ImageNet-1k. All 40 runs agree with the theory. The pair that retrieves the positive at τ = 0 trains on every dataset, and every pair with τ > 0 collapses to one direction (the three IN-1k runs at τ > 0 were stopped early once they had collapsed). (Reply to XyzG, Table R1.)
  2. The weight β set by the theory. We set β by Prop. 29 so that the pairs of DirectPred and DirectCopy, (1/2,1) and (1,2), share one boundary β1/Δτ\beta^{1/\Delta\tau}. We swept the boundary on STL-10 and fixed it for CIFAR-10 and ImageNet-100 by a rule set in advance. As predicted, no run ends below the boundary. The effective rank also grows monotonically with the boundary, and at a fixed boundary it is nearly the same across datasets. At an equal boundary, the pair with the smaller τ is at least as accurate. (Reply to XyzG, Table R2.)
  3. Feature dimension versus batch size. With feature dimension 2048 and 512 representations per batch, the batch second moment has exactly empty directions, which could cause collapse by themselves. At dimension 128 the batch fills every direction, and the collapse of (1,2) at β = 1 persists on every seed, so it does not come from the dimension exceeding the batch. Cor. 1 and App. E.1 predict it at (1,2) for any dimension d ≥ 3. (Reply to XyzG, Table R3.)
  4. Moving averages in the memory and in the target. DirectPred and DirectCopy start from the DMR loss at (1/2,1) and (1,2) and add two moving averages. An EMA of the second moment forms the memory, and a momentum (EMA) encoder produces the positive. We added each to the DMR loss at β = 1, alone and together. At DirectPred's pair, the momentum encoder lets the backbone learn while the projection output keeps about two directions. The EMA second moment alone does not help. At DirectCopy's pair, neither changes the result. The remaining differences include the normalization after the predictor, which lies outside the analysis. (Reply to XyzG, Table R4.)
  5. CIFAR-10 softmax evaluation. We re-evaluated the CIFAR-10 softmax checkpoints exactly as App. F states, with k = 20 for kNN and the App. F linear probe. The order of the three rows is unchanged. (Reply to Ldbh, Table R5.)

Concrete changes to a training recipe that follow from these results are in our reply to E3jw (Q1).

Revisions. The core definitions, propositions and proofs remain unchanged. We add the new results to the paper and make the explanatory changes they require.

  • Table 1: the spectral rows come from the β = 1 runs, and the CIFAR-10 softmax cells from the re-evaluated checkpoints.
  • Table 2: the spectral block comes from the three-seed β = 1 runs, and the β = 20 block is removed. Tables 1–2 now include half ranges.
  • New Table 3 (Sec. 4.2): β is set by Prop. 29. The full sweep is in App. F.
  • Sec. 3 and 4: Sec. 3.2.3 states that β ≠ 1 lies outside Definition 3. Sec. 4.1 states that Tables 1–2 use β = 1, and Sec. 4.2 follows the new tables.
  • Positioning and credit: the abstract, introduction and conclusion drop "one comparable axis", and the introduction and conclusion place methods at their level pairs instead. Sec. 2 and 3.3.2 credit UniGrad, and Sec. 2 cites RDM.
  • Appendices and code: App. B.4 adds the moving-average experiments, and App. F and G are updated, including the settings of the new runs. The code we release follows Eq. (9) and (22).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.