- Rating
- 3
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper studies a gap between regularizing a JEPA representation at the projector output and making that representation attainable through the encoder–projector path during training. SADL models this as a mismatch between endpoint demand and upstream supply, with supply determined by both sample excitation and transmission across the interface. A one-sided penalty updates the encoder when the measured supply falls short. The experiments combine control tasks, collapse diagnostics, controlled bottlenecks, and image classification.
Critical strengths
- Distinguishing a desirable endpoint from the path required to reach it is meaningful. Compared with DirectCLR and VICReg, SADL more explicitly identifies when additional encoder constraints are needed, rather than applying a fixed constraint throughout training.
- The approach is supported by extensive experiments, convincing derivations, and detailed analysis.
Critical weaknesses
- The main experiments are largely excitation-limited, and excitation-only regularization often performs similarly to full SADL. Controlled experiments support the transmission branch, but its importance in ordinary training remains less clear.
- Classification gains are mainly observed early in training: Imagenette improves by 8.91 percentage points at epoch 20 but only 0.44 at epoch 800, while CIFAR-10 shows almost no gain at epoch 100. The practical value therefore depends on whether SADL reduces the cost of reaching a target accuracy.
Questions for authors
While my familiarity with this area is limited, I find this to be a solid paper overall. My questions (and weakness) should be considered in light of that background.
- How does SADL compare with direct encoder regularization when both receive the same tuning budget? Please include the additional training cost and time to reach a fixed performance target.
- How sensitive are the bottleneck diagnosis and resulting updates to covariance regularization and the chosen capacity normalization?
8 comments
Transmission as a Safeguard: Activation and Empirical Evidence (W1)
W1 (transmission term). The main experiments are largely excitation-limited, and excitation-only regularization often performs similarly to full SADL. Controlled experiments support the transmission branch, but its importance in ordinary training remains less clear.
Response: We agree, and the new runs make the reason explicit. Since , at the default weight , was below at every logged training step and in all 25,396 audited records in Appendix C.1. The minimum therefore always picks , and SADL and exc-only have the same loss and the same gradient. The gap between them in the first row of Table R1 (0.8 points at Ep. 1, 0.2 at Ep. 2) is run-to-run variation, and it sets the noise level for the comparisons below.
Table R1. Four-task macro success (%), INTACT with SIGReg as the Demand Regularizer (weight 0.03), training seeds 0, 42 and 3072.
| Ep. 1 SADL | Ep. 1 exc-only | Ep. 2 SADL | Ep. 2 exc-only | Ep. 1 PushT, SADL / exc-only | |
|---|---|---|---|---|---|
| 0.01 (default) | 94.56 ± 1.06 | 95.39 ± 0.60 | 96.47 ± 0.41 | 96.67 ± 0.17 | 82.4 / 84.6 |
| 0.1 | 93.56 ± 1.09 | 94.08 ± 0.98 | 96.22 ± 0.41 | 96.19 ± 0.32 | 77.9 / 79.4 |
| 0.3 | 93.11 ± 0.67 | 92.44 ± 1.73 | 95.81 ± 0.32 | 95.64 ± 0.81 | 76.4 / 72.9 |
| 1.0 | 93.50 ± 0.25 | 93.08 ± 0.08 | 95.83 ± 0.52 | 95.83 ± 0.17 | 76.6 / 76.2 |
The transmission term binds once the encoder's excitation grows faster than what is passed on to . A larger weight produces this: at , SADL selects on a growing share of minibatches as training proceeds (Table R2). A stricter ridge has the same effect at the default weight: with , is selected on 37% to 43% of minibatches (Q2).
Table R2. Share of training minibatches on which SADL selects ().
| Phase | PushT | Cube | Reacher | TwoRoom |
|---|---|---|---|---|
| Ep. 1, steps 0 to 2k | 0% | 0% | 0% | 0% |
| Ep. 1, from step 4k | 14% | 4% | 25% | 45% |
| Ep. 2, from step 4k | 27% | 16% | 44% | 63% |
As grows and starts to be selected, exc-only loses more success than SADL (Table R1). From to , Ep. 1 four-task success drops by 2.95 points for exc-only and 1.45 for SADL, and PushT success by 11.7 and 6.0 points; at Ep. 2 the drops are 1.03 and 0.66. For , SADL is above exc-only at Ep. 1 at every weight. This is the role we intend for the transmission term: excitation carries the correction in the default setting, and transmission acts as a safeguard once excitation outgrows what reaches .
Once it binds, the transmission term keeps transmission level with excitation. Table R3 gives the margin on a fixed held-out probe at Ep. 2. With exc-only, a larger weight keeps expanding while transmission lags behind, and on TwoRoom the margin turns negative. SADL keeps it positive. SADL has the larger margin on all four tasks, at both Ep. 1 and Ep. 2, in every paired run with : three seeds at each of and , and two same-seed reruns at , which we do not count as additional seeds. The erank and cos of and of differ by less than 2 rank units and 0.001, respectively, between the two variants, so the transmission term acts on the link between and rather than on the spectrum of either one.
Table R3. Margin at Ep. 2 on a held-out probe.
| TwoRoom SADL | TwoRoom exc-only | Reacher SADL | Reacher exc-only | |
|---|---|---|---|---|
| 0.01 | +0.089 | +0.093 | +0.119 | +0.120 |
| 0.1 | +0.032 | +0.005 | +0.060 | +0.041 |
| 0.3 | +0.028 | −0.003 | +0.056 | +0.031 |
| 1.0 | +0.028 | −0.006 | +0.055 | +0.028 |
The margin is the quantity this term optimizes, so Table R3 alone does not show that the term matters for a downstream readout. That evidence comes from the controlled test in Appendix F.3 (Table 32(B)). With a built-in transmission deficit and trained encoder weights, SADL raises held-out from −0.018 to 0.535, while exc-only leaves it unchanged. A second case comes from a real image checkpoint. In the forward replay of an ImageNet-LT Ep. 100 checkpoint (Appendix Figure 4), scaling the view residual by 1.5 or more makes , and only the transmission term still responds. In end-to-end control, exc-only loses more success than SADL as grows (Table R1).
Our reading is therefore the reviewer's: in regular training the transmission term is a safeguard. It contributes no gradient while transmission keeps up and takes over when transmission falls behind, at no extra cost in the default regime. The revision presents it this way. It also states that SADL and exc-only coincide in the default setting and that we do not claim a success-rate advantage for SADL over exc-only.
Early Gains and Time-to-Target Training Efficiency (W2)
W2 (early gains). Classification gains are mainly observed early in training: Imagenette improves by 8.91 percentage points at epoch 20 but only 0.44 at epoch 800, while CIFAR-10 shows almost no gain at epoch 100. The practical value therefore depends on whether SADL reduces the cost of reaching a target accuracy.
Response: We agree that practical value depends on the cost of reaching a target. We report this in control and classification.
In control (Table R4), matched -only and SADL runs share code path, data order and evaluation within each model; only backend and SADL vary. For both models/backends, SADL Ep. 1 beats -only Ep. 2 with half the updates (INTACT: 5,695 vs. 11,390); Ep. 2 gains remain 24.4–37.7 points. INTACT SADL Ep. 1 (94.9%–95.4%) also beats -only MATH Ep. 10 (91.58%, SIGReg; Appendix Table 21) with one-tenth the updates, though these are separate runs under the same protocol.
Table R4. Four-task macro success (%), SIGReg Demand Regularizer, seeds 0/42/3072; official evaluation: seeds 0/1/42, 100 episodes/task. INTACT: Direct control; LeWM: Pure-CEM.
| Model | Backend | -only Ep. 1 | SADL Ep. 1 | -only Ep. 2 | SADL Ep. 2 |
|---|---|---|---|---|---|
| INTACT | MATH | 40.5 ± 8.5 | 94.9 ± 0.4 | 70.0 ± 6.1 | 96.3 ± 0.6 |
| INTACT | Flash | 45.8 ± 2.6 | 95.4 ± 0.2 | 66.1 ± 34.5 | 96.3 ± 0.1 |
| LeWM | MATH | 30.9 ± 0.8 | 63.9 ± 1.2 | 46.0 ± 6.0 | 70.4 ± 1.6 |
| LeWM | Flash | 32.1 ± 4.9 | 63.4 ± 5.7 | 31.4 ± 15.8 | 69.1 ± 2.9 |
In classification, Table R5 reports seed-median updates to fractions of baseline final accuracy (3 seeds each for Imagenette/ImageNet-LT; 5 for CIFAR-10). Updates/epoch: 36, 452 and 195, respectively.
Table R5. Updates (thousands; seed medians) to fractions of baseline final accuracy; speed-up = baseline/SADL. No systematic step-time overhead in these runs, so speed-up also reflects training time. At low targets, 1.00 means both reach the target at first evaluation.
| Dataset | 10% | 20% | 30% | 40% | 50% | 60% | 70% | 80% | 90% | 100% | |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Imagenette | baseline | 0.04 | 0.04 | 0.25 | 0.61 | 0.90 | 1.73 | 2.63 | 3.82 | 7.02 | 23.22 |
| SADL | 0.04 | 0.04 | 0.14 | 0.47 | 0.79 | 1.26 | 2.02 | 3.17 | 5.80 | 17.82 | |
| speed-up | 1.00× | 1.00× | 1.75× | 1.31× | 1.14× | 1.37× | 1.30× | 1.20× | 1.21× | 1.30× | |
| ImageNet-LT | baseline | 1.36 | 3.16 | 5.42 | 8.14 | 11.30 | 15.37 | 21.24 | 27.12 | 34.80 | 44.52 (2 of 3 seeds) |
| SADL | 0.90 | 2.71 | 4.97 | 7.23 | 9.94 | 13.56 | 17.63 | 23.50 | 31.19 | 42.94 (3 of 3 seeds) | |
| speed-up | 1.50× | 1.17× | 1.09× | 1.12× | 1.14× | 1.13× | 1.21× | 1.15× | 1.12× | 1.04× | |
| CIFAR-10 | baseline | 0.20 | 0.20 | 0.20 | 0.20 | 0.20 | 0.20 | 0.39 | 1.17 | 3.71 | 16.57 |
| SADL | 0.20 | 0.20 | 0.20 | 0.20 | 0.20 | 0.20 | 0.39 | 0.97 | 3.51 | 16.96 | |
| speed-up | 1.00× | 1.00× | 1.00× | 1.00× | 1.00× | 1.00× | 1.00× | 1.20× | 1.06× | 0.98× |
Imagenette: SADL needs 17.8k vs. 23.2k updates (1.30×) to reach baseline final accuracy; at Ep. 800 it is +0.85 points on three seeds (91.04% vs. 90.19%), or +0.44 on seven seeds (6/7 wins; Appendix Table 38). Peak speed-up is 1.75× at 30%. ImageNet-LT: 10%–90% targets come 1.09–1.50× sooner (largest at 10%); 2/3 baseline vs. 3/3 SADL seeds reach baseline mean final accuracy, with +0.54 points at the final epoch (22.71% vs. 22.17%). CIFAR-10: the 80% target takes about 17% fewer updates, but no speed-up to baseline final accuracy in five-seed runs. Ten-seed gains through Ep. 50 are +1.83, +0.89, +0.68 points (at least 9/10 wins each; Appendix Table 38); the final revised-normalization gain is +0.19 (7/10 wins; Table R12). SADL adds 0.5%–2% step time (Q1), so update savings approximate wall-clock savings.
Table R6 reports paired accuracies on ten CIFAR-10 and seven Imagenette seeds (Appendix Table 38): gains peak early and remain positive on nearly every seed through Ep. 50 and Ep. 100, respectively.
Table R6. Paired classification accuracy (%) from Appendix Table 38; gain = mean paired point difference, wins = seeds with positive gain.
| Dataset (seeds) | Epoch | Baseline | +SADL | Gain | Wins |
|---|---|---|---|---|---|
| CIFAR-10 (10) | 1 | 50.81 ± 0.73 | 52.64 ± 0.80 | +1.83 | 10/10 |
| 10 | 66.87 ± 0.40 | 67.76 ± 0.67 | +0.89 | 9/10 | |
| 50 | 75.16 ± 0.36 | 75.84 ± 0.36 | +0.68 | 9/10 | |
| 100 | 78.50 ± 0.30 | 78.52 ± 0.32 | +0.03 | 4/10 | |
| Imagenette (7) | 5 | 24.23 ± 1.49 | 27.68 ± 1.43 | +3.45 | 7/7 |
| 20 | 35.60 ± 4.48 | 44.51 ± 3.13 | +8.91 | 7/7 | |
| 100 | 69.34 ± 1.99 | 73.15 ± 1.89 | +3.81 | 7/7 | |
| 800 | 90.25 ± 0.17 | 90.69 ± 0.31 | +0.44 | 6/7 |
The revision adds Tables R4–R6 and discusses early gains as time to target.
Brief Summary
We thank the reviewer for finding the paper solid overall, and for a reading of it that matches our intent: distinguishing a desirable endpoint from the path required to reach it, and identifying when additional encoder constraints are needed rather than applying a fixed constraint throughout training. Several of the new experiments test this last point directly. In brief:
New results are numbered Tables R1 to R12, and all results are mean ± SD over training seeds unless stated otherwise. We use the notation of the revised paper.
The Demand Regularizer D(Z) is the existing endpoint regularizer, unchanged by SADL. The Supply Regularizer is
S=[logk+ϵq+ϵ]+2
This is the term SADL adds, where q is the demand capacity and k=min(kexc,ktr) is the supply capacity, built from excitation kexc=Fexc(H) and transmission ktr=Ftr(H,sg(Z)).
"D-only" means λS=0, and "exc-only" means that k is replaced by kexc.