acceptodds
Under review as a conference paper at ICLR 2027

S-JEPA: Self-Sparsifying Representations in Joint-Embedding Predictive Architectures

Abstract

Rectified LpJEPA enables direct control over representation sparsity in joint-embedding predictive architectures (JEPA), yet its performance degrades as representations approach extreme sparsity. We show that much of this degradation is an optimization cost rather than a limit of the sparse code, as aggressive sparsity shrinks gradients and limits early formation of informative features. We therefore introduce S-JEPA, a self-sparsifying JEPA that paces sparsification from loss curvature and corrects optimization strength from code density, adapting both throughout learning without labels or reference runs. At sparsity, reference-free S-JEPA improves frozen encoder and sparse projector accuracy over fixed-sparse training by up to and points, and exceeds its non-sparse counterpart on CIFAR-100 and STL-10, while retaining of non-sparse accuracy on ImageNet-100. Its gains over fixed-sparse training hold across ResNet and ViT backbones and four optimizers. Pretrained on CIFAR-100, S-JEPA improves mean transfer accuracy across five datasets by points over fixed-sparse training, matching non-sparse transfer, and outperforms dense JEPA codes in nearest-neighbor retrieval with less storage.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Joint-Embedding Predictive Architectures, Sparse Representation Learning, Self-Supervised Learning, Adaptive Optimization

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_vic8OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper finds that Rectified LpJEPA's accuracy drop near 95% sparsity is mainly an optimization failure caused by shrunken gradients and overly early sparsification. It proposes S²-JEPA, which paces sparsity by loss curvature and rescales updates by code density during training. At about 96% sparsity, it gains up to 14.4 points over fixed-sparse training and matches or exceeds its dense counterpart on CIFAR-100 and STL-10.

Critical strengths
  1. The paper is clearly written.
  2. The experiments are well designed.
Critical weaknesses
  1. The sparsification method lacks results on large-scale datasets, such as ImageNet-1K and ImageNet-21K.

  2. The claim of "outperforming the non-sparse version" in Table 1 is inaccurate. That non-sparse version is the dense representation from the authors' own method, and its accuracy is already low. In the same table, two other common self-supervised methods, SimCLR and VICReg, achieve higher accuracy.

  3. For this unstructured sparsity, the authors report no speedup, latency, throughput, or FLOPs. Compressing storage alone brings limited benefit, so the practical value is low.

Questions for authors

Please see weaknesses

5 comments
AuthorsOpenReview ↗

Response to Reviewer vic8 (1/2)

Thank you for your review and for raising these points. To address each concern, we ran new experiments and added them to the revised paper. In short:

  • Scale (W1): on ImageNet-1K, with no retuning, S$^2$-JEPA improves encoder accuracy by 10.0 points over the fixed sparse target at the same 95% sparsity, and nearly matches the default-target model with 1% of the labels and in transfer to six datasets (Tab. 2, §5.2).
  • Baseline (W2): the reference is the published Rectified LpJEPA of Kuang et al., reproduced within 0.7 points of their reported accuracy. On CIFAR-100 at 96% sparsity, both label-free and reference-free S$^2$-JEPA beat every other sparse encoder, and the label-free variant nearly matches the encoder accuracy of dense SimCLR (§5.1).
  • Speed (W3): on the full 1.28M-image ImageNet-1K index, S$^2$-JEPA codes cut memory by 10$\times$ and compute by 581$\times$, and make each search 11.4$\times$ faster on CPU and 3.1$\times$ faster on GPU (Tab. 5, §5.4).

W1. Results on large-scale datasets.

S$^2$-JEPA scales to ImageNet-1K (ResNet-50, 100 epochs) with the same constants as every other setting in the paper (Tab. A1). These results are now Tab. 2 and §5.2 of the revised paper, with details in App. D.7. Frozen linear-probe accuracy, protocol of Tab. 1, mean ±\pm std over 3 seeds; in this and the next two tables, bold marks the better of the two 95%-sparse models:

ImageNet-1K, ResNet-50, 100 epSparsity (%)Encoder acc. ↑\uparrowProjector acc. ↑\uparrow
Rectified LpJEPA, default target (μ=0\mu = 0)20.13±0.6020.13 \pm 0.6063.62±0.2663.62 \pm 0.2638.85±0.2338.85 \pm 0.23
Rectified LpJEPA, fixed sparse target (μ=−2.5\mu = -2.5)95.20±0.0995.20 \pm 0.0952.35±0.4652.35 \pm 0.4628.15±0.4128.15 \pm 0.41
S$^2$-JEPA (reference-free)94.97±0.1694.97 \pm 0.1662.35±0.24\mathbf{62.35 \pm 0.24}36.74±0.23\mathbf{36.74 \pm 0.23}

At the same 95% sparsity, S$^2$-JEPA raises encoder accuracy from 52.35 to 62.35 (+10.0 points) and projector accuracy by +8.6 points. This closes nearly 90% of the encoder accuracy drop caused by sparsity, leaving the encoder within 1.3 points and the projector within 2.1 points of the default model, whose codes are only 20% sparse.

The standard large-scale evaluations show the same picture (low-label accuracy in Tab. 2, per-dataset transfer in Tab. A22).

ImageNet-1K, encoder acc. ↑\uparrow1% labels10% labels
Default target (μ=0\mu = 0)38.64±0.2138.64 \pm 0.2154.51±0.1654.51 \pm 0.16
Fixed sparse target (μ=−2.5\mu = -2.5)25.12±0.2925.12 \pm 0.2943.42±0.2643.42 \pm 0.26
S$^2$-JEPA38.39±0.22\mathbf{38.39 \pm 0.22}53.45±0.22\mathbf{53.45 \pm 0.22}
Transfer, encoder acc. ↑\uparrowDTDCIFAR-10CIFAR-100FlowersFood-101PetsMean
Default target (μ=0\mu = 0)72.91±0.5372.91 \pm 0.5386.78±1.2686.78 \pm 1.2666.92±1.1266.92 \pm 1.1289.55±0.0889.55 \pm 0.0866.21±0.1766.21 \pm 0.1779.04±0.7179.04 \pm 0.7176.90
Fixed sparse target (μ=−2.5\mu = -2.5)64.55±0.2164.55 \pm 0.2179.17±0.2679.17 \pm 0.2658.24±0.3458.24 \pm 0.3481.60±0.1981.60 \pm 0.1958.46±0.2058.46 \pm 0.2067.73±0.3367.73 \pm 0.3368.29
S$^2$-JEPA73.12±0.22\mathbf{73.12 \pm 0.22}86.31±0.20\mathbf{86.31 \pm 0.20}66.78±0.27\mathbf{66.78 \pm 0.27}88.56±0.32\mathbf{88.56 \pm 0.32}65.92±0.19\mathbf{65.92 \pm 0.19}78.76±0.24\mathbf{78.76 \pm 0.24}76.58

With only 1% of the labels, S$^2$-JEPA improves on the fixed sparse target by 13.3 points and nearly matches the default model (38.39 vs. 38.64). In transfer to six datasets (the five of Tab. A18 plus CIFAR-100), it improves on the fixed target by 8.3 points on average and again nearly matches the default model (76.58 vs. 76.90). Longer training tells the same story: after 1,000 epochs on ImageNet-100, S$^2$-JEPA at 96.2% sparsity reaches 84.12% encoder accuracy, against 84.58% for the dense model (Tab. A15). All ImageNet-1K results are now in the revised paper (Tab. 2, App. D.7 and Tab. A22). Training on ImageNet-21K was beyond our compute budget during the rebuttal period.

AuthorsOpenReview ↗

Response to Reviewer vic8 (2/2)

W2. The claim of "outperforming the non-sparse version" (Table 1).

The reference in Table 1 is the published method of Kuang et al. (ICML 2026): Rectified LpJEPA at its own default target (μ=0\mu = 0), trained with their recipe. We reproduce it within 0.7 points of their reported accuracy (65.27 vs. 65.97 after 1,000 epochs on CIFAR-100), and tuning its learning rate leaves it essentially unchanged (57.32 vs. 57.24 at 100 epochs; App. D.2, Tab. A10). Its codes are already 44% sparse, so the revised abstract no longer calls it its "non-sparse counterpart" but its "dense counterpart", meaning Rectified LpJEPA at its default target (μ=0\mu = 0).

Within Rectified LpJEPA, S$^2$-JEPA turns the cost of sparsity into a gain: a fixed sparse target (95.5% sparse) loses 12.7 points of encoder accuracy against the default model, while S$^2$-JEPA at 96% sparsity gains 1.7 points (reference-free) and 3.4 points (label-free). Against other methods (CIFAR-100, ResNet-18, Tab. 1; bold marks the best sparse model):

CIFAR-100, ResNet-18 (Tab. 1)Sparsity (%)Encoder acc. ↑\uparrowProjector acc. ↑\uparrow
SimCLR0.000.0060.70±0.1160.70 \pm 0.1154.96±0.0854.96 \pm 0.08
VICReg0.000.0063.04±0.0863.04 \pm 0.0855.64±0.2755.64 \pm 0.27
Rectified LpJEPA, default (μ=0\mu = 0)44.33±0.0944.33 \pm 0.0957.24±0.7857.24 \pm 0.7848.13±0.5748.13 \pm 0.57
Top-k, annealed (kk = 20/80), best other sparse encoder96.09±0.0096.09 \pm 0.0056.16±0.5256.16 \pm 0.5241.40±0.3241.40 \pm 0.32
NCL-ReLU, best other sparse projector95.44±0.0695.44 \pm 0.0652.90±0.0752.90 \pm 0.0746.55±0.1246.55 \pm 0.12
Rectified LpJEPA, μ=−2.5\mu = -2.595.49±0.0495.49 \pm 0.0444.56±0.7644.56 \pm 0.7632.28±0.3032.28 \pm 0.30
S$^2$-JEPA, reference-free95.91±0.0295.91 \pm 0.0258.97±0.3158.97 \pm 0.3146.73±0.3546.73 \pm 0.35
S$^2$-JEPA, label-free96.02±0.0496.02 \pm 0.0460.62±0.29\mathbf{60.62 \pm 0.29}48.38±0.17\mathbf{48.38 \pm 0.17}

Both S$^2$-JEPA variants beat every other sparse encoder, by 4.5 points (label-free) and 2.8 points (reference-free) over annealed Top-k, and match or exceed the projector accuracy of NCL-ReLU. The label-free variant also nearly matches the encoder accuracy of dense SimCLR (60.62 vs. 60.70), while dense VICReg remains 2.4 points higher (63.04). SimCLR and VICReg learn dense codes, so they are not a like-for-like comparison. Note that the label-free variant needs a second run, of the dense reference model, whereas the reference-free variant needs only its own run. We have added this comparison to §5.1 of the revised paper.

W3. Speed-up, latency, throughput and FLOPs.

S$^2$-JEPA codes make retrieval over the full ImageNet-1K gallery (1.28M images, with the 50k validation images as queries) 11.4$\times$ faster on CPU and 3.1$\times$ faster on GPU, with a 10$\times$ smaller index and 581$\times$ fewer multiply-adds per query. Both columns store the same codes, as a dense fp16 matrix and as a sparse matrix, so exact search returns identical neighbours (kNN@1 25.5 in both cases) and all of the gain comes from sparsity. The encoder is the same ResNet-50, so the cost of computing a code is unchanged; the gains come from storing and searching the codes. These results are now Tab. 5 and §5.4 of the revised paper, with the full protocol in App. D.7.

ImageNet-1K, 1.28M gallery, exact cosine searchDense fp16Sparse (S$^2$-JEPA)Gain
Index memory ↓\downarrow5.25 GB0.52 GB10.0×\mathbf{10.0\times}
Multiply-adds per query ↓\downarrow2.6×1092.6 \times 10^{9}4.5×1064.5 \times 10^{6}581×\mathbf{581\times}
Latency per query, CPU (median) ↓\downarrow85.7 ms7.5 ms11.4×\mathbf{11.4\times}
Throughput, single stream, CPU ↑\uparrow11.7 queries/s133 queries/s11.4×\mathbf{11.4\times}
Latency per query, GPU (H200), vs. fp16 ↓\downarrow2.20 ms0.71 ms3.1×\mathbf{3.1\times}
Linear classifier on the code, single input, CPU ↓\downarrow110 μ\mus31 μ\mus3.6×\mathbf{3.6\times}

The advantage grows with scale: the CPU speed-up rises from 3.1×3.1\times at 10k images to 8.3×8.3\times at 126k and 11.4×11.4\times at 1.28M, and the GPU speed-up over dense fp16 from 1.5×1.5\times at 126k to 3.1×3.1\times at 1.28M. All timings use exact cosine search on one H200 node, with 16 CPU threads for dense search and a single thread for the sparse index, and report the median latency over 200 of the queries. Compressing dense codes to a similar size does not reach the same point: on CIFAR-100, PCA to 20–80 dimensions lowers kNN@10 accuracy by 4.5–4.8 points, and int8 quantization keeps accuracy but needs about 6$\times$ more storage than S$^2$-JEPA (Tab. A17). We have not compared approximate search methods such as product quantization, which change the neighbors returned.

Thank you again for your review. We hope these results address your concerns, and we would be happy to answer any further questions.

Dear Reviewer,

We really thank you again for your detailed review of our paper. We know this is a busy period. We have posted a detailed response addressing each of your comments and the requested experiments, and we would be grateful if you could please take a look when convenient. If you feel your concerns have been addressed, we would sincerely appreciate your considering an updated evaluation. If anything remains unconvincing, we would be glad to discuss it together; we are following this thread closely and will respond promptly. We would greatly welcome any additional feedback you might have.

Reviewer_SzNqOpenReview ↗
Rating
4
Confidence
4
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper investigate the reported phenomenon that linear probe performances drops sharply in Rectified LpJEPA as it approaches extreme sparsity in the embeddings. The authors hypothesized that this is because 1) aggressive sparsity shrinks gradients and hence the model parameters are not sufficiently updated and 2) aggressive sparsity limits early formation of informative features.

It's observed in Figure 2 that extremely sparse models actually receive much less total parameter update compared to a dense counterpart, and this is not because of ReLU leading to zero derivatives but rather the layerwise learning rate cap in the LARS optimizer used in Rectified LpJEPA. The authors propose to multiply the learning rate in LARS by \sqrt{active fractions for the dense reference / active fractions for the sparse reference}. With this change, an extreme sparse model recovers dense performance by around 82%, and hence this is only a partial fix.

The authors also hypothesized that the performance drops might be due to fixed sparsity target that overly restrict feature formation early during training. By annealing the mu parameter that control sparsity from 0 to -2.5 during training with scheduling, the online probe accuracy increases by 12.5 and the total parameter update also increases.

Finally, the authors introduce S^2JEPA, which adjust the parameter mu that controls sparsity automatically based on the largest Hessian eigenvalue of the training loss. The scaling constant of learning rate also follows from a more flexible version of \sqrt{active fractions for the dense reference / active fractions for the sparse reference} with a tolerence for larger values. It's shown that S^2JEPA uniformly outperforms Rectified LpJEPA by being both extremely sparse and performant.

Critical strengths
  • The problems of Rectified LpJEPA (fig 2 update magnitude, fig 3 the ability to anneal mu) are well-demonstrated and the solutions are well-designed and also work well empirically.
  • The paper is well-written with very natural transition and good motivation. I enjoy reading it a lot.
Critical weaknesses
  • It would be great to explain the intuition for the LARS optimizer learning rate cap a bit more. The paper will benefit from an appendix section that goes over LARS and how its current form or the form implemented lead to incorrect magnitude scaling.
  • Overall, the new line of distribution-matching based sparsity methods like Rectified LpJEPA has clear correspondence to information maximization, and the RDMReg objective is a sufficient condition of minimizing total correlation / multi-information which encourages complete statistical independence. It would be great to discuss this line of method against more traditional approaches like sparse auto-encoder, where sparsity is not encouraged by distribution-matching. Also, SAE is more reconstruction-based rather than JEPA-style mutual information maximization between views.
Questions for authors
  • are there any connections between your proposed learning rate scaling with sparse muP (https://arxiv.org/abs/2405.15743)? i.e. is it possible to syetematically generalize your proposed fixed into a broader framework of how you should scale per-layer learning rate based on the fan_in, fan_out dimension and their corresponding sparsity levels?
  • There is one more confound in Rectified LpJEPA. Basically, changing the parameter mu will unfortunately change both sparsity and magnitude, as shifting distribution by changing mu couples the sparsity level and overall feature norm. Do you see your magnitude-control approach taking into account of this coupling as well?
3 comments
AuthorsOpenReview ↗

Response to Reviewer SzNq (1/2)

Thank you for the careful reading and the helpful suggestions. We answer each point below and have added the new material to the revised paper.

1. The LARS learning-rate cap and why it gives the wrong update size.

The LARS implementation used by Rectified LpJEPA computes a layer-wise trust ratio λℓ=η∣wℓ∣∣gℓ∣+γ∣wℓ∣\lambda_\ell = \frac{\eta \\|w_\ell\\|}{\\|g_\ell\\| + \gamma \\|w_\ell\\|} and then caps it at the learning rate (Eq. 3–4): uℓ=min⁡(λℓ/lr,1),(gℓ+γwℓ).u_\ell = \min(\lambda_\ell / \mathrm{lr}, 1)\\,(g_\ell + \gamma w_\ell).

The cap is active whenever λℓ≥lr\lambda_\ell \ge \mathrm{lr}, that is, whenever ∣gℓ∣∣wℓ∣≤ηlr−γ≈0.020.03≈0.67,\frac{\\|g_\ell\\|}{\\|w_\ell\\|} \le \frac{\eta}{\mathrm{lr}} - \gamma \approx \frac{0.02}{0.03} \approx 0.67, and this threshold only grows as the learning rate decays. Because gradients are far smaller than weights in every layer, our logs show the cap is active for every LARS tensor at every logged step, in dense, fixed-sparse and S$^2$-JEPA runs alike. LARS therefore behaves like momentum SGD, uℓ≈lr,(gℓ+γwℓ)u_\ell \approx \mathrm{lr}\\,(g_\ell + \gamma w_\ell), so a smaller gradient is never offset by a larger trust ratio and the update shrinks together with the gradient.

The parameter movement measured in the same runs (CIFAR-100, ResNet-18) shows the effect:

Parameter movement, μ=−2.5\mu = -2.5 vs. μ=0\mu = 0 (ratio)Epochs 1–1011–5051–100All
Backbone0.130.170.190.17
Projector0.140.210.200.19
All parameters0.140.180.190.18

Over the whole run, the sparse model moves only 0.18×0.18\times as far as the dense one, and removing the cap raises this ratio to 0.580.58 (§3.1). The new App. A.4 of the revised paper writes out the implemented update term by term and includes this table (Tab. A3).

2. Relation to sparse autoencoders.

SAEs and S$^2$-JEPA reach sparsity through different objectives. An SAE is fitted after pretraining to reconstruct fixed features through a sparse bottleneck, so its code keeps whatever the dense features contain. S$^2$-JEPA learns its code during pretraining, with an invariance objective and a distribution-matching term that pushes coordinates towards independence, so the sparse code is shaped by what can be predicted across views. In line with this, S$^2$-JEPA has lower coordinate dependence (normalized HSIC) than all dense baselines and all fixed Rectified LpJEPA targets (Tab. A20).

For a direct comparison, we fitted a standard TopK SAE with the AuxK dead-latent loss (Gao et al., ICLR 2025; CSR's public implementation) on the 512-d encoder features of dense Rectified LpJEPA (μ=0\mu = 0), using 2,048 latents and k=20k = 20. We chose its configuration from a grid by probe accuracy, which favours the SAE. The table uses CIFAR-100 and the protocol of Tab. A8 (mean ±\pm std over 3 seeds):

Code (≈20\approx 20 active coordinates)Runs neededProbe ↑\uparrowkNN@1 ↑\uparrowPurity@64 ↑\uparrowNever-active units ↓\downarrow
TopK SAE + AuxK on dense featuresdense run + SAE fit47.33±0.5547.33 \pm 0.5533.43±0.4933.43 \pm 0.493.84±0.133.84 \pm 0.131749±221749 \pm 22 / 2048
S$^2$-JEPA, label-freedense reference run + S$^2$-JEPA run48.38±0.17\mathbf{48.38 \pm 0.17}37.03±1.40\mathbf{37.03 \pm 1.40}8.17±0.14\mathbf{8.17 \pm 0.14}0\mathbf{0} / 512
S$^2$-JEPA, reference-freeone run46.73±0.3546.73 \pm 0.3533.80±1.6133.80 \pm 1.618.09±0.208.09 \pm 0.200\mathbf{0} / 512

Fitting the SAE on the projector outputs instead gives a lower probe accuracy (43.98±0.2943.98 \pm 0.29), so we report the version trained on encoder features. The SAE reconstructs its inputs well (fraction of variance unexplained 0.129±0.0010.129 \pm 0.001 on the test set), yet even with AuxK, 1,749 of its 2,048 latents never fire on the test set. Both S$^2$-JEPA codes have about twice the purity of the SAE code and use every coordinate. In probe accuracy, the label-free code is 1.1 points above the SAE, and the reference-free code is 0.6 points below it. The larger difference lies in the encoder: an SAE leaves the encoder unchanged, while S$^2$-JEPA improves it, reaching 58.97–60.62 frozen encoder accuracy compared with 57.24 for the dense model (Tab. 1). We have added this comparison and discussion to App. B.1 (Tab. A5).

AuthorsOpenReview ↗

Response to Reviewer SzNq (2/2)

3. Connection to sparse µP (Dey et al., 2024; arXiv 2405.15743).

We cite this work, and its density rule already appears as one of the settings in Fig. 2 (right). The "full density ratio f0/fμf_0/f_\mu" setting there is a 1/f1/f correction, as in sparse µP's fan-in rule, applied as a target for the update path (×12.5\times 12.5). With our square-root path target (×3.55\times 3.55) instead, online accuracy reaches 51.5%, compared with 44.3% for the 1/f1/f target. The revised Fig. 2 now labels this setting as sparse µP and cites the paper.

The two rules differ because the sparsity is in a different place. Sparse µP deals with sparse weights: when only a fraction of a layer's fan-in is kept, its learning rate is scaled by the inverse of that density, under both SGD and Adam (with an additional width factor for Adam). In S$^2$-JEPA the weights are dense, and it is the activations at the projector output that are sparse. If the active coordinates contribute comparably, the gradient norm then scales with f\sqrt{f} rather than ff (§3.1), which calls for a square-root rather than a full density correction.

A learning-rate multiplier and an update-path multiplier are also different quantities. Multiplying the learning rate by 3.55 increases the path by only 2.56×2.56\times (§3.1), because a larger step also changes the trajectory, so reaching a path of about f0/f\sqrt{f_0/f} takes a learning-rate multiplier above f0/f\sqrt{f_0/f}. This is why the best fixed multiplier in our sweep is ×8\times 8 (Fig. 2; Tab. A10), and why S$^2$-JEPA's α\alpha rises to its upper bound of 10 while m⋆m^\star stays near 3.4 (Fig. 5c).

As for a per-layer version, the table under point 1 shows that the movement shortfall is nearly the same in the backbone (0.17×0.17\times) and the projector (0.19×0.19\times) and in every training phase (0.13–0.21), so a single global factor is enough in our setting. A per-layer rule based on fan-in, fan-out and local sparsity, in the spirit of sparse µP, would be a natural extension for architectures where the shortfall is uneven, and App. A.4 now mentions this direction.

4. Coupling between sparsity and feature norm when μ\mu changes.

For the p=1p = 1 target used throughout the paper, this coupling is exact and has a closed form. The rectified target is y=max⁡(0,Z)y = \max(0, Z), where ZZ is Laplace with location μ\mu and scale σGN=1/2\sigma_{GN} = 1/\sqrt{2} (unit variance). For μ≤0\mu \le 0, a coordinate is active with probability f(μ)=12e2,μf(\mu) = \tfrac{1}{2} e^{\sqrt{2}\\,\mu}. Because the Laplace tail is exponential, the value of an active coordinate follows the same exponential distribution for every μ≤0\mu \le 0, so E[y2∣y>0]=1\mathbb{E}[y^2 \mid y > 0] = 1 and E[y2]=f(μ)=12e2,μ.\mathbb{E}[y^2] = f(\mu) = \tfrac{1}{2} e^{\sqrt{2}\\,\mu}.

μ\mu00−1-1−2-2−2.5-2.5−3-3
Active fraction =E[y2]= \mathbb{E}[y^2]0.50000.12160.02960.01460.0072

In other words, lowering μ\mu changes how many coordinates are active while the active ones keep the same size. The target norm therefore falls exactly as f\sqrt{f}, and the norm ratio between two targets is f0/fμ\sqrt{f_0/f_\mu}, which is the factor our magnitude correction is built on. The learned codes are denser than the target itself: at μ=−2.5\mu = -2.5 the target has 1.46% active coordinates, while the trained projector outputs have about 4.5% (95.5% sparsity, Tab. 1), because the distribution-matching term is balanced against the invariance term and matches the target only approximately. For this reason, the ×12.5\times 12.5 and ×3.55\times 3.55 factors in Fig. 2 use measured densities. During training, S$^2$-JEPA also computes the factor from the measured density of the learned codes (§4.2), so it follows the coupling as μ\mu moves, and the ramp makes up for any remaining shortfall in parameter movement.

We have added this derivation to App. A.5 (Tab. A4). Thank you again for the positive assessment and the constructive suggestions.

Thank you for the additional update. I maintain my score.

Reviewer_3iADOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies the performance degradation of Rectified LpJEPA under extremely sparse representations. The authors argue that much of this “sparsity cliff” is caused by optimization difficulties rather than an inherent limitation of sparse representations. In particular, sparse targets reduce gradient and parameter-update magnitudes, while imposing high sparsity too early restricts feature formation.

To address these issues, the paper proposes S²-JEPA, which combines two adaptive mechanisms. First, sparsity pacing uses changes in the estimated loss curvature to determine how quickly the target representation should become sparse. Second, magnitude correction adjusts the learning rate according to representation density, with eligibility checks deciding whether the correction should remain active.

Critical strengths
  • The sparsity-cliff diagnosis is supported by informative controlled experiments.
    The authors examine update paths, the LARS cap, surrogate backward derivatives, alternative optimizers, and sparsification schedules. The matched-path experiments indicate that update magnitude and sparsification timing have distinct effects. The 1,000-epoch results, in which fixed-sparse training largely recovers encoder performance, further suggest that the short-budget sparsity cliff is not solely a capacity limitation.

  • The method delivers substantial and practically meaningful gains at high sparsity.
    On CIFAR-100, the reference-free method improves frozen encoder accuracy from 44.56 to 58.97 and projector accuracy from 32.28 to 46.73 at 95.91% sparsity. It also achieves 39.07 kNN@10 accuracy, compared with 36.73 for dense Rectified LpJEPA, while reducing storage from 1,024 to approximately 87.9 bytes per image. These results demonstrate that the sparse projector representations themselves remain useful, beyond their effect on downstream encoder performance.

Critical weaknesses
  • The paper starts from a compelling empirical observation and supports the proposed method with an extensive set of experiments. The improvements over fixed-sparse training are substantial and consistent across the evaluated settings, which is the main reason I lean toward acceptance. However, gradual sparsification and density-dependent adjustment of the optimization strength both have precedents in the literature. The main novelty therefore appears to lie in the particular combination of feedback signals, control rules, and eligibility checks.

    In this respect, the use of the largest Hessian eigenvalue as the central feedback signal for sparsity pacing requires stronger justification. The experiments in Fig. 5 and Tables A7 and A9 show that curvature-based pacing is workable and reasonably robust, but they do not establish why this signal is necessary or preferable to other label-free signals that can be obtained more cheaply from the current training run. Proposition 1 only bounds the change in the target distribution under a shift in its mean; it does not connect curvature dynamics to adaptation to the current sparsity target. I would therefore appreciate either a stronger theoretical or mechanistic argument for this connection, or a broader comparison with alternative unsupervised feedback signals, such as changes in the training loss, gradient norms, update-path statistics, or representation stability.

  • The experiments convincingly show that optimization contributes substantially to the sparsity cliff: sparse targets reduce parameter movement, gradual sparsification helps at approximately matched update paths, and longer training recovers much of the encoder gap. However, the experiments do not fully separate optimization effects from representation-capacity or target-distribution effects. After 1,000 epochs, for example, the fixed-sparse encoder approximately matches its dense counterpart, but the projector remains 4.8 points lower. The authors should moderate the causal claim or provide a clearer decomposition of optimization and representational effects.

  • Table A6 does not fully isolate curvature-based pacing from the other controller components. Its linear-annealing baselines use either a tuned constant multiplier or an always-on correction, whereas S²-JEPA uses magnitude correction with eligibility checks. I recommend adding linear annealing with the same density-based correction and eligibility checks as reference-free S²-JEPA. This would directly test whether curvature feedback improves over a predefined schedule when all other components are held fixed. Reporting the associated computational overhead would also clarify whether estimating the largest Hessian eigenvalue provides sufficient practical benefit.

Questions for authors

Please address the concerns raised in the weaknesses above.

5 comments
AuthorsOpenReview ↗

Response to Reviewer 3iAD (1/2)

Thank you for the careful review and the constructive suggestions. We have revised the paper accordingly.

On novelty. We agree that gradual sparsification and density-based learning-rate scaling both have precedents, and we discuss them in §1 and App. B.2. These methods, however, follow rules fixed in advance, set by the density or a schedule. Our contribution is (i) showing that much of the accuracy lost under a sparse target is an optimization cost that training can recover (§3), and (ii) a controller that decides from the current run when to sparsify and whether a correction is needed, so the same constants work across datasets, backbones and optimizers without a per-setting search. The pacing signal is one part of this controller, and we address it next.

1. Why curvature, and how it compares with cheaper signals.

We ran the suggested comparison, replacing curvature with each alternative signal while keeping the pacing rule (Eq. 6), its constants, the magnitude correction and the eligibility checks unchanged. Within the rebuttal compute budget, we ran the two most common signals, training loss and gradient norm, on ImageNet-100. Both tables are now Tab. 4 of the revised paper and are discussed in §5.3 ("Choice of pacing signal").

CIFAR-100, ResNet-18 (Tab. 4, left; mean ±\pm std over 3 seeds):

Pacing signalSparsity (%)Encoder acc. ↑\uparrowProjector acc. ↑\uparrow
Curvature, log⁡λmax⁡\log \lambda_{\max} (ours)95.91±0.0295.91 \pm 0.0258.97±0.31\mathbf{58.97 \pm 0.31}46.73±0.35\mathbf{46.73 \pm 0.35}
Training loss94.83±0.1894.83 \pm 0.1855.95±0.3455.95 \pm 0.3444.37±0.2744.37 \pm 0.27
Gradient norm95.12±0.0995.12 \pm 0.0957.13±0.3257.13 \pm 0.3245.26±0.2245.26 \pm 0.22
Update path (α\alpha)94.79±0.2894.79 \pm 0.2856.92±0.2356.92 \pm 0.2344.43±0.1044.43 \pm 0.10
Representation stability95.03±0.1295.03 \pm 0.1256.64±0.2656.64 \pm 0.2644.52±0.1944.52 \pm 0.19
Fixed μ=−2.5\mu = -2.5, no controller (Tab. 1)95.49±0.0495.49 \pm 0.0444.56±0.7644.56 \pm 0.7632.28±0.3032.28 \pm 0.30

On CIFAR-100, curvature gives the highest encoder and projector accuracy, ahead of the other signals by 1.8–3.0 and 1.5–2.4 points, respectively. Every signal still clearly beats the fixed target, which shows the value of adapting μ\mu from training feedback.

ImageNet-100, ResNet-50 (Tab. 4, right; mean ±\pm std over 3 seeds):

Pacing signalSparsity (%)Encoder acc. ↑\uparrowProjector acc. ↑\uparrow
Curvature (ours)95.26±0.3995.26 \pm 0.3974.73±0.86\mathbf{74.73 \pm 0.86}66.04±0.38\mathbf{66.04 \pm 0.38}
Gradient norm94.75±0.1394.75 \pm 0.1372.88±0.8172.88 \pm 0.8164.80±0.3964.80 \pm 0.39
Training loss94.82±0.1194.82 \pm 0.1172.43±0.2872.43 \pm 0.2863.53±0.3463.53 \pm 0.34

On ImageNet-100, curvature again gives the best accuracy and the sparsest code. With the training loss as the signal, the controller sparsifies too early: by epoch 58 it has already lowered μ\mu to −2.4-2.4, against −1.5-1.5 with curvature, and it ends 2.3 encoder and 2.5 projector points lower. The likely reason is that the loss is measured against the current target, whose norm shrinks as f\sqrt{f} when μ\mu falls, so the loss drops after every step even before the model has adapted. The gradient norm shrinks with the code's density in the same way (it scales with f\sqrt{f}, §3.1), so it can also drop after a μ\mu step before the model has adapted. The controller reads this drop as adaptation and lowers μ\mu too early, which may explain why the gradient norm trails curvature on both datasets.

The choice of signal therefore matters: the pacing rule needs a signal that reflects whether the model has adapted to the current target. Curvature fits this role, the training loss does not, and the gradient norm is not always reliable. For curvature, the link to adaptation rests on what we observe in training: λmax⁡\lambda_{\max} generally falls after each change of the target and can jump sharply after a large one (Fig. 5d), so the rule keeps lowering μ\mu while the trend falls and holds it after a jump. Because λmax⁡\lambda_{\max} can rise right after the target norm falls, it does not simply follow the target norm, unlike the loss and the gradient norm. We have no equally clear explanation for the update-path and representation-stability signals; both end with less sparse codes and lower accuracy than curvature. All signals share the pacing constants set for curvature, so tuning each one separately might narrow these gaps. The revised §5.3 explains the choice of signal, and App. A.3 now explains how Proposition 1 and these observations fit together.

AuthorsOpenReview ↗

Response to Reviewer 3iAD (2/2)

2. Separating optimization from capacity and target effects.

Two experiments separate these effects directly.

Same target, larger updates. Tab. A10 trains the fixed μ=−2.5\mu = -2.5 target with a larger constant learning rate and no pacing. This changes only the update size, while the target stays the same (measured sparsity rises from 95.5% to 97.6%; CIFAR-100, 100 epochs, mean ±\pm std over 3 seeds):

CIFAR-100, 100 epochsSparsity (%)Encoder acc. ↑\uparrowProjector acc. ↑\uparrow
Rectified LpJEPA, μ=0\mu = 0 (Tab. 1)44.33±0.0944.33 \pm 0.0957.24±0.7857.24 \pm 0.7848.13±0.57\mathbf{48.13 \pm 0.57}
Fixed μ=−2.5\mu = -2.5 (Tab. 1)95.49±0.0495.49 \pm 0.0444.56±0.7644.56 \pm 0.7632.28±0.3032.28 \pm 0.30
Fixed μ=−2.5\mu = -2.5, learning rate ×8\times 8 (Tab. A10)97.55±0.0197.55 \pm 0.0158.24±0.4558.24 \pm 0.4544.59±0.2444.59 \pm 0.24
S$^2$-JEPA, reference-free (Tab. 1)95.91±0.0295.91 \pm 0.0258.97±0.31\mathbf{58.97 \pm 0.31}46.73±0.3546.73 \pm 0.35

With larger updates alone, the fixed target exceeds the dense model's encoder accuracy (58.24 vs. 57.24) even at 97.6% sparsity, and recovers most of the projector gap (44.59 vs. 32.28, against 48.13 for the dense model). Capacity therefore does not limit the encoder at this training budget. S$^2$-JEPA, which also paces the target, closes most of the remaining projector gap.

Longer training. After 1,000 epochs on CIFAR-100 (Tab. A15; label-free variant):

CIFAR-100, 1,000 epochs (Tab. A15)Sparsity (%)Encoder acc. ↑\uparrowProjector acc. ↑\uparrow
Rectified LpJEPA, μ=0\mu = 035.55±0.3235.55 \pm 0.3265.27±0.3665.27 \pm 0.3661.30±0.3461.30 \pm 0.34
Fixed μ=−2\mu = -295.13±0.0395.13 \pm 0.0365.40±0.4065.40 \pm 0.4058.71±0.3258.71 \pm 0.32
Fixed μ=−2.5\mu = -2.597.39±0.0297.39 \pm 0.0265.12±0.0965.12 \pm 0.0956.46±0.1056.46 \pm 0.10
S$^2$-JEPA, label-free95.79±0.0495.79 \pm 0.0467.26±0.35\mathbf{67.26 \pm 0.35}61.49±0.41\mathbf{61.49 \pm 0.41}

The 4.8-point projector gap the reviewer mentions belongs to the μ=−2.5\mu = -2.5 target, which ends at 97.4% sparsity. At matched sparsity, the fixed μ=−2\mu = -2 target (95.1%) still trails the dense projector by 2.6 points (58.71 vs. 61.30), while S$^2$-JEPA at 95.8% matches it (61.49). A code this sparse can therefore reach dense-level projector accuracy, and the gap of the fixed target at this sparsity comes from how it is trained. We have added this matched-sparsity comparison to §5.2. We do not, however, claim that the whole cliff is an optimization cost; the paper says "much of" it (§1). We have not shown that the remaining gap of the 97.4%-sparse target comes from optimization, and part of it may reflect the limited capacity of such a sparse code.

3. Linear annealing with the same correction and eligibility checks, and overhead.

We ran this control on ImageNet-100; it is now the "Linear schedule" row of Tab. 4. It lowers μ\mu linearly from 0 to its floor between epochs 10 and 70, as in Tab. A9, a schedule that follows S$^2$-JEPA's path on CIFAR-100. It uses the same reference-free correction and eligibility checks as S$^2$-JEPA, so the two runs differ only in how μ\mu is paced (mean ±\pm std over 3 seeds):

ImageNet-100, ResNet-50Sparsity (%)Encoder acc. ↑\uparrowProjector acc. ↑\uparrow
Linear anneal, same correction and checks (Tab. 4)94.88±0.1394.88 \pm 0.1372.97±0.2272.97 \pm 0.2263.98±0.1963.98 \pm 0.19
S$^2$-JEPA, reference-free (Tab. 1)95.26±0.3995.26 \pm 0.3974.73±0.86\mathbf{74.73 \pm 0.86}66.04±0.38\mathbf{66.04 \pm 0.38}

With everything else the same, feedback pacing gives higher encoder and projector accuracy at higher sparsity. The gap comes from the pace: on ImageNet-100, S$^2$-JEPA sparsifies more slowly (μ=−1.5\mu = -1.5 at epoch 58), while the schedule, which runs to the ImageNet-100 floor of −3-3, is already at −2.4-2.4, the same early sparsification we see with the training loss. On CIFAR-100, where the schedule does follow S$^2$-JEPA's path, the checks keep the correction on, so this control is the "linear + always-on" run (Tab. 4, left; Tab. A9), which ties with S$^2$-JEPA (58.73 / 46.42 vs. 58.97 / 46.73).

So the benefit of feedback pacing is not higher accuracy than every schedule: a schedule matched to a setting does as well there. The benefit is that S$^2$-JEPA finds the right pace in each setting with the same constants, whereas a schedule must be searched for or copied from an earlier S$^2$-JEPA run.

Overhead. λmax⁡\lambda_{\max} is estimated once per epoch with five Hessian-vector products on 64 images, which adds no measurable time. All controller measurements together add about 11% to training time on CIFAR-100 and 17% on ImageNet-100 (App. A.1, §5.3).

All of these results are now in the revised paper (Tab. 4, §5.2 and §5.3). Thank you again for the helpful review.

Dear Reviewer,

We really thank you again for your detailed review of our paper. We know this is a busy period. We have posted a detailed response addressing each of your comments and the requested experiments, and we would be grateful if you could please take a look when convenient. If you feel your concerns have been addressed, we would sincerely appreciate your considering an updated evaluation. If anything remains unconvincing, we would be glad to discuss it together; we are following this thread closely and will respond promptly. We would greatly welcome any additional feedback you might have.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response to All Reviewers

We thank all reviewers for their careful and constructive feedback. To address the main concerns, we ran new experiments and added them to the revised paper. In summary:

  • ImageNet-1K. S$^2$-JEPA scales to ImageNet-1K with the same settings as in all other experiments and no retuning. At the same 95% sparsity, it improves encoder accuracy over the fixed sparse target by 10.0 points (62.35 vs. 52.35), recovering nearly 90% of the accuracy lost to sparsity. It also gains 13.3 points when only 1% of the labels are used and 8.3 points in transfer to six datasets, coming close to the dense Rectified LpJEPA model in both cases (Tab. 2, §5.2).
  • Optimization vs. capacity. Simply using larger updates lets the fixed sparse target exceed dense encoder accuracy, even at 97.6% sparsity (58.24 vs. 57.24). After 1,000 epochs, S$^2$-JEPA also matches the dense projector at similar sparsity (61.49 vs. 61.30). Both results show that much of the accuracy drop under high sparsity is an optimization cost that training can recover (§5.2).
  • Why curvature. We replaced curvature with four other signals (training loss, gradient norm, update path and representation stability) and kept everything else the same. Curvature gives the best accuracy, ahead by 1.8–3.0 points on CIFAR-100 and 1.9–2.3 points on ImageNet-100. The loss and gradient norm naturally shrink as the code becomes sparser, so they make the model sparsify too early (Tab. 4, §5.3).
  • Feedback vs. a fixed schedule. With all other components the same, curvature-based pacing beats a linear schedule on ImageNet-100 by 1.8 encoder and 2.1 projector points. A fixed schedule only works well when it is copied from S$^2$-JEPA's own path in that setting, whereas feedback finds the right pace in every setting without any schedule search (Tab. 4).
  • Real speed-ups. When searching all 1.28M ImageNet-1K images, S$^2$-JEPA codes are 11.4$\times$ faster on CPU and 3.1$\times$ faster on GPU than the same codes stored densely. They also need a 10$\times$ smaller index and 581$\times$ fewer multiply-adds per query, and they return exactly the same neighbors (Tab. 5, §5.4).
  • Baselines. Both S$^2$-JEPA variants outperform every other sparse method on CIFAR-100, by 2.8–4.5 points. Our reference is the published Rectified LpJEPA, which we reproduce within 0.7 points of its reported accuracy (§5.1).
  • Optimizer and theory. A new appendix (App. A.4) explains the LARS update used by Rectified LpJEPA and shows that its learning-rate cap is always active, so sparse training moves the weights only 0.18×0.18\times as far as dense training. Another new appendix (App. A.5) derives exactly how sparsity and target norm are linked, which explains our f\sqrt{f} correction. Fig. 2 now also shows how this correction relates to sparse µP.
  • Sparse autoencoders. Compared with a standard TopK sparse autoencoder trained on dense features, S$^2$-JEPA codes have about twice the purity and use every coordinate, while 1,749 of the autoencoder's 2,048 units never activate (App. B.1).

Detailed replies to each reviewer follow below. We are happy to answer any further questions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.