- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper finds that Rectified LpJEPA's accuracy drop near 95% sparsity is mainly an optimization failure caused by shrunken gradients and overly early sparsification. It proposes S²-JEPA, which paces sparsity by loss curvature and rescales updates by code density during training. At about 96% sparsity, it gains up to 14.4 points over fixed-sparse training and matches or exceeds its dense counterpart on CIFAR-100 and STL-10.
Critical strengths
- The paper is clearly written.
- The experiments are well designed.
Critical weaknesses
-
The sparsification method lacks results on large-scale datasets, such as ImageNet-1K and ImageNet-21K.
-
The claim of "outperforming the non-sparse version" in Table 1 is inaccurate. That non-sparse version is the dense representation from the authors' own method, and its accuracy is already low. In the same table, two other common self-supervised methods, SimCLR and VICReg, achieve higher accuracy.
-
For this unstructured sparsity, the authors report no speedup, latency, throughput, or FLOPs. Compressing storage alone brings limited benefit, so the practical value is low.
Questions for authors
Please see weaknesses
5 comments
Response to Reviewer vic8 (2/2)
W2. The claim of "outperforming the non-sparse version" (Table 1).
The reference in Table 1 is the published method of Kuang et al. (ICML 2026): Rectified LpJEPA at its own default target (), trained with their recipe. We reproduce it within 0.7 points of their reported accuracy (65.27 vs. 65.97 after 1,000 epochs on CIFAR-100), and tuning its learning rate leaves it essentially unchanged (57.32 vs. 57.24 at 100 epochs; App. D.2, Tab. A10). Its codes are already 44% sparse, so the revised abstract no longer calls it its "non-sparse counterpart" but its "dense counterpart", meaning Rectified LpJEPA at its default target ().
Within Rectified LpJEPA, S$^2$-JEPA turns the cost of sparsity into a gain: a fixed sparse target (95.5% sparse) loses 12.7 points of encoder accuracy against the default model, while S$^2$-JEPA at 96% sparsity gains 1.7 points (reference-free) and 3.4 points (label-free). Against other methods (CIFAR-100, ResNet-18, Tab. 1; bold marks the best sparse model):
| CIFAR-100, ResNet-18 (Tab. 1) | Sparsity (%) | Encoder acc. | Projector acc. |
|---|---|---|---|
| SimCLR | |||
| VICReg | |||
| Rectified LpJEPA, default () | |||
| Top-k, annealed ( = 20/80), best other sparse encoder | |||
| NCL-ReLU, best other sparse projector | |||
| Rectified LpJEPA, | |||
| S$^2$-JEPA, reference-free | |||
| S$^2$-JEPA, label-free |
Both S$^2$-JEPA variants beat every other sparse encoder, by 4.5 points (label-free) and 2.8 points (reference-free) over annealed Top-k, and match or exceed the projector accuracy of NCL-ReLU. The label-free variant also nearly matches the encoder accuracy of dense SimCLR (60.62 vs. 60.70), while dense VICReg remains 2.4 points higher (63.04). SimCLR and VICReg learn dense codes, so they are not a like-for-like comparison. Note that the label-free variant needs a second run, of the dense reference model, whereas the reference-free variant needs only its own run. We have added this comparison to §5.1 of the revised paper.
W3. Speed-up, latency, throughput and FLOPs.
S$^2$-JEPA codes make retrieval over the full ImageNet-1K gallery (1.28M images, with the 50k validation images as queries) 11.4$\times$ faster on CPU and 3.1$\times$ faster on GPU, with a 10$\times$ smaller index and 581$\times$ fewer multiply-adds per query. Both columns store the same codes, as a dense fp16 matrix and as a sparse matrix, so exact search returns identical neighbours (kNN@1 25.5 in both cases) and all of the gain comes from sparsity. The encoder is the same ResNet-50, so the cost of computing a code is unchanged; the gains come from storing and searching the codes. These results are now Tab. 5 and §5.4 of the revised paper, with the full protocol in App. D.7.
| ImageNet-1K, 1.28M gallery, exact cosine search | Dense fp16 | Sparse (S$^2$-JEPA) | Gain |
|---|---|---|---|
| Index memory | 5.25 GB | 0.52 GB | |
| Multiply-adds per query | |||
| Latency per query, CPU (median) | 85.7 ms | 7.5 ms | |
| Throughput, single stream, CPU | 11.7 queries/s | 133 queries/s | |
| Latency per query, GPU (H200), vs. fp16 | 2.20 ms | 0.71 ms | |
| Linear classifier on the code, single input, CPU | 110 s | 31 s |
The advantage grows with scale: the CPU speed-up rises from at 10k images to at 126k and at 1.28M, and the GPU speed-up over dense fp16 from at 126k to at 1.28M. All timings use exact cosine search on one H200 node, with 16 CPU threads for dense search and a single thread for the sparse index, and report the median latency over 200 of the queries. Compressing dense codes to a similar size does not reach the same point: on CIFAR-100, PCA to 20–80 dimensions lowers kNN@10 accuracy by 4.5–4.8 points, and int8 quantization keeps accuracy but needs about 6$\times$ more storage than S$^2$-JEPA (Tab. A17). We have not compared approximate search methods such as product quantization, which change the neighbors returned.
Thank you again for your review. We hope these results address your concerns, and we would be happy to answer any further questions.
Follow-up
Dear Reviewer,
We really thank you again for your detailed review of our paper. We know this is a busy period. We have posted a detailed response addressing each of your comments and the requested experiments, and we would be grateful if you could please take a look when convenient. If you feel your concerns have been addressed, we would sincerely appreciate your considering an updated evaluation. If anything remains unconvincing, we would be glad to discuss it together; we are following this thread closely and will respond promptly. We would greatly welcome any additional feedback you might have.
Response to Reviewer vic8 (1/2)
Thank you for your review and for raising these points. To address each concern, we ran new experiments and added them to the revised paper. In short:
W1. Results on large-scale datasets.
S$^2$-JEPA scales to ImageNet-1K (ResNet-50, 100 epochs) with the same constants as every other setting in the paper (Tab. A1). These results are now Tab. 2 and §5.2 of the revised paper, with details in App. D.7. Frozen linear-probe accuracy, protocol of Tab. 1, mean ± std over 3 seeds; in this and the next two tables, bold marks the better of the two 95%-sparse models:
At the same 95% sparsity, S$^2$-JEPA raises encoder accuracy from 52.35 to 62.35 (+10.0 points) and projector accuracy by +8.6 points. This closes nearly 90% of the encoder accuracy drop caused by sparsity, leaving the encoder within 1.3 points and the projector within 2.1 points of the default model, whose codes are only 20% sparse.
The standard large-scale evaluations show the same picture (low-label accuracy in Tab. 2, per-dataset transfer in Tab. A22).
With only 1% of the labels, S$^2$-JEPA improves on the fixed sparse target by 13.3 points and nearly matches the default model (38.39 vs. 38.64). In transfer to six datasets (the five of Tab. A18 plus CIFAR-100), it improves on the fixed target by 8.3 points on average and again nearly matches the default model (76.58 vs. 76.90). Longer training tells the same story: after 1,000 epochs on ImageNet-100, S$^2$-JEPA at 96.2% sparsity reaches 84.12% encoder accuracy, against 84.58% for the dense model (Tab. A15). All ImageNet-1K results are now in the revised paper (Tab. 2, App. D.7 and Tab. A22). Training on ImageNet-21K was beyond our compute budget during the rebuttal period.