- Rating
- 2(+1)
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
EV-JEPA aimed to train an elastic V-JEPA model which can choose a fixed set of width for standalone inference or collaborative inference between edge device and cloud. The core contribution is the Transition-Aware Triplet Training, which joint jointly optimize the triplet: T3 = {f_θ (w_s), f_ θ (w_s,w_l,k), f_θ (w_l)}.
Critical strengths
The adaptive/elastic inference is an important and timely topic to meet latency constraints of different applications, to support heterogenous computing resources.
Critical weaknesses
- The paper trained a fixed set of width, then make a combination of w_s and w_l. The training time will increase when the set increases. The so called “elastic” does not actually provide too much elasticity in the solution.
- The main problem is that the paper did not formulate the objective of elastic inference, what exactly the elastic inference wants to achieve? For example, maximize inference accuracy under memory constraints, or minimize inference time under latency constraints.
- The sets of width and number of layers are pre-selected, no principled way behind it.
- There is no clear performance gain of proposed EV-JEPA compared to baseline.
- The elastic token compression has limited novelty. The performance drop of compression is significant as shown Fig. 6 (middle), meaning that the token compression is not well designed. The result of Fig. 6 (right) is trivial.
- In general, the presentation of the paper is poor. In particular, the text of the experiment section is convoluted. The results are not clearly explained.
- The claims in the abstract and conclusion overstate the achievements and are not insightful. For example, “The elastic principle further extends beyond the backbone, to a token compressor that cuts transmitted payload by up to 512x”. Without mentioning the accuracy loss, this claim does not provide any value.
Questions for authors
- It would be interesting to see the runtime of different model configurations in resource-constrained edge devices. Is the runtime proportional to the model parameters?
- Changing the width in hybrid inference, i.e. collaborative inference, in Fig. 3 and Fig. 4, the accuracy changes are quite small. As in the experiment, the split point is at middle layer (L6 or L7), does it is the same when the split is at earlier layers, for example layer 2?
- How to see “improving over prior elastic training by up to 11 points at intermediate width”?
9 comments
Response to reviewer (pt2)
W3. The sets of width and number of layers are pre-selected, no principled way behind it.
We thank the reviewer and agree that the paper should state the reasoning behind these choices; we will add it to Sec. 4.
Using a predefined set of widths on a fixed-depth backbone is common practice in elastic training [38, 10, 19]. The configuration set is a design choice taken from standard model scales rather than derived from a specific deployment. All these previous works keep the depth of the base architecture fixed. EV-JEPA follows the same practice as its T3 represents a general framework rather than a model tuned to a single deployment scenario, and its configurations follow these principles:
- Standard model scales. The widths {192, 384, 768, 1024, 1280, 1408} are the embedding dimensions of the established ViT family (Ti, S, B, L, H and G), so each configuration corresponds to a familiar model scale. All of them are multiples of the attention-head dimension of 64 (from 3 to 22 heads), so every width slices into whole attention heads.
- Covering the device range. The smallest width is chosen so that the smallest configuration fits on a highly constrained wearable device such as smart glasses, with about ~15 MB of memory. The largest widths correspond to server-class models. The width sets W1–W3 therefore span the range from wearable to server.
- Depth. Depth is fixed per family (Table 1), since EV-JEPA is elastic in width; we acknowledge depth elasticity as a limitation (Sec. 5). We set the depth according to our training budget, which bounds the size of the largest configuration we can train. Having variable depth is usually solved with early-exiting but that is application dependent.
Finally, T3 does not depend on a particular width set or depth. It applies to any width and length, and its behaviour is consistent across the three width sets we evaluate (W1–W3, Figs. 3a–c), which differ in range and, for EV-JEPA-H, in depth.
W4. There is no clear performance gain of proposed EV-JEPA compared to baseline.
We thank the reviewer and believe this concern comes from how our baselines are framed, which we will make clearer in the revision. EV-JEPA is compared against baselines with different roles and outperforms all previous elastic-works with important improvements:
- Over prior elastic training (the baselines to beat). Under identical architecture, teacher, data and configurations, T3 improves over US-Net-style training by up to 11 points on SSv2 (Sec. 4.1, Fig. 3c) and by up to 5 points on K400 and EK100 (Sec. 4.2, Fig. 4). The largest gains come at intermediate widths, where prior elastic training degrades. In hybrid execution, T3 also consistently outperforms US-Net-style training (Sec. 4.2).
- Independently trained models (the reference to match, the standalone baseline). Each standalone model is trained separately and serves only its own width. As in prior elastic work, the goal is to match these models with a single set of weights, not to exceed them. EV-JEPA stays within +/-1 point of them across intermediate widths. At full width it surpasses the non-elastic baseline trained under identical conditions (75.4 vs. 75.0 on SSv2, 86.5 vs. 85.5 on K400; Table 3), while the same weights also serve every smaller width and every hybrid configuration. T3 EV-JEPA is clearly ahead of other methods in this aspect. At the lowest width (192), we surpass the standalone baseline by up to 5.6 points.
- Over the small models, through hybrid execution. A device running at width 192 that hands off to the server gains +9.8 Top-1 points on K400 and +12.6 recall@5 points on EK100 compared with running at width 192 alone, and comes within 1 point of the full-width model (Sec. 4.2). No prior method provides this capability.
Response to reviewer (pt3)
W5. The elastic token compression has limited novelty. The performance drop of compression is significant as shown Fig. 6 (middle), meaning that the token compression is not well designed. The result of Fig. 6 (right) is trivial.
We thank the reviewer and would like to clarify the role of the compressor in the paper. We do not claim token compression itself as a core novelty in our paper. The compressor is an additional component showing that the elastic principle extends beyond the backbone to the compression mechanism. One set of kernel weights covers every compression ratio, and all ratios are trained jointly, mirroring T3 (App. A.5). Its purpose is to show what an end-to-end elastic deployment pipeline looks like: a single network that exposes every trade-off knob, namely width, hybrid pair, split layer and compression ratio. We do not claim this compressor is optimal. To the best of our knowledge, however, no existing approach offers a fully elastic end-to-end pipeline of this kind. We would be grateful if the reviewer could point us to any work that provides all of these capabilities, including compression, so that we can discuss and compare against it.
On the accuracy drop. The compressed path stays within 0.7 points of the uncompressed hybrid path at 64× compression, and within 5.1 points at 512× (Sec. 4.4, Fig. 6 middle). These drops buy a large reduction in transmission: compared with the uncompressed hybrid, 512× compression cuts transmission time by about 98% (e.g., from about 3.1 s to 0.05 s on a 1 Mbps link). The 5.1-point drop at 512× is therefore the extreme end of a controllable trade-off, which a deployment would select only when bandwidth is the bottleneck. The compressor can certainly be improved, and we look forward to doing so in future work.
W6. Presentation
We thank the reviewer for the constructive feedback. We will improve the presentation of the experimental section in the revised manuscript, addressing the reviewers' concerns and making the results easier to follow.
W7. The claims in the abstract and conclusion overstate the achievements and are not insightful. For example, “The elastic principle further extends beyond the backbone, to a token compressor that cuts transmitted payload by up to 512x”. Without mentioning the accuracy loss, this claim does not provide any value.
We thank the reviewer and will adjust the wording accordingly. In particular, the conclusion will state the accuracy cost alongside the payload reduction. As shown in Fig. 6 (middle), compression costs 0.4–0.7 points at 64× and 3.5–5.1 points at 512×, depending on the exit layer, and the earliest exit (k = 4) gives the smallest loss at every ratio. The revised sentence will read: "The elastic principle further extends beyond the backbone, to a token compressor that reduces the transmitted payload by 64× to 512× at an accuracy cost of 0.4 to 5.1 points, depending on the compression ratio and exit layer."
Q1. It would be interesting to see the runtime of different model configurations in resource-constrained edge devices. Is the runtime proportional to the model parameters?
We measured batch-1 latency of EV-JEPA standalone widths (12 layers, 16×256×256 input) on two edge platforms: an Arm Ethos-U85-2048 NPU, a microcontroller-class accelerator representative of smart glasses, and an NVIDIA Jetson AGX Orin.
| Width | Params | ARM Ethos U85 | Jetson Orin |
|---|---|---|---|
| 192 | 5.6M | 234 ms | 4.05 ms |
| 384 | 21.9M | 478 ms | 8.00 ms |
| 768 | 86.2M | 999 ms | 20.13 ms |
Runtime scales approximately linearly with width. Note that larger models require more memory for the model and activation, so it is constrained by device memory capacity. Each step in EV-JEPA's width set therefore gives a predictable, roughly proportional latency saving on edge devices.
Response to reviewer
W1. The paper trained a fixed set of width, then make a combination of w_s and w_l. The training time will increase when the set increases. The so called “elastic” does not actually provide too much elasticity in the solution.
We thank the reviewer for this comment. Training on a predefined set of widths is standard in elastic training (e.g.,MatFormer [10]). We acknowledge that training cost depends on the size of this set, and list this as a limitation (Sec. 5). However, we respectfully disagree that it limits the practical elasticity of the approach and its benefits.
Training time on 128 H100 GPUs. Separate training takes about 4 days per model (3–6 days depending on width). Additional split layers add no training time.
W2. The paper did not formulate the objective of elastic inference, what exactly the elastic inference wants to achieve? Maximize inference accuracy under memory constraints, or minimize inference time under latency constraints.
We thank the reviewer for the question. We respectfully note that the paper defines this objective in Sec. 3.1 ("Problem formulation"). Intuitively, we want to train one large model with multiple submodels with shared weights, so that we can pick the best ones to run either in an edge device or across edge and clouds. Formally, the goal is that a single set of parameters θ must (i) perform well at every standalone width w ∈ W, so that each device can run the configuration that fits its compute budget, and (ii) support distributed execution, where a resource-constrained device runs the first k layers at width ws and a stronger device continues at width wl. Training accordingly minimizes the loss over the entire configuration space C of standalone and hybrid configurations (Eq. 7).
EV-JEPA deliberately does not commit to a single deployment objective. Instead, it provides the means to choose one. A configuration c is one complete way of running the model. Either a standalone width w, or a hybrid tuple (ws,wl,k,ρ) that specifies the source device width, the destination width, the split layer and the compression ratio. All configurations share the same weights and together form the configuration space C. Moving across C trades off three quantities, on-device computation, transmission latency and accuracy (Abstract; Sec. 4.4; App. A.5), with no retraining. Each objective the reviewer mentions is simply a different rule for selecting a configuration from C. For example, to maximize accuracy under a device memory budget M, a latency budget T and a link bandwidth B, we deploy the configuration c∗ that has the highest accuracy among all configurations that fit in the device memory and meet the latency budget:
c∗=argmaxc∈C Acc(c)s.t.Memdev(c)≤M,tdev(c)+BP(c)+tsrv(c)≤T,
where Memdev(c) is the memory required on the device, P(c) is the transmitted payload, and tdev(c) and tsrv(c) are the device and server compute times. Minimizing latency under an accuracy or memory constraint is the same problem with the objective and the constraint swapped. Because all configurations share one set of weights, c∗ can change at run time, for example when bandwidth drops, without retraining or swapping models. We will make this formulation explicit in Sec. 3.1.