acceptodds
Under review as a conference paper at ICLR 2027

EV-JEPA: Elastic V-JEPA for Any Device and Cross-Device Inference

Abstract

Foundation models like V-JEPA are increasingly deployed across a heterogeneous edge–cloud continuum spanning smart glasses, phones, robots, and servers. Yet, they are typically trained as rigid, fixed-capacity networks bound to one device, supporting neither reconfigurable compute budgets nor collaborative execution. Existing approaches address only one axis at a time: slimmable and NAS-based methods provide width elasticity but assume single-device inference, whereas split-computing methods enable cross-device collaboration only for fixed partitions and static capacities. We introduce Elastic V-JEPA (EV-JEPA), a single model trained once and deployable everywhere along the continuum. Its width is reconfigurable, and it is natively partitionable across heterogeneous devices. At its core is Transition-Aware Triplet Training (T3): at every step, a pair of widths is trained both standalone and as endpoints of a hybrid configuration that transitions between them in a split layer, so that the same weights serve the execution of single-width and cross-width. Across various benchmarks, EV-JEPA falls within 1% of independently trained baselines while improving over previous elastic training approaches by up to 11%. Cross-device hybrid execution recovers to within 2% of the highest-capacity standalone model and supports many-to-one, any-to-any, and variable-split configurations. Finally, we show that the elastic principle extends beyond the backbone to an elastic token compression mechanism that exposes controllable trade-offs among on-device computation, latency, and accuracy.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
JEPA, video representations, distributed neural networks, elastic networks, nested networks, slimmable networks

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_t5TUOpenReview ↗
Rating
2(+1)
Confidence
2
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

EV-JEPA aimed to train an elastic V-JEPA model which can choose a fixed set of width for standalone inference or collaborative inference between edge device and cloud. The core contribution is the Transition-Aware Triplet Training, which joint jointly optimize the triplet: T3 = {f_θ (w_s), f_ θ (w_s,w_l,k), f_θ (w_l)}.

Critical strengths

The adaptive/elastic inference is an important and timely topic to meet latency constraints of different applications, to support heterogenous computing resources.

Critical weaknesses
  1. The paper trained a fixed set of width, then make a combination of w_s and w_l. The training time will increase when the set increases. The so called “elastic” does not actually provide too much elasticity in the solution.
  2. The main problem is that the paper did not formulate the objective of elastic inference, what exactly the elastic inference wants to achieve? For example, maximize inference accuracy under memory constraints, or minimize inference time under latency constraints.
  3. The sets of width and number of layers are pre-selected, no principled way behind it.
  4. There is no clear performance gain of proposed EV-JEPA compared to baseline.
  5. The elastic token compression has limited novelty. The performance drop of compression is significant as shown Fig. 6 (middle), meaning that the token compression is not well designed. The result of Fig. 6 (right) is trivial.
  6. In general, the presentation of the paper is poor. In particular, the text of the experiment section is convoluted. The results are not clearly explained.
  7. The claims in the abstract and conclusion overstate the achievements and are not insightful. For example, “The elastic principle further extends beyond the backbone, to a token compressor that cuts transmitted payload by up to 512x”. Without mentioning the accuracy loss, this claim does not provide any value.
Questions for authors
  1. It would be interesting to see the runtime of different model configurations in resource-constrained edge devices. Is the runtime proportional to the model parameters?
  2. Changing the width in hybrid inference, i.e. collaborative inference, in Fig. 3 and Fig. 4, the accuracy changes are quite small. As in the experiment, the split point is at middle layer (L6 or L7), does it is the same when the split is at earlier layers, for example layer 2?
  3. How to see “improving over prior elastic training by up to 11 points at intermediate width”?
9 comments
AuthorsOpenReview ↗

Response to reviewer

W1. The paper trained a fixed set of width, then make a combination of w_s and w_l. The training time will increase when the set increases. The so called “elastic” does not actually provide too much elasticity in the solution.

We thank the reviewer for this comment. Training on a predefined set of widths is standard in elastic training (e.g.,MatFormer [10]). We acknowledge that training cost depends on the size of this set, and list this as a limitation (Sec. 5). However, we respectfully disagree that it limits the practical elasticity of the approach and its benefits.

  • Range and operating points. A single EV-JEPA-H model spans a ~50× range in model size, from configurations suited to wearable devices to server-scale ones (Table 1). Hybrid execution multiplies the operating points beyond the number of widths, |C| = |W| + (#width pairs)·|K|. EV-JEPA-H supports 11 configurations with a single split layer, and up to 51 with any-to-any transitions and three split layers (table below).
  • Training cost compared with the alternative. Without elasticity, every configuration requires its own model. With the same number of data-loading operations and backward calls as training a single model, and three forward passes per step instead of one, T3 produces every configuration at once. As the table shows, the advantage grows with the configuration space: 51 configurations that would take about 204 days to train separately are obtained from a single model in under 9 days, over 22× reduction in training time. This comes with no significant loss in accuracy in contrast to previous works: EV-JEPA stays within +/-1 point of the independently trained models (Sec. 4).
EV-JEPA-H settingConfigurationsTraining separatelyEV-JEPA-H (single model)
Many-to-one, 1 split layer (Table 1)11~44 days< 6 days
Many-to-one, 3 split layers21~84 days< 6 days
Any-to-any, 1 split layer21~84 days< 9 days
Any-to-any, 3 split layers51~204 days< 9 days

Training time on 128 H100 GPUs. Separate training takes about 4 days per model (3–6 days depending on width). Additional split layers add no training time.

W2. The paper did not formulate the objective of elastic inference, what exactly the elastic inference wants to achieve? Maximize inference accuracy under memory constraints, or minimize inference time under latency constraints.

We thank the reviewer for the question. We respectfully note that the paper defines this objective in Sec. 3.1 ("Problem formulation"). Intuitively, we want to train one large model with multiple submodels with shared weights, so that we can pick the best ones to run either in an edge device or across edge and clouds. Formally, the goal is that a single set of parameters θ must (i) perform well at every standalone width w ∈ W, so that each device can run the configuration that fits its compute budget, and (ii) support distributed execution, where a resource-constrained device runs the first k layers at width wsw _ s and a stronger device continues at width wlw _ l. Training accordingly minimizes the loss over the entire configuration space C\mathcal{C} of standalone and hybrid configurations (Eq. 7).

EV-JEPA deliberately does not commit to a single deployment objective. Instead, it provides the means to choose one. A configuration cc is one complete way of running the model. Either a standalone width ww, or a hybrid tuple (ws,wl,k,ρ)(w _ s, w _ l, k, \rho) that specifies the source device width, the destination width, the split layer and the compression ratio. All configurations share the same weights and together form the configuration space C\mathcal{C}. Moving across C\mathcal{C} trades off three quantities, on-device computation, transmission latency and accuracy (Abstract; Sec. 4.4; App. A.5), with no retraining. Each objective the reviewer mentions is simply a different rule for selecting a configuration from C\mathcal{C}. For example, to maximize accuracy under a device memory budget MM, a latency budget TT and a link bandwidth BB, we deploy the configuration c∗c^{\ast} that has the highest accuracy among all configurations that fit in the device memory and meet the latency budget:

c∗=arg⁡max⁡c∈C Acc(c)s.t.Memdev(c)≤M,tdev(c)+P(c)B+tsrv(c)≤T,c^{\ast} = \arg\max _ {c \in \mathcal{C}} \ \mathrm{Acc}(c) \quad \text{s.t.} \quad \mathrm{Mem} _ {\text{dev}}(c) \le M, \quad t _ {\text{dev}}(c) + \frac{P(c)}{B} + t _ {\text{srv}}(c) \le T,

where Memdev(c)\mathrm{Mem} _ {\text{dev}}(c) is the memory required on the device, P(c)P(c) is the transmitted payload, and tdev(c)t _ {\text{dev}}(c) and tsrv(c)t _ {\text{srv}}(c) are the device and server compute times. Minimizing latency under an accuracy or memory constraint is the same problem with the objective and the constraint swapped. Because all configurations share one set of weights, c∗c^{\ast} can change at run time, for example when bandwidth drops, without retraining or swapping models. We will make this formulation explicit in Sec. 3.1.

AuthorsOpenReview ↗

Response to reviewer (pt2)

W3. The sets of width and number of layers are pre-selected, no principled way behind it.

We thank the reviewer and agree that the paper should state the reasoning behind these choices; we will add it to Sec. 4.

Using a predefined set of widths on a fixed-depth backbone is common practice in elastic training [38, 10, 19]. The configuration set is a design choice taken from standard model scales rather than derived from a specific deployment. All these previous works keep the depth of the base architecture fixed. EV-JEPA follows the same practice as its T3 represents a general framework rather than a model tuned to a single deployment scenario, and its configurations follow these principles:

  • Standard model scales. The widths {192, 384, 768, 1024, 1280, 1408} are the embedding dimensions of the established ViT family (Ti, S, B, L, H and G), so each configuration corresponds to a familiar model scale. All of them are multiples of the attention-head dimension of 64 (from 3 to 22 heads), so every width slices into whole attention heads.
  • Covering the device range. The smallest width is chosen so that the smallest configuration fits on a highly constrained wearable device such as smart glasses, with about ~15 MB of memory. The largest widths correspond to server-class models. The width sets W1–W3 therefore span the range from wearable to server.
  • Depth. Depth is fixed per family (Table 1), since EV-JEPA is elastic in width; we acknowledge depth elasticity as a limitation (Sec. 5). We set the depth according to our training budget, which bounds the size of the largest configuration we can train. Having variable depth is usually solved with early-exiting but that is application dependent.

Finally, T3 does not depend on a particular width set or depth. It applies to any width and length, and its behaviour is consistent across the three width sets we evaluate (W1–W3, Figs. 3a–c), which differ in range and, for EV-JEPA-H, in depth.

W4. There is no clear performance gain of proposed EV-JEPA compared to baseline.

We thank the reviewer and believe this concern comes from how our baselines are framed, which we will make clearer in the revision. EV-JEPA is compared against baselines with different roles and outperforms all previous elastic-works with important improvements:

  • Over prior elastic training (the baselines to beat). Under identical architecture, teacher, data and configurations, T3 improves over US-Net-style training by up to 11 points on SSv2 (Sec. 4.1, Fig. 3c) and by up to 5 points on K400 and EK100 (Sec. 4.2, Fig. 4). The largest gains come at intermediate widths, where prior elastic training degrades. In hybrid execution, T3 also consistently outperforms US-Net-style training (Sec. 4.2).
  • Independently trained models (the reference to match, the standalone baseline). Each standalone model is trained separately and serves only its own width. As in prior elastic work, the goal is to match these models with a single set of weights, not to exceed them. EV-JEPA stays within +/-1 point of them across intermediate widths. At full width it surpasses the non-elastic baseline trained under identical conditions (75.4 vs. 75.0 on SSv2, 86.5 vs. 85.5 on K400; Table 3), while the same weights also serve every smaller width and every hybrid configuration. T3 EV-JEPA is clearly ahead of other methods in this aspect. At the lowest width (192), we surpass the standalone baseline by up to 5.6 points.
  • Over the small models, through hybrid execution. A device running at width 192 that hands off to the server gains +9.8 Top-1 points on K400 and +12.6 recall@5 points on EK100 compared with running at width 192 alone, and comes within 1 point of the full-width model (Sec. 4.2). No prior method provides this capability.
AuthorsOpenReview ↗

Response to reviewer (pt3)

W5. The elastic token compression has limited novelty. The performance drop of compression is significant as shown Fig. 6 (middle), meaning that the token compression is not well designed. The result of Fig. 6 (right) is trivial.

We thank the reviewer and would like to clarify the role of the compressor in the paper. We do not claim token compression itself as a core novelty in our paper. The compressor is an additional component showing that the elastic principle extends beyond the backbone to the compression mechanism. One set of kernel weights covers every compression ratio, and all ratios are trained jointly, mirroring T3 (App. A.5). Its purpose is to show what an end-to-end elastic deployment pipeline looks like: a single network that exposes every trade-off knob, namely width, hybrid pair, split layer and compression ratio. We do not claim this compressor is optimal. To the best of our knowledge, however, no existing approach offers a fully elastic end-to-end pipeline of this kind. We would be grateful if the reviewer could point us to any work that provides all of these capabilities, including compression, so that we can discuss and compare against it.

On the accuracy drop. The compressed path stays within 0.7 points of the uncompressed hybrid path at 64× compression, and within 5.1 points at 512× (Sec. 4.4, Fig. 6 middle). These drops buy a large reduction in transmission: compared with the uncompressed hybrid, 512× compression cuts transmission time by about 98% (e.g., from about 3.1 s to 0.05 s on a 1 Mbps link). The 5.1-point drop at 512× is therefore the extreme end of a controllable trade-off, which a deployment would select only when bandwidth is the bottleneck. The compressor can certainly be improved, and we look forward to doing so in future work.

W6. Presentation

We thank the reviewer for the constructive feedback. We will improve the presentation of the experimental section in the revised manuscript, addressing the reviewers' concerns and making the results easier to follow.

W7. The claims in the abstract and conclusion overstate the achievements and are not insightful. For example, “The elastic principle further extends beyond the backbone, to a token compressor that cuts transmitted payload by up to 512x”. Without mentioning the accuracy loss, this claim does not provide any value.

We thank the reviewer and will adjust the wording accordingly. In particular, the conclusion will state the accuracy cost alongside the payload reduction. As shown in Fig. 6 (middle), compression costs 0.4–0.7 points at 64× and 3.5–5.1 points at 512×, depending on the exit layer, and the earliest exit (k = 4) gives the smallest loss at every ratio. The revised sentence will read: "The elastic principle further extends beyond the backbone, to a token compressor that reduces the transmitted payload by 64× to 512× at an accuracy cost of 0.4 to 5.1 points, depending on the compression ratio and exit layer."

Q1. It would be interesting to see the runtime of different model configurations in resource-constrained edge devices. Is the runtime proportional to the model parameters?

We measured batch-1 latency of EV-JEPA standalone widths (12 layers, 16×256×256 input) on two edge platforms: an Arm Ethos-U85-2048 NPU, a microcontroller-class accelerator representative of smart glasses, and an NVIDIA Jetson AGX Orin.

WidthParamsARM Ethos U85Jetson Orin
1925.6M234 ms4.05 ms
38421.9M478 ms8.00 ms
76886.2M999 ms20.13 ms

Runtime scales approximately linearly with width. Note that larger models require more memory for the model and activation, so it is constrained by device memory capacity. Each step in EV-JEPA's width set therefore gives a predictable, roughly proportional latency saving on edge devices.

Reviewer_JeceOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes EV-JEPA, a novel architecture designed to deploy across heterogeneous edge devices. Existing approaches did not allow variable-width split computing, while variable-width methods ran on a single device at a time. This paper proposes Transition-Aware Triplet Training, a novel training approach that enables elastic width and heterogeneous partitioning. At each training step, it uses a pair of widths, both standalone and jointly in a cross-device hybrid configuration. As a result, the method can support both single-width and cross-width execution. We evaluate the method on the SSv2, K400, and EK100 datasets and also test it on VL-JEPA and 3D-CNN models, showing that it maintains accuracy.

Critical strengths
  • Proposed Transition-Aware Triplet Training is novel and enables inference in variable width and cross-device setup together.
  • Strong results on multiple datasets and model architectures.
  • Identification of the transmission bottleneck and compressor module to mitigate it.
Critical weaknesses
  • Some baselines stated in the paper are not included as an elastic training baseline in the evaluation.
  • The method uses a strong teacher model, unlike most prior works; its impact is unclear in the paper.
Questions for authors
  • Explain why some baselines were not included in the evaluation, to ensure that the method is sufficiently compared to prior art.
  • Please provide experiments separating the impact of the teacher model, such as utilizing similar teacher assistance on prior works or the proposed method without a teacher model.
2 comments
AuthorsOpenReview ↗

Response to reviewer

Q1. Explain why some baselines were not included in the evaluation, to ensure that the method is sufficiently compared to prior art.

We thank the reviewer for this question and agree that our choice of baselines should be stated more explicitly. Our evaluation uses three groups of baselines, all trained under identical conditions.

  1. Independently trained standalone models: the target to match. We train one model per width with the same teacher, data, masking and per-model compute (Sec. 4, "Training strategies"). As in prior elastic work, the goal of an elastic model is to match these models at every width with a single set of weights. They are therefore the reference curves in Figs. 3–5.
  2. Sandwich-rule training (US-Net-style). At each step the smallest and largest configurations are optimized together with a randomly sampled one. US-Net [38] introduced this rule, and it has since been adopted by most elastic and weight-sharing methods, including BigNAS [40], AttentiveNAS [35], NASViT [16], Autoformer [7] and the self-supervised US3L [5]. So, while we do not directly compare to other works because we address different problems, we compare to their core supernetwork training method in our experiments and across all widths. Potentially, our T3 can be a replacement of sandwich-rule training for these works.
  3. Distributional sampling (MatFormer-style). One configuration is sampled and optimized per step. This rule underlies the recent line of nested transformer models, from MatFormer [10], which scales nested training to large language and vision transformers.

We write "-style" because none of these methods 1) supports hybrid, cross-width execution, 2) are proven under dense supervision, and are intended as generic elastic training strategies. We extend each sampling rule to the joint space of standalone and hybrid configurations (Sec. 3.3). We then re-implement it with exactly the same architecture, teacher, data, masking, optimizer and configuration set as T3 (Sec. 4), so the training rule is the only variable. US-Net-style training and T3 also optimize the same number of configurations per step (three), so that comparison is compute-matched.

We will make this mapping explicit in the revised Sec. 2 and Sec. 4. Let us know if this addresses the reviewer's question or more details are needed. We are happy to provide more information or even conduct new comparisons.

Q2. Please provide experiments separating the impact of the teacher model, such as utilizing similar teacher assistance on prior works or the proposed method without a teacher model.

We thank the reviewer for giving us the opportunity to clarify this point. The teacher is shared by all compared models, so the reported gains are explained by the training method, not by the teacher.

  • Same teacher for all models. The standalone baselines, MatFormer-style, US-Net-style and T3 are all trained with the same frozen V-JEPA 2 ViT-G teacher, data, masking and optimizer (Sec. 3.4; Sec. 4, "Data & teacher" and "Training strategies"; App. A.2). The only difference between the elastic models is the training rule. The gains of T3 over prior elastic training (up to 11 points on SSv2, Sec. 4.1; up to 5 points on K400 and EK100, Sec. 4.2) are therefore due to T3 itself. We will state explicitly in the revision that the standalone baselines also use the same teacher.
  • Why a teacher is needed. V-JEPA learns by regressing targets from an EMA copy of its own encoder. In an elastic model the student changes configuration at every step, so an EMA target would average parameters across configurations of different capacity and provide an unstable, width-dependent signal (Sec. 3.4). Training without an external teacher would require redesigning V-JEPA's target mechanism for elastic students, which we acknowledge as a limitation (Sec. 5). Teacher-guided elastic training is also common in prior work [38, 40, 20] by either in-place distillation or similar to us, using a full-sized model (Sec. 2).

We will address this accordingly in the revised manuscript.

Let us know if this addresses the reviewer’s question or more details are needed. We are happy to provide more information or even conduct new comparisons.

Thank you for your clarification, and the rebuttal answered my questions clearly. I retain my positive rating.

Reviewer_1JruOpenReview ↗
Rating
4
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper proposes a model with novel learning scheme that enables flexible network width selection and partitioning across heterogenous devices. Specifically, its training scheme jointly optimizes a triplet objective: independent loss on low resource device, high resource device, and the cooperation between low / high resource device. Experimental results show that the model trained with the proposed scheme achieve much better performance than methods either trained with single device or with joint device alone, proving that the proposed scheme helps enhance flexibility of the model to not only being able to adjust computation complexity within a single device, and can be divided into different devices when diverse endpoints with different resource support are available. The idea and implementation are solid, and the experimental results are convincing.

Critical strengths
  1. The idea of triplet training objective makes intuitive sense, and is backed up by strong experimental results
  2. The proposed method addresses complex model deployment in resource constraint environment, which is of high practical value.
Critical weaknesses
  1. no obvious weaknesses
Questions for authors
  1. For token compression, the paper compares accuracy and latency within the proposed model itself under different compression rate. It's better to also show under the optimal configuration, how the overall end to end proposed pipeline compare to existing device partitioned methods in terms of performance and latency
1 comment
AuthorsOpenReview ↗

Response to reviewer

Q1. End-to-end comparison with device-partitioned methods

We thank the reviewer for the positive assessment and for this suggestion. Our interpretation of the question is that it asks how the complete EV-JEPA pipeline, in its optimal configuration, compares with existing device-partitioned methods in accuracy and latency. We see device-partitioned methods as working along three axes:

  1. Split layer: where the model is cut, that is, which layers run on the device and which on the server. Neurosurgeon [22] selects this layer at run time for a fixed model that is identical on both sides.
  2. Compute capacity: how large the network on each side is. A constrained device runs a smaller network and the server a larger one. SplitNets [12] designs these networks jointly for a specific deployment.
  3. Compression at the split: how the features sent over the link are reduced. This is typically done with a learned bottleneck, trained for one split layer and one compression ratio (e.g., BottleFit).

EV-JEPA addresses all three in a single model and makes each of them elastic. The split layer, the width on each side and the compression ratio can all be chosen at run time without retraining.

For the purpose of this experiment, we use the following setup. We use EV-JEPA-B with the device/lower width 192 and the server/higher width 1024, split at k = 6. We assume a 1 Mbit/s link and a wearable-class device budget of about 15 MB (very common in real deployment scenarios). We compare against one baseline per axis:

  • Split layer: Neurosurgeon applied to the width-1024 and width-192 models. It requires no training, since it only selects the split disregarding transmission time.
  • Compute capacity: SplitNets is designed for a different and very specific setting, with two small chips on a wearable device feeding an aggregator, so it cannot be applied directly to ours. As our interpretation of its approach in our scenario, we trained a dedicated small-to-big model for this single configuration (192→1024, k = 6), without elasticity or compression, using the same teacher, data and schedule.
  • Compression: since the optimal configuration depends on the available bandwidth, we report EV-JEPA with its elastic compressor at 64× and 512×.
SetupDevice memoryFitsTransmittedTransmission timeSSv2 Top-1
Full offload of raw video0✓25.2 Mbit25.2 s74.0
Neurosurgeon, w1024, k = 6~77 MB✗16.8 Mbit16.8 s–
Neurosurgeon, w192, k = 6~3.0 MB✓3.15 Mbit3.1 s63.8
Device only, w192~5.6 MB✓0063.8
Dedicated 192→1024 (SplitNets-style)~3.0 MB✓3.15 Mbit3.1 s72.0
EV-JEPA 192→1024~3.0 MB✓3.15 Mbit3.1 s73.2
+ elastic compression 64×~3.5 MB✓0.39 Mbit0.39 s72.5
+ elastic compression 512×~3.5 MB✓0.05 Mbit0.05 s68.1

EV-JEPA-B on SSv2. Device memory counts the int8 weights of the layers that run on the device. Transmission assumes 2048 tokens at 8 bits.

  • Split layer. Splitting a ViT without changing its width saves little transmission, because the large model's intermediate features are almost as large as the raw video (16.8 vs. 25.2 Mbit). Splitting the small model fits on the device but keeps the small model's accuracy. EV-JEPA learns the width transition at the split, so the device runs only narrow layers while accuracy recovers from the wide ones (73.2 vs 63.8).
  • Compute capacity. The dedicated model reaches 72.0. EV-JEPA reaches 73.2 with weights that also serve every other configuration. We attribute this to weight sharing, consistent with the gains in Sec. 4.1.
  • Compression. Bottleneck methods train one bottleneck per ratio. EV-JEPA's compressor covers 64× to 512× with a single set of kernel weights (~0.5 MB at int8), so the ratio can follow the available bandwidth. Transmission time drops from 3.1 s to 0.39 s at 64× (at most 0.7 points lost) and to 0.05 s at 512× (at most 5.1 points lost; Sec. 4.4).

Let us know if this addresses the reviewer's question or if more details are needed.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.