acceptodds
Under review as a conference paper at ICLR 2027

On the Demand-Supply Geometry of Representation Collapse

Abstract

Preventing representation collapse remains a central challenge in Joint-Embedding Predictive Architectures (JEPAs). Endpoint-only constraints cannot fully prevent or repair collapse when intermediate representations lose sample distinctions. We introduce SADL (Supply And Demand Loss), an adaptive one-sided bridge that matches endpoint demand to the available capacity of the representation path. SADL compares demand from the output objective and realized endpoint geometry with the weaker of encoder excitation and cross-endpoint transmission capacity, applying a correction only when demand exceeds supply. It works with different output regularization functions without auxiliary networks, learned gates, or a separate schedule. In single-task and jointly trained four-task LeWorldModel (LeWM) and INTACT experiments, the collapse rate falls from 50% to 0% in a ten-seed collapse test. After only 1 epoch of training with SADL, encoder effective rank increases by 3.4–6.4× and latent prediction error decreases by 20–303× relative to the corresponding endpoint-only baselines. Under multiple regularization functions, INTACT also achieves over 95% mean search-free control success. Across four image datasets, each with multiple paired seeds, SADL delivers consistent seed-wise improvements in accuracy and stability at representative checkpoints, with its largest gains emerging early in training. These findings indicate that demand–supply matching along the representation path is a general principle for faster and more stable representation learning across control and image classification.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
World Models, Joint-Embedding Predictive Architectures, Representation Collapse, Two-Sided Regularization, Numerical Stabilit

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_DWgROpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

This paper studies a gap between regularizing a JEPA representation at the projector output and making that representation attainable through the encoder–projector path during training. SADL models this as a mismatch between endpoint demand and upstream supply, with supply determined by both sample excitation and transmission across the interface. A one-sided penalty updates the encoder when the measured supply falls short. The experiments combine control tasks, collapse diagnostics, controlled bottlenecks, and image classification.

Critical strengths
  • Distinguishing a desirable endpoint from the path required to reach it is meaningful. Compared with DirectCLR and VICReg, SADL more explicitly identifies when additional encoder constraints are needed, rather than applying a fixed constraint throughout training.
  • The approach is supported by extensive experiments, convincing derivations, and detailed analysis.
Critical weaknesses
  • The main experiments are largely excitation-limited, and excitation-only regularization often performs similarly to full SADL. Controlled experiments support the transmission branch, but its importance in ordinary training remains less clear.
  • Classification gains are mainly observed early in training: Imagenette improves by 8.91 percentage points at epoch 20 but only 0.44 at epoch 800, while CIFAR-10 shows almost no gain at epoch 100. The practical value therefore depends on whether SADL reduces the cost of reaching a target accuracy.
Questions for authors

While my familiarity with this area is limited, I find this to be a solid paper overall. My questions (and weakness) should be considered in light of that background.

  1. How does SADL compare with direct encoder regularization when both receive the same tuning budget? Please include the additional training cost and time to reach a fixed performance target.
  2. How sensitive are the bottleneck diagnosis and resulting updates to covariance regularization and the chosen capacity normalization?
8 comments
AuthorsOpenReview ↗

Brief Summary

We thank the reviewer for finding the paper solid overall, and for a reading of it that matches our intent: distinguishing a desirable endpoint from the path required to reach it, and identifying when additional encoder constraints are needed rather than applying a fixed constraint throughout training. Several of the new experiments test this last point directly. In brief:

  • W1 (transmission term): we agree that our main runs are excitation-limited. In the default setting, SADL and its exc-only variant are the same loss. The transmission term binds once excitation outgrows what reaches ZZ, for example at a larger weight or with a stricter ridge, and it then keeps transmission level with excitation. As the weight grows, exc-only loses more success than SADL.
  • W2 (early gains): the early gains do lower the cost of reaching a target. In control, SADL after one epoch exceeds the baseline after two epochs, in both world models and with both attention backends; for INTACT it also exceeds the baseline after ten epochs. On Imagenette, SADL reaches the baseline's final accuracy with 23% fewer updates. On CIFAR-10, SADL leads on 9 of 10 seeds at the early checkpoints up to Ep. 50 (+1.83, +0.89 and +0.68 points; Appendix Table 38), and the final gain under the revised normalization is +0.19 points (7 of 10 seeds).
  • Q1 (direct regularization of HH): when tuned, regularizers applied directly to HH reach success similar to SADL. Over weights up to 1, SADL is the only one whose worst seed stays above 91% at Ep. 1 and 94% at Ep. 2; the others degrade, collapse, or are over-regularized on some seeds when the weight is not tuned. SADL adds 0.5% to 2% to the step time.
  • Q2 (sensitivity): final success and the demand capacity change little when ρ\rho varies over two orders of magnitude or γ\gamma over one. Which supply term is selected does depend on ρ\rho, as expected, and success is unchanged when it switches.

New results are numbered Tables R1 to R12, and all results are mean ± SD over training seeds unless stated otherwise. We use the notation of the revised paper.

The Demand Regularizer D(Z)\mathcal{D}(Z) is the existing endpoint regularizer, unchanged by SADL. The Supply Regularizer is

S=[log⁡q+ϵk+ϵ]+2\mathcal{S}=\left[\log\frac{q+\epsilon}{k+\epsilon}\right]_{+}^{2}

This is the term SADL adds, where qq is the demand capacity and k=min⁡(kexc,ktr)k=\min(k_{exc},k_{tr}) is the supply capacity, built from excitation kexc=Fexc(H)k_{exc}=F_{exc}(H) and transmission ktr=Ftr(H,sg(Z))k_{tr}=F_{tr}(H,\mathrm{sg}(Z)).

"D\mathcal{D}-only" means λS=0\lambda_S=0, and "exc-only" means that kk is replaced by kexck_{exc}.

AuthorsOpenReview ↗

Transmission as a Safeguard: Activation and Empirical Evidence (W1)

W1 (transmission term). The main experiments are largely excitation-limited, and excitation-only regularization often performs similarly to full SADL. Controlled experiments support the transmission branch, but its importance in ordinary training remains less clear.

Response: We agree, and the new runs make the reason explicit. Since k=min⁡(kexc,ktr)k=\min(k_{exc},k_{tr}), at the default weight λS=0.01\lambda_S=0.01, kexck_{exc} was below ktrk_{tr} at every logged training step and in all 25,396 audited records in Appendix C.1. The minimum therefore always picks kexck_{exc}, and SADL and exc-only have the same loss and the same gradient. The gap between them in the first row of Table R1 (0.8 points at Ep. 1, 0.2 at Ep. 2) is run-to-run variation, and it sets the noise level for the comparisons below.

Table R1. Four-task macro success (%), INTACT with SIGReg as the Demand Regularizer (weight 0.03), training seeds 0, 42 and 3072.

λS\lambda_SEp. 1 SADLEp. 1 exc-onlyEp. 2 SADLEp. 2 exc-onlyEp. 1 PushT, SADL / exc-only
0.01 (default)94.56 ± 1.0695.39 ± 0.6096.47 ± 0.4196.67 ± 0.1782.4 / 84.6
0.193.56 ± 1.0994.08 ± 0.9896.22 ± 0.4196.19 ± 0.3277.9 / 79.4
0.393.11 ± 0.6792.44 ± 1.7395.81 ± 0.3295.64 ± 0.8176.4 / 72.9
1.093.50 ± 0.2593.08 ± 0.0895.83 ± 0.5295.83 ± 0.1776.6 / 76.2

The transmission term binds once the encoder's excitation grows faster than what is passed on to ZZ. A larger weight produces this: at λS=0.3\lambda_S=0.3, SADL selects ktrk_{tr} on a growing share of minibatches as training proceeds (Table R2). A stricter ridge has the same effect at the default weight: with ρ=10−2\rho=10^{-2}, ktrk_{tr} is selected on 37% to 43% of minibatches (Q2).

Table R2. Share of training minibatches on which SADL selects ktrk_{tr} (λS=0.3\lambda_S=0.3).

PhasePushTCubeReacherTwoRoom
Ep. 1, steps 0 to 2k0%0%0%0%
Ep. 1, from step 4k14%4%25%45%
Ep. 2, from step 4k27%16%44%63%

As λS\lambda_S grows and ktrk_{tr} starts to be selected, exc-only loses more success than SADL (Table R1). From λS=0.01\lambda_S=0.01 to 0.30.3, Ep. 1 four-task success drops by 2.95 points for exc-only and 1.45 for SADL, and PushT success by 11.7 and 6.0 points; at Ep. 2 the drops are 1.03 and 0.66. For λS≥0.3\lambda_S\ge 0.3, SADL is above exc-only at Ep. 1 at every weight. This is the role we intend for the transmission term: excitation carries the correction in the default setting, and transmission acts as a safeguard once excitation outgrows what reaches ZZ.

Once it binds, the transmission term keeps transmission level with excitation. Table R3 gives the margin ktr−kexck_{tr}-k_{exc} on a fixed held-out probe at Ep. 2. With exc-only, a larger weight keeps expanding HH while transmission lags behind, and on TwoRoom the margin turns negative. SADL keeps it positive. SADL has the larger margin on all four tasks, at both Ep. 1 and Ep. 2, in every paired run with λS≥0.3\lambda_S\ge 0.3: three seeds at each of λS=0.3\lambda_S=0.3 and 1.01.0, and two same-seed reruns at λS=0.3\lambda_S=0.3, which we do not count as additional seeds. The erank and cos of HH and of ZZ differ by less than 2 rank units and 0.001, respectively, between the two variants, so the transmission term acts on the link between HH and ZZ rather than on the spectrum of either one.

Table R3. Margin ktr−kexck_{tr}-k_{exc} at Ep. 2 on a held-out probe.

λS\lambda_STwoRoom SADLTwoRoom exc-onlyReacher SADLReacher exc-only
0.01+0.089+0.093+0.119+0.120
0.1+0.032+0.005+0.060+0.041
0.3+0.028−0.003+0.056+0.031
1.0+0.028−0.006+0.055+0.028

The margin is the quantity this term optimizes, so Table R3 alone does not show that the term matters for a downstream readout. That evidence comes from the controlled test in Appendix F.3 (Table 32(B)). With a built-in transmission deficit and trained encoder weights, SADL raises held-out R2R^2 from −0.018 to 0.535, while exc-only leaves it unchanged. A second case comes from a real image checkpoint. In the forward replay of an ImageNet-LT Ep. 100 checkpoint (Appendix Figure 4), scaling the view residual by 1.5 or more makes ktr<q<kexck_{tr}<q<k_{exc}, and only the transmission term still responds. In end-to-end control, exc-only loses more success than SADL as λS\lambda_S grows (Table R1).

Our reading is therefore the reviewer's: in regular training the transmission term is a safeguard. It contributes no gradient while transmission keeps up and takes over when transmission falls behind, at no extra cost in the default regime. The revision presents it this way. It also states that SADL and exc-only coincide in the default setting and that we do not claim a success-rate advantage for SADL over exc-only.

AuthorsOpenReview ↗

Early Gains and Time-to-Target Training Efficiency (W2)

W2 (early gains). Classification gains are mainly observed early in training: Imagenette improves by 8.91 percentage points at epoch 20 but only 0.44 at epoch 800, while CIFAR-10 shows almost no gain at epoch 100. The practical value therefore depends on whether SADL reduces the cost of reaching a target accuracy.

Response: We agree that practical value depends on the cost of reaching a target. We report this in control and classification.

In control (Table R4), matched D\mathcal{D}-only and SADL runs share code path, data order and evaluation within each model; only backend and SADL vary. For both models/backends, SADL Ep. 1 beats D\mathcal{D}-only Ep. 2 with half the updates (INTACT: 5,695 vs. 11,390); Ep. 2 gains remain 24.4–37.7 points. INTACT SADL Ep. 1 (94.9%–95.4%) also beats D\mathcal{D}-only MATH Ep. 10 (91.58%, SIGReg; Appendix Table 21) with one-tenth the updates, though these are separate runs under the same protocol.

Table R4. Four-task macro success (%), SIGReg Demand Regularizer, seeds 0/42/3072; official evaluation: seeds 0/1/42, 100 episodes/task. INTACT: Direct control; LeWM: Pure-CEM.

ModelBackendD\mathcal{D}-only Ep. 1SADL Ep. 1D\mathcal{D}-only Ep. 2SADL Ep. 2
INTACTMATH40.5 ± 8.594.9 ± 0.470.0 ± 6.196.3 ± 0.6
INTACTFlash45.8 ± 2.695.4 ± 0.266.1 ± 34.596.3 ± 0.1
LeWMMATH30.9 ± 0.863.9 ± 1.246.0 ± 6.070.4 ± 1.6
LeWMFlash32.1 ± 4.963.4 ± 5.731.4 ± 15.869.1 ± 2.9

In classification, Table R5 reports seed-median updates to fractions of baseline final accuracy (3 seeds each for Imagenette/ImageNet-LT; 5 for CIFAR-10). Updates/epoch: 36, 452 and 195, respectively.

Table R5. Updates (thousands; seed medians) to fractions of baseline final accuracy; speed-up = baseline/SADL. No systematic step-time overhead in these runs, so speed-up also reflects training time. At low targets, 1.00 means both reach the target at first evaluation.

Dataset10%20%30%40%50%60%70%80%90%100%
Imagenettebaseline0.040.040.250.610.901.732.633.827.0223.22
SADL0.040.040.140.470.791.262.023.175.8017.82
speed-up1.00×1.00×1.75×1.31×1.14×1.37×1.30×1.20×1.21×1.30×
ImageNet-LTbaseline1.363.165.428.1411.3015.3721.2427.1234.8044.52 (2 of 3 seeds)
SADL0.902.714.977.239.9413.5617.6323.5031.1942.94 (3 of 3 seeds)
speed-up1.50×1.17×1.09×1.12×1.14×1.13×1.21×1.15×1.12×1.04×
CIFAR-10baseline0.200.200.200.200.200.200.391.173.7116.57
SADL0.200.200.200.200.200.200.390.973.5116.96
speed-up1.00×1.00×1.00×1.00×1.00×1.00×1.00×1.20×1.06×0.98×

Imagenette: SADL needs 17.8k vs. 23.2k updates (1.30×) to reach baseline final accuracy; at Ep. 800 it is +0.85 points on three seeds (91.04% vs. 90.19%), or +0.44 on seven seeds (6/7 wins; Appendix Table 38). Peak speed-up is 1.75× at 30%. ImageNet-LT: 10%–90% targets come 1.09–1.50× sooner (largest at 10%); 2/3 baseline vs. 3/3 SADL seeds reach baseline mean final accuracy, with +0.54 points at the final epoch (22.71% vs. 22.17%). CIFAR-10: the 80% target takes about 17% fewer updates, but no speed-up to baseline final accuracy in five-seed runs. Ten-seed gains through Ep. 50 are +1.83, +0.89, +0.68 points (at least 9/10 wins each; Appendix Table 38); the final revised-normalization gain is +0.19 (7/10 wins; Table R12). SADL adds 0.5%–2% step time (Q1), so update savings approximate wall-clock savings.

Table R6 reports paired accuracies on ten CIFAR-10 and seven Imagenette seeds (Appendix Table 38): gains peak early and remain positive on nearly every seed through Ep. 50 and Ep. 100, respectively.

Table R6. Paired classification accuracy (%) from Appendix Table 38; gain = mean paired point difference, wins = seeds with positive gain.

Dataset (seeds)EpochBaseline+SADLGainWins
CIFAR-10 (10)150.81 ± 0.7352.64 ± 0.80+1.8310/10
1066.87 ± 0.4067.76 ± 0.67+0.899/10
5075.16 ± 0.3675.84 ± 0.36+0.689/10
10078.50 ± 0.3078.52 ± 0.32+0.034/10
Imagenette (7)524.23 ± 1.4927.68 ± 1.43+3.457/7
2035.60 ± 4.4844.51 ± 3.13+8.917/7
10069.34 ± 1.9973.15 ± 1.89+3.817/7
80090.25 ± 0.1790.69 ± 0.31+0.446/7

The revision adds Tables R4–R6 and discusses early gains as time to target.

Reviewer_EVHGOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The authors argue that regularizing the output of encoder's projector might not be sufficient to prevent collapse in JEPA models. Instead, they propose SADL, which adds a new loss that encourages encoder features to be diverse and linearly associated with projector features. The resulting model improves across PushT, Cube, TwoRoom, and Reacher environments.

Critical strengths
  • The empirical results in terms of planning performance seem strong.
Critical weaknesses
  • Line 046 -> Official should be official.
  • Line 146-147, Line 155: g_phi and G_phi are inconsistent notation.
  • Line 189 -> what's U? this is not explained.
  • the language used in this paper is very odd and hard to understand. Examples: "encoder excitation", "endpoint", "cross-endpoint transmission", "path-capacity". All of these terms are not defined and as a reader I have to look through the paper back-and-forth very hard for what it means.
  • The paper will benefit from an empirical evidence that there is indeed "collapse" before the endpoint (or projector output). As of now the collapse argument is not convincing.
Questions for authors
7 comments
AuthorsOpenReview ↗

SADL Overview and Notation Clarifications (W1–W3)

We thank the reviewer for finding the planning results strong and for the precise pointers to notation problems, all of which we have fixed. The reviewer summarizes SADL as adding a loss that encourages encoder features to be diverse and linearly associated with projector features. This describes the two quantities SADL measures, but our presentation hid the part that matters most, so we state it plainly here. SADL is not an always-on diversity regularizer. In the notation of the revised paper (Table R1):

  • the Demand Regularizer D(Z)\mathcal{D}(Z) is the endpoint regularizer the model already uses (SIGReg, VISReg, ...), and SADL leaves it unchanged;
  • from D\mathcal{D} and from ZZ, SADL reads how much sample variation the endpoint currently calls for, the demand capacity qq;
  • from HH and ZZ, it measures how much the encoder currently supplies, the supply capacity k=min⁡(kexc,ktr)k=\min(k_{exc},k_{tr}), where kexc=Fexc(H)k_{exc}=F_{exc}(H) measures excitation and ktr=Ftr(H,sg(Z))k_{tr}=F_{tr}(H,\mathrm{sg}(Z)) measures transmission to ZZ;
  • the added Supply Regularizer S=[log⁡q+ϵk+ϵ]+2\mathcal{S}=[\log\frac{q+\epsilon}{k+\epsilon}]_+^2 is zero whenever k≥qk\ge q and grows only as the supply falls short of the demand.

This one-sided, demand-conditioned form is what separates SADL from MCR²-type terms that act at all times. It is also why SADL stays stable over a 100-fold range of its weight while always-on versions of the same quantity do not (Q1). In brief, our responses are:

  • W1 to W3 (notation): all three are fixed.
  • W4 (terminology): the revised paper draws every symbol and term from one notation table and defines each at first use. The four terms the reviewer lists are now defined or renamed (Table R1).
  • W5 (collapse before the endpoint): at the same checkpoint, the encoder representation has lost far more sample variation than the endpoint. Over training, with SIGReg as the Demand Regularizer, the endpoint never dropped below a fixed fraction of its own peak before the encoder did. We also show the mechanism: losses on ZZ have essentially zero gradient along the dominant shared component of HH.
  • Q1 (relation to MCR²): FexcF_{exc} builds on the MCR² expansion term (Yu et al., 2020; Baek et al., 2022), which the revision now cites and compares with in Section 4.3. The key difference in design is that FexcF_{exc} evaluates the rate on centered features while setting the distortion from the uncentered energy, whereas MCR² uses uncentered, unit-normalized features throughout; this makes FexcF_{exc} sensitive to a dominant shared component that MCR² does not separate from sample variation. Beyond this, there is no compression term, and FexcF_{exc} is used as a one-sided constraint rather than an objective.

New results are numbered Tables R1 to R6, and all results are mean ± SD over training seeds unless stated otherwise.


W1 (lowercase "official"). Line 046 -> Official should be official.

Response: Fixed. The revision writes "official" in lowercase throughout. It refers to the upstream evaluator's control protocol and is defined at first use, as opposed to the stricter CLEAR-LeWM criteria in Appendix C.2.


W2 (projector notation). Line 146-147, Line 155: g_phi and G_phi are inconsistent notation.

Response: Fixed. The submission used gϕg_\phi for the projector applied to one sample and GϕG_\phi for the same projector applied to a batch, without saying so. The revision uses a single symbol, PϕP_\phi, for the projector acting on the whole batch, Z=Pϕ(H)Z=P_\phi(H), and JPJ_P for its Jacobian. The batch-level view is not only a matter of notation. BatchNorm uses batch statistics, so PϕP_\phi couples the samples within a batch, and this coupling creates the null space that we measure in our response to W5.


W3 (definition of U). Line 189 -> what's U? this is not explained.

Response: Fixed. U∈RN×dUU\in\mathbb{R}^{N\times d_U} is a generic representation matrix that stands for either HH or ZZ wherever a definition applies to both. The revision states this where UU first appears and lists the quantities defined from it in the notation table: the mean μU\mu_U, the centered features Uˉ\bar U, the centered covariance ΣU\Sigma_U, the uncentered mean energy eUe_U, and fU=tr(ΣU)/(dUeU)f_U=\mathrm{tr}(\Sigma_U)/(d_Ue_U), the fraction of the energy of UU carried by sample variation.

AuthorsOpenReview ↗

Unified Terminology and Notation (W4)

W4 (terminology). the language used in this paper is very odd and hard to understand. Examples: "encoder excitation", "endpoint", "cross-endpoint transmission", "path-capacity". All of these terms are not defined and as a reader I have to look through the paper back-and-forth very hard for what it means.

Response: We agree. The revised paper includes a notation table from which every symbol and term is drawn, and the main text defines each of them at first use. Table R1 lists what changed, and we use the new names in the rest of this response. In plain terms, the four terms the reviewer lists now read as follows.

  • Endpoint. The projector output Z=Pϕ(H)Z=P_\phi(H), where existing anti-collapse regularizers act. The word is kept but defined where it first appears.
  • Encoder excitation is now the excitation function FexcF_{exc}, with supply term kexc=Fexc(H)k_{exc}=F_{exc}(H). For a representation UU, Fexc(U)F_{exc}(U) is high when most of the energy of UU is carried by sample variation and that variation is spread evenly over many directions. It is 0 for a constant representation and at most 1.
  • Cross-endpoint transmission is now the transmission function FtrF_{tr}, with supply term ktr=Ftr(H,sg(Z))k_{tr}=F_{tr}(H,\mathrm{sg}(Z)). It measures, through regularized canonical correlations between HH and ZZ, how much of the variation in HH is linearly carried to the endpoint.
  • Path capacity is now the supply capacity k=min⁡(kexc,ktr)k=\min(k_{exc},k_{tr}). The encoder can only supply variation that it both excites and transmits.

Table R1. Main changes in terminology and notation.

In the submissionIn the revisionMeaning
fθf_\theta; gϕg_\phi, GϕG_\phi, JGϕJ_{G_\phi}EθE_\theta; PϕP_\phi, JPJ_Pencoder; projector (one symbol, acting on the whole batch) and its Jacobian
encoder representation HH, endpoint ZZH=Eθ(X)H=E_\theta(X), endpoint Z=Pϕ(H)Z=P_\phi(H)the endpoint is the projector output
endpoint constraint or objective D(Z)D(Z)Demand Regularizer D(Z)\mathcal{D}(Z)any existing endpoint regularizer (SIGReg, VISReg, QQWorld, ...), unchanged by SADL
supply-shortfall correction, bridgeSupply Regularizer S(H,sg(Z))=[log⁡q+ϵk+ϵ]+2\mathcal{S}(H,\mathrm{sg}(Z))=[\log\frac{q+\epsilon}{k+\epsilon}]_+^2the term SADL adds; zero whenever k≥qk\ge q
Ltotal=Ltask+λDD+λSSL_{total}=L_{task}+\lambda_D D+\lambda_S SL=Ltask+λDD+λSS\mathcal{L}=\mathcal{L}_{task}+\lambda_D\mathcal{D}+\lambda_S\mathcal{S}total objective; λD\lambda_D and λS\lambda_S weight the Demand and Supply Regularizers
encoder excitation CexcC_{exc}excitation function FexcF_{exc}; supply term kexc=Fexc(H)k_{exc}=F_{exc}(H)for a representation UU: how much of its energy is carried by sample variation, and how evenly that variation spreads over directions; in [0,1][0,1]
cross-endpoint transmission CtrC_{tr}transmission function FtrF_{tr}; supply term ktr=Ftr(H,sg(Z))k_{tr}=F_{tr}(H,\mathrm{sg}(Z))how much of the variation in HH is linearly carried to ZZ (regularized canonical correlations); in [0,1][0,1]
path capacity, cutsupply capacity k=min⁡(kexc,ktr)k=\min(k_{exc},k_{tr})the weaker of the two supply terms
qRq_R, qEq_Edemand terms qreg=Freg(sg(Z))q_{reg}=F_{reg}(\mathrm{sg}(Z)), qexc=Fexc(sg(Z))q_{exc}=F_{exc}(\mathrm{sg}(Z))how well D\mathcal{D} is satisfied (rescaled to [0,1][0,1] with D(0)\mathcal{D}(0)), and the excitation already present at ZZ
demand qqdemand capacity q=qreg qexcq=\sqrt{q_{reg}\,q_{exc}}default (α=1/2\alpha=1/2) of the log-space interpolation qreg1−αqexcαq_{reg}^{1-\alpha}q_{exc}^{\alpha}
endpoint-only; Full SADL; excitation-only; transmission-onlyD\mathcal{D}-only; SADL; exc-only; tr-onlyλS=0\lambda_S=0; the full method; kk replaced by kexck_{exc}; kk replaced by ktrk_{tr}
rir_i; r=min⁡(dh,dz)r=\min(d_h,d_z)νi\nu_i; mU=min⁡(dU,N−1)m_U=\min(d_U,N-1), m=min⁡(dH,dZ,N−1)m=\min(d_H,d_Z,N-1)relative eigenvalues; normalization dimensions, equal to the submission's dUd_U and rr whenever N−1N-1 exceeds the feature dimensions, as in all control experiments
rhr_h, rzr_z; chc_h, czc_zerank; coseffective rank; mean off-diagonal sample cosine
E1/E5/E10; OfficialEp. 1/5/10; officialepoch checkpoints; the upstream control-evaluation protocol
MATH, FlashMATH, Flash, defined at first useexplicit versus fused memory-efficient encoder-attention backends, not regularizers

The energy representation (EDE_D, EjE_j) and the phrase "Shape Any Demand-aligned Latent" are also removed.

AuthorsOpenReview ↗

Empirical Evidence of Encoder-Side Collapse (W5)

W5 (collapse before the endpoint). The paper will benefit from an empirical evidence that there is indeed "collapse" before the endpoint (or projector output). As of now the collapse argument is not convincing.

Response: Eq. 1 offered mostly indirect evidence. We provide three direct tests on LeWM: four tasks, ten seeds (40 run-tasks per condition), SIGReg as the Demand Regularizer, and the same runs as Appendix D.4.

At the same checkpoint, the encoder representation has lost far more sample variation than the endpoint. Table R2 compares HH and ZZ at Ep. 1–2. With only the Demand Regularizer, sample variation carries 3%–14% of HH's energy and samples are nearly parallel (mean cosine 0.86–0.97). At the same checkpoints, ZZ has 6–16 times that fraction and higher effective rank. The projector can remix or amplify retained variation but cannot restore state information lost by the encoder, so ZZ shows milder loss. With SADL, HH and ZZ are equally healthy.

Table R2. Encoder representation HH and endpoint ZZ at the same checkpoint (medians over the 40 run-tasks).

Condition and checkpointfHf_HfZf_Zerank(HH)erank(ZZ)cos(HH)
D\mathcal{D}-only, MATH, end of Ep. 10.0390.34317.321.40.961
D\mathcal{D}-only, MATH, end of Ep. 20.1430.80922.045.40.856
D\mathcal{D}-only, Flash, end of Ep. 10.0330.35514.518.50.967
D\mathcal{D}-only, Flash, end of Ep. 20.0380.59616.131.80.962
SADL, Flash, end of Ep. 10.9860.99052.753.30.012
SADL, Flash, end of Ep. 20.9900.99165.264.80.008

Over training, the endpoint never drops first when SIGReg is used as the Demand Regularizer. We logged deep encoder layers (through HH) and ZZ every 569 steps. A drop is the first log below 50% (or 25%) of a layer's own peak, distinct from Section 4.1's end-of-training outcomes. Table R3 gives the crossing order: whenever ZZ crossed, the encoder crossed earlier (5 run-tasks), at the same log (7), or was below threshold from the start (4); never later. In 9 further run-tasks, the encoder fell below 25% while ZZ did not. Thus, with the Demand Regularizer, the drop starts in the encoder, not at the endpoint. With SADL, no layer crossed either threshold.

Table R3. Timing of the drop below 50% (or 25%) of peak in the deep encoder layers relative to ZZ (40 run-tasks per condition, one log every 569 steps).

ConditionThresholdRun-tasks in which ZZ crossedEncoder earlier / same log / later / below threshold from startEncoder crossed, ZZ never did
SIGReg (D\mathcal{D}-only), Flash50%165 / 7 / 0 / 45
SIGReg (D\mathcal{D}-only), Flash25%40 / 3 / 0 / 19
SIGReg with MATH; exc-only; SADLboth0none0

Losses on ZZ cannot see the dominant part of HH. This explains why ZZ can look healthy while HH is not. Initially, HH is dominated by a component shared across inputs; only about 10−410^{-4} of its energy is sample variation (revised Table 1). Training-mode BatchNorm subtracts the batch mean and divides by its standard deviation. Thus a shared shift or common rescaling of HH leaves ZZ unchanged; these directions lie in the null space of JP(H)J_P(H). By Eq. 1, the Demand Regularizer and task loss, which reach HH only through the projector, have zero gradient along these directions. On the same checkpoints, such perturbations change ZZ by about 10−1310^{-13} (shift) and 10−610^{-6} (rescaling) per unit change, versus 2 to 4.5 for generic directions; Table R4 reports gradient shares.

Table R4. Share of each gradient's energy (squared norm with respect to HH) that lies in the null-space directions, on the D\mathcal{D}-only checkpoints.

Gradient with respect to HHShare
Demand Regularizer (0.03⋅0.03\cdotSIGReg)about 10−1010^{-10}
task loss (latent prediction)about 10−910^{-9}
Supply Regularizer S\mathcal{S} (computed at the same checkpoints)0.42 to 0.65

The remaining sample-dependent part of HH grows slowly and in few directions. At Ep. 1 the effective rank of HH is only 14.5 to 17.3 (Table R2), and the Demand Regularizer does little to expand it. SADL acts on HH directly, so it is the only term with a sizable gradient along the shared directions. In practice the bias of the encoder's last LayerNorm learns to cancel the shared mean, and the fraction of the energy of HH carried by sample variation passes 0.8 within the first few hundred steps and settles near 0.99. In the revision, the same-checkpoint comparison of HH and ZZ and the null-space measurements are reported in the main text (Section 4.2 and Table 1), and the timing study of Table R3 is in Appendix D.4 (Table 15).

Reviewer_qH1QOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This paper investigates the challenge of preventing representation collapse and claims that endpoint-only constraints cannot fully prevent or repair collapse when intermediate representations lose sample distinctions. To address this challenge, the authors proposed SADL, a novel loss function for preventing representation collapse in JEPA-style models. It compares the endpoint-derived demand with encoder excitation and cross-endpoint association, penalizing insufficient supply. Experiments on control and image representation learning show that it can improve stability and boost early learning.

Critical strengths
  1. The paper identifies a useful failure mode: endpoint-only constraints cannot fully prevent or repair collapse when intermediate representations lose sample distinctions. It distinguishes regularity at the projected endpoint ZZ from preservation of useful variation in the encoder representation HH, and from transmission of that variation across H→ZH \rightarrow Z. This motivates the need to verify whether the end-point’s present demand matches the path’s attainable supply.

  2. This paper presents SADL (Supply And Demand Loss), an adaptive one-sided bridge that matches endpoint demand to the available capacity of the representation path. The method estimates endpoint demand, measures encoder excitation and cross-endpoint transmission, and applies a one-sided penalty when the weaker capacity falls short. Experiments on control and image representation learning show that it can improve stability and boost early learning.

Critical weaknesses
  1. The paper is not self-contained and the proposed solution is not well presented. Some important terms such as endpoint and encoder excitation are not defined. Many acronyms and mathematical notations are not properly defined before use, which has significantly hampered the readability of this paper.

  2. The presentation and references are not consistent. For example, in the second paragraph of Introduction the author use a “endpoint-only INTACT-VISReg” experiment to show the motivation with the data in Table 2. “In our audited endpoint-only INTACT–VISReg run, Official success falls from 55.67% at epoch one to 24.75% at epoch five, while the effective rank of H falls from 13.74 to 6.64 (Table 2).” However, the reviewer can not find these numbers in Table 2.

  3. The paper presents a theoretical analysis of motivation in Eq (1). It claims that the corrective signal of the endpoint constraint function that prevents representation collapse may vanish, due to its previous projection layer’s gradient (Jacobian matrix). That is, the signal may vanish in the Jacobian null space. However, it lacks empirical evidence demonstrating the vanishing of the corrective signal.

  4. The full demand-supply formulation might be overcomplicated. In Table 3, the excitation-only SADL nearly matches the complete method on both control models. The experiments demonstrate that the transmission branch could be useful, while the additional benefits in the end-to-end training remain unclear.

  5. The design of adaptive demand requires a stronger justification. A matched comparison between a tuned constant demand version using the same capacity scores and penalty would help isolate its contribution. The reported single-seed interpolation study doesn't sufficiently support for the particular demand construction.

Questions for authors
  1. Does full SADL consistently outperform the excitation-only variation?
  2. How does adaptive demand compare with a tuned constant threshold under otherwise identical conditions?
  3. Does the main gain consistently persist with the matched attention backend and training budgets?
8 comments
AuthorsOpenReview ↗

Brief Summary

We thank the reviewer for the careful review and for recognizing the failure mode that motivates SADL: constraints placed only on the endpoint cannot fully prevent or repair collapse when intermediate representations lose sample distinctions. The reviewer also checked our motivating numbers against Table 2, and asked whether each component of SADL is needed and whether the gains survive a matched comparison. We ran all of the requested experiments and respond to each point below. In brief:

  • W1, W2: we agree with both points. The revision draws every symbol and term from a single notation table and defines each at first use, and the motivating example is corrected.
  • W3: with a BatchNorm projector, Eq. 1 has an exact null space. On trained checkpoints, the gradients of the Demand Regularizer and of the task loss carry essentially no energy in it, while 42% to 65% of the SADL gradient lies there. This is why the component shared by all inputs, which dominates HH from initialization, is never reduced by losses computed on ZZ.
  • W4, Q1: in LeWM and INTACT at the default weight (λS=0.01\lambda_S=0.01), SADL and exc-only are the same loss. At larger weights, where ktrk_{tr} is selected, exc-only loses more success than SADL (INTACT; Table R5): the transmission term is a safeguard that comes into play once excitation grows faster than what reaches ZZ.
  • W5, Q2: a tuned constant matches adaptive demand. The best constant ranges from 0.5 to 0.9 across settings, and a poor choice gives no gain, whereas adaptive demand needs no search.
  • Q3: the gains hold with the attention backend and the number of updates matched. SADL raises four-task success by 24 to 54 points in both world models and with both backends.

New results are numbered Tables R1 to R10, and all results are mean ± SD over training seeds unless stated otherwise. From W1 on, we use the notation of the revised paper, summarized in Table R1.

AuthorsOpenReview ↗

Notation and Method Clarification

W1 (terminology and notation). The paper is not self-contained and the proposed solution is not well presented. Some important terms such as endpoint and encoder excitation are not defined. Many acronyms and mathematical notations are not properly defined before use, which has significantly hampered the readability of this paper.

Response: We agree. The submission used its own terms before saying where in the network they act, and several symbols appeared before their definitions. The revision includes a notation table from which every symbol and term in the paper is drawn, and the main text defines each of them at first use, including the abbreviations (Ep. 1/5/10, MATH and Flash, Pure-CEM, Direct control, the official protocol). "Endpoint" is now defined where it first appears, as the projector output Z=Pϕ(H)Z=P_\phi(H). Encoder excitation becomes the excitation function FexcF_{exc}, given with its formula and its range [0,1][0,1]. Some symbols were also overloaded: rr denoted the relative eigenvalues, the interface dimension and, as rhr_h and rzr_z, the effective rank. Table R1 lists the terms that were renamed, and we use the new names in the rest of this response. Section 3 now opens with the collapse mechanism (Section 3.1), defines each measurement function before it is used, and presents the transmission term as a safeguard (Section 3.2; see also W4).

Table R1. Main changes in terminology and notation.

In the submissionIn the revisionMeaning
fθf_\theta; gϕg_\phi, GϕG_\phi, JGϕJ_{G_\phi}EθE_\theta; PϕP_\phi, JPJ_Pencoder; projector (one symbol, acting on the whole batch) and its Jacobian
encoder representation HH, endpoint ZZH=Eθ(X)H=E_\theta(X), endpoint Z=Pϕ(H)Z=P_\phi(H)the endpoint is the projector output
endpoint constraint or objective D(Z)D(Z)Demand Regularizer D(Z)\mathcal{D}(Z)any existing endpoint regularizer (SIGReg, VISReg, QQWorld, ...), unchanged by SADL
supply-shortfall correction, bridgeSupply Regularizer S(H,sg(Z))=[log⁡q+ϵk+ϵ]+2\mathcal{S}(H,\mathrm{sg}(Z))=[\log\frac{q+\epsilon}{k+\epsilon}]_+^2the term SADL adds; zero whenever k≥qk\ge q
Ltotal=Ltask+λDD+λSSL_{total}=L_{task}+\lambda_D D+\lambda_S SL=Ltask+λDD+λSS\mathcal{L}=\mathcal{L}_{task}+\lambda_D\mathcal{D}+\lambda_S\mathcal{S}total objective; λD\lambda_D and λS\lambda_S weight the Demand and Supply Regularizers
encoder excitation CexcC_{exc}excitation function FexcF_{exc}; supply term kexc=Fexc(H)k_{exc}=F_{exc}(H)for a representation UU: how much of its energy is carried by sample variation, and how evenly that variation spreads over directions; in [0,1][0,1]
cross-endpoint transmission CtrC_{tr}transmission function FtrF_{tr}; supply term ktr=Ftr(H,sg(Z))k_{tr}=F_{tr}(H,\mathrm{sg}(Z))how much of the variation in HH is linearly carried to ZZ (regularized canonical correlations); in [0,1][0,1]
path capacity, cutsupply capacity k=min⁡(kexc,ktr)k=\min(k_{exc},k_{tr})the weaker of the two supply terms
qRq_R, qEq_Edemand terms qreg=Freg(sg(Z))q_{reg}=F_{reg}(\mathrm{sg}(Z)), qexc=Fexc(sg(Z))q_{exc}=F_{exc}(\mathrm{sg}(Z))how well D\mathcal{D} is satisfied (rescaled to [0,1][0,1] with D(0)\mathcal{D}(0)), and the excitation already present at ZZ
demand qqdemand capacity q=qreg qexcq=\sqrt{q_{reg}\,q_{exc}}default (α=1/2\alpha=1/2) of the log-space interpolation qreg1−αqexcαq_{reg}^{1-\alpha}q_{exc}^{\alpha}
endpoint-only; Full SADL; excitation-only; transmission-onlyD\mathcal{D}-only; SADL; exc-only; tr-onlyλS=0\lambda_S=0; the full method; kk replaced by kexck_{exc}; kk replaced by ktrk_{tr}
rir_i; r=min⁡(dh,dz)r=\min(d_h,d_z)νi\nu_i; mU=min⁡(dU,N−1)m_U=\min(d_U,N-1), m=min⁡(dH,dZ,N−1)m=\min(d_H,d_Z,N-1)relative eigenvalues; normalization dimensions, equal to the submission's dUd_U and rr whenever N−1N-1 exceeds the feature dimensions, as in all control experiments
rhr_h, rzr_z; chc_h, czc_zerank; coseffective rank; mean off-diagonal sample cosine
E1/E5/E10; OfficialEp. 1/5/10; officialepoch checkpoints; the upstream control-evaluation protocol
MATH, FlashMATH, Flash, defined at first useexplicit versus fused memory-efficient encoder-attention backends, not regularizers

The energy representation (EDE_D, EjE_j) and the phrase "Shape Any Demand-aligned Latent" are removed. The projector now has a single symbol, PϕP_\phi, which maps the whole batch H∈RN×dHH\in\mathbb{R}^{N\times d_H} to Z∈RN×dZZ\in\mathbb{R}^{N\times d_Z}. Because its BatchNorm uses batch statistics, PϕP_\phi couples the samples within a batch, and this coupling is the source of the null space discussed under W3. Where a definition applies to both representations, UU stands for HH or ZZ. For example, fU=tr(ΣU)/(dUeU)f_U=\mathrm{tr}(\Sigma_U)/(d_U e_U) is the fraction of the energy of UU carried by sample variation, which we use under W3.

AuthorsOpenReview ↗

Corrected Evidence and Jacobian Null Space

W2 (inconsistent reference). The presentation and references are not consistent. For example, in the second paragraph of Introduction the author use a "endpoint-only INTACT-VISReg" experiment to show the motivation with the data in Table 2. "In our audited endpoint-only INTACT–VISReg run, Official success falls from 55.67% at epoch one to 24.75% at epoch five, while the effective rank of H falls from 13.74 to 6.64 (Table 2)." However, the reviewer can not find these numbers in Table 2.

Response: The reviewer is right. These numbers come from a single run described in Appendix D.1, so the reference to Table 2 was wrong. The revised introduction no longer relies on this run and instead cites the measurements of Table 1 (Section 4.2): sample variation carries about 0.01% of the energy of HH at initialization and only 3% to 5% after one epoch with a Demand Regularizer alone. The new backend-matched runs (Table R10) give a checkable instance of the failure: with Flash and the Demand Regularizer alone, the INTACT run with training seed 42 drops from 45.3% at Ep. 1 to 27.0% at Ep. 2, while all three seeds trained with SADL stay between 96.2% and 96.4%. In the revision, this run is the collapsed INTACT Flash seed counted in Table 1 (outcomes normal/over-regularized/degraded/collapsed = 0/0/2/1), and it accounts for the large spread of the Flash D\mathcal{D}-only baseline at Ep. 2 (66.1 ± 34.5 in Table R10 and Appendix Table 40); the full backend-matched comparison is in Appendix Table 40 and summarized in Section 4.2.

Comments on the paper

1 comment

Dear Reviewers,

The authors have posted rebuttals. Please read them, comment whether the responses have resolved your concerns and when appropriate adjust your score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.