acceptodds
Under review as a conference paper at ICLR 2027

Adaptive Latent Capacity for World Models

Abstract

We introduce Adaptive LeWorldModel (ALeWM), a world model based on a joint-embedding predictive architecture (JEPA) that learns to concentrate predictive information in compact prefixes of a wide latent representation. To encourage this ordering, ALeWM learns a sequence-conditioned distribution over prefix lengths and trains the predictor to estimate the full next embedding from a sampled input prefix. As standard anti-collapse objectives encourage variation across latent coordinates but do not organize them by predictive importance, we also introduce MixSIGReg. MixSIGReg regularizes the masked embeddings against a prior-weighted mixture with Gaussian active prefixes and zeros in the remaining coordinates. As a result, the ALeWM objective encourages early coordinates to retain information useful for prediction and recursive planning. Our analysis shows that the mixture distribution used by MixSIGReg assigns higher variance to earlier coordinate blocks and lower variance to later ones. In addition, we show that, under specified assumptions, prediction error is minimized by placing the information most useful for prediction in earlier blocks. Empirically, we study the behavior of ALeWM in a controlled dynamical system with known state variables and in goal-conditioned visual control. We show that ALeWM consistently achieves higher mean success rates than tuned fixed-width LeWM, with lower planning capacity on average.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
LeWorldModel, JEPA, Nested dropout, Adaptive latent embedding size

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_SHdfOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper trains a world model to concentrate useful predictive information in the first few coordinates of a wide latent representation. A learned selector chooses how many coordinates to use, and MixSIGReg supports training with different prefix lengths. The authors analyze the resulting representations and use them for goal-conditioned visual planning.

Critical strengths
  • Retaining a wide representation during learning while planning with a compact prefix is a useful design. Predicting the full next embedding from a short prefix encourages the early coordinates to retain the information needed for prediction.
  • The experiments show that compact prefixes can improve planning performance. Comparisons with tuned fixed-width models span four tasks and three backbone sizes, and the controlled dynamical system and ablations help explain the learned representation. The paper also presents the full training scheme, rather than the selector alone, as its contribution, which matches the scope of the experiments.
Critical weaknesses
  • The comparisons do not isolate the benefit of input-dependent capacity selection, one of the paper's central design choices alongside the wide representation and MixSIGReg. Nested Dropout changes both the capacity distribution and the regularizer, while the fixed-prefix evaluation changes only the inference-time capacity of an already trained model. Appendix D.4 shows that the selected capacity varies with goal distance, consistent with input dependence, but not that this variation improves planning over a learned global distribution. A learned input-independent distribution trained with MixSIGReg is the missing control.
Questions for authors
  1. To separate input-dependent selection from the learned capacity distribution, please compare (i) a fixed prefix-length distribution, (ii) a learned distribution shared across all inputs, and (iii) the current input-dependent selector under the same MixSIGReg, capacity support, and tuning budget.
1 comment

We thank the reviewer for the careful review and for recognizing the value of our proposed approach. We also appreciate the positive assessment of the experiments and training scheme. We address the reviewer’s concerns below. Updates based on the reviewer’s comments are incorporated in the revised version of the paper.

(1) Separating capacity allocation from the regularizer:

Thank you for this valuable suggestion. We agree that the original comparison did not clearly separate the two changes. We have expanded the Nested Dropout comparison, now shown in Figure 4 of the revised submission, to include both SIGReg and MixSIGReg. Both Nested Dropout (ND) variants predict the full next embedding and sample capacities from a fixed distribution. In particular, Nested Dropout with MixSIGReg retains the same regularizer as ALeWM but fixes the capacity distribution to the prior, qψ=π0q_\psi = \pi_0. This allows us to compare learned capacity allocation against fixed-prior training.

The following table reports test success rates for the best configuration per capacity shown. Results are mean success rate ± SEM over three training seeds.

MethodCapacity 8Capacity 16Capacity 32Capacity 64Capacity 96Capacity 128Capacity 160Capacity 192
PushT ND + SIGReg76.50 ± 1.7690.83 ± 1.7489.50 ± 1.3290.33 ± 0.6091.50 ± 0.8790.00 ± 0.2989.33 ± 1.0990.50 ± 0.76
PushT ND + MixSIGReg89.67 ± 0.8389.17 ± 0.7388.83 ± 0.4490.50 ± 1.3292.50 ± 0.7690.83 ± 0.8390.83 ± 0.6090.83 ± 0.67
Cube ND + SIGReg70.00 ± 1.8069.50 ± 1.1569.83 ± 1.0968.17 ± 0.9367.17 ± 1.9265.83 ± 1.8867.00 ± 0.5066.50 ± 0.29
Cube ND + MixSIGReg72.17 ± 1.3071.00 ± 1.7668.17 ± 2.1766.83 ± 1.3666.17 ± 0.6770.67 ± 1.7672.17 ± 1.6472.83 ± 1.74

For comparison, ALeWM uses the learned selector:

MethodSuccess rate (%) (↑)Mean planning capacity
PushT ALeWM96.00 ± 0.0053.49 ± 10.51
Cube ALeWM79.00 ± 0.7616.00 ± 0.00

From the tables, neither Nested Dropout variant consistently outperforms the other across all capacities, although MixSIGReg achieves the highest Nested Dropout success on both datasets. ALeWM achieves higher mean success than either variant at every tested fixed capacity, while using shorter average prefixes than those at which Nested Dropout performs best. These results support that both components, the learned capacity allocation and MixSIGReg regularizer, account for the observed improvement.

(2) Comparing input-dependent selection with a learned global distribution:

Thank you for identifying this missing control. We agree that the fixed-prefix evaluation and the observed variation in selected capacities do not isolate the benefit of input conditioning during training. We have now added the requested learned input-independent distribution in Appendix D.5, Table 4 of the revised submission. This variant replaces qψ(k∣w)q_\psi(k\mid\mathbf{w}) with a learned categorical distribution qψ(k)q_\psi(k) shared across all inputs. We retain the capacity network but replace its observation input with one randomly drawn vector that remains fixed throughout training and evaluation. The network remains trainable, so the capacity probabilities can change during learning, while remaining independent of the episode. The below table reports mean ± SEM over three training seeds of both variants:

Capacity selectorPushT success (%) (↑)Reacher success (%) (↑)
Input-independent ALeWM94.67 ± 0.8882.67 ± 1.48
Input-dependent ALeWM96.00 ± 0.0085.83 ± 1.01

Input-dependent ALeWM achieves higher mean success on both datasets, supporting the usefulness of conditioning capacity allocation on the observations.

Reviewer_PCYsOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes Adaptive LeWM, a method built on LeWM that pushes the leading (first k) dimensions of the latent embedding to carry the most predictive information. During training, a capacity network predicts a distribution over prefix lengths k from a grid of predefined values. The encoder still outputs a full d-dimensional embedding, and every coordinate after the first k is zeroed. The predictor takes this masked embedding and predicts the full unmasked next embedding. Instead of SIGReg as in LeWM for the anti-collapse objective, they introduce MixSIGReg, which still prevents collapse but expects later dimensions to be zero more often. At planning time, a prefix length k is chosen with the capacity network and fixed throughout that episode. The paper shows improved planning performance across four environments compared with LeWM, while using much smaller prefixes.

Critical strengths
  • The paper is well written, with insightful toy experiments and ablation experiments.
  • The work fixed the train/val/test episode overlapping issue in LeWM. They partitioned the dataset which is the proper way to conduct those main planning experiments.
  • Comparison with LeWM is sound. They compare with both the max d and the best d where d is the embedding dimension for LeWM.
Critical weaknesses
  • The paper only compares with one baseline, LeWM. The most relevant baseline LpWM is only mentioned in the related work but never compared with. The paper should also compare with SOTA world models like DINO-WM, PLDM, temporal straightening, etc. This affects my overall assessment because it’s hard to ground the effectiveness of the method with only one baseline.
  • Making the leading dimensions more informative is a reasonable idea, but the shorter prefix does not reduce compute. The predictor still takes a d-dimensional input and outputs d dimensions to recover the unmasked target, so CEM rollouts cost the same at planning time. The motivation for keeping a wide embedding while concentrating predictive information in a short prefix is therefore unclear. The paper mentions possible benefits (focusing planning on control-relevant information, supporting multiple latent budgets with one model, leaving the remaining coordinates for downstream tasks), but none is directly tested. By contrast, LpWM shows a concrete benefit of sparse embeddings, for example simpler predictors. The paper would be much stronger with a clearer motivation along these lines, backed by experiments.
Questions for authors
  • In the toy example, which k does the capacity network select? Does it settle on k = 4, the true state dimension?
  • Table 4 shows that the selected k is almost always the same for every test episode within a given environment and seed for most environments. Is the capacity network really adapting per episode, or is it learning one width per task? Beyond the Reacher analysis in Appendix D.4, does the selected k correlate with how hard the initial–goal pair is?
  • The smallest capacity in the grid is 8, and the selected values (8–16 on TwoRoom, mostly 32–64 on PushT) are still well above the intrinsic state dimension of these environments (about 2 for TwoRoom; under 10 for PushT). If smaller options such as {1, 2, 4} are added to the grid, does the model learn an even more compact representation?
4 comments

We thank the reviewer for the careful review and for recognizing the clarity of the paper, the insightful toy and ablation experiments and the sound comparison with LeWM. We address the reviewer’s concerns below. Updates based on the reviewer’s comments are incorporated in the revised version of the paper.

(1) Comparisons with additional world models:

Thank you for this useful suggestion. Our work builds directly on LeWM and aims to improve how its representation is organized for prediction and planning, which motivated our initial focus on that baseline. We agree that comparisons with other relevant methods help place the results in a broader context. We have now added comparisons with PLDM, LpWM, and DINO-WM. LpWM appeared on arXiv approximately one month before our submission, and we consider it concurrent work. Nevertheless, we agree that it is closely related and have included it in the comparison. We also thank the reviewer for highlighting temporal straightening, which we now cite in the related work section.

The following tables report test success rates as mean ± SEM over three training seeds. PLDM and LpWM use the shared LeWM backbone and predictor. DINO-WM retains its frozen pretrained spatial representation and is reported only in the corresponding backbone-size block. Figure 3 in the revised version depicts the same information and Appendix E.2 describes the implementation details. From the tables below, ALeWM achieves higher or comparable mean success on Reacher and higher mean success in every evaluated comparison on PushT. Compared to LpWM, the most relevant baseline, on PushT, ALeWM achieves higher mean planning success across all three latent-width settings, with gains of 16.316.3, 16.716.7, and 6.26.2 percentage points for d=96d = 96, 192192, and 384384, respectively. We believe that these results validate the effectiveness of ALeWM compared to recent state of the art JEPA-based methods.

Reacher, d = 96:

MethodTest success (%) (↑)
ALeWM (Ours)85.83 ± 0.60
LeWM (best d)83.67 ± 0.73
LeWM (max d)82.00 ± 1.76
PLDM84.33 ± 1.36
LpWM85.17 ± 0.73
DINO-WM—

Reacher, d = 192:

MethodTest success (%) (↑)
ALeWM (Ours)85.83 ± 1.01
LeWM (best d)84.50 ± 1.15
LeWM (max d)83.67 ± 1.30
PLDM85.00 ± 0.50
LpWM85.83 ± 1.59
DINO-WM—

Reacher, d = 384:

MethodTest success (%) (↑)
ALeWM (Ours)86.17 ± 0.88
LeWM (best d)85.83 ± 0.44
LeWM (max d)84.67 ± 1.20
PLDM84.00 ± 2.02
LpWM83.50 ± 1.26
DINO-WM78.83 ± 3.33

PushT, d = 96:

MethodTest success (%) (↑)
ALeWM (Ours)94.67 ± 0.83
LeWM (best d)89.33 ± 0.44
LeWM (max d)89.33 ± 0.44
PLDM89.17 ± 0.93
LpWM78.33 ± 3.47
DINO-WM—

PushT, d = 192:

MethodTest success (%) (↑)
ALeWM (Ours)96.00 ± 0.00
LeWM (best d)92.67 ± 0.33
LeWM (max d)92.67 ± 0.33
PLDM90.00 ± 1.04
LpWM79.33 ± 4.21
DINO-WM—

PushT, d = 384:

MethodTest success (%) (↑)
ALeWM (Ours)95.50 ± 0.29
LeWM (best d)94.17 ± 1.01
LeWM (max d)92.83 ± 0.33
PLDM85.50 ± 1.26
LpWM89.33 ± 2.19
DINO-WM72.00 ± 0.29

(2) Motivation for compact prefixes:

The paper presents a training scheme that encourages information useful for predicting how the environment evolves to concentrate in the early latent coordinates. We hypothesize that concentrating this information in a short prefix can make the dynamics easier to learn and recursive predictions more reliable. Table 1 supports the practical benefit of this design through improved mean success relative to full-width and tuned fixed-width LeWM. To further examine this hypothesis, we added to section 4.3 in the revised paper an analysis of the paths formed by the learned embeddings following the analysis in temporal straightening (Wang et al., 2026). We measure the cosine similarity between consecutive changes in the embedding of test examples, each spanning five recorded time steps. Higher similarity means that these changes point in more similar directions, indicating straighter latent paths.

DatasetALeWM selected prefix (↑)LeWM (↑)
TwoRoom0.046 ± 0.002−0.264 ± 0.002
PushT0.566 ± 0.0030.516 ± 0.004
Reacher0.040 ± 0.015−0.078 ± 0.001
OGBench-Cube0.512 ± 0.0100.366 ± 0.008

From the table, ALeWM’s selected prefixes have higher mean cosine similarity on all four datasets. This indicates that consecutive changes in their representations have more consistent directions at the five-step scale, even though ALeWM training objective does not explicitly encourage straight trajectories. Together with the planning results, this provides evidence consistent with our hypothesis. In addition, the toy experiment provides complementary evidence about information concentration. Although ALeWM has eight latent coordinates, its first four recover the physical state with a mean linear-probe R2R^2 of 0.9830.983, close to 0.9800.980 for a separately trained four-dimensional LeWM. This shows that a wider representation can concentrate state information recoverable with a linear model in a short prefix.

(3) Capacity selected in the toy experiment:

In the four-factor toy experiment, the learned capacity distribution has its largest probability at k = 4, matching the true state dimension. This pattern appears on both the training and test sets. The average capacity distribution over test examples is:

Capacity kMean probability (%)
1–3 combined< 0.01
452.37
59.48
614.58
710.38
813.19

The distribution therefore favors the true factor count while retaining non-negligable probability on larger prefixes.

(4) Episode-dependent allocation and episode difficulty:

The selector is conditioned on each episode’s observations and can choose different capacities within the same trained model per episode. This is mostly visible in table 4 (or table 7 in the revised version) in PushT (d=192d=192 and 384384), Reacher (d=192d=192), and TwoRoom (d=384d=384) environments in which the maximal probability issued by the capacity network differs between test episodes. In addition, following Reviewer SHdf suggestion, we incorporated new results in Appendix D.5 comparing input-independent selector networks with ALeWM design. The results support the usefulness of adaptation per episode, rather than task, as conditioning on the observations per episode leads to higher success rate.

In general, we expect that environments with more varied observations or dynamics may produce greater variation in selected capacities, although this remains a hypothesis. Appendix D.3 provides a limited example: when one model is trained jointly on Reacher/DMC and TwoRoom, it learns different capacity distributions for the two datasets. However, TwoRoom performance degrades substantially, so this experiment demonstrates different allocations without establishing successful adaptation across both tasks.

Lastly, relating capacity to episode difficulty is less straightforward. The selector is trained through prediction and regularization, without explicit supervision for difficulty. Moreover, physical start–goal distance captures only one aspect of how difficult an episode may be. Nevertheless, we believe that a broader analysis of how capacity relates to difficulty across environments remains an important and interesting direction for future work.

(5) Allowing smaller capacities:

Thank you for suggesting this experiment. We have added a comparison in Appendix D.6 that expands the original capacity support by adding {1,2,4}\{1, 2, 4\}, while retaining the main training and evaluation protocol.

Capacity supportPushT success (%) (↑)PushT mean capacityTwoRoom success (%) (↑)TwoRoom mean capacity
Original96.00 ± 0.0053.49100.00 ± 0.008.00
Expanded with {1, 2, 4}96.33 ± 0.8364.0099.83 ± 0.1710.67

Results are averaged over three training seeds, with success rates reported as mean ± SEM. From the table, performance remains roughly similar in both success rate and capacity allocation.

We note that the selected prefix size is an operational property of the learned representation and prediction objective. We do not claim that it consistently identifies the intrinsic number of physical state variables. The toy experiment shows that the most probable capacity can agree with the known factor count, while the visual-control experiments evaluate useful planning capacity rather than recovery of the intrinsic state dimension. Nevertheless, we view ALeWM as a possible starting point for future work on latent factor identifiability, which would require further theoretical and empirical investigation.

Dear Reviewer,
As we approach the end of the author–reviewer discussion period, we would appreciate your feedback on our responses and would be happy to address any additional comments. If we have satisfactorily addressed your concerns, we would be grateful if you would consider increasing your score accordingly. Thank you for your time and consideration.

Reviewer_szr6OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper targets how to make compressed representation for world model planning while preserving information needed to predict future states. Their proposed method trains the first few coords of a larger representation to predict next state representation, while learning how many coordinates should be retained. The proposed method is validated on 4 visual control tasks along with a toy dynamical system, and the results show planning achieves higher success rate with less dimensions that starting baseline.

Critical strengths
  • Planning results show higher mean success compared to baseline, while using fewer planning dimensions.
  • Paper performs good ablation across components to validate if each new component actually helps. Ex. MixSigReg shows improvement over SIGReg when everything else is kept the same.
Critical weaknesses
  • The main contribution about capability to pick different capacity for different situations is not validated.
  • Comparison with nested dropout has both capacity selector and regularizer changed, so the ablation is not very clear.
Questions for authors
  • Add an experiment where choosing different capacities per episode shows better results than using one fixed capacity for all episodes
1 comment

We thank the reviewer for the careful review and for recognizing the improved planning success with fewer active dimensions, as well as the value of our component ablations. We address the reviewer’s concerns below. Updates based on the reviewer’s comments are incorporated in the revised version of the paper.

(1) Picking different capacity for different situations:

Thank you for raising this point. We first note that our primary contribution is a training scheme that encourages information useful for predicting how the environment evolves to concentrate in the early latent coordinates. This leads to improved control using a compact planning representation. Table 1 supports this goal through improved planning success with substantially fewer active dimensions. Table 1 also shows that ALeWM’s capacity network indeed picks different capacities per environment, for instance in TwoRoom between 8−138-13 dimensions are selected while on Push the active prefix size is 32−6432-64. Moreover, Table 4 in the original submission, now Table 7 in the revised submission, shows that the same trained model can select different capacities for different episodes, particularly on Reacher.

To further examine the benefit of adaptively selecting the capacity, the fixed-prefix experiment in Section 4.3 of the original submission evaluates the same trained ALeWM checkpoints at every supported fixed capacity, without retraining. Within each fixed-capacity condition, all episodes use the same prefix size. We compare these results with the learned selector, which chooses a capacity from each episode’s initial and goal observations. Across the evaluated tasks, the learned selector matches or exceeds the best reported fixed-prefix mean success. On PushT and Reacher, it also uses smaller prefixes on average than the best-performing fixed-prefix alternative. On OGBench-Cube, it selects the same capacity as the best fixed-prefix alternative. Thus, the selector provides a practical way to choose useful compact prefixes without separately evaluating every supported capacity.

(2) Separating the capacity selector from the regularizer:

Thank you for this valuable suggestion. We agree that the original comparison did not clearly separate the two changes. We have expanded the Nested Dropout comparison, now shown in Figure 4 of the revised submission, to include both SIGReg and MixSIGReg. Both Nested Dropout (ND) variants predict the full next embedding and sample capacities from a fixed distribution. In particular, Nested Dropout with MixSIGReg retains the same regularizer as ALeWM but fixes the capacity distribution to the prior, qψ=π0q_\psi = \pi_0. This allows us to compare learned capacity allocation against fixed-prior training.

The following table reports test success rates for the best configuration per capacity shown. Results are mean success rate ± SEM over three training seeds.

MethodCapacity 8Capacity 16Capacity 32Capacity 64Capacity 96Capacity 128Capacity 160Capacity 192
PushT ND + SIGReg76.50 ± 1.7690.83 ± 1.7489.50 ± 1.3290.33 ± 0.6091.50 ± 0.8790.00 ± 0.2989.33 ± 1.0990.50 ± 0.76
PushT ND + MixSIGReg89.67 ± 0.8389.17 ± 0.7388.83 ± 0.4490.50 ± 1.3292.50 ± 0.7690.83 ± 0.8390.83 ± 0.6090.83 ± 0.67
Cube ND + SIGReg70.00 ± 1.8069.50 ± 1.1569.83 ± 1.0968.17 ± 0.9367.17 ± 1.9265.83 ± 1.8867.00 ± 0.5066.50 ± 0.29
Cube ND + MixSIGReg72.17 ± 1.3071.00 ± 1.7668.17 ± 2.1766.83 ± 1.3666.17 ± 0.6770.67 ± 1.7672.17 ± 1.6472.83 ± 1.74

For comparison, ALeWM uses the learned selector:

MethodSuccess rate (%) (↑)Mean planning capacity
PushT ALeWM96.00 ± 0.0053.49 ± 10.51
Cube ALeWM79.00 ± 0.7616.00 ± 0.00

From the tables, neither Nested Dropout variant consistently outperforms the other across all capacities, although MixSIGReg achieves the highest Nested Dropout success on both datasets. ALeWM achieves higher mean success than either variant at every tested fixed capacity, while using shorter average prefixes than those at which Nested Dropout performs best. These results support that both components, the learned capacity allocation and MixSIGReg regularizer, account for the observed improvement.

Comments on the paper

1 comment

Dear reviewers,

The author-reviewer discussion period closes on November 18 (AoE). Please engage in response periods so that we have our issues discussed thoroughly.

AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.