acceptodds
Under review as a conference paper at ICLR 2027

Learning Perspective Matters in Latent World Models

Abstract

Latent world models often organise prediction along different latent directions, yet the perspective governing these directions and their use is typically predefined rather than learned. This leaves an important design choice outside the learning process, even though both the choice of directions and their relative weighting can shape predictive computation. We introduce orthogonal subspace modulation, which jointly learns an orthonormal basis and adaptive gains conditioned on the current state and action. The basis determines where modulation acts, while the gains determine how strongly each direction is used. We formalise their interaction through Gain–Basis Coupling, showing that the latent perspective becomes functionally consequential when different subspaces receive different gains Across four visual-control tasks, learning this perspective improves control performance over strong latent world-model baselines. Detailed analyses further reveal the empirical validity of Gain–Basis Coupling, functional coordination among subspaces, and physical-dynamics accessibility from subspace combinations. These results show that learning perspective can improve how latent representations are organised and used for prediction and control.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
Latent World Models, Latent Perspective, Orthogonal Subspace Modulation, Gain--Basis Coupling, Visual Control

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_soBvOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The proposed method is built upon LeWorldModel and aim to suggest a modulation of the latent space. The idea is to learn several orthogonal basis QkQ_k in order to project the latent vector into KK different sub-spaces. Then KK different gains are computed from the corresponding projected latents and the action to modulate each sub-latent differently. They prove that their method achieve a gain-basis coupling meaning that cross-subspace perturbation affect the modulation operator depending on cross-subspace gain differences (which are empirically high). Finally they use their modulation method in different environments and compute success rate and action probing.

Critical strengths

None

Critical weaknesses

Empirical results show no clear improvement relatively to related state of the art methods as the suggested method always fall within the same confidence interval of almost all the baseline

Questions for authors
  1. When and why should someone use your method rather than SD-Jepa or Sub-Jepa ?
  2. Section 3.1: experiments reported in Figure 4b compare success rate between the learning setting (λ=1\lambda=1) and a settings never seen during training (λ<1\lambda<1). The training setting will obviously outperforms the rest, how can you conclude any meaningful property ?
  3. Section 3.2: I don't understand why a high error in leave-one-out interventions is a desired property, could you explain ?
  4. Appendix A.2: "α^=−1.41528\widehat{\alpha}=-1.41528 giving α=0.19540\alpha=0.19540" is oddly specific, where does it come from ?
  5. Appendix A.2: what is the purpose of the regularization Lbudget⁡\mathcal L_{\operatorname{budget}} of MM ? What happens without it ? With its parameterization I thought that MM was already bounded
7 comments
AuthorsOpenReview ↗

Clarification on Empirical Improvement and Experimental Interpretation

We thank the reviewer for the careful reading and specific questions.

W1: On the size of the empirical improvement

Ours obtains the highest macro success rate among all locally evaluated methods (87.25%), +8.00 pp over LeWorldModel and +3.17 pp over Sub-JEPA, and improves over LeWorldModel on all four tasks. The error bars in Figure 3 are standard deviations across six CEM planner seeds, not confidence intervals, consistent with the evaluation and reporting protocol of Sub-JEPA (Zhao et al., 2026, Table 1) and QQWorld (Yu et al., 2026, Table 1), so they describe planner variability and overlap does not by itself indicate equal performance. The advantage is also consistent across new controlled comparisons: three additional baselines trained for the rebuttal (Q=IQ=I, capacity-matched FiLM, Non-Orthogonal) reach 84.25--84.58% macro success rate, all below Ours. Following the original LeWorldModel paper (Maes et al., 2026, Table 5), which evaluates robustness by training PushT across three independent seeds, we also did so. Ours achieves an average success rate of 94.34%, compared with 91.78% for FiLM and 91.33% for (Q=IQ=I).

Q1: On when and why to use our method over SD-JEPA or Sub-JEPA

Our method is particularly promising for rotation-sensitive manipulation tasks such as PushT. Figure 6 shows the clearest improvement in the linear accessibility of T-object rotation, while Figure 3 shows our largest control performance margin over competing baselines on PushT. Unlike Sub-JEPA and SD-JEPA, which use predefined subspaces, our method jointly learns where to modulate (QQ) and how strongly (adaptive gains). Furthermore, beyond the choice between methods, the main contribution includes the finding itself: the perspective from which a latent world model views its representation matters, and learning it improves both control (Section 2.2) and the accessibility of the learned dynamics (Section 3.3). We hope this finding can also inspire other latent representation learning methods.

Q2: On the interpretation of Figure 4b

Learning the perspective matters because it determines where the gains apply:

Figure 4b does not compare λ=1\lambda=1 against λ<1\lambda<1. It compares two bases at the same λ\lambda, on the same checkpoint with identical gains. At λ=0\lambda=0 the gains are equal and the two bases give identical results, as Theorem 1 predicts; the gap opens only as gain differences are restored. To address the reviewer's concern, we repeated the same intervention on a separately trained Frozen Haar checkpoint, whose basis was fixed throughout training:

Model (λ=1\lambda=1)Native QQReplaced QQGap
Learned QQ95.6780.3315.34 pp
Frozen Haar90.3387.003.33 pp

Replacing the basis costs 15.34 pp when the basis was learned and 3.33 pp when it was not. Familiarity with the training setting therefore explains only a small part of the gap; the rest reflects the coupling between differentiated gains and the learned directions.

Q3: On why a leave-one-out error increase is desirable

Leave-one-out ablation measures the marginal predictive contribution of each subspace correction. We remove one correction while keeping the other seven active, and measure the resulting increase in prediction error: ΔkLOO=E(without k)−E(all)\Delta_k^{\mathrm{LOO}} = E(\text{without } k) - E(\text{all}). A positive value means that the correction contributes additional predictive value even when combined with the other seven, rather than being redundant. Thus, the desired property is a positive error increase upon removal, not a high prediction error itself. All 8 learned corrections have positive contributions, compared with 6/8 for Frozen Haar. We will clarify this interpretation in Section 3.2.

Q4: On the value −1.41528-1.41528 in Appendix A.2

It is a learned parameter, not a chosen hyperparameter: bα=−1.41528b_\alpha=-1.41528 is the value in the reported PushT checkpoint, and α=σ(bα)=0.19540\alpha=\sigma(b_\alpha)=0.19540 is the resulting bound on the residual gain. We will state its origin and round the displayed value.

Q5: On the purpose of Lbudget\mathcal{L}_{\text{budget}}

The bound and the budget do different jobs:

The parameterization bounds each gain, 1<gt,k<1+α1<g_{t,k}<1+\alpha, but does not prevent all gates from saturating together. The budget controls their mean activation (target ρ=0.3\rho=0.3), which keeps the gates selective. Removing it and retraining three PushT models with matched seeds:

MetricWith budgetNo budget
Success rate93.78%91.56%
Mean activation0.3070.762
Gain CV3.71%2.52%

Without the budget, mean activation rises to 0.762, gain differentiation (CV) falls, and success rate drops by 2.22 pp. The bound limits the maximum gain; the budget limits average gate usage.

[1] Zhao et al. (2026). Sub-JEPA: Subspace Gaussian Regularization for Stable End-to-End World Models. arXiv:2605.09241.

[2] Yu et al. (2026). QQWorld: Quantile-Quantile Matching for World Model Regularization. arXiv:2607.28415.

[3] Maes, L., Le Lidec, Q., Scieur, D., LeCun, Y., & Balestriero, R. (2026). LeWorldModel: Stable End-to-End Joint-Embedding Predictive Architecture from Pixels. arXiv:2603.19312.

[4] Perez et al. (2018). FiLM: Visual Reasoning with a General Conditioning Layer. AAAI 2018. arXiv:1709.07871.

Q1: I struggle to understand why the linear accessibility of one specific action of one specific environment represents a meaningful improvement. The theoretical and experimental findings on the Gain-Basis coupling are interesting but are done relatively to the Frozen-Haar method, not to the baseline (as far as I know) which limits its impact. How can you extend your conclusion that "the perspective from which a latent world model views its representation matters" to the other methods of the literature ?
Q2: Ok, I understand better what you are achieving with this Figure
Q3: I understand that you aim to show that there is no redundancy in subspace correction. However positive LOO can also be interpreted as a low generalization capability, indeed this intervention reminds me of dropout in neural networks where you purposely require the neural networks to have great performance with partially hidden internal representation in order to improve generalization. Is Leave-one-out metric commonly used to measure redundancy ? I can understand that low redundancy could imply positive LOO, however the reciprocal is not obvious. Are you sure that positive LOO is not caused by an undesired property ? The analysis would be easier with an information-based metric rather than prediction-based
Q4: Did you mean α=σ(α^)\alpha=\sigma(\hat\alpha) ? Ok, it was not clear to me that α^\hat\alpha was learned, I thought that it was fixed.
Q5: What do you mean with "keeps the gates selective" ? What prevents to have mˉ=ρ\bar m=\rho with all the gt,kg_{t,k} having the same value ? Did you do the same hyper-parameters (e.g. SIGReg coefficient) search for the setting without budget control than with budget control ? Could you report confidence interval ?

Reviewer_8jWLOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

In latent world models different parts of a learned representation are relevant for predicting the future of the environment. The authors propose a method that allows world models to learn which directions in their latent representation to emphasize while performing prediction. Building on existing approaches that divide latent representations into subspaces, they introduce a learnable orthogonal basis and assign adaptive weights to the resulting subspaces based on the current state and action. The authors also provide a theoretical analysis of how the learned basis interacts with these weights. They evaluate the approach on four visual-control tasks and show improvements over existing world-model baselines.

My first impression after reading this paper was its similarity to Gated SAEs (NeurIPS’24) which separates determining which directions to use from the magnitude of the feature contributions. However, unlike an SAE, this work doesn’t switch off features or subspaces. The model learns how strongly different latent subspaces should be amplified before prediction, rather than which features should survive a sparse reconstruction.

Critical strengths

I appreciate that the authors go beyond reporting improvements in control performance and use targeted ablations and interventions to investigate where these improvements come from. In particular, the Uniform Gain and Frozen Haar ablations investigate the roles of adaptive weighting and learning the orthogonal basis. The additional experiments in Section 3.1 examine how the learned directions interact with the adaptive weights. In Section 3.2 and Appendix B.2, the authors use leave-one-out interventions and Shapley attribution to quantify how much each subspace contributes to reducing prediction error, including when combined with other subspaces.

Furthermore, I find the presentation and the flow of the paper clear, in contrast to many publications & submissions recently.

Critical weaknesses

The main weakness I see is that the benefit of learning an explicit orthogonal basis over existing feature-modulation approaches (e.g. FiLM) is not clear. While the frozen Haar ablation (Table 1 in Appendix) shows that learning the orthogonal basis improves performance compared to keeping it fixed, both variants still rely on the proposed subspace decomposition. Since the vision encoder of the world model is trained end-to-end in both cases, and can adapt its latent representation, I wonder whether the improvements come from learning an orthogonal basis or more generally from adaptive feature modulation.

Questions for authors

Q.1: Have you considered comparing your learned orthogonal perspective against a dictionary learning method with action conditioned feature modulation? E.g. sparse autoencoders together with control vectors can enable steering action predictions. The specific benefit of such a comparison for this work would be that it would help clarify whether the observed gains are due to the orthogonal subspace structure or learning directions for modulation.

Q.2: I wonder whether the learned subspaces also have an interpretable influence on the predicted dynamics. For example, have you investigated how changing individual subspace gains affects predicted (physical) outcomes, and whether these changes follow a systematic relationship (ideally linear)?

Q.3: LeWorldModel has approximately 15 million parameters. So, the training cost shouldn’t be huge, in general. At the same time, the paper reports variability across six CEM planner seeds and not independently trained models. I think it would be interesting to see the robustness of the approach and inspect the differences in the learned bases.

Q.4: In Figure 3, you report a success rate of 81% for LeWorldModel on PushT, whereas the original paper reports 96.0%. Could you clarify where this difference comes from?

3 comments
AuthorsOpenReview ↗

Clarification on the Source of Improvement and Robustness of the Learned Basis

We thank the reviewer for the positive assessment and the thoughtful questions.

W1: On learned orthogonal basis vs. general feature modulation

The improvement comes from learning the orthogonal basis, not from adaptive modulation alone. We trained three controls under the same end-to-end setup, each with a fully trainable encoder and predictor, and each removing one ingredient: Q=IQ=I (same gating, fixed coordinate groups), FiLM (capacity-matched general modulation, parameters within 0.38% of ours), and Non-Orthogonal (learned directions without the orthogonality constraint).

MethodPushTCubeReacherTwoRoomMacro
Q=IQ=I92.6771.3382.0091.3384.33
FiLM92.6769.6782.3393.6784.58
Non-Orthogonal90.0072.3381.3393.3384.25
Ours95.6773.3385.3394.6787.25

Ours is highest on all four tasks, by 2.67--3.00 pp in macro success rate. The encoder adapts freely in every row, yet the three controls land within 0.33 pp of each other and all remain below Ours. Following the original LeWorldModel paper (Maes et al., 2026, Table 5), which evaluates robustness by training PushT across three independent seeds, we also did so. Ours achieves an average success rate of 94.34%, compared with 91.78% for FiLM and 91.33% for (Q=IQ=I).

Q1: On comparison with dictionary learning and action-conditioned modulation

We thank the reviewer for this suggestion. The Non-Orthogonal row above is designed to answer exactly the question it raises: it learns modulation directions with the same action-conditioned gains and removes only the orthogonal structure. It falls 3.00 pp behind Ours, so learning directions alone does not account for the gain. We used this control instead of a full SAE because an SAE adds sparsity and reconstruction objectives, which would change several factors at once and obscure the source of any difference.

Q2: On the interpretable influence of subspace gains on predicted dynamics

The influence is systematic and linear. We froze trained PushT models, varied one subspace gain at a time, and read out predicted object motion with a linear probe (R2R^2 0.76--0.80 for Δx\Delta x, 0.78--0.83 for Δy\Delta y, 0.53--0.62 for Δθ\Delta\theta across three training runs). Within a fixed state--action context, predicted displacement and rotation change linearly with each gain (R2≥0.9999R^2 \geq 0.9999), with median magnitudes of 0.13--0.48 px and 0.13--0.35$^{\circ}$.

Q3: On robustness across independently trained models

The findings are reproduced across training seeds. We trained three independent PushT models and repeated the leave-one-out (LOO) analyses of each subspace.

Training seedsuccess rateGain CVPositive LOO
307295.673.39%8/8
322792.674.02%8/8
340794.673.94%8/8

Every run learns differentiated gains and positive subspace contributions (mean LOO error increase 4.74--5.06%). On the learned bases: after aligning latent spaces across runs, subspace overlap is 0.137--0.139 (random reference 0.134). This is expected, because SIGReg makes the latent isotropic and so leaves no canonical axes to recover. Training reproduces the functional organisation, not one particular orientation.

Q4: On the LeWorldModel PushT result (81.67% vs. 96.0%)

Although the original paper reports 96.0% using three training seeds on one fixed set of 50 trajectories, subsequent works, including ours, obtain lower numbers under multi-seed evaluation: Sub-JEPA (Zhao et al., 2026, Table 1) reports 84.67 ±\pm 6.53% for LeWorldModel, QQWorld (Yu et al., 2026, Table 1) lists the same figure, and we obtain 81.67% over six CEM planner seeds with 50 episodes each. Our result is therefore consistent with independent reproductions. All methods in Figure 3 share the same protocol and planner seeds, so the comparison between them is unaffected. We will state the protocol difference in the revision.

[1] Zhao et al. (2026). Sub-JEPA: Subspace Gaussian Regularization for Stable End-to-End World Models. arXiv:2605.09241.

[2] Yu et al. (2026). QQWorld: Quantile-Quantile Matching for World Model Regularization. arXiv:2607.28415.

[3] Maes, L., Le Lidec, Q., Scieur, D., LeCun, Y., & Balestriero, R. (2026). LeWorldModel: Stable End-to-End Joint-Embedding Predictive Architecture from Pixels. arXiv:2603.19312.

[4] Perez et al. (2018). FiLM: Visual Reasoning with a General Conditioning Layer. AAAI 2018. arXiv:1709.07871.

Thank you for the detailed response.

W1. The Q=I, FiLM, and Non-Orthogonal baselines are helpful, though two points remain unclear. Your three PushT runs span 3.00 pp, as much as your largest per-task margin over the best of the baselines. Could you report the per-run PushT values for FiLM and Q=I? Also, since removing the gate budget drops your PushT score to 91.56% (reply to soBv below), can you please clarify whether Q=I and Non-Orthogonal use the gate budget?

Q1. A full SAE experiment is not needed, and Non-Orthogonal answers my question. Please still discuss Gated SAEs in the paper. Both approaches separate where to act from how strongly, and your method also uses a budget to keep its gates selective.

Q2. Changing one subspace gain barely moves the predicted object (0.13-0.48 px & 0.13-0.35 degrees). Since PushT gains stay between 1 and ~1.2 (Appendix A.2), a smooth predictor should look close to linear over such a small step at a fixed state. Thus, R2 >= 0.9999 is not surprising locally. I asked about linearity; whether a subspace controls the same physical property across different states.

Q3. The three PushT runs are helpful. Alignment should already remove the rotation freedom (from SIGReg isotropy). So overlap at the random level (0.137 to 0.139 vs 0.134) suggests that the learned subspaces differ between runs. Please state this in the paper and report the alignment quality.

Q4. Thank you for the explanation. This gap should have been flagged in the paper. Please note in Figure 3 the published score (96.0) next to your local one (81.67). Or add a footnote or caption, especially because it copies PLDM from Maes et al. (2026) while reporting 81.67% for LeWorldModel beside it.

Scope. All experiments use one backbone (LeWorldModel with SIGReg), and the detailed analyses use PushT only (as you state in conclusion). On Cube, the only 3D task, your method is 1.00 pp above Non-Orthogonal and below Sub-JEPA. Please state these limits in the paper.

Subspace count (reply to Reviewer DTs7). Your two common-K results (85.92 & 86.00) match K=8 and K=16 in Table 2. The other four rows give 84.17 to 84.83, close to FiLM (84.58). Please narrow your claims on K insensitivity.

We thank the reviewer for the constructive follow-up.

W1: Per-seed results and gate budget

PushT results across the same three training seeds:

SeedOursFiLMQ=IQ=I
307295.6792.6792.67
322792.6791.3391.00
340794.6791.3390.33
Mean94.3491.7891.33

Ours outperforms both controls at every matched seed, by 2.56 pp over FiLM and 3.00 pp over Q=IQ=I on average. Both Q=IQ=I and Non-Orthogonal retain the same gate budget (ρ=0.3\rho=0.3, λbudget=0.01\lambda_{\mathrm{budget}}=0.01) as Ours. All other training settings are matched.

Q1: Gated SAEs

We will discuss Gated SAEs in Related Work. Both approaches separate where to act from how strongly, but Gated SAEs learn sparse dictionary features, whereas ours learns orthogonal modulation subspaces and state–action-conditioned gains for latent prediction.

Q2: Cross-state physical interpretation

We agree that local linearity does not establish consistent physical roles across states. We do not claim such consistency. Section 3.2 evaluates the marginal predictive utility of subspace corrections in combination, while Section 3.3 measures physical-dynamics accessibility from subspace combinations, not individual subspace semantics. We will clarify that the gain-response results establish local sensitivity only.

Q3: Cross-seed consistency

Orthogonal Procrustes alignment achieves held-out R2=0.879R^2=0.879–0.8970.897, indicating that the latent representations are largely aligned. However, matched subspace overlap remains only 0.137–0.139, near the random reference (0.134). This weak correspondence is unsurprising, as each run starts from a different random basis and no training loss enforces cross-seed subspace consistency. Nevertheless, differentiated gains and 8/8 positive LOO contributions are reproduced across all three seeds. We will report the alignment quality and clarify this distinction in the paper.

Q4: LeWorldModel PushT score

We will clarify in the Figure 3 caption that LeWorldModel reports 96.0% in Maes et al. (2026), compared with 81.67% in our local multi-seed evaluation. PLDM is taken from Maes et al., while all locally evaluated methods share the same protocol and planner seeds.

Scope

We will explicitly state that all experiments use LeWorldModel with SIGReg, and the detailed analyses focus on PushT. On Cube, Ours exceeds Non-Orthogonal by only 1.00 pp and remains below Sub-JEPA.

Subspace count

We agree that KK insensitivity was overstated. Common K=8K=8 and K=16K=16 achieve 85.92% and 86.00% macro SR, while the other four settings achieve 84.17%–84.83%, close to FiLM (84.58%). We will narrow the claim to the stronger performance of K=8K=8 and K=16K=16 among the tested common-KK settings.

Reviewer_DTs7OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces orthogonal subspace modulation for latent world models. It jointly learns an orthonormal basis and state–action-dependent gains to modulate latent features before prediction. A fixed-gain analysis characterizes how gain differences make the basis consequential. Experiments report improved average control success across four tasks, supported by basis interventions and physical-dynamics probes on PushT.

Critical strengths
  • The method provides a clear, structured way to combine learned latent directions with adaptive modulation. The fixed-gain analysis precisely states when the basis affects the modulation operator, and the uniform-gain and frozen-basis controls help connect this observation to model behavior.
  • The empirical analysis goes beyond reporting control scores. The learned-basis model outperforms the frozen-Haar variant on average, while physical probes use episode-disjoint evaluation and aggregate across subspace combinations rather than selecting the best subset. These choices strengthen the evidence for useful representation changes.
Critical weaknesses
  • The advantage over simpler conditional modulation remains unclear. Uniform Gain and Frozen Haar are useful controls, but the paper lacks coordinate-aligned grouped gating or a comparable FiLM-style baseline. Since the encoder and predictor are also learned, the results do not yet establish why explicitly learning an orthogonal basis is preferable to these simpler alternatives. The fixed-gain theorem characterizes sensitivity, but does not establish a learning advantage.
Questions for authors
  • How does the method compare with Q=IQ=I grouped gating and a capacity-matched FiLM-style modulation baseline under the same training setup?
  • Were the task-specific subspace counts selected on separate validation episodes? How do the main comparisons change with one common subspace count across tasks?
3 comments
AuthorsOpenReview ↗

Clarification on Additional Baselines and the Choice of Subspace Count

We thank the reviewer for the positive assessment and constructive suggestions.

W1.1 & Q1: On additional baselines (Q=IQ=I and FiLM)

We trained both under the same setup with a fully trainable encoder and predictor. Q=IQ=I keeps our grouped gating on fixed coordinate groups; FiLM is capacity-matched (modulation parameters within 0.38% of ours).

MethodPushTCubeReacherTwoRoomMacro
Q=IQ=I92.6771.3382.0091.3384.33
FiLM92.6769.6782.3393.6784.58
Ours95.6773.3385.3394.6787.25

Ours is highest on all four tasks (+2.92 pp over Q=IQ=I, +2.67 pp over FiLM). Following the original LeWorldModel paper (Maes et al., 2026, Table 5), which evaluates robustness by training PushT across three independent seeds, we also did so. Ours achieves an average success rate of 94.34%, compared with 91.78% for FiLM and 91.33% for (Q=IQ=I).

W1.2: On why learning the basis helps when the encoder is also learned

SIGReg encourages an isotropic Gaussian latent distribution without preferred axes (Maes et al., 2026). While the encoder can adapt to fixed coordinate groups, a learned QQ provides a direct optimisation pathway for modulation directions. As shown by the chain-rule analysis in Appendix A.5, predictive gradients jointly update QQ and the encoder. The Q=IQ=I baseline confirms the practical benefit: despite a fully trainable encoder and identical gating, it remains 2.92 pp behind learned QQ.

W1.3: On the role of the theorem

Theorem 1 identifies when the perspective matters, namely under differentiated gains. The experiments then show that training reaches this regime (gain CV 3.39% vs. 0.76% for Frozen Haar) and benefits from it. We will make this division of roles explicit.

Q2: On the choice of subspace count

KK was taken from the sweep in Table 2 rather than a separate validation split. The result does not depend on this choice: with one common KK for all tasks, macro success rate is 85.92% (K=8K=8) and 86.00% (K=16K=16). Both common-(K) results remain above the reported macro success rate of FiLM (84.58%) and Sub-JEPA (84.08%). We will add the common-KK results to the paper.

[1] Perez et al. (2018). FiLM: Visual Reasoning with a General Conditioning Layer. AAAI 2018. arXiv:1709.07871.

[2] Maes, L., Le Lidec, Q., Scieur, D., LeCun, Y., & Balestriero, R. (2026). LeWorldModel: Stable End-to-End Joint-Embedding Predictive Architecture from Pixels. arXiv:2603.19312.

Follow-up on Additional Baselines and Subspace Count

We thank the reviewer again for the constructive feedback. We would like to briefly summarize how we addressed the main concerns:

  • Additional baselines: Ours achieves 87.25% macro SR, compared with 84.33% for Q=IQ=I and 84.58% for capacity-matched FiLM, under the same end-to-end training setup. The learned basis also outperforms both controls across all three PushT training seeds.
  • Subspace count: Common K=8K=8 and K=16K=16 achieve 85.92% and 86.00% macro SR. We also acknowledge that other choices of KK perform closer to FiLM and will narrow our claims accordingly.
  • Theory: Theorem 1 characterizes Gain–Basis Coupling, while the controlled experiments provide empirical evidence for the benefit of learning the basis.

We hope these results address your main concerns and would greatly appreciate any further feedback.

Thank you for adding the Q=IQ=I and capacity-matched FiLM baselines. These comparisons address my main concern about whether learning the basis helps beyond simpler modulation. The smaller gains with a shared subspace count are also informative. Please make clear in the revision that the task-specific counts were selected from the reported sweep. I’ll keep my score of 3.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Rebuttal

Dear Reviewers and Area Chair,

We sincerely thank all reviewers for their time and thoughtful engagement during the discussion. Your feedback has substantially strengthened the paper. Below we briefly summarize our responses.

1. Source of the improvement (DTs7, 8jWL, soBv). We trained three new controls under the same end-to-end setup: Q=IQ=I (same gating on fixed coordinate groups), capacity-matched FiLM, and Non-Orthogonal (learned directions without the orthogonality constraint). They reach 84.25–84.58% macro success rate, compared with 87.25% for ours, which is highest on all four tasks. Across three independently trained PushT models, ours (94.34%) also outperforms FiLM (91.78%) and Q=IQ=I (91.33%) at every matched seed.

2. Robustness and mechanism (8jWL, soBv). Across all three training seeds, the gains stay differentiated and all 8 subspaces have positive leave-one-out contributions. Replacing the basis costs 15.34 pp for the learned basis but only 3.33 pp for Frozen Haar, which supports Gain–Basis Coupling. We also clarified what a positive LOO means (the marginal predictive contribution of each subspace correction), why we use the gate budget, and that α\alpha is learned.

3. Evaluation details (DTs7, 8jWL). We reported results with one common subspace count across tasks (85.92% for K=8K=8, 86.00% for K=16K=16). We also explained the LeWorldModel PushT gap (81.67% vs. 96.0%): it comes from a different evaluation protocol, and our number is consistent with independent reproductions.

Planned revisions. We will:

  • add the new baselines and multi-seed results;
  • state that the task-specific KK was selected from the Table 2 sweep, and narrow the claim on KK insensitivity;
  • discuss Gated SAEs;
  • report cross-seed alignment quality;
  • note the published LeWorldModel score in Figure 3;
  • clarify that the gain-response results show local sensitivity only;
  • state the scope limits: a single LeWorldModel/SIGReg backbone, detailed analyses on PushT only, and a smaller margin on Cube.

We thank Reviewer DTs7 for confirming that the new baselines address the main concern. We would greatly appreciate it if Reviewers 8jWL and soBv could let us know whether our latest responses address your remaining concerns. We are happy to clarify anything further before the discussion period ends.

Best regards,

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.