acceptodds
Under review as a conference paper at ICLR 2027

What Must Quantization Preserve in Latent World Models?

Abstract

What must quantization preserve in latent world models to retain their planning ability? Existing work has primarily characterized sensitivity to precision configuration. To our knowledge, this is the first study to investigate latent world-model quantization from the perspective of representation preservation and calibration-target design. We analyze how quantization-induced representation errors propagate to goal-distance estimates and candidate rankings, and argue that calibration should preserve reusable latent representations rather than merely match scalar planning costs. We derive a bound relating terminal relative-representation error to the second moments of local encoder and predictor reconstruction errors, providing a theoretical basis for Module-MSE. Across LeWM, DINO-WM, and PLDM on four control tasks under symmetric per-channel W4 quantization, Module-MSE reaches 79.33% average success compared with 79.83% for full precision, while achieving the strongest overall planning-decision fidelity. It also maintains strong performance across the tested learning rates and calibration-set sizes. These results show that severe degradation under naive W4 quantization is not inevitable, establish representation preservation as a practical calibration principle, and support Module-MSE as a default choice for latent world-model quantization.

Primary area:
applications to robotics, autonomy, planning
Keywords:
latent world models, post-training quantization, model-based planning, model compression

Reviews

Among:
Shape:
1234this paper, Top 32%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.50 over 4 reviews, above 68% of the others.
Reviewer_EJu2OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
3(+1)
Read review
Paper summary

This paper investigates what information should be preserved during the quantization of latent world models, comparing planning-cost reconstruction, terminal relative-representation reconstruction, and module-level output reconstruction for the encoder and predictor. The authors analyze how local quantization errors propagate to terminal representations and impact candidate action rankings, providing motivation for the Module-MSE calibration objective. Experiments across three latent world models on four control tasks demonstrate that, under symmetric per-channel W4 weight quantization, Module-MSE achieves an average success rate of 79.33%, closely matching the full-precision baseline of 79.83%, while exhibiting favorable performance in planning-decision fidelity and calibration sensitivity evaluations.

Critical strengths
  1. Beyond standard task success rates, the paper evaluates candidate inversion rate, elite set overlap, and selection regret. These metrics provide deeper insight by revealing that even when overall reconstruction error is modest, relative errors among candidates can alter CEM action selection. The authors also distinguish between preserving full-precision model decisions and improving empirical task success rates.

  2. Evaluating across three world models and four control tasks ensures the findings are not confined to a single architecture. Comparing four learnable-rounding objectives under identical quantization parameterizations and optimization steps allows for a controlled study of the calibration objectives. Supplementary analyses on learning rate sensitivity, calibration set size, and adjacent-path supervision provide practical empirical support for module-level reconstruction.

Critical weaknesses
  1. Module-MSE applies reconstruction loss to encoder and predictor outputs within an existing learned rounding framework. While the paper discusses prior layer-wise and block-wise reconstruction approaches, it does not sufficiently isolate and demonstrate what concrete benefits arise specifically from coarsening the reconstruction granularity to the entire encoder or predictor.

    The analysis motivates module-level reconstruction, but does not by itself establish a comparative advantage over alternative calibration objectives. The error bound depends on propagation gains and may be loose, as acknowledged in Appendix A.2. The counterexamples illustrate that matching a scalar terminal cost need not preserve the terminal relative representation, and that matching the terminal relative representation need not preserve intermediate states. However, they do not establish that these ambiguities necessarily cause substantial performance deficits under sufficiently diverse calibration goals and trajectories. Consequently, the paper currently reads more like a useful empirical benchmarking study of calibration targets, and its methodological novelty as well as theoretical explanatory power require clearer delineation.

  2. Tables 2–3 do not report confidence intervals, error bars, or variance across independent calibration seeds, making it difficult to assess whether the marginal performance advantage of Module-MSE is statistically significant. Near-full-precision average performance does not imply uniform preservation across settings. For example, PLDM’s success rate on PushT decreases from 48.0% to 39.0%, although Module-MSE remains tied with GPTQ as the best quantized method in this setting.

    Additionally, the experiments rely largely on a fixed W4 configuration and fixed prediction horizons. Because multi-step error propagation serves as a central motivation for the proposed approach, the lack of evaluations across varying prediction/planning horizons leaves the applicable scope of module-level reconstruction unclear.

Questions for authors
  1. What is the statistical uncertainty surrounding the reported gains of Module-MSE over Module-L1 and GPTQ? How many independent calibration seeds were executed per configuration? Could you provide paired confidence intervals on success rate differences using aligned test episodes across methods?

  2. Operationally, what are the exact distinctions between Module-MSE and standard reconstruction calibration in the utilized AutoRound implementation? Under a comparable calibration budget, could you provide an ablation directly comparing layer-wise, block-wise, and whole encoder/predictor-level reconstruction to isolate the impact of module-level granularity itself?

  3. Does the relative advantage of Module-MSE vary as a function of the planning/prediction horizon? Can empirically measured module errors account for terminal representation errors or ranking flip rates under different calibration targets? Demonstrating this link would substantially strengthen the connection between the theoretical analysis and empirical claims.

9 comments
AuthorsOpenReview ↗

Rebuttal to Reviewer EJu2 Part 1

Part 1

Paper: What Must Quantization Preserve in Latent World Models?

Dear Reviewer EJu2, thank you for your careful review and for raising questions about reconstruction granularity, statistical uncertainty, and performance across prediction horizons. Below, we respond to each weakness and question in turn.

Weakness 1

Reviewer comment

Module-MSE applies reconstruction loss to encoder and predictor outputs within an existing learned rounding framework. While the paper discusses prior layer-wise and block-wise reconstruction approaches, it does not sufficiently isolate and demonstrate what concrete benefits arise specifically from coarsening the reconstruction granularity to the entire encoder or predictor.

The analysis motivates module-level reconstruction, but does not by itself establish a comparative advantage over alternative calibration objectives. The error bound depends on propagation gains and may be loose, as acknowledged in Appendix A.2. The counterexamples illustrate that matching a scalar terminal cost need not preserve the terminal relative representation, and that matching the terminal relative representation need not preserve intermediate states. However, they do not establish that these ambiguities necessarily cause substantial performance deficits under sufficiently diverse calibration goals and trajectories. Consequently, the paper currently reads more like a useful empirical benchmarking study of calibration targets, and its methodological novelty as well as theoretical explanatory power require clearer delineation.

Response

Thank you for examining the reconstruction granularity, the explanatory power of the analysis, and the role of multi-goal supervision.

Our contribution starts from the computational structure of world models and connects the choice of calibration target to multi-step error propagation and planning decisions. It consists of two related analyses.

First, representation reuse determines what quantization should preserve. The encoder and predictor outputs are reused in subsequent prediction steps and goal comparisons. We therefore propose preserving the full representations at these interfaces. We decompose multi-step error into start and goal encoding errors, newly introduced prediction error at each step, and propagation of earlier errors. When local propagation gains are bounded, the second moments of the module reconstruction errors control an upper bound on the terminal relative-representation error. This motivates Module-MSE. The contribution is the choice of what to calibrate and the analysis of its downstream effects, not a new calibration algorithm.

Second, the interaction between score perturbations and candidate margins explains changes in planning. A score error of the same magnitude can have different effects depending on the candidate margin. We characterize when the difference in score perturbations between two candidates exceeds their FP margin, reverses their ranking, and changes the CEM update through elite selection. This connects representation error to planning decisions and provides a common explanation for differences in RTN sensitivity and recovery after calibration.

The following experiments test these two analyses by measuring module error, its accumulation over prediction steps, and candidate decisions, rather than relying only on final task success.

AuthorsOpenReview ↗

Rebuttal to Reviewer EJu2 Part 2

Part 2

Weakness 1 (continued)

Experiment 1. Does reducing module error improve terminal representations and candidate rankings? The bound LRel≤CELE,2+CPLP,2\mathcal L_{\mathrm{Rel}}\le C_E\mathcal L_{E,2}+C_P\mathcal L_{P,2} relates the terminal relative-representation error to the second moments of module residuals. We compare the errors of quantized models and then hold the dynamics fixed to test the roles of residual magnitude and second moment.

(a) Quantized models. We compare six DINO-WM quantization methods using the same start states, candidates, and prediction horizon.

Table 1. DINO-WM: one-step prediction error, terminal representation error, and candidate inversion rate.

MethodOne-step MSE with matched inputs ↓Step-5 free-rollout MSE ↓Terminal relative-representation MSE ↓Candidate inversion rate ↓
RTN0.021550.47534.12645.53%
GPTQ0.0036150.10480.312719.39%
Module-MSE0.002255650.03057190.11475210.66%
Module-L10.00275450.04031210.148411.62%
Relative-MSE0.052950.14510.198613.33%
Cost-MSE0.065650.69530.923832.55%

We use 64 shared start states, 64 common candidates per state, and five prediction steps. One-step MSE with matched inputs compares the FP and quantized predictor outputs on the fifth call, given the same FP history and action. Step-5 free-rollout MSE compares the two models’ latents after their own five-step rollouts. Terminal relative-representation MSE compares the vectors obtained by subtracting each model’s goal latent from its terminal latent. All MSEs are averaged over output components.

Module-MSE has the lowest one-step MSE with matched inputs, step-5 free-rollout MSE, terminal relative-representation MSE, and candidate inversion rate.

(b) Controlled replay of one-step prediction residuals. We hold the FP predictor, initial representation, goal, and candidate actions fixed and change only the prediction residuals injected at each step. On DINO-WM, halving the one-step residuals from Module-MSE reduces terminal prediction MSE by 68.39% and the inversion rate from 9.25% to 5.17%. We also match the residuals from Module-MSE and Module-L1 to the same mean absolute error (MAE) to compare their second moments and how their errors propagate. Table 2 also reports native LeWM W3 rollouts for comparison.

Table 2. Error by prediction step: native rollout and matched-MAE replay with fixed dynamics.

Prediction stepLeWM W3 native rollout: MSE / L1DINO-WM W4 matched MAE: MSE / L1
10.0177986 / 0.01887940.00274727 / 0.00282599
30.0358500 / 0.03788880.0120057 / 0.0132801
50.0644336 / 0.06827050.0249888 / 0.0270521

Each entry is the prediction-latent MSE relative to FP, normalized by output dimension. MSE/L1 identifies the calibration objective or residual source. LeWM uses native rollouts; DINO-WM uses matched-MAE replay. Both use 64 shared candidates.

On LeWM W3, Module-MSE has 6.73% lower mean injected MSE than Module-L1 and lower accumulated error over successive prediction steps. On DINO-WM at matched MAE, the Module-MSE residuals have a 2.72% lower second moment and yield 7.63% lower terminal MSE (95% CI: [1.13%, 13.25%]). The quantized-model comparison and the fixed-predictor replay together support the link between lower second-moment error, better multi-step representations, and better ranking fidelity.

AuthorsOpenReview ↗

Rebuttal to Reviewer EJu2 Part 3

Part 3

Weakness 1 (continued)

Experiment 2. Can score perturbations and candidate margins explain RTN sensitivity and recovery after calibration? For each model, we use 64 start states and 300 shared candidates per state. We compare the score perturbations and decisions of RTN and Module-MSE at the same FP margins.

Table 3. RTN sensitivity and recovery after Module-MSE calibration.

ModelMethodTop-two margin / IQRScore perturbation RMS / IQR ↓Inversion rate ↓Closed-loop success ↑
LeWMRTN0.5549310.1855055.94%79.50%
LeWMModule-MSE0.5549310.1116232.82%84.50%
DINO-WMRTN0.2768110.87503144.95%7.00%
DINO-WMModule-MSE0.2768110.30624211.20%76.00%

RMS is the root mean square of the score perturbation after subtracting its common offset, and IQR is the interquartile range of the FP costs. We normalize the margins and RMS by the IQR before averaging across start states. The horizon is five prediction steps and the elite set has 30 candidates. Closed-loop success rates use the same checkpoints on 200 aligned episodes.

DINO-WM is more sensitive to RTN because its score perturbations are larger relative to its candidate margins. Its normalized perturbation is about 4.7 times LeWM’s, while its top-two margin is about half as large, making rankings easier to reverse. The similar elite-boundary margins (0.01770 for DINO-WM and 0.02004 for LeWM) further highlight the difference in perturbation magnitude. Module-MSE reduces DINO-WM’s normalized score perturbation by 65.0% and its inversion rate from 44.95% to 11.20%. Closed-loop success rises from 7.0% to 76.0%, close to FP’s 76.5%. Both RTN sensitivity and recovery after calibration can thus be understood in terms of the size of the score perturbations relative to the candidate margins.

Calibration data and robustness. More data or a wider variety of goals may improve Cost-MSE, but it was less robust than Module-MSE in Section 5.3: its success-rate range was 17.5–28.5 percentage points, compared with 1.0–7.5 for Module-MSE. With just 13 trajectories, Module-MSE reaches 64.33% mean success across three models, compared with 34.67% for Cost-MSE using 128 trajectories. These results highlight the practical value of deriving the calibration objective from representation reuse and error propagation: Module-MSE combines sample efficiency and calibration robustness with planning fidelity.

Controlled granularity ablation. Layer-MSE, Block-MSE, and Module-MSE share the quantization parameterization, optimizer, calibration data, and FP input source. All use 384 updates per parameter and a learning rate of 1/768. Only the reconstruction target changes: an individual linear layer, a full Transformer residual block, or the output of the entire encoder or a full predictor call.

Table 4. Reconstruction granularity ablation. Each cell reports terminal relative-representation MSE ↓ / inversion rate ↓ / success rate ↑.

Reconstruction granularityLeWM–PushTDINO-WM–PushT
Layer-MSE0.02791 / 4.56% / 80.0%1.525 / 25.83% / 41.0%
Block-MSE0.02173 / 4.18% / 82.5%0.2600 / 12.75% / 74.0%
Module-MSE0.01999 / 3.95% / 83.0%0.1439 / 10.46% / 75.0%

All variants use the same test episodes. Supervising full module outputs reduces terminal representation errors and inversions and increases success on both models. With matched inputs, the DINO-WM predictor’s reconstruction MSE also decreases from 0.007314 for layer-level supervision and 0.004198 for block-level supervision to 0.002924 for module-level supervision. The benefit is thus visible directly at the predictor output.

Reviewer_exzEOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper studies how well post-training quantization preserves the properties of latent world models that are needed for planning. Motivated by the substantial performance drop of DINO-WM under 4-bit quantization, the authors propose changing the quantization calibration objective from matching the final output representation to matching the intermediate representation of each module (Module-MSE). It can provide much richer supervision and leads to significant improvement to quantized model performance for some of the cases tested.

Critical strengths

Using intermediate representation as a quantization optimization goal is intuitively reasonable, and the author provided detailed theoretical analysis to support this change. The performance improvement is significant for some of the models tested, and may have value in practical applications.

Critical weaknesses

One weakness is the novelty and source of the gains. Module-MSE is closely related to existing block-wise reconstruction methods such as GPTQ and AutoRound (the latter is used to implement the experiments), so it would help to clarify what is conceptually new beyond the choice of reconstruction granularity. The paper’s theoretical error analysis could help establish this, but the experiments focus mainly on final task success rates. Experiments that directly test the analysis, such as measuring error accumulation across modules or planning horizons under different calibration objectives, would strengthen the paper’s claims. Another weakness is generality of the gains. The large improvements are concentrated on DINO-WM, and in most other settings simple round-to-nearest (RTN) quantization performs comparably. This suggests Module-MSE may be needed only for certain architectures or hyperparameter choices. A discussion of when and why the method is needed would clarify its practical contribution.

Questions for authors

What specifically does Module-MSE preserve that GPTQ does not? Could the authors provide further analysis of where the performance gain comes from, for instance which modules or which properties of the latent representation benefit most? What practical guidance would the authors offer to practitioners choosing a post-training quantization method for world models? Should Module-MSE always be used, or are there identifiable conditions under which simpler methods such as RTN are sufficient?

9 comments
AuthorsOpenReview ↗

Rebuttal to Reviewer exzE Part 1

Part 1

Paper: What Must Quantization Preserve in Latent World Models?

Dear Reviewer exzE, thank you for your careful review and for raising questions about the novelty of the work, the source of the gains, and when the method is useful in practice. Below, we respond to each weakness and question in turn.

Weakness 1

Reviewer comment

One weakness is the novelty and source of the gains. Module-MSE is closely related to existing block-wise reconstruction methods such as GPTQ and AutoRound (the latter is used to implement the experiments), so it would help to clarify what is conceptually new beyond the choice of reconstruction granularity. The paper’s theoretical error analysis could help establish this, but the experiments focus mainly on final task success rates. Experiments that directly test the analysis, such as measuring error accumulation across modules or planning horizons under different calibration objectives, would strengthen the paper’s claims.

Response

Thank you for asking what is new beyond reconstruction granularity and how the error-propagation analysis can be tested directly.

Our contribution starts from the computational structure of world models and connects the choice of calibration target to multi-step error propagation and planning decisions. It consists of two related analyses.

First, representation reuse determines what quantization should preserve. The encoder and predictor outputs are reused in subsequent prediction steps and goal comparisons. We therefore propose preserving the full representations at these interfaces. We decompose multi-step error into start and goal encoding errors, newly introduced prediction error at each step, and propagation of earlier errors. When local propagation gains are bounded, the second moments of the module reconstruction errors control an upper bound on the terminal relative-representation error. This motivates Module-MSE. The contribution is the choice of what to calibrate and the analysis of its downstream effects, not a new calibration algorithm.

Second, the interaction between score perturbations and candidate margins explains changes in planning. A score error of the same magnitude can have different effects depending on the candidate margin. We characterize when the difference in score perturbations between two candidates exceeds their FP margin, reverses their ranking, and changes the CEM update through elite selection. This connects representation error to planning decisions and provides a common explanation for differences in RTN sensitivity and recovery after calibration.

The following experiments test these two analyses by measuring module error, its accumulation over prediction steps, and candidate decisions, rather than relying only on final task success.

AuthorsOpenReview ↗

Rebuttal to Reviewer exzE Part 2

Part 2

Weakness 1 (continued)

Experiment 1. Does reducing module error improve terminal representations and candidate rankings? The bound LRel≤CELE,2+CPLP,2\mathcal L_{\mathrm{Rel}}\le C_E\mathcal L_{E,2}+C_P\mathcal L_{P,2} relates the terminal relative-representation error to the second moments of module residuals. We compare the errors of quantized models and then hold the dynamics fixed to test the roles of residual magnitude and second moment.

(a) Quantized models. We compare six DINO-WM quantization methods using the same start states, candidates, and prediction horizon.

Table 1. DINO-WM: one-step prediction error, terminal representation error, and candidate inversion rate.

MethodOne-step MSE with matched inputs ↓Step-5 free-rollout MSE ↓Terminal relative-representation MSE ↓Candidate inversion rate ↓
RTN0.021550.47534.12645.53%
GPTQ0.0036150.10480.312719.39%
Module-MSE0.002255650.03057190.11475210.66%
Module-L10.00275450.04031210.148411.62%
Relative-MSE0.052950.14510.198613.33%
Cost-MSE0.065650.69530.923832.55%

We use 64 shared start states, 64 common candidates per state, and five prediction steps. One-step MSE with matched inputs compares the FP and quantized predictor outputs on the fifth call, given the same FP history and action. Step-5 free-rollout MSE compares the two models’ latents after their own five-step rollouts. Terminal relative-representation MSE compares the vectors obtained by subtracting each model’s goal latent from its terminal latent. All MSEs are averaged over output components.

Module-MSE has the lowest one-step MSE with matched inputs, step-5 free-rollout MSE, terminal relative-representation MSE, and candidate inversion rate.

(b) Controlled replay of one-step prediction residuals. We hold the FP predictor, initial representation, goal, and candidate actions fixed and change only the prediction residuals injected at each step. On DINO-WM, halving the one-step residuals from Module-MSE reduces terminal prediction MSE by 68.39% and the inversion rate from 9.25% to 5.17%. We also match the residuals from Module-MSE and Module-L1 to the same mean absolute error (MAE) to compare their second moments and how their errors propagate. Table 2 also reports native LeWM W3 rollouts for comparison.

Table 2. Error by prediction step: native rollout and matched-MAE replay with fixed dynamics.

Prediction stepLeWM W3 native rollout: MSE / L1DINO-WM W4 matched MAE: MSE / L1
10.0177986 / 0.01887940.00274727 / 0.00282599
30.0358500 / 0.03788880.0120057 / 0.0132801
50.0644336 / 0.06827050.0249888 / 0.0270521

Each entry is the prediction-latent MSE relative to FP, normalized by output dimension. MSE/L1 identifies the calibration objective or residual source. LeWM uses native rollouts; DINO-WM uses matched-MAE replay. Both use 64 shared candidates.

On LeWM W3, Module-MSE has 6.73% lower mean injected MSE than Module-L1 and lower accumulated error over successive prediction steps. On DINO-WM at matched MAE, the Module-MSE residuals have a 2.72% lower second moment and yield 7.63% lower terminal MSE (95% CI: [1.13%, 13.25%]). The quantized-model comparison and the fixed-predictor replay together support the link between lower second-moment error, better multi-step representations, and better ranking fidelity.

AuthorsOpenReview ↗

Rebuttal to Reviewer exzE Part 3

Part 3

Weakness 1 (continued)

Experiment 2. Can score perturbations and candidate margins explain RTN sensitivity and recovery after calibration? For each model, we use 64 start states and 300 shared candidates per state. We compare the score perturbations and decisions of RTN and Module-MSE at the same FP margins.

Table 3. RTN sensitivity and recovery after Module-MSE calibration.

ModelMethodTop-two margin / IQRScore perturbation RMS / IQR ↓Inversion rate ↓Closed-loop success ↑
LeWMRTN0.5549310.1855055.94%79.50%
LeWMModule-MSE0.5549310.1116232.82%84.50%
DINO-WMRTN0.2768110.87503144.95%7.00%
DINO-WMModule-MSE0.2768110.30624211.20%76.00%

RMS is the root mean square of the score perturbation after subtracting its common offset, and IQR is the interquartile range of the FP costs. We normalize the margins and RMS by the IQR before averaging across start states. The horizon is five prediction steps and the elite set has 30 candidates. Closed-loop success rates use the same checkpoints on 200 aligned episodes.

DINO-WM is more sensitive to RTN because its score perturbations are larger relative to its candidate margins. Its normalized perturbation is about 4.7 times LeWM’s, while its top-two margin is about half as large, making rankings easier to reverse. The similar elite-boundary margins (0.01770 for DINO-WM and 0.02004 for LeWM) further highlight the difference in perturbation magnitude. Module-MSE reduces DINO-WM’s normalized score perturbation by 65.0% and its inversion rate from 44.95% to 11.20%. Closed-loop success rises from 7.0% to 76.0%, close to FP’s 76.5%. Both RTN sensitivity and recovery after calibration can thus be understood in terms of the size of the score perturbations relative to the candidate margins.

Reviewer_F45vOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper studies which calibration target to use when applying W4 post-training quantization to latent world models used for CEM planning. It shows that quantization changes planning through differential score errors that reorder candidates.

Critical strengths

The paper is well organized. It moves from an observation, to a mechanism, to design choices, to experiments.

Critical weaknesses
  1. The L1-vs-MSE analysis (Section 4.4) is not supported empirically. The two-point distribution in Eq. (4.11) shows only that L1 does not control the second moment in general.
  2. The setting is too easy to separate methods. Symmetric per-channel W4 weight-only quantization barely affects LeWM (82.63% to 81.75% with RTN, Table 1), and GPTQ already restores DINO-WM from 38.13% to 79.38%. The conclusions would be more informative under harder settings: W3 or W2, group-wise quantization, or activation quantization, which the conclusion lists as future work.
  3. Efficiency is not measured. The motivation is the cost of repeated rollouts, but no latency, memory, or throughput numbers are reported.
Questions for authors
  1. Could you evaluate more challenging quantization settings to better distinguish the methods?
  2. Could you report latency, memory usage, and throughput measurements to quantify the practical efficiency gains, particularly for repeated rollouts, whose computational cost motivates the work?
4 comments
AuthorsOpenReview ↗

Rebuttal to Reviewer F45v Part 1

Part 1

Paper: What Must Quantization Preserve in Latent World Models?

Dear Reviewer F45v, thank you for your careful review and for raising questions about the theoretical rationale, more challenging quantization settings, and practical deployment costs. Below, we respond to each weakness and question in turn.

Weakness 1

Reviewer comment

The L1-vs-MSE analysis (Section 4.4) is not supported empirically. The two-point distribution in Eq. (4.11) shows only that L1 does not control the second moment in general.

Response

Thank you for asking whether the theoretical distinction between L1 and MSE is observable in multi-step prediction.

Section 4.4 makes a specific claim: the multi-step error bound depends on the second moment of module residuals, which MSE directly targets; equal first-moment error does not imply equal second-moment error. We test this in two ways: by measuring error over successive steps in quantized models, and by matching the residuals from both methods to the same mean absolute error (MAE) before replaying them through a common FP predictor.

Protocol. In native rollouts, the FP and quantized models start from their own encoder outputs and predict freely for five steps along the same candidate actions. We also pass the same FP prediction history and action to both predictors to obtain the one-step residual ηt=PF(htF,at)−PQ(htF,at)\eta_t=P_F(h_t^F,a_t)-P_Q(h_t^F,a_t). This separates newly introduced error from previously accumulated error. In the matched-MAE replay, we rescale each method’s residuals at each start state and step to the same MAE without changing their directions. Starting from the same FP latent, we subtract the residuals from the outputs of the same FP predictor at each step and measure how the error accumulates.

The table reports three metrics:

  • Mean MAE: the mean absolute magnitude of the one-step residual components, using the rescaled residuals for matched-MAE replay.
  • Injected MSE: the mean square of those same residual components, the second-moment quantity in the bound. Both mean MAE and injected MSE are averaged over candidates, five prediction steps, and start states. Neither includes propagated error from earlier steps.
  • Step-5 MSE: the MSE between the latent after five rollout or replay steps and the corresponding latent from an unperturbed FP rollout, averaged over output components, candidates, and start states. It measures accumulated deviation from FP, not prediction error against a real future observation. Five steps correspond to 25 physical actions.

Table 1. Native rollout errors and matched-MAE replay across calibration methods.

SettingMethod / residual sourceMean MAEInjected MSE ↓Step-5 MSE ↓
LeWM W3, native rollout, 64 start statesModule-MSE0.08037170.01031520.0644336
LeWM W3, native rollout, 64 start statesModule-L10.08281030.01105990.0682705
DINO-WM W4, matched-MAE replay, 136 start statesModule-MSE0.03759590.002556240.0249888
DINO-WM W4, matched-MAE replay, 136 start statesModule-L10.03759590.002627670.0270521

On LeWM W3, Module-MSE reduces injected MSE by 6.73% relative to Module-L1 and also yields lower mean step-5 MSE. More directly, at the same MAE on DINO-WM, residuals from Module-MSE have a 2.72% lower second moment and yield 7.63% lower terminal MSE (95% CI: [1.13%, 13.25%]).

These results directly link the second moment of module residuals to multi-step error and support Module-MSE as a default calibration objective.

AuthorsOpenReview ↗

Rebuttal to Reviewer F45v Part 2

Part 2

Weakness 2

Reviewer comment

The setting is too easy to separate methods. Symmetric per-channel W4 weight-only quantization barely affects LeWM (82.63% to 81.75% with RTN, Table 1), and GPTQ already restores DINO-WM from 38.13% to 79.38%. The conclusions would be more informative under harder settings: W3 or W2, group-wise quantization, or activation quantization, which the conclusion lists as future work.

Response

Thank you for suggesting comparisons at lower weight precision and with activation quantization.

We compare all six quantization methods under W3 weight-only and W4A4 weight-and-activation quantization on LeWM–PushT. LeWM is relatively robust to W4, making this a direct test of whether the gains extend beyond architectures that are especially sensitive to W4.

Table 2. Six-method comparison on LeWM–PushT at W3 and W4A4.

MethodW3: S / O / I / RW4A4: S / O / I / R
RTN45.5% / 64.68% / 15.84% / 28.167842.5% / 58.92% / 21.36% / 35.7171
GPTQ76.0% / 84.65% / 7.12% / 5.6976154.0% / 59.83% / 20.50% / 27.4889
Module-MSE76.5% / 81.87% / 6.64% / 5.4513977.5% / 67.97% / 16.94% / 17.8037
Module-L173.5% / 82.38% / 8.06% / 9.3381869.0% / 66.65% / 16.91% / 19.1072
Relative-MSE69.0% / 78.17% / 10.18% / 13.387266.0% / 65.13% / 18.07% / 22.1066
Cost-MSE39.0% / 54.37% / 21.92% / 44.474863.0% / 62.42% / 19.25% / 25.7087

S, O, and I denote success rate, elite overlap, and inversion rate (all in percent); R denotes selected regret in FP cost units. Higher is better for S and O; lower is better for I and R. Each condition uses 200 closed-loop episodes and 300 fixed candidates at each of 200 states.

Module-MSE achieves the highest success rate (76.5% and 77.5%) and the lowest selected regret in both settings. It also has the lowest inversion rate at W3 and the highest elite overlap at W4A4. Its W4A4 inversion rate is close to Module-L1’s (16.94% vs. 16.91%). These results support its effectiveness under more demanding quantization.

Weakness 3

Reviewer comment

Efficiency is not measured. The motivation is the cost of repeated rollouts, but no latency, memory, or throughput numbers are reported.

Response

Thank you for asking about the practical gains from low-bit deployment.

We measure model storage, rollout latency, and full CEM time for LeWM–PushT and DINO-WM–PushT on an A100-SXM4-80GB. W4A16 uses packed Marlin, while W4A4 uses a fused CUTLASS INT4 Tensor Core kernel. The measurements include runtime dynamic activation quantization and packing.

Table 3. Low-bit execution compared with FP16.

Model / backendModel storage (MiB)Full CEM (s)FP16 / low-bit latency
LeWM H1 · W4A1634.41 → 10.480.5478 → 0.61850.886×
DINO-WM H1 · W4A16106.75 → 51.4516.5007 → 23.01960.717×
DINO-WM H3 · W4A16106.75 → 51.4521.3584 → 29.40430.726×
LeWM H1 · W4A434.41 → 9.310.5624 → 0.50921.104×
DINO-WM H1 · W4A4106.75 → 49.0616.4027 → 13.43951.220×
DINO-WM H3 · W4A4106.75 → 49.0621.4154 → 17.62381.215×

Each row uses its corresponding FP16 baseline. H is the length of the initial observation history, and T = 5. Times are the median of five measured runs after one warm-up.

Weight-only quantization primarily reduces weight storage and the associated GPU memory footprint. The Marlin W4A16 implementation did not yield an end-to-end speedup in this evaluation. Fused W4A4 reduced model storage by 72.9% for LeWM and 54.0% for DINO-WM while speeding up full CEM by 1.104–1.220×.

AuthorsOpenReview ↗

Rebuttal to Reviewer F45v Part 3

Part 3

Question 1

Reviewer comment

Could you evaluate more challenging quantization settings to better distinguish the methods?

Response

Thank you for asking whether more challenging settings can better separate the methods.

Table 2 reports all four metrics for six methods at W3 and W4A4. Module-MSE achieves the highest success rate at both precisions. At W3 it also has the lowest inversion rate and selected regret; at W4A4 it has the highest elite overlap and lowest selected regret.

At W3, its 76.5% success rate exceeds RTN’s 45.5% by 31.0 percentage points. At W4A4, its 77.5% success rate exceeds GPTQ, Module-L1, Relative-MSE, and Cost-MSE by 23.5, 8.5, 11.5, and 14.5 percentage points, respectively. The paired 95% confidence intervals for these differences are [15.5, 31.5], [1.0, 16.0], [4.0, 19.5], and [7.0, 22.0]. All methods use identical test episodes and fixed candidate sets. These results distinguish the calibration objectives more clearly and support Module-MSE under stricter quantization.

Question 2

Reviewer comment

Could you report latency, memory usage, and throughput measurements to quantify the practical efficiency gains, particularly for repeated rollouts, whose computational cost motivates the work?

Response

Thank you for highlighting the latency, throughput, and resource costs of repeated rollouts.

As described in Weakness 3, we measured efficiency with low-bit kernels. Fused W4A4 speeds up cold rollouts by 1.115–1.208×. End-to-end planning throughput increases from 1.778, 0.06097, and 0.04670 to 1.964, 0.07441, and 0.05674 queries/s for LeWM, DINO-WM H1, and DINO-WM H3, respectively. Model storage falls by 72.9% for LeWM and 54.0% for DINO-WM. Table 3 gives the full CEM latencies.

Weight-only quantization mainly reduces storage and the associated GPU memory footprint, whereas quantizing both weights and activations also improves end-to-end speed. The reported timings include runtime activation quantization and packing.

Closing Remarks

We have addressed each weakness and question raised in your review and will incorporate the additional analyses and results into the revised manuscript. Thank you again for your thoughtful feedback.

To our knowledge, this is the first study to link the effects of quantization on planning to calibration design in latent world models. We compare score perturbations with candidate margins to explain ranking flips and use multi-step error analysis to motivate Module-MSE reconstruction of encoder and predictor outputs.

In light of these analyses and results, we invite you to reconsider your assessment of the novelty and value of our work.

Best regards,
Authors of What Must Quantization Preserve in Latent World Models?

AuthorsOpenReview ↗

Follow-up on Our Rebuttal

Dear Reviewer F45v,

As the author–reviewer discussion period draws to a close, we wanted to ask whether you have had a chance to review our rebuttal and the revised manuscript. You asked whether the theoretical distinction between L1 and MSE is observable in multi-step prediction, how the methods compare under more demanding quantization, and what efficiency gains low-bit execution offers. We briefly follow up on these points below; please see our earlier replies for the full point-by-point rebuttal.

To address the first question, we matched the mean absolute error of residuals from Module-MSE and Module-L1 on DINO-WM and replayed them through the same full-precision predictor. The Module-MSE residuals had 2.72% lower one-step MSE and yielded 7.63% lower terminal prediction MSE. This illustrates why equal L1 error does not imply equal second moments or equal rollout error.

On LeWM–PushT under W4A4, Module-MSE achieved 77.5% success, 23.5 percentage points above GPTQ. Our deployment measurements also showed that fused W4A4 sped up full CEM planning by 1.104–1.220× relative to FP16, while Marlin W4A16 reduced model storage without accelerating planning.

If these results address your concerns, we would be grateful if you could reconsider your overall assessment. We would be happy to clarify any remaining questions. Thank you again for your careful review.

Best regards,
Authors of What Must Quantization Preserve in Latent World Models?

Reviewer_Tn4UOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The authors investigate latent dynamics model quantization based on the criteria of representation preservation and calibratino-target design. They first report closed-loop planning results for a couple of JEPA models across a few control tasks and observe that DINO-WM is most sensitive to W4 RTN quantization. Then the authors also argue that quantization for latent world models can affect ordering in the top-k action sequence selection in CEM planning, which changes planning success rate while not changing the final goal-conditioned cost very much. The authors go on to discuss three loss function for quantization supervision: 1) MSE on the cost, 2) MSE on the representations that produce the cost, and 3) MSE on encoder and predictor outputs. The authors gave toy examples for why 1) and 2) are not good ideas, and 3) is optimal with module-mse, i.e. using L2 loss to supervise differences between full-precision output and quantized output of encoders and predictors. Finally, the authors evaluated different quantization calibration targets, varied if we use intermediate prediction for quantization or not, and measured how sensitive is different models to quantizations across some hyper settings.

Critical strengths

strength:

  • the authors gave intuitive examples for why cost-mse and relative presentation-mse are not good ideas. I also like the derivation on the module-mse quantization error bound.
  • in table 3, it's true that module-mse is the best calibration target based on the eval criteria proposed in section 5.1
Critical weaknesses

weakness:

  • in table 2, sometimes quantized models are much better than full precision models in terms of closed-loop planning success rates. It's unclear whether the differences come from quantization or planning related issues. Also there is no mean +- std in the reported numbers, i.e. it's not clear what's the standard deviation here. My understanding is that in some of these planning tasks, variations across planning jobs can be somewhat large.
  • in general, i am not sure if we know how we should measure the effect of quantization on latent controlled dynamics model like JEPA. Planning is the only metric used in the paper to evaluate the goodness of quantization. But what about rollout error? would quantization leads to more significant compounding rollout errors? or would it help reducing rollout error due to discretization of the continuous latent dynamics? what about other linear probe etc? Just listing out a few other options here. I think these are really interesting directions to look into, and I would love to see more discussions on this.
Questions for authors

question

  • In table 1, are there any reasons why DINO-WM is very sensitive to W4 RTN? Is this because of 1) DINO-WM is using patch features or 2) the frozen DINOv2 encoder itself is sensitive to W4 RTN. I think these factors are entangled and it's worth knowing what are the actual cause of the instability.
  • in table 2, why is DINO-WM only bad when using RTN instead of all other quantization techniques like GPTQ etc.
  • it's unclear to me whether planning success rate is the best metric to decide whether a particular quantization method is good or bad. what about decoding, linear probe from representations etc?
6 comments
AuthorsOpenReview ↗

Rebuttal to Reviewer Tn4U Part 1

Part 1

Paper: What Must Quantization Preserve in Latent World Models?

Dear Reviewer Tn4U, thank you for your careful review and for raising questions about planning randomness, DINO-WM’s sensitivity to quantization, and representation quality. Below, we respond to each weakness and question in turn.

Weakness 1

Reviewer comment

in table 2, sometimes quantized models are much better than full precision models in terms of closed-loop planning success rates. It's unclear whether the differences come from quantization or planning related issues. Also there is no mean +- std in the reported numbers, i.e. it's not clear what's the standard deviation here. My understanding is that in some of these planning tasks, variations across planning jobs can be somewhat large.

Response

Thank you for raising the role of planning randomness and the observation that quantized models can outperform full precision (FP) in some settings.

In the main experiment, all methods used the same test episodes, trajectory-window start states and goals, and planning seed. The offline evaluation also used identical candidate actions. These controls minimize the effects of task sampling and planning randomness on the comparison. At the same time, an FP world model paired with a finite-budget cross-entropy method (CEM) search is not an optimal task solver. Quantization changes scores and can lead the search to choose more effective actions on some episodes. Higher success in those cases does not contradict the goal of preserving the pretrained model’s behavior: task success and fidelity to FP decisions are distinct, so we report both.

To test robustness to planning randomness, we kept the paper’s quantized checkpoints fixed and ran paired evaluations with three additional planning seeds. At each seed, all six methods used the same 200 test episodes. Table 1 reports the mean ± standard deviation across seeds. Module-MSE retained the highest mean success rate among the quantized methods on LeWM–PushT (84.17%) and DINO-WM–PushT (75.83%), improving over RTN by 8.00 and 68.33 percentage points, respectively. On DINO-WM, it also exceeded GPTQ by 17.67 percentage points (paired 95% CI: [11.83, 23.50]). Together, the aligned main experiment and the multi-seed evaluation support the effectiveness of Module-MSE and its robustness to planning randomness.

Table 1. Success rates across three planning seeds with fixed quantized checkpoints.

MethodLeWM–PushT success rateDINO-WM–PushT success rate
RTN76.17±1.26%7.50±0.87%
GPTQ83.67±2.75%58.17±1.04%
Module-MSE84.17±2.47%75.83±0.58%
Module-L182.17±0.76%73.50±1.00%
Relative-MSE81.33±1.61%74.00±0.87%
Cost-MSE77.33±1.76%21.67±1.89%
AuthorsOpenReview ↗

Rebuttal to Reviewer Tn4U Part 2

Part 2

Weakness 2

Reviewer comment

in general, i am not sure if we know how we should measure the effect of quantization on latent controlled dynamics model like JEPA. Planning is the only metric used in the paper to evaluate the goodness of quantization. But what about rollout error? would quantization leads to more significant compounding rollout errors? or would it help reducing rollout error due to discretization of the continuous latent dynamics? what about other linear probe etc? Just listing out a few other options here. I think these are really interesting directions to look into, and I would love to see more discussions on this.

Response

Thank you for suggesting ways to assess quantization beyond planning success. We examine rollout error accumulation, whether discretization improves prediction, and whether physical state remains linearly decodable.

1. Does quantization increase multi-step rollout error? On LeWM–PushT W3, we compare FP and six quantized models on the same 100 test trajectories. Each model rolls out the recorded actions for up to 25 physical actions without resetting to a ground-truth observation. At horizon tt, the target is the FP encoding of the observed future frame:

=\frac{1}{N d_z}\sum_{n=1}^{N} \left\|E_F(o_{n,t})-\hat z_{n,t}^{M}\right\|_2^2, \qquad N=100,\quad d_z=192.$$ Here $o_{n,t}$ is the observation after $t$ executed actions in trajectory $n$, and $\hat z_{n,t}^{M}$ is model $M$’s free-running prediction. FP is evaluated against the same target and therefore also has nonzero error. **Table 2. LeWM–PushT W3: latent rollout MSE against FP encodings of observed future frames.** | Method | 5 actions ↓ | 15 actions ↓ | 25 actions ↓ | | --- | ---: | ---: | ---: | | FP | 0.00954549 | 0.06379546 | 0.10837628 | | RTN | 0.07381467 | 0.19920659 | 0.43915779 | | GPTQ | **0.02236101** | 0.08272794 | 0.16784029 | | Module-MSE | 0.02739678 | **0.07712762** | **0.14421201** | | Module-L1 | 0.02746727 | 0.07798243 | 0.14603814 | | Relative-MSE | 0.06541715 | 0.11997690 | 0.21709674 | | Cost-MSE | 0.31573978 | 0.51563745 | 0.81194208 | Each prediction step spans five physical actions. Each model is initialized by its own encoder. Bold marks the lowest mean among quantized methods. Mean rollout error increases with horizon, and quantization adds further error. The excess MSE over FP grows from 0.06427 at 5 actions to 0.33078 at 25 actions for RTN, compared with 0.01785 to 0.03584 for Module-MSE. Module-MSE has the lowest MSE among quantized methods at 15 and 25 actions. At 25 actions, its MSE is 0.14421, 67.16% below RTN. **2. Can discretization improve predictive accuracy?** Every quantized method had higher mean prediction error than FP at every evaluated horizon. In this experiment, Module-MSE reduced the extra error caused by quantization but did not improve on FP predictive accuracy. **3. Do linear probes confirm that state information is retained?** On LeWM–PushT W4, we train an identical linear probe separately for FP and each quantized model to read out object position and orientation. The world models are frozen. Each probe is trained on its model’s latents from observed frames and the corresponding state labels, using the same disjoint sets of 512 training, 128 validation, and 1,000 test trajectories. We then freeze the probes and evaluate both observed-frame encodings and free-running predictions under recorded actions. **Table 3. Physical-state readout with a separately trained linear probe for each model.** | Method | Observed frames: position / orientation | Prediction after 25 actions: position / orientation | | --- | --- | --- | | FP | 11.95 / 7.02 | 15.51 / 8.62 | | RTN | 12.00 / 7.04 | 22.86 / 13.67 | | Module-MSE | **11.87 / 6.89** | **17.01 / 9.22** | | GPTQ | 12.21 / 6.92 | 20.11 / 11.50 | | Module-L1 | 11.88 / 7.16 | 20.95 / 11.28 | | Relative-MSE | 11.99 / 7.11 | 18.87 / 11.36 | | Cost-MSE | 11.92 / 6.96 | 20.92 / 13.64 | Position and orientation errors are measured in pixels and degrees, respectively (lower is better). Observed-frame results average frames at intervals of five actions from 0 to 25; predictions are evaluated after 25 actions. Bold marks the lowest mean among quantized methods. On observed frames, Module-MSE gives position and orientation errors of 11.87 px and 6.89°, close to FP’s 11.95 px and 7.02°. After 25 actions, its errors are 17.01 px and 9.22°, the lowest among the quantized methods and below RTN’s 22.86 px and 13.67°. This indicates that module reconstruction helps preserve linearly decodable position and orientation information, particularly in multi-step predictions.
AuthorsOpenReview ↗

Rebuttal to Reviewer Tn4U Part 3

Part 3

Question 1

Reviewer comment

In table 1, are there any reasons why DINO-WM is very sensitive to W4 RTN? Is this because of 1) DINO-WM is using patch features or 2) the frozen DINOv2 encoder itself is sensitive to W4 RTN. I think these factors are entangled and it's worth knowing what are the actual cause of the instability.

Response

Thank you for asking us to distinguish the effects of patch-based representations from the sensitivity of the DINOv2 encoder.

Section 3.2 traces how latent perturbations affect scores, candidate rankings, elite selection, and CEM updates. Here we examine both the sensitivity of those decisions and the module that contributes most to the error.

1. Decision sensitivity: score perturbations relative to candidate margins. DINO-WM has smaller margins between the top candidates and larger normalized RTN score perturbations. Whether a decision changes depends on the perturbation relative to the margin, not on the absolute error alone. Suppose FP prefers candidate ii to jj, so JiF<JjFJ_i^F<J_j^F. With FP margin gij=JjF−JiFg_{ij}=J_j^F-J_i^F and score error ϵi=JiF−JiQ\epsilon_i=J_i^F-J_i^Q, the quantized margin is JjQ−JiQ=gij−(ϵj−ϵi)J_j^Q-J_i^Q=g_{ij}-(\epsilon_j-\epsilon_i). The ranking flips if the differential perturbation exceeds the original margin.

Table 4. Smaller candidate margins and larger RTN score perturbations in DINO-WM.

ModelTop-two margin / IQRElite-boundary margin / IQRRTN score perturbation RMS / IQRInversion rate
LeWM0.55490.020040.18555.94%
DINO-WM0.27680.017700.875044.95%

For each model, we evaluate 64 start states with 300 shared candidates per state, five prediction steps, and 30 elites. IQR is the interquartile range of FP costs; RMS is the root mean square of the score perturbation after subtracting its common offset. The elite-boundary gap is the cost difference between candidates ranked 30th and 31st.

DINO-WM’s normalized RTN perturbation is about 4.7 times LeWM’s, while its top-two margin is about half as large. The respective inversion rates are 44.95% and 5.94%. The elite-boundary gaps are similar, further highlighting the larger score perturbation in DINO-WM.

2. Error source: the encoder is the more sensitive module. We keep the DINO-WM checkpoint, planning configuration, and patch representation fixed and apply RTN to the encoder or predictor separately.

Table 5. DINO-WM: RTN applied to the encoder, predictor, or both.

EncoderPredictorElite overlap ↑Inversion rate ↓
FPFP100.00%0.00%
RTNFP16.87%45.21%
FPRTN63.73%17.00%
RTNRTN16.78%46.16%

We use 200 test states with 300 shared candidates per state, five prediction steps, and 30 elites.

Encoder-only RTN yields a 45.21% inversion rate, compared with 17.00% for predictor-only RTN. Together with Table 4, this suggests that encoder perturbations feed into the rollout and goal comparison, yielding score changes large enough to cross candidate margins. This effect is observed without changing the patch representation.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.