acceptodds
Under review as a conference paper at ICLR 2027

MAC-v2: Long-horizon Model-based Value Expansion from Pixels

Abstract

We study whether long-horizon model-based value expansion can serve as an effective framework in pixel-based offline reinforcement learning (RL). Long-horizon model-based value expansion from pixels is substantially harder than state-based environments, as the agent must learn both the representation and its dynamics, and errors from either can accumulate recursively over imagined trajectories. We introduce MAC-v2 which enables long-horizon model-based expansion from pixels by training a reconstruction-free latent world model with action chunking. By learning representations from prediction and control objectives, MAC-v2 avoids encoding control-irrelevant visual details that can hinder the dynamics prediction. Action chunking further mitigates the prediction error by reducing the number of recursive model predictions required for value expansion. Across diverse visual benchmarks including visual OGBench, MMBench2, Robomimic, and LIBERO-10, MAC-v2 achieves strong performance against model-free and model-based offline RL baselines. Our analyses show that long-horizon model-based value expansion is critical for the performance of MAC-v2, and that both action chunking and latent learning objectives are essential for reliable long-horizon latent imagination.

Primary area:
reinforcement learning
Keywords:
RL, Offline RL, Model-based RL, Pixel-based RL, Action chunking

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_quYfOpenReview ↗
Rating
3(+2)
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper presents MAC-v2, a sequel to Model-based RL with Action Chunks (MAC), an offline model-based reinforcement learning (RL) algorithm to tackle long-horizon, pixel-based continuous control. MAC-v2 builds upon the recipe of MAC: leveraging a trained action-chunk policy and model for effective horizon reduction in RL and adapting rejection sampling of an expressive flow-matching policy for out-of-distribution action penalization. The authors evaluate MAC-v2 with popular model-free and model-based offline RL baselines across vision-based control benchmarks, where MAC-v2 demonstrates strong or competitive performance.

Critical strengths
  1. Strong empirical performance over benchmarks: MAC-v2 demonstrates strong empirical performance over popular model-free and model-based RL baselines across four offline RL benchmarks: OGBench, MMBench2, Robomimic, and LIBERO-10. Notably, other model-based RL baselines (i.e., LOMPO and OfflineDV2) exhibit almost broken performance on three benchmarks except MMBench2, whereas MAC-v2 shows competitive or the strongest performance on all benchmarks.
  2. Academically impactful challenge in model-based visual RL: Long-horizon control in RL is still one of the long-standing challenges in the RL literature. While recent approaches have tackled this problem (e.g., using expressive policies for horizon reduction), solving the problem under this scheme of learning a pixel-based policy with model-based RL still requires substantial investigation.
Critical weaknesses
  1. Lack of connection from the original backbone---MAC: The authors named their method as MAC-v2, which is a successor version of MAC. Building a sequel method from a previous version is not uncommon in RL; e.g., the Dreamer or TD-MPC family. However, the problem is that one cannot find any connection to the original backbone method in the manuscript. For rigorous investigation and further development of the algorithmic foundation, the authors must describe which parts or components are changed compared to the preceding algorithm. In the manuscript, the authors did not include the differences between MAC-v2 and MAC, nor compare the performance. I conjecture the following differences in the algorithm design: (i) the reward/value/Q network becomes stochastic (originally deterministic), (ii) the one-step policy distilled from the flow-matching policy is removed, and (iii) technical components for visual RL are added (e.g., CNN encoder, data augmentation, frame stacking, stop-gradient, and EMA). Since ablation studies for these components are not provided, I feel unconvinced that MAC-v2 presents significant improvements over the original MAC. In a future manuscript, the authors must clarify the differences.
  2. Weak problem formulation for long-horizon model-based value expansion from vision input: In the Introduction, the authors state that learning both the representation and its dynamics model from fixed offline visual data is challenging for the long-horizon model-based value expansion under the offline RL scheme. The authors subsequently remark that learning a representation that preserves control-relevant information while being predictable by the dynamics model is the key challenge. MAC empirically demonstrates that an action-chunk expressive policy and dynamics with the rejection sampling strategy can mitigate the horizon reduction problem in state-based experiments. However, algorithms for learning with low-dimensional states do not transfer directly to learning with high-dimensional pixels, requiring additional representation learning, as the authors describe in the related work section. In my view, learning a pixel-based policy with an offline model-based RL framework is specifically challenging for the following reasons: (i) the encoder, policy, and dynamics model are jointly coupled in representation learning and behavior learning, while no single representation learning objective dominates the field (e.g., reconstruction-based Dreamer-like methods vs reconstruction-free JEPA-like methods), and (ii) the agent cannot collect corrective behaviors via online interaction, as the authors note. Within the limited evaluation, the claim that MAC-v2 enables long-horizon offline MBRL from pixels feels overclaimed to me. The authors must explain how MAC-v2 mitigates the above challenges I described via theoretical sketches or further experiments. For instance, by incorporating MAC-v2's encoder into MAC and including MAC in the main experiments, one can show that naively incorporating a vision encoder into MAC fails, while MAC-v2 can improve the performance.
Questions for authors
  1. MAC-v2 vs. MAC and evidence for the claim: How do you develop MAC-v2 based on MAC? What are the main differences, and what are the minor design choices that affect performance little? How does the suggested design support the claim that MAC-v2 can solve long-horizon model-based offline RL from pixels effectively?
  2. Experimental setup: Could you compare MAC w/ MAC-v2's vision encoder across benchmarks? As I conjectured in Critical Weaknesses, could you perform the following ablation studies: (i) the effect of determinism and stochasticity of reward/value/Q networks, and (ii) performance and wall-time comparison of using a one-step distillation policy for the flow-matching BC policy? For Q3 & Q4 in Section 5.3, could you provide more description of the setup? How do you remove the gradient from each objective to the encoder? How do you incorporate each representation learning objective into MAC-v2?
  3. Justifications for poor performance of MBRL baselines: Why do LOMPO and OfflineDV2 show poor performance across most benchmarks, which are almost corrupted? Is it attributed to outdated algorithms?
  4. Experimental results: For Q1 & Q2, how does the horizon length HH and actioin chunk length nn correlate with each other? For the same effective horizon length, which of the lengths contributes to better performance?

Clarification requests

  1. If MBRL baselines are outdated, could you consider including single-task DreamerV3(or DreamerV4) or TD-MPC2 as more recent MBRL baselines that support pixel-based offline learning, which do not fall into the scope of this paper but still provide a meaningful performance bottom lines (not during rebuttal, as future baselines)?

Minor requests.

  1. I think choosing block MDP rather than POMDP is overly strict in pixel-based RL, which is unusual as far as I know and may limit broader applications. Is there a special reason for choosing the block MDP?
  2. In Section 4.1, I think moving the notation for chunk data for actions and rewards (i.e., at=at:t+n−1\mathbf{a_t}=a_{t:t+n-1} and rt=∑i=0n−1γirt+i\mathbf{r_t}=\sum_{i=0}^{n-1}{\gamma^i r_{t+i}}) from the pseudo table to the front would increase readability and coherence.
2 comments
AuthorsOpenReview ↗

Response to Reviewer quYf (1/2)

Thank you for your constructive feedback. We address your questions and concerns in detail below and our new manuscript reflects your suggestions. Please refer to the updated PDF for new results and revisions. We believe your suggestions have substantially improved the quality and clarity of our paper, and we would greatly appreciate it if you could reconsider your score in light of the revisions.

 

[Q1, Q2] MAC-v2 vs. MAC and evidence for the claim: How do you develop MAC-v2 based on MAC? What are the main differences, and what are the minor design choices that affect performance little? How does the suggested design support the claim that MAC-v2 can solve long-horizon model-based offline RL from pixels effectively? Experimental setup: Could you compare MAC w/ MAC-v2's vision encoder across benchmarks? As I conjectured in Critical Weaknesses, could you perform the following ablation studies: (i) the effect of determinism and stochasticity of reward/value/Q networks, and (ii) performance and wall-time comparison of using a one-step distillation policy for the flow-matching BC policy?

Thank you so much for raising this concern. We strongly agree that the manuscript should more clearly distinguish the contributions of MAC-v2 from MAC.

The main contribution of MAC-v2 is extending long-horizon model-based value expansion of MAC to visual observations through learned latent world modeling. MAC operates on low-dimensional state observations, whereas MAC-v2 must learn both the latent representation and its dynamics from high-dimensional visual inputs. This setup requires to solve representation learning problem, which was not tackled in MAC. Our representation learning ablations further demonstrate the importance of MAC-v2’s representation learning objectives.

As the reviewer identified, we also introduce distributional value prediction and remove the one-step policy distillation used in MAC. However, these are secondary modifications rather than our main contribution. To show this, we conducted an additional ablation study on the improvements of MAC-v2 from MAC on cube-double-task3.

Success rateMAC-v2ScalarOne-stepMAC (500k)Best baseline
cube-double-task352 ± 626 ± 632 ± 611 ± 1113 ± 4

The results show that replacing distributional value prediction with scalar regression reduces performance and using a one-step policy degrades performance. However, removing the latent modeling of MAC-v2 severely drops the performance, showing that learning a latent world model is critical for MAC-v2’s success in visual environments. We note that MAC with 500k steps already uses 3x more compute budget compared to MAC-v2, due to pixel imagination.

We added a dedicated paragraph stating the contributions of MAC-v2 from MAC in Section 4. We also added the experimental result on MAC vs MAC-v2 in Section 5.3.

 

[Q2-2] For Q3 & Q4 in Section 5.3, could you provide more description of the setup? How do you remove the gradient from each objective to the encoder? How do you incorporate each representation learning objective into MAC-v2?

Thank you for pointing out the ambiguity in the description.

For the gradient ablations (Q3), we apply the stop-gradient operator to the latent representation before passing it to the corresponding module. For example, to remove the policy gradient, we train the policy using sg(z_t) instead of z_t in the flow-matching objective. This allows the policy to be trained normally while preventing its loss from updating the encoder. For the alternative representation learning objectives (Q4), we train the encoder using the representation learning objective instead of MAC-v2's training objectives. For example, in the SimCLR ablation, we first train the encoder using the SimCLR objective, then freeze its parameters and train the world model, policy, and value functions on top of the resulting representation.

We added these implementation details to the Q3 and Q4 discussions in Section 5.3.

AuthorsOpenReview ↗

Response to Reviewer quYf (2/2)

[Q3] Justifications for poor performance of MBRL baselines: Why do LOMPO and OfflineDV2 show poor performance across most benchmarks, which are almost corrupted? Is it attributed to outdated algorithms?

Thank you for raising this question. We found that LOMPO and OfflineDV2 perform especially poorly on the robotic manipulation tasks, including those in visual OGBench. One possible explanation is their reliance on uncertainty-based penalization, whereas MAC-v2 adopts rejection sampling inherited from MAC.

To investigate this, we conducted an additional ablation study in cube-double-task3, in which we replaced MAC-v2's behavior constraint from rejection sampling with MOPO-style uncertainty penalization, while retaining the remaining components.

Success rateMAC-v2LOMPOOfflineDV2MAC-v2 + MOPO
cube-double-task352 ± 60 ± 00 ± 00 ± 0

Replacing rejection sampling with uncertainty penalization substantially reduces the performance. This suggests that the choice of behavior constraint mechanism is an important factor in the performance gap, rather than simply the age of the baseline algorithms.

We added this discussion in Appendix D.

 

[Q4] Experimental results: For Q1 & Q2, how does the horizon length and action chunk length correlate with each other? For the same effective horizon length, which of the lengths contributes to better performance?

Thank you for this question. To answer this question, we additionally conducted an ablation experiment in visual-puzzle-4x4-task1 with Hn=100 being fixed.

(n, H)Success % (4 seeds)
(1, 100)0 ± 0
(4, 25)1 ± 1
(10, 10)16 ± 16
(25, 4)2 ± 1
(100, 1)0 ± 0

The result shows that having a balance on nn and HH is important; too long action chunk size (nn) degrades the performance as the policy becomes less reactive, but we want to be chunk size large enough so that the horizon length HH is small, as too long horizon length degrades the performance because of the accumulating error of the model-based value expansion.

We added this discussion in Appendix D.

 

[Q5] If MBRL baselines are outdated, could you consider including single-task DreamerV3(or DreamerV4) or TD-MPC2 as more recent MBRL baselines that support pixel-based offline learning, which do not fall into the scope of this paper but still provide a meaningful performance bottom lines (not during rebuttal, as future baselines)?

We would like to clarify that DreamerV3 and TD-MPC2 are primarily designed for online model-based RL (MBRL). In particular, TD-MPC2 has only been evaluated in the offline setting for state-based multi-task RL. Therefore, neither method is directly comparable to MAC-v2, which focuses on pixel-based offline RL. While DreamerV4 supports offline RL, its official implementation is not publicly available, and its reported training setup requires eight GPUs, making reproduction computationally expensive.

Nevertheless, to provide a more comprehensive comparison, we are implementing DreamerV4 based on the MMBench2 codebase and tuning TD-MPC2 in our setting. We will incorporate these results into the final manuscript as they become available.

 

[Q6] I think choosing block MDP rather than POMDP is overly strict in pixel-based RL, which is unusual as far as I know and may limit broader applications. Is there a special reason for choosing the block MDP?

We adopt the block MDP assumption because our current implementation of MAC-v2 does not incorporate a history-dependent representation module, such as the RSSM used in Dreamer models. However, we would like to emphasize that this assumption is not an inherent limitation of our method. MAC-v2 can, in principle, be extended to the POMDP setting by incorporating a history-dependent encoder that summarizes past observations and actions into a latent state. Since our method operates on learned latent representations, such an extension would not require fundamental changes to the MAC-v2 framework.

 

[Q7] In Section 4.1, I think moving the notation for chunk data for actions and rewards (i.e., and ) from the pseudo table to the front would increase readability and coherence.

Thank you for the suggestion, we updated the paper accordingly.

Reviewer_MdztOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies long-horizon model-based value expansion on fixed visual offline datasets by combining MAC’s action-chunked world model, flow-based behavior-policy candidate selection, and multi-step latent value targets. Its main extension is a reconstruction-free encoder trained jointly via latent dynamics prediction, chunked reward prediction, and behavior cloning, so the learned representation supports both prediction and control without introducing a wholly new representation-learning objective. Images are mapped to latent states, an n-step dynamics model predicts future latents from current latents and action chunks, the policy generates candidate chunks through flow-based behavior cloning and selects the highest-Q candidate, and the value and Q functions are trained using returns from H imagined chunks and model-predicted rewards. Experiments across 25 OGBench, 10 MMBench2, 3 Robomimic, and 10 LIBERO-10 tasks show the largest improvements on multi-object manipulation and puzzle tasks, mixed gains on MMBench2, and performance competitive with behavior cloning on expert-demonstration benchmarks.

Critical strengths
  1. A major strength is its attempt to address the limited temporal coverage of conventional short-horizon model-based value expansion. By combining action chunking with latent rollouts, MAC-V2 can propagate value information over substantially longer effective horizons without requiring prohibitively deep sequential prediction. The reported results on challenging OGBench tasks, such as Cube, Scene, and Puzzle, demonstrate that this design can translate into substantial empirical gains.
  2. The default chunk length, model-rollout length, and candidate counts are shared across all four benchmark suites. The paper reports four-seed results, per-task numbers, and learning curves. MAC-v2 and the model-free baselines share the same encoder within each benchmark, and the appendix includes baseline tuning and runtime information. The authors clearly acknowledge the method’s inheritance from MAC and prior reconstruction-free representation learning. These materials help assess its potential as a practical visual offline RL method.
  3. The method achieves strong performance on several difficult long-horizon OGBench tasks and provides multiple ablations supporting the importance of long-horizon rollout, action chunking, and the gradient pathways used in training. In particular, the trend of improving performance with longer rollout horizons provides useful evidence that the proposed value expansion mechanism is relevant to the targeted long-horizon setting.
Critical weaknesses
  1. The central claim is that action chunking and joint representation objectives mitigate model error and make long-horizon value expansion effective. The evidence is consistent with this explanation, but does not sufficiently rule out alternatives. Since action chunking, flow-based candidate selection, and long-horizon model-based value expansion already come from previous MAC, the empirical contribution beyond that work particularly needs clear attribution.
  2. Table 4 on page 14 sets γ=0.99 for the model-free baselines, whereas Table 6 on page 15 sets γ=0.999 for MAC-v2. This directly changes the weight of distant sparse rewards:0.99^100≈ 0.366 and 0.999^100≈ 0.905. The comparison therefore does not clearly separate the benefits of model-based value expansion from those of a longer effective discount horizon.
  3. The horizon experiment in Figure 1 on page 8 uses only puzzle-4x4-task5, and the chunk-length experiment uses only task1. If the number of recursive predictions H is fixed, changing n simultaneously changes the environment horizon Hn, the open-loop action length, and the prediction span. The paper does not clearly specify what is held fixed in this ablation, so it cannot by itself establish that fewer recursive predictions reduce error.
  4. On representative tasks, the analysis should also report multi-step reward or return errors under identical action sequences, and calibration of predicted values against actual rollout returns, rather than only policy success rates. In particular, the complete task5 result is 16±16; a single high-variance task provides limited support for a general mechanistic conclusion.
  5. the representation experiments in Figures 3–4 use only cube-double-task3. It is unclear whether replacing the representation objective preserves BC gradients, dynamics gradients, and network capacity. Removing control supervision and adding reconstruction to the original joint objective are different controls. The exact intervention should be specified and checked on a small number of representative tasks to support the roles of reconstruction-free learning and the two supervision sources.
  6. Page 13 defines the predicted termination flag as 1 when termination occurs within a chunk. Equation 13 nevertheless multiplies these termination flags to retain subsequent rewards and the bootstrap term, whereas Equation 14 uses one minus the termination flag. These definitions conflict.
  7. Baseline coverage also limits the conclusions. SeMOPO is discussed in the introduction but omitted from the experiments. Its treatment of control-irrelevant visual information and model uncertainty is closely related to this paper’s motivation [R1]. The omission should be explained, or a comparison added on a representative subset.
  8. The training protocol also needs clarification. Model-free baselines use 500K updates, while the MAC-v2 OGBench curves extend to 1M; Table 6 does not specify the total update count. The appendix reports approximately 16 hours for the model-based baselines and 9 hours for MAC-v2 on the relevant benchmarks. I therefore do not infer a compute advantage for MAC-v2 from update counts alone. Results under matched budgets, final-checkpoint selection, and standard-deviation aggregation should still be specified. [R1] Wan et al. SeMOPO: Learning High-quality Model and Policy from Low-quality Offline Visual Datasets. ICML 2024.
Questions for authors

Q1 How do QC or FQL perform with γ=0.999 under reasonably matched update or wall-clock budgets? What are MAC-v2’s actual update count, batch size, and checkpoint-selection rule? Q2 When n changes, is H or Hn held fixed? At fixed Hn on representative tasks, how do the recursion count, return error, and control performance change? Q3 Which encoder gradients are retained under each alternative objective in Figure 4? Please compare adding reconstruction while retaining the same BC and prediction supervision, and clarify whether the result holds on other representative tasks. Q4 Should the products in Equation 13 use 1−d? Please provide the termination-model loss and multi-step target actually used for the results, including how true termination and time-limit truncation are handled. Q5 Why was SeMOPO omitted, and do the conclusions hold with aligned key input settings? How are checkpoints and across-seed standard deviations aggregated? Please also reconcile the stated λ search set with λ=5 in Table 7.

5 comments
AuthorsOpenReview ↗

Response to reviewer Mdzt (1/4)

Thank you for your constructive feedback. We address your questions and concerns in detail below and our new manuscript reflects your suggestions. Please refer to the updated PDF for new results and revisions. We believe your suggestions have substantially improved the quality and clarity of our paper, and we would greatly appreciate it if you could reconsider your score in light of the revisions.

 

[W1] The central claim is that action chunking and joint representation objectives mitigate model error and make long-horizon value expansion effective. The evidence is consistent with this explanation, but does not sufficiently rule out alternatives. Since action chunking, flow-based candidate selection, and long-horizon model-based value expansion already come from previous MAC, the empirical contribution beyond that work particularly needs clear attribution.

Thank you for raising this concern. We strongly agree that the manuscript should more clearly distinguish the contributions of MAC-v2 from MAC.

The main contribution of MAC-v2 is extending long-horizon model-based value expansion of MAC to visual observations through learned latent world modeling. MAC operates on low-dimensional state observations, whereas MAC-v2 must learn both the latent representation and its dynamics from high-dimensional visual inputs. This setup requires to solve representation learning problem, which was not tackled in MAC. Our representation learning ablations further demonstrate the importance of MAC-v2’s representation learning objectives.

While we also introduce distributional value prediction and remove the one-step policy distillation used in MAC, these are secondary modifications rather than our main contribution. To show this, we conducted an additional ablation study on the improvements of MAC-v2 from MAC on cube-double-task1.

Success rateMAC-v2ScalarOne-stepMAC (500k)Best baseline
cube-double-task352 ± 626 ± 632 ± 611 ± 1113 ± 4

The results show that replacing distributional value prediction with scalar regression reduces performance and using a one-step policy degrades performance. However, removing the latent modeling of MAC-v2 severely drops the performance, showing that learning a latent world model is critical for MAC-v2’s success in visual environments. We note that MAC with 500k steps already uses 3x more compute budget compared to MAC-v2, due to pixel imagination.

We added a dedicated paragraph stating the contributions of MAC-v2 from MAC in Section 4. We also added the experimental result on MAC vs MAC-v2 in Section 5.3.

 

[W2, Q1] Table 4 on page 14 sets γ=0.99 for the model-free baselines, whereas Table 6 on page 15 sets γ=0.999 for MAC-v2. This directly changes the weight of distant sparse rewards:0.99^100≈ 0.366 and 0.999^100≈ 0.905. The comparison therefore does not clearly separate the benefits of model-based value expansion from those of a longer effective discount horizon.

Thank you for pointing out the potential caveat of our evaluation. To address this concern, we evaluated MAC-v2, QC, and FQL with both γ=0.99\gamma=0.99 and γ=0.999\gamma=0.999 on cube-double-task3.

Success rateMAC-v2QCFQL
γ=0.999\gamma = 0.99952 ± 60.0 ± 0.00.0 ± 0.0
γ=0.99\gamma = 0.9942 ± 60.8 ± 0.61.3 ± 1.1

Increasing the discount factor to 0.999 does not improve QC or FQL, having 0% success rate for both tasks. While reducing MAC-v2’s discount factor to 0.99 decreases the success rate, it still retains a substantial advantage over both baselines. This shows that MAC-v2’s success is not attributed from the discount rate, but from the algorithmic improvement.

We also note that our configurations for QC and FQL follow the discount factor that is already tuned in their respective papers (\gamma=0.99). We have added these controlled comparisons to Appendix D.

AuthorsOpenReview ↗

Response to reviewer Mdzt (2/4)

[W3, Q2] The horizon experiment in Figure 1 on page 8 uses only puzzle-4x4-task5, and the chunk-length experiment uses only task1. If the number of recursive predictions H is fixed, changing n simultaneously changes the environment horizon Hn, the open-loop action length, and the prediction span. The paper does not clearly specify what is held fixed in this ablation, so it cannot by itself establish that fewer recursive predictions reduce error. When n changes, is H or Hn held fixed? At fixed Hn on representative tasks, how do the recursion count, return error, and control performance change?

Thank you for this suggestion. In the paper, H is held fixed while n changes. We additionally conducted two experiments for the control performance change and the return error.

For the control performance change, we ran MAC-v2 with various configurations with Hn=100 being fixed.

(n, H)Success % (4 seeds)
(1, 100)0.0 ± 0.0
(4, 25)1 ± 1
(10, 10)16 ± 16
(25, 4)2 ± 1
(100, 1)0.0 ± 0.0

The result shows that having a balance on nn and HH is important; too long action chunk size (nn) degrades the performance as the policy becomes less reactive, but we want to be chunk size large enough so that the horizon length HH is small, as too long horizon length degrades the performance because of the accumulating error of the model-based value expansion.

For the return error, we conducted an experiment comparing the return prediction errors (MSE) of one-step and action-chunked world models to the ground-truth returns over horizons of up to 100 environment steps. To isolate the effect of action chunking in the world model, we fix the policy to MAC-v2's action-chunked policy (chunk length 10) and evaluate both world models using the same action sequences (We use this action-chunked policy because a policy trained without action chunking performs poorly, producing nearly constant returns of -1 per step and making return prediction errors difficult to meaningfully evaluate).

h (env steps)1-stepchunk model
100.260.02
202.280.03
307.920.32
4017.291.70
5029.274.27
6043.108.78
7058.4014.82
8076.1022.41
9094.7930.92
100113.4241.24

The results show that return prediction errors increase with the rollout horizon for both models, but are consistently and substantially lower with action-chunked dynamics. These results provide direct empirical evidence that action chunking mitigates compounding model errors in long-horizon return estimation. This supports one of MAC-v2's central design choices and is consistent with the motivation for horizon reduction (e.g., [1], [2]).

[1] Seohong Park et al., Horizon Reduction Makes RL Scalable, NeurIPS 2026

[2] Kwanyoung Park et al., Scalable Offline Model-Based RL with Action Chunks, ICLR 2026

 

[W4] On representative tasks, the analysis should also report multi-step reward or return errors under identical action sequences, and calibration of predicted values against actual rollout returns, rather than only policy success rates.

Thank you for this suggestion. We agree that evaluating return prediction errors provides important evidence beyond policy success rates. We also report the absolute return error and the relative error under identical action sequences in puzzle-4x4-task1.

(n, H)Return errorRelative error
(1, 10)−96.720.2%
(5, 10)+7.21.4%
(10, 10)−1.10.2%
(10, 5)−4.70.9%
(10, 1)+24.84.5%

The result shows that (n, H) = (10, 10) shows the lowest error, and follows the trend of the task performance, further supporting the ablation result of Q1 (Section 5.3). We added this discussion and the result in Appendix D.

AuthorsOpenReview ↗

Response to reviewer Mdzt (3/4)

[W5, Q3] It is unclear whether replacing the representation objective preserves BC gradients, dynamics gradients, and network capacity. Removing control supervision and adding reconstruction to the original joint objective are different controls. The exact intervention should be specified and checked on a small number of representative tasks to support the roles of reconstruction-free learning and the two supervision sources. Which encoder gradients are retained under each alternative objective in Figure 4? Please compare adding reconstruction while retaining the same BC and prediction supervision, and clarify whether the result holds on other representative tasks.

Thank you for suggesting clarification and the experiments. For clarification, on the alternative representation learning objective ablations, we train the encoder only using the representation learning objective instead of MAC-v2's training objectives. For example, in the SimCLR ablation, we first train the encoder using the SimCLR objective, then freeze its parameters and train the world model, policy, and value functions on top of the resulting representation.

We conducted the experiment that evaluates MAC-v2 with the reconstruction objective while retaining other MAC-v2’s objectives in cube-double-task3.

EnvMAC-v2 + reconstructionMAC-v2reconstruction
cube-double-task353 ± 152 ± 78 ± 3

The result shows that the reconstruction objective only slightly improves the performance of MAC-v2, despite using 50% more parameters due to pixel reconstruction and being 2x slower, showing that MAC-v2’s objective is the primary source of representation learning.

We added this result in Appendix D and clarified the experimental setting in Section 5.3.

 

[W6, Q4] Page 13 defines the predicted termination flag as 1 when termination occurs within a chunk. Equation 13 nevertheless multiplies these termination flags to retain subsequent rewards and the bootstrap term, whereas Equation 14 uses one minus the termination flag. These definitions conflict. Should the products in Equation 13 use 1−d? Please provide the termination-model loss and multi-step target actually used for the results, including how true termination and time-limit truncation are handled.

Thank you for pointing out the error in the equation. We corrected the value target of Equation (13) to use the product of (1-d_t) instead of d_t. We clarify that the termination model is trained using a binary cross-entropy loss to predict the termination, and action chunks containing time-limit truncations are excluded during sampling. We updated these implementation details to Appendix A.1.

 

[W7, Q5] Baseline coverage also limits the conclusions. SeMOPO is discussed in the introduction but omitted from the experiments. Its treatment of control-irrelevant visual information and model uncertainty is closely related to this paper’s motivation [R1]. The omission should be explained, or a comparison added on a representative subset. Why was SeMOPO omitted, and do the conclusions hold with aligned key input settings? How are checkpoints and across-seed standard deviations aggregated?

Thank you for suggesting the SeMOPO as an important baseline. We agree with your suggestion and we run the SeMOPO in task1 of the OGBench environments.

EnvSeMOPOMAC-v2
visual-cube-single-play-task10.0 ± 0.087.2 ± 2.7
visual-cube-double-play-task12.2 ± 2.666.7 ± 4.3
visual-scene-play-task111.5 ± 8.5100.0 ± 0.0
visual-puzzle-3x3-play-task110.2 ± 1.9100.0 ± 0.0
visual-puzzle-4x4-play-task13.7 ± 1.489.7 ± 4.8

The results show that MAC-v2 outperforms SeMOPO with a large margin. We are unlikely to finish the sweep for all tasks during the rebuttal, but we will include the whole taskset in the paper as soon as we get the full results.

For aggregating the results, we average over the last 3 checkpoint’s performances following [3], and we first aggregate the values per-seed and calculate the standard deviation within the aggregated results.

[3] Seohong Park et al., Flow Q-Learning, ICML 2025

Reviewer_rhihOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces MAC-v2, an offline model-based reinforcement learning framework for long-horizon value expansion from visual observations. The method combines a reconstruction-free latent world model with action chunking. The visual encoder is jointly trained through latent dynamics prediction, reward prediction, and flow-matching behavior cloning. The policy selects the highest-valued action chunk among candidates sampled from the learned behavioral policy, while value functions are trained using imagined trajectories in latent space. With the default configuration, ten recursive model predictions cover 100 environment steps.

Critical strengths

The paper addresses an important challenge in visual offline RL and the method design is coherent and well motivated. The evaluation covers multiple benchmarks and the improvements on challenging OGBench tasks are clear.

Critical weaknesses

1.Table 4 specifies a discount factor of 0.99 for the model-free baselines, whereas Table 6 uses 0.999 for MAC-v2. A reward received 100 steps later consequently receives a weight of approximately 0.37 versus 0.90. The current comparisons therefore do not adequately isolate the contribution of model-based value expansion from that of the discounting configuration. 2.The analysis primarily measures task success, without reporting multi-step return prediction error, value calibration, or the accuracy of candidate-action rankings. Consequently, the proposed explanation that action chunking and the learned representation mitigate compounding prediction errors remains insufficiently substantiated. Fewer recursive predictions do not by themselves establish lower overall error, since predicting the outcome of a longer action sequence can also be more difficult. Comparing imagined returns with actual environment returns under matched action sequences would directly test this explanation. Such diagnostics are especially relevant because behavioral action sampling does not ensure that recursively imagined latent states remain within the data distribution. 3.Appendix A.1 defines as a termination indicator, but Equation (13) multiplies future rewards and the bootstrap value by products of , rather than . Under the stated definition, this would incorrectly suppress future returns following nonterminal transitions. Equation (14) instead uses , creating an internal inconsistency. An implementation following Equation (13) would affect the long-horizon value targets and the interpretation of the results.

Questions for authors

See Critical Weaknesses.

2 comments
AuthorsOpenReview ↗

Response to reviewer rhih

Thank you for your constructive feedback. We address your questions and concerns in detail below and our new manuscript reflects your suggestions. Please refer to the updated PDF for new results and revisions. We believe your suggestions have substantially improved the quality and clarity of our paper, and we would greatly appreciate it if you could reconsider your score in light of the revisions.

 

[W1] Table 4 specifies a discount factor of 0.99 for the model-free baselines, whereas Table 6 uses 0.999 for MAC-v2. A reward received 100 steps later consequently receives a weight of approximately 0.37 versus 0.90. The current comparisons therefore do not adequately isolate the contribution of model-based value expansion from that of the discounting configuration.

Thank you for pointing out the potential caveat of our evaluation. To address this concern, we evaluated MAC-v2, QC, and FQL with both γ=0.99\gamma=0.99 and γ=0.999\gamma=0.999 on cube-double-task3.

Success rateMAC-v2QCFQL
γ=0.999\gamma = 0.99952 ± 60.0 ± 0.00.0 ± 0.0
γ=0.99\gamma = 0.9942 ± 60.8 ± 0.61.3 ± 1.1

Increasing the discount factor to 0.999 does not improve QC or FQL, having 0% success rate for both tasks. While reducing MAC-v2’s discount factor to 0.99 decreases the success rate from 52% to 42%, it still retains a substantial advantage over both baselines. This shows that MAC-v2’s success is not attributed from the discount rate, but from the algorithmic improvement. We also note that our configurations for QC and FQL follow the discount factor that is already tuned in their respective papers (=0.99). We have added these controlled comparisons to Appendix D.

 

[W2] The analysis primarily measures task success, without reporting multi-step return prediction error, value calibration, or the accuracy of candidate-action rankings. Consequently, the proposed explanation that action chunking and the learned representation mitigate compounding prediction errors remains insufficiently substantiated. Fewer recursive predictions do not by themselves establish lower overall error, since predicting the outcome of a longer action sequence can also be more difficult. Comparing imagined returns with actual environment returns under matched action sequences would directly test this explanation. Such diagnostics are especially relevant because behavioral action sampling does not ensure that recursively imagined latent states remain within the data distribution.

Thank you for this suggestion. To directly examine this, we conducted an experiment comparing the return prediction errors (MSE) of one-step and action-chunked world models to the ground-truth returns over horizons of up to 100 environment steps in visual-puzzle-4x4-task1. To isolate the effect of action chunking in the world model, we fix the policy to MAC-v2's action-chunked policy (chunk length 10) and evaluate both world models using the same action sequences (We use this action-chunked policy because a policy trained without action chunking performs poorly, producing nearly constant returns of -1 per step and making return prediction errors difficult to meaningfully evaluate).

h (env steps)1-stepchunk model
100.260.02
202.280.03
307.920.32
4017.291.70
5029.274.27
6043.108.78
7058.4014.82
8076.1022.41
9094.7930.92
100113.4241.24

The results show that return prediction errors increase with the rollout horizon for both models, but are consistently and substantially lower with action-chunked dynamics. These results provide direct empirical evidence that action chunking mitigates compounding model errors in long-horizon return estimation. This supports one of MAC-v2's central design choices and is consistent with the motivation for horizon reduction (e.g., [1], [2]).

[1] Seohong Park et al., Horizon Reduction Makes RL Scalable, NeurIPS 2026

[2] Kwanyoung Park et al., Scalable Offline Model-Based RL with Action Chunks, ICLR 2026

 

[W3]. Appendix A.1 defines as a termination indicator, but Equation (13) multiplies future rewards and the bootstrap value by products of , rather than . Under the stated definition, this would incorrectly suppress future returns following nonterminal transitions. Equation (14) instead uses , creating an internal inconsistency. An implementation following Equation (13) would affect the long-horizon value targets and the interpretation of the results.

Thank you for pointing out the error in the equation. We corrected the value target of Equation (13) to use the product of (1-d_t) instead of d_t. We updated these implementation details to Appendix A.1.

As the discussion period is coming to an end, we would like to remind and ask whether our responses have addressed your concerns.

We would be happy to clarify anything further or answer any remaining questions before the discussion period ends.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Response to all reviewers

We thank all reviewers for their constructive feedback! We included new experimental results and revisions in the PDF following the reviewers’ suggestions. Please check out our revised paper. We highlighted the changes in the revised version.

Here, we summarize the major changes:

  • Clarification of MAC-v2’s improvement compared to MAC, and corresponding ablation study (Section 4, Section 5.3)
  • MAC-v2 and the baselines with matched discount factors and training steps (Appendix D)
  • Comparison of return prediction error between MAC-v2 and MAC-v2 without action chunking (Appendix D)
  • More extensive search on (n, H) pairs for long-horizon model-based value expansion analysis with fixed rollout length nH=100 (Appendix D).
  • MAC-v2 with uncertainty penalization results (Appendix D).
  • Correct the value target of Equation (13)

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.