- Rating
- 3(+2)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper presents MAC-v2, a sequel to Model-based RL with Action Chunks (MAC), an offline model-based reinforcement learning (RL) algorithm to tackle long-horizon, pixel-based continuous control. MAC-v2 builds upon the recipe of MAC: leveraging a trained action-chunk policy and model for effective horizon reduction in RL and adapting rejection sampling of an expressive flow-matching policy for out-of-distribution action penalization. The authors evaluate MAC-v2 with popular model-free and model-based offline RL baselines across vision-based control benchmarks, where MAC-v2 demonstrates strong or competitive performance.
Critical strengths
- Strong empirical performance over benchmarks: MAC-v2 demonstrates strong empirical performance over popular model-free and model-based RL baselines across four offline RL benchmarks: OGBench, MMBench2, Robomimic, and LIBERO-10. Notably, other model-based RL baselines (i.e., LOMPO and OfflineDV2) exhibit almost broken performance on three benchmarks except MMBench2, whereas MAC-v2 shows competitive or the strongest performance on all benchmarks.
- Academically impactful challenge in model-based visual RL: Long-horizon control in RL is still one of the long-standing challenges in the RL literature. While recent approaches have tackled this problem (e.g., using expressive policies for horizon reduction), solving the problem under this scheme of learning a pixel-based policy with model-based RL still requires substantial investigation.
Critical weaknesses
- Lack of connection from the original backbone---MAC: The authors named their method as MAC-v2, which is a successor version of MAC. Building a sequel method from a previous version is not uncommon in RL; e.g., the Dreamer or TD-MPC family. However, the problem is that one cannot find any connection to the original backbone method in the manuscript. For rigorous investigation and further development of the algorithmic foundation, the authors must describe which parts or components are changed compared to the preceding algorithm. In the manuscript, the authors did not include the differences between MAC-v2 and MAC, nor compare the performance. I conjecture the following differences in the algorithm design: (i) the reward/value/Q network becomes stochastic (originally deterministic), (ii) the one-step policy distilled from the flow-matching policy is removed, and (iii) technical components for visual RL are added (e.g., CNN encoder, data augmentation, frame stacking, stop-gradient, and EMA). Since ablation studies for these components are not provided, I feel unconvinced that MAC-v2 presents significant improvements over the original MAC. In a future manuscript, the authors must clarify the differences.
- Weak problem formulation for long-horizon model-based value expansion from vision input: In the Introduction, the authors state that learning both the representation and its dynamics model from fixed offline visual data is challenging for the long-horizon model-based value expansion under the offline RL scheme. The authors subsequently remark that learning a representation that preserves control-relevant information while being predictable by the dynamics model is the key challenge. MAC empirically demonstrates that an action-chunk expressive policy and dynamics with the rejection sampling strategy can mitigate the horizon reduction problem in state-based experiments. However, algorithms for learning with low-dimensional states do not transfer directly to learning with high-dimensional pixels, requiring additional representation learning, as the authors describe in the related work section. In my view, learning a pixel-based policy with an offline model-based RL framework is specifically challenging for the following reasons: (i) the encoder, policy, and dynamics model are jointly coupled in representation learning and behavior learning, while no single representation learning objective dominates the field (e.g., reconstruction-based Dreamer-like methods vs reconstruction-free JEPA-like methods), and (ii) the agent cannot collect corrective behaviors via online interaction, as the authors note. Within the limited evaluation, the claim that MAC-v2 enables long-horizon offline MBRL from pixels feels overclaimed to me. The authors must explain how MAC-v2 mitigates the above challenges I described via theoretical sketches or further experiments. For instance, by incorporating MAC-v2's encoder into MAC and including MAC in the main experiments, one can show that naively incorporating a vision encoder into MAC fails, while MAC-v2 can improve the performance.
Questions for authors
- MAC-v2 vs. MAC and evidence for the claim: How do you develop MAC-v2 based on MAC? What are the main differences, and what are the minor design choices that affect performance little? How does the suggested design support the claim that MAC-v2 can solve long-horizon model-based offline RL from pixels effectively?
- Experimental setup: Could you compare MAC w/ MAC-v2's vision encoder across benchmarks? As I conjectured in Critical Weaknesses, could you perform the following ablation studies: (i) the effect of determinism and stochasticity of reward/value/Q networks, and (ii) performance and wall-time comparison of using a one-step distillation policy for the flow-matching BC policy? For Q3 & Q4 in Section 5.3, could you provide more description of the setup? How do you remove the gradient from each objective to the encoder? How do you incorporate each representation learning objective into MAC-v2?
- Justifications for poor performance of MBRL baselines: Why do LOMPO and OfflineDV2 show poor performance across most benchmarks, which are almost corrupted? Is it attributed to outdated algorithms?
- Experimental results: For Q1 & Q2, how does the horizon length and actioin chunk length correlate with each other? For the same effective horizon length, which of the lengths contributes to better performance?
Clarification requests
- If MBRL baselines are outdated, could you consider including single-task DreamerV3(or DreamerV4) or TD-MPC2 as more recent MBRL baselines that support pixel-based offline learning, which do not fall into the scope of this paper but still provide a meaningful performance bottom lines (not during rebuttal, as future baselines)?
Minor requests.
- I think choosing block MDP rather than POMDP is overly strict in pixel-based RL, which is unusual as far as I know and may limit broader applications. Is there a special reason for choosing the block MDP?
- In Section 4.1, I think moving the notation for chunk data for actions and rewards (i.e., and ) from the pseudo table to the front would increase readability and coherence.
2 comments
Response to Reviewer quYf (2/2)
[Q3] Justifications for poor performance of MBRL baselines: Why do LOMPO and OfflineDV2 show poor performance across most benchmarks, which are almost corrupted? Is it attributed to outdated algorithms?
Thank you for raising this question. We found that LOMPO and OfflineDV2 perform especially poorly on the robotic manipulation tasks, including those in visual OGBench. One possible explanation is their reliance on uncertainty-based penalization, whereas MAC-v2 adopts rejection sampling inherited from MAC.
To investigate this, we conducted an additional ablation study in cube-double-task3, in which we replaced MAC-v2's behavior constraint from rejection sampling with MOPO-style uncertainty penalization, while retaining the remaining components.
| Success rate | MAC-v2 | LOMPO | OfflineDV2 | MAC-v2 + MOPO |
|---|---|---|---|---|
| cube-double-task3 | 52 ± 6 | 0 ± 0 | 0 ± 0 | 0 ± 0 |
Replacing rejection sampling with uncertainty penalization substantially reduces the performance. This suggests that the choice of behavior constraint mechanism is an important factor in the performance gap, rather than simply the age of the baseline algorithms.
We added this discussion in Appendix D.
[Q4] Experimental results: For Q1 & Q2, how does the horizon length and action chunk length correlate with each other? For the same effective horizon length, which of the lengths contributes to better performance?
Thank you for this question. To answer this question, we additionally conducted an ablation experiment in visual-puzzle-4x4-task1 with Hn=100 being fixed.
| (n, H) | Success % (4 seeds) |
|---|---|
| (1, 100) | 0 ± 0 |
| (4, 25) | 1 ± 1 |
| (10, 10) | 16 ± 16 |
| (25, 4) | 2 ± 1 |
| (100, 1) | 0 ± 0 |
The result shows that having a balance on and is important; too long action chunk size () degrades the performance as the policy becomes less reactive, but we want to be chunk size large enough so that the horizon length is small, as too long horizon length degrades the performance because of the accumulating error of the model-based value expansion.
We added this discussion in Appendix D.
[Q5] If MBRL baselines are outdated, could you consider including single-task DreamerV3(or DreamerV4) or TD-MPC2 as more recent MBRL baselines that support pixel-based offline learning, which do not fall into the scope of this paper but still provide a meaningful performance bottom lines (not during rebuttal, as future baselines)?
We would like to clarify that DreamerV3 and TD-MPC2 are primarily designed for online model-based RL (MBRL). In particular, TD-MPC2 has only been evaluated in the offline setting for state-based multi-task RL. Therefore, neither method is directly comparable to MAC-v2, which focuses on pixel-based offline RL. While DreamerV4 supports offline RL, its official implementation is not publicly available, and its reported training setup requires eight GPUs, making reproduction computationally expensive.
Nevertheless, to provide a more comprehensive comparison, we are implementing DreamerV4 based on the MMBench2 codebase and tuning TD-MPC2 in our setting. We will incorporate these results into the final manuscript as they become available.
[Q6] I think choosing block MDP rather than POMDP is overly strict in pixel-based RL, which is unusual as far as I know and may limit broader applications. Is there a special reason for choosing the block MDP?
We adopt the block MDP assumption because our current implementation of MAC-v2 does not incorporate a history-dependent representation module, such as the RSSM used in Dreamer models. However, we would like to emphasize that this assumption is not an inherent limitation of our method. MAC-v2 can, in principle, be extended to the POMDP setting by incorporating a history-dependent encoder that summarizes past observations and actions into a latent state. Since our method operates on learned latent representations, such an extension would not require fundamental changes to the MAC-v2 framework.
[Q7] In Section 4.1, I think moving the notation for chunk data for actions and rewards (i.e., and ) from the pseudo table to the front would increase readability and coherence.
Thank you for the suggestion, we updated the paper accordingly.
Response to Reviewer quYf (1/2)
Thank you for your constructive feedback. We address your questions and concerns in detail below and our new manuscript reflects your suggestions. Please refer to the updated PDF for new results and revisions. We believe your suggestions have substantially improved the quality and clarity of our paper, and we would greatly appreciate it if you could reconsider your score in light of the revisions.
[Q1, Q2] MAC-v2 vs. MAC and evidence for the claim: How do you develop MAC-v2 based on MAC? What are the main differences, and what are the minor design choices that affect performance little? How does the suggested design support the claim that MAC-v2 can solve long-horizon model-based offline RL from pixels effectively? Experimental setup: Could you compare MAC w/ MAC-v2's vision encoder across benchmarks? As I conjectured in Critical Weaknesses, could you perform the following ablation studies: (i) the effect of determinism and stochasticity of reward/value/Q networks, and (ii) performance and wall-time comparison of using a one-step distillation policy for the flow-matching BC policy?
Thank you so much for raising this concern. We strongly agree that the manuscript should more clearly distinguish the contributions of MAC-v2 from MAC.
The main contribution of MAC-v2 is extending long-horizon model-based value expansion of MAC to visual observations through learned latent world modeling. MAC operates on low-dimensional state observations, whereas MAC-v2 must learn both the latent representation and its dynamics from high-dimensional visual inputs. This setup requires to solve representation learning problem, which was not tackled in MAC. Our representation learning ablations further demonstrate the importance of MAC-v2’s representation learning objectives.
As the reviewer identified, we also introduce distributional value prediction and remove the one-step policy distillation used in MAC. However, these are secondary modifications rather than our main contribution. To show this, we conducted an additional ablation study on the improvements of MAC-v2 from MAC on cube-double-task3.
The results show that replacing distributional value prediction with scalar regression reduces performance and using a one-step policy degrades performance. However, removing the latent modeling of MAC-v2 severely drops the performance, showing that learning a latent world model is critical for MAC-v2’s success in visual environments. We note that MAC with 500k steps already uses 3x more compute budget compared to MAC-v2, due to pixel imagination.
We added a dedicated paragraph stating the contributions of MAC-v2 from MAC in Section 4. We also added the experimental result on MAC vs MAC-v2 in Section 5.3.
[Q2-2] For Q3 & Q4 in Section 5.3, could you provide more description of the setup? How do you remove the gradient from each objective to the encoder? How do you incorporate each representation learning objective into MAC-v2?
Thank you for pointing out the ambiguity in the description.
For the gradient ablations (Q3), we apply the stop-gradient operator to the latent representation before passing it to the corresponding module. For example, to remove the policy gradient, we train the policy using sg(z_t) instead of z_t in the flow-matching objective. This allows the policy to be trained normally while preventing its loss from updating the encoder. For the alternative representation learning objectives (Q4), we train the encoder using the representation learning objective instead of MAC-v2's training objectives. For example, in the SimCLR ablation, we first train the encoder using the SimCLR objective, then freeze its parameters and train the world model, policy, and value functions on top of the resulting representation.
We added these implementation details to the Q3 and Q4 discussions in Section 5.3.