- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3(+1)
Read review
Paper summary
This paper introduces multi-scale predictive representations, which try to predict different properties of offline behavior trajectories with the goal of finding a representation suitable for downstream offline goal-conditioned RL. The paper proposes to combine different alignment objectives in order to learn a good representation. Namely, the proposed goal representations is based on dynamical (next-state prediction), temporal (return prediction) and behavioral (inverse model, goal dynamics prediction) alignment. The authors give intuition why these alignment objectives might be helpful for offline GCRL and show the advantage of their representation empirically on OGBench. Ablation studies underlie the importance of each representation objective, showing that the required alignment objectives are task specific.
Critical strengths
- Combining different representation objectives in order to structure the representation allows for a general approach that is agnostic to downstream RL methods
- Convincing empirical results. The approach beats multiple baselines on a variety of OGBench tasks. The additional analysis of failure cases shows limitations of the representation learning approach itself. The ablations show that each objective has task-specific benefits, indicating that a combination of alignment objectives is required to solve different tasks.
- The paper has a good structure and is well written
Critical weaknesses
- W1: While intuition is given for the different alignment objectives, an overarching principle is missing and it remains unclear which of these objectives contribute to the final performance or are redundant. For instance, the dynamic losses were never ablated in total isolation (i.e. no dynamics at all). In summary, the approach feels like a combination of different representation objectives without a clear overall picture.
- W2: The dynamics losses and latent unrolling is closely related to model-based RL while the approach is only compared against model-free GCRL approaches.
- W3: Novelty unclear. The architecture and objectives are quite similar to MR.Q (as acknowledged by the authors) and extend the approach to the GCRL setting with additional behavioral alignment.
- W4: The idea should be agnostic to the downstream GCRL approach as stated by the authors, however only TD3+BC as backbone is considered.
Questions for authors
- Q1: How important are the dynamics losses? Especially as they resemble a latent world model (W3) and are not ablated fully (W2). An additional ablation without any dynamic losses could be helpful.
- Q2: What is the contribution of this paper? The motivation of the different objectives is not clear (W1) and the approach seems very close to MR.Q (W3).
- Q3: Is the approach agnostic to the down-stream GCRL approach? Could the representation be used with e.g. CRL or GCIQL?
5 comments
(2/3) Response to Reviewer ccxi
W2. "The dynamics losses and latent unrolling is closely related to model-based RL while the approach is only compared against model-free GCRL approaches."
We added MAC (Park et al., 2025)[1], a recent offline model-based method evaluated on OGBench. MAC learns a dynamics model over action chunks and trains its value function on model rollouts. We implemented MAC in the same goal-conditioned setting as the other baselines: the same data, network sizes, DDPG+BC policy extraction, and evaluation as OGBench. These are our runs in this shared setting. Details are in App. E.5 (all tasks in Table 9):
| Environment | GCTD3 | MAC | MSPR |
|---|---|---|---|
antmaze-medium | 79 | 68 | 99 |
antmaze-large | 25 | 36 | 98 |
antmaze-giant | 0 | 1 | 68 |
humanoidmaze-medium | 26 | 6 | 67 |
humanoidmaze-large | 6 | 0 | 58 |
| Avg. (12) | 46 | 14 | 67 |
The two methods use a dynamics model in different ways. MAC generates rollouts with its model and trains the critic on them, so the goal signal has to travel back along predicted trajectories. MSPR uses its dynamics heads, including the latent unroll, only to train the encoder. The agent never uses them to act or to compute value targets, so model errors do not enter the critic, and the distance to the goal comes directly from the Monte-Carlo return. The gap grows with the horizon (Figure 7): 68 vs. 99 on antmaze-medium, 1 vs. 68 on antmaze-giant. In this setting MAC is also low on manipulation (App. E.5, Table 9).
W3 / Q2. "Novelty unclear. The architecture and objectives are quite similar to MR.Q" / "What is the contribution of this paper?"
MSPR builds on MR.Q, and the paper says so. To test whether the MR.Q objectives are enough for goal-reaching, we trained the same GCTD3 agent (, same BC weights and discounts) on two other encoders: one trained with MR.Q's losses adapted to goals (GC-MR.Q), and the MSPR network with no predictive losses (No-aux) (selected tasks; all tasks in Table 8, App. E.4):
| Environment | GCTD3 | No-aux | GC-MR.Q | MSPR |
|---|---|---|---|---|
antmaze-large | 25 | 0 | 1 | 98 |
antmaze-giant | 0 | 0 | 0 | 68 |
humanoidmaze-medium | 26 | 24 | 21 | 67 |
humanoidmaze-large | 6 | 0 | 4 | 58 |
puzzle-4x4 | 88 | 83 | 87 | 93 |
| Avg. (12) | 46 | 33 | 38 | 67 |
- GC-MR.Q. GC-MR.Q reaches 38 on average. Here the reward predictor only estimates the single-step binary reward (whether the agent has reached the goal), so its latent space gets no direct signal of the distance to the goal. GC-MR.Q therefore fails on the long-horizon tasks, the same pattern as removing , while short-horizon manipulation stays close to MSPR.
- No-aux. The MSPR network, all four loss weights set to 0, encoder trained by the critic's TD loss. No-aux reaches 33 on average, below the backbone (46). Its critic's value falls to the lower bound on both
antmazetasks in Figure 6 (App. E.4) within 75k steps, and does not recover. This shows that the gain comes from MSPR's predictive losses, not from the network.
How the analysis differs. MR.Q's latent-MDP results are for a single-task policy that reads only ; they ask the latent to predict the reward and the next state. A goal-conditioned policy reads the state and the goal together, and a latent can predict reward and next state well while still does not show the goal-directed action (Sec. 4, "The three terms are distinct"). Prop. 3 states this extra condition: if is small, there is a policy on whose one-step gap to the optimal action is bounded. MR.Q's analysis has no such condition. Prop. 2 also differs: it uses the Monte-Carlo return, which carries the distance to the goal, instead of the one-step reward.
Our contribution is therefore:
- which predictions help a goal-conditioned encoder under sparse reward, with one loss and one upper bound for each (W1);
- controlled comparisons that place the gain in these losses, not in the network (No-aux), the MR.Q losses (GC-MR.Q), or model-based value learning (MAC);
- evidence that the same encoder raises the average of other backbones (GCIQL 45 to 56, GCBC 10 to 16; W4).
Reference
- Park, Kwanyoung, et al. "Scalable offline model-based RL with action chunks." International Conference on Learning Representations. Vol. 2026. (arXiv:2512.08108)
(3/3) Response to Reviewer ccxi
W4 / Q3. "Only TD3+BC as backbone is considered" / "Could the representation be used with e.g. CRL or GCIQL?"
Yes. We kept the encoder and its four losses unchanged and replaced GCTD3 with GCIQL or GCBC. Each agent keeps its own hyperparameters (selected tasks; all tasks in Table 7, App. E.3):
| Environment | GCBC | + MSPR | GCIQL | + MSPR | GCTD3 | MSPR |
|---|---|---|---|---|---|---|
antmaze-large | 24 | 42 | 40 | 86 | 25 | 98 |
antmaze-giant | 0 | 10 | 1 | 63 | 0 | 68 |
humanoidmaze-medium | 8 | 24 | 34 | 59 | 26 | 67 |
scene | 5 | 10 | 35 | 60 | 42 | 45 |
pointmaze-medium | 9 | 13 | 66 | 43 | 77 | 54 |
cube-single | 6 | 6 | 60 | 42 | 63 | 65 |
| Avg. (12) | 10 | 16 | 45 | 56 | 46 | 67 |
- GCIQL rises from 45 to 56, with the largest gains on long-horizon locomotion. It loses on
pointmazeandcube, where its own networks are already strong; we report this in App. E.3. - GCBC rises from 10 to 16. GCBC has no critic and uses only , so it cannot use the distance information that puts into , and a smaller gain is expected.
- GCIQL and GCTD3 both learn from state and action, so they use the whole encoder ( and ), and that is where the encoder helps most.
Changes in the paper
- Sec. 2: how MSPR differs from MR.Q (Monte-Carlo return vs. one-step reward) and from MAC (dynamics used only to train the encoder).
- Sec. 4: each proposition is an upper bound, not a guarantee on the return; the meaning of "multi-scale".
- Sec. 5 (RQ1): the role of each loss; No-dyn. Sec. 5 (RQ2): No-aux, GC-MR.Q, and other backbones.
- App. E.3: MSPR with GCIQL and GCBC (Table 7).
- App. E.4: No-aux, No-dyn, GC-MR.Q (Table 8), and No-aux training curves (Figure 6).
- App. E.5: comparison with MAC (Table 9, Figure 7).
(1/3) Response to Reviewer ccxi
We thank the reviewer for the careful review. We answer each point below. Changes in the revised paper are shown in blue.
W1. "An overarching principle is missing and it remains unclear which of these objectives contribute to the final performance or are redundant."
The principle is simple. The MSPR encoder gets no gradient from the actor or the critic. So whatever the agent needs must already be in the representation, and each loss puts one piece of it there:
Propositions 1 to 3 bound the error that each missing piece can cause, and none of the three error terms controls another (Sec. 4, "The three terms are distinct"). So in the analysis no condition is redundant. The ablation agrees (selected tasks; all tasks in Table 2):
antmaze-gianthumanoidmaze-mediumhumanoidmaze-largepointmaze-largecube-singlevisual-puzzle-3x3humanoidmazetasks andvisual-puzzle-3x3drop to 0.pointmaze-largeandcube-single, where the next state depends on small details (a wall, the contact with the cube).Sec. 5 (RQ1) now explains each drop by the role of the loss.
Q1. "How important are the dynamics losses? … An additional ablation without any dynamic losses could be helpful."
We ran it. No-dyn removes both Ldyn and Lg-dyn and keeps the other two losses. For reference, No-aux removes all four losses (same network, encoder trained by the critic's TD loss). Same agent and hyperparameters (Table 8, App. E.4):
pointmaze-mediumpointmaze-largeantmaze-mediumantmaze-largeantmaze-gianthumanoidmaze-mediumhumanoidmaze-largecube-singlecube-doublescenepuzzle-3x3puzzle-4x4The dynamics losses matter most on long horizons. Without them,
antmaze-giantdrops from 68 to 32, and bothhumanoidmazetasks fall to near 0. On shorter tasks such asantmaze-mediumandpuzzle-3x3, the other two losses already do most of the work. The three variants give a clear order: no losses 33, all but the dynamics losses 47, all four losses 67.