acceptodds
Under review as a conference paper at ICLR 2027

Latent Successor Models

Abstract

An effective reinforcement learning (RL) agent must reason about both the immediate outcomes and long-term consequences of its actions. This challenge is especially pronounced when learning from pixel observations. Many prior RL methods learn visual representations for control by combining representation learning with reward prediction and value estimation. How can we acquire such representations without directly training them to predict rewards or values? In this paper, we propose *Latent Successor Models* (LSM) an approach that learns actionable visual representations through generative modeling of future outcomes at arbitrary horizons and behavioral cloning. This removes the need for direct reward and value prediction or pixel reconstruction when learning these representations. By sampling distant future outcomes under a policy, we construct targets for value estimation without predicting every intermediate state. Our analysis shows that directly predicting future outcomes yields more accurate value estimates at longer horizons. Experiments across a range of visual offline RL benchmarks ( tasks) demonstrate substantial improvements over prior model-free and model-based approaches, with particularly large gains on challenging long-horizon tasks.

Primary area:
reinforcement learning
Keywords:
offline reinforcement learning, visual reinforcement learning, representation learning, successor model, flow matching

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_RgAQOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper introduces Latent Successor Models (LSM) for offline visual reinforcement learning. The main successor model adapts a flow matching based method predicts future latent states with a given horizon and current state. The successor model is trained in a boostrapped method whose target network is the EMA version of the online network. The training target is only the encoded future state, without reward or value. This enables critic model to learn truncated λ\lambda-return using latent states directly sampled from the successor model at selected horizons, without generating every intermediate state. The experiments show that the proposed method can predict future latent and values in a long horizon with a much smaller error compare to autoregressive methods, which results in a higher success rate than other baselines.

Critical strengths

The presentation of the paper is generally clear. The core mechanism using bootstrapped flow matching as a successor model is novel and the proposed method achieves better performance than other baselines over multiple benchmarks.

Critical weaknesses

The main concern is about the missing ablations, for example, "Bootstrapped velocity at all noise levels" (Eq. 8) vs clean-future-state bootstrap (Eq. 7, LUHM-style) is the claimed main novelty, but the proposed method is different from LUHM in many aspects including the reward estimator and actor desgin. The authors are encouraged to conduct more detailed ablations to help understand the effect of each module.

Questions for authors

No

2 comments

We thank the reviewer for the detailed feedback and suggestions. The reviewer raised the question about the contributions of the three different components between LSM and LUHM, which we have addressed by running additional experiments to evaluate three variants of LSM, each replacing only its training objective, return estimator, or policy with the corresponding component from LUHM. We conduct experiments on visual-cube-double-task3\texttt{visual-cube-double-task3} and visual-scene-task3\texttt{visual-scene-task3} using the same training setup and evaluation protocols as in Sec. 5.3. Below, we report success rates as the mean and standard deviation across 4 random seeds.

Methodvisual-cube-double-task3\texttt{visual-cube-double-task3}visual-scene-task3\texttt{visual-scene-task3}Average
LSM (Ours)53±1953 \pm 1976±1476 \pm 1465±965 \pm 9
LSM w/ coupled CFM objective39±1239 \pm 1267±967 \pm 953±1053 \pm 10
LSM w/ ν\nu-return estimator49±749 \pm 778±1078 \pm 1064±964 \pm 9
LSM w/ deterministic actor4±14 \pm 10±10 \pm 12±12 \pm 1

These results suggest that policy extraction and the bootstrapped velocity objective contribute to LSM’s performance, while the choice of return estimator has little effect on these two tasks. This observation is compatible with the discussion in Chung et al. (2026) [1], where the λ\lambda-return is a special case of the ν\nu-return. Are there additional ablation experiments that the reviewer would like to see?

[1] Chung, H., Lee, J., & Oh, S. (2026). Offline reinforcement learning with universal horizon models. arXiv preprint arXiv:2605.15603.

I appreciate the authors new ablations and I will update the recommendation to weak acceptance.

Reviewer_edZwOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents Latent Successor Models (LSM), an offline RL method for learning from pixel observations. The key idea is to learn a visual representation without pixel reconstruction or direct reward/value supervision of the encoder, by jointly training the encoder with a horizon-conditioned flow-based successor model in latent space and a flow behavioral-cloning policy. The successor model is trained with a bootstrapped flow-matching objective that uses the next latent state’s velocity as the target, and it is then used to sample future latent states directly at chosen horizons to construct Monte Carlo λ\lambda-return targets for critic learning, avoiding step-by-step rollouts. The paper reports that direct sampling yields lower future-prediction and value-estimation errors than autoregressive rollout, that the learned representation supports more accurate value estimation than alternative encoder objectives, and that LSM outperforms prior model-free and model-based methods on 38 visual offline RL tasks across Visual OGBench, MMBench2, and Visual Robomimic, with the largest gains on long-horizon manipulation tasks.

Critical strengths
  1. Clear and well-motivated method. The paper combines bootstrapped flow matching, horizon conditioning, and self-predictive representation learning into a single coherent objective, and shows in Appendix A.1 that the objective recovers the horizon-conditioned successor measure. The three challenges in Sec. 4.1 map directly onto the design choices in Secs. 4.2-4.4, making the method easy to follow.

  2. Thorough evaluation with strong results on long-horizon tasks. The method is evaluated on 38 tasks across three benchmarks against 11 model-free and model-based baselines. The gains on long-horizon manipulation are large and well outside the variance as shown in Table 1.

  3. Supporting analyses. The experiments show that direct sampling reduces future-prediction and value-estimation errors compared with autoregressive rollout (Sec. 5.1), and compare the learned representation against four alternative encoder objectives (Sec. 5.2). The appendix further ablates the gradient sources for the encoder and the successor-model architecture.

Critical weaknesses
  1. The source of the gain over LUHM is not isolated: LUHM is the closest baseline, it learns a horizon-conditioned latent successor model with the same encoder and successor-model architecture as latent successor models (LSM), and differs in three things: the training objective, the return estimate, and the policy parametrization. The paper’s technical contribution is the bootstrapped velocity objective (Eq. 8), but the 54 vs. 13 gap on visual-cube-double and 88 vs. 22 on visual-scene (Table 1) could come from the λ\lambda-return construction or the rejection-sampling policy extraction instead. Without an ablation that changes each of these components separately, the main claim that the objective is what matters is unsupported.

  2. The supporting analyses are limited, and the main claim is overstated: The analyses in Secs. 5.1 and 5.2 are conducted on a single task (visual-cube-double-task3). Sec. 5.1 only compares LSM with its own autoregressive variant, not with the one-step model baselines, and Sec. 5.2 measures value-estimation error rather than policy performance. The abstract claims "substantial improvements over prior model-free and model-based approaches," but the results in Table 1 do not fully support this: LSM performs worse than FQL on dmcontrol and LOMPO on maniskill, achieves 0 on visual-puzzle-4x4 where QC reaches 20, and shows high variance on visual-puzzle-3x3 (30 ±\pm 11). The improvements are mainly on long-horizon manipulation tasks, and the claims should be scoped to that setting.

Questions for authors
  1. Isolating the contribution: LSM differs from LUHM in the training objective (Eq. 8 vs. coupled CFM), the return estimate (λ\lambda-return vs. ν\nu-return), and the policy extraction (rejection sampling vs. deterministic actor). Could the authors ablate these three components individually (e.g., LUHM with LSM's objective only) to show which one accounts for the gap in Table 1?

  2. Generality of the analyses: Sections 5.1 and 5.2 use only visual-cube-double-task3. Could the authors extend these analyses to at least one task from a different domain (e.g., visual-scene or an MMBench2 task), and additionally report final success rates for each encoder objective in Section 5.2, so that the representation comparison is tied to policy performance rather than only value-estimation error?

  3. Comparison against one-step models: Section 5.1 compares direct sampling with LSM's own autoregressive variant. Could the authors include the one-step model baselines (e.g., OffDV2 or LOMPO) in the same future-prediction and value-estimation comparison?

  4. Failure cases and hyperparameter sensitivity: What explains the weaker results on dmcontrol, maniskill, and visual-puzzle-4x4, and the high variance on visual-puzzle-3x3? Also, how sensitive is LSM to the final horizon HfinalH_{\text{final}} and λfinal\lambda_{\text{final}}, which differ across benchmarks (40 vs. 100), and were these tuned per benchmark?

5 comments

We thank the reviewer for the detailed feedback and suggestions. The reviewer raised questions about (1) the contributions of the three components that differ between LSM and LUHM, (2) the breadth of the analyses in Secs. 5.1 and 5.2 and their connection to policy performance, and (3) failure cases and hyperparameter sensitivity. To address (1), we ablate each component in LSM individually. For (2), we extend the analyses to an additional Visual OGBench task, visual-scene-task3\texttt{visual-scene-task3}, and observe similar trends. We also report policy performance alongside value estimation error for each encoder objective. For (3), we add diagnostic analyses and ablations to investigate the failure cases and the sensitivity to different hyperparameters. Together with the discussion below, do these revisions fully address the reviewer’s concerns?

Could the authors ablate these three components individually (e.g., LUHM with LSM's objective only) to show which one accounts for the gap in Table 1?

Yes. To study the contributions of the three components that differ between LSM and LUHM, we ran additional experiments to evaluate three variants of LSM, each replacing only its training objective, return estimator, or policy with the corresponding component from LUHM. We conduct experiments on visual-cube-double-task3\texttt{visual-cube-double-task3} and visual-scene-task3\texttt{visual-scene-task3} using the same training setup and evaluation protocols as in Sec. 5.3. Below, we report success rates as the mean and standard deviation across 4 random seeds.

Methodvisual-cube-double-task3\texttt{visual-cube-double-task3}visual-scene-task3\texttt{visual-scene-task3}Average
LSM (Ours)53±1953 \pm 1976±1476 \pm 1465±965 \pm 9
LSM w/ coupled CFM objective39±1239 \pm 1267±967 \pm 953±1053 \pm 10
LSM w/ ν\nu-return estimator49±749 \pm 778±1078 \pm 1064±964 \pm 9
LSM w/ deterministic actor4±14 \pm 10±10 \pm 12±12 \pm 1

Replacing LSM’s objective with coupled CFM reduces average success from 6565 to 5353, while using the ν\nu-return estimator yields similar success (6464). Replacing rejection sampling with a deterministic actor causes the largest drop, from 6565 to 22. These results suggest that policy extraction and the bootstrapped velocity objective contribute to LSM’s performance, while the choice of return estimator has little effect on these two tasks. This observation is compatible with the discussion in Chung et al. (2026) [1], where the λ\lambda-return is a special case of the ν\nu-return.

Could the authors extend these analyses to at least one task from a different domain (e.g., visual-scene or an MMBench2 task), and additionally report final success rates for each encoder objective in Section 5.2, so that the representation comparison is tied to policy performance rather than only value-estimation error?

As suggested by the reviewer, we have extended the ablation studies in Secs. 5.1 and 5.2 to visual-scene-task3\texttt{visual-scene-task3}, following the same experimental setup. We also report final policy success rates for each encoder objective in Sec. 5.2. Please see the revised Figs. 4 and 5 for the aggregated results, with accompanying descriptions highlighted in blue. Below, we report the latent MMD$^2$ and value estimate MSE averaged over both tasks in Sec. 5.1, and the value estimate MSE and the success rate for each task in Sec. 5.2.

Sec. 5.1: Long-horizon future prediction and value estimation. Results are averaged over visual-cube-double-task3\texttt{visual-cube-double-task3} and visual-scene-task3\texttt{visual-scene-task3}. Horizons are measured in environment steps.

HorizonLatent MMD$^2$ (×10−2\times 10^{-2}): LSM (direct)Latent MMD$^2$ (×10−2\times 10^{-2}): LSM (autoregressive)Value estimate MSE (×105\times 10^{5}): LSM (direct)Value estimate MSE (×105\times 10^{5}): LSM (autoregressive)
205.29±0.635.29 \pm 0.634.88±0.474.88 \pm 0.470.62±0.080.62 \pm 0.080.63±0.080.63 \pm 0.08
606.44±0.266.44 \pm 0.266.98±0.346.98 \pm 0.340.64±0.090.64 \pm 0.090.96±0.100.96 \pm 0.10
805.72±0.205.72 \pm 0.207.16±0.437.16 \pm 0.430.65±0.090.65 \pm 0.091.87±0.271.87 \pm 0.27
1205.19±0.235.19 \pm 0.238.44±1.058.44 \pm 1.050.65±0.090.65 \pm 0.094.29±1.134.29 \pm 1.13
1604.92±0.224.92 \pm 0.229.33±1.269.33 \pm 1.260.64±0.090.64 \pm 0.097.20±1.867.20 \pm 1.86
2005.05±0.265.05 \pm 0.269.97±1.399.97 \pm 1.390.64±0.090.64 \pm 0.0910.17±2.4310.17 \pm 2.43

Sec. 5.2: Encoder objectives and downstream policy performance.

Encoder objectivevisual-cube-double-task3\texttt{visual-cube-double-task3}: value estimate MSEvisual-scene-task3\texttt{visual-scene-task3}: value estimate MSEvisual-cube-double-task3\texttt{visual-cube-double-task3}: success ratevisual-scene-task3\texttt{visual-scene-task3}: success rate
LSM (Ours)(1.33±0.18)×105\mathbf{(1.33 \pm 0.18) \times 10^5}(5.87±1.48)×104\mathbf{(5.87 \pm 1.48) \times 10^4}53±19\mathbf{53 \pm 19}76±14\mathbf{76 \pm 14}
Reconstruction(2.84±0.23)×105(2.84 \pm 0.23) \times 10^5(2.61±0.65)×105(2.61 \pm 0.65) \times 10^58±28 \pm 20±00 \pm 0
SimCLR(4.61±0.47)×105(4.61 \pm 0.47) \times 10^5(4.03±0.65)×105(4.03 \pm 0.65) \times 10^50±00 \pm 04±24 \pm 2
One-step BYOL(4.06±0.05)×105(4.06 \pm 0.05) \times 10^5(1.84±0.15)×106(1.84 \pm 0.15) \times 10^60±00 \pm 00±00 \pm 0
Bisimulation(3.07±0.16)×105(3.07 \pm 0.16) \times 10^5(2.16±0.04)×106(2.16 \pm 0.04) \times 10^60±00 \pm 00±00 \pm 0

Across the two tasks, direct sampling from LSM achieves lower long-horizon prediction and value estimation errors than autoregressive rollouts. In the encoder comparison, LSM achieves the lowest mean MSE and highest mean success on both tasks. Together, these results support long-horizon prediction accuracy of LSM and show that its learned representations are good for both value estimation and downstream control on the evaluated tasks.

Could the authors include the one-step model baselines (e.g., OffDV2 or LOMPO) in the same future-prediction and value-estimation comparison?

Following the reviewer’s suggestion, we add a controlled comparison with a separately trained one-step transition model in Sec. 5.1, following the same experimental setup. Specifically, we freeze LSM’s encoder and train a separate one-step latent transition model in its latent space. We then train a critic using λ\lambda-return targets constructed from its autoregressive rollouts.

For future prediction, we compute MMD$^2$ between the distributions of future latent states generated by autoregressive rollouts and the corresponding ground-truth distributions. For value estimation, we measure the MSE between the truncated λ\lambda-return estimates and ground-truth discounted returns under the reference policy. Please see the revised Fig. 4 for the additional results, with accompanying descriptions highlighted in blue.

HorizonLatent MMD$^2$ (×10−2\times 10^{-2}): LSM (direct)Latent MMD$^2$ (×10−2\times 10^{-2}): LSM (autoregressive)Latent MMD$^2$ (×10−2\times 10^{-2}): One-step model (autoregressive)Value estimate MSE (×105\times 10^{5}): LSM (direct)Value estimate MSE (×105\times 10^{5}): LSM (autoregressive)Value estimate MSE (×105\times 10^{5}): One-step model (autoregressive)
205.29±0.635.29 \pm 0.634.88±0.474.88 \pm 0.473.37±0.343.37 \pm 0.340.62±0.080.62 \pm 0.080.63±0.080.63 \pm 0.083.53±0.963.53 \pm 0.96
606.44±0.266.44 \pm 0.266.98±0.346.98 \pm 0.345.60±0.515.60 \pm 0.510.64±0.090.64 \pm 0.090.96±0.100.96 \pm 0.103.52±0.903.52 \pm 0.90
805.72±0.205.72 \pm 0.207.16±0.437.16 \pm 0.436.23±0.586.23 \pm 0.580.65±0.090.65 \pm 0.091.87±0.271.87 \pm 0.273.53±0.903.53 \pm 0.90
1205.19±0.235.19 \pm 0.238.44±1.058.44 \pm 1.056.78±1.036.78 \pm 1.030.65±0.090.65 \pm 0.094.29±1.134.29 \pm 1.133.54±0.913.54 \pm 0.91
1604.92±0.224.92 \pm 0.229.33±1.269.33 \pm 1.267.41±0.967.41 \pm 0.960.64±0.090.64 \pm 0.097.20±1.867.20 \pm 1.863.55±0.923.55 \pm 0.92
2005.05±0.265.05 \pm 0.269.97±1.399.97 \pm 1.397.06±0.807.06 \pm 0.800.64±0.090.64 \pm 0.0910.17±2.4310.17 \pm 2.433.55±0.933.55 \pm 0.93

The one-step model achieves lower latent MMD$^2$ at horizons of 2020 and 6060 steps, while LSM’s direct predictions are more accurate at the longer evaluated horizons (80–200 steps). LSM also achieves lower value estimation MSE across all tested horizons. These results suggest that LSM produces more accurate long-horizon future prediction and value estimation than the one-step model when both use the same visual representation.

Also, how sensitive is LSM to the final horizon HfinalH_\text{final} and λfinal\lambda_\text{final}, which differ across benchmarks (40 vs. 100), and were these tuned per benchmark?

To study LSM’s sensitivity to HfinalH_\text{final} and λfinal\lambda_\text{final}, we conduct ablation experiments on selected tasks from each benchmark, varying Hfinal∈10,40,100H_\text{final} \in \\{10, 40, 100\\} and λfinal∈0.8,0.95,0.99\lambda_\text{final} \in \\{0.8, 0.95, 0.99\\}. We evaluate visual-cube-double-task3\texttt{visual-cube-double-task3} and visual-scene-task3\texttt{visual-scene-task3} from Visual OGBench, pygame-dungeon-explorer1\texttt{pygame-dungeon-explorer1} and cup-catch-var1\texttt{cup-catch-var1} from MMBench2, and visual-square\texttt{visual-square} from Visual Robomimic. We select hyperparameters separately for each benchmark based on average performance across its selected tasks, aggregating results over 4 random seeds. We have revised Appendix E.2 and have added Appendix D.4 to clarify the selection procedure.

HfinalH_\text{final} ablations:

HfinalH_\text{final}Visual OGBench (2 tasks)MMBench2 (2 tasks)Visual Robomimic (1 task)
1046±846 \pm 836±636 \pm 673±573 \pm 5
4065±9\mathbf{65 \pm 9}56±156 \pm 172±8\mathbf{72 \pm 8}
10068±268 \pm 257±7\mathbf{57 \pm 7}67±167 \pm 1

λfinal\lambda_\text{final} ablations:

λfinal\lambda_\text{final}Visual OGBench (2 tasks)MMBench2 (2 tasks)Visual Robomimic (1 task)
0.860±260 \pm 259±359 \pm 366±466 \pm 4
0.9565±9\mathbf{65 \pm 9}57±7\mathbf{57 \pm 7}72±8\mathbf{72 \pm 8}
0.9959±659 \pm 653±253 \pm 272±472 \pm 4

Increasing HfinalH_\text{final} from 1010 to 4040 improves performance on Visual OGBench and MMBench2, with small additional gains at 100100, while Visual Robomimic does not benefit from longer horizons. We retain Hfinal=40H_\text{final}=40 on Visual OGBench and Visual Robomimic, which use larger action chunks of size 1010, and use Hfinal=100H_\text{final}=100 on MMBench2, which uses smaller action chunks of size 55. We select shared λfinal=0.95\lambda_\text{final}=0.95 because it performs generally well across all three benchmarks.

Reviewer_7d5xOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper introduces Latent Successor Models (LSM) to learn visual representations without directly training to predict rewards or values. LSM constructs long-horizon return targes for critic learning without generating every intermediate state. Evaluating across a wide range of benchmarkds demostrates the substantial improvement of their method.

Critical strengths
  • The authors proposed a principled way to get long-horizon targets withouth compounding rollouts. LSM removes the need for direct reward and value prediction or pixel reconstruction.
  • The method shows strong emperical gains.
Critical weaknesses

The paper presents the velocity-bootstrap objective in Eq. 8 as a key methodological contribution. However, the comparison with LUHM also changes the return estimator and action-selection procedure, despite using the same encoder and successor-model architectures. Is there a run that changes only the objective and what would be the results?

Questions for authors

It seems discount factors vary across models and benchmarks. Is this a specific design? On Visual OGBench and Robomimic, LSM uses γ\gamma = 0.999 while the model-free and one-step baselines use γ\gamma = 0.99. If all methods use the same γ\gamma, will LSM keep its advantage? Also, how were γ\gamma, H_final, λ\lambda, and K chosen for LSM, and were they tuned on the evaluation tasks?

3 comments

We thank the reviewer for the detailed feedback and suggestions. The reviewer raised questions about (1) isolating the contribution of LSM’s training objective and (2) hyperparameter sensitivity and selection. We attempt to address (1) by ablating the three components that differ between LSM and LUHM individually. To address (2), we add ablations to investigate the sensitivity to different hyperparameters and clarify the selection criteria. Together with the discussion below, do these revisions fully address the reviewer’s concerns?

However, the comparison with LUHM also changes the return estimator and action-selection procedure, despite using the same encoder and successor-model architectures. Is there a run that changes only the objective, and what would be the results?

To study the contributions of the three components that differ between LSM and LUHM, we ran additional experiments to evaluate three variants of LSM, each replacing only its training objective, return estimator, or policy with the corresponding component from LUHM. We conduct experiments on visual-cube-double-task3\texttt{visual-cube-double-task3} and visual-scene-task3\texttt{visual-scene-task3} using the same training setup and evaluation protocols as in Sec. 5.3. Below, we report success rates as the mean and standard deviation across 4 random seeds.

Methodvisual-cube-double-task3\texttt{visual-cube-double-task3}visual-scene-task3\texttt{visual-scene-task3}Average
LSM (Ours)53±1953 \pm 1976±1476 \pm 1465±965 \pm 9
LSM w/ coupled CFM objective39±1239 \pm 1267±967 \pm 953±1053 \pm 10
LSM w/ ν\nu-return estimator49±749 \pm 778±1078 \pm 1064±964 \pm 9
LSM w/ deterministic actor4±14 \pm 10±10 \pm 12±12 \pm 1

Replacing LSM’s objective with LUHM’s coupled CFM reduces average success from 6565 to 5353, while using the ν\nu-return estimator in LUHM yields similar success (6464). Replacing rejection sampling in LSM with a deterministic actor in LUHM causes the largest drop, from 6565 to 22. These results suggest that LSM benefits from its bootstrapped velocity objective and rejection-sampling policy compared with the corresponding components in LUHM, while the choice of return estimator has little effect on these two tasks. This observation is compatible with the discussion in Chung et al. (2026) [1], where the λ\lambda-return is a special case of the ν\nu-return.

It seems discount factors vary across models and benchmarks. Is this a specific design? On Visual OGBench and Robomimic, LSM uses γ=0.999\gamma = 0.999 while the model-free and one-step baselines use γ=0.99\gamma = 0.99.

To ensure fair comparisons, we have run additional experiments sweeping γ∈0.99,0.999\gamma \in \\{0.99, 0.999 \\} for both LSM and two strong model-free baselines, FQL and QC. We evaluate on visual-cube-double-task3\texttt{visual-cube-double-task3} and visual-scene-task3\texttt{visual-scene-task3} from Visual OGBench, following the training setup and evaluation protocols in Sec. 5.3. Below, we report success rates averaged across the two tasks and four random seeds. Bold entries indicate each method’s default discount factor in our original experiments.

γ\gammaLSM (Ours)QCFQL
0.9964±464 \pm 445±5\mathbf{45 \pm 5}10±3\mathbf{10 \pm 3}
0.99965±9\mathbf{65 \pm 9}42±642 \pm 60±00 \pm 0

The performance of LSM remains similar across the two discount factors, while increasing γ\gamma reduces the mean performance of both baselines. These results show that LSM’s average advantage over QC and FQL on the selected tasks persists when the discount factor is matched.

If all methods use the same γ\gamma, will LSM keep its advantage? Also, how were γ\gamma, HfinalH_\text{final}, λ\lambda, and KK chosen for LSM, and were they tuned on the evaluation tasks?

We select γ\gamma and HfinalH_\text{final} separately for each benchmark, while using shared values λfinal=0.95\lambda_\text{final}=0.95, Ktrain=8K_\text{train}=8, and Keval=32K_\text{eval}=32, chosen to perform generally well across all three benchmarks. We have revised Appendix D.2 and have added Appendix C.4 to clarify the selection procedure.

To study LSM’s sensitivity to different hyperparameters, we conduct ablation experiments on selected tasks from each benchmark, varying γ∈0.9,0.99,0.999\gamma \in \\{ 0.9, 0.99, 0.999 \\}, Hfinal∈10,40,100H_\text{final} \in \\{10, 40, 100\\}, λfinal∈0.8,0.95,0.99\lambda_\text{final} \in \\{0.8, 0.95, 0.99\\}, Ktrain∈4,8,16K_\text{train} \in \\{ 4, 8, 16 \\}, and Keval∈8,16,32,64K_\text{eval} \in \\{8, 16, 32, 64 \\}. We evaluate visual-cube-double-task3\texttt{visual-cube-double-task3} and visual-scene-task3\texttt{visual-scene-task3} from Visual OGBench, pygame-dungeon-explorer1\texttt{pygame-dungeon-explorer1} and cup-catch-var1\texttt{cup-catch-var1} from MMBench2, and visual-square\texttt{visual-square} from Visual Robomimic. We select hyperparameters separately for each benchmark based on average performance across its selected tasks, aggregating results over 4 random seeds. Below, we bold the results for the default hyperparameters on each benchmark.

γ\gamma ablations:

γ\gammaVisual OGBench (2 tasks)MMBench2 (2 tasks)Visual Robomimic (1 task)
0.940±240 \pm 256±256 \pm 253±753 \pm 7
0.9964±464 \pm 457±7\mathbf{57 \pm 7}64±664 \pm 6
0.99965±9\mathbf{65 \pm 9}52±652 \pm 672±8\mathbf{72 \pm 8}

HfinalH_\text{final} ablations:

HfinalH_\text{final}Visual OGBench (2 tasks)MMBench2 (2 tasks)Visual Robomimic (1 task)
1046±846 \pm 836±636 \pm 673±573 \pm 5
4065±9\mathbf{65 \pm 9}56±156 \pm 172±8\mathbf{72 \pm 8}
10068±268 \pm 257±7\mathbf{57 \pm 7}67±167 \pm 1

λfinal\lambda_\text{final} ablations:

λfinal\lambda_\text{final}Visual OGBench (2 tasks)MMBench2 (2 tasks)Visual Robomimic (1 task)
0.860±260 \pm 259±359 \pm 366±466 \pm 4
0.9565±9\mathbf{65 \pm 9}57±7\mathbf{57 \pm 7}72±8\mathbf{72 \pm 8}
0.9959±659 \pm 653±253 \pm 272±472 \pm 4

KtrainK_\text{train} ablations:

KtrainK_\text{train}Visual OGBench (2 tasks)MMBench2 (2 tasks)Visual Robomimic (1 task)
463±1863 \pm 1852±252 \pm 276±276 \pm 2
865±9\mathbf{65 \pm 9}57±7\mathbf{57 \pm 7}72±8\mathbf{72 \pm 8}
1662±1562 \pm 1557±457 \pm 466±866 \pm 8

KevalK_\text{eval} ablations:

KevalK_\text{eval}Visual OGBench (2 tasks)MMBench2 (2 tasks)Visual Robomimic (1 task)
860±260 \pm 250±250 \pm 266±466 \pm 4
1662±462 \pm 458±258 \pm 270±970 \pm 9
3265±9\mathbf{65 \pm 9}57±7\mathbf{57 \pm 7}72±8\mathbf{72 \pm 8}
6459±359 \pm 357±557 \pm 569±869 \pm 8

First, with γ=0.99\gamma=0.99, LSM achieves an average success rate of 64±464 \pm 4 on the selected Visual OGBench tasks, close to 65±965 \pm 9 with γ=0.999\gamma=0.999, while performance on visual-square\texttt{visual-square} decreases from 72±872 \pm 8 to 64±664 \pm 6. On MMBench2, γ=0.99\gamma=0.99 yields the highest mean normalized score among the swept values, 57±757 \pm 7. Based on these results, we select γ=0.999\gamma=0.999 for Visual OGBench and Visual Robomimic, and γ=0.99\gamma=0.99 for MMBench2.

Second, increasing HfinalH_\text{final} from 1010 to 4040 improves performance on Visual OGBench and MMBench2, with small additional gains at 100100, while Visual Robomimic does not benefit from longer horizons. We retain Hfinal=40H_\text{final}=40 on Visual OGBench and Visual Robomimic, which use larger action chunks of size 1010, and use Hfinal=100H_\text{final}=100 on MMBench2, which uses smaller action chunks of size 55.

Finally, we select shared values λfinal=0.95\lambda_\text{final}=0.95, Ktrain=8K_\text{train}=8, and Keval=32K_\text{eval}=32 because they perform generally well across all three benchmarks. Increasing KtrainK_\text{train} to 1616 or KevalK_\text{eval} to 6464 provides no consistent improvement.

[1] Chung, H., Lee, J., & Oh, S. (2026). Offline reinforcement learning with universal horizon models. arXiv preprint arXiv:2605.15603.

Thanks for the rebuttal. I've read it and will keep my score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.