acceptodds
Under review as a conference paper at ICLR 2027

From Search to Policy: Model-Based Improvement for Offline Visual Control

Abstract

LeWorldModel (LeWM) provides predictive visual representations as a foundation for goal-directed control. Temporal reachability gives these representations a geometry based on the interaction cost of reaching one state from another. We introduce Directed Temporal Representations for Control (DTRC) to learn this temporal geometry from offline trajectories. Using temporal supervision from offline trajectories, we learn a control representation and a directed temporal distance on frozen LeWM features. We show that learning temporal reachability aligns latent distance with goal-reaching cost and turns distance reduction into a direct measure of action progress. We demonstrate empirically that this temporal geometry leads to higher success rates across a diverse suite of visual goal-conditioned control tasks.

Primary area:
applications to robotics, autonomy, planning
Keywords:
offline reinforcement learning, model-based reinforcement learning, world models, goal-conditioned control, visual control, temporal quasimetric, flow matching

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_5QHLOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces Directed Temporal Representations for Control (DTRC), an offline visual goal-conditioned control framework built on frozen LeWorldModel (LeWM) features. DTRC trains a control representation and a metric-residual temporal critic via local calibration, multistep expectile bootstrapping, and latent dynamics consistency. The critic guides a conditional flow-matching policy through advantage weighting on offline transitions and filtered model-generated rollouts, enabling direct test-time execution without planning. Experiments across ten visual control benchmarks evaluate goal-reaching success, learned temporal geometry, online inference latency, and component ablations against latent planning and direct policy baselines.

Critical strengths

The paper addresses a practical limitation of latent world models such as LeWM, where test-time control relies on costly online trajectory optimization. DTRC instead learns a goal-relative temporal critic on frozen visual features to guide a conditional flow-matching policy during training. This method enables fast closed-loop execution at test time and outperforms CEM-based latent planning on several manipulation and reaching tasks. In addition, the controlled comparison in Table 6 holds the flow-matching policy fixed to isolate the learned control representation and temporal advantage weighting. Unlike the main baseline comparisons, which use different policy architectures, this ablation clearly shows that temporal critic supervision improves direct policy learning on frozen visual features.

Critical weaknesses
  1. The experiments do not convincingly justify the directional residual and the model-based improvement over a simpler symmetric temporal critic. In Table 6, removing the directional residual performs just as well as the full model and slightly better on Cube. Moreover, three of the four ablated tasks have near-zero learned asymmetry in Table 2, and two are already saturated at 100% under "Flow BC + temporal critic". Adding the model-assisted mechanism mainly helps on PushT while slightly hurting Cube. Because it combines policy target augmentation with bootstrap target refinement in Appendix E.1, their individual effects cannot be separated. Stronger geometry diagnostics in Table 2 also do not guarantee better control. For instance, DTRC gets its highest progress score on PointMaze at 88.0%, but the lowest success rate in Table 1 at 11.3%. Finally, the value-based baselines (GC-IQL, HILP) use unimodal Gaussian MLPs rather than the flow-matching policy used by DTRC.

  2. There are several inconsistencies in the evaluation tables and missing algorithmic details make the empirical gains hard to verify. Tables 1 and 6 state that each method is evaluated over 5 seeds with 50 start-goal pairs per seed. However, many reported means (such as 11.3%, 50.7%, and 90.7%) look like averages over 3 seeds rather than 5. The six task averages computed from Table 1 also do not match the offset 25 averages reported for the same tasks in Table 7. In addition, Appendix E.2 notes that all error bars come from a single fixed training checkpoint rather than multiple training runs. Finally, line 18 of Algorithm 1, "Refine eligible bootstrapped temporal targets", is never mathematically defined in the paper.

Questions for authors
  1. Were 3 or 5 evaluation seeds used in Figure 2, Table 1, and Table 6, given values such as 11.3% and 50.7% with 50 pairs per seed? Also, why do the offset 25 averages in Table 7 differ from the six task means in Table 1?

  2. In Table 6, three of the four tasks have near-zero asymmetry and two are already at 100%. Can you ablate the directional residual across multiple training seeds on tasks with strong asymmetry (e.g., AntMaze-Medium) or at goal offset 100? Also, why does DTRC get only 11.3% success on PointMaze in Table 1 despite an 88.0% progress score in Table 2?

  3. What is the exact update rule and eligibility condition for "Refine eligible bootstrapped temporal targets" in line 18 of Algorithm 1? Can you separate policy target augmentation from this target refinement in Table 6, and report what fraction of candidate actions pass the filters?

  4. How do GC-IQL and HILP perform when paired with the same flow-matching policy architecture as DTRC and standard Q-V extraction for GC-IQL? Can you also confirm that all baselines in Figure 2 were trained and evaluated on visual LeWM inputs rather than the state vector checkpoints mentioned in Appendix F?

6 comments
AuthorsOpenReview ↗

Response to W1 & Q2: Directionality and Geometry versus Control

Thank you for recognizing the controlled evidence for temporal supervision and for asking whether directionality itself improves control. We have added the requested AntMaze and longer-offset comparisons, and separate this question from what the geometry diagnostics measure.

Directionality beyond saturated tasks. We compare the directed critic with dsym(s,g)=[d(s,g)+d(g,s)]/2d_{sym}(s,g)=[d(s,g)+d(g,s)]/2. Both learned heads and the exact parameter count are retained. This removes directionality without the capacity reduction of the original head-removal ablation.

All models use 30k updates, batch 1024 and final checkpoints. Success (%) is mean ± sample SD across three training seeds, after averaging five evaluation seeds × 50 recorded-future start-goal pairs per model and offset. The action budget is 50.

TaskOffsetDirectedSymmetrized
AntMaze-Medium2534.0 ± 1.231.9 ± 1.7
AntMaze-Medium5024.7 ± 0.623.5 ± 1.2
AntMaze-Medium10020.0 ± 1.419.2 ± 0.4
PointMaze10031.9 ± 0.232.1 ± 0.2
PushT10012.7 ± 2.916.7 ± 0.2

AntMaze uses native 64×64 pixels and the back camera. These are recorded-future goal evaluations, not official fixed-goal benchmark scores. The unsaturated AntMaze results show modest mean gains of 2.1/1.2/0.8 percentage points. PointMaze and PushT show that directionality is not uniformly beneficial; we have not rerun Cube and do not overturn its original negative result.

A symmetric estimator cannot represent unequal forward and reverse costs exactly, which motivates allowing asymmetry. This representational argument does not guarantee a control gain. Also, a mean signed difference such as Table 2's d(g,s)−d(s,g)d(g,s)-d(s,g) is not a complete measure of pairwise asymmetry, because positive and negative differences can cancel. Neither a large signed mean nor favorable geometry diagnostics establish policy superiority; the control ablation above is the relevant evidence.

Why 88% progress does not imply high PointMaze success. Table 2's statistic is the fraction of recorded transitions for which d(st+1,g)<d(st,g)d(s_{t+1},g)<d(s_t,g), with goals 25 recorded transitions ahead. The successor is produced by the dataset action, not by the learned policy. This checks local ordering along recorded trajectories; it does not test whether the policy selects those actions, stays on suitable trajectories or reaches the success region within budget. Even a small positive distance decrease counts as progress, whereas temporal advantage subtracts the elapsed transition cost. We will make these distinctions explicit and avoid treating progress as a surrogate success rate.

We also tested budget sensitivity without retraining: at offset 100, raising the action budget from 50 to 100 increases new PointMaze success from 31.9% to 98.0% for the directed critic and from 32.1% to 98.0% for the symmetric control. The policy's horizon input remains clipped at 50. This identifies a budget effect at offset 100; it does not establish the cause of Table 1's offset-25 result.

The broader temporal-supervision result is distinct from the directional residual. At offset 50, adding learned representation conditioning and temporal weighting to Flow BC, with both model-assisted branches disabled, raises PointMaze success from 65.6% to 99.3% and PushT from 35.6% to 55.7%. Thus the new evidence supports this learning path while qualifying the additional benefit of directionality as task-dependent.

AuthorsOpenReview ↗

Response to W1, W2 & Q3: Model-Assisted Updates and Separate Branch Effects

Thank you for asking us to define Algorithm 1's target refinement and separate the two model-assisted branches. We provide the exact implemented rule, measured filter acceptance and a factorial ablation of their effects.

Exact target-refinement rule. Write dˉ(z,g)=min⁡(200,dθˉ(ψˉ(z),ψˉ(g)))\bar d(z,g)=\min(200,d_{\bar\theta}(\bar\psi(z),\bar\psi(g))), with stopped gradients. Let Δ\Delta be a known positive recorded goal offset and E=1≤Δ≤nE=\\{1\leq\Delta\leq n\\}. The unrefined goal target is

y0={Δ,E,min⁡(200,n+dˉ(zt+n,zg)),otherwise.y_0=\begin{cases}\Delta,&E,\\\\ \min(200,n+\bar d(z_{t+n},z_g)),&\text{otherwise}.\end{cases}

Here n=8n=8. From update 10,000, sample 16 policy candidates per condition. In these experiments each candidate is one action (k=1k=1). Let VV contain candidates passing both behavior-support and dynamics-disagreement thresholds. Define

CD=k+dˉ(zt+k,zg),C∗=min⁡a∈Vk+dˉ(z^t+k(a),zg).C_D=k+\bar d(z_{t+k},z_g),\qquad C_*=\min_{a\in V}\\{k+\bar d(\hat z_{t+k}(a),z_g)\\}.

The predicted successor is the mean of four separately trained dynamics models in frozen LeWM feature space. Set C∗=+∞C_*=+\infty when VV is empty. When refinement is enabled, the target is

y={C∗,¬E∧C∗<CD∧C∗<y0,\y0,otherwise.y=\begin{cases}C_*,&\neg E\\ \land\\ C_*<C_D\\ \land\\ C_*<y_0,\\\y_0,&\text{otherwise}.\end{cases}

Before update 10,000, or with refinement disabled, y=y0y=y_0. Thus an eligible target must be bootstrapped, and the selected candidate must improve upon both the recorded continuation and the existing target. Direct observed-offset targets are unchanged. The critic fits this detached target by expectile regression of y−dθy-d_\theta, with expectile 0.3; target parameters use EMA rate 0.005.

The policy branch is separate: when enabled, a selected candidate with C∗<CDC_*<C_D adds a flow-matching target with weight min⁡(20,exp⁡((CD−C∗)/β))\min(20,\exp((C_D-C_*)/\beta)), where β=1\beta=1. It does not require the critic-refinement condition C∗<y0C_*<y_0. Neither branch operates at test time.

Filter definition and measured acceptance. Behavior support is the auxiliary behavior flow's mean squared residual, estimated with two noise/time samples at times in [0.1,0.9]. Dynamics disagreement averages ensemble sample standard deviations over predicted steps and latent coordinates. Each threshold initializes from the recorded-action 90th percentile and updates every 50 steps as δ←0.99δ+0.01q0.9\delta\leftarrow0.99\delta+0.01q_{0.9}. Candidates must satisfy both scores ≤ their thresholds.

For full DTRC, averaging the 21 logged minibatches from steps 10k–30k within each model, then the three training seeds, gives:

Rate (%)PointMazePushT
Candidates passing behavior filter78.568.2
Candidates passing dynamics filter89.891.0
Candidates passing both filters72.963.3
Conditions receiving a policy target83.371.8
Conditions receiving critic refinement69.263.2

The first three denominators are candidate actions; the last two are all sampled conditions, each with 16 candidates. These are logged-batch diagnostics, not a census of every training update. Passing the filters indicates model agreement and behavior compatibility, not guaranteed physical improvement.

Separating the two effects. All four arms retain the same auxiliary training and candidate generation; only policy-target use (P) and critic-target refinement (R) change. These main-batch results use the same 30k protocol, five evaluation seeds × 50 pairs per model/offset and budget 50. Values are success (%) as mean ± sample SD across three training seeds.

TaskPROffset 25Offset 50Offset 100
PointMazeOffOff99.9 ± 0.299.3 ± 0.628.7 ± 0.5
PointMazeOnOff99.6 ± 0.099.1 ± 0.631.2 ± 0.4
PointMazeOffOn99.9 ± 0.298.7 ± 0.527.5 ± 0.5
PointMazeOnOn100.0 ± 0.099.7 ± 0.231.9 ± 0.2
PushTOffOff83.5 ± 1.655.7 ± 1.89.6 ± 0.8
PushTOnOff83.6 ± 0.854.3 ± 0.914.9 ± 0.6
PushTOffOn80.5 ± 1.554.8 ± 2.67.5 ± 0.2
PushTOnOn84.3 ± 1.555.6 ± 2.412.7 ± 2.9

Additional policy targets improve both offset-100 means, but not offset 50. Refinement alone does not improve these results, and combining the branches is not uniformly best: policy-only exceeds full DTRC on PushT at offset 100. These results support a conditional benefit from model-generated policy supervision, not a general benefit from target refinement. We will define the rule explicitly and report the two branches separately.

AuthorsOpenReview ↗

Response to W1 & Q4: Baseline Fidelity, Flow Policies, and Visual Inputs

Thank you for requesting a comparison that preserves the baselines' mechanisms and controls the policy family. We have trained GC-IQL and HILP with both Gaussian and Flow actors on the same frozen visual inputs as DTRC.

Mechanisms retained. The new GC-IQL uses expectile V learning, TD Q learning and the standard AWR advantage A=min⁡(Q1,Q2)−VA=\min(Q_1,Q_2)-V, with weight min⁡(100,exp⁡(10A))\min(100,\exp(10A)), following the reference implementation. This replaces the value-difference extraction in our original adaptation.

For HILP, the new implementation retains Hilbert representation learning, random directional skills, representation-displacement intrinsic rewards, skill-conditioned Q/V and actor learning, and goal-directed execution using the normalized direction between current and goal representations. This restores the skill-learning mechanism omitted by the original direct goal-conditioned adaptation, following the HILP reference. It uses HILP's non-planning core.

Within each algorithm, Gaussian and Flow versions retain the same critic, goal/skill sampling and advantage weights. The actor objective changes from weighted Gaussian likelihood to weighted flow matching. The Flow backbone matches DTRC's width/depth (512 × 3) and eight integration steps; algorithm-specific conditioning means total parameter counts are not identical. Gaussian actors execute their mean, and Flow actors execute one sample without candidate search.

Results with matched inputs and update budgets. Every model uses the same task-specific frozen LeWM encoder and offline data, 30k updates, batch 1024 and its final checkpoint. Values are success (%) as mean ± sample SD across three training seeds, after averaging five evaluation seeds × 50 paired recorded-future start-goal samples per model and offset; budget is 50 actions.

PushTOffset 25Offset 50Offset 100
GC-IQL Gaussian68.9 ± 0.841.2 ± 0.46.1 ± 1.2
GC-IQL Flow75.3 ± 2.741.7 ± 1.46.3 ± 2.1
HILP Gaussian35.1 ± 0.68.8 ± 1.10.8 ± 0.0
HILP Flow38.0 ± 2.49.1 ± 0.50.7 ± 0.5
DTRC (reference)84.3 ± 1.556.0 ± 1.812.5 ± 2.7
PointMazeOffset 25Offset 50Offset 100
GC-IQL Gaussian98.9 ± 0.298.9 ± 0.631.1 ± 0.6
GC-IQL Flow99.3 ± 0.298.3 ± 0.228.9 ± 0.8
HILP Gaussian100.0 ± 0.096.3 ± 0.229.3 ± 0.2
HILP Flow100.0 ± 0.087.3 ± 4.426.0 ± 1.1
DTRC (reference)100.0 ± 0.099.7 ± 0.231.9 ± 0.2

Flow improves the short-offset PushT means, but is not universally better: both baselines' longer-offset PointMaze means decrease. DTRC remains ahead of the Flow variants on the unsaturated settings shown, so the difference cannot be explained solely by assigning DTRC a Flow actor. This does not identify which DTRC component explains the remaining gap. We also do not claim superiority to every direct baseline: GC-IDM reaches 71.6% on PushT at offset 50 versus DTRC's 56.0%.

These are PyTorch visual adaptations at a fixed update budget, not equal-convergence or equal-FLOP comparisons. HILP receives 30k joint representation/skill/policy updates, not 30k per separately trained stage. Its low PushT scores do not establish a general limitation of HILP. The DTRC references here are reevaluations of the same full-model checkpoints, so their PushT means differ slightly from the main ablation batch. We retain each batch's reference results separately; repeatability is addressed in our statistics response.

Visual inputs and Appendix F. We confirm that all methods in the new tables are trained and evaluated from image-derived LeWM features. Native simulator state is used only for reset/rendering and outcome measurement, not as actor or critic input. Appendix F's state-vector checkpoint statement concerns the Figure 7 representation visualization, so it does not by itself establish Figure 2's inputs. We cannot yet verify every original Figure 2 baseline's checkpoint-to-result mapping. The new comparisons have explicit encoder/checkpoint provenance and will be clearly distinguished from the historical figures.

Reviewer_QQkbOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Directed Temporal Representations for Control (DTRC), a framework for learning directed temporal control representations on frozen predictive visual features. DTRC jointly learns a control representation and a directed temporal critic, using long-horizon temporal targets, local temporal calibration, and action-conditioned consistency to capture temporal reachability from offline trajectories. The learned temporal critic is then used to guide advantage-weighted policy learning. Experiments across ten visual control tasks show that DTRC outperforms several existing baselines on certain tasks.

Critical strengths
  1. The paper is generally well-written and clearly organized. The motivation, methodology, and implementation details are presented in an accessible and structured manner.
  2. The paper provides a relatively comprehensive empirical evaluation across ten visual control tasks, covering different environments and task settings.
Critical weaknesses
  1. Several key components of DTRC closely follow existing methods. The directed temporal representation adopts the metric-residual construction from MRN, while the long-horizon temporal targets and local temporal calibration share substantial similarities with MQE. Moreover, the advantage-weighted policy objective resembles the energy-weighted Flow Matching formulation in EFM, and its temporal advantage follows the distance-based weighting principle in DWSL. The paper does not sufficiently clarify the methodological differences from these prior works or demonstrate what fundamentally new capabilities arise from their combination. Consequently, the overall novelty of DTRC remains unclear.
  2. The necessity of several proposed components is not adequately established. In Table 6, the directional residual yields only 0.7% improvement on PushT and reduces performance by 2.7% on Cube. Furthermore, since LeWM is already an action-conditioned predictive world model, the motivation for introducing an additional action-conditioned consistency loss requires further justification. The paper also lacks independent ablations of the temporal learning objectives and systematic hyperparameter sensitivity analyses.
  3. The implementations of GC-IQL and HILP substantially modify their original mechanisms. In addition, these baselines use Gaussian policies, whereas DTRC employs a flow-based policy.
  4. Figure 1 contains potentially misleading or inaccurate illustrations. For example, the directional relationships between (s) and (g) are not clearly represented, and the trajectory in Figure 1(c) does not appear consistent with the PushT task. In addition, the result highlighting in Table 1 is inconsistent with its caption: some best results are not bolded, the Puzzle-3x3 column is improperly marked, and the second-best result in the Cube-Double column appears to be incorrectly highlighted.
Questions for authors
  1. Beyond combining existing methods, what are the key novel contributions of this work, and how does it fundamentally differ from existing approaches?
  2. Could the authors provide additional theoretical analysis or experimental results to justify the use of the directional residual?
  3. Could the authors provide ablation studies to evaluate the individual contributions of the proposed components?
  4. Could the authors explain why GC-IDM, GCBC, GC-IQL, and HILP underperform LeWM on the PushT task in Figure 2?
6 comments
AuthorsOpenReview ↗

Response to W1 & Q1: Novelty and Relation to Prior Work

We agree that metric-residual parameterization, multistep learning, advantage weighting, and flow matching are established ingredients. DTRC's contribution is a training mechanism that uses a learned temporal-reaching critic to construct policy supervision from both recorded continuations and carefully screened model-predicted alternatives. This connects temporal geometry on frozen visual features to action-target selection and weighting.

What is distinctive in DTRC? Observed short-range offsets, long-range bootstrapping, and action-conditioned consistency train the temporal cost. The same critic evaluates two types of policy targets. Let dd denote the target temporal critic. For a recorded kk-step continuation and a candidate sequence a\mathbf a with predicted successor e^t+k(a)\hat e_{t+k}(\mathbf a), define

CD=k+d(eˉt+k,eˉg),C(a)=k+d(e^t+k(a),eˉg).C_{\mathcal D}=k+d(\bar e_{t+k},\bar e_g),\qquad C(\mathbf a)=k+d(\hat e_{t+k}(\mathbf a),\bar e_g).

Recorded actions are weighted by their temporal advantage Adata=d(eˉt,eˉg)−CDA^{\rm data}=d(\bar e_t,\bar e_g)-C_{\mathcal D}. Candidate sequences are first screened for behavior support and dynamics agreement; only candidates with Aim=CD−C(a∗)>0A^{\rm im}=C_{\mathcal D}-C(\mathbf a^{*}) > 0 provide additional flow-matching targets. Accepted candidates can also refine eligible bootstrapped critic targets, without changing the directly observed short-offset anchors. Thus, temporal reachability determines both how strongly to imitate recorded actions and when a predicted alternative is eligible for policy supervision. Candidate evaluation is training-only.

Distinction from prior work.

  • MRN provides the metric-residual architecture, which we adopt without claiming architectural novelty. DTRC contributes the temporal supervision and policy rules built on it.
  • MQE also combines multistep learning with quasimetric structure. Its state/state-action Q/V formulation, action-invariance constraint, and DDPG+BC extraction differ from DTRC's expectile fitting of temporal reaching costs and progress-weighted direct flow policy. In particular, our local calibration fits observed transition counts, whereas MQE's action-invariance constraint aligns state and state-action embeddings. DTRC also screens predicted targets against recorded continuations using the temporal critic.
  • DWSL establishes distance-reduction weighting. We retain this principle, but the weights here come from a structured, bootstrapped temporal critic rather than solely supervised trajectory-distance distributions. The same critic evaluates predicted alternatives and conditions the policy.
  • EFM establishes energy-weighted flow learning and includes QIPO's iterative Q-guided optimization. We do not claim weighted flow matching or candidate generation as new. DTRC instead defines improvement by temporal reaching cost relative to the recorded continuation and admits predicted targets only after behavior-support and dynamics-agreement checks.

Controlled evidence. To separate temporal supervision from the choice of flow policy, we disable both model-assisted branches and compare against Flow BC using the same flow backbone width/depth and sampling. At goal offset 50 with action budget 50, success rises from 65.6 ± 3.6 to 99.3 ± 0.6 on PointMaze and from 35.6 ± 2.5 to 55.7 ± 1.8 on PushT. These comparisons test representation conditioning and temporal weighting jointly, not either ingredient in isolation. All models use 30k updates, batch 1024, and fixed checkpoints; results are mean ± SD over three training seeds, each evaluated with five seeds × 50 recorded-future pairs. Our separate bootstrap ablation shows lower success at all three tested offsets on both tasks when bootstrap supervision is removed, with both model-assisted branches off.

We then isolate additional policy targets, keeping critic-target refinement off while retaining auxiliary training and candidate generation. At offset 100/budget 50, PointMaze/PushT success improves from 28.7 ± 0.5 / 9.6 ± 0.8 to 31.2 ± 0.4 / 14.9 ± 0.6. Gains are not universal: offset-50 means do not improve, and both branches yield 31.9 ± 0.2 / 12.7 ± 2.9 at offset 100. We therefore claim a demonstrable benefit for particular learning paths under the tested conditions, not that every component is essential or that the full combination always wins.

In summary, our novelty claim is the operational link from learned temporal reaching cost to recorded-action weighting and constrained predicted-target supervision, rather than the individual building blocks. The controlled results support the temporal-supervision pathway and show additional gains from predicted targets in the longer-offset setting. Direct comparisons with the cited methods remain an important limitation; we do not claim untested superiority.

AuthorsOpenReview ↗

Response to W2, Q2 & Q3: Directionality, Component Ablations, and Sensitivity

Thank you for the questions on directionality, individual objectives, and sensitivity. We conducted controlled experiments and distinguish architectural motivation from measured control benefits. Unless noted otherwise, all runs use 30k updates, batch 1024, and final checkpoints. Success (%) is mean ± SD over three training seeds, each averaged across five evaluation seeds and 50 recorded-future start–goal pairs per offset (action budget 50).

1. Directionality (Q2). If temporal reaching costs differ between s→gs\to g and g→sg\to s, a symmetric function cannot represent both exactly. Asymmetry removes this representational restriction, but need not improve policy success. We test dsym(s,g)=[d(s,g)+d(g,s)]/2d_{\mathrm{sym}}(s,g)=[d(s,g)+d(g,s)]/2, retaining both heads and the parameter count, unlike the original residual-removal ablation.

AntMaze-Medium (native pixels)Offset 2550100
Directed34.0 ± 1.224.7 ± 0.620.0 ± 1.4
Symmetrized31.9 ± 1.723.5 ± 1.219.2 ± 0.4

Gains are modest (+2.1/+1.2/+0.8 percentage points). At offset 100, directed/symmetrized scores are 31.9/32.1 on PointMaze and 12.7/16.7 on PushT. Thus, we do not claim a uniform performance advantage. Cube was not rerun, and its original negative result remains a limitation.

2. Independent objectives (Q3). To test bootstrap supervision, both model-assisted branches are disabled in both arms. We change only the coefficient on non-exact/bootstrap goal targets (1 to 0); exact-target normalization, local and consistency losses, policy weighting, sampling, and architecture remain fixed.

Bootstrap targetsOffset 2550100
PointMaze on99.9 ± 0.299.3 ± 0.628.7 ± 0.5
PointMaze off91.1 ± 1.660.7 ± 2.419.3 ± 0.6
PushT on83.5 ± 1.655.7 ± 1.89.6 ± 0.8
PushT off75.2 ± 0.842.4 ± 2.04.0 ± 0.4

Bootstrap supervision contributes substantially in this controlled, model-assisted-off setting; this is not an isolated effect within full DTRC.

We separately remove auxiliary local calibration and/or consistency from full DTRC. The main goal loss still contains exact short-offset targets, so “local off” does not eliminate all local temporal supervision.

PushT (main ablation batch)Offset 2550100
Full84.3 ± 1.555.6 ± 2.412.7 ± 2.9
Local off84.0 ± 1.856.0 ± 3.113.9 ± 1.2
Consistency off83.5 ± 2.055.9 ± 3.113.5 ± 1.4
Both off83.3 ± 0.857.1 ± 4.513.9 ± 0.5

Neither auxiliary loss consistently improves PushT. PointMaze offset-100 means are likewise 31.9/31.7/31.9/32.0 in table order. We therefore regard them as motivated regularizers, not components whose individual necessity is established by these success rates.

3. Why consistency beyond LeWM? LeWM predicts next-step visual latents from zt,atz_t,a_t, whereas our loss constrains the learned control representation et=ψ(zt)e_t=\psi(z_t). Predictability from zz need not survive projection by ψ\psi, which may discard action-relevant information:

Lcons=E,MSE(fξ(ψ(zt),at),sg[ψˉ(zt+1)]).L_{\mathrm{cons}}=\mathbb E\\,\mathrm{MSE}(f_\xi(\psi(z_t),a_t),\mathrm{sg}[\bar\psi(z_{t+1})]).

The target is the observed successor represented by the target head; gradients stop there. This trains ψ\psi and its one-step transition head, not LeWM's frozen encoder. It is a representation-level constraint, not evidence that the auxiliary loss is necessary for success.

4. Sensitivity. We trained 18 additional PushT models (three seeds × six one-factor changes) around local coefficient 1, consistency coefficient 0.1, and weighting temperature 1. All other settings are fixed. The default reference was reevaluated from the same full-model checkpoints in this supplementary batch and is distinct from the main-ablation evaluation.

PushT sensitivityOffset 2550100
Default84.3 ± 1.556.0 ± 1.812.5 ± 2.7
Local 0.384.7 ± 0.957.2 ± 1.416.3 ± 1.3
Local 382.5 ± 1.652.7 ± 1.213.7 ± 1.8
Consistency 0.0382.7 ± 2.455.2 ± 1.114.1 ± 1.9
Consistency 0.382.8 ± 1.852.5 ± 3.713.1 ± 2.3
Temperature 0.582.8 ± 1.752.8 ± 4.216.0 ± 3.1
Temperature 285.2 ± 0.856.1 ± 0.812.4 ± 0.4

Defaults are not uniformly best. This is one-factor sensitivity on PushT, not exhaustive tuning or cross-task robustness evidence. PushT resets remain stochastic even for matched dataset indices, so small score differences do not establish definitive rankings.

AuthorsOpenReview ↗

Response to W3 & Q4: Baseline Fidelity, Policy Fairness, and PushT Performance

Thank you for highlighting these comparison issues. We agree that baseline mechanisms and policy-family differences must be accounted for. We added mechanism-preserving visual adaptations and Gaussian/Flow controls, while distinguishing what these experiments can and cannot explain about the original LeWM comparison.

1. Restoring GC-IQL and HILP mechanisms. Our original GC-IQL actor used value-difference rather than Q–V weights; our original HILP adaptation replaced skill learning/selection with direct goal-conditioned extraction. These changes were consequential. The revised adaptations restore:

  • GC-IQL: expectile V, TD Q, and actor weights from Q(s,a,g)−V(s,g)Q(s,a,g)-V(s,g), following the reference implementation.
  • HILP: Hilbert representation learning, random directional skills, displacement-based intrinsic rewards, skill-conditioned Q/V and policies, and goal-directed skill selection using the normalized representation displacement.

Both use frozen LeWM visual features in PyTorch. HILP uses the non-planning core and 30k joint representation/skill/policy updates, not 30k per separately trained stage. These are closer adaptations, not exact reproductions of every reference training procedure. We will document the differences and distinguish them from our original adaptations.

2. Gaussian versus Flow actors. We trained both actor types for GC-IQL and HILP on PointMaze and PushT. Within each algorithm, the Flow variant preserves the critic, goal/skill sampling, and advantage weights, but replaces weighted Gaussian likelihood with weighted flow matching. Its network uses DTRC's width/depth (512 × 3) and eight integration steps. Conditioning and total parameter counts differ by algorithm. Gaussian actors execute their mean; Flow actors execute one sample without search. Thus, this controls the algorithm-specific learning signals, not every aspect of actor training or execution.

Within each task, all methods use the same frozen encoder and offline data, 30k updates, batch 1024, three training seeds, and final checkpoints. Evaluation uses five seeds × 50 paired recorded-future start–goal pairs per offset, with a 50-action budget. PushT success (%) below is mean ± SD across training seeds after averaging evaluation seeds.

PushT methodOffset 2550100
GCBC77.6 ± 1.735.3 ± 0.62.5 ± 0.8
GC-IDM78.9 ± 1.471.6 ± 2.812.4 ± 0.7
GC-IQL Gaussian68.9 ± 0.841.2 ± 0.46.1 ± 1.2
GC-IQL Flow75.3 ± 2.741.7 ± 1.46.3 ± 2.1
HILP Gaussian35.1 ± 0.68.8 ± 1.10.8 ± 0.0
HILP Flow38.0 ± 2.49.1 ± 0.50.7 ± 0.5
DTRC (supplementary reference)84.3 ± 1.556.0 ± 1.812.5 ± 2.7

Flow improves GC-IQL at offset 25 (68.9 to 75.3) and HILP (35.1 to 38.0), with little or no gain at longer offsets. DTRC exceeds both revised Flow baselines at all three offsets, so adopting Flow alone does not account for those gaps. It does not isolate which DTRC component explains them. DTRC is not uniformly superior: GC-IDM is stronger at offset 50 (71.6 vs. 56.0) and nearly tied at 100 (12.4 vs. 12.5).

These are fixed-update comparisons, not equal-convergence, equal-FLOP, or best-attainable-performance claims; low HILP scores should not be read as a general limitation of HILP. They are recorded-future goal tests, not official fixed-goal benchmark results. The DTRC reference comes from the supplementary sensitivity evaluation, not the main ablation batch. Stochastic PushT resets also limit conclusions from small differences.

3. Why were the original PushT baselines below LeWM (Q4)? A plausible factor is test-time planning versus direct action prediction: LeWM searches action sequences through its predictive model at execution, whereas these direct policies act without that search. However, the original comparison also confounded planning with baseline adaptations and actor choices; it cannot establish planning as the cause. GCBC and GC-IDM used deterministic MSE regression, so Gaussian-policy limitations cannot explain all four original baselines.

Our new controls restore core GC-IQL/HILP mechanisms and examine actor-family effects, but do not causally identify the original LeWM gap. We have not reevaluated LeWM under this protocol, and therefore neither compare its old Figure 2 score to this table nor claim that the gap has closed. The revised results support a more controlled direct-policy comparison under a fixed update budget; explaining the original LeWM gap remains open.

Reviewer_pwo3OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposed DTRC for visual goal-conditioned control framework based on offline trajectories. It learns a temporal critic using trajectory offsets and bootstrapped targets. Evaluated over 10 tasks show the goal-reaching performance gain and efficiency of DTRC.

Critical strengths
  • Idea is clear and the proposed temporal supervision improves the flow policy over four tasks.
  • The prposed method shows pratcial improvement over LeWM baselines.
Critical weaknesses

The efficiency and trajectory-quality comparisons seem incomplete: Table 3 compares DTRC only with LeWM, omitting the other direct-policy baselines. Since these methods also avoid test-time search and achieve comparable or better success rates, the reported speedups do not establish a practical advantage over the strongest alternatives. Are there any specific reasons that the authors did not compare with those baselines?

Questions for authors

See the weakness above

3 comments
AuthorsOpenReview ↗

Response to Weakness: Direct-policy efficiency and trajectory quality

Thank you for pointing out that the comparison with LeWM does not establish an advantage over other direct policies. Table 3 focused on the cost of online planning versus direct execution. That scope was too narrow for a comparison of practical alternatives, so we have added both latency and trajectory-quality measurements for the direct-policy baselines.

Matched comparisons. We evaluate GCBC, GC-IDM, Flow BC, and GC-IQL/HILP with both Gaussian and Flow actors on PointMaze and PushT. The new GC-IQL retains standard Q-V advantage extraction, and HILP retains skill learning. All methods use the same task-specific frozen visual LeWM encoder and offline data, 30k updates, batch 1024 and final checkpoints. The table below uses offset 50 and a 50-action budget. Each of three independently trained models is evaluated with five seeds × 50 paired recorded-future start-goal samples. Success is first averaged within each model; mean ± sample SD is then computed across training seeds.

Online latency alongside success. Timing uses the same exclusively occupied GPU, batch size 1, 40 warm-up calls and 240 timed calls per checkpoint, with CUDA synchronization. The reported pixels-to-action latency includes current-image encoding, policy computation and transfers, with the goal encoding cached as within an episode; environment rendering/stepping is excluded. Latency entries summarize the three checkpoint-level means. Concurrent training times and batched rollout timings are not used.

MethodPointMaze success (%)Latency (ms)PushT success (%)Latency (ms)
GCBC55.1 ± 2.23.92 ± 0.0535.3 ± 0.63.95 ± 0.04
GC-IDM93.7 ± 1.64.00 ± 0.0271.6 ± 2.84.01 ± 0.01
GC-IQL Gaussian98.9 ± 0.63.92 ± 0.0241.2 ± 0.43.95 ± 0.01
GC-IQL Flow98.3 ± 0.25.89 ± 0.0341.7 ± 1.45.91 ± 0.04
HILP Gaussian96.3 ± 0.24.37 ± 0.148.8 ± 1.14.37 ± 0.01
HILP Flow87.3 ± 4.46.20 ± 0.059.1 ± 0.56.23 ± 0.05
Flow BC65.6 ± 3.66.00 ± 0.0435.7 ± 2.06.06 ± 0.02
DTRC99.7 ± 0.26.25 ± 0.0556.0 ± 1.86.48 ± 0.24

DTRC is not the fastest direct policy. Compared with Flow BC, its mean success increases from 65.6% to 99.7% on PointMaze and from 35.7% to 56.0% on PushT, for approximately 0.25/0.42 ms more per decision. This is a favorable trade-off relative to Flow BC, not dominance over all alternatives: GC-IDM is both faster and more successful on PushT, while Gaussian GC-IQL has similar PointMaze success at lower latency. All these direct policies avoid online search, so the speedup over LeWM cannot be credited uniquely to DTRC.

Trajectory quality under the same evaluations. Jerk is the mean norm of the second action difference (lower is smoother). Monotonicity is the percentage of episodes with non-increasing goal distance at every recorded step. We use the same distance for every method within a task: native task distance for PointMaze and frozen-LeWM cosine distance for PushT, not each method's learned critic. The following values are means across the three training seeds; quality statistics are computed over evaluated trajectories, not selected successful examples.

MethodPointMaze jerkMonotonicity (%)PushT jerkMonotonicity (%)
GCBC0.07480.50.0383.1
GC-IDM0.08772.00.0411.5
GC-IQL Gaussian0.07296.50.0385.6
GC-IQL Flow0.86458.50.2694.1
HILP Gaussian0.05896.80.0282.8
HILP Flow1.12334.70.4021.1
Flow BC1.22710.90.2131.1
DTRC0.70471.10.2487.9

Relative to Flow BC, DTRC has lower jerk and higher episode monotonicity on PointMaze. On PushT, it improves monotonicity and success but has higher jerk. Gaussian policies are smoother than DTRC in both tasks, and several have higher PointMaze monotonicity. Conversely, GC-IDM's higher PushT success despite lower monotonicity shows why monotonicity alone is not a control-quality ranking. We will report these metrics alongside success rather than claim uniformly superior trajectories.

We will expand the efficiency/trajectory comparison accordingly and distinguish the planning-versus-policy result from trade-offs among direct policies. These new measurements cover two tasks under a fixed 30k, recorded-future-goal protocol; they are not a new ten-task benchmark or equal-training-compute comparison. PushT reset variability also cautions against interpreting close differences as definitive rankings. The supported result is improved control over Flow BC with modest additional inference cost, not a universal speed or trajectory-quality advantage.

Dear Reviewer,

Thank you for your efforts in reviewing this submission. As the discussion period is approaching its end, I would like to kindly remind you to read the authors' rebuttal and engage in the discussion. Your input is important for reaching a well-informed final assessment.

Thank you for your time and cooperation.

Thanks for the rebuttal. I also read other reviews, and will keep my original score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.