- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces Directed Temporal Representations for Control (DTRC), an offline visual goal-conditioned control framework built on frozen LeWorldModel (LeWM) features. DTRC trains a control representation and a metric-residual temporal critic via local calibration, multistep expectile bootstrapping, and latent dynamics consistency. The critic guides a conditional flow-matching policy through advantage weighting on offline transitions and filtered model-generated rollouts, enabling direct test-time execution without planning. Experiments across ten visual control benchmarks evaluate goal-reaching success, learned temporal geometry, online inference latency, and component ablations against latent planning and direct policy baselines.
Critical strengths
The paper addresses a practical limitation of latent world models such as LeWM, where test-time control relies on costly online trajectory optimization. DTRC instead learns a goal-relative temporal critic on frozen visual features to guide a conditional flow-matching policy during training. This method enables fast closed-loop execution at test time and outperforms CEM-based latent planning on several manipulation and reaching tasks. In addition, the controlled comparison in Table 6 holds the flow-matching policy fixed to isolate the learned control representation and temporal advantage weighting. Unlike the main baseline comparisons, which use different policy architectures, this ablation clearly shows that temporal critic supervision improves direct policy learning on frozen visual features.
Critical weaknesses
-
The experiments do not convincingly justify the directional residual and the model-based improvement over a simpler symmetric temporal critic. In Table 6, removing the directional residual performs just as well as the full model and slightly better on Cube. Moreover, three of the four ablated tasks have near-zero learned asymmetry in Table 2, and two are already saturated at 100% under "Flow BC + temporal critic". Adding the model-assisted mechanism mainly helps on PushT while slightly hurting Cube. Because it combines policy target augmentation with bootstrap target refinement in Appendix E.1, their individual effects cannot be separated. Stronger geometry diagnostics in Table 2 also do not guarantee better control. For instance, DTRC gets its highest progress score on PointMaze at 88.0%, but the lowest success rate in Table 1 at 11.3%. Finally, the value-based baselines (GC-IQL, HILP) use unimodal Gaussian MLPs rather than the flow-matching policy used by DTRC.
-
There are several inconsistencies in the evaluation tables and missing algorithmic details make the empirical gains hard to verify. Tables 1 and 6 state that each method is evaluated over 5 seeds with 50 start-goal pairs per seed. However, many reported means (such as 11.3%, 50.7%, and 90.7%) look like averages over 3 seeds rather than 5. The six task averages computed from Table 1 also do not match the offset 25 averages reported for the same tasks in Table 7. In addition, Appendix E.2 notes that all error bars come from a single fixed training checkpoint rather than multiple training runs. Finally, line 18 of Algorithm 1, "Refine eligible bootstrapped temporal targets", is never mathematically defined in the paper.
Questions for authors
-
Were 3 or 5 evaluation seeds used in Figure 2, Table 1, and Table 6, given values such as 11.3% and 50.7% with 50 pairs per seed? Also, why do the offset 25 averages in Table 7 differ from the six task means in Table 1?
-
In Table 6, three of the four tasks have near-zero asymmetry and two are already at 100%. Can you ablate the directional residual across multiple training seeds on tasks with strong asymmetry (e.g., AntMaze-Medium) or at goal offset 100? Also, why does DTRC get only 11.3% success on PointMaze in Table 1 despite an 88.0% progress score in Table 2?
-
What is the exact update rule and eligibility condition for "Refine eligible bootstrapped temporal targets" in line 18 of Algorithm 1? Can you separate policy target augmentation from this target refinement in Table 6, and report what fraction of candidate actions pass the filters?
-
How do GC-IQL and HILP perform when paired with the same flow-matching policy architecture as DTRC and standard Q-V extraction for GC-IQL? Can you also confirm that all baselines in Figure 2 were trained and evaluated on visual LeWM inputs rather than the state vector checkpoints mentioned in Appendix F?
6 comments
Response to W1, W2 & Q3: Model-Assisted Updates and Separate Branch Effects
Thank you for asking us to define Algorithm 1's target refinement and separate the two model-assisted branches. We provide the exact implemented rule, measured filter acceptance and a factorial ablation of their effects.
Exact target-refinement rule. Write , with stopped gradients. Let be a known positive recorded goal offset and . The unrefined goal target is
Here . From update 10,000, sample 16 policy candidates per condition. In these experiments each candidate is one action (). Let contain candidates passing both behavior-support and dynamics-disagreement thresholds. Define
The predicted successor is the mean of four separately trained dynamics models in frozen LeWM feature space. Set when is empty. When refinement is enabled, the target is
Before update 10,000, or with refinement disabled, . Thus an eligible target must be bootstrapped, and the selected candidate must improve upon both the recorded continuation and the existing target. Direct observed-offset targets are unchanged. The critic fits this detached target by expectile regression of , with expectile 0.3; target parameters use EMA rate 0.005.
The policy branch is separate: when enabled, a selected candidate with adds a flow-matching target with weight , where . It does not require the critic-refinement condition . Neither branch operates at test time.
Filter definition and measured acceptance. Behavior support is the auxiliary behavior flow's mean squared residual, estimated with two noise/time samples at times in [0.1,0.9]. Dynamics disagreement averages ensemble sample standard deviations over predicted steps and latent coordinates. Each threshold initializes from the recorded-action 90th percentile and updates every 50 steps as . Candidates must satisfy both scores ≤ their thresholds.
For full DTRC, averaging the 21 logged minibatches from steps 10k–30k within each model, then the three training seeds, gives:
| Rate (%) | PointMaze | PushT |
|---|---|---|
| Candidates passing behavior filter | 78.5 | 68.2 |
| Candidates passing dynamics filter | 89.8 | 91.0 |
| Candidates passing both filters | 72.9 | 63.3 |
| Conditions receiving a policy target | 83.3 | 71.8 |
| Conditions receiving critic refinement | 69.2 | 63.2 |
The first three denominators are candidate actions; the last two are all sampled conditions, each with 16 candidates. These are logged-batch diagnostics, not a census of every training update. Passing the filters indicates model agreement and behavior compatibility, not guaranteed physical improvement.
Separating the two effects. All four arms retain the same auxiliary training and candidate generation; only policy-target use (P) and critic-target refinement (R) change. These main-batch results use the same 30k protocol, five evaluation seeds × 50 pairs per model/offset and budget 50. Values are success (%) as mean ± sample SD across three training seeds.
| Task | P | R | Offset 25 | Offset 50 | Offset 100 |
|---|---|---|---|---|---|
| PointMaze | Off | Off | 99.9 ± 0.2 | 99.3 ± 0.6 | 28.7 ± 0.5 |
| PointMaze | On | Off | 99.6 ± 0.0 | 99.1 ± 0.6 | 31.2 ± 0.4 |
| PointMaze | Off | On | 99.9 ± 0.2 | 98.7 ± 0.5 | 27.5 ± 0.5 |
| PointMaze | On | On | 100.0 ± 0.0 | 99.7 ± 0.2 | 31.9 ± 0.2 |
| PushT | Off | Off | 83.5 ± 1.6 | 55.7 ± 1.8 | 9.6 ± 0.8 |
| PushT | On | Off | 83.6 ± 0.8 | 54.3 ± 0.9 | 14.9 ± 0.6 |
| PushT | Off | On | 80.5 ± 1.5 | 54.8 ± 2.6 | 7.5 ± 0.2 |
| PushT | On | On | 84.3 ± 1.5 | 55.6 ± 2.4 | 12.7 ± 2.9 |
Additional policy targets improve both offset-100 means, but not offset 50. Refinement alone does not improve these results, and combining the branches is not uniformly best: policy-only exceeds full DTRC on PushT at offset 100. These results support a conditional benefit from model-generated policy supervision, not a general benefit from target refinement. We will define the rule explicitly and report the two branches separately.
Response to W1 & Q4: Baseline Fidelity, Flow Policies, and Visual Inputs
Thank you for requesting a comparison that preserves the baselines' mechanisms and controls the policy family. We have trained GC-IQL and HILP with both Gaussian and Flow actors on the same frozen visual inputs as DTRC.
Mechanisms retained. The new GC-IQL uses expectile V learning, TD Q learning and the standard AWR advantage , with weight , following the reference implementation. This replaces the value-difference extraction in our original adaptation.
For HILP, the new implementation retains Hilbert representation learning, random directional skills, representation-displacement intrinsic rewards, skill-conditioned Q/V and actor learning, and goal-directed execution using the normalized direction between current and goal representations. This restores the skill-learning mechanism omitted by the original direct goal-conditioned adaptation, following the HILP reference. It uses HILP's non-planning core.
Within each algorithm, Gaussian and Flow versions retain the same critic, goal/skill sampling and advantage weights. The actor objective changes from weighted Gaussian likelihood to weighted flow matching. The Flow backbone matches DTRC's width/depth (512 × 3) and eight integration steps; algorithm-specific conditioning means total parameter counts are not identical. Gaussian actors execute their mean, and Flow actors execute one sample without candidate search.
Results with matched inputs and update budgets. Every model uses the same task-specific frozen LeWM encoder and offline data, 30k updates, batch 1024 and its final checkpoint. Values are success (%) as mean ± sample SD across three training seeds, after averaging five evaluation seeds × 50 paired recorded-future start-goal samples per model and offset; budget is 50 actions.
| PushT | Offset 25 | Offset 50 | Offset 100 |
|---|---|---|---|
| GC-IQL Gaussian | 68.9 ± 0.8 | 41.2 ± 0.4 | 6.1 ± 1.2 |
| GC-IQL Flow | 75.3 ± 2.7 | 41.7 ± 1.4 | 6.3 ± 2.1 |
| HILP Gaussian | 35.1 ± 0.6 | 8.8 ± 1.1 | 0.8 ± 0.0 |
| HILP Flow | 38.0 ± 2.4 | 9.1 ± 0.5 | 0.7 ± 0.5 |
| DTRC (reference) | 84.3 ± 1.5 | 56.0 ± 1.8 | 12.5 ± 2.7 |
| PointMaze | Offset 25 | Offset 50 | Offset 100 |
|---|---|---|---|
| GC-IQL Gaussian | 98.9 ± 0.2 | 98.9 ± 0.6 | 31.1 ± 0.6 |
| GC-IQL Flow | 99.3 ± 0.2 | 98.3 ± 0.2 | 28.9 ± 0.8 |
| HILP Gaussian | 100.0 ± 0.0 | 96.3 ± 0.2 | 29.3 ± 0.2 |
| HILP Flow | 100.0 ± 0.0 | 87.3 ± 4.4 | 26.0 ± 1.1 |
| DTRC (reference) | 100.0 ± 0.0 | 99.7 ± 0.2 | 31.9 ± 0.2 |
Flow improves the short-offset PushT means, but is not universally better: both baselines' longer-offset PointMaze means decrease. DTRC remains ahead of the Flow variants on the unsaturated settings shown, so the difference cannot be explained solely by assigning DTRC a Flow actor. This does not identify which DTRC component explains the remaining gap. We also do not claim superiority to every direct baseline: GC-IDM reaches 71.6% on PushT at offset 50 versus DTRC's 56.0%.
These are PyTorch visual adaptations at a fixed update budget, not equal-convergence or equal-FLOP comparisons. HILP receives 30k joint representation/skill/policy updates, not 30k per separately trained stage. Its low PushT scores do not establish a general limitation of HILP. The DTRC references here are reevaluations of the same full-model checkpoints, so their PushT means differ slightly from the main ablation batch. We retain each batch's reference results separately; repeatability is addressed in our statistics response.
Visual inputs and Appendix F. We confirm that all methods in the new tables are trained and evaluated from image-derived LeWM features. Native simulator state is used only for reset/rendering and outcome measurement, not as actor or critic input. Appendix F's state-vector checkpoint statement concerns the Figure 7 representation visualization, so it does not by itself establish Figure 2's inputs. We cannot yet verify every original Figure 2 baseline's checkpoint-to-result mapping. The new comparisons have explicit encoder/checkpoint provenance and will be clearly distinguished from the historical figures.
Response to W1 & Q2: Directionality and Geometry versus Control
Thank you for recognizing the controlled evidence for temporal supervision and for asking whether directionality itself improves control. We have added the requested AntMaze and longer-offset comparisons, and separate this question from what the geometry diagnostics measure.
Directionality beyond saturated tasks. We compare the directed critic with dsym(s,g)=[d(s,g)+d(g,s)]/2. Both learned heads and the exact parameter count are retained. This removes directionality without the capacity reduction of the original head-removal ablation.
All models use 30k updates, batch 1024 and final checkpoints. Success (%) is mean ± sample SD across three training seeds, after averaging five evaluation seeds × 50 recorded-future start-goal pairs per model and offset. The action budget is 50.
AntMaze uses native 64×64 pixels and the back camera. These are recorded-future goal evaluations, not official fixed-goal benchmark scores. The unsaturated AntMaze results show modest mean gains of 2.1/1.2/0.8 percentage points. PointMaze and PushT show that directionality is not uniformly beneficial; we have not rerun Cube and do not overturn its original negative result.
A symmetric estimator cannot represent unequal forward and reverse costs exactly, which motivates allowing asymmetry. This representational argument does not guarantee a control gain. Also, a mean signed difference such as Table 2's d(g,s)−d(s,g) is not a complete measure of pairwise asymmetry, because positive and negative differences can cancel. Neither a large signed mean nor favorable geometry diagnostics establish policy superiority; the control ablation above is the relevant evidence.
Why 88% progress does not imply high PointMaze success. Table 2's statistic is the fraction of recorded transitions for which d(st+1,g)<d(st,g), with goals 25 recorded transitions ahead. The successor is produced by the dataset action, not by the learned policy. This checks local ordering along recorded trajectories; it does not test whether the policy selects those actions, stays on suitable trajectories or reaches the success region within budget. Even a small positive distance decrease counts as progress, whereas temporal advantage subtracts the elapsed transition cost. We will make these distinctions explicit and avoid treating progress as a surrogate success rate.
We also tested budget sensitivity without retraining: at offset 100, raising the action budget from 50 to 100 increases new PointMaze success from 31.9% to 98.0% for the directed critic and from 32.1% to 98.0% for the symmetric control. The policy's horizon input remains clipped at 50. This identifies a budget effect at offset 100; it does not establish the cause of Table 1's offset-25 result.
The broader temporal-supervision result is distinct from the directional residual. At offset 50, adding learned representation conditioning and temporal weighting to Flow BC, with both model-assisted branches disabled, raises PointMaze success from 65.6% to 99.3% and PushT from 35.6% to 55.7%. Thus the new evidence supports this learning path while qualifying the additional benefit of directionality as task-dependent.