- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes CoTA-DT, a task-aware Decision Transformer framework for language-conditioned offline meta-reinforcement learning. A GRU context encoder learns task representations through trajectory contrastive learning, alignment with task descriptions, and auxiliary prediction of rewards, state transitions, and empirical returns-to-go. A frozen pretrained language model with a shared trainable LoRA adapter supports language encoding and auxiliary prediction, and is subsequently adapted for task-conditioned policy learning. For zero-shot deployment, the method initializes a task latent by optimizing its projected alignment with the target task description, then updates the latent after sufficient rollout history becomes available. Few-shot adaptation additionally incorporates support trajectories and permits lightweight fine-tuning of the LoRA adapter and action head. Experiments on Point-Robot, Cheetah-Vel, Ant-Dir, and Meta-World report higher mean returns than the evaluated baselines on Mixed datasets in both zero-shot and few-shot settings. Additional experiments investigate dataset quality on selected environments, component contributions, backbone choices, description robustness, and few-shot adaptation modes.
Critical strengths
1.A useful integration of language-based initialization and trajectory-based task inference. The framework connects task descriptions and interaction experience through a common task latent. This allows deployment without target-task support trajectories while subsequently incorporating observed transitions. The design is practically relevant: language provides initial task information, while experience may reveal characteristics not fully specified by the description. Using these complementary sources to condition a common policy is a useful extension of language-conditioned offline meta-RL.
2.Consistent reported improvements supported by several component analyses. Tables 1–2 report favorable mean performance across four benchmarks on Mixed datasets, while Table 4 examines the contribution of each representation-learning objective across all four environments. The small-DT control in Table 10 still exceeds the reproduced T2DA-T baseline on Meta-World, suggesting that the gains do not depend entirely on a large decision backbone. Appendix F further distinguishes initialization, rollout-time latent updates, and policy fine-tuning, providing useful evidence about few-shot adaptation.
Critical weaknesses
1.Potential target-dependent conditioning during policy training remains unresolved. Section 4.1.1 includes , , and in the context used to construct , while Section 4.2 conditions the prediction of on that latent. Algorithm 2 further holds one latent fixed across a prediction window without defining its context relative to every action target. Deployment instead uses past transitions only. This may be a notation issue rather than an implementation error, but the description does not rule out a consequential training shortcut. Separately, the world-model objective appears to reconstruct transitions already included in the encoder input. Reconstruction is not inherently invalid, but it should not alone be interpreted as evidence of predictive dynamics inference.
2.Mechanism attribution and some robustness conclusions exceed the available evidence. Appendix F provides useful few-shot controls, but the independent benefit of zero-shot rollout-time latent updates is not isolated. Point-Robot cannot demonstrate this benefit because , as the appendix acknowledges. Table 4 also removes reward and transition prediction jointly, preventing attribution specifically to dynamics supervision. Dataset-quality evidence is limited to two environments in the zero-shot setting, despite broader wording in Section 5. The description-robustness discussion is also overstated: Table 9 reports a decline from to with noisy descriptions, which does not support the accompanying characterization as “remarkably stable.”
Questions for authors
1.Causal information access: For each predicted action , which exact transition indices are used to compute its conditioning latent? Can the encoder access , , , or later information? Please specify the relationship between the fixed latent in Algorithm 2 and every action target in the prediction window, and clarify whether world-model targets appear in the corresponding latent inputs. If the implementation is causal, please reconcile the descriptions; otherwise, results excluding the relevant shortcuts would materially affect my assessment.
2.Independent contribution of zero-shot self-adaptation: On longer-horizon environments such as Cheetah-Vel, Ant-Dir, or Meta-World, how does keeping the language-initialized fixed throughout the episode compare with updating the latent from observed transitions? This would clarify whether the zero-shot gains arise primarily from initialization or from the proposed rollout-time adaptation mechanism.
3.Scope of the generalization evidence: Could the authors provide a controlled evaluation that separates dynamics variation from goal variation? I would particularly encourage adding Hopper-Param and Walker-Param experiments, with an appropriate baseline and a comparison that isolates the contribution of dynamics supervision. Could the authors also provide the missing Medium/Expert results for Cheetah-Vel and Ant-Dir, or explicitly delimit the scope of their dataset-quality conclusions? These additions would directly address existing claims rather than introduce a separate research direction.
4.Evaluation and reproducibility: Which tasks and metrics determine the “best-performing checkpoint”? How are initial target returns selected for unseen tasks, and do these choices use held-out evaluation outcomes or target-task offline data? How are context-dependent baselines initialized without target-task support trajectories? Finally, given the implementation discrepancy acknowledged in Appendix D, can the authors provide the exact implementation and configurations used for Tables 1–4?
4 comments
Q3. Scope of the generalization evidence
We added the two benchmarks suggested. In Hopper-Param and Walker-Param the reward function is shared and the tasks differ only in the dynamics (body mass, inertia, damping and friction), which is complementary to Point-Robot, Cheetah-Vel and Ant-Dir, where the dynamics are shared and only the goal varies. Following Meta-DT, each benchmark has 45 training tasks and 5 held-out test tasks with an episode length of 200, and the offline datasets are collected as in Appendix C. We compare with Meta-DT, which was originally evaluated on these two benchmarks.
Zero-shot test return on Mixed datasets (mean ± std over 5 seeds):
| Method | Hopper-Param | Walker-Param |
|---|---|---|
| Meta-DT | 345.6 ± 12.8 | 452.3 ± 14.9 |
| CoTA-DT | 361.5 ± 9.7 | 463.4 ± 16.6 |
CoTA-DT is above Meta-DT by 15.9 on Hopper-Param and by 11.1 on Walker-Param; the latter is within one standard deviation, so we regard the two methods as comparable on Walker-Param.
To isolate dynamics supervision, we removed and separately on the two dynamics-varying benchmarks and on Meta-World:
| Variant | Hopper-Param | Walker-Param | Meta-World |
|---|---|---|---|
| CoTA-DT (full) | 361.5 ± 9.7 | 463.4 ± 16.6 | 1292.5 ± 28.2 |
| w/o | 348.0 ± 13.9 | 436.3 ± 14.5 | 1124.3 ± 47.0 |
| w/o | 334.2 ± 10.5 | 412.2 ± 19.8 | 1088.7 ± 39.4 |
When only the dynamics vary, removing costs 27.3 and 51.2, about twice the cost of removing (13.5 and 27.1; the former is within one standard deviation). Transition prediction is thus the more important one-step signal there, while reward prediction still contributes because the reward depends on the forward velocity and hence on the dynamics. On Meta-World, where both vary, the two terms cost 168.2 and 203.8, against 345.2 for their joint removal in Table 4. In Point-Robot, Cheetah-Vel and Ant-Dir the transition function is identical across tasks, so cannot carry task-identifying information there by construction.
Both tables are added to Appendix E. For dataset quality, we restrict the conclusions of Sec. 5 to Point-Robot and Meta-World, as noted under W2.
Q4. Evaluation and reproducibility
Checkpoint. As in the released implementations of Meta-DT and T2DA, each run is evaluated every 500 training steps on the held-out test tasks. For every method, including all baselines, we report the checkpoint with the highest mean zero-shot return over these evaluations; the full test-return curves are shown in Fig. 2.
Initial target return. We follow T2DA: the initial return-to-go is a single constant per environment, shared by all tasks of that environment and by all DT-based methods. It uses neither evaluation outcomes nor target-task offline data.
Context-based baselines. We follow the aligned zero-shot protocol of T2DA: no data from the target task is available before evaluation, and a context-based method can infer its task representation only from the samples generated by its own policy during the evaluation episode. UNICORN is run in this way. Meta-DT is run in its zero-shot mode, i.e. without the prompt, with its context encoder fed by the rollout history. Prompt-DT is run under the same protocol as in Meta-DT, with its prompt constructed from the trajectories generated by its own policy during evaluation instead of from target-task demonstrations.
Implementation. The configurations used for Tables 1–4 are those listed in Table 6, and they are identical for the two variants mentioned in Appendix D. The variants differ only in the code path that feeds to the DT: a prefix token with one latent per training window (used for Tables 1–4) versus concatenation with updated at every step. The anonymous repository now includes the prefix-token path, selectable by a single option, alongside the concatenation variant (concat_z); the note in Appendix D is rewritten accordingly.
Thank you for the clarifications and additional experiments. The causal indexing explanation, fixed-latent comparison, and Hopper-Param/Walker-Param results substantially address my main concerns. The strengthened evidence makes me inclined to raise my rating from 2 (Weak rejection) to 3 (Weak acceptance).
Thank you for going through our response. Your questions made the paper clearer, and we really appreciate you raising the rating.
We thank the reviewer for the detailed reading. The questions on causal access, on mechanism attribution and on the evaluation protocol are the right ones to ask of this paper, and we answer each below.
W1 / Q1. Causal information access
The implementation is causal; the inconsistency is in our notation, which the revised PDF corrects. We answer the points of Q1 in turn.
Indices. The latent that conditions the prediction of at is computed from the H transitions preceding the action, τt−H:t−1=(sj,aj,rj,sj+1)j=t−Ht−1, whose last element is (st−1,at−1,rt−1,st). The encoder therefore sees st, which the policy receives at step t in any case, and has no access to at, rt, st+1 or any later quantity. Sec. 4.1.1 wrote the upper index as t, whereas Algorithm 3 wrote t−1; the latter is what the code does.
Fixed latent in Algorithm 2. For a training window covering steps t0,…,t0+Tc−1, the latent is Eϕ(τt0−H:t0−1), computed from the H transitions preceding the first step of the window, and it is held fixed for all Tc action targets. For example, with H=20, Tc=40 and t0=40, transitions 20–39 produce the latent that conditions a40,…,a79. No action target of the window, and no transition inside it, enters the encoder input. At deployment the latent is refreshed at every step (Algorithm 3), so the training-time latent is older than the deployment-time one, never newer.
World-model targets. The predicted transition (st,at,rt,st+1) is the one immediately following the context window, so rt, st+1 and Rt do not appear in the latent input; the objective is predictive rather than reconstructive. The phrase "sampled from a sub-trajectory" in Sec. 4.1.3 was inaccurate.
In addition, Eϕ is frozen during policy learning, so the policy loss cannot shape it to carry action information. None of the prediction targets is thus visible to the encoder at any stage of training, and all reported numbers are returns of the fixed policy on held-out tasks under Algorithm 3. The revised PDF uses τt−H:t−1 throughout (Secs. 4.1.1, 4.1.3, 4.2, Algorithms 1–2), and line 5 of Algorithm 2 now states the indices above.
W2 / Q2. Independent contribution of zero-shot self-adaptation
We agree that Appendix F isolates rollout-time updates only in the few-shot setting. We ran the requested comparison: the same zero-shot policies are evaluated with the language-initialized latent held fixed for the whole episode (zt=z0), on the three longer-horizon benchmarks (Mixed, 5 seeds).
Updating the latent from observed transitions improves the return by 10.7 on Cheetah-Vel and 114.4 on Meta-World; on Ant-Dir the difference (12.5) is within one standard deviation. With the latent fixed, CoTA-DT remains at or above T2DA-T (−90.7, 955.0 and 1103.5 in Table 1), so the zero-shot gains come from both sources: the language-grounded initialization, and on Cheetah-Vel and Meta-World a larger part from the rollout-time updates. The effect is larger than in the few-shot setting of Appendix F, where the initial latent is already computed from target-task trajectories. On Point-Robot (T=H) no update takes place, so the results there reflect the initialization alone. The comparison is added to Appendix F.
On the other points of W2: Sec. 5 now restricts the dataset-quality conclusions to Point-Robot and Meta-World, and the description-robustness paragraph in Appendix E.2 no longer says "remarkably stable"; it reports that a minimal description costs 0.9 and a distractor costs 3.6. The joint removal of reward and transition prediction in Table 4 is addressed under Q3.