acceptodds
Under review as a conference paper at ICLR 2027

Decision Transformer with Task-Aware Representations for Offline Meta-Reinforcement Learning

Abstract

To enable effective generalization in offline meta-RL, it is essential to accurately infer the underlying task through latent task representations, which provide a compact and transferable summary of task-specific characteristics for guiding policy generation. In this paper, we propose Contrastive Task-Aware Decision Transformer (CoTA-DT), a DT-based offline meta-RL framework that learns task representations by jointly integrating four complementary components: (i) contrastive learning from offline trajectories, (ii) alignment with natural-language task descriptions, (iii) one-step supervision via a task-conditioned world model, and (iv) long-term supervision via a task-conditioned Q function. The learned task representations are used as conditioning signals for a DT-based policy, enabling task-aware meta-policy learning. To further support generalization in both zero-shot and few-shot settings, we design a language-grounded, self-adaptive strategy that enables effective and reliable adaptation to unseen tasks. Notably, in CoTA-DT, the pretrained LLM serves as a shared backbone for language encoding, world modeling, Q-function estimation, and policy learning. This unified design facilitates parameter sharing, improving both efficiency and generalization. Extensive experiments across diverse offline meta-RL benchmarks demonstrate that CoTA-DT consistently outperforms prior methods, particularly in challenging zero-shot generalization scenarios.

Primary area:
reinforcement learning
Keywords:
Offline Meta-Reinforcement Learning, Decision Transformer, Task Representation Learning, Contrastive Learning, Large Language Models

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_UiffOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes CoTA-DT, a task-aware Decision Transformer framework for language-conditioned offline meta-reinforcement learning. A GRU context encoder learns task representations through trajectory contrastive learning, alignment with task descriptions, and auxiliary prediction of rewards, state transitions, and empirical returns-to-go. A frozen pretrained language model with a shared trainable LoRA adapter supports language encoding and auxiliary prediction, and is subsequently adapted for task-conditioned policy learning. For zero-shot deployment, the method initializes a task latent by optimizing its projected alignment with the target task description, then updates the latent after sufficient rollout history becomes available. Few-shot adaptation additionally incorporates support trajectories and permits lightweight fine-tuning of the LoRA adapter and action head. Experiments on Point-Robot, Cheetah-Vel, Ant-Dir, and Meta-World report higher mean returns than the evaluated baselines on Mixed datasets in both zero-shot and few-shot settings. Additional experiments investigate dataset quality on selected environments, component contributions, backbone choices, description robustness, and few-shot adaptation modes.

Critical strengths

1.A useful integration of language-based initialization and trajectory-based task inference. The framework connects task descriptions and interaction experience through a common task latent. This allows deployment without target-task support trajectories while subsequently incorporating observed transitions. The design is practically relevant: language provides initial task information, while experience may reveal characteristics not fully specified by the description. Using these complementary sources to condition a common policy is a useful extension of language-conditioned offline meta-RL.

2.Consistent reported improvements supported by several component analyses. Tables 1–2 report favorable mean performance across four benchmarks on Mixed datasets, while Table 4 examines the contribution of each representation-learning objective across all four environments. The small-DT control in Table 10 still exceeds the reproduced T2DA-T baseline on Meta-World, suggesting that the gains do not depend entirely on a large decision backbone. Appendix F further distinguishes initialization, rollout-time latent updates, and policy fine-tuning, providing useful evidence about few-shot adaptation.

Critical weaknesses

1.Potential target-dependent conditioning during policy training remains unresolved. Section 4.1.1 includes ata_t, rtr_t, and st+1s_{t+1} in the context used to construct ztz_t, while Section 4.2 conditions the prediction of ata_t on that latent. Algorithm 2 further holds one latent fixed across a prediction window without defining its context relative to every action target. Deployment instead uses past transitions only. This may be a notation issue rather than an implementation error, but the description does not rule out a consequential training shortcut. Separately, the world-model objective appears to reconstruct transitions already included in the encoder input. Reconstruction is not inherently invalid, but it should not alone be interpreted as evidence of predictive dynamics inference.

2.Mechanism attribution and some robustness conclusions exceed the available evidence. Appendix F provides useful few-shot controls, but the independent benefit of zero-shot rollout-time latent updates is not isolated. Point-Robot cannot demonstrate this benefit because T=H=20T=H=20, as the appendix acknowledges. Table 4 also removes reward and transition prediction jointly, preventing attribution specifically to dynamics supervision. Dataset-quality evidence is limited to two environments in the zero-shot setting, despite broader wording in Section 5. The description-robustness discussion is also overstated: Table 9 reports a decline from −7.9±0.4-7.9\pm0.4 to −11.5±0.5-11.5\pm0.5 with noisy descriptions, which does not support the accompanying characterization as “remarkably stable.”

Questions for authors

1.Causal information access: For each predicted action ata_t, which exact transition indices are used to compute its conditioning latent? Can the encoder access ata_t, rtr_t, st+1s_{t+1}, or later information? Please specify the relationship between the fixed latent in Algorithm 2 and every action target in the prediction window, and clarify whether world-model targets appear in the corresponding latent inputs. If the implementation is causal, please reconcile the descriptions; otherwise, results excluding the relevant shortcuts would materially affect my assessment.

2.Independent contribution of zero-shot self-adaptation: On longer-horizon environments such as Cheetah-Vel, Ant-Dir, or Meta-World, how does keeping the language-initialized z0z_0 fixed throughout the episode compare with updating the latent from observed transitions? This would clarify whether the zero-shot gains arise primarily from initialization or from the proposed rollout-time adaptation mechanism.

3.Scope of the generalization evidence: Could the authors provide a controlled evaluation that separates dynamics variation from goal variation? I would particularly encourage adding Hopper-Param and Walker-Param experiments, with an appropriate baseline and a comparison that isolates the contribution of dynamics supervision. Could the authors also provide the missing Medium/Expert results for Cheetah-Vel and Ant-Dir, or explicitly delimit the scope of their dataset-quality conclusions? These additions would directly address existing claims rather than introduce a separate research direction.

4.Evaluation and reproducibility: Which tasks and metrics determine the “best-performing checkpoint”? How are initial target returns selected for unseen tasks, and do these choices use held-out evaluation outcomes or target-task offline data? How are context-dependent baselines initialized without target-task support trajectories? Finally, given the implementation discrepancy acknowledged in Appendix D, can the authors provide the exact implementation and configurations used for Tables 1–4?

4 comments

We thank the reviewer for the detailed reading. The questions on causal access, on mechanism attribution and on the evaluation protocol are the right ones to ask of this paper, and we answer each below.

W1 / Q1. Causal information access

The implementation is causal; the inconsistency is in our notation, which the revised PDF corrects. We answer the points of Q1 in turn.

Indices. The latent that conditions the prediction of ata_t is computed from the HH transitions preceding the action, τt−H:t−1=(sj,aj,rj,sj+1)j=t−Ht−1\tau_{t-H:t-1}=\\{(s_j,a_j,r_j,s_{j+1})\\}_{j=t-H}^{t-1}, whose last element is (st−1,at−1,rt−1,st)(s_{t-1},a_{t-1},r_{t-1},s_t). The encoder therefore sees sts_t, which the policy receives at step tt in any case, and has no access to ata_t, rtr_t, st+1s_{t+1} or any later quantity. Sec. 4.1.1 wrote the upper index as tt, whereas Algorithm 3 wrote t−1t-1; the latter is what the code does.

Fixed latent in Algorithm 2. For a training window covering steps t0,…,t0+Tc−1t_0,\dots,t_0+T_c-1, the latent is Eϕ(τt0−H:t0−1)E_\phi(\tau_{t_0-H:t_0-1}), computed from the HH transitions preceding the first step of the window, and it is held fixed for all TcT_c action targets. For example, with H=20H=20, Tc=40T_c=40 and t0=40t_0=40, transitions 20–39 produce the latent that conditions a40,…,a79a_{40},\dots,a_{79}. No action target of the window, and no transition inside it, enters the encoder input. At deployment the latent is refreshed at every step (Algorithm 3), so the training-time latent is older than the deployment-time one, never newer.

World-model targets. The predicted transition (st,at,rt,st+1)(s_t,a_t,r_t,s_{t+1}) is the one immediately following the context window, so rtr_t, st+1s_{t+1} and RtR_t do not appear in the latent input; the objective is predictive rather than reconstructive. The phrase "sampled from a sub-trajectory" in Sec. 4.1.3 was inaccurate.

In addition, EϕE_\phi is frozen during policy learning, so the policy loss cannot shape it to carry action information. None of the prediction targets is thus visible to the encoder at any stage of training, and all reported numbers are returns of the fixed policy on held-out tasks under Algorithm 3. The revised PDF uses τt−H:t−1\tau_{t-H:t-1} throughout (Secs. 4.1.1, 4.1.3, 4.2, Algorithms 1–2), and line 5 of Algorithm 2 now states the indices above.

W2 / Q2. Independent contribution of zero-shot self-adaptation

We agree that Appendix F isolates rollout-time updates only in the few-shot setting. We ran the requested comparison: the same zero-shot policies are evaluated with the language-initialized latent held fixed for the whole episode (zt=z0z_t = z_0), on the three longer-horizon benchmarks (Mixed, 5 seeds).

RolloutCheetah-VelAnt-DirMeta-World
zt=z0z_t = z_0 fixed−87.2 ± 6.0968.8 ± 15.11178.1 ± 33.5
Self-adaptive (Table 1)−76.5 ± 4.7981.3 ± 16.81292.5 ± 28.2

Updating the latent from observed transitions improves the return by 10.7 on Cheetah-Vel and 114.4 on Meta-World; on Ant-Dir the difference (12.5) is within one standard deviation. With the latent fixed, CoTA-DT remains at or above T2DA-T (−90.7, 955.0 and 1103.5 in Table 1), so the zero-shot gains come from both sources: the language-grounded initialization, and on Cheetah-Vel and Meta-World a larger part from the rollout-time updates. The effect is larger than in the few-shot setting of Appendix F, where the initial latent is already computed from target-task trajectories. On Point-Robot (T=HT=H) no update takes place, so the results there reflect the initialization alone. The comparison is added to Appendix F.

On the other points of W2: Sec. 5 now restricts the dataset-quality conclusions to Point-Robot and Meta-World, and the description-robustness paragraph in Appendix E.2 no longer says "remarkably stable"; it reports that a minimal description costs 0.9 and a distractor costs 3.6. The joint removal of reward and transition prediction in Table 4 is addressed under Q3.

Q3. Scope of the generalization evidence

We added the two benchmarks suggested. In Hopper-Param and Walker-Param the reward function is shared and the tasks differ only in the dynamics (body mass, inertia, damping and friction), which is complementary to Point-Robot, Cheetah-Vel and Ant-Dir, where the dynamics are shared and only the goal varies. Following Meta-DT, each benchmark has 45 training tasks and 5 held-out test tasks with an episode length of 200, and the offline datasets are collected as in Appendix C. We compare with Meta-DT, which was originally evaluated on these two benchmarks.

Zero-shot test return on Mixed datasets (mean ± std over 5 seeds):

MethodHopper-ParamWalker-Param
Meta-DT345.6 ± 12.8452.3 ± 14.9
CoTA-DT361.5 ± 9.7463.4 ± 16.6

CoTA-DT is above Meta-DT by 15.9 on Hopper-Param and by 11.1 on Walker-Param; the latter is within one standard deviation, so we regard the two methods as comparable on Walker-Param.

To isolate dynamics supervision, we removed LrewL_{\mathrm{rew}} and LdynL_{\mathrm{dyn}} separately on the two dynamics-varying benchmarks and on Meta-World:

VariantHopper-ParamWalker-ParamMeta-World
CoTA-DT (full)361.5 ± 9.7463.4 ± 16.61292.5 ± 28.2
w/o LrewL_{\mathrm{rew}}348.0 ± 13.9436.3 ± 14.51124.3 ± 47.0
w/o LdynL_{\mathrm{dyn}}334.2 ± 10.5412.2 ± 19.81088.7 ± 39.4

When only the dynamics vary, removing LdynL_{\mathrm{dyn}} costs 27.3 and 51.2, about twice the cost of removing LrewL_{\mathrm{rew}} (13.5 and 27.1; the former is within one standard deviation). Transition prediction is thus the more important one-step signal there, while reward prediction still contributes because the reward depends on the forward velocity and hence on the dynamics. On Meta-World, where both vary, the two terms cost 168.2 and 203.8, against 345.2 for their joint removal in Table 4. In Point-Robot, Cheetah-Vel and Ant-Dir the transition function is identical across tasks, so LdynL_{\mathrm{dyn}} cannot carry task-identifying information there by construction.

Both tables are added to Appendix E. For dataset quality, we restrict the conclusions of Sec. 5 to Point-Robot and Meta-World, as noted under W2.

Q4. Evaluation and reproducibility

Checkpoint. As in the released implementations of Meta-DT and T2DA, each run is evaluated every 500 training steps on the held-out test tasks. For every method, including all baselines, we report the checkpoint with the highest mean zero-shot return over these evaluations; the full test-return curves are shown in Fig. 2.

Initial target return. We follow T2DA: the initial return-to-go is a single constant per environment, shared by all tasks of that environment and by all DT-based methods. It uses neither evaluation outcomes nor target-task offline data.

Context-based baselines. We follow the aligned zero-shot protocol of T2DA: no data from the target task is available before evaluation, and a context-based method can infer its task representation only from the samples generated by its own policy during the evaluation episode. UNICORN is run in this way. Meta-DT is run in its zero-shot mode, i.e. without the prompt, with its context encoder fed by the rollout history. Prompt-DT is run under the same protocol as in Meta-DT, with its prompt constructed from the trajectories generated by its own policy during evaluation instead of from target-task demonstrations.

Implementation. The configurations used for Tables 1–4 are those listed in Table 6, and they are identical for the two variants mentioned in Appendix D. The variants differ only in the code path that feeds zz to the DT: a prefix token with one latent per training window (used for Tables 1–4) versus concatenation with sts_t updated at every step. The anonymous repository now includes the prefix-token path, selectable by a single option, alongside the concatenation variant (concat_z); the note in Appendix D is rewritten accordingly.

Thank you for the clarifications and additional experiments. The causal indexing explanation, fixed-latent comparison, and Hopper-Param/Walker-Param results substantially address my main concerns. The strengthened evidence makes me inclined to raise my rating from 2 (Weak rejection) to 3 (Weak acceptance).

Thank you for going through our response. Your questions made the paper clearer, and we really appreciate you raising the rating.

Reviewer_YCWBOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes CoTA-DT, a Decision Transformer-based framework for offline meta-RL that learns task representations from multiple complementary signals for reliable task identification. A GRU-based context encoder maps sub-trajectories to task latents, which are jointly trained with a sub-trajectory InfoNCE loss, a CLIP-style alignment with natural-language task descriptions, a task-conditioned world model, and a return-prediction objective. A single LoRA-adapted LLM serves as the shared backbone for these objectives and for the DT policy, which is conditioned on the task latent as a prefix token. For zero-shot adaptation, the task latent is initialized by aligning it with the target-task description (latent inversion) and refined during rollout from recent transitions, while the few-shot setting initializes it from the average latent of the support trajectories. Experiments on Point-Robot, Cheetah-Vel, Ant-Dir, and Meta-World report the best returns against language-conditioned, prompt-based, and context-based baselines, including T2DA-T.

Critical strengths

S1. A well-motivated adaptation mechanism.

The paper addresses an important problem in offline meta-RL: reliable task identification when little or no target-task data is available. I find the language-grounded latent inversion (Eq. 9), combined with the self-adaptive rollout, to be a simple mechanism. It initializes the task latent from the task description and then hands task inference over to the trajectory encoder. The same procedure naturally extends to the few-shot setting by replacing the zero initialization with the support-set average zˉ\bar z, which unifies both settings within a single framework.

S2. Consistent empirical improvements across diverse benchmarks.

CoTA-DT achieves the best reported returns on all four benchmarks in both the zero-shot and few-shot settings (Tables 1–2), as well as on the Expert and Medium datasets (Table 3). The gains are particularly notable on Meta-World (+189.0 zero-shot and +213.1 few-shot over T2DA-T). This suggests that the approach may be especially beneficial when tasks are highly heterogeneous.

Critical weaknesses

W1. Novelty.

The methodological novelty of CoTA-DT does not seem entirely clear, as each representation-learning component seems to build closely on existing work. Contrastive or metric-learning objectives for task representation learning have been widely adopted in context-based offline meta-RL (e.g., FOCAL, CORRO, and UNICORN), and the sub-trajectory InfoNCE objective (Eq. 1) appears to be a fairly standard instance of this approach. Likewise, the language alignment (Eqs. 2–4), the task-conditioned world model (Eq. 5), and the prefix-token DT (Sec. 4.2) appear to closely correspond to the contrastive language-decision pre-training (Sec. 3.3), the generalized world model (Sec. 3.2), and the conditioning scheme of T2DA-T in T2DA, respectively (see also Meta-DT and Prompt-DT). Although Appendices A-B describe differences from T2DA, Sections 4.1.2–4.1.3 do not appear to cite T2DA. In addition, the introduction seems to characterize T2DA as a single-source method, even though T2DA already combines trajectory dynamics with language.

W2. Analysis of the learned representations.

While the paper centers on task-aware representations, the direct analysis of these representations is currently provided mainly through a t-SNE plot on Point-Robot (Figure 3) and a retrieval accuracy reported only for CoTA-DT (Appendix G). Since no representation-level comparison with baselines such as T2DA, Meta-DT, or UNICORN seems to be provided, it remains somewhat unclear whether the representations of CoTA-DT are indeed more discriminative or informative. In addition, the latent visualization appears to be provided only for Point-Robot, a relatively simple environment, and not for Cheetah-Vel, Ant-Dir, or Meta-World. It therefore remains unclear whether the well-structured representations in Figure 3 also emerge in these more complex environments.

Questions for authors

Q1. T2DA (Appendix F, Table 9) reported that jointly training the decision embedding and the language alignment degraded performance compared with staged training. Could the authors explain why joint optimization is expected to be beneficial in CoTA-DT, and provide a direct comparison between staged and joint training under the same setting?

Q2. Given that the paper centers on task-aware representations, could the authors provide a more comprehensive representation analysis? For example, a representation-level comparison with baselines such as T2DA, Meta-DT, and UNICORN would help clarify whether CoTA-DT indeed learns more discriminative or informative task representations. In addition, latent visualizations in environments beyond Point-Robot, particularly Meta-World with its heterogeneous manipulation tasks, would help show whether the well-structured representations in Figure 3 generalize to more complex settings.

Q3. Could the authors report the values of κα\kappa_\alpha, δ\delta, and mm, which do not appear in Table 6? In addition, the default values were selected via a sweep on Point-Robot Mixed, Table 7 reports zero-shot test returns, and the best-performing checkpoint is reported. Could the authors clarify which metric was used for this selection, and whether a separate set of validation tasks was used for hyperparameter and checkpoint selection?

7 comments

We thank the reviewer for the detailed summary and for the questions, which concern the positioning of the method, the analysis of the representations, and the evaluation protocol.

W1 / Q1. Novelty, relation to T2DA, and joint versus staged training

We agree that the individual objectives belong to established families; the revised PDF cites the related work where each is introduced (Secs. 4.1.1–4.1.3, 4.2). The losses are close in form and differ in what they are applied to. Eq. (1) is standard InfoNCE, in the family of FOCAL, CORRO and UNICORN, but each anchor is paired with a nearby sub-trajectory of the same task and contrasted with the other sub-trajectories in the minibatch, rather than with samples grouped by task label (FOCAL) or relabeled negatives (CORRO). Eqs. (2)–(4) share the symmetric CLIP form of T2DA, which fixes the trajectory encoder and moves only the text side; here the alignment gradient reaches Eϕ\mathcal{E}_\phi. Eq. (5) follows the world models of Meta-DT and T2DA, plus return-to-go supervision (Eq. 6). The prefix token follows T2DA-T, carrying a trajectory-side latent instead of the text embedding. We also corrected the introduction: T2DA combines dynamics with language in training, but at deployment T2DA-T conditions on the text embedding alone, which cannot be updated from observed transitions.

Our contribution is the role given to the latent. One zz is at once the conditioning variable of the policy, the variable inverted from the description (Eq. 9), and the variable refreshed from transitions during rollout. The description is not fed to the policy. Eq. (9) maps it to a latent in the output space of Eϕ\mathcal{E}_\phi, so the policy receives z0z_0 and the later encoder outputs through the same prefix token, and the few-shot setting only changes the starting point of Eq. (9) from 0\mathbf{0} to zˉ\bar z. T2DA-T has nothing trajectory-derived to refresh at deployment; Meta-DT has no language anchor to initialize from.

The appendix shows the two sources acting on one variable on Meta-World. In the zero-shot setting, the language-initialized latent held fixed for the whole episode gives 1178.1 (T2DA-T: 1103.5), and refreshing it from the rollout gives 1292.5 (Table 15). At k=5k{=}5, a data-initialized latent refreshed from rollout gives 1318.0, and adding language-grounded inversion gives 1384.7 with no parameter update (Table 14).

Q1. With architecture, supervision signals and hyperparameters unchanged, we followed the staged order of T2DA: (i) train Eϕ\mathcal{E}_\phi with the trajectory-side losses; (ii) freeze it and train Projz\mathrm{Proj}_z, Projh\mathrm{Proj}_h, θL\theta_L with LalignL_{\mathrm{align}}; (iii) train the policy. Zero-shot return on Mixed (5 seeds):

SchedulePt-RobotMeta-World
w/o LalignL_{\mathrm{align}} (Table 4)−12.3 ± 0.5868.0 ± 36.7
Staged−8.3 ± 0.21137.3 ± 41.6
Joint (CoTA-DT)−7.9 ± 0.41292.5 ± 28.2

Joint training is better on both benchmarks, slightly on Point-Robot (0.4) and by 155.2 on Meta-World, where the staged variant falls close to T2DA-T (1103.5 ± 36.3). We believe this is consistent with Table 9 of T2DA: there the policy consumes the text embedding and the decision embedding is only an alignment target, so fixing it first is natural. In CoTA-DT the language-initialized z0z_0 is later replaced by outputs of Eϕ\mathcal{E}_\phi, so the two must agree in the latent space itself, which requires the alignment gradient to reach Eϕ\mathcal{E}_\phi.

With Eϕ\mathcal{E}_\phi frozen, the latents are arranged by the trajectory-side losses alone, without reference to the descriptions, and Projz\mathrm{Proj}_z and Projh\mathrm{Proj}_h have to fit the correspondence from 45 or 18 training tasks. Such a fit matches the training tasks but does not determine where an unseen description is inverted. When the gradient reaches Eϕ\mathcal{E}_\phi, the latents themselves are arranged consistently with the descriptions, and the inverted z0z_0 falls closer to what Eϕ\mathcal{E}_\phi produces for that task. This agrees with the table: the gap is small on Point-Robot, where the held-out goals lie between the 45 training goals, and large on Meta-World, where the 18 training tasks are heterogeneous. Joint training is kept stable by switching LalignL_{\mathrm{align}} on only after the first 10 epochs of representation learning and by keeping the trajectory-side losses active throughout. Details are in the new Appendix E.4.

W2 / Q2. Analysis of the learned representations

We agree that Fig. 3 and the retrieval accuracy concern CoTA-DT only. The revised Appendix G (now "Representation Analysis") adds the two analyses requested.

Comparison with baselines (Appendix G.2). For each method we take the encoder that produces its task representation (the context encoders of Meta-DT and UNICORN, the trajectory encoder of T2DA, Eϕ\mathcal{E}_\phi for CoTA-DT), encode the same Mixed trajectories of the held-out test tasks, and report the kk-NN task-identification accuracy (%, k=5k{=}5, leave-one-out), i.e., how often the nearest representations of a trajectory belong to the same task. With 5 and 4 held-out tasks, the chance level is 20% on Point-Robot and 25% on Meta-World.

MethodPt-RobotMeta-World
Meta-DT61.177.1
UNICORN92.592.4
T2DA90.894.4
CoTA-DT96.297.9

CoTA-DT is the most discriminative on both benchmarks, 3.7 and 3.5 points above the best baseline (UNICORN on Point-Robot, T2DA on Meta-World), on tasks unseen in training. The advantage is kept on Meta-World, whose tasks are heterogeneous manipulation skills rather than points on a low-dimensional goal manifold. Discriminability alone does not determine zero-shot return: UNICORN is close to T2DA here but well below it in Table 1, since a context-based latent cannot be initialized from a description. Alignment with language is therefore reported separately (Appendix G.1): on Mixed, the rank-1 retrieval accuracy between zˉi\bar z_i and wiw_i is 82% on Point-Robot and 78% on Meta-World.

Visualization on Meta-World (Appendix G.3, Fig. 4). We repeat the visualization of Fig. 3 for the 22 heterogeneous Meta-World tasks, with and without language alignment. Without alignment, trajectory representations and text embeddings form two disjoint groups, and sub-trajectory latents are only loosely grouped by task. With the full objective, zˉi\bar z_i and wiw_i appear as adjacent pairs for most tasks, and sub-trajectory latents form compact, well-separated clusters at their text anchors. The structure seen in Fig. 3 therefore does not depend on the goal manifold of Point-Robot. The correspondence is looser for the 4 held-out tasks, whose descriptions are not seen in training; this is the case in which the rollout-time updates add task information (Table 15: 1178.1 with the latent held fixed, 1292.5 with the updates).

We also note how these analyses relate to the main results. In our setting (Sec. 3.1) the objective is the return of the meta-policy on unseen tasks, and the task representation is an intermediate variable that matters through the policy conditioned on it. We therefore regard the policy-level results as the main evidence for the representations: Tables 1–3 for the full method, and Table 4, where each representation-learning loss is removed and the return is measured. The kk-NN accuracy and the visualizations are supplementary. They help explain these results, but, as the UNICORN case shows, they do not by themselves predict the return. The two levels are consistent: the representation without alignment in Fig. 4 (left) corresponds to the w/o LalignL_{\mathrm{align}} ablation, which loses 424.5 on Meta-World.

Q3. Hyperparameters and selection protocol

The InfoNCE temperature is fixed at κα=0.1\kappa_\alpha = 0.1. The offset of the positive sub-trajectory is sampled uniformly, δ∼U1,…,20\delta \sim \mathcal{U}\\{1,\dots,20\\}, independently for each anchor. For zˉi\bar z_i, the set of sub-trajectories is the one contained in the current minibatch: each minibatch holds 128 sub-trajectories sampled across the training tasks, and zˉi\bar z_i averages those of task ii. mm is therefore determined by the batch size and equals 128/N128/N on average, about 2.8 on the three 45-task benchmarks and 7.1 on Meta-World. These values are now listed in Table 6.

Selection. A single configuration is used for all four benchmarks. It was chosen from the one-at-a-time sweep on Point-Robot Mixed (Table 7), scored by zero-shot return, and was not re-tuned on Cheetah-Vel, Ant-Dir or Meta-World. The results on these three benchmarks therefore involve no hyperparameter tuning on their tasks, and we do not claim that the values are optimal there. Table 7 also shows that moderate changes around the defaults have a small effect, which is why one configuration can be shared. We did not hold out separate validation tasks; as in Meta-DT and T2DA, the best-performing checkpoint is reported for every method, including all baselines.

Reviewer_JB3MOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes CoTA-DT, an offline meta-RL framework that aims to improve task identification and generalization to unseen tasks. CoTA-DT combines contrastive learning, language alignment, world modeling, and Q-function supervision to learn task-aware representations, which are used to condition a Decision Transformer policy for zero-shot and few-shot adaptation. Experiments on four offline meta-RL benchmarks show that CoTA-DT consistently outperforms existing methods, particularly in zero-shot settings.

Critical strengths

1.The paper proposes a unified framework that combines contrastive learning, language alignment, world modeling, and Q-function supervision to learn task-aware representations. The language-grounded initialization and self-adaptive rollout provide a practical approach to zero-shot and few-shot generalization in offline meta-RL.

2.The authors conduct extensive experiments across four offline meta-RL benchmarks, demonstrating consistent improvements over existing methods, particularly on Meta-World. The component ablations, backbone comparisons, and adaptation studies further support the effectiveness of the proposed framework.

Critical weaknesses

1.The proposed method builds upon several techniques already explored in Meta-DT and T2DA, including world-model-based task representation learning and language alignment. Although Table 4 demonstrates the contribution of individual losses, there is no direct comparison between joint and staged training under the same architecture and supervision signals. It remains unclear whether joint optimization provides additional benefits over existing training approaches.

2.The consistency of task representations between training and inference is not sufficiently explained. In Sec. 4.1.1, the task representation ztz_t is computed from τt−H\tau_{t-H}, which includes the current transition, while the same representation is used to predict the corresponding action, reward, and next state. However, inference only uses past transitions. Furthermore, Eq. (9) optimizes the initial task representation through projected language similarity without explicitly constraining it to the distribution of trajectory-derived representations. These issues raise concerns about the reliability of task conditioning during zero-shot adaptation.

Questions for authors

Q1. How does joint optimization of the representation learning objectives compare with the staged training used in T2DA? Can you provide a comparison under the same architecture and supervision signals?

Q2. In Sec. 4.1.1, ztz_t is computed from τt−H\tau_{t-H}, which includes the current transition. Does the training implementation use this transition when predicting the corresponding action, reward, and next state? If so, how is information leakage prevented?

Q3. In Eq. (9), what ensures that the optimized z0z_0 is consistent with the distribution of trajectory-derived representations used during policy training? How sensitive is zero-shot performance to this initialization?

4 comments

We thank the reviewer for the positive assessment and for the two questions on training schedule and on train/inference consistency of the latent, which we answer below.

W1 / Q1. Joint versus staged training

We ran this comparison with the architecture, the four supervision signals and all hyperparameters unchanged, replacing only the schedule by the three-stage order of T2DA: (i) train Eϕ\mathcal{E}_\phi with the trajectory-side losses (Eq. (7) without LalignL_{\mathrm{align}}); (ii) freeze Eϕ\mathcal{E}_\phi and train Projz\mathrm{Proj}_z, Projh\mathrm{Proj}_h, θL\theta_L with LalignL_{\mathrm{align}} alone; (iii) train the policy as in Algorithm 2. Zero-shot return on Mixed (mean ± std, 5 seeds):

SchedulePt-RobotMeta-World
w/o LalignL_{\mathrm{align}} (Table 4)−12.3 ± 0.5868.0 ± 36.7
Staged−8.3 ± 0.21137.3 ± 41.6
Joint (CoTA-DT)−7.9 ± 0.41292.5 ± 28.2

Joint training is better on both benchmarks, by a small margin on Point-Robot (0.4) and a large one on Meta-World (155.2). Aligning on top of a frozen encoder recovers most of the benefit of alignment on Point-Robot but only part of it on Meta-World, where the staged variant is close to T2DA-T (1103.5 ± 36.3).

The gap follows from how zz is used. The policy is conditioned on zz, and z0z_0 is obtained by inverting Projz\mathrm{Proj}_z at an unseen description (Eq. 9). With Eϕ\mathcal{E}_\phi frozen, the projection heads alone must fit the map between descriptions and latents from 45 or 18 task means, and nothing constrains the fit away from the training tasks; under joint training Eϕ\mathcal{E}_\phi itself moves toward the description geometry, and the inverted z0z_0 lands closer to trajectory-derived latents. This matters little on Point-Robot, whose 45 goals densely cover a 2D space, and much on Meta-World, whose 18 tasks are heterogeneous. Staging suits T2DA because its policy is conditioned on the text embedding; in CoTA-DT the language-initialized z0z_0 is later replaced by outputs of Eϕ\mathcal{E}_\phi, so the two must agree in the latent space itself. The full protocol and analysis are in the new Appendix E.4 of the revised PDF.

W2 / Q2. Consistency of the latent between training and inference

The training implementation does not use the current transition; the inconsistency is in our notation, and we thank the reviewer for pointing it out. In both training and inference, the latent at step tt is computed from the HH transitions that precede the action, τt−H:t−1=(sj,aj,rj,sj+1)j=t−Ht−1\tau_{t-H:t-1}=\\{(s_j,a_j,r_j,s_{j+1})\\}_{j=t-H}^{t-1}, whose last element is (st−1,at−1,rt−1,st)(s_{t-1},a_{t-1},r_{t-1},s_t). It contains sts_t, which is an input to the policy at step tt in any case, and contains neither ata_t, rtr_t nor st+1s_{t+1}. Sec. 4.1.1 wrote the upper index of this window as tt, whereas Algorithm 3 wrote t−1t-1; the latter is what the code does, and it is the only reading consistent with the stated count of HH transitions.

Accordingly: (i) for the world model and the Q function, the predicted transition (st,at,rt,st+1)(s_t,a_t,r_t,s_{t+1}) is the one that immediately follows the context window, so rtr_t, st+1s_{t+1} and RtR_t are not in the encoder input; the phrase "sampled from a sub-trajectory" in Sec. 4.1.3 was inaccurate. (ii) For policy training, the latent of a DT window is computed from the HH transitions preceding the window and held fixed within it, so no action target of the window enters the encoder input. (iii) Eϕ\mathcal{E}_\phi is frozen during policy learning, so the policy loss cannot shape it to carry action information. Hence none of the prediction targets (ata_t, rtr_t, st+1s_{t+1}, RtR_t) is visible to the encoder at any stage of training, and there is no information leakage. All reported numbers are environment returns of the fixed policy on held-out tasks under Algorithm 3, where only past transitions exist. The revised PDF uses τt−H:t−1\tau_{t-H:t-1} throughout (Secs. 4.1.1, 4.1.3, 4.2, Algorithms 1–2) and states the convention once in Sec. 4.1.1.

Q3. Consistency and sensitivity of z0z_0

Eq. (9) has no explicit constraint; consistency comes from training Projz\mathrm{Proj}_z only on trajectory-derived task means zˉi\bar z_i, jointly with Eϕ\mathcal{E}_\phi. Three results test this. (i) With the same policy, a latent computed by Eϕ\mathcal{E}_\phi from k=5k{=}5 target-task trajectories (Appendix F, online) gives −8.6-8.6 / 1318.01318.0 on Point-Robot / Meta-World, against −7.9-7.9 / 1292.51292.5 for the inverted z0z_0: the two are on par. (ii) Applying Eq. (9) to that latent improves it (−8.6→−6.9-8.6\to-6.9, 1318.0→1384.71318.0\to1384.7), so inversion stays within the region the policy was trained on. (iii) On Point-Robot (T=HT=H) the whole episode uses z0z_0 alone and still exceeds all baselines.

Zero-shot return is sensitive to the quality of z0z_0: it falls to −12.3-12.3 / 868.0868.0 without alignment (Table 4), by 3.63.6 with a distractor in the description (Table 9). On the longer-horizon benchmarks z0z_0 covers only the first 2020 of 200200 steps.

Reviewer_JB3MOpenReview ↗

Response to Authors' Rebuttal

Thanks for the author's reply. All my concerns have been solved. I maintain my weak accept score.

Thank you for taking the time to read our response, and for the helpful comments along the way. We're glad the concerns have been addressed.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.