acceptodds
Under review as a conference paper at ICLR 2027

Language Model Augmented Decision Transformer for Multi-Task Offline Reinforcement Learning

Abstract

Offline multi-task reinforcement learning (MTRL) aims to learn a unified policy from heterogeneous offline datasets that generalizes across tasks. However, existing offline MTRL methods often struggle with two fundamental issues: severe per-task data scarcity and limited cross-task generalization. In this paper, we propose Language-Model Augmented Decision Transformer (LaMA-DT), a unified sequence-modeling framework that leverages a single large language model (LLM) for both trajectory augmentation and multi-task policy learning. LaMA-DT uses natural-language task descriptions to guide task-aligned trajectory augmentation through a set of supervised generation and alignment objectives, and incorporates a scoring-and-filtering mechanism to retain high-quality augmented trajectories. The same LLM is then repurposed as a language-conditioned Decision Transformer and fine-tuned on a mixture of real and augmented data. Experiments on Meta-World and MuJoCo multi-task benchmarks demonstrate that LaMA-DT consistently outperforms competitive baselines, particularly in low-data regimes, showing that language-guided data augmentation substantially improves cross-task generalization.

Primary area:
reinforcement learning
Keywords:
Multi-Task Offline Reinforcement Learning, Decision Transformer, Large Language Models, Trajectory Augmentation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_ngLHOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes LaMA-DT, a framework that uses a pretrained language model for trajectory augmentation and policy learning in offline multi-task reinforcement learning. The model is adapted through six objectives covering dynamics prediction, trajectory generation, and task–trajectory alignment, using objective-specific and shared LoRA adapters. Generated trajectories are scored across these objectives and filtered before being combined with real data. The same language model is then adapted into a language-conditioned Decision Transformer. Experiments on Meta-World and MuJoCo report improvements over the evaluated baselines, particularly in low-data settings, with additional analyses of augmentation quality, component contributions, and few-shot adaptation to unseen tasks.

Critical strengths
  1. The framework offers a coherent way to combine language-conditioned trajectory augmentation with multi-task policy learning. Reusing the language model and shared adapter connects the two stages and provides a reasonable approach to addressing limited per-task data.

  2. The low-data results provide some empirical support for the augmentation pipeline. In Table 12, removing augmentation while retaining shared-adapter initialization reduces performance on both ML30 and Cheetah-dir, suggesting that the gains are not solely attributable to policy initialization.

Critical weaknesses
  1. The trajectory-generation procedure is insufficiently specified. The main text describes forward generation from an initial state–action pair, whereas Appendix E.3 introduces intermediate teacher-forced states and preservation of the final state. The source of these states and their role in generation are unclear. This distinction materially affects whether the reported quality reflects learned long-horizon generation or reconstruction anchored to real trajectories.

  2. The coverage results appear inconsistent with the stated metric. Appendix E.6 defines coverage through grid cells occupied by valid generated transitions, yet Table 4 reports an increase from 99% before filtering to 109% afterward. If filtering selects a subset of the same candidates under fixed validity criteria and grid construction, coverage cannot increase. Clarifying any resampling or budget differences is necessary to substantiate the claim that filtering improves quality while expanding valid coverage.

Questions for authors
  1. What information is actually used to generate a trajectory? The main text says generation starts from an initial state–action pair, but Appendix E.3 mentions intermediate teacher-forced states and a preserved final state. Are these taken from the original real trajectory? If so, how much of the generated trajectory is anchored to real data?

  2. Why does Valid Coverage increase from 99% to 109% after filtering in Table 4? If filtering only removes candidates, the remaining subset should not cover more valid grid cells under the same metric. Were additional samples generated, or were the two results computed using different candidate sets?

  3. Table 12 shows that augmentation helps, but is the proposed pipeline better than other ways of generating additional data? For example, have you compared it with MTDIFF-generated data while keeping the same LLM policy, amount of synthetic data, and policy-training budget? This would help establish what the proposed pipeline adds.

5 comments

We thank the reviewer for the careful reading.

W1 & Q1. What information is used to generate a trajectory

Yes. The intermediate teacher-forced states are taken from the same real trajectory that provides the seed pair (s0, a0), and the preserved final state is the final state of that trajectory. As described in App. E.3, long horizons are generated in chunks of 50 steps (Tab. 6). Each chunk starts from the real state at its first step, and all remaining states, actions, and rewards of the chunk are rolled out autoregressively by the O3 adapter at temperature 0.4. The part of an augmented trajectory that is anchored to real data is therefore the seed pair, one state in every 50 steps, and the final state. This reduces wall-clock time by roughly 3× relative to fully autoregressive generation (App. E.3).

The quality reported in the paper reflects the generated content and not the anchors:

(i) Dyn. MSE in Tab. 4 is computed at every generated transition, by resetting the simulator to the generated state and applying the generated action (App. E.6). An anchor affects only the first transition of a chunk.

(ii) The N = 4 candidates of a seed share the same anchors. Selecting among them with the scores changes the policy result from 883.1 to 937.8 on Cheetah-dir (Tab. 3), and the sampling temperature changes it from 896.8 (T = 0) to 937.8 (T = 0.4) (Tab. 7). Neither would happen if the trajectories were determined by their anchors.

(iii) The generated states are not copies of real states. Their average distance to the nearest real state is 0.4376 in normalized space, and a Valid Coverage above 100% means that validated generated transitions fall in cells that contain no real state (Tab. 4).

W2 & Q2. Valid Coverage in Tab. 4

We thank the reviewer for pointing out this inconsistency. The reasoning is correct: filtering only selects among the generated candidates, so it cannot increase coverage under the same validity criterion and grid. We checked the entry against our logs, and the unfiltered value in Tab. 4 was indeed filled in incorrectly. The correct value is 117%, not 99%. To answer Q2, no additional samples were generated, and the filtered set is the top-1 selection from the same candidates. The other entries of Tab. 4 are unaffected.

With the correct value, Valid Coverage is 117% before filtering and 109% after filtering, which is consistent with the definition in App. E.6. The unfiltered candidates reach more cells, but they are less faithful to the dynamics and to the real state distribution (Dyn. MSE 0.0089, State KL 0.42). The filter removes the least reliable candidates, which lowers Dyn. MSE to 0.0031 and State KL to 0.18, and the removed candidates account for 8 points of coverage.

Two points follow. First, the filtered data are not confined to the real data. Their coverage remains above 100% and above MTDIFF (102%), so they still contain validated transitions in cells that no real state occupies. Second, the filter gives up a small part of the coverage for a large improvement in fidelity, and the policy results show that this is beneficial: with the same amount of augmented data, filtering improves ML30 from 62.1 to 69.8, Cheetah-dir from 883.1 to 937.8, and Ant-dir from 285.8 to 328.5 (Tab. 3). These policy results do not depend on the corrected entry. We have corrected Tab. 4 and the corresponding sentence in Sec. 4.

Q3. Comparison with MTDIFF-generated data

Tab. 4 compares the two generators directly on data quality, and Tab. 1 compares MTDIFF as a complete method (57.5 vs. 69.8 on ML30, 536.8 vs. 937.8 on Cheetah-dir). What the paper contains is the effect of data fidelity with the LLM policy, the amount of synthetic data, and the training budget all fixed (w/o Filtering, Tab. 3): our unfiltered data (Dyn. MSE 0.0089, State KL 0.42) give 883.1 on Cheetah-dir, and the filtered data (0.0031, 0.18) give 937.8. MTDIFF data are less faithful than our unfiltered data on both metrics (0.0107, 0.51).

What the proposed pipeline adds over generating additional data in general is therefore the multi-objective filter and the description-conditioned generation. Without task-discriminative descriptions, the same pipeline drops to 746.9 on Cheetah-dir and 40.4 on ML30 (Tab. 10).

Dear Reviewer, we would be grateful if you could let us know whether our response has addressed your concerns, and we are happy to provide any further clarification before the discussion period ends.

Reviewer_91TTOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

In this work, the authors aim to address per-task data scarcity and improve generalization in offline multi-task RL. To this end, they propose Language-Model Augmented Decision Transformer (LaMA-DT), which first trains six objective-specific LoRA adapters on an LLM to predict dynamics, fill in missing trajectory segments, generate complete trajectories, reconstruct trajectories backward, explain trajectories in natural language, and generate trajectories from task descriptions. They then generate candidate trajectories and score them using the trained adapters. The highest-scoring trajectories are retained as augmented data and combined with the original data for policy learning. Experimental results demonstrate good generalization across different tasks.

Critical strengths
  • The overall idea of training different LoRA adapters to support learning across different subtasks is interesting.

  • The authors provide comprehensive experiments, including numerous ablation studies, to support their claims.

Critical weaknesses
  • The underlying idea is relatively simple, but the overall design seems overly complicated. The different LoRA adapters primarily contribute to trajectory generation and scoring for offline data augmentation, rather than directly to policy training. The method would be more interesting if its components played a more active role in policy learning.

  • The scoring functions come from the same model trained on the same dataset, so they may not provide an independent assessment and could favor trajectories that overfit the original data. It may be worth exploring different LLMs as independent judges of the generated trajectories.

Questions for authors

Q1. Have you conducted an ablation study that retains all LoRA training objectives but removes one score at a time from the overall scoring function?

Q2. Could you explain how the method specifically handles multiple tasks, particularly unseen tasks? Does its generalization ability come entirely from conditioning on task prompts?

Q3. Have you explored alternative weighting schemes instead of simply averaging all six scores?

3 comments

We thank the reviewer for the constructive comments and questions.

W1. Design complexity and the role of the adapters in policy learning

The design is simpler than it may appear. LaMA-DT uses one frozen LLM throughout, the adapters are shared by all tasks and trained in a single loop that samples one objective per step (Alg. 2), and the deployed policy is a plain DT with one adapter (App. F.7). What makes the design look complicated is the number of objectives, and each of them is included because it supervises a different aspect of trajectory generation: O1 captures the environment dynamics, O2 local temporal consistency, O3 long-horizon forward generation, O4 backward reasoning from the final state, and O5 and O6 the alignment between trajectories and task descriptions (Sec. 3.2). Each aspect is valuable: removing any single objective lowers performance on all three benchmarks, and removing a group lowers it further (w/o O2–O4: 69.8 → 50.2 on ML30; w/o O5 and O6: 57.8; Tab. 3).

That the objective-specific adapters are not used directly in policy training is a deliberate choice. Each of them is optimized for its own target, e.g., next-state prediction for O1, backward reconstruction for O4, and text generation for O5, and these targets differ from the policy objective, which is action prediction conditioned on the return-to-go (Eq. 5). The objectives are heterogeneous and their gradients interfere, which is the reason they are given separate adapters in the first place (Sec. 3.2.3), and attaching these adapters to the policy would introduce the same interference into policy learning. LaMA-DT instead transfers what the objectives have in common. The shared adapter is updated jointly by all six objectives and aggregates the knowledge that transfers across them, and the policy adapter is initialized from it (Sec. 3.3.2). All six objectives therefore take part in policy learning through the shared adapter. The experiments confirm its effect: removing the shared adapter lowers ML30 by 6.7, Cheetah-dir by 59.8, and Ant-dir by 15.3 (Tab. 3), and with this initialization LaMA-DT converges faster than the baselines in the early phase of training (Fig. 4, App. F.1).

W2. Independence of the scoring functions

We address the three parts of this comment in turn.

(1) Whether the scores provide an independent assessment. The scoring adapters share the backbone and the training data with the generator, but the assessment is not a self-evaluation. By design, candidates are sampled from the forward model O3 only, and five of the six scores come from other adapters that model the trajectory in a different direction: one-step prediction (O1), interpolation toward the final state (O2), backward reconstruction (O4), trajectory-to-description (O5), and description-to-trajectory (O6). A forward rollout that drifts receives a low likelihood from the one-step and backward models even when the forward model rates it as likely. Empirically, we verified the filter against a judge that is independent of both the model and the offline data, the MuJoCo simulator. Dyn. MSE in Tab. 4 resets the simulator to each generated state, applies the generated action, and compares the true next state with the generated one. The filter never queries the simulator, yet the trajectories it keeps have 2.9× lower true dynamics error (0.0031 vs. 0.0089). The scores therefore agree with an independent assessment.

(2) Whether the filter favors trajectories that overfit the original data. A filter with this bias would select near-copies of the training trajectories, and the selected set would collapse onto the real data. Tab. 4 shows that this does not happen. The average distance to the nearest real state changes only from 0.4831 before filtering to 0.4376 after filtering and remains comparable to MTDIFF (0.4552), while the dynamics error is reduced by 2.9×. The Valid Coverage of the filtered set is 109%, i.e., it still includes simulator-validated transitions in regions that contain no real state. The filter thus removes unreliable rollouts and does not reduce the augmented data to the training data. Consistently, the policies trained on the filtered data perform better in the environment on all three benchmarks (Tab. 3).

(3) Different LLMs as independent judges. We thank the reviewer for this suggestion. A pretrained LLM cannot score numeric states and actions without numeric projection layers and adapters, which have to be trained on the same per-task trajectories. A judge built on a different LLM would therefore change the model but would still depend on the same dataset. For this reason we used the simulator, which is independent of both, to verify the filter. Cross-model judging is a natural extension of the scoring mechanism, and we will note it as future work.

Q1. Removing one score at a time

Yes. We ablate the scoring function in two ways in Tab. 3. First, the w/o Filtering variant retains all six LoRA training objectives and removes the scores, so that the candidates are not ranked. It lowers ML30 from 69.8 to 62.1, Cheetah-dir from 937.8 to 883.1, and Ant-dir from 328.5 to 285.8, which shows that the scoring function as a whole is necessary. Second, the single-objective variants (w/o O1 to w/o O6) remove one objective at a time from the pipeline, including its score. Each of them degrades all three benchmarks: on ML30, the result drops from 69.8 to between 56.4 (w/o O3) and 66.3 (w/o O4). No objective can therefore be removed without a loss.

Q2. Multiple tasks and unseen tasks

LaMA-DT handles multiple tasks with a single model, at three points of the method. First, every task is specified by a natural-language description p_i (Sec. 3.1). Unlike task IDs, the descriptions specify all tasks in one shared semantic space. Second, all parameters are shared across tasks. There is one frozen LLM, and the adapters are objective-specific and not task-specific (App. F.7). Stage I samples tasks uniformly (Alg. 2), so the adapters learn the dynamics and temporal structure that the tasks have in common, while the description that conditions every objective lets the same adapters capture the task-specific data distributions. Third, the method explicitly prevents tasks from being confused. Different tasks often share similar state–action distributions and differ in their goals (Sec. 3.2.2), so O5 and O6 couple trajectories with their descriptions in both directions. In Stage II, candidates are generated per task from the seeds and the description of that task, and candidates that drift toward another task receive low alignment scores and are discarded. The final policy is a single DT conditioned on p_i and trained on the data of all tasks (Sec. 3.3.2).

For unseen tasks, the generalization ability does not come entirely from conditioning on task prompts. LaMA-DT is not applied zero-shot. It is adapted with a few trajectories of the new task by repeating the same three stages (Sec. 3.3.2, App. B.5): the adapters are adapted on the small support set starting from the Stage I adapters, the adapted generator produces candidates for the new task, which are filtered by the six objectives, and the policy is trained on the union with its adapter initialized from the adapted shared adapter. The generalization therefore has three sources: (i) the knowledge stored in the shared adapters, which is learned on the training tasks and only needs a short adaptation; (ii) the description, which specifies the new task in the same semantic space as the training tasks; and (iii) the task-aligned augmentation, which expands the small support set of the new task in the same way it addresses data scarcity for seen tasks.

The experiments are consistent with this analysis. LaMA-DT reaches 52.3 on unseen ML30 tasks against 43.5 for Prompt-DT, a prompt-conditioned baseline (Tab. 11). Replacing the descriptions with a learnable task-ID embedding lowers ML30 from 69.8 to 61.0, and using one shared text for all tasks lowers it to 40.4 (Tab. 10).

Q3. Weighting of the six scores

All six scores are length-normalized log-likelihoods (Sec. 3.3.1) and are therefore on a common scale, so the unweighted mean is a deliberate choice that introduces no hyperparameter. We do not tune any aggregation weights, and the same rule is used for all benchmarks, covering both manipulation and locomotion (Tab. 6). Its effectiveness is verified in two ways. First, the w/o Filtering ablation shows the contribution of this aggregation to policy performance: +7.7 on ML30, +54.7 on Cheetah-dir, and +42.7 on Ant-dir (Tab. 3). Second, its effect on data quality is measured directly: filtering with the unweighted mean reduces Dyn. MSE from 0.0089 to 0.0031 and State KL from 0.42 to 0.18, while Valid Coverage remains above 100% (Tab. 4). We do not claim that the mean is optimal. Learned or task-adaptive weights are a sensible extension, which we will note in the revision, but the simplest tuning-free aggregation already yields high-quality augmented data.

Dear Reviewer, we would be grateful if you could let us know whether our response has addressed your concerns, and we are happy to provide any further clarification before the discussion period ends.

Reviewer_YwfAOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

LaMA-DT is a three-stage framework using one frozen LLM for both trajectory augmentation and multi-task Decision Transformer learning. Six objectives train shared and objective-specific LoRA adapters; candidate rollouts are filtered by aggregated likelihoods before policy training. Meta-World and MuJoCo experiments show especially strong low-data gains, supported by ablations, quality diagnostics, description-robustness tests, unseen-task evaluation, and preliminary compute analysis.

Critical strengths
  1. The consistent low-data improvements directly address the paper’s central motivation. They indicate that the method solves an important practical bottleneck rather than only improving an already data-rich setting.

  2. The evaluation extends beyond policy returns to component ablations, augmentation-ratio and temperature sweeps, quality metrics, description perturbations, unseen tasks, and compute analysis. These analyses substantially support the paper’s soundness.

Critical weaknesses
  1. The central causal claim is insufficiently isolated. There is no matched control using the same LLM and synthetic-data budget without language conditioning, nor another generator integrated into the identical policy pipeline.

  2. The evidence is limited to synthetic, low-dimensional benchmarks with deterministic templated descriptions, while unseen tasks remain within the same benchmark families. The comparison set also lacks a common-protocol evaluation of the closest language-conditioned, retrieval-based, and data-centric DT methods.

  3. The contribution and novelty are insufficient. I do not see a fundamentally new learning principle, formulation, or algorithmic insight that clearly advances beyond applying an LLM as both a synthetic-data generator and policy backbone. Given the limited direct comparisons with prior DT methods, the experiments demonstrate that this engineering combination works, but they do not convincingly establish that its conceptual contribution is substantial enough.

Questions for authors
  1. Could you compare against task-ID-conditioned augmentation and a non-language trajectory generator using the same backbone, augmentation budget, and filtering procedure? Which component drives the low-data improvement?

  2. Why are closely related methods such as LPDT, T2DA, retrieval-augmented prompting, and model-based augmentation not evaluated under a common protocol?

  3. Are O5 explanation targets derived from the same task templates? If so, what demonstrates semantic alignment beyond template reconstruction?

  4. Evaluation on real robotic data with independently written or ambiguous task descriptions would be valuable for establishing broader applicability.

4 comments

We thank the reviewer for the positive assessment of the low-data results and the evaluation.

W1 & Q1. Matched controls and the source of the low-data improvement

The controls asked for here are in the paper (Tab. 10, Tab. 3, and Tab. 12). The rows below use the same LLM backbone, augmentation budget, and policy training, and differ from the full method only in the stated factor.

SettingML30Cheetah-dir
Full LaMA-DT69.8937.8
No description: one shared text for all tasks (Tab. 10)40.4746.9
Learnable task-ID embedding in place of language (Tab. 10)61.0913.4
w/o alignment objectives O5 and O6 (Tab. 3)57.8835.1
w/o filtering, same amount of synthetic data (Tab. 3)62.1883.1

The no-description row is the control without language conditioning. It loses 29.4 points on ML30 and 190.9 on Cheetah-dir. The task-ID row is the task-ID-conditioned counterpart with the same backbone, budget, and filtering, and it is 8.8 points lower on ML30. A generator of a different family, MTDIFF, is compared on data quality in Tab. 4 (3.5× higher dynamics error than our filtered data).

Which component drives the low-data improvement is isolated in Tab. 12 (5% sub-optimal data). The full method reaches 39.1 on ML30 and 521.5 on Cheetah-dir. Removing augmentation while keeping the LLM policy, the descriptions, and the shared-adapter initialization gives 29.8 and 359.6. Removing the shared-adapter initialization while keeping augmentation gives 37.0 and 514.9. The improvement is therefore driven by the augmented data (9.3 and 161.9) and not by the initialization (2.1 and 6.6). Without augmentation, the LLM-based policy is at the level of the DT baselines (HarmoDT: 27.0 and 452.2; Tab. 2), so the gain does not come from the LLM backbone itself.

W2, Q2 & Q4. Benchmarks, descriptions, and compared methods

Meta-World and the MuJoCo multi-task suites are the benchmarks used by the DT baselines, which allows a like-for-like comparison, and held-out tasks within a family are the standard unseen-task protocol under a shared state and action space.

Tab. 10 evaluates descriptions that depart from the template. A shortened paraphrase with different wording stays within 1.9 points on ML30 (67.9), while descriptions that carry less task information degrade gradually (task name only: 55.5; typos and grammar errors: 49.6). Performance follows the task-discriminative content and not the template wording. No stage of LaMA-DT queries a simulator, so the method applies to logged robot data as is, and we agree that this is a valuable next step.

On Q2, the problem setting of this paper is offline multi-task RL, not offline meta-RL. A single policy is trained on the offline data of a fixed set of tasks and evaluated on the same tasks (Sec. 2), and the difficulty we address is the scarcity of per-task data. Tab. 1 and Tab. 2 therefore compare with the state-of-the-art offline MTRL methods under this protocol, where prompt-based DTs are represented by Prompt-DT and HarmoDT and data synthesis by MTDIFF. T2DA and LPDT target a different problem setting, offline meta-RL: they are trained on a set of training tasks and evaluated on unseen test tasks, zero-shot from text for T2DA and few-shot from trajectory prompts for LPDT, and neither performs data augmentation (App. A). For completeness, App. F.6 also studies the generalization of LaMA-DT to unseen tasks. With few-shot adaptation (App. B.5), LaMA-DT outperforms Prompt-DT, the method that LPDT builds on, on all four benchmarks (e.g., 52.3 vs. 43.5 on unseen ML30 tasks; Tab. 11).

Q3. O5 targets

Yes, the O5 target of a trajectory is the description of its task, built from the same template. The template tokens are identical across tasks, so the only part of the target that the model cannot produce without the trajectory is the task-specific content. Lowering the O5 loss therefore requires inferring from the numeric states and actions which task the trajectory belongs to. Two results show that this is not template reconstruction. In the no-description variant, the target is the same text for every trajectory and can be reconstructed perfectly, yet ML30 drops from 69.8 to 40.4. The paraphrased variant changes the wording but keeps the content, and stays at 67.9 (Tab. 10). Removing O5 lowers ML30 by 6.0, and removing O5 and O6 by 12.0 (Tab. 3).

W3. Contribution and novelty

We would like to clarify the novelty of LaMA-DT, which lies in its formulation and its method and goes beyond applying an LLM as a generator and a policy backbone.

First, the paper proposes a new way to address data scarcity in offline MTRL. Existing methods operate on the trajectories that are available: they share transitions across tasks (CDS), generate rollouts from a latent skill space learned from the same data (SRTD), or improve parameter sharing in DTs (HarmoDT, SoCo-DT), and all of them degrade when per-task data are limited (Sec. 1). Our observation is that task descriptions are almost always available even when trajectories are scarce, and we use natural language as the task-specification medium that guides trajectory augmentation. Existing language-based DTs (LaMo, LPDT, T2DA) use the language model for initialization or task embedding, and none of them generates data (App. A). To our knowledge, language-guided trajectory augmentation has not been studied in offline MTRL.

Second, this idea cannot be realized by applying an LLM directly, since LLMs are not pretrained on trajectory data. The generated trajectories must remain faithful to the limited real data while expanding coverage, and must stay aligned with the task description to prevent cross-task drift (Sec. 1). The method is designed around these two requirements:

(i) Four trajectory-generation objectives (O1–O4) teach the LLM the task-specific trajectory distribution from complementary directions: one-step dynamics, interpolation, forward generation, and backward reconstruction (Sec. 3.2.1).

(ii) A bidirectional task–trajectory alignment pair, trajectory-to-description (O5) and description-to-trajectory (O6), forms a semantic feedback loop that prevents goal-agnostic generation (Sec. 3.2.2).

(iii) An objective-aware fine-tuning strategy uses objective-specific adapters to isolate the gradients of the heterogeneous objectives and a shared adapter to aggregate the knowledge that transfers across them (Sec. 3.2.3).

(iv) A scoring-and-filtering mechanism reuses the trained adapters as likelihood evaluators and keeps only the candidates that are consistent under all six objectives, with no environment access and no additional model (Sec. 3.3.1).

(v) The policy learning stage connects augmentation to control. The policy adapter is initialized from the shared adapter, and a provenance token enables source-aware behavior cloning that benefits from augmented data while remaining anchored to real data (Sec. 3.3.2).

The experiments support that these designs, and not the LLM itself, are responsible for the results. With 5% data, the same LLM policy without augmentation is comparable to or below the DT baselines (Tab. 12, Tab. 2). Removing the generation objectives O2–O4 or the alignment objectives O5–O6 reduces ML30 from 69.8 to 50.2 and 57.8, and augmentation without filtering falls below HarmoDT (Tab. 3). Regarding direct comparisons, Tab. 1 and Tab. 2 compare with three prior DT methods (MTDT, Prompt-DT, HarmoDT) under both standard and low-data regimes.

Dear Reviewer, we would be grateful if you could let us know whether our response has addressed your concerns, and we are happy to provide any further clarification before the discussion period ends.

Thank you for the detailed response. The ablations clarified the roles of language conditioning and trajectory augmentation, and I appreciate the explanation of how real-state anchors are used. Having considered the rebuttal and the other reviewers' comments, I still have reservations about the methodological contribution and will keep my original rating.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.