- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper proposes LaMA-DT, a framework that uses a pretrained language model for trajectory augmentation and policy learning in offline multi-task reinforcement learning. The model is adapted through six objectives covering dynamics prediction, trajectory generation, and task–trajectory alignment, using objective-specific and shared LoRA adapters. Generated trajectories are scored across these objectives and filtered before being combined with real data. The same language model is then adapted into a language-conditioned Decision Transformer. Experiments on Meta-World and MuJoCo report improvements over the evaluated baselines, particularly in low-data settings, with additional analyses of augmentation quality, component contributions, and few-shot adaptation to unseen tasks.
Critical strengths
-
The framework offers a coherent way to combine language-conditioned trajectory augmentation with multi-task policy learning. Reusing the language model and shared adapter connects the two stages and provides a reasonable approach to addressing limited per-task data.
-
The low-data results provide some empirical support for the augmentation pipeline. In Table 12, removing augmentation while retaining shared-adapter initialization reduces performance on both ML30 and Cheetah-dir, suggesting that the gains are not solely attributable to policy initialization.
Critical weaknesses
-
The trajectory-generation procedure is insufficiently specified. The main text describes forward generation from an initial state–action pair, whereas Appendix E.3 introduces intermediate teacher-forced states and preservation of the final state. The source of these states and their role in generation are unclear. This distinction materially affects whether the reported quality reflects learned long-horizon generation or reconstruction anchored to real trajectories.
-
The coverage results appear inconsistent with the stated metric. Appendix E.6 defines coverage through grid cells occupied by valid generated transitions, yet Table 4 reports an increase from 99% before filtering to 109% afterward. If filtering selects a subset of the same candidates under fixed validity criteria and grid construction, coverage cannot increase. Clarifying any resampling or budget differences is necessary to substantiate the claim that filtering improves quality while expanding valid coverage.
Questions for authors
-
What information is actually used to generate a trajectory? The main text says generation starts from an initial state–action pair, but Appendix E.3 mentions intermediate teacher-forced states and a preserved final state. Are these taken from the original real trajectory? If so, how much of the generated trajectory is anchored to real data?
-
Why does Valid Coverage increase from 99% to 109% after filtering in Table 4? If filtering only removes candidates, the remaining subset should not cover more valid grid cells under the same metric. Were additional samples generated, or were the two results computed using different candidate sets?
-
Table 12 shows that augmentation helps, but is the proposed pipeline better than other ways of generating additional data? For example, have you compared it with MTDIFF-generated data while keeping the same LLM policy, amount of synthetic data, and policy-training budget? This would help establish what the proposed pipeline adds.
5 comments
Q3. Comparison with MTDIFF-generated data
Tab. 4 compares the two generators directly on data quality, and Tab. 1 compares MTDIFF as a complete method (57.5 vs. 69.8 on ML30, 536.8 vs. 937.8 on Cheetah-dir). What the paper contains is the effect of data fidelity with the LLM policy, the amount of synthetic data, and the training budget all fixed (w/o Filtering, Tab. 3): our unfiltered data (Dyn. MSE 0.0089, State KL 0.42) give 883.1 on Cheetah-dir, and the filtered data (0.0031, 0.18) give 937.8. MTDIFF data are less faithful than our unfiltered data on both metrics (0.0107, 0.51).
What the proposed pipeline adds over generating additional data in general is therefore the multi-objective filter and the description-conditioned generation. Without task-discriminative descriptions, the same pipeline drops to 746.9 on Cheetah-dir and 40.4 on ML30 (Tab. 10).
Dear Reviewer, we would be grateful if you could let us know whether our response has addressed your concerns, and we are happy to provide any further clarification before the discussion period ends.
We thank the reviewer for the careful reading.
W1 & Q1. What information is used to generate a trajectory
Yes. The intermediate teacher-forced states are taken from the same real trajectory that provides the seed pair (s0, a0), and the preserved final state is the final state of that trajectory. As described in App. E.3, long horizons are generated in chunks of 50 steps (Tab. 6). Each chunk starts from the real state at its first step, and all remaining states, actions, and rewards of the chunk are rolled out autoregressively by the O3 adapter at temperature 0.4. The part of an augmented trajectory that is anchored to real data is therefore the seed pair, one state in every 50 steps, and the final state. This reduces wall-clock time by roughly 3× relative to fully autoregressive generation (App. E.3).
The quality reported in the paper reflects the generated content and not the anchors:
(i) Dyn. MSE in Tab. 4 is computed at every generated transition, by resetting the simulator to the generated state and applying the generated action (App. E.6). An anchor affects only the first transition of a chunk.
(ii) The N = 4 candidates of a seed share the same anchors. Selecting among them with the scores changes the policy result from 883.1 to 937.8 on Cheetah-dir (Tab. 3), and the sampling temperature changes it from 896.8 (T = 0) to 937.8 (T = 0.4) (Tab. 7). Neither would happen if the trajectories were determined by their anchors.
(iii) The generated states are not copies of real states. Their average distance to the nearest real state is 0.4376 in normalized space, and a Valid Coverage above 100% means that validated generated transitions fall in cells that contain no real state (Tab. 4).
W2 & Q2. Valid Coverage in Tab. 4
We thank the reviewer for pointing out this inconsistency. The reasoning is correct: filtering only selects among the generated candidates, so it cannot increase coverage under the same validity criterion and grid. We checked the entry against our logs, and the unfiltered value in Tab. 4 was indeed filled in incorrectly. The correct value is 117%, not 99%. To answer Q2, no additional samples were generated, and the filtered set is the top-1 selection from the same candidates. The other entries of Tab. 4 are unaffected.
With the correct value, Valid Coverage is 117% before filtering and 109% after filtering, which is consistent with the definition in App. E.6. The unfiltered candidates reach more cells, but they are less faithful to the dynamics and to the real state distribution (Dyn. MSE 0.0089, State KL 0.42). The filter removes the least reliable candidates, which lowers Dyn. MSE to 0.0031 and State KL to 0.18, and the removed candidates account for 8 points of coverage.
Two points follow. First, the filtered data are not confined to the real data. Their coverage remains above 100% and above MTDIFF (102%), so they still contain validated transitions in cells that no real state occupies. Second, the filter gives up a small part of the coverage for a large improvement in fidelity, and the policy results show that this is beneficial: with the same amount of augmented data, filtering improves ML30 from 62.1 to 69.8, Cheetah-dir from 883.1 to 937.8, and Ant-dir from 285.8 to 328.5 (Tab. 3). These policy results do not depend on the corrected entry. We have corrected Tab. 4 and the corresponding sentence in Sec. 4.