acceptodds
Under review as a conference paper at ICLR 2027

Explicit Trajectory Diversity for RL-Based Post-Training of LLM Agents

Abstract

Large language model (LLM) agents often admit multiple high-quality solutions to the same task, differing in reasoning structure, tool-use pattern, or interaction trajectory. Yet existing notions of diversity in LLM post-training are mostly implicit, arising from general stochasticity and regularization mechanisms rather than explicitly targeting task-relevant behavioral variation. While such implicit diversity can be useful, it does not directly specify which forms of behavioral variation should be encouraged for a given task. In this work, we study explicit trajectory diversity in RL-based post-training for LLMs. Our key idea is to define diversity through user-specified, task-specific trajectory descriptors, which map each sampled trajectory to an interpretable behavioral representation, and then measure diversity as a set-level functional over the resulting descriptor matrix. Building on this formulation, we introduce Trajectory-guided Joint Policy Optimization (TJPO), a single-policy framework that optimizes explicit diversity over sampled trajectory groups, avoiding the need for population-based policy training, and instantiate it within group-based policy optimization through trajectory-level learning signals. This design makes the diversity objective both interpretable and controllable: practitioners can specify whether diversity should correspond to action composition, location visitation, spatial coverage, or other task-relevant behavioral dimensions. Experiments across different environments, including Sokoban and agentic tasks, show that TJPO consistently improves explicit diversity over strong group-based RL baselines while maintaining strong task performance. Further analysis shows that different descriptors induce different forms of behavioral variation, confirming that the learned diversity is aligned with the intended task-specific definition rather than generic randomness. These results suggest that explicit, task-aligned diversity can be effectively optimized in a unified RL framework for LLMs.

Primary area:
foundation or frontier models, including LLMs
Keywords:
large language model agents, reinforcement learning, LLM post-training, trajectory diversity, group-based policy optimization

Reviews

Among:
Shape:
1234this paper, Bottom 9%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.50 over 2 reviews, above 9% of the others.
Reviewer_5C9AOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes TJPO, a framework for explicitly optimizing trajectory diversity in RL-based LLM agent training. It uses task-specific trajectory descriptors and converts group-level diversity into trajectory-level advantages, building on GiGPO. Experiments on ALFWorld and Sokoban show improved diversity while maintaining competitive task performance.

Critical strengths

The core idea is interesting and intuitive. Explicitly optimizing task-relevant trajectory diversity is a promising direction, and the proposed framework is simple and compatible with existing group-based RL algorithms. The robustness and user-constraint experiments also suggest potential practical benefits beyond diversity metrics.

Critical weaknesses

1. Insufficient methodological justification.

The trajectory descriptors are manually designed, introducing considerable task-specific prior knowledge. How can they generalize to tasks where the desired behavioral dimensions are unknown? If such prior knowledge is available, why not directly incorporate it into the prompt?

Section 3.4 should also better explain why GiGPO is necessary rather than standard GRPO. The quality-diversity trade-off controlled by λ deserves more discussion in the method section. Additionally, the original GRPO paper is missing from the references.

2. Weak experimental evaluation.

This is my main concern. Tables 1–4 primarily evaluate diversity using metrics closely related to the training objectives, making the improvements somewhat expected. More independent evaluations are needed.

The baselines are also limited. Beyond GiGPO and simple decoding penalties (PP/RP), the authors should compare against stronger sampling-based methods, such as higher temperatures and verbalized sampling, as well as prompting-based approaches.

Furthermore, some variants sacrifice task success for diversity, and the experiments lack multiple training seeds. Although Table 5 provides more meaningful evidence, it covers only a small subset of ALFWorld tasks.

Overall, I find the method promising, but the experimental evidence is not sufficiently convincing to support the paper's claims.

Questions for authors

See Weaknesses

Reviewer_rXV9OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces TJPO, which adds a behavioral diversity signal to GiGPO training for LLM agents. Users define trajectory features such as action frequencies, visited locations or movement patterns. For each group of eight rollouts, TJPO measures diversity using pairwise distances or a kernel determinant, assigns each trajectory a credit, and adds the standardized credit to its original advantage. Experiments use Qwen2.5-7B on ALFWorld and Qwen3-VL-4B on Sokoban, with one training run per configuration. TJPO increases the targeted diversity, though task success sometimes drops. On ALFWorld, location diversity rises from 0.086 to 0.132 while success falls from 91.6% to 90.5%. On a separate suite of 15 tasks with verified alternative solutions, compliant success under location constraints improves from 77.1% to 85.4%, and success under location disruptions rises from 77.9% to 85.4%.

Critical strengths
  1. The practical evaluation goes beyond measuring diversity. The authors verify that each selected ALFWorld task has two independent solution branches, then test location constraints and location unavailability without additional training. The gains of 8.3 and 7.5 percentage points come from a small suite, but this is a useful way to test whether behavioral diversity helps in practice.
  2. The method is straightforward to integrate into GiGPO and requires no additional policy or model. Appendix A clearly derives the pairwise credit as an affine transformation of the exact change caused by removing a trajectory. The case studies also provide complete rollouts for both methods, making the behavioral differences easier to assess.
Critical weaknesses
  1. The evidence does not establish a reliable balance between diversity and task success. Each configuration has one training run and no uncertainty estimates. ALFWorld success falls from 91.6% to 86.8% for the action variant and to 87.8% for the determinantal location variant. Most diversity metrics closely follow the training objective, so their improvement alone does not establish more useful strategies. The practical evaluation covers only 15 tasks and uses λ = 0.5 rather than the main setting of 0.3. Repeated seeds, independent behavioral measures and uncertainty estimates are needed. The repeated evaluation of checkpoints on the same 128 prompts also requires a clearer account of checkpoint selection and final evaluation.

  2. The conceptual contribution is narrow, and the comparisons do not isolate its value. TJPO adds a dispersion bonus over predefined behavioral histograms to an existing RL objective. The experiments omit training with entropy regularization and a generic descriptor under the same credit rule, leaving unclear whether the benefit comes from explicit task alignment or simply an additional diversity incentive. The determinantal implementation also needs verification. With normalized histograms and kernel bandwidth 10, nearly identical kernel entries can push the determinant below the fallback threshold of 10^(-10), switching the credit to pairwise kernel dissimilarity. Without reporting fallback frequency and bandwidth sensitivity, the reported gains cannot be confidently attributed to determinantal optimization.

Questions for authors

Please evaluate diversity beyond the descriptors used for training. Across all 128 prompts, how many distinct successful routes or action sequences does each method produce? An additional embedding-based measure would help assess whether the gains extend beyond the optimized metrics.

How does TJPO compare with training using entropy regularization and tuned rollout temperature, KL penalty and clipping settings? Please also test a generic descriptor with the same credit rule. Does the task-specific descriptor retain an advantage at comparable task success and training cost?

How often does determinantal training trigger the fallback in Appendix A? Please report the distribution of regularized kernel determinants and test bandwidths chosen from the observed squared pairwise distances. Do the gains persist when the determinant credit is actually used?

Why was λ = 0.5 chosen for Table 5 rather than the main setting of 0.3? Please report the same evaluation at 0.3, with uncertainty estimates clustered by task or scene to preserve dependence between intervention directions. Does the standard-success difference of 97.5% versus 99.2% persist across training seeds?

Failed trajectories also receive diversity credit. Have you tested assigning credit only to successful rollouts? Does this reduce the success drop for the action-type variant while retaining useful diversity?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.