- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper proposes TJPO, a framework for explicitly optimizing trajectory diversity in RL-based LLM agent training. It uses task-specific trajectory descriptors and converts group-level diversity into trajectory-level advantages, building on GiGPO. Experiments on ALFWorld and Sokoban show improved diversity while maintaining competitive task performance.
Critical strengths
The core idea is interesting and intuitive. Explicitly optimizing task-relevant trajectory diversity is a promising direction, and the proposed framework is simple and compatible with existing group-based RL algorithms. The robustness and user-constraint experiments also suggest potential practical benefits beyond diversity metrics.
Critical weaknesses
1. Insufficient methodological justification.
The trajectory descriptors are manually designed, introducing considerable task-specific prior knowledge. How can they generalize to tasks where the desired behavioral dimensions are unknown? If such prior knowledge is available, why not directly incorporate it into the prompt?
Section 3.4 should also better explain why GiGPO is necessary rather than standard GRPO. The quality-diversity trade-off controlled by λ deserves more discussion in the method section. Additionally, the original GRPO paper is missing from the references.
2. Weak experimental evaluation.
This is my main concern. Tables 1–4 primarily evaluate diversity using metrics closely related to the training objectives, making the improvements somewhat expected. More independent evaluations are needed.
The baselines are also limited. Beyond GiGPO and simple decoding penalties (PP/RP), the authors should compare against stronger sampling-based methods, such as higher temperatures and verbalized sampling, as well as prompting-based approaches.
Furthermore, some variants sacrifice task success for diversity, and the experiments lack multiple training seeds. Although Table 5 provides more meaningful evidence, it covers only a small subset of ALFWorld tasks.
Overall, I find the method promising, but the experimental evidence is not sufficiently convincing to support the paper's claims.
Questions for authors
See Weaknesses