- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes CrossPO, a method for training user-centric interactive large language model (LLM) agents using reinforcement learning (RL). CrossPO aims to improve the performance of user-centric LLM agents by training them to proactively acquire missing information in user requests. To achieve this goal, CrossPO uses cross-trajectory credit estimation that finds similar task trajectories in a training batch and estimates action-level credits by comparing trajectories with and without information-seeking actions. After cross-trajectory credit estimation, CrossPO performs policy optimization by combining the action-level credits and the trajectory-level credits of Group-Relative Policy Optimization (GRPO).
Critical strengths
-
S1. This paper identifies the limitations of Group Relative Policy Optimization (GRPO) (e.g., coarse-grained trajectory-level advantage estimation) and proposes CrossPO that estimates fine-grained action-level advantages by comparing similar task trajectories with and without information seeking actions.
-
S2. This paper provides comprehensive experiment results on three user-centric LLM agent benchmarks such as UserGym, ColBench, and -bench using proprietary LLMs and open-weight LLMs such as Qwen2.5-7B-Instruct and Qwen3-4B. Furthermore, this paper provides additional mechanism analysis, ablation study, and efficiency analysis.
-
S3. This paper shows that CrossPO can considerably improve the performance compared to GRPO. More specifically, on ColBench, Qwen3-4B trained by CrossPO achieves 48.5% of success rates, while Qwen3-4B trained by GRPO provides 32.6% of success rates.
Critical weaknesses
- W1. Although CrossPO can achieve better performance compared to vanilla GRPO, its performance gain seems modest when compared to advanced algorithms such as InfoPO. More specifically, on ColBench, Qwen3-4B trained by CrossPO achieves 48.5% of success rates, while Qwen3-4B trained by InfoPO provides 43.9% of success rates.
Questions for authors
- Q1. What is the main difference between CrossPO and InfoPO?
Response to Reviewer vozR
We thank the reviewer for the careful review and constructive suggestions. Below, we discuss the performance gains over InfoPO and clarify the differences between the two methods in credit assignment and computational cost.
Response to Weakness 1
We thank the reviewer for the valuable feedback. While the improvement over InfoPO may appear modest in absolute terms, CrossPO consistently achieves stronger overall performance across both model backbones. On ColBench, CrossPO improves the success rate from 43.9% to 48.5% (+4.6 percentage points) on Qwen3-4B, corresponding to a 10.5% relative improvement. The gain is larger on Qwen2.5-7B-Instruct, where success increases from 42.6% to 49.1% (+6.5 percentage points). As shown in Table 1, CrossPO also achieves higher aggregate scores across UserGym and τ²-Bench, indicating that its benefits extend beyond a single benchmark.
Importantly, these improvements are achieved without additional rollout generation or a separately trained critic, making CrossPO a practical approach to improving interactive agent training.
Response to Question 1
We thank the reviewer for this insightful question, which helps us clarify the key differences between CrossPO and InfoPO. The key difference lies in how the two methods construct auxiliary credit for information requests. InfoPO estimates turn-level information gain by comparing the likelihood of the realized next action under factual and masked-feedback histories, and combines this signal with task-outcome supervision. This requires additional policy forward evaluations. CrossPO derives request-level credit directly from observed downstream task returns. It aligns equivalent information requests across sampled trajectories and compares subsequent returns between trajectories containing and omitting each request. A request can receive positive credit even when its own trajectory ultimately fails, provided the corresponding return contrast is positive. The estimated credit augments the GRPO advantage while retaining supervision for overall task completion.
In long-horizon agent training, collecting additional rollouts can be costly, as each trajectory requires repeated autoregressive generation and environment or simulator interactions. CrossPO constructs its auxiliary signal through request identification, grouping, and return aggregation, reusing existing rollout data without a learned critic or additional environment interactions. Although InfoPO also reuses collected rollouts, CrossPO further avoids the additional counterfactual policy evaluations required by InfoPO.
On τ2-Bench with Qwen3-4B, CrossPO averages 863.84 s per training step, with embedding and clustering adding 0.47% to the 859.83 s spent on components shared with GRPO (Figure 4). Under matched training settings, InfoPO averages 1,472.63 s per step, corresponding to a 71.27% increase over this baseline and approximately 1.70× CrossPO’s training time for the same number of training steps.
Thank you for your careful review and constructive feedback. We hope these clarifications address your concerns. We appreciate the reviewer's constructive feedback and suggestions for improving our work.