acceptodds
Withdrawn from ICLR 2027

CrossPO: Cross-Trajectory Policy Optimization for User-Centric Agents

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

User-centric LLM agents often need to acquire missing information through interaction before they can complete a task. Learning this behavior is challenging because task outcomes reflect both information acquisition and subsequent execution, leaving trajectory-level rewards too coarse to assess individual information requests. We therefore introduce CrossPO, which learns effective information-seeking behavior by aligning equivalent information requests across trajectories. Within each task's rollout batch, it groups approximately equivalent requests and compares average subsequent returns between trajectories containing and omitting each request class. It shrinks the resulting credit toward zero when either group has limited sample support and uses it to augment GRPO advantages while retaining trajectory-level supervision. This allows equivalent requests to share supervision across dialogue histories, so that a request can receive positive local credit even in an unsuccessful trajectory. Experiments on UserGym, ColBench, and -Bench demonstrate strong overall performance across intent clarification, collaborative coding, and tool use. On ColBench, CrossPO improves pass and success rates over the strongest baseline by up to 5.5 and 6.5 percentage points, respectively. Ablations support the effectiveness of credit assignment and calibration, while behavioral analyses show substantially fewer duplicate requests. Further efficiency analysis shows that CrossPO adds less than 0.5% to per-step training time, making it practical for multi-turn agent training.

Primary area:
foundation or frontier models, including LLMs
Keywords:
User-Centric LLM Agents, Reinforcement Learning, Credit Assignment

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_vozROpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes CrossPO, a method for training user-centric interactive large language model (LLM) agents using reinforcement learning (RL). CrossPO aims to improve the performance of user-centric LLM agents by training them to proactively acquire missing information in user requests. To achieve this goal, CrossPO uses cross-trajectory credit estimation that finds similar task trajectories in a training batch and estimates action-level credits by comparing trajectories with and without information-seeking actions. After cross-trajectory credit estimation, CrossPO performs policy optimization by combining the action-level credits and the trajectory-level credits of Group-Relative Policy Optimization (GRPO).

Critical strengths
  • S1. This paper identifies the limitations of Group Relative Policy Optimization (GRPO) (e.g., coarse-grained trajectory-level advantage estimation) and proposes CrossPO that estimates fine-grained action-level advantages by comparing similar task trajectories with and without information seeking actions.

  • S2. This paper provides comprehensive experiment results on three user-centric LLM agent benchmarks such as UserGym, ColBench, and τ2\tau^2-bench using proprietary LLMs and open-weight LLMs such as Qwen2.5-7B-Instruct and Qwen3-4B. Furthermore, this paper provides additional mechanism analysis, ablation study, and efficiency analysis.

  • S3. This paper shows that CrossPO can considerably improve the performance compared to GRPO. More specifically, on ColBench, Qwen3-4B trained by CrossPO achieves 48.5% of success rates, while Qwen3-4B trained by GRPO provides 32.6% of success rates.

Critical weaknesses
  • W1. Although CrossPO can achieve better performance compared to vanilla GRPO, its performance gain seems modest when compared to advanced algorithms such as InfoPO. More specifically, on ColBench, Qwen3-4B trained by CrossPO achieves 48.5% of success rates, while Qwen3-4B trained by InfoPO provides 43.9% of success rates.
Questions for authors
  • Q1. What is the main difference between CrossPO and InfoPO?
1 comment
AuthorsOpenReview ↗

Response to Reviewer vozR

We thank the reviewer for the careful review and constructive suggestions. Below, we discuss the performance gains over InfoPO and clarify the differences between the two methods in credit assignment and computational cost.

Response to Weakness 1

Although CrossPO can achieve better performance compared to vanilla GRPO, its performance gain seems modest when compared to advanced algorithms such as InfoPO. More specifically, on ColBench, Qwen3-4B trained by CrossPO achieves 48.5% of success rates, while Qwen3-4B trained by InfoPO provides 43.9% of success rates.

We thank the reviewer for the valuable feedback. While the improvement over InfoPO may appear modest in absolute terms, CrossPO consistently achieves stronger overall performance across both model backbones. On ColBench, CrossPO improves the success rate from 43.9% to 48.5% (+4.6 percentage points) on Qwen3-4B, corresponding to a 10.5% relative improvement. The gain is larger on Qwen2.5-7B-Instruct, where success increases from 42.6% to 49.1% (+6.5 percentage points). As shown in Table 1, CrossPO also achieves higher aggregate scores across UserGym and τ²-Bench, indicating that its benefits extend beyond a single benchmark.

Importantly, these improvements are achieved without additional rollout generation or a separately trained critic, making CrossPO a practical approach to improving interactive agent training.

Response to Question 1

What is the main difference between CrossPO and InfoPO?

We thank the reviewer for this insightful question, which helps us clarify the key differences between CrossPO and InfoPO. The key difference lies in how the two methods construct auxiliary credit for information requests. InfoPO estimates turn-level information gain by comparing the likelihood of the realized next action under factual and masked-feedback histories, and combines this signal with task-outcome supervision. This requires additional policy forward evaluations. CrossPO derives request-level credit directly from observed downstream task returns. It aligns equivalent information requests across sampled trajectories and compares subsequent returns between trajectories containing and omitting each request. A request can receive positive credit even when its own trajectory ultimately fails, provided the corresponding return contrast is positive. The estimated credit augments the GRPO advantage while retaining supervision for overall task completion.

In long-horizon agent training, collecting additional rollouts can be costly, as each trajectory requires repeated autoregressive generation and environment or simulator interactions. CrossPO constructs its auxiliary signal through request identification, grouping, and return aggregation, reusing existing rollout data without a learned critic or additional environment interactions. Although InfoPO also reuses collected rollouts, CrossPO further avoids the additional counterfactual policy evaluations required by InfoPO.

On τ2\tau^2-Bench with Qwen3-4B, CrossPO averages 863.84 s per training step, with embedding and clustering adding 0.47% to the 859.83 s spent on components shared with GRPO (Figure 4). Under matched training settings, InfoPO averages 1,472.63 s per step, corresponding to a 71.27% increase over this baseline and approximately 1.70× CrossPO’s training time for the same number of training steps.

Thank you for your careful review and constructive feedback. We hope these clarifications address your concerns. We appreciate the reviewer's constructive feedback and suggestions for improving our work.

Reviewer_owxtOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

CrossPO studies learning to request information in user-centric agents. Across trajectories of the same task, it groups approximately equivalent requests into anchors using native IDs or frozen-embedding clustering. It compares downstream returns of trajectories containing versus not containing a request, then applies support-dependent shrinkage, normalization, and clipping to obtain local credit. The assistant turn containing the first request receives this adjustment in addition to its GRPO advantage; repeated requests incur only an interaction cost. The authors evaluate Qwen2.5-7B-Instruct and Qwen3-4B on UserGym, ColBench, and tau-squared-Bench, with held-out UserGym environments, component ablations, a shuffled-credit control, and three-seed training curves. Table 1 also includes closed-source reference models, and the supplement provides the training implementation.

Critical strengths
  1. The method reuses sampled trajectories and task outcomes to provide local supervision, without additional environment rollouts or a separately trained critic. First and repeated requests are treated differently, and the algorithm and main hyperparameters are specified concretely.

  2. Two backbones, three benchmarks, held-out UserGym environments, component ablations, a shuffled-credit control, and three-seed training curves provide empirical support. The supplementary code also includes public/private context separation, leakage checks, data preflight checks, and an independent ColBench grading execution path. The work therefore provides both protective mechanisms and reproducibility materials.

Critical weaknesses
  1. Incremental value for stronger, newer agents and the validity of request-level attribution remain insufficiently validated. The trainable backbones are limited to Qwen2.5-7B-Instruct and Qwen3-4B, from the 2024/2025 model series and the same Qwen family. Evaluation on newer, similarly sized backbones and another independent family available before the experimental cutoff would be informative; this is not a request to include every post-submission release. Table 1 does include closed-source references such as Gemini-3-Flash and GPT-4.1, but their scores do not validate the training method's incremental benefit on newer backbones. Cross-trajectory return differences between asking and not asking may also conflate prior histories and subsequent execution ability; shrinkage does not eliminate this comparison bias. The shuffled-credit degradation supports the usefulness of credit correspondence, but does not by itself validate causal attribution. Cost-only controls, threshold/manual equivalence checks, and interventions or independent continuations from matched histories are still needed. Appendix B acknowledges matching bias and the limitations of the fixed-class assumption, so the concern is insufficient validation rather than complete neglect. Uncertainty in main test scores, relative advantages across simulators, and the theoretical interpretation at G=8 also need clarification.

  2. Evidence for resistance to reward gaming and for data/experimental reproducibility remains incomplete. The implementation includes first-request credit, repeated-request costs, separation of public observations from private tasks, optional canary checks, failure markers, and separate-process/Ray grading for ColBench. However, I did not find systematic adversarial results for prompt injection, inducing simulated users to reveal reference answers or evaluation information, or splitting paraphrased requests across anchors to obtain extra credit. Fewer repeated requests and shuffled-credit controls do not replace such tests. Static inspection also indicates conditional protection: the adapter converts observations to strings before checking, while private-key-name scanning does not parse strings; canaries are empty by default and depend on external task provision. This does not establish leakage in actual tasks or demonstrate an exploitable attack. A separate ColBench process is not automatically a filesystem/network security sandbox; the upstream grader's isolation configuration needs specification.

For reproducibility, the ZIP contains 59 files, configurations, fixed-seed row-level splits, and metadata preflight checks on at most the first 512 rows. It does not include training/validation/test data with exact ID/hash manifests, a reproducible preprocessing chain, checkpoints, or per-run results, and I did not find a complete entry point for reconstructing the main table and all RL baselines. External environments/verl are not pinned to commits, dependencies mostly specify lower bounds, and the simulator is identified only by the gpt-4o-mini alias. Training scripts alone therefore do not suffice to reproduce the results. Public upstream data and code are valuable, and missing materials do not establish fabrication or inevitable contamination, but the current release is insufficient to independently verify data boundaries and main-table scores.

Questions for authors
  1. Please provide the experimental/model cutoff, exact checkpoint revisions, and inference modes, and verify CrossPO's gains over GRPO/InfoPO under matched training budgets on a newer similarly sized backbone and an independent model family available at that time. Direct scores of closed-source reference models do not replace this test.

  2. Can you add cost-only controls, clustering-threshold/manual equivalence checks, and asking-versus-not-asking interventions or independent continuations from matched histories, to separate interaction-cost, clustering, and cross-trajectory-credit contributions?

  3. At G=8, how does the appendix theorem yield a nontrivial high-confidence sign interpretation? Please report effective sample sizes and matching bias, distinguishing native anchors from semantic clustering within the same batch. Can you provide independent checks using leave-one-trajectory-out estimation or cross-fitting?

  4. Please report adversarial tests for reward gaming and information leakage: simulator prompt injection, requests for reference answers/evaluation information, paraphrases or compound questions that evade repeat detection, and forged tool returns. How are canaries generated, and do they cover all private fields? How are leakage checks on already-rendered strings and the upstream ColBench sandbox configured? Please distinguish implemented protections from experimentally demonstrated defenses.

  5. Please provide per-benchmark data versions, sample counts, exact train/dev/test IDs or hashes, deduplication/task-level separation checks, the pipeline from upstream data to training parquet, all split/training seeds, pinned upstream commits/dependencies, and complete commands, checkpoints/logs, or per-task results for reconstructing the main table and all RL baselines. For data that cannot be redistributed, verifiable sources and acquisition/reconstruction procedures would still help.

  6. Please specify exact simulator/evaluator API versions, prompts, sampling, and retry settings. Evaluate the main baselines as well when changing the simulator, report consistent treatment and proportions of failed trajectories, and provide cross-seed intervals or paired per-task results for the main test scores.

9 comments
AuthorsOpenReview ↗

Response to Reviewer owxt

We thank the reviewer for the careful review and constructive suggestions. Below, we address each question together with the relevant concerns raised in the corresponding weaknesses.

Response to Question 1 and Weakness 1

Question 1: Please provide the experimental/model cutoff, exact checkpoint revisions, and inference modes, and verify CrossPO's gains over GRPO/InfoPO under matched training budgets on a newer similarly sized backbone and an independent model family available at that time. Direct scores of closed-source reference models do not replace this test.

Weakness 1 (Backbone Generalization): The trainable backbones are limited to Qwen2.5-7B-Instruct and Qwen3-4B, from the 2024/2025 model series and the same Qwen family. Evaluation on newer, similarly sized backbones and another independent family available before the experimental cutoff would be informative; this is not a request to include every post-submission release. Table 1 does include closed-source references such as Gemini-3-Flash and GPT-4.1, but their scores do not validate the training method's incremental benefit on newer backbones.

We thank the reviewer for this constructive suggestion. We have added matched-budget comparisons of GRPO, InfoPO, and CrossPO using google/gemma-4-12B-it, extending our evaluation to a newer backbone from the independent Gemma family. Below, we clarify the model configurations and describe the additional comparisons.

(i) Experimental cutoff and model configurations. Our model-availability cutoff was September 1, 2026 (AoE), before which Gemma-4-12B-it had already been released.

Table 4 already specifies that actor thinking is disabled in the original experiments. We additionally provide the exact initialization checkpoint revisions and the inference mode used for Gemma below.

BackboneExact checkpoint revisionInference mode
Qwen/Qwen2.5-7B-Instructa09a35458c702b33eeacc393d103063234e8bc28Standard (non-thinking)
Qwen/Qwen3-4B1cfa9a7208912126459214e8b04321603b3df60cThinking disabled
google/gemma-4-12B-it43c9d014b231474fa7212dbb48a762566c2b255dThinking disabled

Within each backbone, GRPO, InfoPO, and CrossPO use the same initialization revision and inference mode.

(ii) Matched training and evaluation budgets. Section 4.1 and Appendix C (Table 4) already report the shared optimization settings, benchmark-specific training schedules, batch sizes, rollout configurations, and sequence-length limits. Appendix C also specifies that CrossPO and the baselines use the same rollout budgets and interaction limits within each benchmark.

(iii) Results on Gemma-4-12B-it. The table below reports performance on google/gemma-4-12B-it. For Gemma-4-12B-it, we compare CrossPO with GRPO and InfoPO under matched training budgets.

MethodUserGym Travel ↑UserGym Function ↑UserGym Persuasion ↑UserGym Tau ↑UserGym Turtle ↑UserGym Search ↑UserGym Intention ↑UserGym Telepathy ↑ColBench Pass ↑ColBench Succ. ↑τ²-Bench Telecom ↑τ²-Bench Retail ↑τ²-Bench Airline ↑
GRPO0.6210.3210.7140.0670.2830.9041.7900.7070.6140.5220.4060.5690.463
InfoPO0.6340.3970.7700.0970.3130.8801.8430.7560.6720.5540.4940.5940.475
CrossPO0.6410.3850.7860.1330.3250.9361.9200.7800.6870.5910.5310.5750.513

On Gemma-4-12B-it, CrossPO outperforms GRPO on all 13 metrics and InfoPO on 11, showing broadly consistent gains across benchmarks. These controlled training comparisons directly evaluate CrossPO’s incremental benefit on a newer backbone outside the Qwen family. We have added the results to Appendix D.1(Table 5).

AuthorsOpenReview ↗

Response to Reviewer owxt (2)

Response to Question 2 and Weakness 1

Question 2: Can you add cost-only controls, clustering-threshold/manual equivalence checks, and asking-versus-not-asking interventions or independent continuations from matched histories, to separate interaction-cost, clustering, and cross-trajectory-credit contributions?

Weakness 1 (Credit Attribution): Cross-trajectory return differences between asking and not asking may also conflate prior histories and subsequent execution ability; shrinkage does not eliminate this comparison bias. The shuffled-credit degradation supports the usefulness of credit correspondence, but does not by itself validate causal attribution. Cost-only controls, threshold/manual equivalence checks, and interventions or independent continuations from matched histories are still needed.

We thank the reviewer for suggesting these complementary checks. Below, we separately examine interaction cost, semantic grouping, and cross-trajectory credit.

(i) Interaction cost. We add the cost-only and credit-only controls requested by the reviewer. The table below compares GRPO, cost-only (Aq=0A_q=0, request cost retained), credit-only (c0=0c_0=0, request credit retained), and full CrossPO on Qwen3-4B. Cost-only reduces duplicate requests but lowers task performance; credit-only improves task performance but does not consistently reduce duplicate requests; full CrossPO achieves both. This supports the complementary roles of request credit and the interaction cost, rather than attributing the duplicate reduction solely to credit. The full results have been added to Appendix D.2 (Table 7).

ConfigurationUserGym Avg. ↑UserGym Dup.(per task) ↓ColBench Succ. ↑ColBench Dup.(per task) ↓τ²-Bench Avg. ↑τ²-Bench Dup.(per task) ↓
GRPO0.4890.230.3260.270.1290.30
Cost-only0.4420.090.3180.170.1060.12
Credit-only0.5940.280.4250.230.1980.32
Full CrossPO0.6150.070.4850.040.2460.05

(ii) Clustering. We report the proportions of auxiliary credit assignments from native identifiers and semantic clustering. To assess the reliability of semantic grouping, we manually audit 200 matched request pairs per benchmark together with their preceding contexts. The audit judges whether two matched requests seek the same information about the same relevant entity or requirement under their respective contexts. Two annotators independently assess each pair, and a pair is counted as equivalent only when both agree. The native/semantic breakdown and manual audit results have been added to Appendix D.2 (Table 8).

BenchmarkNative assignments (%)Semantic assignments (%)Audited semantic pairsEquivalent pairs (%)
UserGym16.183.920093.5
ColBench010020094
τ²-Bench45.454.620090.5

We report a sensitivity analysis of the semantic clustering threshold using Qwen3-4B on all three benchmarks. The threshold is set to 0.65, 0.70, or 0.75 during training, while all other training settings and the evaluation protocol remain unchanged. The threshold sensitivity results have been added to Appendix D.2 (Table 9).

Training thresholdUserGym ↑ColBench Pass ↑ColBench Succ ↑τ²-Bench ↑
0.650.6040.5620.4530.221
0.70 (original setting)0.6150.5760.4850.246
0.750.6120.5830.4870.233
AuthorsOpenReview ↗

Response to Reviewer owxt (3)

(iii) Cross-trajectory credit. We appreciate the reviewer’s thoughtful comments on the interpretation of request-level credit. In CrossPO, the return contrast between trajectories with and without equivalent requests provides auxiliary supervision that distinguishes requests by their estimated downstream utility and complements the trajectory-level GRPO advantage. This signal reuses sampled rollouts and task outcomes, requiring no additional environment rollouts or separately trained critic. We interpret AqA_q as a lightweight auxiliary signal for improving information seeking, without assuming that it provides an unbiased causal estimate of request utility in every context.

To further evaluate this claim, we test whether estimated credit predicts the return difference between immediately requesting qq and suppressing equivalent requests throughout the continuation, starting from the same restored state. For each anchor, we restore the full pre-request state associated with history hh, including the environment, simulator, evaluation state, and the policy’s observation history. From the restored state hh, we construct two independent continuation branches: (i) Ask-q, where a fixed request for qq is forcibly inserted at hh and the policy then continues freely; (ii) Suppress-q, where all requests semantically equivalent to qq are masked throughout the entire continuation until termination. We sample 4 independent continuations per branch and estimate the intervention effect as the difference in mean suffix return between the two branches. We then correlate this effect with AqA_q, estimated from 8 naturally sampled rollouts that are disjoint from the intervention continuations.

The table below reports Spearman correlations and sign agreement between estimated credit AqA_q and independently estimated intervention return differences. The 95% confidence intervals for Spearman correlations are obtained by resampling tasks.

Benchmark# matched contextsSpearman correlation[95% CI]Sign agreement
UserGym10920.7934(0.7752,0.8119)86.7%
ColBench37190.8021(0.7928,0.8084)89.3%
τ²-Bench5300.6914(0.6410,0.7392)77.2%

Overall, these results support the complementary roles of request credit and interaction cost and show that performance gains persist across the tested clustering thresholds. The contextual audit and matched-history interventions further support the reliability of semantic grouping and the predictive value of the estimated credit.

Reviewer_bWYiOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Summary

CrossPO augments GRPO with request-level credit estimated from sibling trajectories of the same task. It groups approximately equivalent information requests, compares subsequent returns between trajectories containing and omitting each request, and applies shrinkage, normalization, and clipping to the contrast. Credit is assigned to the first occurrence, while every request incurs a fixed cost. Experiments cover UserGym, ColBench, and tau2-Bench with two Qwen backbones, including held-out UserGym environments and component ablations.

Strengths

S1. The method reuses existing rollouts and does not require a learned critic or additional counterfactual interactions. This is a useful design choice for interactive environments where rollout generation dominates training time. The measured embedding and clustering overhead is small in the reported tau2-Bench configuration.

S2. The empirical gains extend across tasks and backbones. On ColBench, success increases from 0.426 to 0.491 for Qwen2.5-7B and from 0.439 to 0.485 for Qwen3-4B relative to InfoPO. The held-out UserGym results also support transfer beyond the training environments.

S3. The support-dependent shrinkage addresses a concrete weakness of comparisons built from only eight trajectories. The ablations and shuffled-credit control provide evidence that the calibration and assignment of the auxiliary signal matter. Appendix B also states the history-mismatch and fixed-class assumptions needed for its advantage interpretation.

Weaknesses

W1. The mechanism analysis does not independently validate whether the credit identifies useful requests. Figure 3(a,b) bins training samples using credit computed from their subsequent returns and then measures the success of those samples. With terminal-only rewards, the contrast is itself a difference in success rates, so this association partly follows from how the score is constructed. Appendix B.2 correctly allows a mismatch term between histories, but does not measure it. A small diagnostic that estimates credit on one set of rollouts and checks it on separate rollouts, preferably comparing similar pre-request contexts, would provide stronger evidence for the claimed request utility. The positive task results remain useful, but do not by themselves establish that interpretation.

W2. The reduction in duplicate requests is not isolated from the explicit request penalty. Equation 10 charges every request a fixed cost and gives repeated occurrences no additional anchor benefit. Table 3 removes standardization, shrinkage, and clipping, but does not include a cost-only version of the same training objective. The shuffled-credit control still injects potentially misleading advantages, so its deterioration does not settle this question. Setting A_q = 0 while retaining c_0 and the same loss normalization, then separately testing c_0 = 0 with credit retained, would clarify how much of the reported reduction comes from outcome-based credit rather than directly penalizing interaction.

W3. The reliability of the semantic anchors needs more evidence. Section 3.2 groups requests across different histories using a frozen encoder and a 0.70 threshold. Similar wording can refer to different entities or missing requirements. Complete linkage controls embedding similarity, but does not establish that the grouped requests seek equivalent information. A small annotated audit of matched requests, together with a sensitivity check around the chosen threshold, would help assess the central alignment assumption. It would also be useful to distinguish results using native request identifiers from those relying on semantic clustering.

Critical strengths

S1. The method reuses existing rollouts and does not require a learned critic or additional counterfactual interactions. This is a useful design choice for interactive environments where rollout generation dominates training time. The measured embedding and clustering overhead is small in the reported tau2-Bench configuration.

Critical weaknesses

W1. The mechanism analysis does not independently validate whether the credit identifies useful requests. Figure 3(a,b) bins training samples using credit computed from their subsequent returns and then measures the success of those samples. With terminal-only rewards, the contrast is itself a difference in success rates, so this association partly follows from how the score is constructed. Appendix B.2 correctly allows a mismatch term between histories, but does not measure it. A small diagnostic that estimates credit on one set of rollouts and checks it on separate rollouts, preferably comparing similar pre-request contexts, would provide stronger evidence for the claimed request utility. The positive task results remain useful, but do not by themselves establish that interpretation.

W2. The reduction in duplicate requests is not isolated from the explicit request penalty. Equation 10 charges every request a fixed cost and gives repeated occurrences no additional anchor benefit. Table 3 removes standardization, shrinkage, and clipping, but does not include a cost-only version of the same training objective. The shuffled-credit control still injects potentially misleading advantages, so its deterioration does not settle this question. Setting A_q = 0 while retaining c_0 and the same loss normalization, then separately testing c_0 = 0 with credit retained, would clarify how much of the reported reduction comes from outcome-based credit rather than directly penalizing interaction.

Questions for authors

Q1. Does anchor credit predict request utility on rollouts excluded from its estimation, after accounting for the history mismatch discussed in Appendix B.2?

Q2. How do the cost-only version of the same objective and CrossPO without the request cost compare on task success and duplicate requests?

Q3. What fraction of the auxiliary supervision uses native identifiers versus semantic clusters, and how accurately do the latter group equivalent requests? Please also clarify whether Table 1 reports means across training seeds; Figure 2 specifies three seeds for training curves, but the table's aggregation and uncertainty are not stated.

3 comments
AuthorsOpenReview ↗

Response to Reviewer bWYi

We thank the reviewer for the careful review and constructive suggestions. We hope the responses and clarifications below help address your concerns.

Response to question 1 and weakness 1

Question 1: Does anchor credit predict request utility on rollouts excluded from its estimation, after accounting for the history mismatch discussed in Appendix B.2?

Weakness 1: The mechanism analysis does not independently validate whether the credit identifies useful requests. Figure 3(a,b) bins training samples using credit computed from their subsequent returns and then measures the success of those samples. With terminal-only rewards, the contrast is itself a difference in success rates, so this association partly follows from how the score is constructed. Appendix B.2 correctly allows a mismatch term between histories, but does not measure it. A small diagnostic that estimates credit on one set of rollouts and checks it on separate rollouts, preferably comparing similar pre-request contexts, would provide stronger evidence for the claimed request utility. The positive task results remain useful, but do not by themselves establish that interpretation.

We thank the reviewer for suggesting independent validation of request credit with context matching. Question 1 and Weakness 1 both concern the validity of request-level credit and the influence of history mismatch, so we address them together. We appreciate the reviewer’s careful distinction between the within-sample association in Figure 3 and independent evidence of request utility.

Regarding history mismatch, we agree that cross-trajectory contrasts compare trajectories with potentially different pre-request histories. Appendix B.2 explicitly accounts for this through the mismatch bound bqb_q and does not assume bq=0b_q=0. We therefore do not claim that AqA_q is an unbiased causal estimate of request utility at every context. Our claim is that aligned requests provide calibrated auxiliary credit that improves information seeking when combined with the trajectory-level GRPO advantage. To keep this auxiliary signal conservative, we retain the GRPO advantage, shrink low-support credits, normalize and clip AqA_q, assign credit only to the first occurrence, and penalize repeated requests.

To further evaluate the predictive value of this auxiliary credit, we have added an independent-rollout diagnostic with context matching. For each task, we estimate request credit from eight rollouts and use another 32 independent rollouts for validation. To examine history mismatch, we cluster these validation rollouts based on the similarity of their pre-request hidden states, recorded from Qwen3-4B during rollout generation. If a trajectory first requests qq at step tt, we compare its hidden state immediately before that request with the hidden states immediately before the same step in trajectories that never request qq and are still active at step tt.

Within each cluster containing both requesting and non-requesting trajectories, we compute the difference in mean subsequent return. We aggregate these cluster-specific differences using a weighted average, with weights wc=nc,qnc,¬qnc,q+nc,¬qw_c = \frac{n_{c,q}n_{c,\neg q}}{n_{c,q}+n_{c,\neg q}}, where nc,qn_{c,q} and nc,¬qn_{c,\neg q} denote the numbers of requesting and non-requesting trajectories in cluster cc, respectively.We then compute the Spearman correlation between the credit estimated from the original eight rollouts and the aggregated return differences on the 32 validation rollouts. This diagnostic tests whether estimated credit retains predictive value when comparisons are restricted to similar pre-request contexts. The table below reports Spearman correlations with 95% confidence intervals obtained by resampling tasks.

BenchmarkEligible anchorsSpearman correlation (95% CI)
UserGym2,8350.7876 (0.7693, 0.8149)
ColBench8,4960.7692 (0.7589, 0.7802)
τ²-Bench6910.6503 (0.5973, 0.6965)

The results show moderate-to-strong positive rank correlations between estimated credit and aggregated return differences from context-matched validation across all three benchmarks, with all 95% confidence intervals well above zero. These findings indicate that credit estimated from eight rollouts retains predictive value on independent validation trajectories when comparisons are restricted to similar pre-request contexts, providing evidence beyond the within-sample association in Figure 3. We have added the new context-matched validation experiment to Appendix D.2(Table 6).

AuthorsOpenReview ↗

Response to Reviewer bWYi (2)

Response to question 2 and weakness 2

Question 2: How do the cost-only version of the same objective and CrossPO without the request cost compare on task success and duplicate requests?

Weakness 2: The reduction in duplicate requests is not isolated from the explicit request penalty. Equation 10 charges every request a fixed cost and gives repeated occurrences no additional anchor benefit. Table 3 removes standardization, shrinkage, and clipping, but does not include a cost-only version of the same training objective. The shuffled-credit control still injects potentially misleading advantages, so its deterioration does not settle this question. Setting Aq=0A_q = 0 while retaining c0c_0 and the same loss normalization, then separately testing c0=0c_0 = 0 with credit retained, would clarify how much of the reported reduction comes from outcome-based credit rather than directly penalizing interaction.

Thank you for recommending these ablations to clarify the respective roles of request credit and interaction cost. Question 2 and Weakness 2 both concern what drives task performance and the reduction in duplicate requests, so we address them together. We agree that cost-only and credit-only controls provide a direct way to separate the contributions of request credit and interaction cost.

We conducted these controlled experiments using Qwen3-4B on UserGym, ColBench, and τ²-Bench, comparing GRPO (neither credit nor cost), cost-only (cost retained, credit disabled), credit-only (credit retained, cost disabled), and full CrossPO. Credit estimation, calibration, and first-occurrence assignment remain unchanged wherever credit is enabled. Within each benchmark, all configurations retain the base GRPO objective and use the same initialization, rollout budgets, optimization settings, loss normalization, and evaluation protocol.

ConfigurationUserGym Avg. ↑UserGym Dup.(per task) ↓ColBench Succ. ↑ColBench Dup.(per task) ↓τ²-Bench Avg. ↑τ²-Bench Dup.(per task) ↓
GRPO0.4890.230.3260.270.1290.30
Cost-only0.4420.090.3180.170.1060.12
Credit-only0.5940.280.4250.230.1980.32
Full CrossPO0.6150.070.4850.040.2460.05

CrossPO achieves the best task performance and the fewest duplicate requests among the four configurations on all three benchmarks. Compared with cost-only, CrossPO adds request-level supervision that distinguishes requests by their estimated utility and assigns credit only to the first occurrence of each request, providing more targeted guidance for information acquisition beyond the uniform request penalty. Compared with credit-only, CrossPO penalizes repeated requests, discouraging unnecessary interactions and encouraging more efficient use of the interaction budget. Together, these comparisons support the complementary contributions of request credit and interaction cost.We have added these credit and interaction cost ablation results to Appendix D.2(Table 7).

AuthorsOpenReview ↗

Response to Reviewer bWYi (3)

Response to question 3 and weakness 3

Question 3: What fraction of the auxiliary supervision uses native identifiers versus semantic clusters, and how accurately do the latter group equivalent requests? Please also clarify whether Table 1 reports means across training seeds; Figure 2 specifies three seeds for training curves, but the table's aggregation and uncertainty are not stated.

Weakness 3: The reliability of the semantic anchors needs more evidence. Section 3.2 groups requests across different histories using a frozen encoder and a 0.70 threshold. Similar wording can refer to different entities or missing requirements. Complete linkage controls embedding similarity, but does not establish that the grouped requests seek equivalent information. A small annotated audit of matched requests, together with a sensitivity check around the chosen threshold, would help assess the central alignment assumption. It would also be useful to distinguish results using native request identifiers from those relying on semantic clustering.

We appreciate your suggestions on anchor validation and clearer reporting of uncertainty. Question 3 and Weakness 3 both concern anchor reliability and the use of native identifiers versus semantic clustering, so we address them together. We agree that semantic equivalence should be assessed in context and that the contribution of each anchor source should be reported explicitly. Below, we report the anchor-source breakdown, contextual audit, and threshold sensitivity, and clarify the aggregation and uncertainty in Table 1.

We measure the native/semantic proportions over supported first-occurrence request-level credit assignments, counting each assignment once and excluding repeated occurrences. This reports their contribution to auxiliary supervision.

For the semantic audit, two annotators independently inspect sampled pairs of matched requests together with their preceding contexts, without seeing credits or outcomes. A pair is counted as equivalent only if both annotators judge that the requests seek the same information about the same relevant entity or requirement, with compatible answers in their respective contexts.

The table below summarizes the native/semantic assignment proportions and the semantic audit results.

BenchmarkNative assignments (%)Semantic assignments (%)Audited semantic pairsEquivalent pairs (%)
UserGym16.183.920093.5
ColBench010020094
τ²-Bench45.454.620090.5

We report a sensitivity analysis of the semantic clustering threshold using Qwen3-4B on all three benchmarks. The threshold is set to 0.65, 0.70, or 0.75 during training, while all other training settings and the evaluation protocol remain unchanged.

Training thresholdUserGym ↑ColBench Pass ↑ColBench Succ ↑τ²-Bench ↑
0.650.6040.5620.4530.221
0.70 (original setting)0.6150.5760.4850.246
0.750.6120.5830.4870.233

CrossPO outperforms strong baselines across the three tested thresholds. Together with the audit, these results characterize both the accuracy of semantic grouping and the robustness to threshold selection.

The original entries in Table 1 report mean performance across three independent training seeds. We report the standard deviation of the corresponding evaluation scores to quantify variability across training runs. We will explicitly state the aggregation in the Table 1 caption and provide uncertainty for the main test results. A summary for Qwen3-4B across 3 independent training seeds is shown below.

Test metricGRPOInfoPOCrossPO
UserGym average0.489 ± 0.0180.595 ± 0.0070.615 ± 0.006
ColBench pass rate0.423 ± 0.0160.553 ± 0.0120.576 ± 0.010
ColBench success rate0.326 ± 0.0100.439 ± 0.0050.485 ± 0.018
τ²-Bench average0.129 ± 0.0190.167 ± 0.0130.246 ± 0.012

Together, these analyses provide further evidence on anchor reliability, threshold sensitivity, and variability across training seeds. We have incorporated these results and clarifications into the revised manuscript and hope they address your concerns. We appreciate the reviewer's constructive feedback and suggestions for improving our work.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.