acceptodds
Withdrawn from ICLR 2027

CurioPO: Curbing Spurious Exploration for Stable Multi-Turn Agentic Reinforcement Learning

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Recent progress in multi-turn reinforcement learning (RL) has significantly advanced LLM agents on complex interactive tasks across web, embodied, and search environments. Despite improvements in credit assignment, rollout filtering, and exploration control, training instability remains pervasive and frequently leads to collapse. We identify spurious exploration as a distinct source of this instability: agents produce interactions that appear valid and diverse, yet the resulting observations fail to shift the agent's next-action distribution, injecting unreliable turn-level learning signals into policy updates. To address this issue, we propose \ourmethod, a regime-aware policy optimization framework for stable multi-turn agentic RL. \ourmethod measures each interaction through two complementary signals: action belief shift captures how much an observation changes the model's distribution over next-action tokens, and excess surprise captures whether the observation carries genuine context-dependent novelty beyond surface token rarity. Their gated fusion classifies turns into distinct exploration regimes and enables targeted intervention. At the turn level, \ourmethod evaluates candidates in sandboxed environments before committing them to rollouts and curbs spurious turns according to their regime. At the trajectory-group level, \ourmethod reallocates rollout budget after consecutive non-productive regimes. During policy optimization, a bounded curiosity gate dampens noisy intermediate turns. Experiments on WebShop, ALFWorld, and Search QA show consistent improvements in task performance, training stability, and exploration efficiency over strong multi-turn agentic RL baselines, with reduced KL divergence spikes, suppressed gradient-norm explosion, and fewer spurious turns throughout training.

Primary area:
reinforcement learning
Keywords:
large language models, reinforcement learning, reasoning, RLVR

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_U1JoOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

CurioPO measures feedback productivity with restricted next-action belief shift and context-adjusted observation surprise. These signals guide sandboxed candidate resampling, trajectory restarts, and intermediate-turn advantage weighting. Experiments on WebShop, ALFWorld, and search QA report task performance, stability diagnostics, counterfactual observation masking, matched-call controls, and compute costs.

Critical strengths
  1. The distinction between valid interaction and useful feedback is valuable. Observation masking, usefulness labels, and matched-environment-call controls offer evidence beyond the method's own spurious-turn classification.
  2. The task gains are accompanied by component ablations and training-stability diagnostics across two backbone sizes. The appendix also reports the extra compute, allowing a realistic reading of the exploration-efficiency claim.
Critical weaknesses
  1. The belief-shift proxy uses only first tokens of admissible actions. It can miss useful confirmation, distinctions later in an action string, and delayed bridge information. The reported Heat regression and delayed-usefulness analysis show that this is an actual limitation of the default intervention, not only a hypothetical edge case.
  2. Fewer committed turns do not mean lower total training cost. The reported WebShop comparison increases GPU-hours from 38.2 to 49.5 and wall-clock time from 9.7 to 12.6 hours. Matched-call controls are helpful, but a matched-total-compute learning curve is still needed to assess the efficiency benefit.
Questions for authors
  1. How does the default method compare with a confirmation-aware or richer action representation under matched computation?
  2. At equal GPU-hours or wall-clock budget, how do success and collapse frequency compare with T2PO and GiGPO?
Reviewer_xtFZOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes CurioPO, a method to alleviate the issue of spurious exploration in multi-turn agentic reinforcement learning. The proposed method combines action belief shift and excess surprise to assess feedback productivity, guiding candidate resampling, trajectory budget allocation, and turn-level advantage weighting. This paper conducts experiments on WebShop, ALFWorld, three benchmarks of single-hop QA, and four benchmarks of multi-hop QA. The experimental results report improved task performance and training stability over several baselines.

Critical strengths
  • Novel perspective for agentic RL diagnosis: This paper establishes its main motivation on spurious exploration, which focuses on the effect of observation on the agent's subsequent decision. Compared to some classical perspectives such as performance and diversity, this motivation is insightful.
  • Broad empirical evaluation: This paper conducts experiments on multiple benchmarks, and shows improvements over several RL baselines. This paper also includes component ablations, hyperparameters analysis. And the further experiments such as observation masking experiments and human assessment provide complementary evidence about the proposed signals and mechanisms.
Critical weaknesses
  • Lack sufficient evidence for spurious exploration: Based on the section of introduction, I find spurious exploration to be a plausible phenomenon. However, this paper does not seem to provide sufficient evidence establishing its existence or its causal relationship with training instability. First, Figure 1 is able to suggest that performance degradation cannot be explained solely by malformed or repetitive actions, but it does not measure the informativeness of observations, influence on subsequent actions, or the contribution of these interactions to task progress. Therefore, these three metrics alone do not establish that many interactions appear exploratory while offering little practical value. Second, the subsequent analysis still provide insufficient evidence to establish spurious exploration as a distinct failure mode, particularly as a source of training instability. I suggest that a more thorough investigation of this issue would strengthen the paper’s contribution.
  • Numerical inconsistencies present in Table 9: Based on the experimental setup described in section 5.1, which involves three random seeds, the two sets of "Pick" results for CurioPO in Table 9 appear to be mathematically inconsistent. Specifically, the cells of (RL from Qwen3-4B-RFT & CurioPO, Pick) and (RL from Qwen3-8B-RFT & CurioPO, Pick). Given that the success rates are capped at 100, their sample standard deviation cannot exceed 3(100−xˉ)\sqrt{3}(100-\bar{x}). Consequently, the maximum possible standard deviations for the two rows would be approximately 4.57 and 1.47, respectively—both of which are lower than the reported values ​​of 6.94 and 2.05. I request that the author should conduct a further check.
Questions for authors

Please refer to the above Weaknesses.

Reviewer_4S5eOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper proposes CurioPO, a framework for stabilizing multi-turn agentic reinforcement learning by reducing “spurious exploration”,interactions that appear diverse or novel but provide little useful information for subsequent decisions. It combines excess surprise and action belief shift to assess feedback productivity, guiding turn-level resampling, trajectory-budget allocation, and intermediate-turn advantage weighting. Experiments on WebShop, ALFWorld, and search-augmented QA report improved task performance, training stability, and exploration efficiency.

Critical strengths

None

Critical weaknesses

Dear Area Chair,

I would like to confidentially raise a serious concern regarding the experimental results in submission 35468 (CurioPO).

Across WebShop, ALFWorld, and Search QA, 25 distinct result entries attributed to CurioPO exactly match those reported as “T²PO (Ours)” in arXiv:2605.02178v1, including all associated standard deviations. T²PO has been published at ICML 2026 (proceedings record). The corresponding locations are:

Submission 35468T²PO v1
Table 1, p. 7, lines 339 and 343Table 1, p. 6
Table 9, p. 18, lines 934 and 939Table 1, p. 6
Table 2, p. 7, line 358Table 2, p. 7

These comprise 18 mean–standard-deviation pairs and seven QA scores, excluding repeated entries.

However, the submission separately reports different, generally lower T²PO baseline results (p. 7, lines 338, 342, and 357), against which CurioPO’s improvements are presented. The attribution of the prior T²PO results to CurioPO, alongside these different baseline values, requires explanation.

The papers also share similar figure layouts, table structures, and experimental presentation. The plotted curves differ, so I regard these similarities as supporting context rather than evidence of identical figure reuse.

Given the extensive exact numerical overlap and the unresolved result attribution, I have serious concerns about the reliability of the submission’s central empirical claims and recommend desk rejection. If author clarification is required procedurally, I suggest requesting verifiable original experimental records, including per-seed logs, complete configurations, and checkpoint identifiers.

Sincerely,
Reviewer

Questions for authors

None

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.