acceptodds
Under review as a conference paper at ICLR 2027

Trajectory-Aware Policy Optimization: Closing the State Distribution Gap in Long-Horizon LLM RL

Abstract

Reinforcement learning (RL) has become a dominant paradigm for enhancing LLMs' reasoning capabilities. Yet long-horizon autoregressive rollouts introduce a fundamental limitation: local policy mismatch can compound into severe state distribution shift over trajectories. Existing LLM RL methods, such as GRPO and GSPO, control policy mismatch at the token or sequence level, but overlook this trajectory-induced shift, causing training instability and eventual collapse as the horizon grows. To close the state distribution gap, we propose Trajectory-Aware Policy Optimization (TAPO), a principled framework that theoretically derives the state-distribution correction from the off-policy policy gradient and incorporates it into the RL objective. TAPO further employs a geometric-mean estimator for variance reduction, together with a length-aware dynamic clipping that prevents overly deviated updates across the trajectory. Extensive experiments demonstrate that TAPO significantly surpasses existing LLM RL algorithms across eight competition-level benchmarks, achieving up to 48% and 12% improvement over GRPO on mathematical reasoning and algorithmic coding, with substantially enhanced training stability.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Reinforcement Learning with Verifiable Rewards, LLM RL, RLVR

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_uTCBOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes a new algorithm TAPO. The paper first identifies the state distribution approximation in TRPO/PPO/GRPO works, and demonstrates that such approximation is not negligible in LLM post-training. The paper then derives prefix-level importance sampling that correctly fixes the state distribution mismatch in the objective, apply standard empirical transformations like geometric average to reduce computation instability, and demonstrate the advantage of TAPO over the other existing methods.

Critical strengths
  1. The motivation experiments in figure 1 clearly demonstrates the state distribution shift problem. Previous algorithms (PPO, GRPO family) have been taking the state distribution approximation in PPO as granted, while such shift problem has been rarely visualized.

  2. The design of the algorithm is well-motivated and easy to understand. Each design component of the algorithm is either principled (e.g., using state distribution importance sampling) or a widely-accepte practice (e.g., using geometric average, length adaptive clipping).

  3. In experiments, a large number of existing algorithms are tested as baselines to support the findings. A wide variety of ablation studies were conducted.

Critical weaknesses
  1. The pharsing "achieving up to 48% and 12% improvement" is misleading. Most papers state absolution accuracy improvement instead of relative improvement in percentage. Such phrasing is eyecatching but causes misinterpretation.

  2. In evaluation, Avg@8 still bears noise in accuracy. Having more responses per problem during evaluation is preferred, e.g. Avg@32.

  3. The novelty of the paper is moderate but acceptable. Prior works such as [1] have identified the same problem in LLM post-training.

[1] A Step Back: Prefix Importance Ratio Stabilizes Policy Optimization

Questions for authors

Do authors have comments on the weaknesses listed above?

Reviewer_U6VWOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies state distribution shift in LLM reinforcement learning on long sequences and proposes TAPO. It weights tokens using the geometric mean of prefix importance ratios and applies dynamic clipping that tightens with token position. Experiments on math and coding tasks across several Qwen3 model sizes report improvements in accuracy and training stability.

Critical strengths
  1. The problem is practically relevant. Expressing the state distribution ratio as a prefix probability ratio offers a clear view of how policy drift accumulates over long sequences.
  2. The method is straightforward, and the experiments cover eight benchmarks, multiple model sizes, and a broad set of baselines. TAPO shows competitive performance and relatively stable training in the tested settings.
Critical weaknesses
  1. The strongest baseline, RIPO, is not analyzed in enough depth. RIPO is a token-level method, yet it performs similarly to TAPO on the reported benchmarks and even better on some metrics. However, the analysis only covers GRPO, GSPO, and TAPO, without checking whether RIPO shows the same degradation on long sequences. Similar performance does not rule out the value of trajectory-aware methods, but the paper has not made clear what TAPO adds over a strong token-level baseline.
  2. The ablations do not fully separate the contributions of prefix information and clipping. Figure 3 compares different prefix weights and clipping settings—no clipping, two fixed bounds, and dynamic clipping—but misses a key control: token-level ratios with the same dynamic clipping. The gains could therefore come mainly from better update constraints rather than prefix state information. The ablations also show only training curves, not final scores, leaving the full method’s performance gains over the ablated variants unclear.
Questions for authors
  1. Does RIPO show the degradation seen in Figure 2? Could you add its training curves and prefix drift diagnostics to clarify what TAPO addresses that RIPO does not?
  2. Could you add a GRPO baseline with the same dynamic clipping and report final results for all ablation settings to show how much each component actually helps?
Reviewer_fhRqOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

In this paper, the authors investigate state distribution shift caused by local policy mismatch in long-horizon LLM RL. To address this problem, they propose Trajectory-Aware Policy Optimization (TAPO). Specifically, TAPO (1) uses a geometric-mean estimator of the trajectory correction ratio and (2) applies length-aware dynamic clipping. Experiments on Qwen3-8B-Base report average Avg@8 scores of 41.0 on mathematical reasoning and 58.6 on code generation, exceeding GRPO’s 27.7 and 52.4, respectively.

Critical strengths

The motivation for considering state distribution shift is well justified. Figure 1 shows that the state-level ratio can deviate substantially while the token-level ratio remains relatively stable. This motivates considering the preceding trajectory when optimizing long-horizon LLM RL.

The experiments provide evidence across model scales and task domains. Table 1 reports the highest average Avg@8 for TAPO on both mathematical reasoning and code generation with Qwen3-8B-Base, while Table 2 reports the highest math averages at 1.7B and 4B. These comparisons suggest that the observed benefit is not confined to one model scale or task domain.

Critical weaknesses
  1. The link between Theorem 1 and the final TAPO objective is incomplete. Eqs. (5)–(7) require R_t for exact state-distribution and action-level correction, while Eq. (14) uses R_t^(1/t). Theorems 2–3 explain variance reduction and concentration, but do not establish how accurately the resulting objective corrects state distribution shift.

  2. The robustness of the gains over RIPO is unclear. Table 1 reports average improvements of 0.5 percentage points on math and 1.4 on code. Tables 1–2 provide no uncertainty estimates, and §4.1 does not specify the number of independent training runs. Further evidence is needed to assess whether these gains are reliable.

Questions for authors
  1. Length-aware dynamic clipping: Theorem 3 describes concentration around E[log R̃_t], whereas Eq. (14) clips around a ratio of 1. How does Eq. (13) account for this difference in center?

  2. Checkpoint selection: How was the “best checkpoint” selected in §4.1? Was selection based on training scores, a separate validation set, or the reported evaluation benchmarks?

  3. Clipping ablations: Were fixed clipping ranges between the two settings in Figure 3 evaluated? Could you report Avg@8 for these ablations to establish whether length-aware dynamic clipping improves benchmark performance as well as training stability?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.