- Rating
- 2(+1)
- Confidence
- 3
- Informativeness
- 2(+1)
- Presentation
- 2
- Soundness
- 2(+1)
Read review
Paper summary
This paper proposes Behavior-Aware Policy Optimization (BAPO) for credit assignment in long-horizon LLM agents. BAPO partitions trajectories using changes in behavioral descriptor distributions, scores the resulting intervals through success–failure likelihood ratios, and redistributes advantages toward the highest-scoring interval while preserving the mean token-level advantage. The paper also introduces an inference-time harness that compresses history at behavioral boundaries. Experiments report improvements on ALFWorld and Terminal-Bench, alongside improved performance and reduced inference cost on ScienceWorld.
Critical strengths
- The paper addresses an important problem: extracting useful intermediate learning signals from sparse, trajectory-level rewards in long-horizon agent tasks.
- The proposed mechanism uses behavioral statistics and terminal outcomes without requiring an additional learned reward model.
- The reported results are promising, with improvements over GiGPO on ALFWorld at both model sizes and over DAPO on Terminal-Bench.
Critical weaknesses
- Equation (4) measures the association between behavioral descriptors and eventual success, rather than their contribution conditional on the agent’s current state. For example, successful verification feedback may receive a high score without identifying the earlier actions that enabled success. Moreover, Equations (7)–(9) increase relative credit for only one interval while decreasing it for the remaining intervals, even when several stages are necessary for completion. The paper provides limited evidence that this selection reliably captures the relevant contributions.
- The fixed-interval comparison in Figure 2 evaluates inference-time context compression rather than RL credit assignment. Controlled training ablations are therefore needed to distinguish the effects of behavioral segmentation, contrastive scoring, and selecting only the highest-scoring interval. Comparisons with other segment-level methods are also missing: SPO [1] estimates advantages over trajectory segments, while BEACON [2] partitions agent trajectories at milestones and assigns credit at multiple temporal scales. Comparisons under appropriately controlled compute and supervision budgets would help establish whether BAPO offers benefits beyond adopting an intermediate temporal granularity.
- Segment-level optimization has been explored by SPO [1] and BEACON [2], while GiGPO already derives fine-grained credit from return differences across trajectories sharing the same state. For context management, ReSum [3] compresses interaction histories into summaries and combines this with segmented RL training, while AgentFold [4] maintains compressed historical segments alongside recent interactions. BAPO introduces different implementation choices, particularly JS-based boundaries and likelihood-ratio scoring, but their additional methodological value is not convincingly established. Furthermore, credit redistribution and context compression are evaluated separately, leaving the claimed unification insufficiently supported by evidence of a meaningful interaction between the components.
[1] Guo Y, Xu L, Liu J, et al. Segment policy optimization: Effective segment-level credit assignment in rl for large language models. NIPS2025.
[2] Wang Z, Yan Y, Li H, et al. Milestone-guided policy learning for long-horizon language agents. ICML2026.
[3] Wu X, Li K, Zhao Y, et al. Resum: Unlocking long-horizon search intelligence via context summarization[J]. arXiv preprint arXiv:2509.13313, 2025.
[4] Ye R, Zhang Z, Li K, et al. Agentfold: Long-horizon web agents with proactive context management[J]. arXiv preprint arXiv:2510.24699, 2025.
Questions for authors
- Could the authors provide an example showing raw interactions, their behavioral descriptors, detected boundaries, interval scores, and the resulting advantage adjustments? An example would be particularly helpful for understanding how different Bash operations are represented.
- Which model performs inference-time compression, and what exactly does the normalized “Cost” measure include? Does it account for all summarization calls as well as the agent’s input and output tokens?
6 comments
Response to Reviewer mVQF (Part2/4)
W2. Controlled training ablations are needed to distinguish the effects of behavioral segmentation, contrastive scoring and the selection of credited intervals; comparisons with SPO and BEACON under controlled compute and supervision budgets are missing.
Response: We thank the reviewer for this constructive suggestion. Following the reviewer's suggestion, we conducted a controlled ablation study on ALFWorld (Qwen2.5-1.5B) and Terminal-Bench 2.1 (Qwen3.5-9B). Starting from BAPO, we vary one component at a time while keeping the rollouts, , and training budget fixed. Specifically, the fixed-length variant replaces behavioral boundaries with intervals of steps, BAPO* credits all above-average intervals, and GRPO removes credit redistribution entirely.
| Variant | Segmentation | Scoring | Credited intervals | ALFWorld (1.5B) | TB2.1 (9B) |
|---|---|---|---|---|---|
| GRPO | – | – | – | 72.8 | 28.1 |
| Fixed-length | Fixed () | Contrastive | Eq. (7) | 87.3 | 31.5 |
| BAPO* | Behavioral | Contrastive | All above | 88.4 | 38.2 |
| BAPO | Behavioral | Contrastive | Eq. (7) | 90.8 | 33.3 |
We draw three observations:
-
Contrastive scoring and credit redistribution provide substantial gains. Since segmentation alone does not modify the advantages, the improvement of the fixed-length variant over GRPO (+14.5 on ALFWorld and +3.4 on Terminal-Bench 2.1) reflects the benefit of contrastive scoring and redistribution, even without behavioral segmentation.
-
Behavioral segmentation outperforms fixed-length segmentation. With the same scoring and redistribution strategy, BAPO improves over the fixed-length variant by 3.5 points on ALFWorld and 1.8 points on Terminal-Bench 2.1. Fixed-length intervals may split a coherent behavioral phase or merge unrelated phases, leading to less targeted credit assignment.
-
The optimal number of credited intervals depends on task horizon. Crediting all above-average intervals improves performance on the longer Terminal-Bench tasks (+4.9 points) but slightly reduces performance on ALFWorld (−2.4 points), as discussed in our response to Weakness 1.
Overall, these results show that BAPO's improvements go beyond simply introducing an intermediate temporal granularity. Both behavioral segmentation and contrastive credit redistribution contribute to its effectiveness, while the choice of credited intervals offers a further trade-off across task settings.
Due to the limited rebuttal period, the newly added results are based on a single run, while the GRPO result on ALFWorld is taken from the original paper.
Response to Reviewer mVQF (Part3/4)
W3. Segment-level optimization has been explored by SPO and BEACON, GiGPO derives fine-grained credit from shared states, and ReSum and AgentFold address context management. The additional value of JS-based boundaries and likelihood-ratio scoring, and the claimed unification, are not convincingly established.
Response: We thank the reviewer for this thoughtful comment. We believe that BAPO differs from these methods in where its credit structure comes from, which makes it applicable to settings that they do not address.
SPO estimates segment advantages using Monte Carlo rollouts from each segment boundary, requiring the environment to be restored to intermediate states. Its released code focuses on single-turn mathematical reasoning. For multi-turn agents, restoring intermediate states is difficult, especially in terminal environments where costly snapshots are needed. Moreover, the additional rollouts substantially increase the interaction budget, making a comparison under matched compute impractical. BEACON partitions trajectories into segments using hand-designed, environment-specific milestone detectors and performs reward shaping within each segment. GiGPO, in contrast, derives step-level credit by exploiting identical states that recur across rollouts.
BAPO differs fundamentally from both approaches. It requires neither environment restoration, hand-crafted milestone detectors, nor recurring states. Instead, BAPO identifies credit assignment units through behavioral changes and estimates their importance through success–failure contrasts, using only the rollouts already collected by GRPO. Moreover, its mean-preserving redistribution preserves the original trajectory-level advantage. Together, these properties enable fine-grained credit assignment in open-ended agent environments, including terminal, coding, and web-based tasks.
Following your suggestion, we further compared BAPO with BEACON under controlled compute and supervision budgets. On Terminal-Bench 2.1, we trained BEACON with the same TMax setup and budget as BAPO:
| Method (Qwen3.5-9B) | Terminal-Bench 2.1 |
|---|---|
| GRPO | 28.1 |
| DAPO | 28.8 |
| BEACON | 32.5 |
| BAPO | 33.3 |
| BAPO$^*$ | 38.2 |
As shown in the table, BAPO outperforms BEACON by 0.8 points without any milestone supervision. Together with the ablation in Weakness 2, where likelihood-ratio scoring improves over GRPO by 14.5 / 3.4 points and JS-based boundaries add a further 3.5 / 1.8 points, and with the gains over GiGPO on ALFWorld (+4.1 at 1.5B and +3.2 at 7B), these results provide clear evidence of the additional value of JS-based boundaries and likelihood-ratio scoring.
For context management, ReSum compresses the entire history into a single summary when the context approaches its limit, while AgentFold learns when and how to fold the history through supervised fine-tuning (SFT). In contrast, our harness triggers compression at behavioral boundaries without additional training, ensuring that each summary captures a completed sub-task.
Following the reviewer's suggestion, we reproduced ReSum on ScienceWorld using Qwen3.5-9B. Since its dedicated summarizer (ReSumTool-30B) has not been released, we use the same backbone for summarization, as in our harness, while following ReSum's summarization procedure and using the same step budget. ReSum achieves 62.2% success rate (56/90 tasks), outperforming ReAct (57.0%) and fixed-(k) compression (60.7%). Our harness further improves the success rate to 66.3%, a gain of 4.1 percentage points over ReSum. This suggests that compressing at behavioral boundaries, after completing a sub-task, better preserves task-relevant information than compressing the entire history when the context fills up.
For AgentFold, only inference scripts are available, without model weights or SFT data. Since its folding strategy requires training on curated trajectories, a faithful reproduction without additional training is not feasible within the rebuttal period. We will discuss AgentFold in the related work.
Regarding unification, we refer to a shared behavioral abstraction: the same segmentation defines the units for credit assignment during training and the boundaries for context compression during inference. We evaluate these two settings separately to isolate the contribution of each component. The fact that the same segmentation improves both training and inference across different benchmarks and backbones suggests that it captures general behavioral structure, rather than being tailored to a specific task or setting.
Response to Reviewer mVQF (Part1/4)
We sincerely thank the reviewer for the thorough and thoughtful review.
W1. Eq. (4) measures association with success rather than contribution conditional on the current state, so successful verification feedback may receive a high score without identifying the earlier actions that enabled success. Moreover, Eqs. (7)–(9) increase relative credit for only one interval, even when several stages are necessary.
Response: We thank the reviewer for this insightful comment. We highlight three aspects of our design that make the success–failure contrast in Eq. (4) a practical credit signal for long-horizon agents while reducing the risk of over-crediting verification feedback.
First, the contrast requires no counterfactual rollouts. Estimating each action's contribution conditional on the current state would require restarting the agent from intermediate states, which is costly and often infeasible in terminal environments. Instead, BAPO reuses the rollouts already collected by GRPO, enabling credit assignment without additional environment interactions.
Second, the contrast identifies behaviors that distinguish success from failure within the same task. Routine verification steps, such as running tests or submitting results, often appear in both successful and failed rollouts and therefore receive lower scores. In contrast, behaviors that distinguish successful trajectories, including earlier actions that enable success, receive higher scores. In our worked example (Question 1), inspecting the archived logs receives the highest score, while routine verification steps contribute little.
To provide direct evidence that the selection captures the relevant contributions, we used an LLM judge (Qwen3.8-Max) that sees neither BAPO's boundaries nor its scores. In 78% of the training trajectories, the interval credited by BAPO contains a step that the judge identifies as critical to success.
Regarding the case where several stages are necessary, we truly appreciate this insightful point. BAPO is designed to sharpen credit on the most decisive behavioral transition; as the horizon grows, other transitions can also carry complementary signal. Motivated by your comment, we evaluated a variant, BAPO*, which credits every interval scoring above the trajectory mean, uk=α1[Lk>Lˉ](Lk−Lˉ), with the same centering as Eq. (9):
Due to the limited rebuttal period and computational resources, BAPO* results are from a single run, whereas BAPO results are averaged over three runs as in the paper. Both variants remain above all baselines (e.g., GiGPO 86.7 on ALFWorld and DAPO 28.8 on Terminal-Bench 2.1). BAPO* improves Terminal-Bench 2.1 by 4.9 points, while BAPO remains 2.4 points higher on ALFWorld. These results suggest that the longer the task horizon, the more it benefits from identifying critical states at a finer granularity. We will include BAPO* and this analysis in the revision, and in future work we plan to adapt the number of credited intervals to the horizon and difficulty of each task.