acceptodds
Under review as a conference paper at ICLR 2027

BAPO: Behavior-Aware Policy Optimization for Fine-Grained Credit Assignment in Long-Horizon Agents

Abstract

Reinforcement learning (RL) has become an effective approach for improving the sequential decision-making capabilities of large language model (LLM) agents. While outcome-based methods such as Group Relative Policy Optimization (GRPO) offer a simple way to optimize agent policies using task-level rewards, their trajectory-level supervision often fails to distinguish the contributions of intermediate interactions. Recent methods therefore exploit finer-grained structures within agent trajectories to improve credit assignment. A key challenge is to distinguish intermediate states that reflect meaningful task progress from those that contribute little to the final outcome. Changes in an agent's tool-use behavior can provide useful signals for locating such task-relevant states along the trajectory. Building on this insight, we propose **Behavior-Aware Policy Optimization** (BAPO), a simple method for identifying task-relevant intermediate states and using them to construct fine-grained learning signals for policy optimization. BAPO first detects coarse changes in an agent's tool-use behavior to localize candidate transition regions. It then attributes task-level outcomes to these candidate transitions by contrasting behavioral patterns across successful and failed trajectories, identifying transitions associated with task success. Based on this attribution, BAPO redistributes trajectory-level advantages toward actions associated with reaching these states. In this way, BAPO uses task outcomes to provide differentiated supervision for intermediate interactions. Experiments on ALFWorld and Terminal-Bench with Qwen-family models show that BAPO consistently improves agent performance, with success-rate gains of up to 4.5 %. Beyond policy optimization, we find that these transition signals can also guide inference-time trajectory compression, suggesting that they capture behaviorally meaningful structure useful for both training and inference.

Primary area:
foundation or frontier models, including LLMs
Keywords:
agentic rl, credit assignment

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_mVQFOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
2(+1)
Presentation
2
Soundness
2(+1)
Read review
Paper summary

This paper proposes Behavior-Aware Policy Optimization (BAPO) for credit assignment in long-horizon LLM agents. BAPO partitions trajectories using changes in behavioral descriptor distributions, scores the resulting intervals through success–failure likelihood ratios, and redistributes advantages toward the highest-scoring interval while preserving the mean token-level advantage. The paper also introduces an inference-time harness that compresses history at behavioral boundaries. Experiments report improvements on ALFWorld and Terminal-Bench, alongside improved performance and reduced inference cost on ScienceWorld.

Critical strengths
  1. The paper addresses an important problem: extracting useful intermediate learning signals from sparse, trajectory-level rewards in long-horizon agent tasks.
  2. The proposed mechanism uses behavioral statistics and terminal outcomes without requiring an additional learned reward model.
  3. The reported results are promising, with improvements over GiGPO on ALFWorld at both model sizes and over DAPO on Terminal-Bench.
Critical weaknesses
  1. The proposed attribution score does not sufficiently establish that the selected intervals represent task-critical progress.\textbf{The proposed attribution score does not sufficiently establish that the selected intervals represent task-critical progress.} Equation (4) measures the association between behavioral descriptors and eventual success, rather than their contribution conditional on the agent’s current state. For example, successful verification feedback may receive a high score without identifying the earlier actions that enabled success. Moreover, Equations (7)–(9) increase relative credit for only one interval while decreasing it for the remaining intervals, even when several stages are necessary for completion. The paper provides limited evidence that this selection reliably captures the relevant contributions.
  2. The experiments do not isolate the proposed mechanism’s benefits or establish its advantage over existing segment-level approaches.\textbf{The experiments do not isolate the proposed mechanism’s benefits or establish its advantage over existing segment-level approaches.} The fixed-interval comparison in Figure 2 evaluates inference-time context compression rather than RL credit assignment. Controlled training ablations are therefore needed to distinguish the effects of behavioral segmentation, contrastive scoring, and selecting only the highest-scoring interval. Comparisons with other segment-level methods are also missing: SPO [1] estimates advantages over trajectory segments, while BEACON [2] partitions agent trajectories at milestones and assigns credit at multiple temporal scales. Comparisons under appropriately controlled compute and supervision budgets would help establish whether BAPO offers benefits beyond adopting an intermediate temporal granularity.
  3. The framework appears to combine established components without sufficiently demonstrating a distinct methodological advance.\textbf{The framework appears to combine established components without sufficiently demonstrating a distinct methodological advance.} Segment-level optimization has been explored by SPO [1] and BEACON [2], while GiGPO already derives fine-grained credit from return differences across trajectories sharing the same state. For context management, ReSum [3] compresses interaction histories into summaries and combines this with segmented RL training, while AgentFold [4] maintains compressed historical segments alongside recent interactions. BAPO introduces different implementation choices, particularly JS-based boundaries and likelihood-ratio scoring, but their additional methodological value is not convincingly established. Furthermore, credit redistribution and context compression are evaluated separately, leaving the claimed unification insufficiently supported by evidence of a meaningful interaction between the components.

[1] Guo Y, Xu L, Liu J, et al. Segment policy optimization: Effective segment-level credit assignment in rl for large language models. NIPS2025.

[2] Wang Z, Yan Y, Li H, et al. Milestone-guided policy learning for long-horizon language agents. ICML2026.

[3] Wu X, Li K, Zhao Y, et al. Resum: Unlocking long-horizon search intelligence via context summarization[J]. arXiv preprint arXiv:2509.13313, 2025.

[4] Ye R, Zhang Z, Li K, et al. Agentfold: Long-horizon web agents with proactive context management[J]. arXiv preprint arXiv:2510.24699, 2025.

Questions for authors
  1. Could the authors provide an example showing raw interactions, their behavioral descriptors, detected boundaries, interval scores, and the resulting advantage adjustments? An example would be particularly helpful for understanding how different Bash operations are represented.
  2. Which model performs inference-time compression, and what exactly does the normalized “Cost” measure include? Does it account for all summarization calls as well as the agent’s input and output tokens?
6 comments
AuthorsOpenReview ↗

Response to Reviewer mVQF (Part1/4)

We sincerely thank the reviewer for the thorough and thoughtful review.

W1. Eq. (4) measures association with success rather than contribution conditional on the current state, so successful verification feedback may receive a high score without identifying the earlier actions that enabled success. Moreover, Eqs. (7)–(9) increase relative credit for only one interval, even when several stages are necessary.

Response: We thank the reviewer for this insightful comment. We highlight three aspects of our design that make the success–failure contrast in Eq. (4) a practical credit signal for long-horizon agents while reducing the risk of over-crediting verification feedback.

First, the contrast requires no counterfactual rollouts. Estimating each action's contribution conditional on the current state would require restarting the agent from intermediate states, which is costly and often infeasible in terminal environments. Instead, BAPO reuses the rollouts already collected by GRPO, enabling credit assignment without additional environment interactions.

Second, the contrast identifies behaviors that distinguish success from failure within the same task. Routine verification steps, such as running tests or submitting results, often appear in both successful and failed rollouts and therefore receive lower scores. In contrast, behaviors that distinguish successful trajectories, including earlier actions that enable success, receive higher scores. In our worked example (Question 1), inspecting the archived logs receives the highest score, while routine verification steps contribute little.

To provide direct evidence that the selection captures the relevant contributions, we used an LLM judge (Qwen3.8-Max) that sees neither BAPO's boundaries nor its scores. In 78% of the training trajectories, the interval credited by BAPO contains a step that the judge identifies as critical to success.

Regarding the case where several stages are necessary, we truly appreciate this insightful point. BAPO is designed to sharpen credit on the most decisive behavioral transition; as the horizon grows, other transitions can also carry complementary signal. Motivated by your comment, we evaluated a variant, BAPO*, which credits every interval scoring above the trajectory mean, uk=α 1[Lk>Lˉ] (Lk−Lˉ)u_k = \alpha \, \mathbb{1}[L_k > \bar{L}] \, (L_k - \bar{L}), with the same centering as Eq. (9):

MethodALFWorld (Qwen2.5-1.5B)Terminal-Bench 2.1 (Qwen3.5-9B)
BAPO90.833.3
BAPO*88.438.2

Due to the limited rebuttal period and computational resources, BAPO* results are from a single run, whereas BAPO results are averaged over three runs as in the paper. Both variants remain above all baselines (e.g., GiGPO 86.7 on ALFWorld and DAPO 28.8 on Terminal-Bench 2.1). BAPO* improves Terminal-Bench 2.1 by 4.9 points, while BAPO remains 2.4 points higher on ALFWorld. These results suggest that the longer the task horizon, the more it benefits from identifying critical states at a finer granularity. We will include BAPO* and this analysis in the revision, and in future work we plan to adapt the number of credited intervals to the horizon and difficulty of each task.

AuthorsOpenReview ↗

Response to Reviewer mVQF (Part2/4)

W2. Controlled training ablations are needed to distinguish the effects of behavioral segmentation, contrastive scoring and the selection of credited intervals; comparisons with SPO and BEACON under controlled compute and supervision budgets are missing.

Response: We thank the reviewer for this constructive suggestion. Following the reviewer's suggestion, we conducted a controlled ablation study on ALFWorld (Qwen2.5-1.5B) and Terminal-Bench 2.1 (Qwen3.5-9B). Starting from BAPO, we vary one component at a time while keeping the rollouts, α\alpha, and training budget fixed. Specifically, the fixed-length variant replaces behavioral boundaries with intervals of k=5k=5 steps, BAPO* credits all above-average intervals, and GRPO removes credit redistribution entirely.

VariantSegmentationScoringCredited intervalsALFWorld (1.5B)TB2.1 (9B)
GRPO–––72.828.1
Fixed-lengthFixed (k=5k=5)ContrastiveEq. (7)87.331.5
BAPO*BehavioralContrastiveAll above Lˉ\bar{L}88.438.2
BAPOBehavioralContrastiveEq. (7)90.833.3

We draw three observations:

  1. Contrastive scoring and credit redistribution provide substantial gains. Since segmentation alone does not modify the advantages, the improvement of the fixed-length variant over GRPO (+14.5 on ALFWorld and +3.4 on Terminal-Bench 2.1) reflects the benefit of contrastive scoring and redistribution, even without behavioral segmentation.

  2. Behavioral segmentation outperforms fixed-length segmentation. With the same scoring and redistribution strategy, BAPO improves over the fixed-length variant by 3.5 points on ALFWorld and 1.8 points on Terminal-Bench 2.1. Fixed-length intervals may split a coherent behavioral phase or merge unrelated phases, leading to less targeted credit assignment.

  3. The optimal number of credited intervals depends on task horizon. Crediting all above-average intervals improves performance on the longer Terminal-Bench tasks (+4.9 points) but slightly reduces performance on ALFWorld (−2.4 points), as discussed in our response to Weakness 1.

Overall, these results show that BAPO's improvements go beyond simply introducing an intermediate temporal granularity. Both behavioral segmentation and contrastive credit redistribution contribute to its effectiveness, while the choice of credited intervals offers a further trade-off across task settings.

Due to the limited rebuttal period, the newly added results are based on a single run, while the GRPO result on ALFWorld is taken from the original paper.

AuthorsOpenReview ↗

Response to Reviewer mVQF (Part3/4)

W3. Segment-level optimization has been explored by SPO and BEACON, GiGPO derives fine-grained credit from shared states, and ReSum and AgentFold address context management. The additional value of JS-based boundaries and likelihood-ratio scoring, and the claimed unification, are not convincingly established.

Response: We thank the reviewer for this thoughtful comment. We believe that BAPO differs from these methods in where its credit structure comes from, which makes it applicable to settings that they do not address.

SPO estimates segment advantages using Monte Carlo rollouts from each segment boundary, requiring the environment to be restored to intermediate states. Its released code focuses on single-turn mathematical reasoning. For multi-turn agents, restoring intermediate states is difficult, especially in terminal environments where costly snapshots are needed. Moreover, the additional rollouts substantially increase the interaction budget, making a comparison under matched compute impractical. BEACON partitions trajectories into segments using hand-designed, environment-specific milestone detectors and performs reward shaping within each segment. GiGPO, in contrast, derives step-level credit by exploiting identical states that recur across rollouts.

BAPO differs fundamentally from both approaches. It requires neither environment restoration, hand-crafted milestone detectors, nor recurring states. Instead, BAPO identifies credit assignment units through behavioral changes and estimates their importance through success–failure contrasts, using only the rollouts already collected by GRPO. Moreover, its mean-preserving redistribution preserves the original trajectory-level advantage. Together, these properties enable fine-grained credit assignment in open-ended agent environments, including terminal, coding, and web-based tasks.

Following your suggestion, we further compared BAPO with BEACON under controlled compute and supervision budgets. On Terminal-Bench 2.1, we trained BEACON with the same TMax setup and budget as BAPO:

Method (Qwen3.5-9B)Terminal-Bench 2.1
GRPO28.1
DAPO28.8
BEACON32.5
BAPO33.3
BAPO$^*$38.2

As shown in the table, BAPO outperforms BEACON by 0.8 points without any milestone supervision. Together with the ablation in Weakness 2, where likelihood-ratio scoring improves over GRPO by 14.5 / 3.4 points and JS-based boundaries add a further 3.5 / 1.8 points, and with the gains over GiGPO on ALFWorld (+4.1 at 1.5B and +3.2 at 7B), these results provide clear evidence of the additional value of JS-based boundaries and likelihood-ratio scoring.

For context management, ReSum compresses the entire history into a single summary when the context approaches its limit, while AgentFold learns when and how to fold the history through supervised fine-tuning (SFT). In contrast, our harness triggers compression at behavioral boundaries without additional training, ensuring that each summary captures a completed sub-task.

Following the reviewer's suggestion, we reproduced ReSum on ScienceWorld using Qwen3.5-9B. Since its dedicated summarizer (ReSumTool-30B) has not been released, we use the same backbone for summarization, as in our harness, while following ReSum's summarization procedure and using the same step budget. ReSum achieves 62.2% success rate (56/90 tasks), outperforming ReAct (57.0%) and fixed-(k) compression (60.7%). Our harness further improves the success rate to 66.3%, a gain of 4.1 percentage points over ReSum. This suggests that compressing at behavioral boundaries, after completing a sub-task, better preserves task-relevant information than compressing the entire history when the context fills up.

For AgentFold, only inference scripts are available, without model weights or SFT data. Since its folding strategy requires training on curated trajectories, a faithful reproduction without additional training is not feasible within the rebuttal period. We will discuss AgentFold in the related work.

Regarding unification, we refer to a shared behavioral abstraction: the same segmentation defines the units for credit assignment during training and the boundaries for context compression during inference. We evaluate these two settings separately to isolate the contribution of each component. The fact that the same segmentation improves both training and inference across different benchmarks and backbones suggests that it captures general behavioral structure, rather than being tailored to a specific task or setting.

Reviewer_AjRXOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes Behavior-Aware Policy Optimization (BAPO), a simple framework to improve credit assignment for long-horizon LLM agents by leveraging changes in tool-use behavior along trajectories. BAPO segments each rollout into behavioral intervals using a local Jensen–Shannon divergence over compact action/target/feedback descriptors, attributes outcome credit to intervals via a contrastive frequency ratio between successful and failed trajectories, and redistributes GRPO-style advantages to the most success-associated interval in a mean-preserving, zero-sum fashion. The same segmentation signal is used at inference time to compress completed history into semantic milestones, yielding lower context cost and improved robustness. Experiments on ALFWorld, Terminal-Bench 2.1, and ScienceWorld show consistent gains over several baselines.

Critical strengths

Technical novelty and innovation

  • The paper articulates a clean, lightweight pipeline for (i) unsupervised behavioral boundary detection via local distributional shifts in tool-use descriptors and (ii) outcome-conditioned interval scoring that requires no external reward model or verifier.

Clarity of presentation

  • The method is well-motivated, with clear equations outlining boundary detection, contrastive attribution, and mean-preserving redistribution.

  • The high-level flow (Figure 1) effectively communicates the three components: boundary detection, redistribution, and context management.

Problem Significance

  • Addresses an important bottleneck—credit assignment under sparse terminal rewards—and suggests a practical alternative to learned critics or external verifiers/judges.

  • The inference harness is likely useful to practitioners who struggle with long-context drift and cost in deployment scenarios.

Critical weaknesses

Technical limitations or concerns

  • Concentrating additional credit solely on the single top-scoring interval may be brittle and can ignore multi-step or multi-interval progress (e.g., preparatory and execution phases both necessary for success).

  • The descriptor design (action, target, feedback) relies on a structured and finite action space; its generalization to tool-rich, unstructured settings or noisier feedback remains under-specified.

Experimental gaps

  • Claims about accelerated convergence and enhanced training stability are not backed by learning curves or stability diagnostics in the main text.

  • TB 2.1 results compare mainly against the base and DAPO; more targeted agent-RL credit assignment baselines on TB 2.1 would strengthen the case.

Clarity or presentation issues

  • Key implementation details are under-specified: probability estimation for P(x|r) (smoothing, priors), descriptor vocabulary construction and normalization, thresholds (δ) and peak radius (w) selection, and computational overhead.

Missing related work or comparisons

  • A discussion of how BAPO’s interval-level signal differs from milestone-based reward shaping (BEACON) or adaptive granularity (GACA), and why BAPO is preferable in particular settings, is missing.

  • ScienceWorld harness comparisons omit more recent context-management/compression baselines (e.g., ReSum, AgentFold, ACON), which directly target the same problem.

Questions for authors
  • Why does BAPO assign additional credit only to the top-scoring interval? Could this overlook multiple intervals that jointly contribute to task success?

  • How well does the proposed behavioral descriptor generalize to tool-rich environments with unstructured actions or noisy feedback?

  • Could the authors provide learning curves and stability analyses to support the claims of faster convergence and improved training stability?

  • Could the authors include stronger agent-RL credit assignment baselines on Terminal-Bench 2.1 to better demonstrate BAPO's effectiveness?

  • Could the authors clarify the estimation and smoothing of P(x∣r)P(x \mid r), descriptor vocabulary construction, hyperparameter selection (δ\delta, ww), and computational overhead?

  • How does BAPO differ from milestone-based reward shaping (BEACON) and adaptive granularity methods (GACA)? In which settings would BAPO be preferable?

  • How does the proposed context-management harness compare with recent compression methods such as ReSum, AgentFold, and ACON on ScienceWorld?

5 comments
AuthorsOpenReview ↗

Response to Reviewer AjRX (Part 1/3)

We highly appreciate the reviewer for the careful and constructive comments.

Q1: Why does BAPO assign additional credit only to the top-scoring interval? Could this overlook multiple intervals that jointly contribute to task success?

Response: We thank the reviewer for this insightful question. BAPO is designed to concentrate credit on the most decisive behavioral transition. However, we agree that longer trajectories may contain multiple critical transitions that jointly contribute to success. Motivated by the reviewer's suggestion, we evaluated a multi-interval variant, BAPO*, which credits all intervals scoring above the trajectory mean:

uk=α 1[Lk>Lˉ](Lk−Lˉ),u_k = \alpha\,\mathbb{1}[L_k > \bar{L}](L_k-\bar{L}),

followed by the same mean-preserving centering as in Eq. (9). BAPO* introduces no additional hyperparameters, and the number of credited intervals adapts to each trajectory.

MethodALFWorld (Qwen2.5-1.5B)Terminal-Bench 2.1 (Qwen3.5-9B)
BAPO90.833.3
BAPO*88.438.2

Due to the limited rebuttal period and computational budget, BAPO* results are based on a single run, while BAPO results are averaged over three runs as reported in the paper. Both variants outperform the reported baselines, including GiGPO (86.7) on ALFWorld and DAPO (28.8) on Terminal-Bench 2.1.

BAPO* improves Terminal-Bench 2.1 by 4.9 points, where tasks typically involve multiple necessary stages, such as exploration, implementation, and verification. In contrast, BAPO remains 2.4 points higher on ALFWorld, where tasks are shorter and often hinge on a single decisive transition.

These results suggest that the appropriate granularity of credit assignment depends on task horizon and complexity: concentrating credit on one transition can be effective for shorter tasks, while longer, multi-stage tasks may benefit from crediting multiple transitions. We will include BAPO* and this analysis in the revision. In future work, we plan to adapt the number of credited intervals to task horizon and difficulty.

Q2: How well does the proposed behavioral descriptor generalize to tool-rich environments with unstructured actions or noisy feedback?

Response: We thank the reviewer for this important question. Terminal-Bench 2.1 already provides a challenging setting with highly unstructured actions. Agents can issue arbitrary Bash commands, including pipelines, chained commands, inline Python, and heredoc file writes, while receiving free-form outputs. We use three design choices to handle such environments:

First, descriptors capture functional behavior rather than surface syntax. The action represents the command category (e.g., list, read, search, edit, test, run, submit), or the tool name for function-calling agents. The target captures the argument type (e.g., file extension, directory, or test suite) rather than the literal path, while the feedback records execution status (ok, error, or timeout). Thus, functionally similar operations on different files map to the same descriptor.

Second, composite commands are decomposed into basic operations. For example, a && b produces one descriptor per executed sub-command, while a heredoc file write is treated as a single edit:

Raw Bash commandDescriptor(s)
ls -la logs/ && cat incident.py(list, dir, ok), (read, .py, ok)
python3 -c "import gzip; ..."(decompress, .gz, ok)
cat > logparse/parser.py << 'EOF' ...(edit, .py, ok)
python3 -m unittest discover -s tests(test, tests, error)
python3 incident.py && cat out/incidents.json(run, .py, ok), (read, .json, ok)

Third, contrastive scoring reduces the influence of non-discriminative feedback. Since scores are computed by contrasting successful and failed rollouts of the same task, common feedback patterns, such as benign warnings, receive scores close to zero when they occur at similar frequencies in both groups. Thus, credit redistribution focuses on behavioral patterns that systematically distinguish success from failure.

In a representative terminal task with 16 rollouts and 363 steps, these rules produce only 31 distinct descriptors, while still distinguishing exploration, editing, testing, and verification phases. This illustrates that the descriptor vocabulary can remain compact even with free-form Bash interactions.

For larger tool sets, the action field can be extended using tool categories or clusters, without changing the boundary detection, contrastive scoring, or credit redistribution mechanisms. We will add the descriptor rules and examples to the appendix.

AuthorsOpenReview ↗

Response to Reviewer AjRX (Part 2/3)

Q3: Could the authors provide learning curves and stability analyses to support the claims of faster convergence and improved training stability?

Response: We thank the reviewer for this valuable suggestion. Our statement on training stability is supported by the multi-seed results in the paper. On ALFWorld, BAPO not only achieves higher success than GRPO but also shows markedly smaller variance across seeds: ±1.9\pm 1.9 vs. ±3.6\pm 3.6 at 1.5B and ±1.4\pm 1.4 vs. ±5.2\pm 5.2 at 7B (Table 2). This is consistent with the design of BAPO: the mean-preserving redistribution in Eq. (9) leaves the trajectory-level GRPO advantage unchanged and only reshapes credit within a trajectory, so it does not introduce an additional source of instability. "Accelerated convergence" refers to BAPO reaching clearly higher success than the GRPO-family baselines under the same training budget and number of updates (GRPO in Table 2 and DAPO in Table 1), i.e., a more efficient use of the same optimization steps. We will make the meaning of both terms explicit in the revision.

Q4: Could the authors include stronger agent-RL credit assignment baselines on Terminal-Bench 2.1 to better demonstrate BAPO's effectiveness?

Response: We thank the reviewer for this helpful suggestion. Many fine-grained credit-assignment methods, such as GiGPO and G2PO, rely on identical states recurring across rollouts. In Terminal-Bench, the file system, processes and outputs seldom repeat exactly, so these methods largely reduce to GRPO. Following your suggestion, we additionally evaluated GRPO, a segment-level credit baseline that applies our contrastive scoring to fixed-length segments (one interval every k=5k=5 steps), and the milestone-based method BEACON, all under the same TMax training setup and budget as BAPO:

Method (Qwen3.5-9B)Terminal-Bench 2.1
Base model16.1
GRPO28.1
DAPO28.8
Fixed-length segment credit31.5
BEACON (hand-designed milestones)32.5
BAPO33.3
BAPO$^*$38.2

We make the following observations:

  1. BAPO improves over the outcome-based baselines GRPO and DAPO by 5.2 and 4.5 points, respectively.

  2. BAPO also outperforms segment-level credit on fixed-length segments by 1.8 points, showing that aligning the credit units with behavioral phases matters on terminal tasks.

  3. BAPO outperforms BEACON by 0.8 points without any milestone supervision, and BAPO$^*$ further raises the success rate to 38.2.

As shown in the table, these results demonstrate that behavior-derived credit units are effective even when neither state recurrence nor milestone supervision is available.

Q5: Could the authors clarify the estimation and smoothing of P(x∣r)P(x \mid r), the descriptor vocabulary construction, the hyperparameter selection (δ\delta, ww), and the computational overhead?

Response: We thank the reviewer for these questions. P(x∣r)P(x \mid r) is estimated within each task group at every training step from the rollouts already sampled for the policy update, with Laplace smoothing P(x∣r)=cr(x)+ϵ∑x′cr(x′)+ϵ∣V∣P(x \mid r) = \frac{c_r(x) + \epsilon}{\sum_{x'} c_r(x') + \epsilon |V|}, where cr(x)c_r(x) counts descriptor xx in rollouts with outcome rr and ϵ=[X]\epsilon=[X]; groups that are all-success or all-failure provide no contrast, so we set λ(x)=0\lambda(x)=0 and BAPO reduces to GRPO. The descriptor vocabulary is not predefined but built online from the observed (action, target, feedback) tuples, with arguments mapped to their types (Question 2). The window W=3W=3 is selected in Figure 2 (right), while δ=[X]\delta=[X] and w=[X]w=[X] are set once on held-out training rollouts and kept fixed across all experiments. BAPO requires no additional model calls, rollouts, critic or reward model: boundary detection and scoring are pure counting operations that take only milliseconds per group, which is negligible compared with rollout generation.

AuthorsOpenReview ↗

Response to Reviewer AjRX (Part 3/3)

Q6: How does BAPO differ from milestone-based reward shaping (BEACON) and adaptive granularity methods (GACA)? In which settings would BAPO be preferable?

Response: We thank the reviewer for pointing us to these related works. BEACON partitions trajectories at milestones identified by hand-designed, environment-specific detectors, and combines within-segment reward shaping with segment- and trajectory-level advantages. GACA does not segment trajectories; it adaptively mixes episode- and step-level credit for each decision, using the policy's normalized per-step NLL as a criticality score, and its step-level credit reuses GiGPO's grouping of identical states across rollouts. BAPO differs from both in that it discovers its credit units from behavioral change and scores them by contrasting successful and failed rollouts, requiring neither milestone detectors nor recurring states.

Our experiments support this distinction:

  1. Compared with BEACON, BAPO matches its reported results on ALFWorld (90.8 vs. 91.4 at 1.5B and 94.0 vs. 94.5 at 7B) without any milestone supervision, and outperforms it by 0.8 points on Terminal-Bench 2.1 when trained under the same TMax setup (Question 4).

  2. Compared with state-grouping credit, on which GACA's step-level signal is built, BAPO outperforms GiGPO on ALFWorld by 4.1 and 3.2 points. On Terminal-Bench 2.1, where identical states rarely recur and such step-level credit largely reduces to episode-level credit, BAPO improves over GRPO by 5.2 points, and BAPO$^*$ by 10.1 points.

  3. Under identical supervision, behavior-derived intervals outperform fixed-length segments by 3.5 and 1.8 points on ALFWorld and Terminal-Bench 2.1 (see also our response to Reviewer mVQF), confirming that the choice of credit units matters.

These results show that BAPO brings the benefits of fine-grained credit assignment to open-ended environments, such as terminal, coding and web tasks, where milestones are hard to define and states rarely recur. GACA's per-step weighting is complementary and could be combined with BAPO's interval scores. We will add this discussion to the related work.

Q7: How does the proposed context-management harness compare with recent compression methods such as ReSum, AgentFold, and ACON on ScienceWorld?

Response: We thank the reviewer for this helpful suggestion. These methods differ mainly in when and what they compress, and whether additional training or optimization is required. ReSum summarizes the entire history when the context approaches its limit; AgentFold learns when and how to fold the history through supervised fine-tuning (SFT); and ACON optimizes compression guidelines (i.e., what information to retain) using an optimizer LLM on training tasks. In contrast, our harness determines when to compress based on the agent's behavior, triggering compression at behavioral boundaries so that each summary captures a completed sub-task, without additional training or guideline optimization.

Following the reviewer's suggestion, we reproduced ReSum and ACON on ScienceWorld using Qwen3.5-9B, with the same backbone for compression and the same step budget across methods. For ReSum, whose dedicated summarizer (ReSumTool-30B) has not been released, we follow its summarization procedure. For ACON, we follow its official pipeline to optimize compression guidelines on training tasks disjoint from the 90 evaluation tasks.

Method (Qwen3.5-9B)SR
ReAct57.0
ClaudeCode58.9
Fixed-kk60.7
ReSum62.2
ACON64.4
Ours66.3

We make three observations:

1.All compression methods outperform ReAct, suggesting that managing long interaction histories benefits long-horizon agents.

2.Our harness outperforms ACON by 1.9 points, despite requiring neither an optimizer LLM nor additional training tasks. ACON benefits from optimized compression guidelines, while our harness relies solely on behavioral boundaries to determine when to compress.

3.Our harness also improves over ReSum and fixed-(k) by 4.1 and 5.6 points, respectively. These results suggest that triggering compression at meaningful behavioral transitions is more effective than compressing at fixed intervals or when the context approaches its limit.

Overall, our harness achieves the highest success rate among the compared methods without additional training or guideline optimization. AgentFold releases only inference scripts, without model weights or SFT data. Since its folding strategy requires SFT on curated trajectories, a faithful reproduction is not feasible within the rebuttal period. Moreover, ACON and our harness are complementary: ACON optimizes what to retain, while our harness determines when to compress. Its optimized guidelines could therefore be incorporated into our compressor.

Reviewer_KNYKOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper addresses credit assignment for long-horizon agents trained with trajectory-level rewards. The authors explore whether behavioral shifts can identify meaningful units for credit assignment. They propose Behavior-Aware Policy Optimization (BAPO), which detects behavioral changes to segment trajectories, scores the segments by their association with task success, and redistributes advantages across them. Behavioral boundaries are also used for inference-time context compression. Experiments show improvements over the evaluated baselines on ALFWorld and Terminal-Bench 2.1.

Critical strengths
  1. The paper uses behavioral changes to guide credit assignment. Behavioral boundaries are detected automatically from interaction traces, without requiring manual annotations.
  2. The method shows performance improvements over the baselines across multiple environments.
Critical weaknesses
  1. The author states “To pinpoint the critical bottleneck that drives task progression, BAPO next identifies the interval with the strongest empirical association with task success.” Long-horizon tasks may involve multiple critical bottlenecks. Why does BAPO select only one interval for additional credit, and how would allocating credit to multiple high-scoring intervals affect performance?

  2. The ablation study is insufficient to show the contributions of the core design choices. The segmentation comparison only evaluates the inference task.

Questions for authors
  1. Long-horizon tasks may involve multiple critical bottlenecks. Why does BAPO select only one interval for additional credit, and how would allocating credit to multiple high-scoring intervals affect performance?

  2. The ablation study comparing behavioral segmentation with fixed-length segmentation is conducted only for inference task on ScienceWorld. It remains unclear whether behavioral segmentation also improves credit assignment during training compared with fixed-length segmentation. Could the authors extend this comparison to the training setting?

  3. Could the authors use model assessments of completed trajectories to examine whether the detected boundaries correspond to meaningful task stages and whether higher-scoring segments contain behaviors critical to task success? This could provide supporting evidence for both the segmentation and scoring mechanisms.

4 comments
AuthorsOpenReview ↗

Response to Reviewer KNYK (Part 1/2)

We sincerely thank the reviewer for the positive assessment and the constructive feedback, and for recognizing that BAPO detects behavioral boundaries automatically from interaction traces without manual annotation and improves over the baselines across multiple environments. We address each comment in detail below.

W1 & Q1. Long-horizon tasks may involve multiple critical bottlenecks. Why does BAPO select only one interval for additional credit, and how would allocating credit to multiple high-scoring intervals affect performance?

Response: We sincerely thank the reviewer for raising this insightful point. BAPO is designed to sharpen credit on the most decisive behavioral transitions; as the horizon grows, other transitions can also carry complementary signal. Motivated by your insightful comment, we evaluated a variant, BAPO$^*$, which credits every interval scoring above the trajectory mean, uk=α 1[Lk>Lˉ] (Lk−Lˉ)u_k = \alpha \, \mathbb{1}[L_k > \bar{L}] \, (L_k - \bar{L}), with the same centering as Eq. (9).

MethodALFWorld (Qwen2.5-1.5B)Terminal-Bench 2.1 (Qwen3.5-9B)
BAPO90.833.3
BAPO$^*$88.438.2

Due to the limited rebuttal period and computational resources, BAPO* results are from a single run, whereas BAPO results are averaged over three runs as in the paper.

Both variants remain clearly above all baselines (e.g., GiGPO 86.7 on ALFWorld and DAPO 28.8 on Terminal-Bench 2.1). BAPO* improves Terminal-Bench 2.1 by 4.9 points, while BAPO remains 2.4 points higher on ALFWorld. This suggests that the longer the task horizon, the more it benefits from identifying critical states at a finer granularity. We will include BAPO$^*$ and this analysis in the revision. In future work, we plan to study this trade-off more systematically, for example by adapting the number of credited intervals to the horizon and difficulty of each task.

W2 & Q2. The comparison between behavioral and fixed-length segmentation is conducted only for inference on ScienceWorld. Does behavioral segmentation also improve credit assignment during training?

Response: We thank the reviewer for this valuable suggestion. The comparison in Figure 2 (left) isolates the effect of boundary placement on context management. Motivated by your comment, we conducted the corresponding training-time ablation: we replace the JS boundaries with fixed-length segmentation (one interval every k=5k=5 steps), while keeping the contrastive scoring, the redistribution, α\alpha and the training budget identical to BAPO.

MethodALFWorld (Qwen2.5-1.5B)Terminal-Bench 2.1 (Qwen3.5-9B)
GRPO (no redistribution)72.828.1
Fixed-length segmentation + contrastive scoring87.331.5
BAPO (behavioral segmentation + contrastive scoring)90.833.3

Behavioral segmentation outperforms fixed-length segmentation by 3.5 points on ALFWorld and 1.8 points on Terminal-Bench 2.1. Fixed-length intervals often cut through a behavioral phase or merge two phases into one, so the interval score mixes success-associated and unrelated behavior and the redistributed credit becomes less targeted.

AuthorsOpenReview ↗

Response to Reviewer KNYK (Part 2/2)

Q3. Could the authors use model assessments of completed trajectories to examine whether the detected boundaries correspond to meaningful task stages and whether higher-scoring segments contain behaviors critical to task success?

Response: We are grateful for this constructive suggestion. Following it, we used Qwen3.8-Max as a judge to assess all completed training trajectories. Without seeing BAPO's boundaries or scores, the judge received the task, the final outcome and the raw trajectory, and was asked to (i) mark the boundaries between task stages and (ii) identify the step(s) most critical to the outcome. We then checked whether the interval credited by BAPO contains a step that the judge identified as critical.

The credited interval matches the judge's annotation in 78% of the trajectories compared with 51% for a randomly chosen interval. This confirms that the higher-scoring intervals identified by BAPO indeed contain the behaviors critical to task success, and, together with the case study below, that the behavioral boundaries delimit coherent task stages. This is consistent with the design of the contrastive score in Eq. (4): behaviors that both successful and failed rollouts perform, such as running tests or submitting, are down-weighted, so credit concentrates on the behaviors that actually distinguish success from failure.

In the revision, we will also provide a detailed case study in the appendix. In a representative terminal task, the agent must fix a log-summary tool so that it counts errors over all logs, including gzip-archived ones. BAPO splits the rollout into three stages: exploration, test-driven fixing of the parser, and inspection of the archived logs followed by the final fix. The credited interval contains the archive inspection, which reveals a timestamp format not covered by the unit tests. This is the behavior on which the task hinges: it appears in all successful rollouts of the group and in none of the failed ones. The appendix traces this rollout step by step, from the raw bash commands to the behavioral descriptors, boundaries, interval scores and resulting advantages.

Thank you to the authors for their response, which addresses my concerns. I will keep my score unchanged.

AuthorsOpenReview ↗

Sincere Appreciation for Your Review and Guidance

Dear Reviewer,

We sincerely appreciate your thoughtful feedback and continued support of our work. We are glad that our additional experiments and clarifications have addressed your main concerns. Thank you again for your time, insight, and contributions to the review process.

Best regards,

All authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.