acceptodds
Under review as a conference paper at ICLR 2027

Action Context Grouping and Echo Penalization for Agent Policy Optimization

Abstract

Stepwise group relative reinforcement learning has emerged as a computationally efficient paradigm for training Large Language Model (LLM) agents in interactive environments with multiple turns. However, relying solely on environmental observations for state grouping introduces severe credit assignment biases. In this paper, we identify two fundamental optimization bottlenecks in this paradigm centered on observations: (1) spatial degeneracy, where ambiguous observations confound structurally different states, and (2) temporal blindness, where discounted future returns do not explicitly quantify historical redundancy associated with repetitive behavioral loops (the “Echo Trap”). To overcome these limitations without triggering the sample sparsity inherent in dense history matching, we propose Action Echo Policy Optimization (AEPO). At the step level, AEPO introduces Action Conditioned Spatial Disambiguation (ACSD), constructing a soft weighted baseline via action text similarities to precisely isolate spatial noise. At the trajectory level, AEPO incorporates Difficulty Aware Echo Regularization (DAER), a dynamically calibrated penalty that squeezes out trajectory redundancy while preserving necessary exploration. Under matched training and evaluation settings, AEPO improves overall success rates over GRPO, GiGPO, RAGEN (StarPO-S), and ProxMO on ALFWorld and WebShop. With a 1.5B model, AEPO achieves a 93.0% ALFWorld success rate and adds 5.60% to advantage-computation time relative to GiGPO. Additional Search-R1 experiments extend the evaluation to multi-turn retrieval with free-form queries.

Primary area:
reinforcement learning
Keywords:
reinforcement learning in agents, planning in agents

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_6mEvOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Action Echo Policy Optimization (AEPO), a critic-free reinforcement learning method for multi-turn LLM agents. It addresses two limitations of observation-based stepwise credit assignment: observation aliasing across different action contexts and temporal redundancy in inefficient trajectories. AEPO combines Action-Conditioned Spatial Disambiguation (ACSD), which uses preceding-action similarity to construct a soft weighted baseline within each observation group, and Difficulty-Aware Echo Regularization (DAER), which adjusts trajectory-level advantages using arrival delays relative to successful rollouts and scales the adjustment by the group success rate. The two advantages are combined for policy optimization. Experiments on ALFWorld, WebShop, and Search-R1 report improvements over GRPO, GiGPO, RAGEN, and ProxMO with limited additional advantage-computation cost.

Critical strengths
  1. Well-motivated and complementary design: The paper identifies concrete spatial and temporal limitations of observation-based credit assignment and supports them with empirical analyses. ACSD and DAER form a complementary design that addresses step-level observation ambiguity and trajectory-level inefficiency, respectively.

  2. Efficient and empirically supported: AEPO remains critic-free and introduces limited additional advantage-computation cost. It reports consistent improvements over several RL baselines across ALFWorld, WebShop, and Search-R1, supported by ablations of both components and analyses of key hyperparameters, action encoders, and computational efficiency.

Critical weaknesses
  1. Potential inconsistency between ACSD and DAER: ACSD is motivated by the observation that the current observation alone is insufficient to identify the underlying task state, because identical observations may arise from different preceding actions and historical contexts. However, DAER computes the earliest successful arrival time tmin⁡(z)t_{\min}(z) using state keys without conditioning on preceding actions, task progress, or solution strategy. In tasks with state revisitation or multiple valid solution modes, the same state key may occur at different semantic stages. A later arrival may therefore reflect a different valid strategy rather than temporal redundancy. It is unclear why historical context is necessary for state comparison in ACSD but can be ignored when comparing arrival times in DAER.

  2. Coarse trajectory-level penalization of localized temporal errors: Although DAER is motivated by historical redundancy, its delay penalty is computed at the trajectory level and consequently affects every action in the trajectory. In a rollout containing a few early repetitive actions followed by an optimal recovery sequence, the same negative trajectory-level adjustment is applied not only to the actions that caused the delay, but also to the necessary and correct recovery actions. Therefore, DAER identifies that a trajectory is inefficient without identifying which actions are responsible for the inefficiency. This coarse assignment may suppress useful recovery behavior after an error.

Questions for authors

In addition to the issues I mentioned under "Critical Weaknesses," I also have the following questions:

  1. Why is ACSD conditioned only on the immediately preceding action? ACSD is motivated by the observation that the current observation alone may be insufficient to identify the underlying task state. However, the method augments it only with the immediately preceding action at−1a_{t-1}. Could the authors clarify why this single action is considered a sufficient contextual cue? Have they compared it with richer alternatives, such as the preceding observation–action pair (ot−1,at−1)(o_{t-1},a_{t-1}), the last kk interaction steps, or an embedding of the full interaction history? Such comparisons would help establish whether the gains arise specifically from the immediately preceding action or more generally from adding historical context.

  2. Does ACSD weaken step-level credit after rare exploratory transitions? When the immediately preceding action at−1a_{t-1} is rare and has no similar counterparts in the observation group, the target step's self-weight may approach one, causing the ACSD advantage for the current action ata_t to approach zero. This does not directly suppress the rare preceding action itself, whose advantage is computed at the previous step, but it may remove fine-grained step-level credit from actions that follow rare exploratory transitions.

5 comments
AuthorsOpenReview ↗

Response to Reviewer 6mEv (1/4)

We sincerely thank the reviewer for the careful reading and for recognizing the complementary roles of ACSD and DAER, the critic-free design, and the empirical support across benchmarks. We address each question below with new experiments and diagnostics, and we post our responses as four separate comments for readability.

Response to Weakness 1: Preceding-Action Baseline in ACSD vs. Observation Key in DAER

Response. Yes. Two valid strategies can reach the same observation at different stages, so a later arrival can reflect an alternative route rather than wasted steps. ACSD and DAER use their keys for different estimates. For ACSD, the reference is a step-level return baseline for the current action; Figure 1(a) shows that conditioning on the preceding action reduces within-group outcome variance at both model scales on both benchmarks. For DAER, the reference is the earliest arrival at an observation among successful rollouts for the same task. Adding the preceding action to this temporal key can separate semantic stages, but it also discards informative timing comparisons whenever two trajectories reach the same stage through different actions. In addition, within-group centering and scaling by group success pp in Eq. (8) buffer shared exploration offsets and relax temporal pressure on hard multi-strategy tasks.

We directly compared the two DAER keys, oto_t and (ot,at−1)(o_t,a_{t-1}), across 154,632 rollouts spanning model scales, task categories, and training steps. The resulting trajectory advantages had a correlation of 0.946, and the redundancy rankings agreed in 92.5% of comparisons, so adding the preceding action altered the ordering in only a small fraction of cases.

We also compared both temporal keys in end-to-end training while holding ACSD and all remaining settings fixed (reporting success rate across three random seeds, mean steps over all episodes and successful episodes separately, and repeated observation-action pairs per 100 steps in successful episodes):

Qwen2.5-1.5B-Instruct

DAER keyALFWorld (All)ALF All StepsALF Succ. StepsALF Rep./100WebShop (Succ.)WS All StepsWS Succ. StepsWS Rep./100
oto_t (AEPO, ours)92.97 ± 1.3521.9119.7812.6971.88 ± 1.9613.0210.504.80
(ot,at−1)(o_t,a_{t-1})92.71 ± 1.6322.0219.8213.0172.14 ± 1.9713.1610.684.94

Qwen2.5-7B-Instruct

DAER keyALFWorld (All)ALF All StepsALF Succ. StepsALF Rep./100WebShop (Succ.)WS All StepsWS Succ. StepsWS Rep./100
oto_t (AEPO, ours)95.31 ± 1.3620.0718.6011.9080.47 ± 2.8611.639.605.64
(ot,at−1)(o_t,a_{t-1})94.79 ± 1.6320.1818.5412.5280.47 ± 2.0711.659.576.02

Adding the preceding action to DAER changes mean task success by at most 0.52 points across all four settings, yields a slightly higher repetition rate in each setting, and increases advantage-computation cost by 6.2%. These results support using the observation key oto_t as the default temporal reference in the evaluated environments.

Revision. We will define both temporal keys, distinguish the step-level return baseline in ACSD from the trajectory-level arrival-time reference in DAER, clarify how valid multi-strategy revisits interact with measured delay and group centering, and report the ranking, training, and runtime comparisons.

AuthorsOpenReview ↗

Response to Reviewer 6mEv (2/4)

Response to Weakness 2: Trajectory-Level Delay Penalty and Error Recovery

Response. It can. Because DAER applies a single trajectory-level adjustment across all steps in a rollout, delay accumulated during early mistakes reduces the trajectory-level component assigned to a later action that correctly advances the task. In the combined objective (Eq. 10), however, a productive recovery action at a post-error state still receives step-level discrimination through ωAACSD(at)\omega A_{\mathrm{ACSD}}(a_t) relative to rollouts that remain stuck. To test the net behavioral effect directly, we evaluated trained policies from identical post-error states and against trajectory-length penalty baselines.

(1) Recovery from shared error histories. We selected 2,048 interaction histories containing early repetitions and resumed both the ACSD-only and AEPO (ACSD + DAER) policies from the exact same environment states, measuring whether the immediate next action advanced the task and whether the continuation ultimately completed it:

Qwen2.5-1.5B-Instruct

MethodALFWorld next-step recoveryALFWorld final completionWebShop next-step recoveryWebShop final completion
ACSD only73.4468.7560.9450.00
AEPO (ACSD + DAER)75.0070.3161.7250.78

Qwen2.5-7B-Instruct

MethodALFWorld next-step recoveryALFWorld final completionWebShop next-step recoveryWebShop final completion
ACSD only87.5081.2574.2263.28
AEPO (ACSD + DAER)87.5081.2573.4462.50

At 1.5B, DAER increases both next-step recovery and final completion by 1.56 points on ALFWorld and 0.78 points on WebShop. At 7B, both rates are unchanged on ALFWorld and 0.78 points lower on WebShop. Across all four settings, the largest difference in either direction is 1.56 points, so DAER does not materially impair post-error recovery.

(2) Efficiency controls. Holding ACSD fixed across three random seeds under the same validation budget for selecting penalty weights, we compared DAER against a step-1 linear length penalty and a DAPO-style soft overlong penalty active in the final 20% of the step budget (reporting overall success rate and mean steps and repetitions per 100 steps on successful episodes):

Qwen2.5-1.5B-Instruct

MethodALFWorld (All)ALF Succ. StepsALF Rep./100WebShop (Succ.)WS Succ. StepsWS Rep./100
ACSD only91.41 ± 0.7820.8018.7567.19 ± 4.1311.205.78
ACSD + linear penalty90.36 ± 0.4520.4518.3165.89 ± 6.6410.886.34
ACSD + soft overlong penalty90.89 ± 0.9020.6918.2566.93 ± 3.9311.175.82
AEPO (ACSD + DAER)92.97 ± 1.3519.7812.6971.88 ± 1.9610.504.80

Qwen2.5-7B-Instruct

MethodALFWorld (All)ALF Succ. StepsALF Rep./100WebShop (Succ.)WS Succ. StepsWS Rep./100
ACSD only93.75 ± 3.4119.1017.4776.82 ± 0.9010.307.10
ACSD + linear penalty92.45 ± 0.9018.8317.1176.04 ± 2.269.917.72
ACSD + soft overlong penalty93.23 ± 1.9719.0517.0576.56 ± 2.0710.127.34
AEPO (ACSD + DAER)95.31 ± 1.3618.6011.9080.47 ± 2.869.605.64

Across all four settings, DAER achieves the highest task success, the shortest successful episodes, and the lowest repetition rate (18.75 to 12.69 and 17.47 to 11.90 on ALFWorld; 5.78 to 4.80 and 7.10 to 5.64 on WebShop at 1.5B/7B), whereas both direct length penalties reduce success relative to ACSD alone. Together with the shared-history continuations above, this shows that DAER trims repetitive loops without suppressing recovery after early errors.

Revision. We will clarify that DAER applies its delay adjustment at the trajectory level while ACSD preserves step-level differentiation, detail the shared-error-history protocol and both recovery metrics, and report the recovery and length-penalty comparisons.

AuthorsOpenReview ↗

Response to Reviewer 6mEv (3/4)

Response to Question 1: Preceding Action vs. Richer Interaction Histories in ACSD

Response. ACSD conditions on the immediately preceding action (PA) because PA identifies the transition operation that produced the current observation while keeping the cross-trajectory reference pool sufficiently large; the policy itself retains its full interaction history. In the POMDP formulation (Section 2.1), the preceding observation (PO) informs the prior belief, whereas PA selects the transition operator applied to that belief:

bt(s)∝Z(ot∣s)∑s′T(s∣s′,at−1) bt−1(s′).b_t(s) \propto \mathcal{Z}(o_ t\mid s)\sum_ {s'}\mathcal{T}(s\mid s',a_ {t-1})\,b_ {t-1}(s').

Even under the same prior and current observation, distinct actions induce different posteriors, and Proposition 1 (Appendix A) shows that conditioning the population baseline on PA reduces return residual variance whenever expected returns differ across preceding actions. For instance, on a WebShop rug page with size and color already selected, one rollout clicks runner to select the missing shape while another re-clicks the already selected size; the page text before and after the click is identical, yet clicking buy now next yields episode scores of 10 and 0, which PA separates even when both observations are identical.

(1) History-context ablation. Holding DAER, the policy history window, observation grouping, TF-IDF encoder, and temperature fixed on Qwen2.5-7B-Instruct (averaging per-field L2-normalized cosine similarities at matching temporal positions for multi-field contexts; GiGPO and AEPO from Table 1, remaining rows over three sets of 128 evaluation episodes):

Qwen2.5-7B-Instruct

MethodALFWorld (All)WebShop (Succ.)
GiGPO (no context)92.19 ± 3.8075.78 ± 3.83
AEPO (PA + DAER)95.31 ± 1.3680.47 ± 2.86
PO + DAER93.75 ± 1.5678.91 ± 1.56
One observation-action pair + DAER95.05 ± 1.9780.21 ± 1.19
Last two observation-action pairs + DAER91.67 ± 2.9675.26 ± 2.39

PA outperforms PO by 1.56 points on both benchmarks. Adding PO to PA (one observation-action pair) lowers mean success by only 0.26 points on each benchmark, whereas extending the context to the last two observation-action pairs reduces mean success relative to PA by 3.64 points on ALFWorld and 5.21 points on WebShop. Moreover, under the same TF-IDF tokenizer, PA and PO average 2.83 vs. 7.43 tokens on ALFWorld (2.6x longer for PO) and 5.22 vs. 111.30 tokens on WebShop (21.3x longer due to product-page text). A learned embedding over the full interaction history remains an open comparison for future work.

(2) Effect on the reference pool. Across 12,392 target steps, we measured effective sample size (ESSi=1/∑jwij2\mathrm{ESS}_ i = 1 / \sum_ j w_ {ij}^2, including the target visit), the share of steps with self-weight above 90%, and mean total weight on other trajectories on ALFWorld (ALF) and WebShop (WS):

ContextALF ESSALF self-weight (>90%)ALF cross-traj. weightWS ESSWS self-weight (>90%)WS cross-traj. weight
PA (ours)17.824.00%58.95%4.8513.79%52.79%
PO13.7312.14%57.25%5.0513.49%46.12%
One observation-action pair10.3917.83%49.51%4.4832.84%39.04%
Last two observation-action pairs5.7429.68%34.68%3.2835.22%31.90%

On the same sampled rollouts, expanding the baseline context from PA to the last two observation-action pairs lowers the mean weight contributed by other trajectories from 58.95% to 34.68% on ALFWorld and from 52.79% to 31.90% on WebShop. In both environments, more steps also rely primarily on their own return to form the baseline. Using PO alone spreads weights more broadly on WebShop, yet identical product-page text can still correspond to different selected options (as in the rug page above, where ot−1o_{t-1} and oto_t remain identical after different option clicks). PA provides the cue to separate these visits while retaining more references from other trajectories than the two-step window. Together with the 7B task-success results above, this supports PA as an empirically effective and sample-efficient default for ACSD.

Revision. We will distinguish the policy's full history from ACSD's baseline context, note full-history learned embeddings as future work, add the 7B history-context ablation, token-length comparison, and 12,392-step weight diagnostics, and clarify the trade-off between history length and reference-pool support.

Reviewer_wyF6OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes AEPO, a critic-free method that improves step comparisons and trajectory-efficiency preferences using existing rollouts. ACSD weights the step-level baseline by the similarity of preceding actions, while DAER adjusts trajectory advantages according to arrival delays relative to successful rollouts and the task group's success rate. Experiments cover ALFWorld, WebShop, and additional Search-R1 tasks.

Critical strengths
  1. The main strength is the efficient reuse of existing rollouts. AEPO improves credit assignment and encourages efficient behavior without introducing a learned critic or additional branch sampling. The lightweight computations make the method a practical extension of GiGPO.
  2. The experiments show consistent overall improvements over strong baselines on ALFWorld and WebShop, with three random seeds and two model sizes. Component ablations, action-encoder comparisons, and the additional Search-R1 results provide useful support for the proposed design.
Critical weaknesses
  1. The extent to which ACSD resolves state ambiguity remains unclear. The immediately preceding action provides useful context, but its lexical similarity may not distinguish states whose relevant differences lie further back in the history. The encoder and temperature comparisons are useful, but the effect of the history-window length is not examined. The observation-matching procedure and construction of normalized state keys also need more explicit specification, since both directly affect the comparisons.
  2. DAER's arrival delay is a proxy for redundancy, but may also reflect useful exploration or alternative successful routes. The difficulty scaling and treatment of unseen states mitigate this concern, and the component ablations support the method's effectiveness. However, a comparison with a simple trajectory-length penalty is missing. Moreover, the reported average step counts include failed episodes, so they do not directly establish how efficiently successful tasks are completed.
Questions for authors
  1. Could you compare conditioning on one preceding action with a short history window, and specify the observation-matching procedure and state-key construction?
  2. How does DAER compare with a simple length penalty at a similar success–efficiency trade-off, and do the efficiency gains persist when step counts are measured only on successful episodes?
6 comments
AuthorsOpenReview ↗

Response to Reviewer wyF6 (1/3)

We sincerely thank the reviewer for recognizing that AEPO reuses existing rollouts efficiently without a learned critic or extra branch sampling, and for noting its consistent multi-seed gains, component ablations, and Search-R1 results. We address each point below and post our responses in three parts for readability.

Response to Weakness 1 and Question 1 (Part 1): Preceding Action vs. Longer History Contexts in ACSD

Response. ACSD conditions on the immediately preceding action (PA) because PA identifies the action that produced the current observation while keeping the cross-trajectory reference pool well-populated. By Proposition 1 (Appendix A), conditioning the baseline on PA lowers return residual variance whenever expected returns differ across actions leading to the same observation. Meanwhile, the policy itself retains its interaction history to track earlier state dependencies. Still, if two visits share both the current observation and PA after diverging earlier, they receive the same reference weights; we therefore tested whether a longer history window improves ACSD in practice.

(1) History-window ablation. Holding DAER and the policy's interaction history fixed, we compared PA against the preceding observation (PO), one observation-action pair, and the last two observation-action pairs on Qwen2.5-7B-Instruct (GiGPO and AEPO from Table 1; mean ± SD across three random seeds):

Qwen2.5-7B-Instruct

MethodALFWorld (All)WebShop (Succ.)
GiGPO (no context)92.19 ± 3.8075.78 ± 3.83
AEPO (PA + DAER)95.31 ± 1.3680.47 ± 2.86
PO + DAER93.75 ± 1.5678.91 ± 1.56
One observation-action pair + DAER95.05 ± 1.9780.21 ± 1.19
Last two observation-action pairs + DAER91.67 ± 2.9675.26 ± 2.39

PA achieves the highest mean success on both benchmarks and outperforms PO by 1.56 points on each. Adding PO to PA (one observation-action pair) lowers mean success by only 0.26 points on each benchmark, whereas extending the window to the last two observation-action pairs reduces mean success relative to PA by 3.64 points on ALFWorld and 5.21 points on WebShop.

(2) Effect on the reference pool. Across 12,392 target steps, we measured effective sample size (ESSi=1/∑jwij2\mathrm{ESS}_i = 1 / \sum_j w_{ij}^2, including the target visit), the share of steps with self-weight above 90%, and mean total weight on other trajectories on ALFWorld (ALF) and WebShop (WS):

ContextALF ESSALF self-weight (>90%)ALF cross-traj. weightWS ESSWS self-weight (>90%)WS cross-traj. weight
PA (ours)17.824.00%58.95%4.8513.79%52.79%
PO13.7312.14%57.25%5.0513.49%46.12%
One observation-action pair10.3917.83%49.51%4.4832.84%39.04%
Last two observation-action pairs5.7429.68%34.68%3.2835.22%31.90%

On the same sampled rollouts, expanding the baseline context from PA to the last two observation-action pairs lowers the mean weight contributed by other trajectories from 58.95% to 34.68% on ALFWorld and from 52.79% to 31.90% on WebShop. In both environments, more steps also rely primarily on their own return to form the baseline. Using PO alone spreads weights more broadly on WebShop, yet identical product-page text can still correspond to different selected options: on a rug page with size and color already chosen, clicking runner versus re-clicking the selected size leaves both ot−1o_{t-1} and oto_t identical, whereas buy now next scores 10 versus 0. PA provides the cue to separate these visits while retaining more references from other trajectories than the two-step window. Together with the 7B task-success results above, this supports PA as an empirically effective and sample-efficient default for ACSD.

Revision. We will add the 7B history-window comparison, report the 12,392-step ESS and reference-weight statistics, and clarify why PA is a sample-efficient default context for ACSD.

AuthorsOpenReview ↗

Response to Reviewer wyF6 (2/3)

Response to Weakness 1 and Question 1 (Part 2): Observation Matching and State-Key Construction

Response. Observation grouping determines which visits are compared: an overly coarse key merges distinct rooms or products, whereas matching the full prompt (with step counters and history) splits identical views into singletons. To isolate the current environment view while keeping cross-trajectory groups well-populated, we extract a normalized observation key before adding history or admissible actions to the prompt. ACSD and DAER share this key.

(1) State-key construction by benchmark.

  • ALFWorld (exact match). The key is the raw environment observation text, which retains object identifiers (mug 1, desklamp 1) and the reset task goal without prompt history or admissible-action lists. Visits match by exact string equality within each task instance.
  • WebShop (exact match). At reset, we record the task instruction from the initial " [SEP] "-delimited observation. At each step, we strip the header through that instruction and rejoin the remaining " [SEP] " page fields in single quotes, so visits to the same product page match by exact string equality at any step.
  • Search-R1 (similarity match). Because free-form queries return variable text passages, we follow GiGPO and group task-local observations with SequenceMatcher similarity ≥0.9\geq 0.9 (Appendix D).

(2) How PA weighting resolves ambiguity within a matched group. Within each group, ACSD extracts the action text from <action> tags (omitting <think> text; empty at step 0) and computes L2-normalized TF-IDF cosine similarities against every visit in the group, including the target itself. A softmax at τ=0.1\tau = 0.1 weights those reference returns to form the baseline for the current action:

  • ALFWorld example. In a lamp-inspection task, examine mug 1 and go to desklamp 1 both return Nothing happens. and share a group, but ACSD assigns much higher baseline weight to peers reached via the same preceding command.
  • WebShop example. In the rug task from (1/3), two visits share identical formatted page text before and after an option click and both then issue buy now. Page matching pools the two purchases into one group, while PA weighting gives higher weight to peers preceded by the runner click for the first purchase and reverses the ordering for the repeated size click, so each buy now action is evaluated against a matching transition.

Revision. We will add the exact key-extraction and matching specifications for all three benchmarks, note that ACSD and DAER share the same observation key, and include both examples.

AuthorsOpenReview ↗

Response to Reviewer wyF6 (3/3)

Response to Weakness 2 and Question 2: Length-Penalty Controls and Successful-Episode Efficiency in DAER

Response. Measured strictly on successful episodes, DAER retains its efficiency advantage across both benchmarks and model scales, achieving higher task success and fewer completion steps than step-level linear and soft overlong length penalties. Arrival delay is an empirical proxy for redundancy, so a longer exploratory route can incur delay if another rollout reaches the same observation earlier via a shortcut; however, successful-rollout references, group centering, and difficulty scaling limit this pressure.

(1) Exploration, alternative routes, and difficulty scaling. DAER measures delay against the earliest arrival among successful rollouts in the task group (Eq. 8). Unseen observations incur zero delay, group centering cancels exploration steps shared across rollouts on the same task, and scaling by group success pp relaxes pressure on hard tasks (extending Table 8 to 7B; mean steps over all evaluation episodes):

Qwen2.5-1.5B-Instruct

MethodALFWorld (All)ALF All StepsWebShop (Succ.)WS All Steps
AEPO w/o difficulty scaling89.5318.2669.8111.24
AEPO (full)92.9721.9171.8813.02

Qwen2.5-7B-Instruct

MethodALFWorld (All)ALF All StepsWebShop (Succ.)WS All Steps
AEPO w/o difficulty scaling91.4116.8578.9110.05
AEPO (full)95.3120.0780.4711.63

With pp-scaling, both models take more steps per episode on average while improving success by 3.44/3.90 points on ALFWorld and 2.07/1.56 points on WebShop at 1.5B/7B.

(2) Comparison with trajectory-length penalties. Whereas pp-scaling avoids truncating hard episodes prematurely, on solvable tasks DAER trims redundant steps more selectively than penalizing length directly. Holding ACSD and the validation budget for penalty tuning fixed across three seeds, we compared DAER against a step-1 linear penalty (Golubev et al. [1], Eq. 4, threshold zero) and a DAPO-style soft overlong penalty (Yu et al. [2], Eq. 13, final 20% of the step budget), reporting mean steps over all episodes (All Steps) and completed episodes (Succ. Steps):

Qwen2.5-1.5B-Instruct

MethodALFWorld (All)ALF All StepsALF Succ. StepsWebShop (Succ.)WS All StepsWS Succ. Steps
GiGPO89.06 ± 2.6224.2221.0568.75 ± 3.5314.5411.70
ACSD only91.41 ± 0.7823.3120.8067.19 ± 4.1314.4111.20
ACSD + linear penalty90.36 ± 0.4523.3020.4565.89 ± 6.6411.7810.88
ACSD + soft overlong penalty90.89 ± 0.9023.3620.6966.93 ± 3.9312.2711.17
AEPO (ACSD + DAER)92.97 ± 1.3521.9119.7871.88 ± 1.9613.0210.50

Qwen2.5-7B-Instruct

MethodALFWorld (All)ALF All StepsALF Succ. StepsWebShop (Succ.)WS All StepsWS Succ. Steps
GiGPO92.19 ± 3.8021.8819.5075.78 ± 3.8313.0310.80
ACSD only93.75 ± 3.4121.0319.1076.82 ± 0.9012.5210.30
ACSD + linear penalty92.45 ± 0.9021.1818.8376.04 ± 2.2611.259.91
ACSD + soft overlong penalty93.23 ± 1.9721.1519.0576.56 ± 2.0711.5510.12
AEPO (ACSD + DAER)95.31 ± 1.3620.0718.6080.47 ± 2.8611.639.60

Both length penalties lower success relative to ACSD alone in all four settings, since penalizing raw length can suppress useful exploration. DAER achieves the highest success rate and the fewest steps per completed episode in every setting. On WebShop, the linear penalty has fewer All Steps than DAER (11.78 vs. 13.02 at 1.5B; 11.25 vs. 11.63 at 7B) only because failed rollouts end earlier; among completed episodes the ordering reverses, with DAER averaging 10.50/9.60 Succ. Steps versus 10.88/9.91 (linear) and 11.17/10.12 (soft overlong). At comparable completion lengths on 7B ALFWorld (18.60 vs. 18.83 Succ. Steps), DAER exceeds the linear penalty by 2.86 points (95.31% vs. 92.45%; each Succ. Steps mean covers that method's completed tasks).

Revision. We will add the 7B scaling ablation and both length-penalty controls to Table 8, report all-episode and successful-episode step counts separately, and clarify when exploratory detours or alternative routes can incur delay.

References

[1] Alexander Golubev et al. Training Long-Context, Multi-Turn Software Engineering Agents with Reinforcement Learning. arXiv:2508.03501, 2025.

[2] Qiying Yu et al. DAPO: An Open-Source LLM Reinforcement Learning System at Scale. NeurIPS, 2025.

Reviewer_8mrKOpenReview ↗
Rating
2(−1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Action Echo Policy Optimization (AEPO), a framework for improving credit assignment in multi-turn LLM agent reinforcement learning. The authors identify two limitations of observation-based stepwise policy optimization: state aliasing caused by identical observations under different action contexts, and inefficient repetitive behaviors during long-horizon interactions. To address these issues, AEPO introduces Action-Conditioned Spatial Disambiguation (ACSD), which computes action-aware step-level baselines using preceding-action similarity, and Difficulty-Aware Echo Regularization (DAER), which penalizes trajectory delays according to task difficulty. Experiments on ALFWorld, WebShop, and Search-R1 demonstrate improvements over existing RL baselines.

Critical strengths

The paper addresses important and practical challenges in multi-turn agent reinforcement learning, particularly observation aliasing and inefficient repetitive behaviors.

AEPO provides a simple and intuitive framework that combines action-aware credit assignment with difficulty-adaptive trajectory regularization.

Experiments across multiple model scales and agentic benchmarks demonstrate consistent improvements over strong baselines, including GRPO, GiGPO, and ProxMO.

Critical weaknesses

ACSD relies solely on the immediately preceding action to disambiguate observations. However, in partially observable environments, the underlying state may depend on the entire interaction history. It remains unclear whether preceding-action similarity is sufficient to resolve state aliasing in more complex scenarios.

Using TF-IDF similarity to construct action-aware baselines may not reliably capture the semantic or causal relationships between actions. Lexically similar actions can have different effects on the environment, while semantically equivalent actions may be expressed differently. The robustness of this design requires further investigation.

DAER uses arrival delays relative to successful trajectories as a proxy for inefficient behavior. However, longer trajectories do not necessarily indicate redundant actions, as additional steps may be required for exploration or information gathering. The proposed penalty could therefore discourage useful exploration, especially when successful trajectories are not optimal.

Questions for authors

Why is only the immediately preceding action considered in ACSD? Would incorporating longer action histories provide more reliable state disambiguation?

How does DAER distinguish genuinely redundant actions from necessary exploration? Could the delay penalty negatively affect tasks where longer trajectories are beneficial?

Can the authors provide more direct evidence that AEPO reduces observation aliasing and repetitive action loops, beyond improvements in overall task success rates?

8 comments

I have not received a response to my questions, so my concerns about state disambiguation and potentially penalizing useful exploration remain unresolved. These issues leave me less confident in the paper’s conclusions. I am therefore revising my rating from weak accept to weak reject.

AuthorsOpenReview ↗

Response to Reviewer 8mrK (1/5)

We sincerely thank you for recognizing the importance of observation aliasing and repetitive behavior in multi-turn agent training, the simplicity of AEPO, and its consistent gains across model scales and benchmarks. We address each point below with new experiments, combining Weakness 3 and Question 2, which both concern exploration under delay regularization.

Response to Weakness 1: Preceding-Action Similarity in More Complex Environments

Response. The immediately preceding action can distinguish visits that look alike even when solving the task requires a longer history: the policy chooses each action conditioned on its available interaction history, while ACSD uses preceding-action similarity to reweight reference returns within the same current observation without fragmenting groups (analyzed in Question 1). To evaluate this design in more complex settings with multi-stage dependencies and open-ended actions, we evaluated AEPO on ScienceWorld [1] and the six-dataset Search-R1 benchmark [2] under the same training and evaluation settings.

On ScienceWorld, we report task success rate (mean ± SD across three random seeds) on seen task types and variants (L0), unseen variants of seen task types (L1), unseen task types (L2), and their equal-weight average:

Qwen2.5-1.5B-Instruct

MethodL0L1L2ScienceWorld (Avg.)
GRPO21.09 ± 1.3513.54 ± 4.306.77 ± 0.4513.80 ± 1.38
GiGPO25.78 ± 0.7815.36 ± 4.448.33 ± 0.9016.49 ± 1.57
AEPO28.13 ± 1.3517.19 ± 4.359.90 ± 0.4518.40 ± 1.17

Qwen2.5-7B-Instruct

MethodL0L1L2ScienceWorld (Avg.)
GRPO49.22 ± 2.3430.21 ± 5.0825.52 ± 1.6334.98 ± 1.95
GiGPO53.39 ± 1.1935.42 ± 4.3026.82 ± 1.1938.54 ± 1.82
AEPO55.73 ± 1.9737.50 ± 3.4128.13 ± 1.5640.45 ± 1.95

AEPO outperforms GiGPO on every ScienceWorld split at both scales. The average gain is 1.91 points at both 1.5B and 7B, including 1.57/1.31 points on unseen task types (L2).

On Search-R1, we reran all methods across six retrieval datasets (128 questions per dataset, three seeds; extending Appendix D, Table 6) and report exact-match score (mean ± SD) and their equal-weight average:

Qwen2.5-3B-Instruct

MethodNQTriviaQAPopQAHotpotQA2WikiMultiHopQAMuSiQueSearch (Avg.)
Search-R1 (GRPO)39.84 ± 3.5856.25 ± 3.5839.06 ± 2.7133.33 ± 2.9631.25 ± 3.9112.50 ± 3.1335.37 ± 1.12
GiGPO41.93 ± 5.8659.38 ± 4.3542.45 ± 4.3036.98 ± 8.2137.24 ± 7.4312.76 ± 3.1638.45 ± 2.83
AEPO43.23 ± 5.0860.68 ± 5.0843.75 ± 3.4138.28 ± 7.4539.06 ± 7.0313.80 ± 2.9639.80 ± 2.42

Qwen2.5-7B-Instruct

MethodNQTriviaQAPopQAHotpotQA2WikiMultiHopQAMuSiQueSearch (Avg.)
Search-R1 (GRPO)42.97 ± 2.0762.24 ± 5.0842.71 ± 1.1938.54 ± 5.3234.64 ± 5.4916.41 ± 2.0739.58 ± 2.73
GiGPO46.61 ± 0.4564.84 ± 0.7846.09 ± 4.3541.67 ± 4.7143.75 ± 4.7518.75 ± 4.1343.62 ± 1.93
AEPO48.44 ± 1.5666.41 ± 1.3547.66 ± 4.8843.23 ± 5.4945.31 ± 4.7520.31 ± 3.5845.23 ± 2.32

AEPO achieves the highest score on all six search datasets at both 3B and 7B, improving over GiGPO by 1.35/1.61 points on average. Even when underlying states depend on multi-stage experimental procedures (ScienceWorld) or free-form retrieval histories (Search-R1), AEPO consistently improves over observation-only grouping (GiGPO). This supports preceding-action similarity as an empirically effective and sample-efficient default context for ACSD in the evaluated environments.

Revision. We will add the ScienceWorld and six-dataset Search-R1 results and clarify how the full interaction history available to the policy complements the preceding-action context used to estimate the step-level baseline.

References

[1] Ruoyao Wang, Peter Jansen, Marc-Alexandre Côté, and Prithviraj Ammanabrolu. ScienceWorld: Is your Agent Smarter than a 5th Grader? EMNLP, 2022.

[2] Bowen Jin et al. Search-R1: Training LLMs to Reason and Leverage Search Engines with Reinforcement Learning. COLM, 2025.

AuthorsOpenReview ↗

Response to Reviewer 8mrK (2/5)

Response to Weakness 2: Semantic Action Similarity in ACSD

Response. We agree that TF-IDF similarity may not always capture semantic or causal relationships when lexically similar actions have different effects on the environment or semantically equivalent actions are expressed differently. To directly test the robustness of this design, we conducted a controlled encoder ablation using two representative dense encoders: all-MiniLM-L6-v2, a lightweight sentence embedding model, and bge-large-en-v1.5, a larger and stronger text embedding model. Because ACSD accepts any action representation, we changed only the action encoder while keeping all other training and evaluation settings fixed. Appendix E (Table 7) reports this comparison on ALFWorld and WebShop (Qwen2.5-1.5B-Instruct), and we extend it here to ScienceWorld (Qwen2.5-1.5B-Instruct) and the six-dataset Search suite (Qwen2.5-3B-Instruct), with overhead measured relative to GiGPO's advantage-computation time:

Similarity methodALFWorld (All)Adv. overheadWebShop (Succ.)Adv. overheadScienceWorld (Avg.)Adv. overheadSearch (Avg.)Adv. overhead
GiGPO (reference)89.060.00%68.750.00%16.490.00%38.450.00%
TF-IDF (ours)92.975.60%71.884.11%18.404.34%39.8015.42%
all-MiniLM-L6-v292.9437.32%72.1425.80%18.7529.28%40.02109.91%
bge-large-en-v1.593.50312.58%72.56260.42%19.27274.88%40.63978.58%

MiniLM performs nearly identically to TF-IDF across all four benchmarks, and all three encoders consistently outperform GiGPO, including on ScienceWorld and Search. Although BGE achieves the best results, it improves over TF-IDF by only 0.53, 0.68, 0.87, and 0.83 points on ALFWorld, WebShop, ScienceWorld, and Search while increasing the overhead of advantage computation from 5.60%/4.11%/4.34%/15.42% to 312.58%/260.42%/274.88%/978.58%. Thus, AEPO achieves consistent gains in more complex scenarios and its performance is robust to the encoder choice, while TF-IDF provides the best balance between accuracy and efficiency for the evaluated environments.

The action distribution helps explain this result. Action strings in the evaluated environments are mostly short and canonical, so richer semantic encoders have limited room to help:

Action subsetShareMedian wordsMax words
ALFWorld admissible actions100%46
WebShop click[...]78.75%15
WebShop search[...] (queries in free form)21.25%1529
ScienceWorld expert actions100%426
Search-R1 <search> calls (3B HotpotQA)63.69%1240
Search-R1 <answer> final answers (3B HotpotQA)36.31%218

Specifically, ALFWorld actions, ScienceWorld actions, WebShop click[...] (78.75%), and Search-R1 <answer> turns (36.31%) have medians of only 1 to 4 words, where IDF downweights shared template words within an observation group and the low softmax temperature (τ=0.1\tau = 0.1) sharpens differences in the verb or target argument. Only the free-form WebShop search[...] queries (21.25%, median 15 words) and Search-R1 <search> calls (63.69%, median 12 words) are substantially longer. Even with these longer queries included, dense encoders provide only marginal aggregate gains at substantially higher cost. This supports TF-IDF as an empirically effective and efficient default for ACSD in the evaluated environments.

Revision. We will add the ScienceWorld and Search encoder comparisons alongside Appendix E, report their computational costs and the action length statistics, and clarify that TF-IDF is an efficient default for the evaluated environments while ACSD can use stronger semantic encoders when the action space is more complex.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General response from authors

We thank all reviewers, the Area Chair, and the Senior Area Chair for their time and constructive feedback. Reviewer 8mrK noted that AEPO addresses observation aliasing and repetitive behaviors in multi-turn agent RL through a simple framework with consistent gains over GRPO, GiGPO, and ProxMO across scales and benchmarks. Reviewer wyF6 emphasized AEPO's efficient reuse of existing rollouts without a learned critic or extra branch sampling, supported by consistent multi-seed gains, ablations, and Search-R1 results. Reviewer 6mEv recognized the complementary design of step-level ACSD and trajectory-level DAER, together with AEPO's critic-free formulation and low advantage-computation overhead.

In the initial reviews, both Reviewer 8mrK and Reviewer wyF6 gave positive recommendations (initial Rating: 3, Weak acceptance; Confidence: 3). Completing all requested multi-turn RL runs and three-seed evaluations across scales and benchmarks required over 10,000 GPU-hours, delaying our responses until near the deadline; we sincerely apologize for the late posting and thank Reviewer wyF6 for following up to express understanding of this effort and confirm maintaining a positive recommendation. Following the reviewers' suggestions, we have posted point-by-point responses and incorporated all new analyses and experiments into the newly uploaded revision, with additions marked in blue:

Generalization to ScienceWorld and Full Search-R1: Added three-seed evaluations on ScienceWorld at 1.5B and 7B across seen (L0), unseen-variant (L1), and unseen-task (L2) splits, plus unified three-seed evaluations across all six Search-R1 retrieval datasets at 3B and 7B. AEPO outperforms GRPO and GiGPO on every ScienceWorld split (+1.91 average points over GiGPO at both scales) and across all six Search-R1 datasets at 3B and 7B (+1.35 and +1.61 average points over GiGPO) (Appendices B.2, D.1, and D.2; Tables 6 and 8).

Action-Encoder Robustness and Action-Length Analysis: Extended controlled ACSD ablations with all-MiniLM-L6-v2 and bge-large-en-v1.5 to ScienceWorld (1.5B) and six-dataset Search-R1 (3B), alongside action-length distributions. MiniLM matches TF-IDF and BGE adds only 0.53–0.87 points while raising advantage-computation overhead from 4.11%–15.42% to 260.42%–978.58%, confirming TF-IDF as an efficient default and showing ACSD's compatibility with denser encoders (Appendix E; Tables 9 and 10).

State-Key Specification and History-Window Analysis in ACSD: Specified the normalized observation-key rules shared by ACSD and DAER across all four benchmarks, clarified how full policy history complements preceding-action baseline conditioning, and added rollout examples of degenerate feedback, hidden option selection, and latent inventory. Controlled 7B ablations and 12,392-step weight diagnostics show that preceding-action conditioning achieves the highest success (95.31% on ALFWorld; 80.47% on WebShop), beating preceding-observation conditioning (+1.56 points on both) while avoiding the cross-trajectory support drop of two-step windows (Section 3.1; Appendices B.3, G.3, and H.1; Tables 19 and 20).

Rare Exploratory Transitions and Aliasing Dynamics in ACSD: With DAER disabled at 1.5B and 7B, continuous preceding-action softmax weighting outperforms uniform observation grouping and hard exclusion of unmatched transitions. Across seven checkpoints (steps 0–150), AEPO also maintains lower within-observation action disagreement than GiGPO at every post-initial step across scales and benchmarks (Appendices H.2 and H.3; Tables 21 and 22).

Temporal-Key Choice and Length-Penalty Controls in DAER: Comparing oto_t with (ot,at−1)(o_t, a_{t-1}) across 154,632 rollouts yields a 0.946 advantage correlation and 92.5% ranking agreement, and 1.5B/7B training confirms matching success within 0.52 points with lower repetition rates and 6.2% lower overhead. Extending difficulty scaling to 7B and comparing against tuned step-level linear and DAPO-style soft overlong penalties across three seeds confirm that raw length penalties lower success below ACSD alone, whereas DAER achieves the highest success, the fewest steps per completed episode, and 16.96%–32.32% fewer repeated steps per 100 steps (Sections 3.2 and 4.3; Appendices F, F.1, and F.2; Tables 11, 13, and 14).

Post-Error Recovery Behavior in DAER: Resuming ACSD-only and AEPO policies from 2,048 shared error histories with early repetitions at 1.5B and 7B yields next-step recovery and final completion rates within 1.56 points across all settings, confirming that DAER trims loops without impairing recovery (Appendix F.3; Table 15).

We thank all reviewers again for their constructive feedback, which has strengthened the analytical foundation, empirical validation, and clarity of our work.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.