- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes Action Echo Policy Optimization (AEPO), a critic-free reinforcement learning method for multi-turn LLM agents. It addresses two limitations of observation-based stepwise credit assignment: observation aliasing across different action contexts and temporal redundancy in inefficient trajectories. AEPO combines Action-Conditioned Spatial Disambiguation (ACSD), which uses preceding-action similarity to construct a soft weighted baseline within each observation group, and Difficulty-Aware Echo Regularization (DAER), which adjusts trajectory-level advantages using arrival delays relative to successful rollouts and scales the adjustment by the group success rate. The two advantages are combined for policy optimization. Experiments on ALFWorld, WebShop, and Search-R1 report improvements over GRPO, GiGPO, RAGEN, and ProxMO with limited additional advantage-computation cost.
Critical strengths
-
Well-motivated and complementary design: The paper identifies concrete spatial and temporal limitations of observation-based credit assignment and supports them with empirical analyses. ACSD and DAER form a complementary design that addresses step-level observation ambiguity and trajectory-level inefficiency, respectively.
-
Efficient and empirically supported: AEPO remains critic-free and introduces limited additional advantage-computation cost. It reports consistent improvements over several RL baselines across ALFWorld, WebShop, and Search-R1, supported by ablations of both components and analyses of key hyperparameters, action encoders, and computational efficiency.
Critical weaknesses
-
Potential inconsistency between ACSD and DAER: ACSD is motivated by the observation that the current observation alone is insufficient to identify the underlying task state, because identical observations may arise from different preceding actions and historical contexts. However, DAER computes the earliest successful arrival time using state keys without conditioning on preceding actions, task progress, or solution strategy. In tasks with state revisitation or multiple valid solution modes, the same state key may occur at different semantic stages. A later arrival may therefore reflect a different valid strategy rather than temporal redundancy. It is unclear why historical context is necessary for state comparison in ACSD but can be ignored when comparing arrival times in DAER.
-
Coarse trajectory-level penalization of localized temporal errors: Although DAER is motivated by historical redundancy, its delay penalty is computed at the trajectory level and consequently affects every action in the trajectory. In a rollout containing a few early repetitive actions followed by an optimal recovery sequence, the same negative trajectory-level adjustment is applied not only to the actions that caused the delay, but also to the necessary and correct recovery actions. Therefore, DAER identifies that a trajectory is inefficient without identifying which actions are responsible for the inefficiency. This coarse assignment may suppress useful recovery behavior after an error.
Questions for authors
In addition to the issues I mentioned under "Critical Weaknesses," I also have the following questions:
-
Why is ACSD conditioned only on the immediately preceding action? ACSD is motivated by the observation that the current observation alone may be insufficient to identify the underlying task state. However, the method augments it only with the immediately preceding action . Could the authors clarify why this single action is considered a sufficient contextual cue? Have they compared it with richer alternatives, such as the preceding observation–action pair , the last interaction steps, or an embedding of the full interaction history? Such comparisons would help establish whether the gains arise specifically from the immediately preceding action or more generally from adding historical context.
-
Does ACSD weaken step-level credit after rare exploratory transitions? When the immediately preceding action is rare and has no similar counterparts in the observation group, the target step's self-weight may approach one, causing the ACSD advantage for the current action to approach zero. This does not directly suppress the rare preceding action itself, whose advantage is computed at the previous step, but it may remove fine-grained step-level credit from actions that follow rare exploratory transitions.
5 comments
Response to Reviewer 6mEv (2/4)
Response to Weakness 2: Trajectory-Level Delay Penalty and Error Recovery
Response. It can. Because DAER applies a single trajectory-level adjustment across all steps in a rollout, delay accumulated during early mistakes reduces the trajectory-level component assigned to a later action that correctly advances the task. In the combined objective (Eq. 10), however, a productive recovery action at a post-error state still receives step-level discrimination through relative to rollouts that remain stuck. To test the net behavioral effect directly, we evaluated trained policies from identical post-error states and against trajectory-length penalty baselines.
(1) Recovery from shared error histories. We selected 2,048 interaction histories containing early repetitions and resumed both the ACSD-only and AEPO (ACSD + DAER) policies from the exact same environment states, measuring whether the immediate next action advanced the task and whether the continuation ultimately completed it:
Qwen2.5-1.5B-Instruct
| Method | ALFWorld next-step recovery | ALFWorld final completion | WebShop next-step recovery | WebShop final completion |
|---|---|---|---|---|
| ACSD only | 73.44 | 68.75 | 60.94 | 50.00 |
| AEPO (ACSD + DAER) | 75.00 | 70.31 | 61.72 | 50.78 |
Qwen2.5-7B-Instruct
| Method | ALFWorld next-step recovery | ALFWorld final completion | WebShop next-step recovery | WebShop final completion |
|---|---|---|---|---|
| ACSD only | 87.50 | 81.25 | 74.22 | 63.28 |
| AEPO (ACSD + DAER) | 87.50 | 81.25 | 73.44 | 62.50 |
At 1.5B, DAER increases both next-step recovery and final completion by 1.56 points on ALFWorld and 0.78 points on WebShop. At 7B, both rates are unchanged on ALFWorld and 0.78 points lower on WebShop. Across all four settings, the largest difference in either direction is 1.56 points, so DAER does not materially impair post-error recovery.
(2) Efficiency controls. Holding ACSD fixed across three random seeds under the same validation budget for selecting penalty weights, we compared DAER against a step-1 linear length penalty and a DAPO-style soft overlong penalty active in the final 20% of the step budget (reporting overall success rate and mean steps and repetitions per 100 steps on successful episodes):
Qwen2.5-1.5B-Instruct
| Method | ALFWorld (All) | ALF Succ. Steps | ALF Rep./100 | WebShop (Succ.) | WS Succ. Steps | WS Rep./100 |
|---|---|---|---|---|---|---|
| ACSD only | 91.41 ± 0.78 | 20.80 | 18.75 | 67.19 ± 4.13 | 11.20 | 5.78 |
| ACSD + linear penalty | 90.36 ± 0.45 | 20.45 | 18.31 | 65.89 ± 6.64 | 10.88 | 6.34 |
| ACSD + soft overlong penalty | 90.89 ± 0.90 | 20.69 | 18.25 | 66.93 ± 3.93 | 11.17 | 5.82 |
| AEPO (ACSD + DAER) | 92.97 ± 1.35 | 19.78 | 12.69 | 71.88 ± 1.96 | 10.50 | 4.80 |
Qwen2.5-7B-Instruct
| Method | ALFWorld (All) | ALF Succ. Steps | ALF Rep./100 | WebShop (Succ.) | WS Succ. Steps | WS Rep./100 |
|---|---|---|---|---|---|---|
| ACSD only | 93.75 ± 3.41 | 19.10 | 17.47 | 76.82 ± 0.90 | 10.30 | 7.10 |
| ACSD + linear penalty | 92.45 ± 0.90 | 18.83 | 17.11 | 76.04 ± 2.26 | 9.91 | 7.72 |
| ACSD + soft overlong penalty | 93.23 ± 1.97 | 19.05 | 17.05 | 76.56 ± 2.07 | 10.12 | 7.34 |
| AEPO (ACSD + DAER) | 95.31 ± 1.36 | 18.60 | 11.90 | 80.47 ± 2.86 | 9.60 | 5.64 |
Across all four settings, DAER achieves the highest task success, the shortest successful episodes, and the lowest repetition rate (18.75 to 12.69 and 17.47 to 11.90 on ALFWorld; 5.78 to 4.80 and 7.10 to 5.64 on WebShop at 1.5B/7B), whereas both direct length penalties reduce success relative to ACSD alone. Together with the shared-history continuations above, this shows that DAER trims repetitive loops without suppressing recovery after early errors.
Revision. We will clarify that DAER applies its delay adjustment at the trajectory level while ACSD preserves step-level differentiation, detail the shared-error-history protocol and both recovery metrics, and report the recovery and length-penalty comparisons.
Response to Reviewer 6mEv (3/4)
Response to Question 1: Preceding Action vs. Richer Interaction Histories in ACSD
Response. ACSD conditions on the immediately preceding action (PA) because PA identifies the transition operation that produced the current observation while keeping the cross-trajectory reference pool sufficiently large; the policy itself retains its full interaction history. In the POMDP formulation (Section 2.1), the preceding observation (PO) informs the prior belief, whereas PA selects the transition operator applied to that belief:
Even under the same prior and current observation, distinct actions induce different posteriors, and Proposition 1 (Appendix A) shows that conditioning the population baseline on PA reduces return residual variance whenever expected returns differ across preceding actions. For instance, on a WebShop rug page with size and color already selected, one rollout clicks runner to select the missing shape while another re-clicks the already selected size; the page text before and after the click is identical, yet clicking buy now next yields episode scores of 10 and 0, which PA separates even when both observations are identical.
(1) History-context ablation. Holding DAER, the policy history window, observation grouping, TF-IDF encoder, and temperature fixed on Qwen2.5-7B-Instruct (averaging per-field L2-normalized cosine similarities at matching temporal positions for multi-field contexts; GiGPO and AEPO from Table 1, remaining rows over three sets of 128 evaluation episodes):
Qwen2.5-7B-Instruct
| Method | ALFWorld (All) | WebShop (Succ.) |
|---|---|---|
| GiGPO (no context) | 92.19 ± 3.80 | 75.78 ± 3.83 |
| AEPO (PA + DAER) | 95.31 ± 1.36 | 80.47 ± 2.86 |
| PO + DAER | 93.75 ± 1.56 | 78.91 ± 1.56 |
| One observation-action pair + DAER | 95.05 ± 1.97 | 80.21 ± 1.19 |
| Last two observation-action pairs + DAER | 91.67 ± 2.96 | 75.26 ± 2.39 |
PA outperforms PO by 1.56 points on both benchmarks. Adding PO to PA (one observation-action pair) lowers mean success by only 0.26 points on each benchmark, whereas extending the context to the last two observation-action pairs reduces mean success relative to PA by 3.64 points on ALFWorld and 5.21 points on WebShop. Moreover, under the same TF-IDF tokenizer, PA and PO average 2.83 vs. 7.43 tokens on ALFWorld (2.6x longer for PO) and 5.22 vs. 111.30 tokens on WebShop (21.3x longer due to product-page text). A learned embedding over the full interaction history remains an open comparison for future work.
(2) Effect on the reference pool. Across 12,392 target steps, we measured effective sample size (, including the target visit), the share of steps with self-weight above 90%, and mean total weight on other trajectories on ALFWorld (ALF) and WebShop (WS):
| Context | ALF ESS | ALF self-weight (>90%) | ALF cross-traj. weight | WS ESS | WS self-weight (>90%) | WS cross-traj. weight |
|---|---|---|---|---|---|---|
| PA (ours) | 17.82 | 4.00% | 58.95% | 4.85 | 13.79% | 52.79% |
| PO | 13.73 | 12.14% | 57.25% | 5.05 | 13.49% | 46.12% |
| One observation-action pair | 10.39 | 17.83% | 49.51% | 4.48 | 32.84% | 39.04% |
| Last two observation-action pairs | 5.74 | 29.68% | 34.68% | 3.28 | 35.22% | 31.90% |
On the same sampled rollouts, expanding the baseline context from PA to the last two observation-action pairs lowers the mean weight contributed by other trajectories from 58.95% to 34.68% on ALFWorld and from 52.79% to 31.90% on WebShop. In both environments, more steps also rely primarily on their own return to form the baseline. Using PO alone spreads weights more broadly on WebShop, yet identical product-page text can still correspond to different selected options (as in the rug page above, where and remain identical after different option clicks). PA provides the cue to separate these visits while retaining more references from other trajectories than the two-step window. Together with the 7B task-success results above, this supports PA as an empirically effective and sample-efficient default for ACSD.
Revision. We will distinguish the policy's full history from ACSD's baseline context, note full-history learned embeddings as future work, add the 7B history-context ablation, token-length comparison, and 12,392-step weight diagnostics, and clarify the trade-off between history length and reference-pool support.
Response to Reviewer 6mEv (1/4)
We sincerely thank the reviewer for the careful reading and for recognizing the complementary roles of ACSD and DAER, the critic-free design, and the empirical support across benchmarks. We address each question below with new experiments and diagnostics, and we post our responses as four separate comments for readability.
Response to Weakness 1: Preceding-Action Baseline in ACSD vs. Observation Key in DAER
Response. Yes. Two valid strategies can reach the same observation at different stages, so a later arrival can reflect an alternative route rather than wasted steps. ACSD and DAER use their keys for different estimates. For ACSD, the reference is a step-level return baseline for the current action; Figure 1(a) shows that conditioning on the preceding action reduces within-group outcome variance at both model scales on both benchmarks. For DAER, the reference is the earliest arrival at an observation among successful rollouts for the same task. Adding the preceding action to this temporal key can separate semantic stages, but it also discards informative timing comparisons whenever two trajectories reach the same stage through different actions. In addition, within-group centering and scaling by group success p in Eq. (8) buffer shared exploration offsets and relax temporal pressure on hard multi-strategy tasks.
We directly compared the two DAER keys, ot and (ot,at−1), across 154,632 rollouts spanning model scales, task categories, and training steps. The resulting trajectory advantages had a correlation of 0.946, and the redundancy rankings agreed in 92.5% of comparisons, so adding the preceding action altered the ordering in only a small fraction of cases.
We also compared both temporal keys in end-to-end training while holding ACSD and all remaining settings fixed (reporting success rate across three random seeds, mean steps over all episodes and successful episodes separately, and repeated observation-action pairs per 100 steps in successful episodes):
Qwen2.5-1.5B-Instruct
Qwen2.5-7B-Instruct
Adding the preceding action to DAER changes mean task success by at most 0.52 points across all four settings, yields a slightly higher repetition rate in each setting, and increases advantage-computation cost by 6.2%. These results support using the observation key ot as the default temporal reference in the evaluated environments.
Revision. We will define both temporal keys, distinguish the step-level return baseline in ACSD from the trajectory-level arrival-time reference in DAER, clarify how valid multi-strategy revisits interact with measured delay and group centering, and report the ranking, training, and runtime comparisons.