- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The paper proposes ACRE, a method for constructing process credit in multi-turn interactive tasks to supplement trajectory-level outcome supervision with local action signals. Building on GiGPO, ACRE uses a rubric library to distinguish action categories, applies a frozen LLM to score actions according to these rubrics, groups actions sharing the same environment state to calculate local advantages, and combines them with trajectory-level advantages. The method is evaluated on ALFWorld and WebShop.
Critical strengths
- The research problem is valuable. Long-horizon cumulative rewards can make local action evaluation depend heavily on subsequent decisions. ACRE instead evaluates an action from its pre-decision context and the action itself, aiming to better reflect its local quality.
- The experiments cover two environments and multiple model scales. ACRE consistently improves over the GiGPO baseline across these settings.
- The paper is clearly written and easy to understand.
Critical weaknesses
-
GiGPO uses only environment rewards, whereas ACRE introduces additional supervision through rubrics and LLM scoring. It is therefore unclear whether the gains come from the proposed credit-assignment design or simply from access to an additional semantic supervision signal.
-
ACRE aims to avoid interference from later actions when assigning local credit. BEACON addresses a closely related credit-assignment problem, so a comparison would help clarify ACRE's empirical contribution.
-
Grouping actions by identical pre-decision states may limit the method's applicability when states rarely repeat.
-
The accumulated-rubric-score ablation may change the numerical scale of the learning signal. For the ablation that provides the entire trajectory to the rubric scorer, the performance decrease may also reflect a mismatch with rubrics designed specifically for local action evaluation, rather than showing that future information is inherently harmful.
Questions for authors
- How much of ACRE's improvement comes from the proposed credit-assignment mechanism rather than from introducing additional LLM-based supervision beyond environment rewards?
- How does ACRE compare with BEACON under a matched setting?
- How often do repeated pre-decision states occur, and how does this affect applicability outside the evaluated environments?
- How are the rubrics generated: manually, by an LLM, or through a combination of the two?
8 comments
W2/Q2. Comparison with BEACON
W2: ACRE aims to avoid interference from later actions when assigning local credit. BEACON addresses a closely related credit-assignment problem, so a comparison would help clarify ACRE's empirical contribution.
Q2: How does ACRE compare with BEACON under a matched setting?
BEACON assigns credit to trajectory segments delimited by subgoal milestones, whereas ACRE constructs action-level process advantages from the quality of each action in its pre-decision context. We evaluate our BEACON reproduction using ACRE's protocol and include BEACON's published results (Tables 1 and 3) for reference.
WebShop
| Model | Method | Score (%) | SR (%) |
|---|---|---|---|
| 1.5B | ACRE | 87.18 ± 1.17 | 78.84 ± 2.60 |
| 1.5B | BEACON (published) | 86.1 | 75.6 |
| 1.5B | BEACON (our reproduction) | 87.60 ± 1.58 | 77.08 ± 3.72 |
| 3B | ACRE | 89.29 ± 1.66 | 81.97 ± 2.93 |
| 3B | BEACON (our reproduction) | 85.29 ± 2.45 | 77.15 ± 3.90 |
| 7B | ACRE | 91.84 ± 1.49 | 85.48 ± 2.76 |
| 7B | BEACON (published) | 87.7 | 79.7 |
| 7B | BEACON (our reproduction) | 88.35 ± 1.25 | 80.47 ± 1.98 |
ALFWorld
| Model | Method | Pick | Look | Clean | Heat | Cool | Pick2 | All |
|---|---|---|---|---|---|---|---|---|
| 1.5B | ACRE | 97.26 ± 1.26 | 86.84 ± 6.20 | 92.50 ± 5.19 | 92.11 ± 8.57 | 88.30 ± 6.18 | 80.25 ± 1.81 | 90.36 ± 2.15 |
| 1.5B | BEACON (published) | 100.0 | 88.2 | 86.7 | 100.0 | 78.9 | 92.9 | 91.4 |
| 1.5B | BEACON (our reproduction) | 94.48 ± 7.68 | 78.15 ± 9.15 | 98.69 ± 1.21 | 94.41 ± 3.90 | 82.52 ± 6.26 | 80.79 ± 4.01 | 89.06 ± 2.82 |
| 3B | ACRE | 97.53 ± 2.30 | 88.32 ± 5.83 | 91.40 ± 6.29 | 81.25 ± 9.22 | 94.29 ± 1.01 | 73.80 ± 11.03 | 89.19 ± 0.81 |
| 3B | BEACON (our reproduction) | 96.72 ± 3.00 | 94.17 ± 3.00 | 87.16 ± 3.88 | 86.11 ± 8.70 | 62.15 ± 10.31 | 65.87 ± 6.01 | 81.51 ± 3.18 |
| 7B | ACRE | 100.00 ± 0.00 | 100.00 ± 0.00 | 100.00 ± 0.00 | 96.70 ± 3.50 | 94.44 ± 0.86 | 78.36 ± 3.20 | 94.66 ± 0.81 |
| 7B | BEACON (published) | 100.0 | 81.8 | 96.3 | 92.9 | 94.7 | 90.0 | 94.5 |
| 7B | BEACON (our reproduction) | 99.49 ± 0.87 | 98.33 ± 2.89 | 99.48 ± 0.90 | 100.00 ± 0.00 | 99.33 ± 1.15 | 95.71 ± 3.44 | 98.70 ± 0.60 |
Published BEACON results are included for reference only; the direct comparison is based on our reproduction under ACRE's evaluation protocol.
Compared with our BEACON reproduction, ACRE achieves higher SR on WebShop at all three scales and on ALFWorld at 1.5B/3B, with particularly large gains on the 3B Cool and Pick2 tasks. BEACON is stronger on ALFWorld 7B and slightly higher on WebShop 1.5B Score. These results support ACRE as a competitive approach to fine-grained credit assignment.
W3/Q3. State Recurrence, Coverage, and Alternative Grouping
W3: Grouping actions by identical pre-decision states may limit the method's applicability when states rarely repeat.
Q3: How often do repeated pre-decision states occur, and how does this affect applicability outside the evaluated environments?
We measured recurrence within the rollout groups used during WebShop training. After excluding each trajectory's first action, the table reports the fraction of scored actions whose state appears in at least two distinct trajectories:
| Model | Scored actions in recurring state groups |
|---|---|
| 1.5B | 72.64% |
| 3B | 74.35% |
| 7B | 82.43% |
Without a matching state, ACRE's current rule produces zero process advantage and retains episode-level supervision. A possible extension is to group task-relevant, similar states, allowing local comparisons without exact state matches.
To examine whether the credit signal can be used with a different grouping rule, we combine ACRE with HGPO[1], which constructs hierarchical groups based on increasingly longer shared histories and adaptively aggregates their advantages.. We retain its hierarchy and combine environment-return advantages with process advantages derived from immediate rubric scores.
WebShop results:
| Method | 1.5B Score (%) | 1.5B SR (%) | 3B Score (%) | 3B SR (%) | 7B Score (%) | 7B SR (%) |
|---|---|---|---|---|---|---|
| GiGPO | 85.11 ± 2.10 | 76.69 ± 3.54 | 87.35 ± 2.10 | 80.08 ± 2.56 | 87.89 ± 2.07 | 78.97 ± 4.01 |
| HGPO (our reproduction) | 82.76 ± 2.35 | 74.35 ± 2.44 | 88.35 ± 1.81 | 81.18 ± 3.28 | 88.84 ± 1.61 | 81.32 ± 3.03 |
| HGPO + ACRE | 87.12 ± 1.82 | 77.86 ± 2.65 | 90.67 ± 2.17 | 84.77 ± 2.82 | 92.10 ± 1.93 | 85.55 ± 3.07 |
| ACRE | 87.18 ± 1.17 | 78.84 ± 2.60 | 89.29 ± 1.66 | 81.97 ± 2.93 | 91.84 ± 1.49 | 85.48 ± 2.76 |
HGPO + ACRE achieves higher Score and SR than HGPO at all three scales. These results support grouping and credit-signal design as orthogonal choices: grouping determines which decisions are compared, while ACRE uses immediate action quality to assign local credit.
[1] He, S., Feng, L., Cheng, X., Feng, L., & An, B. (2026, April). Hierarchy-of-groups policy optimization for long-horizon agentic tasks. In International Conference on Learning Representations (Vol. 2026, pp. 27572-27593).
W1/Q1. Semantic Supervision versus Credit Assignment
To isolate the contribution of credit assignment, we hold the Judge and scoring prompt fixed within each comparison and vary how the resulting action-quality scores enter policy optimization. We use the same frozen Qwen3.5-35B-A3B Judge with a General prompt requesting an action-quality score or a Criteria-guided prompt that additionally specifies evaluation dimensions.
For each prompt, we compare two ways of using this supervision:
Additional semantic supervision is useful: with the General prompt, adding the mean Judge score to the episode reward improves SR over GRPO by 2.02/3.06/2.54 points at 1.5B/3B/7B. The controlled comparisons further reveal the contribution of process credit assignment: under the same Judge and prompt, constructing process advantages further improves SR by 0.20/0.58/1.89 points with the General prompt and 1.24/1.75/5.99 points with the Criteria-guided prompt.
The controlled comparisons support contributions from both semantic supervision and using that supervision to construct action-level process advantages.