acceptodds
Under review as a conference paper at ICLR 2027

ACRE: Action-level Credit Assignment with Rubric Evaluation for Long-Horizon Agents

Abstract

Long-horizon agentic tasks require reinforcement learning (RL) methods to optimize both local decisions and final task completion. While group-based RL methods provide an effective framework for learning from sparse outcome rewards, it remains difficult to determine which intermediate actions contribute to success or failure. Recent stepwise policy optimization methods improve credit assignment by introducing intermediate signals, but these signals are still derived from downstream consequences, leaving local decision quality entangled with future outcomes. We introduce Action-level Credit Assignment with Rubric Evaluation for Long-Horizon Agents (ACRE). Rather than estimating process advantages from outcome-derived returns, ACRE uses reusable rubrics for different functional action families to turn semantic evaluations of intermediate actions into process rewards. These rewards are mean-centered within matched environment-state groups to form relative process advantages, which are combined with the episode-level outcome advantage for policy optimization. In this way, ACRE provides a direct step-level learning signal while preserving episode-level supervision for task completion. Across ALFWorld and WebShop with Qwen2.5-1.5B/3B/7B-Instruct, ACRE consistently improves over return-based process advantage estimation. We further show that rubric-based outcome evaluation can approach verifiable environment rewards, suggesting a practical direction for agentic RL in more open-ended settings where deterministic outcome verification is unavailable.

Primary area:
reinforcement learning
Keywords:
long-horizon agent, reinforcement learning, credit assignment, rubric-as-reward

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_5zGrOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

The paper proposes ACRE, a method for constructing process credit in multi-turn interactive tasks to supplement trajectory-level outcome supervision with local action signals. Building on GiGPO, ACRE uses a rubric library to distinguish action categories, applies a frozen LLM to score actions according to these rubrics, groups actions sharing the same environment state to calculate local advantages, and combines them with trajectory-level advantages. The method is evaluated on ALFWorld and WebShop.

Critical strengths
  • The research problem is valuable. Long-horizon cumulative rewards can make local action evaluation depend heavily on subsequent decisions. ACRE instead evaluates an action from its pre-decision context and the action itself, aiming to better reflect its local quality.
  • The experiments cover two environments and multiple model scales. ACRE consistently improves over the GiGPO baseline across these settings.
  • The paper is clearly written and easy to understand.
Critical weaknesses
  1. GiGPO uses only environment rewards, whereas ACRE introduces additional supervision through rubrics and LLM scoring. It is therefore unclear whether the gains come from the proposed credit-assignment design or simply from access to an additional semantic supervision signal.

  2. ACRE aims to avoid interference from later actions when assigning local credit. BEACON addresses a closely related credit-assignment problem, so a comparison would help clarify ACRE's empirical contribution.

  3. Grouping actions by identical pre-decision states may limit the method's applicability when states rarely repeat.

  4. The accumulated-rubric-score ablation may change the numerical scale of the learning signal. For the ablation that provides the entire trajectory to the rubric scorer, the performance decrease may also reflect a mismatch with rubrics designed specifically for local action evaluation, rather than showing that future information is inherently harmful.

Questions for authors
  1. How much of ACRE's improvement comes from the proposed credit-assignment mechanism rather than from introducing additional LLM-based supervision beyond environment rewards?
  2. How does ACRE compare with BEACON under a matched setting?
  3. How often do repeated pre-decision states occur, and how does this affect applicability outside the evaluated environments?
  4. How are the rubrics generated: manually, by an LLM, or through a combination of the two?
8 comments
AuthorsOpenReview ↗

W1/Q1. Semantic Supervision versus Credit Assignment

W1: GiGPO uses only environment rewards, whereas ACRE introduces additional supervision through rubrics and LLM scoring. It is therefore unclear whether the gains come from the proposed credit-assignment design or simply from access to an additional semantic supervision signal.

Q1: How much of ACRE's improvement comes from the proposed credit-assignment mechanism rather than from introducing additional LLM-based supervision beyond environment rewards?

To isolate the contribution of credit assignment, we hold the Judge and scoring prompt fixed within each comparison and vary how the resulting action-quality scores enter policy optimization. We use the same frozen Qwen3.5-35B-A3B Judge with a General prompt requesting an action-quality score or a Criteria-guided prompt that additionally specifies evaluation dimensions.

For each prompt, we compare two ways of using this supervision:

  • w/o process credit: Add the mean action score to the episode outcome reward and optimize through trajectory-level GRPO.
  • w/ process credit: Mean-center action scores within state groups to construct action-level process advantages, combined with episode-level advantages from environment rewards.
Judge ConfigurationProcess Credit1.5B (Score / SR)3B (Score / SR)7B (Score / SR)
General promptw/o87.06 ± 1.17 / 76.69 ± 2.2087.00 ± 1.72 / 80.47 ± 2.3888.12 ± 0.92 / 80.14 ± 2.28
General promptw/86.82 ± 1.72 / 76.89 ± 3.0487.28 ± 1.96 / 81.05 ± 3.2388.17 ± 1.82 / 82.03 ± 2.44
Criteria-guided promptw/o86.13 ± 2.20 / 77.41 ± 3.8487.03 ± 2.09 / 79.17 ± 3.1186.18 ± 1.46 / 76.37 ± 2.54
Criteria-guided promptw/86.36 ± 1.57 / 78.65 ± 2.8389.41 ± 1.03 / 80.92 ± 2.3387.65 ± 2.20 / 82.36 ± 2.78
ACRE (action-specific rubrics)w/87.18 ± 1.17 / 78.84 ± 2.6089.29 ± 1.66 / 81.97 ± 2.9391.84 ± 1.49 / 85.48 ± 2.76
GRPO (no Judge)w/o84.52 ± 1.37 / 74.67 ± 2.9086.36 ± 2.51 / 77.41 ± 3.1887.47 ± 0.61 / 77.60 ± 1.19

Additional semantic supervision is useful: with the General prompt, adding the mean Judge score to the episode reward improves SR over GRPO by 2.02/3.06/2.54 points at 1.5B/3B/7B. The controlled comparisons further reveal the contribution of process credit assignment: under the same Judge and prompt, constructing process advantages further improves SR by 0.20/0.58/1.89 points with the General prompt and 1.24/1.75/5.99 points with the Criteria-guided prompt.

The controlled comparisons support contributions from both semantic supervision and using that supervision to construct action-level process advantages.

AuthorsOpenReview ↗

W2/Q2. Comparison with BEACON

W2: ACRE aims to avoid interference from later actions when assigning local credit. BEACON addresses a closely related credit-assignment problem, so a comparison would help clarify ACRE's empirical contribution.

Q2: How does ACRE compare with BEACON under a matched setting?

BEACON assigns credit to trajectory segments delimited by subgoal milestones, whereas ACRE constructs action-level process advantages from the quality of each action in its pre-decision context. We evaluate our BEACON reproduction using ACRE's protocol and include BEACON's published results (Tables 1 and 3) for reference.

WebShop

ModelMethodScore (%)SR (%)
1.5BACRE87.18 ± 1.1778.84 ± 2.60
1.5BBEACON (published)86.175.6
1.5BBEACON (our reproduction)87.60 ± 1.5877.08 ± 3.72
3BACRE89.29 ± 1.6681.97 ± 2.93
3BBEACON (our reproduction)85.29 ± 2.4577.15 ± 3.90
7BACRE91.84 ± 1.4985.48 ± 2.76
7BBEACON (published)87.779.7
7BBEACON (our reproduction)88.35 ± 1.2580.47 ± 1.98

ALFWorld

ModelMethodPickLookCleanHeatCoolPick2All
1.5BACRE97.26 ± 1.2686.84 ± 6.2092.50 ± 5.1992.11 ± 8.5788.30 ± 6.1880.25 ± 1.8190.36 ± 2.15
1.5BBEACON (published)100.088.286.7100.078.992.991.4
1.5BBEACON (our reproduction)94.48 ± 7.6878.15 ± 9.1598.69 ± 1.2194.41 ± 3.9082.52 ± 6.2680.79 ± 4.0189.06 ± 2.82
3BACRE97.53 ± 2.3088.32 ± 5.8391.40 ± 6.2981.25 ± 9.2294.29 ± 1.0173.80 ± 11.0389.19 ± 0.81
3BBEACON (our reproduction)96.72 ± 3.0094.17 ± 3.0087.16 ± 3.8886.11 ± 8.7062.15 ± 10.3165.87 ± 6.0181.51 ± 3.18
7BACRE100.00 ± 0.00100.00 ± 0.00100.00 ± 0.0096.70 ± 3.5094.44 ± 0.8678.36 ± 3.2094.66 ± 0.81
7BBEACON (published)100.081.896.392.994.790.094.5
7BBEACON (our reproduction)99.49 ± 0.8798.33 ± 2.8999.48 ± 0.90100.00 ± 0.0099.33 ± 1.1595.71 ± 3.4498.70 ± 0.60

Published BEACON results are included for reference only; the direct comparison is based on our reproduction under ACRE's evaluation protocol.

Compared with our BEACON reproduction, ACRE achieves higher SR on WebShop at all three scales and on ALFWorld at 1.5B/3B, with particularly large gains on the 3B Cool and Pick2 tasks. BEACON is stronger on ALFWorld 7B and slightly higher on WebShop 1.5B Score. These results support ACRE as a competitive approach to fine-grained credit assignment.

AuthorsOpenReview ↗

W3/Q3. State Recurrence, Coverage, and Alternative Grouping

W3: Grouping actions by identical pre-decision states may limit the method's applicability when states rarely repeat.

Q3: How often do repeated pre-decision states occur, and how does this affect applicability outside the evaluated environments?

We measured recurrence within the rollout groups used during WebShop training. After excluding each trajectory's first action, the table reports the fraction of scored actions whose state appears in at least two distinct trajectories:

ModelScored actions in recurring state groups
1.5B72.64%
3B74.35%
7B82.43%

Without a matching state, ACRE's current rule produces zero process advantage and retains episode-level supervision. A possible extension is to group task-relevant, similar states, allowing local comparisons without exact state matches.

To examine whether the credit signal can be used with a different grouping rule, we combine ACRE with HGPO[1], which constructs hierarchical groups based on increasingly longer shared histories and adaptively aggregates their advantages.. We retain its hierarchy and combine environment-return advantages with process advantages derived from immediate rubric scores.

WebShop results:

Method1.5B Score (%)1.5B SR (%)3B Score (%)3B SR (%)7B Score (%)7B SR (%)
GiGPO85.11 ± 2.1076.69 ± 3.5487.35 ± 2.1080.08 ± 2.5687.89 ± 2.0778.97 ± 4.01
HGPO (our reproduction)82.76 ± 2.3574.35 ± 2.4488.35 ± 1.8181.18 ± 3.2888.84 ± 1.6181.32 ± 3.03
HGPO + ACRE87.12 ± 1.8277.86 ± 2.6590.67 ± 2.1784.77 ± 2.8292.10 ± 1.9385.55 ± 3.07
ACRE87.18 ± 1.1778.84 ± 2.6089.29 ± 1.6681.97 ± 2.9391.84 ± 1.4985.48 ± 2.76

HGPO + ACRE achieves higher Score and SR than HGPO at all three scales. These results support grouping and credit-signal design as orthogonal choices: grouping determines which decisions are compared, while ACRE uses immediate action quality to assign local credit.

[1] He, S., Feng, L., Cheng, X., Feng, L., & An, B. (2026, April). Hierarchy-of-groups policy optimization for long-horizon agentic tasks. In International Conference on Learning Representations (Vol. 2026, pp. 27572-27593).

Reviewer_mpGuOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes ACRE, which adds a "process reward" to GRPO-style training of LLM agents. A frozen LLM judge scores each action against hand-written rubrics for its action family, seeing only the context before the action. ACRE is evaluated on ALFWorld and WebShop with Qwen2.5-1.5B/3B/7B-Instruct against GRPO and GiGPO.

Critical strengths

The paper proposes a practical, simple recipe to improve action evaluation in LLMs, using rubrics per action family, and GIGPO state grouping. This is easy to expand to other agents. Experiment ask interesting questions, e.g., agreement with other judges, how often each signal gives credit in Section 5.2, the cosine similarity with rollout-estimated value (App. D.3), repeated actions, and a compute table (Table 6).

Critical weaknesses

Method

  • Rubrics are hand-made by the user, making the method not scalable. This is trivially solvable with having another LLM create the rubric, but also opens a whole new subset of investigations to understand the effect of rubric proposition on the method (and of the LLM that creates them).

Evidence

  • Every result comes from one training run per method: the reported ± are over three evaluation task streams, "rather than independent training runs" (l. 1101–1102, 1118–1119). Training variance is never measured, and on WebShop the gaps over GiGPO at 1.5B and 3B (+2.2 and +1.9 SR) are within the evaluation spread alone.
  • Ablations (Table 2) are single runs on one setting (3B, WebShop), with differences of 1.2–3.3 SR points, within the evaluation spread of Table 1.
  • ALFWorld is evaluated on "valid seen" tasks (Table 7), i.e. in distribution, and checkpoints are selected on "the highest validation success rate" (l. 1015) without saying which split that is.
  • The gains are not attributed. ACRE adds supervision from a 35B judge and hand-written rubrics encoding task heuristics (e.g. "non-repetition", "no revisit excluded product", App. B) that the baselines don't receive. Without a judge-matched baseline — the same judge without rubrics, or rubric-based process rewards such as ARCO or TRCA, which the paper cites Section 2., the gains may come from distilling the judge's knowledge rather than from the credit construction.
  • The rubrics were developed on "observed policy behaviour" and "development rollouts", and revised "when systematic ambiguities are observed" (lines 807–817). On which tasks, and with what budget, is not stated. Since rubrics are the cornerstone of the method, this makes me question the paper methodology.
  • Compute is not matched: time per update is 2.8–3.3× GiGPO's on WebShop (Table 6), with the judge on two extra H100s and its tokens excluded from throughput.
Questions for authors

Additionally to the points in the weaknesses:

  • On which tasks were the rubrics developed, and how much effort went into them?
  • If the process signal is uncorrelated with QπQ^\pi (App. D.3), what makes it useful for the task, beyond the judge's knowledge?
14 comments
AuthorsOpenReview ↗

Rubric Construction and Development Effort (Method Weakness; W5/Q1)

Method weakness: Rubrics are hand-made by the user, making the method not scalable. This is trivially solvable with having another LLM create the rubric, but also opens a whole new subset of investigations to understand the effect of rubric proposition on the method (and of the LLM that creates them).

E5: The rubrics were developed on "observed policy behaviour" and "development rollouts", and revised "when systematic ambiguities are observed" (lines 807–817). On which tasks, and with what budget, is not stated. Since rubrics are the cornerstone of the method, this makes me question the paper methodology.

Q1: On which tasks were the rubrics developed, and how much effort went into them?

Our rubrics are developed through an LLM-driven workflow with lightweight human oversight. LLMs perform the analysis, drafting, and iterative refinement, while the authors guide the design and review the generated rubrics to ensure their quality.

  1. Action-family identification: An LLM (GPT 5.5) groups the environment's supported actions by their functional roles.
  2. Failure analysis and drafting: Policy rollouts collected during RL serve as development data. The LLM analyzes failure cases, identifies evaluation dimensions for each action family, and drafts the rubrics.
  3. Author review: The authors check that the proposed rubrics reflect task requirements and the intended notion of action quality.
  4. Offline validation and refinement: We apply the candidate rubrics to development cases and inspect the judge's decisions. Systematic errors or ambiguities guide further LLM revisions.
  5. Final approval: Once the offline checks are satisfactory, the authors approve and freeze the rubric library and judge configuration. Both remain unchanged during the reported training run.

For WebShop, we used Qwen2.5-3B as the development policy and conducted five refinement rounds. We then reused the workflow and design experience on ALFWorld, which required two refinement rounds. This transfer illustrates reuse across environments; within each environment, the finalized action-family library is shared across tasks and trajectories.

We agree that further automating rubric generation is important for scaling this approach, particularly to open-ended tasks. Our action-family rubric library is reused across tasks within each environment, while adaptation to new environments still requires human review and validation. More automated generation and refinement could reduce this development effort without changing ACRE's credit-assignment formulation.

AuthorsOpenReview ↗

W1. Variation Across Training Runs

Every result comes from one training run per method: the reported ± are over three evaluation task streams, "rather than independent training runs" (l. 1101–1102, 1118–1119). Training variance is never measured, and on WebShop the gaps over GiGPO at 1.5B and 3B (+2.2 and +1.9 SR) are within the evaluation spread alone.

To address the concern about training stability, we repeated ACRE training using three rollout seeds: 42, 142, and 242. The following tables report the individual results and the mean and sample standard deviation across the three training runs.

WebShop

Each seed entry reports Score and SR, respectively. The ± values in individual seed entries represent evaluation-stream standard deviations.

ModelSeed 42 (Score / SR)Seed 142 (Score / SR)Seed 242 (Score / SR)Across-run Mean ± SD (Score / SR)Original ACRE (Score / SR)Δ (Score / SR)
1.5B87.63 ± 1.57 / 79.04 ± 2.7487.50 ± 0.69 / 78.58 ± 2.3387.43 ± 1.35 / 77.80 ± 2.5587.52 ± 0.10 / 78.47 ± 0.6387.18 / 78.84+0.34 / −0.37
3B89.84 ± 1.81 / 82.42 ± 3.1388.53 ± 2.52 / 79.82 ± 3.7988.60 ± 1.83 / 81.84 ± 3.2288.99 ± 0.74 / 81.36 ± 1.3689.29 / 81.97−0.30 / −0.61
7B91.36 ± 2.18 / 85.68 ± 2.9092.05 ± 2.17 / 86.59 ± 3.0091.24 ± 0.97 / 85.94 ± 1.7491.55 ± 0.44 / 86.07 ± 0.4791.84 / 85.48−0.29 / +0.59

ALFWorld

ModelSeed 42 (SR)Seed 142 (SR)Seed 242 (SR)Across-run Mean ± SDOriginal ACREΔ
1.5B91.28 ± 0.4590.89 ± 1.6390.49 ± 2.2690.89 ± 0.4090.36+0.53
3B90.23 ± 1.7990.62 ± 1.3587.76 ± 0.9889.54 ± 1.5589.19+0.35
7B94.79 ± 0.4594.79 ± 0.6095.96 ± 0.4595.18 ± 0.6894.66+0.52

The results show that across-run standard deviations on WebShop range from 0.10–0.74 for Score and 0.47–1.36 for SR. The repeated-run means differ from the original results by at most 0.34 Score and 0.61 SR points. On ALFWorld, the repeated-run means are also close to the original results across all three model sizes, with differences of only 0.35–0.53 SR points and across-run standard deviations of 0.40–1.55.

Overall, these results demonstrate that ACRE achieves consistent performance across repeated training runs, supporting the reliability of our reported results.

AuthorsOpenReview ↗

W2 & W3. Replication of Ablation Studies and ALFWorld Evaluation Split Clarification

W2. Replication of Ablation Studies

Ablations (Table 2) are single runs on one setting (3B, WebShop), with differences of 1.2–3.3 SR points, within the evaluation spread of Table 1.

We conduct additional experiments to examine the reliability of the ablation findings in two settings:

  1. WebShop 3B: We conduct three training runs with rollout seeds 42, 142, and 242.
  2. ALFWorld 7B: Given the substantially longer training time of ALFWorld, we additionally evaluate different ablation settings with one training run per variant to examine whether the findings extend to another environment.

WebShop 3B: Three-run ablation results

VariantScore (42 / 142 / 242)Score Mean ± SDSR (42 / 142 / 242)SR Mean ± SD
Full89.84 / 88.53 / 88.6088.99 ± 0.7482.42 / 79.82 / 81.8481.36 ± 1.36
Common-only87.54 / 87.13 / 86.5687.08 ± 0.4978.71 / 78.06 / 77.7378.17 ± 0.50
Shared-full87.63 / 89.16 / 87.7388.17 ± 0.8679.36 / 81.38 / 80.6680.47 ± 1.02
Trajectory88.56 / 89.83 / 87.9488.78 ± 0.9680.79 / 82.62 / 78.6580.69 ± 1.99

ALFWorld 7B: Cross-environment ablation

The ± values for ALFWorld denote standard deviations across the three fixed evaluation streams, as each ablation variant uses one training run.

VariantSRΔ vs. Full
Full (ACRE)94.66 ± 0.81—
Common-only93.36 ± 1.03−1.30
Shared-full92.58 ± 1.70−2.08
Trajectory94.27 ± 0.60−0.39

Across both tasks, Common-only and Shared-full perform below Full, while Trajectory remains closest. Common-only omits action-specific quality dimensions, whereas Shared-full may distort their relative weights by including overlapping criteria. Trajectory retains action-level scoring, which may explain its smaller gap, but Full achieves higher mean SR using only decision-time context. These consistent trends support the ablation conclusions in Table 2(a).

W3. ALFWorld Evaluation Split Clarification

ALFWorld is evaluated on "valid seen" tasks (Table 7), i.e. in distribution, and checkpoints are selected on "the highest validation success rate" (l. 1015) without saying which split that is.

Validation for checkpoint selection and final evaluation both use ALFWorld's official valid_seen split, which contains 140 games. Their fixed sampling seeds and game coverage are summarized below.

StageSampling seedEpisodesDistinct games covered
Validation (checkpoint selection)0256119
Final evaluation123256123
Final evaluation456256119
Final evaluation789256115
Final evaluation (combined)123, 456, 789768140

We will explicitly document the split and sampling protocol to make the evaluation procedure transparent.

Reviewer_hepyOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes ACRE, a rubric-based action-level credit assignment framework for long-horizon LLM agents. It evaluates local action quality using reusable action-family rubrics and constructs state-conditioned relative process advantages, which are combined with episode-level outcome advantages for policy optimization. Experiments on WebShop and ALFWorld across three Qwen2.5 model scales demonstrate consistent improvements over GRPO and GiGPO, supported by additional analyses and ablation studies. The paper also explores rubric-based outcome supervision as an alternative to official environment rewards.

Critical strengths
  1. Well-motivated analysis and rubric-based process supervision. The paper clearly identifies the limitations of outcome-derived process credit and proposes ACRE to evaluate intermediate actions independently of downstream outcomes using reusable action-family rubrics. This provides a meaningful approach to addressing delayed credit assignment in long-horizon agentic RL.
  2. An insightful design for state-conditioned relative process advantages. ACRE applies mean-centering to rubric rewards within groups of actions encountered under the same environment state, converting local quality assessments into relative process advantages. This simple yet effective design provides an intuitive way to integrate semantic process supervision into group-based policy optimization.
  3. Informative analyses supporting the value of direct process credit. The paper provides insightful analyses showing that direct process credit can distinguish local action quality more accurately than return-to-go, provide additional supervision when outcome advantages vanish, and reduce redundant actions. These findings offer useful empirical evidence for the distinct role of local process supervision in long-horizon agent learning.
Critical weaknesses
  1. Section 5.2 shows that Direct Process Credit provides supervision when Outcome Advantage vanishes, but does not establish how much this contributes to performance gains. Moreover, the near-zero cosine similarity (−0.005) with Downstream Utility demonstrates signal differences rather than effective complementarity.
  2. Appendix B.3 lacks details on query-specific constraint construction, the division between LLM judgments and deterministic checks, and the exact rubrics and prompts. Section 5.4 reports comparable performance to official environment rewards but does not analyze reward-level differences, leaving the distinct supervision provided by rubric rewards unclear.
  3. Experiments cover only WebShop and ALFWorld, limiting evidence of generalization to open-ended agent tasks. Comparisons are also restricted to GRPO and GiGPO, without recent rubric-based credit assignment methods such as ARCO, TRCA, and DRACO, making ACRE's advantages over related approaches difficult to assess.
Questions for authors

Clarification requests

  1. Contribution of Direct Process Credit. Section 5.2 shows that Direct Process Credit provides additional supervision when the Outcome Advantage is zero. Could the authors further analyze how much these process-only groups contribute to the overall performance gains? For example, would enabling Process Credit only in these groups achieve performance comparable to full ACRE?
  2. Downstream Utility Analysis. Appendix D.3 estimates downstream utility using only eight continuations per candidate action at the initial policy checkpoint, where task success is relatively sparse. Given that prior work often employs substantially more rollouts when constructing Monte Carlo reference values for analysis, using more continuations would provide a more reliable estimate of downstream utility and strengthen the evidence supporting the reported near-zero cosine similarity (−0.005).
  3. Episode-level Rubric Reward Design. Could the authors provide the complete outcome-level rubrics, judge prompts, and details on how query-specific constraints are constructed and assigned to LLM-based or deterministic evaluation? A concrete trajectory example showing criterion-level judgments, the resulting rubric reward, and a comparison with the official environment reward would help clarify the differences between the two supervision signals. Suggestions for future work
  4. Broader Benchmark Evaluation. Evaluating ACRE on more diverse long-horizon agent benchmarks, particularly those involving more complex interactions, open-ended objectives, and limited programmatic outcome verification, would help establish its generalizability.
  5. Comparisons with Additional Baselines. Comparisons with recent rubric-based step-wise credit assignment methods, such as ARCO, TRCA, and DRACO, would help better establish ACRE's distinctive contributions and advantages over existing process supervision approaches.
6 comments
AuthorsOpenReview ↗

W1/Q1. Contribution of Direct Process Credit

W1 Section 5.2 shows that Direct Process Credit provides supervision when Outcome Advantage vanishes, but does not establish how much this contributes to performance gains.

Q1 Could the authors further analyze how much these process-only groups contribute to the overall performance gains? For example, would enabling Process Credit only in these groups achieve performance comparable to full ACRE?

We agree that supervision coverage alone does not quantify its contribution to final performance. Section 5.2 (Lines 354–359) reports that, on WebShop, 38.14% of trajectory groups have nonzero process credit despite zero outcome advantages. We assess the practical value of this supervision through two comparisons below.

Here, an outcome-zero group contains trajectories with identical outcome rewards (e.g., all succeed or all fail), so all episode-level advantages are zero. An outcome-nonzero group contains different outcome rewards.

Comparison 1: GRPO → Zero-only — the benefit of enabling process credit exclusively in outcome-zero groups. GRPO uses outcome advantages alone. Zero-only adds process credit in outcome-zero groups, while leaving it disabled in outcome-nonzero groups. This comparison tests whether process supervision in groups where outcome rewards cannot distinguish trajectories can improve final task performance.

Method1.5B Score (%)1.5B SR (%)3B Score (%)3B SR (%)7B Score (%)7B SR (%)
GRPO84.52 ± 1.3774.67 ± 2.9086.36 ± 2.5177.41 ± 3.1887.47 ± 0.6177.60 ± 1.19
Zero-only87.29 ± 1.4876.04 ± 2.3188.65 ± 1.9480.99 ± 2.8189.65 ± 2.1684.11 ± 3.36

Zero-only achieves higher SR than GRPO by 1.37, 3.58, and 6.51 percentage points at 1.5B, 3B, and 7B, respectively. These results provide performance evidence for the usefulness of process supervision in outcome-zero groups.

Comparison 2: Nonzero-only → Full ACRE — the added value of supervising outcome-zero groups within the full method. Nonzero-only enables process credit in outcome-nonzero groups and disables it in outcome-zero groups. Full ACRE enables process credit in both types of groups. This comparison assesses the value of also supervising outcome-zero groups when process credit is already available in outcome-nonzero groups.

Method1.5B Score (%)1.5B SR (%)3B Score (%)3B SR (%)7B Score (%)7B SR (%)
Nonzero-only87.31 ± 2.2777.34 ± 4.1686.86 ± 0.7980.66 ± 0.9088.19 ± 1.2980.99 ± 1.68
Full ACRE87.18 ± 1.1778.84 ± 2.6089.29 ± 1.6681.97 ± 2.9391.84 ± 1.4985.48 ± 2.76

Full ACRE achieves higher mean SR than Nonzero-only by 1.50, 1.31, and 4.49 percentage points, respectively. This supports the additional value of process supervision in outcome-zero groups.

Together, these comparisons connect the supervision coverage observed in Section 5.2 to measurable differences in final success rate. At 3B and 7B, enabling process credit only in outcome-zero groups already achieves mean SR close to Full ACRE. Full ACRE achieves the highest mean SR across all three scales, supporting the benefit of retaining process supervision in both outcome-zero and outcome-nonzero groups.

AuthorsOpenReview ↗

W1/Q2. Downstream Utility Analysis

W1: Moreover, the near-zero cosine similarity (−0.005) with Downstream Utility demonstrates signal differences rather than effective complementarity.

Q2: Appendix D.3 estimates downstream utility using only eight continuations per candidate action at the initial policy checkpoint, where task success is relatively sparse.

Following your suggestion, we collected 64 continuations per candidate action, retaining the original 100 decision contexts, four candidates per context, initial Qwen2.5-3B policy, and within-context centering from Appendix D.3. We report the first M = 8/16/32/64 continuations from this new collection, under the original limits (A) and increased token/action limits (B/C).

The cosine similarities with downstream utility are:

Credit signalMA: 15 actions / 12,288 tokensB: 15 actions / 28,672 tokensC: 30 actions / 28,672 tokens
Direct process credit8−0.0118−0.0065−0.0165
Direct process credit16−0.0210−0.0135−0.0149
Direct process credit32−0.0188−0.0128−0.0146
Direct process credit64−0.0156−0.0076−0.0101
Return-to-go640.77670.80000.7604

For the RTG comparison, the utility reference excludes the continuation used to compute the return. The cosine for direct process credit remains near zero as M increases, whereas RTG remains positively aligned with downstream utility. This preserves the contrast reported in Section 5.2 (−0.005 for direct credit versus 0.655 for RTG).

We agree that the cosine result characterizes signal differences. In Section 5.2, our interpretation of complementary supervision also relies on local-quality ranking and process-only supervision coverage. The W1/Q1 ablations above add performance evidence: Zero-only improves SR over GRPO, and Full ACRE improves SR over Nonzero-only across all three scales. Together, these results support combining supervision of action quality at decision time with episode-level outcome supervision.

AuthorsOpenReview ↗

W2/Q3. Episode-level Rubric Reward Design (Part 1)

W2 Appendix B.3 lacks details on query-specific constraint construction, the division between LLM judgments and deterministic checks, and the exact rubrics and prompts. Section 5.4 reports comparable performance to official environment rewards but does not analyze reward-level differences, leaving the distinct supervision provided by rubric rewards unclear.

Q3 Could the authors provide the complete outcome-level rubrics, judge prompts, and details on how query-specific constraints are constructed and assigned to LLM-based or deterministic evaluation? A concrete trajectory example showing criterion-level judgments, the resulting rubric reward, and a comparison with the official environment reward would help clarify the differences between the two supervision signals.

The outcome rubric checks whether the final purchase satisfies the user's request. For WebShop, it instantiates criteria for product type, descriptive attributes, selected options, and price, with a separate purchase-completion check.

1. Constraint construction and evaluation

RequirementConstructionEvaluation
Purchase completionFixed requirementTerminal environment record
Product typeLLM extracts category and source spanLLM judgment + type-compatibility checks
Descriptive attributesLLM extracts individual properties and source spansEvidence-based LLM judgment; exact catalog matches can override
Explicit optionsCode parses color, size, etc.Actual selected-option record
PriceCode parses upper boundLLM comparison with actual purchase price

The judge evaluates each instantiated criterion against the request and structured purchase evidence, returning pass, fail, or null with supporting evidence.

2. Worked trajectories

We provide two representative training episodes:

Example 1: Complete matchExample 2: Partial match
RequestMen's slim-fit button-down shirt, aqua blue, x-small, below $50Men's sweater with specified washing and material attributes, xx-large, below $60
Purchased itemMatching Oxford shirt, $18.50T-shirt instead of sweater, $19.99
TrajectorySearch → open → select size/color → buySearch → open → select fit/size → buy
Constraint results8/8 pass3 pass, 1 fail, 3 null
Rubric reward10.4286
Official reward10 (raw Score 1.0)0 (raw Score 0)

The second example illustrates how the rubric preserves credit for confirmed requirements despite an incorrect product type. Specifically, Q=3/7=0.4286Q=3/7=0.4286, whereas the official verifier assigns zero.

Following Appendix B.3, a valid purchase receives 10 when all constraints pass and otherwise receives QQ; an invalid or missing purchase receives zero. null requirements remain in the denominator. Purchase completion acts as a separate validity gate rather than an additional constraint in QQ.

3. Judge prompt

The pipeline uses separate prompts for constraint extraction and outcome evaluation. Below are condensed versions of their instructions.

Constraint extraction

Input: request text after code has extracted explicit options and the price bound.
Extract exactly one product type and each remaining requested attribute.
Copy requirement text and source spans verbatim; add no inferred requirements.
Return JSON with product_type and attributes.

Outcome evaluation

Input: user_request, constraint_contract, terminal_rubrics, and purchase evidence (product facts, actual selected options, purchase price, and Buy completion).

Evaluate only the final outcome; judge each supplied criterion independently.
- Product type: require the requested kind, a clear synonym, or an ordinary subtype.
- Attributes: pass if entailed by facts; fail if contradicted; null if evidence is absent or conflicting.
- Options: require the exact value in the actual selected-option record.
- Price: compare the actual purchase price with the requested bound.
- Completion: require an executable Buy; buying alone does not imply other passes.

Return JSON with a rubrics list of {id, verdict, evidence}, in the supplied order, with verdict in {pass, fail, null} and brief evidence for every judgment.

The judge supplies criterion-level verdicts; the deterministic checks and aggregation described above produce the final reward.

We will provide the complete prompts and deterministic checks in Appendix B.3.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Manuscript Revisions

We thank all reviewers for their constructive feedback. We have highlighted revisions in blue for ease of review. The main updates are:

  • Process credit: Added controlled experiments and analyses of the contribution of action-level process credit (Sections 5.4.2–5.4.3 and Appendices D.2–D.3).
  • Rubrics: Clarified the construction workflow and added rubric ablations and an outcome-reward example (Section 5.4.1 and Appendices B.1 and B.3).
  • Related methods: Added comparisons with fine-grained credit-assignment methods and evidence of compatibility with state-grouping improvements (Section 5.3 and Appendix C.5).
  • Experimental reliability: Added repeated training and ablation results, and clarified evaluation coverage (Appendices C.4 and E).

We welcome any further questions during the discussion period.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.