acceptodds
Under review as a conference paper at ICLR 2027

Supervising Agent Evolution with Prefix Potentials

11 views

Abstract

Agent evolution through continued interaction and parameter updates requires informative supervision. However, environmental feedback is often sparse and concentrated at terminal outcomes, offering limited guidance for intermediate decisions. Typically, existing approaches alleviate this sparsity via separately trained evaluators or task-specific reference structures to construct step-level supervision. Instead, we propose **A**gent **S**elf-supervision through **C**ompletion-potential **E**stimation and **N**eighboring-prefix **D**ifferencing (ASCEND), which derives this fine-grained supervision from assessments by the evolving agent. Concretely, the evolving agent estimates a prefix potential at each step through an additional scoring call, assessing task-completion prospects based on the task objective and the actions and observations available up to that step. Differences between adjacent prefix potentials yield step-level signals, which are normalized, accumulated from each step onward, and combined with terminal outcome feedback to guide parameter updates. Evaluations on long-horizon interactive tasks and multi-hop reasoning demonstrate that ASCEND outperforms representative evolution baselines, with improvements extending to additional benchmarks without further adaptation. Further analysis shows that prefix potentials correlate with final task outcomes and that differences between adjacent prefix potentials align with independently assessed task progress. Code is available at *Anonymous*.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Agent Evolution, Parametric Self-Evolution, Self-Supervision, Completion Potential

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_AKnLOpenReview ↗
Rating
3(+1)
Confidence
2(−1)
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper proposes a method for constructing fine-grained process rewards in agentic RL scenarios. The authors first point out that current methods either rely on auxiliary evaluation or task-specific reference construction beyond the available interaction evidence. The authors then propose ASCEND, a fine-grained supervision for agent evolution under explicit information constraints. ASCEND segments each interaction in the trajectory, obtaining multiple prefixes of varying lengths. Then, using prompts, the same model is asked to evaluate whether prefixes of different lengths can complete the task, obtaining the potential of each prefix. The process reward for each segment is obtained by differentiating the potential. Finally, GRPO is performed by combining process rewards and outcome rewards. The authors evaluate the effectiveness of the method on agent and multi-hop reasoning benchmarks.

Critical strengths
  1. This paper is quite concise and easy to understand, and the method proposed in the paper is also simple and effective.
  2. The authors conduct extensive experiments and analyses, including potential-outcome alignment analysis, alignment analysis with human judgment, various sensitivity analyses, and a discussion of self-assessment bias and cost. Some of these analyses are insightful.
Critical weaknesses
  1. Some of the authors' conclusions are exaggerated. They first point out that current methods are limited by auxiliary evaluation or task-specific reference construction beyond the available interaction evidence. Regarding the latter, it doesn't seem to be information-constrained as the authors claim. Numerous methods [1-2] can obtain process rewards simply by calculating within a single trajectory or between different trajectories within a group in GRPO, incurring minimal costs. In comparison, the authors' proposed method requires scoring each prefix, making it more costly. This weakens the motivation of this paper. Furthermore, they haven't compared their method with these well-known baselines such as GIGPO.
  2. The method's performance is not significant. On almost all benchmarks, ASCEND's improvement over the baseline is not significant, and it is even within the noisy range.
  3. The idea of ​​obtaining process rewards by evaluating trajectory segments is not new, and the authors have not compared it with related work, such as [3-4].
  4. The method is standard post-training in agent scenarios. However, the paper emphasizes the word "evolution" extensively and repeatedly, claiming that it is online adaptation after deployment, but this is not actually the case, so it seems a bit strange.

[1] Group-in-Group Policy Optimization for LLM Agent Training.

[2] RLVMR: Reinforcement Learning with Verifiable Meta-Reasoning Rewards for Robust Long-Horizon Agents.

[3] Meta-Reasoner: Dynamic Guidance for Optimized Inference-time Reasoning in Large Language Models.

[4] Optimizing Test-Time Compute via Meta Reinforcement Fine-Tuning.

Questions for authors

See Weaknesses.

8 comments
AuthorsOpenReview ↗

Response Part-1/3

We sincerely thank Reviewer #AKnL for the detailed comments. Below, we clarify the research positioning and provide additional comparisons.


W1. Some of the authors' conclusions are exaggerated. They first point out that current methods are limited by auxiliary evaluation or task-specific reference construction beyond the available interaction evidence. Regarding the latter, it doesn't seem to be information-constrained as the authors claim. Numerous methods [1-2] can obtain process rewards simply by calculating within a single trajectory or between different trajectories within a group in GRPO, incurring minimal costs. In comparison, the authors' proposed method requires scoring each prefix, making it more costly. This weakens the motivation of this paper. Furthermore, they haven't compared their method with these well-known baselines such as GIGPO.

A: First, we would like to clarify that our information-constrained setting concerns the resources and comparison structures required for supervision construction, distinct from computational cost. GiGPO [1] depends on recurring anchors with varying returns. RLVMR [2] requires teacher-annotated warmup demonstrations and predefined behavioral reward rules. Conversely, ASCEND constructs prefix potentials directly from visible interaction histories through additional scoring calls, without these matching or demonstration requirements.

The following table summarizes process-supervision requirements. Terminal feedback is treated separately, and teacher demonstrations refer to method-specific supervised warmup.

MethodStep-level signalsAnchor-state matchingTeacher demonstrationsBehavioral reward rulesPrefix model scoring
GiGPO [1]✓\checkmark✓\checkmark×\times×\times×\times
RLVMR [2]✓\checkmark×\times✓\checkmark✓\checkmark×\times
ASCEND✓\checkmark×\times×\times×\times✓\checkmark

Second, we additionally evaluate GiGPO on APPWorld and multi-hop reasoning, with Qwen3.5-9B as the foundation model. Anchor grouping follows exact observation matching on APPWorld and text-similarity matching with a threshold of 0.9 on multi-hop reasoning.

Results on APPWorld are as follows:

MethodBest SR ↑\uparrowΔ\DeltaSR ↑\uparrowAvg. Steps ↓\downarrowΔ\DeltaSteps ↓\downarrow
Base Model30.35—20.66—
Vanilla GRPO44.64+14.2918.83−1.83
GiGPO45.83+15.4818.95−1.71
ASCEND50.00+19.6518.22−2.44

Results on multi-hop reasoning are as follows:

Method2WikiHotpotQAMuSiQueMacro Avg.
EM ↑\uparrowF1 ↑\uparrowEM ↑\uparrowF1 ↑\uparrowEM ↑\uparrowF1 ↑\uparrowEM ↑\uparrowF1 ↑\uparrow
Base Model38.8454.6836.1553.834.3813.0426.4640.52
Vanilla GRPO62.2171.6149.3864.166.0814.6139.2250.13
GiGPO62.6972.2648.7164.276.6215.6839.3450.74
ASCEND63.4972.8850.3365.737.0716.5740.3051.73

Third, we further audit anchor coverage. GiGPO compares terminal outcome rewards discounted by the remaining number of steps. Missing matches or identical returns within matched groups provide no return contrasts for differentiated anchor-based signals.

SettingNo matching anchorMatched group with identical returnsMatched group with varying returns
APPWorld30.86% (59,443/192,635)18.87% (36,345/192,635)50.27% (96,847/192,635)
2Wiki20.79% (8,705/41,862)16.94% (7,091/41,862)62.27% (26,066/41,862)
  • Steps without differentiated anchor-based signals account for 49.73% (95,788/192,635) of APPWorld steps and 37.73% (15,796/41,862) of 2Wiki steps.
  • ASCEND exceeds GiGPO by 4.17% SR with 0.73 fewer average execution steps on APPWorld, and by 0.99% macro-average F1 on multi-hop reasoning.

ASCEND constructs process signals without matching anchors or varying group returns, at the cost of additional prefix scoring. Appendix D discusses shared-prefix caching as an opportunity to reduce repeated computation.

We have expanded the related-work discussion in Section 2 and added GiGPO comparisons in Table 1 and Table 2. Implementation details and anchor-coverage results are provided in Appendix B.2 and Appendix C.6, Figure 6, of the revised version.

AuthorsOpenReview ↗

Response Part-2/3

W2. The method's performance is not significant. On almost all benchmarks, ASCEND's improvement over the baseline is not significant, and it is even within the noisy range.

A: First, ASCEND improves APPWorld SR from 44.64% to 50.00%, with corresponding Avg. Steps decreasing from 18.83 to 18.22. On multi-hop reasoning, macro-average F1 increases from 50.13% to 51.73%, with higher F1 on all three benchmarks.

Second, to assess evaluation-sample uncertainty on multi-hop reasoning, we conduct a paired bootstrap analysis with 20,000 resamples of evaluation questions for the reported round-4 agent instances. The table below reports performance and pointwise 95% confidence intervals for the improvements, with equal benchmark weights for macro averages. Here, Δ\Delta denotes the improvement of ASCEND over Vanilla GRPO in EM/F1 points.

Method2WikiHotpotQAMuSiQueMacro Avg.
EMF1EMF1EMF1EMF1
Vanilla GRPO62.2171.6149.3864.166.0814.6139.2250.13
ASCEND63.4972.8850.3365.737.0716.5740.3051.73
95% CI of Δ\Delta[0.73, 1.84][0.76, 1.79][0.16, 1.73][0.86, 2.30][0.00, 1.99][0.86, 3.07][0.61, 1.54][1.14, 2.07]

The macro-average EM and F1 improvements have 95% confidence intervals of [0.61, 1.54] and [1.14, 2.07] points, respectively. The lower confidence bound for the F1 improvement is positive on all three benchmarks. These results support the reported improvements under evaluation-sample resampling, with the trained agent instances and recorded predictions held fixed.

We have added this analysis in Appendix C.9, Table 23, of the revised version.


W3. The idea of ​​obtaining process rewards by evaluating trajectory segments is not new, and the authors have not compared it with related work, such as [3-4].

A: The key distinction concerns the requirements for constructing process supervision. Their reported implementations depend on auxiliary progress evaluation or verifiable answer completion.

Meta-Reasoner scores generated progress reports through a separate evaluator and updates a contextual bandit to select inference-time strategies such as continuing, restarting, or backtracking. Its feedback guides strategy selection instead of directly supervising parameter updates of the executing agent. Adapting this mechanism to parametric evolution requires both auxiliary evaluation and a formulation that translates controller feedback into action-aligned supervision.

MRT estimates progress through changes in the probability of a correct answer under a meta-prover policy. Its open-ended RL implementation samples candidate answers from intermediate reasoning prefixes and verifies their correctness. This procedure depends on an answer-completion interface, whereas purely operational APPWorld tasks require changes to the environment without textual answer submission. An analogous estimator based on completed task outcomes would require continuation rollouts from intermediate states and evaluation of the resulting environment states.

The following table summarizes these requirements. ✓\checkmark indicates that the capability or requirement applies to the reported procedure, and ×\times indicates its absence.

MethodBackbone parameter updatesSeparate progress evaluatorOracle outcome checks for progressCompletion sampling for progressAdditional model calls
Meta-Reasoner×\times✓\checkmark×\times×\times✓\checkmark
MRT✓\checkmark×\times✓\checkmark✓\checkmark✓\checkmark
ASCEND✓\checkmark×\times×\times×\times✓\checkmark

ASCEND directly assesses task-completion prospects from visible interaction histories through the evolving agent. Probability-weighted prefix potentials and neighboring log-potential differences yield step-aligned supervision without an auxiliary evaluator or meta-prover completion sampling. This construction accommodates both stateful interaction and multi-hop reasoning, while retaining additional prefix-scoring computation. Our contribution concerns how visible-prefix assessments are constructed and integrated into learning under the specified information constraints.

We have included Meta-Reasoner [3] and MRT [4] in Section 2 of the revised version and distinguished their supervision requirements from ASCEND.

AuthorsOpenReview ↗

Response Part-3/3

W4. The method is standard post-training in agent scenarios. However, the paper emphasizes the word "evolution" extensively and repeatedly, claiming that it is online adaptation after deployment, but this is not actually the case, so it seems a bit strange.

A: First, we would like to clarify that our research concerns agent evolution, extending adaptation beyond capability preparation before release through repeated interaction and update cycles. In each cycle, the current agent collects trajectories, assesses their visible prefixes, and updates its parameters with the resulting supervision. RL provides a mechanism for parametric updates within this process. Our experiments instantiate these cycles through a finite sequence of GRPO-based parametric updates. ASCEND enables the current agent to construct process supervision from its own visible interaction history and support iterative improvement through step-level supervision construction.

Second, beyond the task-performance improvements on APPWorld and multi-hop reasoning, additional ablations support the contribution of the supervision design within the evaluated evolution settings:

  • Probability-weighted scoring preserves finer distinctions between prefix assessments. ASCEND achieves higher SR and macro-average F1 than Argmax and Direct Scalar variants, supporting this scoring design.
  • Neighboring-prefix differencing translates changes in assessed task-completion prospects into local signals. The improvement over the Linear variant supports the learning benefit of expressing these changes in log space.
  • Suffix aggregation accumulates normalized signals from each action onward to construct process advantages. Improvements over No-suffix, Broadcast, and Shuffle variants support suffix aggregation and step-dependent credit assignment.

We have added the component ablations in Section 5.4, Tables 3-4, of the revised version, with detailed definitions provided in Appendix B.3.


Reference

[1] Feng, Lang, et al. "Group-in-group policy optimization for llm agent training." Advances in Neural Information Processing Systems 38 (2026): 46375-46408.

[2] Zhang, Zijing, et al. "Rlvmr: Reinforcement learning with verifiable meta-reasoning rewards for robust long-horizon agents." International Conference on Learning Representations. Vol. 2026. 2026.

[3] Sui, Yuan, et al. "Meta-reasoner: Dynamic guidance for optimized inference-time reasoning in large language models." Findings of the Association for Computational Linguistics: ACL 2026. 2026.

[4] Yuxiao Qu, et al. "Optimizing Test-Time Compute via Meta Reinforcement Finetuning." Proceedings of the 42nd International Conference on Machine Learning, PMLR 267:50893-50925, 2025.

Reviewer_rfqnOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes ASCEND, which uses an agent’s self-evaluation of task-completion prospects from partial trajectories and combines changes in these estimates with terminal outcome rewards for GRPO training.

Critical strengths
  • This paper addresses an important challenge in agent learning: providing useful feedback for intermediate actions when rewards are mainly available at the end of a task.

  • The proposed method offers a simple way to turn the agent’s own assessments into learning signals, without requiring a separately trained process evaluator or an external teacher.

Critical weaknesses
  • The probability-weighted scoring method is not sufficiently justified. The paper does not compare it with simply using the highest-probability A–F bin or directly generating a score, so the benefit of using the output distribution remains unclear.
  • The human audit supports the association between positive potential changes and progress, but does not validate the magnitude of these changes. It remains unclear whether log-space differences meaningfully reflect how much each action improves the chances of completing the task.
  • Although environmental outcome rewards are available, the paper does not compare self-evaluated potentials with intermediate values estimated from continuation rollouts, as in IPR [1]. Such a comparison would help assess how accurately self-evaluation captures expected task reward.
  • The interaction between process and outcome signals is not sufficiently analyzed. The paper does not report how often they disagree or whether normalization and suffix aggregation lead to advantages that reinforce unhelpful actions or penalize useful ones.
  • APPWorld lacks comparisons with other step-level RL methods. The comparison with outcome-only GRPO shows that adding the proposed signal helps, but does not establish its advantage over other ways of constructing intermediate rewards.

[1] Weimin Xiong et al. Watch Every Step! LLM Agent Learning via Iterative Step-level Process Refinement. EMNLP 2024.

Questions for authors
  • How does probability-weighted scoring compare with taking the highest-probability bin or directly generating a scalar score? An ablation would help justify this design.
  • What evidence supports interpreting larger log-potential changes as greater improvements in task-completion prospects? The current human audit mainly assesses progress discrimination rather than the accuracy of the change magnitudes.
  • Have the authors compared their potentials with expected rewards estimated from multiple continuation rollouts, as in IPR? Even an analysis on a subset of prefixes would help assess the quality of the estimates.
  • How often do the process and outcome signals disagree? When the process term reverses the sign of the final advantage, does this reflect useful credit assignment or an assessment error?
  • Why are comparisons with other intermediate reward methods limited to multi-hop QA? Including such baselines on APPWorld would strengthen the evidence for the proposed method.
9 comments
AuthorsOpenReview ↗

Response Part-1/5

We sincerely thank Reviewer #rfqn for the constructive feedback. Below, we address the concerns with additional experiments and analyses.


W1. The probability-weighted scoring method is not sufficiently justified. The paper does not compare it with simply using the highest-probability A–F bin or directly generating a score, so the benefit of using the output distribution remains unclear.

Q1. How does probability-weighted scoring compare with taking the highest-probability bin or directly generating a scalar score? An ablation would help justify this design.

A: First, we compare probability-weighted scoring with Argmax, which selects the highest-probability A-F bin, and Direct Scalar, which generates a numerical potential. Both retain differencing, normalization, suffix aggregation, and squared terminal feedback.

The ablation results on APPWorld are as follows:

MethodBest SR ↑\uparrowΔ\DeltaSR ↑\uparrowAvg. Steps ↓\downarrowΔ\DeltaSteps ↓\downarrow
Base Model30.35—20.66—
Vanilla GRPO44.64+14.2918.83−1.83
ASCEND (Argmax)48.21+17.8617.63−3.03
ASCEND (Direct Scalar)45.23+14.8817.83−2.83
ASCEND50.00+19.6518.22−2.44

The ablation results on multi-hop reasoning are as follows:

Method2WikiHotpotQAMuSiQueMacro Avg.
EM ↑\uparrowF1 ↑\uparrowEM ↑\uparrowF1 ↑\uparrowEM ↑\uparrowF1 ↑\uparrowEM ↑\uparrowF1 ↑\uparrow
Base Model38.8454.6836.1553.834.3813.0426.4640.52
Vanilla GRPO62.2171.6149.3864.166.0814.6139.2250.13
ASCEND (Argmax)60.8270.3448.5064.076.7415.9538.6950.12
ASCEND (Direct Scalar)62.3573.2448.1464.877.2816.6339.2651.58
ASCEND63.4972.8850.3365.737.0716.5740.3051.73
  • ASCEND achieves the highest APPWorld SR, exceeding Argmax and Direct Scalar by 1.79% and 4.77%.
  • ASCEND also achieves the highest macro-average F1 on multi-hop reasoning, with improvements of 1.61% and 0.15%, respectively.

Second, we further examine scoring resolution across these three settings. Argmax on ASCEND records (offline) recomputes potentials from the recorded ASCEND scoring distributions through highest-probability bin selection, holding both the prefixes and distributions fixed. The metrics are as follows:

  • Top-3 score share measures the frequency of the three most common scores after rounding to two decimal places.
  • Same-score rate measures the proportion of adjacent intermediate prefix pairs whose original-precision scores differ by at most 10−1210^{-12}.
  • Near-zero difference rate measures the proportion of adjacent intermediate prefix pairs with absolute log-potential differences at most 0.020.02, with a potential floor of 0.010.01.

Scoring-resolution results on APPWorld:

MethodTop-3 score shareSame-score rateNear-zero difference rate
Argmax93.09%70.29%70.29%
Direct Scalar75.98%55.95%55.95%
Argmax on ASCEND records (offline)87.60%64.52%64.52%
ASCEND11.00%0.32%19.48%

Scoring-resolution results on 2Wiki:

MethodTop-3 score shareSame-score rateNear-zero difference rate
ASCEND (Argmax)95.31%36.37%36.37%
ASCEND (Direct Scalar)76.77%28.32%28.32%
Argmax on ASCEND records (offline)95.44%35.73%35.73%
ASCEND15.12%0.57%6.49%

(i) Across both settings, ASCEND produces less concentrated scores and fewer tied or near-zero neighboring differences.

(ii) On identical records, probability weighting reduces same-score rates from 64.52% to 0.32% on APPWorld and from 35.73% to 0.57% on 2Wiki.

The paired comparisons show that probability weighting retains finer distinctions between neighboring assessments. Together with the improvements in SR and macro-average F1, these diagnostics support probability-weighted scoring for fine-grained prefix-potential supervision.

We have added the scoring ablations in Section 5.4, Table 3, and the scoring-resolution analysis in Appendix C.6, Figure 5, of the revised version.

AuthorsOpenReview ↗

Response Part-2/5

W2. The human audit supports the association between positive potential changes and progress, but does not validate the magnitude of these changes. It remains unclear whether log-space differences meaningfully reflect how much each action improves the chances of completing the task.

Q2. What evidence supports interpreting larger log-potential changes as greater improvements in task-completion prospects? The current human audit mainly assesses progress discrimination rather than the accuracy of the change magnitudes.

A: First, we interpret a larger positive log-potential change as a larger relative improvement in assessed task-completion prospects. As ASCEND captures this through δi(ℓ)=log⁡max⁡(ω~i(ℓ),ϵ)max⁡(ω~i(ℓ−1),ϵ)\delta_i^{(\ell)}=\log\frac{\max(\tilde{\omega}_i^{(\ell)},\epsilon)}{\max(\tilde{\omega}_i^{(\ell-1)},\epsilon)}, the same absolute increase has different relative significance depending on the preceding assessment. For example, increases from 0.2 to 0.4 and from 0.6 to 0.8 yield log differences of 0.693 and 0.288. The former doubles the assessment and receives a larger signal. This can emphasize intermediate progress that improves a weak completion prospect while further reasoning remains necessary.

Second, we examine learning utility with the Linear variant, which computes δi(ℓ)=max⁡(ω~i(ℓ),ϵ)−max⁡(ω~i(ℓ−1),ϵ)\delta_i^{(\ell)}=\max(\tilde{\omega}_i^{(\ell)},\epsilon)-\max(\tilde{\omega}_i^{(\ell-1)},\epsilon). The numerical floor, normalization, suffix aggregation, terminal processing, and 4-round budget remain the same.

Method2WikiHotpotQAMuSiQueMacro Avg.
EM ↑\uparrowF1 ↑\uparrowEM ↑\uparrowF1 ↑\uparrowEM ↑\uparrowF1 ↑\uparrowEM ↑\uparrowF1 ↑\uparrow
Base Model38.8454.6836.1553.834.3813.0426.4640.52
Vanilla GRPO62.2171.6149.3864.166.0814.6139.2250.13
Linear62.1171.1649.2563.945.3012.8638.8949.32
ASCEND63.4972.8850.3365.737.0716.5740.3051.73

ASCEND achieves higher F1 on all three benchmarks, improving macro-average F1 from 49.32% to 51.73%. The human audit evaluates progress discrimination, while the ablation supports the learning utility of this construction.

We have added the Linear comparison in Section 5.4, Table 4, and specified its construction in Appendix B.3 of the revised version.

AuthorsOpenReview ↗

Response Part-3/5

W3. Although environmental outcome rewards are available, the paper does not compare self-evaluated potentials with intermediate values estimated from continuation rollouts, as in IPR [1]. Such a comparison would help assess how accurately self-evaluation captures expected task reward.

Q3. Have the authors compared their potentials with expected rewards estimated from multiple continuation rollouts, as in IPR? Even an analysis on a subset of prefixes would help assess the quality of the estimates.

A: IPR [1] evaluates alternative actions along expert trajectories through Monte Carlo (MC) continuations and constructs contrastive pairs for iterative refinement. We adapt its MC estimation component as IPR (adapted).

First, we examine same-prefix alignment on 30,791 intermediate prefixes across four rounds of IPR (adapted) on 2Wiki. The same fixed base-model agent provides ASCEND-style probability-weighted self-assessments and 4 continuation rollouts per prefix. The MC estimate is their mean squared terminal F1. The metrics are defined as follows:

  • Spearman correlation measures agreement between the two rankings.
  • Mean MC Return averages MC estimates for prefixes at or below the 25th percentile (Lower) and at or above the 75th percentile (Higher) of self-assessed potential. We report this metric in each group to examine whether higher self-assessments correspond to better continuation outcomes.

Statistics are calculated separately by prefix length and pooled for Overall.

Prefix LengthSpearman ρ\rho ↑\uparrowMean MC Return
LowerHigher
After 1 action0.1630.3320.506
After 2 actions0.2680.2500.444
After 3 actions0.4310.0970.378
After 4 actions0.6410.0300.489
Overall0.2280.2540.440
  • Higher self-assessments correspond to higher mean MC returns at every prefix length, with 0.440 versus 0.254 overall.
  • Overall ranking agreement is modest, with stronger association among longer prefixes in this sample, increasing from 0.163 after one action to 0.641 after four.

Second, we examine task performance by replacing self-assessed potentials with MC estimates, retaining log-space differencing, normalization, suffix aggregation, terminal processing, and the four-round budget. The continuation agent remains fixed, while ASCEND obtains potentials from the evolving agent.

Method2WikiHotpotQAMuSiQueMacro Avg.
EM ↑\uparrowF1 ↑\uparrowEM ↑\uparrowF1 ↑\uparrowEM ↑\uparrowF1 ↑\uparrowEM ↑\uparrowF1 ↑\uparrow
Base Model38.8454.6836.1553.834.3813.0426.4640.52
Vanilla GRPO62.2171.6149.3864.166.0814.6139.2250.13
IPR (adapted)62.1371.2550.2765.366.4515.2239.6250.61
ASCEND63.4972.8850.3365.737.0716.5740.3051.73
  • IPR (adapted) improves macro-average F1 over Vanilla GRPO. Notably, ASCEND achieves higher F1 across all three benchmarks, exceeding the adapted IPR by 1.12 macro-average F1.
  • MC estimation requires 4 continuation rollouts per intermediate prefix, including generation, retrieval, and terminal evaluation. Conversely, ASCEND only requires 1 prefix-scoring call per potential, achieving higher task performance in this comparison while avoiding these additional rollouts.

We have added the same-prefix comparison in Appendix C.7, Table 21, and the IPR (adapted) performance results in Table 2, with the adaptation protocol in Appendix B.2 of the revised version.

Reviewer_g8r9OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

this paper constructs process supervision reward for multi-turn agents by turn-level rubrics judge, in a separate scoring call, to assess task-completion prospects from each turn. Adjacent log-potential differences are clipped, normalized, accumulated into step-dependent returns, and combined with terminal outcome advantages for PPO/GRPO-style updates. The method avoids a separately trained process evaluator and does not use reference answers to construct its prefix scores, although training still uses environmental outcome feedback and task-specific scoring rubrics. The main contribution is a practical implementation of prefix-based self-assessment under restricted information and eval-train requirements.

Critical strengths
  1. A useful reduction in the requirements for process supervision. The method uses the executing model and only the currently visible prefix, avoiding separate process-reward training, answer likelihoods, or repeated-state matching. This is particularly relevant to state-changing interaction tasks such as AppWorld, where answer-based process rewards do not transfer directly. The paper and supplementary implementation describe a complete path from prefix assessment to parameter updates.

  2. An attempt to validate the supervision itself, across distinct interaction settings. Beyond final task performance, the paper studies outcome association and independently assessed local progress in both tool interaction and multi-hop retrieval. Hiding future trajectories and final outcomes from the local progress audit is a useful design choice. These analyses provide preliminary evidence that the scores contain task-relevant information, while not establishing the reliability of every negative signal or of the final normalized advantages.

Critical weaknesses
  1. the central QA comparison is not currently reproducible as a controlled test of process supervision. Appendix B.2 states that GRPO and ASCEND apply the same squared transformation to terminal rewards. However, the supplied QA configurations set outcome_reward_pow_2 to true for ASCEND and false for GRPO, and this option is used by the reward pipeline. Squaring is not innocuous under group normalization: it changes relative advantages and can even change their signs. Thus, either the released configurations differ from those used for the reported experiments, or the comparison changes both process supervision and the terminal objective. Given the macro-average gain of 1.60 F1 points, this ambiguity materially limits attribution of the improvement. Comparisons with methods using self-induced rather than reference-answer-based terminal supervision do not resolve it.

  2. the AppWorld evaluation appears to use the reported test set for adaptive model selection. The manuscript discloses repeated evaluation on test_normal and best-round reporting. The supplied evolution path additionally uses these metrics to promote or restore global_best, which determines the starting checkpoint of the next round. If this is the experimental path, the same set influences both training continuation and final reporting; it is therefore not an independent test of generalization. Applying the same selection protocol to GRPO does not remove this limitation. Original checkpoint-selection records and evaluation on a set that did not influence selection are needed to establish the principal AppWorld gain. This concern is about adaptive selection, not an allegation that test answers were placed in the scoring prompt.

Questions for authors
  1. Which terminal-reward transformation was used for the runs behind Tables 1, 2 and 12? Please reconcile Appendix B.2 (the same squared transformation) with config/baselines_4gpu/README.md and the outcome_reward_pow_2 values in the released yaml.

  2. Is there a comparison using the same judge and core rubric to score the complete visible trajectory once, combining that score with the same terminal feedback? I think it's more compelling to matching terminal-reward processing and training budgets

  3. I understand that adjacent potential differences form the immediate process signals, followed by suffix accumulation. Holding the prefix scores and process signals fixed, how does this compare with using only the current step’s difference, or aggregating the process signal and broadcasting it to the trajectory? These controls would isolate the value of the step-dependent assignment. The already-cited SIOP provides a related step-wise versus trajectory-broadcast ablation in Section 4.4/Table 3.

  4. pp. 31–32 contains no regression examples. For AppWorld no-change steps, the reported 80% non-reward rate and 5% neutral rate imply that approximately 75% receive a negative local signal. What do these negatives represent—new information about remaining risk, genuine harmful actions, or scoring variability?

If the authors can satisfactorily resolve the two inconsistencies identified in my weaknesses section and provide additional experimental evidence addressing one or two of questions 2–4, demonstrating that the differential prefix-scoring mechanism itself contributes to the observed improvements through well-controlled ablation studies, I would be willing to raise my score.

6 comments
AuthorsOpenReview ↗

Response Part-1/4

We sincerely thank Reviewer #g8r9 for the careful review. Below, we provide point-by-point responses.


W1. The central QA comparison is not currently reproducible as a controlled test of process supervision. Appendix B.2 states that GRPO and ASCEND apply the same squared transformation to terminal rewards. However, the supplied QA configurations set outcome_reward_pow_2 to true for ASCEND and false for GRPO, and this option is used by the reward pipeline. Squaring is not innocuous under group normalization: it changes relative advantages and can even change their signs. Thus, either the released configurations differ from those used for the reported experiments, or the comparison changes both process supervision and the terminal objective. Given the macro-average gain of 1.60 F1 points, this ambiguity materially limits attribution of the improvement. Comparisons with methods using self-induced rather than reference-answer-based terminal supervision do not resolve it.

Q1. Which terminal-reward transformation was used for the runs behind Tables 1, 2 and 12? Please reconcile Appendix B.2 (the same squared transformation) with config/baselines_4gpu/README.md and the outcome_reward_pow_2 values in the released yaml.

A: First, all reported GRPO and ASCEND runs in Tables 1, 2, and 14 in the revised version (Tables 1, 2, and 12 in the initial version) adopt the setting outcome_reward_pow_2: true. We apologize for uploading an outdated GRPO configuration and an inconsistent README, which we will correct.

Second, since squaring affects group-normalized advantages, we additionally evaluate GRPO without squaring under the same evolution budgets.

The ablation results on APPWorld are as follows.

MethodEvolution TypeFeedback UtilizationSquared terminal outcomeBest SR ↑\uparrowΔ\DeltaSR ↑\uparrowAvg. Steps ↓\downarrowΔ\DeltaSteps ↓\downarrow
Base ModelNone——30.35—20.66—
Vanilla GRPOParametricOutcome supervision✓\checkmark44.64+14.2918.83−1.83
Vanilla GRPOParametricOutcome supervision×\times44.04+13.6921.23+0.57
ASCENDParametricOutcome + prefix-potential supervision✓\checkmark50.00+19.6518.22−2.44

The ablation results on multi-hop reasoning are as follows:

MethodSquared terminal outcome2WikiHotpotQAMuSiQueMacro Avg.
EM ↑\uparrowF1 ↑\uparrowEM ↑\uparrowF1 ↑\uparrowEM ↑\uparrowF1 ↑\uparrowEM ↑\uparrowF1 ↑\uparrow
Base Model—38.8454.6836.1553.834.3813.0426.4640.52
Vanilla GRPO✓\checkmark62.2171.6149.3864.166.0814.6139.2250.13
Vanilla GRPO×\times61.2070.9447.4863.246.5015.9338.3950.04
ASCEND✓\checkmark63.4972.8850.3365.737.0716.5740.3051.73

(i) On APPWorld, squaring increases GRPO SR from 44.04% to 44.64% and reduces Avg. Steps from 21.23 to 18.83.

(ii) On multi-hop reasoning, GRPO macro-average F1 changes from 50.04% to 50.13%.

With matched squared terminal outcome rewards, ASCEND exceeds Vanilla GRPO by 5.36% SR on APPWorld and 1.60% macro-average F1 on multi-hop reasoning, supporting the contribution of prefix-potential supervision.

We have added the terminal-outcome squaring ablations in Appendix C.5, Tables 19-20, of the revised version.

AuthorsOpenReview ↗

Response Part-2/4

W2. The AppWorld evaluation appears to use the reported test set for adaptive model selection. The manuscript discloses repeated evaluation on test_normal and best-round reporting. The supplied evolution path additionally uses these metrics to promote or restore global_best, which determines the starting checkpoint of the next round. If this is the experimental path, the same set influences both training continuation and final reporting; it is therefore not an independent test of generalization. Applying the same selection protocol to GRPO does not remove this limitation. Original checkpoint-selection records and evaluation on a set that did not influence selection are needed to establish the principal AppWorld gain. This concern is about adaptive selection, not an allegation that test answers were placed in the scoring prompt.

A: First, our APPWorld experiment evaluates closed-loop adaptation, with the test_normal split serving as a proxy for the post-deployment operating environment. Its feedback intentionally guides agent-instance selection as part of the evolutionary strategy, determining the starting instance for subsequent rounds. The results therefore measure adaptation under repeated access to this split, distinct from generalization to an independent held-out APPWorld set.

Second, each round starts from the retained best agent instance. After candidate evaluation, the best instance is retained for subsequent updates. The table below summarizes this selection procedure for the reported ASCEND run on APPWorld, where A0A_0 denotes the initial agent instance and AtA_t the candidate produced in round tt.

Evolution RoundStarting InstanceCandidate SRRetained Best Instance
1A0A_041.66A1A_1
2A1A_143.45A2A_2
3A2A_248.21A3A_3
4A3A_350.00A4A_4
5A4A_448.21A4A_4
6A4A_449.40A4A_4
7A4A_448.80A4A_4
8A4A_447.61A4A_4

ASCEND retains A4A_4 after reaching 50.00% SR, with subsequent rounds starting from this instance. The evolution curve in Figure 3 reports candidate performance before selection, while the best-round result reports the retained instance. We have clarified the evaluation scope and agent-instance selection procedure in Section 5.1 and Appendix B.5 of the revised version (Appendix B.4 in the initial version).

AuthorsOpenReview ↗

Response Part-3/4

Q2. Is there a comparison using the same judge and core rubric to score the complete visible trajectory once, combining that score with the same terminal feedback? I think it's more compelling to matching terminal-reward processing and training budgets

A: Thank you for the helpful suggestion. We compare two full-trajectory scoring variants with ASCEND, retaining the same judge, core rubric, and squared terminal outcome feedback, with an 8-round evolution budget on APPWorld and a 4-round budget on multi-hop reasoning. The variants are detailed as follows:

  • Single Score produces one full-trajectory score, group-normalizes it, and broadcasts it across steps alongside terminal feedback.
  • Process Scores produces potentials for all positions in one call with the full trajectory visible, followed by the same differencing, normalization, and suffix aggregation as ASCEND.

Results on APPWorld:

MethodBest SR ↑\uparrowΔ\DeltaSR ↑\uparrowAvg. Steps ↓\downarrowΔ\DeltaSteps ↓\downarrow
Base Model30.35—20.66—
Vanilla GRPO44.64+14.2918.83−1.83
Full-trajectory scoring (Single Score)48.21+17.8618.39−2.27
Full-trajectory scoring (Process Scores)42.26+11.9119.59−1.07
ASCEND50.00+19.6518.22−2.44

Results on multi-hop reasoning:

Method2WikiHotpotQAMuSiQueMacro Avg.
EM ↑\uparrowF1 ↑\uparrowEM ↑\uparrowF1 ↑\uparrowEM ↑\uparrowF1 ↑\uparrowEM ↑\uparrowF1 ↑\uparrow
Base Model38.8454.6836.1553.834.3813.0426.4640.52
Vanilla GRPO62.2171.6149.3864.166.0814.6139.2250.13
Full-trajectory scoring (Single Score)62.4472.0049.3565.016.8716.4039.5551.14
Full-trajectory scoring (Process Scores)62.4571.7249.8365.216.6615.7039.6550.88
ASCEND63.4972.8850.3365.737.0716.5740.3051.73

(i) On APPWorld, ASCEND exceeds Single Score and Process Scores by 1.79% and 7.74% SR, with 0.17 and 1.37 fewer average execution steps, respectively.

(ii) On multi-hop reasoning, ASCEND achieves higher F1 across all three benchmarks, improving macro-average F1 over Single Score and Process Scores by 0.59% and 0.85%, respectively.

These comparisons support assessing each prefix from the interaction evidence available at that step for supervision construction across both evaluated settings.

We have added these comparisons in Section 5.4, Table 3, of the revised version, with variant settings provided in Appendix B.3.

Comments on the paper

1 comment
AuthorsOpenReview ↗

To ALL Reviewers

General Response To All Reviewers


Dear reviewers,

Thank you for your time and the detailed, constructive feedback, which has helped strengthen our work. We are fully committed to incorporating your suggestions into the revision. At this juncture, we would like to re-emphasize two contributions of our work on supervision construction for agent evolution under explicit information constraints:

  • Develop ASCEND to construct fine-grained supervision from the evolving agent and visible interaction prefixes, without separately trained process evaluators or task-specific reference structures.
  • Demonstrate learning benefits across stateful interaction and multi-hop reasoning, supported by cross-benchmark transfer and analyses of outcome and step-level alignment.

Also, here we would like to address some common concerns raised by the reviewers:

Concern 1: Contribution and distinction from existing methods.

A: First, we clarify how the current agent constructs supervision from visible interaction histories for iterative improvement, with GRPO providing the parametric update mechanism. Second, we expanded Related Work to distinguish ASCEND from dependencies such as anchor-state matching in GiGPO and auxiliary progress evaluation in Meta-Reasoner.

Concern 2: Effectiveness and generality across settings.

A: First, to strengthen comparisons with other intermediate-supervision methods, we conducted additional experiments with GiGPO in APPWorld and multi-hop reasoning and an MC-adapted IPR on multi-hop reasoning. ASCEND achieves higher APPWorld SR and multi-hop reasoning macro-average F1 in these comparisons. Second, we clarify the complementary evaluation protocols: experiments on APPWorld examine closed-loop adaptation within a deployment proxy, while evaluations on HotpotQA and MuSiQue assess cross-benchmark transfer after evolution on 2Wiki, without further adaptation.

Concern 3: Design justification and informativeness of supervision.

A: First, to examine the contribution of individual design choices, we conducted additional ablations on full-trajectory scoring, probability weighting, log-space differencing, and step-dependent credit assignment. The results consistently favor ASCEND on the reported SR and macro-average F1 comparisons. Second, to assess supervision signal quality, we added paired scoring diagnostics, same-prefix MC comparisons, and process-outcome analyses linked to the existing human audit. These analyses show finer scoring resolution, positive alignment with MC continuation estimates, and examples of progress-consistent credit assignment, complementing the task-performance evidence.


Again, we express our gratitude for the thoughtful reviews. During the rebuttal, we conducted additional experiments and analyses, incorporated the results and clarifications into the revised manuscript, and prepared the detailed responses below. We hope our responses address your concerns and support a positive reassessment of our work. We look forward to your feedback and remain available for further discussion.

Best regards,

Authors of #18523

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.