acceptodds
Under review as a conference paper at ICLR 2027

ProcEvo: Self-Evolving Multi-Agent Orchestration via Process-Guided Attribution

Abstract

LLM-based multi-agent systems increasingly rely on learnable orchestration to coordinate specialized agents for complex tasks. Self-evolving orchestration further seeks to improve the orchestration policy from accumulated execution experience, making effective credit assignment over past trajectories critical. However, a single task- or trajectory-level outcome cannot distinguish the heterogeneous effects of individual agent transitions: the same execution may contain useful, redundant, and harmful transitions. Inspecting local state changes provides finer-grained information, but a state change alone does not reveal whether a transition constitutes meaningful progress, since repetition or unsupported reasoning may also alter the execution state. Therefore, we propose ProcEvo, a framework for self-evolving multi-agent orchestration that performs transition-level process attribution. ProcEvo first compares the task states before and after each agent invocation to estimate its local effect. It then uses an external LLM to extract evidence for a checklist instantiated with the task, applying relevance, consistency, support, and advancement to assess the evidence. The resulting process evidence is converted into transition-level process credit and combined with terminal supervision to optimize the learnable orchestrator. Across evaluated subsets of MMLU-Pro, GSM-Hard, and GPQA Main, ProcEvo achieves an average accuracy of 72.76%, compared with 67.47% for the highest-averaging competing baseline. Component ablations and process-level analyses further highlight the distinct roles of progress estimation by comparing task states and checklist verification. The implementation is available in the supplementary anonymous repository: https://anonymous.4open.science/r/ProcEvo-2EC7

Primary area:
foundation or frontier models, including LLMs
Keywords:
Large Language Models, Multi-Agent Systems, Agent Orchestration, Process-Guided Learning, Reinforcement Learning

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_c4SnOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes ProcEvo, a framework for training a sequential multi-agent orchestrator with transition-level process supervision. It combines state-quality differences (Skywork-Reward-V2) with checklist-based evidence verification using an external LLM, then incorporates these signals into PPO alongside terminal rewards. Experiments on sampled subsets of MMLU-Pro, GSM-Hard, and GPQA Main report improvements over several orchestration and multi-agent baselines, supported by component ablations and diagnostics.

Critical strengths
  • I find the motivation clear and well illustrated. Terminal rewards conflate useful, redundant and harmful transitions, and raw state change does not imply progress.
  • I like the design principle behind the checklist reducer. It separates evidence extraction from credit assignment and explicitly suppresses self-certification.
  • I appreciate that the authors report variance over five independent runs and per-comparison p-values, including non-significant ones.
Critical weaknesses
  • I am concerned that the state-progress signal, presented as transition-level, is partly trajectory-level. In Eq. 19, each step’s state progress is scaled by a factor based on how well its whole trajectory scored relative to the other rollouts for the same task. As a result, two steps with identical local progress can receive different credit depending on how their trajectories performed. Because the same factor scales negative progress, a harmful step also receives a smaller penalty in a poorly performing trajectory. This seems inconsistent with the goal of identifying which individual steps help or hurt progress.
  • The evidence-reduction procedure is underspecified because several key decisions rely on LLM judgments, such as whether sources are independent, tool results are trustworthy, or new evidence genuinely supports a claim. Given this, I have further concerns about what the checklist signal capture below.
  • All executable agents share the GPT-4.1-mini backbone and differ only in role prompts and actions, so their errors are likely correlated. Agent-to-agent support may therefore be a weaker form of the self-certification.
  • 36.4% of credit events come from trusted tool, thus, the signal may partly reward tool use rather than verified progress.
  • The evaluator (DeepSeek-V3.2) may be stronger than the executor (GPT-4.1-mini), so some gains may come from transferring the stronger model’s judgments.
Questions for authors
  • How do the results change if the σ(z_j) trajectory gating in Eq. 19 is removed, so that state progress is used without trajectory-level scaling?
  • Is the DeepSeek-V3.2 evaluator used only during training, or also at test time? How do results change if GPT-4.1-mini is used as the evaluator?
  • How do agent-selection frequencies change during training?
6 comments
AuthorsOpenReview ↗

Response to Reviewer c4Sn - Part1

We appreciate your detailed and constructive analysis of the proposed process signals and orchestration policy. Your questions offer an opportunity to clarify the distinction between local progress estimation and trajectory-aware optimization, as well as the safeguards used for evidence credit. We respond to each concern below and supplement the original explanations with targeted ablations and trajectory-level analyses.

Q1

Reviewer — Critical Weakness W1: I am concerned that the state-progress signal, presented as transition-level, is partly trajectory-level. In Eq. 19, each step’s state progress is scaled by a factor based on how well its whole trajectory scored relative to the other rollouts for the same task. As a result, two steps with identical local progress can receive different credit depending on how their trajectories performed. Because the same factor scales negative progress, a harmful step also receives a smaller penalty in a poorly performing trajectory. This seems inconsistent with the goal of identifying which individual steps help or hurt progress.

Reviewer — Question 1: How do the results change if the σ(zj)\sigma(z_j) trajectory gating in Eq. 19 is removed, so that state progress is used without trajectory-level scaling?

A1: Thank you for carefully examining Eq. (19). We would like to clarify the distinction between estimating local progress and weighting that estimate during policy learning. Eq. (5) defines the local State Progress measurement ptp_t from adjacent task states and the historical best. Eq. (19) does not change that measurement: it scales the signal's influence on policy optimization, p~t=σ(zj)pt\tilde{p}_t = \sigma(z_j)p_t, based on the relative performance of sampled trajectories for the same question. Equal local progress therefore remains equal at the measurement stage, even if the orchestrator learns from it with different weights.

Why calibrate that weight? A state-quality estimate provides evidence of local improvement, but not of whether the corresponding agent choice is a useful use of the overall workflow budget. A step that appears locally favorable inside a long, unsuccessful rollout should not necessarily shape the policy as strongly as one embedded in a productive trajectory. Eqs. (18)–(19) incorporate step-level feedback, terminal outcome, and trajectory length to calibrate an auxiliary learning signal. Task-relative comparisons across sampled outcomes are also used in group-relative policy optimization such as GRPO [1]. The distinction is between identifying local change and deciding how heavily to optimize for it, not between transition supervision and learning from terminal outcomes alone.

As noted in the review, the positive gate factor also attenuates negative local progress in lower-scoring trajectories. This is a deliberate weighting choice rather than an attempt to redefine a harmful step as beneficial: multiplication by a positive factor preserves the sign of the local estimate. The terminal reward also remains in the GAE-based policy update under Eqs. (9)–(10). Negative outcomes therefore still affect the policy independently of the auxiliary State Progress weight. Calibrating both positive and negative estimates limits the contribution of potentially noisy local judgments from poor trajectories rather than requiring that every such estimate receive equal optimization weight.

The requested ablation directly tests whether this trajectory context is useful:

Table 1. GSM-Hard trajectory-gating ablation. Accuracy is reported as mean ± SD over five Full runs and three ablation runs, using 100 validation questions per run.

ConfigurationVal accuracy
Full (with trajectory gating)77.20 ± 0.84%
Without trajectory gating72.67 ± 1.53%

Removing the trajectory gate lowers mean accuracy by 4.53 percentage points. This supports trajectory-aware calibration while retaining the same local progress estimator. The results are also included in revised Appendix A.9, Table 9.

Reference
[1] Shao et al., DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models, 2024.

AuthorsOpenReview ↗

Response to Reviewer c4Sn - Part2

Q2

Reviewer — Critical Weakness W2: The evidence-reduction procedure is underspecified because several key decisions rely on LLM judgments, such as whether sources are independent, tool results are trustworthy, or new evidence genuinely supports a claim. Given this, I have further concerns about what the checklist signal capture below.

A2: Thank you for this question. The distinction between extracting candidate evidence and deterministically granting credit is central to our design. ProcEvo does not rely on an LLM's unconstrained decision to assign reward. The LLM extracts structured candidate claims, source provenance, and support. The deterministic reducer tracks checklist-item maturity and applies eligibility, scope, contradiction, duplication, and evidence-source rules described in Section 3.3 and Appendix B.2–B.5. Only a net eligible increase in checklist potential yields positive Checklist Credit under Eqs. (7)–(8). A self-confirming assertion is not automatically independent verification.

The key distinction is that an LLM-proposed support relation is not itself a reward decision: positive credit requires an eligible evidence-maturity increase under the explicit reducer rules. To examine how this distinction operates on actual trajectories, we added the 80-transition external-content assessment reported in eZM1, A3. Under the unified review criterion, 29/40 positive-credit transitions were supported by independently examined new evidence, including 24/26 Python-supported cases under a conservative standard. The assessor evaluated the transition content rather than taking the extractor's labels as the reference.

Q3

Reviewer — Critical Weakness W3: All executable agents share the GPT-4.1-mini backbone and differ only in role prompts and actions, so their errors are likely correlated. Agent-to-agent support may therefore be a weaker form of the self-certification.

A3: Thank you for raising the possibility of correlated mistakes. The homogeneous GPT-4.1-mini executor pool serves as an experimental control. Holding the task-solving backbone fixed helps attribute performance differences to agent selection and process supervision rather than mixed model capacity, as detailed in Appendix A.5. The reducer evaluates evidence provenance and novelty rather than treating cross-role agreement as independent confirmation. Repeated agreement among similarly prompted agents does not itself advance checklist maturity without eligible supporting evidence.

We also evaluated a heterogeneous execution-agent pool with DeepSeek-V3.2 for the main reasoning roles, Gemini 2.5 Flash for Critic and Reflect, and GPT-4.1-mini for the other roles. The process-supervision mechanism remained unchanged. This configuration achieved 81% on the 100-question GSM-Hard validation subset. Separately, replacing the shared execution backbone with Gemini 2.5 Flash yielded 85% for ProcEvo versus 76% for Puppeteer on the same 100-question evaluation. These results show that the method also operates beyond a homogeneous GPT-4.1-mini pool. Checklist Credit still depends on new, eligible evidence, not on agreement between agents alone.

Q4

Reviewer — Critical Weakness W4: 36.4% of credit events come from trusted tool, thus, the signal may partly reward tool use rather than verified progress.

A4: Thank you for raising this concern. The 36.4% in Figure 4(d) measures the share of evidence-maturity promotions sourced from trusted tools, not the share of tool calls receiving reward. Under Eq. (8), positive credit requires an eligible increase in evidence maturity. Invoking a tool alone does not earn credit.

Table 2. Trusted-tool executions versus positive Checklist Credit in logged GSM-Hard training transitions.

DiagnosticObserved count
Tool-invoking transitions with zero positive Checklist Credit429/571 (75.1%)
Successful tool invocations with zero positive Checklist Credit215/352 (61.1%)
Exact repeated evidence on the same item suppressed13/13

These observations show that calling a tool alone is insufficient to earn reward. As a further independent content check, 24/26 positive-credit Python steps had valid supporting evidence under the conservative external audit. The mechanism rewards what a tool establishes, not the fact that a tool was invoked.

AuthorsOpenReview ↗

Response to Reviewer c4Sn - Part3

Q5

Reviewer — Critical Weakness W5: The evaluator (DeepSeek-V3.2) may be stronger than the executor (GPT-4.1-mini), so some gains may come from transferring the stronger model’s judgments.

Reviewer — Question 2: Is the DeepSeek-V3.2 evaluator used only during training, or also at test time? How do results change if GPT-4.1-mini is used as the evaluator?

A5: Thank you for suggesting the most direct way to separate a strong training evaluator from a strong executing model. DeepSeek-V3.2 is used to extract training evidence. It is not called to solve tasks during evaluation. The same is true of the local Skywork state scorer. At test time, the trained orchestrator selects the task-executing agents. Evaluator calls do not contribute to validation answers. The corresponding training API expenditures appear in eZM1, A2.

We additionally replaced the evidence evaluator with GPT-4.1-mini and retrained ProcEvo:

Table 3. GSM-Hard evidence-evaluator ablation. Accuracy is reported as mean ± SD over five Full runs and three ablation runs, using 100 validation questions per run.

Evidence evaluatorVal accuracy
DeepSeek-V3.2 (Full)77.20 ± 0.84%
GPT-4.1-mini (weaker evaluator)75.67 ± 2.08%

With GPT-4.1-mini as the evidence evaluator, ProcEvo reaches 75.67%, 1.53 percentage points below the submitted Full mean. The process-supervision benefit remains observable with a weaker evidence evaluator. The revised Appendix A.9, Table 9, includes this comparison.

Q6

Reviewer — Question 3: How do agent-selection frequencies change during training?

A6: We appreciate the request for an observable policy-level check. The learned policy selects the next agent conditional on the evolving task state, and its selections change over training. We analyzed 1,781 recorded selections from the GSM-Hard run:

Table 4. Agent-selection changes between the first and last 25-question stages of logged GSM-Hard training.

Policy behaviorFirst stageLast stage
Terminator selection share5.1%16.0%
Retrieval-oriented agent selection share29.2%12.5%
Reasoning + Reflect selection share16.2%23.2%
Mean trajectory length4.694.01

These are changes in actual chosen actions, not merely changed evaluator scores. Increased termination and reduced low-value retrieval, combined with a shorter average workflow, are consistent with a learned policy that makes more selective agent-calling decisions. This behavioral evidence complements the measured final-accuracy improvements.

We hope the mechanism clarification, targeted ablations, and logged agent/tool statistics address the concerns about trajectory gating and credit reliability. We thank the reviewer again for the detailed suggestions and welcome further discussion.

Reviewer_Qk6FOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes ProcEvo, a self-evolving multi-agent orchestration framework that assigns process-level credit to individual agent transitions. It combines state-quality differences with checklist-based evidence verification, using these signals alongside terminal rewards to train an orchestrator with PPO. On MMLU-Pro, GSM-Hard, and GPQA Main, ProcEvo achieves an average accuracy of 72.76%, outperforming the strongest baseline at 67.47%. Ablations and transition-level analyses further investigate the complementary roles of its two process signals.

Critical strengths
  • I like the motivation of separating state changes from meaningful progress. A transition can introduce new content without actually improving the solution, and terminal rewards cannot distinguish useful and harmful contributions within a failed trajectory. This is a concrete credit-assignment problem that the paper addresses directly.
  • The experiments provide useful evidence beyond final accuracy. In particular, the observation that only 13.9% of transitions with positive state progress receive verified credit (Figure 3) shows that the two signals capture substantially different behaviors. The ablation results also support their complementary value.
Critical weaknesses
  • I'm concerned about whether the credit is assigned to the right agent. The checklist rewards the transition that increases evidence maturity, which may be a later verification step rather than the agent that originally produced the useful reasoning. This could encourage the orchestrator to favor verification agents over agents that actually generate solutions.
  • The method may reward repeated recovery and verification without establishing whether they are worthwhile. Equation 5 assigns positive credit to local quality improvements even when they merely recover a previously reached state. Meanwhile, checklist verification can reward later confirmation of existing evidence. I would like to see whether these mechanisms produce more productive workflows or simply encourage additional reasoning and checking steps. The current experiments focus on accuracy without adequately exposing this trade-off.
Questions for authors
  • When one agent proposes a correct intermediate result and another agent verifies it, which agent actually receives credit? Have you considered distributing credit backward to the original contributor?
  • Does the historical-best term in Equation 5 sufficiently discourage regression–recovery cycles? Could you analyze how often the trained orchestrator revisits previously achieved solution quality?
4 comments
AuthorsOpenReview ↗

Response to Reviewer Qk6F - Part1

Thank you for your careful assessment of our credit-assignment mechanism. We appreciate your attention to whether the original contributor receives appropriate learning credit and whether later verification or regression–recovery cycles could lead to unnecessary actions. We address these issues using the reward definitions, GAE-based optimization, and observations from recorded training trajectories.

Q1

Reviewer — Critical Weakness W1: I'm concerned about whether the credit is assigned to the right agent. The checklist rewards the transition that increases evidence maturity, which may be a later verification step rather than the agent that originally produced the useful reasoning. This could encourage the orchestrator to favor verification agents over agents that actually generate solutions.

Reviewer — Question 1: When one agent proposes a correct intermediate result and another agent verifies it, which agent actually receives credit? Have you considered distributing credit backward to the original contributor?

A1: Thank you for raising the important distinction between producing a useful result and subsequently strengthening its evidence. ProcEvo assigns direct process credit to the transition where the corresponding contribution occurs, not automatically to the last agent or to every verifier. An agent that constructs a useful intermediate result can receive State Progress for improving the solution state. A later check receives Checklist Credit only when it adds eligible, new supporting evidence and advances checklist maturity under Eqs. (7)–(8). Agreement or repetition alone does not suffice. The reward rules are specified in Eq. (5) and Appendix B.3–B.5.

Why not assign the verifier's direct evidence gain retrospectively to the proposer? The two actions are complementary but not interchangeable: a plausible proposal is valuable, yet independently verifying it is a separate achievement. Automatically moving the verification reward backward would blur the boundary between having a candidate answer and having evidence that it is justified. ProcEvo instead learns the effect of earlier decisions through GAE [1], which propagates later returns into the advantages of earlier orchestration actions under Eqs. (9)–(10). Immediate evidence credit remains with the transition that produced it.

A logged GSM-Hard telephone-pricing trajectory illustrates this distinction. Reasoning correctly derives $20 and receives +0.162 State Progress, with no direct Checklist Credit. A subsequent Critic checks the two pricing segments and receives +0.00035 Checklist Credit for new supporting evidence. The revised Appendix C.3, Table 13, reports both transitions and their corresponding credit. Across 1,781 training selections, Critic and Reflect account for 19.5% of selected actions but 14.0% of direct Checklist reward, providing no evidence of an automatic allocation preference for verification roles.

Reference
[1] Schulman et al., High-Dimensional Continuous Control Using Generalized Advantage Estimation, ICLR 2016.

AuthorsOpenReview ↗

Response to Reviewer Qk6F - Part2

Q2

Reviewer — Critical Weakness W2: The method may reward repeated recovery and verification without establishing whether they are worthwhile. Equation 5 assigns positive credit to local quality improvements even when they merely recover a previously reached state. Meanwhile, checklist verification can reward later confirmation of existing evidence. I would like to see whether these mechanisms produce more productive workflows or simply encourage additional reasoning and checking steps. The current experiments focus on accuracy without adequately exposing this trade-off.

Reviewer — Question 2: Does the historical-best term in Equation 5 sufficiently discourage regression–recovery cycles? Could you analyze how often the trained orchestrator revisits previously achieved solution quality?

A2: Distinguishing a useful recovery from repeatedly re-earning the same reward is central to efficient orchestration. Eq. (5) deliberately keeps these separate. Its local-difference term allows a policy to learn from recovery after a mistake, while the historical-best bonus is triggered only when the new quality exceeds the earlier peak by margin δ=0.02\delta=0.02. A return to the old peak does not receive the historical-best increment. This matters for the cases in Figure 3 and Appendix C.3, where a failed trajectory can still contain a valuable intermediate step.

Likewise, a later verification may legitimately increase evidence maturity, but repeatedly presenting the same evidence does not continually increase the checklist potential. The reducer checks eligible novelty, scope, source, and maturity, as specified in Appendix B.3–B.5. In the logged tool subset, all 13 identified exact repeats on the same item were assigned zero additional item-level credit. Thus, ProcEvo incentivizes new support, not an unbounded sequence of confirmations.

We examined recovery behavior directly in the recorded training trajectories:

Table 1. Regression, recovery, and workflow behavior in logged ProcEvo training trajectories on GSM-Hard.

DiagnosticObservation
Trajectories that exceed the prior quality peak by 0.02 after regression91/333, or 27.3%
Trajectories with at least two recoveries but no subsequent new peak44/333 (13.2%)
Steps strictly revisiting a previously achieved quality level57/1,445 (3.9%)
Mean trajectory length, first → last 25-question training stage4.69 → 4.01 steps

The strict revisits are uncommon, and the orchestrator's trajectories become shorter, not longer, across the logged training stages. Together with the accuracy gains in Tables 1–2 of the paper, these observations support the intended behavior: the policy can recover from mistakes while the historical-best term and evidence-maturity rules discourage cyclic re-earning of the same progress.

We hope the explicit credit-allocation rules and the observed recovery and workflow statistics clarify how ProcEvo balances generating new solutions, verifying evidence, and avoiding redundant actions. We appreciate the opportunity to make these design choices more transparent.


AuthorsOpenReview ↗

We Are Anticipating Your Feedback!

Dear Reviewer Qk6F,

We appreciate your thoughtful review of our submission and the time you have taken to provide us with your feedback. We have carefully considered all of your comments and suggestions and addressed them accordingly in our responses.

As the discussion period approaches its end, we would like to kindly request your feedback on our rebuttal. We would greatly appreciate it if you could review our responses and let us know whether we have adequately addressed your concerns. Any further comments and discussions are sincerely appreciated!

Thank you for your consideration and valuable time.

Sincerely,

Authors

Thank you for your response. Most of my concerns are addressed and I would like to maintain my original positive score.

Reviewer_eZM1OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

ProcEvo learns a sequential agent-selection policy from execution trajectories using PPO. It combines terminal correctness feedback with two forms of transition-level supervision: state-progress estimates from a frozen reward model, and checklist-based evidence accumulated by a deterministic reducer after extraction by an external LLM. Checklist gains become bounded step rewards, while calibrated state progress contributes an auxiliary advantage. The paper evaluates the system on fixed subsets of MMLU-Pro, GSM-Hard and GPQA Main, reports five-run comparisons with six baselines, and studies the two supervision components through a factorial ablation and descriptive trajectory analyses.

Critical strengths
  1. The supervision design addresses a meaningful distinction in multi-agent reasoning. State changes alone can reward repetition or unsupported assertions. Sections 3.3–3.4 instead combine a state-progress signal with evidence maturity, explicitly exclude final-answer correctness from checklist process credit, and suppress self-certification. This makes the proposed feedback more specific than simply attaching a terminal label to every invocation.
  2. The controlled component ablation provides useful evidence beyond the external-system comparison. Table 1 reports the highest mean accuracy on all three evaluated subsets, with a three-benchmark average of 72.76% versus 67.47% for AOrchestra. More importantly for attribution, Table 2 holds the task instances, agent configuration and optimization settings fixed: on GSM-Hard, the full system achieves 77.20%, compared with 71.80% without state progress, 70.40% without checklist verification, and 68.60% without both. The five-run statistics support the contribution of the combined supervision within this setting.
Critical weaknesses
  1. The experimental scope and resource accounting limit the strength of the main performance claim. Appendix A.1 uses only 70 MMLU-Pro, 100 GSM-Hard and 100 GPQA Main evaluation questions, and the task-executing agents use GPT-4.1-mini throughout. Appendix A.3 appropriately treats runs as the unit of its Welch tests, but this measures variation on a fixed question set rather than uncertainty over benchmark instances. Moreover, Appendix A.2 explicitly preserves different baseline prompts, agent pools and calling patterns, and the paper does not report a matched resource comparison including the extra evaluators. The factorial ablation mitigates the attribution concern on GSM-Hard, but the evidence remains too narrow to establish how reliably the gains transfer or what they cost.
  2. The process analyses characterize the proposed signals more directly than they validate their correctness. The verified-progress rates, semantic regressions and suppression events in Sections 4.4–4.5 and Appendix C are defined through the same state/evidence machinery that constructs supervision. Their disagreement demonstrates that the two pathways are different; it does not independently establish that a credited transition is genuinely useful or that suppressed support is invalid. The qualitative example in Table 9 is helpful, but a systematic external audit of transition judgments is not reported. Since reliable transition attribution is the central motivation, this leaves an important gap between improved end-task accuracy and the proposed explanation for that improvement.
Questions for authors
  1. Can you provide the sampled question identifiers and an explicit train/evaluation disjointness check for the “additional subsets” in Appendix A.1, together with paired per-question results or uncertainty over question sampling? This is a request to verify the split and scope, not an allegation of overlap.
  2. What are the training and inference costs, including calls/tokens for both evaluators, and how does ProcEvo compare with terminal-only orchestration under the same execution and supervision budget? Please distinguish the controlled Table 2 comparison from the external baselines.
  3. Is there an independently checked sample of transitions, covering useful, redundant and harmful steps, against which checklist credit and state progress can be assessed? Agreement/error rates and representative disagreements would materially strengthen the attribution claim.
6 comments
AuthorsOpenReview ↗

Response to Reviewer eZM1 - Part1

We are grateful for your thoughtful review and for recognizing the importance of process-guided credit and controlled ablations. Your comments raise several important questions about evaluation scope, baseline comparability, computation, and the independent validation of transition-level feedback. Below, we clarify the original experimental design and present the corresponding supplementary evidence point by point.

Q1

Reviewer — Critical Weakness W1: The experimental scope and resource accounting limit the strength of the main performance claim. Appendix A.1 uses only 70 MMLU-Pro, 100 GSM-Hard and 100 GPQA Main evaluation questions, and the task-executing agents use GPT-4.1-mini throughout. Appendix A.3 appropriately treats runs as the unit of its Welch tests, but this measures variation on a fixed question set rather than uncertainty over benchmark instances. Moreover, Appendix A.2 explicitly preserves different baseline prompts, agent pools and calling patterns, and the paper does not report a matched resource comparison including the extra evaluators. The factorial ablation mitigates the attribution concern on GSM-Hard, but the evidence remains too narrow to establish how reliably the gains transfer or what they cost.

Reviewer — Question 1: Can you provide the sampled question identifiers and an explicit train/evaluation disjointness check for the “additional subsets” in Appendix A.1, together with paired per-question results or uncertainty over question sampling? This is a request to verify the split and scope, not an allegation of overlap.

A1: Thank you for highlighting the importance of benchmark coverage and question-level uncertainty. We selected fixed subsets of public benchmarks to compare repeated PPO optimization and multi-agent execution on identical questions while keeping repeated model calls feasible. Related work adopts similar cost-conscious evaluation. Huang et al. [1] sampled 200 examples for GPT-4 self-correction. Bo et al. [2] evaluated multi-agent reflection on 100 randomly selected test instances under computational constraints. Fixed questions provide a consistent and reproducible basis for comparing orchestration policies. The paired analysis below further examines question-level uncertainty.

The five independent runs in the original paper address training-run variation. To examine which questions were sampled, we checked the complete training and evaluation partitions and found zero overlap in both question IDs and normalized question texts. The revised Appendix A.1, Tables 3–4, reports the overlap counts and representative question IDs. We additionally resampled paired question-level correctness for ProcEvo and Puppeteer on the same evaluated questions:

Table 1. Paired ProcEvo–Puppeteer accuracy differences with question-level bootstrap 95% confidence intervals on fixed evaluation subsets.

BenchmarkQuestionsProcEvoPuppeteerΔ accuracy95% paired CI
GSM-Hard10078.00%64.00%+14.00 pp[+6.00, +23.00] pp
GPQA Main10060.00%57.00%+3.00 pp[−8.00, +14.00] pp
MMLU-Pro7082.86%77.14%+5.71 pp[−1.43, +14.29] pp

The GSM-Hard gain remains positive under question resampling, while GPQA and MMLU-Pro also show positive point estimates. This complements the run-to-run significance tests already reported in Appendix A.3.

The submission standardizes task-executing agents to GPT-4.1-mini to isolate orchestration rather than mixing backbone capacity with policy design. To check transfer beyond this backbone, we additionally repeated the comparison using Gemini 2.5 Flash execution agents:

Table 2. Cross-backbone ProcEvo–Puppeteer accuracy on GSM-Hard (100 evaluation questions per setting).

Executing backboneProcEvoPuppeteerGain
GPT-4.1-mini (submitted Table 1)77.20%64.00%+13.20 pp
Gemini 2.5 Flash (additional experiment)85.00%76.00%+9.00 pp

The Gemini comparison also favors ProcEvo and provides additional evidence that the improvement is not confined to the original GPT-4.1-mini backbone. The paired analysis addresses variation across questions, while the backbone experiment examines transfer across executing models. The Gemini result is from an additional single run.

References
[1] Huang et al., Large Language Models Cannot Self-Correct Reasoning Yet, ICLR 2024.
[2] Bo et al., Reflective Multi-Agent Collaboration based on Large Language Models, NeurIPS 2024.

AuthorsOpenReview ↗

Response to Reviewer eZM1 - Part2

Q2

Reviewer — Question 2: What are the training and inference costs, including calls/tokens for both evaluators, and how does ProcEvo compare with terminal-only orchestration under the same execution and supervision budget? Please distinguish the controlled Table 2 comparison from the external baselines.

A2: Thank you for raising the important distinction between comparing orchestration systems and isolating the effect of process feedback. In the original Table 1, external systems retain their published orchestration protocols, while using the same execution backbone, evaluation questions, and answer-scoring rules (Appendix A.2). Table 2 instead fixes the ProcEvo agent configuration and optimization protocol and compares the process-supervision components, including w/o Both Components, giving a direct mechanism comparison. ProcEvo achieves 77.20 ± 0.84%, against 68.60 ± 0.55% for w/o Both Components (+8.60 pp, five runs).

We also evaluated combined process supervision beyond GSM-Hard. The additional GPQA Main w/o Both Components experiment provides a second benchmark-level check:

Table 3. ProcEvo versus w/o Both Components on GSM-Hard and GPQA Main, with 100 evaluation questions per benchmark.

BenchmarkProcEvow/o Both ComponentsGain
GSM-Hard77.20 ± 0.84%68.60 ± 0.55%+8.60 pp
GPQA Main60.80 ± 0.84%56.00%+4.80 pp

The GPQA comparison extends this process-supervision check beyond GSM-Hard. GPQA w/o Both Components is an additional single-run result. Full follows the submitted multi-run evaluation.

To test whether spending more execution budget on terminal outcomes closes the gap, we trained w/o Both Components variants with increased rollout counts:

Table 4. GSM-Hard validation accuracy and estimated model API cost in USD across rollout budgets, using 100 training questions and 100 validation questions per setting.

Method / rollout settingVal accuracyTrain API costVal API cost
ProcEvo78%$1.68$0.46
w/o Both Components, 8 rollouts73%$1.65$2.06
w/o Both Components, 16 train / 4 validation rollouts73%$2.90$0.78
w/o Both Components, 16 train / 16 validation rollouts78%$2.90$3.42

At essentially the same training API expenditure, the 8-rollout w/o Both Components variant obtains 73%, versus 78% for ProcEvo. Reaching 78% with w/o Both Components optimization requires 16 validation rollouts in this test, at 1.73× the training API expenditure and 7.44× the validation API expenditure. The cost comparison supports a concrete finding. Under comparable training API expenditure, the tested w/o Both Components rollout expansion does not reach ProcEvo’s accuracy. The controlled component ablation separately isolates the benefit of process feedback.

For completeness, we also provide the detailed supervision-cost accounting. The two process signals serve different training functions. Skywork-Reward-V2 scores state quality locally. The evidence evaluator extracts candidate support for the deterministic reducer, as described in Sections 3.3–3.4 and Appendix B. Both evaluators are used in training and neither is invoked during the reported validation runs.

Table 5. Model API token use and estimated cost in USD for ProcEvo training on 100 GSM-Hard questions. The total corresponds to the ProcEvo training row of Table 4. Displayed component values are rounded.

Training usageAPI tokensAPI cost
Executing agents1.54M$0.63
DeepSeek-V3.2 evidence evaluator2.83M$0.57
GPT-4.1-mini evaluator fallback1.30M$0.47
Total model API usage5.66M$1.68

The $1.68 training cost includes evidence-evaluation API overhead. The $0.46 validation API cost covers task-executing agents selected by the trained policy. Skywork state scoring runs locally.

AuthorsOpenReview ↗

Response to Reviewer eZM1 - Part3

Q3

Reviewer — Critical Weakness W2: The process analyses characterize the proposed signals more directly than they validate their correctness. The verified-progress rates, semantic regressions and suppression events in Sections 4.4–4.5 and Appendix C are defined through the same state/evidence machinery that constructs supervision. Their disagreement demonstrates that the two pathways are different; it does not independently establish that a credited transition is genuinely useful or that suppressed support is invalid. The qualitative example in Table 9 is helpful, but a systematic external audit of transition judgments is not reported. Since reliable transition attribution is the central motivation, this leaves an important gap between improved end-task accuracy and the proposed explanation for that improvement.

Reviewer — Question 3: Is there an independently checked sample of transitions, covering useful, redundant and harmful steps, against which checklist credit and state progress can be assessed? Agreement/error rates and representative disagreements would materially strengthen the attribution claim.

A3: We appreciate the request for independent evidence that the process signals correspond to meaningful changes in task state and supporting evidence. We would like to clarify the intended role of the analyses in the submitted paper: Figure 3 illustrates how State Progress and Checklist Credit capture complementary aspects of a transition, while the five-run component ablation in Table 2 evaluates their contribution to learning. To further substantiate the semantic basis of these signals, we conducted a blinded semantic audit using Claude Sonnet 5.5 on 80 GSM-Hard transitions from 52 questions. We sampled 20 transitions from each of the four State Progress and Checklist Credit sign combinations. The external evaluator inspected the transition content without seeing ProcEvo’s scores or credit labels. We then applied a consistent evidence criterion to disputed judgments.

Table 6. Claude Sonnet 5.5 blinded semantic audit on 80 GSM-Hard transitions from 52 questions. Each State Progress and Checklist Credit sign group contains 20 transitions.

Independently checked distinctionOutcome
Positive Checklist Credit accompanied by supported new evidence29/40 (72.5%)
With Checklist Credit = 0, material solution improvement: State+ vs. State≤013/20 (65%) vs. 4/20 (20%)
With State≤0, eligible evidence advancement: Credit+ vs. Credit=015/20 (75%) vs. 4/20 (20%)
Positive-credit Python evidence conservatively supported24/26 (92.3%)

The audit directly checks whether rewarded transitions introduce supportable evidence. Of 40 positive-credit steps, 29 passed the unified check, including 24 of 26 Python-supported cases under the conservative criterion. The conditional comparisons offer a complementary finding. With zero Checklist Credit, positive State Progress is associated with more material solution improvements, at 65% versus 20%. With nonpositive State Progress, positive Checklist Credit is associated with more eligible evidence advances, at 75% versus 20%. These results connect both signals to independently assessed transition content, beyond the internal signal statistics in the submission.

A separate GPQA trace illustrates a concrete evidence–answer disagreement. In T057, Python reports 1.44 g of carbon, while the selected answer corresponds to 0.72 g. The reducer assigns +0.00020 Checklist Credit to tool-derived answer_support, but does not mark the selected task_answer as matching that evidence. This distinction matters: credit for an intermediate piece of support does not certify the final answer. Our independent GSM-Hard assessment, reported in the revised Appendix C.5, Tables 14–15, provides the broader quantitative check of transition-level signals. Together with the controlled ablation, it supports both the learning contribution and the evidence grounding of process supervision.

We hope these clarifications and the new paired, resource-budget, cross-backbone, and external-audit results address the concerns about evaluation rigor and process-credit validity. We would be pleased to provide further clarification during the discussion.


From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.