- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes ProcEvo, a framework for training a sequential multi-agent orchestrator with transition-level process supervision. It combines state-quality differences (Skywork-Reward-V2) with checklist-based evidence verification using an external LLM, then incorporates these signals into PPO alongside terminal rewards. Experiments on sampled subsets of MMLU-Pro, GSM-Hard, and GPQA Main report improvements over several orchestration and multi-agent baselines, supported by component ablations and diagnostics.
Critical strengths
- I find the motivation clear and well illustrated. Terminal rewards conflate useful, redundant and harmful transitions, and raw state change does not imply progress.
- I like the design principle behind the checklist reducer. It separates evidence extraction from credit assignment and explicitly suppresses self-certification.
- I appreciate that the authors report variance over five independent runs and per-comparison p-values, including non-significant ones.
Critical weaknesses
- I am concerned that the state-progress signal, presented as transition-level, is partly trajectory-level. In Eq. 19, each step’s state progress is scaled by a factor based on how well its whole trajectory scored relative to the other rollouts for the same task. As a result, two steps with identical local progress can receive different credit depending on how their trajectories performed. Because the same factor scales negative progress, a harmful step also receives a smaller penalty in a poorly performing trajectory. This seems inconsistent with the goal of identifying which individual steps help or hurt progress.
- The evidence-reduction procedure is underspecified because several key decisions rely on LLM judgments, such as whether sources are independent, tool results are trustworthy, or new evidence genuinely supports a claim. Given this, I have further concerns about what the checklist signal capture below.
- All executable agents share the GPT-4.1-mini backbone and differ only in role prompts and actions, so their errors are likely correlated. Agent-to-agent support may therefore be a weaker form of the self-certification.
- 36.4% of credit events come from trusted tool, thus, the signal may partly reward tool use rather than verified progress.
- The evaluator (DeepSeek-V3.2) may be stronger than the executor (GPT-4.1-mini), so some gains may come from transferring the stronger model’s judgments.
Questions for authors
- How do the results change if the σ(z_j) trajectory gating in Eq. 19 is removed, so that state progress is used without trajectory-level scaling?
- Is the DeepSeek-V3.2 evaluator used only during training, or also at test time? How do results change if GPT-4.1-mini is used as the evaluator?
- How do agent-selection frequencies change during training?
6 comments
Response to Reviewer c4Sn - Part2
Q2
Reviewer — Critical Weakness W2: The evidence-reduction procedure is underspecified because several key decisions rely on LLM judgments, such as whether sources are independent, tool results are trustworthy, or new evidence genuinely supports a claim. Given this, I have further concerns about what the checklist signal capture below.
A2: Thank you for this question. The distinction between extracting candidate evidence and deterministically granting credit is central to our design. ProcEvo does not rely on an LLM's unconstrained decision to assign reward. The LLM extracts structured candidate claims, source provenance, and support. The deterministic reducer tracks checklist-item maturity and applies eligibility, scope, contradiction, duplication, and evidence-source rules described in Section 3.3 and Appendix B.2–B.5. Only a net eligible increase in checklist potential yields positive Checklist Credit under Eqs. (7)–(8). A self-confirming assertion is not automatically independent verification.
The key distinction is that an LLM-proposed support relation is not itself a reward decision: positive credit requires an eligible evidence-maturity increase under the explicit reducer rules. To examine how this distinction operates on actual trajectories, we added the 80-transition external-content assessment reported in eZM1, A3. Under the unified review criterion, 29/40 positive-credit transitions were supported by independently examined new evidence, including 24/26 Python-supported cases under a conservative standard. The assessor evaluated the transition content rather than taking the extractor's labels as the reference.
Q3
Reviewer — Critical Weakness W3: All executable agents share the GPT-4.1-mini backbone and differ only in role prompts and actions, so their errors are likely correlated. Agent-to-agent support may therefore be a weaker form of the self-certification.
A3: Thank you for raising the possibility of correlated mistakes. The homogeneous GPT-4.1-mini executor pool serves as an experimental control. Holding the task-solving backbone fixed helps attribute performance differences to agent selection and process supervision rather than mixed model capacity, as detailed in Appendix A.5. The reducer evaluates evidence provenance and novelty rather than treating cross-role agreement as independent confirmation. Repeated agreement among similarly prompted agents does not itself advance checklist maturity without eligible supporting evidence.
We also evaluated a heterogeneous execution-agent pool with DeepSeek-V3.2 for the main reasoning roles, Gemini 2.5 Flash for Critic and Reflect, and GPT-4.1-mini for the other roles. The process-supervision mechanism remained unchanged. This configuration achieved 81% on the 100-question GSM-Hard validation subset. Separately, replacing the shared execution backbone with Gemini 2.5 Flash yielded 85% for ProcEvo versus 76% for Puppeteer on the same 100-question evaluation. These results show that the method also operates beyond a homogeneous GPT-4.1-mini pool. Checklist Credit still depends on new, eligible evidence, not on agreement between agents alone.
Q4
Reviewer — Critical Weakness W4: 36.4% of credit events come from trusted tool, thus, the signal may partly reward tool use rather than verified progress.
A4: Thank you for raising this concern. The 36.4% in Figure 4(d) measures the share of evidence-maturity promotions sourced from trusted tools, not the share of tool calls receiving reward. Under Eq. (8), positive credit requires an eligible increase in evidence maturity. Invoking a tool alone does not earn credit.
Table 2. Trusted-tool executions versus positive Checklist Credit in logged GSM-Hard training transitions.
| Diagnostic | Observed count |
|---|---|
| Tool-invoking transitions with zero positive Checklist Credit | 429/571 (75.1%) |
| Successful tool invocations with zero positive Checklist Credit | 215/352 (61.1%) |
| Exact repeated evidence on the same item suppressed | 13/13 |
These observations show that calling a tool alone is insufficient to earn reward. As a further independent content check, 24/26 positive-credit Python steps had valid supporting evidence under the conservative external audit. The mechanism rewards what a tool establishes, not the fact that a tool was invoked.
Response to Reviewer c4Sn - Part3
Q5
Reviewer — Critical Weakness W5: The evaluator (DeepSeek-V3.2) may be stronger than the executor (GPT-4.1-mini), so some gains may come from transferring the stronger model’s judgments.
Reviewer — Question 2: Is the DeepSeek-V3.2 evaluator used only during training, or also at test time? How do results change if GPT-4.1-mini is used as the evaluator?
A5: Thank you for suggesting the most direct way to separate a strong training evaluator from a strong executing model. DeepSeek-V3.2 is used to extract training evidence. It is not called to solve tasks during evaluation. The same is true of the local Skywork state scorer. At test time, the trained orchestrator selects the task-executing agents. Evaluator calls do not contribute to validation answers. The corresponding training API expenditures appear in eZM1, A2.
We additionally replaced the evidence evaluator with GPT-4.1-mini and retrained ProcEvo:
Table 3. GSM-Hard evidence-evaluator ablation. Accuracy is reported as mean ± SD over five Full runs and three ablation runs, using 100 validation questions per run.
| Evidence evaluator | Val accuracy |
|---|---|
| DeepSeek-V3.2 (Full) | 77.20 ± 0.84% |
| GPT-4.1-mini (weaker evaluator) | 75.67 ± 2.08% |
With GPT-4.1-mini as the evidence evaluator, ProcEvo reaches 75.67%, 1.53 percentage points below the submitted Full mean. The process-supervision benefit remains observable with a weaker evidence evaluator. The revised Appendix A.9, Table 9, includes this comparison.
Q6
Reviewer — Question 3: How do agent-selection frequencies change during training?
A6: We appreciate the request for an observable policy-level check. The learned policy selects the next agent conditional on the evolving task state, and its selections change over training. We analyzed 1,781 recorded selections from the GSM-Hard run:
Table 4. Agent-selection changes between the first and last 25-question stages of logged GSM-Hard training.
| Policy behavior | First stage | Last stage |
|---|---|---|
| Terminator selection share | 5.1% | 16.0% |
| Retrieval-oriented agent selection share | 29.2% | 12.5% |
| Reasoning + Reflect selection share | 16.2% | 23.2% |
| Mean trajectory length | 4.69 | 4.01 |
These are changes in actual chosen actions, not merely changed evaluator scores. Increased termination and reduced low-value retrieval, combined with a shorter average workflow, are consistent with a learned policy that makes more selective agent-calling decisions. This behavioral evidence complements the measured final-accuracy improvements.
We hope the mechanism clarification, targeted ablations, and logged agent/tool statistics address the concerns about trajectory gating and credit reliability. We thank the reviewer again for the detailed suggestions and welcome further discussion.
Response to Reviewer c4Sn - Part1
We appreciate your detailed and constructive analysis of the proposed process signals and orchestration policy. Your questions offer an opportunity to clarify the distinction between local progress estimation and trajectory-aware optimization, as well as the safeguards used for evidence credit. We respond to each concern below and supplement the original explanations with targeted ablations and trajectory-level analyses.
Q1
A1: Thank you for carefully examining Eq. (19). We would like to clarify the distinction between estimating local progress and weighting that estimate during policy learning. Eq. (5) defines the local State Progress measurement pt from adjacent task states and the historical best. Eq. (19) does not change that measurement: it scales the signal's influence on policy optimization, p~t=σ(zj)pt, based on the relative performance of sampled trajectories for the same question. Equal local progress therefore remains equal at the measurement stage, even if the orchestrator learns from it with different weights.
Why calibrate that weight? A state-quality estimate provides evidence of local improvement, but not of whether the corresponding agent choice is a useful use of the overall workflow budget. A step that appears locally favorable inside a long, unsuccessful rollout should not necessarily shape the policy as strongly as one embedded in a productive trajectory. Eqs. (18)–(19) incorporate step-level feedback, terminal outcome, and trajectory length to calibrate an auxiliary learning signal. Task-relative comparisons across sampled outcomes are also used in group-relative policy optimization such as GRPO [1]. The distinction is between identifying local change and deciding how heavily to optimize for it, not between transition supervision and learning from terminal outcomes alone.
As noted in the review, the positive gate factor also attenuates negative local progress in lower-scoring trajectories. This is a deliberate weighting choice rather than an attempt to redefine a harmful step as beneficial: multiplication by a positive factor preserves the sign of the local estimate. The terminal reward also remains in the GAE-based policy update under Eqs. (9)–(10). Negative outcomes therefore still affect the policy independently of the auxiliary State Progress weight. Calibrating both positive and negative estimates limits the contribution of potentially noisy local judgments from poor trajectories rather than requiring that every such estimate receive equal optimization weight.
The requested ablation directly tests whether this trajectory context is useful:
Table 1. GSM-Hard trajectory-gating ablation. Accuracy is reported as mean ± SD over five Full runs and three ablation runs, using 100 validation questions per run.
Removing the trajectory gate lowers mean accuracy by 4.53 percentage points. This supports trajectory-aware calibration while retaining the same local progress estimator. The results are also included in revised Appendix A.9, Table 9.
Reference
[1] Shao et al., DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models, 2024.