acceptodds
Under review as a conference paper at ICLR 2027

EvoSteer: Online Self-Evolving Graph Orchestration via Reference-Anchored Credit Assignment

Abstract

In recent years, LLM-based multi-agent systems have been widely applied to orchestrate tool-using agents into executable communication graphs. However, existing self-evolving orchestration still faces key challenges, including post-hoc evolution that revises the team only after the trajectory ends, credit diffusion that gives every action the same terminal advantage under confounded baselines, and skill admission that is uncalibrated and never retired. To address these challenges, we propose EvoSteer, a new paradigm of Online Self-Evolving Graph Orchestration—the orchestrator builds a running team and repairs its plausible but failing steps from execution features and a learned value estimate. To support this paradigm, we introduce Anchored Trajectory Balance (AnchorTB), a regression-style flow-matching loss that assigns each orchestration action a coefficient by balancing subtrajectories against a frozen reference. Built on the learned flow, we further propose Validated Skill Admission, in which a candidate skill is tried before promotion and promoted only if paired evidence passes a sequential test under a shared nominal testing budget. Moreover, AnchorTB combines measured task-level reference reward statistics with prefix-dependent corrections. Experimental results on twelve datasets show that EvoSteer significantly outperforms baselines across question answering, mathematical reasoning, code generation, and interactive decision making. Our code is available at https://anonymous.4open.science/r/evosteer-C312.

Primary area:
foundation or frontier models, including LLMs
Keywords:
muti-agent

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_mHBGOpenReview ↗
Rating
4(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper presents EvoSteer, an online self-evolving graph orchestration framework for LLM-based multi-agent systems. In contrast to existing post-hoc optimization approaches, EvoSteer employs a trainable orchestrator to dynamically construct and refine communication graphs during task execution, guided by runtime feedback and learned value estimates. The framework incorporates online graph editing and repair, Anchored Trajectory Balance for reference-relative credit assignment across subtrajectories, and Validated Skill Admission, which leverages paired rollouts and sequential statistical testing to regulate the admission of reusable skills. Extensive experiments across 12 benchmarks spanning question answering, mathematical reasoning, code generation, and interactive decision-making demonstrate consistent improvements over existing orchestration and skill-evolution baselines. Further ablation studies and cross-backbone evaluations substantiate the effectiveness and transferability of the proposed framework.

Critical strengths
  • Clear problem formulation and compelling research motivation. The paper clearly identifies the limitations of post-hoc orchestration and provides a well-motivated formulation that emphasizes the importance of dynamic graph adaptation and repair during task execution.
  • Interesting and well-designed optimization approach. The proposed AnchorTB introduces a reference-anchored subtrajectory credit assignment mechanism that provides more informative learning signals for optimizing orchestration policies.
  • Comprehensive experimental evaluation. The paper presents extensive experiments across 12 benchmarks, complemented by thorough ablation studies and cross-backbone evaluations that demonstrate empirical performance and generalizability.
Critical weaknesses
  • Gap between theoretical guarantees and practical implementation. The theoretical guarantees of AnchorTB rely on shared state flows, whereas the practical implementation employs trajectory-dependent estimates. This discrepancy raises concerns about the extent to which the theoretical analysis supports the implemented optimization procedure.
  • Substantial computational overhead and limited efficiency analysis. The framework introduces considerable computational overhead through repeated training rollouts and skill validation. However, comprehensive end-to-end cost-efficiency comparisons with existing baselines are lacking, making it difficult to assess whether the performance improvements justify the additional computational expense.
Questions for authors
  1. Could the authors clarify whether the theoretical guarantees of AnchorTB extend to its practical implementation, particularly when trajectory-dependent estimates are employed?
  2. How does EvoSteer perform relative to existing baselines under matched inference-time token and computational budgets?
4 comments
AuthorsOpenReview ↗

Response to Reviewer mHBG (1/2): Theory and the implemented optimization

We thank the reviewer for the positive assessment and valuable suggestions, and answer each point below.

W1 / Q1. Theory and implementation.

In short, yes: the guarantees that describe the implemented optimization step hold exactly for the trajectory-dependent estimates, and the distribution-level guarantee holds for the shared flow used on the trained policy's own trajectories.

Why a shared flow is required, and where the implementation departs from it. A shared flow assigns one value to each state, whichever continuation is later scored. This is the definition of a flow under which the classical balance results are proven (Malkin et al., 2022; Bengio et al., 2023; Madan et al., 2023), including the reference-relative form we build on (Venkatraman et al., 2024), and without it zero loss would say nothing about the sampled distribution (Remark B.3). Our implementation meets it except in one deliberate place. The structure correction is read from the previous batch and the learned residual is a function of the state, so both are shared, and the task anchor is the same for every trajectory of the trained policy. Only the reference's own rollouts, which also supply the task anchor, leave themselves out, so that a rollout's reward does not enter its own anchor, as in RLOO (App. B.1–B.2). Keeping it in would make the anchor exactly shared but pull it toward that rollout's outcome, and a learned shared scalar is biased: in Prop. B.9 it settles at Elog⁡Rβ\mathbb E\log R_\beta rather than log⁡Z\log Z.

Q1. Which guarantees extend to the implementation. The first group concerns a single recorded trajectory. These results use only the anchor values on that trajectory and never compare trajectories, so a different value on another trajectory does not affect them (Prop. B.1), and they are exactly what the implementation computes. The loss on a trajectory is zero exactly when every span on it balances; each action's coefficient is the sum of the residuals of the spans containing it, which is the per-action credit of Prop. 2; and the loss gradient for the orchestrator is this coefficient times the action's log-probability gradient, with the measured anchors detached (Remark A.6, Lemma A.17). Changing the anchors moves residuals, loss and coefficients only by a bounded amount (Prop. B.5), and the estimator properties, such as the measured anchor's range and the bound on v^\hat v's calibration error (Prop. B.8), also hold as implemented.

The second group, that balanced flows give the target distribution and nearly balanced ones a nearly correct distribution (Props. A.8, A.15), compares one state across all its continuations and therefore needs a shared flow. The flow on the trained policy's own trajectories is exactly such a flow, so whenever it nearly balances on every reference-supported history, the policy is provably close to the target (Prop. A.15); on reference rollouts, the self-exclusion changes residuals and credit within the bound of Prop. B.5. The theory thus specifies the exact update the implementation performs, and what a nearly balanced solution guarantees about the trained policy.

AuthorsOpenReview ↗

Response to Reviewer mHBG (2/2): Cost and matched budgets

W2 / Q2. Cost and matched budgets.

EvoSteer scores highest at a per-problem test cost within the range of the other orchestrators, and its one-time training needs fewer GPU hours than any other trained orchestrator, with skill validation taking 13.7% of its training tokens. The table reports both (IID / OOD: mean of the six benchmarks), with each baseline in its authors' configuration on the same executor and training tasks:

MethodTraining: H800 hours / tokensTest tokens / problemIID avgOOD avg
Qwen3.5-9B–3.2k54.5247.14
Self-consistency@6–19.7k56.9248.26
Best-of-N@4–20.3k55.4747.97
SFT7 / 3.0M3.6k52.3248.41
GRPO†35 / 0.8B4.9k55.9149.35
AFlow (search)API / 2.75B16.0k77.4670.52
AgentFlow124 / 1.73B31.6k79.8075.89
FlowSteer231 / 4.67B43.4k82.1177.84
SkillFlow189 / 1.42B22.0k83.0678.05
EvoSteer96 / 1.58B21.7k87.8482.40

Q2. Matched test-time budgets. Test tokens count every orchestrator and agent call, prefill plus decode, averaged over the twelve benchmarks. With the same backbone and nearly the same tokens (90% and 94% of EvoSteer's), self-consistency and best-of-N add at most 2.4 points on average over a single call, whereas EvoSteer adds 33.3 IID and 35.3 OOD points: extra samples cannot supply retrieval, execution feedback or in-run repair. Among orchestrators, EvoSteer uses fewer tokens than AgentFlow and FlowSteer, about as many as SkillFlow and more than AFlow (which scores over 10 points lower), yet is best on all twelve benchmarks (Table 1).

Training cost. EvoSteer trains once, for 240 steps, in 96 H800 hours, fewer than AgentFlow (124), SkillFlow (189) and FlowSteer (231); its 1.58B tokens are below those of AgentFlow, FlowSteer and AFlow's search (2.75B) and close to SkillFlow's (1.42B). Only the single-model fine-tunes, SFT and GRPO†, use fewer GPU hours, and they score over 30 points lower. Of EvoSteer's tokens, 26.5% are natural rollouts, 13.7% paired rollouts for skill validation, 59.8% prefill-only rescoring for the AnchorTB loss, and 0.004% the skill author. This cost is paid once: the trained orchestrator transfers unchanged to six other executors (Sec. 5.2), and at test time no reference, paired or validation rollouts are drawn (App. D). Skill validation is also what keeps admitted skills useful, since admitting every candidate instead lowers MedQA and AIME (Fig. 6d).


We thank the reviewer again. All additional experiments and revisions have been incorporated into the revised version. If our responses and revisions address your concerns, we would be deeply grateful if you could kindly reconsider raising the score further. Thank you very much!

Reviewer_mHBGOpenReview ↗

Response for the Rebuttal

Thank you for the detailed response. The authors have addressed most of my concerns by clarifying the relationship between AnchorTB’s theoretical guarantees and its practical implementation. They have also provided valuable computational cost comparisons that further strengthen the paper.

These clarifications have increased my confidence in the paper, and I will raise my score.

Thank you very much for your kind feedback and for raising your score.We are glad that our response has addressed your concerns, and we would greatly appreciate it if you could also raise your confidence score. Thank you again for your time and constructive comments.

Reviewer_2G5POpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

EvoSteer allows a team of agents to change its structure while solving a task. It can add or remove agents, change their connections, and rerun earlier steps. The paper introduces AnchorTB to train these decisions and uses paired runs to select reusable skills. Experiments show improvements across several task types.

Critical strengths
  1. Changing the team during execution is useful, especially when an earlier step needs to be corrected.
  2. The experiments include comparisons under the same architecture and ablations of the main components.
  3. The paper reports training costs and discusses several limitations in the appendix.
Critical weaknesses
  1. The main method offer little new contribution. There are quite a few papers related to anchors and corrections base on exectation history, failures, etc. The contribution is mainly an improved training objective.
  2. The theory guaratees contain strong assumptions. Some results require a consistent value estimate for the same history, while the implemented anchor can vary across trajectories.
  3. Not very practical for real environment. The method needs scored execution records and paired runs from matching initial conditions. These requirements are harder to meet for open-ended tasks or tools that change the real environment.
  4. The reported evaluation does not establish continued skill discovery and updating after deployment.
Questions for authors
  1. What can the reference anchors tell us ?
  2. Which theoretical guarantees apply to the trajectory-dependent anchors used in the implementation?
  3. Does the implemented skill-selection procedure satisfy the conditions needed for the stated error control?
  4. Which applications satisfy the requirements for reliable scoring and repeatable execution?
7 comments
AuthorsOpenReview ↗

Response to Reviewer 2G5P (1/3): Anchoring and in-run correction versus prior work

We thank the reviewer for the careful review and valuable suggestions, and answer each point below.

W1 / Q1. Anchoring and correction versus prior use of execution history.

We read the recent work in full: correction, i.e., DoVer (ICLR 2026), AgenTracer (ICLR 2026), MANTA (concurrent), Puppeteer (NeurIPS 2025) and AOrchestra (ICML 2026); and anchoring, i.e., GiGPO (NeurIPS 2025) and Tree-GRPO (ICLR 2026). EvoSteer's correction and anchors differ from them in kind:

Prior workEvoSteer
Correction is triggered byan observed failure or anomaly: a failed trace with the ground truth (DoVer), a failed trajectory (AgenTracer), LLM audit flags (MANTA), an LLM-written summary (AOrchestra)predicted risk, before any failure shows: v^\hat v estimates the frozen reference's continuation reward from execution features (Eq. 5)
Correction happensafter the fact, or at most once: after the run, by replaying a checkpoint (DoVer); in the next attempt (AgenTracer); once per run (MANTA)mid-run, as often as the policy decides: every edit executes as it is issued (Prop. 1), and the repair count follows from the policy under the shared budget (Sec. 4.1)
Correction targetstext or the team as a whole, not the failing agent: orchestrator messages and plans (DoVer); text feedback for the next attempt (AgenTracer); added agents or links (MANTA); a fresh sub-agent, possibly losing earlier work (AOrchestra)the failing agent itself: RERUN_AGENT / DROP_AGENT act on one node of the running team (Eq. 4)
When to correct is decided byhand-designed rules, or learning without per-decision credit: prompts (DoVer, AOrchestra); a severity gate and fixed rules (MANTA); one trajectory-level reward (Puppeteer: "coarse-grained rewards", App. D)one learned policy, with its own credit for each repair: each action gets its own coefficient (Prop. 2)
An anchor isa position in a trajectory: a repeated state (GiGPO), a shared prefix (Tree-GRPO), a blamed failure step (AgenTracer), an intervention point in a failed trace (DoVer)a value estimated at every state: the reference's log expected reward u∗(s)u^\ast(s), which equals log⁡Z\log Z at the root (Sec. 4.2)
The anchor comes fromthe policy's own samples or failed traces: the current policy's group (GiGPO) or tree (Tree-GRPO); counterfactual replay of failed traces (AgenTracer, DoVer)a fixed reference: frozen-reference rollouts on the same task (uqu_q) and team structure (c(g)c(g)) (Eqs. 8–9)
The anchor separatesone undifferentiated signal: a group mean (GiGPO, Tree-GRPO); one blamed step per failure (AgenTracer)the three sources of a score: task difficulty (uqu_q), team structure (c(g)c(g)) and the decision (residual) (Sec. 1)
Requiresrepeated states, mid-run branching or replay: identical states (GiGPO, §6); branched rollouts (Tree-GRPO); a checkpoint/replay interface and the ground truth (DoVer)none of these: only two reference rollouts per task in each batch (Table 3)

Correction. Prior methods correct only after a failure or anomaly is observed, redo text, an attempt or the whole team, and decide when to correct by rules or by one trajectory-level reward. In orchestration this comes too late, since the decisive error is locally indistinguishable from a recoverable step (Sec. 1). EvoSteer turns correction into a learned decision inside the run: v^\hat v flags risk before any failure shows (Eq. 5), the policy repairs only the failing agent while the team keeps running (Eq. 4, Prop. 1), and each repair receives its own credit (Prop. 2). The contribution is thus more than an objective: the architecture creates mid-run repair decisions, and AnchorTB makes them learnable.

Anchors. Prior anchors are positions in a trajectory, taken from the policy's own samples or failed traces. They need repeated states, mid-run branching or replay, which orchestration rarely offers because each state carries the full history (Def. 2), and they yield one signal that mixes task difficulty with the policy's own choices. EvoSteer's anchor instead estimates, at every state and from a frozen reference, the flow at which every span balances (Prop. A.8). It needs no repeated state, separates task difficulty, team structure and the decision, and gives each action its own coefficient (Prop. A.14).

Q1. What the anchors tell us. The anchor estimates u∗(s)=log⁡Eρ[Rβ∣s]u^\ast(s)=\log\mathbb E_\rho[R_\beta\mid s], the frozen reference's log expected reward from state ss, and its parts read separately: uqu_q is the task's difficulty (how often ρ\rho solves it), c(g)c(g) is whether the team structure raises or lowers reference success relative to that difficulty, and bψb_\psi holds the rest. A low score can thus be traced to a hard task, a weak structure or a poor decision, which a learned state value cannot tell apart (Sec. 1).

AuthorsOpenReview ↗

Response to Reviewer 2G5P (2/3): Scope of the theoretical guarantees

W2 / Q2 / Q3. Scope of the guarantees.

Our guarantees rest on the standard condition of flow-balance theory, one value per state; the implementation meets it except for one deliberate self-exclusion, and every guarantee used in training holds for it exactly.

In GFlowNet theory a flow is a function of the state, and the results that balanced flows yield the target distribution are proven under exactly this condition (Malkin et al., 2022; Bengio et al., 2023; Madan et al., 2023), including the reference-relative form we build on (Venkatraman et al., 2024). It concerns how the estimate is built, not the tasks or the executor, and any flow written as a function of the state meets it. It is also what makes zero loss informative: without it, any policy admits estimates with zero loss (Remark B.3).

Where the anchor varies, and why. Of the anchor's three parts, two are always shared and the third varies only on the reference's own rollouts:

  1. Structure correction c(g)c(g): read from the previous batch.
  2. Learned residual bψb_\psi: a function of the state.
  3. Task anchor uqu_q: the same for every trajectory of the trained policy and every paired path; only the reference's own rollouts, which also supply it, each leave themselves out, so that a rollout's reward does not enter its own anchor, as in RLOO (App. B.1–B.2).

Keeping it in would make the anchor exactly shared but pull it toward that rollout's own outcome, and a learned shared scalar is biased: in Prop. B.9 it settles at Elog⁡Rβ\mathbb E\log R_\beta rather than log⁡Z\log Z.

Q2. Which guarantees apply. Every guarantee the training update uses holds exactly for the implemented anchors, and the distribution-level guarantee holds for the flow on the trained policy's trajectories.

  1. Per-trajectory guarantees. They use only the anchor values on that trajectory, each reused in every span of it, and never compare trajectories, so a different value on another trajectory does not affect them (Prop. B.1). On each trajectory the loss is zero exactly when every span balances, each action's coefficient is the sum of the residuals of the spans containing it, which is the per-action credit of Prop. 2, and changing the anchors moves residuals, loss and coefficients only by a bounded amount (Prop. B.5). The estimator properties, such as the measured anchor's range and the bound on v^\hat v's calibration error (Prop. B.8), hold as implemented too.
  2. Distribution-level guarantees. Balanced flows give the target distribution and nearly balanced ones a nearly correct distribution (Props. A.8, A.15). These results compare one prefix across all its continuations, so they need one value per prefix, as in the work above. The flow on the trained policy's own trajectories has exactly this property, so whenever it nearly balances on every reference-supported history, the policy is provably close to the target (Prop. A.15); on reference rollouts, the self-exclusion changes residuals and credit within the bound of Prop. B.5.

Q3. Skill admission. Yes: the error bound needs one condition, and each implemented rule serves it. The condition is that each test is valid given what was known when its candidate was registered (Eq. C.8); in particular, its validation outcomes must be new, neither used to choose the candidate nor kept selectively (Prop. C.6; Assumption C.2 is one sufficient protocol for it).

  1. Candidate fixed first: a candidate is fixed before any of its validation pairs is run, and only pairs launched afterwards count, so no outcome that chose it is reused.
  2. New, unselected outcomes: each pair is a new rollout, kept for being complete and side-effect-free, never for its outcome, so outcomes are neither duplicated nor kept selectively.
  3. Matched arms: both arms start from the same task, configuration and snapshots and are run by the frozen reference, so they differ only in the candidate, and the trained policy acts in neither (Def. C.1).
  4. Budget per look: each look of each candidate has its own share of the error budget, so a whole run spends at most α\alpha, with no independence needed between tests or looks (Props. C.5–C.6).

The two failure cases in the appendix, one outcome copied into many pairs and reuse of selection outcomes (Remarks C.2–C.3), therefore cannot occur.

AuthorsOpenReview ↗

Response to Reviewer 2G5P (3/3): Practical requirements and the evaluation protocol

W3 / Q4. Practical requirements.

Scores and repeated runs are used only to train the orchestrator and to admit skills; evaluation uses neither, on the IID and the OOD test sets alike, and "online" refers to edits made inside a run from its own execution feedback.

At evaluation. The trained orchestrator runs with v^\hat v and the skill library frozen, and no reference, paired or validation rollouts are drawn (App. D). Each mid-run edit is chosen from the execution feedback and v^\hat v, which reads execution features rather than a grader (Sec. 4.1), so a run needs nothing beyond the task; the grader is used only afterwards, to report the score. The OOD benchmarks test exactly this setting: six task sets never used in training, including SWE-Bench Verified and WebShop, whose tools edit a code repository or act in a web store. EvoSteer is best on all of them (Table 1), and the same orchestrator transfers unchanged to six other executors (Sec. 5.2).

In training. The requirements are those of any reward-trained orchestrator, including AgentFlow, FlowSteer and GRPO in our comparison:

  1. A score per episode: computed once, after the stop (Sec. 4.1); any bounded score in [0, 1] fits, binary or graded (App. B).
  2. Repeatable starts: repeated runs only restart a task from its initial state. AnchorTB uses a few rollouts per task, as group-based RL does, and a skill pair adds two runs of the same task. Nothing is restored mid-run, unlike branching (Tree-GRPO) or checkpoint replay (DoVer, AgenTracer), and runs need not be deterministic, since the two arms of a pair may diverge after the first action (Def. C.1).

Q4. Suitable applications.

  1. For training: any task family with an automatic bounded score and a resettable start, i.e., question answering with references, mathematics with answer checking, code with tests, and interactive environments run in simulators or containers, the four types our six training benchmarks cover.
  2. For use: any task of a trained type, since a run needs no score; the OOD benchmarks are such tasks.

W4. Skill discovery and updating.

EvoSteer follows the protocol of recent self-evolving and self-improving systems: the system evolves automatically from feedback during training, and what evolved is then evaluated, frozen, on held-out tasks. We test it both on unseen tasks of the training families (IID) and on entirely new families (OOD), and the team still evolves inside every test run.

Evolution during training. Training is a continuing stream of tasks on which skills are discovered and updated without human input: the author distills candidates from scored runs, and paired evidence promotes or retires each one as it accumulates. Over training, 14 candidates were proposed, 9 promoted and 5 retired, with none left pending, so admission both accepted and rejected candidates.

The same protocol as recent work. Two lines of recent work evolve in this way and are evaluated in this way. Skill and prompt evolution methods evolve their skills or prompts on training rollouts and report the frozen result on held-out tests, including transfer to new task families (SkillOpt, NeurIPS 2026; GEPA, ICLR 2026; SkillFlow). Self-improving coding agents evolve the agent itself and take the transfer of the evolved agent to new benchmarks or backbones as their held-out test (Darwin Gödel Machine and Huxley–Gödel Machine, ICLR 2026). Our evaluation covers both forms: IID on unseen items of the training families, OOD on families never used in training, and transfer to six other executors (Sec. 5.2). Freezing also keeps the test clean, since admitting skills on test items would require their graders, and removing the evolved skills lowers both IID and OOD scores, also with the paired budget kept (Table 2).

After deployment. Continued admission needs only what training needs, a score per episode and repeatable starts (W3), so it applies unchanged wherever a deployment provides them, e.g., code with tests. Evolving at test time without ground-truth scores, from self-judgment or execution feedback, is a separate test-time-learning protocol, studied by ReasoningBank and the online setting of ACE (both ICLR 2026), which the paper does not claim; we have made this explicit and revised "evolves while it runs" to "evolves within each run".


We thank the reviewer again. All additional experiments and revisions have been incorporated into the revised version. If our responses and revisions address your concerns, we would be deeply grateful if you could kindly reconsider raising the score further. Thank you very much!

Reviewer_b91dOpenReview ↗
Rating
3
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper trains an orchestrator (Qwen3.5-9B with LoRA) that builds a multi-agent graph step by step over a frozen executor. Each edit is executed immediately, and rerun and drop are ordinary actions. The state also includes execution features and a small value head that predicts the reference policy's reward. Training uses AnchorTB, a reference-relative subtrajectory balance loss whose state flows are anchored to a per-task success-rate estimate. New skills, written by an external LLM, are admitted through paired rollouts and a sign test. Experiments cover 12 benchmarks with a 9B executor and show gains over AFlow, AgentFlow, FlowSteer, SkillFlow and SkillOpt, plus transfer to six other executors.

Critical strengths

The empirical study is broad. It covers QA, math, code and interactive tasks, with five-run means, executor transfer and per-component ablations, and the reported averages and ablation gaps are internally consistent. Making skill admission an explicit paired test, rather than relying on an LLM judge, is also a reasonable direction.

Critical weaknesses
  1. The main premise overstates the gap in prior work. The paper claims existing orchestrators keep the running team fixed and can only add agents. That is not accurate:

    • FlowSteer, one of the baselines, already edits a workflow step by step with add, delete and modify actions driven by execution feedback.
    • Puppeteer (NeurIPS 2025) trains an orchestrator that re-activates agents within a single run. Section 2 wrongly describes it as evolving between runs.
    • AgentFlow and Magentic-One also revise their plan mid-run.

    The paper itself says interleaved execution with feedback is "as in FlowSteer". The new parts reduce to rerun and drop actions, a feature vector and a value head. In addition, nothing evolves at test time, since the policy, value head and skill library are all frozen.

  2. AnchorTB combines known ideas, and the theory does not cover what is implemented. Relative TB, subtrajectory balance for LLMs (Hu et al., 2024) and flow-balance RL for LLMs (FlowRL) already exist. The anchor is essentially a per-task baseline, as in GRPO or RLOO. The appendix's own remarks show the gap: the leave-one-out anchor is not a shared flow (B.1, B.3), the loss optimum is not log Z (B.9), and the executor is stochastic, so the target is not reachable (A.10, A.11). The main text still claims training toward that target. Propositions 1 to 3 mostly restate the design.

  3. The skill admission test is not shown to be valid or useful in practice.

    • The appendix admits the error guarantee depends on an unverified premise (C.4). The implementation reuses the same task families to both propose and validate candidates, which is exactly the setting Remarks C.2 and C.3 show breaks the test.
    • Both arms are run by the reference policy, not the trained one.
    • With the alpha-spending schedule, a later candidate needs about 16 straight wins to be promoted, yet the paper never reports how many skills were promoted or retired.
    • The "no skill evolution" ablation also removes the paired rollouts, about a third of the training episodes, so its 5.27-point drop is confounded.
  4. The evaluation is not compute-matched, and the mechanism evidence is weak.

    • Budgets: EvoSteer uses up to 98k tokens and 50 tool calls per episode plus a stronger skill-writing model. Baseline budgets are not given, and there are no self-consistency or best-of-N baselines.
    • Token accounting: Fig. 6a reports about 19k tokens per problem, while the training totals in Appendix D imply about 34k per episode.
    • Variance: several cells show zero standard deviation across five sampled runs, with no significance tests on margins under 2 points.
    • Fig. 6c: the comparison is a strawman, since a per-dataset constant has AUC 0.5 by design.
    • Fig. 6e: any edit that produces a first answer counts as an improvement.
    • Case study: the value head, by the authors' own account, does not see the code, so its drop cannot reflect the bug. The "3 of 3 tests" are the original MBPP tests, not MBPP+.
Questions for authors
  1. What can EvoSteer do within a single run that FlowSteer, Puppeteer and AgentFlow cannot?
  2. How many skills were proposed, promoted and retired during training?
  3. What were the inference budgets for each baseline, and what is EvoSteer's actual cost per benchmark?
  4. Is MBPP+ scored with the EvalPlus extended tests? Why do some cells show zero variance?
  5. Can you rerun the skill ablation while keeping the paired rollout budget?
5 comments
AuthorsOpenReview ↗

Response to Reviewer b91d (1/3): Differences from prior orchestrators and objectives

We thank the reviewer for the careful review and valuable suggestions, and answer each point below.

W1 / Q1. Differences from prior orchestrators.

The table summarizes how EvoSteer differs from these systems, most notably in that a plausible but failing agent is corrected within the run instead of surfacing only when the task fails. We first clarify that "as in FlowSteer" (Sec. 5.3) refers only to the format of the text feedback, not to interleaved execution: that ablation keeps textual feedback after each edit and removes EvoSteer's execution features, whereas FlowSteer's add, delete and modify actions edit a workflow that has not yet run: its feedback only checks whether an edit is valid, and the workflow is executed once finished (its Eq. 2). Our revised Section 2 describes how Puppeteer updates its policy across tasks; its within-run behavior is in the table.

FlowSteerPuppeteerAgentFlowMagentic-OneEvoSteer
Paradigmworkflow design; answer from the finished graphsequential agent calls; outputs aggregatedplanner loop until the verifier stopsledger-driven loop, replanningbuild, execute, repair in one loop
Orchestration unithand-written operatorsone pool agent per stepplanner + fixed executor / verifier / generatorfixed five-agent teamtool-using agents with roles and skills on a typed graph
Observed after each actionedit validity, repair hintsagent outputtool result, verifier verdictprompted progress ledgerexecution result, execution features, value estimate
Redo an executed stepno rerunre-call as a new stepnew sub-goal next turnreplan, reset all agentsrerun / drop one agent node
Repair / stop signal––LLM verifier (binary)prompted stall checklearned value estimate v^\hat v
Credit assignmenttrajectory-level advantagetrajectory returnsame signal per turn– (not trained)per-action (AnchorTB)
Capability growthhand-added operators–––validated skill admission

Within a single run (Q1). The running team itself evolves: after each edit the orchestrator sees the executed result, its features and v^\hat v, and can rerun or drop the failing agent while the rest of the team keeps running, whereas the other systems add a step or turn, reset the team, or cannot redo a step.

Beyond rerun/drop, features and value head. These three are only the decision interface; the difference runs through every row of the table: one policy decides team size, communication, repairs and stopping under a shared budget, each decision is credited separately, and capabilities grow through validated skills rather than hand-added operators.

Evaluation protocol. What self-evolves at test time is the team: every episode starts from an empty team that is built, executed and repaired within it (Sec. 4.1), while the policy, value head and library stay frozen (App. D). This is the standard and stricter protocol: test items never update the model and six benchmarks are held out entirely, so the gains on them (Table 1(b)) come from what the frozen system does inside each episode.

W2. AnchorTB versus prior objectives.

AnchorTB builds on (sub)trajectory balance (Sec. 2) and changes where the state flows come from, so that each action receives its own credit and a low score can be traced to the task or to the team structure (Sec. 1).

ObjectiveSpansIntermediate flowsCredit
Relative TB / FlowRLfull trajectory– (learned log⁡Z\log Z)shared by all actions
SubTB for LLMs (Hu et al., 2024)all subtrajectoriesreward of each terminated prefixper step; every prefix must be scorable
GRPO / RLOO–– (baseline from the current policy's group)shared by all actions
AnchorTBall subtrajectoriesmeasured on frozen-reference rollouts, + c(g)+bψc(g)+b_\psiper action (Prop. A.14)

Relative TB and FlowRL give all actions one coefficient and learn log⁡Z\log Z, biased under a stochastic executor (Prop. B.9); AnchorTB gives each action its own coefficient from the spans that contain it (Prop. 2) and measures the normalizer from reference rollouts, learning only a residual. SubTB for LLMs needs a reward at every prefix, which a partially built team does not have; AnchorTB instead measures intermediate flows from reference visits to the same team structure. GRPO-style objectives (AgentFlow, FlowSteer) and RLOO share one advantage and take their baseline from the policy being trained, which mixes task difficulty with that policy's own structural choices; our anchor is measured under a fixed reference and is only one part of the flow. Each of these choices matters (Table 2), and with only the loss changed AnchorTB outperforms GRPO, PPO, TB and tempered TB at comparable compute (Fig. 6a,b). We have added both tables, with discussion, at the start of the appendix.

AuthorsOpenReview ↗

Response to Reviewer b91d (2/3): Theory, validated skill admission, and mechanism evidence

W2 (theory). The appendix separates three levels. First, Prop. 2's per-action credit holds exactly for the implemented loss with any anchor values, leave-one-out included, since it uses only one trajectory's values. Second, leave-one-out keeps a trajectory's own reward out of its anchor, as in RLOO; a freely learned normalizer, the exactly shared alternative, is biased (Prop. B.9), which is why AnchorTB measures it (Eq. 8). Third, recovering the target distribution needs a shared flow, as in all balance objectives (Remark B.3), and the flow on the trained policy's own trajectories is shared; only reference rollouts exclude themselves. As Sec. 3 states, a stochastic executor makes the target unreachable only if it violates the condition of Prop. A.10; App. A then gives the best attainable policy and the remaining gap. Props. 1–3 deliberately summarize, at the method level, results proven in App. C.6: Lemma A.1, Prop. A.14, and Props. C.2, C.5 and C.6, respectively.

W3 / Q2 / Q5. Validated skill admission.

Error control (C.4, C.2–C.3). The premise of Remark C.4 is that each test is valid given what was known when its candidate was registered; here, validation pairs must be fresh outcomes, neither used to choose the candidate nor kept selectively (Assumption C.2). Remarks C.2 and C.3 show two ways to break this premise, and neither occurs in our implementation:

  1. C.3 reuses the outcomes that selected a candidate. Our author distills a candidate from past trajectories, and the test counts only pairs launched after the candidate is registered. Proposal and validation share the task family the skill serves, not the outcomes.
  2. C.2 copies one outcome across many pairs. Each of our pairs is a new rollout on a task drawn at that step, kept whenever it is complete and side-effect-free, regardless of its outcome.

Validated admission raises all six IID datasets, whereas admitting every candidate lowers MedQA and AIME (Fig. 6d), where the test retired one candidate each (Q2).

Reference arms. Running both arms with the frozen ρ\rho is what makes the test valid: arms run by the updating πθ\pi_\theta would change the pair distribution between pairs and mix the skill's effect with policy updates. The trained policy is not left out: candidates can be bound in-run before validation (Sec. 4.3), so πθ\pi_\theta tries each skill in its own rollouts while ρ\rho measures whether it helps.

Q2. Over training, no candidate was left pending:

HotpotQANQ-OpenMedQAAIMEMBPP+ALFWorldTotal
Proposed / promoted / retired2 / 1 / 11 / 1 / 02 / 1 / 12 / 1 / 14 / 3 / 13 / 2 / 114 / 9 / 5
Pairs N (discordant D)1,299 (143)1,232 (292)1,281 (416)1,281 (406)1,324 (344)1,293 (323)7,710 (1,924)

The rising threshold is met by accumulated evidence, not an unbroken run of wins: every family gathers over a hundred discordant pairs.

Q5. We reran the skill ablation without skills but with the full paired-rollout budget (five-run means, now in Table 2; IID / OOD: mean of the six benchmarks).

VariantHotpotQANQMedQAAIMEMBPP+ALFWorldIIDTriviaQAMuSiQueGPQAMATHSWEWebShopOOD
EvoSteer (full)92.3485.0093.1374.6792.1989.6987.8495.6384.5385.6395.4742.5090.6382.40
No skills, no pairs88.2880.9490.1668.6784.5382.8182.5694.3880.6383.2892.8138.5985.1679.14
No skills, paired budget kept88.9181.0990.9468.6784.2283.7582.9394.6981.2583.5993.1338.9186.8879.74
Fixed-count, k = 590.4781.7289.2266.0090.0078.5982.6794.8481.2583.2894.5335.0086.0979.17

With the paired budget kept, removing the skills still costs most of the drop, so the drop comes from the skills. The fixed-count row keeps the skill mechanism and changes only the admission rule, so its drop reflects the sequential test itself.

W4 (mechanism evidence).

Fig. 6c. The comparator is the category level of the anchor's own shrinkage hierarchy (App. B.2), so the figure isolates what the task-level leave-one-out term adds. It is recomputed from the reference pool at every step, so it is not constant within a dataset (AUC 0.48–0.53, not exactly 0.5).

Fig. 6e. The add-agent slice includes the edit that produces the first answer, which we have stated in App. D. The claim about in-run correction rests on the repair slice: a repair reruns or drops an agent that has already executed, so the score before it reflects an executed result rather than an empty team, and repairs raise it in 80.7% of cases and lower it in 7.0%. The ablations agree without this definition: masking rerun and drop costs 3.57 IID points (Table 2), and value-guided replanning helps on every IID dataset (Fig. 6f).

AuthorsOpenReview ↗

Response to Reviewer b91d (3/3): Budgets, variance, significance, and the MBPP+ case

W4 (evaluation) / Q3 / Q4.

Budgets and cost (Q3). Each baseline runs with its authors' budget. The table gives the budget and the measured test-time tokens per problem (mean over the twelve benchmarks):

MethodBudget cap (tokens / calls / time)Tokens / problemIID avgOOD avg
Qwen3.5-9B8k / 1 / –3.2k54.5247.14
Self-consistency@66 samples19.7k56.9248.26
Best-of-N@44 samples + self-scoring20.3k55.4747.97
SFT8k / 1 / –3.6k52.3248.41
GRPO†8k / 1 / –4.9k55.9149.35
AFlow32k context / 5 calls / –16.0k77.4670.52
AgentFlow8k per call / 10 steps / 1,800 s31.6k79.8075.89
FlowSteer0.5k + 4k per call / 20 rounds / 2,400 s43.4k82.1177.84
SkillFlow0.5k + 8k per call / ≤28 steps / 600 s22.0k83.0678.05
SkillOpt8k / 1 (ALFWorld: 50 steps) / –20.1k84.0579.58
EvoSteer0.25k + 12k per call / 50 / 600 s21.7k87.8482.40

EvoSteer scores highest at a per-problem cost within the range of the other orchestrators. At nearly the same tokens, self-consistency and best-of-N on the same backbone add only a few points over a single call: extra samples cannot supply retrieval, execution feedback or in-run repair, which is where EvoSteer's gains come from. EvoSteer's measured cost per problem is:

HotpotQANQMedQAAIMEMBPP+ALFWorldTriviaQAMuSiQueGPQAMATHSWEWebShop
Tokens (k)23.67.611.820.417.335.06.536.112.811.259.419.2

The skill author is called only in training author windows; at test time the library is frozen, the author is not called, and no reference, paired or validation rollouts are drawn (App. D). With the executor itself as author, EvoSteer still stays above the strongest baseline in every column (App. D).

Token accounting. The 19.3k per problem in Fig. 6a counts one rollout of the trained policy and matches the IID average of the measured costs above. The 1.58B training total is 40% rollouts (natural and paired) and 60% prefill-only rescoring of every action under θ\theta and ρ\rho for the AnchorTB loss, with the skill author at 0.004%; dividing it by the number of episodes counts these rescoring passes as rollout cost, which is where the 34k comes from.

Zero variance (Q4). EvoSteer samples at test time as in training (orchestrator 1.0, executor 0.3; Table 3), and each score is the mean of five independent runs. A ±0.00 means that the same number of items was solved in all five runs, which happens when most items are solved, or failed, in every run; F1 on the same runs still varies. Six of the seven such cells are baselines; we have added their per-run counts and the test temperatures to App. D.

Significance. EvoSteer minus the strongest baseline on every benchmark (Welch's t-test over the five runs; EM / F1 for question answering):

HotpotQANQMedQAAIMEMBPP+ALFWorldTriviaQAMuSiQueGPQAMATHSWEWebShop
Δ\Delta+3.12 / +0.84+2.34 / +2.36+1.41+8.67+2.03+3.91+3.29 / +3.40+3.44 / +1.75+1.88+0.78+0.94+4.69
p<.001 / .004.002 / .001.023<.001.011<.001.003 / .003<.001 / .009.002.15.059<.001

All six average rows of Table 1 give p < 0.001. On MATH-Hard both methods exceed 94.6, close to the ceiling.

Case study. The case study does not claim that v^\hat v reads the bug; App. E states that it sees execution features and the task type, not the code. v^\hat v estimates the reference continuation reward given these features (Prop. B.8), and at Round 4 it falls to 0.353: under ρ\rho, such states are estimated to succeed about a third of the time. In the same batch, both reference rollouts of this problem submitted code that passed the visible test and failed a hidden one. The orchestrator acts on this risk by rerunning the agent, and the bound skill's first pitfall guides the fix.

MBPP+ (Q4). Yes. Both the training reward and the reported Pass@1 run each problem's full EvalPlus test, with original and extended inputs, as implemented in our anonymous repository (_code_score); the prompt shows only the first original assertion, as in the EvalPlus MBPP+ prompt. In the case study, the corrected code passes all 113 EvalPlus inputs of that problem; the three tests listed there are the original assertions, shown for readability, and App. E has been adjusted accordingly.


We thank the reviewer again. All additional experiments and revisions have been incorporated into the revised version. If our responses and revisions address your concerns, we would be deeply grateful if you could kindly reconsider raising the score further. Thank you very much!

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.