acceptodds
Under review as a conference paper at ICLR 2027

SReF: Structure–Resource Factorization for Multi-Agent LLM Orchestration

Abstract

Multi-agent systems built from heterogeneous large language models decide both what computation to perform and which model should perform it. Coupling these decisions can make changes in the available experts unnecessarily affect an otherwise useful planning policy. We introduce SReF, a Structure-Resource Factorization framework that separates a model-identity-agnostic Graph Planner from a descriptor-conditioned Expert Router. The Planner generates computation graphs and issues Patches, while the Router assigns available experts to reached nodes. To train these distinct decisions, the Planner receives feedback over verified segments along executed dependencies, and the Router learns from model-relative comparisons under matched node contexts. Segment-level verification supports local training feedback and history-preserving correction of pending computation without verifying every node independently. This separation supports resource-local adaptation through the Router while retaining the learned Planner and its ability to Patch pending computation in response to execution feedback. Across four benchmarks, SReF improves Pass@1 over the strongest routing or orchestration baselines by 1.8-4.5 percentage points and over its NoPatch variant by 1.6-4.9 percentage points. Planner-credit experiments compare terminal-only and segment-level Planner credit under the same verification and Patch protocol, and examine the additional training time of finer node-level feedback. Under expert-pool insertion and removal, direct transfer with the Planner retained approaches the performance obtained by Router-only and joint adaptation. Our code for SReF is released at https://anonymous.4open.science/r/SReF-007E

Primary area:
foundation or frontier models, including LLMs
Keywords:
Multi-Agent Systems, LLM Orchestration, Reinforcement Learning, Large Language Models

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_rSPAOpenReview ↗
Rating
2
Confidence
2
Informativeness
2(+1)
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies whether computation planning and model allocation can be learned separately in heterogeneous LLM orchestration. The authors propose SReF, which combines a model-identity-agnostic Graph Planner with a descriptor-conditioned Expert Router. Using this design, a frozen verifier evaluates executed segments and can trigger history-preserving modifications of pending computation. The Planner is then trained using dependency-aware segment feedback attributed to the source of executed node versions, while the Router learns from matched-context comparisons between candidate experts. The empirical results validate the improvements on four benchmarks and necessary ablations and analysis.

Critical strengths

S1. The manuscript successfully demonstrates a substantial technical improvement to an existing methodological design.

S2. The writing is clear and readable.

S3. The paper presents encouraging empirical results and useful controls. Specifically, the reported improvements span various benchmarks and the ablations are technically informative.

Critical weaknesses

W1. The authors should state that why should this particular separation of planning, routing, verification, and credit assignment be preferable to simpler alternatives with comparable supervision and computation. The proposed SReF’s more distinctive contribution appears to be its dependency- and provenance-aware credit under graph repair. However, Table 2 changes local reward availability as well as credit construction, leaving unclear whether the proposed propagation mechanism is responsible for the gains rather than intermediate supervision more generally.

W2. The justification of factorization and structural credit remains limited. Though Appendix C.3 establishes that the dependency trace is well-defined, while Eq. 35 establishes conditional Planner invariance by construction. This is not a requirement for a universal optimality theorem, but the paper would benefit from clearer assumptions and targeted empirical evidence for these design choices.

W3. End-to-end efficiency and resource matching should be taken into consideration. As presented in the manuscript, SReF relies on an external verifier with substantial invocation overhead, while the presneted cost breakdown omits Planner calls and does not provide corresponding totals for the baselines. It is also unclear whether the closest baselines receive comparable verification, repair, and training-supervision resources.

Questions for authors

See the weaknesses above.

5 comments
AuthorsOpenReview ↗

Response to Reviewer rSPA(1/4)

Thank you for your careful review and constructive comments. We appreciate your positive assessment of the technical improvement, writing clarity, and empirical results. We respond to your concerns and questions point by point below.

W1: Design motivation and the independent contribution of credit propagation

The authors should state that why should this particular separation of planning, routing, verification, and credit assignment be preferable to simpler alternatives with comparable supervision and computation. The proposed SReF’s more distinctive contribution appears to be its dependency- and provenance-aware credit under graph repair. However, Table 2 changes local reward availability as well as credit construction, leaving unclear whether the proposed propagation mechanism is responsible for the gains rather than intermediate supervision more generally.

SReF addresses two related issues: execution failures may arise from subtasks and their dependencies or from the choice of executing experts; after graph repair, the final result may also be jointly produced by the initial plan and subsequent Patches. We therefore separate planning from routing and construct different feedback for these two types of decisions.

For the Router, we fix the node and its dependency inputs and compare outputs from different experts to learn model selection for the same subtask. For the Planner, we reuse the segment-level verification scores collected during execution to provide more local structural feedback than terminal rewards alone.

Our credit mechanism further accounts for execution over a dependency graph whose computation structure may be modified by Patch. Nodes at different positions in the graph have different responsibilities: upstream nodes provide the basis for subsequent computation, while downstream nodes use, check, or integrate existing results. A segment’s local quality and the outcomes of the downstream execution it supports are therefore related but distinct feedback signals. Assigning each segment only the terminal reward and its own verification score provides intermediate supervision, but does not use inter-segment dependencies to distinguish the paths along which downstream feedback should propagate. The full method therefore propagates credit along actual execution dependencies, allowing upstream structural decisions to receive feedback from downstream segments that depend on them.

Meanwhile, Patch changes pending computation, so a single executed segment may mix nodes from the initial plan and later repairs. We attribute each node’s credit to the planning output that last created or modified its actually executed version: retained nodes continue to update their original source, while newly created or modified nodes update the corresponding Patch output. In this way, feedback is not assigned wholesale to the most recent Patch merely because the nodes appear in the latest graph.

Thus, segment-level scoring provides local quality feedback, dependency propagation organizes feedback across segments using the executed graph, and provenance attribution returns feedback to the planning decisions that actually produced the node versions. Together, these designs address the training requirements of graph execution and history-preserving repair.

To distinguish the contributions of local supervision and dependency propagation, we added a control that retains the same segment-level supervision but removes inter-segment credit propagation. Each segment directly combines the terminal structural signal with its own local verification reward to construct training credit, without aggregating feedback from other segments. The verification and Patch protocol, reward weights, structural regularization terms, and provenance-attribution rules are unchanged. Given the rebuttal time constraints, we added retraining and evaluation in the MATH-500 setting. Below, we report the mean ± sample standard deviation of Pass@1 (%) across three evaluation runs:

Credit assignmentMATH-500
Global92.1 ± 0.2
Segment, without propagation93.3 ± 0.4
Segment-level (SReF)93.5 ± 0.2
AuthorsOpenReview ↗

Response to Reviewer rSPA(2/4)

Adding segment-level supervision without credit propagation already improves accuracy by 1.2 percentage points, showing that intermediate supervision effectively improves training outcomes in the current setting. The full method provides a further 0.2-percentage-point improvement, but this difference is small relative to the evaluation variability. We will investigate these issues further in subsequent experiments. Taken together, dependency propagation and provenance attribution provide a way to organize feedback using the executed graph and Patch history. We will accordingly distinguish the design motivation of these mechanisms from their empirically demonstrated performance contribution, rather than attributing all gains to propagation.

Revision. We have added the segment-supervision control without propagation to Appendix G.4.

W2: Assumptions, theoretical scope, and empirical evidence for factorization and structural credit

The justification of factorization and structural credit remains limited. Though Appendix C.3 establishes that the dependency trace is well-defined, while Eq. 35 establishes conditional Planner invariance by construction. This is not a requirement for a universal optimality theorem, but the paper would benefit from clearer assumptions and targeted empirical evidence for these design choices.

Our theoretical results establish the mathematical properties of the credit recurrence and the conditional invariance of the Planner interface, respectively. They do not prove that structure–resource separation is universally optimal.

Theoretical scope of dependency credit. The result in Appendix C.3 is based on a finite executed segment-level DAG. Given rewards, execution durations, edge-attribution coefficients, and old critic predictions, it establishes that the dependency-credit recurrence has a unique finite solution that can be computed in reverse topological order. Historical execution records and retry nodes remain in the executed graph, so the result also applies to trajectories containing Patches. This ensures that credit computation is well-defined; it does not imply an unbiased estimate of the causal contribution of structural decisions, nor does it guarantee superiority over other credit-assignment methods.

Conditions for Planner invariance. Equation 35 applies with the same Planner input, fixed parameters, and identical decoding settings. The Planner’s input excludes expert identities and resource descriptors, so changing only the expert pool does not change the conditional planning policy for a given input. However, different expert outputs can change verification feedback and subsequent Patch contexts, so the executed graph may still change. This interface property does not imply that a frozen Planner remains optimal for the changed expert pool, nor does it guarantee that Router-only adaptation is equivalent to joint training.

The practical premise of this design is that existing task decompositions and dependency structures retain value after an expert-pool change, and that some differences in execution capability can be handled through resource reallocation or feedback-driven Patch. If an expert-pool change makes the original decomposition unsuitable, updating the Planner may still be necessary.

Targeted supplementary experiments. We added an experiment removing Qwen3.5-27B, the expert with the highest mean calibrated capability in the original pool, and a joint-policy control trained on randomized pools. Below, we report the mean ± sample standard deviation of Pass@1 (%) across three evaluation runs:

MethodPrePost-RPost-All
SReF92.8 ± 0.292.8 ± 0.292.9 ± 0.3
Fixed-ID Router90.1 ± 0.192.0 ± 0.493.0 ± 0.2
Planner-Integrated91.7 ± 0.3N/A92.0 ± 0.0
Planner-Integrated + randomized-pool training92.7 ± 0.2N/A92.6 ± 0.1

SReF’s direct-transfer performance is close to its Router-only and joint-adaptation results, supporting the feasibility of retaining the Planner in this expert-removal setting. Meanwhile, randomized-pool training improves the joint policy’s direct-transfer performance from 91.7% to 92.7%, close to SReF. This shows that joint policies can also acquire robustness through training. We therefore limit our conclusion to the effectiveness of the factorized design under the tested conditions, without claiming that it universally outperforms joint policies.

We will explicitly distinguish these theoretical properties from the empirical conclusions in the revised manuscript. The contribution of dependency propagation itself is addressed separately in W1.

AuthorsOpenReview ↗

Response to Reviewer rSPA(3/4)

W3: Complete inference overhead and resource matching

End-to-end efficiency and resource matching should be taken into consideration. As presented in the manuscript, SReF relies on an external verifier with substantial invocation overhead, while the presneted cost breakdown omits Planner calls and does not provide corresponding totals for the baselines. It is also unclear whether the closest baselines receive comparable verification, repair, and training-supervision resources.

We thank the reviewer for pointing out the insufficient reporting of cost breakdowns and resource allocation. SReF’s segment-level verification and Planner Patch are part of its complete execution procedure and do introduce additional overhead. We have added inference token statistics that include Planner calls and a GraphPlanner control with verification and repair.

Planner token usage

DatasetMean input tokens/questionMean output tokens/question
MATH-500790.0143.0
MMLU-Pro900.1140.0
HumanEval861.3236.7
GSM-Hard778.7144.1

Mean token usage by method

We must emphasize that these statistics were collected from our local implementations using the same accounting convention as SReF. Each method is accounted for using the calls actually made during inference, including the inputs and outputs of planning, verification, repair, retry, and reward-model calls where applicable. Some methods were also adapted to our small-model pool; without such adaptation, their direct execution performed substantially worse than running an individual model. SReF’s total includes Planner, expert, and verifier usage. Verification accounts for a large share of our token consumption, so finding an inexpensive and accurate verifier is a direction for our next stage of work.

MethodMean tokens/question
Phi-4604.8
Qwen3.5-27B486.3
DeepSeek V4.0 Pro379.0
RouterDC543.8
GraphRouter533.2
Router-R11423.1
xRouter1392.6
R2-Reasoner3205.6
MasRouter3964.7
GraphPlanner5767.4
GraphPlanner + verification and repair8916.1
Puppeteer3751.3
Uno-Orchestra2761.3
SReF4,777.8

Verification and repair resources. The original baselines retain their respective execution procedures, without additional access to SReF’s segment-level verifier and Planner Patch. To examine the effect of the additional verification and repair resources, we integrated the same verifier used by SReF after GraphPlanner’s final output, keeping the verification threshold and maximum number of repairs identical. If verification fails, the relevant context is passed to a designated model to regenerate the answer. Evaluation uses the same expert pool.

Reviewer_JBRSOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

SReF separates graph planning and verification-triggered repair, done by a Planner that never sees model identities, from descriptor-conditioned expert routing. The Planner is trained with dependency-aware segment credit and cross-Patch provenance, and the Router with matched-context expert comparisons. The paper reports Pass@1 gains of 1.8–4.5 points over the strongest compared routing or orchestration baseline on four benchmarks and presents initial evidence that the Planner can remain fixed under the tested pool changes.

Critical strengths
  1. A clean factorization of structure from resources. Separating a Planner that never sees model identities from a Router that assigns experts by their descriptors gives a simple, enforceable interface: when the expert pool changes, the learned Planner is retained and adaptation is confined to the routing side. The invariance claim is carefully scoped.

  2. A principled credit design with favorable ablations. Dependency-aware segment credit generalizes advantage estimation to executed graphs, with a finite-trace proposition; provenance attribution handles patched segments that mix node versions; and the Router learns from matched-context comparisons of candidate experts. Under the same verification and Patch protocol, the segment-level variant scores higher than the terminal-only variant in both tested settings, and the variant without Patch still leads the strongest compared routing or orchestration baseline on most benchmarks, so inference-time repair alone does not explain the gains.

Critical weaknesses
  1. The evidence that the Planner can stay fixed when the expert pool changes is narrow. This is the paper's central argument, but Table 3 tests it only on MATH-500. It adds one expert, which is the same model as the verifier, run as a separate instance, and removes one that is not the highest-mean original expert (Table 8). Removal favors SReF, whose Pass@1 is unchanged while the controls drop, but under insertion SReF's direct transfer beats the Planner-Integrated control by only 0.4 points (94.1 versus 93.7), and the changed-pool results give neither run counts nor dispersion. A closely related design also goes untested: Conductor (Nielsen et al., 2026, "Learning to Orchestrate Agents in Natural Language with the Conductor"), an RL orchestrator over pools of worker LLMs, trains one coordinating policy on randomized agent pools so that it adapts to new agent sets, but it is neither cited nor compared.

  2. Comparative inference cost and resource matching are not established. At inference, SReF runs a verifier whose tokens amount to 0.82–1.24 times the expert-pool tokens (Tables 16–17), Planner calls are not itemized, and no baseline costs are reported, even though the single-model DeepSeek V4.0 Pro reference already exceeds SReF on MMLU-Pro (74.1 versus 73.7, Table 1). The comparison may also favor SReF in training resources. Its Planner is warm-started on filtered GPT-5.6 trajectories and trained with rewards from the same verifier (Appendices C.6 and D.3), and the supervised SReF-SFT variant, before decision-specific policy training, already beats the strongest compared routing or orchestration baseline on GSM-Hard and MMLU-Pro. The baselines are adapted to the SReF Planner backbone "as far as possible," using the shared expert pool and deployment protocol, but their teacher data and verifier and Patch access are not specified.

Questions for authors
  1. Resources: What teacher data, verifier access, and training, verification, and repair budgets did each baseline receive, and what are the complete inference costs of every method, including Planner and verifier calls?
  2. Matched control: Do SReF's gains over the strongest routing or orchestration baseline hold when that baseline receives the same verifier and Patch access, or when SReF's Planner is trained without the GPT-5.6 teacher trajectories?
  3. Uncertainty: How many runs underlie Tables 1 and 2 and the Insert and Remove means in Table 3, and do paired intervals over the evaluation questions support the smaller differences, such as the 1.8-point HumanEval margin and the 1.4-point MATH-500 credit gain?
  4. Patch outcomes: For Table 1's evaluation samples, how often is a Patch triggered and how often does it repair the triggering failure, reported separately from Table 16's cost run, and how are invalid graphs and Patches counted?
  5. Pool robustness: How does a Planner-Integrated control trained on randomized pools, as in Conductor, perform under the insertion and removal changes, and how does SReF fare when the highest-mean expert is removed?
7 comments
AuthorsOpenReview ↗

Response to Reviewer JBRS(1/5)

Thank you for your careful review and constructive comments. We appreciate your positive assessment of the structure–resource factorization, credit design, and empirical controls. We respond to your concerns and questions point by point below.

Q3: Number of evaluation runs and variability across tables

Uncertainty: How many runs underlie Tables 1 and 2 and the Insert and Remove means in Table 3, and do paired intervals over the evaluation questions support the smaller differences, such as the 1.8-point HumanEval margin and the 1.4-point MATH-500 credit gain?

For Table 1, we performed two independent complete system executions for each evaluation question and computed Pass@1 and Pass@2 from these two samples. Each complete execution includes the method’s planning and expert calls, as well as verification and repair where applicable.

We have supplemented each experimental setting in Tables 2 and 3 to include three evaluation runs. Below, we report the mean ± sample standard deviation across these three evaluations, in Pass@1 (%).

Table 2: Credit-assignment controls

Credit assignmentMATH-500HumanEval
Global92.1 ± 0.289.2 ± 0.7
Segment-level93.5 ± 0.295.3 ± 0.3
Node-level93.7 ± 0.395.7 ± 0.6

Compared with Global, Segment-level improves performance by 1.4 and 6.1 percentage points on MATH-500 and HumanEval, respectively. The further gains from Node-level are smaller: 0.2 and 0.4 percentage points, respectively.

Table 3: Expert-pool changes

MethodOrig.Insert: PreInsert: Post-RInsert: Post-AllRemove: PreRemove: Post-RRemove: Post-All
SReF93.5 ± 0.194.1 ± 0.193.8 ± 0.294.4 ± 0.493.3 ± 0.293.1 ± 0.393.3 ± 0.3
Fixed-ID Router93.4 ± 0.093.3 ± 0.194.0 ± 0.094.2 ± 0.090.7 ± 0.292.5 ± 0.293.3 ± 0.2
Planner-Integrated93.1 ± 0.593.7 ± 0.3N/A94.1 ± 0.291.9 ± 0.2N/A92.6 ± 0.2

These results show that, under the expert-pool changes tested, SReF’s mean direct-transfer performance is close to its Router-only and joint-adaptation performance, and Router-only updates do not provide a consistent improvement. We therefore limit our conclusion to the following: direct transfer while retaining the Planner remains competitive in these settings.

Regarding the reliability of smaller performance differences, our comparisons primarily concern mean accuracy. The 1.8-percentage-point difference on HumanEval in Table 1 corresponds to approximately 3 questions out of 164 per run, while the 1.4-percentage-point difference on MATH-500 in Table 2 corresponds to approximately 7 questions out of 500 per run. We have added repeated evaluations and report standard deviations to show the variability of the results and reduce the influence of randomness.

Revision. We have updated Tables 2 and 3 with means and sample standard deviations over three evaluations.

W1 & Q5: More demanding expert-pool changes and the relationship to Conductor

The evidence that the Planner can stay fixed when the expert pool changes is narrow. This is the paper's central argument, but Table 3 tests it only on MATH-500. It adds one expert, which is the same model as the verifier, run as a separate instance, and removes one that is not the highest-mean original expert (Table 8). Removal favors SReF, whose Pass@1 is unchanged while the controls drop, but under insertion SReF's direct transfer beats the Planner-Integrated control by only 0.4 points (94.1 versus 93.7), and the changed-pool results give neither run counts nor dispersion. A closely related design also goes untested: Conductor (Nielsen et al., 2026, "Learning to Orchestrate Agents in Natural Language with the Conductor"), an RL orchestrator over pools of worker LLMs, trains one coordinating policy on randomized agent pools so that it adapts to new agent sets, but it is neither cited nor compared.

Pool robustness: How does a Planner-Integrated control trained on randomized pools, as in Conductor, perform under the insertion and removal changes, and how does SReF fare when the highest-mean expert is removed?

AuthorsOpenReview ↗

Response to Reviewer JBRS(2/5)

Removing the expert with the highest mean calibrated capability in the original pool. Following the reviewer’s suggestion, we additionally removed Qwen3.5-27B. Below, we report the mean ± sample standard deviation of Pass@1 (%) across three evaluation runs:

MethodPrePost-RPost-All
SReF92.8 ± 0.292.8 ± 0.292.9 ± 0.3
Fixed-ID Router90.1 ± 0.192.0 ± 0.493.0 ± 0.2
Planner-Integrated91.7 ± 0.3N/A92.0 ± 0.0

After removing Qwen3.5-27B, SReF’s mean direct-transfer accuracy decreases by 0.7 percentage points from the original-pool result of 93.5% and remains close to its Router-only and joint-adaptation results. With direct transfer, SReF outperforms Fixed-ID Router by 2.7 percentage points and Planner-Integrated by 1.1 percentage points. Fixed-ID Router recovers to a similar level after adaptation. These results support the conclusion that, in this expert-removal setting, retaining the Planner and transferring directly can maintain competitive performance without requiring a Planner update to recover performance.

Relationship to Conductor and the randomized-pool training control. We thank the reviewer for pointing out this related work. Conductor uses reinforcement learning to train a coordinating policy in which the same model generates subtasks, assigns executing experts, and specifies communication relationships. It then fine-tunes the trained coordinator on randomized expert subsets to improve adaptation to different expert combinations. This shows that a joint orchestration policy can also acquire expert-pool adaptability through appropriate training. We will add a citation to this work in the revised manuscript. Conductor, Nielsen et al., 2026.

SReF separates structural planning from resource allocation through its interfaces: the Planner generates a computation structure without model identities, while the Router selects executing models based on expert descriptors. Our experiments focus on whether the trained Planner can be retained after expert-pool changes. Randomized-pool training and this interface separation are compatible and can also be combined.

To address the reviewer’s suggestion, we adopted Conductor’s idea of expert-pool randomization and added a Planner-Integrated + randomized-pool training control. The original expert pool contains five models. During training, three are randomly selected each time to form the available pool, and the joint policy is given the list of currently available experts. During Post-All adaptation after insertion, each training pool always includes the new model and two randomly selected models from the original five. Insert: Pre involves no training on the new model. Below, we report the mean ± sample standard deviation of Pass@1 (%) across three evaluation runs.

MethodOrig.Insert: PreInsert: Post-AllRemove-9B: PreRemove-9B: Post-AllRemove-27B: PreRemove-27B: Post-All
SReF93.5 ± 0.194.1 ± 0.194.4 ± 0.493.3 ± 0.293.3 ± 0.392.8 ± 0.292.9 ± 0.3
Planner-Integrated93.1 ± 0.593.7 ± 0.394.1 ± 0.291.9 ± 0.292.6 ± 0.291.7 ± 0.392.0 ± 0.0
Planner-Integrated + randomized-pool training93.2 ± 0.293.6 ± 0.494.0 ± 0.292.5 ± 0.192.7 ± 0.592.7 ± 0.292.6 ± 0.1

Orig. denotes the original full expert pool; Pre denotes direct evaluation on the target pool without additional parameter updates; Post-All denotes further updating all applicable policy parameters on the target pool.

AuthorsOpenReview ↗

Response to Reviewer JBRS(3/5)

Randomized-pool training helps reduce the joint policy’s dependence on particular experts. When Qwen3.5-9B and Qwen3.5-27B are removed, its direct-transfer performance improves from 91.9% to 92.5% and from 91.7% to 92.7%, respectively. The latter is already close to SReF’s 92.8%. This shows that joint policies can also improve robustness to expert removal through randomized-pool training. When a new expert unseen during training is inserted, the randomized-pool version performs similarly to the original Planner-Integrated: 93.6% versus 93.7% for Pre, and 94.0% versus 94.1% for Post-All. Thus, in the current setting, randomized-pool training is more helpful for removing existing experts, while its benefits for inserting an unseen expert are limited. SReF retains competitive direct-transfer performance under the tested changes, but these results also show that randomized-pool training is an effective way to improve the adaptability of joint policies.

Revision. We have added the expert-removal and randomized-pool controls to Appendices G.5 and G.6.

W2 & Q1: Teacher data, verification and repair resources, and complete inference cost

Comparative inference cost and resource matching are not established. At inference, SReF runs a verifier whose tokens amount to 0.82–1.24 times the expert-pool tokens (Tables 16–17), Planner calls are not itemized, and no baseline costs are reported, even though the single-model DeepSeek V4.0 Pro reference already exceeds SReF on MMLU-Pro (74.1 versus 73.7, Table 1). The comparison may also favor SReF in training resources. Its Planner is warm-started on filtered GPT-5.6 trajectories and trained with rewards from the same verifier (Appendices C.6 and D.3), and the supervised SReF-SFT variant, before decision-specific policy training, already beats the strongest compared routing or orchestration baseline on GSM-Hard and MMLU-Pro. The baselines are adapted to the SReF Planner backbone "as far as possible," using the shared expert pool and deployment protocol, but their teacher data and verifier and Patch access are not specified.

Resources: What teacher data, verifier access, and training, verification, and repair budgets did each baseline receive, and what are the complete inference costs of every method, including Planner and verifier calls?

Teacher data and training procedures. SReF’s Planner is initialized through supervised training on filtered initial-planning and Patch trajectories generated by GPT-5.6. The Router uses node contexts from the retained trajectories for matched-context expert comparisons and initialization. During subsequent training, the frozen verifier provides segment-level feedback and scores candidate Router outputs.

Each baseline retains the training objective and initialization procedure of its original method; Uno-Orchestra and R2-Reasoner also use SFT initialization. The use of SFT by different methods does not imply access to the same teacher data or supervision budget. Thus, sharing the Planner backbone and expert pool is insufficient to establish fully matched training resources. However, because the frameworks follow different procedures, it is difficult to make the initial conditions identical for every baseline.

Reviewer_XYBEOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

SReF splits orchestration into two parts. A Planner writes a task graph without naming models, and a Router picks one of five models for each node. A judge model checks parts of the graph and can trigger one rewrite of the steps not yet run. Across four benchmarks it beats twelve baselines by 1.8 to 4.5 points. The paper claims that when models are added or removed, only the Router needs retraining.

Critical strengths

The rewrite and credit mechanism is precisely specified and well controlled, with the same checkpoint run with and without rewrites and three crdeit schemes compared under one protocol. Segment-level credit clearly wins.

Critical weaknesses

The paper's own control undercuts its thesis: on MATH-500, a jointly trained planner scores 93.0 against SReF's 93.4. In the pool-change test, retraining the Router actually lowers accuracy. Only SReF gets the inference-time judge, which uses about as many tokens as all the experts combined. Phi-4 scores 38.1 on MMLU-Pro against about 70 in its own report. Most worrying, the Phi-4 token counts in Table 17 are an exact rescaling of the Phi-4-mini counts, so they weren't measured independently.

Questions for authors

Did any baseline get the judge or the teacher data? How was Table 17 produced? How many runs are there per table? For future work, test harsher pool changes, such as removing the strongest model.

3 comments
AuthorsOpenReview ↗

Response to Reviewer XYBE(1/3)

Thank you for your careful review and constructive comments. We appreciate your recognition of the clearly specified rewrite and credit mechanism and the controlled comparisons. We respond to your concerns and questions point by point below.

Q3: Number of evaluation runs for each table

How many runs are there per table?

For Table 1, we performed two independent complete system executions for each evaluation question and computed Pass@1 and Pass@2 from these two samples. We have supplemented each experimental setting in Tables 2 and 3 to include three evaluation runs. Below, we report the mean ± sample standard deviation across these three evaluations, in Pass@1 (%).

Table 2: Credit-assignment controls

Credit assignmentMATH-500HumanEval
Global92.1 ± 0.289.2 ± 0.7
Segment-level93.5 ± 0.295.3 ± 0.3
Node-level93.7 ± 0.395.7 ± 0.6

Table 3: Expert-pool changes

MethodOrig.Insert: PreInsert: Post-RInsert: Post-AllRemove: PreRemove: Post-RRemove: Post-All
SReF93.5 ± 0.194.1 ± 0.193.8 ± 0.294.4 ± 0.493.3 ± 0.293.1 ± 0.393.3 ± 0.3
Fixed-ID Router93.4 ± 0.093.3 ± 0.194.0 ± 0.094.2 ± 0.090.7 ± 0.292.5 ± 0.293.3 ± 0.2
Planner-Integrated93.1 ± 0.593.7 ± 0.3N/A94.1 ± 0.291.9 ± 0.2N/A92.6 ± 0.2

Revision. We have updated Tables 2 and 3 with means and sample standard deviations over three evaluations.

W3, W4 & Q2: Phi-model evaluation results and the token statistics in Table 17

Phi-4 scores 38.1 on MMLU-Pro against about 70 in its own report.

Most worrying, the Phi-4 token counts in Table 17 are an exact rescaling of the Phi-4-mini counts, so they weren't measured independently.

How was Table 17 produced?

During our experiments, we found that the old answer-extraction rules incorrectly processed some outputs from the Phi models. This mainly affected tasks such as MMLU-Pro that require an option letter or a numerical answer. For example, for the correct response “The answer is A,” the old rule could extract the initial letter “T” rather than the final option “A,” causing a correct response to be marked incorrect. We subsequently corrected the extraction rules and conducted multiple rounds of experiments. However, when preparing the manuscript, we mistakenly selected an earlier set of experimental results, which led to some outdated values being included in Tables 1 and 8.

We have corrected the relevant entries in Tables 1 and 8; the errors were mainly concentrated in the Phi models. After correction, Phi-4 achieves 60.3% Pass@1 and 63.9% Pass@2 on MMLU-Pro. All other experiments in the paper used the correct capability vectors, so the correction to Table 8 does not change the configurations actually used in those experiments.

The data in Table 17 come from the Pass@1 evaluation process for Table 1 and were aggregated from the actual invocation logs of each model. The token counts for the two models were independently aggregated from their respective per-invocation logs; neither was obtained by scaling the other model’s average usage. We were also surprised by the proportionality. Our investigation identified the following details of the execution procedure.

Our experiments use a pool of small models. When a question exceeds the pool’s capabilities, for example in mathematical or multiple-choice tasks, verification can fail at the final output. At this point, Patch generates a subtask named “Check the output and answer.” This subtask depends on the final node in the graph before patching, whose output is already a number or an answer option. We also found that, when candidate models offer similarly limited quality gains in these contexts, the Router’s cost term favors lower-cost models such as Phi-4, Phi-4-mini, and Qwen3.5-4B. Except on HumanEval, the Phi models were largely uncompetitive outside these subtasks and appeared only in these Patch subtasks.

AuthorsOpenReview ↗

Response to Reviewer XYBE(2/3)

Because the Phi models sometimes do not follow the required final-answer format, when they are assigned to the final-output nodes in mathematical or multiple-choice tasks, we make an additional call to the same model after it completes the node execution to summarize and format its answer. This is part of the actual execution procedure and incurs additional model calls and token usage. These additional formatting calls are not included in the Calls column. In the original table, both their input and output tokens were combined into the Output column of the corresponding model row. Accordingly, Mean/call includes the formatting-token overhead in its numerator, while its denominator excludes these additional formatting calls.

These subtasks typically use a fixed prompt for each dataset and receive the number or answer option produced by the preceding node; their final outputs are also mostly short numbers or fixed option phrases. These factors make the two Phi models’ average token usage per counted call nearly identical within the same dataset. By contrast, HumanEval has greater variation in code content, context, and output length, so it does not exhibit the same pattern. We will include relevant output examples in the revised appendix.

Revision. We have corrected the results in Table 1 and Appendix E.3 and clarified token accounting in Appendix G.3.

W1 & Q4: Joint-planning control and more demanding expert-pool changes

The paper's own control undercuts its thesis: on MATH-500, a jointly trained planner scores 93.0 against SReF's 93.4. In the pool-change test, retraining the Router actually lowers accuracy.

For future work, test harsher pool changes, such as removing the strongest model.

Our central claim is that, under the expert-pool changes studied, the trained Planner can be retained while maintaining competitive performance without updating its parameters. This claim does not require SReF to substantially outperform joint planning on the original pool, nor does it imply that Router-only adaptation must improve accuracy.

In our supplementary evaluations on the original pool, SReF and Planner-Integrated achieve 93.5 ± 0.1% and 93.1 ± 0.5%, respectively. This difference is small.

With expert insertion, SReF achieves 94.1 ± 0.1% with direct transfer, 93.8 ± 0.2% with Router-only adaptation, and 94.4 ± 0.4% with joint adaptation. Under the original removal setting, the corresponding results are 93.3 ± 0.2%, 93.1 ± 0.3%, and 93.3 ± 0.3%. Thus, direct transfer performs similarly to adaptation, and Router-only updates do not provide a consistent improvement.

Following the reviewer’s suggestion, we additionally removed Qwen3.5-27B, the expert with the highest mean calibrated capability in the original pool. The results below are the mean ± sample standard deviation of Pass@1 (%):

MethodPrePost-RPost-All
SReF92.8 ± 0.292.8 ± 0.292.9 ± 0.3
Fixed-ID Router90.1 ± 0.192.0 ± 0.493.0 ± 0.2
Planner-Integrated91.7 ± 0.3N/A92.0 ± 0.0

After removing Qwen3.5-27B, SReF’s mean direct-transfer accuracy decreases by approximately 0.7 percentage points relative to the original pool and remains close to its Router-only and joint-adaptation results. With direct transfer, it outperforms Fixed-ID Router by 2.7 percentage points and Planner-Integrated by 1.1 percentage points. After adaptation, Fixed-ID Router recovers to a level close to SReF. These results further support retaining the Planner and transferring directly as an effective option under the expert-pool changes tested here.

Revision. We have added the Qwen3.5-27B removal experiment to Appendix G.5.

W2 & Q1: Verification and repair, teacher data, and inference cost

Only SReF gets the inference-time judge, which uses about as many tokens as all the experts combined.

Did any baseline get the judge or the teacher data?

SReF uses an external verifier at inference time, which does introduce additional overhead. The verifier and the Planner Patch it triggers are part of the complete method: the verifier checks executed segments, and the Planner uses failure feedback to revise pending computation while preserving the executed history. We reproduced the baselines using the training and execution procedures of their original methods, without additionally integrating SReF’s segment-level verifier and Planner Patch. Based on our current findings, identifying an inexpensive yet accurate verifier is a key factor in improving multi-agent systems built on our small-model pool.

AuthorsOpenReview ↗

Response to Reviewer XYBE(3/3)

Teacher data and initialization. We use teacher trajectories for supervised initialization to help the small Planner learn to generate executable Graph-IR that satisfies the interface requirements, covering graph structures with different node counts, depths, and dependencies. We attempted direct training from initial weights without this fine-tuning stage, but approximately 35% of the outputs contained structural errors. These invalid trajectories are discarded by the training procedure, reducing the efficiency of valid-sample collection. The reported SReF-Base results also rely on additional processing to improve graph validity.

Teacher trajectories may also convey structural planning priors, so we do not attribute their role solely to format learning. Among the relevant baselines, Uno-Orchestra and R2-Reasoner also use SFT initialization; we retain their respective original training procedures.

Adding verification and repair to GraphPlanner. To further examine the effect of inference-time verification and repair, we invoke the verifier after GraphPlanner produces its final output, keeping the relevant settings consistent with SReF. If verification fails, the relevant context is passed to a designated model to regenerate the answer. Evaluation uses the same expert pool.

The table below reports Pass@1 (%). Results for the original methods are taken from Table 1, and the added control is reported separately.

MethodGSM-HardHumanEvalMMLU-ProMATH-500
GraphPlanner67.293.365.889.2
GraphPlanner + verification and repair71.994.867.193.0
SReF73.395.173.793.4

Inference token cost. We separately count the Planner’s input and output tokens and combine them with expert and verifier usage. The averages are macro-averages with equal weight assigned to the four evaluation datasets.

Planner token usage

DatasetMean input tokens/questionMean output tokens/question
MATH-500790.0143.0
MMLU-Pro900.1140.0
HumanEval861.3236.7
GSM-Hard778.7144.1

Mean token usage by method

We must emphasize that these statistics were collected from our local implementations using the same accounting convention as SReF. We include all additional prompt-related token usage, including the inputs and outputs of reward-model, planning, and judging calls. Some methods were also adapted to our small-model pool; without such adaptation, their direct execution performed substantially worse than running an individual model. SReF’s total includes Planner, expert, and verifier usage. Verification accounts for a large share of our token consumption, so finding an inexpensive and accurate verifier is a direction for our next stage of work.

MethodMean tokens/question
Phi-4604.8
Qwen3.5-27B486.3
DeepSeek V4.0 Pro379.0
RouterDC543.8
GraphRouter533.2
Router-R11423.1
xRouter1392.6
R2-Reasoner3205.6
MasRouter3964.7
GraphPlanner5767.4
GraphPlanner + verification and repair8916.1
Puppeteer3751.3
Uno-Orchestra2761.3
SReF4,777.8

Revision. We have added complete inference-token statistics and the GraphPlanner repair control to Appendices G.3 and G.7.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.