acceptodds
Under review as a conference paper at ICLR 2027

LEARNING TO ORCHESTRATE WITH COUNTERFACTUAL PREFERENCES

Abstract

Large language model (LLM) orchestration requires jointly deciding which workers to call, what subtasks to assign, and how to pass information between them. Terminal task rewards make this hard to learn, since one outcome must be credited across many coupled choices. We introduce CPO-Orch, an offline preference-learning approach that localizes this supervision. It constructs decision-localized counterfactuals by editing exactly one decision in an executed workflow (a worker, a subtask instruction, or an access edge), re-executing both variants, and retaining pairs whose outcomes differ consistently under repeated execution. It combines these with natural preference pairs, structurally matched to suppress worker-identity shortcuts, and trains an 8B orchestrator with direct preference optimization, without online rollouts. Across four reasoning domains, CPO-Orch reaches 85.56% macro accuracy, outperforming the strongest fixed worker in its pool (83.18%), the strongest single model evaluated outside the pool (82.97%), the best fixed multi-agent scaffold (84.02%), and an oracle that selects the best worker per domain, while ablations show that natural and counterfactual pairs are complementary.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Agent Orchestration, Multi-Agent Systems, Counterfactual Preference Learning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_xALYOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes the offline preference learning framework CPO-ORCH. By constructing decision-localized counterfactual preference pairs and combining them with structurally matched natural preference pairs to suppress model-identity shortcuts, the method fine-tunes an 8B orchestrator offline using DPO. Across multiple benchmark tasks, CPO-ORCH achieves better accuracy compared to the baselines. The ablation studies further validate the complementarity between natural and counterfactual preferences.

Critical strengths
  • By intervening on a single decision within an executed workflow and performing paired re-executions, the paper transforms an end-to-end outcome verifier into localized decision supervisory signals.
  • Compared to the high costs of online reinforcement learning, which requires frequent multi-model rollouts at every iteration, this approach incurs inference overhead during the data construction phase and subsequently optimizes offline via DPO.
  • The comparative experiments and ablations across multiple tasks substantiate the effectiveness of the proposed method.
Critical weaknesses
  • Recent studies [1, 2] have explored dynamic topologies and role assignments using graph search or reinforcement learning. Also, leveraging single-step perturbations or counterfactual interventions to build paired data for multi-step reasoning error correction has precedent [3, 4]. [1] Agent-pro: Learning to evolve via policy-level reflection and optimization; [2] Architectural Foundations for Multistage Task Orchestration in Engineering Educational Systems; [3] Towards Generalizable Reasoning: Group Causal Counterfactual Policy Optimization for LLM Reasoning; [4] IG-Search: Step-Level Information Gain Rewards for Search-Augmented Reasoning

  • For a 7-node worker pool containing multiple commercial API models, the repeated re-execution scheme in this work incurs substantial API billing costs and network latency. In environments involving real-time interactive tools, external databases, or state mutations, repeated execution cannot always ensure an identical initial state. How does the proposed method address these challenges?

  • The five defined families of counterfactual edit operators rely on hard-coded rule replacements or static template prefixes. When real-world tasks require complex long-horizon reasoning, isolating changes to a single decision while enforcing an identical workflow depth TT may disrupt the semantic coherence of the workflow.

  • Natural preference negatives conflate distinct reward levels without a margin-based DPO objective, e.g., 0.5 vs. 0.0, treating both identically as z−z^-. The text clarifies neither whether this quality discrepancy introduces gradient dominance bias nor how the model avoids learning formatting shortcuts from parse-failure negatives.

  • For complex agent systems involving multi-step collaborative invocations, sample sizes such as 18 coding problems or several dozen multiple-choice questions are relatively small. The baselines omit standard dynamic reactive multi-agent systems, focusing instead on fixed models and static scaffolds. Besides, experiments are confined to Qwen3-8B; evaluating across diverse backbone models is essential to substantiate cross-architecture generalizability.

Questions for authors

Please see the above comments.

1 comment

Dear Reviewer xALY,

Thank you for raising these questions about prior work, replay, and the treatment of negative examples.

W1. Relationship to prior work

Prior work has used counterfactual perturbations and rewards at individual steps for reasoning and search. Our contribution is to construct preferences over decisions inside an executable multiworker workflow. We edit one decision, such as which worker runs a subtask or which earlier outputs it can see. We then execute both plans and use the final task outcome to label the pair for offline DPO. We will discuss the suggested work and state more clearly where our contribution fits.

W2. API cost and replay

Initial mining cost USD 827 for 77,200 workflow executions and 173,303 worker calls, including rejected candidates. It used one execution per plan. The separate validation pass required agreement in four of five trials and cost about USD 320. In a matched comparison, the score with repeated validation was 0.4 points lower. The mixing and operator studies used one execution per plan. The full counterfactual campaign cost about USD 1,600 in API charges, including the additional LiveCodeBench runs. We do not have a comparable measure of total collection time.

Our tasks let both plans start from the same input. If a tool writes to a database or changes external state, each plan would need to start from the same snapshot and run in isolation. Our evaluation did not cover those environments.

W3. Workflow coherence

The validator checks that both plans parse, keep the same depth, and differ in one decision. These checks do not guarantee that the edited plan remains coherent. For example, changing an earlier worker can leave a later step expecting an output the replacement worker does not produce. Execution captures the effect on the final outcome, but repairing the plan may require changing more than one decision.

W4. Reward margins and formatting shortcuts

Natural pair mining gives priority to parse failures. A plan that produces a correct answer earns 1.0, a parseable plan with a wrong answer earns 0.5, and a parse failure earns 0.0. DPO uses the resulting preference order without weighting either reward gap explicitly. The two types of negatives can still contribute different gradients because their frequencies and model scores differ.

Matching on plan length, depth, and worker identity leaves room for formatting shortcuts. Both plans in a counterfactual pair must parse, so these pairs provide a signal about choices within valid plans. A direct test for the shortcut would train on natural pairs with only parseable negatives under the same settings, then report parse rate and accuracy among parsed plans separately.

W5. Evaluation scope and reactive baselines

The main evaluation covers 198 tasks, including 18 LiveCodeBench problems, with one Qwen3 model with 8B parameters and one worker pool. We also report a separate set of 144 later LiveCodeBench problems. Across the evaluated runs, at least one worker solved 59 of those problems. That study examines how worker capability limits orchestration. Our GRPO baseline uses the same upfront planning setup. Other backbones and reactive systems that can revise a plan after seeing intermediate outputs remain open comparisons.

If these clarifications address your main concerns about the method and bring our claims in line with the evidence, we would appreciate it if you would consider a higher rating.

Reviewer_poLWOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

CPO-ORCH trains a Qwen3-8B orchestrator with offline DPO. It combines structurally matched natural preferences with pairs formed by editing one workflow decision and re-executing both versions. Repeated execution filters unstable preferences. On 198 tasks, it reports 85.56% macro accuracy versus 84.02% for the best fixed scaffold, using about 2.21 worker calls per task.

Critical strengths
  1. The paper is generally well organized, and the figures help explain the overall pipeline.

  2. The appendix provides detailed implementation settings and prompt templates, which are helpful for reproducibility.

Critical weaknesses

(1) The paper largely follows the workflow formulation of Conductor and uses standard DPO for optimization. Its main contribution is constructing counterfactual preference pairs through single-decision edits and repeated execution. However, these edits rely on manually designed rules and fixed instruction templates rather than a learned intervention mechanism. Moreover, it remains unclear whether the improvements come from decision-localized supervision or simply from selecting more reliable preference pairs. Stronger ablations are needed to establish the novelty and effectiveness of the proposed approach.

(2) The main comparison does not fully isolate the benefit of counterfactual preferences. CPO-ORCH and Natural DPO use different LoRA ranks, training epochs, and numbers of distinct preference pairs. These differences make it difficult to attribute the improvement in Table 1 to the proposed preference construction. A comparison under identical training configurations and matched data budgets would be more convincing.

(3) The paper mainly compares against Conductor and simple baselines, but lacks comparisons with MAS-Orchestra [1] and ToolOrchestra[2].

[1] Ke Z, Ming Y, Xu A, et al. Mas-orchestra: Understanding and improving multi-agent reasoning through holistic orchestration and controlled benchmarks[J]. arXiv preprint arXiv:2601.14652, 2026. [2] Su H, Diao S, Lu X, et al. Toolorchestra: Elevating intelligence via efficient model and tool orchestration[J]. arXiv preprint arXiv:2511.21689, 2025.

(4) Several important references are missing. The paper does not cite the original sources of its Qwen3-8B backbone or key evaluation benchmarks, including MMLU, RLPR, and LiveCodeBench.

Questions for authors

See Weaknesses.

1 comment

Dear Reviewer,

Thank you for raising the distinction between decision localization and label reliability, and for pointing out the missing comparisons and citations.

W1. Contribution and label reliability

We use Conductor’s workflow format and standard DPO. Our contribution is the way we construct preference pairs. A rule changes one worker assignment, instruction, or access edge. We execute both plans and use the final task outcome to assign the preference. Either plan can be preferred.

The mixing and operator studies used one execution per plan. In a separate matched comparison, the score with repeated validation was 0.4 points lower. The operator study also found similar scores for routing edits alone and for instruction and access edits alone. Measuring the effect of localization itself would require a comparison that holds label reliability and task composition fixed.

W2. Matched training

Table 1 compares two separately tuned systems, so we cannot attribute the full gap to counterfactual data. In Sec. 4.3, every arm used the same training settings and 2,034 distinct pairs. Natural data alone scored 82.61%, the equal mixture scored 84.06%, and counterfactual data alone scored 81.91%. The equal mixture scored highest among seven ratios, but the sweep gives limited evidence that mixing improves accuracy. Counterfactual mining also found decisive pairs on 110 tasks where natural mining found none. We will move the training settings and pair counts into the main text and train both Table 1 pair sets under both configurations.

W3. Baseline comparisons

MAS Orchestra learns to compose callable subagents. ToolOrchestra learns to select models and tools over multiple turns. Conductor GRPO lets us compare training approaches under the same workflow format, worker pool, and execution environment. A direct comparison with the other two methods would require adapting their interfaces and matching worker access and execution budgets. We will discuss both methods in Related Work and clarify the scope of our comparison.

W4. Citations and presentation

We will cite Yang et al. for Qwen3, Hendrycks et al. for MMLU, Yu et al. for RLPR, and Jain et al. for LiveCodeBench. We will also specify which experiments used one execution per plan and which used repeated validation.

If the ablations and our clarification of the contribution address your main concerns, we would appreciate it if you would consider a higher rating. Thank you.

Reviewer_Nk5nOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper addresses credit assignment in natural-language multi-LLM orchestration, where a single terminal task reward fails to distinguish which coupled decisions actually drove task success or failure.The authors propose CPO-ORCH, an offline preference-learning data construction pipeline that combines structurally matched natural pairs to mitigate dominant-worker shortcuts with counterfactual pairs constructed through single-decision workflow edits.Counterfactual pairs are curated through repeated execution, and a Qwen3-8B orchestrator is optimized offline via standard DPO over serialized workflow plans. The paper reports improved macro accuracy over fixed workers, fixed scaffolds, and its GRPO controls, together with mixing and operator-family ablations.

Critical strengths
  1. Practical formulation of localized credit assignment. CPO-ORCH constructs targeted preference supervision through single-decision workflow edits and re-execution, without introducing a new RL objective or a separate process reward model. Five-run consistency filtering helps reduce execution noise in preference labels.
  2. Thoughtful controls against worker-selection shortcuts. Natural pairs are matched on final-worker identity and frontier-worker presence, with workflow depth differing by at most one step. These constraints provide a sensible safeguard against preferences driven primarily by dominant-worker selection.
  3. Useful empirical controls and transparent limitations. The evaluation includes fixed scaffolds, oracle routing diagnostics, a call-count-matched random control, and cold- and warm-start GRPO. The controlled ablations use multiple training seeds, while the appendices explicitly document differences in data composition and execution stability.
Critical weaknesses
  1. The main comparison does not isolate the contribution of counterfactual supervision. CPO-ORCH and Natural DPO differ in LoRA rank, learning rate, and training duration (Table 7), as well as data diversity and effective supervision (Table 5: 1,472 vs. 816 distinct pairs and 105 vs. 217 identical chosen/rejected rows). Although both systems were tuned separately, these differences prevent attributing the full 4.04-percentage-point gain specifically to counterfactual supervision. A comparison controlling these factors would strengthen the attribution.

  2. Cost claims exclude data construction, whose execution volume already exceeds GRPO’s rollout budget. The reported 4.5 GPU-hours and zero worker calls apply only to DPO optimization. Natural preferences are mined from prior GRPO rollouts, while counterfactual preferences require repeated execution of both workflow variants. Although Appendix I acknowledges these upfront costs and their reuse, it does not quantify total workflow executions, worker calls, or API expenditure, including discarded candidates. This limits assessment of the method’s end-to-end efficiency relative to online GRPO.

Questions for authors

Offline data-construction cost. Could you report the total workflow executions, worker calls, and token/API costs required to construct the preference pools, including source-rollout collection, rejected candidates, and repeated validation? Please distinguish one-time collection costs from amortized reuse and clarify the dependence on rollouts generated during GRPO training.

1 comment

Dear reviewer Nk5n,

We are glad to address your concerns as following.

W1. What Table 1 measures

We agree that Table 1 cannot tell us how much of the 4.04 point gap comes from counterfactual data. Both training sets had 2,046 rows, but our method used 1,472 distinct pairs and Natural DPO used 816. The sets also had 105 and 217 rows with identical chosen and rejected plans, respectively. These rows contribute zero gradient. LoRA rank, learning rate, and training duration also differed. We will show these settings next to Table 1 and run each pair set with the other system’s training settings.

Sec. 4.3 fixes the training settings and uses 2,034 distinct pairs per arm, with three training seeds and three evaluation draws per seed. Natural data alone scored 82.61%, the equal mix scored 84.06%, and counterfactual data alone scored 81.91%. The two sources gave similar scores at the same pair count, though task coverage and the number of pairs that contribute a gradient still differed. Counterfactual mining found decisive pairs on 110 tasks where natural mining found none, including 97 RLPR tasks.

W2 and Q1. What the preference data cost

The natural pool came from 76,800 previously collected Conductor GRPO rollouts. They used 167,676 worker calls, cost about USD 700 in API charges, and yielded 2,602 matched pairs. The GPU cost of collecting those rollouts is separate.

Initial counterfactual mining ran 38,600 candidate pairs once per side. It took 77,200 workflow executions and 173,303 worker calls, cost USD 827 including discarded candidates, and retained 2,925 pairs. Rerunning LiveCodeBench with the corrected verifier cost about USD 350 for 19,320 executions. Additional LiveCodeBench mining cost about USD 110. Repeated validation cost about USD 320 for roughly 30,000 executions. The counterfactual campaign cost about USD 1,600 in API charges. We have cost estimates for the later stages, but we lack a complete count of their worker calls and tokens.

Training one DPO model costs about USD 45 and takes 4.8 GPU hours, with zero worker calls. One GRPO run costs about USD 770 and takes 152 GPU hours, with roughly 11,000 worker calls. The first DPO model costs about USD 1,650 with the existing natural pool, or USD 2,350 if we also charge the natural pool’s API calls. The 30 ablation runs reused the pools without new worker calls. We will report the cost of the first model and the cost of each additional run in the paper.

If the matched experiments and the full cost breakdown address your main concerns, we would appreciate it if you would consider a higher rating. Thank you for your time!

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.