- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper proposes the offline preference learning framework CPO-ORCH. By constructing decision-localized counterfactual preference pairs and combining them with structurally matched natural preference pairs to suppress model-identity shortcuts, the method fine-tunes an 8B orchestrator offline using DPO. Across multiple benchmark tasks, CPO-ORCH achieves better accuracy compared to the baselines. The ablation studies further validate the complementarity between natural and counterfactual preferences.
Critical strengths
- By intervening on a single decision within an executed workflow and performing paired re-executions, the paper transforms an end-to-end outcome verifier into localized decision supervisory signals.
- Compared to the high costs of online reinforcement learning, which requires frequent multi-model rollouts at every iteration, this approach incurs inference overhead during the data construction phase and subsequently optimizes offline via DPO.
- The comparative experiments and ablations across multiple tasks substantiate the effectiveness of the proposed method.
Critical weaknesses
-
Recent studies [1, 2] have explored dynamic topologies and role assignments using graph search or reinforcement learning. Also, leveraging single-step perturbations or counterfactual interventions to build paired data for multi-step reasoning error correction has precedent [3, 4]. [1] Agent-pro: Learning to evolve via policy-level reflection and optimization; [2] Architectural Foundations for Multistage Task Orchestration in Engineering Educational Systems; [3] Towards Generalizable Reasoning: Group Causal Counterfactual Policy Optimization for LLM Reasoning; [4] IG-Search: Step-Level Information Gain Rewards for Search-Augmented Reasoning
-
For a 7-node worker pool containing multiple commercial API models, the repeated re-execution scheme in this work incurs substantial API billing costs and network latency. In environments involving real-time interactive tools, external databases, or state mutations, repeated execution cannot always ensure an identical initial state. How does the proposed method address these challenges?
-
The five defined families of counterfactual edit operators rely on hard-coded rule replacements or static template prefixes. When real-world tasks require complex long-horizon reasoning, isolating changes to a single decision while enforcing an identical workflow depth may disrupt the semantic coherence of the workflow.
-
Natural preference negatives conflate distinct reward levels without a margin-based DPO objective, e.g., 0.5 vs. 0.0, treating both identically as . The text clarifies neither whether this quality discrepancy introduces gradient dominance bias nor how the model avoids learning formatting shortcuts from parse-failure negatives.
-
For complex agent systems involving multi-step collaborative invocations, sample sizes such as 18 coding problems or several dozen multiple-choice questions are relatively small. The baselines omit standard dynamic reactive multi-agent systems, focusing instead on fixed models and static scaffolds. Besides, experiments are confined to Qwen3-8B; evaluating across diverse backbone models is essential to substantiate cross-architecture generalizability.
Questions for authors
Please see the above comments.
Dear Reviewer xALY,
Thank you for raising these questions about prior work, replay, and the treatment of negative examples.
W1. Relationship to prior work
Prior work has used counterfactual perturbations and rewards at individual steps for reasoning and search. Our contribution is to construct preferences over decisions inside an executable multiworker workflow. We edit one decision, such as which worker runs a subtask or which earlier outputs it can see. We then execute both plans and use the final task outcome to label the pair for offline DPO. We will discuss the suggested work and state more clearly where our contribution fits.
W2. API cost and replay
Initial mining cost USD 827 for 77,200 workflow executions and 173,303 worker calls, including rejected candidates. It used one execution per plan. The separate validation pass required agreement in four of five trials and cost about USD 320. In a matched comparison, the score with repeated validation was 0.4 points lower. The mixing and operator studies used one execution per plan. The full counterfactual campaign cost about USD 1,600 in API charges, including the additional LiveCodeBench runs. We do not have a comparable measure of total collection time.
Our tasks let both plans start from the same input. If a tool writes to a database or changes external state, each plan would need to start from the same snapshot and run in isolation. Our evaluation did not cover those environments.
W3. Workflow coherence
The validator checks that both plans parse, keep the same depth, and differ in one decision. These checks do not guarantee that the edited plan remains coherent. For example, changing an earlier worker can leave a later step expecting an output the replacement worker does not produce. Execution captures the effect on the final outcome, but repairing the plan may require changing more than one decision.
W4. Reward margins and formatting shortcuts
Natural pair mining gives priority to parse failures. A plan that produces a correct answer earns 1.0, a parseable plan with a wrong answer earns 0.5, and a parse failure earns 0.0. DPO uses the resulting preference order without weighting either reward gap explicitly. The two types of negatives can still contribute different gradients because their frequencies and model scores differ.
Matching on plan length, depth, and worker identity leaves room for formatting shortcuts. Both plans in a counterfactual pair must parse, so these pairs provide a signal about choices within valid plans. A direct test for the shortcut would train on natural pairs with only parseable negatives under the same settings, then report parse rate and accuracy among parsed plans separately.
W5. Evaluation scope and reactive baselines
The main evaluation covers 198 tasks, including 18 LiveCodeBench problems, with one Qwen3 model with 8B parameters and one worker pool. We also report a separate set of 144 later LiveCodeBench problems. Across the evaluated runs, at least one worker solved 59 of those problems. That study examines how worker capability limits orchestration. Our GRPO baseline uses the same upfront planning setup. Other backbones and reactive systems that can revise a plan after seeing intermediate outputs remain open comparisons.
If these clarifications address your main concerns about the method and bring our claims in line with the evidence, we would appreciate it if you would consider a higher rating.