- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes ORCAS, a CLAWs-style multi-agent framework for automated optimization modeling. It employs a multilayer architecture with four role-specialized agents (Planner, Modeler, Coder, and Reviewer), each equipped with an isolated workspace and role-specific skills. A routed double-loop correction mechanism enables targeted error repair, while persistent memory supports experience reuse across tasks. The framework also preserves intermediate artifacts for step-level provenance. Experiments on eight optimization modeling benchmarks show that ORCAS outperforms the strongest agentic baseline, with ablation studies evaluating the contributions of the main components.
Critical strengths
- ORCAS achieves competitive performance across eight optimization modeling benchmarks, outperforming the strongest agentic baseline by 3.0 percentage points on average. The extensive ablation studies further support the effectiveness of the overall system.
- The framework preserves intermediate modeling artifacts and supports error localization and targeted correction. This structured and inspectable workflow has practical value for improving the reliability of automated optimization modeling.
Critical weaknesses
The paper's motivation and methodological novelty are not sufficiently established. Existing methods such as CoE and OptiMUS already employ role-specialized collaboration and iterative correction, while AlphaOPT incorporates a reusable experience library. The paper does not clearly explain why these existing mechanisms are inadequate or how ORCAS advances beyond their integration.
Questions for authors
- Given that CoE and OptiMUS already employ role-specialized collaboration and iterative correction, while AlphaOPT supports experience reuse, what specific limitations of these methods does ORCAS address? How do the proposed mechanisms differ from existing approaches beyond their integration?
- Removing role-specific skills reduces ORCAS's average accuracy from 76.1% to 64.5% across four benchmarks, below OptiTree's 74.9%. How were these skills constructed, and to what extent do they rely on manually curated domain knowledge? Were comparable domain knowledge resources retained in the evaluated agentic baselines?
- The routed double-loop correction mechanism directs errors either to the Coder or back to the Planner. How does the Reviewer reliably identify the root cause and the appropriate repair stage, particularly when errors originate in the Modeler?
6 comments
2. Skills documents
[Q2]
These materials were curated and finalized by the authors through an iterative drafting and manual review process. They are based on general optimization-modeling knowledge and commonly observed modeling/coding pitfalls, and are intended to describe generic principles and failure modes rather than benchmark-specific solutions. We did not consult any instance from the eight evaluation benchmarks or from sources overlapping with them. We have audited the documents to ensure that no benchmark-specific content is included.
3. The routed double-loop correction mechanism
[Q3]
- Root-cause localization is a grounded LLM judgment. The routing decision is made by the Reviewer, which is an LLM agent. It does not apply a hand-coded rule. The judgment is grounded in three sources. The first is the full artifact chain of the current cycle: the plan, the model, the program, and the execution trace, each stored as a separate file. The second is the original problem statement, against which every artifact is checked. The third is the Reviewer's skills, including the semantic-audit skill, which defines the three alignment checks (problem–plan, plan–model, model–program) and the routing table.
- Errors that originate in the Modeler. The Modeler has no separate repair entry point. When the Reviewer judges the plan correct but the mathematical derivation faulty, it routes the fault to the Planner and fills the field modeler_fix_instruction with the specific correction. Routing through the Planner has two benefits. The Modeler receives a targeted correction rather than regenerating the same derivation from an unchanged plan. The Planner can also revise the plan when the apparent Modeler error actually comes from an ambiguous step.
- manual verification of routing accuracy. To measure how reliable this judgment is, we randomly sampled 100 problems (A total of 112 cycles) from the four ablation benchmarks. For every rejected cycle, we traced the artifact chain and manually determined the stage at which the fault originated. We then compared this label with the Reviewer's routing decision. The Reviewer's routing was correct in 94.6% of cases.
Finally, we sincerely thank you again for your constructive feedback and suggestions on our manuscript. We wish you all the best.
Dear Reviewers,
Thank you for your time and efforts in evaluating our work. As the discussion window is nearing its ending point, we would like to gently check if our responses and revisions have adequately addressed the concerns you raised.
We would be very grateful for any additional feedback or clarification you might have, as your expertise is vital to enhancing the quality of our submission. We look forward to hearing from you.
Best regards, The Authors
We thank the reviewer again for the constructive feedback. Below we provide detailed explanations and specific revisions, and we hope these clarifications resolve the concerns raised.
1. Novelty
[W1, Q1]
The contribution is a representational principle, not an assembly of components. ORCAS treats modeling competence as an externalized, stage-separated, and addressable state: plans, models, programs, verdicts, and agent functions are written as typed files indexed by problem and cycle. The multilayer architecture, the routed double-loop correction mechanism, and agent-isolated workspaces are not independent additions; they follow from this principle.
This principle yields a planning stage before formulation and makes plan-level routed correction possible. Unlike prior pipelines that pass every problem through a fixed element-extraction schema (e.g., OptiMUS extracts parameters, formulates constraints and objective, then generates code; LLMOPT uses a five-element formulation), ORCAS inserts a planning stage before formulation. For each problem, the Planner produces a problem-specific workflow zP=(τ1,…,τL), where each step specifies its input, expected output, success criterion, and problem-specific precautions. Because the plan is a separate artifact, it is the target of the outer loop: interpretation errors are repaired by revising the decomposition and answering the Reviewer's diagnostic questions against the source text, rather than by patching extracted elements. Unlike CoE's dynamic expert selection or OptiTree's hierarchical decomposition, ORCAS's plan is an explicit, typed, persisted contract that downstream agents execute step by step.
Specific differences from existing methods. | Dimension | CoE | OptiMUS | AlphaOPT | ORCAS | |---|---|---|---|---| | What happens before formulation | Conductor selects the next expert | Fixed element extraction (parameters → constraints/objective → code) | Single pipeline guided by retrieved insights | Problem-specific plan; no mathematics committed | | How intermediate state is represented | Shared reasoning state along the expert chain | Shared state held by a manager agent | Single pipeline context | Typed files per stage, indexed by problem and cycle | | What each agent can see | Outputs of the preceding experts | Shared state | Not applicable (single agent) | Role-specific projection only; the Coder never sees the plan, and the Planner never sees code | | Source of the correction signal | Backward reflection along the chain | Execution and debugging feedback | Solver verification of candidate insights | Adversarial Reviewer auditing problem–plan, plan–model, and model–program alignment | | Where a correction is sent | Back along the expert chain | The component that failed to execute | No per-instance repair loop | The earliest defective stage, chosen by root-cause class (inner loop: Coder; outer loop: Planner) | | Verified upstream artifacts kept during repair | No | Partially | Not applicable | Yes; the inner loop reuses the plan and model unchanged | | Experience | None | None | Global library distilled from successful trajectories | Along with usage, generate memories related to the work of each agent |
Evidence that the design choices matter. Removing the Planner lowers mean accuracy by 6.7 points, and by 12.0 and 11.9 points on IndustryOR and OptMATH. Removing workspace isolation while keeping routed correction, skills, and memory lowers accuracy by 7.2 points with independent contexts and by 11.4 points with a shared context. Without memory, ORCAS(0) averages 78.6 over eight benchmarks, compared with 70.1 for AlphaOPT. On the three hardest benchmarks it averages 58.7, compared with 47.5 for AlphaOPT, 40.7 for CoE, and 40.6 for OptiMUS. Under a matched inference budget, the strongest baseline reaches 71.8, compared with 76.1 for ORCAS.