acceptodds
Under review as a conference paper at ICLR 2027

ORCAS: A CLAWs-Style Multi-Agent Framework with Routed Correction and Inspectable Representation for Optimization Modeling

Abstract

Optimization modeling is a long-horizon task that relies on professional knowledge and consists of multiple independent steps, making it one of the most critical yet technically demanding tasks in operations research (OR). Large language models (LLMs) have demonstrated immense potential in automated modeling across various scenarios. Existing agentic modeling methods, relying solely on step-by-step reasoning or prompts, are unable to overcome the difficulties of ineffective collaboration among agents and lack of persistent guidance, and often fail to achieve high performance. To address these challenges, we propose ORCAS, a CLAWs-style modeling system, representing modeling competence as an externalized, inspectable state that is routable, persistent, and improvable without fine-tuning. ORCAS adopts a multilayer architecture with agent-isolated workspaces and a routed double-loop correction mechanism that classifies each detected fault and routes it to the stage at which it could have arisen, thereby achieving separation of concerns and step-level provenance while enabling modeling capability to improve with use. Experiments on eight optimization modeling benchmarks show that ORCAS achieves an average accuracy of 79.5% and surpasses state-of-the-art agentic methods, with particularly large gains on challenging tasks.

Primary area:
optimization
Keywords:
Optimization Modeling, Large Language Models, Artificial Intelligence

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_jvo2OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes ORCAS, a CLAWs-style multi-agent framework for automated optimization modeling. It employs a multilayer architecture with four role-specialized agents (Planner, Modeler, Coder, and Reviewer), each equipped with an isolated workspace and role-specific skills. A routed double-loop correction mechanism enables targeted error repair, while persistent memory supports experience reuse across tasks. The framework also preserves intermediate artifacts for step-level provenance. Experiments on eight optimization modeling benchmarks show that ORCAS outperforms the strongest agentic baseline, with ablation studies evaluating the contributions of the main components.

Critical strengths
  1. ORCAS achieves competitive performance across eight optimization modeling benchmarks, outperforming the strongest agentic baseline by 3.0 percentage points on average. The extensive ablation studies further support the effectiveness of the overall system.
  2. The framework preserves intermediate modeling artifacts and supports error localization and targeted correction. This structured and inspectable workflow has practical value for improving the reliability of automated optimization modeling.
Critical weaknesses

The paper's motivation and methodological novelty are not sufficiently established. Existing methods such as CoE and OptiMUS already employ role-specialized collaboration and iterative correction, while AlphaOPT incorporates a reusable experience library. The paper does not clearly explain why these existing mechanisms are inadequate or how ORCAS advances beyond their integration.

Questions for authors
  1. Given that CoE and OptiMUS already employ role-specialized collaboration and iterative correction, while AlphaOPT supports experience reuse, what specific limitations of these methods does ORCAS address? How do the proposed mechanisms differ from existing approaches beyond their integration?
  2. Removing role-specific skills reduces ORCAS's average accuracy from 76.1% to 64.5% across four benchmarks, below OptiTree's 74.9%. How were these skills constructed, and to what extent do they rely on manually curated domain knowledge? Were comparable domain knowledge resources retained in the evaluated agentic baselines?
  3. The routed double-loop correction mechanism directs errors either to the Coder or back to the Planner. How does the Reviewer reliably identify the root cause and the appropriate repair stage, particularly when errors originate in the Modeler?
6 comments

We thank the reviewer again for the constructive feedback. Below we provide detailed explanations and specific revisions, and we hope these clarifications resolve the concerns raised.

1. Novelty

[W1, Q1]

  1. The contribution is a representational principle, not an assembly of components. ORCAS treats modeling competence as an externalized, stage-separated, and addressable state: plans, models, programs, verdicts, and agent functions are written as typed files indexed by problem and cycle. The multilayer architecture, the routed double-loop correction mechanism, and agent-isolated workspaces are not independent additions; they follow from this principle.

  2. This principle yields a planning stage before formulation and makes plan-level routed correction possible. Unlike prior pipelines that pass every problem through a fixed element-extraction schema (e.g., OptiMUS extracts parameters, formulates constraints and objective, then generates code; LLMOPT uses a five-element formulation), ORCAS inserts a planning stage before formulation. For each problem, the Planner produces a problem-specific workflow zP=(τ1,…,τL)z_P=(\tau_1,\ldots,\tau_L), where each step specifies its input, expected output, success criterion, and problem-specific precautions. Because the plan is a separate artifact, it is the target of the outer loop: interpretation errors are repaired by revising the decomposition and answering the Reviewer's diagnostic questions against the source text, rather than by patching extracted elements. Unlike CoE's dynamic expert selection or OptiTree's hierarchical decomposition, ORCAS's plan is an explicit, typed, persisted contract that downstream agents execute step by step.

  3. Specific differences from existing methods. | Dimension | CoE | OptiMUS | AlphaOPT | ORCAS | |---|---|---|---|---| | What happens before formulation | Conductor selects the next expert | Fixed element extraction (parameters → constraints/objective → code) | Single pipeline guided by retrieved insights | Problem-specific plan; no mathematics committed | | How intermediate state is represented | Shared reasoning state along the expert chain | Shared state held by a manager agent | Single pipeline context | Typed files per stage, indexed by problem and cycle | | What each agent can see | Outputs of the preceding experts | Shared state | Not applicable (single agent) | Role-specific projection only; the Coder never sees the plan, and the Planner never sees code | | Source of the correction signal | Backward reflection along the chain | Execution and debugging feedback | Solver verification of candidate insights | Adversarial Reviewer auditing problem–plan, plan–model, and model–program alignment | | Where a correction is sent | Back along the expert chain | The component that failed to execute | No per-instance repair loop | The earliest defective stage, chosen by root-cause class (inner loop: Coder; outer loop: Planner) | | Verified upstream artifacts kept during repair | No | Partially | Not applicable | Yes; the inner loop reuses the plan and model unchanged | | Experience | None | None | Global library distilled from successful trajectories | Along with usage, generate memories related to the work of each agent |

  4. Evidence that the design choices matter. Removing the Planner lowers mean accuracy by 6.7 points, and by 12.0 and 11.9 points on IndustryOR and OptMATH. Removing workspace isolation while keeping routed correction, skills, and memory lowers accuracy by 7.2 points with independent contexts and by 11.4 points with a shared context. Without memory, ORCAS(0) averages 78.6 over eight benchmarks, compared with 70.1 for AlphaOPT. On the three hardest benchmarks it averages 58.7, compared with 47.5 for AlphaOPT, 40.7 for CoE, and 40.6 for OptiMUS. Under a matched inference budget, the strongest baseline reaches 71.8, compared with 76.1 for ORCAS.

2. Skills documents

[Q2]

These materials were curated and finalized by the authors through an iterative drafting and manual review process. They are based on general optimization-modeling knowledge and commonly observed modeling/coding pitfalls, and are intended to describe generic principles and failure modes rather than benchmark-specific solutions. We did not consult any instance from the eight evaluation benchmarks or from sources overlapping with them. We have audited the documents to ensure that no benchmark-specific content is included.

3. The routed double-loop correction mechanism

[Q3]

  1. Root-cause localization is a grounded LLM judgment. The routing decision is made by the Reviewer, which is an LLM agent. It does not apply a hand-coded rule. The judgment is grounded in three sources. The first is the full artifact chain of the current cycle: the plan, the model, the program, and the execution trace, each stored as a separate file. The second is the original problem statement, against which every artifact is checked. The third is the Reviewer's skills, including the semantic-audit skill, which defines the three alignment checks (problem–plan, plan–model, model–program) and the routing table. 
  2. Errors that originate in the Modeler. The Modeler has no separate repair entry point. When the Reviewer judges the plan correct but the mathematical derivation faulty, it routes the fault to the Planner and fills the field modeler_fix_instruction with the specific correction. Routing through the Planner has two benefits. The Modeler receives a targeted correction rather than regenerating the same derivation from an unchanged plan. The Planner can also revise the plan when the apparent Modeler error actually comes from an ambiguous step. 
  3. manual verification of routing accuracy. To measure how reliable this judgment is, we randomly sampled 100 problems (A total of 112 cycles) from the four ablation benchmarks. For every rejected cycle, we traced the artifact chain and manually determined the stage at which the fault originated. We then compared this label with the Reviewer's routing decision. The Reviewer's routing was correct in 94.6% of cases.

Finally, we sincerely thank you again for your constructive feedback and suggestions on our manuscript. We wish you all the best.

Dear Reviewers,

Thank you for your time and efforts in evaluating our work. As the discussion window is nearing its ending point, we would like to gently check if our responses and revisions have adequately addressed the concerns you raised.

We would be very grateful for any additional feedback or clarification you might have, as your expertise is vital to enhancing the quality of our submission. We look forward to hearing from you.

Best regards, The Authors

Reviewer_XAtdOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents ORCAS, a multi-agent framework for translating natural-language optimization problems into mathematical formulations and executable solver code. The system separates planning, modeling, coding, and verification into four roles, each with its own workspace, skills, and persistent memory. A routed correction mechanism directs implementation errors to the Coder and semantic or formulation errors back to the planning stage. Intermediate artifacts are preserved for inspection and error diagnosis. Experience is collected from separate problems and frozen before evaluation. Across eight benchmarks, the paper reports approximately 79.5% average accuracy, supported by ablations examining role separation, correction routing, skills, and memory.

Critical strengths

-The distinction between implementation errors and modeling errors is well motivated. Routing corrections to the relevant stage is a practical design, and the routing ablations support its usefulness.

-The experimental coverage is substantial, including multiple benchmarks and backbones, component ablations, and matched-compute comparisons. Freezing memory during evaluation is also a useful control against cross-instance leakage.

-Saving formulations, code, and review feedback makes the system easier to inspect. The human assessment of intermediate models complements evaluation based on final objective values.

Critical weaknesses

-My main concern is the scoring protocol. Appendix A.1 allows continuous and integer variable declarations to be switched after an objective mismatch, with the revised result counted as correct. Although intended to handle ambiguous benchmark specifications, this means Pass@1 does not directly measure the correctness of the formulation originally produced. The effect of this adjustment on the results is unclear.

-The false-infeasibility analysis needs a clearer distinction between solver outcomes and system-generated labels. The Coder memory instructs the system to return INFEASIBLE when data are considered incomplete, even without running a solver. This instruction may itself contribute to the reported failure pattern.

-The architectural contribution is partly entangled with the domain-specific skills. The matched-compute baselines control inference budget, but it is unclear whether they receive equivalent modeling guidance. A simpler agent with the same skills and memory would provide a stronger comparison.

Questions for authors

-What are the results before variable-type correction, and how many predictions become correct through this adjustment? Is the same procedure applied to every baseline?

-How many false-infeasibility cases originate from the data-completeness gate? What happens if these receive a separate insufficient-information status?

-Does ORCAS retain its advantage over a single-agent refinement baseline with the same skills, memory, and inference budget?

4 comments

We thank the reviewer again for the constructive feedback. Below we provide detailed explanations and specific revisions, and we hope these clarifications resolve the concerns raised.

1. Variable-type re-check. 

[W1, Q1]

Upon checking, we confirm that the variable-type consistency check was never performed for ORCAS or for any baseline, and the sentence about the variable-type re-check in Appendix A.1 is an editing error. Table 1 results are therefore unaffected, and there is no “before variable-type correction” condition to compare. We clarified the actual protocol in Appendix A.1, and audit other dataset/baseline descriptions.

2. The false-infeasibility supplementary Analysis

[W2, Q2]

  1. All INFEASIBLE labels in the main run are system-generated.

We re-examined every INFEASIBLE output on the four analyzed benchmarks (NL4Opt, MAMO-EasyLP, IndustryOR, OptMATH; 1,083 instances). In every case, the generated program terminated at the data-completeness gate (Coder SOUL.md, Rule 15) without invoking a solver. None of the INFEASIBLE labels in the main run is a solver outcome.

BenchmarkNumeric, correctNumeric, wrongINFEASIBLE, correctINFEASIBLE, wrongAcc.
NL4Opt220517396.7
MAMOEasy519260095.2
IndOR573101257.0
OptM.1035643055.4

Computed over all 163 incorrect instances, wrong abstentions account for 45 (27.6%). We will correct the rates in Table 15 and Table 14 accordingly.

  1. Ablation: removing the gate.

We disabled the gate and required the system to always construct a complete model, inferring any data it judged missing, and then invoke the solver. All other factors were held fixed. We find removing the gate globally is not a net gain.

BenchmarkNumeric, correctNumeric, wrongSolver INFEASIBLE, correctSolver INFEASIBLE, wrongAcc. (Δ)
NL4Opt220817096.7 (0.0)
MAMOEasy518270095.0 (−0.2)
IndOR57410257.0 (0.0)
OptM.101804854.4 (−1.0)

Removing the gate, raises the share of numeric outputs from 93.9% to 97.1%; leaves only solver-certified INFEASIBLE labels (31 outputs, 21 correct); slightly lowers mean accuracy, from 76.1 to 75.8. The errors are not eliminated; they change form. Errors induced by inferred parameters (E2) rise from 20.3% to 39.2% of all incorrect instances. With the gate, the system marks a data gap with an explicit, inspectable label that the Reviewer can overturn (Cases B and C). Without it, the same gap becomes a plausible-looking numeric answer that cannot be distinguished from a correct one. The instruction change also affects instances that never triggered the gate: MAMO-EasyLP, which issued no abstentions, changes by one instance. This is why removing the gate globally is not a net gain.

  1. How many abstentions were genuinely false?

To isolate the gate's effect, we re-ran only the 45 wrongly abstained instances with forced data inference and find that the correct results will indeed occur:

BenchmarkAbstained (wrong)Correct after forcingStill wrong
NL4Opt303
MAMOEasy000
IndOR12210
OptM.301119
Total4513 (28.9%)32
13 of the 45 abstentions (28.9%) concerned data that were in fact recoverable. These are genuine false abstentions, and they confirm that the gate itself contributes to the observed failures. The remaining 32 stay incorrect even with forced inference, so for these instances the abstention flagged a real modeling difficulty rather than hiding a solvable problem.

As a post-hoc estimate of headroom, an "abstain-then-retry" policy would raise IndustryOR from 57.0 to 59.0 and OptMATH from 55.4 to 61.1. We report this only as an indication of headroom, not as a result of ORCAS, because it combines two runs after the fact.

3. Comparison with a single-agent baseline with the same skills, memory, and inference budget

[W3, Q3]

We ran a factorial comparison on the four ablation benchmarks with DeepSeek-V4 and found that ORCAS retain its advantage. The single-agent configuration is iterative self-refinement under the same budget as ORCAS (about 63K input tokens, the same sandbox). In the guided cells it receives exactly the ORCAS(200) guidance: all 14 skills, including the Reviewer's audit and infeasibility-ruling skills, retrieved on demand through the same index, plus the union of the four frozen role memories.

ArchitectureGuidanceNL4OptMAMOEasyIndOROptM.Mean
Single agent (self-refine)none90.894.052.050.571.8
Single agent (self-refine)skills + memory91.894.55353.973.5
ORCASskills + memory96.795.257.055.476.1

Giving the simpler agent identical skills and memory improves it by 1.7 points, but it remains 2.6 points below ORCAS, indicating that skills are most effective when each role loads only the subset relevant to its stage, which a single agent cannot do.

Finally, we sincerely thank you again for your constructive feedback and suggestions on our manuscript. We wish you all the best.

Dear Reviewers,

Thank you for your time and efforts in evaluating our work. As the discussion window is nearing its ending point, we would like to gently check if our responses and revisions have adequately addressed the concerns you raised.

We would be very grateful for any additional feedback or clarification you might have, as your expertise is vital to enhancing the quality of our submission. We look forward to hearing from you.

Best regards, The Authors

Dear Reviewers,

Thank you for your time and efforts in evaluating our work. As the discussion window is nearing its ending point, we would like to gently check if our responses and revisions have adequately addressed the concerns you raised.

We would be very grateful for any additional feedback or clarification you might have, as your expertise is vital to enhancing the quality of our submission. We look forward to hearing from you.

Best regards, The Authors

Reviewer_csUXOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces ORCAS, a multi-agent system for optimization modeling built on a "CLAWs-style" framework. It uses four agents (Planner, Modeler, Coder, and Reviewer), each with its own workspace, skills, and memory. The Reviewer checks the plan, formulation, code, and solution, then sends errors back to the Coder or Planner for correction, with up to K=3K = 3 cycles. Memory is built from 200 OR-Instruct problems and frozen during evaluation.

Using DeepSeek-V4, ORCAS(200) achieves 79.5% average Pass@1 across eight benchmarks, compared with 76.5% for OptiTree, with larger gains on ComplexOR, IndustryOR, and OptMATH. The paper also includes ablations of agent roles, error routing, workspace isolation, and skills, along with compute-matched comparisons, statistical tests, human evaluation of 200 model artifacts, and failure analysis.

Critical strengths
  1. Careful evaluation design. Memory is built from a separate dataset and kept read-only during evaluation, with checksums ensuring no changes across runs. The paper also includes a matched-compute comparison (Table 6), a paired McNemar test for memory improvements (Table 7), and a reliability check of the Reviewer (Table 10). The reported 98.3% precision for APPROVED decisions provides useful evidence that the verification step works, even when using the same backbone model.

  2. Transparent and inspectable design. Each correction cycle saves its intermediate outputs, making it possible to trace errors and understand how repairs work. The case studies in Appendix C illustrate the difference between code-level and planning-level corrections, while the failure analysis identifies where modeling errors arise. The released prompts, workspace configurations, and skill descriptions (Appendices D–E) also help with reproducibility and make the design easier to assess.

Critical weaknesses

Critical Weaknesses

  1. Baseline comparisons may not be fair, and the evaluation protocol is unclear. Appendix A.1 describes several choices that appear to benefit ORCAS, but it is unclear whether the same rules apply to the baselines.

    • Variable-type re-check: Incorrect solutions are retried with continuous and integer variable types switched, effectively giving ORCAS a second chance.
    • Corrected references: The authors correct 40 reference answers in NL4Opt and MAMO, making comparisons with previously reported baseline results questionable.
    • Baseline setup: DeepSeek-V4 is used "whenever applicable," but the paper does not clarify which baselines were re-run or how backbone-dependent knowledge libraries in OptiTree and AlphaOPT were handled.

    These choices matter because ORCAS leads OptiTree by only about 3 points. The macro-average also gives equal weight to ComplexOR (18 problems) and MAMO EasyLP (545 problems). On ComplexOR, the difference of 66.7% vs. 50.0% amounts to just 12 vs. 9 correct answers. All results come from a single temperature-0 run, without variance estimates.

    Another concern is the amount of hand-written domain knowledge. ORCAS includes 14 skill documents, trap catalogs, and a "Common Failures to Avoid" list, but the paper does not explain how these were developed or whether benchmark examples influenced them. Some memory entries also resemble OptMATH problem wording. The matched-compute comparison in Table 6 does not address this, since the baselines lack the same skills and role prompts. It is therefore difficult to tell how much improvement comes from the architecture itself.

  2. Several contributions are overstated or insufficiently supported.

    • Memory and self-evolution: The paper emphasizes learning from experience, but memory improves macro-average accuracy by only about 0.9 points (79.4 vs. 78.6 from Table 1, reported as +1.0), or a net 18 additional correct answers out of 1,893. Memory is also frozen during evaluation, so improvement through continued use is not actually tested.

    • False-positive infeasibility: The reported 41.3% failure rate may largely reflect ORCAS's own design. The Coder's hard rule 15 and memory instruct it to output "Answer: INFEASIBLE" without running the solver when a model is marked PARTIAL. The error taxonomy is also inconsistent: Table 16 percentages are multiples of 1/46 and sum to 95.7%, while ORCAS(200) has roughly 163 errors across the four audited benchmarks, despite the claim that all instances were reviewed.

    • Workspace isolation: The underlying CLAWs platform already provides persistent workspaces and file-based state. Table 3 compares isolated workspaces with configurations that combine conflicting role instructions. The NL4Opt drop from 96.7% to 84.9% may therefore reflect instruction conflicts rather than the benefit of workspace isolation itself.

    • Novelty: The main remaining contribution is routing corrections between Planner and Coder alongside role-specific skills. Similar ideas already appear in Chain-of-Experts and OptiMUS (specialized agents), ReLoop and OptiMUS (verification and repair), and AlphaOPT, MIRROR, and Agora-Opt (experience reuse). MIRROR, Agora-Opt, and COOPA are discussed but not included in the comparisons.

    • Human evaluation: The 200 rated artifacts were sampled only from successful ORCAS outputs, with no baseline comparison. The rubric in Table 13 also rewards features already built into ORCAS's output template, such as provenance annotations, parameter tables, and assumptions. Without inter-annotator agreement, these ratings provide limited independent evidence of better modeling quality.

Questions for authors
  1. Were the variable-type re-check and the corrected reference targets applied to every baseline in Table 1? Please state, for each baseline, whether its numbers were re-run (with which backbone and protocol) or taken from the original paper. How does the ranking change if the re-check is disabled for all methods?

  2. How were the 14 skill documents, the trap catalogs, and the Modeler's "Common Failures to Avoid" list written, and by whom (authors or agents)? Were any instances from the eight evaluation benchmarks, or from sources overlapping with them (e.g., OptMATH's generator), consulted during their development?

7 comments

We thank the reviewer for the constructive feedback.

1. Baseline comparisons. Variable-type re-check was an editing error. The variable-type consistency check was never performed for ORCAS or any baseline; Table 1 results are therefore unaffected, and there is no “re-check disabled” condition to compare. We clarified the actual protocol in Appendix A.1, and audit other dataset/baseline descriptions.

The 40 corrections affect only reference objective values, not problem text. All methods were scored by the same script against the same corrected files.

All prompt-based baselines in Table 1 were re-run with DeepSeek-V4 under identical decoding. For OptiTree and AlphaOPT, we built their knowledge libraries following their papers using the same 200 OR-Instruct problems, matching construction budget.

2. Statistical reliability. Macro-averaging gives ComplexOR (18) the same weight as MAMO EasyLP (545). Instance-weighted accuracy is ORCAS 85.8% vs. OptiTree 85.2% (+0.6 pp). On Hard-3 (ComplexOR, IndustryOR, OptMATH; 311 instances), ORCAS reaches 56.6% vs. 52.7% (+3.9 pp). We report Hard-3 as a supplementary, not pre-registered, analysis because these benchmarks are less saturated.

Under our fixed checkpoint and API version, temperature 0 yields deterministic outputs in our runs. Memory McNemar: 29 vs. 11 discordant cases, χ²=7.225, p≈0.007. ORCAS vs. OptiTree McNemar: b−c=11, b+c≈39, χ²=2.564, p=0.109. We note that the per-instance pairwise gap against a single strong baseline may not reach conventional significance thresholds, as the two systems perform comparably on the saturated easy benchmarks. The more robust evidence is the consistent directional advantage across all three hard benchmarks (ComplexOR, IndustryOR, OptMATH) and across different models tested, which together provide convergent evidence that the architectural contribution of ORCAS is genuine.

3. Skills documents. These materials were curated and finalized by the authors through iterative drafting and manual review, based on general optimization-modeling knowledge and common pitfalls. No instance from the eight benchmarks or overlapping sources (e.g., OptMATH generator) was consulted or used as input, reference, or few-shot example.

4. Memory contribution. We froze memory to ensure reproducibility and to prevent test-time leakage (an evaluation instance influencing its own score via memory deposited mid-evaluation). The experiment demonstrates cross-corpus generalization: memory constructed on 200 OR-Instruct problems transfers to 1,893 held-out problems. The source ablation in the Table 8 shows that the gain depends on what is remembered: memory built from 200 MAMO problems reaches 97.0% on MAMO EasyLP, compared with 95.2% for OR-Instruct memory. This argues against attributing the gain to longer context alone. Consequently, they do not test online updating on the evaluation stream itself.

5. False-positive infeasibility. Full reannotation of 163 errors gives: E1 (27.6%), E2 (20.2%), E3 (12.9%), E4 (11.7%), E5 (10.4%), E6 (9.2%), E7 (4.9%), E8 5 (3.1%). The root cause is upstream: the Modeler marked a recoverable parameter as missing. The pipeline already includes several measures against over-declaring missing data. We will list better calibration between missing and recoverable data as the primary direction for future work.

6. Human evaluation. We agree that four of the seven dimensions (M3 constraint provenance, M5 parameter documentation, M6 assumption disclosure, M7 completeness) partly reward features that the ORCAS output template explicitly asks for, but they are also advantages compared to other systems. In the revision we will split the rubric into template-dependent dimensions, reported as an inspectability audit, and template-independent dimensions (M1, M2, M4).

7. Workspace isolation. In CLAWs, workspace and instruction isolation are coupled by design; separating them introduces new confounds (rewritten instructions, altered delivery). Table 3's ablation therefore demonstrates the following: Under CLAWs's file-based state paradigm, ORCAS's four-role decomposition achieves effective separation of concerns through isolated workspaces;the shared-workspace configuration breaks this separation and performance degrades substantially.

8. Novelty. 1) The contribution is a representational principle: modeling competence as an externalized, stage-separated, addressable state, with typed files indexed by problem and cycle; the architecture, routed double-loop correction, and isolated workspaces follow from it. 2) Unlike fixed extraction schemas (OptiMUS, LLMOPT), ORCAS inserts a planning stage: the Planner outputs a problem-specific workflow zP=(τ1,…,τL)z_P=(\tau_1,\ldots,\tau_L) specifying inputs, outputs, criteria, and precautions. Because the plan is a separate artifact, the outer loop repairs interpretation at the plan level. Unlike CoE or OptiTree, the plan is an explicit, typed, persisted contract.

We thank the reviewer again for the constructive feedback. While the response above was concise, below we provide more detailed explanations and specific revisions. We hope these clarifications resolve the concerns raised.

1. Variable-type re-check.

[W1.1, Q1]

Upon checking, we confirm that the variable-type consistency check was never performed for ORCAS or for any baseline, and the sentence about the variable-type re-check in Appendix A.1 is an editing error: when preparing the dataset description, we mistakenly copied a paragraph from an early internal document, and it does not describe the actual protocol used in this study. Therefore, the results in Table 1 are not based on this step, and the reported ranking is unaffected; there is no “re-check disabled” condition to compare. We have removed the sentence in the revision and clarified the actual evaluation protocol in Appendix A.1. We are also auditing the remaining dataset and baseline descriptions in the appendix to avoid similar misstatements. We apologize for the confusion.

2. Corrected reference answers and fairness of comparison.

[W1.2]

The corrections shift the absolute accuracy of all methods equally and do not alter relative rankings. The 40 corrected instances (2 in NL4Opt, 14 in MAMO EasyLP, 24 in MAMO ComplexLP) affect only the reference objective values, not the natural-language problem descriptions. Critically, all methods — both ORCAS and every baseline — were evaluated against the same corrected reference files using the same unified scoring script. We will make this explicit in a revised Appendix A.1 with the sentence: "All baselines reported in Table 1 are evaluated against the same corrected benchmark files used for ORCAS."

3. Baseline setup and knowledge-library handling for OptiTree and AlphaOPT.

[W1.3]

All prompt-based methods (CoE, OptiMUS, ReLoop, OptiTree, AlphaOPT) were re-run with DeepSeek-V4 as the backbone under identical decoding parameters (temperature 0, top-pp 1.0, single sample). For OptiTree and AlphaOPT, whose performance depends on an external knowledge library, we constructed those libraries following the procedures described in their respective papers, using the same OR-Instruct source data (200 problems) as ORCAS, thereby matching both the data source and the construction budget. This design ensures that any difference in accuracy reflects the architectural contribution of each system rather than differences in backbone capability or knowledge-construction advantage. We will add a supplementary table in the revision listing for each baseline.

4. Macro-average weighting and statistical reliability of the 3-point gap.

[W1]

Instance-weighted accuracy. Weighting each benchmark by its number of instances, ORCAS(200) achieves 85.8% and OptiTree achieves 85.2% , a gap of +0.6 pp. We will report both macro- and instance-weighted averages in the revision.

MethodMacro-avgInstance-weighted avg
OptiTree76.5%85.2%
ORCAS(200)79.5%85.8%

Hard-benchmark aggregate. The more practically meaningful signal comes from the three benchmarks that genuinely stress formulation capability: ComplexOR, IndustryOR, and OptMATH (combined 311 instances). On this subset, ORCAS(200) achieves 56.6% versus OptiTree's 52.7%, a gap of +3.9 pp over a sample large enough to support reliable comparison (Table R2). The easy benchmarks (NL4Opt, MAMO EasyLP) are largely saturated for both methods; the hard-benchmark gap reflects the core design advantage of ORCAS.

MethodComplexORIndustryOROptMATHHard-3 (311 inst.)
OptiTree50.0%54.0%52.4%52.7%
ORCAS(200)66.7%57.0%55.4%56.6%

5. The statistical significance of the gap.

[W1]

At temperature 0, decoding is deterministic given a fixed model checkpoint and API version, which is why single-run evaluation is standard practice in this literature (OptiTree, SIRL, StepORLM all report single-run results). And we also conducted the McNemar test to demonstrate its effectiveness

For statistical reliability of ORCAS's key claim — that accumulated memory improves modeling — we already report a McNemar test on paired per-instance outcomes across all 1,893 problems (Appendix A.3): χ2=7.225\chi^2 = 7.225, p≈0.007p \approx 0.007. This confirms the memory contribution is not a statistical artifact.

For the pairwise comparison between ORCAS(200) and OptiTree, we have conducted the analogous McNemar test using per-instance results. The net discordant advantage is b−c=11b - c = 11 (ORCAS correct, OptiTree incorrect: bb; OptiTree correct, ORCAS incorrect: cc). With $b + c \approx$39, χ2=\chi^2 = 2.564, p=p = 0.109. We note that the per-instance pairwise gap against a single strong baseline may not reach conventional significance thresholds, as the two systems perform comparably on the saturated easy benchmarks. The more robust evidence is the consistent directional advantage across all three hard benchmarks (ComplexOR, IndustryOR, OptMATH) and across all three backbone models tested (Qwen2.5-72B, DeepSeek-R1, DeepSeek-V4; Appendix A.6), which together provide convergent evidence that the architectural contribution of ORCAS is genuine rather than benchmark-specific.

We will add the McNemar contingency table for ORCAS(200) vs. OptiTree to the revision alongside the existing Table 7.

6. Skills documents

[W1, Q2]

These materials were curated and finalized by the authors through an iterative drafting and manual review process. They are based on general optimization-modeling knowledge and commonly observed modeling/coding pitfalls, and are intended to describe generic principles and failure modes rather than benchmark-specific solutions. We did not consult any instance from the eight evaluation benchmarks or from sources overlapping with them. We have audited the documents to ensure that no benchmark-specific content is included.

7. Comparison with a single-agent baseline with the same skills, memory, and inference budget

[W1]

We ran a factorial comparison on the four ablation benchmarks with DeepSeek-V4 and found that ORCAS retain its advantage. The single-agent configuration is iterative self-refinement under the same budget as ORCAS (about 63K input tokens, the same sandbox). In the guided cells it receives exactly the ORCAS(200) guidance: all 14 skills, including the Reviewer's audit and infeasibility-ruling skills, retrieved on demand through the same index, plus the union of the four frozen role memories.

ArchitectureGuidanceNL4OptMAMOEasyIndOROptM.Mean
Single agent (self-refine)none90.894.052.050.571.8
Single agent (self-refine)skills + memory91.894.55353.973.5
ORCASskills + memory96.795.257.055.476.1

Giving the simpler agent identical skills and memory improves it by 1.7 points, but it remains 2.6 points below ORCAS, indicating that skills are most effective when each role loads only the subset relevant to its stage, which a single agent cannot do.

Comments on the paper

1 comment

Dear Reviewers,

Thank you for your time and efforts in evaluating our work. As the discussion window is nearing its ending point, we would like to gently check if our responses and revisions have adequately addressed the concerns you raised.

We would be very grateful for any additional feedback or clarification you might have, as your expertise is vital to enhancing the quality of our submission. We look forward to hearing from you.

Best regards, The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.