- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
DecisionBrain orchestrates heterogeneous optimization methods through explicit strategy design, a validated algorithm library, and an operationally separate feasibility-review stage. It evaluates natural-language optimization workflows on FrontierOR65-Fea and the targeted Hard32-Fea stress test, with nested ablations of the proposed mechanisms.
Critical strengths
The system addresses an important failure of optimization agents: an executable model need not satisfy the original requirements. The shared backbone and explicit execution budgets improve interpretability of the open-source comparisons. Hard32 feasibility increases to 26/32 instances, and the review ablation demonstrates a useful distinction between producing feasible decisions and preventing infeasible returns. The appendices document benchmark selection, solver interfaces, and failure records in considerable detail.
Critical weaknesses
The central attribution to adaptive orchestration remains incomplete. Hard32 deliberately selects instances where a particular monolithic configuration fails while a family-specific method succeeds. Table 3 attributes 46.9 percentage points to library access; the design-stage comparison is restricted to Gurobi and improves feasibility from 8/32 to 11/32. Appendix G.1 further identifies standard family solvers behind the full system's accepted decisions. These results demonstrate the package's utility in the selected regime, but do not sufficiently distinguish adaptive composition from straightforward family-based dispatch within the same library.
Questions for authors
- Clarification from existing run records: What evidence distinguishes adaptive composition from dispatching each problem family to a fixed method in the same full library? Please relate this to the Gurobi-restricted design ablation and Appendix G.1. Evidence of composition-specific gains would strengthen the main contribution.
- Clarification: What caused the three internal/external feasibility disagreements on FrontierOR65-Fea: contract errors, checker errors, or another failure? Their causes would affect the assessment of the verification contribution.
3 comments
Orchestration Contribution 2: Adaptive Design and Findings
Part 1 presents the experimental setup and the three result tables.
PDPTW results
DecisionBrain solves all 20 PDPTW instances, whereas Family Dispatch solves 16. On those same 16 feasible instances, mean reference gap improves from 4.44% to 3.62%. Across all 20 tasks, mean runtime falls from 40.46 to 29.23 minutes. Thus orchestration improves delivery success, solution quality, and overall task cost in this family. This is not dominance on every instance or faster solving conditional on success. On the 16 shared successes, DecisionBrain averages 26.99 minutes versus 20.55 for Family Dispatch. Its runtime advantage across all tasks includes avoiding unsuccessful attempts. The extra time on shared successes also motivates DecisionBrain-Flash, since orchestration can be redundant when a direct method already succeeds.
Problems directly supported by specialist solvers
VRPTW reaches 6/6 feasibility with DecisionBrain and Family Dispatch. DecisionBrain improves mean reference gap from 1.19% to 0.85%, but increases mean runtime from 12.15 to 17.23 minutes. This is a balance between quality and runtime. On JSSP, DecisionBrain and Family Dispatch each reach 6/6, but DecisionBrain has a worse gap, 8.97% versus 7.97%, and longer runtime, 19.15 versus 18.36 minutes. Family Dispatch returns a better objective on all six JSSP instances.
The final DecisionBrain designs select the hybrid schema on 11 of these 12 VRPTW and JSSP instances. This documents a tendency to introduce a design with multiple components even when direct specialist methods succeed. It does not identify the causal contribution of each extra component.
DecisionBrain-Flash design and results
DecisionBrain-Flash gives the model up to 45 minutes to select one installed package and solve the complete problem with one algorithm. Implementation repairs are allowed within this same budget. If a candidate passes verification, the task finishes. If feasibility review requests a new algorithm design, or the 45 minutes expire without an accepted result, the system switches to the unrestricted DecisionBrain workflow. Both phases share the same total task budget of 120 minutes. The second phase receives the actual remaining time and may choose either one algorithm or a composition.
Of the 32 tasks, 28 finish without escalation. The other four trigger escalation because feasibility review requests a new algorithm design, not because the initial time budget expires. All four subsequently succeed.
DecisionBrain-Flash retains 32/32 feasibility and reduces mean runtime across all tasks from 25.09 to 21.49 minutes, a 14.34% reduction relative to DecisionBrain. On their common set of all 32 instances, mean reference gap increases from 3.56% to 5.53%. The quality loss is concentrated in PDPTW, which accounts for approximately 93% of the net increase in the overall mean gap. This is the same regime in which the DecisionBrain versus Family Dispatch comparison shows a quality benefit from allowing orchestration. DecisionBrain-Flash therefore gives up quality primarily in this regime to reduce overall runtime while preserving feasibility, rather than delivering a lossless acceleration.
The balance differs by family. On PDPTW, DecisionBrain-Flash achieves 20/20 feasibility in 24.67 minutes on average, but its gap is 6.86% on the subset solved by all three systems, worse than both alternatives. On VRPTW it averages 12.45 minutes, close to Family Dispatch, but its 2.51% gap is worse. On JSSP its gap of 8.05% improves on DecisionBrain and its objective is better on all six instances, but it is slightly worse than Family Dispatch on average. Its mean runtime is 19.95 minutes. These results are consistent with our expectation. For instances well matched to solvers in the library, DecisionBrain-Flash can improve solving performance, further illustrating the boundary of orchestration and the value of an adaptive policy.
Summary
Orchestration is useful when a single packaged method struggles, but it is not always beneficial when a specialist already solves the problem effectively. On PDPTW, DecisionBrain improves feasibility, quality on shared successes, and mean runtime across all tasks relative to Family Dispatch. The VRPTW and JSSP results show that extra orchestration can add cost or worsen the objective. DecisionBrain remains meaningful in practical settings because requirements vary in their constraints and the solver interfaces they need, so a single package may not directly cover the complete problem. DecisionBrain-Flash responds by trying one package first and allowing unrestricted strategies when needed. It retains 32/32 feasibility and reduces overall mean runtime by 14.34%, while giving up quality mainly on PDPTW, the same regime in which DecisionBrain shows a quality advantage over Family Dispatch. The higher success rate reflects API model improvements. These results and discussion are now in the main text.
Internal and External Feasibility Disagreements
Thank you for raising this concern. Following the end of our previous appointment, the original traces are no longer accessible. We reran all three cases and inspected their descriptions and checkers. We infer that inconsistencies between the descriptions and the dataset checkers are the most likely source of the disagreements.[ZL1.1] earl2005: The description requires immediate deactivation upon interception or defense-zone entry but leaves the discrete update order unspecified. The internal checker updates a[k], while the dataset checker updates a[k+1]. All 11 violations reported by the dataset checker match this one-step difference.[ZL2.1]
bollapragada2001: The evaluation statement specifies supplied direction cosines, but the dataset checker recomputes them from coordinates. With supplied coefficients, all 144 equilibrium equations satisfy the external tolerance of 10⁻⁵ (maximum residual 1.17 × 10⁻¹¹). With recomputed coefficients, 86 fail (maximum residual 2.78 × 10⁻³).
kress2019: The ambiguity concerns whether stationary positions are required at the operation's end time.[ZL3.1] For an operation lasting p time units and starting at t, the checker generated by the model requires[ZL4.1][ZL5.1] stationary positions through t+p−1; the dataset checker also checks t+p. We suspect this endpoint convention caused the original disagreement. The rerun satisfies both conventions and passes both checkers.
All four evaluated baselines also failed external acceptance on earl2005 and bollapragada2001, providing supporting context. These findings identify benchmark contract issues; the original external scores remain unchanged.
Orchestration Contribution 1: Experimental Comparison
Thank you for suggesting a comparison against fixed dispatch by problem family. This was particularly useful and directly motivated the new comparison and DecisionBrain-Flash policy. The earlier design ablation restricted to Gurobi examines strategy design within a fixed backend. The experiment below instead compares specialist dispatch by problem family with unrestricted orchestration. Appendix G.1 identifies the packages behind accepted decisions, but package identity alone does not describe whether their use involved construction, warm starts, or subproblem coordination. We apologize that the original run records are difficult to retrieve, but the new experiments directly address this question and provide evidence on the contribution of orchestration beyond fixed dispatch by problem family.
This comparison was not included in the original submission because restricting the model to prescribed packages while excluding composition of algorithms required additional execution controls. We have now implemented constrained design schemas and runtime import guards and completed the comparison on all 32 Hard32 instances.
We compare three systems. DecisionBrain (full system) can use one algorithm or compose several algorithms. Family Dispatch selects one algorithm within the package assigned to each problem family. DecisionBrain-Flash first tries one package and can later switch to the unrestricted DecisionBrain workflow. Family Dispatch assigns OR-Tools to PDPTW and JSSP, and PyVRP to VRPTW. It does not prescribe a particular solver within OR-Tools. DecisionBrain and DecisionBrain-Flash use the same available library. Family Dispatch uses the documentation for its assigned package. All three use the same backbone model and checking mechanism. Each task has a total budget of 120 minutes, and each solver execution is limited to 30 seconds. Each task has a CPU quota of three cores. Gurobi is unavailable in all three settings.
The results distinguish a regime where the prescribed single method is less effective from regimes directly handled by a specialist. On PDPTW, DecisionBrain improves feasibility, mean quality on the common feasible subset, and mean runtime across all tasks over Family Dispatch. When one solver is already effective, composition may add little or even reduce performance. VRPTW exhibits a balance between quality and runtime, while JSSP shows worse quality and longer runtime. Because real OR requirements vary in their constraints and interface needs, the ability to compose remains useful, but composition should not be the default for every instance. This motivates DecisionBrain-Flash.
Table 1. Feasibility on all tasks
Table 2. Mean reference gap on instances solved by all three systems (%)
Table 3. Mean total task runtime across all tasks (minutes)
Reference gap is (objective − reference objective) / |reference objective|, not a certified optimality gap. Table 2 uses identical instances for all three settings. Table 3 includes failed tasks and timeouts and measures runtime rather than monetary or token cost. Updated statistics use the PDPTW-12 Family Dispatch and JSSP-1 DecisionBrain-Flash reruns. These point estimates do not measure variation between independent runs.
Part 2 discusses the results and the design of DecisionBrain-Flash.