- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3(+1)
Read review
Paper summary
This paper presents AutoHOpt, a framework that connects natural-language optimization modeling with task-specific heuristic search. It first constructs an executable model, OPTSPEC, and checks its interpretation of the source requirements using behavioral probes and retained evidence from previous failures. The accepted model then guides the generation of initialization, destroy, and repair operators in SEARCHIR. These operators run within adaptive large neighborhood search while the model remains fixed. Experiments evaluate model recovery and search performance on ten benchmarks and the new 1,440-task CoPBench, with additional component ablations and fixed-model comparisons.
Critical strengths
- A useful connection between model recovery and search design AutoHOpt gives the recovered model a concrete role in search design. OPTSPEC supplies the objective and feasibility evaluator together with the decision structure used to construct search operators. Constraints that couple decisions can therefore guide which variables are released together and how partial solutions are reconstructed. This makes the recovered mathematical structure directly useful in designing the search procedure.
- Gains in feasible coverage and solution quality The main benchmark comparisons are based on 30 independent runs of the complete pipeline. On CoPBench, AutoHOpt achieves 87.4% full-suite feasibility which is higher than 79.9% for NEMO. On each run's common feasible intersection, its mean gap to the best feasible objective observed across the compared methods is 2.4% , a substantial improvement over NEMO's 4.8%. These results support gains in feasible coverage across the full suite and in objective quality on the common feasible subsets.
Critical weaknesses
- Details of the agent interactions The paper explains the modeling and search loops, and Appendices A and K provide a useful execution example. A fuller explanation of the interactions between agents would help readers understand and evaluate the procedure. In particular, please specify the complete prompts and response schemas, and explain how diagnostic feedback, retained probes, and previous responses are selected for subsequent requests. These choices affect what the agent knows about a failed attempt and how it is guided toward a correction. A complete interaction trace would help make the process clearer and easier to follow.
- Construction and validation of CoPBench CoPBench covers several problem families and instance sizes. For the full 1,440-task suite, the construction and validation of the natural-language descriptions merit further explanation. Please describe how the instances and descriptions were produced, and how their agreement on the intended objectives and constraints was checked. This is important because the reported feasibility and objective comparisons depend on evaluating the tasks specified by those descriptions.
Questions for authors
- Appendices A and K describe an execution of the current system. Could the authors provide the complete prompts and response schemas for the evaluated modeling, source-review, and search-design stages, together with a full request-response trace? The trace should show how diagnostic feedback, retained probes, and earlier responses are incorporated into subsequent requests. This would clarify the information supplied to each agent and the revision procedure used in the experiments.
- How were the instances and natural-language descriptions in the full 1,440-task CoPBench suite generated or collected? Could the authors describe how their correspondence was verified, particularly the checks used to establish agreement on the intended objectives and constraints?
- Could the authors provide results by problem family and instance size for the main 30-run CoPBench evaluation, including feasibility rates, runtime, and the conditional objective gaps reported in Table 9? This breakdown would show whether the aggregate improvements are consistent across different families and instance sizes.
- Could the authors examine a representative sample of OPTSPEC models before and after source review, reporting which semantic errors were corrected and which remained? A returned solution may satisfy the source requirements and match the reference objective even when the accepted model omits or adds a constraint. So it would be helpful to examine whether the recovered models are correct.
- Appendix E gives a default initial temperature of 1. Since changing objective units alters the acceptance probability of uphill moves when temperature is fixed, could the authors clarify whether the temperature is calibrated to the objective scale in the reported experiments?
4 comments
Response to Reviewer BFSV (2/2)
Q3. Results by problem family and instance size
Main-text Table 4 reports the 30-run evaluation, with the metric defined in Section 5.5. It establishes broader full-suite feasibility, 87.4% versus 79.9%, and a smaller conditional mean gap, 2.4% versus 4.8%, for AutoHOpt relative to NEMO. The common feasible intersection across all twelve methods includes all four families. Its existing 30-run composition is:
| Family | Tasks per run | Mean common-feasible tasks | Mean common coverage |
|---|---|---|---|
| Facility location | 360 | 127.6 | 35.5% |
| Manufacturing scheduling | 360 | 77.9 | 21.6% |
| Chip placement | 360 | 127.6 | 35.5% |
| UAV planning | 360 | 90.3 | 25.1% |
All four families therefore contribute to the matched objective comparison, and all four scales contribute to the full-suite feasibility denominator. The reported method comparisons are at suite level; the family table quantifies the composition of the common feasible set. We interpret the 7.5-percentage-point feasibility gain and reduction in conditional gap from 4.8% to 2.4% at this suite-level scope.
Q4. Inspecting model semantics before and after review
Model semantics and returned-solution validity are separate evaluation targets. We inspect the declared domains, objective terms, and constraint relationships against the source, using distinguishing assignments to test their behavior. The added diagnostics identify the type and provenance of each change.
- IndustryOR94: the first model passes four recorded source-derived probes covering marginal purchasing costs and grade-specific composition requirements. The appendix displays the complete model, expressions, and tests.
- IndustryOR47: the retained feasibility probe drives the numerical rewrite described above. The variable domains and mathematical relationship are preserved; this is a numerical representation repair.
- OptMATH153: a separate archived diagnostic run contains two objective coefficients with positive signs where the source specifies negative signs. Its original objective probe evaluates the zero decision, where either sign yields zero, so the discrepancy remains after that review. A subsequent direct comparison with the signed source expressions identifies the error; a separately recorded retry restores both signs. Appendix R displays the original coefficients and the first and revised models' objective values at distinguishing nonzero assignments. These observations test objective terms and are not asserted to satisfy the full constraint system.
The cases make successful tests, an observed repair, and a remaining semantic error independently inspectable. We report them as traceable model-level diagnostics. Population-level semantic-correction rates would require a separately sampled before/after model audit; solution-level success rates are not used as estimates of model equivalence.
Q5. Temperature and objective units
The reported 30-run experiments followed Appendix E, using and geometric cooling . Temperature was expressed in evaluator-cost units, with no automatic calibration to objective scale. Acceptance used the unnormalized difference in minimization-oriented penalized cost.
For an uphill move with unchanged feasibility and hard-residual rank components, the acceptance probability at iteration is
-\frac{\widetilde f_M(x')-\widetilde f_M(x)} {\max(T_0\gamma^{k-1},10^{-12})} \right].$$ Under a positive cost rescaling $\widetilde f'_M=a\widetilde f_M+b$, preserving these probabilities requires scaling both $T_0$ and the numerical temperature floor by $a$, with $\gamma$ unchanged. Appendix E now states the experimental settings, cost units, absence of automatic scale calibration, and rescaling relationship explicitly.Thanks for the response. The added prompts and interaction trace make it easier to see how the system retries after a failed attempt. The revision also explains how the 1,440 CoPBench tasks were generated, how the numerical instances were matched to their descriptions, and how two implementations checked feasible solutions. The comparison is clearer now: feasibility is measured across the full suite, while objective gaps are computed on the tasks where all methods found a feasible solution in a given run, using the best observed feasible objective as the reference.
The OptMATH153 example clears up my question about the correction. The two nonzero checks were done after the fact, and the corrected model came from a separate retry. Appendix E specifies the initial temperature , the cooling factor , and that temperature is measured in the evaluator’s cost units. It also clarifies that temperature was not automatically calibrated to the objective scale. These revisions address the main concerns from my original review.
Thank You for Your Constructive Feedback and Updated Assessment
We sincerely thank you for your careful follow-up and for reconsidering your assessment and raising your score. We appreciate your recognition that the revisions address the main concerns from your original review.
Your detailed and constructive suggestions have helped us improve the clarity, precision, and overall quality of the paper, particularly in presenting the interaction process, benchmark construction, and evaluation protocol. We are grateful for the time and care you have devoted to reviewing our work and helping us strengthen the manuscript.
Response to Reviewer BFSV (1/2)
Dear Reviewer BFSV,
We thank the reviewer for taking the time to carefully review our work and for the constructive feedback that has helped us improve the quality of our manuscript.
We appreciate your recognition of the connection between model recovery and search design, and the improvements in feasible coverage and solution quality. AutoHOpt makes recovered mathematical structure directly usable for both candidate evaluation and task-specific search. The revision makes the information exchanged between stages, the construction of CoPBench, and the scope of the validation explicit.
W1 / Q1. Complete prompts, response contracts, and interaction context
Appendix O presents the complete recorded IndustryOR94 execution: the original task, modeling prompt and response, compiled evaluator, source-review prompt and probes, and search-design prompt and generated initialization, destroy, and repair functions. It specifies the compact declaration's field grammar, its compilation into OPTSPEC, and the six-field SEARCHIR response contract.
Sections 4.2 and 4.4 now state the context supplied to model and search revisions. Appendix O.7 gives the exact field-level rules of the frozen implementation:
IndustryOR47 supplies a concrete feedback-driven revision. A source-valid assignment with 1.1 hectares and 55 whole labor-days failed the expression
work == labor * areabecause floating-point multiplication produced55.00000000000001. The next modeling request contained the failed diagnostic and all four retained probes. The generated edit rewrote the same positive-rate relationship aswork / labor == area; all four retained tests then passed, followed by two new passing tests. The appendix displays the actual follow-up fields, complete edit, and replay outcomes. This example shows how diagnostic evidence changes the next request and remains available after repair.W2 / Q2. CoPBench construction and objective–constraint correspondence
CoPBench is a constructed benchmark of 1,440 industrial optimization tasks across four domains, ten subtypes, four scales, and three configuration profiles. Each domain and each scale contains 360 tasks; each domain–scale–profile cell contains 30.
For each task, a frozen configuration and seed determine the numerical instance, active operational rules, and a private feasible construction witness. The public input contains an LLM-assisted scenario description, explicit calculation rules, and a complete numerical annex. The calculation rules and annex are rendered from the same numerical instance used by the reference evaluator. This connects the task statement to the evaluated coefficients, entities, and policy scopes.
The construction paragraph in Section 5.1 and Appendix Q.4 now explain the correspondence with concrete examples. In facility location, transport cost includes customer demand, customer weight, and distance, whereas emissions use demand, distance, and the facility's emission factor. In scheduling, a due date contributes to tardiness, while an active hard deadline constrains completion. In UAV tasks, hard windows apply to service start, and an explicit flag specifies whether the final depot-return leg is required.
The release records schema checks and independently verified feasible witnesses for all 1,440 instances. A primary evaluator and separately implemented verifier check feasibility and recompute the objective and metrics, with absolute and relative agreement tolerances of 10−8. The revision reports the scope of these numerical checks and exposes the objective terms, coefficients, and constraint scopes for direct inspection of the description-to-instance correspondence.