- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 3(+1)
Read review
Paper summary
The paper addresses the problem of missing external rules in LLM-based optimisation modelling. Real-world optimisation tasks may depend on regulations, policies, thresholds and exceptions that are not clearly stated in the problem description. The authors introduce SearchWorthyOR (SWOR), a benchmark comprising 100 source tasks across 10 optimisation scenario categories, each evaluated under three conditions: the rule is inapplicable, applicable but omitted, or applicable and provided. They also propose SearchWorthy Agent (SWAgent), which maintains an information evidence table linking unresolved conditions, supporting sources, applicability assessments, and mathematical model modifications. The agent iteratively searches external sources, verifies rule applicability, revises the optimisation formulation when justified and resolves it using Gurobi.
The evaluation covers 300 instances and compares SWAgent against several LLM-based optimisation and search workflows. SWAgent achieves 79% overall answer accuracy, outperforming the strongest performing workflow, codex-native-search (67.67%), by 11.33 percentage points. Ablation studies indicate that the information evidence table, initial model formulation, and closure criterion contribute to performance. However, accuracy on the most challenging condition, where an applicable rule is omitted, is only 45%, and trajectory audits reveal difficulties in translating retrieved evidence into correct mathematical modifications. Overall, the paper contributes a benchmark and an evidence-based agent architecture for proactive external rules discovery, with empirical results demonstrating improvements while highlighting persistent challenges in applicability assessment and model revision.
Critical strengths
-
The paper formulation and benchmark design. The paper identifies an important limitation of LL-based optimisation: models may produce mathematically plausible solutions while overlooking the external rules that fundamentally change the optimal decisions. The SWOR benchmark addresses this gap through three paired conditions that distinguish rule applicability from rule visibility. Requiring the optimal action sets to be disjoint before and after applying an applicable rule ensures that missing information has measurable consequences, making the benchmark a useful contribution.
-
Evidence-grounded optimisation with empirical validation. SWAgent introduces an information evidence table that explicitly connects unresolved conditions, supporting evidence, applicability assessments, and mathematical model modifications. Its iterative search, verification, and model-revision workflow provides a structured approach to integrating external knowledge into optimisation. The reported 79% overall answer outperforms the strongest compared workflow by 11.33 points, and the ablation studies provide empirical support for the proposed design. These results approach a useful step towards more reliable LLM-based optimisation agents.
Critical weaknesses
-
Limited benchmark scale and external validity. The benchmark contains 100 source tasks across 10 optimisation categories, with three conditions per task. Although the paired conditions provide a useful controlled evaluation, the relatively small set of constructed scenarios leaves uncertainty about whether the reported gains generalise to real-world optimisation problems, where regulatory requirements may be ambiguous, interacting, or difficult to discover. Consequently, the reported 79% accuracy and 11.33 percentage points improvement over the strongest comparison do not yet establish robust performance in realistic deployments.
-
The central challenge, which is discovering and incorporating omitted rules, remains inadequately solved. SWAgent achieves only 45% accuracy in C2 where a relevant rule applies but is omitted from the initial problem statement. The semantic audit also reports complete evidence-chain pass rates of 48.9% for C2 trajectories, indicating persistent difficulties connecting retrieved evidence to the optimisation model correctly. This is consequential because proactive discovery and evidence-based model revision are the paper’s central contribution; the results suggest that the proposed approach still fails on a substantial fraction of precisely the cases it is designed to address. The paper would be more convincing with stronger evidence that the method reliably discovers relevant rules and translates them into correct model changes, beyond improving aggregate accuracy.
Questions for authors
-
Clarification: Benchmark generalisation. How were the 100 source tasks selected, and do the authors have evidence that the results generalise beyond the constructed scenarios? In particular, have the authors evaluated SWAgent on independently sourced, realistic optimisation problems not used during benchmark construction?
-
Clarification: C2 failures and evidence grounding. Given that SWAgent achieves only 45% accuracy on C2 and the semantic audit reports a 48.9% complete evidence-chain pass rate, what are the dominant failure modes? How often does the agent fail to retrieve a relevant rule vs retrieve the correct rule but incorrectly translate it into the optimisation model? This distinction is important for assessing whether the proposed approach addresses the core problem or primarily improves performance in easier conditions.
-
Clarification - comparison fairness. Were the SWAgent and the baseline workflows evaluated using equivalent search access, computational budgets, and solver configurations? To what extent does the reported 11.33 percentage point improvement reflect the information evidence table and associated reasoning procedure, rather than differences in search strategy or other system components?
15 comments
Response to Reviewer dqz1(2/8)
W1: Benchmark coverage and realism
Thank you for your comments on benchmark coverage and realism.
SearchWorthyOR addresses a specific practical gap: business requirements may state the objective, resources, and candidate decisions without listing all external rules that determine feasibility. We use verifiable real-world rules to construct controlled optimization instances that test rule discovery, applicability judgments, and mathematical model updates.
(a) Application categories
The current benchmark contains 100 source tasks. Each has C1/C2/C3, giving 300 condition-specific instances. Each category contains 9–12 source tasks. The categories are listed below; we also included them in the appendix of the original submission.
Table 1. Application coverage.
| Application category | Source tasks | Specific scenarios covered |
|---|---|---|
| Assignment and matching | 10 | Lighting recalls, room matching, hazardous materials |
| Energy and environment | 9 | Interregional grid connections, oil storage design, secondary containment design |
| Facilities and networks | 11 | Parking facilities, oil pipelines, control-function configuration |
| Finance and portfolios | 11 | ISA investment accounts, portfolios of positions |
| Healthcare resources | 10 | Treatment slots, eligibility matching, imaging equipment restoration |
| Inventory and supply chains | 9 | Hazardous materials, road freight consolidation, temperature control |
| Production and capacity | 12 | Product ingredients, composition thresholds, vehicle production, certificate resources |
| Routing and transportation | 9 | Cross-sea transportation, ferries, shuttle route planning |
| Scheduling and workforce | 10 | Cabin crew scheduling, vaccination services |
| Telecommunications services | 9 | Project delivery, SMS providers, message routing |
These ten categories correspond to typical areas such as energy, finance, healthcare, production, logistics, and telecommunications in the ORSSA overview of application areas. They also cover classic problem types such as transportation and assignment, network optimization, and inventory, discussed in Chapters 9, 10, and 18 of Hillier and Lieberman's Introduction to Operations Research, 11th edition. These links between application areas and problem types show that our approach covers a broad range of applications and constructed scenarios.
(b) Rule complexity and interactions
Thank you and Reviewer RF3z for raising the issue of rule interactions. We checked all 100 C2 tasks and obtained the following results.
Table 2. Requirement structure in C2.
| Rule structure | Tasks |
|---|---|
| Single requirement | 64 |
| Multiple requirements, no clear interaction observed | 29 |
| Multiple requirements with clear interactions | 7 |
We count requirements as obligations or restrictions that can be implemented separately. An interaction is a dependency, exception, or priority relation between requirements. Multiple requirements occur in 36% of tasks, and clear interactions occur in 7%.
For example,
under the ADA standards, the 240-space parking facility in R005 first requires at least 7 accessible spaces. This count then determines that at least 2 must be van-accessible spaces. The two requirements exclude options A and B in sequence, changing the optimal choice from A (utility 30) to C (utility 20).
This shows how rule dependencies affect the mathematical model and the decision. Each task has a fixed assessment date. Its rule materials record version dates and items awaiting verification, so that thresholds, exceptions, and scope can be distinguished across versions.
Overall, the benchmark covers different business domains and tests multiple requirements, rule interactions, and temporal applicability.
Response to Reviewer dqz1(3/8)
W1 / Q1: Task sources and independent evaluation
Thank you for this specific question. SearchWorthyOR serves as an evaluation set in this study and is not used to train SWAgent. We evaluate how a shared workflow for condition discovery, evidence checking, and mathematical model updates performs across optimization scenarios.
(a) Task sources and selection
The initial backgrounds and mathematical structures draw on IndustryOR, the OptiMiner benchmark, LogiOR, and scenarios from our team. We use assignment, scheduling, selection, and capacity models as starting points. We then use real-world materials to reconstruct the business setting, assessment date, candidate options, and numerical values, and add external requirements that change the decision. Each source task stores the issuing organization, URL, supporting source excerpts, rule description, Base/Full reference mathematical models, typed changes, and complete sets of optimal actions. Content and mathematical checks examine the interpretation of evidence, applicability to the instance, correspondence with constraints, and solver results.
(b) Information differences across C1/C2/C3
-
C1: The target rule is not applicable. We change applicability facts such as object attributes or time to test applicability judgments and preservation of the correct mathematical model.
-
C2: The target rule applies but is omitted from the problem statement. This tests whether the agent can discover the information gap, obtain the rule, and update the mathematical model.
-
C3: The applicable rule is provided. It shares the business facts and reference answer with C2, and adds a rule summary to the input. It tests modeling when the rule is known.
The three conditions deliberately control information visibility. The business rules and the grounds for the decisions come from real-world materials. The C2 Base model contains the conditions in the problem statement, while the Full model adds the applicable requirements. Construction requires their optimal action sets to be disjoint, so that omitting the rule changes the optimal decision.
C2 and C3 keep the business facts the same. Providing or withholding the rule allows us to examine the effect of filling the information gap.
(c) Generality and independent validation
SWAgent reuses the same workflow across application categories and backbones. Its inputs contain the public problem statement and output format, which the agent uses to determine what to check. These results provide evidence across domains and backbones on the current evaluation tasks.
Evaluation on independently sourced real-world optimization tasks is still pending. Future work will further test external validity using larger-scale real-world tasks, longer rule chains, and dynamically updated tasks.
Overall, this study uses an evaluation set to test how a shared workflow handles missing external rules. The current evidence comes from experiments across domains and backbones on these tasks.
Response to Reviewer dqz1(1/8)
Dear Reviewer dqz1,
Thank you for your thoughtful review. We appreciate your recognition of the problem formulation, benchmark design, and experimental evaluation. Below, we provide additional explanations and experiments addressing your comments. We have also updated the paper PDF. If you have any further questions, we would be happy to discuss and address them.