acceptodds
Under review as a conference paper at ICLR 2027

Search Before You Optimize: Proactive Rule Discovery for Evidence-Grounded Optimization

Abstract

Constraints required by real-world optimization tasks may originate outside the problem description, and their applicability varies with jurisdiction, time, and task conditions. However, existing methods neither proactively assess whether the available information is sufficient for modeling nor actively search for missing external information, which can lead to modeling failures. To address this gap, we introduce SearchWorthyOR (SWOR), a benchmark built around real-world external rules with three paired conditions: rule inapplicable, applicable but omitted, and applicable and provided. SWOR evaluates rule discovery, applicability assessment, and information integration. We further propose SearchWorthy Agent (SWAgent), which uses an information–evidence table to link conditions requiring verification, supporting evidence, and modeling effects, organize on-demand search, and revise or retain the model according to verified findings. On SWOR, LLM answer accuracy is substantially lower in cases requiring proactive search than in those with rules provided, demonstrating the importance of this challenge for LLM-based automated modeling. SWAgent achieves 79.00% answer accuracy, the highest among the compared workflows, demonstrating its advantage in proactive external search.

Primary area:
optimization
Keywords:
Large language models, operations research, optimization modeling, evidence-grounded reasoning

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_dqz1OpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
3
Presentation
2
Soundness
3(+1)
Read review
Paper summary

The paper addresses the problem of missing external rules in LLM-based optimisation modelling. Real-world optimisation tasks may depend on regulations, policies, thresholds and exceptions that are not clearly stated in the problem description. The authors introduce SearchWorthyOR (SWOR), a benchmark comprising 100 source tasks across 10 optimisation scenario categories, each evaluated under three conditions: the rule is inapplicable, applicable but omitted, or applicable and provided. They also propose SearchWorthy Agent (SWAgent), which maintains an information evidence table linking unresolved conditions, supporting sources, applicability assessments, and mathematical model modifications. The agent iteratively searches external sources, verifies rule applicability, revises the optimisation formulation when justified and resolves it using Gurobi.

The evaluation covers 300 instances and compares SWAgent against several LLM-based optimisation and search workflows. SWAgent achieves 79% overall answer accuracy, outperforming the strongest performing workflow, codex-native-search (67.67%), by 11.33 percentage points. Ablation studies indicate that the information evidence table, initial model formulation, and closure criterion contribute to performance. However, accuracy on the most challenging condition, where an applicable rule is omitted, is only 45%, and trajectory audits reveal difficulties in translating retrieved evidence into correct mathematical modifications. Overall, the paper contributes a benchmark and an evidence-based agent architecture for proactive external rules discovery, with empirical results demonstrating improvements while highlighting persistent challenges in applicability assessment and model revision.

Critical strengths
  1. The paper formulation and benchmark design. The paper identifies an important limitation of LL-based optimisation: models may produce mathematically plausible solutions while overlooking the external rules that fundamentally change the optimal decisions. The SWOR benchmark addresses this gap through three paired conditions that distinguish rule applicability from rule visibility. Requiring the optimal action sets to be disjoint before and after applying an applicable rule ensures that missing information has measurable consequences, making the benchmark a useful contribution.

  2. Evidence-grounded optimisation with empirical validation. SWAgent introduces an information evidence table that explicitly connects unresolved conditions, supporting evidence, applicability assessments, and mathematical model modifications. Its iterative search, verification, and model-revision workflow provides a structured approach to integrating external knowledge into optimisation. The reported 79% overall answer outperforms the strongest compared workflow by 11.33 points, and the ablation studies provide empirical support for the proposed design. These results approach a useful step towards more reliable LLM-based optimisation agents.

Critical weaknesses
  1. Limited benchmark scale and external validity. The benchmark contains 100 source tasks across 10 optimisation categories, with three conditions per task. Although the paired conditions provide a useful controlled evaluation, the relatively small set of constructed scenarios leaves uncertainty about whether the reported gains generalise to real-world optimisation problems, where regulatory requirements may be ambiguous, interacting, or difficult to discover. Consequently, the reported 79% accuracy and 11.33 percentage points improvement over the strongest comparison do not yet establish robust performance in realistic deployments.

  2. The central challenge, which is discovering and incorporating omitted rules, remains inadequately solved. SWAgent achieves only 45% accuracy in C2 where a relevant rule applies but is omitted from the initial problem statement. The semantic audit also reports complete evidence-chain pass rates of 48.9% for C2 trajectories, indicating persistent difficulties connecting retrieved evidence to the optimisation model correctly. This is consequential because proactive discovery and evidence-based model revision are the paper’s central contribution; the results suggest that the proposed approach still fails on a substantial fraction of precisely the cases it is designed to address. The paper would be more convincing with stronger evidence that the method reliably discovers relevant rules and translates them into correct model changes, beyond improving aggregate accuracy.

Questions for authors
  1. Clarification: Benchmark generalisation. How were the 100 source tasks selected, and do the authors have evidence that the results generalise beyond the constructed scenarios? In particular, have the authors evaluated SWAgent on independently sourced, realistic optimisation problems not used during benchmark construction?

  2. Clarification: C2 failures and evidence grounding. Given that SWAgent achieves only 45% accuracy on C2 and the semantic audit reports a 48.9% complete evidence-chain pass rate, what are the dominant failure modes? How often does the agent fail to retrieve a relevant rule vs retrieve the correct rule but incorrectly translate it into the optimisation model? This distinction is important for assessing whether the proposed approach addresses the core problem or primarily improves performance in easier conditions.

  3. Clarification - comparison fairness. Were the SWAgent and the baseline workflows evaluated using equivalent search access, computational budgets, and solver configurations? To what extent does the reported 11.33 percentage point improvement reflect the information evidence table and associated reasoning procedure, rather than differences in search strategy or other system components?

15 comments
AuthorsOpenReview ↗

Response to Reviewer dqz1(1/8)

Dear Reviewer dqz1,

Thank you for your thoughtful review. We appreciate your recognition of the problem formulation, benchmark design, and experimental evaluation. Below, we provide additional explanations and experiments addressing your comments. We have also updated the paper PDF. If you have any further questions, we would be happy to discuss and address them.

AuthorsOpenReview ↗

Response to Reviewer dqz1(2/8)

W1: Benchmark coverage and realism

Thank you for your comments on benchmark coverage and realism.

SearchWorthyOR addresses a specific practical gap: business requirements may state the objective, resources, and candidate decisions without listing all external rules that determine feasibility. We use verifiable real-world rules to construct controlled optimization instances that test rule discovery, applicability judgments, and mathematical model updates.

(a) Application categories

The current benchmark contains 100 source tasks. Each has C1/C2/C3, giving 300 condition-specific instances. Each category contains 9–12 source tasks. The categories are listed below; we also included them in the appendix of the original submission.

Table 1. Application coverage.

Application categorySource tasksSpecific scenarios covered
Assignment and matching10Lighting recalls, room matching, hazardous materials
Energy and environment9Interregional grid connections, oil storage design, secondary containment design
Facilities and networks11Parking facilities, oil pipelines, control-function configuration
Finance and portfolios11ISA investment accounts, portfolios of positions
Healthcare resources10Treatment slots, eligibility matching, imaging equipment restoration
Inventory and supply chains9Hazardous materials, road freight consolidation, temperature control
Production and capacity12Product ingredients, composition thresholds, vehicle production, certificate resources
Routing and transportation9Cross-sea transportation, ferries, shuttle route planning
Scheduling and workforce10Cabin crew scheduling, vaccination services
Telecommunications services9Project delivery, SMS providers, message routing

These ten categories correspond to typical areas such as energy, finance, healthcare, production, logistics, and telecommunications in the ORSSA overview of application areas. They also cover classic problem types such as transportation and assignment, network optimization, and inventory, discussed in Chapters 9, 10, and 18 of Hillier and Lieberman's Introduction to Operations Research, 11th edition. These links between application areas and problem types show that our approach covers a broad range of applications and constructed scenarios.

(b) Rule complexity and interactions

Thank you and Reviewer RF3z for raising the issue of rule interactions. We checked all 100 C2 tasks and obtained the following results.

Table 2. Requirement structure in C2.

Rule structureTasks
Single requirement64
Multiple requirements, no clear interaction observed29
Multiple requirements with clear interactions7

We count requirements as obligations or restrictions that can be implemented separately. An interaction is a dependency, exception, or priority relation between requirements. Multiple requirements occur in 36% of tasks, and clear interactions occur in 7%.

For example,

under the ADA standards, the 240-space parking facility in R005 first requires at least 7 accessible spaces. This count then determines that at least 2 must be van-accessible spaces. The two requirements exclude options A and B in sequence, changing the optimal choice from A (utility 30) to C (utility 20).

This shows how rule dependencies affect the mathematical model and the decision. Each task has a fixed assessment date. Its rule materials record version dates and items awaiting verification, so that thresholds, exceptions, and scope can be distinguished across versions.

Overall, the benchmark covers different business domains and tests multiple requirements, rule interactions, and temporal applicability.

AuthorsOpenReview ↗

Response to Reviewer dqz1(3/8)

W1 / Q1: Task sources and independent evaluation

Thank you for this specific question. SearchWorthyOR serves as an evaluation set in this study and is not used to train SWAgent. We evaluate how a shared workflow for condition discovery, evidence checking, and mathematical model updates performs across optimization scenarios.

(a) Task sources and selection

The initial backgrounds and mathematical structures draw on IndustryOR, the OptiMiner benchmark, LogiOR, and scenarios from our team. We use assignment, scheduling, selection, and capacity models as starting points. We then use real-world materials to reconstruct the business setting, assessment date, candidate options, and numerical values, and add external requirements that change the decision. Each source task stores the issuing organization, URL, supporting source excerpts, rule description, Base/Full reference mathematical models, typed changes, and complete sets of optimal actions. Content and mathematical checks examine the interpretation of evidence, applicability to the instance, correspondence with constraints, and solver results.

(b) Information differences across C1/C2/C3

  • C1: The target rule is not applicable. We change applicability facts such as object attributes or time to test applicability judgments and preservation of the correct mathematical model.

  • C2: The target rule applies but is omitted from the problem statement. This tests whether the agent can discover the information gap, obtain the rule, and update the mathematical model.

  • C3: The applicable rule is provided. It shares the business facts and reference answer with C2, and adds a rule summary to the input. It tests modeling when the rule is known.

The three conditions deliberately control information visibility. The business rules and the grounds for the decisions come from real-world materials. The C2 Base model contains the conditions in the problem statement, while the Full model adds the applicable requirements. Construction requires their optimal action sets to be disjoint, so that omitting the rule changes the optimal decision.

C2 and C3 keep the business facts the same. Providing or withholding the rule allows us to examine the effect of filling the information gap.

(c) Generality and independent validation

SWAgent reuses the same workflow across application categories and backbones. Its inputs contain the public problem statement and output format, which the agent uses to determine what to check. These results provide evidence across domains and backbones on the current evaluation tasks.

Evaluation on independently sourced real-world optimization tasks is still pending. Future work will further test external validity using larger-scale real-world tasks, longer rule chains, and dynamically updated tasks.

Overall, this study uses an evaluation set to test how a shared workflow handles missing external rules. The current evidence comes from experiments across domains and backbones on these tasks.

Reviewer_ix8zOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates whether large LLMs can proactively discover and incorporate external rules into optimization problems. The authors introduce SWOR, a benchmark consisting of 100 source tasks and 300 problems under three conditions, designed to evaluate rule applicability, proactive rule discovery, and rule integration. The paper also proposes SWAgent, which combines active search, evidence management, and iterative model refinement. Experimental results show that SWAgent achieves the highest overall answer accuracy among the compared workflows.

The paper addresses a practically important problem and introduces a benchmark with paired evaluation conditions and multiple evaluation metrics. However, the representativeness of the benchmark and the reliability of its reference annotations require further investigation. In addition, the influence of pretrained knowledge on rule discovery remains unclear, and the effectiveness of SWAgent across different backbone models has not been sufficiently evaluated.

Critical strengths
  1. The paper addresses the practical challenge of discovering and incorporating external rules into optimization models, rather than assuming that all necessary constraints are provided in the problem description.

  2. The benchmark uses three paired conditions to evaluate rule applicability, proactive discovery, and rule integration. It also considers answer accuracy, model correctness, and evidence support, providing a relatively comprehensive evaluation.

Critical weaknesses
  1. Some models achieve relatively high accuracy on C2 without external retrieval. Since the external rules are drawn from publicly available sources, models may rely on pretrained knowledge rather than proactive search. Could the authors further investigate the influence of pretrained knowledge and clarify whether the benchmark effectively measures proactive rule discovery?

  2. Table 2 shows that Claude Opus 5 achieves 64% accuracy on C2 without external retrieval, outperforming SWAgent (45%), which uses GPT-5.6-luna/xhigh. This suggests that backbone model capabilities may substantially influence performance. Could the authors evaluate SWAgent with additional backbone models, such as Claude Opus 5 and GPT-6 Astra, and compare it against their corresponding base models to assess its effectiveness and generalizability across different LLMs?

Questions for authors
  1. The repository's README and documentation are primarily written in Chinese, I suggest providing English versions to make the code easier for international researchers to understand and reproduce.

  2. Although the appendix describes an LLM-based cross-review process for validating the reference answers, it is unclear whether the rule applicability judgments and mathematical constraints were independently verified by domain experts. Could the authors clarify whether human expert validation was conducted and how the reliability of the ground truth was ensured?

  3. The strict evidence-matching metric requires retrieved evidence to match reference quotations under the same official URLs. Although the authors also conduct semantic evaluations, could this strict criterion underestimate valid evidence obtained from other authoritative sources? It would be helpful to examine how different matching criteria affect the evaluation results.

13 comments
AuthorsOpenReview ↗

Response to Reviewer ix8z (1/6)

Dear Reviewer ix8z,

Thank you for your thoughtful review. We appreciate your recognition of the practical problem, paired evaluation conditions, and evaluation metrics. Below, we provide additional explanations and experiments addressing your comments. We have also updated the paper PDF. If you have any further questions, we would be happy to discuss and address them.

AuthorsOpenReview ↗

Response to Reviewer ix8z (2/6)

W1: Pretrained knowledge and proactive rule discovery

Thank you for raising this question, which helps clarify the respective roles of existing knowledge and external acquisition.

We added two observational analyses.

  • Analysis A asks whether a model can accurately reproduce the task-relevant content when the omitted rule is explicitly named.

  • Analysis B asks whether external acquisition improves optimization answers on tasks whose rules the model failed to reproduce accurately.

(a). Analysis A: Closed-book rule knowledge

Because the omitted rule is known for each source task, we supplied its name, the task's assessment date, and the relevant subject matter, and asked the model to reproduce the rule from its existing knowledge.

DeepSeek Flash and Claude Opus 5 were each prompted on the 100 source tasks. C1/C2/C3 share the same target rule, and both models had an output limit of 65,536 tokens. We froze a task-specific rubric before collecting the responses, covering three dimensions.

Table 1. Closed-book assessment rubric.

DimensionAssessment
Rule gistWhether the response correctly states what is required, prohibited, or permitted
Task-critical contentWhether thresholds, exceptions, deadlines, and other necessary details are accurate and complete
ApplicabilityWhether the covered entities, objects, jurisdiction, and temporal version are correct

For example, the R009 probe was approximately as follows.

As of August 4, 2026, what age and disease eligibility requirements does the FDA specify for Casgevy, particularly the crisis-related condition for sickle cell disease and the transfusion-dependence requirement for beta-thalassemia? Please answer from existing knowledge alone, without browsing the web or formulating or solving an optimization model.

For this task, the rubric assessed the following.

  • Rule gist. Treatment eligibility depends jointly on age and the approved disease indications.

  • Task-critical content. The reference version specified for the task sets a minimum age of 2 years, together with the corresponding disease requirements.

  • Applicability. The response identifies sickle cell disease (SCD) with recurrent vaso-occlusive crises or transfusion-dependent beta-thalassemia and uses the version applicable at the task's specified date.

A response counted as full recall only if it passed all three dimensions. Missing details received a partial pass, while an explicit error or an explicit inability to provide the required content received a fail.

Table 2. Closed-book rule recall.

BackboneRule gist pass ratePass rate for task-critical contentApplicability pass rateFull recall rate
DeepSeek Flash54.55%20.20%34.34%20.20%
Opus 543.43%3.03%14.14%3.03%

The results show that DeepSeek Flash has a higher full recall rate. More importantly, a substantial proportion of the tasks require information that is absent from both models' pretrained knowledge.

(b). Analysis B: Online/offline results after unsuccessful recall

We linked Analysis A to the existing C2 results by backbone and source-task ID using two predefined subsets. The first includes source tasks with incomplete recall, meaning that at least one dimension falls below a full pass. The second requires an explicit error or inability to reproduce the rule gist or task-critical content. The two subsets contain 79 and 73 source tasks for DeepSeek, and 96 and 87 for Opus, respectively. SWAgent-offline retains the information–evidence table, model updates, and solving.

Table 3. C2 accuracy on unsuccessful-recall subsets.

BackboneClosed-book probe subsetSWAgent-offline C2SWAgent-online C2Online gain
DeepSeek FlashIncomplete recall12.66%62.87%+50.21 pp
DeepSeek FlashRule gist or critical content failed12.33%64.84%+52.51 pp
Claude Opus 5Incomplete recall17.71%80.21%+62.50 pp
Claude Opus 5Rule gist or critical content failed14.94%80.46%+65.52 pp

The results show that our model substantially improves performance on tasks where the required content is absent from pretrained knowledge, supporting the role of search and reading on these tasks.

AuthorsOpenReview ↗

Response to Reviewer ix8z (3/6)

W2: Cross-backbone and base-model comparisons

Thank you for this suggestion. We agree that the method should be evaluated within each backbone. We have added Astra, Opus 5, and DeepSeek Flash experiments and report their corresponding base-model results and SWAgent online/offline results. The tables use Luna for GPT-5.6-luna, Astra for GPT-6 Astra, and Opus 5 for Claude Opus 5.

(a). Base-model performance

The base models directly formulate and solve the problems using the descriptions and their existing knowledge. The following results are those reported in the original paper.

Table 4. Base-model accuracy.

ModelOverallC1C2C3
Luna62.67%70.00%31.00%87.00%
Astra77.33%79.00%57.00%96.00%
Opus 581.00%79.00%64.00%100.00%
DeepSeek Flash68.33%76.00%32.00%97.00%

(b). SWAgent online/offline performance

The offline variant retains the information–evidence table, model updates, and solving. All configurations evaluate the same 300 problems, with 100 in each condition. Entries below are percentages, reported in online / offline order.

Table 5. SWAgent online/offline accuracy.

ModelOverallC1C2C3
Luna79.00 / 65.0093.00 / 95.0045.00 / 3.0099.00 / 97.00
Astra73.67 / 62.0097.00 / 97.0033.00 / 3.0091.00 / 86.00
Opus 589.67 / 66.6792.00 / 90.0081.00 / 17.0096.00 / 93.00
DeepSeek Flash84.00 / 67.4488.00 / 92.0066.33 / 12.3397.67 / 98.00

SWAgent-online exceeds the corresponding reported base-model overall accuracy for Luna, DeepSeek, and Opus.

In the additional same-backbone comparisons, DeepSeek SWAgent improves over Standard-online by 9.11 percentage points overall and 22.00 points on C2. Opus SWAgent improves over Claude Code by 5.00 points overall, with equal C2 accuracy. Enabling external acquisition raises C2 accuracy by 54 points for DeepSeek, 64 points for Opus, and 30 points for Astra. Astra's online C2 accuracy is 33%, below its base-model result of 57%, indicating room for improvement in its end-to-end performance.

Reviewer_RF3zOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper introduces SWOR, a benchmark for external-rule discovery in LLM-based optimization modeling: 100 Chinese OR tasks with 3 paired conditions (rule inapplicable / applicable-but-omitted / provided), with each rule constructed to change the optimal decision set. It also proposes SW Agent, built around an information–evidence table to guide information search and formulation revision. Experiments demonstrate that SW Agent reaches better performance under different conditions.

Critical strengths
  1. Constructing a benchmark with external rules is an interesting idea for OR area. Most of existing OR benchmarks assume that task description contains sufficient information for modeling, which may not hold on in realistic scenarios.
  2. In the experiments, each ablated component drops consistently (Table 4: −11.0 to −16.7 pts), and Figure 4 cleanly separates "retrieved the source" from "completed an evidence-grounded model" (source support 86.7/81.1% vs. full chain 66.7/48.9%), exposing the evidence-binding bottleneck without disguising it.
Critical weaknesses
  1. Poor writing style. The abstract is clear. However, the introduction and method sections contain many obscure and awkward sentences. For example, lines 32–42 of the introduction consist of numerous short, poorly connected sentences. In Eqs. (3.2) and (3.3) of the Problem Formulation section, the notations Compatible, Complete, and Consistent lack clear definitions or explanations. Sections 4.1 and 4.3 introduce at least 15 different symbols, some of which differ only in superscripts without sufficient explanation (e.g. V0,Vt,VrefV_0, V_t, V_{ref} and M,Ms0,Ms+M, M_s^0, M_s^+). These writing issues severely hindered my understanding of the paper's specific methodology and prevented me from properly assessing its substantive novelty.
  2. Over-strong benchmark construction assumptions. If I understand correctly, Eq. (4.3) assumes that each task has exactly one applicable rule that necessarily flips the optimum. Real-world settings, however, may involve zero or several interacting rules with ambiguous applicability.
  3. Insufficient cost-performance analysis. CoE and OptiMUS do not perform search by design; the only comparable baseline, Codex-Native-Search, is a black-box system with an incompatible cost-accounting convention and no trajectory-level failure attribution, and the C2 margin over it is only 4 points. Moreover, SW-Agent issues 2.6× more searches than Codex without any accompanying cost analysis. I suggest that the authors add comparative experiments using the same backbone with naive web search, together with a cost analysis.
Questions for authors

See my questions in weakness

7 comments
AuthorsOpenReview ↗

Response to Reviewer RF3z(1/4)

Dear Reviewer RF3z,

Thank you for your thoughtful review. We appreciate your recognition of the benchmark contribution and component evaluation. Below, we provide additional explanations and experiments addressing your comments. We have also updated the paper PDF. If you have any further questions, we would be happy to discuss and address them.

AuthorsOpenReview ↗

Response to Reviewer RF3z(2/4)

W1: Writing clarity, terminology, and notation

Thank you for identifying these presentation issues. We agree that the concepts and notation need more intuitive explanations. SWAgent's workflow can be summarized as follows. It identifies missing real-world conditions that could affect the decision, searches for and verifies those conditions, and then revises and solves the optimization model accordingly.

(a) Concept definitions

Table 1. Concept definitions.

ConceptIntuitive meaningRole in closure
CompatibleA candidate interpretation agrees with the problem facts, verified rules, and other conditions. Interpretations with the same modeling effect are grouped together.Determines whether a candidate interpretation is admissible.
ConsistentThe current information contains no unresolved contradictions.Provides the consistency requirement for closure.
closedWithin a specified set of conditions, each condition has a resolved compatible modeling effect, and the information is consistent.Unifies the original terms Complete and Closed under closed.

In the revision, subscripts will specify the scope of closure. Equation (3.3) will use closed⁡C(P)(P,Et)\operatorname{closed}_{\mathcal C(P)}(P,E_t), and Equation (5.3) will use closed⁡^t\widehat{\operatorname{closed}}_t for the operational assessment during execution.

(b) Notation

The symbols you highlighted describe information states, reference models, and models generated during execution. Their relationships are summarized below.

Table 2. Notation.

NotationExplanation
V0(ci),Vt(ci),Vref(ci)V_0(c_i),V_t(c_i),V_{\mathrm{ref}}(c_i)Sets of compatible answers to condition cic_i under the initial information, evidence accumulated through round tt, and reference evidence, respectively. Answers with the same modeling effect count as one element.
Ms0,Ms+M_s^0,M_s^+Reference models for source task ss before and after incorporating the applicable requirements.
M^0,M^T\widehat M_0,\widehat M_TThe initial and final models actually generated by the agent.
P,C(P),ciP,\mathcal C(P),c_iThe public task facts together with a revisable initial formulation, the relevant set of conditions, and an individual condition in that set.
E0,Et,EsrefE_0,E_t,E_s^{\mathrm{ref}}Initial external evidence, evidence accumulated through round tt, and reference evidence for source task ss. E0=∅E_0=\varnothing means that no external evidence has yet been acquired.
Δs,Δ^T\Delta_s,\widehat\Delta_TThe model modification supported by reference evidence and the final modification generated by the agent. These may affect variables, constraints, or the objective.
A⋆(M)\mathcal A^\star(M)All optimal public-action combinations for model MM, used to determine whether incorporating a rule changes the optimal decisions.

Here, ss indexes source tasks, while tt and TT denote the current and final iteration stages. We will provide a consolidated notation table in the appendix and briefly explain each symbol at its first occurrence in the main text.

(c) Planned revisions

Table 3. Planned manuscript revisions.

LocationRevision
Introduction and Section 3Rewrite the transitions in lines 32–42 to clarify the connection among missing conditions, evidence acquisition, and model revision. Use closed consistently in Sections 3 and 5.
Workflow descriptions in Sections 3 and 5Give a verbal overview in Section 3 and consolidate the search, reading, and update equations in the method section, introducing qtq_t and DtD_t consistently.
Benchmark construction in Section 4Explain the organization directly as 100 source tasks, each yielding C1/C2/C3. Replace the concatenation operator with a plain-language explanation that C3 adds a rule summary to the C2 description, and simplify dataset and evaluation indices.
Section 5 and the algorithm appendixPresent an actual record using four columns for the condition, current assessment, evidence, and modeling effect before introducing the update equation. Define the initial solver output and the information–evidence table in detail in the algorithm appendix and notation table.
AuthorsOpenReview ↗

Response to Reviewer RF3z(3/4)

W2: Rule complexity and benchmark assumptions

Thank you for highlighting the construction assumptions that require clarification. We agree that realistic tasks require applicability assessment and reasoning about interacting requirements, and the benchmark includes cases addressing both.

(a) The roles of C1, C2, and C3

Table 4. C1/C2/C3 evaluation conditions.

ConditionStatus of the target ruleRequired behavior
C1Inapplicable to the given factsVerify its scope and retain the original model.
C2Applicable but omitted from the problem descriptionIdentify the information gap, acquire the rule, and incorporate the required model modification.
C3Applicable and provided in the problem descriptionFormulate the model using the supplied rule.

C1 is constructed around a designated target rule. Additional external requirements can be examined in future extensions.

(b) Multiple requirements and interactions in C2

We reviewed all 100 C2 tasks and classified them by the number of independently implementable requirements and their interactions. Interactions include dependencies, exceptions, and precedence relationships.

Table 5. Requirement structure in C2.

Rule structureNumber of tasks
Single requirement64
Multiple requirements with no explicit interaction identified29
Multiple requirements with explicit interactions7

Multiple requirements occur in 36% of tasks, and explicit interactions occur in 7%, with 1–7 requirements per task. Requirements are counted as distinct obligations or restrictions. One requirement may translate into several mathematical constraints.

For example, R005 concerns parking provision at a commercial development and illustrates a dependency between requirements. The task describes a newly constructed 240-space parking facility with four configuration options. Under §208.2 of the ADA Standards, a facility with 201–300 spaces requires at least 7 accessible spaces. Section 208.2.4 then uses that total to determine the minimum number of van-accessible spaces, requiring one for every six accessible spaces or fraction thereof, which gives at least 2. U.S. Access Board guidance The first requirement excludes option A, which provides no accessible spaces. The second also excludes option B, which provides only one van-accessible space. The optimum changes from A, with utility 30, to C, with utility 20. This is a concrete dependency in which the result of one rule calculation becomes the input to another.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.