acceptodds
Under review as a conference paper at ICLR 2027

Instance Evidence Matters: Tool-Integrated LLM Reasoning for Data-Informed Optimization Formulation

Abstract

Large language models (LLMs) for optimization modeling typically treat instance data as values to be bound to a formulation inferred from problem descriptions. Yet instance data can play a more fundamental role: when interpreted under problem semantics, their structural and numerical properties become instance evidence that can directly inform formulation decisions, affecting both model correctness and solver efficiency. We introduce **InForm** (**IN**stance-informed **FORM**ulation), a tool-integrated framework that progressively acquires and uses such evidence during optimization formulation. Rather than analyzing the entire instance upfront, InForm interleaves formulation reasoning with targeted tool calls, acquiring decision-relevant evidence before the corresponding formulation decisions are finalized. Its formulation-oriented tools transform raw instance data into mathematically meaningful evidence, while experience retrieval provides reusable guidance for translating recurring evidence patterns into local formulation decisions. Across four large-scale OR benchmarks, InForm achieves 84.1% average formulation accuracy, outperforming the strongest baseline by 6.8 percentage points, and improving 16.2 percentage points on MIPLIB-NL against base LLM. InForm also yields 1.07–2.18 solver-time improvements on commonly correct instances. These results show that instance data should not merely instantiate an optimization model: the evidence they reveal can actively shape how the model is formulated.

Primary area:
optimization
Keywords:
Large Language Models, Optimization Modeling, Instance-Informed Formulation, Tool Use

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_ZbRcOpenReview ↗
Rating
2
Confidence
4
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies the problem of large-scale optimization modeling, which is an important but under-explored topic. To do this, they propose an agentic system which can make better use of the instance data files than existing agentic/fine-tuned systems. On this basis, they can find more useful patterns/evidence from the instance data files to help build the math models more accurately. They achieve this by calling LLMs to perform "tool analysis". Next, they build a so-called experience memory in terms of these useful data evidence, which can be reused for solving subsequent test instances by doing RAG. They test their agentic system on four large-scale datasets, showing improved pass@1 accuracy and reduced solver-running time.

Critical strengths
  1. This paper is very well-written, very well motivated, and thus easy to understand and follow;

  2. They tackle an important but under-explored problem in LLM-based optimization modeling, that is, large-scale optimization modeling;

Critical weaknesses
  1. [Coverage of Evaluated Benchmarks]: I was wondering how InForm can be extended to conventional (usually small-scale) optimization modeling benchmarks such as NL4Opt, OptMath-Bench, NL4LP, MAMO, ReScroatic, OptiBench, etc.? And what is the performance of InForm if it is possible?

  2. [Adding cost-matched Baselines]: From the description of InForm, we can see that InForm can invoke the LLM multiple times. Therefore the authors should add some cost-matched baselines (which take approximately the same tokens, or the same number of times in terms of LLMs invoking, tool calls, etc.) to isolate the effect that the gains of InForm may be attributed to additional LLM calls or tool calls. (For example, if InForm call LLMs three times, the reported pass@1 score is actually more related to a pass@3 score. Therefore, it is unfair to compare it with the bare baseline's/ DeepSeek-V4-Flash's pass@1)

  3. [Failure analysis of Baselines]: From Table 1, it can be seen that some agentic baselines perform worse than the bare baseline/ DeepSeek-V4-Flash. For example, on MIPLIB-NL, LEAN-LLM-OPT is worse than DeepSeek-V4-Flash? Why did this happen? Besides, why the improvements of other agentic baselines on this dataset were so limited: they achieved a score between 30.3 to 42.5, compared to 32.1 of DeepSeek-V4-Flash? (The same hold on other datasets such as Large-Scale-OR) Table 3 presents some error analysis, but it could not answer these questions.

  4. [Adding more metrics and baselines]: 1) It can be seen that InForm largely improved the performance in term of pass@1: for example, pass@1 = 32.1 by DeepSeek-V4-Flash vs pass@1 = 63.3 by InForm, which is very impressive. However, will this gains still hold when we look at pass@4 (and/or pass@8)? 2) We should also measure the cost in terms of generation, including the time used (for generation, and for solver running that is already included), and the token used for fair comparisons; 3) Zero-shot LLMs: why not taking models from Gemini (such as Gemini-3.1-pro), Claude (such as Sonnet-4.6 and/or Opus-4.6) and Kimi (such as Kimi K-2.6) as baselines?

  5. [More Explanations on Experience Memory:] The authors indicate that they maintain an Experience Memory that distills reusable evidence-to-decision associations from verified correct trajectories. I have the following questions: 1) how many items were there at the beginning in the Experience Memory? If not zero, how were they obtained? 2) How to obtain such "verified correct trajectories" as I assume that the ground-truth for early solved test instances are not available (as the labels of test instances should not be exposed)?

Questions for authors

Please check the weak points;

4 comments

1. Response to W1: Coverage of Evaluated Benchmarks

W1: I was wondering how InForm can be extended to conventional (usually small-scale) optimization modeling benchmarks such as NL4Opt, OptMath-Bench, NL4LP, MAMO, ReScroatic, OptiBench, etc.?

Thank you for this suggestion. Our original evaluation focused on large-scale optimization modeling with separate instance data. Many conventional benchmarks embed numerical values in the problem text, limiting the use of InForm’s instance-analysis tools.

To test whether InForm transfers to smaller problems, we evaluated it on 49 NLP4LP Hard, 47 IndustryOR, and 130 ReSocratic instances. We chose these subsets because they contain structured instance data, including Markdown tables embedded in the problem descriptions, that can be converted into separate input files for the InForm pipeline. These are the pass@1 accuracy on the selected instances.

MethodNLP4LP HardIndustryORReSocraticAvg.
DeepSeek-v4-Flash (zero-shot)67.368.152.362.6
CoE71.480.964.672.3
OptiMUS67.374.560.867.5
OptiTree73.572.360.068.6
AlphaOPT65.372.359.265.6
LEAN-LLM-OPT71.470.255.465.7
ORPilot69.468.157.765.1
InForm73.576.662.370.8

InForm ties with OptiTree for the highest accuracy on NLP4LP Hard and ranks second behind CoE on both IndustryOR and ReSocratic, suggesting that its instance-analysis approach is also useful for smaller optimization problems. We added this anlysis in Appenidx B.2 of the revised manuscript.


2. Response to W2: Adding cost-matched Baselines

W2: ... Therefore the authors should add some cost-matched baselines (which take approximately the same tokens, or the same number of times in terms of LLMs invoking, tool calls, etc.) to isolate the effect that the gains of InForm may be attributed to additional LLM calls or tool calls.

Thank you for pointing this out. To control for LLM call count, we added a four-call DeepSeek-V4-Flash baseline: three calls generate candidate formulations, and a fourth selects one without seeing reference answers or solver outcomes.

MethodLarge-Scale-ORBench4OptText2OptMIPLIB-NLAvg.
DeepSeek-V4-Flash: three candidates + LLM selection83.280.783.638.071.4
InForm: online memory with LLM verification (Seen in Response 5)92.786.190.363.883.2

The extra calls improve DeepSeek-V4-Flash from 68.8% to 71.4%. InForm achieves 83.2%, a 11.8-point gain, with the largest gap on MIPLIB-NL (63.8% vs. 38.0%). CoE and OptiMUS also score below InForm despite averaging 7.0 and 13.4 LLM calls per instance. These results suggest that additional calls alone do not explain InForm’s gains and support the value of using instance evidence to guide formulation.


3. Response to W3: Failure analysis of Baselines

W3: From Table 1, it can be seen that some agentic baselines perform worse than the bare baseline/ DeepSeek-V4-Flash. For example, on MIPLIB-NL, LEAN-LLM-OPT is worse than DeepSeek-V4-Flash? Why did this happen? ...

Thank you for pointing out this. We manually reviewed 50 randomly sampled failures across the agentic baselines and found three ways in which their workflows can leave modeling errors unresolved or introduce new ones.

  • Decomposition can propagate incorrect structural assumptions. CoE and OptiMUS split formulation into subtasks, while OptiTree uses retrieved templates to guide decomposition. If an early abstraction treats overlapping groups as independent, later stages can produce internally consistent constraints that still omit the coupling required by the problem.
  • Data processing can preserve values while misinterpreting their role. LEAN-LLM-OPT and ORPilot can prepare usable inputs, yet the formulation may still reverse a parameter’s sign convention or mistake a zero coefficient for an absent relation. We observed such errors in LEAN-LLM-OPT on instances that DeepSeek-V4-Flash solved correctly, offsetting gains on other instances.
  • Execution feedback can miss modeling errors. OptiTree and ORPilot use execution feedback to repair outputs, but a model with missing constraints or incorrect indices can still return an optimal solution. In those cases, execution provides no signal that the formulation needs revision.

InForm addresses these failure modes before solving: it checks instance facts while modeling choices are still open and retrieves guidance based on those facts and the decision at hand.


4. Response to W4: Adding more metrics and baselines

W1: 1) It can be seen that InForm largely improved the performance in term of pass@1... 2) We should also measure the cost in terms of generation... 3) Zero-shot LLMs: why not taking models from Gemini, Claude and Kimi as baselines?

Thank you for your suggestion.

  1. Pass@k accuracy. We compared DeepSeek-V4-Flash and InForm through pass@4. We report MIPLIB-NL, the benchmark highlighted by the reviewer, and the macro-average across all four benchmarks. InForm was tested under an online-memory with LLM verifications settings; seen in response 5. The gap narrows as more attempts are allowed, but InForm still leads at pass@4 by 17.1 percentage points on MIPLIB-NL and 4.6 points on macro-average. Thus, additional attempts help the baseline close part of the gap, while a clear advantage remains on MIPLIB-NL.

    BenchmarkMethodPass@1 (%)Pass@2 (%)Pass@3 (%)Pass@4 (%)
    MIPLIB-NLDeepSeek-V4-Flash32.148.452.554.8
    MIPLIB-NLInForm64.368.369.771.9
    Macro averageDeepSeek-V4-Flash68.379.581.984.1
    Macro averageInForm83.287.288.188.7
  2. Generation cost. We apologize for not highlighting this analysis in the main text. Appendix B.4 (Table 7) reports both generation time and token usage. InForm averages 104.5 seconds and 36.3K tokens per instance. It incurs additional cost over zero-shot prompting, but generating faster than all six agentic baselines and using fewer tokens than five.

  3. Additional zero-shot baseline. Taking experimental costs into account, we initially selected five representative models spanning four model families. To broaden the zero-shot comparison, we additionally evaluated Kimi-K3.1-Pro on the same four benchmarks. Each cell reports ER (%) / Acc (%). Kimi-K3.1-Pro achieves a macro-average accuracy comparable to DeepSeek-V4-Pro and ranks second among the evaluated zero-shot models, behind GPT-5.6-Sol.

    MethodLarge-Scale-ORBench4OptText2OptMIPLIB-NLAvg.
    Kimi-K3.1-Pro96.7 / 85.599.5 / 84.397.3 / 90.060.0 / 42.888.4 / 75.6

We added the pass@K anlysis in Appenidx B.3 and the Kimi results in Table 1 main results of the revised manuscript.


5. Response to W5: More Explanations on Experience Memory

W5: I have the following questions: 1) how many items were there at the beginning in the Experience Memory? If not zero, how were they obtained? 2) How to obtain such "verified correct trajectories" as I assume that the ground-truth for early solved test instances are not available?

Thank you for raising this question.

We first clarify how Experience Memory was constructed for the original Table 1. The memory started empty. After each prediction was fixed and executed, we checked its solver output against the benchmark reference and distilled verified trajectories into memory. Although reference-based verification has been studied as an oracle setting for agent memory (e.g., ReasoningBank [1]), it gave InForm access to test-set feedback that the baselines did not receive.

To address this concern, we ran an additional online experiment using an LLM judge instead of reference answers to determine which trajectories to use for memory updates after each prediction is fixed and executed. In the results, InForm retains the highest average accuracy at 83.2%, 5.2 percentage points above the second-best baseline and 14.4 points above its backbone.

Table 1. Main Results on InForm and two zero-shot baselines. Each cell reports execution rate (ER) / accuracy (Acc), in percent.

MethodLarge-Scale-ORBench4OptText2OptMIPLIB-NLAvg.
GPT-5.6-Sol94.4 / 86.599.8 / 88.897.0 / 88.865.2 / 48.089.1 / 78.0
DeepSeek-v4-Flash98.0 / 82.298.0 / 82.487.9 / 78.253.8 / 32.384.4 / 68.8
InForm99.7 / 92.799.7 / 86.198.2 / 90.397.1 / 63.898.7 / 83.2

To verify that InForm can also benefit from Experience Memory built offline, we used DeepSeek-v4-Flash to generate trajectories for 110 original Text2Opt benchmark instances outside the evaluation sets. We distilled the trajectories into a memory and kept it frozen during evaluation. In the results, offline memory also raised average accuracy from 80.1% to 82.5%.

Table 2. InForm’s modeling accuracy across four benchmarks under different Experience Memory settings.

Memory settingLarge-Scale-ORBench4OptText2OptMIPLIB-NLAvg.
Without memory90.884.687.357.880.1
Offline frozen memory92.183.891.862.482.5
Online memory with LLM verification92.786.190.363.883.2
Online memory with reference verification93.187.595.262.684.6

All three memory variants improved average accuracy, suggesting that transferable experience in instance analysis and formulation benefits across the memory sources and verification methods tested. Even without memory, InForm without memory still outperformed the strongest baseline, highlighting the value of using instance evidence to guide formulation.

We clarified our memory settings in Section 2.4 and added these detailed analysis of the Experience Memory mechanism in Section 3.3 of the revised manuscript.

[1] Ouyang, S., Yan, J., Hsu, I., Chen, Y., Jiang, K., Wang, Z., ... & Pfister, T. Reasoningbank: Scaling agent self-evolving with reasoning memory. ICLR, 2026.


We again thank the reviewers for their careful evaluation and constructive feedback. If our responses have addressed your concerns, we would greatly appreciate your reconsideration of the score.

AuthorsOpenReview ↗

Gentle reminder: the discussion period is nearing its end, and we would welcome your feedback

Dear Reviewer ZbRc,

Thank you again for your careful review. As the ICLR discussion period is approaching its conclusion, we would greatly appreciate any feedback on whether our rebuttal has addressed your concerns.

If any concern remains unresolved, we would be happy to provide further clarification. If our responses and new evidence have addressed your concerns, we would sincerely appreciate your reconsideration of the score.

Thank you for your time and consideration.

Best regards, The Authors

Reviewer_NtJUOpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper presents a tool-integrated large language model (LLM) framework for optimization-model formulation that combines formulation reasoning (where experience memory retrieves guidance from successful trajectories) with targeted tool calls to extract instance evidence from external data tables. The authors report average accuracy performance gains on multiple OR benchmarks with solver speedups.

Critical strengths

First, the paper provides a well-motivated problem with clear examples (e.g., fixed-charge network flow). Additionally, the distinction between instance data as model coefficients versus evidence that shapes optimization model structure is very intuitive and practically relevant.

Second, the author's tool design is practical, and the paper has strong ablations that show how progressive tool use, specialized tools, and memory each contribute meaningfully. Further, their evaluation on multiple backbones and diverse benchmarks adds credibility.

Critical weaknesses

In the paper, online experience memory is extracted from verified correct test-set trajectories and used to help later test instances within a fixed evaluation order. While same-batch leakage is avoided using this approach, this is an online evaluation and not independent test-set evaluation. As such, the reported memory improvement may reflect benchmark-specific learning rather than transferable reasoning.

Questions for authors
  1. Can you report results with experience memory frozen on a disjoint training set?
  2. How does this compare to online updating?
  3. What is the sensitivity to test-instance ordering?
3 comments

1. Response to W / Q1 / Q2

W: In the paper, online experience memory is extracted from verified correct test-set trajectories and used to help later test instances within a fixed evaluation order. ... As such, the reported memory improvement may reflect benchmark-specific learning rather than transferable reasoning. Q1: Can you report results with experience memory frozen on a disjoint training set? Q2: How does this compare to online updating?

Thank you for this suggestion. In the original online evaluation, Experience Memory started empty. After each prediction was finalized and executed, we checked its solver output against the test reference and used the verified trajectory and its reference to update memory for later batches. The reported gains may therefore partly reflect adaptation to the benchmark.

To test whether experience in instance analysis and formulation transfers to disjoint problems, we ran two follow-up experiments.

  • Offline frozen memory. We used DeepSeek-v4-Flash to generate trajectories for 110 Text2Opt instances disjoint from the evaluation sets. We distilled the solver-verified trajectories into memory and froze it before evaluation.
  • Online memory with LLM verification. A separate LLM judged completed trajectories without seeing test references. We distilled the accepted trajectories into memory for later batches.
Memory settingLarge-Scale-ORBench4OptText2OptMIPLIB-NLAvg.
Without memory90.884.687.357.880.1
Offline frozen memory92.183.891.862.482.5
Online memory with LLM verification92.786.190.363.883.2
Online memory with reference verification93.187.595.262.684.6

Offline frozen memory and online memory with LLM verification improved average accuracy over w/o memory variant by 2.4 and 3.1 percentage points, respectively, and outperformed the strongest baseline by 4.5 and 5.2 points. All three memory variants improved average accuracy, suggesting that transferable experience in instance analysis and formulation benefits modeling across the memory sources and verification methods tested.

To more specify this, we added a detailed analysis of Experience Memory mechanism in Section 3.3 of the revised manuscript.


2. Response to Q3

Q3: What is the sensitivity to test-instance ordering?

Thank you for this suggestion. We ran three independent experiments for each online memory variant, using shuffle seeds 2024, 2025, and 2026 to vary the test-instance order.

Table 1. Accuracy (%) across test-instance orderings with LLM verification

Shuffle seedLarge-Scale-ORBench4OptText2OptMIPLIB-NLMacro-average
202491.187.886.463.882.3
202594.185.394.563.384.3
202693.185.390.064.383.2

Table 2. Accuracy (%) across test-instance orderings with reference verification

Shuffle seedLarge-Scale-ORBench4OptText2OptMIPLIB-NLMacro-average
202492.187.397.365.285.5
202594.188.894.559.384.2
202693.186.393.663.384.1

Across the tested orderings, macro-average accuracy varied by 2.0 percentage points with LLM verification and 1.4 points with reference verification. Both variants outperformed InForm without memory (80.1%) and the strongest baseline (78.0%) under every tested ordering. InForm’s advantage is robust to changes in test-instance order, consistent with the benefit of transferable experience in instance analysis and formulation.


We again thank the reviewers for their careful evaluation and constructive feedback. If our responses have addressed your concerns, we would greatly appreciate your reconsideration of the score.

Thank you very much for addressing my three questions. Your additional experimentation does a nice job addressing them.

Thank you again for your careful review and valuable suggestions, which have greatly helped us improve the quality and clarity of our paper. We are glad that our additional experiments have addressed your concerns, and sincerely appreciate your recognition and positive assessment of our work.

Reviewer_bwmxOpenReview ↗
Rating
3(+1)
Confidence
2(−2)
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

InForm interleaves optimization formulation with deterministic tools that extract relevant properties from external instance data. It retrieves evidence-to-decision guidance from a memory updated using previously verified trajectories, without changing the backbone parameters. Across four benchmarks, the method reports 84.1% macro-average accuracy and 63.3% on MIPLIB-NL. Component ablations, an additional backbone, case studies, and cost analyses examine the contributions of progressive analysis, specialized tools, and experience retrieval, alongside solver-time gains on commonly solved instances.

Critical strengths
  • Formulation-oriented tools expose relevant data structure without requiring the model to inspect complete raw tables.
  • Strong no-memory results and component ablations support the practical value of progressive instance analysis itself.
  • Shared-backbone agent comparisons and separate inference-cost measurements help assess the practical value of the framework.
Critical weaknesses
  • Online memory uses verified test trajectories, but its validation oracle and baseline access are insufficiently specified.
  • Objective matching and unstated reformulation assumptions leave semantic correctness of generated formulations insufficiently verified.
  • Unreported variability across repeated runs limits confidence in the reported accuracy and solver-time improvements.
Questions for authors
  1. Table 2 attributes a 3.9-point gain to memory built from verified trajectories. Does verification use test references, and does the gain persist with memory built on disjoint problems and frozen before evaluation?

  2. The abstract's 16.2-point MIPLIB-NL gain matches the comparison with GPT-5.6-Sol, while Table 1 implies a 31.2-point gain over the backbone. Should the abstract name GPT-5.6-Sol as the comparator or report the backbone gain?

  3. Which assumptions justify Figure 1's flow bound, and are returned assignments independently checked against the original constraints? Are formulation audits available?

  4. Were the results in Tables 1–2 averaged over multiple runs with consistent experimental settings? Please report the number of runs, means, and standard deviations.

  5. Equation (6) compares each baseline on a different subset of jointly solved instances. Please report these subset sizes and whether the speedups persist on a shared subset.

7 comments

1. Response to W1 / Q1

W1: Online memory uses verified test trajectories, but its validation oracle and baseline access are insufficiently specified. Q1: Table 2 attributes a 3.9-point gain to memory built from verified trajectories. Does verification use test references, and does the gain persist with memory built on disjoint problems and frozen before evaluation?

Thank you for raising this concern. In the original online setting, Experience Memory started empty. Once each prediction was finalized and executed, we checked its solver output against the test reference and used the verified trajectory to build memory for later batches. Ground-truth trajectory verification, previously studied as an oracle setting for online agent memory (e.g., ReasoningBank [1]), however, gave InForm access to post-submission feedback unavailable to the baselines.

To address this concern, we ran two follow-up experiments.

  • Online memory with LLM verification. A separate LLM judged completed trajectories without seeing test references. Accepted trajectories were distilled into memory for later batches.
  • Offline frozen memory. We distilled solver-verified DeepSeek-v4-Flash trajectories from 110 Text2Opt instances disjoint from the evaluation sets and froze the memory before evaluation.
Memory settingLarge-Scale-ORBench4OptText2OptMIPLIB-NLAvg.
Without memory90.884.687.357.880.1
Offline frozen memory92.183.891.862.482.5
Online memory with LLM verification92.786.190.363.883.2
Online memory with reference verification93.187.595.262.684.6

Both settings improved macro-average accuracy over InForm without memory and exceeded the strongest baseline by 4.5 and 5.2 percentage points, respectively. This suggests that transferable experience in instance analysis and formulation benefits across the memory settings tested. Even without memory, InForm outperformed the strongest baseline by 2.1 points, highlighting the value of using instance evidence to guide formulation.

We updated Tables 1 and 2 with the LLM-verified online results and added a detailed analysis of Experience Memory mechanism in Section 3.3 of the revised manuscript.

[1] Ouyang, S., Yan, J., Hsu, I., Chen, Y., Jiang, K., Wang, Z., ... & Pfister, T. Reasoningbank: Scaling agent self-evolving with reasoning memory. ICLR, 2026.


2. Response to Q2

Q2: The abstract's 16.2-point MIPLIB-NL gain matches the comparison with GPT-5.6-Sol, while Table 1 implies a 31.2-point gain over the backbone. Should the abstract name GPT-5.6-Sol as the comparator or report the backbone gain?

Thank you for pointing this out. We apologize for the error in the abstract. The 16.2-point figure came from comparing the rounded accuracies of InForm (63.3%) and GPT-5.6-Sol (47.1%) in Table 1. Our intended comparison was with the backbone, DeepSeek-V4-Flash (32.1%), giving a gain of 31.2 percentage points on MIPLIB-NL. We have corrected the wording in the abstract of the revised manuscript. We also checked the related comparisons throughout the manuscript for consistency. Thank you again for your careful reading.


3. Response to W2 / Q3

W2: Objective matching and unstated reformulation assumptions leave semantic correctness of generated formulations insufficiently verified. Q3: Which assumptions justify Figure 1's flow bound, and are returned assignments independently checked against the original constraints? Are formulation audits available?

Thank you for raising this question. Pass@1 Acc measures whether a single generated program returns the reference objective value within tolerance. Objective matching is a widely used measure in LLM-based optimization modeling, including in ORLM, SIRL, OptiTree, and AlphaOPT. We agree that objective matching alone does not establish semantic correctness. However, direct verification remains challenging: some benchmarks lack reference formulations, and structurally different formulations can be equivalent, as studied by EquivaMap [2]. We apply the same criterion to InForm and all baselines, so their reported solving accuracies are compared on the same basis.

To further address this concern, we therefore let human experts review 25 objective-matching InForm formulations per benchmark against the original problem statements. Among the 100 conclusive cases, we found only one material modeling error: a missing unit conversion in the transmission loss term that went undetected because the total objective value was within the scoring tolerance.

Large-Scale-ORBench4OptText2Opt-110MIPLIB-NL
Modeling error rate0% (0/25)0% (0/25)4% (1/25)0% (0/25)

For the question, as noted above, we scored this case by objective matching without independently checking the returned assignment. When selecting it for Figure 1, we checked InForm’s formulation against the original requirements. The instance data show balanced supply and demand of Q=1,965Q=1,965 and zero per-unit flow costs. Since circulation is not required, removing cycles with link activations fixed preserves flow balance and cost. The remaining paths carry QQ units in total, so the data-derived bound fe≤Qyef_e\le Qy_e enforces the original activation rule while preserving an optimum.

In the revised manuscript, we clarify the Figure 1 example, explicitly define Acc, and discuss the limitations of using objective matching to assess formulation correctness.

[2] Zhai, H., Lawless, C., Vitercik, E., & Leqi, L. (2025). Equivamap: Leveraging llms for automatic equivalence checking of optimization formulations. ICML, 2025.


4. Response to Q4

Q4: Were the results in Tables 1–2 averaged over multiple runs with consistent experimental settings? Please report the number of runs, means, and standard deviations.

Thank you for this suggestion. The original manuscript reported results from a single run per experimental setting. We now report results from three independent runs with the mean and standard deviation. In the results, InForm still achieves the highest averaged accuracy across the four benchmarks among all the baselines.

4. Response to Q4 (continuing)

Table 1. Main results across three rounds. Each cell reports executable rate and Pass@1 accuracy as ER / Acc, with each metric shown as mean (%) ± std (%).

MethodLarge-Scale-ORBench4OptText2OptMIPLIB-NLAvg.
GPT-5.6-Sol-Codex94.4 ± 1.14 / 86.5 ± 1.5199.8 ± 0.29 / 88.8 ± 0.5197.0 ± 1.39 / 88.8 ± 1.8965.2 ± 3.59 / 48.0 ± 4.1589.1 ± 0.64 / 78.0 ± 1.95
DeepSeek-v4-Flash98.0 ± 0.99 / 82.2 ± 4.9598.0 ± 1.52 / 82.4 ± 3.1087.9 ± 0.52 / 78.2 ± 3.2853.8 ± 1.97 / 32.3 ± 1.1484.4 ± 0.45 / 68.8 ± 2.30
DeepSeek-v4-Pro99.3 ± 0.57 / 86.8 ± 0.5797.6 ± 1.06 / 82.6 ± 1.5597.6 ± 1.39 / 87.0 ± 0.5268.5 ± 0.26 / 42.4 ± 1.3190.8 ± 0.66 / 74.7 ± 0.30
DouBao-Seed2.0-Pro97.4 ± 1.51 / 79.9 ± 2.8695.6 ± 1.78 / 79.9 ± 2.6093.0 ± 1.89 / 83.3 ± 2.1054.4 ± 2.23 / 31.4 ± 3.4685.1 ± 1.45 / 68.6 ± 1.37
Kimi-K3.1-Pro96.7 ± 0.57 / 85.5 ± 2.4999.5 ± 0.88 / 84.3 ± 3.1797.3 ± 0.91 / 90.0 ± 3.2860.0 ± 1.71 / 42.8 ± 4.5388.4 ± 0.81 / 75.6 ± 0.83
Qwen3-Max98.3 ± 1.14 / 79.2 ± 4.5495.6 ± 1.55 / 79.0 ± 0.7896.1 ± 1.05 / 85.5 ± 2.7354.9 ± 0.94 / 30.2 ± 1.0486.2 ± 0.37 / 68.5 ± 0.60
ORLM35.3 ± 5.45 / 19.8 ± 2.6232.7 ± 1.06 / 21.3 ± 0.5130.6 ± 1.89 / 4.8 ± 1.3928.2 ± 2.28 / 1.5 ± 1.0431.7 ± 0.82 / 11.9 ± 0.56
OptMATH34.3 ± 4.46 / 23.4 ± 2.8624.2 ± 3.06 / 13.4 ± 0.7817.9 ± 4.20 / 6.7 ± 1.0512.1 ± 2.28 / 0.8 ± 0.2622.1 ± 3.40 / 11.1 ± 1.12
SIRL56.4 ± 0.99 / 28.7 ± 0.9963.5 ± 1.83 / 33.0 ± 1.7637.9 ± 1.89 / 5.5 ± 1.8234.5 ± 7.01 / 3.8 ± 0.6948.1 ± 1.14 / 17.7 ± 0.32
StepORLM47.9 ± 0.57 / 25.7 ± 0.9948.1 ± 3.38 / 27.7 ± 3.7433.0 ± 3.78 / 11.5 ± 4.5822.6 ± 0.90 / 2.7 ± 0.4537.9 ± 1.70 / 16.9 ± 2.39
CoE88.4 ± 5.45 / 77.6 ± 6.3797.6 ± 0.29 / 81.4 ± 2.8990.3 ± 3.19 / 77.0 ± 4.3072.7 ± 0.52 / 38.5 ± 2.9787.3 ± 2.25 / 68.6 ± 3.95
OptiMUS91.7 ± 2.86 / 77.9 ± 1.1496.6 ± 1.28 / 72.3 ± 1.6391.5 ± 0.52 / 73.9 ± 3.6758.7 ± 0.26 / 34.5 ± 1.1484.6 ± 0.61 / 64.7 ± 1.37
OptiTree95.0 ± 1.98 / 80.9 ± 5.0898.6 ± 0.29 / 83.4 ± 2.1197.9 ± 1.39 / 82.1 ± 3.7881.3 ± 2.14 / 36.3 ± 2.4993.2 ± 1.29 / 70.7 ± 3.28
AlphaOPT91.4 ± 0.57 / 78.9 ± 2.0695.9 ± 0.51 / 75.1 ± 1.3494.2 ± 1.39 / 78.2 ± 2.7383.1 ± 0.52 / 43.6 ± 5.2191.2 ± 0.60 / 68.9 ± 0.85
LEAN-LLM-OPT94.7 ± 2.06 / 86.1 ± 0.9999.2 ± 0.59 / 85.6 ± 1.7892.4 ± 1.05 / 77.9 ± 3.4469.1 ± 1.04 / 31.7 ± 2.7588.8 ± 0.75 / 70.3 ± 1.17
ORPilot92.4 ± 3.02 / 76.9 ± 7.6997.5 ± 0.51 / 76.8 ± 0.5996.4 ± 1.57 / 81.8 ± 0.9170.0 ± 1.45 / 41.6 ± 1.9789.1 ± 0.87 / 69.3 ± 1.92
InForm99.7 ± 0.57 / 92.7 ± 1.5199.7 ± 0.29 / 86.1 ± 1.4798.2 ± 0.91 / 90.3 ± 4.1097.1 ± 1.59 / 63.8 ± 0.4598.7 ± 0.57 / 83.2 ± 1.02

Table 2. Ablation study of InForm across three rounds. Each cell reports Pass@1 accuracy as mean (%) ± std (%).

VariantLarge-Scale-ORBench4OptText2OptMIPLIB-NLAvg.
Baseline82.2 ± 4.9582.4 ± 3.1078.2 ± 3.2832.3 ± 1.1468.8 ± 2.30
w/o Progressive90.1 ± 0.9984.4 ± 3.8481.5 ± 1.0553.1 ± 2.5777.3 ± 0.21
w/o OR-Tools86.8 ± 1.5182.6 ± 0.7884.5 ± 3.9655.5 ± 2.7377.4 ± 1.82
w/o Memory90.8 ± 0.5784.6 ± 0.7887.3 ± 0.9157.8 ± 1.4580.1 ± 0.18
Full Ours92.7 ± 1.5186.1 ± 1.4790.3 ± 4.1063.8 ± 0.4583.2 ± 1.02

We also updated the appendix Table 6 to report the results of the Doubao experiments averaged across three runs. All updated results are included in the revised manuscript.


Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Rebuttal

Dear Reviewers and Area Chairs,

We sincerely thank the three reviewers for their careful evaluation and constructive feedback. Their comments are valuable in improving our paper.

Following the discussion, our final reviewer ratings were 3, 4, and 2, with an average of 3.0.

  • bwmx (2 → 3) addressed their initial concerns and shifted to supporting acceptance of our paper.
  • NtJU (3 → 4) acknowledged that our additional evidence adequately addressed all questions and expressed stronger support.
  • ZbRc (2) maintained the original rating. We provided further clarifications, but received no further feedback within the discussion period.

The key strengths recognized by the reviewers are summarized below.

  • Paper Writing Quality (ZbRc, NtJU): The paper was described as well-written, well-motivated, and easy to follow.
  • Problem Significance (ZbRc, NtJU): Reviewers recognized large-scale optimization modeling as an important and under-explored research problem.
  • Methodological Design (bwmx, NtJU, ZbRc): Reviewers appreciated the practical formulation-oriented tools and the use of instance evidence to guide modeling decisions.
  • Empirical Effectiveness (bwmx, NtJU): Reviewers recognized the effectiveness demonstrated across multiple backbones and diverse benchmarks, supported by comprehensive component ablations.

The key concerns raised by the reviewers and our corresponding responses are summarized below.

  • Experience Memory Mechanism
    • Memory generalization (ZbRc W5; NtJU Q1–Q2; bwmx W1/Q1): Both offline frozen memory and online memory with LLM verification improved performance, demonstrating the transferability of experience.
    • Instance ordering (NtJU Q3): Multi-seed results showed consistent improvements across different instance orderings, confirming the robustness of the gains.
    • Memory-free performance (ZbRc W5; bwmx W1/Q1): InForm without memory still outperforms the strongest baseline, confirming that instance analysis itself drives the core improvements. We consequently adopted the memory-free results in Table 1 and revised the corresponding discussion of the Memory mechanism.
  • Broader Experimental Evaluation
    • Benchmark generalization (ZbRc W1): InForm achieved competitive results on three smaller optimization benchmarks, demonstrating its applicability beyond large-scale problems.
    • Evaluation coverage (ZbRc W2/W4): Cost-matched and Pass@K comparisons confirmed that the performance gains cannot be explained by additional LLM calls alone. We also included an additional zero-shot baseline.
    • Reliability (bwmx Q4/Q5): We reported results over three independent runs and solver-time comparisons on a shared subset.
  • Further Analysis and Clarifications
    • Baseline failure analysis (ZbRc W3): We identified and quantified major failure patterns in existing agentic baselines.
    • Objective-matching limitations (bwmx W2/Q3): Clarified this common limitation of existing OR modeling methods. Expert audits found only 1 material error among 100 cases (1%).
  • Presentation Correction
    • Abstract comparison (bwmx Q2): We corrected the MIPLIB-NL performance comparison and clarified the intended baseline.

All additional results, analyses, and clarifications have been incorporated into the revised manuscript. We respectfully ask the Area Chairs to consider our additional experiments, clarifications and the positive score updates from two reviewers in the final assessment.

Thank you for your time and consideration.

Best regards, The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.