- Rating
- 2
- Confidence
- 4
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies the problem of large-scale optimization modeling, which is an important but under-explored topic. To do this, they propose an agentic system which can make better use of the instance data files than existing agentic/fine-tuned systems. On this basis, they can find more useful patterns/evidence from the instance data files to help build the math models more accurately. They achieve this by calling LLMs to perform "tool analysis". Next, they build a so-called experience memory in terms of these useful data evidence, which can be reused for solving subsequent test instances by doing RAG. They test their agentic system on four large-scale datasets, showing improved pass@1 accuracy and reduced solver-running time.
Critical strengths
-
This paper is very well-written, very well motivated, and thus easy to understand and follow;
-
They tackle an important but under-explored problem in LLM-based optimization modeling, that is, large-scale optimization modeling;
Critical weaknesses
-
[Coverage of Evaluated Benchmarks]: I was wondering how InForm can be extended to conventional (usually small-scale) optimization modeling benchmarks such as NL4Opt, OptMath-Bench, NL4LP, MAMO, ReScroatic, OptiBench, etc.? And what is the performance of InForm if it is possible?
-
[Adding cost-matched Baselines]: From the description of InForm, we can see that InForm can invoke the LLM multiple times. Therefore the authors should add some cost-matched baselines (which take approximately the same tokens, or the same number of times in terms of LLMs invoking, tool calls, etc.) to isolate the effect that the gains of InForm may be attributed to additional LLM calls or tool calls. (For example, if InForm call LLMs three times, the reported pass@1 score is actually more related to a pass@3 score. Therefore, it is unfair to compare it with the bare baseline's/ DeepSeek-V4-Flash's pass@1)
-
[Failure analysis of Baselines]: From Table 1, it can be seen that some agentic baselines perform worse than the bare baseline/ DeepSeek-V4-Flash. For example, on MIPLIB-NL, LEAN-LLM-OPT is worse than DeepSeek-V4-Flash? Why did this happen? Besides, why the improvements of other agentic baselines on this dataset were so limited: they achieved a score between 30.3 to 42.5, compared to 32.1 of DeepSeek-V4-Flash? (The same hold on other datasets such as Large-Scale-OR) Table 3 presents some error analysis, but it could not answer these questions.
-
[Adding more metrics and baselines]: 1) It can be seen that InForm largely improved the performance in term of pass@1: for example, pass@1 = 32.1 by DeepSeek-V4-Flash vs pass@1 = 63.3 by InForm, which is very impressive. However, will this gains still hold when we look at pass@4 (and/or pass@8)? 2) We should also measure the cost in terms of generation, including the time used (for generation, and for solver running that is already included), and the token used for fair comparisons; 3) Zero-shot LLMs: why not taking models from Gemini (such as Gemini-3.1-pro), Claude (such as Sonnet-4.6 and/or Opus-4.6) and Kimi (such as Kimi K-2.6) as baselines?
-
[More Explanations on Experience Memory:] The authors indicate that they maintain an Experience Memory that distills reusable evidence-to-decision associations from verified correct trajectories. I have the following questions: 1) how many items were there at the beginning in the Experience Memory? If not zero, how were they obtained? 2) How to obtain such "verified correct trajectories" as I assume that the ground-truth for early solved test instances are not available (as the labels of test instances should not be exposed)?
Questions for authors
Please check the weak points;
4 comments
3. Response to W3: Failure analysis of Baselines
W3: From Table 1, it can be seen that some agentic baselines perform worse than the bare baseline/ DeepSeek-V4-Flash. For example, on MIPLIB-NL, LEAN-LLM-OPT is worse than DeepSeek-V4-Flash? Why did this happen? ...
Thank you for pointing out this. We manually reviewed 50 randomly sampled failures across the agentic baselines and found three ways in which their workflows can leave modeling errors unresolved or introduce new ones.
- Decomposition can propagate incorrect structural assumptions. CoE and OptiMUS split formulation into subtasks, while OptiTree uses retrieved templates to guide decomposition. If an early abstraction treats overlapping groups as independent, later stages can produce internally consistent constraints that still omit the coupling required by the problem.
- Data processing can preserve values while misinterpreting their role. LEAN-LLM-OPT and ORPilot can prepare usable inputs, yet the formulation may still reverse a parameter’s sign convention or mistake a zero coefficient for an absent relation. We observed such errors in LEAN-LLM-OPT on instances that DeepSeek-V4-Flash solved correctly, offsetting gains on other instances.
- Execution feedback can miss modeling errors. OptiTree and ORPilot use execution feedback to repair outputs, but a model with missing constraints or incorrect indices can still return an optimal solution. In those cases, execution provides no signal that the formulation needs revision.
InForm addresses these failure modes before solving: it checks instance facts while modeling choices are still open and retrieves guidance based on those facts and the decision at hand.
4. Response to W4: Adding more metrics and baselines
W1: 1) It can be seen that InForm largely improved the performance in term of pass@1... 2) We should also measure the cost in terms of generation... 3) Zero-shot LLMs: why not taking models from Gemini, Claude and Kimi as baselines?
Thank you for your suggestion.
-
Pass@k accuracy. We compared DeepSeek-V4-Flash and InForm through pass@4. We report MIPLIB-NL, the benchmark highlighted by the reviewer, and the macro-average across all four benchmarks. InForm was tested under an online-memory with LLM verifications settings; seen in response 5. The gap narrows as more attempts are allowed, but InForm still leads at pass@4 by 17.1 percentage points on MIPLIB-NL and 4.6 points on macro-average. Thus, additional attempts help the baseline close part of the gap, while a clear advantage remains on MIPLIB-NL.
Benchmark Method Pass@1 (%) Pass@2 (%) Pass@3 (%) Pass@4 (%) MIPLIB-NL DeepSeek-V4-Flash 32.1 48.4 52.5 54.8 MIPLIB-NL InForm 64.3 68.3 69.7 71.9 Macro average DeepSeek-V4-Flash 68.3 79.5 81.9 84.1 Macro average InForm 83.2 87.2 88.1 88.7 -
Generation cost. We apologize for not highlighting this analysis in the main text. Appendix B.4 (Table 7) reports both generation time and token usage. InForm averages 104.5 seconds and 36.3K tokens per instance. It incurs additional cost over zero-shot prompting, but generating faster than all six agentic baselines and using fewer tokens than five.
-
Additional zero-shot baseline. Taking experimental costs into account, we initially selected five representative models spanning four model families. To broaden the zero-shot comparison, we additionally evaluated Kimi-K3.1-Pro on the same four benchmarks. Each cell reports ER (%) / Acc (%). Kimi-K3.1-Pro achieves a macro-average accuracy comparable to DeepSeek-V4-Pro and ranks second among the evaluated zero-shot models, behind GPT-5.6-Sol.
Method Large-Scale-OR Bench4Opt Text2Opt MIPLIB-NL Avg. Kimi-K3.1-Pro 96.7 / 85.5 99.5 / 84.3 97.3 / 90.0 60.0 / 42.8 88.4 / 75.6
We added the pass@K anlysis in Appenidx B.3 and the Kimi results in Table 1 main results of the revised manuscript.
5. Response to W5: More Explanations on Experience Memory
W5: I have the following questions: 1) how many items were there at the beginning in the Experience Memory? If not zero, how were they obtained? 2) How to obtain such "verified correct trajectories" as I assume that the ground-truth for early solved test instances are not available?
Thank you for raising this question.
We first clarify how Experience Memory was constructed for the original Table 1. The memory started empty. After each prediction was fixed and executed, we checked its solver output against the benchmark reference and distilled verified trajectories into memory. Although reference-based verification has been studied as an oracle setting for agent memory (e.g., ReasoningBank [1]), it gave InForm access to test-set feedback that the baselines did not receive.
To address this concern, we ran an additional online experiment using an LLM judge instead of reference answers to determine which trajectories to use for memory updates after each prediction is fixed and executed. In the results, InForm retains the highest average accuracy at 83.2%, 5.2 percentage points above the second-best baseline and 14.4 points above its backbone.
Table 1. Main Results on InForm and two zero-shot baselines. Each cell reports execution rate (ER) / accuracy (Acc), in percent.
| Method | Large-Scale-OR | Bench4Opt | Text2Opt | MIPLIB-NL | Avg. |
|---|---|---|---|---|---|
| GPT-5.6-Sol | 94.4 / 86.5 | 99.8 / 88.8 | 97.0 / 88.8 | 65.2 / 48.0 | 89.1 / 78.0 |
| DeepSeek-v4-Flash | 98.0 / 82.2 | 98.0 / 82.4 | 87.9 / 78.2 | 53.8 / 32.3 | 84.4 / 68.8 |
| InForm | 99.7 / 92.7 | 99.7 / 86.1 | 98.2 / 90.3 | 97.1 / 63.8 | 98.7 / 83.2 |
To verify that InForm can also benefit from Experience Memory built offline, we used DeepSeek-v4-Flash to generate trajectories for 110 original Text2Opt benchmark instances outside the evaluation sets. We distilled the trajectories into a memory and kept it frozen during evaluation. In the results, offline memory also raised average accuracy from 80.1% to 82.5%.
Table 2. InForm’s modeling accuracy across four benchmarks under different Experience Memory settings.
| Memory setting | Large-Scale-OR | Bench4Opt | Text2Opt | MIPLIB-NL | Avg. |
|---|---|---|---|---|---|
| Without memory | 90.8 | 84.6 | 87.3 | 57.8 | 80.1 |
| Offline frozen memory | 92.1 | 83.8 | 91.8 | 62.4 | 82.5 |
| Online memory with LLM verification | 92.7 | 86.1 | 90.3 | 63.8 | 83.2 |
| Online memory with reference verification | 93.1 | 87.5 | 95.2 | 62.6 | 84.6 |
All three memory variants improved average accuracy, suggesting that transferable experience in instance analysis and formulation benefits across the memory sources and verification methods tested. Even without memory, InForm without memory still outperformed the strongest baseline, highlighting the value of using instance evidence to guide formulation.
We clarified our memory settings in Section 2.4 and added these detailed analysis of the Experience Memory mechanism in Section 3.3 of the revised manuscript.
[1] Ouyang, S., Yan, J., Hsu, I., Chen, Y., Jiang, K., Wang, Z., ... & Pfister, T. Reasoningbank: Scaling agent self-evolving with reasoning memory. ICLR, 2026.
We again thank the reviewers for their careful evaluation and constructive feedback. If our responses have addressed your concerns, we would greatly appreciate your reconsideration of the score.
Gentle reminder: the discussion period is nearing its end, and we would welcome your feedback
Dear Reviewer ZbRc,
Thank you again for your careful review. As the ICLR discussion period is approaching its conclusion, we would greatly appreciate any feedback on whether our rebuttal has addressed your concerns.
If any concern remains unresolved, we would be happy to provide further clarification. If our responses and new evidence have addressed your concerns, we would sincerely appreciate your reconsideration of the score.
Thank you for your time and consideration.
Best regards, The Authors
1. Response to W1: Coverage of Evaluated Benchmarks
Thank you for this suggestion. Our original evaluation focused on large-scale optimization modeling with separate instance data. Many conventional benchmarks embed numerical values in the problem text, limiting the use of InForm’s instance-analysis tools.
To test whether InForm transfers to smaller problems, we evaluated it on 49 NLP4LP Hard, 47 IndustryOR, and 130 ReSocratic instances. We chose these subsets because they contain structured instance data, including Markdown tables embedded in the problem descriptions, that can be converted into separate input files for the InForm pipeline. These are the pass@1 accuracy on the selected instances.
InForm ties with OptiTree for the highest accuracy on NLP4LP Hard and ranks second behind CoE on both IndustryOR and ReSocratic, suggesting that its instance-analysis approach is also useful for smaller optimization problems. We added this anlysis in Appenidx B.2 of the revised manuscript.
2. Response to W2: Adding cost-matched Baselines
Thank you for pointing this out. To control for LLM call count, we added a four-call DeepSeek-V4-Flash baseline: three calls generate candidate formulations, and a fourth selects one without seeing reference answers or solver outcomes.
The extra calls improve DeepSeek-V4-Flash from 68.8% to 71.4%. InForm achieves 83.2%, a 11.8-point gain, with the largest gap on MIPLIB-NL (63.8% vs. 38.0%). CoE and OptiMUS also score below InForm despite averaging 7.0 and 13.4 LLM calls per instance. These results suggest that additional calls alone do not explain InForm’s gains and support the value of using instance evidence to guide formulation.