acceptodds
Under review as a conference paper at ICLR 2027

OptDiff: Formulating Optimization Problems with Masked Diffusion Language Models

Abstract

Turning a natural language description into a solver-ready optimization model demands faithful interpretation of entities, quantities, and relations, as well as generation fast enough for practical use. Existing approaches generate and revise such formulations with autoregressive language models, whose sequential decoding makes long formulations and iterative correction increasingly costly. We observe that an optimization formulation involves three tightly coupled kinds of decisions: variable semantics, relation structure, and numerical grounding. To make these decisions explicit, we design an intermediate representation (IR) that separates them while preserving their dependencies through shared semantic declarations. On this IR, masked diffusion can reconstruct multiple decisions in parallel and revise selected regions through remasking. Building on it, we present OptDiff, which fixes variable semantics in a shared node, reconstructs objective and constraint structures in parallel with confidence-guided denoising, binds numerical values in a separate stage, and repairs flagged statements through localized remasking. Across four optimization modeling benchmarks, OptDiff achieves competitive modeling accuracy with the lowest generation time among the compared methods. Compared with a matched autoregressive baseline, it improves objective-value accuracy by 5.3 percentage points on average and reduces generation time by roughly , reaching nearly on complex formulations. These results indicate that optimization formulations become well suited to diffusion once their semantic, structural, and numerical decisions are made explicit.

Primary area:
foundation or frontier models, including LLMs
Keywords:
optimization modeling, large language models, diffusion language models, masked diffusion, autoformulation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_3GUPOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper studies the efficiency of LLM-based optimization modeling (converting optimization problems given in natural language into math models and corresponding solver code). Rather than using conventional autoregressive language models, they propose to use diffusion language models. To achieve this, they design an intermediate representation to handle variable semantics, relation structure, and numerical grounding. On this basis, the masked diffusion can reconstruct multiple decisions at the same time and revise selected parts through remasking, and thus can generate contents in a faster way. They also conduct experiments to show the effectiveness of their method, and conduct ablation studies to show the necessity of each design component.

Critical strengths
  1. This paper is very easy to follow: the writing is smooth and the sections are well organised.

  2. The topic (efficiency of optimization modeling) is very interesting, but this topic is under-explored in the optimization modeling community.

Critical weaknesses
  1. [Design Choice]: "we design an intermediate representation (IR) that separates them while preserving their dependencies through shared semantic declarations." Is this design necessary? This kind of design (namely explicitly modeling intermediate representatio/IR) appeared at the early stage of LLM-based automated optimization modeling (say 2022 or 2023). However, few researchers employ this kind of design as they found that: using IR is not necessary and it may degrade the performance of LLM in optimization modeling (I did not remember which paper(s) exactly. But from the trend of previously published papers in this direction, we know that few studies employs IR to help optimization modeling anymore).

  2. [Selection of Baselines]: The selection of baselines is confusing. For commercial/open LLMs with APIs, they only compare it with GPT-4o (which is a very old model in the year of 2024) and Qwen3-3B. Why not more recent models such as GPT5.4, 5.5, 5.6, 6? Why not Claude families? Why not Gemini families (especially the Gemini Flash Series)? Why not DeepSeek families (DeepSeek Flash Models:)? Besides, for a given model such as GPT-5.6, how to set the thinking mode (low, medium, high, ultra) as we know that more thinking need more generation time? Besides, why SIRL and OptMATH (both are open-sourced) were omitted for comparison?

  3. [Considering only Generation Time is Not Enough]: The efficiency of optimization modeling should be measured in at least two dimensions (see SAGE:https://arxiv.org/pdf/2605.02545): 1) the time to generate solver-executable code and 2)the time to execute it with Gurobi/COPT. However, this paper only considers/compares the time of generation (without consideirng the time of generation), which can be misleading. For example, if an LLM (say GPT-5.6-Sol-Ultra) takes 5 minutes to generate code and the generated solver code takes 1 minute to solve; On the contrary, OptDiff takes 1 seconds to generate the solver code, but takes 1 hour to run the solver code. I think most users will perfer GPT-5.6-Sol-Ultra in this case.

  4. [Insufficient Benchmarks]: Most papers in optimization modeling in the year of 2026 perform experiments on at least 7/8 benchmarks (see StepORLM (https://arxiv.org/pdf/2509.22558), SAGE(https://arxiv.org/pdf/2605.02545), optiTree(https://arxiv.org/pdf/2510.22192)). The authors only conducted experiments on 4 benchmarks. Therefore, I do not trust the generalibility of their conclusions.

  5. [How to Handle Large-Scale Optimization Problem?] 1) In large-scale optimization modeling scenario, the data and problem description are usually stored separately: How can the diffusion LLMs handle it? (Adding the data into the problem description will lead to a context length of millions tokens or even more); 2) in large scale optimization modeling, the number of variables and/or constraints can be millions or even more, how to handle this? (This weak point is not a specific weak point to this paper, but to almost all optimization modeling papers. So I did not consider this point when I gave the ratings. However, considering that this paper is tackling the problem of efficiency, I think this is closely related).

I am willing to raise my ratings if these concerns are (partially) addressed.

Questions for authors

Please check the weak points.

2 comments
AuthorsOpenReview ↗

Response to Reviewer 3GUP — Part 1/2

CW1: "Is this design necessary?"

Role of IR in diffusion.
Thank you for raising this design question. We acknowledge that an IR is not generally necessary for autoregressive optimization modeling. In contrast to AR approaches, our diffusion-based framework uses the IR to preserve dependencies among modeling decisions while enabling parallel generation and localized reconstruction, making it necessary for our proposed generation and repair workflow. When generating solver code directly, variable declarations and their dependent relations can be reconstructed simultaneously before the declarations are settled, making consistency harder to maintain. Our IR establishes shared variable declarations first, supports parallel objective and constraint generation under this shared context, and then binds numerical values. It also allows selected fields to be remasked and corrected while unaffected content remains fixed.

Evidence from the ablations.
Table 2 of the manuscript supports this design within the same iLLaDA backbone. After task SFT and adaptive denoising, cumulatively adding shared semantics, branch decoupling, and separate numerical binding raises NL4Opt accuracy from 69.0% to 89.7% and reduces generation time from 4.51 s to 2.44 s, before local repair. We will clarify this role of IR in Sec. 3.1.

CW2: "Why not more recent models such as GPT5.4, 5.5, 5.6, 6?"

Expanded comparisons and reasoning budgets.
Thank you for highlighting these gaps. We have added SIRL, OptMATH, and recent GPT baselines; the results show that OptDiff retains a generation-speed advantage across the evaluated methods and reasoning settings. We evaluate GPT-5.6-Sol and GPT-6-Sol at Medium reasoning effort in Table 3.1 and report Low, Medium, and High reasoning-effort results on the four original benchmarks in Table 3.2.

Comparison scope.
We compare our 8B diffusion model with similarly sized open models, including ORLM, StepORLM, and Qwen3.5-9B + SFT. GPT models add practical accuracy–latency reference points. Since their capacity, internal architecture, and serving resources cannot be matched to our 8B setup, we interpret their scores as system-level comparisons that complement the open-model results.

Accuracy and generation efficiency.
Across all eight benchmarks, OptDiff is both faster and more accurate than Qwen3.5-9B + SFT. Its speed advantage also holds against both GPT models on all eight benchmarks: on NL4Opt, for example, OptDiff takes 2.81 s, compared with 28.82 s and 26.28 s for GPT-5.6-Sol and GPT-6-Sol at Medium effort. We will add Tables 3.1 and 3.2 with this analysis to the appendix.

Table 3.1. Expanded baseline comparison and additional benchmark evaluation.

Original four benchmarks

MethodNL4Opt N=213MAMO-EasyLP N=545MAMO-ComplexLP N=111NLP4LP N=178
Acc. (%)Gen. (s)Acc. (%)Gen. (s)Acc. (%)Gen. (s)Acc. (%)Gen. (s)
Zero-shot LLMs
GPT-4o (closed)61.012.2370.315.0157.725.7073.611.12
GPT-5.6-Sol (Medium)92.028.8296.727.7282.072.7497.834.43
GPT-6-Sol (Medium)92.026.2897.434.6382.043.1398.322.89
Qwen3-32B77.546.9792.345.7146.890.4993.847.25
Fine-tuned LLMs
ORLM-Llama3 (8B)90.613.3591.615.5763.127.3088.217.50
StepORLM (8B)96.721.5497.622.0777.530.3297.225.66
LLMOPT-Qwen2.5 (14B)85.422.9887.722.9464.040.0286.533.13
SIRL-Qwen2.5 (7B)89.713.1891.913.0276.629.6297.213.46
OptMATH-Qwen2.5 (7B)85.429.2390.627.1436.026.8887.127.56
Prompting and Agentic Methods
Chain-of-Experts (GPT-4o)66.767.7494.382.8050.575.3886.562.01
OptiMUS-v0.3 (GPT-4o)76.161.9678.082.6046.889.6188.871.86
CAFA (GPT-4o)68.17.5271.27.5645.09.4850.08.66
Controlled Comparison
Qwen3.5-9B + SFT89.25.5181.56.5959.553.7191.05.42
OptDiff (8B)93.92.8187.73.0767.69.0093.32.79

Table 3.2. GPT models across reasoning effort levels.

ModelEffortNL4Opt N=213MAMO-EasyLP N=545MAMO-ComplexLP N=111NLP4LP N=178
Acc. (%)Gen. (s)Acc. (%)Gen. (s)Acc. (%)Gen. (s)Acc. (%)Gen. (s)
GPT-5.6-SolLow93.432.2194.926.9379.359.9095.526.73
GPT-5.6-SolMedium92.028.8296.727.7282.072.7497.834.43
GPT-5.6-SolHigh91.537.0497.444.2485.6142.0598.947.69
GPT-6-SolLow93.026.9595.422.8682.932.4597.823.60
GPT-6-SolMedium92.026.2897.434.6382.043.1398.322.89
GPT-6-SolHigh92.035.4697.448.5986.598.8598.930.02
AuthorsOpenReview ↗

Response to Reviewer 3GUP — Part 2/2

CW3: "Considering only Generation Time Is Not Enough"; "the time to generate solver-executable code" and "the time to execute it with Gurobi/COPT".

Compilation and complete execution time.
Thank you for this helpful suggestion. We report supplementary end-to-end timing results in Table 3.3 below, showing that OptDiff retains its speed advantage after compilation and solving are included. Specifically, we conducted additional timing experiments on NL4Opt and MAMO-ComplexLP, comparing OptDiff with five baselines under matched solver settings, CPU resources, and time limits. OptDiff's deterministic compiler converts the generated IR into executable solver code (Sec. 3.5). Process Overhead covers process management, library loading, file/result handling, and coordination between stages.

End-to-end results and accounting.
OptDiff achieves the lowest end-to-end latency among the six methods: 2.852 s on NL4Opt and 9.050 s on MAMO-ComplexLP, corresponding to 1.95× and 5.94× speedups over Qwen3.5-9B + SFT. Its mean solving time is 0.005 s on each dataset, indicating that the generation savings are not offset by solver execution. End-to-end latency is measured directly through completion or termination, with failures and timeouts included in the averages. Repair-related operations are assigned to their respective stages without double counting. We have supplemented Appendix B with this end-to-end table and a short explanation of the stage definitions and timing scope.

Table 3.3. End-to-end latency and stage-wise breakdown. All times are in seconds.

NL4Opt (N = 213)

MethodGen.Extract & CheckCompile & BuildSolveProcess OverheadEnd-to-end
GPT-5.6-Sol (Medium)28.8210.0010.0240.0050.02528.876
StepORLM (8B)21.5410.0020.0360.0050.02821.612
SIRL-Qwen2.5 (7B)13.1750.0010.0460.0060.03313.261
OptMATH-Qwen2.5 (7B)29.2330.0010.0410.0040.03229.311
Qwen3.5-9B + SFT5.5080.0020.0050.0060.0305.551
OptDiff (8B)2.8060.0030.0060.0050.0322.852

MAMO-ComplexLP (N = 111)

MethodGen.Extract & CheckCompile & BuildSolveProcess OverheadEnd-to-end
GPT-5.6-Sol (Medium)72.7440.0020.0250.0040.03472.809
StepORLM (8B)30.3150.0080.0340.0050.03330.395
SIRL-Qwen2.5 (7B)29.6200.0020.0450.0050.03529.707
OptMATH-Qwen2.5 (7B)26.8810.0020.0410.0030.03226.959
Qwen3.5-9B + SFT53.7090.0060.0080.0030.03453.760
OptDiff (8B)8.9980.0040.0080.0050.0359.050

CW4: "The authors only conducted experiments on 4 benchmarks."

Broader benchmark coverage.
Thank you for raising this concern. We have expanded the evaluation from four to eight benchmarks, adding ComplexOR, IndustryOR, ReSocratic, and OptMATH Bench. The additional results are reported in Table 3.1 below; we will include the full comparison and analysis in the appendix.

Table 3.1 (continued). Additional four benchmarks.

MethodComplexOR N=18IndustryOR N=100ReSocratic N=403OptMATH Bench N=166
Acc. (%)Gen. (s)Acc. (%)Gen. (s)Acc. (%)Gen. (s)Acc. (%)Gen. (s)
Zero-shot LLMs
GPT-4o (closed)44.424.9838.021.4542.419.9117.528.63
GPT-5.6-Sol (Medium)50.064.1750.060.6787.331.7128.959.99
GPT-6-Sol (Medium)72.231.7762.039.9887.628.2130.764.82
Qwen3-32B27.866.7435.080.0768.054.3914.587.56
Fine-tuned LLMs
ORLM-Llama3 (8B)50.010.6235.016.9875.413.073.626.29
StepORLM (8B)38.923.3042.038.7682.422.7012.775.48
LLMOPT-Qwen2.5 (14B)27.829.9331.034.2380.124.105.454.84
SIRL-Qwen2.5 (7B)55.618.3339.025.3781.414.8525.353.09
OptMATH-Qwen2.5 (7B)44.425.1126.027.8269.034.2722.938.97
Prompting and Agentic Methods
Chain-of-Experts (GPT-4o)55.685.1533.072.1172.265.3018.775.50
OptiMUS-v0.3 (GPT-4o)50.082.7344.079.0287.670.2222.381.54
CAFA (GPT-4o)44.49.8141.08.8362.07.8812.09.08
Controlled Comparison
Qwen3.5-9B + SFT38.97.7536.018.8868.76.035.483.81
OptDiff (8B)50.03.8340.04.5180.12.2121.76.60

CW5: "the data and problem description are usually stored separately. How can the diffusion LLMs handle it?" "the number of variables and/or constraints can be millions or even more, how to handle this?"

Large-scale data and model construction.
Thank you for highlighting this broader setting. Our evaluation covers text-specified benchmark problems; external-data interfaces and scalable model construction are future directions. This setting requires external-data integration, long-range dependency management, and memory-efficient construction of large variable and constraint systems. We will state this limitation and these directions in the Conclusion (Sec. 5).

Reviewer_ixaYOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper converts natural-language optimization problems into solver-ready formulations using masked diffusion. Specifically, the authors introduce an intermediate representation separating variable semantics, relation structure and numerical binding, thereby enabling parallel denoising and efficient generation. The work achieves speedup over an autoregressive baseline while maintaining competitive accuracy.

Critical strengths

First, the paper's intermediate representation approach using structured diffusion modeling is architecturally sound and exploits problem structure in a way that suits masked reconstruction. This is important because it demonstrates a credible case for where diffusion outperforms autoregressive generation beyond mere speed.

Second, the author's approach consistently reduces mean generation time while improving accuracy relative to a supervised fine-tuning baseline on the same hardware. This is important because latency is critical for interactive optimization.

Critical weaknesses

First, the paper's correctness metric is weak. The formulations are counted correct if they compile, solve, and match the reference optimal value rather than the optimal solution or feasible region. Further, semantically incorrect formulations can achieve the same objective, thereby inflating accuracy and directly undermining claims of solver-ready formulations and modeling accuracy.

Second, it appears that comparison is not properly controlled within the experimentation, as different pretrained families, tokenizers, and instruction tuning can confound decoding-strategy effects. Additionally, critical training details are missing, preventing reproduction.

Questions for authors
  1. What is the end-to-end latency breakdown including parameter extraction, validation, repair, compilation, and solving?
  2. Can you compare diffusion versus autoregression on the same backbone?
5 comments
AuthorsOpenReview ↗

Response to Reviewer ixaY — Part 1/2

CW1: "The formulations are counted correct if they compile, solve, and match the reference optimal value rather than the optimal solution or feasible region."

Manual semantic audit.
Thank you for raising this concern. Our manual evaluation on all four benchmarks confirms that OptDiff retains its accuracy advantage after checking formulation semantics. We audited all objective-matching formulations from both methods against the problem descriptions using identical criteria for variables, objectives, constraints, bounds, units, and integrality. Verified Acc. is the percentage of the full test set that both matches the reference optimal value and passes this audit (Table 2.1).

Accuracy after semantic verification.
OptDiff's accuracy remains unchanged on all four benchmarks. On MAMO-ComplexLP, its Verified Acc. is 67.6%, while Qwen3.5-9B + SFT decreases from 59.5% to 57.7%, exposing semantic errors missed by objective agreement. We will add Table 2.1, these audit criteria, and the definition of Verified Acc. to the appendix.

Table 2.1. Formulation correctness beyond objective agreement.

DatasetMethodObjective Acc. (%)Verified Acc. (%)Time (s)
NL4OptQwen3.5-9B + SFT89.289.25.51
NL4OptOptDiff (8B)93.993.92.81
MAMO-EasyLPQwen3.5-9B + SFT81.581.56.59
MAMO-EasyLPOptDiff (8B)87.787.73.07
MAMO-ComplexLPQwen3.5-9B + SFT59.557.753.71
MAMO-ComplexLPOptDiff (8B)67.667.69.00
NLP4LPQwen3.5-9B + SFT91.091.05.42
NLP4LPOptDiff (8B)93.393.32.79

CW2 / Q2: "different pretrained families, tokenizers, and instruction tuning can confound decoding-strategy effects." "Can you compare diffusion versus autoregression on the same backbone?"

Matched controls and ablations.
Thank you for this suggestion. Our controlled comparison and within-backbone ablations support OptDiff and its components, although the cross-model comparison cannot fully exclude backbone effects. The controlled comparison in manuscript Table 1 matches task-training data, IR, branch parallelism, separate binding, local-repair scope, repair prompts, and hardware. Table 2 evaluates adaptive denoising and other components with the iLLaDA backbone fixed.

Backbone attribution.
We agree that a same-backbone AR-diffusion comparison would strengthen attribution and requires matched initialization and training. Our matched-workflow comparison and within-iLLaDA ablations provide complementary evidence for the framework and its components. We will state these controls and the remaining backbone limitation in Sec. 4.4, and prioritize the same-backbone comparison if revision time permits.

AuthorsOpenReview ↗

Response to Reviewer ixaY — Part 2/2

CW2: "Additionally, critical training details are missing, preventing reproduction."

Training data and adapters.
Thank you for requesting these details. We independently train the relation, binding, and objective-form LoRA adapters from the task-adapted model. Task SFT uses 1,482 training problems: 498 from NL4Opt, 122 from MAMO (93 EasyLP and 29 ComplexLP), 569 from OR-Instruct, and 293 from public OptMATH data. Their 4,446 stage-specific records, together with 2,269 auxiliary numerical-model records, form 6,715 SFT records. Task SFT uses LoRA rank 64 and alpha 128; the specialized adapters use rank 8 and alpha 16.

Masking and reproducibility.
Task SFT combines semantic-row, full-response, and random masking according to annotation availability. Specialized objectives mask relation responses, binding slots, or objective-form/variable-domain classification codes, preserving fixed syntax during binding. We will provide a compact listing in the appendix, covering source counts and splits, stage initialization and training schedules, optimizer settings, learning rates, batch sizes, steps, seeds, model/tokenizer versions, and checkpoint selection, with shared and model-specific settings identified for Qwen3.5-9B + SFT.

Q1: "What is the end-to-end latency breakdown including parameter extraction, validation, repair, compilation, and solving?"

Timing scope and repair accounting.
Thank you for requesting this breakdown. Table 2.2 reports the full execution-time breakdown, showing that OptDiff retains its end-to-end speed advantage after costs beyond generation are included. Repair-related model calls are counted in Gen.; repeated checks, compilation/model building, and solving are assigned to their respective stages without double counting. Process Overhead covers process management, library loading, file/result handling, and coordination between stages.

End-to-end results.
We measure latency directly from receiving the problem to returning a result or termination status, retaining failed and timed-out instances in the averages. OptDiff achieves 1.95× and 5.94× speedups over Qwen3.5-9B + SFT on NL4Opt and MAMO-ComplexLP, respectively. We have supplemented Appendix B with the end-to-end results shown in Table 2.2, together with the stage definitions and timing scope.

Table 2.2. End-to-end latency and stage-wise breakdown. All times are in seconds.

NL4Opt (N = 213)

MethodGen.Extract & CheckCompile & BuildSolveProcess OverheadEnd-to-end
GPT-5.6-Sol (Medium)28.8210.0010.0240.0050.02528.876
StepORLM (8B)21.5410.0020.0360.0050.02821.612
SIRL-Qwen2.5 (7B)13.1750.0010.0460.0060.03313.261
OptMATH-Qwen2.5 (7B)29.2330.0010.0410.0040.03229.311
Qwen3.5-9B + SFT5.5080.0020.0050.0060.0305.551
OptDiff (8B)2.8060.0030.0060.0050.0322.852

MAMO-ComplexLP (N = 111)

MethodGen.Extract & CheckCompile & BuildSolveProcess OverheadEnd-to-end
GPT-5.6-Sol (Medium)72.7440.0020.0250.0040.03472.809
StepORLM (8B)30.3150.0080.0340.0050.03330.395
SIRL-Qwen2.5 (7B)29.6200.0020.0450.0050.03529.707
OptMATH-Qwen2.5 (7B)26.8810.0020.0410.0030.03226.959
Qwen3.5-9B + SFT53.7090.0060.0080.0030.03453.760
OptDiff (8B)8.9980.0040.0080.0050.0359.050

Thank you for taking the time to review and respond to my feedback. The addition of Table 2.1 to the paper makes it a stronger claim, so thanks for adding. Further, I appreciate you acknowledging my comment on confounding, so adding the limitation to Section 4.4 would be good. I am hopeful, however, that you can conduct the same-backbone comparison in the revision.

Reviewer_t3SqOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

OptDiff formulates optimization problems through an intermediate representation separating variable semantics, relation structure, and numerical binding. A task-adapted masked diffusion model generates shared declarations, reconstructs objective and constraint branches, binds parameter slots, and repairs flagged regions by remasking. A deterministic compiler produces solver code. Four benchmarks evaluate objective-value accuracy and generation time.

Critical strengths
  1. The shared variable records, separated relation skeletons, and explicit coefficient slots align the generation procedure with meaningful dependencies in optimization formulations. They provide a coherent basis for parallel generation and localized reconstruction.

  2. The same-data, same-IR comparison reports an approximately 5.3-point mean accuracy gain and substantial generation-time reductions. The cumulative ablation also helps distinguish gains from adaptive decoding, shared semantics, and numerical binding.

Critical weaknesses
  1. The inference protocol needs to separate decoder gains from workflow gains. Matching data, IR, repair prompts, and hardware is valuable. However, Section 4.3 specifies OptDiff's precision and denoising budgets without an equally complete account of the autoregressive engine, caching, output budget, stopping rules, or branch execution. It is unclear whether the AR comparator also uses the same parallel objective/constraint branches, separate binding, and localized editing procedure. If it serializes the whole IR while OptDiff generates several short responses, part of the speedup may come from the workflow. That would still be useful, but should be attributed accordingly. Equation (4) motivates reporting the existing runs' dependent update counts, output lengths, and stage timings. The repair loop also needs an iteration limit and a clear account of how failed cases and all their generation calls enter the timing metric.

  2. The deterministic components perform substantial modeling work, but their scope is underspecified. Sections 3.1–3.5 rely on source-quantity extraction and conversions, indexed declarations, source-clause checks, and deterministic compilation. Appendix A gives a scalar example and prompts, but not a complete grammar or an operational specification for indexed formulations and compound numerical expressions. Checking inequality direction against a natural-language clause, or recovering bounds and proportion relations, involves semantic decisions beyond syntax validation. The repair boundary also matters: binding coefficients into a fixed skeleton cannot always recover a missing variable or a wrongly generated relation. Please specify which diagnoses permit rebuilding earlier stages and which cause rejection. Exact training-set counts and splits, stage training schedules, and adapter settings are also needed to reproduce the learned and deterministic parts together.

Questions for authors
  1. Does the AR comparator use the same branch parallelism, binding, and local-edit workflow? What do the existing stage timings, forward-pass counts, and repair limits show, including failed cases?

  2. Can the full IR, source-extraction and verification rules, and recovery behavior for variable or skeleton errors be specified together with the actual training and adapter configurations?

2 comments
AuthorsOpenReview ↗

Response to Reviewer t3Sq — Part 1/2

CW1 / Q1: "Does the AR comparator use the same branch parallelism, binding, and local-edit workflow?"

Matched workflow between AR and OptDiff.
Thank you for highlighting this ambiguity. The AR comparator, Qwen3.5-9B + SFT, matches OptDiff's branch parallelism, separate numerical binding, and local-repair scope. Both batch objective/constraint branches for generation and binding, sharing task-training data, IR, repair prompts, hardware, and objective-form/variable-domain modules (timed in Gen.). Local repair preserves unaffected content: Qwen generates replacement content for the selected field; OptDiff remasks and reconstructs it.

AR baseline specification.
We run Qwen in BF16 using Hugging Face Transformers, with KV caching enabled, greedy decoding, and thinking disabled. Branches stop independently at EOS or their output limit: 192 tokens for variables, 448 for constraints, 128 for the objective, and 24 for source-reference completion; binding limits depend on slot count and parameter-identifier length.

Within the same iLLaDA backbone, adaptive denoising reduces NL4Opt generation time from 15.55 s to 4.51 s, with accuracy changing from 69.5% to 69.0% (manuscript Table 2). Cumulative ablations assess the effects of shared semantics, branch decoupling, numerical binding, and local repair, complementing the matched-workflow AR comparison.

We will clarify the matched workflow in Sec. 4.3 and document the controlled comparison's inference settings in the appendix.

CW1 / Q1: "What do the existing stage timings, forward-pass counts, and repair limits show, including failed cases?"

Run accounting and repair limits.
Thank you for this suggestion. We have conducted additional timing experiments (Tables 1.1 and 1.2), covering all generation calls and failed or timed-out instances, with local repair capped at four rounds. Repair stops earlier when checks, compilation, and solving succeed, or no acceptable modification remains.

Stage-wise generation statistics.
For Eq. (4), Table 1.1 separates initial variable, relation, and binding generation from repair-related generation, and reports cumulative tokens and forward passes across all four benchmarks. Forward passes decrease from 136.56 to 24.47 on NL4Opt and from 158.16 to 25.82 on MAMO-EasyLP, showing fewer sequential model updates. We include repair in the forward-pass count. When both branches run in one batch, each forward pass counts once and corresponds to one dependent update under our schedule. We time the parallel branches together.

Table 1.1. Fine-grained generation statistics.

MetricNL4Opt (N=213)MAMO-EasyLP (N=545)MAMO-ComplexLP (N=111)NLP4LP (N=178)
Qwen3.5-9B + SFTOptDiff (8B)Qwen3.5-9B + SFTOptDiff (8B)Qwen3.5-9B + SFTOptDiff (8B)Qwen3.5-9B + SFTOptDiff (8B)
Gen. (s)5.512.816.593.0753.719.005.422.79
Variable gen. (s)1.690.901.971.042.251.221.670.88
Relation gen. (s)1.370.701.880.8319.905.191.210.71
Numerical binding (s)0.760.731.200.621.721.290.900.70
Repair (s)1.690.481.540.5829.841.301.640.50
Generated tokens153.70105.91175.68117.271536.81323.17156.70104.20
Forward passes136.5624.47158.1625.821034.2528.26131.0124.11

End-to-end latency.
Table 1.2 measures end-to-end latency directly through completion or termination. OptDiff retains 1.95× and 5.94× speedups over Qwen3.5-9B + SFT on NL4Opt and MAMO-ComplexLP, respectively. Repeated operations enter their respective stages without double counting. Process Overhead covers process management, library loading, file/result handling, and coordination between stages. We have added the end-to-end comparison and stage-wise timing breakdown to Appendix B; we will also add Table 1.1 and the stopping and counting rules above.

Table 1.2. End-to-end latency and stage-wise breakdown. All times are in seconds.

NL4Opt (N=213)

MethodGen.Extract & CheckCompile & BuildSolveProcess OverheadEnd-to-end
GPT-5.6-Sol (Medium)28.8210.0010.0240.0050.02528.876
StepORLM (8B)21.5410.0020.0360.0050.02821.612
SIRL-Qwen2.5 (7B)13.1750.0010.0460.0060.03313.261
OptMATH-Qwen2.5 (7B)29.2330.0010.0410.0040.03229.311
Qwen3.5-9B + SFT5.5080.0020.0050.0060.0305.551
OptDiff (8B)2.8060.0030.0060.0050.0322.852

MAMO-ComplexLP (N=111)

MethodGen.Extract & CheckCompile & BuildSolveProcess OverheadEnd-to-end
GPT-5.6-Sol (Medium)72.7440.0020.0250.0040.03472.809
StepORLM (8B)30.3150.0080.0340.0050.03330.395
SIRL-Qwen2.5 (7B)29.6200.0020.0450.0050.03529.707
OptMATH-Qwen2.5 (7B)26.8810.0020.0410.0030.03226.959
Qwen3.5-9B + SFT53.7090.0060.0080.0030.03453.760
OptDiff (8B)8.9980.0040.0080.0050.0359.050
AuthorsOpenReview ↗

Response to Reviewer t3Sq — Part 2/2

CW2 / Q2: "Can the full IR, source-extraction and verification rules, and recovery behavior for variable or skeleton errors be specified together with the actual training and adapter configurations?"

Learned and deterministic components.
Thank you for requesting a clearer operational specification. Learned modules generate variable declarations, relations, and numerical bindings. Deterministic modules process source information, check the IR, and compile it into solver code. Structural repair targets the affected declarations or relations. Source-based rules cover quantity extraction/conversion, explicit bounds, and proportion relations; variable-domain decisions combine adapter predictions with source checks.

Structural repair and IR specification.
Within the four-round repair limit, we locally correct fields, indices, or domains in existing declarations. For identifiable relation errors, we repair affected statements, update their bindings, and reassemble, compile, and solve, preserving unaffected records. Cases without an acceptable correction, including missing-variable diagnoses, terminate as failures. To make this behavior reproducible, we will supplement Appendix A with a compact specification of record syntax, index scopes, operators, and source-extraction, conversion, and verification rules, including applicability conditions and the handling of inequality direction, bounds, and proportions. An error-to-action summary will identify what is edited, rebound, rechecked, or rejected. We have added a complete indexed tour example to Appendix A.4, including its IR, compiled constraints, and verified optimum (158).

CW2 / Q2: "Exact training-set counts and splits, stage training schedules, and adapter settings are also needed to reproduce the learned and deterministic parts together."

Training data and adapters.
Thank you for requesting these details. Task SFT uses 1,482 training problems; the relation, binding, and objective-form LoRA adapters are then trained independently from the task-adapted model. The sources are 498 NL4Opt problems, 122 MAMO problems (93 EasyLP and 29 ComplexLP), 569 OR-Instruct problems, and 293 public OptMATH problems. Three stage-specific records per problem yield 4,446 records; adding 2,269 auxiliary numerical-model records gives 6,715 SFT records. Task SFT uses LoRA rank 64 and alpha 128; the four specialized adapters use rank 8 and alpha 16.

Masking objectives and reproducibility.
Task SFT combines semantic-row, full-response, and random masking according to annotation availability. Specialized objectives mask relation responses, binding slots, or objective-form/variable-domain codes, preserving fixed syntax during binding. We will provide a compact training-configuration listing in the appendix, covering source counts and train/validation splits, stage initialization and training schedules, optimizer settings, learning rates, batch sizes, steps, seeds, model/tokenizer versions, and checkpoint selection, with shared and model-specific settings identified for Qwen3.5-9B + SFT.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.