- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper presents MLLM4OR, a 9B multimodal large language model designed to translate complementary text-image optimization specifications into mathematical formulations and executable solver code. The authors construct MLLM4OR-Corpus, a 22,107-problem multimodal optimization corpus, and investigate several supervised fine-tuning strategies, including Direct-SFT, Compact-SFT, Detailed-SFT, and a Mixture-SFT strategy.
The paper further proposes SG-DPO, a solver-grounded direct preference optimization procedure. The central idea is to construct preference pairs using model-generated candidate solutions that are audited through execution and solver-based checks, with verified reference solutions serving as positive responses. The final preference dataset contains 498 preference pairs from 315 training problems.
The resulting model achieves strong performance on the proposed MLLM4OR-Test benchmark and demonstrates meaningful cross-dataset generalization on LLM4OR-VL and MM-OptBench. In particular, Mixture-SFT reaches 91.0% on MLLM4OR-Test, while SG-DPO obtains 49.5% on LLM4OR-VL and 44.3% on MM-OptBench, outperforming the corresponding Mixture-SFT models on the two external benchmarks. The paper also reports Pass@4 results and comparisons with several strong general-purpose multimodal models.
Overall, the paper addresses an interesting and practically relevant problem at the intersection of multimodal reasoning, mathematical optimization, structured generation, and solver-based verification.
Critical strengths
- Substantial task-specific multimodal corpus and end-to-end task formulation.
A major strength is the construction of a relatively large 22,107-problem corpus for a difficult and underexplored task: translating complementary text-image optimization specifications into mathematical formulations and executable solver code. The work goes beyond generic visual question answering by requiring structured mathematical reasoning and executable outputs.
The combination of multimodal inputs, mathematical formulation, and solver execution creates a practically meaningful benchmark for studying reliable multimodal structured generation.
- Strong empirical performance and evidence of cross-dataset generalization.
The paper provides a broad empirical evaluation rather than relying only on the proposed benchmark. Mixture-SFT substantially improves performance over the baseline on MLLM4OR-Test, and SG-DPO provides additional gains on LLM4OR-VL and MM-OptBench. The external benchmark results are especially valuable because they suggest that the learned capabilities are not completely confined to the training distribution.
The use of solver-based verification and Pass@4 analysis also provides a more informative evaluation protocol than simple language-generation metrics.
Critical weaknesses
- The methodological novelty of SG-DPO is not yet sufficiently isolated.
The core DPO objective is standard, and the main methodological contribution is the solver-grounded construction of preference pairs. However, the current experiments do not establish that solver-grounded preference construction itself is responsible for the observed gains.
For example, it would be important to compare SG-DPO against equally sized preference datasets constructed using: (a) random model-generated negatives, (b) hard negatives without solver verification, and (c) rejection-sampled supervised fine-tuning or another supervised approach using the same verified/generated candidates. Without such controls, the improvement could plausibly result from additional targeted training or hard-negative exposure rather than the specific DPO formulation.
This is the main issue affecting my assessment of the paper's methodological novelty.
- The multimodal contribution and semantic fidelity are not sufficiently isolated.
Although the task is multimodal, the paper does not appear to provide direct controlled evidence quantifying how much the image contributes to the final solution. Text-only, image-only, and image-shuffled/corrupted-image experiments would substantially strengthen the claim that the model learns genuine multimodal grounding.
Moreover, successful solver execution does not by itself guarantee semantic fidelity. A formulation may be syntactically valid and solvable while encoding an incorrect parameter, variable domain, objective function, or individual constraint. A more granular evaluation of these semantic components would therefore provide stronger evidence for the claimed text-image-to-optimization capability.
The source composition of the evaluation set is another limitation: approximately 85.6% of MLLM4OR-Test is derived from OptMATH, so the headline aggregate score is heavily influenced by this source. In addition, the external LLM4OR-VL benchmark is reconstructed from text-only optimization sources, which makes it useful for cross-source generalization but not a direct test of naturally occurring multimodal optimization problems.
Questions for authors
- Can you isolate the specific contribution of solver-grounded preference construction in SG-DPO?
In particular, could you report controlled comparisons using the same number of preference pairs and the same initialization, with negatives constructed by:
random model-generated responses; hard model-generated responses without solver-based verification; solver-verified hard negatives but trained with a supervised/rejection-sampling objective instead of DPO?
This would clarify whether the gains come specifically from solver-grounded preference optimization or primarily from additional targeted training on difficult examples.
- How much of the final performance actually depends on the visual input?
Could you provide text-only, image-only, and image-shuffled/corrupted-image ablations under the same training and evaluation protocol? This would help establish whether the model is genuinely using visual information to resolve optimization specifications, rather than relying predominantly on the textual description.
- What exactly is checked by the solver-based correctness evaluation?
Could you clarify whether correctness is evaluated at the level of individual numerical values, variable names and domains, objective expressions, and individual constraints, or whether the primary criterion is successful execution/solver completion? If the latter, can you provide a semantic-fidelity analysis or component-level error breakdown?
- How robust are the reported gains to the source imbalance in MLLM4OR-Test?
Since approximately 85.6% of the test set is derived from OptMATH, could you report source-balanced or per-source aggregate metrics, such as macro-average accuracy across OptMATH-derived and OR-Instruct-derived subsets? This would help determine whether the overall 91.0% score reflects broad generalization or is primarily driven by the dominant source.
Author- Reviewer Discussion
Dear Reviewers, the authors have submitted their response. Kindly review it and update your review where appropriate.