- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
OR-ARK constructs optimization-modeling training data from academic literature. It converts approximately 60,000 operations research papers into Markdown and uses DeepSeek to extract problem descriptions, mathematical derivations, optimization formulations, and executable solver code. The resulting structured dataset provides consistent training examples, with experimental results demonstrating its effectiveness in improving optimization-modeling capabilities.
Critical strengths
1.The data source is a real gap. Published OR papers contain formulations that seed-based synthesis does not, and the pipeline is automatic.
Critical weaknesses
-
Potential data contamination. The paper does not report decontamination against IndustryOR, OptMATH, NL4OPT, or MAMO. This is particularly concerning for IndustryOR, a key evaluation benchmark containing only 100 problems.
-
Limited evidence for effective self-reflection. ERC introduces scripted errors and corrections that ultimately recover the original answers. However, the paper does not demonstrate that the trained model can autonomously detect and correct errors at inference time, or outperform an iterative solver-feedback baseline using the same model.
-
Confounded reflection ablation. The reflection ablation modifies both the SFT data and RL training, making their individual contributions unclear. Moreover, the resulting RL model achieves only 36.0% on IndustryOR, underperforming the main SFT model (40.0%).
-
Insufficient experimental rigor. Table 1 relies on previously reported baselines with different model backbones and incomplete agent comparisons. The proposed 8B model also underperforms SIRL-7B on NL4OPT (86.9% vs. 96.3%) and MAMO-Easy (86.2% vs. 91.7%). No run-to-run variance is reported. Additionally, MIPLIB-NL is discussed but not evaluated, while the LLM-judged reflection reward and dynamic reward weights lack dedicated ablations.
-
Presentation quality needs improvement. Several parts of the paper would benefit from clearer explanations and more informative figures. For instance, the caption of Figure 4 is overly brief and does not adequately explain the figure's key findings.
Questions for authors
See Weakness
7 comments
Response: Q2. Evidence of Autonomous Self-Reflection and Error Correction
Thank you for this important observation. To demonstrate autonomous self-correction at inference time, we provide a representative example in which the trained model identifies an intermediate reasoning error, reflects on its cause, and corrects its solution within a single generation process, without external solver feedback.
The complete inference trajectory is available in our anonymous repository as ERC_example.json for independent inspection. This provides qualitative evidence of autonomous error detection and correction, beyond merely reproducing scripted ERC training trajectories.
Response: Q3. Clarification of the Reflection Ablation
1. Regarding the Confounded Reflection Ablation
We would like to clarify that Table 3 already compares RL models trained with and without the reflection mechanism. Both configurations undergo SFT and RL training, and the performance gap directly reflects the overall contribution of the proposed reflection framework, rather than differences arising from whether RL is applied.
Specifically, the reflection mechanism is designed to operate across both training stages. ERC supervision during SFT introduces error–reflection–correction trajectories, while the reflection reward during RL reinforces and improves reflective reasoning. Therefore, removing reflection-related components from both stages is necessary to evaluate the effectiveness of the complete reflection framework.
To further distinguish their contributions, we conducted additional token-matched and reflection-reward ablations. The updated results are summarized below.
| Model | MIPLIB-NL | IndustryOR | OptMATH | NL4OPT | MAMO-Complex | MAMO-Easy |
|---|---|---|---|---|---|---|
| Ours-RL-8B w/o reflection | 15.7% | 36.0% | 26.5% | 74.7% | 45.5% | 81.3% |
| Ours-RL-8B w/o reflection (token-matched) | 16.1% | 37.0% | 25.4% | 76.3% | 47.9% | 80.5% |
| Ours-RL-8B w/o reflection reward | 21.1% | 45.0% | 29.0% | 83.7% | 56.9% | 85.7% |
| Ours-RL-8B | 24.7% | 49.0% | 31.6% | 86.9% | 60.1% | 86.2% |
The results show that the complete reflection framework consistently outperforms the non-reflection RL baseline. Moreover, retaining ERC supervision while removing the reflection reward still yields substantial improvements, while adding the reflection reward provides further gains across all six benchmarks. These findings support the complementary contributions of ERC supervision and reflection rewards.
2. Regarding the Lower Performance of Non-Reflection RL
The non-reflection RL model achieves 36.0% on IndustryOR, compared with 40.0% for Ours-SFT-8B. As discussed in our original Limitations section, academic-derived training data may contain mathematical modeling errors and supervision noise due to the complexity of extracting optimization formulations from academic publications.
Without explicit reflection and correction mechanisms, RL training may reinforce imperfect modeling patterns, limiting its effectiveness. In contrast, incorporating the complete reflection framework increases IndustryOR accuracy to 49.0%, exceeding both the non-reflection RL model and Ours-SFT-8B.
This result highlights the importance of reflection mechanisms for effectively utilizing complex and potentially noisy academic optimization knowledge. Improving the mathematical accuracy of academic-data extraction remains an important direction for future work.
Response: Q1. Data Contamination and Benchmark Decontamination
Thank you for raising this important concern. To assess potential contamination, particularly on the 100-instance IndustryOR benchmark, we conducted exact-match and semantic similarity checks between our self-constructed training data and all six evaluation benchmarks, followed by manual verification.
1. Training Data Provenance. Our training data consists of two sources: our self-constructed OR dataset and publicly available data from SIRL. For the SIRL-derived portion, we preserve the original train–test separation established in SIRL. For our constructed dataset, we conducted additional decontamination checks against IndustryOR, OptMATH, NL4OPT, MAMO, and MIPLIB-NL.
2. Exact-Match Detection. We extracted problem descriptions, removed shared instruction templates, normalized text formatting, and applied SHA-256 hashing. No exact duplicates were found between our constructed training data and any evaluation benchmark.
3. Semantic Similarity Detection. We used BGE-M3 embeddings to retrieve the top-20 most similar training instances for each test problem, flagging pairs with cosine similarity above 0.80. The results are summarized below:
Notably, IndustryOR exhibits zero exact matches and zero semantic overlap candidates, directly addressing the reviewer's concern about this benchmark.
4. Manual Verification. We manually examined all 13 candidate pairs, comparing their problem descriptions, mathematical formulations, and numerical parameters. No identical numerical instances or direct paraphrases were identified. The similarities primarily reflected shared optimization problem categories rather than instance-level duplication. All candidate pairs are available in
candidate_pairs.csvin our anonymous repository for independent verification.5. Train–Test Independence. For SIRL-derived data, we follow the original train–test partition. For our self-constructed data, the decontamination analyses provide additional empirical evidence of instance-level independence. While distinct optimization problems may share mathematical structures, such similarities do not inherently indicate data leakage. Together, these findings support the reliability of our evaluation and the independence of our training and test instances.