- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The authors propose StarOR, a framework that integrates Monte Carlo Tree Search (MCTS) with test-time reinforcement learning for optimization modeling. StarOR decomposes the modeling process into four stages and updates an instance-specific, transient LoRA adapter using GRPO after each non-terminal search expansion. Sibling candidates generated during search serve as local comparison groups for policy optimization, allowing search feedback to guide subsequent generation. The framework also introduces an unsupervised, multi-component reward that evaluates intermediate modeling decisions through executable completions without requiring ground-truth labels. Experiments on five optimization-modeling benchmarks demonstrate improved accuracy over the evaluated same-backbone test-time baselines.
Critical strengths
- Well-motivated integration of structured search and test-time adaptation. StarOR uses sibling candidates from the same MCTS node as comparison groups for GRPO, allowing search-generated feedback to update the policy that produces subsequent candidates. The staged formulation also reflects dependencies among modeling decisions. This provides a coherent mechanism for instance-specific adaptation beyond fixed-policy search and constitutes a meaningful methodological contribution to automated optimization modeling.
- Consistent empirical gains with informative ablations. Across five benchmarks, StarOR achieves a macro-average accuracy of 65.0%, outperforming the evaluated same-backbone test-time baselines. Removing LoRA adaptation reduces this average to 60.8%, while replacing the full reward with a majority-voting reward also yields 60.8%. These results support the practical value of combining online adaptation with task-specific feedback, strengthening the empirical case for the proposed framework.
Critical weaknesses
The reward design does not sufficiently distinguish consensus from formulation correctness. Objective-value agreement and structural similarity can reward candidates that share the same modeling error, while the objective-scale priors and perturbation-test expectations are generated by the backbone itself. Consequently, these signals may reinforce correlated mistakes rather than identify faithful formulations. The paper does not sufficiently quantify how reliably its rewards distinguish correct models from executable but semantically incorrect ones. This is consequential because both search and policy adaptation optimize these rewards, making their reliability central to the claimed improvement in modeling fidelity.
Questions for authors
What evidence indicates that the proposed rewards distinguish faithful formulations from executable but semantically incorrect models? In particular, have you examined cases where multiple candidates share an incorrect objective value or omit the same constraint?
3 comments
(4) Mechanism: How do the signals complement one another?
1. Beyond executability. Execution feedback checks whether the program runs successfully. Perturbation tests assess its behavior under input changes, while structural feedback compares variable types, constraint counts, and objective direction. Together, they help distinguish executable but incorrect formulations.
2. Beyond consensus. Scale priors assess whether objective values are plausible, and tests assess behavior under perturbations. Candidates sharing an objective value can therefore receive different total rewards. Soft weighting (λ = 0.5) down-weights out-of-scale values without excluding candidates, preserving the influence of other signals.
3. Continued exploration. GRPO learns from reward differences within same-prefix, same-stage groups. PUCT, historical-branch revisits, and one-shot cluster suppression preserve alternative paths when current feedback is insufficient.
Table 4 supports this complementarity: the full reward improves macro-average accuracy over majority-voting reward from 60.8% to 65.0%, with gains of 5.0 and 6.0 percentage points on IndustryOR and OptMATH, respectively.
Conclusion: Complementary feedback distinguishes candidates beyond consensus, while continued exploration preserves opportunities for correction.
Thank you for your response. I have raised my score.
Can the reward distinguish correct formulations from shared errors? How reliable are priors generated by the same model?
Thank you for raising the question of reward reliability. StarOR evaluates formulation quality by combining candidate agreement, execution and test feedback, and scale and structural information. We address this concern through empirical audits, controlled constructions, prior comparisons, and an explanation of signal complementarity.
We drew a fixed random sample of 20 problems from IndustryOR and collected 119 candidate groups comprising 952 nodes. Among these, 15 groups comprising 120 nodes were manually audited for the semantics of their current-stage formulations. The logs were used to characterize incorrect consensus, while manual semantic labels and total rewards were used to analyze the reward ranking of correct and incorrect nodes.
Conclusion:
In the empirical audit, semantically correct nodes receive the highest reward in 81.0% of cases, whereas the highest reward is assigned to the largest incorrect cluster in 13.3% of cases, supporting the ability of the full reward to distinguish modeling semantics.Setup. On IndustryOR, we constructed four error types: deleting one constraint, reversing one inequality, deleting two constraints, and combining deletion with reversal. Each type uses 5 distinct problems, yielding 20 problem–construction instances per setting across 11 unique problems, selected without reward-based filtering.
A-in uses the correct prior. A-out shifts only the original-problem prior to exclude all candidate objective values, keeping candidates and tests unchanged. B–D vary the candidate composition and whether incorrect candidates share an objective value, as shown below.
Conclusion:
In the controlled constructions with correct test intervals, correct candidates receive higher total rewards in all 20 instances of every setting, even when incorrect candidates form a majority consensus or the prior for the original problem is miscalibrated.1. Prior prediction quality. Priors are generated from the problem text and numerical information; reference answers are used for post hoc coverage evaluation. OptMATH contains 166 problems, using the existing Easy/Hard split of 65/101 problems:
The comparison between the weaker and stronger models evaluates the prediction quality of prior intervals. The three experiments below further examine how scale weighting affects final modeling accuracy.
2. Effect of the prior on final performance.
Conclusion:
The default prior yields a 5.02-percentage-point accuracy gain, while performance under substantial prior perturbation remains close to that with scale weighting disabled, **supporting both the effectiveness of the objective-scale prior and its robustness to prediction errors**.