acceptodds
Under review as a conference paper at ICLR 2027

StarOR: Synergizing Tree Search and Test-Time Reinforcement Learning for Optimization Modeling

Abstract

Optimization modeling is inherently hierarchical, requiring a precise sequence of symbolic commitments. Traditional learning-based automated optimization modeling methods improve modeling policies through large-scale annotated or curated training data, but are costly to adapt to new problem distributions. Meanwhile, one-shot generation remains brittle in hierarchical modeling, where early symbolic errors can propagate into invalid formulations. Test-time scaling offers a promising alternative by enabling structural exploration with additional instance-level computation; however, existing search-based methods typically rely on a fixed policy, causing repeated rollouts to inherit similar modeling biases and providing limited credit assignment for intermediate decisions. To address these limitations, we propose StarOR, a synergistic search-and-adaptation framework that couples MCTS with Test-Time Reinforcement Learning for optimization modeling. StarOR decomposes the modeling process into four stages and updates a transient LoRA adapter via GRPO at each non-terminal node. By using MCTS-generated siblings as local comparison sets, StarOR transforms search-time exploration into instance-specific policy refinement. Moreover, an unsupervised multi-faceted reward system provides fine-grained feedback for intermediate formulation decisions without ground-truth labels. Across five optimization benchmarks, StarOR attains a 65.0% average accuracy with a 4B backbone, matching the highest average and outperforming the evaluated same-backbone test-time baselines.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM Reasoning, Optimization Modeling, Test-Time Learning

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_qfEMOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors propose StarOR, a framework that integrates Monte Carlo Tree Search (MCTS) with test-time reinforcement learning for optimization modeling. StarOR decomposes the modeling process into four stages and updates an instance-specific, transient LoRA adapter using GRPO after each non-terminal search expansion. Sibling candidates generated during search serve as local comparison groups for policy optimization, allowing search feedback to guide subsequent generation. The framework also introduces an unsupervised, multi-component reward that evaluates intermediate modeling decisions through executable completions without requiring ground-truth labels. Experiments on five optimization-modeling benchmarks demonstrate improved accuracy over the evaluated same-backbone test-time baselines.

Critical strengths
  1. Well-motivated integration of structured search and test-time adaptation. StarOR uses sibling candidates from the same MCTS node as comparison groups for GRPO, allowing search-generated feedback to update the policy that produces subsequent candidates. The staged formulation also reflects dependencies among modeling decisions. This provides a coherent mechanism for instance-specific adaptation beyond fixed-policy search and constitutes a meaningful methodological contribution to automated optimization modeling.
  2. Consistent empirical gains with informative ablations. Across five benchmarks, StarOR achieves a macro-average accuracy of 65.0%, outperforming the evaluated same-backbone test-time baselines. Removing LoRA adaptation reduces this average to 60.8%, while replacing the full reward with a majority-voting reward also yields 60.8%. These results support the practical value of combining online adaptation with task-specific feedback, strengthening the empirical case for the proposed framework.
Critical weaknesses

The reward design does not sufficiently distinguish consensus from formulation correctness. Objective-value agreement and structural similarity can reward candidates that share the same modeling error, while the objective-scale priors and perturbation-test expectations are generated by the backbone itself. Consequently, these signals may reinforce correlated mistakes rather than identify faithful formulations. The paper does not sufficiently quantify how reliably its rewards distinguish correct models from executable but semantically incorrect ones. This is consequential because both search and policy adaptation optimize these rewards, making their reliability central to the claimed improvement in modeling fidelity.

Questions for authors

What evidence indicates that the proposed rewards distinguish faithful formulations from executable but semantically incorrect models? In particular, have you examined cases where multiple candidates share an incorrect objective value or omit the same constraint?

3 comments
AuthorsOpenReview ↗

Can the reward distinguish correct formulations from shared errors? How reliable are priors generated by the same model?

Thank you for raising the question of reward reliability. StarOR evaluates formulation quality by combining candidate agreement, execution and test feedback, and scale and structural information. We address this concern through empirical audits, controlled constructions, prior comparisons, and an explanation of signal complementarity.


(1) Empirical audit: Can the reward distinguish semantically correct from incorrect modeling nodes?

We drew a fixed random sample of 20 problems from IndustryOR and collected 119 candidate groups comprising 952 nodes. Among these, 15 groups comprising 120 nodes were manually audited for the semantics of their current-stage formulations. The logs were used to characterize incorrect consensus, while manual semantic labels and total rewards were used to analyze the reward ranking of correct and incorrect nodes.

Audit metricResult
Rate at which semantically correct nodes receive the highest reward81.0%
Rate at which an incorrect objective-value cluster forms a strict majority40.0%
Rate at which the highest reward is assigned to the largest incorrect objective-value cluster13.3%

Conclusion: In the empirical audit, semantically correct nodes receive the highest reward in 81.0% of cases, whereas the highest reward is assigned to the largest incorrect cluster in 13.3% of cases, supporting the ability of the full reward to distinguish modeling semantics.


(2) Controlled constructions: How does reward change after deleting or reversing?

Setup. On IndustryOR, we constructed four error types: deleting one constraint, reversing one inequality, deleting two constraints, and combining deletion with reversal. Each type uses 5 distinct problems, yielding 20 problem–construction instances per setting across 11 unique problems, selected without reward-based filtering.

A-in uses the correct prior. A-out shifts only the original-problem prior to exclude all candidate objective values, keeping candidates and tests unchanged. B–D vary the candidate composition and whether incorrect candidates share an objective value, as shown below.

SettingCandidates (correct / incorrect / non-executable)Comparison conditionMean R, correctMean R, incorrectMean R, non-executableRate of correct > incorrect > non-executable
A-in2/1/1Correct objective value within the prior interval0.71280.44010.0000100% (20/20)
A-out2/1/1Correct objective value outside the prior interval0.55030.44010.0000100% (20/20)
B2/2/0Two incorrect candidates share the same objective value0.65730.4779—100% (20/20)*
C1/2/1Two incorrect candidates share the same objective value0.58510.50530.0000100% (20/20)
D1/2/1Two incorrect candidates have different objective values0.58510.44240.0000100% (20/20)

Conclusion: In the controlled constructions with correct test intervals, correct candidates receive higher total rewards in all 20 instances of every setting, even when incorrect candidates form a majority consensus or the prior for the original problem is miscalibrated.


(3) Prior reliability: Prediction quality and prior interventions.

1. Prior prediction quality. Priors are generated from the problem text and numerical information; reference answers are used for post hoc coverage evaluation. OptMATH contains 166 problems, using the existing Easy/Hard split of 65/101 problems:

Prior-generating modelEasy coverageHard coverageOverall coverage
Qwen3-4B-Instruct57/65 (87.7%)60/101 (59.4%)117/166 (70.5%)
DeepSeek-V4-Pro63/65 (96.9%)75/101 (74.3%)138/166 (83.1%)

The comparison between the weaker and stronger models evaluates the prediction quality of prior intervals. The three experiments below further examine how scale weighting affects final modeling accuracy.

2. Effect of the prior on final performance.

Prior settingEasy Acc (%)Hard Acc (%)Overall Acc (%)
Default prior: Qwen3-4B-Instruct44.62 ± 1.5426.40 ± 2.8633.53 ± 1.25
Deliberately perturbed: center shifted by ±30%, interval width halved36.41 ± 0.8923.76 ± 2.9728.71 ± 1.52
Objective-scale weighting disabled40.00 ± 2.6621.12 ± 2.8628.51 ± 0.70

Conclusion: The default prior yields a 5.02-percentage-point accuracy gain, while performance under substantial prior perturbation remains close to that with scale weighting disabled, **supporting both the effectiveness of the objective-scale prior and its robustness to prediction errors**.


(4) Mechanism: How do the signals complement one another?

1. Beyond executability. Execution feedback checks whether the program runs successfully. Perturbation tests assess its behavior under input changes, while structural feedback compares variable types, constraint counts, and objective direction. Together, they help distinguish executable but incorrect formulations.

2. Beyond consensus. Scale priors assess whether objective values are plausible, and tests assess behavior under perturbations. Candidates sharing an objective value can therefore receive different total rewards. Soft weighting (λ = 0.5) down-weights out-of-scale values without excluding candidates, preserving the influence of other signals.

3. Continued exploration. GRPO learns from reward differences within same-prefix, same-stage groups. PUCT, historical-branch revisits, and one-shot cluster suppression preserve alternative paths when current feedback is insufficient.

Table 4 supports this complementarity: the full reward improves macro-average accuracy over majority-voting reward from 60.8% to 65.0%, with gains of 5.0 and 6.0 percentage points on IndustryOR and OptMATH, respectively.

Conclusion: Complementary feedback distinguishes candidates beyond consensus, while continued exploration preserves opportunities for correction.

Thank you for your response. I have raised my score.

Reviewer_68wVOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents StarOR, which combines structured search with test-time reinforcement learning for optimization modeling. Partial formulations are completed into executable code and evaluated, with the feedback used to guide MCTS and update a LoRA adapter for the current problem. Experiments on five benchmarks with a 4B model show improvements over search without adaptation and other test-time baselines.

Critical strengths
  1. I see this as a sensible next step for LLM-based optimization modeling. It connects the motivation behind process-supervised methods such as StepORLM with test-time adaptation, allowing feedback on partial formulations to improve subsequent modeling decisions.

  2. Adaptation improves performance on all five benchmarks, raising macro-average accuracy from 60.8% to 65.0%. The reward and budget ablations provide useful support for the overall design.

Critical weaknesses
  1. Table 11 leaves the one-stage baseline’s GRPO updates and stopping behavior unclear. The reward ablations also show performance gains without establishing whether the rewards distinguish correct formulations from executable but incorrectly formulated ones.

  2. The paper’s scope remains tied to MCTS. Exploring the adaptation mechanism with other search structures, such as beam search in BPP-Search or evolutionary search in FormuEvo, would help establish its broader value for optimization modeling.

Questions for authors
  1. Can the rewards distinguish executable but incorrectly formulated models from correct ones? Please test this by deleting constraints or reversing inequalities in a small set of correct models, and report whether the incorrect variants receive lower rewards.

  2. Does the one-stage variant in Table 11 retain multiple GRPO updates per instance? Please clarify its update schedule, reward settings, and stopping rule.

Suggestion: I encourage the authors to discuss in the conclusion or future work how this approach could extend to other search structures, such as beam search or evolutionary search.

4 comments

Thank you for your careful review and constructive suggestions on reward reliability, the one-stage baseline, and extensions to other search structures.


W1 & Q1: Can the reward distinguish correct formulations from shared errors? How reliable are priors generated by the same model?

(1) Empirical audit: Can the reward distinguish semantically correct from incorrect modeling nodes?

We drew a fixed random sample of 20 problems from IndustryOR and collected 119 candidate groups comprising 952 nodes. Each group contains 8 candidates sharing the same prefix and modeling stage. Among these, 15 groups comprising 120 nodes were manually audited for the semantics of their current-stage formulations. The logs were used to characterize incorrect consensus, while manual semantic labels and total rewards were used to analyze the reward ranking of correct and incorrect nodes.

Audit metricResult
Rate at which semantically correct nodes receive the highest reward81.0%
Rate at which an incorrect objective-value cluster forms a strict majority40.0%
Rate at which the highest reward is assigned to the largest incorrect objective-value cluster13.3%

Conclusion: In the empirical audit, semantically correct nodes receive the highest reward in 81.0% of cases, whereas the highest reward is assigned to the largest incorrect cluster in 13.3% of cases, supporting the ability of the full reward to distinguish modeling semantics.


(2) Controlled constructions: How does reward change after deleting or reversing?

Setup. On IndustryOR, we constructed four error types: deleting one constraint, reversing one inequality, deleting two constraints, and combining deletion with reversal. Each type uses 5 distinct problems, yielding 20 problem–construction instances per setting across 11 unique problems, selected without reward-based filtering.

A-in uses the correct prior. A-out shifts only the original-problem prior to exclude all candidate objective values, keeping candidates and tests unchanged. B–D vary the candidate composition and whether incorrect candidates share an objective value, as shown below.

SettingCandidates (correct / incorrect / non-executable)Comparison conditionMean R, correctMean R, incorrectMean R, non-executableRate of correct > incorrect > non-executable
A-in2/1/1Correct objective value within the prior interval0.71280.44010.0000100% (20/20)
A-out2/1/1Correct objective value outside the prior interval0.55030.44010.0000100% (20/20)
B2/2/0Two incorrect candidates share the same objective value0.65730.4779—100% (20/20)*
C1/2/1Two incorrect candidates share the same objective value0.58510.50530.0000100% (20/20)
D1/2/1Two incorrect candidates have different objective values0.58510.44240.0000100% (20/20)

Conclusion: In the controlled constructions with correct test intervals, correct candidates receive higher total rewards in all 20 instances of every setting, even when incorrect candidates form a majority consensus or the prior for the original problem is miscalibrated.


(3) Prior reliability: Prediction quality and prior interventions.

1. Prior prediction quality. Priors are generated from the problem text and numerical information; reference answers are used for post hoc coverage evaluation. OptMATH contains 166 problems, using the existing Easy/Hard split of 65/101 problems:

Prior-generating modelEasy coverageHard coverageOverall coverage
Qwen3-4B-Instruct57/65 (87.7%)60/101 (59.4%)117/166 (70.5%)
DeepSeek-V4-Pro63/65 (96.9%)75/101 (74.3%)138/166 (83.1%)

The comparison between the weaker and stronger models evaluates the prediction quality of prior intervals. The three experiments below further examine how scale weighting affects final modeling accuracy.

2. Effect of the prior on final performance.

Prior settingEasy Acc (%)Hard Acc (%)Overall Acc (%)
Default prior: Qwen3-4B-Instruct44.62 ± 1.5426.40 ± 2.8633.53 ± 1.25
Deliberately perturbed: center shifted by ±30%, interval width halved36.41 ± 0.8923.76 ± 2.9728.71 ± 1.52
Objective-scale weighting disabled40.00 ± 2.6621.12 ± 2.8628.51 ± 0.70

Conclusion: The default prior yields a 5.02-percentage-point accuracy gain, while performance under substantial prior perturbation remains close to that with scale weighting disabled, **supporting both the effectiveness of the objective-scale prior and its robustness to prediction errors**.


W1 & Q2: GRPO updates, reward settings, and stopping rules in the one-stage variant


Thank you for requesting this clarification. The one-stage variant retains multiple GRPO updates within each instance. “One-stage” means that the complete formulation is generated at once; the search still contains a formulation layer and a code layer. In Table 11, this variant uses at most 4 iterations per problem, 10 parallel candidates per iteration, and a generation cap of 12K tokens per candidate.

(1) Nodes and updates.

  • Each child of the root contains a complete formulation: problem type, sets, parameters, variables, objective, and constraints.
  • Candidates are completed into code, evaluated using the same reward as StarOR, and used for a within-group GRPO update.
  • Code completion for reward computation is distinct from search-tree expansion into the code layer; formulation nodes can still expand into that layer.

(2) Subsequent expansion and stopping rules.

  • Iteration 2 is forced to expand from the root, generating complete formulations with the updated policy and performing another GRPO update. Later iterations follow StarOR’s node-selection procedure.
  • Rewards, dynamic weighting, and stopping criteria follow StarOR, including code-stage termination, consensus-based early stopping when applicable, and the maximum iteration budget.
  • Repeated root expansions remain within the same modeling stage and do not count as consecutive consensus across distinct stages.
SettingOne-stage variantFour-stage StarOR
Node granularityComplete formulation → codeThree modeling stages → code
Maximum iterations410
Candidates per iteration108
Token cap per candidate12K6K
GRPO update triggerSame criteria as StarORAfter each eligible nonterminal expansion
Content used for updatesThe complete formulation and its associated reasoningThe current-stage formulation and its associated reasoning
Stopping conditionsSame criteria as StarORCode-stage termination and consecutive consensus across distinct stages, subject to the maximum iteration budget

All other settings remain the same as in StarOR. In this comparison, the one-stage and four-stage variants achieve 31.93 ± 3.35% and 33.53 ± 1.25% accuracy, with average generation costs of 123.2K and 117.4K tokens, respectively. Both use search feedback for instance-level adaptation; the main difference is whether each update targets the complete formulation or the modeling decisions at the current stage.

Similarly, the six-stage variant divides formulation construction into 6 modeling stages, followed by 1 code layer, yielding a search structure with 7 layers (6+1).

W2 & Suggestion: Extension to other search structures


Thank you for this suggestion. StarOR’s process of generating comparable candidates, evaluating them through execution, and updating the policy using within-group differences offers opportunities for integration with other search structures. We will discuss the following two directions in the future-work section of the appendix.

(1) Beam search.

  • Candidate generation. With a beam width of B, each retained partial formulation generates K candidates for the next stage, producing B × K candidates in total.
  • Group construction and adaptation. The K candidates sharing a prefix form a GRPO comparison group. Rewards obtained from executable completions are used both to select the next beam and to compute within-group relative advantages for updating LoRA before the next generation step.
  • Search organization. Standard beam search proceeds stage by stage, providing a regular workflow that facilitates batched generation and parallel evaluation of candidates at the same depth.

(2) Evolutionary search.

  • Revision generation. One possible extension is to generate a group of revised candidates conditioned on the same parent formulation and its existing feedback.
  • Adaptation and selection. StarOR’s full reward is used to compute within-group advantages and perform a GRPO update, followed by population selection to retain the next generation. Evolutionary search thus retains candidates, while instance-level adaptation seeks to turn search feedback into improved subsequent revisions.
  • Potential complementary feedback. Drawing on OPSD (On-Policy Self-Distillation), a teacher branch could additionally receive feedback from the current round of execution and perturbation tests and provide token-level distillation signals on revision trajectories generated by the current policy, complementing GRPO’s candidate-level rewards. This is a preliminary direction offered for further discussion.

Thank you for the detailed response and additional experiments. I appreciate the authors' efforts to address my concerns. My only remaining minor suggestion is to qualify the claim regarding prior robustness, as accuracy under the perturbed prior (28.71%) drops to a level close to that of the unweighted baseline (28.51%). A more interpretation of these results would help strengthen the discussion.

Overall, the response has adequately addressed my concerns, and I am inclined to raise my score.

Thank you for pointing out that the prior-perturbation results require a more precise interpretation.


regarding prior robustness, as accuracy under the perturbed prior (28.71%) drops to a level close to that of the unweighted baseline (28.51%). A more interpretation of these results would help strengthen the discussion.

Prior coverage deteriorates substantially under perturbation, yet final accuracy remains close to the baseline with scale weighting disabled. We interpret this result through both coverage and final performance.

PriorEasy coverageHard coverageOverall coverage
Default57/65(87.7%)60/101(59.4%)117/166(70.5%)
Perturbed18/65(27.7%)27/101(26.7%)45/166(27.1%)

Perturbation reduces overall coverage from 70.5% to 27.1%, leaving most reference optima outside the prior intervals. Nevertheless, final accuracy remains at 28.71 ± 1.52%, close to 28.51 ± 0.70% with scale weighting disabled. The default prior achieves 33.53 ± 1.25%, a gain of 5.02 percentage points.

Thus, the scale prior provides additional gains when informative, while performance remains near the no-scale-weighting baseline under the tested substantial miscalibration.

Conclusion: The default prior improves modeling accuracy; even when perturbed coverage falls to 27.1%, performance remains close to disabling scale weighting, combining useful gains with tolerance to prior errors in the tested setting.

We will add this detailed explanation and interpretation to the paper.


Thank you again for your valuable and constructive feedback. We hope this further discussion addresses your concerns. We remain committed to addressing any remaining points you may have during the discussion phase!

Reviewer_uGrDOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes a framework for automated optimization modeling that couples hierarchical Monte Carlo Tree Search (MCTS) with test-time reinforcement learning. The modeling process is decomposed into four stages: (1) problem type and sets, (2) parameters and variables, (3) objective and constraints, and (4) executable code. At each non-terminal MCTS node, StarOR samples a sibling group of candidate continuations, completes them into executable Gurobi programs, computes an unsupervised multi-faceted reward, and updates a transient LoRA adapter via GRPO. The reward combines semantic consensus, executability, test-case robustness, and structural consistency, with dynamic reward shaping over search iterations. Experiments on five OR modeling benchmarks show that StarOR with a 4B backbone reaches 65.0% macro-average accuracy, matching DeepSeek-V3.1 and outperforming several same-backbone test-time baselines.

Critical strengths
  1. The combination of MCTS, transient LoRA adaptation, and node-level GRPO is described clearly. The four-stage decomposition is intuitive and aligns with the dependency structure of optimization formulations.
  2. The multi-faceted reward avoids ground-truth labels at test time and combines complementary signals: execution validity, objective-value consensus, structural signatures, and perturbation-based test cases. The dynamic weighting schedule is a reasonable attempt to balance exploration and convergence.
  3. The paper evaluates on five benchmarks with varying difficulty, compares against model-based, learning-based, and test-time baselines, and includes ablations for LoRA adaptation, reward design, stage granularity, repair budget, and scaling behavior.
Critical weaknesses
  1. The core ingredients—hierarchical MCTS, test-time adaptation with LoRA, and GRPO—are individually well established. OR-R1 already applies test-time GRPO to OR modeling, and AutoFormulator already uses MCTS for optimization formulation. The main novelty is the combination and the node-level reward granularity, which is useful but incremental. The claim of being “the first architecture to integrate hierarchical MCTS with test-time training for OR modeling” is plausible but should be qualified more carefully given closely related work.
  2. Reward correctness is not guaranteed and may reinforce consensus errors. The semantic and structural rewards primarily reward agreement among sampled candidates. If the sibling group consistently makes the same modeling mistake, StarOR may reinforce an incorrect but self-consistent formulation. The objective-scale prior and test-case rewards partially mitigate this, but they are also generated by the same backbone and can inherit its biases. The paper does not provide a systematic failure analysis of consensus-locking or reward hacking.
  3. StarOR uses additional pre-generation, test-case construction, repeated execution, repair rounds, and per-node LoRA updates. The main comparison against Best-of-N and other test-time methods is not fully compute-matched. The compute-budget-controlled experiment in Appendix C.3 is helpful, but it is limited to OptMATH and still shows StarOR has higher wall-clock latency. The main table also mixes single-seed StarOR results with multi-seed Best-of-N results in places, which weakens statistical reliability. 4.The stage-decomposition ablation and several reward ablations are only on OptMATH, and some reward components have neutral or negative effects on NL4OPT. The four-stage choice is motivated but not conclusively established as optimal. The LoRA adaptation gain is consistent but modest relative to the gain from search itself; the paper would benefit from a clearer decomposition of where the improvement comes from.
  4. Statistical significance is insufficiently reported. Many main results appear to be single-run or lack confidence intervals. Given the stochasticity of sampling, MCTS, and GRPO updates, the reported differences—especially the 65.0% vs. 64.6% comparison with DeepSeek-R1—should be supported by multiple seeds and variance estimates.
  5. Scalability and practicality. StarOR’s runtime grows substantially with expansion size, and the method requires repeated solver executions and LoRA updates per instance. The paper acknowledges this, but the practical trade-off against simpler Best-of-N or repair-based methods is not fully resolved.
Questions for authors
  1. How does StarOR behave when all sampled siblings converge to the same incorrect objective value? Can the authors provide a failure-mode analysis or a metric for “consensus but wrong” cases?
  2. In the main results, how many seeds are used for StarOR? Can the authors report mean ± standard deviation for the main table and for the LoRA ablation?
  3. How exactly is the compute budget matched between StarOR and Best-of-N/AutoFormulator in the main comparison? The appendix provides a budget-controlled experiment on OptMATH, but it is unclear whether the same control is applied to all five benchmarks.
  4. What fraction of instances rely on the repair loop, and how much of the final accuracy is attributable to repair versus search-and-adaptation?
  5. How sensitive is StarOR to the objective-scale prior? What happens when the prior is intentionally perturbed or disabled beyond the reported ablation?
  6. The four-stage decomposition is compared with one-stage and six-stage variants only on OptMATH. Would the conclusion hold on IndustryOR or MAMO-Complex?
  7. Is the transient LoRA adapter updated using only the current instance’s sibling group? How stable is the GRPO update with K=8, and are there cases where the adapter degrades subsequent search?
  8. Can the authors clarify the relationship to OR-R1 and TTT-Discover more explicitly, especially regarding the claimed novelty of node-level test-time training for OR modeling?
6 comments

I. Reward reliability and incorrect consensus (W2 & Q1 & Q5)


(1) Empirical audit: Can the reward distinguish semantically correct from incorrect modeling nodes?

We drew a fixed random sample of 20 problems from IndustryOR and collected 119 candidate groups comprising 952 nodes. Each group contains 8 candidates sharing the same prefix and modeling stage. Among these, 15 groups comprising 120 nodes were manually audited for the semantics of their current-stage formulations. The logs were used to characterize incorrect consensus, while manual semantic labels and total rewards were used to analyze the reward ranking of correct and incorrect nodes.

Audit metricResult
Rate at which semantically correct nodes receive the highest reward81.0%
Rate at which an incorrect objective-value cluster forms a strict majority40.0%
Rate at which the highest reward is assigned to the largest incorrect objective-value cluster13.3%

Conclusion: In the empirical audit, semantically correct nodes receive the highest reward in 81.0% of cases, whereas the highest reward is assigned to the largest incorrect cluster in 13.3% of cases, supporting the ability of the full reward to distinguish modeling semantics.

We further audited trajectories after a unique largest incorrect objective-value cluster appeared in the IndustryOR search logs.

Audit metricAudit result
Immediate escape at the current stage: the current group's best_rollout lies outside the largest incorrect objective-value cluster21.8%
Escape at a later stage: an objective value selected at a subsequent, deeper stage lies outside the original incorrect cluster44.4%
Escape from the original incorrect cluster at least once at the current or a later stage41.3%

If sampled siblings converge to the same incorrect objective value, GRPO would rely on feedback from those incorrect outputs; however, this situation was virtually absent in the audit. Furthermore, even when the current selection is incorrect, 41.3% of the audited records escape the original incorrect cluster, indicating robustness to incorrect consensus.

Conclusion: The search trajectories show that the largest incorrect cluster does not necessarily determine subsequent selections; cross-branch exploration and reevaluation at later stages provide actual opportunities to escape incorrect consensus.


(2) Controlled constructions

Setup. On IndustryOR, we constructed four error types: deleting one constraint, reversing one inequality, deleting two constraints, and combining deletion with reversal. Each type uses 5 distinct problems, yielding 20 problem–construction instances per setting across 11 unique problems, selected without reward-based filtering.

A-in uses the correct prior. A-out shifts only the original-problem prior to exclude all candidate objective values, keeping candidates and tests unchanged. B–D vary the candidate composition and whether incorrect candidates share an objective value, as shown below.

SettingCandidates (correct / incorrect / non-executable)Comparison conditionMean R, correctMean R, incorrectMean R, non-executableRate of correct > incorrect > non-executable
A-in2/1/1Correct objective value within the prior interval0.71280.44010.0000100% (20/20)
A-out2/1/1Correct objective value outside the prior interval0.55030.44010.0000100% (20/20)
B2/2/0Two incorrect candidates share the same objective value0.65730.4779—100% (20/20)*
C1/2/1Two incorrect candidates share the same objective value0.58510.50530.0000100% (20/20)
D1/2/1Two incorrect candidates have different objective values0.58510.44240.0000100% (20/20)

Conclusion: In the controlled constructions with correct test intervals, correct candidates receive higher total rewards in all 20 instances of every setting, even when incorrect candidates form a majority consensus or the prior for the original problem is miscalibrated.


(3) Effect of the prior on final performance.

Prior settingEasy Acc (%)Hard Acc (%)Overall Acc (%)
Default prior: Qwen3-4B-Instruct44.62 ± 1.5426.40 ± 2.8633.53 ± 1.25
Deliberately perturbed: center shifted by ±30%, interval width halved36.41 ± 0.8923.76 ± 2.9728.71 ± 1.52
Objective-scale weighting disabled40.00 ± 2.6621.12 ± 2.8628.51 ± 0.70

Conclusion: The default prior yields a 5.02% gain, while performance under substantial prior perturbation remains close to that with scale weighting disabled, **supporting both the effectiveness of the objective-scale prior and its robustness to prediction errors**.

II. Experimental reliability and validation across datasets (W3 & W4 & Q2 & Q3 & Q6)


W4 & Q2: Random seeds for the main results and LoRA ablation.

Thank you for suggesting that we report variability across runs. Table 2 reports single-seed results for StarOR; the OptMATH result for Best-of-N (N = 16) also comes from a single run, as noted below the table. Baseline results marked with an asterisk are taken from the corresponding papers. The scaling experiment in Table 6 presents single-seed StarOR results alongside three-seed averages for Best-of-N. To directly compare performance across repeated runs, we provide additional three-seed results on OptMATH.

We conducted three-seed experiments on all 166 OptMATH problems, including a corresponding three-seed LoRA ablation. Easy and Hard denote difficulty groups used for analysis.

OptMATH groupProblemsWithout LoRA (%)With LoRA (%)Mean gain (percentage points)
Easy6542.56 ± 0.8944.62 ± 1.54+2.06
Hard10118.15 ± 2.0626.40 ± 2.86+8.25
All16627.71 ± 1.0433.53 ± 1.25+5.82

Conclusion: Under the same search and reward settings, LoRA provides a mean gain of 5.82 percentage points overall and 8.25 percentage points on Hard problems, demonstrating its benefits, especially on difficult problems.

Q3: How exactly is the compute budget matched between StarOR and Best-of-N/AutoFormulator in the main comparison?

Search settings in the main table. All three methods in Table 2 use Qwen3-4B-Instruct-2507 across the five benchmarks, with the following search or sampling settings.

MethodNumber of candidatesMaximum search iterationsGeneration limit per candidate
Best-of-NN = 16, sampled in parallel once—8K tokens
AutoFormulatorN = 8T = 106K tokens
StarORK = 8 candidates per expansionT = 106K tokens

Generation-budget comparison on OptMATH. We configure the methods with similar target generation budgets.

MethodSearch or sampling settingsGeneration limit per candidateAccuracy (%, mean ± std.)Actual generated tokens (K)Runtime (s)
BoN-ConsensusN = 608K30.72 ± 1.59265.0119.76
BoN-ORMN = 608K28.51 ± 1.94279.9125.51
AutoFormulatorN = 10, T = 128K13.86 ± 1.20129.4317.15
StarORK = 8, T = 106K33.53 ± 1.25117.4209.1

Conclusion: In this comparison, StarOR improves accuracy over BoN-Consensus by 2.81 percentage points while generating 55.7% fewer tokens; runtimes are reported in the table.

Q6: Stage decomposition and validation across datasets.

The four-stage decomposition follows modeling dependencies while balancing feedback granularity and error propagation. The problem type and sets establish the modeling paradigm and index domains; parameters and variables are then defined on this basis; the objective and constraints use these established symbols; finally, solver code is generated. This decomposition keeps closely related decisions within the same stage and provides search and update feedback at major dependency boundaries. Generating an entire formulation at once makes errors harder to localize to individual decisions, whereas an overly fine decomposition lengthens the dependency chain and separates decisions that should be made jointly.

We provide additional comparisons of the four-stage decomposition on OptMATH and IndustryOR.

DecompositionMax. iterations × candidates per iteration × token limit per candidateNominal generation limit (K)OptMATH accuracy (%, mean ± std.)OptMATH actual generated tokens (K)IndustryOR accuracy (%, mean ± std.)
One-stage4 × 10 × 12K48031.93 ± 3.35123.249.33 ± 1.53
Four-stage10 × 8 × 6K48033.53 ± 1.25117.452.00 ± 1.73
Six-stage18 × 6 × 4K43231.12 ± 0.92183.650.33 ± 1.53

Conclusion: The four-stage decomposition improves accuracy on IndustryOR and outperforms both the one-stage and six-stage variants on OptMATH while generating fewer tokens, supporting its balance between modeling granularity and search efficiency in these settings.

III. Compute fairness and practical costs (W3 & W5 & Q3)


W3 & Q3: How are the budgets matched between StarOR, Best-of-N, and AutoFormulator?

On OptMATH, all methods use the same Qwen3-4B-Instruct-2507 backbone, vLLM, decoding settings, Gurobi environment, and execution timeout. Each method retains its own search or selection procedure and runs with the target generation-budget configuration specified in the appendix. We report both actual generated tokens and runtime.

MethodSearch/sampling configurationAccuracy (%, mean ± std.)Generated tokens (K)Runtime (s)
BoN-ConsensusN = 60, 8K limit per candidate30.72 ± 1.59265.0119.76
BoN-ORMN = 60, 8K limit per candidate28.51 ± 1.94279.9125.51
AutoFormulatorN = 10, T = 12, 8K limit per candidate13.86 ± 1.20129.4317.15
StarORK = 8, T = 10, 6K limit per candidate33.53 ± 1.25117.4209.1

W5: How do the additional costs of search and updates affect practical use?

Compared with BoN-Consensus, StarOR improves accuracy by 2.81 percentage points while generating 55.7% fewer tokens in this setting. Sequential search, repeated execution, and GRPO updates also require additional time: the average runtime is 209.1 s, compared with 119.76 s for BoN-Consensus. Relative to AutoFormulator, StarOR achieves higher accuracy, generates fewer tokens, and has a shorter runtime. Table 6 further illustrates the practical trade-off in expansion width: StarOR (K = 4) achieves 25.9% accuracy in 93.9 s, whereas Best-of-N (N = 32) achieves 24.9% in 104.2 s.

Conclusion: We prioritize high-value, difficult industrial modeling tasks, where modeling accuracy directly affects downstream decisions and is more important than generation latency; even under the controlled comparison, StarOR achieves higher accuracy with fewer actual generated tokens.

IV. Component contributions and adaptation behavior (Q4 & Q7)


Q4: Repair trigger frequency and contribution to accuracy.

We also ablate the repair loop, with results shown below.

Maximum repair roundsAccuracy (%)Gain over no repair (percentage points)Average runtime (s)
030.7—194.6
2 (default)32.5+1.8205.2
433.1+2.4219.8
833.1+2.4252.1

Conclusion: The default two-round repair loop improves accuracy by 1.8 percentage points at an average additional cost of 10.6 seconds, providing an effective balance between modeling accuracy and repair cost.

Q7: Scope of GRPO updates, stability at K = 8, and per-problem effects.

Each update uses 8 sibling candidates within the current instance that share the same prefix and modeling stage. After the completions are executed, total rewards are normalized within the group to obtain relative advantages, which are then used to update the policy for generating the current-stage formulation. To characterize the stability and impact of GRPO LoRA updates, we examine changes in correctness in the LoRA ablation across three random-seed runs, totaling 498 problem–run pairs.

Without LoRA → with LoRANumber of pairsProportion of all pairs
Incorrect → correct499.84%
Correct → incorrect204.02%
Net increase in correct results295.82%

Pass@3 across three runs and per-problem changes. We further aggregate the three random-seed runs by problem: a problem passes under pass@3 if at least one of the three complete runs produces a correct answer. Both settings use the same 166 OptMATH problems, and we report coverage rates and changes in the covered sets.

GroupWithout LoRA pass@3 (%)With LoRA pass@3 (%)Newly covered problemsLost coverageNet newly covered problemsGain (percentage points)
Easy (65 problems)58.4661.5453+2+3.08
Hard (101 problems)31.6849.50224+18+17.82
All (166 problems)42.1754.22277+20+12.05

Conclusion: In the three-seed comparison at K = 8, LoRA turns 49 incorrect results into correct ones and 20 correct results into incorrect ones, yielding 29 additional correct results overall; pass@3 also increases by 12.05 percentage points, showing that despite per-problem variation, instance-level updates improve search outcomes overall and expand coverage of difficult problems.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response and Summary

We thank the reviewers for their time and constructive feedback. They highlighted three strengths of StarOR.

  1. Search informs subsequent modeling decisions. Sibling formulations from the same MCTS node provide local comparisons for updating the policy that generates the next candidates on the same problem. 68wV, qfEM
  2. The stages and feedback reflect the modeling task. The four stages follow dependencies among modeling decisions. Execution, objective-value consensus, structural consistency, and perturbation tests provide complementary feedback without correctness labels. uGrD
  3. Both adaptation and reward design contribute. LoRA improves accuracy on all five benchmarks. In separate ablations, removing LoRA or replacing the full reward with majority voting reduces macro-average accuracy from 65.0% to 60.8%. 68wV, qfEM

We address the main concerns below.

  1. StarOR’s two contributions to OR modeling. (uGrD W1/Q8)
    StarOR integrates tree search and test-time adaptation and provides label-free feedback by combining executability, objective-value consensus, scale priors, perturbation tests, and structural consistency. We will remove the “first” claim and define our contribution through these two designs.

  2. Reward reliability in controlled constructions and natural search. (uGrD W2/Q1; 68wV W1/Q1; qfEM W1/Q1)
    On IndustryOR, with correctly calibrated test intervals, the full reward ranked correct formulations above executable variants with deleted constraints or reversed inequalities in 20/20 instances per setting. Across five benchmarks (4,521 groups, 1,319 problems), a unique largest cluster with an incorrect objective value contained a top-reward candidate in 5.27%–22.29% of all groups per benchmark. When such a cluster formed, the next deeper stage selected outside it in 46.15%–62.38% of cases. These results support reward discrimination and the ability of subsequent search to escape incorrect consensus.

  3. Benefits of the scale prior and performance under miscalibration. (uGrD Q5; 68wV follow-up)
    On OptMATH, the default scale prior improves accuracy by 5.02 pp over disabling scale weighting. Under the tested perturbation, coverage falls from 70.5% to 27.1%, yet accuracy remains close to the unweighted baseline (28.71% vs. 28.51%). The default prior improves accuracy, while performance under substantial miscalibration stays near the unweighted baseline.

  4. Three-seed results and gains from LoRA beyond search alone. (uGrD W3/W4, Q2/Q7)
    We report the mean and standard deviation of accuracy over three seeds for each of the five benchmarks. The 4B model achieves 65.06% macro-average accuracy, competitive with the reported DeepSeek-R1 result. Across three seeds, LoRA improves OptMATH accuracy by 5.82 pp overall (pass@3 +12.05pp) and 8.25 pp on Hard problems (pass@3 +17.82pp), showing that policy updates add value beyond search, particularly on difficult problems.

  5. Accuracy, token use, and end-to-end runtime under matched generation budgets. (uGrD W3/W5, Q3)
    Under the same candidate generation token cap, StarOR improves accuracy over BoN-Consensus by 2.81–16.59 pp across the evaluated benchmarks while generating 47%–56% fewer tokens. All reported runtimes are end-to-end, including generation, solver execution, adaptation, and repair as applicable. We focus on high-value, difficult modeling problems where accuracy takes priority over latency.

  6. Rationale for the four-stage decomposition and evidence across benchmarks. (uGrD Q6; 68wV W1/Q2, W2)
    The four stages follow dependencies among modeling decisions, allowing StarOR to compare candidates at each stage and evaluate closely related decisions together. The one-stage baseline also retains multiple GRPO updates per problem. In comparisons across three benchmarks, the four-stage variant achieved higher accuracy and used fewer generated tokens than the one- and six-stage variants where token use was measured. We also discussed how groups of candidates sharing a parent could support policy updates in beam search or evolutionary search.

All three reviewers now give positive ratings. In response to Reviewer uGrD’s follow-up, we addressed all five points with

  1. three-seed main results across five benchmarks,
  2. budget-controlled comparisons on three benchmarks and clarification of end-to-end runtime,
  3. a clearer contribution statement and a commitment to remove the “first” claim,
  4. natural-search audits across five benchmarks,
  5. stage-decomposition results on MAMO-Complex.

We thank the reviewers and the area chair for their time and consideration. We look forward to hearing your thoughts and would be happy to clarify any remaining questions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.