- Rating
- 2
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper proposes a self-training framework for optimization modeling. The current model is evaluated with and without hints, and the resulting performance gap determines the allocation of subsequent training data. A fixed copy of the initial model generates hint-assisted solution trajectories, which are filtered by verifiers. The accepted solutions are then used for LoRA-based supervised fine-tuning with the hints removed from the training inputs. This assessment, allocation, and training cycle is repeated. The method is evaluated on three optimization-modeling benchmarks.
Critical strengths
None
Critical weaknesses
- The effectiveness of the performance-gap allocation metric is insufficiently established.
The comparison against uniform allocation provides useful evidence, but it does not establish whether the proposed gap offers information beyond ordinary problem difficulty. In particular, comparison with allocation based on the unassisted failure rate, expressed as 1 - p_plain, is needed to distinguish the value of hint-based assessment from simply prioritizing problems the model struggles to solve.
Moreover, the paper does not directly demonstrate that a larger performance gap predicts greater subsequent learning gains. This leaves the central interpretation of the allocation signal insufficiently validated.
- The contribution of recursive adaptation is not sufficiently isolated.
The experiments do not compare repeated reassessment against a curriculum determined by the initial assessment and then held fixed. Consequently, it remains unclear whether updating the allocation throughout training provides a meaningful advantage over a static curriculum.
Without this control, the benefits of recursive adaptation are difficult to distinguish from those of additional supervised fine-tuning on verified, externally guided solutions. A comparison using the same solution-generation procedure and matched training budgets would help establish the specific contribution of recursive reassessment.
Questions for authors
-
Could the authors compare the proposed allocation rule against allocation based on the unassisted failure rate, 1 - p_plain, while keeping the solution-generation procedure, accepted-example budget, and training settings consistent?
-
Could the authors provide paired statistical tests and confidence intervals for the improvements on NL4Opt and OptMATH-bench-193, both over the initial model and over uniform allocation? Since the reported variation reflects evaluation seeds, could the authors also assess variability across independent runs of the training pipeline?
3 comments
Response to Reviewer wPa9 (2/2): Training variability and allocation controls
Response to the concern about variability across independent training runs
Thank you for highlighting the distinction between evaluation randomness and training randomness. The currently reported standard deviations are computed across different decoding seeds at the same trained checkpoint and reflect variability during evaluation. The paired analyses likewise characterize evaluation uncertainty at fixed checkpoints. We have not yet completed independent training replications in this round of revisions. We therefore limit our conclusions to the training runs evaluated and do not interpret these standard deviations as evidence of stability across independent training runs.
Response to the concern about failure-rate and frozen initial-gap controls
Dynamic failure-rate allocation selects samples using 1 - p_plain from its own model at each round, with the same exploration and capacity constraints as the original dynamic control. The frozen initial-gap condition retains the assistance-gap information from R0 but does not use post-training measurements to update priorities.
In the order NL4Opt / MAMO / OptMATH, the differences between dynamic gap and failure-rate allocation are +0.41 / +1.81 / +1.56 percentage points. These results show consistently positive differences in mean performance in favor of dynamic gap.
Thank you for the additional experiments.
However, the central interpretation of the performance gap remains insufficiently validated: the results do not directly establish that larger gaps predict greater subsequent learning gains. The allocation comparisons provide useful evidence, but do not fully justify this predictive interpretation.
I therefore maintain my rating.
Response to Reviewer wPa9 (1/2): Additional experiments and statistical evidence
Overview of the additional experiments
Thank you for your specific and constructive comments on our work. To address these concerns, we added three controls with matched training budgets: allocation based on the frozen initial gap, dynamic failure-rate allocation, and offline SFT. All three additional experiments use 15 rounds, 128 training records per round, an effective batch size of 8, and 240 optimizer updates, with evaluation fixed at round 15 (R15). This matches the number of training examples and the update budget of the main experiment.
The central design of our method is to translate the learner's current capability deficiencies into subsequent training decisions. The assisted condition provides a way to assess how amenable failures are to intervention; round-by-round evaluation feeds the updated model state back into training allocation; and execution-based verification checks the outputs. The resulting recursive loop is "model state → deficiency measurement → training selection → model update → remeasurement." The additional controls examine different aspects of this design, rather than treating all components as a single factor.
The mean pass@1 scores (%) on NL4Opt, MAMO, and OptMATH are as follows:
Dynamic gap achieves the highest mean accuracy among the four conditions on NL4Opt and MAMO. Compared with the frozen initial-gap condition, it improves performance by 0.41 and 2.46 percentage points, respectively. Compared with dynamic failure-rate allocation, it improves performance by 0.41, 1.81, and 1.56 percentage points across the three benchmarks, respectively.
Thank you for suggesting a failure-rate baseline, a frozen initial-allocation control, and predictive analyses. We added the corresponding controls and present descriptive results separately from statistical inference.
Response to the concern about paired statistical evidence and gains over uniform allocation
Thank you for recommending paired analyses at the problem level. We checked the archived outputs and performed 20,000 paired cluster bootstrap resamples, using problems as the resampling unit and retaining the three decoding outcomes for each problem under each of the two methods. We also conducted paired sign-flip tests.
On OptMATH-bench-193, the main model improves from 35.58% for the initial model with available paired outputs to 40.07%, a gain of 4.49 percentage points, with a 95% confidence interval of [0.35, 8.12] percentage points. On NL4Opt, the mean improvement is approximately 0.41 percentage points; the 95% problem-cluster bootstrap interval is approximately [0, 0.95] percentage points, and the two-sided paired sign-flip test yields p = 0.25.
Overall, our method improves mean performance over the initial model on both OptMATH and NL4Opt. The improvement on OptMATH is statistically supported, whereas NL4Opt shows a smaller numerical gain. Relative to uniform allocation, our method improves performance by 4.10, 0.69, and 0.14 percentage points on MAMO, OptMATH, and NL4Opt, respectively. These mean differences are consistently positive, but their magnitudes vary substantially across tasks, with the largest gain on MAMO. We therefore distinguish the capability improvements from training as a whole from the incremental benefits of dynamic task allocation; the significance of these gains is task-dependent.