- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes RELIVE-OR, a framework for improving LLM-based operations research modeling through synthetic experience learning. It combines structural data synthesis, closed-loop verification, and influence-based sample selection for fine-tuning, while introducing FREE for update-free test-time candidate selection. Experiments across six OR modeling benchmarks demonstrate competitive performance with an 8B model.
Critical strengths
-
Meaningful research problem. The paper addresses an important challenge in LLM-based OR modeling: generating structurally diverse training data while maintaining modeling correctness.
-
Comprehensive framework and competitive results. The paper presents an integrated framework covering data synthesis, validation, training, and inference. Experiments across six OR benchmarks demonstrate competitive performance with an 8B model.
Critical weaknesses
-
Several key aspects of the proposed method remain unclear. Although Equations (8)–(9) define seed-relative OR model-specification preservation, the paper does not sufficiently explain how these specifications are constructed and verified in practice. Similarly, Section 2.1.3 introduces structural consistency checks, but their actual implementation is not clearly described. These missing details make it difficult to assess the reliability and reproducibility of the validation framework.
-
The framework combines several existing techniques, such as influence-based data selection, repeated SFT, and reference-guided candidate selection. However, the connection between these components and the paper's central motivation of preserving OR model specifications is not sufficiently clear. Moreover, the experiments do not adequately isolate the contributions of individual components, making it difficult to determine whether the improvements arise from the proposed mechanisms or from increased synthetic data, repeated SFT, and other training choices.
-
Section 2.4 emphasizes curvature-aware alignment as the main FREE mechanism, yet Table 1 shows that it underperforms simpler gradient- and embedding-based selectors. Moreover, even the best FREE variant (77.6%) falls below the reported SF@3 result (78.3%). Therefore, the additional benefit of FREE over conventional multi-candidate selection remains unclear, particularly given the emphasis on improvements over First-only accuracy.
Questions for authors
-
How are the seed-relative specifications constructed and verified, and how is structural consistency evaluated in practice? Please clarify the implementation of , and $S(z',\hat z')$, including the specific validation criteria and procedures.
-
Can the authors provide more systematic ablation studies to quantify each component's contribution? In particular, comparisons between different validation strategies, random versus influence-based sample selection, and single-stage versus two-stage SFT under comparable training budgets would help clarify the sources of improvement.
-
Why does the proposed curvature-aware alignment method underperform simpler alternatives, such as gradient alignment and embedding-based selection, in Table 1?
9 comments
Response to Reviewer LQGT for Q2
Q2. Ablations of selection and validation
We compare the broad model 1, Random 50%, and Retrospective 50%, with matched subset size and training budget for the latter two. Retrospective selection improves First accuracy on five of six benchmarks, from 66.39% to 69.43% (+3.04 points), without additional inference cost. SF@3 increases from 77.88% to 78.28%;
Macro-average accuracy (%).
| Training checkpoint | First | SF@3 | Mean LogLik | Trusted Emb. | Gradient Align. | Curvature Align. | Oracle@3 |
|---|---|---|---|---|---|---|---|
| Broad model 1 | 62.58 | 68.82 | 69.08 | 67.71 | 68.02 | 66.52 | 70.35 |
| Random 50% | 66.39 | 77.88 | 77.33 | 78.80 | 78.10 | 77.05 | 81.51 |
| Retrospective top 50% (2) | 69.43 | 78.28 | 78.58 | 77.72 | 77.45 | 77.64 | 81.42 |
Unweighted mean across six benchmarks. Each checkpoint generates three candidates per problem: seed 44, temperature 0.1, top-p 0.8. First uses the first candidate; SF@3 uses majority voting; Oracle@3 counts success if any candidate is correct. All selectors rescore the same saved candidates without parameter updates. MAMO uses the cleaned and fixed datasets.
All selectors improve macro-average accuracy over First across the three checkpoints (+3.94–12.41 points), although none consistently outperforms SF@3. Reference-based FREE variants permit bank updates without retraining and provide inspectable reference formulations.
Q2: Detailed results (%)
(a) Broad model 1
| Dataset | First | SF@3 | Mean LogLik | Trusted Emb. | Gradient Align. | Curvature Align. | Oracle@3 |
|---|---|---|---|---|---|---|---|
| NL4OPT | 78.87 | 81.22 | 81.22 | 80.28 | 80.28 | 80.28 | 82.16 |
| NL4LP | 77.53 | 79.21 | 80.90 | 80.90 | 78.65 | 78.65 | 81.46 |
| MAMO-Easy | 89.17 | 92.48 | 92.66 | 92.11 | 92.48 | 92.29 | 93.39 |
| MAMO-Complex | 71.17 | 77.48 | 74.77 | 78.38 | 76.58 | 75.68 | 80.18 |
| IndustryOR | 30.95 | 38.10 | 40.48 | 35.71 | 35.71 | 33.33 | 40.48 |
| ComplexOR | 27.78 | 44.44 | 44.44 | 38.89 | 44.44 | 38.89 | 44.44 |
| Macro-average | 62.58 | 68.82 | 69.08 | 67.71 | 68.02 | 66.52 | 70.35 |
(b) Random 50%
| Dataset | First | SF@3 | Mean LogLik | Trusted Emb. | Gradient Align. | Curvature Align. | Oracle@3 |
|---|---|---|---|---|---|---|---|
| NL4OPT | 80.28 | 87.32 | 86.38 | 87.32 | 86.85 | 86.38 | 87.32 |
| NL4LP | 89.89 | 98.31 | 98.31 | 98.31 | 97.19 | 96.63 | 98.31 |
| MAMO-Easy | 88.62 | 95.41 | 95.60 | 95.96 | 95.05 | 94.86 | 97.25 |
| MAMO-Complex | 52.25 | 71.17 | 63.06 | 72.97 | 72.07 | 69.37 | 78.38 |
| IndustryOR | 59.52 | 59.52 | 59.52 | 57.14 | 61.90 | 59.52 | 66.67 |
| ComplexOR | 27.78 | 55.56 | 61.11 | 61.11 | 55.56 | 55.56 | 61.11 |
| Macro-average | 66.39 | 77.88 | 77.33 | 78.80 | 78.10 | 77.05 | 81.51 |
(c) Retrospective top 50% (2)
| Dataset | First | SF@3 | Mean LogLik | Trusted Emb. | Gradient Align. | Curvature Align. | Oracle@3 |
|---|---|---|---|---|---|---|---|
| NL4OPT | 81.69 | 86.85 | 87.79 | 87.79 | 87.32 | 87.79 | 88.26 |
| NL4LP | 91.01 | 96.63 | 97.19 | 96.63 | 97.19 | 96.07 | 97.19 |
| MAMO-Easy | 90.64 | 94.86 | 95.60 | 95.41 | 95.41 | 94.50 | 96.88 |
| MAMO-Complex | 54.05 | 73.87 | 70.27 | 72.97 | 72.07 | 74.77 | 78.38 |
| IndustryOR | 54.76 | 61.90 | 59.52 | 52.38 | 57.14 | 57.14 | 66.67 |
| ComplexOR | 44.44 | 55.56 | 61.11 | 61.11 | 55.56 | 55.56 | 61.11 |
| Macro-average | 69.43 | 78.28 | 78.58 | 77.72 | 77.45 | 77.64 | 81.42 |
Pooled accuracy (%).
Total correct instances divided by total evaluated instances; each instance receives equal weight.
| Training checkpoint | First | SF@3 | Mean LogLik | Trusted Emb. | Gradient Align. | Curvature Align. | Oracle@3 |
|---|---|---|---|---|---|---|---|
| Broad model 1 | 80.23 | 83.75 | 83.94 | 83.57 | 83.30 | 82.94 | 85.11 |
| Random 50% | 81.41 | 89.80 | 88.99 | 90.25 | 89.53 | 88.90 | 91.79 |
| Retrospective top 50% | 83.12 | 89.53 | 89.80 | 89.62 | 89.62 | 89.35 | 91.61 |
| Retrospective − Random (pp) | +1.71 | −0.27 | +0.81 | −0.63 | +0.09 | +0.45 | −0.18 |
Response to Reviewer LQGT for Q3
FREE’s central contribution is reusing the verified experience bank for customizable, update-free test-time program selection. FREE-Trusted Embedding, FREE-Gradient Alignment, and FREE-Curvature Alignment provide alternative reference-based scoring rules; FREE-Mean LogLik provides the model-confidence baseline.
1. Role of curvature-aware alignment.
Curvature correction accounts for differences in scale and correlation among reference-gradient directions. It may help when these patterns are informative and reliably estimated. Because the regularized matrix uses at most eight retrieved references, its effectiveness can depend on reference quality and regularization. This is a possible explanation for the observed differences, rather than an experimentally established cause.
In the original Table 1, FREE-Gradient Alignment achieves 77.6%, FREE-Trusted Embedding 77.4%, and FREE-Curvature Alignment 76.19%. The updated Q2 comparison gives a different ordering for the retrospective checkpoint: FREE-Trusted Embedding 77.72%, FREE-Curvature Alignment 77.64%, and FREE-Gradient Alignment 77.45%. We will clarify that curvature-aware alignment is an explored variant, rather than a consistently superior scoring rule.
2. Comparison with self-consistency.
SF@3 achieves 78.3% in the original results and 78.28% in the updated retrospective-checkpoint evaluation, compared with 77.72% for the best reference-based selector in the latter comparison. Dataset-level results show complementary strengths.
Dataset-level win rates against SF@3. Each selector is compared with SF@3 on six benchmarks across three checkpoints. A win means strictly higher accuracy; ties are counted separately. Each dataset–checkpoint comparison receives equal weight.
| Selector | Broad model: wins/6 | Random 50%: wins/6 | Retrospective 50%: wins/6 | Overall win rate | Wins / ties / losses |
|---|---|---|---|---|---|
| FREE-Mean LogLik | 3/6 (50.0%) | 2/6 (33.3%) | 4/6 (66.7%) | 9/18 (50.0%) | 9 / 4 / 5 |
| FREE-Trusted Embedding | 2/6 (33.3%) | 3/6 (50.0%) | 3/6 (50.0%) | 8/18 (44.4%) | 8 / 3 / 7 |
| FREE-Gradient Alignment | 0/6 (0.0%) | 2/6 (33.3%) | 3/6 (50.0%) | 5/18 (27.8%) | 5 / 4 / 9 |
| FREE-Curvature Alignment | 0/6 (0.0%) | 0/6 (0.0%) | 2/6 (33.3%) | 2/18 (11.1%) | 2 / 3 / 13 |
For the retrospective checkpoint, FREE-Trusted Embedding and FREE-Gradient Alignment each outperform SF@3 on three benchmarks and tie on one; FREE-Curvature Alignment wins on two and ties on one. At least one reference-based variant wins on five of six benchmarks. This describes coverage across variants, not the performance of a single fixed selector.
Pooled accuracy (%). All selectors rescore the same saved candidates within each checkpoint. Pooled accuracy weights each test instance equally.
| Checkpoint | First | SF@3 | FREE-Mean LogLik | FREE-Trusted Embedding | FREE-Gradient Alignment | FREE-Curvature Alignment | Oracle@3 |
|---|---|---|---|---|---|---|---|
| Broad model | 80.23 | 83.75 | 83.94 | 83.57 | 83.30 | 82.94 | 85.11 |
| Random 50% | 81.41 | 89.80 | 88.99 | 90.25 | 89.53 | 88.90 | 91.79 |
| Retrospective 50% | 83.12 | 89.53 | 89.80 | 89.62 | 89.62 | 89.35 | 91.61 |
| Retrospective − Random (pp) | +1.71 | −0.27 | +0.81 | −0.63 | +0.09 | +0.45 | −0.18 |
Pooled accuracy = total correctly solved instances / total evaluated instances. SF@3 denotes self-consistent majority voting over three candidates. FREE-Mean LogLik is the model-confidence baseline; the other three FREE variants use retrieved verified references.
For the retrospective checkpoint, FREE-Trusted Embedding and FREE-Gradient Alignment achieve 89.62%, FREE-Curvature Alignment 89.35%, and SF@3 89.53%. The reference-based selectors are therefore within 0.18 percentage points of SF@3 under this aggregation.
3. Additional benefits of the verified experience bank.
Reference-based FREE complements answer agreement with an external reference signal:
- Customization: adding domain-relevant or expert-verified experiences can adjust candidate rankings without retraining the model or regenerating candidates.
- Inspectability: users can examine the retrieved formulations and their modeling structures when reviewing a selection.
A correct numerical answer does not always imply a correct formulation. For example, an omitted constraint may be inactive on one instance but become important when the data change. Reference-based FREE adds a compatibility signal from trusted formulations through embeddings or gradients, encouraging alignment with validated modeling behavior.
Response to Reviewer LQGT for Q1
Dear Reviewer LQGT,
Thank you for your thoughtful review. We clarify how the intended specification, structural checks, and retrospective selection are implemented.
1. Constructing the seed-relative specification (Eqs. 8–9).
Γj=Specsyn(rπ(j)0,sj,hj).
These requirements are supplied through the synthesis prompt: the seed provides a validated problem–code example, the scenario defines the new decision setting, and the scaffold supplies modeling guidance. Equation (9), rj∈Fspec(Γj), expresses the intended specification-preservation condition.
To support diversity, synthesis may change variables, objectives, and constraints. We therefore impose no universal list of unchanged seed structures. Once generated, z′ becomes the executable anchor for checking consistency with its textual description and reconstructed code.
2. Evaluating structural consistency S(z′,z^′).
Rule-based checks extract features from solver-API calls and recognizable code patterns. They compare:
A required structure detected in z′ but absent from z^′ triggers failure. Structures not detected in z′ impose no additional requirements. Passing S means that no conflict was detected by these checks; identical code, variable names, and code organization are unnecessary.
3. Applying Vfinal.
We render z′ as text x′ and reconstruct z^′ through a separate call to the same LLM. Acceptance requires structural consistency, successful execution with finite objective values, and numerical agreement:
\leq \max(10^{-6},10^{-3}\max(|f(z')|,|f(\hat z')|)).$$ When both objective magnitudes are below $10^{-6}$, the absolute difference must instead be at most $10^{-8}$. Structural or execution failures trigger bounded repair and rechecking. A finite objective mismatch normally triggers a new textual rendering from the unchanged anchor. We try up to **five textual views** and require at least one passing reconstruction; otherwise, the candidate is rejected. Accepted experiences form $\mathcal D_V$. On **119 known-good pairs**, multi-view verification retains **49 pairs**, compared with **36** under single-view verification. This recovers **13 additional valid pairs** and reduces false rejection from **69.75% to 58.82%** without relaxing the acceptance criteria. The experiment demonstrates why multiple views help retain more valid synthetic training examples. **4. Connecting seed-relative alignment to retrospective learning.** Trusted problem–code pairs provide the reference objective: $$L_0(\theta)=\frac{1}{|\mathcal D_0|} \sum_{(x,z)\in\mathcal D_0}\ell(f_\theta(x),z).$$ For each verified experience $r_j=(x_j',z_j')$, define its training loss as $$\ell_j(\theta)=\ell(f_\theta(x_j'),z_j').$$ At the broad-training checkpoint $\theta_1$, let $$g_0=\nabla L_0(\theta_1), g_j=\nabla\ell_j(\theta_1).$$ Its approximate influence on trusted-seed loss is $$\widetilde I_0(r_j) =-g_0^\top(G_1+\lambda I)^{-1}g_j,$$ where $G_1$ is the empirical gradient second-moment matrix over the broad-training data and $\lambda>0$ provides regularization. Under this local approximation, increasing the experience’s training weight by a small $\epsilon$ predicts $$\Delta L_0\approx\epsilon\,\widetilde I_0(r_j).$$ We retain the **50% with the smallest estimated influences**, corresponding to the greatest predicted loss reductions. Because the reference targets are validated formulations, this favors experiences expected to reinforce trusted modeling behavior. Curvature accounts for the model’s local response to parameter changes, beyond gradient similarity alone. **5. Scope, scalability, and reproducibility.** Reconstruction checks determine admission to $\mathcal D_V$; retrospective selection then scores each experience against the aggregate trusted-seed loss. Regularized regression avoids explicitly constructing and inverting the full curvature matrix, providing a shared selection criterion across formulation families. We will clarify that Eqs. (8)–(9) define the intended admissible set, while these procedures provide consistency checks and an indirect signal of specification alignment and it is not proof of every requirement in $\Gamma_j$.