- Rating
- 2
- Confidence
- 3
- Informativeness
- 3(+1)
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
OptArchitect synthesizes optimization problems by extracting executable modeling skills and recursively composing them through decision-level couplings. Candidates undergo structural checks, reference execution, and independent reconstruction from their natural-language descriptions before admission. The resulting benchmark contains 116 atomic and 79 composite problems, with average accuracy across four reasoning models decreasing from 74.8% to 46.5%. The paper also reports closed-world recomposition, size-matched comparisons, expert assessments, synthesis ablations, and downstream fine-tuning results.
Critical strengths
- Executable skill representations and recursive composition provide a concrete approach to generating structurally varied optimization problems.
- Independent reconstruction and blinded expert assessment check semantic ambiguity beyond whether the generated reference model executes.
- Closed-world recomposition and synthesis ablations provide useful controls for evaluating the contribution of structured composition.
Critical weaknesses
- Residual ambiguity in accepted composites could inflate the benchmark's apparent difficulty and distort model comparisons.
- The bridge acceptance test remains operationally unclear, limiting confidence that accepted composites require genuine coupling.
- Limited matching controls and unmatched training comparisons leave the specific benefits of structural composition insufficiently isolated.
Questions for authors
-
Section 4.4 reports lower clarity and completeness ratings for composite problems. Does the Atomic-to-Composite accuracy gap persist when both groups are restricted to verified unambiguous problems?
-
How is Equation (9)'s condition on feasible regions or optimal solution sets checked computationally, particularly with tied optima? Are acceptance witnesses retained for each composite?
-
The size-matched results in Section 4.3 control variable and constraint counts. Does the gap persist after controlling formulation class, description length, and semantic clarity?
-
Section 4.4 reports 71.2% macro accuracy using 1K OR-Instruct and 1K OptArchitect examples. Are equal-budget Qwen3-8B comparisons available using 2K OR-Instruct examples alone or replacing synthetic composites with atomic examples?
-
Does Table 2 apply Equation (9)'s structural acceptance criterion to both recomposition conditions? If not, are results available for random pairs satisfying the same criterion?
-
Figure 1's caption claims accuracy "above 95% for all four models," but DeepSeek-V4-Pro scores 94.0% on MAMO EasyLP. Correct the caption to match the plotted results.
4 comments
Response to Reviewer ZNkR (2/3)
W2 / Q2. How is Equation 9 checked, in particular with tied optima? Are acceptance witnesses retained?
The submitted version did not describe this step, and Appendix L.3 spoke of a change in the returned decisions, which does not rule out ties. The revision reports the evidence by type for all 79 Composite problems (Section 3.2, Appendix A.6).
- 56 problems. The bridge enters only through constraints, so the model without the bridge is a relaxation of the composite model, and removing the bridge strictly improves the optimal value. Then no optimal solution of the relaxed model is feasible in the composite model, and the optimal sets differ whichever optimum the solver returns. The witness is the pair of optimal values with and without the bridge.
- 3 problems. The bridge enters only through constraints, but the optimal value does not change. These satisfy Equation 9 through the feasible-region condition, which also does not depend on ties, but the bridge is not binding at the optimum. We flag these three explicitly.
- 20 problems. The bridge also modifies the objective. Acceptance was based on a change in the returned optimal decisions. With multiple optima this is a necessary sign, not a proof, and we list it as a limitation.
On witnesses: for the 59 problems whose bridge enters only through constraints, our records keep the optimal values with and without the bridge. For the other 20, they keep the decisions returned by the two solves.
W3 / Q3. Does the gap persist after controlling formulation class, description length and semantic clarity?
Table 17 reports these controls on the clean problems. Restricting to clean problems fixes semantic clarity.
| Setting | Atomic n | Atomic acc. (%) | Composite n | Composite acc. (%) | Gap |
|---|---|---|---|---|---|
| All clean problems | 67 | 72.0 | 50 | 47.5 | 24.5 |
| LP | 16 | 78.1 | 22 | 59.1 | 19.0 |
| MILP | 29 | 69.8 | 14 | 37.5 | 32.3 |
| Nonlinear | 5 | 65.0 | 3 | 25.0 | 40.0 |
| Other | 17 | 72.1 | 11 | 43.2 | 28.9 |
| Length-matched | 10 | 70.0 | 10 | 40.0 | 30.0 |
Classes are assigned from the mathematical structure of the reference model. Other covers reference models solved by closed-form evaluation, dynamic programming, enumeration or policy search, without an LP or MIP formulation. The gap is present in every class. The Nonlinear row has only three Composite problems and we draw no conclusion from it. Composite problems are not concentrated in harder classes: 44% of the clean Composite problems are LP, against 24% of the clean Atomic problems. The length-matched row uses the ten clean Atomic and ten clean Composite problems closest in statement length (1,264 and 1,265 words on average). With ten problems on each side, it shows that the gap does not shrink, not how large it is.
On the weakness that the benefit of structural composition is not isolated, Appendix D.4 adds a same-parent comparison. Among the clean problems we manually identified 21 for which an uncoupled version can be built, and drew 10 with a fixed random seed. The uncoupled version keeps the same two parent skills and removes the bridge. Average accuracy is 70.0% on the uncoupled versions and 50.0% on the coupled versions, and the coupled version is harder for all four models (Table 8). The sample is small and we treat this as directional evidence.
Response to Reviewer ZNkR (3/3)
Q4. Equal-budget downstream comparisons
These are the right controls and the submitted version did not have them. The experiment is still running. We were worried about running out of time, so we are posting the rest of the response now and will add the results in this thread as soon as they are ready.
Q5. Does Table 2 apply Equation 9 to both recomposition conditions?
Yes. Both conditions build a bridge for each pair and accept a problem only if it passes the structural test of Equation 9. They differ only in how pairs are selected: uniformly at random in one case, by compatibility screening in the other. The submitted text was ambiguous on this point, because it mentioned bridges only for the Structured condition. Section 4.2 and Appendix C now state that both use the same acceptance criterion. The 16.6-point difference between Random (65.5%) and Structured (48.9%) therefore reflects compatibility-based pairing under the same criterion.
Q6. Figure 1 caption
Corrected. The caption, and the matching sentences in the introduction and Appendix B, now say that the four models are at or above 94%, in line with the 94.0% of DeepSeek-V4-Pro on MAMO EasyLP. Thank you for catching it.
Thank you again. Where the evidence is still incomplete, as for the bridge test on 20 problems and the downstream controls that are not finished, we have said so in the paper or in this response.
Thank you for the careful rechecking and renewed human evaluation. The average Atomic–Composite gap remains 24.5 points on the clean subset, and the same-parent comparison provides preliminary evidence of a coupling effect. I have adjusted my component scores accordingly.
I maintain my overall score because 20 of 79 composites rely on solver-returned decisions that cannot distinguish changed optimal sets from tied optima. For three other composites, please provide a witness that the feasible region changes. The same-parent study covers only ten pairs, and the equal-budget downstream comparisons are still pending. The PDF available to me matches the original submission, and Figure 2 remains difficult to read at normal zoom because the text is small and the panels cited in the caption are unlabeled. Releasing the clean subset or flagging its instances would also help benchmark users.
Response to Reviewer ZNkR (1/3)
Thank you for these specific questions. Each one points to a control or a clarification that the paper needed. After reading the reviews we rechecked every number in the paper and redid the human evaluation. This took a long time, which is why our response comes late. We hope you will still have time to look at it. Table numbers below refer to the revised PDF.
W1 / Q1. Does the gap persist on verified unambiguous problems?
We redid the human evaluation. Three reviewers each rated the same 80 Atomic and 60 Composite problems under a written rubric (Appendix G). We call a problem clean when all three answer yes on correctness, completeness and clarity. Our clarity criterion also rejects statements that are unambiguous but hard to read, so the clean problems are a subset of the unambiguous ones. 67 Atomic and 50 Composite problems are clean.
The gap persists and is positive for all four models. Your concern was also justified. The average gap falls from 32.7 to 24.5 points, so statement problems did inflate the apparent difficulty by about a quarter. The gap shrinks for every model, by between 5.6 and 10.8 points. Section 4.4 now reports the result on clean problems next to the result on the full sample, and we removed the sentence that read lower clarity as a sign of compositional difficulty.
Questions about statement quality are common for benchmarks in this area. We ourselves removed invalid problems from IndustryOR and other benchmarks when preparing the evaluation sets (Table 3). That is why we used the strictest rule we could here: a single no from any of the three reviewers on any of the three dimensions removes the problem.