acceptodds
Under review as a conference paper at ICLR 2027

OptArchitect: Building Verifiable and Challenging Optimization Problems through Recursive Skill Composition

Abstract

Large language models are increasingly capable of solving existing optimization problems, making many established benchmarks less effective for evaluating and improving advanced reasoning. We present **OptArchitect**, a framework for automatically constructing challenging and verifiable optimization problems through recursive composition of reusable reasoning skills. Instead of directly prompting a language model to generate problems, OptArchitect first extracts structured skills from optimization papers, industrial documents, and solver cases, representing each skill through its problem semantics, mathematical structure, executable solver implementation, and transfer constraints. The framework then composes compatible skills through a structure-aware matchmaker and introduces explicit bridge operations to create nontrivial interactions between them. Generated problems are validated through executable optimization models, independent solver probes, structural consistency checks, and iterative skill refinement. This process supports recursive composition, enabling problems to be generated at increasing depths of structural complexity. Experiments show that current frontier reasoning models perform well on existing optimization benchmarks but exhibit substantial degradation on OptArchitect-generated problems. Across four reasoning models, average accuracy decreases from 86.1% on existing IndustryOR problems to 56.1% on recursively composed OptArchitect problems. The degradation becomes more pronounced as composition depth increases, with GPT-5.5 accuracy falling from 84.5% at depth to 33.3% at , while GLM-5.2 falls from 61.2% to 0%. The generated problems also exhibit substantially greater structural complexity, with the number of decision variables increasing from 40 at to 651 at , and the number of constraints increasing from 40 to 909. Ablation studies further show that structured skill representations, compatibility matching, bridge operations, executable solver verification, independent probing, and iterative refinement are all important for maintaining generation quality. Human evaluation confirms high problem correctness and modeling completeness, while cross-backbone experiments demonstrate that extracted skills transfer across different language models. Finally, models fine-tuned on OptArchitect-generated problems show improved performance on downstream optimization reasoning tasks. These results suggest that recursive, verifiable skill composition provides a scalable approach for constructing difficult reasoning data beyond the capabilities and limitations of existing optimization benchmarks.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
Optimization Modeling

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 4 reviews, above 27% of the others.
Reviewer_ZNkROpenReview ↗
Rating
2
Confidence
3
Informativeness
3(+1)
Presentation
2
Soundness
2
Read review
Paper summary

OptArchitect synthesizes optimization problems by extracting executable modeling skills and recursively composing them through decision-level couplings. Candidates undergo structural checks, reference execution, and independent reconstruction from their natural-language descriptions before admission. The resulting benchmark contains 116 atomic and 79 composite problems, with average accuracy across four reasoning models decreasing from 74.8% to 46.5%. The paper also reports closed-world recomposition, size-matched comparisons, expert assessments, synthesis ablations, and downstream fine-tuning results.

Critical strengths
  • Executable skill representations and recursive composition provide a concrete approach to generating structurally varied optimization problems.
  • Independent reconstruction and blinded expert assessment check semantic ambiguity beyond whether the generated reference model executes.
  • Closed-world recomposition and synthesis ablations provide useful controls for evaluating the contribution of structured composition.
Critical weaknesses
  • Residual ambiguity in accepted composites could inflate the benchmark's apparent difficulty and distort model comparisons.
  • The bridge acceptance test remains operationally unclear, limiting confidence that accepted composites require genuine coupling.
  • Limited matching controls and unmatched training comparisons leave the specific benefits of structural composition insufficiently isolated.
Questions for authors
  1. Section 4.4 reports lower clarity and completeness ratings for composite problems. Does the Atomic-to-Composite accuracy gap persist when both groups are restricted to verified unambiguous problems?

  2. How is Equation (9)'s condition on feasible regions or optimal solution sets checked computationally, particularly with tied optima? Are acceptance witnesses retained for each composite?

  3. The size-matched results in Section 4.3 control variable and constraint counts. Does the gap persist after controlling formulation class, description length, and semantic clarity?

  4. Section 4.4 reports 71.2% macro accuracy using 1K OR-Instruct and 1K OptArchitect examples. Are equal-budget Qwen3-8B comparisons available using 2K OR-Instruct examples alone or replacing synthetic composites with atomic examples?

  5. Does Table 2 apply Equation (9)'s structural acceptance criterion to both recomposition conditions? If not, are results available for random pairs satisfying the same criterion?

  6. Figure 1's caption claims accuracy "above 95% for all four models," but DeepSeek-V4-Pro scores 94.0% on MAMO EasyLP. Correct the caption to match the plotted results.

4 comments

Response to Reviewer ZNkR (1/3)

Thank you for these specific questions. Each one points to a control or a clarification that the paper needed. After reading the reviews we rechecked every number in the paper and redid the human evaluation. This took a long time, which is why our response comes late. We hope you will still have time to look at it. Table numbers below refer to the revised PDF.

W1 / Q1. Does the gap persist on verified unambiguous problems?

We redid the human evaluation. Three reviewers each rated the same 80 Atomic and 60 Composite problems under a written rubric (Appendix G). We call a problem clean when all three answer yes on correctness, completeness and clarity. Our clarity criterion also rejects statements that are unambiguous but hard to read, so the clean problems are a subset of the unambiguous ones. 67 Atomic and 50 Composite problems are clean.

ModelAtomic, all rated (80)Composite, all rated (60)GapAtomic, clean (67)Composite, clean (50)Gap
GPT-5.5 (Medium)83.851.732.180.658.022.6
DeepSeek-V4-Pro77.550.027.574.654.020.6
Claude Opus 4.878.853.325.474.660.014.6
GLM-5.262.516.745.858.218.040.2
Average75.642.932.772.047.524.5

The gap persists and is positive for all four models. Your concern was also justified. The average gap falls from 32.7 to 24.5 points, so statement problems did inflate the apparent difficulty by about a quarter. The gap shrinks for every model, by between 5.6 and 10.8 points. Section 4.4 now reports the result on clean problems next to the result on the full sample, and we removed the sentence that read lower clarity as a sign of compositional difficulty.

Questions about statement quality are common for benchmarks in this area. We ourselves removed invalid problems from IndustryOR and other benchmarks when preparing the evaluation sets (Table 3). That is why we used the strictest rule we could here: a single no from any of the three reviewers on any of the three dimensions removes the problem.

Response to Reviewer ZNkR (2/3)

W2 / Q2. How is Equation 9 checked, in particular with tied optima? Are acceptance witnesses retained?

The submitted version did not describe this step, and Appendix L.3 spoke of a change in the returned decisions, which does not rule out ties. The revision reports the evidence by type for all 79 Composite problems (Section 3.2, Appendix A.6).

  • 56 problems. The bridge enters only through constraints, so the model without the bridge is a relaxation of the composite model, and removing the bridge strictly improves the optimal value. Then no optimal solution of the relaxed model is feasible in the composite model, and the optimal sets differ whichever optimum the solver returns. The witness is the pair of optimal values with and without the bridge.
  • 3 problems. The bridge enters only through constraints, but the optimal value does not change. These satisfy Equation 9 through the feasible-region condition, which also does not depend on ties, but the bridge is not binding at the optimum. We flag these three explicitly.
  • 20 problems. The bridge also modifies the objective. Acceptance was based on a change in the returned optimal decisions. With multiple optima this is a necessary sign, not a proof, and we list it as a limitation.

On witnesses: for the 59 problems whose bridge enters only through constraints, our records keep the optimal values with and without the bridge. For the other 20, they keep the decisions returned by the two solves.

W3 / Q3. Does the gap persist after controlling formulation class, description length and semantic clarity?

Table 17 reports these controls on the clean problems. Restricting to clean problems fixes semantic clarity.

SettingAtomic nAtomic acc. (%)Composite nComposite acc. (%)Gap
All clean problems6772.05047.524.5
LP1678.12259.119.0
MILP2969.81437.532.3
Nonlinear565.0325.040.0
Other1772.11143.228.9
Length-matched1070.01040.030.0

Classes are assigned from the mathematical structure of the reference model. Other covers reference models solved by closed-form evaluation, dynamic programming, enumeration or policy search, without an LP or MIP formulation. The gap is present in every class. The Nonlinear row has only three Composite problems and we draw no conclusion from it. Composite problems are not concentrated in harder classes: 44% of the clean Composite problems are LP, against 24% of the clean Atomic problems. The length-matched row uses the ten clean Atomic and ten clean Composite problems closest in statement length (1,264 and 1,265 words on average). With ten problems on each side, it shows that the gap does not shrink, not how large it is.

On the weakness that the benefit of structural composition is not isolated, Appendix D.4 adds a same-parent comparison. Among the clean problems we manually identified 21 for which an uncoupled version can be built, and drew 10 with a fixed random seed. The uncoupled version keeps the same two parent skills and removes the bridge. Average accuracy is 70.0% on the uncoupled versions and 50.0% on the coupled versions, and the coupled version is harder for all four models (Table 8). The sample is small and we treat this as directional evidence.

Response to Reviewer ZNkR (3/3)

Q4. Equal-budget downstream comparisons

These are the right controls and the submitted version did not have them. The experiment is still running. We were worried about running out of time, so we are posting the rest of the response now and will add the results in this thread as soon as they are ready.

Q5. Does Table 2 apply Equation 9 to both recomposition conditions?

Yes. Both conditions build a bridge for each pair and accept a problem only if it passes the structural test of Equation 9. They differ only in how pairs are selected: uniformly at random in one case, by compatibility screening in the other. The submitted text was ambiguous on this point, because it mentioned bridges only for the Structured condition. Section 4.2 and Appendix C now state that both use the same acceptance criterion. The 16.6-point difference between Random (65.5%) and Structured (48.9%) therefore reflects compatibility-based pairing under the same criterion.

Q6. Figure 1 caption

Corrected. The caption, and the matching sentences in the introduction and Appendix B, now say that the four models are at or above 94%, in line with the 94.0% of DeepSeek-V4-Pro on MAMO EasyLP. Thank you for catching it.

Thank you again. Where the evidence is still incomplete, as for the bridge test on 20 problems and the downstream controls that are not finished, we have said so in the paper or in this response.

Thank you for the careful rechecking and renewed human evaluation. The average Atomic–Composite gap remains 24.5 points on the clean subset, and the same-parent comparison provides preliminary evidence of a coupling effect. I have adjusted my component scores accordingly.

I maintain my overall score because 20 of 79 composites rely on solver-returned decisions that cannot distinguish changed optimal sets from tied optima. For three other composites, please provide a witness that the feasible region changes. The same-parent study covers only ten pairs, and the equal-budget downstream comparisons are still pending. The PDF available to me matches the original submission, and Figure 2 remains difficult to read at normal zoom because the text is small and the panels cited in the caption are unlabeled. Releasing the clean subset or flagging its instances would also help benchmark users.

Reviewer_Qr46OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

OptArchitect synthesizes optimization problems by representing modeling mechanisms as executable skills and recursively composing them through explicit decision-level bridges. Generated instances undergo solver checks and independent reconstruction. The evaluation includes 116 atomic and 79 composite problems, four reasoning models, agent methods, fixed-source recomposition, size matching, human assessment, and downstream fine-tuning. Reported mean accuracy decreases from 74.8% on atomic problems to 46.5% on composites. We find the research direction worthwhile, but remain concerned about benchmark validity, the specification of the evaluation protocol, and attribution of the gains to recursive composition.

Critical strengths

1 A concrete approach to creating interacting modeling tasks

The skill representation, explicit bridge operations, and reuse of accepted composites provide a coherent approach to generating problems with interacting decisions (Section 3). This is a meaningful research direction for optimization-modeling evaluation.

2 Several relevant controls for the difficulty claim

The reported atomic-to-composite gap is accompanied by fixed-source recomposition and size-matched analyses (Tables 2 and 7). In the fixed-source experiment, mean accuracy decreases from 83.4% for atomic reconstruction to 49.0% for structured recomposition. These controls help address simple domain and size explanations for the observed difficulty.

Critical weaknesses

1 Uncertainty about semantic validity and the bridge criterion

We are concerned that the current evidence does not clearly separate increased modeling difficulty from residual ambiguity in the generated specifications. Appendix G reports lower completeness and clarity ratings for composites (88% and 82%) than for atomic problems (100% and 95%). These are reported rating averages, not estimates of the fraction of invalid tasks. Nevertheless, it is unclear whether the main accuracy gap persists on a subset independently confirmed to be complete and unambiguous.

The operational bridge test also needs clarification. Equation (9) concerns projected feasible regions or optimal-solution sets, while Appendix L.3 discusses changes in returned decision signatures. When multiple optima exist, different returned decisions need not imply different optimal sets. We would appreciate a precise account of the implemented test and its treatment of degeneracy and tolerances, since verified decision coupling is central to the method.

2 Unclear evaluation denominators and aggregation procedures

We have difficulty reconciling some reported values with the stated protocol. Section 4.1 gives 116 atomic problems, and Table 1 describes OptArchitect scores as means over three decoding runs. Under equally weighted binary scoring, Opt-Verifier’s 48.1% does not appear consistent with an integer total over 348 evaluations: 167 correct gives 48.0% after rounding, while 168 gives 48.3%. Similarly, Appendix E describes five candidates per composite problem; if Table 10 uses one such batch over 79 problems, its Pass@5 values would need to correspond to integer success counts out of 79. The reported values are not readily reconciled with that interpretation.

Additional repetitions, different subsets, or aggregation stages may explain these values, but are not clearly specified. Table 11 also uses both per-transfer and candidate-level descriptions. These could explain why overall success exceeds the problem-correctness gate rate for two baselines, but the denominators and aggregation stages need to be distinguished explicitly. We would appreciate per-run counts and gate records to assess the affected comparisons. This concern does not by itself negate the main atomic-to-composite difficulty trend.

3 Limited isolation of the recursive and component contributions

The existing controls support the usefulness of structured recomposition, but it remains unclear how much recursive reuse contributes beyond composing the same mechanisms in one step. A comparison using the same atomic mechanisms, comparable problem size, and matched generation budgets would help distinguish these explanations.

The atomic interventions in Table 13 also need explanation: removing Matchmaker or Bridge lowers atomic success from 65% to 30% or 20%, respectively, although the described atomic path uses a single skill. We are unsure what changes in these variants. Finally, the downstream mixture of 1K OR-Instruct and 1K synthesized examples is promising, but the contribution of recursive data is difficult to isolate without matched Qwen3-8B controls using 2K OR-Instruct or 1K OR-Instruct plus 1K atomic examples. These uncertainties limit our assessment of the distinctive methodological contribution.

Questions for authors

1 Benchmark and verification validity. Does the main difficulty gap remain on a subset independently confirmed to be complete and unambiguous? How is Equation (9) implemented in practice, and how are multiple optima, alternative defensible formulations, and numerical tolerances handled?

2 Evaluation protocol. Could the authors specify the effective sample sizes, repetitions, and aggregation formulas for Tables 1 and 10, including the 48.1% atomic score and Pass@5 results? Please also distinguish candidate-level from per-transfer gate counts in Table 11. Per-run integer counts and representative logs would help reconcile these results.

3 Component attribution. What exactly changes in the atomic Matchmaker and Bridge ablations? Are there matched comparisons of recursive versus flat composition and downstream training with recursive versus atomic or OR-Instruct-only data? Clarifying existing controls, or providing these comparisons, could materially affect the assessment of the proposed contributions.

5 comments

Response to Reviewer Qr46 (1/4)

Thank you for the detailed review. You separated what was unclear from what was wrong, and you wrote the ethics note with care. We are grateful for both.

We should say first how the errors arose. The experiments were run about four months ago. Results were computed and then copied into the tables by hand, and we did not check the tables against the counts afterwards. Mistakes entered in these intermediate steps. This is our fault and we accept the criticism. After reading the reviews we rederived every reported percentage from integer counts and redid the human evaluation. This took a long time, which is why our response comes late. We hope you will still have time to look at it.

Table numbers below refer to the revised PDF. The submitted Tables 10, 11, 12 and 13 are now Tables 13, 14, 15 and 18. The Structured Recomposition average that you quote as 49.0 is 48.9 after recomputation.

W1 (a). Does the gap persist on problems confirmed complete and unambiguous?

You are right that the submitted figures were averages over ratings and not estimates of the share of invalid problems. The new evaluation was designed to answer at the level of problems. Three reviewers each rated the same 80 Atomic and 60 Composite problems (Appendix G). A problem is clean only if all three answer yes on correctness, completeness and clarity. Our clarity criterion also rejects statements that are unambiguous but hard to read, so the clean problems are a subset of the unambiguous ones. 67 of 80 Atomic and 50 of 60 Composite problems are clean, so about 16% and 17% of the rated problems fail the check.

On the clean problems the average gap is 24.5 points, against 32.7 points on all rated problems. It is positive for all four models: 22.6, 20.6, 14.6 and 40.2 points (Table 16). The gap persists, and about a quarter of it was due to problems that fail the reviewer check. We rewrote Section 4.4 and Appendix G accordingly and removed the sentence that read lower clarity as a sign of compositional difficulty.

Table 17 adds two controls on the clean problems. By formulation class, the gap is 19.0 points for LP, 32.3 for MILP and 28.9 for problems without an LP or MIP formulation. The Nonlinear class has only three Composite problems and we draw no conclusion from it. By statement length, on the ten clean Atomic and ten clean Composite problems closest in length (1,264 and 1,265 words on average), accuracy is 70.0% versus 40.0%.

Questions about statement quality are common for benchmarks in this area. We ourselves removed invalid problems from IndustryOR and other benchmarks when preparing the evaluation sets (Table 3). That is why we used the strictest rule we could to answer your question.

Response to Reviewer Qr46 (2/4)

W1 (b). The implemented bridge test

Your reading is exact. Equation 9 is about feasible regions and optimal sets, while Appendix L.3 described a comparison of returned decisions, and the two are not equivalent when optima are tied. The revision reports what was actually checked for each of the 79 Composite problems (Section 3.2, Appendix A.6).

Bridge typeProblemsEvidenceIndependent of ties
Constraints only, and removing the bridge strictly improves the optimal value56The model without the bridge is a relaxation. A strictly better value means none of its optimal solutions is feasible in the composite model, so the optimal sets differ.Yes
Constraints only, optimal value unchanged3The feasible region differs, which is the first condition of Equation 9. The bridge is not binding at the optimum.Yes, but the coupling is weak
Bridge also modifies the objective20The returned optimal decisions change.No

For the 20 problems in the last row we do not claim more than the evidence supports. A change in the returned decisions is a necessary sign, not a proof that the optimal sets differ. We now list this as a limitation, and L.3 has been reworded.

On tolerances: 10−410^{-4} for reference self-consistency, 1% for the reconstruction gate, and 5% for model evaluation (Appendix A.6). Alternative defensible formulations are handled by the reconstruction gate. If a reconstruction supports a materially different answer under a defensible reading, the candidate is rejected as ambiguous.

W2 (a). Opt-Verifier and the 348 evaluations

You are right. 48.1% cannot come from 348 equally weighted binary evaluations. The correct value is 167/348 = 48.0%. When rechecking this method we also corrected IndustryOR from 66.2 to 66.3, OptiMath from 51.7 to 51.9, and its Drop from 26.6 to 26.5.

The aggregation is now stated: accuracy is the total number of correct answers over three runs, divided by three times the number of problems. The new Table 10 gives per-run counts for all agent-based methods.

MethodSetRun 1Run 2Run 3TotalEvaluationsAccuracy (%)
CoEAtomic61666218934854.3
CoEComposite2724247523731.6
Opt-VerifierAtomic58525716734848.0
Opt-VerifierComposite1619165123721.5
OptiTreeAtomic65606418934854.3
OptiTreeComposite1317154523719.0
AlphaOptAtomic46514714434841.4
AlphaOptComposite2521236923729.1

All three runs cover all 116 Atomic and all 79 Composite problems, so the effective sample size is the same in every run.

Response to Reviewer Qr46 (3/4)

W2 (b). Pass@5

Your reading of the protocol is the correct one. With one batch of five candidates over 79 problems, Pass@5 must be an integer over 79, and the submitted values were not. The revision states the estimator (Eq. 15). For problem ii, let cic_i be the number of correct candidates among five. Then

Pass@k=179∑i=179[1−(5−cik)/(5k)].\text{Pass@}k=\frac{1}{79}\sum_{i=1}^{79}\left[1-\binom{5-c_i}{k}\Big/\binom{5}{k}\right].

Table 12 reports the distribution of cic_i, and every entry of Table 13 follows from it.

Modelci=0c_i=012345Pass@1Pass@3Pass@5
GPT-5.5 (Medium)25841437220/395 = 55.763.854/79 = 68.4
DeepSeek-V4-Pro261010240220/395 = 55.761.953/79 = 67.1
Claude Opus 4.8231101440230/395 = 58.265.356/79 = 70.9
GLM-5.25511124665/395 = 16.524.724/79 = 30.4

All three columns use the same five candidates per problem, which are sampled separately from the three decoding runs of Table 1.

W2 (c). Per-transfer and candidate-level counts in the submitted Table 11

The submitted caption mixed the two levels and gave no denominators, so the table could not be checked. Table 14 now reports counts. Each strategy runs 100 transfers with at most three attempts per transfer. The two gate rates are computed over attempts, and the success rate over transfers.

MethodTransfersAttemptsProblem CorrectnessExecutable-Model CorrectnessSuccessful transfers
Direct Generation10029318 (6%)44 (15%)7 (7%)
Summary Prompting10028932 (11%)78 (27%)12 (12%)
OptArchitect-Atomic100217178 (82%)191 (88%)65 (65%)
OptArchitect-Composite100232167 (72%)181 (78%)52 (52%)

This explains the apparent contradiction you pointed out. For Direct Generation, 18 of 293 attempts pass the problem-correctness gate and 7 of 100 transfers succeed. The two rates have different denominators. The two gate columns cover two of the five acceptance criteria. An attempt must also pass independent reconstruction, the leakage check and the non-triviality check, which is why there are fewer successful transfers than gate passes.

OpenReview does not let us attach files at this stage. If per-attempt gate records or sample logs would help, we will post them in a follow-up comment.

Thank you for noting that these reporting problems do not by themselves negate the main trend. We agree that they had to be fixed before the trend could be trusted.

Reviewer_kJvVOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces OptArchitect which generates optimization problems by combining reusable modeling mechanisms and checking them with a solver and independent reconstruction. Across four models, accuracy falls from 74.8% on atomic problems to 46.5% on composite problems. Controlled comparisons suggest that interactions between mechanisms contribute to this difficulty. The authors also report modest fine-tuning gains using the generated data.

Critical strengths
  1. The requirement that a bridge change feasible or optimal decisions is a good design choice. It gives a concrete criterion for checking whether the mechanisms actually interact, which is central to generating meaningful composite problems.
  2. I found the closed-world and size-matched comparisons useful. The accuracy gap remains when the mechanisms come from the same source pool and when model sizes are similar. This makes the claim that composition adds modeling difficulty more convincing.
Critical weaknesses
  1. Some of the measured difficulty may come from unclear problem statements. Human reviewers give composite problems lower ratings for clarity and completeness than atomic problems. The paper does not show whether composite problems remain harder than atomic problems after excluding those that human reviewers flagged as incorrect, incomplete, or ambiguous.
  2. The paper needs to explain the structural verification step more concretely. It states that removing a bridge must change the feasible region or optimal solution set, but does not clearly describe how this is checked, particularly when multiple optimal solutions exist.
Questions for authors
  1. Among human reviewed problems, do composite problems still have lower model accuracy than atomic problems when both groups are restricted to examples judged correct, complete, and unambiguous?
  2. When checking whether bridge removal changes optimal decisions, how do you distinguish a genuine change from the solver returning different solutions that are equally optimal under both models?
  3. Could you compare independent and decision-coupled versions built from the same parent skills, with similar model sizes and specification quality? This would help isolate the contribution of the bridge itself to modeling difficulty.
  4. Could you compare the mixed training data with equally sized OR-Instruct-only dataset (like 1K OR-Instruct examples + 1K OptArchitect examples vs 2K OR-Instruct examples) under the same training setup?
4 comments

Response to Reviewer kJvV (1/2)

Thank you for the careful and constructive review. Each of your four questions points to evidence the paper was missing, and the same-parent comparison in Q3 is one we had not thought of running. After reading the reviews we rechecked every number in the paper and redid the human evaluation. This took a long time, which is why our response comes late. We hope you will still have time to look at it. Table numbers below refer to the revised PDF.

W1 / Q1. Are Composite problems still harder once problems judged incorrect, incomplete or ambiguous are excluded?

The submitted version did not show this, and it needed to be shown.

We redid the human evaluation on a larger sample. Three reviewers each rated the same 80 Atomic and 60 Composite problems under a written rubric (Appendix G). We call a problem clean when all three reviewers answer yes on correctness, completeness and clarity. Our clarity criterion also rejects statements that are unambiguous but hard to read, so the clean problems are a subset of the unambiguous ones. 67 Atomic and 50 Composite problems are clean.

ModelAtomic, all rated (80)Composite, all rated (60)GapAtomic, clean (67)Composite, clean (50)Gap
GPT-5.5 (Medium)83.851.732.180.658.022.6
DeepSeek-V4-Pro77.550.027.574.654.020.6
Claude Opus 4.878.853.325.474.660.014.6
GLM-5.262.516.745.858.218.040.2
Average75.642.932.772.047.524.5

So the answer is yes, with a qualification that we now state in the paper. The gap shrinks from 32.7 to 24.5 points. As you suspected, part of the measured difficulty did come from problems that fail the reviewer check, about a quarter of it. The remaining three quarters are still there on the clean problems, for all four models (Table 16, Section 4.4, Appendix G.4). The submitted version read the lower clarity of Composite problems as a sign of compositional difficulty; we removed that sentence.

Some context, which we do not offer as an excuse. Questions about statement quality are common for benchmarks in this area. When preparing our own evaluation sets we removed 5 of the 100 IndustryOR problems and 8 of the 211 MAMO ComplexLP problems as invalid, and dropped missing or ambiguous OptiMath instances (Table 3). Experts also often disagree on whether a given statement is clear, and a flagged statement is not always unusable: one of our two reasons for a clarity "no" is that the statement is hard to read although it has only one reading. For this reason we chose the strictest rule we could, where a single "no" from any of the three reviewers on any of the three dimensions removes the problem, and we report the gap under that rule.

Response to Reviewer kJvV (2/2)

W2 / Q2. How is the bridge test checked, and how is a genuine change told apart from a different but equally optimal solution?

The submitted version did not describe this step, and Appendix L.3 spoke of a change in the "returned decision signature", which is not enough when optima are tied. You are right on both points.

We went through all 79 Composite problems again and now report the evidence by type (Section 3.2, Appendix A.6).

  • In 59 problems the bridge enters only through constraints, so the model without the bridge is a relaxation of the composite model.
    • In 56 of them, removing the bridge strictly improves the optimal value. Then no optimal solution of the relaxed model is feasible in the composite model, since it would otherwise attain the same value there. The optimal sets therefore differ, whichever optimum the solver returns.
    • In 3 of them, the optimal value does not change. The bridge changes the feasible region, which satisfies Equation 9 and also does not depend on ties, but it is not binding at the optimum. We say this explicitly because the coupling in these three problems is weaker.
  • In 20 problems the bridge also modifies the objective. A change in the optimal value is not informative here, and acceptance was based on a change in the returned optimal decisions. With multiple optima this is a necessary sign, not a proof. Appendix A.6 lists it as a limitation, and L.3 has been reworded.

In short, the test is independent of ties for 59 of the 79 problems and is not for the other 20.

Q3. Independent versus decision-coupled versions from the same parent skills

We ran this comparison (Appendix D.4, Table 8). Among the clean problems we manually identified 21 for which an uncoupled version can be built, and drew 10 of them with a fixed random seed. The uncoupled version keeps the same two parent skills and removes the bridge, so the two mechanisms are solved side by side. A response is correct only if both sub-answers are correct.

ModelUncoupled (10)Coupled (10)Gap
GPT-5.5 (Medium)80.0 (8)60.0 (6)20.0
DeepSeek-V4-Pro70.0 (7)60.0 (6)10.0
Claude Opus 4.870.0 (7)60.0 (6)10.0
GLM-5.260.0 (6)20.0 (2)40.0
Average (40)70.0 (28)50.0 (20)20.0

The coupled version is harder for every model, by 20 points on average. Ten problems give a coarse estimate, and we treat the result as directional evidence that the bridge adds difficulty beyond having two mechanisms in one task.

Q4. Mixed training data versus an equally sized OR-Instruct-only set

This control is needed before the downstream gain can be attributed to the synthesized data, and the submitted version did not have it. The experiment is still running. We were worried about running out of time, so we are posting the rest of the response now and will add the results in this thread as soon as they are ready.

Thank you again. The bridge criterion and the two controls you found useful are where we put most of the work in this revision, and where the evidence is still incomplete we have said so in the paper.

Thank you for the rebuttal, would it be still possible to see the mixed training data experiments in Q4? Also I think it would be better to highlight the changes text/tables in the revised manuscript.

Thank you for the follow-up and for your interest in this experiment.

Q4. The equal-budget runs (2K OR-Instruct versus 1K OR-Instruct + 1K OptArchitect, with the same Qwen3-8B setup) are still in progress. We will do our best to post the full table in this thread tomorrow.

Highlighting changes. Thank you for the suggestion. The final revised manuscript will show all changes from the submitted version in blue, including the new tables in Appendices D.4, E, F.1 and G.

We are grateful for your engagement throughout the discussion, and we hope these comments help clarify the work.

Reviewer_9EJbOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

OptArchitect represents optimization-modeling mechanisms as executable skills, composes compatible skills through decision-level bridges, and promotes accepted composites for further composition. Generated problems undergo reference execution and independent reconstruction from their descriptions. The paper studies benchmark difficulty across recursive depths, includes closed-world and size-matched comparisons, and evaluates synthesized data for downstream training.

Critical strengths
  1. Composing executable modeling mechanisms through decision-relevant bridges is a meaningful alternative to concatenating problem descriptions. Promoting composites for further composition gives the synthesis process a reusable structural unit.

  2. The closed-world IndustryOR study and solver-size matching address important confounds in measuring compositional difficulty. These controls are well motivated and could yield useful evidence if the reporting and problem-validity issues are resolved.

Critical weaknesses
  1. Several numerical results are incompatible with the stated counting protocols. Appendix E says that five candidates are sampled for each of the 79 Composite problems and that the standard Pass@k estimator is used. With five candidates and k=5, each problem contributes either zero or one, so aggregate Pass@5 must be an integer divided by 79. None of the reported 68.6%, 66.8%, 70.1%, and 30.8% in Table 10 rounds to an attainable value under this protocol. For example, 54/79 is 68.35% and 55/79 is 69.62%. Appendix G.1 likewise specifies three reviewers, each providing 30 binary ratings per set and dimension, followed by averaging reviewer percentages. This gives an integer numerator over 90; values such as 95.0% and 94.3% in Table 12 cannot arise as stated. Overlap between reviewers' samples does not change that arithmetic. An unreported aggregation protocol could explain these discrepancies, but it must be specified and supported by the original counts.

  2. Residual ambiguity remains a competing explanation for benchmark difficulty. Even taking Table 12's reported rates at face value, Composite problems receive lower clarity and completeness ratings than Atomic problems: 82% versus 95%, and 88% versus 100%. A difficult but underspecified problem does not provide the same evidence of compositional reasoning difficulty as an unambiguous problem. Solver-size matching does not address this confound. The acceptance threshold for independent reconstruction is also unclear. Appendix M.2 accepts 52.46 against a reference of 52.40, whereas the specified tolerances concern reference self-consistency and external-model evaluation, not clearly the reconstruction gate. The paper should explain adjudication of incomplete or ambiguous accepted problems and establish whether the principal gap remains on the unambiguous subset.

Questions for authors
  1. Can the per-problem five-candidate success counts and the original binary reviewer counts reconcile Tables 10 and 12 with their stated protocols?

  2. What threshold and adjudication procedure govern independent-reconstruction disagreements, and does the difficulty gap remain on problems judged complete and unambiguous?

3 comments

Response to Reviewer 9EJb (1/3)

Thank you for checking our numbers against the stated protocols. Both arithmetic observations are correct.

We should say first how the errors arose. The experiments were run about four months ago. Results were computed and then copied into the tables by hand, and we did not check the tables against the counts afterwards. Mistakes entered in these intermediate steps. This is our fault and we accept the criticism. After reading your review we went back and rederived every reported percentage from integer counts, and redid the human evaluation. This manual work is the reason our response comes so late. We hope you will still have time to look at it.

Table numbers below refer to the revised PDF. The submitted Table 10 is now Table 13, and the submitted Table 12 is now Table 15.

W1 (a). Pass@5 in the submitted Table 10

You are right. With five candidates and k=5k=5, Pass@5 must be an integer over 79, and 68.6, 66.8, 70.1 and 30.8 are not.

Appendix E now states the estimator (Eq. 15). For problem ii, let cic_i be the number of correct candidates among five. Then

Pass@k=179∑i=179[1−(5−cik)/(5k)],\text{Pass@}k=\frac{1}{79}\sum_{i=1}^{79}\left[1-\binom{5-c_i}{k}\Big/\binom{5}{k}\right],

so Pass@5 is the fraction of problems with ci≥1c_i\ge 1, and Pass@1 is ∑ici/395\sum_i c_i/395. The new Table 12 gives the distribution of cic_i:

Modelci=0c_i=012345ProblemsCorrect candidates
GPT-5.5 (Medium)2584143779220
DeepSeek-V4-Pro26101024079220
Claude Opus 4.823110144079230
GLM-5.2551112467965

Every entry of Table 13 follows from these counts:

ModelPass@1Pass@3Pass@5
GPT-5.5 (Medium)220/395 = 55.750.4/79 = 63.854/79 = 68.4
DeepSeek-V4-Pro220/395 = 55.748.9/79 = 61.953/79 = 67.1
Claude Opus 4.8230/395 = 58.251.6/79 = 65.356/79 = 70.9
GLM-5.265/395 = 16.519.5/79 = 24.724/79 = 30.4

All three columns use the same five candidates per problem, which are sampled separately from the three decoding runs of Table 1. The conclusion of this analysis does not change: more samples help every model, and Pass@5 on Composite stays below single-sample accuracy on Atomic. OpenReview does not let us attach files at this stage. If the per-problem values of cic_i would be useful, we will post them in a follow-up comment.

Response to Reviewer 9EJb (2/3)

W1 (b). Human evaluation in the submitted Table 12

You are right here as well. Three reviewers with 30 binary ratings each give an integer over 90, and 95.0% and 94.3% are not of that form.

This evaluation was done long ago, and we could not trace how the reported figures were derived. We prefer not to offer an explanation we cannot verify. We redid this part of the paper instead.

Protocol (Appendix G.1). Three reviewers with operations research backgrounds and at least three years of optimization-modeling experience. The sample has 80 Atomic problems drawn at random from the 116, and 60 Composite problems drawn from the 79 with stratification by depth (28 at D1, 21 at D2, 11 at D3). Every reviewer rates all 140 problems, in randomized order.

Criteria (Appendix G.2). Each dimension is rated yes or no. A reviewer answers yes only if the condition holds.

DimensionCondition for yes
Problem correctnessThe statement is logically consistent and agrees with the data.
Modeling completenessVariables, objective and constraints are all specified; no value or rule has to be assumed.
Semantic clarityThe statement has a single reasonable reading and can be followed in one careful reading. A no is recorded when a second defensible reading exists that changes the feasible region or the objective, or when the statement is unambiguous but hard to read.
Practical plausibilityThe decision context and operational relationships are realistic.
Solution accuracyThe certified answer is the quantity asked for, in the stated units.

Aggregation. Each value is k/nk/n, with kk the number of yes ratings and n=240n=240 for Atomic, n=180n=180 for Composite. Since all reviewers rate the same problems, k/nk/n equals the mean of the three reviewer percentages.

Counts behind the new Table 15.

SetDimensionR-AR-BR-Ckknnk/nk/n (%)
AtomicCorrectness78787923524097.9
Completeness79808023924099.6
Clarity70757622124092.1
Plausibility78788023624098.3
Solution accuracy77777723124096.3
CompositeCorrectness57576017418096.7
Completeness55585817118095.0
Clarity52545616218090.0
Plausibility58586017618097.8
Solution accuracy57575917318096.1

W2 (a). Residual ambiguity as an explanation for difficulty

We agree that this is the main threat to the difficulty claim, and the submitted version did not deal with it. It also read the lower clarity and completeness of Composite problems as a sign of compositional difficulty. That reading was not justified and we removed it.

We now test the alternative directly. We call a problem clean when all three reviewers answer yes on correctness, completeness and clarity. The clarity criterion also rejects statements that are unambiguous but hard to read, so the clean problems are a subset of the unambiguous ones. 67 of the 80 Atomic problems and 50 of the 60 Composite problems are clean.

Table 16 repeats the comparison on these problems (correct answers in parentheses):

ModelAtomic, all rated (80)Composite, all rated (60)GapAtomic, clean (67)Composite, clean (50)Gap
GPT-5.5 (Medium)83.8 (67)51.7 (31)32.180.6 (54)58.0 (29)22.6
DeepSeek-V4-Pro77.5 (62)50.0 (30)27.574.6 (50)54.0 (27)20.6
Claude Opus 4.878.8 (63)53.3 (32)25.474.6 (50)60.0 (30)14.6
GLM-5.262.5 (50)16.7 (10)45.858.2 (39)18.0 (9)40.2
Average75.642.932.772.047.524.5

Two things follow, and we report both. First, your concern was justified. The gap falls from 32.7 to 24.5 points, so problems that fail the reviewer check account for about a quarter of the gap on the rated sample. On the ten Composite problems that are not clean, the four models are correct in only 8 of 40 cases. Second, the gap does not go away. On the clean problems it is positive for every model, between 14.6 and 40.2 points.

Section 4.4 and Appendix G.3 and G.4 have been rewritten along these lines.

Response to Reviewer 9EJb (3/3)

W2 (b). Reconstruction threshold and the M.2 example

You are right that the submitted version gave tolerances for reference self-consistency (10−410^{-4}) and for model evaluation (5%), but none for the reconstruction gate. Appendix A.6 now states the rule. A reconstruction is accepted only if (i) it selects the same discrete decisions as the reference, or, when optima are tied, decisions that are also optimal in the reference model; (ii) its continuous decisions agree up to the search tolerance; and (iii) its objective value is within 1% of the reference. A reconstruction with different discrete decisions is rejected whatever the objective gap.

The M.2 example needs a precise account. The certified answer stored in the frozen problem package is 52.40 dollars per week. It was obtained by searching the order quantity on a grid with step 0.72 vials, and the best grid point is 105.84 vials per cycle. The reconstruction selects the same single-lot policy and the same 8-week cycle, orders 106 whole vials, and obtains 52.46. If we evaluate the reference formulation at 106 vials, we also get 52.46. So the two formulations agree, and the 0.11% difference comes from the step of the reference search. Put plainly, the stored reference value is slightly below the optimum of its own formulation. The difference is inside the 1% reconstruction tolerance and far below the 5% evaluation tolerance, so it cannot change a correctness judgment. We kept the frozen answer, as our audit rule requires, and M.2 now explains the difference.

W2 (c). Adjudication of incomplete or ambiguous accepted problems

During synthesis, a candidate is rejected as ambiguous when its reconstruction supports a materially different answer under a defensible reading. It is not forced to agree with the reference (Section 3.3), and disagreements are classified by a neutral adjudication step (Appendix A.6).

The human evaluation shows that this filter is not perfect: 13 of the 80 rated Atomic problems and 10 of the 60 rated Composite problems are not clean. We plan to release the benchmark and code, and problems that fail the reviewer check will not be included in the release.

Q1. Can the per-problem five-candidate counts and the original reviewer counts reconcile Tables 10 and 12?

For Pass@k, yes. The distribution of per-problem counts in Table 12 reproduces every entry of Table 13, as shown under W1 (a). For the human evaluation, we could not recover the original rating sheets, so we cannot reconcile the submitted table. We replaced it with a new evaluation and report its counts under W1 (b).

Q2. What threshold and adjudication govern reconstruction disagreements, and does the gap remain on complete and unambiguous problems?

The threshold and rule are given under W2 (b). On clean problems the average gap is 24.5 points, against 32.7 points on all rated problems, and it is positive for every model.

Thank you again for the care you put into this review. The soundness concerns were deserved. We have tried to make every number checkable, and we are happy to provide anything further that would help.

Comments on the paper

1 comment

General response to all reviewers

We thank all four reviewers for the time and care they put into this submission. We answer every reviewer and every question separately below, and we do not avoid any of them. We sincerely hope you will have time to read the individual responses. Here we only summarize the points that several reviewers raised. The tables and counts are in the individual responses.

1. The numerical errors. We apologize. Several reported numbers could not arise under the protocols we described, and the reviewers were right to point this out. These were careless mistakes on our side. They came from how we worked: results passed between coauthors, intermediate files saved in different places, tables copied by hand, and versions mixed up. We take this very seriously and accept all the criticism. We do not want to make excuses. We went back through every number the reviewers questioned. What could be verified from the records was verified and is now reported with integer counts and the aggregation formula. What could not be verified was redone, in particular the whole human evaluation.

2. The clarity criterion. Disagreement about clarity is common in this area. Experts often judge the same operations research problem differently, and existing benchmarks have been questioned on this point in earlier work. In the first evaluation the experts judged clarity by the standard of mathematical modeling competition problems, which is very demanding, and a statement that was hard for a person to read also counted as unclear. The submitted paper did not say this. It only described clarity as admitting a single unambiguous interpretation, which was too vague, and that is our omission. In the revision we keep the emphasis on semantics: a statement fails if it has a second defensible reading or if it is hard to read. We no longer require the competition standard, and the rubric is now written out in Appendix G. Because the criterion and the sample both changed, the old and new ratings should not be compared directly.

3. Is the gap still there on clean problems? All reviewers asked this, and we tested it. Three reviewers rated the same 80 Atomic and 60 Composite problems, and we call a problem clean only if all three approve its correctness, completeness and clarity. On the clean problems the average Atomic-to-Composite gap is 24.5 points, against 32.7 points on all rated problems. So part of the gap, about a quarter, did come from statement quality, as the reviewers suspected, and the rest remains for all four models. The gap also remains within each formulation class and after matching statement length.

4. The bridge test under multiple optima. The reviewers are right that different returned decisions do not imply different optimal sets. We re-examined all 79 Composite problems. For 59 of them the bridge enters only through constraints, and the evidence does not depend on which optimum the solver returns. For the other 20 the test relied on returned decisions. We describe the check in Section 3.2 and Appendix A.6 and state the 20 problems as a limitation.

5. Independent versus coupled versions. We added this experiment. For ten problems we built an uncoupled version from the same two parent skills. Average accuracy is 70.0% on the uncoupled versions and 50.0% on the coupled ones, and the coupled version is harder for all four models. The sample is small and we present it as directional evidence.

6. Table 2 and Equation 9. Both recomposition conditions apply the same structural acceptance criterion and differ only in how pairs are selected. The submitted text was unclear about this and has been corrected.

What is still missing. One additional experiment is not finished: the equal-budget downstream comparison asked for by three reviewers. We are sorry about this. It is running, and we will post the results in the individual threads before the deadline.

We are also sorry to be posting this late. Time was short and the rechecking and the new experiments took a great deal of work. We value this opportunity very much, and we know the first version gave the reviewers good reason to doubt our numbers. Thank you for the chance to correct them.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.