- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper proposes OptRoute, a framework for organizing and retrieving reusable formulation skills for optimization modeling. Skills are extracted from reference solutions and organized into an evolving tree, where paradigm nodes guide retrieval and leaf nodes store formulation knowledge. Retrieved skills are synthesized into a modeling plan for solver-code generation. Experiments on eight benchmarks and two backbones compare overall performance, memory organization, retrieval strategies, and memory scaling.
Critical strengths
The paper presents a clear framework for organizing and retrieving reusable optimization-modeling skills. Table 3 provides a useful controlled comparison by fixing the leaf skills and downstream generation pipeline while varying retrieval strategies. Improvements across eight benchmarks and two backbones further suggest that the approach is useful beyond a single model or problem setting.
Critical weaknesses
-
The contribution of structural evolution is not sufficiently separated from other changes. Table 2 jointly changes skill refinement, consolidation, and tree organization. Consequently, the improvement does not establish how much comes from evolving the hierarchy rather than improving the skill content. Table 3 controls the leaf skills, but does not compare against a stronger retrieval pipeline such as dense retrieval followed by LLM reranking. Given the existing work on hierarchical retrieval and dynamic tree memory discussed in Section 4.2, the paper needs clearer evidence of what the proposed organization–retrieval coupling adds.
-
The analysis does not yet establish the robustness or scope of the advantage. The macro-average improvement over the strongest agentic baseline is 3.0 percentage points with DeepSeek but only 0.9 with DouBao, and the main tables provide no uncertainty estimates. The scaling study increases both available knowledge and the evolved structure, so it does not isolate the benefit of tree evolution as memory grows. Beyond one illustrative case, there is little systematic analysis of retrieval failures, routing errors, or which formulation distinctions benefit most from the hierarchy. This limits the insight beyond showing an aggregate performance improvement.
Questions for authors
-
Could you separate the effects of structural evolution and skill-content updates? For example, compare an evolving tree with frozen leaf content against a fixed tree with enriched skills.
-
How does OptRoute compare with dense retrieval followed by LLM reranking under comparable inference budgets? What additional benefit does it provide over existing dynamic tree-memory approaches?
-
How stable are the gains across different construction samples and insertion orders? Uncertainty estimates would help assess the smaller improvements, particularly with DouBao.
-
Could you provide systematic retrieval-error analysis and comparisons at each memory scale using the same skills? This would clarify when hierarchical routing helps and whether the scaling gains come from better organization or additional knowledge.
I would consider raising my rating if the responses and supporting evidence address these concerns, particularly by isolating the benefit of structural evolution and demonstrating that the gains are robust.
3 comments
3. W2/Q3 — Repeated runs, construction samples, and insertion order
W2: The analysis does not yet establish the robustness or scope of the advantage. The macro-average improvement over the strongest agentic baseline is 3.0 percentage points with DeepSeek but only 0.9 with DouBao, and the main tables provide no uncertainty estimates. Q3: How stable are the gains across different construction samples and insertion orders? Uncertainty estimates would help assess the smaller improvements, particularly with DouBao.
Thank you for emphasizing the importance of robustness across runs, construction samples, and insertion orders.
- Repeated runs
We added three-run means and standard deviations for the main comparisons (revised Table 2). OptRoute achieves 79.3 ± 0.6 versus OptiTree 75.2 ± 0.7 on DeepSeek, and 79.2 ± 1.0 versus 77.5 ± 0.5 on DouBao. The DouBao advantage is smaller (The previously reported 0.9 was obtained from a single run. Across multiple independent runs, our method consistently outperformed OptiTree, with a substantial improvement in one run) .
- construction samples and insertion order
We also added construction sensitivity in Appendix C.4 (Table 8):
| Variation | Macro Pass@1 across the three settings |
|---|---|
| Construction sample seed | 79.2 / 81.3 / 78.4 |
| Skill insertion order | 79.2 / 79.8 / 75.0 |
The construction-sample results show a 2.9-point range. More importantly, we acknowledge a 4.8-point insertion-order range, including one weaker order. Inspection of that run suggests that some formulation-distinguishing skills were skipped as redundant, leading to retrieval of continuous-variable guidance where integer formulations were needed; MAMO-Complex fell to 59.5. This is a real limitation of sequential, LLM-guided memory construction. At the same time, the other tested order reaches 79.8. The revised Appendix C.4 reports these outcomes rather than hiding the adverse case.
4. W2/Q4 — Matched-skill scaling and retrieval-error analysis
W2: Beyond one illustrative case, there is little systematic analysis of retrieval failures, routing errors, or which formulation distinctions benefit most from the hierarchy. This limits the insight beyond showing an aggregate performance improvement. Q4: Could you provide systematic retrieval-error analysis and comparisons at each memory scale using the same skills This would clarify when hierarchical routing helps and whether the scaling gains come from better organization or additional knowledge.
Thank you for requesting a more controlled scaling comparison and clearer analysis of retrieval failures.
- Comparisons at each memory scale using the same skills
We expanded the memory-scaling study in revised Section 3.4 (Figure 3). At each construction scale, the retrieval methods have access to the same leaf-skill set and use the same downstream generator:
| Construction cases | OptRoute | Flat LLM | Dense |
|---|---|---|---|
| 100 | 76.3 | 74.7 | 76.8 |
| 200 | 78.1 | 76.2 | 75.9 |
| 400 | 78.3 | 73.8 | 72.4 |
| 800 | 79.2 | 76.9 | 76.6 |
OptRoute is not best at the smallest scale; its advantage emerges at the larger tested scales, but OptRoute improves from 76.3% to 79.2% macro accuracy as memory grows. In contrast, Flat LLM and Dense show no consistent improvement despite access to the same growing skill sets. This suggests that the gains are not simply due to more knowledge; effective skill organization and retrieval also play an important role.
- Error analysis
We also expanded Appendix C.5 (Error Analysis and Limitations) to distinguish three observed failure sources: missing formulation-family coverage, confusion between similar skills with different applicability conditions, and incorrect synthesis/code generation even after relevant retrieval.
The benchmarks provide neither gold retrieval labels nor standardized intermediate modeling traces, making these failure sources difficult to disentangle. For example, the LLM may sometimes recover missing fine-grained formulation details on its own, whereas similar omissions may lead to incorrect or unbounded models. We therefore provide a qualitative analysis rather than a fully quantified error decomposition.
Besides, revised Appendix C.3 (Figures 6–8) gives concrete examples of content refinement, restructuring, and cross-branch synthesis. We acknowledge that a fully quantified retrieval-versus-generation error decomposition remains open.
In summary, the revision directly addresses the reviewer’s two most actionable requests: (i) separate content from structure, and (ii) compare against LLM-reranked retrieval with matched skills and budget. We have also added repeated runs, sample/order sensitivity, matched-skill scaling, and a candid discussion of remaining failure modes.
We sincerely thank the reviewer for these valuable suggestions. If our responses have addressed your concerns, we would greatly appreciate your reconsideration of the score.
Response to Reviewer HwdR
We particularly appreciate the reviewer’s concrete suggestions and the indication that stronger evidence could change the assessment. We have prioritized directly isolating the proposed contribution, testing stronger retrieval baselines, and reporting robustness limitations transparently.
1. W1/Q1 — Isolating structural evolution from skill-content updates
Thank you for this constructive suggestion to disentangle structural evolution from skill-content updates.
We followed the suggested comparison and added a factor-isolating ablation in revised Section 3.3 (Table 3). The configurations use the same validated construction skills and downstream modeling pipeline:
The full model improves by 4.5 and 3.7 points over the two restricted variants, respectively. The particularly large difference on MAMO-Complex (88.0 versus 65.5 and 76.6) suggests that the joint design is valuable when closely related formulations must be distinguished. We agree this is evidence of complementarity, rather than a precise additive attribution to each component.
2. W1/Q2 — Stronger retrieval baselines and dynamic-memory novelty
Thank you for suggesting stronger retrieval baselines and asking us to clarify the contribution beyond existing dynamic memories.
We strengthened the controlled retrieval comparison in revised Section 3.3 (Table 4). BM25 and dense retrieval are each followed by LLM-based selection under a matched budget (the LLM selects at most three skills from the top 30 retrieved candidates). The leaf-skill memory and downstream generation pipeline are held fixed:
OptRoute exceeds Dense + LLM by 2.5 points and BM25 + LLM by 4.0 points. On IndustryOR, the gap to Dense + LLM is 5.5 points (73.0 versus 67.5). Since the skill set and downstream generator are held fixed, these comparisons more directly demonstrate the value of paradigm-guided routing rather than gains from access to additional modeling knowledge.
We also revised Section 4.2 and Appendix A.2 (Table 5) to avoid equating novelty with “dynamic trees.” Unlike OptiTree’s subproblem hierarchy or AlphaOPT’s insight taxonomy, OptRoute uses formulation-family paradigms as both the evolving organization units and the routing decisions for retrieving complete skill leaves. We recognize that prior work also updates hierarchical memories; the novelty claim is specifically about this formulation-aware organization–retrieval coupling.