- Rating
- 1
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
EvoORSkill studies how reusable optimization modeling skills can evolve while preserving their intended mathematical meaning. It represents versioned skills as hyperedges over shared mathematical objects, traces dependencies affected by revisions, and applies specification, requirement, and retention checks before admitting updates. Historical replay improves revision strategies by reusing compatible evaluation records. Experiments across six datasets and frozen-library transfer to OptMATH report improvements in solution accuracy and token efficiency.
Critical strengths
- The paper identifies an important challenge: revising one modeling component can affect other components through shared mathematical objects, even when the resulting code remains executable.
- Connecting skill evolution with dependency-aware mathematical validation is a promising direction for reliable optimization modeling agents.
- The component ablations and frozen-library transfer experiment provide useful evidence supporting the evaluated framework.
Critical weaknesses
- The central validation procedures are insufficiently specified. Section 3 defines specification, requirement, and retention checks, but does not explain their implementations in sufficient detail. It remains unclear how requirement records are constructed and confirmed, how executable model fragments are compared with declared mathematical relations, and how retained tasks are selected. These procedures are central to the claimed reliability benefits.
- The presentation and missing experimental details substantially limit reproducibility. Section 3 moves between task-level models, skill collections, and revision strategies using dense notation without a complete worked example. The supplied manuscript has no implementation appendix resolving these ambiguities. Task splits, initial-library construction, prompts, revision budgets, solver settings, and checker configurations are insufficiently documented. Regenerating aggregate tables does not establish reproducibility of the underlying agent runs.
- The computational cost of revision checking is not sufficiently characterized. The paper reports aggregate library-building tokens and ablation totals, but does not separately report the costs of specification, requirement, and retention checks or clarify which operations invoke an LLM. Solver-based checks also require runtime accounting beyond token usage. Differences between ablation totals cannot isolate checking overhead because removing checks changes candidate admission and subsequent evolution. Consequently, the computational trade-off underlying the reliability gains remains unclear.
- Cross-problem dependencies and conflicts are insufficiently analyzed. The method primarily explains dependencies among components assembled for a single task. Retention checks and frozen-library transfer provide cross-task evidence, but do not explain how revisions to shared skills propagate across different problem families or how conflicting applicability conditions are handled. This limits the conclusions about the broader evolution of a reusable skill library.
- The ablations do not fully isolate the proposed mechanisms. Removing cross-skill checking supports checking related components, but does not establish the specific benefit of a hypergraph over an ordinary dependency graph or full rechecking. Specification and requirement checks are removed together, leaving their individual contributions unresolved. Repeated-run variability is also not reported.
Questions for authors
- How are requirement records constructed and confirmed? What roles do humans, LLMs, and reference specifications play, and how are the three admission checks implemented?
- Can the authors provide a complete revision example showing the original and revised code and mathematical relations, affected dependencies, detected counterexamples, and admission decision?
- Please break down token usage, solver runtime, and invocation counts for the three checks. Which operations use an LLM, and which costs are included in the reported build-token totals?
- How are cross-problem effects and conflicting applicability conditions handled when a shared skill is revised? Can the authors provide a concrete example?
- Can the authors provide reproducible experimental configurations and clarify the benefit of hypergraph-based tracing relative to an ordinary dependency graph or full rechecking?