- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper introduces CO-Agent, a compact language-model agent for combinatorial optimization. It combines cost-aware planner training (CARP), verifier-guided distillation from teacher repairs on student-visited states (CO-VGOPD), and repository-level skill evolution (AUSE). The agent generates, executes, verifies, and repairs solver code within a computational budget. Experiments on CO-Bench, FrontierCO, and HeuriGym report improved solution quality and feasibility, supported by component ablations, quality–cost comparisons, and continual skill-evolution studies.
Critical strengths
-
Repository-level skill evaluation addresses a meaningful failure mode. AUSE evaluates whether inserting a skill improves the deployed agent under natural retrieval, rather than assuming that standalone skill quality translates into system-level gains. The comparisons with and without RMU admission, together with the negative-transfer analysis, provide useful evidence for this design.
-
The experiments examine several aspects of the system. Beyond final performance, the paper includes training-objective comparisons, component ablations, budget generalization, and previous-task retention. These studies help distinguish the intended roles of planning, corrective distillation, and skill evolution.
Critical weaknesses
-
The evaluation protocol does not sufficiently establish fairness or generalization. The paper states that training and repository evolution use training/development data, but does not specify the task and instance counts, split construction, or separation of problem families across training, skill evolution, admission, and final testing. This is particularly important for the claimed generalization on HeuriGym. Table 1 also does not identify the backbone models, inference budgets, or solver access used by the agent baselines. Comparing a trained agent with a learned skill repository against larger direct-generation models demonstrates system capability, but does not isolate the contribution of the proposed learning mechanisms. Stronger controls would use the same trained generator, tools, initial repository, and explicit inference limits.
-
The central cost-efficiency claims are difficult to assess from the reported measurements. Appendix A.1 defines cost as a weighted combination of normalized model calls, tokens, solver time, execution time, and wall-clock time, but does not provide the numerical weights and reference quantities needed to reproduce the comparison. Absolute resource measurements and sensitivity to these choices are also needed, since the components partly overlap. Excluding the teacher from deployment cost is reasonable, but training and repository-evolution costs should be reported separately to establish the practical trade-off. There is also an unresolved reporting discrepancy: Table 3 gives the full system a gap of 4.7 and FR of 95.2%, whereas Table 1 reports 4.7/92.0% on CO-Bench and 26.4/95.2% on FrontierCO. The setting behind Table 3 is not identified.
Questions for authors
-
Could you provide the task and instance counts for each training, development, skill-evolution, admission, and test split? Are evaluation problem families unseen during both model training and repository construction?
-
Which backbone, token budget, candidate budget, solver limits, and tool access are used for each agent baseline? Do the gains remain when these methods share the same trained generator and initial skill repository?
-
What numerical weights and normalization constants define the reported cost? Could you provide absolute token usage, model calls, solver time, and end-to-end latency, alongside separate training and repository-construction costs?
-
Which benchmark and configuration does Table 3 evaluate, and how should its full-system numbers be reconciled with Table 1?
-
How are admission neighborhoods sampled and refreshed during repeated skill evolution? Are the reported RMU and NTR measured on separate evaluation tasks? A 95% bootstrap interval for one candidate does not by itself address repeated adaptive selection on the same admission set.
2 comments
Comment 4 Which benchmark and configuration does Table 3 evaluate, and how should its full-system numbers be reconciled with Table 1?
Response: We thank the reviewer for this question. We conduct ablation studies separately on all three benchmarks. In the original manuscript, Table 3 only presented ablation results from CO-Bench and contained an entry error where the FR value was incorrectly taken from FrontierCO. We have reprocessed the ablation experiments and revised Table 3 to report metrics averaged across the three benchmarks. Meanwhile, we have added the unified trajectory cost metric to Table 1 to clearly demonstrate the cost advantage of CO-Agent.
Modifications: Table 1, Table 3.
Comment 5 How are admission neighborhoods sampled and refreshed during repeated skill evolution? Are the reported RMU and NTR measured on separate evaluation tasks? A 95% bootstrap interval for one candidate does not by itself address repeated adaptive selection on the same admission set.
Response: We thank the reviewer for raising this important question regarding the reliability of repository-level admission in AUSE. We appreciate this observation and agree that the original manuscript did not sufficiently clarify the admission protocol and the statistical scope of the bootstrap evaluation.
(1) Admission-neighborhood construction and refreshing. In the current implementation, AUSE is designed as a two-stage process consisting of local skill evolution followed by repository-level admission. During local evolution, multiple rounds of parent selection, offspring generation, and fitness evaluation are performed on the evolution neighborhood. Once this process terminates, a single locally selected candidate is submitted for repository-level admission. Admission is therefore performed once at the end of each skill-evolution process, rather than repeatedly for intermediate offspring. The current implementation does not include an additional dynamic refreshing mechanism for admission neighborhoods, which we now clarify in the revised manuscript.
(2) RMU and NTR evaluation. Both RMU and NTR are computed on the same admission evaluation task set through paired counterfactual evaluations. Specifically, we execute the original repository and the augmented repository on identical problem instances under the same natural-retrieval mechanism and execution protocol. This paired design controls for differences in problem difficulty and isolates the marginal effect of adding the candidate skill. More importantly, it evaluates the behavior of the complete agent after repository augmentation, rather than relying solely on the candidate skill's standalone fitness. This distinction is central to AUSE: a skill is admitted based on its actual contribution to the deployed agent, accounting for both retrieval behavior and potential negative transfer.
(3) Repeated adaptive selection and bootstrap interpretation. We agree with the reviewer that a 95% bootstrap interval for an individual candidate does not, by itself, provide statistical guarantees against repeated adaptive selection on a shared admission set. In our current implementation, the bootstrap interval is intended to quantify the uncertainty of the paired RMU estimate for the candidate being evaluated, rather than to claim control over arbitrary repeated adaptive testing. We acknowledge that repeatedly reusing the same admission tasks across multiple skill-evolution processes could introduce a risk of overfitting to the admission set. Addressing this issue through more rigorous admission-task refreshing and statistical control across repeated admission decisions is an important direction for future work.
We sincerely appreciate the reviewer for highlighting this issue, which has helped us improve the transparency of AUSE and identify an important direction for its further development.
Thank you again for your suggestion.
We thank the reviewer for these thoughtful and detailed questions covering data partitioning, baseline fairness, cost accounting, table interpretation, and the statistical protocol of our skill evolution pipeline. Below we address each comment point-by-point.
Comment 1 Could you provide the task and instance counts for each training, development, skill-evolution, admission, and test split? Are evaluation problem families unseen during both model training and repository construction?
Response: We thank the reviewer for this careful question regarding data splits and potential leakage. We answer from two separate perspectives, covering model training and AUSE skill evolution: (1) For model training of CARP and CO-VGOPD, we adopt three combinatorial optimization benchmarks: CO-Bench (36 problem families, 6482 instances), FrontierCO (8 problem families, 416 instances), and HeuriGym (9 problem families, 218 instances). Splitting is performed at the problem-family level following an 8:2 partition. Within the 80% of problem families allocated to the training pool, we randomly sample 80% of their instances for training and the remaining 20% for validation. The held-out 20% of problem families together with all their associated instances constitute the test set. (2) For the AUSE skill-evolution pipeline, we initialize the skill repository with 46 clustered optimization skills derived from OptSkills. All skill evolution and admission procedures are carried out using the training splits of the three benchmarks, while final evaluation is performed on the test split.
Crucially, all problem families and instances in the test set remain unseen during both model training and skill repository construction, which effectively prevents data leakage.
Modifications: Appendix C.1, Dataset partitions.
Comment 2 Which backbone, token budget, candidate budget, solver limits, and tool access are used for each agent baseline? Do the gains remain when these methods share the same trained generator and initial skill repository?
Response: Thank you for this valuable question. We report token consumption and solving turns for CO-Agent and all baselines in Appendix B.1 (Cost reporting), with visualizations in Figure 6–8. We further add a controlled ablation in Appendix C.6 and Table 6 on the CO-Bench test set, where we adopt the same trained Qwen3-4B generator and initial skill repository. Although using our skill repository or our generator individually improves the baselines, these variants still lag behind the full CO-Agent. This confirms our gains are not only from the generator or static skills, but rely on the CARP planner and AUSE evolution mechanism. Moreover, we have explicitly documented the backbone, token budget, candidate budget, solver limits and tool access rules for each agent baseline in Appendix C.1 to ensure comparison fairness. We hope these additions address your concern, and we would appreciate your favorable consideration.
Modifications: Appendix B.1, Appendix C.6, Table 6.
Comment 3 What numerical weights and normalization constants define the reported cost? Could you provide absolute token usage, model calls, solver time, and end-to-end latency, alongside separate training and repository-construction costs?
Response: Thank you for this question. We have supplemented the absolute raw cost statistics for CO-Agent and all baselines in Appendix B.1 (Cost reporting), including token consumption, model call counts, solver runtime and end-to-end wall-clock latency, visualized in Figure 6–8. The numerical aggregation weights and fixed normalization constants for the unified trajectory cost are also explicitly stated there in: αm=0.30,αt=0.30,αs=0.25,αe=0.10,αw=0.05, with reference constants Cmodelref=120.0, Ctokenref=3500.0, Csolverref=320.0, Cexecref=100.0, Cwallref=1050.0. The one-time separate training of CARP, CO-VGOPD and the AUSE skill repository construction costs approximately 120 GPU-hours on RTX 4090, excluding the inference evaluation overhead on held-out test instances. We hope these details fully clarify our cost definition and resource consumption. We would appreciate your favorable consideration if the response addresses your concern.
Modifications: Appendix B.1