- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper proposes AutoMIP, an autoresearch framework for mixed-integer linear and nonlinear programming. It combines a persistent idea pool with an algorithm tree to manage research directions and reusable experimental states. The agent can refine the current approach or switch to an alternative direction by restoring a compatible historical state. The paper evaluates AutoMIP on MIPLIB Open and Hard instances and MINLPLib, comparing success rates, runtime, token consumption, and final solution quality. Ablation experiments examine the contributions of idea pooling and algorithm tree search.
Critical strengths
-
The persistent idea pool is an interesting design. It preserves untested research hypotheses independently of the current experiment, allowing the agent to explore alternative directions without losing previous ideas.
-
The algorithm tree explicitly connects research directions with compatible historical states and executable artifacts. This allows the agent to resume experiments from earlier states rather than always continuing from the latest implementation.
Critical weaknesses
-
The reported efficiency statistics contain internal inconsistencies. In Table 1, EvoX's overall mean token consumption is 2,297,150, exceeding every reported interval mean, whose maximum is 1,586,613. If the intervals form disjoint groups covering all runs, the overall mean must be a weighted average and cannot exceed every group mean. Table 2 reports an AutoMIP mean runtime of 2,035 seconds. However, the 30 successful runtimes in Table 9 sum to 27,398 seconds. Counting the remaining 30 runs at the stated 3,600-second deadline gives an overall mean of 2,256.63 seconds. The reported value therefore requires a different aggregation or failure-handling rule that is not clearly explained. These inconsistencies directly affect the claimed runtime and token advantages. The authors should clarify the aggregation rules and reconcile the reported statistics with the per-instance results.
-
The claims of new best-known solutions are not sufficiently supported by the reported experimental evidence. The paper claims new best solutions on 31/60 MIPLIB instances and 52/60 MINLPLib instances. However, the per-instance tables mainly report success labels, runtime, and tokens, without providing the reference objectives, final objectives, or improvement magnitudes. Improving a supplied starting incumbent does not necessarily establish a new best-known benchmark result. The paper also does not clearly specify the success criterion for MINLPLib. Although an independent verification procedure is described, the corresponding per-instance numerical verification results are not reported. Without these results, it is difficult to verify the claimed new records and final-objective wins.
Questions for authors
-
Could the authors clarify the exact aggregation rules for runtime and token consumption, including how unsuccessful runs are handled? In particular, how are EvoX's overall token mean in Table 1 and AutoMIP's 2,035-second mean runtime in Table 2 obtained from the detailed results?
-
Could the authors provide the reference objective, final verified objective, and improvement magnitude for each successful MIPLIB Open and MINLPLib instance? How are new best-known solutions distinguished from improvements over starting incumbents, and what exact success criterion is used for MINLPLib?
5 comments
[W2, Q2] Reference and final verified objectives, success criterion for MINLPLib, and improvements over starting incumbents.
Reference and final verified objectives. We have added the public reference objective and final verified objective for each instance in the Open and MINLPLib per-instance tables (Appendices E and J). The public reference objectives were released as recently as January 26, 2026, and September 2026, respectively. For these minimization instances, we define the absolute improvement as . Our criterion is: a result is counted as a new best-known solution only when its independently verified objective strictly improves the public best-known objective. Therefore, improving only the agent's own starting incumbent does not meet this criterion. Under this criterion, we confirm that the 31 Open and 52 MINLPLib successes are improvements over their respective public references; for example, 'allcolor58' improves from 258 to 221 (), and 'z26' from -1198 to -1201 ().
Success criterion for MINLPLib. MINLPLib uses the same success criterion as Open in MIPLib: within different intervals in 43,200 seconds, a research agent finds and independently verifies a feasible solution strictly better than the public best-known objective. If no feasible reference solution is known, finding and verifying a feasible solution counts as success. We have clarified this point in the revised manuscript, with the corresponding changes highlighted in blue on page 7. For your reference, also, we have added the corresponding MINLPLib task template to Appendix B, highlighted in blue on page 21.
Improvements over starting incumbents. Following your suggestion, we also have added results in Tables 23-25 with the corresponding analysis in Appendix K to quantify the improvements achieved over the starting incumbents on the Open, Hard, and MINLPLib instances. The results demonstrate that AutoMIP can substantially improve existing feasible solutions, with maximum relative improvements of 26.78% (‘allcolor58’), 8.01% (‘neos-5196530-nuhaka’), and 7.13% (‘acopf_caseactivsg70k_qcqp’), respectively. Notably, improvements exceeding 5% are achieved on 8 Open, 3 Hard, and 5 MINLPLib instances. These findings provide further evidence that AutoMIP can go beyond simply retaining the initial solutions and discover effective solving strategies that yield better objective values.
Reminder
Dear Reviewer SWrL,
As the rebuttal period is approaching its deadline tomorrow, we would greatly appreciate it if you could take a moment to review our responses and the revised manuscript. Highlights in our responses will make it easier for you to locate and review the corresponding revisions in the paper. We appreciate any further feedback, so that we can still reply them before the discussion closes. Thank you very much for your time and consideration!
Response to Reviewer SWrL
Dear Reviewer SWrL,
We sincerely thank the reviewer for recognizing the novel design of AutoMIP, particularly the persistent idea pool and algorithm tree for managing research directions and reusable experimental states. We acknowledge the concerns regarding the paper's presentation and soundness and address them point by point below.
[W1, Q1] Clarification of the inconsistent results.
Thank you for identifying the aggregation inconsistency. Indeed, the time intervals form disjoint groups covering all runs. As we stated in Metrics paragraph on page 7, w/s column counts wins/solved runs in each interval, whereas time and token columns average over the runs assigned to that interval, including unsuccessful runs. The columns under Total average over all 60 instances. For an unsuccessful run, we use its recorded elapsed wall-clock time and token consumption rather than substituting fixed budget values.
This aggregation rule illustrates the number of instances solved within each time interval, the number of wins when multiple methods solve the same instance, and thus the number of instances that remain unsolved after the full long-term run. It also successfully provides a comprehensive comparison of different autoresearch from perspectives of time and token in different time intervals and over the entire execution period.
EvoX token consumption in Table 1. The previous inconsistency resulted from a miscalculation. We have corrected EvoX's 6--12 h token mean in Table 1 from 1,244,128 to 2,774,485. The previous value did not include the token consumption of all unsuccessful runs assigned to that interval, which caused the inconsistency. For EvoX, we recorded the solved 6, 5, 3, and 1 runs in the four disjoint groups, respectively. The final group however contains the other 45 unsuccessful runs, which we should have included in the average token calculation. Consequently, the corrected interval mean give
$\frac{6\times (306{,}763)+5\times (720{,}454)+3\times (1{,}586{,}613)+46\times (2{,}774{,}485)}{60} \approx 2{,}297{,}150$
which now agrees with the reported Total token mean. We have updated the results in our paper.
AutoMIP mean runtime in hard set. We apologize for the misleading results. The maximum budget for the Hard instance is 3,600 seconds. An autoresearch agent iteratively tries different methods for attaining the best-known objective per Hard instance, as we stated in the experimental setting. However, in our original paper, we recorded the time point for the agent finding the latest improved objective values, for each of the 30 unsuccessful runs (i.e., those that did not reach the target objective within the allocated budget), since it fails to find a better objective value till 3,600s. These recorded time points are less than 3,600s. Consequently, these time points lead to a smaller average value (2,035) over the total 60 instances than otherwise 2,256.63 seconds if we assign exactly 3,600 seconds to each of the 30 unsuccessful runs.
We agree that assigning the full 3,600-second budget to each unsuccessful run provides a more consistent and transparent measure of the total computational budget. Accordingly, we have updated the reported AutoMIP mean runtime from 2,035 to 2,257 (rounded from 2,256.63) seconds in Table 2 of the revised paper, calculated using the 27,398 seconds accumulated across the 30 successful runs and the full 3,600-second budget for each of the remaining 30 unsuccessful runs. We thank the reviewer for pointing out this issue. Inspired by your comment, we will release all computation logs upon acceptance to facilitate reproducibility and verification of the reported results.