W1. Outdated baselines. The main algorithmic baselines are classical QR/Cholesky, Blendenpik, and sparse-sketch LSQR, with core methods dating back to 2013 or earlier. The paper omits recent solvers it cites [1–3] and a closely related surrogate-based LS autotuning method [4]. These comparisons are needed to establish competitiveness with current solvers and LS-specific tuning approaches.
W2. Unconvincing advantage over TPE. Figure 4 shows only a modest final gain over TPE, without explicit held-out test results or variation across independent searches; Appendix C also notes that training and validation rankings can differ. Moreover, LLM evolution can change code implementations whereas TPE searches a fixed implementation. A controlled test-set comparison is needed to isolate the benefit of LLM search itself.
W3. Missing end-to-end cost evaluation. The search process for each task requires substantial time, along with LLM API fees and considerable CPU computation, yet the comparison focuses on the final solver runtime. Equal evaluation counts do not mean equal search time or compute budgets. Please compare LLM evolution and conventional tuning by wall-clock/compute budget and quantify how many downstream solves are needed to amortize discovery.
W4. Unclear novelty and limited LS-specific adaptation. Is the main novelty simply applying the existing ShinkaEvolve framework [5] to LS? Beyond an LS runtime/accuracy evaluator, an existing solver seed, and basic task statistics, I do not see a substantive LS-specific modification to candidate generation or the search procedure. Most discovered changes recombine established numerical techniques. The authors should clearly identify and justify the methodological contribution beyond this application.
W5. Narrow data distribution and limited OOD evaluation. Synthetic tasks vary size, conditioning, and accuracy but use the same generator: exponentially decaying singular values, normalized inputs, and fixed optimal residual norm (0.01). Figure 5 primarily tests other parameter settings within this generator, not genuinely different matrix distributions; one real-world dataset is insufficient to establish broad generalization. Please test different spectra, residual scales, and data distributions.
References
[1] Fast and forward stable randomized algorithms for linear least-squares problems.
[2] Fast randomized least-squares solvers can be just as accurate and stable as classical direct solvers.
[3] Adaptive LSQR preconditioning from one small sketch.
[4] Surrogate-based autotuning for randomized sketching algorithms in regression problems.
[5] ShinkaEvolve: Towards open-ended and sample-efficient program evolution.
Kind reminder: updated manuscript addressing the Reviews
Dear reviewer 9Fb6,
We again thank you for your time and feedback. Given that the ICLR discussion period will come to an end on November 18th, we kindly want to ask whether you could assess our rebuttal?
Furthermore, we want to again highlight the extensive revision of the paper and the accompanying additional experiments we did to address your review comments. We hope that these additions have addressed your concerns with respect to our paper.
Please let us know if any further questions may have come up, we would be happy to answer them.
Have a good week, The authors.