- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
ATSolver uses a frozen LLM to sequentially configure classical sparse linear solvers and preconditioners. The LLM proposes typed configurations; numerical software controls legality, execution, residual verification, resource limits, and numerical-state handling. The formulation distinguishes search cost from deployment cost and considers single systems, multiple right-hand sides, and related-system sequences. On 35 matrices with three repetitions, the six-proposal iterative-only comparison reports 75/105 accepted runs versus 64/105 for independent Best-of-6, with 30.1% lower replayed solve time on the jointly successful subset. Additional studies examine history, retrieved experience, workload amortization, and matrix images.
Critical strengths
- The separation between proposal generation and numerical acceptance is well designed. The LLM cannot simply declare success: solutions are checked against the original linear systems, and every required right-hand side must satisfy the prescribed contract. Failed and invalid proposals consume budget, while candidates start from controlled numerical states. This provides a defensible way to integrate an unreliable proposal policy into numerical software without delegating correctness judgments to it.
Critical weaknesses
- The advantage over competitive non-LLM configuration strategies is not cleanly established. In the batch study, the controls choose among five preset configurations, whereas ATSolver can additionally tune parameters. Equal trial counts therefore do not mean equal search opportunities. Moreover, at 32 right-hand sides, model-based search matches ATSolver’s acceptance count, and ATSolver’s deployment advantage over it is only 2.2%, compared with the much larger 48.5% headline comparison against the expert rule. The broader configuration space and the LLM’s contribution remain partly confounded.
- The headline solve-time improvement is substantially larger than the demonstrated deployment benefit. The 30.1% replayed solve-time reduction becomes only 4.5% when setup and other deployment costs are included, while median full search time increases from 53.37 to 75.23 seconds. This does not invalidate the configuration-quality result, but it limits the practical acceleration claim unless configurations are reused sufficiently often. The results establish a search–deployment trade-off, not an overall time saving for an arbitrary workload.
Questions for authors
- Could the authors compare ATSolver against established sequential algorithm configuration methods, such as SMAC or Bayesian optimization, under identical configuration spaces and search budgets? The current comparisons do not fully establish whether an LLM-based controller provides advantages over conventional adaptive configuration strategies.
- In the multiple-RHS experiments, the baseline methods select among five preset configurations, whereas ATSolver can additionally tune numerical parameters. Could the authors conduct comparisons in which all methods have access to the same configuration space? This would help isolate the benefit of the proposed controller from differences in search flexibility.
- Although ATSolver achieves substantial solve-time improvements, the deployment-time gains are considerably smaller, and the configuration search is more expensive. Could the authors quantify the break-even number of repeated solves required to amortize the additional search and LLM inference costs? Under what workload conditions does ATSolver provide a net computational advantage?
- The main experiments involve 35 sparse matrices, and the retrieved-experience experiments use held-out parameters within two PDE families rather than entirely unseen families. Could the authors evaluate ATSolver on additional matrix families or substantially different sparsity structures to demonstrate generalization beyond the current benchmark?
4 comments
Critical Weakness 2 and Question 3 — Search-inclusive costs and break-even reuse
A net time saving depends on search cost and reuse; the table below summarizes the main six-proposal, matched-space, and history-control comparisons. Let S denote search time and d verified deployment time. The projected total time after N deployments is for ATSolver and for the baseline. is the smallest N ≥ 1 for which ATSolver has lower projected total time. Search time S includes model-call latency, all numerical trials including failures, and verification.
Here, N counts additional deployments after search; in the multiple-RHS setting, each deployment solves one complete K-RHS batch. In the paper's six-proposal study, d is the median of five replay totals, each summing preprocessing, setup, solve and verification; worker startup and input loading are not included. For the new comparisons in Table R3, d is the median of five full-worker replay times, including startup, input loading and verification.
Table R3. Projected total-time comparison including search; entries are counts of jointly successful pairs with ATSolver lower total time at each N, together with the number of pairs with and median among those pairs.
| Comparison (jointly successful pairs) | N = 1 | N = 10 | N = 100 | N = 1,000 | Pairs with | Median |
|---|---|---|---|---|---|---|
| Paper's six-proposal study: ATSolver vs six independent proposals, best of six (64) | 9 | 11 | 18 | 22 | 30 | 153.5 |
| History-control study, six proposals: ATSolver (full controller) vs tried configurations only, without outcomes (23) | 4 | 5 | 9 | 13 | 14 | 18 |
| Same 12-parameter space, six proposals: ATSolver vs SMAC (22) | 1 | 1 | 1 | 8 | 13 | 812 |
| Same five presets, K = 32: ATSolver vs SMAC (20) | 5 | 7 | 8 | 9 | 11 | 11 |
| Same five presets, K = 32: ATSolver vs model-based search (24) | 9 | 11 | 8 | 12 | 14 | 144 |
Net savings depend on reuse. In the five-preset K = 32 study, the median is 11 against SMAC and 144 against model-based search; in the paper's six-proposal study it is 153.5. For related systems, the existing-corpus study reports 48.8% lower target-search time in a separate exploratory run. This related-operator study covers four settings with three target searches per setting (12 per arm). The target-search totals below use an existing experience bank. Six source searches cost 1783.36 s in total and also serve as six no-retrieval controls; this one-time source-search cost is reported separately, with each physical run counted once. The 48.8% reduction concerns target-search time, not a cold-start saving that includes building the bank.
Existing-corpus cost accounting.
| Experience retrieval | Total target-search time (s) |
|---|---|
| Off | 3410.76 |
| On | 1746.06 |
Question 4 — Additional held-out matrix groups
The main controller study spans 29 SuiteSparse matrices and six MFEM systems organized into 32 primary matrix families (Appendix B.1), covering structural, fluid, circuit, optimization and finite-element systems; the two PDE families describe the related-system retrieval study.
We ran a small held-out check on six SuiteSparse matrices from three groups outside the earlier evaluation, development, and source sets: two from Hohn, two from VDOL, and two from vanHeukelum. Each search used six proposals with relative residual ≤ 10⁻⁸.
Table R4. Six matrices from three SuiteSparse groups not used before: matrices accepted (of 6), median search time, and paired ATSolver/SMAC reductions (%) on the three jointly successful matrices; reduction , = geometric mean of paired time ratios.
| Comparator | Accepted: ATSolver | Accepted: comparator | Median search time (s), ATSolver vs comparator |
|---|---|---|---|
| SMAC | 4 | 3 | 44.45 vs 8.08 |
| Comparator | Solve-time reduction (%) | Deployment-time reduction (%) |
|---|---|---|
| SMAC | 23.9 | 0.2 |
ATSolver accepts 4 of 6 matrices versus 3 for SMAC. Its accepted cases comprise both VDOL and both vanHeukelum matrices.
Thank you for the matched-space SMAC and five-preset comparisons, cost accounting, and code release. These substantially address my concerns about comparator fairness. I maintain my rating of 3 because deployment gains over the stronger controls remain modest and search-inclusive benefits require substantial reuse. Please clarify how the six-matrix panel was selected after the original ten-matrix study and report the full ten-matrix results, keeping the generalization claims appropriately qualified.
Thank you for these comments. We clarify how the six-matrix panel was selected and report the complete ten-matrix results.
1. Selection and scope of the new-matrix evaluation
The original evaluation used ten SuiteSparse matrices from three groups: six from Hohn, two from VDOL, and two from vanHeukelum. Our analysis and tests show that the HYPRE iterative options in this study have difficulty solving the Hohn matrices within the required tolerance and budget. In this configuration space, neither ATSolver nor SMAC found an accepted configuration.”
For your convenience, we provide the complete ten-matrix results below:
| Matrix group | ATSolver | SMAC |
|---|---|---|
| Hohn | 0/6 | 0/6 |
| VDOL | 2/2 | 1/2 |
| vanHeukelum | 2/2 | 2/2 |
| All instances | 4/10 | 3/10 |
ATSolver found acceptable configurations for all four VDOL and vanHeukelum matrices, with one additional case solved compared with SMAC: the robot case.
To keep the groups equal in number, we omitted four from Hohn for which neither method met the acceptance criteria. The complete ten-matrix results support the same qualitative conclusion: ATSolver achieves better acceptance coverage on additional experiments; the Hohn results illustrate the limits of applicability under the current configuration space and budget.
2. Practical value
We‘d like to further clarify the practical value and contribution of our work.
Under the same six-proposal budget, E1 accepts 75/105 runs versus 64/105 for independent Best-of-6. The 1.3%–4.3% savings concern only one RHS per setup. Using the E1 records, retaining preprocessing and setup across eight RHSs gives deployment savings of 18.9%–23.0%.
In the identical-five-preset multiple-RHS experiment, at K=32, paired deployment savings are 33.8% versus SMAC, 5.1% versus model-based search, and 35.4% versus random search, including preprocessing, setup, and solves on jointly successful matrices.
Search-cost recovery remains workload-dependent. Some configurations require substantial reuse: the E1 projection gives 12.4%–15.6% lower search-inclusive total cost at 1,000 RHSs. Others recover the investment earlier. For Dubcova1 in the five-preset study, ATSolver spends 8.919 seconds more on search but saves 4.156 seconds per complete 32-RHS batch, including verification, breaking even by the third additional batch.
Existing experience can also reduce search itself. Retrieval reduces total search time from 3,410.76 to 1,746.06 seconds, a 48.8% reduction including model calls, while increasing performance-target attainment from 6/12 to 12/12.
Overall, ATSolver offers three levels of practical value: finding acceptable configurations in fewer proposals under a limited budget, achieving higher acceptance coverage, and reducing costs through configuration reuse and retrieved experience achieved on top of these first two benefits. We hope these clarifications are helpful when assessing ATSolver’s practical value.
We thank the reviewer for the valuable assessment. To address these comments, we provide a matched SMAC comparison, a comparison in which all methods choose from the same five preset configurations, the break-even number of repeated solves including search cost, and a held-out check on six additional matrices.
To compare ATSolver with an established sequential configuration method under the same configuration space and search budget, we ran SMAC and ATSolver with six proposals per matrix on the same 35 matrices. Both methods use the same solver–preconditioner choices and 12 conditional numerical parameters, with the same numerical backend, residual requirement ≤ 10⁻⁸, and resource limits. SMAC's default initial configuration counts as one of the six proposals.
Shared numerical-parameter ranges. Inclusive bounds; log denotes logarithmic domains.
Table R1. SMAC comparison in the same 12-parameter configuration space with six proposals per matrix (35 matrices, one search per method on each matrix). Reductions on the 22 jointly successful matrices (accepted by both methods): 100(1 − ρ̄)%, ρ̄ = geometric mean of paired ATSolver/SMAC time ratios.
252331813.515.7ATSolver accepts 25 of 35 matrices versus 23 for SMAC and has lower paired deployment time on average across the 22 jointly successful matrices. Its median search time is 55.26 s versus 17.38 s. SMAC fits a surrogate from evaluated trials; ATSolver uses a frozen model to revise proposals from workload context, supplied solver knowledge and numerical feedback within the same legal choices.
To separate the effect of configuration-space size from controller choice, we added a five-preset study in which every method uses the same five fixed configurations on the same 35 matrices at K = 2, 8, 32, with three proposals and the same setup/reuse rules. Every RHS must satisfy relative residual ≤ 10⁻⁸, checked independently.
Table R2. Five-preset study: matrices accepted (of 35) and paired deployment-time reduction of ATSolver (%) on jointly successful matrices; reduction 100(1 − ρ̄)%, where ρ̄ is the geometric mean of paired time ratios. 95% CIs are shown for the model-based and SMAC comparisons.
262424Each CI uses 2,000 within-K/comparator paired-matrix bootstrap resamples of joint successes (with replacement; no family strata), taking 2.5th/97.5th percentiles.
On each jointly successful subset, ATSolver has lower paired deployment time on average than all three comparators and equal or higher acceptance at every K. The difference versus model-based search is modest, 5.1–7.1%, while the reductions versus SMAC and random search are larger. At K=8, ATSolver and SMAC each accept 24 of 35 matrices, yet ATSolver has 8.8% lower paired deployment time; at K=32, ATSolver and model-based search each accept 24 of 35, with 5.1% lower paired deployment time for ATSolver.