acceptodds
Under review as a conference paper at ICLR 2027

Large Matrix Model: A Pretrained Sparse Solver Across Equation Families

Abstract

Sparse linear solvers are traditionally specialized to the algebraic structure of their operators, and existing learned solvers largely inherit this specialization by training for individual matrices or within a single equation family. We ask whether sparse solving can instead admit a pretraining paradigm: can a single model learn reusable numerical priors that transfer across equation families and to previously unseen real-world operators? We introduce the Large Matrix Model (LMM), a pretrained neural–numerical solver that maps a sparse operator and its current residual to a correction subspace and is deployed with frozen weights. LMM combines residual-conditioned graph representations with node-wise gated Chebyshev directions, which overcome the rank ceiling of a shared polynomial dictionary, and couples the resulting subspace with restart-free incremental orthogonalization and an exact float64 minimal-residual projection. To evaluate cross-equation-family generalization, we construct a corpus of 240 equation families spanning five algebraic classes and 15 structured subfamilies, pretrain LMM on 80 families, and hold out the remaining 160 families by identity. A single frozen checkpoint converges on 1,590 of 1,600 systems from unseen equation families and, without adaptation, on 489 of 556 systems in a large SuiteSparse split disjoint from our smaller adaptation test. Under matched downstream data and compute, pretrained initialization improves convergence on the group-disjoint 200-system SuiteSparse test from 174 to 181 and reduces geometric-mean residual by relative to random initialization. These results provide evidence that reusable solver priors can be learned across heterogeneous equation families, moving learned sparse solvers from problem-specific training toward pretrained generalist numerical models.

Primary area:
applications to physical sciences (physics, chemistry, biology, etc.)
Keywords:
Sparse linear solvers, pretrained numerical methods, scientific machine learning, graph neural networks, mixture of experts, learned iterative methods, Krylov subspace methods, polynomial basis functions, Chebyshev basis, sparse matrices, numerical linear algebra, zero-shot generalization, solver acceleration, scientific computing

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_XurZOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The paper proposes a hybrid neural-numerical subspace solver for sparse linear systems. The approach uses residual-conditioned graph directions and an explicit polynomial numerical prior to form search directions. Incremental block orthogonalization reduces the computational cost of retraining directions across iterations without restart. Empirical studies on a synthetic benchmark and a subset of SuiteSparse problems show that the hybrid solver can generalize across held-out equation families, addressing an important problem when ML approaches applied in practice. However, it is not clear which components are responsible for the generalization capability, and the computational benefits have not been demonstrated against sufficiently competitive practical baselines.

Critical strengths
  • The research problem is important and well motivated.
  • The method is generally described clearly, with detailed information on architectural choices and the training objective.
  • The paper includes extensive evaluation and comparisons with several baselines.
Critical weaknesses
  • The benefit of the hybrid construction is not sufficiently isolated. LMM is close to G-RANS, but adds a hybrid graph–polynomial basis and an accumulated search space. Although the paper includes some component ablations, removing the polynomial branch also reduces 60 candidate directions. What happens if all 360 directions are graph-generated, or if the solver uses a polynomial-only construction? The current evidence does not fully distinguish the benefit of the hybrid combination from the benefit of a larger search space. This is important because learning reusable solver priors is the paper’s central claim.
  • The classical baselines are insufficient to establish a practical advantage. The comparisons primarily use unpreconditioned methods and Jacobi-GMRES. Stronger preconditioned baselines are needed to establish whether LMM justifies the coverage advantage over conventional solvers as well as the additional cost of neural-network inference. This limits the practical significance of the results.
Questions for authors
  • What defines an “equation family”? How different are the held-out families from the training families?
  • How does LMM compare with an ungated polynomial-only numerical solver using the same accumulation and projection procedure? Such a comparison would help establish how much of the observed generalization comes from learning.
  • Can LMM work with conventional preconditioners to further improve performance?
  • How do the results change under different residual tolerances and different wall-clock budgets?
Reviewer_3vUNOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes LMM, a pretrained solver for sparse linear systems. A graph network and a gated Chebyshev branch generate 360 correction directions per step. These are accumulated without restart by incremental block orthogonalization, and the update is an exact float64 minimal-residual projection. The model is pretrained on 80 synthetic equation families and tested on 160 held-out families and on SuiteSparse matrices, where it solves 1590/1600 and 489/556 systems at relative residual 1e-3.

Critical strengths
  1. The question (one frozen solver across equation families) is interesting, and the family-level holdout protocol is a useful idea.
  2. The update is a real least-squares projection, so the reported residuals are true residuals.
  3. The evaluation is careful in several ways: seed variance is reported, failures are counted, and a non-reproducible speedup is openly retracted.
Critical weaknesses
  1. No strong classical baseline. The comparison is with unpreconditioned GMRES/BiCGSTAB (CPU, 2.5 s budget), and Jacobi-GMRES. There is no ILU, AMG, or sparse direct solver. The matrices are small (20k–60k rows). A sparse direct solver would solve nearly all of them, to far tighter tolerance than 1e-3. The tolerance is also loose, and the time budget penalizes CPU baselines. The claim “coverage without knowing the class” is weak without these baselines.
  2. The held-out families are not truly out of distribution. The 80/160 split is made inside the same 15 subfamilies (6/5/5 families taken per subfamily for training), so test families are close relatives of training families (e.g., screened Poisson vs. Poisson). On the only split with distinct equations, Syn-OOD-S, LMM solves 148/200 and BiCGSTAB 150/200, and three seeds of LMM give 148/132/131.
  3. Baselines were adapted by the authors and are unequal. G-RANS, GNP and GMP are the authors’ multi-matrix adaptations of per-matrix or narrower methods. GNP has 10k and GMP 45k parameters, against 1.74M for LMM. GNP and GMP were not designed for this protocol, so these rows say little about the original methods.
  4. Pretraining benefit is weakly supported. The 181 vs. 174 result uses 200 systems from only 50 matrices (4 right-hand sides each, so strongly correlated), a single seed, and a scratch model that the authors say had not converged. This does not show that pretraining matters.
  5. Missing ablation and cost reporting. There is no “without graph branch” ablation, although the paper claims a hybrid design. The polynomial branch has the largest effect, and the gate effect is small (about 13 of 1600 systems). Training cost (data generation, GPU hours) and per-solve cost are not reported. Network forward time dominates (about 42 ms per action), and no end-to-end comparison with good classical solvers is given.
  6. Novelty is mostly combination. The pieces are a graph branch (G-RANS-style), Chebyshev probes with gating, and block orthogonalization updating. Proposition 1 is a trivial containment fact, and the “effective rank” evidence (Fig. 3) is measured at initialization, so it does not show that the extra rank is useful. IBO is compared only with full recomputation, which is a weak baseline (Arnoldi-type methods already do incremental updates).
  7. Name and scale are overstated. “Large Matrix Model” suggests scale, but the model has 1.7M parameters and the matrices have at most 60k rows. The training data is synthetic and author-generated. Training uses the exact solution x*, which real problems do not have.
Questions for authors
  1. How does LMM compare with ILU/AMG-preconditioned Krylov and a sparse direct solver, at tighter tolerances and on GPU?
  2. What are the training cost and the break-even number of solves?
  3. What is the result with the graph branch removed?
  4. Can you test on families that are truly outside the 15 subfamilies, and on larger matrices?
  5. Can you repeat the pretrained-vs-scratch comparison over several seeds and more matrices?
Reviewer_S7uzOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents the Large Matrix Model (LMM), a learned solver for sparse linear systems that is trained once and then used with frozen weights. At each step, a graph neural network looks at the matrix and the current residual and proposes a set of 360 search directions (300 from the graph network and 60 from Chebyshev polynomials of the matrix, each scaled by a learned per-node gate). The update itself is computed by an exact least-squares step over all directions generated so far, and an incremental QR update (IBO) keeps this cheap without restarting. The authors build a synthetic corpus of 240 equation families across five matrix classes, train LMM on 80 of them, and test it on the other 160 (10 systems per family, 1,600 in total). At a relative residual tolerance of 1e-3, LMM solves 1,590 of these 1,600 systems. Without any adaptation, it also solves 489 of 556 SuiteSparse systems.

Critical strengths
  • The paper studies an important problem. Most learned solvers are trained for one matrix or one PDE family, and understanding whether one frozen model can transfer across other equation families is a critical step forward.
  • The network only suggests search directions, and the update is a standard least-squares step over all directions so far (Section 2.1). As a result, the residual never increases between steps, and every reported solution is checked against the true residual.
  • The rank argument for the polynomial branch is simple and holds in practice. Without gating, 60 polynomial columns span at most 16 directions, and gating raises the measured rank to 48-60 (Proposition 1, Figure 3).
  • The ablations are retrained, and effects are reported in units of seed standard deviation (Table 3).
  • The paper includes results that do not favor it, for example BiCGSTAB matching LMM on Syn-OOD-S (150 vs. 148), MINRES and CG solving every system they apply to, and a routed-attention speedup that did not replicate (Appendix B.4).
  • The data pipeline is documented in detail, with fixed seeds, content hashes and a group-disjoint SuiteSparse split (Appendix B.1).
Critical weaknesses
  • The comparison with classical solvers favors LMM. Success requires reaching the tolerance within a 2.5 s wall-clock budget, but the classical methods are timed on CPU (Table 2), while LMM appears to run on H200 GPUs (Appendix B.2). The classical baselines are also unpreconditioned GMRES and BiCGSTAB, plus Jacobi-GMRES, even though the introduction itself names ILU and AMG as the standard choices. The GMRES restart length is not given. Many of the GMRES failures may be timeouts on slower hardware. The paper should compare on the same hardware, or by matrix-vector product count, and include ILU-preconditioned GMRES and AMG-preconditioned CG.

  • The tolerance of 1e-3 is loose for a linear solver. Most applications need much lower tolerance (say 1e-6). The SuiteSparse right-hand sides are stored in float32 (Appendix B.1), so tighter tolerances could not be tested on this data anyway. It is unclear whether LMM keeps its advantage, or converges at all, at the accuracies classical solvers usually target.

  • The cost per step is not reported in a hardware-independent way. Each action needs 15 Chebyshev products plus a product of A with all 360 new directions, so one LMM step costs several hundred sparse matrix-vector products. LMM also keeps every direction it generates, about 2,100 per solve on Syn-ID (Table 4), far more than a typical GMRES restart. Reporting matrix-vector products and memory per solve would show whether LMM is more efficient or spends more work per step.

  • The held-out families are closer to the training families than the framing suggests. The 80 training families are drawn from all 15 subfamilies. Hence, every held-out family shares its subfamily with at least some of the training families. For example, HP1 lists Poisson, screened Poisson and weighted Laplacian side by side. As far as I can tell, the held-out systems (Syn-OOD-L) also have the same size range as the in-distribution systems (Syn-ID). The test therefore does not check transfer to larger systems.

  • On Syn-OOD-S, where LMM scores lowest, it does not beat BiCGSTAB (solves 150 of 200 systems). Table 2 reports LMM at 148, but this is the best of three training seeds; the other two solve 132 and 131 (Table 8).

  • The matrices are small for a model referred to as Large Matrix Model. All test systems have 20k-60k rows and at most 2M nonzeros. The restart-free basis grows by 360 dense columns per action, so memory grows quickly for systems that need many actions or have millions of rows. The paper does not test larger systems.

  • The novelty over G-RANS is limited. G-RANS already learns residual-conditioned correction subspaces and shows cross-equation transfer (Table 1, Appendix D). LMM's additions are a wider basis, gated Chebyshev directions, a standard incremental QR update and a larger training corpus. Table 10 also shows that LMM with only k = 20 directions already solves 1,548/1,600 Syn-OOD-L systems, against 831 for G-RANS at the same width. Most of the gain therefore does not come from the wide basis and IBO, and the paper does not show what does.

  • The learned baselines are weakened by the adaptation. GNP and GMP are per-matrix methods in their original papers, and the authors retrain them across matrices (Appendix B.3). This is acknowledged, but a comparison with GNP under its original per-matrix protocol, including training time, would be more informative on SS-S.

Questions for authors
  1. On what hardware were LMM and the classical solvers timed? How do the results in Table 2 change if all methods run on the same device, or are compared by matrix-vector product count?
  2. How does LMM compare with ILU-preconditioned GMRES and AMG-preconditioned CG on SuiteSparse? What restart length was used for GMRES?
  3. How does LMM perform at tolerances of 1e-6 and 1e-8? Can you rebuild the SuiteSparse right-hand sides in float64 to test this?
  4. What are the matrix-vector product count, memory use and subspace size per solve for LMM compared with GMRES?
  5. How does LMM perform when a whole subfamily, or a whole matrix class, is held out from training?
  6. Can you report the mean and spread over the three seeds in Table 2, rather than only seed 43?
  7. Since LMM with k = 20 already solves 1,548/1,600 Syn-OOD-L systems, which parts of LMM account for the gap to G-RANS at the same width?
  8. How does LMM scale to systems with millions of rows, in time and in memory?
3 comments
AuthorsOpenReview ↗

Reply to Reviewer S7uz (Part 2 of 3)

  • Q5 — Whole-subfamily and whole-class exclusion.

    We performed genuine matched-budget retraining, rather than inference masks, while preserving the original tests. Removing all six NM1 training families leaves 100/100 withheld successes versus 100/100 for the control; the five superclass exclusions give:

    Removed superclassControl on withheld classClass-out on withheld classControl-minus-class-out rate95% family-cluster CI
    H+320/320320/3200.0000[0.0000, 0.0000]
    B320/320320/3200.0000[0.0000, 0.0000]
    N+320/320320/3200.0000[0.0000, 0.0000]
    H-310/320310/3200.0000[0.0000, 0.0000]
    N-309/320264/3200.1406[0.0500, 0.2469]

    Each class-out arm removes 16 training families and retains 64, using the original 60-epoch/37500-step budget and seed 43. N-minus exhibits a clear negative transfer result; the other four show no success-count drop, which does not establish equivalence or arbitrary unseen-class robustness.

  • Q6 — Means and spread over all three main-table seeds.

    The table below uses all three archived main checkpoints, with failed/nonfinite outcomes retained in success denominators. SD is the sample SD across seeds; the residual column averages the per-seed finite-outcome geometric means, using the same 1e-30 logarithm floor.

    TrackSystems/seedSuccesses, seeds 43 / 44 / 45Success-count mean ± SDResidual gmean across-seed mean ± SD
    Syn-OOD-S200148 / 132 / 131137.00 ± 9.543.464e-03 ± 4.06e-04
    Syn-ID800800 / 800 / 800800.00 ± 0.005.242e-04 ± 6.17e-06
    Syn-OOD-L16001590 / 1590 / 15841588.00 ± 3.465.370e-04 ± 5.70e-06
    SS-S200179 / 179 / 180179.33 ± 0.585.129e-04 ± 2.41e-05
    SS-L556489 / 491 / 490490.00 ± 1.001.304e-04 ± 6.21e-06

    These are the recovered paper-source runs: seed 43 used one H200 host and seeds 44/45 another, so the observed spread is not a pure training-seed-only uncertainty estimate. In particular, Syn-OOD-S instability is retained rather than presenting only the best seed.

AuthorsOpenReview ↗

Reply to Reviewer S7uz (Part 3 of 3)

  • Q7 — Which parts of LMM plausibly explain the G-RANS gap at k=20.

    Our working explanation is that LMM's training formulation better uses a fixed training budget for long inference trajectories: single-step correction-subspace supervision plus rollout replay. Replay supplies residual states visited by actual solves while keeping the per-example gradient horizon bounded; G-RANS progressive residual-loss training instead backpropagates through increasingly many correction steps.

    A progressive G-RANS curriculum that trains every horizon j=1,...,T with the same number of examples/updates per stage performs 1 + 2 + ... + T = T(T+1)/2 = O(T²) correction evaluations. This conditional cost model assumes roughly constant cost per correction: limited compute may restrict training coverage of later residual states, although G-RANS can still run longer inference than its trained horizon.

    G-RANS training stage / unrolled correctionsCompleted epochsMedian logged epoch train secondsSum logged epoch train hours
    140358.03.98
    240723.68.04
    3401117.612.42

    The actual 1/2/3-step stages become substantially more costly, with 3.98/8.04/12.42 hours of logged epoch-training time respectively (24.44 hours in total). The k=20 checkpoint completed all 120 planned epochs; the proposed explanation concerns long-horizon state coverage and optimization under fixed compute, rather than premature termination of that run.

    The LMM replay rollout cap is ten solver actions (Appendix A.4), with early stopping on convergence. In a same-configuration main-model seed44 replication, the 58 logged rank0 refresh batches from epochs 3–60 average 5.17 actions (the logged batch means range from 4.0 to 6.2). Unlike G-RANS's 1/2/3-step training stages, these rollout actions generate replay states without backpropagating through the complete rollout. Replay depth and differentiable training horizon are therefore different quantities.

    Training-state coverage / curriculumDepth or stagesTotal epochsLogged or projected training time on the original 2-GPU G-RANS configuration
    LMM replay, measured main-model replicationTypically about 5 actions; cap 10Not a G-RANS curriculumNo cross-model training-time equivalence claimed
    G-RANS, completed runStages 1, 2, 3; 40 epochs each12024.44 h, measured logged epoch-training time
    G-RANS, hypothetical progressive extensionStages 1 through 5; 40 epochs each200About 61.1 h (2.5 days), extrapolated; not run
    G-RANS, hypothetical progressive extensionStages 1 through 10; 40 epochs each400About 224 h (9.3 days), extrapolated; not run

    The projections scale the measured 24.44 h by [T(T+1)/2]/6, retaining the original implementation. They exclude additional setup/evaluation overhead and do not predict accuracy. Under this unchanged per-stage training quota, extending G-RANS to depths comparable to typical or maximum LMM replay would require roughly 2.5 or 9.2 times the completed run's training time, making that extension impractical within our existing training allocation and rebuttal window.

    LMM rollout replay offers a plausible alternative by sampling residuals encountered during actual solves without backpropagating through the complete trajectory; rollout generation and orthogonalization still have real costs. The structured polynomial directions, broader training corpus and larger architecture may also contribute, whereas the shared accumulated-subspace evaluation does not isolate IBO as the explanation.

    Given the remaining rebuttal time, this is a mechanistic working hypothesis supported by the observed stage-cost trend, not a measured causal allocation of the gap. A same-data, same-architecture, matched-compute progressive-depth versus replay intervention remains unperformed; we will prioritize that comparison if further evidence is requested.

  • Q8 — Time and memory at million-row scale.

    We tested the frozen model without retraining on unchanged structured synthetic operator recipes at larger sizes, with tolerance 1e-3. HP1/NP1 rows aggregate two samples per family (four total); NM1 rows contain two samples.

    Rows nOperator group2.5 s successes10 s successes10 s median elapsed, s (range for HP1/NP1)10 s peak allocated GiB
    100,000HP1/NP14/44/40.20–0.213.0
    100,000NM12/22/20.9614.0
    500,000HP1/NP14/44/40.61–0.6214.9
    500,000NM10/22/26.1079.6
    1,000,000HP1/NP14/44/41.16–1.2929.7
    1,000,000NM10/20/2OOMOOM; no completed peak

    At one million rows, HP1/NP1 take 1.16–1.29 s and peak at 29.7 GiB, while NM1 has no successful solves under either deadline and encounters OOM in the 10 s runs.

Reviewer_S7uzOpenReview ↗

Additional Questions

Thank you for the detailed response and for the many new experiments. The same-GPU comparison, the seed statistics, the class-holdout runs and the graph-only and polynomial-only ablations address several of my concerns, and I appreciate the effort that went into them.

The new results show that LMM performs well at a tolerance of 1e-3, but the advantage does not hold at tighter tolerances. With float64 right-hand sides, LMM solves 69 of 200 SS-S systems at 1e-6 and 2 of 200 at 1e-8, and SA-AMG GMRES already performs better at 1e-5. The cost is also high. The table reports a median of 1,875 column-equivalent matrix-vector products for LMM, against 149.5 for GMRES(100), and a peak memory of about 25 GiB, against 0.2 GiB. I may be misreading it, but I do not think this table supports the statement that LMM uses half the matrix-vector products of GMRES, unless each 360-column block product is counted as one.

Unfortunately, I am not able to raise my score at this point without the following:

  1. Results at 1e-6 or tighter where LMM is competitive with well-tuned preconditioned solvers, or a multithreaded sparse direct solver. The current GPU ILU and AMG baselines are custom implementations, and the CPU baselines use a single core. A sparse direct solver failing on 92 of 200 matrices of this size within 10 s suggests these baselines are not well tuned.
  2. A clear accuracy-versus-cost comparison, in operator work and memory, showing a regime where LMM is the better choice.
  3. An experiment that explains the large gap to G-RANS at the same width, even on a smaller scale.

I understand that some of these may not fit in the rebuttal period. Thank you again for the careful and open discussion.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.