- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper introduces LinAlg-Bench, a diagnostic benchmark of 1,194 SymPy-verified linear algebra problems across nine tasks on 3X3, 4X4, 5X5 matrices. The goal is to measure mathematical execution depth (and when it breaks). An LLM-as-a-Judge is used to categorize the types of errors made and the analysis (after testing on 10 LLMs) shows a shift in the type of error as the challenge increases.
Critical strengths
- It is important to have granular, controled-difficulty, measurements of LLM skills. This work suggests another angle for that.
- The method seems easy to extend further as models become stronger (6X6 matrices, more advanced matrix operations, etc).
Critical weaknesses
-
There seems to be inconsistencies in the data/claims throught the paper. For example, figure 2 claims that arithmetic errors persist - but the total number of problems per dimension is not the same, so in %, arithmetic errors drop substantially. The "Declared-pass protocol" is an obvious bias upwards for the 2 frontier models - why not just take relative % instead of assuming pass on problems which we not attempted? Keeping only parseable boxed answers creates another data bias. "Judge-leniency" looks like an ad-hock crutch. Etc.
-
Writing quality: The abstract and introduction contains many terms which are not defined and not widely known. It sells results without explaining what was done. Almost all citations are in the 2 paragraphs of previous works - and missing obvious papers (e.g. Charton 2020 is cited but Charton 2022 "Linear Algebra With Transformers" is not). Figure captions not self-contained with no clear message. Figure fonts too small for human eyes. Looks very "Claudeish".
Questions for authors
- In table 1 the core challenge design is described. Why do you say that each level strictly adds a "computational requirement"? e.g. Gaussian elimination requires multi-step memory. Determinant calculation doesn't necesseraly require tracking a similar number of steps, and the steps are different. The generality of the method depends on the ability to order the challenges by difficulty, but that's not obvious to do.