- Rating
- 2
- Confidence
- 4(+1)
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper introduces Multilingual GSM-Symbolic, a multilingual extension of GSM-Symbolic containing 100 underlying mathematical templates translated and localized into 15 languages. The authors render 20 instances from each translated template, yielding 30,000 question–answer instances, and state that the templates, generation code, validation tests, model outputs, and annotation provenance are released through an anonymous repository. The benchmark is evaluated with a broad collection of open-weight instruction-tuned models.
The paper then fits a generalized linear mixed model to estimate how model size, reasoning mode, language resource level, typological distance from English, tokenizer fertility, and their interactions relate to accuracy. It reports that model size, language resource level, and reasoning are the largest positive predictors, while typological distance is negatively associated with performance. The authors additionally perform leave-one-language-out prediction, symbolic-versus-original comparisons, and analyses of where errors occur within generated solutions.
The benchmark construction is useful, and the joint quantitative analysis is more systematic than reporting separate multilingual accuracy tables. However, the methodological novelty is limited, most headline conclusions are already expected from prior multilingual evaluation work, and the paper overstates both the independence of the benchmark instances and the causal or transferable meaning of its regression coefficients.
Critical strengths
-
A controlled and reusable multilingual evaluation resource. Using the same underlying mathematical structures across 15 languages is preferable to comparing unrelated monolingual benchmarks. Releasing editable symbolic templates and generation code, rather than only a fixed set of rendered questions, may support future contamination-resistant evaluation and correction of dataset errors.
-
Broad evaluation with a joint analysis of model and language factors. The paper evaluates a diverse set of model families and places model size, reasoning, resource level, typological distance, and tokenizer fertility in one statistical framework. The leave-one-language-out analysis is also more informative than reporting only aggregate multilingual accuracy.
Critical weaknesses
-
The effective dataset novelty, structural diversity, and translation validation are overstated. The reported 30,000 instances are generated from only 100 underlying GSM-style templates. The paper's own variance analysis shows that approximately 93% of the remaining uncertainty comes from variation across templates and that generating additional instances from the same templates provides little benefit. Most headline findings are computed on a fixed set of rendered instances and could plausibly have been obtained with a sufficiently large static item-matched multilingual dataset. The paper does not demonstrate which central scientific conclusion specifically requires its symbolic, extensible infrastructure. Given the availability of MGSM, MGSM-Pro, mGSM-Symbolic, and related resources, the incremental contribution is closer to a useful engineering consolidation than a fundamentally new benchmark paradigm. In addition, all translators used LLM assistance, but the paper does not clearly report independent double review, inter-annotator agreement, or language-wise error rates for naturalness and semantic equivalence.
-
The robustness, external validity, and operational usefulness of the regression analysis are limited. Common Crawl page count is a model-agnostic proxy rather than the actual language composition of each model's training corpus, while model size, model family, training compute, data mixture, tokenizer, and post-training recipe remain partly confounded. Only 15 languages support the language-level conclusions, and typological distance is highly correlated with tokenizer fertility. Moreover, the appendix reports a large template-by-language random effect that substantially improves AIC, yet this term is omitted from the primary model, and the shared difficulty of individual symbolic instances is not modeled. The paper reports only the standard deviations of the random effects, not the fitted effects required to reproduce a concrete prediction from the published formula. Finally, leave-one-language-out evaluation considers known models in an unseen language, but does not establish performance for a new model family or a jointly unseen model-language pair. The fitted model therefore functions more convincingly as a retrospective summary of the evaluated matrix than as a generally usable prediction tool.
Questions for authors
-
Which principal result specifically depends on having symbolic and extensible templates rather than a static item-matched multilingual dataset? Can the authors demonstrate that template-based resampling materially improves model ranking reliability, contamination resistance, or estimation of the reported fixed effects? Given that the paper's own analysis finds that template-level variance dominates, why was structural coverage limited to only 100 underlying templates?
-
How many templates in each language were independently reviewed by a second native speaker? Please report language-wise grammaticality, naturalness, semantic-equivalence, and answer-correctness error rates. All translators used LLM assistance, so a translator reviewing their own LLM-assisted output is not sufficient to establish independent validation.
-
How do the primary coefficients, confidence intervals, leave-one-language-out errors, and explained-variance estimates change when the template-by-language effect and an instance-level random effect are included? The reported template-by-language standard deviation is large, indicating that translation or localization changes template difficulty across languages.
-
The main text reports approximately 92% of between-language and 23% of model-by-language variation explained, whereas the appendix reports different values for the original and alternative specifications. Please reconcile these numbers and identify the exact model underlying each headline result.
-
How well does the regression model predict performance when an entire model family is held out, or when both the model family and language are unseen? For a genuinely new model-language pair, how should a practitioner use the published formula when the family, model, and model-by-language random effects are unavailable? Please provide the complete prediction procedure and the information required to reproduce the reported forecasts from the released model fit.
3 comments
Response to the Authors
We thank the authors for the detailed response. Two concerns remain.
Presentation: the manuscript still falls short of publication standards — headline coefficients disagree (size β=1.77 in the abstract vs. 1.88 in Finding 1; −0.27 vs. −0.25 in the conclusion), typos persist ("eThis" §3.1; "shortcormings" §2.1), Fig. 12 lacks model labels on its x-axis, and several pages contain unexplained blank regions.
Contribution: the response acknowledges extensibility is not required for the findings, and Findings 1–3 are explicitly consistent with prior work. Beyond a localized translation of GSM-Symbolic and a regression fit, what precisely is the novel contribution?
the manuscript still falls short of publication standards — headline coefficients disagree (size β=1.77 in the abstract vs. 1.88 in Finding 1; −0.27 vs. −0.25 in the conclusion), typos persist ("eThis" §3.1; "shortcormings" §2.1),
We have corrected typos. While the inconsistencies were indeed true in the original version it is not true in the revised version.
Fig. 12 lacks model labels on its x-axis, and several pages contain unexplained blank regions.
Fig 12 has models on its x-axis and contains no unexplained blank regions. This is true in both the review version. I had sent the PDF to a third party to confirm that this is indeed the case. I have also opened it in multiple pdf readers (zotero, mozilla firefox and Preview). I would be happy to address this issue but cannot reproduce it.
the response acknowledges extensibility is not required for the findings
Indeed, but it argued that is valuable. Releasing expandable high-quality resources allows others to utilize, contribute and thus forward science. This in itself is valuable.
, and Findings 1–3 are explicitly consistent with prior work. Beyond a localized translation of GSM-Symbolic and a regression fit, what precisely is the novel contribution?
While 3 of our 5 findings extend those of previous work, previous work tackled these individually and not nec. in a multilingual and symbolic setting. As many of these attributes interact modelling them jointly establishes exact relative ordering which allow prioritization of resources. We also mention this in line 304-306. Reproduction especially under different conditions strengthens existing claims and reproduction of existing results is itself valuable to our science.
However, besides these contributions, we also provide an important distinction of resource level and typological distance (also noted by VT6k), which have important implications for how to target development in low-resource languages. Similarly, we also show that using just a few target language template can drastically improve downstream estimates which is of importance for benchmark developers. Besides these headline finding we examine several other dimensions including the difference imperial and metric, the effect of validation on scores and explore the positive effect of synthetic samples in appendix A as well as provide a detailed solution gap analysis of where the error happens ($4.4, appendix I)
We thank the authors for the detailed response. Two concerns remain.
We are happy to hear that we have addressed most of the reviewers concerns and hope these last comments can clarify the remaining. If these address the reviewer concerns we hope that the reviewer will consider updating their scores accordingly.
We thank the reviewer for the detailed review and are glad the item-matched design, released code and joint analysis were seen as useful.
We suspect the 93% comes from Appendix A.5: "At n = 20 the template-level term is already 93% of the total…". The argument is that 20 instances per template is a reasonable budget because more would add little, not that instances are worthless. Equivalently sized benchmarks (e.g. MGSM) are widely used for model evaluation; 100 templates gives a strong enough signal while balancing validator effort across 18 languages.
A static dataset may give a similar signal on model competence, but ours adds two things it lacks: (1) human-validated templates that can be scaled to arbitrary size with full control over which problem aspects vary, and (2) evaluation under variability, which reduces the effect of benchmark leakage. Extensibility is not required for our findings, but the open pipeline lets practitioners vary aspects of their choice (number ranges, gender of names, …), and since submission it has grown from 15 to 18 languages (Punjabi, Swedish, Urdu).
Most languages start with one human validator, who checks and improves the initial machine translation. The open repository lets further native speakers validate and extend it, which has happened for five languages; there, template edits were merged only once all validators were satisfied. As this is an open discussion on GitHub, inter-annotator agreement is hard to report. Naturalness is hard to measure objectively, but validators were asked to actively correct templates for grammaticality, naturalness and semantic equivalence. The public git history (Added analysis in Appendix K.3) documents this: reviewers edited 98.4% of templates but changed the answer rule in only 2.9%, and even such non-wording edits often only accommodate target-language grammar without changing the computation (e.g. Danish nouns are sampled with their article because of grammatical gender, template 0032).
Training-corpus composition is not consistently reported, and data quality would remain; we note this in the limitations. The model × language effect captures model-specific deviations, e.g. the English-centric OLMo models do better on English than their size and features imply (+1.11 log-odds).
We therefore use relative fertility (deviation from the language mean, §3.3).
We agree and have made it the primary model (see the general response); everything was refitted, with leave-one-language-out errors of 3.1 pp (language mean) and 6.0 pp (model in language). Under the previous specification, significant coefficients keep their sign and significance except distance × reasoning (p = .03 → .06), and size, resource and reasoning are about 10% smaller (Appendix A.1, Table 2).
The fitted models were already available through the released code and rendered analyses (
transfer_analysis.html,model_ablations.html) and we now also release all fixed effects and every random-effect level as CSV inpaper/artifacts/analysis/fitted_effects/.The 92% / 23% came from the original specification (recomputed: 91.7% / 22.6%). The appendix values (91% / 16%, 89% / 20%) did not reproduce, an error on our side. Every headline number now comes from the primary model (93% / 19%), and Appendix A.1 reports the original specification separately.
This evaluation simulates the case where the model is given but data in the target language is not, so holding a model or family out would not match the real-world use case; the goal is to estimate a known model's performance in a new language. For this, the procedure (Appendix F) is to refit on the other languages and predict with the fixed effects and the family, model, variant and template effects, setting the language, model × language and template × language terms to zero. Evaluating about 10 templates in the target language reduces the error to 4.4 pp. Refitting takes minutes on a laptop.
Finally, we are surprised by the presentation score of 1 and would be glad to improve it if you can point to passages that were hard to follow.