acceptodds
Under review as a conference paper at ICLR 2027

BharatMath: Culturally Regenerated Mathematical Reasoning for Twelve Indic Languages.

Abstract

Mathematics benchmarks for Indian languages are translations: they carry the dollars, miles and bake sales of their English sources into every language, they are small, and their English halves are saturated. We introduce BharatMath, 88,066 word problems in twelve Indic languages that are regenerated rather than translated. An executable formula is extracted from each English source problem, fresh operands are sampled and the gold answer is computed in code, so no language model can alter the stored contract; whether the generated question expresses it is audited separately. The question is then rebuilt around a record from a Context Bank: a finite, typed, enumerable relation of 782 curated cells that allocates regional units and scenarios (and, as advisory guidance, magnitudes) by language and mathematical domain, so that what the corpus is about is specified before generation and auditable after it. On identical contracts rated blind, the whole pipeline clears a cultural-plausibility criterion on 52% of items against 0.7% for literal translation; the bank's own contribution is +9 points of regional appropriateness, no significant change in cultural plausibility, and -5 points of mathematical equivalence. Golds are machine-checkable (99.94% reproduce from the stored formula); a two-family gold-blind LLM audit of test items finds 81.6% fully valid, 12.4% ambiguous, 3.7% with an altered operand and 2.1% with a differing answer, and the main effects below persist on the valid subset. Seven open-weight models span five points on MGSM-English but 2 to 86 points between their best and worst Indic language, and regenerated Bengali orders them differently from translated Bengali. Because every item is a contract plus a rendering, the benchmark also supports single-factor interventions, which locate six gaps: question comprehension (up to 52 points), numeral script (up to 34), a natural-language execution component for one model family, sampling-accessible headroom, the cost of forcing native-script thought under a fixed token budget (4-20 points, most of it truncation), and distractor robustness. We release the corpus, the bank, the lineage of every item, and keep a private held-out split.

Primary area:
datasets and benchmarks
Keywords:
multilingual mathematical reasoning benchmark, Indic languages, culturally grounded data generation

Reviews

Among:
Shape:
1234this paper, Bottom 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.25 over 4 reviews, above 49% of the others.
Reviewer_myQCOpenReview ↗
Rating
3(−1)
Confidence
4
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposed BHARATMATH. It skipped the simple translation process for the multilingual math benchmark. It explicitly strips away the core mathematical formulas and the surrounding context. I think this is crucial for the development of multilingual evaluation and training. It ensures that the data remains free from English-centric drift.

Critical strengths
  1. The core idea that separates formula from context is inspiring for the whole multilingual community. Translation tax and English drift are major issues with multilingual data today. This paper offers an excellent approach.
  2. Against a translated benchmark, the proposed BharatMath is more challenging and culturally suitable.
  3. The corpus, the bank, and the lineage of every item are useful for research.
  4. The six observations regarding the gaps derived from the experiments are highly insightful.
Critical weaknesses
  1. How are the mathematical equivalence, cultural plausibility, and regional appropriateness of Figure 3 measured? Is this a manual process? This should be described more clearly in the main text.
  2. Table 3 shows that the inference mode (off vs. on) significantly impacts model rankings; how can this phenomenon be explained? While Figure 5 illustrates a reversal in model rankings on BharatMath, this reversal does not appear to occur when the inference mode is set to "on," as shown in Table 3.
Questions for authors
  1. There is a paper about cultural reasoning, which has a similar idea that separates what must never change from what should. It is suitable for discussion within the context of relevant work: https://arxiv.org/pdf/2606.01879.
  2. The cultural bank is specific to math. A more generalised cultural bank would further enhance the paper's impact.
  3. The construct process/code will be released with the corpus/bank/lineage?
1 comment
AuthorsOpenReview ↗

How Figure 3 is measured, the reasoning-mode correction, CultureForest, and code release

Thank you for the careful and encouraging review. Your question on reasoning mode led us to a decoding artefact; correcting it makes every model comparison in the paper more reliable.

How Figure 3 is measured: automated, blind and re-rated

PanelMethodReliability
LeftA blind rater from a third model family scores shuffled candidates, with method labels hidden, on anchored 1–5 rubrics (4–5 passes)10% re-rating: κ 0.92 / 0.88 / 0.77
RightBlind Qwen3-235B judgeOut-of-family check (Gemma-4-31B): κ 0.83

This is now described in §4 (pp. 4–5) and the Fig. 3 caption. Native speakers independently rate regional fit 4–5 of 5 on 84% of reads (Appendix E).

Why reasoning mode changes the rankings

You are right that the reversal does not hold with reasoning on. There are two causes:

  • The Sarvam chat templates ignore the off switch. 97–100% of their "off" outputs contain reasoning, so their "off" rows were in effect "on". Reasoning helps Qwen (+0.09), Evon 3.3 (+0.17) and Nemotron (+0.33), but Sarvam by about +0.01.
  • Gemma loops under greedy decoding. 12% and 34% of the two Gemma models' reasoning-on outputs are truncated.

With sampled decoding, the order is:

QwenGemma-31BGemma-26BSarvam-105BEvon 3.3Sarvam-30BNemotron
0.9260.9210.8810.8730.8430.8200.591

With reasoning on for every model (Table 45):

  • Bengali. MGSM and BharatMath rank models alike (ρ = 0.86).
  • Telugu. BharatMath widens both Gemma–Sarvam-30B gaps by about 10 points.

§5, Fig. 5 and Tables 4 and 21–22 are corrected accordingly.

BharatMath is harder beyond MGSM

It is harder than IndicGSM8K, the machine-translated GSM8K in the same twelve languages, for five of seven open models (3.4–8.3 points) and for four of five closed models (6.8–14.3 points; Fig. 6, Tables 46–47).

Questions

  • CultureForest. It is now cited in Related Work (p. 2). It grounds questions in atomic cultural norms; BharatMath instead holds the mathematics fixed and varies only the setting.
  • A more general bank. A new §3 paragraph (p. 3) notes that only the domain key D is specific to mathematics. Keyed by locale and task domain, the same relation can condition any templated generation with a code-checkable invariant, such as unit-bearing science items.
  • Code release. Yes (Reproducibility statement, p. 10; Appendix I). We will release the generation pipeline, the bank builder, the audit, the evaluation harness and every revision experiment, with the corpus, the bank, the lineage and the anonymised human labels. The supplementary material already contains the evaluation prompt and grader, the anonymised human labels and 120 example items with their full construction record.
Reviewer_7TPyOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper probably developed a benchmark for Mathematical reasoning in 12 Indian languages. They designed a four-stage pipeline to construct the benchmark. They tested seven LLMs on the regenerated benchmark.

Critical strengths

I couldn't find any strengths in the paper because its content is overshadowed by its poor writing. None of the lines of the paper are understandable.

  • They developed a benchmark of mathematical reasoning for 12 Indian languages.
  • They tested six LLMS for benchmarking.
Critical weaknesses
  • First, I would like to mention that this paper is not of a standard of writing that can be comprehended by a reader. A paper's general structure should revolve around the problem, motivation, and solution, with a hypothesis and experimental rigor, which is missing in this paper.
  • I hardly understand the problem that the authors were trying to solve in this paper. Their benchmark construction process and all the experiments are obscured by sloppy writing. The sentences lack grammatical structure and use so many unnecessary semicolons.
  • I believe this paper needs substantial revision in presentation and writing so that a reader can understand the work.
Questions for authors

See Weaknesses

1 comment
AuthorsOpenReview ↗

Paper rewritten around problem, motivation, hypothesis, approach and evidence

Thank you. We agree the submitted text was too dense and have rewritten it. The paper's argument in five sentences:

  1. Problem. Maths benchmarks in Indian languages are translations of English sets, so they keep the dollars, miles and bake sales of the source.
  2. Motivation. Such a benchmark cannot tell a failure at mathematics from a failure to read an unfamiliar scene, and asking a model to "localise" the items brings that model's biases into the benchmark.
  3. Hypothesis. If each item's mathematics is frozen in code and only the setting is regenerated from an explicit table of regional conventions (the Context Bank), the items stay correct and become regionally appropriate.
  4. Approach. We build 88,066 such items in twelve languages and validate them with a blind four-method study, a two-family model audit and a pre-registered native-speaker study.
  5. Evidence. Native speakers find no wrong answer among audit-valid items. BharatMath separates seven models that MGSM-English cannot, is harder than a translated benchmark for most open and closed models, and training on it improves held-out Indic benchmarks by up to 11 points.

The revised presentation

WhereRevision
Introduction (pp. 1–2)The problem → Why localisation by a model is not the fix → Hypothesis and approach → What we test → contributions
Abstract (p. 1)About a fifth shorter, with fewer numbers
ThroughoutSemicolon chains split into sentences; parentheticals removed
§4 and §5Organised around explicit questions; each paragraph opens with its claim
Table 3 (p. 5)All validity evidence in one table
Figures 6–7 (pp. 7–8)External evidence in one figure; interventions as an annotated heatmap

Experimental rigour

The revision adds a pre-registered human study with planted controls, frozen-plan comparisons against a translated benchmark and five closed models, and RL training with pre-specified controls (general response).

We would be grateful if you could look at the revised introduction (pp. 1–2), which follows the structure you describe.

Reviewer_1a8KOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The authors proposed an Indic-language benchmark of math word problems. In contrast to previous work, the questions in the benchmark are regenerated rather than being translated. The translation has this drawback that some elements like dollar, miles, etc. are carried to the translated version of the question. This makes the evaluation in the target language ineffective. The merits of the proposed benchmark is presented in table 1: it has relatively large number of questions, and covers higher number of Indic languages. It also has scenario-level grounding, sub-regional control, recomputed answers on newly sampled values, and released item lineage. As a validation, for instance, the authors show that "the two Indic-specialised models move from third and fourth to first and second, and the general-purpose models lose 4–20 points, compared to translated MGSM-Bengali. Many other validations showed the merits of the proposed benchmark.

Critical strengths
  • The benchmark is made through a comprehensive and systematic pipeline (Figure 2). The question, answer pairs are high quality, through the "contract", Fig. 1 top part.
  • Comprehensive evaluations show more reasonable resolution and ordering of various models based on the proposed benchmark. In addition, the experiments were run by controlling for confounders such as the reasoning levels of the models.
Critical weaknesses
  • I believe a benchmark is useful whenever an optimization, such as model selection, or RL-post training based on it, results in higher quality models. This though needs a second evaluation, e.g. Arena style, to prove the benchmark usefulness. This seems to be missing in the paper.
  • The fact that specialized Indic models perform better in the proposed benchmark, raises this question that is such improvement is made possible because of the model better reasoning abilities, or its Indic knowledge, or both.
Questions for authors
  • Why some proprietary models such as GPT, Claude, Gemini, etc. models were not tested?
2 comments
AuthorsOpenReview ↗

Downstream use tested: GRPO gains on held-out Indic benchmarks, checkpoint selection, and per-model closed-model results

Thank you for the constructive review. We ran both downstream uses you suggest and a closed-model comparison; all are new in §5, Fig. 6 and Appendix H. Scoring is on benchmarks BharatMath was not built from.

Training on BharatMath improves held-out Indic benchmarks (Table 48)

BharatMath supplies verified native-language supervision at scale: every answer is computed in code, in twelve languages, with a lineage-disjoint training split. GRPO on it gives:

ModelBase, MGSM-Bn/TeGain, MGSM-Bn/Te [95% CI]Gain, IndicGSM8K
Qwen3.5-4B71.2+10.9 [7.7, 14.2]+11.4
Qwen3.5-9B76.8+8.4 [5.4, 11.6]+5.4
gemma-4-E4B84.2+0.9 [−1.3, 2.8]+1.3
gemma-4-12B81.2+0.2 [−2.0, 2.2]+0.4

Controls fixed before training show where the gain comes from:

  • Native text beats the English sources of the same items: +5.3 on MGSM-Bn/Te and +5.1 on an Indic MATH-500 (Qwen3.5-4B); +2.3 on MGSM-Bn/Te (gemma-4-E4B).
  • A random reward lowers every benchmark (Qwen3.5-4B: −6.5 on IndicGSM8K).
  • A machine translation of the same verified items matches the gain. The value lies in verified items in the target language, which BharatMath provides.

BharatMath picks better checkpoints (70 pre-registered pairs)

Share of pairs on which each benchmark picks the checkpoint that is better on MGSM-Bn/Te:

BharatMathIndicGSM8KMGSM-EnglishGSM8KGSM-Symbolic
90%89%69%67%53%

Closed models, per model (Table 47)

Plan frozen before the first call: 100 items per language from each benchmark, the same prompt and grader, and each API's lowest reasoning setting. Accuracy in %:

ModelBharatMathIndicGSM8KGap [95% CI]MGSM Bn / Te
gpt-6-luna79.293.5−14.3 [−16.9, −11.6]92 / 90
gemini-3.5-flash-lite81.994.5−12.6 [−15.2, −10.1]91 / 87
claude-haiku-5-581.993.0−11.1 [−13.8, −8.6]91 / 86
gpt-6.1-terra89.896.7−6.8 [−8.8, −4.9]96 / 92
gemini-3.8-flash*95.796.9−1.3 [−2.7, +0.4]97 / 94

*Cannot disable reasoning; at ceiling on both. Across the four other models, BharatMath spreads accuracy over 10.6 points, against 3.7 on IndicGSM8K.

Reasoning, Indic knowledge, or both?

Paired interventions on identical items separate the parts (§5, "What the separation measures"):

FactorInterventionFinding
Reading the languageSame Indian scenario, in EnglishAbout +5 for Gemma-31B and Sarvam-105B; +52 for Nemotron
Regional framingRegional item vs. translation of its sourceRegional item solved slightly more often (+3.8 pooled)
Executing the arithmeticModel writes a programGemma-31B solves 72% of its Bengali failures

With reasoning on for every model, the Indic-specialised models gain no relative advantage in Bengali (ρ = 0.86 with MGSM; Table 45). In Telugu, BharatMath widens the Gemma–Sarvam-30B gap by about 10 points. BharatMath therefore measures comprehension of Indic text and execution of the arithmetic, which is what an Indic mathematics benchmark should measure.

After going through author response and other reviewer comments, I will keep my score unchanged.

Reviewer_mJiBOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

Overall: The authors present a method of converting English based mathematical QA questions to 12 Indic languages. The paper starts from an English math dataset, preserves the formula/arithmetic, samples new numerical values and executes the arithmetic using code to get a new calculation. Then, using the proposed 'Context Bank' convert this newly generated calculation to a regional focused question.

Contributions:

  1. A Multilingual math benchmark, with around 88k entries, spanning 12 languages, 15 domains, with lineage-disjoint train, public test and private test splits.
  2. 'Context Bank', a cultural regional setup which can be used along with the proposed methodology to generate regional specifc multilingual questions.

Validation:

  1. Comparison across seven open weight models, with experiments examining how question language, numeral scripts, and reasoning methods affect accuracy.
  2. Automated checks and LLM judges assess whether the generated questions preserve the intended mathematics and fit the regional context.
Critical strengths
  1. The paper has a good coverage across broad indic languages, and is decently sized with 88,066 word problems. The stored formulas, numerical values provide an opportunity for controlled evaluation setup, which is a strength of the proposed benchmark.
Critical weaknesses
  1. The authors’ own LLM-based judges only 82.7% of labelled test questions as valid, with 13.2% classified as ambiguous or missing necessary information (Table 11). This is important because a model could be penalized for a problem with the question rather than a failure to solve it.
  2. The stable rankings on the filtered subset are encouraging, but the filtering also relies on LLM judgments (Appendix D). Without native-speaker checks or any human in the loop to verify LLM-based judges, confidence in the questions’ correctness, language quality, and cultural appropriateness remains limited. Independent human evaluation would help establish whether the model-based quality checks are reliable. This is the biggest weakness according to me, for the soundness of the evaluation of this benchmark.
  3. In Table 7, adding the bank improves regional appropriateness by 9.1 percentage points, but reduces the share of questions judged to preserve the intended mathematics by 5.3 points. Moreover, the cultural-plausibility increase is not statistically significant (p = 0.579). The gains in regional appropriateness are promising, although further analysis would help clarify the tradeoff with mathematical correctness.
Questions for authors
  1. Could the authors provide an independent native-speaker evaluation of a small sample across languages, including both questions labelled valid and those flagged by the LLM-based audit? How closely do human assessments agree with the model judgments on mathematical correctness, ambiguity, and cultural appropriateness?
1 comment
AuthorsOpenReview ↗

Native-speaker evaluation run as proposed: no wrong gold among audit-valid items; audit-human agreement equals human-human

Thank you for the precise suggestion. We ran the native-speaker evaluation you proposed, covering both audit-valid and flagged items, and measured its agreement with the LLM audit (§4(iii), Table 3, Appendix E). Humans find no wrong gold among audit-valid items, and the LLM audit agrees with humans as closely as humans agree with each other.

Study design

  • Pre-registered. The plan was frozen before the first label. It takes 20 items per language, covering all 78 problem archetypes, from every audit stratum: valid, ambiguous, kept after a flag, and re-golded.
  • Blind, with controls. Annotators never see the stratum. Four planted defective items per language check attention.
  • Gold seen last. Annotators rate the language and whether the question is well posed, and record an independent chat assistant's answer. They then lock these ratings, and only then judge the stored gold and rate regional fit, realism and naturalness (1–5).

Results

There were 172 reads of 123 items in nine languages, by 14 annotators; 81% of reads were by native or fluent speakers.

MeasureResult
Wrong gold, audit-valid items0 reads. All 7 wrong-answer reads fall on items the audit had already flagged ambiguous
Fatal defect, audit-valid items3 of 136 reads (2.2% [0.8, 6.3]); the pre-registered rule accepts
Agreement, audit vs. humansκ 0.44, equal to human–human (0.44, 52 pairs); PABAK 0.73 vs. 0.77
Agreement on the goldPABAK 0.96
Regional fit / realism / naturalness rated 4–583.9% / 86.3% / 86.3% of reads
Language fine or minor issues97.7% of reads

The result is robust. Excluding the two annotators who caught fewer than half their controls, 1 read in 101 finds a fatal defect; on native-proficiency reads only, 1 in 67.

The 82.7% figure and fair evaluation

The validity labels separate the items a model could be penalised on:

  • Items that humans flag are the ones models fail (mean accuracy 0.571 vs. 0.865), so the labels track real defects, not rater noise.
  • On the valid core, every model gains 2–5 points and the ranking is identical (Tables 12–13). Every item ships its label, so the valid core is one filter away.

What the Context Bank contributes (Table 8)

  • Regional appropriateness +9.1 points (p ≤ 0.002), confirmed by native speakers (regional fit 4–5 on 84% of reads).
  • Cultural plausibility comes from regeneration itself: 52% of items, against 0.7% for literal translation.
  • The −5.3 is a rubric shift, not wrong answers. 2.7% of items pose a different problem and 4.8% are answerable but not uniquely (§4, p. 5). Released items additionally pass a numeral gate and the two-family audit.

Scope

The study bounds the pooled defect rate, not per-language rates. It covers nine of the eleven languages offered (Urdu was not in the annotation tool). Appendix E reports the instructions, consent and annotator characteristics. The supplementary material contains all 172 anonymised reads with each item's audit stratum, so the wrong-gold and fatal-defect counts above can be recomputed.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.