- Rating
- 3(−1)
- Confidence
- 4
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposed BHARATMATH. It skipped the simple translation process for the multilingual math benchmark. It explicitly strips away the core mathematical formulas and the surrounding context. I think this is crucial for the development of multilingual evaluation and training. It ensures that the data remains free from English-centric drift.
Critical strengths
- The core idea that separates formula from context is inspiring for the whole multilingual community. Translation tax and English drift are major issues with multilingual data today. This paper offers an excellent approach.
- Against a translated benchmark, the proposed BharatMath is more challenging and culturally suitable.
- The corpus, the bank, and the lineage of every item are useful for research.
- The six observations regarding the gaps derived from the experiments are highly insightful.
Critical weaknesses
- How are the mathematical equivalence, cultural plausibility, and regional appropriateness of Figure 3 measured? Is this a manual process? This should be described more clearly in the main text.
- Table 3 shows that the inference mode (off vs. on) significantly impacts model rankings; how can this phenomenon be explained? While Figure 5 illustrates a reversal in model rankings on BharatMath, this reversal does not appear to occur when the inference mode is set to "on," as shown in Table 3.
Questions for authors
- There is a paper about cultural reasoning, which has a similar idea that separates what must never change from what should. It is suitable for discussion within the context of relevant work: https://arxiv.org/pdf/2606.01879.
- The cultural bank is specific to math. A more generalised cultural bank would further enhance the paper's impact.
- The construct process/code will be released with the corpus/bank/lineage?
How Figure 3 is measured, the reasoning-mode correction, CultureForest, and code release
Thank you for the careful and encouraging review. Your question on reasoning mode led us to a decoding artefact; correcting it makes every model comparison in the paper more reliable.
How Figure 3 is measured: automated, blind and re-rated
This is now described in §4 (pp. 4–5) and the Fig. 3 caption. Native speakers independently rate regional fit 4–5 of 5 on 84% of reads (Appendix E).
Why reasoning mode changes the rankings
You are right that the reversal does not hold with reasoning on. There are two causes:
With sampled decoding, the order is:
With reasoning on for every model (Table 45):
§5, Fig. 5 and Tables 4 and 21–22 are corrected accordingly.
BharatMath is harder beyond MGSM
It is harder than IndicGSM8K, the machine-translated GSM8K in the same twelve languages, for five of seven open models (3.4–8.3 points) and for four of five closed models (6.8–14.3 points; Fig. 6, Tables 46–47).
Questions