acceptodds
Under review as a conference paper at ICLR 2027

GLOBENUM: Benchmarking Context-Aware Numerical Formatting Reasoning Across Cultures

Abstract

Numerical notation varies across cultures: 1.234 denotes one point two three four in India or the US, but one thousand two hundred and thirty-four in Belgium or Colombia. For many such expressions, the intended value cannot be recovered from the string alone; only context can resolve it. We present GLOBENUM, a benchmark that isolates these genuinely ambiguous representations and tests whether linguistic, geographic, and cultural context enables models to recover their intended value. Across 7 model families, 10 numerical formats, 12 countries, and 7 languages, using both symbolic operations and localised word problems, we find a strong preference for dot-decimal interpretations that persists despite geographic cues. Localisation further exposes a striking asymmetry: cultural context improves accuracy in dot-decimal locales while reducing it by up to 13 points in comma-decimal locales. Even with geographic and formatting cues provided, accuracy on genuinely ambiguous representations reaches only 32.78%. GLOBENUM reframes numerical reasoning as a cultural task: regional context must shape not only how numbers are written, but how their values are interpreted and computed, a distinction current models fail to make reliably.

Primary area:
datasets and benchmarks
Keywords:
Number formatting, arithmetic, culture, reasoning, mathematics, global

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_VK7POpenReview ↗
Rating
2(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper presents GlobeNum, a dataset/benchmark that asks whether language models can read and write numbers in the notation used in different parts of the world. A string can mean different values based on the region: "1,234" would be "thousand two hundred thirty four" in the US but "one point two three four" in France. The benchmark covers ten notation formats, twelve countries and seven languages. It has 7,000 symbolic arithmetic items and ~35k localized word problems built from FERMAT templates. Items are sorted into tiers by how much the number string reveals about its own notation. The ambiguity sits in both decimal/fractional separators (dot or comma) and digits grouping (triplets, duplets , quadruplets). Five prompt conditions add context step by step, from a plain instruction to the user's country and a hint to watch the formatting. Eleven models are tested. The authors report a strong preference for dot-decimal readings that persists with country cues, a drop in comma-decimal locales when the stories are localized, and a best pooled score of ~32% on the most ambiguous tier.

This paper takes on a real and neglected problem. The same written number can mean different things in different countries, simply because of different cultural notations for decimals. A language model can get the size of an answer wrong by an error of several orders of magnitude, yet still sound confident. The authors build a benchmark. hey sort items by how much the number itself reveals, then add context step by step, such as the country and a hint about formatting. They conclude that models lean toward the US style even when told where the user lives.

I value the question, the clear logic behind the design and the wide choice of languages and models. I also thank the authors for releasing their data. It allowed me to recompute the numbers and several tables, and those checks mostly came out right. A revised version of the paper could become a very good benchmark, but I do not think the current evidence supports the main claims. Also I am concerned about significant flaws or caveats in the design, which are undocumented at this point. Since the paper and the data are build around special symbols in the notation (commas, dots, apostrophes) I sometimes use triangular brackets further in the review '< >', in order to avoid ambiguity in the data representation.

My first concern is that in the hardest tier, many items cannot show which reading a model used. For multiplication and division, both readings of an ambiguous number give the same digits. For example, 6,722 divided by 2 is written 3,361 under either reading. A model can earn credit for the right digits without reading the number correctly. The paper drops one tier for exactly this reason, but the same reason applies to half of the items in the tier it keeps. The "any format" (V-any) score meant to separate notation from value has the same blind spot. The paper also never checks whether its word problems can be solved for the wrong reasons, a danger that earlier work on math benchmarks has shown.

My second concern is that the paper offers no reference points. It does not say what a reader that always picks one reading would score, or what the models score on the same arithmetic without any ambiguity. Many results sit near the floor. The claim that bias shrinks as context grows looks like a floor effect to me. In the released numbers, the models mostly stop using the requested notation once the stories are added, so both groups fall together, while the value-only score of the favored reading hardly changes.

My third concern is that the scoring asks for something no prompt requests. The output notation is never named, yet an answer is marked wrong if it uses a different one. Other design choices add noise. Multiplication numbers always end in zero, correct answers are padded with zeros, one format for China does not match the standard locale data, and the Swiss separator differs from that data. Some word problems turn into odd statements under one reading, such as a fraction of a doughnut, which the authors' own annotation guideline says should be avoided. More than a quarter of the symbolic country-cue runs repeat an identical prompt under a different language label.

My fourth concern is reproducibility. The release holds data and summary counts, but it has no code for building items, prompting or scoring, no raw answers and no annotation records. The paper gives no decoding settings, seeds or confidence intervals. One large model ran from a third-party 4-bit copy, and the precision of the others is not stated. Several tables depend on pooling choices that the paper does not explain.

Most of these problems come from how the benchmark is built and scored, so fixing them would change the main tables and the claims. That is a job for a revision and not for a rebuttal. The released numbers already hint at a clear finding, which is that writing the problem in the local language moves the model's reading far more than the country line does. A revised paper could lead with that finding, backed by results for addition and subtraction alone, reference baselines, the scoring code and clean pooling choices. For now I lean toward rejection, and I would welcome a resubmission.

Critical strengths
  1. The problem is real. A number such as 1,234 or 1.234 can be read at the wrong size without any warning, and the answer still looks valid. This matters for any service that reaches users in more than one region.
  2. The topic has been little studied. Multilingual math benchmarks usually translate the words and leave the numbers in one notation. This paper moves the focus to the number itself. I did a rigorous literature search regarding the problem, and it is quite understudied indeed.
  3. The benchmark is new within its stated scope. I know of no other set that mixes strings whose value depends on the notation with context that could settle them, such as country, language and a formatting hint.
  4. The intention of the design is stated clearly. Operands are sorted into unambiguous, format-ambiguous and value-ambiguous types, and the paper explains why the tier with two value-ambiguous numbers is left out. This gives readers a clear vocabulary and a reason for each tier.
  5. The set of languages and countries is good. It covers seven languages, including Tamil, Korean and Chinese, 23 country-language pairs and ten notation formats. A parallel symbolic and word-problem version lets the paper separate the numbers from the story around them. The appendix describes a review of every template by native speakers.
  6. The model set is varied. Eleven models cover hosted and open weights, small and large sizes, and region-focused models such as Apertus, EuroLLM and Aya. The finding that the region-focused models show no clear advantage is useful.
  7. The data release made the checking possible. I recomputed all 7,000 referenced answers from the released files and every one was correct. The item counts match the paper. The symbolic rows of Table 5 and the numbers in Figure 1 reproduce exactly from the aggregate file.
  8. The released numbers already hold a clear finding. Pooled over locales, the less-favored reading scores 3.7 on the value-only metric in English-language stories and 21.4 in local-language stories (comma versus thousands split), and 11.4 against 30.3 in the dot versus dot-grouping split. The country line alone moves very little. This language effect deserves to lead the paper, although it needs a per-country check.
Critical weaknesses
  1. Most problems come from how the benchmark is built and scored, so a revision could fix them but I have doubts that a rebuttal could do it. Repairing them changes the central tables and several claims. The review guidelines say new experiments should not significantly change the content of a submission.

  2. The paper does not ask whether its word problems can be answered for the wrong reasons. Patel et al. (2021) is cited once, as an example of a benchmark that gets translated (Section 2.1). Its main finding goes unused. Trained solvers that never saw the question still solved ~65% of ASDiv-A and ~75% of MAWPS (a Mathematical Word Problem could be reduced to a bag-of-words + arithmetic operation classifier, providing surprisingly high accuracy on the simple tasks; it proves that lots of tasks could be answered by the models without any actual reasoning). A similar false positive exists at this paper in another form. In the High tier, every multiplication and division has the same digits under both readings of the ambiguous number, so a model can earn digit credit without reading that number at all. The paper has no question-free, operator-only or reading-blind baseline that would show how much credit is real. The tests authors propose could have a decent amount of false-positive resolution acts and that should be addressed

  3. There are no baselines or null results, and some findings sit on a floor that the design creates.

    • A "cell" is one notation format inside one conflict mode. Twelve of the 20 High-tier cells have a comma-decimal target and eight have a thousands target. A reader that ignores context and always picks one reading therefore matches the target in 60% or 40% of cells before any arithmetic. By my calculation, a context-blind reader with perfect arithmetic would reach 55 to 88% digit credit in this tier, depending on how zeros are scored, which the paper does not say. The models reach 40 to 45%. The paper cannot say how much of that gap is reading and how much is arithmetic on long numbers.
    • The claim that format bias decreases with context (Table 4) is a floor effect. In the comma versus thousands split, the value-correct score falls from 55.6 to 11.6 for the favored group and from 11.8 to 2.0 for the other group, while stories and the JSON wrapper are added. At the same step the favored group format score drops from 84% to 23%, so the models stop using the target notation. On the value-only metric the favored group barely moves (68.6 to 63.2) and the other group falls from 12.2 to 3.0. The claim authors make as a finding doesn't exist: nothing became less biased.
  4. Almost nothing needed to reproduce the study is released. The README lists the benchmark files, the word-problem templates and one aggregate results file. There is no code that generates the items, no prompting or inference code (only examples in the appendix), no scoring code, no raw model answers, no annotation records and no run configuration. The paper gives no decoding settings, seeds or reasoning modes. A reader cannot check how an answer was extracted, how zeros were handled or how borderline strings were scored. The main numbers can be checked only through 28,424 aggregate rows, each a count out of 100.

  5. The "any format" score does not do its job on the High tier, especially for a division operation. V-Any is meant to forgive the wrong notation while still checking the value. In the High tier the value itself depends on the notation. Take 6,722 / 2 from the data. The reference is 3,361.000 if the comma is a thousands mark and 3,361 (meaning 3.361) if it is a decimal comma. The same answer string, 3,361, is correct in both cells, and the model never had to choose a reading. A model that read 6,722 as 6.722 and wrote 3.361 could also pass the thousands cell, because 3.361 is a valid way to write 3,361 in the dot-grouping formats. V-Any is informative mainly on addition and subtraction (see point 10). The released file also contains a counter named 'reading_ambiguous' that the paper never mentions. It averages about 35 per 100 answers in the High tier and 1.5 in the Standard tiers.

  6. The choice of separator characters is debatable and undocumented. CLDR (the Unicode locale data, checked against the current repository) gives comma U+002C and full stop U+002E for the decimal and grouping marks in most locales. It gives the apostrophe U+0027 as the grouping mark for the Swiss locales (de-CH, fr-CH, en-CH, it-CH), U+202F for French and U+00A0 for Russian and South African English. The dataset's Swiss format (AC) uses the typographic apostrophe U+2019 instead. The Colombian format (ADC) uses U+0027 together with the full stop, which CLDR does not list for es-CO. The space formats (SD, SC) use a plain space U+0020, the same character that separates operands from operators. The JSON files store all of these as raw UTF-8 text with no '\uXXXX' escapes, so the code point can only be learned from the bytes. For example, 'AC.json' contains <1’477’138,90551 + 840’118’893,88336> (U+2019, bytes E2 80 99) and 'ADC.json' contains <1'477.138,90551 + 840'118.893,88336> (U+0027). In my checks of eight tokenizers, look-alike characters received different token IDs. Pipelines that normalize Unicode also turn U+00A0, U+202F and U+2009 into a plain space. The paper should state each code point and give a source for it.

  7. Some word problems become odd under one of the two readings, and the paper does not control for this. A value-ambiguous number attached to a countable object turns into a fraction under the decimal reading. One item reads as: "Mr. Rutledge bought 2,936in of doughnut dough, he splits them equally into 2in for each doughnut. How many doughnuts does he have on average?" The answer is 1,468 doughnuts, or 1.468 doughnuts if the comma is a decimal mark. Other items give absurd sizes under one reading, such as "Each sausage cost 38,320 Dollars" (38.32 under the decimal reading) and "her 433lb dog" with "7,100g of dog treats for each lb". The authors own annotation guideline (Appendix C.3, check 3) says a decimal must attach to something measurable and not to a whole item. The High tier needs both readings to be possible, so it produces exactly the sentences the guideline rules out. This hands the model a plausibility cue that the benchmark does not account for. It may also make some items harder, because the reader meets a surprising quantity either way. The paper does not test whether this helps or hurts. A simple check would compare accuracy on items where one reading is absurd with items where both are plausible.

  8. The generator leaves two kinds of trailing-zero artifacts. Firstly, in all 75 multiplications of the High tier (25 in each conflict mode) the value-ambiguous number ends in zero, for example <4 * 19,700> and <8 * 2,000>. For the other operators only 3 to 8 of 25 do. Such a product has the same digits under both readings (78,800 or 78.8). Secondly, the reference answers for the thousands reading carry zeros that nobody would write. For 8,881 + 6 the reference is 8,887.000, against 14,881 for the decimal-comma reading. 762 reference answers in the dataset are whole numbers written with a zero fraction. The paper explains neither rule and does not say how 8,887 is scored against 8,887.000.

  9. Accuracy is the wrong yardstick for the combined country-and-hint condition (paper's P5), and repeated prompts distort it.

    • The symbolic P3 and P5 runs contain 25,400 instances per model, but 6,900 of them (27%) repeat an earlier prompt. The prompt is in English and names only the country, so CH-de, CH-en and CH-fr (and CN-en/CN-zh, IN-en/IN-ta, US-en/US-es) send identical text. For apertus-8b under P5, on the comma versus thousands split with format C, the three Swiss rows (IDs 25918, 25929, 25940) are identical: 3 exact, 86 format-correct, 11 digit-correct and 9 value-correct out of 100. The German row (id 25962) differs only slightly, with 10 and 8. Switzerland therefore counts three times, and any interval that treats the 25,400 as independent would be too narrow.
    • One prompt is also scored against several references. With <Location: United States>, the item <382 + 819> is scored against 1201 in the D cell, 1 201 in the SD cell and 1,201 in the CD(333) cell, and no single answer can match all three. Accuracy under P5 has a ceiling below 100% that comes from the scoring and not from the models.
  10. Reading-blindness is visible directly in the released reference answers. For each expression I compared the reference under the thousands reading with the reference under the decimal-comma reading in the comma versus thousands split. For addition and subtraction the digits always differ (0 of 500 pairs match). For <8,881 + 6> the answer is 8,887 or 14.881. For multiplication and division the digits always match once zeros are stripped (490 of 490 pairs). For <14,596 / 8> the gold is 1,824.500 or 1,8245, and both are made of the digits 18245. The authors leave out the tier with two value-ambiguous numbers for this very reason (footnote to Table 2), but the same reason applies to half of the items in the tier they keep. Only 2,000 of the 7,000 symbolic items (one third roughly) can show through the digit sequence which reading a model used.

  11. The paper does not say at what precision most local models were run. Appendix G notes that the Mixtral-8x22B checkpoint is a third-party AWQ quantization, and its model card is tagged 4-bit. Nothing is said about the other local models in the released file (Qwen2.5-72B, Apertus-70B, Aya-Expanse-32B, Qwen3.8-27B, Ministral-3-8B). Published studies find that 4-bit weight-only quantization usually stays within about 2% of full precision on average, while math and multi-step reasoning are more sensitive than other abilities and Mixtral-style mixture-of-experts models are more sensitive than dense ones (Li et al., 2024). This benchmark depends on exact digit strings, so the effect may matter, and a weak Mixtral score cannot be separated from the effect of compression. A same-model comparison of the quantized and full-precision checkpoints on the F×F tier would settle it cheaply.

  12. No confidence intervals, seeds, bootstraps or noise estimates are given, so most figures cannot be read properly. Each model-cell has 100 items, so a rate near 10% already carries a standard error of about 3 points, and the same 100 operand pairs are reused in all ten formats, so cells are not independent. The paper interprets differences of one to three points, for example High-tier digit credit of 40.95 (P2) against 39.89 (P3) in Table 5. The repeated prompts in the release give a free noise estimate for hosted models. On identical prompts, deepseek-v3-flash differs by 1.8 items out of 100 on average and by up to 12, and gpt-5.6-luna by 1.2 and up to 6. Most open models repeat exactly, so decoding was probably greedy, but the paper does not say so.

  13. Several headline numbers depend on pooling choices that the paper does not describe. The word problem rows of Table 5 match the released numbers only if the 12 English-language locales are used. Pooling all 23 locales misses the printed values by up to 4.9 points. In Tables 4 and 13, the printed numbers reproduce exactly only if the "comma-only" format is left out of the comma group and the "dot-only" and "space" formats are left out of the dot group. The text defines these groups with those formats included. The "+Lang" column mixes English locales back in, so it is not a target language subset. The right side of Figure 2 cannot be reproduced from the release.

  14. One of the three conflict modes rests on a notation that does not match the locale data. CD444 (comma separated decimals, digits are grouped in quadruplets) puts a comma every four digits (1,2345,6789) and is assigned to China. CLDR gives three-digit groups for Chinese, Japanese and Korean. Four-digit units belong to how numbers are read aloud (万, 亿), not to how digits are grouped on the page. Six of the 20 High-tier cells and the whole first conflict mode depend on this format, although Table 3 says each locale is paired with attested conventions.

Questions for authors
  1. Could you release the scoring code and the per-item model answers? In particular, how does the digit check treat zeros and the position of the decimal mark, and how is '8,887' scored against the reference value of '8,887.000'?
  2. Could you report the High tier separately for addition and subtraction and for multiplication and division, on all four metrics? Do the main claims still hold on addition and subtraction alone?
  3. What do simple reference readers score (always the dot reading, always the comma reading), and what do the models score on the same operands written without any ambiguity? Have you tested whether the word problems can be solved without the question text or from the operator alone, as Patel et al. (2021) did for other benchmarks?
  4. How does V-Any score <3,361> against the references <3,361.000> and <3.361>, and how does it score <3.361> against <3,361>? What does the 'reading_ambiguous' counter in the released file count, and why is it not reported?
  5. Why does every High-tier multiplication contain a number ending in zero, and why are the thousands-reading references padded with zeros (ex: <8,887.000>)?
  6. No prompt asks for an output notation, yet value-correct requires one. For <789 + 456> the correct would be <1245>, <1 245>, <1,245>, <1.245> or <1'245> depending on a hidden format code. Why was the target notation scored? Could Figure 1 and Tables 4 and 13 also be given on V-Any, and could you add an upper-bound prompt that names the notation?
  7. What source supports each country-format pair? In particular, what supports a comma every four digits for China and Japan (CLDR gives three), the apostrophe-plus-dot format for Colombia, and U+2019 instead of U+0027 for Switzerland? Please list the code point of every separator and say whether Unicode normalization was applied.
  8. Are CH-de, CH-en and CH-fr (and the other same-country pairs) the same prompt under P3 and P5? If so, could you remove the repeats, recompute the totals and give intervals?
  9. Which locales and formats are pooled in the word-problem rows of Table 5, in the group definitions of Tables 4 and 13, in the "+Lang" column and in the right side of Figure 2? Why are the comma-only, dot-only and space formats left out of the groups?
  10. Which decoding settings, seeds, reasoning modes, maximum lengths and exact model versions and dates were used? At what precision was each local model run, and does the full-precision Mixtral give the same scores as the 4-bit copy?
  11. How many annotators reviewed each language, and how many edits and rejections did they make? The guideline says decimals belong only on measurable things. How did reviewers treat High-tier items where one reading gives a fraction of a countable object, such as 1.468 doughnuts?
  12. How was "plausible scale" checked when units are swapped without converting the numbers (for example centimeters for the UK and inches for the US in the same template)?
  13. Two rows look odd. At the bare prompt apertus-70b has a format score of 75.7 and a digit score of 8.0. On word problems gpt-5.6-luna has a digit score of 51.9 and a format score of 18.6. Are these answer-extraction failures, and how were failed extractions counted?
7 comments

Thank you for your long and detail feedback, we will answer each of your questions which reflect that weakness you highlight below.

  1. Could you release the scoring code and the per-item model answers? In particular, how does the digit check treat zeros and the position of the decimal mark, and how is '8,887' scored against the reference value of '8,887.000'?

In our evaluation depending on the metric, we use all the number formatting conventions to test what the number can mean. Given a number format, we use the decimal marker to remove an non-significant zeroes before comparing the numbers.

  1. Could you report the High tier separately for addition and subtraction and for multiplication and division, on all four metrics? Do the main claims still hold on addition and subtraction alone?

The table below has the requested results. We observe no large difference in trend between operations, we've also provided the standard deviation.

OperationD-onlySDCD (333)CD (2223)CD (444)C-onlySCDCACADCAll Formats
Addition77.27%71.64%68.73%69.45%57.33%31.15%29.94%24.18%30.67%24.36%45.13%
Subtraction72.55%72.73%71.76%72.36%59.27%25.82%25.33%19.91%25.33%19.73%42.96%
Multiplication48.73%21.45%33.82%20.73%28.73%32.36%21.70%17.45%20.85%17.45%25.15%
Division66.00%64.73%63.52%64.73%48.73%35.88%34.06%29.09%34.30%28.55%44.52%
All Operations66.14%57.64%59.45%56.82%48.52%31.30%27.76%22.66%27.79%22.52%39.44%
Std Dev12.49%24.38%17.43%24.26%13.96%4.17%5.39%5.11%5.91%4.94%9.57%
  1. What do simple reference readers score (always the dot reading, always the comma reading), and what do the models score on the same operands written without any ambiguity? Have you tested whether the word problems can be solved without the question text or from the operator alone, as Patel et al. (2021) did for other benchmarks?

Reference readers: Our goal is to characterise behavioural preferences across 11 diverse models (including region-specialised models e.g., Apertus, EuroLLM, Qwen, DeepSeek, Aya) rather than benchmark against naive heuristics. Because our conflict splits are balanced, a fixed single-convention reader achieves a known analytical ceiling (~50% on two-way conflicts, ~33.3% on three-way conflicts), which we will state in Section 5.1 as an anchor for our cross-model comparisons.

Unambiguous counterparts: Rewriting ambiguous operands into unambiguous forms is non-unique (e.g., 1 as 1.0, 1.00, or 01) and alters string length, precision, and tokenisation, confounding the exact surface ambiguity our research questions isolate. Instead, our Unambiguous × Unambiguous tier (Table 5) provides the controlled baseline for unambiguous notation across all 10 formats.

Question-stripped / operator-only ablation (Patel et al., 2021): Figure 1 and Table 5 already perform this comparison by contrasting Symbolic Operations (raw operands and operators only) with Localised Word Problems. Rather than exploiting shallow shortcuts, models drop in accuracy when narrative text is added (e.g., High-Ambiguity P1 VC drops from 27.79% to 13.70%), confirming that models cannot solve the word problems via operator-only heuristics. We will cite Patel et al. (2021) when discussing this comparison in Section 5.2.

Due to the character limit, we continue our response here part 2

  1. How does V-Any score <3,361> against the references <3,361.000> and <3.361>, and how does it score <3.361> against <3,361>? What does the 'reading_ambiguous' counter in the released file count, and why is it not reported?

V-Any (value_correct_under_any) scoring: V-Any evaluates whether the model's output parses to the ground-truth scalar value under any of the 10 supported formatting conventions, with non-significant trailing zeros normalised (stripped) across all formats:

  • <3,361> vs. <3,361.000>: Scores correct (1). Once trailing zeros are stripped, <3,361.000> represents 3361, and <3,361> parses to 3361 under dot-decimal grouping formats (CD (333), D-only).
  • <3,361> vs. <3.361> and <3.361> vs. <3,361>: Both score correct (1). Because V-Any checks whether the prediction matches the target scalar value under any valid format, <3,361> (parsed as 3361 under CD (333) or 3.361 under comma-decimal formats) and <3.361> (parsed as 3.361 under D-only or 3361 under DC) both admit a valid format parse matching the underlying scalar target (3361 or 3.361). We deliberately report V-Any alongside strict VC and DS in Table 5 to show that even under this permissive upper bound, performance remains low.

reading_ambiguous column: reading_ambiguous is a legacy diagnostic counter left over from early exploratory experiments (tracking whether a model's surface output string admits multiple distinct numerical readings across formats). It is not used in our evaluation or claims, and we will document this clearly in the repository README and remove/annotate the unused column in the released summary file.

  1. Why does every High-tier multiplication contain a number ending in zero, and why are the thousands-reading references padded with zeros (ex: <8,887.000>)? Trailing zeros in High-Ambiguity multiplication operands are included deliberately to ensure outcome divergence across competing conventions. When an operand is parsed as a decimal (e.g., 1.230), trailing zeros are non-significant and vanish, whereas when parsed as a grouped integer (1,230 or 1.230 as thousands), those zeros remain significant, guaranteeing that dot-decimal and comma-decimal interpretations yield distinct numerical products and preventing accidental value collisions. Reference answers are formatted to preserve uniform precision across competing readings of the same ambiguous input (for example, when a three-decimal-place reading such as 8.887 is contrasted with an integer thousands reading such as 8,887.000), ensuring consistent decimal-place representation across formats before normalisation.

  2. No prompt asks for an output notation, yet value-correct requires one. For <789 + 456> the correct would be <1245>, <1 245>, <1,245>, <1.245> or <1'245> depending on a hidden format code. Why was the target notation scored? Could Figure 1 and Tables 4 and 13 also be given on V-Any, and could you add an upper-bound prompt that names the notation?

Different metrics capture different information, value-correctness captures this directly and provides critical signal on the Standard-Ambiguity subset (where separator-free operands like 789 + 456 test whether models adapt their output format to regional narrative cues), whereas V-Any already isolates format-agnostic numerical accuracy. Moreover, we observed that the trends between V-Any and VC were similar but VC gave better distinctions. We will include the V-Any versions of Figure 1, Table 4, and Table 13 in the Appendix.

An upper-bound prompt that names the notation would mean that we would need to give explicit instruction which would make the results fall into a instruction-following measure, that is why we do not provide that level of prompting as stronger instruction-following model may incorrectly appear to be better at number format distinction.

  1. What source supports each country-format pair? In particular, what supports a comma every four digits for China and Japan (CLDR gives three), the apostrophe-plus-dot format for Colombia, and U+2019 instead of U+0027 for Switzerland? Please list the code point of every separator and say whether Unicode normalization was applied.

We do not use CLDR as this didn't agree with our native annotators. Instead we simply asked our annotators for the conventions uses in their locale. Regarding the unicode, we believe that the models used are robust enough to surmount unicode specific issues.

  1. Are CH-de, CH-en and CH-fr (and the other same-country pairs) the same prompt under P3 and P5? If so, could you remove the repeats, recompute the totals and give intervals? For the worded version, they are different as they are in different languages. However, for the symbolic version they are the same, exactly like P1, P2, P3, P4 and P5 in fact. But these are counted in repeats.

Due to the character limit, we continue our response here part 3 9. Which locales and formats are pooled in the word-problem rows of Table 5, in the group definitions of Tables 4 and 13, in the "+Lang" column and in the right side of Figure 2? Why are the comma-only, dot-only and space formats left out of the groups?

Pooled locales and formats: Table 5 pools across all locales and the applicable formats within each ambiguity tier, while Tables 4 and 13, the +Lang column, and Figure 2 (right panel) pool all locales belonging to each format comparison group.

Inclusion of C-only, D-only, and space-grouped formats (SD, SC): These formats are not excluded from the groups: in Table 13, C and D are shorthand labels for C-only and D-only, and space-grouped formats (SD, SC) are likewise included in their respective convention groups. We will standardise the format labels (C-only, D-only, SD, SC) across all tables and captions to avoid any ambiguity.

  1. Which decoding settings, seeds, reasoning modes, maximum lengths and exact model versions and dates were used? At what precision was each local model run, and does the full-precision Mixtral give the same scores as the 4-bit copy?

We used deterministic decoding (temperature = 0) wherever supported by the model or API, with explicit reasoning/thinking modes disabled and all remaining parameters (including top_p, seeds, and maximum generation lengths) kept at each model's or provider's default settings. While Appendices F and G currently outline our experimental setup, we will add these exact decoding settings, maximum token limits, and default parameters explicitly to Appendix G. Model versions, APIs, and numerical precision: Exact open-weights checkpoint identifiers, API endpoints/versions, and dates are listed in Appendices F and G. All local models were evaluated in their default native precision (BF16/FP16). We tried to stay within the model default settings as we believe these are what optimise the models and most user's default settings too.

  1. How many annotators reviewed each language, and how many edits and rejections did they make? The guideline says decimals belong only on measurable things. How did reviewers treat High-tier items where one reading gives a fraction of a countable object, such as 1.468 doughnuts?

Annotators reviewed the localisations; no questions were rejected, and items were only edited where needed. Tracking the number of edits would primarily reflect Gemini's localisation ability, which lies outside the scope of this research. With regard to measurable quantities, we made every effort to keep them plausible, though occasional edge cases may affect how a sentence reads; however, most items include an explicit unit to ensure they are measurable from the outset.

  1. How was "plausible scale" checked when units are swapped without converting the numbers (for example centimeters for the UK and inches for the US in the same template)?

Plausibility is not check in this work as the numbers are generated automatically and restricting the number to plausibility is in direct conflict with the comparisons between the many different axes that we look at.

  1. Two rows look odd. At the bare prompt apertus-70b has a format score of 75.7 and a digit score of 8.0. On word problems gpt-5.6-luna has a digit score of 51.9 and a format score of 18.6. Are these answer-extraction failures, and how were failed extractions counted?

Neither anomaly seem to come from answer-extraction failures. An extraction failure would mean that all metrics are set to zero.

  • apertus-70b (bare prompt: FS = 75.7, DS = 8.0): The model consistently produced extractable outputs adhering to the target regional number format (75.7%), but frequently failed the underlying arithmetic or separator disambiguation (8.0% digit accuracy). This is possible by producing a number like 52 which means that it does adhere to the number format all the time, but the answer is incorrect.
  • gpt-5.6-luna (word problems: DS = 51.9, FS = 18.6): The model frequently arrived at the right numerical digits (51.9%), but normalised its final output to a default/unformatted representation rather than preserving the prompt's regional formatting (18.6%).
Reviewer_UU5UOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents GLOBENUM, a benchmark for evaluating how LLMs understand numerical formats across different countries and languages. The authors mainly focus on ambiguous numbers where the same number can have different values depending on the regional format. They evaluate 11 models using symbolic operations and localised word problems. The results show that models have a strong preference for dot-decimal formats and often fail to use contextual information correctly.

Critical strengths
  • The motivation of the paper is clear. Numerical formatting is an important issue which has not recieved much attention in existing numerical reasoning benchmarks.
  • The ambiguity taxonomy is interesting, especially the distinction between format ambiguity and value ambiguity. It helps to understand which cases actually need contextual information.
  • The benchmark covers a good number of numerical formats, languages and models. The human verification of the localised problems is also a positive aspect.
Critical weaknesses
  1. The authors mainly use arithmetic operations to check whether models can understand the numbers correctly. But if a model gives a wrong answer, it is not clear whether it failed to interpret the number or just made an arithmetic mistake. Although they have separate metrics, I think this has not been properly analyzed.

  2. The cultural context analysis does not seem to be testing the correct behavior. In Figure 2, the authors compare symbolic operations with localised word problems and then consider the performance difference as the effect of cultural context. But these two are different tasks and also word problems require more language understanding. So I think this comparison does not properly isolate the impact of cultural information.

  3. Another thing I found is that the benchmark always expects a numerical answer, even when the number is ambiguous without enough context. For example, 1.234 can have two different meanings based on the convention. And In such cases, a model that identifies the ambiguity and asking for clarification could also be considered correct. But the benchmark does not consider this pattern, which I believe is important for context-aware numerical reasoning.

  4. I also noticed an issue with the prompting setup. In P1, the authors only provide the mathematical expression, but from P2 onwards they also introduce system instructions and JSON output format. So when comparing P1 with P2, multiple things are changing at the same time. It is not clear whether the performance difference comes from the additional instruction or the change in output format.

Questions for authors
  • Can the authors provide additional analysis to distinguish numerical interpretation errors from arithmetic calculation errors?
  • How do the authors justify comparing symbolic operations with localised word problems in order to measure the effect of cultural context, as both tasks have different language complexity?
  • Did the authors consider evaluating whether models can identify ambiguous numbers and ask for clarification when the given context is not sufficient?
  • In the prompting experiments, why does P1 use a different output format and system instruction compared to P2–P5? Could this affect the comparison between prompting strategies?
2 comments

Thank you for your comments, we address the questions you posed directly as these reflect the weakness you highlighted.

  1. The authors mainly use arithmetic operations to check whether models can understand the numbers correctly. But if a model gives a wrong answer, it is not clear whether it failed to interpret the number or just made an arithmetic mistake. Although they have separate metrics, I think this has not been properly analyzed.

Instead of comparing whether the models get the arithmetic right or wrong, we compare the difference between getting it more correct in different number format, mainly US/UK format compared to other regional formats. To give a more categorical comparison, below you can find a table with the average results per number format for unamb x unamb under the most basic prompt P1 on symbolic problems. Here the numbers are always the same just written under different format, and the expected answers are always the same. We use the Digit Sequence Correctness metric which strips all formatting from the model output and purely looks at the sequence of digits. We can clearly see that the model is strongest with CD (333) and D-only which are the conventional UK/US number formats and weaker in the other ones. This shows that the models perform better when the numbers are express in that way than in other number formats.

Number FormatD-onlySDCD (333)CD (2223)CD (444)C-onlySCDCACADC
Average Value Correct Rate (%)41.438.347.424.025.835.030.737.336.733.1
  1. The cultural context analysis does not seem to be testing the correct behavior. In Figure 2, the authors compare symbolic operations with localised word problems and then consider the performance difference as the effect of cultural context. But these two are different tasks and also word problems require more language understanding. So I think this comparison does not properly isolate the impact of cultural information.

This is true, the ideal setting would be to add culture without having to add words but in a symbolic setting the only way to do it is by providing external information such as number format, location and instruction to particularly consider those, yet this is what we do. An alternative would be to compare the original text translated into the locale’s language without localising it, UK-xx. For example, having the problem in French but with the names, currencies and objects translated from the UK questions. The results for this experiment will be ready for the camera-ready version.

  1. Another thing I found is that the benchmark always expects a numerical answer, even when the number is ambiguous without enough context. For example, 1.234 can have two different meanings based on the convention. And In such cases, a model that identifies the ambiguity and asking for clarification could also be considered correct. But the benchmark does not consider this pattern, which I believe is important for context-aware numerical reasoning.

This is an interesting behaviour to look at and investigate in future work especially now that we’ve established some key issues of LLM regarding number formatting. We ask models to give a numerical answer so that we can see what their base answer is, even under ambiguous formatting to understand their preference/biases. This also allows us to evaluate the model in a more consistent way.

  1. I also noticed an issue with the prompting setup. In P1, the authors only provide the mathematical expression, but from P2 onwards they also introduce system instructions and JSON output format. So when comparing P1 with P2, multiple things are changing at the same time. It is not clear whether the performance difference comes from the additional instruction or the change in output format.

As mentioned in Appendix D, we are aware of this and will clarify our motivation for this choice. The choice was to ensure that P1 was completely context and language free baseline, hence why we didn’t even include an English system prompt. This ensures that P1 is a true baseline, and any non-number format specific instruction should support that model to activate any internal knowledge it has. Moreover, we believe that this might affect the specific accuracy values but not the trend we observe. P2 acts as baseline for worded problems.

I appreciate the authors' responses. Some concerns required significant revisions to the paper, and after considering other reviewers' concerns and responses, I will maintain my initial rating.

Reviewer_qn7rOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces GLOBENUM, a benchmark for evaluating whether large language models can correctly interpret numerical representations across different cultural and regional conventions. The authors distinguish between unambiguous, format-ambiguous, and value-ambiguous numerical expressions, and construct both symbolic arithmetic tasks and culturally localised word problems.

Critical strengths
  1. An interesting and underexplored problem.

  2. Broad evaluation coverage.

Critical weaknesses
  1. The experimental investigation is rather shallow, particularly regarding prompting strategies.

    Although the underlying research question is interesting, the experimental design is relatively straightforward and offers limited insight into why the models fail. The five prompting conditions (P1–P5) mainly vary whether a location or a generic formatting reminder is included. For example, P5 simply adds the location and the instruction "Pay attention to the numerical formatting."

    I find this insufficient to support the broader conclusions about models' inability to perform context-aware numerical reasoning. In particular, the experiments do not distinguish whether models lack knowledge of regional numerical conventions or simply fail to activate and apply that knowledge under weak prompting.

    More informative comparisons would include explicit formatting rules, few-shot demonstrations, and instructions requiring models to first identify the numerical convention, parse the operands, and then perform the calculation. These are fairly natural prompting strategies for this task, yet none are evaluated.

    Consequently, it remains unclear whether the reported failures represent a fundamental limitation in numerical understanding or a limitation of the particular prompting setup. In my view, this substantially limits the insights offered by the experimental results.

  2. Insufficient analysis of the causes of numerical-formatting failures and the role of localisation.

    The paper acknowledges that a single country may use multiple numerical conventions. However, the geographic prompting conditions specify only the country, without necessarily identifying which convention is intended. For genuinely value-ambiguous expressions, such context may still be insufficient to determine a unique interpretation. This raises concerns about attributing incorrect predictions to failures of cultural reasoning.

    Furthermore, the reported Target Value Correctness metric requires both numerical correctness and exact adherence to the expected output format. While the authors provide additional metrics, the analysis does not sufficiently disentangle failures in format identification, numerical parsing, arithmetic reasoning, and answer formatting.

    The analysis of cultural context is also somewhat limited. Comparing symbolic expressions against localised word problems changes several aspects of the task simultaneously, making it difficult to isolate the contribution of cultural knowledge itself.

    Finally, while the evaluation includes several multilingual and regionally oriented models, it provides limited evidence about whether models specifically adapted to individual linguistic or regional communities would behave differently.

Questions for authors
  1. How much of the reported performance degradation can be resolved through stronger prompting? In particular, have the authors considered evaluating explicit numerical-format instructions, few-shot examples, or a two-stage procedure in which the model first parses the operands and then performs the arithmetic? I believe such experiments are important for distinguishing a lack of regional numerical knowledge from a failure to apply existing knowledge.

  2. How is the intended numerical convention uniquely determined when multiple formats are valid for the same locale? Given that some country–language pairs are associated with multiple formatting conventions, it is unclear whether geographic context alone provides sufficient information to resolve all high-ambiguity examples. How would the conclusions change if alternative valid interpretations were considered or the intended convention were explicitly specified?

  3. Are the observed limitations specific to general-purpose multilingual models? Would the authors expect models trained or adapted specifically for particular regional communities to exhibit similar biases? Additional evidence on this distinction would help clarify the scope of the paper's conclusions.

1 comment

Thank you for your comments, we address the questions you posed directly as these encompass the weakness you highlighted.

  1. How much of the reported performance degradation can be resolved through stronger prompting? In particular, have the authors considered evaluating explicit numerical-format instructions, few-shot examples, or a two-stage procedure in which the model first parses the operands and then performs the arithmetic? I believe such experiments are important for distinguishing a lack of regional numerical knowledge from a failure to apply existing knowledge.

You raise an interesting point. We believe that giving this kind of information would test models’ instruction-following capability rather than understanding what the models’ fundamental preferences/biases are or testing whether the context helps the model adapt. As a scenario, models’ robustness should be tested in the context that a user naturally writes the number in their locale’s formatting and the model should, at best, know their location and know what they mean. In this case, we wouldn’t expect the user to even realise that what they write might be ambiguous in a different location.

  1. How is the intended numerical convention uniquely determined when multiple formats are valid for the same locale? Given that some country–language pairs are associated with multiple formatting conventions, it is unclear whether geographic context alone provides sufficient information to resolve all high-ambiguity examples. How would the conclusions change if alternative valid interpretations were considered or the intended convention were explicitly specified?

To ensure that we allow the model to express its final answer in whichever format it desires (even one that doesn’t belong to the locale) we use Any-Format correctness (V-any) which reads the output into all the formats and decides whether the final answer can be read as the correct value. All the metrics are described in lines 324-334. We even look as Digit-Sequence correctness which omits any effect of the number format. However, when the question is in a specific format that can be identified (Standard ambiguity) we also look at Format-value which verifies is the same format as the one used in the question has been reproduced in the answer.

  1. Are the observed limitations specific to general-purpose multilingual models? Would the authors expect models trained or adapted specifically for particular regional communities to exhibit similar biases? Additional evidence on this distinction would help clarify the scope of the paper's conclusions.

We find that regional specific models which we explicitly test for this purpose i.e. Apertus (Switzerland), EuroLLM (European conventions BE, CH, FR, ES, DE), and also Aya-expanse which is not region specific but trained to be a multilingual model that accounts for lots of languages and cultures. All these models show similar biases to the general-purpose multilingual models with strong preference towards UK/US conventions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.