- Rating
- 2(+1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper presents GlobeNum, a dataset/benchmark that asks whether language models can read and write numbers in the notation used in different parts of the world. A string can mean different values based on the region: "1,234" would be "thousand two hundred thirty four" in the US but "one point two three four" in France. The benchmark covers ten notation formats, twelve countries and seven languages. It has 7,000 symbolic arithmetic items and ~35k localized word problems built from FERMAT templates. Items are sorted into tiers by how much the number string reveals about its own notation. The ambiguity sits in both decimal/fractional separators (dot or comma) and digits grouping (triplets, duplets , quadruplets). Five prompt conditions add context step by step, from a plain instruction to the user's country and a hint to watch the formatting. Eleven models are tested. The authors report a strong preference for dot-decimal readings that persists with country cues, a drop in comma-decimal locales when the stories are localized, and a best pooled score of ~32% on the most ambiguous tier.
This paper takes on a real and neglected problem. The same written number can mean different things in different countries, simply because of different cultural notations for decimals. A language model can get the size of an answer wrong by an error of several orders of magnitude, yet still sound confident. The authors build a benchmark. hey sort items by how much the number itself reveals, then add context step by step, such as the country and a hint about formatting. They conclude that models lean toward the US style even when told where the user lives.
I value the question, the clear logic behind the design and the wide choice of languages and models. I also thank the authors for releasing their data. It allowed me to recompute the numbers and several tables, and those checks mostly came out right. A revised version of the paper could become a very good benchmark, but I do not think the current evidence supports the main claims. Also I am concerned about significant flaws or caveats in the design, which are undocumented at this point. Since the paper and the data are build around special symbols in the notation (commas, dots, apostrophes) I sometimes use triangular brackets further in the review '< >', in order to avoid ambiguity in the data representation.
My first concern is that in the hardest tier, many items cannot show which reading a model used. For multiplication and division, both readings of an ambiguous number give the same digits. For example, 6,722 divided by 2 is written 3,361 under either reading. A model can earn credit for the right digits without reading the number correctly. The paper drops one tier for exactly this reason, but the same reason applies to half of the items in the tier it keeps. The "any format" (V-any) score meant to separate notation from value has the same blind spot. The paper also never checks whether its word problems can be solved for the wrong reasons, a danger that earlier work on math benchmarks has shown.
My second concern is that the paper offers no reference points. It does not say what a reader that always picks one reading would score, or what the models score on the same arithmetic without any ambiguity. Many results sit near the floor. The claim that bias shrinks as context grows looks like a floor effect to me. In the released numbers, the models mostly stop using the requested notation once the stories are added, so both groups fall together, while the value-only score of the favored reading hardly changes.
My third concern is that the scoring asks for something no prompt requests. The output notation is never named, yet an answer is marked wrong if it uses a different one. Other design choices add noise. Multiplication numbers always end in zero, correct answers are padded with zeros, one format for China does not match the standard locale data, and the Swiss separator differs from that data. Some word problems turn into odd statements under one reading, such as a fraction of a doughnut, which the authors' own annotation guideline says should be avoided. More than a quarter of the symbolic country-cue runs repeat an identical prompt under a different language label.
My fourth concern is reproducibility. The release holds data and summary counts, but it has no code for building items, prompting or scoring, no raw answers and no annotation records. The paper gives no decoding settings, seeds or confidence intervals. One large model ran from a third-party 4-bit copy, and the precision of the others is not stated. Several tables depend on pooling choices that the paper does not explain.
Most of these problems come from how the benchmark is built and scored, so fixing them would change the main tables and the claims. That is a job for a revision and not for a rebuttal. The released numbers already hint at a clear finding, which is that writing the problem in the local language moves the model's reading far more than the country line does. A revised paper could lead with that finding, backed by results for addition and subtraction alone, reference baselines, the scoring code and clean pooling choices. For now I lean toward rejection, and I would welcome a resubmission.
Critical strengths
- The problem is real. A number such as 1,234 or 1.234 can be read at the wrong size without any warning, and the answer still looks valid. This matters for any service that reaches users in more than one region.
- The topic has been little studied. Multilingual math benchmarks usually translate the words and leave the numbers in one notation. This paper moves the focus to the number itself. I did a rigorous literature search regarding the problem, and it is quite understudied indeed.
- The benchmark is new within its stated scope. I know of no other set that mixes strings whose value depends on the notation with context that could settle them, such as country, language and a formatting hint.
- The intention of the design is stated clearly. Operands are sorted into unambiguous, format-ambiguous and value-ambiguous types, and the paper explains why the tier with two value-ambiguous numbers is left out. This gives readers a clear vocabulary and a reason for each tier.
- The set of languages and countries is good. It covers seven languages, including Tamil, Korean and Chinese, 23 country-language pairs and ten notation formats. A parallel symbolic and word-problem version lets the paper separate the numbers from the story around them. The appendix describes a review of every template by native speakers.
- The model set is varied. Eleven models cover hosted and open weights, small and large sizes, and region-focused models such as Apertus, EuroLLM and Aya. The finding that the region-focused models show no clear advantage is useful.
- The data release made the checking possible. I recomputed all 7,000 referenced answers from the released files and every one was correct. The item counts match the paper. The symbolic rows of Table 5 and the numbers in Figure 1 reproduce exactly from the aggregate file.
- The released numbers already hold a clear finding. Pooled over locales, the less-favored reading scores 3.7 on the value-only metric in English-language stories and 21.4 in local-language stories (comma versus thousands split), and 11.4 against 30.3 in the dot versus dot-grouping split. The country line alone moves very little. This language effect deserves to lead the paper, although it needs a per-country check.
Critical weaknesses
-
Most problems come from how the benchmark is built and scored, so a revision could fix them but I have doubts that a rebuttal could do it. Repairing them changes the central tables and several claims. The review guidelines say new experiments should not significantly change the content of a submission.
-
The paper does not ask whether its word problems can be answered for the wrong reasons. Patel et al. (2021) is cited once, as an example of a benchmark that gets translated (Section 2.1). Its main finding goes unused. Trained solvers that never saw the question still solved ~65% of ASDiv-A and ~75% of MAWPS (a Mathematical Word Problem could be reduced to a bag-of-words + arithmetic operation classifier, providing surprisingly high accuracy on the simple tasks; it proves that lots of tasks could be answered by the models without any actual reasoning). A similar false positive exists at this paper in another form. In the High tier, every multiplication and division has the same digits under both readings of the ambiguous number, so a model can earn digit credit without reading that number at all. The paper has no question-free, operator-only or reading-blind baseline that would show how much credit is real. The tests authors propose could have a decent amount of false-positive resolution acts and that should be addressed
-
There are no baselines or null results, and some findings sit on a floor that the design creates.
- A "cell" is one notation format inside one conflict mode. Twelve of the 20 High-tier cells have a comma-decimal target and eight have a thousands target. A reader that ignores context and always picks one reading therefore matches the target in 60% or 40% of cells before any arithmetic. By my calculation, a context-blind reader with perfect arithmetic would reach 55 to 88% digit credit in this tier, depending on how zeros are scored, which the paper does not say. The models reach 40 to 45%. The paper cannot say how much of that gap is reading and how much is arithmetic on long numbers.
- The claim that format bias decreases with context (Table 4) is a floor effect. In the comma versus thousands split, the value-correct score falls from 55.6 to 11.6 for the favored group and from 11.8 to 2.0 for the other group, while stories and the JSON wrapper are added. At the same step the favored group format score drops from 84% to 23%, so the models stop using the target notation. On the value-only metric the favored group barely moves (68.6 to 63.2) and the other group falls from 12.2 to 3.0. The claim authors make as a finding doesn't exist: nothing became less biased.
-
Almost nothing needed to reproduce the study is released. The README lists the benchmark files, the word-problem templates and one aggregate results file. There is no code that generates the items, no prompting or inference code (only examples in the appendix), no scoring code, no raw model answers, no annotation records and no run configuration. The paper gives no decoding settings, seeds or reasoning modes. A reader cannot check how an answer was extracted, how zeros were handled or how borderline strings were scored. The main numbers can be checked only through 28,424 aggregate rows, each a count out of 100.
-
The "any format" score does not do its job on the High tier, especially for a division operation. V-Any is meant to forgive the wrong notation while still checking the value. In the High tier the value itself depends on the notation. Take
6,722 / 2from the data. The reference is 3,361.000 if the comma is a thousands mark and 3,361 (meaning 3.361) if it is a decimal comma. The same answer string,3,361, is correct in both cells, and the model never had to choose a reading. A model that read 6,722 as 6.722 and wrote3.361could also pass the thousands cell, because3.361is a valid way to write 3,361 in the dot-grouping formats. V-Any is informative mainly on addition and subtraction (see point 10). The released file also contains a counter named 'reading_ambiguous' that the paper never mentions. It averages about 35 per 100 answers in the High tier and 1.5 in the Standard tiers. -
The choice of separator characters is debatable and undocumented. CLDR (the Unicode locale data, checked against the current repository) gives comma U+002C and full stop U+002E for the decimal and grouping marks in most locales. It gives the apostrophe U+0027 as the grouping mark for the Swiss locales (de-CH, fr-CH, en-CH, it-CH), U+202F for French and U+00A0 for Russian and South African English. The dataset's Swiss format (AC) uses the typographic apostrophe U+2019 instead. The Colombian format (ADC) uses U+0027 together with the full stop, which CLDR does not list for es-CO. The space formats (SD, SC) use a plain space U+0020, the same character that separates operands from operators. The JSON files store all of these as raw UTF-8 text with no '\uXXXX' escapes, so the code point can only be learned from the bytes. For example, 'AC.json' contains <1’477’138,90551 + 840’118’893,88336> (U+2019, bytes E2 80 99) and 'ADC.json' contains <1'477.138,90551 + 840'118.893,88336> (U+0027). In my checks of eight tokenizers, look-alike characters received different token IDs. Pipelines that normalize Unicode also turn U+00A0, U+202F and U+2009 into a plain space. The paper should state each code point and give a source for it.
-
Some word problems become odd under one of the two readings, and the paper does not control for this. A value-ambiguous number attached to a countable object turns into a fraction under the decimal reading. One item reads as: "Mr. Rutledge bought 2,936in of doughnut dough, he splits them equally into 2in for each doughnut. How many doughnuts does he have on average?" The answer is 1,468 doughnuts, or 1.468 doughnuts if the comma is a decimal mark. Other items give absurd sizes under one reading, such as "Each sausage cost 38,320 Dollars" (38.32 under the decimal reading) and "her 433lb dog" with "7,100g of dog treats for each lb". The authors own annotation guideline (Appendix C.3, check 3) says a decimal must attach to something measurable and not to a whole item. The High tier needs both readings to be possible, so it produces exactly the sentences the guideline rules out. This hands the model a plausibility cue that the benchmark does not account for. It may also make some items harder, because the reader meets a surprising quantity either way. The paper does not test whether this helps or hurts. A simple check would compare accuracy on items where one reading is absurd with items where both are plausible.
-
The generator leaves two kinds of trailing-zero artifacts. Firstly, in all 75 multiplications of the High tier (25 in each conflict mode) the value-ambiguous number ends in zero, for example <4 * 19,700> and <8 * 2,000>. For the other operators only 3 to 8 of 25 do. Such a product has the same digits under both readings (78,800 or 78.8). Secondly, the reference answers for the thousands reading carry zeros that nobody would write. For
8,881 + 6the reference is8,887.000, against14,881for the decimal-comma reading. 762 reference answers in the dataset are whole numbers written with a zero fraction. The paper explains neither rule and does not say how8,887is scored against8,887.000. -
Accuracy is the wrong yardstick for the combined country-and-hint condition (paper's P5), and repeated prompts distort it.
- The symbolic P3 and P5 runs contain 25,400 instances per model, but 6,900 of them (27%) repeat an earlier prompt. The prompt is in English and names only the country, so CH-de, CH-en and CH-fr (and CN-en/CN-zh, IN-en/IN-ta, US-en/US-es) send identical text. For apertus-8b under P5, on the comma versus thousands split with format C, the three Swiss rows (IDs 25918, 25929, 25940) are identical: 3 exact, 86 format-correct, 11 digit-correct and 9 value-correct out of 100. The German row (id 25962) differs only slightly, with 10 and 8. Switzerland therefore counts three times, and any interval that treats the 25,400 as independent would be too narrow.
- One prompt is also scored against several references. With <Location: United States>, the item <382 + 819> is scored against 1201 in the D cell, 1 201 in the SD cell and 1,201 in the CD(333) cell, and no single answer can match all three. Accuracy under P5 has a ceiling below 100% that comes from the scoring and not from the models.
-
Reading-blindness is visible directly in the released reference answers. For each expression I compared the reference under the thousands reading with the reference under the decimal-comma reading in the comma versus thousands split. For addition and subtraction the digits always differ (0 of 500 pairs match). For <8,881 + 6> the answer is 8,887 or 14.881. For multiplication and division the digits always match once zeros are stripped (490 of 490 pairs). For <14,596 / 8> the gold is 1,824.500 or 1,8245, and both are made of the digits 18245. The authors leave out the tier with two value-ambiguous numbers for this very reason (footnote to Table 2), but the same reason applies to half of the items in the tier they keep. Only 2,000 of the 7,000 symbolic items (one third roughly) can show through the digit sequence which reading a model used.
-
The paper does not say at what precision most local models were run. Appendix G notes that the Mixtral-8x22B checkpoint is a third-party AWQ quantization, and its model card is tagged 4-bit. Nothing is said about the other local models in the released file (Qwen2.5-72B, Apertus-70B, Aya-Expanse-32B, Qwen3.8-27B, Ministral-3-8B). Published studies find that 4-bit weight-only quantization usually stays within about 2% of full precision on average, while math and multi-step reasoning are more sensitive than other abilities and Mixtral-style mixture-of-experts models are more sensitive than dense ones (Li et al., 2024). This benchmark depends on exact digit strings, so the effect may matter, and a weak Mixtral score cannot be separated from the effect of compression. A same-model comparison of the quantized and full-precision checkpoints on the F×F tier would settle it cheaply.
-
No confidence intervals, seeds, bootstraps or noise estimates are given, so most figures cannot be read properly. Each model-cell has 100 items, so a rate near 10% already carries a standard error of about 3 points, and the same 100 operand pairs are reused in all ten formats, so cells are not independent. The paper interprets differences of one to three points, for example High-tier digit credit of 40.95 (P2) against 39.89 (P3) in Table 5. The repeated prompts in the release give a free noise estimate for hosted models. On identical prompts, deepseek-v3-flash differs by 1.8 items out of 100 on average and by up to 12, and gpt-5.6-luna by 1.2 and up to 6. Most open models repeat exactly, so decoding was probably greedy, but the paper does not say so.
-
Several headline numbers depend on pooling choices that the paper does not describe. The word problem rows of Table 5 match the released numbers only if the 12 English-language locales are used. Pooling all 23 locales misses the printed values by up to 4.9 points. In Tables 4 and 13, the printed numbers reproduce exactly only if the "comma-only" format is left out of the comma group and the "dot-only" and "space" formats are left out of the dot group. The text defines these groups with those formats included. The "+Lang" column mixes English locales back in, so it is not a target language subset. The right side of Figure 2 cannot be reproduced from the release.
-
One of the three conflict modes rests on a notation that does not match the locale data. CD444 (comma separated decimals, digits are grouped in quadruplets) puts a comma every four digits (
1,2345,6789) and is assigned to China. CLDR gives three-digit groups for Chinese, Japanese and Korean. Four-digit units belong to how numbers are read aloud (万, 亿), not to how digits are grouped on the page. Six of the 20 High-tier cells and the whole first conflict mode depend on this format, although Table 3 says each locale is paired with attested conventions.
Questions for authors
- Could you release the scoring code and the per-item model answers? In particular, how does the digit check treat zeros and the position of the decimal mark, and how is '8,887' scored against the reference value of '8,887.000'?
- Could you report the High tier separately for addition and subtraction and for multiplication and division, on all four metrics? Do the main claims still hold on addition and subtraction alone?
- What do simple reference readers score (always the dot reading, always the comma reading), and what do the models score on the same operands written without any ambiguity? Have you tested whether the word problems can be solved without the question text or from the operator alone, as Patel et al. (2021) did for other benchmarks?
- How does V-Any score <3,361> against the references <3,361.000> and <3.361>, and how does it score <3.361> against <3,361>? What does the 'reading_ambiguous' counter in the released file count, and why is it not reported?
- Why does every High-tier multiplication contain a number ending in zero, and why are the thousands-reading references padded with zeros (ex: <8,887.000>)?
- No prompt asks for an output notation, yet value-correct requires one. For <789 + 456> the correct would be <1245>, <1 245>, <1,245>, <1.245> or <1'245> depending on a hidden format code. Why was the target notation scored? Could Figure 1 and Tables 4 and 13 also be given on V-Any, and could you add an upper-bound prompt that names the notation?
- What source supports each country-format pair? In particular, what supports a comma every four digits for China and Japan (CLDR gives three), the apostrophe-plus-dot format for Colombia, and U+2019 instead of U+0027 for Switzerland? Please list the code point of every separator and say whether Unicode normalization was applied.
- Are CH-de, CH-en and CH-fr (and the other same-country pairs) the same prompt under P3 and P5? If so, could you remove the repeats, recompute the totals and give intervals?
- Which locales and formats are pooled in the word-problem rows of Table 5, in the group definitions of Tables 4 and 13, in the "+Lang" column and in the right side of Figure 2? Why are the comma-only, dot-only and space formats left out of the groups?
- Which decoding settings, seeds, reasoning modes, maximum lengths and exact model versions and dates were used? At what precision was each local model run, and does the full-precision Mixtral give the same scores as the 4-bit copy?
- How many annotators reviewed each language, and how many edits and rejections did they make? The guideline says decimals belong only on measurable things. How did reviewers treat High-tier items where one reading gives a fraction of a countable object, such as 1.468 doughnuts?
- How was "plausible scale" checked when units are swapped without converting the numbers (for example centimeters for the UK and inches for the US in the same template)?
- Two rows look odd. At the bare prompt apertus-70b has a format score of 75.7 and a digit score of 8.0. On word problems gpt-5.6-luna has a digit score of 51.9 and a format score of 18.6. Are these answer-extraction failures, and how were failed extractions counted?
7 comments
Due to the character limit, we continue our response here part 2
- How does V-Any score <3,361> against the references <3,361.000> and <3.361>, and how does it score <3.361> against <3,361>? What does the 'reading_ambiguous' counter in the released file count, and why is it not reported?
V-Any (value_correct_under_any) scoring: V-Any evaluates whether the model's output parses to the ground-truth scalar value under any of the 10 supported formatting conventions, with non-significant trailing zeros normalised (stripped) across all formats:
- <3,361> vs. <3,361.000>: Scores correct (1). Once trailing zeros are stripped, <3,361.000> represents 3361, and <3,361> parses to 3361 under dot-decimal grouping formats (CD (333), D-only).
- <3,361> vs. <3.361> and <3.361> vs. <3,361>: Both score correct (1). Because V-Any checks whether the prediction matches the target scalar value under any valid format, <3,361> (parsed as 3361 under CD (333) or 3.361 under comma-decimal formats) and <3.361> (parsed as 3.361 under D-only or 3361 under DC) both admit a valid format parse matching the underlying scalar target (3361 or 3.361). We deliberately report V-Any alongside strict VC and DS in Table 5 to show that even under this permissive upper bound, performance remains low.
reading_ambiguous column: reading_ambiguous is a legacy diagnostic counter left over from early exploratory experiments (tracking whether a model's surface output string admits multiple distinct numerical readings across formats). It is not used in our evaluation or claims, and we will document this clearly in the repository README and remove/annotate the unused column in the released summary file.
-
Why does every High-tier multiplication contain a number ending in zero, and why are the thousands-reading references padded with zeros (ex: <8,887.000>)? Trailing zeros in High-Ambiguity multiplication operands are included deliberately to ensure outcome divergence across competing conventions. When an operand is parsed as a decimal (e.g., 1.230), trailing zeros are non-significant and vanish, whereas when parsed as a grouped integer (1,230 or 1.230 as thousands), those zeros remain significant, guaranteeing that dot-decimal and comma-decimal interpretations yield distinct numerical products and preventing accidental value collisions. Reference answers are formatted to preserve uniform precision across competing readings of the same ambiguous input (for example, when a three-decimal-place reading such as 8.887 is contrasted with an integer thousands reading such as 8,887.000), ensuring consistent decimal-place representation across formats before normalisation.
-
No prompt asks for an output notation, yet value-correct requires one. For <789 + 456> the correct would be <1245>, <1 245>, <1,245>, <1.245> or <1'245> depending on a hidden format code. Why was the target notation scored? Could Figure 1 and Tables 4 and 13 also be given on V-Any, and could you add an upper-bound prompt that names the notation?
Different metrics capture different information, value-correctness captures this directly and provides critical signal on the Standard-Ambiguity subset (where separator-free operands like 789 + 456 test whether models adapt their output format to regional narrative cues), whereas V-Any already isolates format-agnostic numerical accuracy. Moreover, we observed that the trends between V-Any and VC were similar but VC gave better distinctions. We will include the V-Any versions of Figure 1, Table 4, and Table 13 in the Appendix.
An upper-bound prompt that names the notation would mean that we would need to give explicit instruction which would make the results fall into a instruction-following measure, that is why we do not provide that level of prompting as stronger instruction-following model may incorrectly appear to be better at number format distinction.
- What source supports each country-format pair? In particular, what supports a comma every four digits for China and Japan (CLDR gives three), the apostrophe-plus-dot format for Colombia, and U+2019 instead of U+0027 for Switzerland? Please list the code point of every separator and say whether Unicode normalization was applied.
We do not use CLDR as this didn't agree with our native annotators. Instead we simply asked our annotators for the conventions uses in their locale. Regarding the unicode, we believe that the models used are robust enough to surmount unicode specific issues.
- Are CH-de, CH-en and CH-fr (and the other same-country pairs) the same prompt under P3 and P5? If so, could you remove the repeats, recompute the totals and give intervals? For the worded version, they are different as they are in different languages. However, for the symbolic version they are the same, exactly like P1, P2, P3, P4 and P5 in fact. But these are counted in repeats.
Due to the character limit, we continue our response here part 3 9. Which locales and formats are pooled in the word-problem rows of Table 5, in the group definitions of Tables 4 and 13, in the "+Lang" column and in the right side of Figure 2? Why are the comma-only, dot-only and space formats left out of the groups?
Pooled locales and formats: Table 5 pools across all locales and the applicable formats within each ambiguity tier, while Tables 4 and 13, the +Lang column, and Figure 2 (right panel) pool all locales belonging to each format comparison group.
Inclusion of C-only, D-only, and space-grouped formats (SD, SC): These formats are not excluded from the groups: in Table 13, C and D are shorthand labels for C-only and D-only, and space-grouped formats (SD, SC) are likewise included in their respective convention groups. We will standardise the format labels (C-only, D-only, SD, SC) across all tables and captions to avoid any ambiguity.
- Which decoding settings, seeds, reasoning modes, maximum lengths and exact model versions and dates were used? At what precision was each local model run, and does the full-precision Mixtral give the same scores as the 4-bit copy?
We used deterministic decoding (temperature = 0) wherever supported by the model or API, with explicit reasoning/thinking modes disabled and all remaining parameters (including top_p, seeds, and maximum generation lengths) kept at each model's or provider's default settings. While Appendices F and G currently outline our experimental setup, we will add these exact decoding settings, maximum token limits, and default parameters explicitly to Appendix G. Model versions, APIs, and numerical precision: Exact open-weights checkpoint identifiers, API endpoints/versions, and dates are listed in Appendices F and G. All local models were evaluated in their default native precision (BF16/FP16). We tried to stay within the model default settings as we believe these are what optimise the models and most user's default settings too.
- How many annotators reviewed each language, and how many edits and rejections did they make? The guideline says decimals belong only on measurable things. How did reviewers treat High-tier items where one reading gives a fraction of a countable object, such as 1.468 doughnuts?
Annotators reviewed the localisations; no questions were rejected, and items were only edited where needed. Tracking the number of edits would primarily reflect Gemini's localisation ability, which lies outside the scope of this research. With regard to measurable quantities, we made every effort to keep them plausible, though occasional edge cases may affect how a sentence reads; however, most items include an explicit unit to ensure they are measurable from the outset.
- How was "plausible scale" checked when units are swapped without converting the numbers (for example centimeters for the UK and inches for the US in the same template)?
Plausibility is not check in this work as the numbers are generated automatically and restricting the number to plausibility is in direct conflict with the comparisons between the many different axes that we look at.
- Two rows look odd. At the bare prompt apertus-70b has a format score of 75.7 and a digit score of 8.0. On word problems gpt-5.6-luna has a digit score of 51.9 and a format score of 18.6. Are these answer-extraction failures, and how were failed extractions counted?
Neither anomaly seem to come from answer-extraction failures. An extraction failure would mean that all metrics are set to zero.
- apertus-70b (bare prompt: FS = 75.7, DS = 8.0): The model consistently produced extractable outputs adhering to the target regional number format (75.7%), but frequently failed the underlying arithmetic or separator disambiguation (8.0% digit accuracy). This is possible by producing a number like 52 which means that it does adhere to the number format all the time, but the answer is incorrect.
- gpt-5.6-luna (word problems: DS = 51.9, FS = 18.6): The model frequently arrived at the right numerical digits (51.9%), but normalised its final output to a default/unformatted representation rather than preserving the prompt's regional formatting (18.6%).
Thank you for your long and detail feedback, we will answer each of your questions which reflect that weakness you highlight below.
In our evaluation depending on the metric, we use all the number formatting conventions to test what the number can mean. Given a number format, we use the decimal marker to remove an non-significant zeroes before comparing the numbers.
The table below has the requested results. We observe no large difference in trend between operations, we've also provided the standard deviation.
Reference readers: Our goal is to characterise behavioural preferences across 11 diverse models (including region-specialised models e.g., Apertus, EuroLLM, Qwen, DeepSeek, Aya) rather than benchmark against naive heuristics. Because our conflict splits are balanced, a fixed single-convention reader achieves a known analytical ceiling (~50% on two-way conflicts, ~33.3% on three-way conflicts), which we will state in Section 5.1 as an anchor for our cross-model comparisons.
Unambiguous counterparts: Rewriting ambiguous operands into unambiguous forms is non-unique (e.g., 1 as 1.0, 1.00, or 01) and alters string length, precision, and tokenisation, confounding the exact surface ambiguity our research questions isolate. Instead, our Unambiguous × Unambiguous tier (Table 5) provides the controlled baseline for unambiguous notation across all 10 formats.
Question-stripped / operator-only ablation (Patel et al., 2021): Figure 1 and Table 5 already perform this comparison by contrasting Symbolic Operations (raw operands and operators only) with Localised Word Problems. Rather than exploiting shallow shortcuts, models drop in accuracy when narrative text is added (e.g., High-Ambiguity P1 VC drops from 27.79% to 13.70%), confirming that models cannot solve the word problems via operator-only heuristics. We will cite Patel et al. (2021) when discussing this comparison in Section 5.2.