- Rating
- 3
- Confidence
- 4
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper explains why LLMs failed to solve the math problems in multilingual settings. It found that LLMs can execute supplied calculations correctly but have difficulty retrieving numerical inputs and subsequent solutions. Inspired by these findings, this paper used a two-stage pipeline, AccessPath-then-TargetLanguage SFT, to improve multilingual math ability.
Critical strengths
- Beyond Acc, this paper analyses the reason why LLMs fail in multilingual reasoning. The distinction among autonomous solving, conditional computation, and continued solving provides a more informative diagnosis than final-answer accuracy alone.
- Compared with Full-CoT training in both stages, this approach improves mean MGSM accuracy by approximately 3 pp in Swahili and Bengali under matched data order, source exposure, and training updates.
Critical weaknesses
- The controlled models are trained from scratch with source-language solutions but only target-language questions. Why only target-language questions? These designs are useful for diagnosis but differ substantially from realistic multilingual training.
- The three synthetic configurations also show highly variable and non-monotonic autonomous accuracy, from 10.55% to 89.55% under full lexical sharing (§3–4; Figure 3).
- The mean gains are only 2.93 and 3.07 points, and the 95% seed-level intervals in Figure 7 cross zero. The post-training diagnostic intervals also include zero. The improvement is not so solid.
Questions for authors
See above.
3 comments
Response to Reviewer FV82 [Part.2]
W3: The mean training gains are only 2.93 and 3.07 percentage points, and the confidence intervals for both these gains and the post-training diagnostic differences include zero.
We extended both the main evaluation and the post-training diagnostics from three to eight paired training seeds, keeping the training recipe and fixed endpoint unchanged. The combined mean gains are 1.60 percentage points in Swahili and 2.70 in Bengali.
Table F2. MGSM results across eight paired seeds, with 250 questions per language and a 512-token limit. Differences and intervals are in percentage points.
| Language | Full-CoT→Full-CoT | AccessPath→Full-CoT | Difference | 95% paired-seed CI | 95% question-bootstrap CI |
|---|---|---|---|---|---|
| Swahili | 60.85% | 62.45% | +1.60 | [−0.46, 3.66] | [−1.45, 4.55] |
| Bengali | 63.15% | 65.85% | +2.70 | [0.63, 4.77] | [−0.15, 5.55] |
Paired-seed CIs assess training variability on fixed questions. Question-bootstrap CIs use 10,000 paired resamples with the trained models fixed.
The expanded comparison retains positive mean gains in both languages. Bengali’s paired-seed interval now excludes zero, providing stronger evidence for its gain across training repetitions on the fixed test set. Swahili’s paired-seed interval and both question-bootstrap intervals include zero.
Table F3. Post-training differences on the same 165 Swahili and 164 Bengali diagnostic instances, across eight paired seeds. Values are AccessPath-route minus Full-CoT-route accuracy in percentage points, with 95% paired-seed intervals.
| Language | Autonomous solving | Continued solving |
|---|---|---|
| Swahili | +3.18 [0.63, 5.73] | +0.30 [−2.52, 3.13] |
| Bengali | +1.37 [−1.96, 4.71] | +0.84 [−1.57, 3.24] |
The diagnostics also show higher mean autonomous accuracy: +3.18 points in Swahili and +1.37 in Bengali, with the Bengali effect still uncertain. Continued-solving differences are small and show no clear route advantage. The expanded results retain modest positive mean differences in autonomous solving, with the strength of the evidence varying by language and evaluation cohort.
Looking Forward to Your Feedback
Dear Reviewer FV82,
Thank you again for recognizing the value of our diagnostic approach and for raising questions about the controlled setting, configuration variability, and training gains. Our responses in Parts 1–2 address these points:
- We explained why the controlled experiments withhold paired target-language solutions and discussed the Chinese-supervised calibration that confirms the tasks are learnable with paired supervision.
- We clarified how the three synthetic configurations differ and examined value retrieval alongside autonomous solving to interpret their contrasting results (Table F1).
- We expanded the training comparison and post-training diagnostics from three to eight paired seeds, reporting both the mean improvements and their uncertainty (Tables F2–F3).
As the discussion period draws to a close, we would appreciate your feedback on whether these responses address your concerns. We would be happy to provide further clarification on any remaining point.
Thank you again for your time and thoughtful review. Have a nice day!:)
Best regards,
Authors
Response to Reviewer FV82 [Part.1]
Thank you for recognizing the value of separating arithmetic execution from autonomous solving. We clarify the purpose of the controlled setting and the configuration variability, and report the expanded training comparison below.
The controlled setting tests whether source-language solution supervision transfers without paired target-language solutions. Providing such pairs would introduce direct task supervision in the target language, making successful solving harder to attribute to transfer. The English–Chinese models receive Chinese questions and independent solution-format text, providing exposure to Chinese and its solution format while withholding question–solution pairing.
The Chinese-supervised calibration confirms that the tasks and formats are learnable with paired supervision: all five calibration models achieve at least 98.24% autonomous accuracy on both Chinese question forms (Table 5). Section 5 then examines the same diagnostic distinction in pretrained Qwen and Phi models, extending the analysis beyond the controlled supervision setting.
Full lexical sharing does not guarantee successful autonomous transfer in these experiments. The three configurations are independently constructed settings that differ in language forms, sharing rankings, task instances, and training randomness. Across these settings, autonomous accuracy varies substantially, while conditional computation and continued solving remain near ceiling.
Table F1. Target-language accuracy at full lexical sharing (%), reproduced from Table 3.
Value retrieval identifies a specific difficulty in Configuration 3: accuracy is 99.61% in the source language but only 17.19% in the target language. Thus, poor recovery of stated inputs coexists with accurate computation from supplied states and successful continuation from a correct prefix.
Within each sharing sweep, the source corpus, sample order, tokenizer, initialization, and optimizer settings are fixed (Appendix B.1). Full lexical sharing still leaves source–target word-order differences. Configuration 3 also exhibits a non-monotonic response: autonomous accuracy falls from 48.83% at 50% sharing to 10.55% at full sharing (Table 2). These runs do not isolate the cause of this reversal. We will clarify that the supported finding is the coexistence of accurate supplied computation and continuation with poor input retrieval and autonomous solving, even under full lexical sharing.