- Rating
- 3
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes CELT to address missing reasoning spans in multilingual reasoning. It uses counterfactual distillation to align target-language reasoning with English teacher distributions while preserving existing capabilities. Experiments across three backbones and benchmarks show consistent improvements over seven baselines, particularly in low-resource languages.
Critical strengths
The paper provides a clear and empirically grounded characterization of missing reasoning spans in multilingual reasoning. Its Frontier Measurement offers a useful fine-grained diagnostic beyond the proposed method. CELT also achieves strong and consistent improvements across models, benchmarks, and languages, particularly in low-resource settings.
Critical weaknesses
1.The paper lacks a comparison with directly fine-tuning on English questions and reasoning traces, making it unclear whether the gains stem from counterfactual distillation or additional English supervision.
2.The paper does not analyze unreachable cases, leaving it unclear whether these failures arise from language understanding, mathematical computation, or reasoning limitations.
3.Frontier Measurement requires substantial additional decoding, but the paper does not adequately report its computational cost or wall-clock time.
4.The reading loss is essentially a question-translation objective, but the paper does not establish whether its benefits come from improved question understanding or general cross-lingual representation alignment.
Questions for authors
-
How does CELT compare with directly fine-tuning on English questions and reasoning traces under the same training budget?
-
What causes the unreachable cases: language understanding, mathematical computation, or reasoning limitations?
-
What are the computational cost and wall-clock time of Frontier Measurement?
-
Does the reading loss improve question understanding or primarily enhance cross-lingual representation alignment?
4 comments
What causes the unreachable cells
- The paper does not analyze unreachable cases, leaving it unclear whether these failures arise from language understanding, mathematical computation, or reasoning limitations.
- What causes the unreachable cases: language understanding, mathematical computation, or reasoning limitations?
Thank you for this question. We annotated every unreachable MGSM test cell of the three backbones, 751 cells in total (Table C.1). Two annotators per language, fluent in it and in English, read the English and target-language questions and the model's own English solution. They also read the frozen model's output and its continuation from the whole English solution. The order of the causes was fixed before annotation, and each cell gets the first applicable cause:
- Language understanding: a quantity, a relation or the asked quantity of the target-language question is misread.
- Computation: the question is read correctly and an arithmetic step is wrong.
- Reasoning: the question is read correctly and the arithmetic is right, but the plan is wrong.
- Question divergence: the benchmark's target-language question states a different number or relation than the English one.
- Answer form: the right value in another unit, rounding or form.
Each label describes the continuation from the whole English solution. In computation cells it recomputes a supplied step wrongly, and in reasoning cells it drops the supplied plan.
The annotators agreed on 89.21% of the cells (Cohen's kappa 0.84). Cells per cause:
| Backbone | Group | cells | understanding | computation | reasoning | question divergence | answer form |
|---|---|---|---|---|---|---|---|
| Qwen2.5-3B-Instruct | HRL | 125 | 49 | 17 | 30 | 16 | 13 |
| Qwen2.5-3B-Instruct | URL | 121 | 64 | 12 | 21 | 14 | 10 |
| Qwen2.5-7B-Instruct | HRL | 102 | 44 | 11 | 22 | 15 | 10 |
| Qwen2.5-7B-Instruct | URL | 114 | 63 | 8 | 17 | 16 | 10 |
| Llama3-8B-Instruct | HRL | 155 | 63 | 22 | 37 | 18 | 15 |
| Llama3-8B-Instruct | URL | 134 | 69 | 15 | 24 | 15 | 11 |
| all | all | 751 | 352 (46.87%) | 85 (11.32%) | 151 (20.11%) | 94 (12.52%) | 69 (9.19%) |
Which method answers which cause. On Qwen2.5-7B-Instruct MGSM, we scored the unreachable cells under the runs of Table 1. We also trained polyGRPO on the merged CELT model, with three seeds. Correct cell-runs (%), both groups pooled; the frozen model answers none of these cells:
| understanding | computation | reasoning | question divergence | answer form | all | |
|---|---|---|---|---|---|---|
| CELT | 15.58 | 8.77 | 10.26 | 2.15 | 10.00 | 11.57 |
| polyGRPO | 12.46 | 45.61 | 38.46 | 6.45 | 71.67 | 24.69 |
| CELT, then polyGRPO | 17.13 | 40.35 | 40.17 | 7.53 | 65.00 | 26.39 |
Findings.
First, language understanding is the largest cause on every backbone and in both groups, 39.20% to 55.26% of the cells and 46.87% overall. Computation accounts for 11.32% and reasoning for 20.11%.
Second, the share of understanding does not come from the order of the causes. In a free ranking, with the causes shown in random order, the main cause matches the ordered label in 732 of 751 cells (97.47%). Understanding stays the largest cause everywhere, at 44.34% overall.
Third, two automatic checks, which look at the same misreading in another output, are consistent with the label. The frozen model restated each target-language question in English. The restatement repeats the misreading in 77.27% to 81.25% of the understanding cells, against 6.58% to 8.77% of the other cells. In 85.71% to 87.50% of the understanding cells, the continuation from the whole English solution restates the misread quantity or relation and computes from it.
Fourth, this explains why no prefix helps the understanding cells. After the supplied steps, the model reads the question again and returns to its misreading, so even the whole English solution does not bring it to the answer. In a further 12.52% of the cells, the benchmark's own target-language question differs from the English one.
Fifth, each method does better on the causes it trains on. CELT answers more understanding cells than polyGRPO, 15.58% against 12.46%, the cause Question Reading targets. The answer reward of polyGRPO answers more computation, reasoning and answer-form cells, where the question is read correctly. CELT has no distillation row for unreachable cells, of which 19 are computation and 39 reasoning cells on Qwen2.5-7B-Instruct. The two compose: polyGRPO after CELT answers the most unreachable cell-runs and raises MGSM AVG from 81.75 to 83.83.
Revision. We will add this analysis to Appendix C, with the free ranking, the two checks, examples of each cause and the table by method. It will also list the MGSM items of the divergence and answer-form cells.
Computational cost and wall-clock time of Frontier Measurement
- Frontier Measurement requires substantial additional decoding, but the paper does not adequately report its computational cost or wall-clock time.
- What are the computational cost and wall-clock time of Frontier Measurement?
Thank you for asking for concrete numbers. We measured Frontier Measurement and training on one node with 8 A100 80GB GPUs. Decoding uses vLLM (greedy, prefix caching) and training the LoRA configuration of Appendix A.4, both data parallel over the 8 GPUs. GPU-hours are busy time summed over the GPUs, and wall-clock on the node includes about 5% for scheduling and loading. Frontier Measurement has two stages. The own traces decode every cell once (82203 decodings). The prefix probes decode only failing cells with a reference trace (Table A.1).
| Backbone | own traces: tokens (M) | GPU-h | wall-clock (min) | prefix probes: decodings | tokens (M) | GPU-h | wall-clock (min) | training: GPU-h | wall-clock (min) |
|---|---|---|---|---|---|---|---|---|---|
| Qwen2.5-3B-Instruct | 23.35 | 1.25 | 9.8 | 79300 | 15.30 | 0.82 | 6.4 | 4.62 | 36.4 |
| Qwen2.5-7B-Instruct | 21.54 | 1.93 | 15.2 | 61738 | 10.99 | 0.98 | 7.8 | 8.14 | 64.1 |
| Llama3-8B-Instruct | 24.74 | 2.45 | 19.3 | 88617 | 18.17 | 1.80 | 14.2 | 9.38 | 73.9 |
Tokens are generated tokens.
What the numbers say.
First, Frontier Measurement is short. The probes take 6.4 to 14.2 minutes of wall-clock on one node, 8.9% to 13.2% of CELT's total GPU-hours (own traces, probes and training). The own traces take 9.8 to 19.3 minutes, and preservation uses them as well.
Second, CELT as a whole is cheaper than every trained baseline. On Qwen2.5-7B-Instruct the full pipeline takes 11.05 GPU-hours on the same node. The baselines take 19.64 (xRFT), 24.31 (LIDR), 21.73 (MAPO), 52.86 (GRPO) and 61.47 (polyGRPO). These include their own sampling, translation or rollouts.
Third, the measurement runs once before training and is shared by all seeds and design variants. Its size grows only with the failing cells, so the backbone that fails fewer cells needs fewer probes: 61738 on Qwen2.5-7B-Instruct against 79300 on Qwen2.5-3B-Instruct.
Revision. We will add this table to Appendix A.2 and the total cost of every method to Appendix A.7. We hope this gives the cost picture you asked for.
Question understanding and representation alignment under the reading loss
- The reading loss is essentially a question-translation objective, but the paper does not establish whether its benefits come from improved question understanding or general cross-lingual representation alignment.
- Does the reading loss improve question understanding or primarily enhance cross-lingual representation alignment?
Thank you, this is a good question, and the two explanations predict different things. We separated them with a control and with direct measures of each.
Control. CELT with replaced by an alignment loss . For every reading row, is one minus the cosine similarity of two mean-pooled hidden states, averaged over layers 10 to 18 of 28. The two states come from the target-language question and from the English question (stop-gradient). optimizes alignment directly and never produces a restatement. It has the unit row weight of , and the layers were fixed in advance. We also ran weights 0.5 and 2, and all layers instead of 10 to 18.
Measures (Qwen2.5-7B-Instruct, MGSM test set, three seeds):
- Restatement fidelity: the share of target-language test questions whose English restatement keeps every quantity, every relation and the asked quantity. Qwen2.5-72B-Instruct judges it against the English question, and agreed with two annotators on 191 of 200 pairs.
- Misread rate: the share of outputs on the 481 cells with a frontier that misread a quantity or relation, same judge.
- Alignment: cosine similarity of mean-pooled hidden states at layer 14, and top-1 retrieval of the parallel question among the 250 English questions.
| restatement fidelity, HRL | restatement fidelity, URL | misread rate | cosine, URL | retrieval, URL | MGSM URL | recovered | |
|---|---|---|---|---|---|---|---|
| Base | 88.40 | 61.70 | 71.10 | 0.612 | 57.30 | 48.80 | 0.00 |
| CELT w/o (Table 2) | 88.91 | 63.20 | 33.68 | 0.641 | 60.80 | 65.07 | 50.94 |
| CELT with | 89.11 | 64.00 | 31.95 | 0.887 | 95.60 | 65.83 | 52.39 |
| CELT | 93.60 | 82.30 | 18.30 | 0.689 | 68.90 | 71.60 | 64.45 |
Findings.
First, the gain of goes with better question understanding. Relative to CELT without it, raises URL restatement fidelity by 19.10 points. Restatement is close to what trains. The stronger evidence is the misread rate on the task outputs, which does not train directly: it falls by 15.38 points. The URL cosine rises by only 0.048.
Second, alignment alone does not deliver the gain. raises the URL cosine by 0.246, about five times as much, and URL retrieval to 95.60. Yet restatement fidelity rises by only 0.80 and URL accuracy by 0.76, against 6.53 for . Across the six weight and layer settings, raises them by at most 1.10 and 0.90, while the URL cosine reaches 0.851 to 0.926.
So the reading loss improves how the target-language question is understood, and the alignment it adds is small.
Revision. We will add the control, its sweep and these measures to Appendix E. We hope this separates the two explanations clearly.
Fine-tuning on English questions and traces under the same training budget
Thank you, this is the natural alternative to rule out. We ran two controls on all three backbones of Table 1. Each uses the same optimizer updates and training tokens as CELT on that backbone, the same adapter and the same three seeds.
AVG (%), mean of three seeds; 3B and 7B are the Qwen2.5 backbones, L8B is Llama3-8B:
On Qwen2.5-7B-Instruct MGSM, with kept and recovered as in Table 2:
What we see.
First, more English supervision raises English but barely reaches the target languages. English SFT reaches 94.80 on English MGSM and 78.47 on English MATH500 (Base 70.80, CELT 78.80), and recovers 11.02% of the frontier cells.
Second, the same English traces as hard targets under the target-language question recover 40.96%. CELT recovers 23.49 points more. It differs in its target and in its rows. The hard-token row of Appendix E keeps CELT's rows and recovers 56.27, so the rows account for 15.31 points and the soft target for 8.18. Both hard-target controls also lose some cells that the frozen model answers, while CELT keeps 99.04%. So cross-lingual SFT recovers more cells than the w/o Ldist row of Table 2 (40.96 against 28.90) but keeps fewer (95.89 against 98.77). Both reach the same AVG (74.27 and 74.32).
Third, in the nine columns CELT is 1.66 to 17.60 points above English SFT and 1.01 to 11.82 above it with reading. It is 1.75 to 10.36 above cross-lingual SFT. The remaining cell, with preservation and reading kept and only distillation removed, is the w/o Ldist row of Table 2: 74.32 AVG and 28.90 recovered. In CELT, English enters only through preservation, and no distillation row has an English question on the student side (Section 4.2). Under the same budget, more English supervision therefore does not explain the gain.
Revision. We will add both controls on all three backbones to Table 2 and Appendix B. We hope this answers the question, and we are happy to add further controls.