- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
The paper identifies "transfer spillover" where cross-lingual reasoning transfer often overwrites culturally grounded answers the base model previously got right. The authors propose X-OPSD as a solution. X-OPSD builds two teachers from one frozen backbone using activation interventions. The first is a reasoning teacher that attenuates a language-related subspace, the second is a culture preserving teacher that intervenes only along directions orthogonal to a culture subspace. A single student is student is trained by on-policy self-distillation, with math inputs routed to the first teacher and the cultural inputs routed to the second. There is no need for steering or routing at inference. The contributions are the spillover framing with a strong paired forgetting metric. The validation covers three backbones on three culture-sensitive benchmarks against Base, OPD, OPSD, COPSD, and MRRE. X-OPSD improves both averages and reduces CulturALL forgetting relative to COPSD. However all results are single runs on small evaluation sets and the baselines are not trained on matching data.
Critical strengths
-
The ablations in table 3 hold the training data fixed and show that cultural protection and task-specific teacher routing each matter. Full X-OPSD has the best joint averages and the lowest forgetting.
-
Measuring "transfer spillover" and pairing it with forgetting gives a better failure than aggregate accuracy. This framing is useful even beyond this method.
Critical weaknesses
-
X-OPSD trains and calibrates on INCLUDE data while OPSD and COPSD see only math. So the culture gains over those baselines can't be attributed to the method rather than to the data itself.
-
The gains are often 1-2 points from single runs with no variance reported. The forgetting rates rest on 65-81 items and the per-language coefficients are hand picked (frequently to be zero). The Gemma culture teacher is inactive for six of eight languages.
-
The teacher preference metric (eq. 7) has a drift problem away from base. For languages where the culture teacher is 0 (table 2) the culture teacher is identical to base. Any fine-tuning that lowers the student's overlap with Base inflates the Delta_route, whether or not the student moves toward the reasoning teacher. Table 4 shows that the largest preference scores are exactly in those languages. A controlled student distilled without reasoning intervention is needed.
Not critical, but observations, I think referencing limitations in the main body would be beneficial to the overall structure of the paper. I also noticed that two citations are showing ("?"). One for mAce and the other for MRRE in the appendix.
Questions for authors
How do OPSD and COPSD perform when trained on INCLUDE as well?
What is the variance across multiple runs and seeds?
How were the coefficients chosen and how sensitive are they to the results?
Why does language fidelity on the 35B model fall below base?
A Gentle Reminder
Dear Reviewer QArP,
Thank you again for your thoughtful comments and constructive suggestions throughout the discussion. As the author–reviewer discussion period is coming to an end, we would like to kindly ask whether our responses have addressed your main concerns. If there are still aspects that remain unclear or require further clarification, we would be very happy to discuss them further.
We sincerely appreciate your time and valuable feedback.