- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper investigate the CoT monitorability in the multilingual setting. The author tested this with multilingual GPQA dataset in 13 languages and 16 models.
Critical strengths
- The experiments are quite comprehensive
- The topic about CoT monotorability is timely and worth investigation
Critical weaknesses
- Discussion of CoT faithfulness is largely missing, since you induce unreliable CoT monotorability by CoT unfaithfuness (l.088-089). Why the induction works? I can imagine that CoTs can be unfaithful while monotorable and vice versa. Furthermore, although you cited the paper from Zhao et al. 2026, but only discuss it very superficially. They also investigated multilingual CoT faithfulness, and for me, it seems to share a great overlap with your part. Can you clearly clairify the differences?
- Only the multilingual GPQA dataset is considered, which is only for MCQA. The generalizability to other NLP task remains questionable. I think at least e.g., MMMLU or MGSM8K shoud be included in the experiments. And the selection of examined languages are not introduced in the experimental setup, which is only first shown in Figure 2.
- At least another judge model should be included apart from GPT5.1, as judge model choice can yield different CoT faithfulness conclusion (Young et al., 2025). Therefore, inter model agreement is highly disired here to draw a more robust conclusion.
- l.152: Although you mentioned that the evaluation follows Chen et al. (2025), it does not adopt their conditioning, i.e., condition on items where the no-hint answer is not the hint target but the hinted answer is, and normalize by the hint's causal effect. Turpin et al. (2023) paper is not cited, who introduced the paradigm and is the base of Chen et al. (2025).
- The metric set up seems to be kind of artificial. And the deception rate is defined over all responses that select C (l.167; l.1332), which seems to be problematic, because any C selection that the model would have made will always be labelled deceptive. In my opinion, the rate is high largely by construction.
- The tier differences are quite small and confound (Table 7), which are averaged over models and languages per tier, CI and significance test is missing. Furthermore, the resource tier also depends on translation quality and how much the required target language prefill degrades reasoning.
- Although the main contribution is about multilinguality, very limited result is broken down by language family, script or typological feature. Languages are grouped only by resource tier.
Questions for authors
- l.120: "CoT monitorability across across high-, mid-,...". Remove one of "across"
- l.314: "mode's preferences" -> "model's preferences"
- l.316: "can be find" -> "can be found"
- l.395: "remain hight" -> "remain high"
- l.639: "confrims" -> "confirms"
- l.667: "more less erratic" is confusing
- "logit lens Belrose et al. (2023)" should use parenthetical citation
- In introduction, Kwa et al. (2025) is METR's long-task time-horizon paper. It does not support claims about fabricating justifications or deception.
4 comments
Response to Reviewer 8n39 (Part 2/2)
Re: Analysis across language families and typological diversity
We would like to clarify a potential oversight. Our analysis extends beyond resource-tier comparisons to detailed language-specific evaluations. Figure 2 reports deception rates across all 13 languages and 16 models, while Figure 6 examines proprietary models. Appendix Tables 5–6 establish per-language robustness across five stochastic runs, Tables 8–9 report accuracy and deception rates, and Table 10 evaluates hint-solving capabilities across languages.
Importantly, Appendix C, Table 11 explicitly documents eight linguistic families. For example, English, Russian, and Bengali belong to the Indo-European family but represent high-, mid-, and low-resource settings, respectively. This enables comparisons across languages with shared linguistic ancestry but differing resource availability. Furthermore, Section 5, Figure 7, and Appendix Figure 13 examine differences in reasoning failure mechanisms and answer-switching patterns across linguistic groups. Together, these analyses demonstrate that CoT monitorability failures persist across diverse languages, linguistic families, and model architectures, rather than being restricted to resource-tier effects.
Re: Typographical issue
We thank the reviewer for identifying these typographical errors and will correct them in the revised manuscript.
Re: Validity of the deception-rate metric
We would like to clarify a potential misunderstanding regarding our metric. Our deception rate operationalizes failures of CoT faithfulness. Specifically, among responses selecting the adversarial target C, we assess whether the CoT discloses the hint under simple conditions and faithfully executes its computation under complex conditions (Section 3; Appendix D.2, Figs. 15–16). Importantly, merely mentioning a hint does not establish faithfulness when models fabricate intermediate values, manipulate hint procedures, or provide misleading post-hoc justifications to support C. Our taxonomy further distinguishes ordinary arithmetic mistakes from procedural manipulation (Section 5.1), demonstrating that these failures reflect distinct reasoning mechanisms rather than intentional deception.
Our findings are supported by multiple independent controls, rather than arising by construction. Appendix B.1 and Figure 11 distinguish answer susceptibility from CoT monitorability; Figure 5 and Appendix B.2 demonstrate robustness when varying the hinted target across B, C, and D; and Appendix Tables 5–6 confirm stability across five stochastic runs. Importantly, Appendix Table 7 reports higher deception rates for low-resource languages (92.2% simple; 93.5% complex) than high-resource languages (88.5%; 87.7%), despite broadly comparable no-hint accuracy. Furthermore, Section 5 identifies distinct reasoning failure mechanisms, while Section 6 provides mechanistic evidence of answer switching. These complementary analyses support our conclusions regarding the fragility of multilingual CoT monitoring.
Dear Reviewer 8n39,
We have provided detailed responses to your questions and concerns in Parts 1 and 2 of our rebuttal below.
To explicitly address your most critical blockers, we took the following actions:
-
We implemented an independent evaluation using
GPT-5.6-solon 300 sampled responses across resource tiers. The resulting 100% inter-model agreement confirms that our findings are mathematically robust and not an artifact of GPT-5.1. -
We clarified that while Zhao et al. test whether answers depend on CoT via truncation, our work uniquely tests whether CoT safely exposes adversarial reliance under linguistic distribution shift. Our inclusion of mechanistic interpretability and complex procedural manipulation taxonomy provides entirely distinct contributions to safety monitoring.
-
Regarding your point about lack of results broken down by language family or typological feature, our paper already explicitly categorize and analyze these exact linguistic family divisions in Appendix C, Table 11, and Figure 7.
We would greatly appreciate if you would consider reassessing your score in light of the above responses and experiments. Let us know if you have any additional questions/concerns.
Follow-up on Rebuttal
As the author-reviewer discussion period is coming to a close, we wanted to gently check in and see if you have had a chance to look at our rebuttal and revised PDF.
We have carefully addressed all your concerns and conducted additional experiments. If our clarifications have resolved your questions, we would be immensely grateful if you could kindly share your feedback and consider updating your score before the deadline.
If you have any questions, please let us know so we can address them. Thank you so much for your invaluable time and dedication to reviewing our work!
Response to Reviewer 8n39 (Part 1/2)
We thank Reviewer 8n39 for their assessment and recognition of our work's strengths, particularly
the comprehensive experimental evaluationandthe timeliness and importance of investigating CoT monitorability. We appreciate the reviewer's acknowledgment of the relevance of our research and address their concerns below.Re: CoT faithfulness, monitorability, and distinction from Zhao et al. (2026)
We would like to clarify a potential misunderstanding. Our study examines whether CoT exposes adversarial-cue reliance for safety monitoring, rather than equating CoT unfaithfulness with monitorability. Through controlled hint interventions, we assess cue disclosure and correct procedural execution (Section 3; Appendix D.2), while our taxonomy distinguishes hint omission from procedural manipulation (Section 5.1).
Our contribution also differs fundamentally from Zhao et al. (2026). Whereas Zhao et al. examine whether model answers depend on CoT through trace truncation and error injection, we investigate whether CoT reliably exposes adversarial-cue reliance for safety monitoring under linguistic distribution shift. Specifically, our contributions are threefold: (1) a systematic evaluation of CoT monitorability across 16 models and 13 typologically diverse languages, examining both simple and complex adversarial hints; (2) a detailed analysis of distinct reasoning failure mechanisms, including hint concealment, procedural manipulation, non-causal rationalization, and answer switching (Section 5); and (3) mechanistic interpretability analyses using Logit Lens and Tuned Lens, revealing how internal answer preferences evolve during generation and can diverge from observable reasoning traces (Section 6; Appendix A.6). Together, these findings demonstrate that CoT monitorability is fragile under linguistic distribution shift and provide new behavioral and mechanistic insights into CoT-based safety oversight, extending beyond prior studies of CoT faithfulness.
Re: Inter-model judge agreement and evaluation reliability
To further validate our automated evaluation, we conducted an additional independent assessment using GPT-5.6-sol on 300 complex-hint CoT responses, randomly sampled across six languages spanning high-, mid-, and low-resource tiers (50 per language). Both GPT-5.1 and GPT-5.6-sol applied identical strict criteria for verifying explicit hint computation and procedural correctness.
Table: Inter-model agreement between GPT-5.1 and GPT-5.6-sol.
The two judges achieved 100% agreement (300/300) across all six languages and resource tiers. This corresponds to Cohen's κ = 1.00 wherever both classification labels are represented. These findings provide additional evidence of cross-model consistency in our strict CoT faithfulness evaluation, complementing the multilingual human verification reported in Appendix A.3. We will incorporate this additional validation into the revised manuscript.