- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper asks whether looped language models (LoopLMs) are less CoT-monitorable than standard models. It uses eight MonitorBench tasks. Each task is run under a standard setting and two stress tests, with three monitor scopes. Monitorability is MonitorBench’s input-intervention score: how often the monitor flags cue use among correct cued responses, relative to the fraction of successes attributable to the cue. Qwen3.8-27B serves as both judge and monitor. This paper explored loop depth within one model family. Under Direct Concealment, Ouro-1.4B’s CoT-only monitorability falls from T=4 to T=8 on Logic (77.9→41.8), Science (40.8→23.6), and Engineering (54.4→43.6). Other tasks are stable or move in the other direction. The authors argue that pass rates, task difficulty, and CoT length do not explain the drops. Qualitative examples suggest deeper models treat the cue as a “check” or omit it. In addition, the study compared looped models with non-looped models.
Critical strengths
- Good writing and structure of the papers, the appendix also show lots of the details.
- Varying loop depth on a fixed checkpoint is the cleanest available way to isolate the effect of recurrent computation on verbalisation. Checking pass rates and token counts as alternative explanations is the right instinct.
- This paper frames the depth effect as task-dependent and the architecture comparison as confounded. The Figure 4 progression and the appendix examples show concretely what reduced monitorability looks like and this makes the metric interpretable.
Critical weaknesses
- Unreported sample sizes in this paper. Scores are computed only on questions with TE > 0 at all three depths, with TE estimated from four rollouts per condition. The number of retained questions per task, model, and condition is never reported.
- Under the action-only scope, the 1.4B Preference and Judgment scores fall sharply (e.g., Preference 46.2→2.6), but this is not discussed in the main content.
Questions for authors
- How many questions are retained per task, model, setting, and depth after the TE > 0 filter?
- Could you report paired bootstrap CIs or permutation tests for the T=4 vs T=8 differences, with a multiple-comparison correction?
- Under the standard (non-stress) setting, does 1.4B also verbalise less with depth?
- In Table 12a, Qwen3.5-35B-A3B scores 34.91 on Law Judgment (all-messages) versus 100 under CoT-only (Table 10a). Could you explain about this point?
2 comments
Dear Reviewer,
Thank you again for your review. As the discussion period ends today, we wanted to check whether our response and the revised manuscript address your concerns, and we would be glad to clarify anything before the deadline. If they do, we would appreciate it if you would consider updating your score.
Authors
1/N
In the rebuttal, we revise our manuscript and report the number of retained questions in Table 8 in the paper. We also report the numbers below.
Table 1: Number of retained questions to compute the monitorability score across different tasks under standard/direct concealment/monitor-aware evasion settings. To ensure a fair comparison, we only keep questions with a positive total effect under all three loop depths T=4,6,and 8.
In the rebuttal, we further conduct the qualitative analysis on two Cue Condition tasks: Preference and Judgment. Each example compares responses to an identical prompt at loop depths 4, 6, and 8. In several examples of Ouro-1.4B-Thinking, we notice that the final answer becomes shorter and omits the explanation linking the answer to the supplied cue, despite the fact that we instruct the model to provide some explanation. Those qualitative examples explain why action monitorability under Preference and Judgement falls sharply, as the models tend to only give the option as the final answer for some instances. More qualitative analysis is in Appendix A2.6.
In this rebuttal, we additionally compute paired bootstrap confidence intervals and conduct two-sided paired hypothesis tests for the monitorability differences between (T=4) and (T=8). As reported in Table 9 of the revised manuscript, under the CoT-only monitoring scope, we observe statistically significant differences in Engineering/Logic/Science for Ouro-1.4B-Thinking, and Logic/Science for Ouro-2.6B-Thinking under Direct Concealment. Specifically, for the Science task for Ouro-1.4B-Thinking, the paired 95% CI is [-34.65, -14.90] under the Direct Concealment and [-25.66, -8.31] under the Monitor-aware Evasion. These results provide additional statistical support for the task-dependent reduction in CoT monitorability observed in our experiments. More results can be found in Tables 9-11 in the revised manuscript.
As reported in Table 13 in the paper, in the original (no-intervention) and standard (non-stress) settings, we find that the Ouro-1.4B-Thinking model does not generate fewer CoT tokens with increased loop depth. In particular, tokens averaged over tasks slightly decrease from 4367 to 4307 when loop depth increases from 4 to 8 in the standard setting.
Thanks for pointing this out. We miscopied the results from Science (all-messages) to the Law Judgment (all-messages) for Qwen3.5-35B-A3B. So for Qwen3.5-35B-A3B in the Law Judgment task, the monitorability score is 97.62, 100.00, and 100.00 for action-only, CoT-only, and all-message scopes. We have already fixed this in our revised manuscript.