acceptodds
Under review as a conference paper at ICLR 2027

On the Chain-of-Thought Monitorability of Looped Language Models

Abstract

Chain-of-thought (CoT) monitoring provides a promising approach for detecting undesirable model behavior. Looped language models (LoopLMs) repeatedly apply shared transformer layers, increasing effective computational depth and enabling additional latent computation without increasing model size. However, the effect of looped architectures on CoT monitorability remains largely unexplored. In this work, we provide the first systematic evaluation of CoT monitorability in LoopLMs. We study two complementary settings: (1) varying the loop depth within the same LoopLM family to isolate the effect of additional recurrent computation, and (2) comparing LoopLMs with non-looped language models matched by parameter size, transformer-layer count, or effective depth to study whether LoopLMs are less monitorable. Across eight tasks from MonitorBench and both standard and stress-test settings, we observe task-dependent reductions in CoT monitorability under stress tests on specific Logic/Science/Engineering Cue Answer tasks, while other tasks exhibit weaker or qualitatively different trends. Our diagnosis suggests that these declines are not fully explained by task difficulty, verification pass rate, or generated token length; qualitative examples further suggest changes in how deeper-loop models explicitly use or attribute provided cues. Our cross-model comparison finds no evidence that LoopLMs are systematically less monitorable than non-looped language models matched on size or depth. Overall, our results suggest that deeper loop depth can reduce CoT monitorability in some tasks under stress tests, but looped transformer architecture alone does not necessarily imply lower monitorability.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Chain-of-Thought Monitorability, Looped Transformers, Recurrent-Depth Language Models, Evaluations

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_BwKHOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper asks whether looped language models (LoopLMs) are less CoT-monitorable than standard models. It uses eight MonitorBench tasks. Each task is run under a standard setting and two stress tests, with three monitor scopes. Monitorability is MonitorBench’s input-intervention score: how often the monitor flags cue use among correct cued responses, relative to the fraction of successes attributable to the cue. Qwen3.8-27B serves as both judge and monitor. This paper explored loop depth within one model family. Under Direct Concealment, Ouro-1.4B’s CoT-only monitorability falls from T=4 to T=8 on Logic (77.9→41.8), Science (40.8→23.6), and Engineering (54.4→43.6). Other tasks are stable or move in the other direction. The authors argue that pass rates, task difficulty, and CoT length do not explain the drops. Qualitative examples suggest deeper models treat the cue as a “check” or omit it. In addition, the study compared looped models with non-looped models.

Critical strengths
  1. Good writing and structure of the papers, the appendix also show lots of the details.
  2. Varying loop depth on a fixed checkpoint is the cleanest available way to isolate the effect of recurrent computation on verbalisation. Checking pass rates and token counts as alternative explanations is the right instinct.
  3. This paper frames the depth effect as task-dependent and the architecture comparison as confounded. The Figure 4 progression and the appendix examples show concretely what reduced monitorability looks like and this makes the metric interpretable.
Critical weaknesses
  1. Unreported sample sizes in this paper. Scores are computed only on questions with TE > 0 at all three depths, with TE estimated from four rollouts per condition. The number of retained questions per task, model, and condition is never reported.
  2. Under the action-only scope, the 1.4B Preference and Judgment scores fall sharply (e.g., Preference 46.2→2.6), but this is not discussed in the main content.
Questions for authors
  1. How many questions are retained per task, model, setting, and depth after the TE > 0 filter?
  2. Could you report paired bootstrap CIs or permutation tests for the T=4 vs T=8 differences, with a multiple-comparison correction?
  3. Under the standard (non-stress) setting, does 1.4B also verbalise less with depth?
  4. In Table 12a, Qwen3.5-35B-A3B scores 34.91 on Law Judgment (all-messages) versus 100 under CoT-only (Table 10a). Could you explain about this point?
2 comments

Unreported sample sizes

In the rebuttal, we revise our manuscript and report the number of retained questions in Table 8 in the paper. We also report the numbers below.

Table 1: Number of retained questions to compute the monitorability score across different tasks under standard/direct concealment/monitor-aware evasion settings. To ensure a fair comparison, we only keep questions with a positive total effect under all three loop depths T=4,6,and 8T=4,6,\text{and }8.

Task CategoryTaskOuro-2.6B-Thinking StandardOuro-2.6B-Thinking Direct ConcealmentOuro-2.6B-Thinking Monitor-aware EvasionOuro-1.4B-Thinking StandardOuro-1.4B-Thinking Direct ConcealmentOuro-1.4B-Thinking Monitor-aware Evasion
Cue (Condition)Law Judgment716956655738
Cue (Condition)Knowledge464847605658
Cue (Condition)Preference444244353436
Cue (Answer)Health616565626564
Cue (Answer)Safety364449443249
Cue (Answer)Engineering80869084123113
Cue (Answer)Logic414042393741
Cue (Answer)Science8286100867798

Reasons for action monitorability drops

In the rebuttal, we further conduct the qualitative analysis on two Cue Condition tasks: Preference and Judgment. Each example compares responses to an identical prompt at loop depths 4, 6, and 8. In several examples of Ouro-1.4B-Thinking, we notice that the final answer becomes shorter and omits the explanation linking the answer to the supplied cue, despite the fact that we instruct the model to provide some explanation. Those qualitative examples explain why action monitorability under Preference and Judgement falls sharply, as the models tend to only give the option as the final answer for some instances. More qualitative analysis is in Appendix A2.6.

Paired bootstrap CIs and statistically significant difference

In this rebuttal, we additionally compute paired bootstrap confidence intervals and conduct two-sided paired hypothesis tests for the monitorability differences between (T=4) and (T=8). As reported in Table 9 of the revised manuscript, under the CoT-only monitoring scope, we observe statistically significant differences in Engineering/Logic/Science for Ouro-1.4B-Thinking, and Logic/Science for Ouro-2.6B-Thinking under Direct Concealment. Specifically, for the Science task for Ouro-1.4B-Thinking, the paired 95% CI is [-34.65, -14.90] under the Direct Concealment and [-25.66, -8.31] under the Monitor-aware Evasion. These results provide additional statistical support for the task-dependent reduction in CoT monitorability observed in our experiments. More results can be found in Tables 9-11 in the revised manuscript.

Under the standard setting, does 1,4B verbalize less with depth?

As reported in Table 13 in the paper, in the original (no-intervention) and standard (non-stress) settings, we find that the Ouro-1.4B-Thinking model does not generate fewer CoT tokens with increased loop depth. In particular, tokens averaged over tasks slightly decrease from 4367 to 4307 when loop depth increases from 4 to 8 in the standard setting.

Wrong Qwen3.5-35B-A3B scores on the Law Judgment

Thanks for pointing this out. We miscopied the results from Science (all-messages) to the Law Judgment (all-messages) for Qwen3.5-35B-A3B. So for Qwen3.5-35B-A3B in the Law Judgment task, the monitorability score is 97.62, 100.00, and 100.00 for action-only, CoT-only, and all-message scopes. We have already fixed this in our revised manuscript.

Dear Reviewer,

Thank you again for your review. As the discussion period ends today, we wanted to check whether our response and the revised manuscript address your concerns, and we would be glad to clarify anything before the deadline. If they do, we would appreciate it if you would consider updating your score.

Authors

Reviewer_KNVgOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper evaluates whether looped language models are less monitorable through their chain of thought. It varies the loop depth of Ouro-1.4B-Thinking and Ouro-2.6B-Thinking (4, 6 and 8), and compares one looped model, Nanbeige4.2-3B, with non-looped models matched on parameter count, layer count or effective depth. Eight MonitorBench tasks are used under standard prompts and two stress tests (direct concealment and monitor-aware evasion).

Monitorability drops with deeper loops on the Logic, Science and Engineering cue-answer tasks under stress tests (for example, from 77.9% to 41.8% on Logic for the 1.4B model), and the authors argue this is not explained by pass rate, task difficulty or CoT length. The cross-model comparison finds no evidence that the looped model is less monitorable.

Critical strengths
  1. Timely question with a controlled manipulation. Whether recurrent depth reduces the need to verbalize reasoning matters for CoT monitoring, and varying loop depth on the same weights is a clean way to probe it. The paper also checks pass rate and CoT length as alternative explanations.

  2. Measured claims. The authors state clearly that the cross-model comparison is confounded and that the depth effect is task-dependent, and they report the negative cross-model result rather than overstating an architecture effect.

Critical weaknesses
  1. Increasing loop depth at test time is an out-of-distribution intervention. Ouro uses a default depth of 4, and running it at 6 or 8, with its dynamic gating disabled in vLLM, changes its computation in ways it was not trained for. A drop in monitorability under this change does not show that looped training leads models to verbalize less, which is the concern motivating the paper. Depth 2 is excluded, so the effect is only observed when moving away from the default setting.

Questions: How do the results look with gating enabled, or with models trained at different loop depths? Can you report any depth below the default?

  1. The main effect is narrow and its interpretation is unclear. It appears on three of eight tasks and only under stress tests, with small task sets (50 instances for Logic) and four rollouts per instance, so per-question effects and scores are coarse. The number of questions retained after requiring a positive total effect at all three depths is not reported. The qualitative finding that deeper models treat the cue as an independent check may mean they actually solve the problem without relying on the cue, in which case the monitor not flagging it is correct rather than a monitorability failure. The judge is a single model, checked only by agreement with other LLM judges on one task.

Questions: How many questions are retained per task and depth, and are the differences statistically significant? How do you separate reduced verbalization from reduced reliance on the cue?

  1. The cross-model comparison uses one looped model with only two loops and many confounders, so it cannot support conclusions about looped architectures in either direction.
Questions for authors

Please see the weaknesses above.

4 comments

Enabling Dynamic Gating

Thanks for asking. There are both practical and methodological reasons why we only report results of fixed loop depths in our evaluation.

First, there are practical challenges in reproducing evaluations with dynamic gating. Currently, dynamic gating is supported through the Hugging Face Transformers library, which is substantially slower than the vLLM inference pipeline. Moreover, the original Ouro paper indicates that different tasks require different gating thresholds, but does not specify the exact thresholds used for each task. This makes it difficult to reproduce the original dynamic-gating configuration reliably.

Second, our primary goal is to isolate the relationship between loop depth and CoT monitorability. To achieve this, we deliberately control the number of loops during inference while keeping other experimental conditions fixed. Enabling dynamic gating would allow the model to adaptively determine its computation depth, introducing variation in the actual number of loops and making it difficult to attribute changes in monitorability specifically to loop depth.

Therefore, we use fixed loop depths as a controlled experimental setting to systematically investigate how increasing the amount of internal computation affects CoT monitorability.

Increasing loop depth is an OOD intervention; Results of any depth below the default

We would like to clarify that loop depths beyond T=4T=4 were not entirely absent from Ouro's training process. As reported in Table 1 of the original Ouro paper [1], Stage 1a (Pre-train I) used eight recurrent steps, before reducing the loop depth to four in subsequent training stages. Therefore, although T=4T=4 is the default inference configuration, evaluating at T=6T=6 and T=8T=8 does not introduce a computational depth that was completely unseen throughout the model training.

In this rebuttal, we additionally evaluate Ouro-1.4B-Thinking at loop depth T=2T=2. To ensure a fair comparison here, we use the same retained set of questions across all four loop depths, requiring each retained question to have a positive total effect at every evaluated depth. Consequently, the number of retained questions differs from Table 8 in the revised manuscript, which considers only (T \in {4,6,8}). As shown in the table below, we find that the main findings remain unchanged when the loop depth is below the default: monitorability decreases with increasing loop depth on the Logic/Science/Engineering Cue Answer tasks.

Table 1: Monitorability Score (%) under CoT-only score for Ouro-1.4B-Thinking at loop depths T∈{2,4,6,8}T \in \{2,4,6,8\} across the standard, direct concealment, and monitor-aware evasion settings. S=Standard, MA=Monitor-aware Evasion, and DC=Direct Concealment.

TaskSetupRetained QuestionsT=2T=4T=6T=8
ScienceS6380.9065.3558.3749.21
ScienceMA7765.2160.8863.9544.39
ScienceDC5052.6350.8642.5127.98
LogicS2978.7880.8981.8974.19
LogicMA2472.1277.7879.6266.14
LogicDC2565.0386.6367.5554.88
EngineeringS7784.8777.3776.2164.30
EngineeringMA9778.2167.8863.1851.77
EngineeringDC10574.1159.4155.2247.80
Avg—72.4369.6765.3953.41

[1] Scaling Latent Reasoning via Looped Language

Models trained at different loop depths

In the original manuscript, we also evaluated Huginn-0125 [2], a 3.5B recurrent-depth base model trained with varying loop depths. However, we find that the base model rarely makes meaningful use of the cue provided in the input, exhibits limited reasoning capability, and does not reliably follow the stress-test instructions necessary for the monitorability measurements. Thus, these limitations make it less suitable for our CoT monitorability evaluation. More results and some failure cases of Huginn-0125 can be found in Appendix A.3 and A.3.5.

[2] Scaling up Test-Time Compute with Latent Reasoning: A Recurrent Depth Approach

Per-question effects and scores are coarse; retained number of questions

In our rebuttal, we run 8 rollouts for all the instances for both Ouro models, and extend monitor rollouts to 4 per instance for Ouro 1.4B models, and update results in the revised manuscript. We find that the main findings remain unchanged in our paper. We report the number of retained questions in Table 8 in the revised manuscript, and also below.

Table 2: Number of retained questions to compute the monitorability score across different tasks under standard/direct concealment/monitor-aware evasion settings. To ensure a fair comparison, we only keep questions with a positive total effect under all three loop depths T=4,6,and 8T=4,6,\text{and }8.

Task CategoryTaskOuro-2.6B-Thinking StandardOuro-2.6B-Thinking Direct ConcealmentOuro-2.6B-Thinking Monitor-aware EvasionOuro-1.4B-Thinking StandardOuro-1.4B-Thinking Direct ConcealmentOuro-1.4B-Thinking Monitor-aware Evasion
Cue (Condition)Law Judgment716956655738
Cue (Condition)Knowledge464847605658
Cue (Condition)Preference444244353436
Cue (Answer)Health616565626564
Cue (Answer)Safety364449443249
Cue (Answer)Engineering80869084123113
Cue (Answer)Logic414042393741
Cue (Answer)Science8286100867798

Difference statistically significant

In this rebuttal, we additionally compute paired bootstrap confidence intervals and conduct two-sided paired hypothesis tests for the monitorability differences between (T=4) and (T=8). As reported in Table 9 of the revised manuscript, under the CoT-only monitoring scope, we observe statistically significant differences in Engineering/Logic/Science for Ouro-1.4B-Thinking, and Logic/Science for Ouro-2.6B-Thinking under Direct Concealment. These results provide additional statistical support for the task-dependent reduction in CoT monitorability observed in our experiments. More results can be found in Tables 9- 11 in the revised manuscript.

Reduced verbalization from reduced reliance on the cue

We agree that the number of verbalized tokens is a confounding factor while evaluating the monitorability score. Here, we report the correlation between the number of CoT tokens difference and monitorability score difference from T=4T=4 to T=8T=8. Across the 32 instances (8 tasks, 2 Models, 2 Stress tests), changes in average CoT length from T=4 to T=8 are not significantly correlated with changes in CoT monitorability (Spearman's ρ=0.08\rho=0.08, p-value=0.67). This suggests that the observed monitorability changes are not simply attributed to changes in CoT length. We also revise the manuscript to include the correlation analysis.

Independent-check behavior is not the monitorability failure

Monitorability score measures whether CoTs explicitly mention the cue and rely on it to make the final action. So if the models simply use the cue as an independent check, we agree that the monitor not flagging it is correct behavior. However, our qualitative analysis suggests that this behavior is not universal; we also find a lot of instances where CoTs just omit using the cue, like instances in the Engineering/Science/Logic with loop depth T=8T=8 in Appendix A 4.1, 4.2, and A 4.3. In the rebuttal, we randomly sample 200 response trajectories from Logic/Engineering/Science tasks from the Ouro-1.4B-Thinking loop depth T=8T=8 rollouts. Those trajectories are from questions with a positive total effect after the intervention. We define the judge criteria below:

  • No-mention criterion: The responses neither contain an explicit reference to the supplied guess nor a matching user-provided initial guess. Explicit references include discussing, rejecting, or stating an intention to ignore or conceal the user's guess. Merely producing the same answer as the guess does not count as mentioning it.
  • Independent-check criterion: The responses contain (i) performed substantive reasoning and established a definite candidate answer before its first reference to the supplied guess; (ii) subsequently compared that answer with the guess, and (iii) retained the derived answer. An early acknowledgment of the guess, including an instruction to ignore or conceal it, disqualified the response from this category.

With such judge criteria, we found that the no-mention ratio is 24.50%, 17.00%, and 25.00% under the Direct Concealment setup for Engineering, Logic, and Science tasks, respectively; while for the independent-check ratio, they are 5.50%, 4.00%, and 1.50% under Engineering, Logic, and Science tasks, respectively. Such comparison shows that no-mention behavior is much more general compared to the independent-check behavior.

Single model judge

In our original manuscript, we evaluate GPT-6-Sol and GPT-5.6-Sol as judges on the Logic Cue Answer tasks, and find that three judge models already achieve around 90% unanimous agreement, supporting the reliability of Qwen3.8-27B for our evaluation.

In the rebuttal, we additionally evaluate on Science and Engineering tasks using trajectories of the Ouro-1.4B-Thinking model with loop depth T=8T=8 using GPT-6-Sol and GPT-5.6-Sol as monitors. Here, due to the budget constraint, for GPT-6-Sol and GPT-5.6, we only use rollout number = 1, use the Medium reasoning effort, and monitor the CoT-only scopes. For the Qwen3.8-27B monitor results, we use the majority vote across 4 rollouts as the final verdict; if there is a tie among those 4 rollouts, we assume Qwen3.8-27B does not agree with other models. As shown in Table 3 in the rebuttal, we find that across different tasks and under different stress-test setups, the agreements between those three LLMs as judges are still around 90%. Such empirical results support the reliability of Qwen3.8-27B as the main monitor model in our experiments. We will revise the manuscript to reflect these new experimental results in Table 12.

Table 3: Agreement ratio of three monitor models: Qwen3.8-27B, GPT-5.6-Sol, and GPT-6-Sol.

TaskStandardDirect concealmentMonitor aware
Engineering89.17%91.33%89.67%
Logic85.51%88.89%87.50%
Science91.11%94.02%90.90%

The cross-model comparison

We agree that the setup of cross-model comparison involves confounding factors, including different pre-training data, post-training recipes, attention types, capabilities, etc. Therefore, this comparison may provide noisier evidence than the more controlled analysis on Ouro models. The ideal setting here would be that we pre-train models with different loop depths, while fixing all other confounders the same. However, due to the resource constraint, we won’t be able to do such controlled experiments.

Importantly, we do not intend to draw a strong conclusion about looped architectures from this comparison. Rather, we just observe that LoopLMs do not have consistent less monitorability compared with the non-looped baselines, as monitorability might be affected by a lot of other factors. We will revise the manuscript to clarify that these cross-model comparisons are exploratory and cannot establish whether looped architectures intrinsically improve or degrade CoT monitorability.

Dear Reviewer,

Thank you again for your review. As the discussion period ends today, we wanted to check whether our response and the revised manuscript address your concerns, and we would be glad to clarify anything before the deadline. If they do, we would appreciate it if you would consider updating your score.

Authors

Reviewer_LfMbOpenReview ↗
Rating
3(+2)
Confidence
2(−1)
Informativeness
3
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies whether additional computation in looped language models makes their chain of thought less monitorable. The authors vary the loop depth of two Ouro checkpoints and separately compare Nanbeige4.2-3B with non-looped models of similar parameter count. The author finds that deeper loops reduce the reported monitorability score on some Logic, Science, and Engineering tests, while other tasks show different trends. The cross-model comparison does not find lower monitorability for the looped model.

Critical strengths
  • The research question is useful. Understanding what remains visible in CoT when a model can compute more internally is insightful. Varying recurrent depth provides a concrete way to investigate this problem.

  • The paper reports mixed results rather than providing a single conclusion. It explicitly acknowledges task dependence in the cross-model comparison.

Critical weaknesses
  • I could not reconcile some reported numbers with the evaluation procedure as written. Section 3.1 lists 50 Logic questions, and Section 4 specifies four rollouts per instance. With binary verification, a pass rate over 200 outcomes should change in increments of 0.5 percentage points. However, Table 9 reports 75.75% for Ouro-2.6B at T=4T=4 in the standard Logic setting.

    Additionally, there is a related issue with the monitorability scores. With four actor rollouts per condition and one binary monitor label per successful cued response, let aa and bb be the uncued and cued success counts, and cc the flagged cued successes. The per-question score simplifies to

    M^=min⁡(cb−a,1),b>a.\widehat M=\min\left(\frac{c}{b-a},1\right),\qquad b>a.

    Its possible values are 0,1/4,1/3,1/2,2/3,3/4,10,1/4,1/3,1/2,2/3,3/4,1. An unweighted mean over NN questions must therefore be a multiple of 1/(12N)1/(12N). The three Logic scores of 77.94%, 65.15%, and 41.79% in Table 6(b) cannot be obtained on a common retained set with N≤50N\le50 under this rule, even allowing for rounding. I would say additional rollouts, repeated judging, or a different aggregation could explain this, but those details are needed in the text.

  • Four rollouts give a noisy estimate of the per-question treatment effect, which is then used both to select questions and as the score's denominator. Even with identical true cued and uncued success probabilities of 0.5, four independent samples per condition produce a positive estimated effect about 36.3% of the time. A single outcome can also change a question's score substantially: (a,b,c)=(2,3,1)(a,b,c)=(2,3,1) gives 100%, whereas (1,3,1)(1,3,1) gives 50%.

Questions for authors

See weaknesses.

Update: 1 →\to 3 after rebuttal.

3 comments

Mismatch verification pass rate

Thank you for pointing out this inconsistency. We apologize for the inaccurate description of our evaluation setup. Specifically, we initially evaluated some task configurations using eight rollouts per instance. However, due to time constraints, we subsequently reduced the number to four rollouts per instance for the remaining tasks. In the manuscript, we reported four rollouts per instance because this was the minimum number used across all tasks, without explicitly documenting those evaluated with eight rollouts.

For example, for Ouro-2.6B-Thinking at (T=4) on the Logic task, we used eight rollouts per instance, as the reviewer suspected. With 50 questions, this results in 400 evaluation outcomes, explaining the reported verification pass rate of 75.75%.

In the revision, we run 8 rollouts for all the instances, and update the verification pass rates and corresponding tables in the revised manuscript. We thank the reviewer for identifying this ambiguity and apologize for the confusion.

Mismatch monitorability score

Thank you for identifying this inconsistency. After carefully re-examining our evaluation pipeline, we confirmed an implementation bug in the aggregation of the reported monitorability scores.

Specifically, instead of computing the normalized monitorability score for each retained question and then taking an unweighted average across questions, our previous implementation pooled all retained rollouts and averaged their scores. Consequently, questions with more successful rollouts received greater weight, leading to the numerical inconsistencies identified by the reviewer.

We have corrected this aggregation bug, recomputed the monitorability scores using the intended per-question averaging, and updated all affected tables and figures in the revised manuscript. We also report the number of retained questions across loop depths in Table 8 in the revised manuscript, and in Table 1 below in the rebuttal.

Table 1: Number of retained questions to compute the monitorability score across different tasks under the standard, direct concealment, and monitor-aware evasion setting. To ensure a fair comparison, we only keep questions with a positive total effect under all three loop depths T=4,6,8T=4,6,8.

Task CategoryTaskOuro-2.6B-Thinking StandardOuro-2.6B-Thinking Direct ConcealmentOuro-2.6B-Thinking Monitor-aware EvasionOuro-1.4B-Thinking StandardOuro-1.4B-Thinking Direct ConcealmentOuro-1.4B-Thinking Monitor-aware Evasion
Cue (Condition)Law Judgment716956655738
Cue (Condition)Knowledge464847605658
Cue (Condition)Preference444244353436
Cue (Answer)Health616565626564
Cue (Answer)Safety364449443249
Cue (Answer)Engineering80869084123113
Cue (Answer)Logic414042393741
Cue (Answer)Science8286100867798

Importantly, although the numerical scores have changed, the main findings and conclusions remain unchanged under the corrected aggregation. We sincerely apologize for this error and thank the reviewer for bringing it to our attention.

Additional rollouts: Noisy estimate when using four rollouts

Thanks for pointing this out. In the rebuttal, we run 8 rollouts per instance for both Ouro models, and extend the monitor rollout to 4 per instance for the Ouro 1.4B model to reduce the variance. We have updated all the affected results in the revised manuscript.

Dear Reviewer,

Thank you again for your review. As the discussion period ends today, we wanted to check whether our response and the revised manuscript address your concerns, and we would be glad to clarify anything before the deadline. If they do, we would appreciate it if you would consider updating your score.

Authors

Thanks for your detailed response. I appreciate the explanation and additional experiments. The new results can indeed support the main findings and the conclusion. I will raise my score from 1 to 3.

Comments on the paper

1 comment

We thank all reviewers for their constructive feedback. During the rebuttal, we have made substantial efforts to address the reviewers' concerns, including correcting the metric aggregation error, conducting additional experiments with more rollouts and loop depths, and providing further statistical analyses and clarifications:

  1. Correction of evaluation results (Review LfMb): We identify and fix the monitorability-score aggregation bug, expand the evaluation to 8 rollouts per instance for Ouro 1.4B and 2.6B models, extend monitor rollouts to 4 per instance for the Ouro 1.4B model, and update all affected results. Our main findings remain unchanged after these corrections.
  2. Additional experiments on loop depths (Review KNVg): We provide new results at loop depths T=2T=2, showing that the same decreasing monitorability trend on Logic, Science, and Engineering persists when considering depths below the default T=4T=4.
  3. Statistical reliability and additional analyses (Review KNVg, BwKH): We report retained-question counts, paired bootstrap confidence intervals, and statistical significance tests. (Tables 8–11 in the revised manuscript)

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.