- Rating
- 1
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
- This paper presents a definition of monitorability and framing evaluations as proxy metrics for the ideal definition.
- This paper shows how small changes in evaluation configuration can change the results of monitorability evaluations on a given benchmark. A range of experiments are run across up to 6 non-frontier actor models and up to 3 different benchmarks. These results show that correcting for "false, false positives" (FFPs) and the summary statistic used changes ranking between models. These findings show that care needs to be taken when running monitorability evaluations and that experiment design is important for getting robust results. It recommends that results should be corrected for FFPs and that unelicited benign transcripts should be used for more reliably evaluating if a monitor can discriminate between transcripts.
Critical strengths
- I found the investigation into how differing the evaluation set-up in seemingly reasonable ways can result in different outcomes interesting. It is important to understand how accurate these evaluations are, and working towards a robust method for evaluation is necessary for high quality monitoring techniques to be developed for effective AI control.
Critical weaknesses
- This paper has formatting issues. There are many visual errors, e.g. Figure 4.3 (b) is unreadable due to the legend partially blocking the bars, on page 7 there is unexplained blue text. I also spotted an inconsistency in the results presented, in Figure 4.2 the caption states "The GPT-OSS-120B bar in the “No supp. (FFP correct.)” subplot is labeled N/A" but the figure shows a value in this column. In addition to this there are errors in the references, e.g. Guan et al. is cited from arxiv but is published in ICML. These presentation issues undermine the content presented and may make the reader doubt the correctness of the results.
- The definition of monitorability is presented as a contribution however it appears to be similar to the definition of monitorability in "Monitoring Monitorability" by Guan et al. (ICML 2026). It would be beneficial to the reader to frame how the formalisation of monitorability is adding onto pre-existing work.
Questions for authors
- Can you explain clearly how your definition of monitorability is distinct from previous works and what exactly is the new contribution?
- Can you rectify the formatting and quality of writing issues so as to better showcase the quality of your work (e.g. plot formatting, text formatting, referencing works that have been published outside of arxiv correctly)