- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The main point is that monitors trained in one setting can fail in another. They show this by taking a monitor from one setting and showing that it behaves in an unpredictable way in another setting.
Critical strengths
Monitors and monitoribility is important
Critical weaknesses
- Lack of novelty (isn’t that standard distribution shift?)
- Poor evidence (3 models and 3 tasks?)
- Poor statistics
- Poor communication (difficult to read, no detail about the monitor)
Somme comments:
Abstract: a bit difficult to understand (AI-generated?) Some examples:
- Terms used that have not been introduced e.g., portability, select trust, transfer grid, breaker
- Not sure what it means:
- “Finally, an independently fitted and calibrated HarnessAudit breaker raises alarms before the audited harm locus on 91.7% of harmful evaluation episodes at 10.9% false cuts.”
- “Reversing the GPT 5.5 transfer gives 0.815”?
- “This separation changes what is required for reuse”.
Introduction:
- Difficult to understand
- Terms introduced without explanation: score (is it a function in this context? It should be explained), target harm, runtime representation, operating point, monotone model ordering, FIN, …
- “Conflating these properties under a single notion of robustness obscures both the source of failure and the evidence needed to respond to it”: Who mentioned a single notion of robustness? Is the point that this is how it is currently done? This should be mentioned.
- “The preregistered monotone model ordering therefore fails on this grid: the middle model transfers strongly while its two neighbours lie near chance.” Who are the neighbours and what grid?
- Not a proper introduction: The introduction reads like a summary of the paper, this is not the goal of an introduction. e.g., “Once monitorability and portability are separated, the next question is whether portability can be inferred without measuring the target ranking directly. The tested evidence becomes progressively more informative, but none resolves the problem as accurately as direct target measurement.” => This belongs in the methods and results.
Section 2:
- Lack of important information: No information about the monitor. We know that it’s called SEQ and that it has been preregistered apparently, but we don’t know what model it is and what it does. Most of the models are based on this so it should be defined somewhere.
- Poor statistics: Some of the results do not contain statistics, they are just point estimate comparisons. Intervals are also not reported. A bit inconsistent given that the main point of the paper is that we need to make decisions rigorously and account for uncertainty.
Section 4:
- Why is there a theorem, isn’t it a standard method? It’s atypical to introduce it this was, makes it sound novel
References:
- Interesting choice to cite a French book from 1939
Questions for authors
NA