- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper presents a mechanism for separating self-reported model assessments from what is actually permitted by a policy. They provide results showing the mechanism improves results on several benchmarks across multiple tasks.
Critical strengths
- The core idea is well supported and relates to the much discussed topic of self-assessment by language models.
- The presented evaluation results are thorough with relevant ablations.
Critical weaknesses
- The presentation of the method is slightly disorganized and includes more jargon than necessary. In addition, the three theorems are more linked to safety claims than the performance claims of the paper.
- Many of the reported performance improvements reported are fairly small. For example, the Brier reductions in Table 2 are around 0.01 and the ToolSandbox gain in Table 3 is under 0.04 improvement.
Questions for authors
- What is the token cost overhead for this mechanism?
- How do you estimate the latency would change when used in more complex schemas than the ones evaluated here?
- How were the thresholds in Section 4.2 chosen?
Thank you for the authors' response. The paper would benefit from another round of revisions to more thoroughly address the feedback. At this stage, I will maintain my original assessment.