- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The authors propose CIRCLE, which addresses all three while staying black-box (no internals, logits or gradients), free at the margin (every feature is a by-product of harness logs) and self-calibrating (valid after it starts intervening). Randomizing the irreversible action is not a regularizer; it is what keeps the risk estimate a function of observable data once the controller is live.
Critical strengths
1.The authors express the experimental setup in detail. Experiments are pretty comprehensive and multi-facet.
- Strong mathematical analysis that makes the method loves more reliable.
Critical weaknesses
To be honest, I scanned all assigned papers before writing reviews, this paper has the best framing and figures, which is the only paper that leaves a positive impression on me. However, it was disappointing to see that the abstract was purely generated by AI, detected by GPTZero, which is just like ALL other assigned papers. I will never give positive assessments on papers like this: Abstract being the very core part of a paper, is purely generated by AI, I can image how much effort is put into the paper. Also, the authors say that “Large language models were used only to polish the language of the manuscript.” The code can not be totally hand-crafted.
Questions for authors
- The author said the problem arises from "agents fall into doom loops (re-running the same failing test, re-reading the same file, re-proposing a patch already rejected)", can this be solved or mitigated by Recursive Self-Improvement (RSI)?