- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
My reading is that the paper tries to clarify what runtime defenses for LLM agents can observe, decide and actually enforce. It brings together 77 version-pinned records with 25 core mechanisms mapped in detail. I see the external coding and small handoff tests as support for this comparison.
Critical strengths
I appreciate the care put into the source records. Separating a learned safety judgment from an enforced restriction also helps me make sense of the different systems. The authors show where the coders disagreed, which I found usefull. The paper has value as a reference.
Critical weaknesses
-
My main concern is that I don't see enough of a new contribution. Section 2 explains how this map differs from earlier surveys but the main takeaways still feel familiar: local checks may not compose, permission doesn't establish intent, and a safety verdict doesn't tell us what actually happened. Organizing these ideas is useful. What I'm missing is a substantial conclusion or design insight that emerges from putting them together. A survey can certainly make that contribution and I don't think the finer categories here are enough on their own.
-
I'm also not convinced that someone else could use the framework reliably. In Appendix B.1, exact-set agreement is 54.5% for target labels and 47.7% for inferred-premise labels across 44 jointly applicable records. The core horizon, modality, authority-scope labels do agree, which is reassuring. The harder distinctions remain the issue: having an author settle the disagreements doesn't show that another reader would reach the same decisions. Nor do the 80 handoff runs in Section 6.2 close that gap. They check expected outcomes on examples the authors constructed, including deliberately permissive controls. I'd find a case where the framework exposed a previously overlooked problem much more convincing. The authors are upfront about these limits but the evidence is still too limited.
Questions for authors
-
I may be missing something: which finding goes furthest beyond what the surveys in Section 2 already tell us?
-
Do you have an example where someone independently used the framework and found an overlooked issue or changed a design decision? I'm asking about existing evidence here rather than proposing another experiment for the rebuttal.