- Rating
- 2
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
LLM agents can take harmful actions, so a guard model labels recorded trajectories as safe or unsafe, usually against written safety policies. Existing guards are handed the right policy, but in practice you have a library and must pick one, and a wrong pick means a wrong judgment. The proposed RePolicy reads short policy descriptions, fetches the relevant one, then judges the trajectory against it. It’s trained with SFT and then RL on a synthetic 20k-trajectory dataset, with shuffled policy IDs, order and decoys so it learns to match on content.
Critical strengths
Strengths
- Splitting “which rules apply?” from “was this a violation?” is a real, practical distinction. Most policy-aware guards quietly assume the first part is solved, and making it an explicit, inspectable step means you can actually audit what the guard consulted. Reminds me of the arguement used in formal verifcation works.
- Good metric selection beyond the typical F1score: decoy selection rate, number of policies retrieved, FP/FN breakdowns, library-size sensitivity. Experimental evaluation seems decent.
- Reproducibility is strong. Code, splits, policy library, rewards, hyperparameters and a very detailed baseline protocol are all released.
Critical weaknesses
Weaknesses
- Unsafe-class F1 ignores true negatives, so over-flagging is never penalized. An “always unsafe” classifier scores about 79.8, so RePolicy’s 88.15 is only about 8 points above trivial, and it’s below trivial on OpenAgentSafety and AgentHazard. No per-benchmark false-positive rates are reported.
- The policy library is fixed at 30 policies for both training and test. A 4B model can memorize them, so “selection” may really be 30-way classification, and nothing tests unseen, rewritten, or user-defined policies, which is exactly the motivating scenario.
- Correct me if I am wrong but the policy reward is defined only for a single policy, yet the model retrieves about 3.2 on average. If the reward just checks that the right one is included, retrieving more is a reward hack, and the rising retrieval count looks suspiciously like that.
- No error bars or confidence intervals.
Questions for authors
Questions
- Unsafe-class F1 ignores true negatives. Can you report FPR, FNR, balanced accuracy (or macro-F1) and AUROC on each of the six benchmarks, not just on your synthetic validation set?
- What does an “always unsafe” classifier score per benchmark and overall? Given your split sizes it’s roughly 79.8 Overall, and above RePolicy’s score on OpenAgentSafety and AgentHazard. How should readers interpret the 88.15 against that?
- Can you report the pooled F1 you already compute (Eq. 10), and Overall with AgentHazard excluded, since it has no safe examples and can’t penalize false alarms?