- Rating
- 2
- Confidence
- 1
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The paper investigates how increasing inference-time compute can inadvertently increase safety violations in language models. It identifies two inter-related failure mechanisms: contamination, where an imperfect safety filter passes some unsafe responses, and amplification, where reward maximization selects these responses with high probabilities during Best-of- selection. The authors derive finite-sample bounds and asymptotic conditions showing that the probability of selecting an unsafe response can approach one when unsafe, proxy-feasible responses have sufficiently high reward under the learned reward model, even when false-acceptance probabilities and average proxy errors are small. They further establish that coverage control can bound amplification independently of the sampling budget but cannot eliminate the underlying contamination. Based on this insight, they propose constrained pessimistic sampling (cPes), which combines reward clipping with regularized selection to limit concentration on high-reward candidates. Toy simulations and LLM experiments using Qwen2.5-7B and Mistral-7B, Llama Guard and an alternative safety filter, and two reward models provide empirical evidence for the proposed mechanism.
Critical strengths
Although I do not fully understand the theoretical contributions, I believe the paper addresses an important problem, as inference-time scaling is an increasingly widely used paradigm. The paper demonstrates that increasing the computational budget can worsen safety despite safety filtering and reward-based selection, highlighting a potential failure mode that deserves greater attention in the literature.
Critical weaknesses
The paper is difficult to follow in its current form. I suggest that the authors explain their theoretical findings more clearly in plain language to make them accessible to a broader readership. I also recommend including a summary table of the notations for ease of reference.
Questions for authors
How does cPes compare empirically with standard constrained Best-of-(N) selection in terms of safety and utility? Are its effects consistent across different reward models and safety filters?