acceptodds
Under review as a conference paper at ICLR 2027

Safety Hacking in Constrained Best-of- Inference-time Scaling

Abstract

Inference-time scaling pipelines often sample multiple outputs, filter them with a learned safety model, and return the proxy-feasible output with the highest learned reward. We show that this pipeline can fail in two stages: an imperfect safety proxy first contaminates the feasible set with unsafe outputs, and reward maximization can then amplify this residual contamination. We call this failure safety hacking, in which the selected output passes the learned filter but violates the true safety constraint. For constrained Best-of- sampling, we derive finite- bounds determined by the joint upper reward tails of safe and unsafe outputs within the proxy-feasible set. If unsafe-but-feasible outputs have the heavier tail, the probability of safety hacking approaches as grows, even when false-positive mass and average safety and reward proxy errors are arbitrarily small. We also show that policies within a bounded divergence from the proxy-feasible reference distribution admit an -independent safety-hacking bound. Coverage control limits amplification but cannot repair a contaminated feasible set: admitted unsafe outputs may still be favored, and regularized selection is not necessarily safer than constrained Best-of-. Toy and language-model experiments characterize both contamination and its reward-tail amplification, which exposes an inherent difficulty in inference-time scaling with imperfect reward and safety models.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Inference-time scaling, AI safety, Best-of-$N$ sampling

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_okVNOpenReview ↗
Rating
2
Confidence
1
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The paper investigates how increasing inference-time compute can inadvertently increase safety violations in language models. It identifies two inter-related failure mechanisms: contamination, where an imperfect safety filter passes some unsafe responses, and amplification, where reward maximization selects these responses with high probabilities during Best-of-NN selection. The authors derive finite-sample bounds and asymptotic conditions showing that the probability of selecting an unsafe response can approach one when unsafe, proxy-feasible responses have sufficiently high reward under the learned reward model, even when false-acceptance probabilities and average proxy errors are small. They further establish that coverage control can bound amplification independently of the sampling budget but cannot eliminate the underlying contamination. Based on this insight, they propose constrained pessimistic sampling (cPes), which combines reward clipping with regularized selection to limit concentration on high-reward candidates. Toy simulations and LLM experiments using Qwen2.5-7B and Mistral-7B, Llama Guard and an alternative safety filter, and two reward models provide empirical evidence for the proposed mechanism.

Critical strengths

Although I do not fully understand the theoretical contributions, I believe the paper addresses an important problem, as inference-time scaling is an increasingly widely used paradigm. The paper demonstrates that increasing the computational budget can worsen safety despite safety filtering and reward-based selection, highlighting a potential failure mode that deserves greater attention in the literature.

Critical weaknesses

The paper is difficult to follow in its current form. I suggest that the authors explain their theoretical findings more clearly in plain language to make them accessible to a broader readership. I also recommend including a summary table of the notations for ease of reference.

Questions for authors

How does cPes compare empirically with standard constrained Best-of-(N) selection in terms of safety and utility? Are its effects consistent across different reward models and safety filters?

Reviewer_aGF4OpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

This paper studies safety failures in constrained Best-of-N inference, where candidates are filtered by a learned safety model and the highest-reward accepted candidate is returned. It separates safety-filter contamination from its amplification through reward maximization. The authors derive finite-budget probability bounds and sufficient tail conditions under which unsafe selection becomes asymptotically certain, even with small average proxy errors. They also establish coverage-based bounds and study constrained pessimistic sampling (cPes), a regularized alternative that limits concentration. Simulations and language-model experiments examine the dependence of amplification on reward ranking.

Critical strengths
  1. The distinction between filter contamination and downstream amplification is useful. It explains why average safety-filter accuracy alone is insufficient to characterize the safety of reward-based selection.

  2. Random-feasible selection controls for exposure without reward maximization. The reward-model ablation is particularly informative: replacing Beaver with Skywork reverses the scaling trend while holding candidates and filtering fixed, showing that increasing the sampling budget is not intrinsically unsafe.

Critical weaknesses
  1. The asymptotic result provides a sufficient condition, while the language-model experiments demonstrate finite-budget behavior. The pooled survival curves are useful diagnostics, but do not establish prompt-specific tail conditions or show whether amplification can be predicted before running large-budget selection. The paper would be stronger with an out-of-sample evaluation of its tail-based diagnostics.

  2. The coverage-based guarantee is independent of the sampling budget, but may still be loose or vacuous depending on residual contamination, acceptance probability, and the concentration bound. The experiments do not quantitatively establish how well this guarantee predicts observed risk. Budget independence alone does not demonstrate a practically useful safety guarantee.

Questions for authors
  1. Can the authors report numerical values of the safety bounds under the experimental settings, alongside observed hacking rates? When are these bounds non-vacuous?

  2. Can tail statistics estimated from a smaller candidate pool predict amplification on an independent, larger pool, particularly at the prompt level?

  3. How does cPes compare with clipping-only CBoN, softmax-based selection, and smaller-budget CBoN at matched safety or utility levels?

  4. Would the conclusions hold on a mixed benchmark of benign and adversarial requests? Please also report abstention and unconditional unsafe-output rates alongside conditional hacking rates.

Reviewer_iwFwOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies safety hacking in constrained Best-of-N inference-time scaling, where sampled responses are first filtered by a safety model and then ranked using a learned reward model. The authors identify two sources of failure: contamination of the feasible set by unsafe responses that pass the safety filter, and amplification of the errors through reward maximization. The paper derives theoretical bounds characterizing safety-hacking probability. It further shows that controlling distributional deviation from the proxy-feasible reference policy can bound amplification independently of the sampling budget, and studies constrained pessimistic sampling as an instantiation. Some toy and LLM experiments support the analysis, demonstrating that safety-hacking behavior depends on reward-model ranking and safety-filter errors.

Critical strengths
  • Clear problem formulation: The paper explains how small safety-filter errors can become more serious when selecting the highest-reward response from many candidates under certain assumptions.
Critical weaknesses
  • Oversimplified toy setting: The toy experiment assigns substantially heavier reward-proxy noise to unsafe responses, making their dominance under extreme-value selection pretty much expected. The experiment provides limited evidence that such adverse reward-tail distributions arise in practical settings.

  • Strong dependence on reward-model choice: In fig4a, replacing Beaver with Skywork reverses the observed scaling trend: safety hacking decreases rather than increases with larger sampling budgets. So the proposed issue might just sometime appear in some ill-trained reward model.

  • Limited empirical validation of theoretical assumptions: The main LLM experiments do not directly characterize the relative upper-tail distributions, which somehow motivates and incurs the theoretical results. These experiments demonstrate finite-budget amplification in selected configurations without establishing whether the sufficient conditions for safety hacking hold.

Questions for authors
  • Do the author verify the key assumptions made in the theoretical analysis exist in a wide range of safety and reward models?
  • Do the author consider extending experiments to consider specific combination of safety and reward models?
  • Can the author provide justification that the constrained BoN is still a valid method for the most modern/frontier way to select safety outcome? If a paradigm is abandoned or not used in practical setting, then the issue might not be that serious.
  • Do the correlation or interaction between the safety model and reward model affect the theory or the empirical resuls?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.