acceptodds
Under review as a conference paper at ICLR 2027

Value Probes: Learned Value Functions for Beam-Search Red Teaming

Abstract

Automated red-team auditing of language models is an important way to evaluate safety-relevant behaviors. These audits implicitly search a tree of possible conversation pathways: at each turn an auditor model decides which conversation to continue, and an LLM judge scores the dialogue. We emphasize that this auditing procedure is a Markov decision process, where states are the conversation history, actions are candidate responses, and rewards are judge scores for the target behavior. By adopting this perspective, we introduce two major improvements to auditing. First, we propose adopting beam search for the audit rather than relying on the auditor alone. We then interpret the problem of finding states that are likely to lead to problematic behavior as value estimation. Surprisingly, we show that this value function can be estimated by linear probing from the target model's activations to predict discounted future rewards—a method we term “Value Probes.” By using Value Probes as a heuristic in our beam-search audit, we are able to substantially improve auditing success. Across 18 target behaviors and 3 scales of models, our approach raises peak audit scores by roughly 50% compared to (token-matched) standard auditing approaches. Furthermore, we show that Value Probes can be fit using off-policy data from smaller models. We also ablate individual components, and show that our method offers a comparably-performant and more efficient alternative to more expensive language model follow-ups. We hope these methods will improve language model auditing and lead to safer models.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Automated Red Teaming, Linear Probes, Markov Decision Process, Multi-Turn Jailbreaks, Interpretability, Value Functions

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_cxooOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper frames automated red-team auditing as a Markov decision process, where the state is the conversation so far and the reward is an LLM judge's score for a target behavior. The authors replace the auditor-driven search of Petri with beam search, and rank branches with Value Probes: linear probes on the target model's mean-pooled activations at one layer, trained to predict discounted future judge scores. The method is evaluated on 18 behaviors with Gemma 3 4B, 12B, and 27B as targets, against a token-matched Petri audit, beam search with random selection, and beam search ranked by a Claude Opus 4.8 judge. The probe arms reach a mean peak judge score of 0.79 to 0.85, against 0.51 to 0.57 for Petri (Table 1). Probes trained on Gemma 3 4B rollouts are also tested on Gemma 3 12B, Gemma 3 27B, and Gemma 4 31B (Table 2).

Critical strengths
  1. The MDP framing separates the search from the ranking signal, and the random-selection arm shows how much of the gain over Petri comes from the search structure alone (Section 5.2).

  2. The Max Value Probe is within 0.04 of the Claude Opus 4.8 judge in mean peak score at every model size, using a single linear readout instead of 288 extra judge calls per audit (Section 5.3).

Critical weaknesses
  1. The results suggest that much of the gain over Petri comes from beam search itself, and the probe's own contribution is not separated from simpler rankers. Beam search with random selection outperforms Petri by 0.24 at 4B and 0.15 at 12B and 27B, while the probes add 0.06 to 0.10 (Section 5.2). There is no ranker that predicts only the current node's judge score, and none trained on the same targets from the transcript text alone, so it is unclear how much the forward-looking target and the activations each contribute.

  2. The transfer claim for Gemma 4 31B is not supported by the experiment. On 31B the transferred probes are compared only with random selection (Table 2), yet Section 5.5 states that transfer to a new model comes at no measurable cost relative to an on-policy probe. The Conclusion also compares the 31B scores with the Gemma 3 scores, although the Figure 3 caption says the 31B runs use a different judge and should be compared only within that panel.

Questions for authors
  1. Could the authors compare the probe with a simpler ranker trained on the same data, such as a probe that predicts only the current node's judge score, or a small model that reads the transcript text? This would show how much the forward-looking target and the activations each contribute.

  2. Would a Petri audit on Gemma 4 31B be feasible? This would help support the claim that transfer comes at no measurable cost for a new model.

1 comment

Thank you very much for your review, these were very good comments. We attempted to acquire budget for these follow up experiments, but were unable to be approved before the deadline. However, although we do not have results from a full sweep, we do have some partial results from experiments we ran earlier we can share in our comment. We did not train probes for this, but we did find that using smaller judge LLMs with the blackbox method we detail, produced lower scores during auditing. We have notes from early experimental runs that suggest that a small enough capacity judge provides little to no lift over the random heuristic. As far as a full Petri audit on Gemma 4 31B, this is a good call out, we really want to include results for these experiments, but due to the above budget constraints, we weren't able to provide these results before the review. We've added an additional cost analysis section which gives a sense of the costs for conducting our audits, harvesting rollouts, and training new probes on new rollouts. If our budget request is approved later, we are happy to post results as a follow up for completeness. Thank you again!

Reviewer_q3iKOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a more effective way to uncover unsafe LLM/Agent behavior by combining beam search with predictions from the model’s internal activations about which conversations may become problematic.

Critical strengths

Auditing conversations and their potential rollouts for red teaming is an interesting idea that could be used as a mechanism to assess whether models are ready to react to those type of attacks.

The paper shows clear improvements over the Petri baseline.

Critical weaknesses

Jailbreaking and prompt injection auditing is usually something you want to catch on the fly, meaning when the prompt arrives and when the trajectory is actually guided by the user. In this, this an auditing that would happen offline. While this is useful from a study point of view, I question the value from an actual application point of view.

The authors could have used many other potential models, they tested 3 variants of Gemma, and they show that their approach does better than the Petri audit, which is by Fronsdal et al., (2025)

The paper is hard to follow in many places. For example:

  1. the entire paragraph 455 to 466 is really hard to distinguish what the authors would like to explore in future. It feels a list of citations
  2. Lines 365 to 370, the authors cite a bunch of results that can be seen in the plots but they don't really explain what this means and refers to section 6. The explanation in Section 6, in lines 435 to 442 is not better and makes it worse.
  3. Line 431 "The transfer results add that the direction a probe finds is not specific to the model whose rollouts trained it" Can you just say that the probe generalizes across models?
Questions for authors

Line 366 to 367. Ok so the max value probe is ahead in some models and ~same in others, what does this mean for the paper? This entire paragraph is also hard to follow.

1 comment

Thank you for your review, we appreciate the writing feedback, and have made edits in our revision with your suggestions. Regarding your point on the 3 variants of Gemma, we have preliminary, but incomplete results for other models using our method, but due to budget constraints, and context window differences in smaller models, we were unable to run the full sweep for all models we wanted to. We have added a section in the appendix with details on the models we ran preliminary audits for (Qwen, Llama, Phi).

Jailbreaking and prompt injection auditing is usually something you want to catch on the fly, meaning when the prompt arrives and when the trajectory is actually guided by the user. In this, this an auditing that would happen offline. While this is useful from a study point of view, I question the value from an actual application point of view.

Thank you for pointing this out, we have updated our paper with a note on this; in addition to offline auditing, our probes are usable in online detection flows as well. We did not emphasize this point due to our focus on the offline auditing piece which we believe to still be an important step to check for alignment of a new model pre-release, but we've added a note in the introduction pointing out that auditing is considered an offline flow, as well as a note in future work that usage of our probes in monitoring can be further explored.

Reviewer_8DC6OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper formulates multi-turn model auditing as a search problem. Conversation branches represent alternative auditing strategies, and a learned value function helps determine which branches to retain and extend.

The proposed value functions are lightweight linear probes trained on the audited model’s hidden activations. They predict discounted future behavioral scores and guide beam search, reducing the need for expensive LLM-based branch evaluation.

Experiments span 18 target behaviors and several model sizes. A substantial part of the improvement over the token-matched Petri baseline comes from the beam-search structure itself. Adding probe-based ranking produces a further improvement of approximately 0.06–0.10 in peak judge score over random branch selection.

Critical strengths
  1. A useful control for search structure. Random branch selection helps separate the benefit of beam search from the additional benefit of the learned ranking signal.
  2. Evaluation across behaviors and model sizes. Paired comparisons across 18 behavior categories provide more informative evidence than a single aggregate score.
  3. A practical data-transfer experiment. Conversations and labels collected using a smaller model can support probe training for larger models. This may reduce the cost of collecting new auditing trajectories.
Critical weaknesses
  1. A current-reward-only probe is an important missing control. A probe trained with the same features and data but with γ=0\gamma=0 would help determine whether future-value prediction contributes beyond recognizing behavior already present in the conversation. This matters because training targets include the current reward, while evaluation uses the maximum score over the dialogue.
  2. Efficiency should include preparation costs. Trajectory collection, LLM judge labels, activation extraction, and per-behavior probe training all require resources. Reporting a break-even point would clarify how much repeated auditing is needed to recover these costs.
  3. The method requires internal model access. Its applicability is limited when hidden activations are unavailable.
Questions for authors

If I understand correctly, I think the following distinction can be remained explicit: transfer is not direct probe-weight transfer. The larger model’s activations are used to train new probes on reused conversations and labels.

1 comment

Thank you for your review, regarding your question for authors, that is correct, transfer is not direct probe weight transfer, it is using the larger models activations to train the new probes on the conversations and labels reused from rollouts in the smaller/older model.

Regarding the weaknesses:

  1. We agree with the assessment that exploring the shape of how these change over time. In the past week we attempted to acquire budget to run these experiments, but could not be approved in time by the deadline. We believe understanding the shape of performance as gamma changes is still important, and we have added this mention to our future work section.
  2. This is a great call out, we've added a new section in the appendix for this which is for cost analysis and breakeven points for our experiments. There are several variables here, for which we've included our values, but we've also included the methodology we use to calculate the breakeven points, so anyone who wants to run a variant of our experiments can run their own calculations.
  3. For the value probes, this is correct. We hope that for blackbox access, that the beam search improvement coupled with a high capacity LLM judge, although more expensive per audit, serves as a means to perform audits at comparable efficacy to our whitebox method.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.