acceptodds
Under review as a conference paper at ICLR 2027

Eliciting Model Behaviors by Optimizing the Latent Posterior

Abstract

In the latent posterior model of transformer behavior, the next-token distribution arises from a posterior over latent predictive models conditioned on the context, mixed to generate continuations. We exploit this model in settings where it is exact, namely Bayes-filtered transformers (BFTs) meta-trained on sequences from a hierarchical prior, to introduce Posterior Prefix Tuning (PPT), a new method for eliciting behavior from a transformer: given a utility function on continuations, find a prompt under which the model's expected utility is large. For a BFT, the elicitation objective factors through the latent posterior, and we show that the gradient of a tilted surrogate is a covariance over the tilted posterior between the per-model utility and the score of the tilt. PPT optimizes the parameters of a distribution over hard prompts, drawing latent-model samples once from the BFT's prior via predictive Monte Carlo (PMC) and computing its gradient by importance sampling. The inner loop performs no transformer forwards and no backpropagation through the model, and the prior samples are utility-independent, so the cost of characterizing the BFT amortizes across utilities. We validate PPT and a Rao–Blackwellized variant on Beta–Bernoulli and reinforced urn BFTs across three utility families (reverse cross-entropy, frequency matching, Dyck validity).

Primary area:
probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
Keywords:
Bayesian inference, in-context learning, meta-learning, elicitation, prefix tuning, importance sampling, transformers

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_Q8cYOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies elicitation problem: given a utility function on model continuation, find a prompt that maximizes the expected utility of the continuation. The authors study the problem on Bayes filtered transformers, whose next token prediction approximate a bayesian posterior predictive and where the objective is an expectation of a per latent unity over the posterior of the prompt. They propose Posterior Prefix Tuning, that replaces the hard prompt with a prompt generating kernel. It defines a tiled posterior by weighting the prior with the expected likelihood of prompts and maximizes the resulting surrogate by gradient ascent. The gradient is a covariance under the tilted posterior, estimated by importance sampling from a fixed set of prior samples. Therefor prior samples can be reused across utilizes. Experiments use a binary vocabulary, continuation length 4, prompt length 6 and 50, three utilities, with GCG as the baseline.

Critical strengths

The paper takes a clear probabilistic view of elicitation: under the latent-posterior model, a prompt matters only through the posterior it induces over latent predictors, so elicitation becomes moving that posterior toward high-utility latents. I find this framing natural and useful, and it links elicitation to standard tools from Bayesian inference.

Critical weaknesses

I like the framing, but in its current form I find it hard to judge what the method adds, either conceptually or empirically.

  1. Positioning relative to prior work. Choosing inputs so that a Bayesian learner’s posterior moves toward a target is closely related to machine teaching for Bayesian learners (Zhu, NeurIPS 2013). That work also relaxes the discrete choice and then rounds. Machine teaching has also been used to find adversarial training sets (Mei & Zhu, AAAI 2015). Neither is discussed. Apart from the tilted surrogate, the components (the Bayesian view of in-context learning, predictive Monte Carlo, importance sampling) are existing tools. A discussion of what PPT adds beyond this line of work would help.

  2. Surrogate without guarantees. As the authors note, the surrogate can differ from the true objective, with no bound on the gap, and the rounded prompt has no optimality guarantee. Improving the surrogate therefore need not improve elicitation.

  3. Scale of the evaluation. The experiments use a binary vocabulary, short prompts and continuations, and conjugate priors. In these settings the posterior depends on a few counts and the problem is small. Results against GCG are mixed, including one setting where PPT does much worse. The authors also note that the conditions PPT needs are open for real LMs. So it is not yet clear in which regime the method is needed.

Questions for authors
  1. How does PPT relate to machine teaching for Bayesian learners? Could you compare against relaxing the counts (sufficient statistics) and rounding?
  2. What can PPT handle that machine teaching cannot, for example priors that are only accessible through samples, or non-conjugate priors? An experiment in such a setting would make the case clearer.
  3. Can anything be said about the gap between the surrogate and the true objective, or about the quality of the rounded prompt?
  4. What would be needed to extend PPT beyond binary vocabularies and low-order Markov latents?
Reviewer_pwSAOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

In the paper, the authors propose a new method called Posterior Prefix Tuning (PPT), which tries to find prompts that can increase a certain desired behavior from a language model without requiring backpropagation through the transformer. The main idea comes from the Bayesian interpretation of in-context learning, where a transformer is assumed to maintain a posterior distribution over different latent predictive models, and the prompt changes this distribution. Now the authors argue that instead of directly searching over the prompt tokens, they can optimize this latent posterior to favor the behaviors that is wanted from the model. The authors evaluate their method on two synthetic Bayes transformers, with different utility functions and prompt lengths. The results are mixed, where PPT performs better in certain settings but the baselines remains competitive or significantly better in others.

Critical strengths
  1. I think the paper is reasonably well motivated and the idea of optimizing the latent posterior rather than directly searching over prompts is interesting. I also appreciate that the authors try to use an existing theoretical understanding of in-context learning to actually develop an optimization method, rather than just using it to explain model behavior.
  2. The paper is well written and easy to follow.
Critical weaknesses
  1. I think the biggest weakness of the paper is that the experiments are extremely limited compared to the motivation of the work. I understand that the authors want to study the problem in a controlled setting where the Bayesian interpretation is more accurate, but I find it difficult to understand how much these results actually tell us about eliciting behaviors from real LLMs. At least some experiments on more realistic sequence distributions or pretrained models would have made the contribution much stronger.

  2. The entire approach relies on the assumption that the transformer behaves approximately like a Bayesian mixture over latent predictive models. In the experiments, they train the transformers on data generated from such distributions, so the underlying structure is already known and deliberately introduced during training. I think this makes the problem much simpler than the actual elicitation problem for real language models. I am not sure how one would identify or parameterize these latent models for a normal pretrained transformer.

Questions for authors

Please check the weaknesses

Reviewer_Ri7tOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies the problem of elicitation in the specific setting of finding a prompt that makes a language model behave in a particular way, e.g. searching prompts that result in undesirable behavior for red-teaming. The authors adopt the common Bayesian view of in-context learning, where the model implicitly maintains a posterior over latent tasks and updates it as it reads the context, with the prompt steering the posterior towards a target behavior. The authors consider models trained on sequences from a known hierarchical priors (e.g. biased coin flips, binary Markov chains with random transitions). The authors propose optimizing over a distribution of prompts like an i.i.d. token distribution or a Markov transition matrix instead of searching over prompts. The proposed objective is a surrogate that reweights the latent prior with likelihood of the latent task under the prompt and its gradient is a covariance that can be estimated by importance sampling over prior samples. These samples are generated as long unconditional generations from the transformer and estimating the token frequencies.The same samples can then be reused for any target behavior. The approach is empirically evaluated on a few synthetic settings, with GCG as a baseline. The effectiveness of the approach is mixed and the comparisons are limited.

Critical strengths
  • The perspective of framing the elicitation problem as a latent posterior inference problem is principled and follows prior work. This work uses it to connect the Bayesian interpretation of in-context learning with red-teaming. and offers a different view of empirical phenomena such as persona-modulation attacks.
  • Procedurally, the method requires no gradient access from the model and can amortize the cost over different target behaviors with a long unconditional samples that can be reused.
Critical weaknesses
  • The method is primarily motivated in terms of safety elicitation in LLMs, but the specific theoretical setting studied in the paper make several strong assumptions which do not easily translate to LLM (e.g. exchangeable or Markov-exchangeable predictive behavior, a convergent martingale predictive for PMC, and a tractable finite-dimensional WW). So I feel in terms of significance the paper is limited by the actual applicability of the approach. This is acknowledged by the authors in their conclusion but does hold the paper back.
  • Even beyond the limited applicability, the empirical results have some issues. The paper relies on a single weak GCG baselines for their comparisons and as the authors acknowledge for the ∣Y∣=2|\mathcal{Y}| = 2 case GCG's top-kk step is meaningless. Additionally, in practice PPT relies on a large number of unconditional samples which in practice might outweigh the amortization benefits if there are not enough target behaviors. The paper also misses other natural comparisons like BDPL (policy gradient over per-token categoricals using forward passes), GBDA (Gumbel-softmax token distributions), the cross-entropy method, random search, and soft-prompt tuning as an upper bound.
  • Based on my understanding, the surrogate with r=0r=0 is the expected utility of hard prompts drawn from κ\kappa reweighted by the model's own prior probability of each prompt. This means Jtilt(κ)≤max⁡zJ(z)J_{\text{tilt}}(\kappa) \le \max_z J(z), with equality only when κ~\tilde\kappa concentrates on optimal prompts. For the kernel classes used, that means point masses, i.e. deterministic prompts and would result in a systematic bias towards low-entropy prompts likely under the prior.
Questions for authors
  • Is my understanding of the r=0r = 0 case? If so, why not adopt that interpretation and its bound?
  • Why round using JtiltJ_{\text{tilt}} rather than evaluating JJ with a handful of forward passes on the candidate prompts?
  • Is there any setting, even a toy one that isn't exchangeable, where PMC + PPT applies to a model not meta-trained on a known prior?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.