- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper studies elicitation problem: given a utility function on model continuation, find a prompt that maximizes the expected utility of the continuation. The authors study the problem on Bayes filtered transformers, whose next token prediction approximate a bayesian posterior predictive and where the objective is an expectation of a per latent unity over the posterior of the prompt. They propose Posterior Prefix Tuning, that replaces the hard prompt with a prompt generating kernel. It defines a tiled posterior by weighting the prior with the expected likelihood of prompts and maximizes the resulting surrogate by gradient ascent. The gradient is a covariance under the tilted posterior, estimated by importance sampling from a fixed set of prior samples. Therefor prior samples can be reused across utilizes. Experiments use a binary vocabulary, continuation length 4, prompt length 6 and 50, three utilities, with GCG as the baseline.
Critical strengths
The paper takes a clear probabilistic view of elicitation: under the latent-posterior model, a prompt matters only through the posterior it induces over latent predictors, so elicitation becomes moving that posterior toward high-utility latents. I find this framing natural and useful, and it links elicitation to standard tools from Bayesian inference.
Critical weaknesses
I like the framing, but in its current form I find it hard to judge what the method adds, either conceptually or empirically.
-
Positioning relative to prior work. Choosing inputs so that a Bayesian learner’s posterior moves toward a target is closely related to machine teaching for Bayesian learners (Zhu, NeurIPS 2013). That work also relaxes the discrete choice and then rounds. Machine teaching has also been used to find adversarial training sets (Mei & Zhu, AAAI 2015). Neither is discussed. Apart from the tilted surrogate, the components (the Bayesian view of in-context learning, predictive Monte Carlo, importance sampling) are existing tools. A discussion of what PPT adds beyond this line of work would help.
-
Surrogate without guarantees. As the authors note, the surrogate can differ from the true objective, with no bound on the gap, and the rounded prompt has no optimality guarantee. Improving the surrogate therefore need not improve elicitation.
-
Scale of the evaluation. The experiments use a binary vocabulary, short prompts and continuations, and conjugate priors. In these settings the posterior depends on a few counts and the problem is small. Results against GCG are mixed, including one setting where PPT does much worse. The authors also note that the conditions PPT needs are open for real LMs. So it is not yet clear in which regime the method is needed.
Questions for authors
- How does PPT relate to machine teaching for Bayesian learners? Could you compare against relaxing the counts (sufficient statistics) and rounding?
- What can PPT handle that machine teaching cannot, for example priors that are only accessible through samples, or non-conjugate priors? An experiment in such a setting would make the case clearer.
- Can anything be said about the gap between the surrogate and the true objective, or about the quality of the rounded prompt?
- What would be needed to extend PPT beyond binary vocabularies and low-order Markov latents?