- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
Test-time selection improves LLM accuracy by choosing one response across a set randomly generated ones. For example, the Best-of-N method assigns a score to each response and selects the highest one as the final prediction. One way to design this score is to train an auxiliary model on labelled data to predict which responses of the original model were correct or not [1]. The current paper argues that, since this labelled bank is generated using the original distribution of LLM responses, the accuracy measurements estimated from it will differ from the operational model accuracy. This is because, during operation, the model chooses one answer over , which induces a distributional shift between prompts seen in labelled data and during operation. The paper investigates this distributional shift theoretically and empirically.
[1] Cobbe, K., Kosaraju, V., Bavarian, M., Chen, M., Jun, H., Kaiser, L., ... & Schulman, J. (2021). Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168.
Critical strengths
- The paper tackles a very novel problem: how choosing answers (possibly using an auxiliary model called a verifier) induces distributional shifts between operational data and labelled data collected previously to estimate answer accuracy. To the best of my knowledge, no prior work has previously observed or adressed this issue.
- The theoretical results make very loose assumptions about the distributions of selected answers (e.g. Theorem 3.1). Consequently, they hold for many selection approaches: choosing randomly across , using Best-of-N with a continuous score etc.
Critical weaknesses
My main concern is that the paper can be hard to follow for reader who are not familiar with measure theory and certain notations. For instance
- The decreasing rearrangement of a probability density ratio is an critical concept to understand Theorem 3.1. It would be beneficial to quickly define and explained this notion in the appendix. For example, you could clarify that the decreasing rearrangement maps to and measures the upper quantile of the ratio with . loosely implies that for of the probability mass of , the new density is at least twice as large. With this interpretation, equation (5) actually makes a lot of sense.
- Additionally, the symbol is introduced in complexity analysis () and other times on functions ( in equation 10 for example). Is it not clear what this symbol represents and why it can be applied over functions and scalars.
- My other critiques are questions related to specific parts of the paper, which I put in the Questions For Authors section bellow.
Questions for authors
-
Line 164 states that the ratio is fixed independently of the i.i.d. labelled draws that make up the bank. I assume this bank is then used to train the verifier and that this verifier is used to select one answer over . If this is the case, then doesn't the ratio depend on the verifier and thus its training data.? Perhaps I am missing something.
-
Equation 8 is not very clear to me.
- It is the first introduction to , which was never defined beforehand. Is the LLM reponse accuracy on labelled data or the accuracy estimate returned by the verifier? These notions of estimate accuracy have different sources of error: finite sample, performance of the verifier etc. so more details are required.
- Over what space is the infimum taken?
- I am also unsure of what the expectation represents. I think only the term is a random variable in so is it an expectation over all possible labelled subsets? Why is a subscript necessary?