- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
In this paper, the authors propose a fine-tuning method optimized for inference time methods such as Best-of-n or Pass-at-k. They highlight a limitation in sampling each n/k samples from the same distribution, and propose an approach, inspired by stratified sampling, to jointly train a population of policies for better performance on inference time sampling strategies. They implement this strategy with LoRA adapters, and provide theoretical results in linear settings highlighting the necessity of different policies to achieve zero error on specific target functions. Through experiments, they highlight the efficacy of their approach across a variety of reasoning tasks and model scales.
Critical strengths
- The method is intuitive and well-motivated.
- The experiments are very compelling, I really appreciated the Mixture baseline as providing more context for the efficacy of the method: sampling from each policy is necessary to leverage a diversity of policies that can subsequently specialize, otherwise in the Mixture setting there is no guarantee that the LoRA "expert" for a given problem will be invoked.
Critical weaknesses
- You say your goal isn't to improve Pass@k. Why is this? Aren't you explicitly optimizing for this and trying to improve inference time methods?
- Presentation: You have some hanging 1/2 line phrases, such as "Unlike LOO, this modification does not give an unbiased estimate of the best-of-k policy gradient." at the end of your pages. They're slightly jarring and would probably save you space if you were to merge them with the paragraphs above them.
- The normalized color bars in Fig. 4 do not visualize problems which are solved by more than 1 policy, they just communicate the distribution of unique problems. I would be interested to see unnormalized forms to see proportion of problems that are not uniquely solved, otherwise its hard to understand what this is communicating, and potentially misleading readers into thinking that it is providing evidence towards individual policy specialization (which Fig. 6 shows does not always happen).
Questions for authors
I am curious about the bimodal pattern you observed in the Jaccard similarity plots. Do you think this indicates that, for certain tasks, fewer than 5 total policies are required to solve the task, leading to redundant policies?
3 comments
Reviewer Response
Thank you for clarifying things! I appreciate the last experiment (point 4) and the effort put in to responding and updating the paper. Maybe this last result could go in the Appendix of the paper.
I think this paper deserves an acceptance at the conference!
Thank you for your positive feedback! Based on your suggestion, we have added the analysis of the bimodal pattern in Knights-&-Knaves to Appendix H, Table 5. We would appreciate it if you could consider increasing your rating, if you feel the additional experiments and analysis have strengthened the paper. Please let us know if you have additional comments or feedback. Thanks!
Rebuttal
We thank the reviewer for carefully reviewing our paper! We greatly appreciate the feedback. Please see below our responses to the comments.
We have also updated the paper by adding additional experiments and analysis in the appendices based on the feedback (updated text in lavender).
1. You say your goal isn't to improve Pass@k. Why is this? Aren't you explicitly optimizing for this and trying to improve inference time methods?
We would like to clarify that our goal is indeed to improve pass@k. Lines 108-112 on page 3 emphasize that we do not aim to design a state-of-the-art method for pass@k. Instead, we view using stratified policies as a complementary approach that could also benefit other pass@k optimization methods.
2. ... You have some hanging 1/2 line phrases... at the end of your pages.
We thank the reviewer for noting this. We have merged these sentences with the preceding paragraphs.
3. The normalized color bars in Fig. 4 do not visualize problems which are solved by more than 1 policy... I would be interested to see unnormalized forms to see proportion of problems that are not uniquely solved...
Based on the reviewer's suggestion, we have revised Fig. 6 to show each adapter’s contribution to problems solved by exactly one, two, three, four, or five policies. The heights of the bars show the percentage of solved problems at each count. Colors show the contribution of each policy (e.g., if exactly m policies solve a problem, each is considered to have contributed equally).
4. I am curious about the bimodal pattern you observed in the Jaccard similarity plots. Do you think this indicates that, for certain tasks, fewer than 5 total policies are required to solve the task, leading to redundant policies?
Although the bimodal pattern in the Jaccard similarity heatmap for Knights-&-Knaves shows that some policies solve similar sets of problems, it does not necessarily imply that two policies are sufficient. Based on the reviewer's suggestion, to make this more concrete, we evaluate the sets of solved problems for every subset of the five policies on Knights-&-Knaves using greedy decoding, and report below the best subset's coverage (how many problems are solved) for each subset size, along with the gap w.r.t. using the full population:
Even the best pair of adapters (selected in hindsight) is severely suboptimal compared to using the full population.
We hope that our responses address your concerns and are helpful in improving your rating. If you have any other comments or feedback, please let us know! We are looking forward to hearing back from you! Thank you again for the review.