- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper provides a detailed and complete analysis for test-time answers/responses selection. They analyze what the failure is and how it may occur when a verifier selects from lots of generated responses. Then they propose a better metric or perspective to identify when and how selection failures occur. The extensive results can support what they claim.
Critical strengths
-
They find an interesting and important problem and analyze the failure mode/pattern. This may help to advance future related research and realistic applications of LLMs/agents.
-
The empirical results also support their claim in main text. Based on the results and their findings/conclusion, the performance of LLMs can be improved without substantial additional compute, by refining the selection process at test time.
Critical weaknesses
-
They provides a detailed analysis for selection failure pattern but did not offer a valid solution for this failure, which makes their contribution limited.
-
Also their experiments are only conducted on four small benchmarks. I know it's test-time. So it won't lead to much time and cost for experiments. They may conduct more extensive experiments to support their claims. Therefore, I am not convinced that the current contribution meets the bar for ICLR.
Questions for authors
N/A