- Rating
- 1
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The key idea seems to be that LLMs are often evaluated based on whether they get the correct answer under some number of attempts. This can obfuscate performance differences, since, if a model is evaluated on k attempts, each model will have a distribution over the number of correct answers in its k attempts. Depending on how you choose to evaluate, the rankings of different models may change: some models may get one correct answer out of k more frequently, while others get higher percentages of their k attempts correct. As such, reporting the full distribution of successes over k attempts would be more informative about model behavior.
Critical strengths
None
Critical weaknesses
I have serious concerns with the presentation of this paper and found it very difficult to understand: the style is excessively verbose yet often lacking in crucial explanatory detail, and relatively simple concepts are obfuscated with breezy statements that are somehow both matter-of-fact and contentless. This isn't helped by the sudden notation and results dumps. I recommend that the authors do a full, human rewrite. As a reviewer, I struggled with how much effort I should put into deciphering this and ultimately decided that, regardless of the content, I would strongly recommend rejection based on the presentation alone.
Questions for authors
Did you use an LLM to generate the entire paper? If so, did you proofread the generation? Did you run any of the experiments yourselves?
2 comments
The issue is not that the appendix is long or that your paper has math, it's that the entire manuscript was written in Claudish. I'm not sure how the conference publication system should be adjusted to handle this kind of thing, but, in the current paradigm, relying on peer review to parse your slop for you is antisocial behavior. If you don't have time to write the paper (or clean up the AI writing), don't submit. I maintain my score.
Request for specific feedback
First and foremost, your criticisms of
the presentation of this paperanddifficulty to understandare well taken. We revised the paper so that it can now be read without a background in Bayesian methods or advanced statistics. Also, the appendices only provide supporting details on the proposed method, additional evaluations, and mathematical proofs. The main text is sufficient to communicate the idea behind TailPass and its utility without needing to go back and forth between the main text and the appendices. Please let us know if you find the revised version clearer and easier to follow.We understand that your decision, as you stated, is
based on the presentation aloneand that your recommendation holdsregardless of the content, but we would like to ask for more specific feedback: Did you find a claim or result that is incorrect, or were you unable to determine its correctness from the exposition?Questions
We used AI no more than what is disclosed in our AI use statement.
The proposed methods and all mathematical derivations are the authors' own work.There are 15 distinct experimental setups (7 empirical and 8 numerical; Section 4 and Appendices C, F-H, J-K) on 22 LLM configurations, as follows:
The experiments involved generating over 9.5 billion tokens and took us several months to complete. How could AI conduct such a variety of experiments to examine different aspects of evaluation? We think that if we could scaffold AI models to do so, that would be a contribution in its own right!
Please do let us know if you would like any details or explanations about any of the above experiments. Is there a particular result whose provenance you question? If so, please identify it so that we can address the concern directly.
Request
We understand that every reviewer is busy and that reviewing a long paper like ours takes more time, but you can skip the appendices and focus on the main text. In the main text, we discuss the problems with existing evaluation methods and how our proposed framework can address them. We welcome any criticism and will respond to questions about any technical aspect. But please provide specific criticisms and identify the weaknesses. If you wish to maintain your score, please at least clarify your specific concerns.