- Rating
- 1
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper studies the research problem of how to measure the marginal utility of LLM agent skills. The marginal utility of a skill is defined as a fixed package plus announcement protocol as it changes the expected verifier reward. At each checkpoint the paper replays one realised prefix, apply a protocol arm, independently continue it, and compare verifier rewards. Across 22 screened realised states from eleven SkillsBench tasks under one model and agent, the experiments show the calculated marginal utility score.
Critical strengths
- The marginal utility of a specific LLM agent skill seems interesting.
- The research reports comprehensive related works.
- The paper reports comprehensive experimental details.
Critical weaknesses
- The method used to measure such marginal utility is rather simple and follows the standard definition of marginal utility. It is unknown if this kind of simple empiricial study consistitues significant research progress that meets the publication standard at premier machine learning conferences. It may be more suitable for a workshop paper, rather than a conference paper.
- While the research topic is interesting, what this paper essentially does is to apply marginial utility measurements to LLM agent skills. The experiments simply report the calculated scores, without significant or insightful observations that can be helpful for LLM agent skill community.
- The experimental settings are not diverse enough using only one model and one benchmark.
- Uncertainty and flucation across different experiment runs and settings are not sufficiently regulated or measured. This brings concerns on the rebliability of the calculation method.
- The reliablity of the verifier is not sufficiently validated. Since the score is dependent on the quality of the verifier, this also brings concerns on the rebliability of the calculation method.
Questions for authors
- The paper essentially applies standard marginal utility measurements to LLM agent skills. Could the authors clarify what constitutes the significant research progress beyond this application?
- Given that the method follows the standard definition of marginal utility and the study is relatively simple empirically, could the authors explain why they believe this work meets the publication standard of a premier machine learning conference, rather than being more suitable for a workshop paper?
- The experiments use only one model and one benchmark. Could the authors explain whether this limited setting is sufficient to support the paper's claims?
- The paper does not sufficiently regulate or measure uncertainty and fluctuation across different experiment runs and settings. How do the authors address the resulting concerns about the reliability of the calculation method? Could they provide variance estimates or confidence intervals across multiple runs?
- How do the authors validate the verifier's reliability? How sensitive are the marginal utility scores to verifier errors or biases?
- How could the authors assure that the proposed calculation method produces reliable and reproducible results? What additional experiments or analyses could be provided to verify its reliability?