- Rating
- 1
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper studies whether a no-repeat prompt reduces the value of hypothesis generated. On an earnings-announcement benchmark, independent restarts perform better than sequential generation with a no-repeat instruction. The paper also develops a measurement-error model to explain when repeated formulations can be more useful than broader coverage. At a high level this paper examines the classical exploration vs exploitation tradeoff.
Critical strengths
- The main empirical finding is somewhat interesting. The authors provide robustness checks and withheld-label evaluation to improve validity of the results. Overall the results are reasonably supported for the experiment setting chosen.
- The paper hints at some interesting ideas, and the distinction between repeated ideas and redundant information makes sense. As in, similar hypothesis may provide different measurements of the same mechanism. The paper further shows that explanatory value and significant hypothesis counts give different policy rankings. These observations could inspire future works.
Critical weaknesses
- The main evidence comes from one chosen benchmark, and one proposer-annotator pair, while there are various discovery / hypothesis generation benchmarks out there (DiscoveryBench, HypoBench, to name a few). The comparisons also change memory or hisotry framing together with permission to repeat. Therefore, it's difficult to assess how much the results come from the repetition itself, rather than particular prompts and benchmark.
- The theory and additional experiments do not provide a clear explanation of the main result. The prediction that re-annotation would reduct the restart advantage is not supported. Section 5 studies feedback and selection rather than directly testing the proposed repetition-coverage boundary. From section 3 to section 5, the presentation of the paper moves between these different questions without sufficiently explaining their connection. This gives readers a difficult time parsing the core arguments, and make it difficult to identify what general conclusion can be made beyond the quantitative results and benchmark-specific observation.
Questions for authors
- What's the main contribution beyond the known tradeoff between exploring new ideas and refining promising ones (exploration vs exploitation)?
- What properties of the chosen benchmark make repetition useful, and where would you expect the findings to generalize?
- How would you interpret the prompt comparisons, given that they also seem to change memory or the framing of previous hypothesis?
- Which theoretical conclusions are supported by the experiments, and which remain untested?
- In general I would also recommend a re-structuring of the paper's presentation to better surface the key argument, what each experiment examines, what each conclusion supports, to improve the readability of the manuscript.