- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This work aims to measure a language model’s idea-generation capability through the lens of compression, specifically by quantifying how many additional bits are required to guide the model toward the core idea of a target future paper. To this end, the authors propose Priced Guidance, a framework for quantifying the compression cost of ideas. In this framework, a language model, referred to as the generator, generates a predefined idea step by step. At each step, it produces a question and a probability distribution over a finite set of candidate answers. Another language model, referred to as the guide, which has access to the predefined idea, selects an answer. The cost of each step is defined as (-\log_2 p) bits, where (p) is the probability assigned to the selected answer by the generator. The costs across all steps are then aggregated to obtain the overall compression cost.
Critical strengths
- The proposed Priced Guidance framework is an interesting approach to quantifying the compression cost of ideas.
- The experiment cover several mainstream LLMs.
- Overall, the paper is easy to follow.
Critical weaknesses
- The research idea is interesting, particularly the goal of measuring a language model’s idea-generation capability. However, the claims appear to be overstated. The proposed framework seems to primarily evaluate how efficiently a language model can be guided toward generating a predefined text, which is not necessarily equivalent to measuring its ability to generate novel and meaningful research ideas. The authors should clarify the scope of the proposed framework and moderate their claims accordingly.
- The practical value of this research direction needs to be better established. How could the proposed framework advance research in this community? For example, could the evaluation results provide actionable insights for improving LLMs’ research idea-generation capabilities? The authors should clarify the potential applications of the framework and explain how it could contribute to future LLM development.
- The paper does not appear to fully address the question posed by its title, which seems somewhat overstated. Rather than directly investigating whether language models can generate research ideas, the work proposes a framework for quantifying the compression cost of guiding a model toward a predefined idea. The authors should clarify the relationship between the proposed metric and the actual capability of generating research ideas, and ensure that the title and claims accurately reflect the scope of the work.
- The motivations behind several core design choices remain unclear. For example, the formulation of the bid and the use of a logarithmic function to calculate the compression cost require stronger justification. The authors should explain the theoretical foundations of these design choices and clarify why they are appropriate for quantifying the difficulty of guiding a model toward a target idea.
- There are concerns regarding the experimental settings. The framework employs LLMs as both the generator and the guide, without clear ground-truth labels or an independent evaluation mechanism. This setup makes it difficult to determine whether the measured compression costs reliably reflect the practical quality of idea generation. The authors should consider incorporating human-guided evaluation or other independent assessment methods to validate the proposed framework and its results.
- Although the authors claim to investigate LLMs’ ability to generate research ideas, the framework appears to capture only limited aspects of the challenges specific to research idea generation. For example, it is unclear how the proposed approach accounts for the novelty, feasibility, significance, and scientific soundness of a research idea. In its current form, the task resembles a word-guessing game in which one model guides another toward a predefined target. The authors should better characterize the unique challenges of research idea generation and demonstrate how the proposed framework captures these challenges beyond general target-guessing or text-reconstruction tasks.
Questions for authors
Please refer to the above content.
3 comments
Reply to the rebuttal
Thanks authors for their responses. It is highly suggested to add the detailed human assessment in the revision of this work. My major concerns still retain, such as the overstated novelty, limited practical value and lacking strong motivation for techniques. Thus, I tend to maintain the scores in this time. I will join the discussion with other reviewers before making the final decision.
Thank you for the prompt reply. We have added the detailed human evaluation to the revised paper. While we believe our compression-motivated perspective offers a novel way to evaluate open-ended generation tasks, including idea generation, we appreciate your constructive feedback.
Response to Reviewer NRtr
We thank the reviewer for the review.
W1 / W3 / W6. Research idea recovery, title and evaluation scope
We respectfully disagree. The generator aims to recreate a research idea rather than the exact text of its summary. This distinction is evident from the compression cost of the summary text, which is ten times higher. The title reflects the research question we investigate: whether models can generate ideas that later appear in published research. Our framework addresses this question by measuring how much guidance a model needs to recover those ideas. Nevertheless, we agree that proposing ideas that are subsequently published captures only one aspect of evaluating idea generation for novelty and value, and we have discussed this in the limitations section.
W2. Practical value and model improvement
The framework provides a graded metric for tracking model development. For example, GLM 5.3 reduces median Directional cost from 100.6 to 53.4 bits compared with GLM 5.2. On a more recent 43-paper subset, Opus 5.5 reduces the 80th-percentile Essence cost from 141.33 to 107.72 bits compared with Opus 5. These comparisons reveal which aspects of recovery improve across model versions. Information cost could also potentially provide a training signal for reinforcement learning, while successful recovery trajectories could supply synthetic training examples. Finally, priced guidance could extend to other domains with defined targets and verifiable success criteria, such as program synthesis. These training applications and extensions remain directions for future work.
W4. Why bids and logarithmic costs?
A bid specifies the next question, the available answers, and their probabilities before the guide selects an answer. This makes the information supplied by the guide explicit and priced, while allowing the generator to choose which uncertainty to resolve. It also defines the corresponding unguided process, which samples those same answers without access to the target.
The logarithmic cost is motivated theoretically. It follows from coding and sampling the same answer sequence. Under fixed generator randomness, the probability of that sequence is the product of its conditional action probabilities. Taking minus log base two makes this product additive in bits. Theorems 2.1 and 2.2 formalize the connection to description length and unguided acceptance, including the terminal judge cost; it is not an arbitrary scoring transform.
W5. Independent human assessment of the evaluation.
Thank you for suggesting human validation. We have now completed an evaluation with four researchers, who were blinded to the automated judge’s labels. Each researcher evaluated 10 ideas, giving 40 assessments across 20 ideas, with two researchers per idea. They assessed Directional match—alignment in the core research problem and contribution type—and the stricter Essence match, which additionally requires agreement in the core method or finding and its defining mechanism.
Among the 14 ideas labeled as Essence matches by the automated judge, 13 (92.9%) received a positive Directional judgment from at least one researcher, and 9 (64.3%) received a positive Essence judgment from at least one researcher. These results provide human support for recovery of the target research direction in this evaluated subset, with a majority of these ideas also receiving at least one positive Essence judgment.
The qualitative feedback adds useful detail: some proposals captured the overall approach while omitting particular mechanisms. We will include the full annotation results and illustrative examples in the revision, distinguishing directional alignment from the stricter requirement of recovering the core mechanism.