acceptodds
Under review as a conference paper at ICLR 2027

Priced Guidance: Can Language Models Generate Future Research Ideas?

Abstract

We evaluate language models' capability to generate novel research ideas through the lens of compression. Our goal is to lower-bound the potentially tiny probability that a language model generates the essence of a future research idea without any hints. Rather than estimate this probability through expensive repeated sampling, our Priced Guidance framework measures the compression cost: how many additional bits of information are needed to guide the model to recover the target idea. We prove that if the model can recover the target idea with at most bits of guidance in expectation, then it can generate the idea without any guidance with probability at least . In our framework, the language model, called the generator, can pose a sequence of multiple-choice questions, e.g., “what is the topic of the research idea?”, and specify a probability distribution over possible answers. A guide, which is a language model with access to the target idea, selects answers. If a selected answer has prior probability , the generator pays bits. The generator's goal is to minimize the cumulative cost it needs to pay to produce an idea that matches the essence of the target idea. This cumulative cost equals to, up to a constant, the number of bits of information sent by the guide. Using this methodology, we evaluate five generator models (Opus 5, Fable 5.1, GPT-6 Astra, GPT-5.6 Sol, and GLM 5.3) on the core ideas in 87 recent high-quality deep learning papers. We use a frontier model as a judge to determine whether the generated idea matches the target in terms of the central research object and defining mechanism. Fable 5.1 achieves the lowest median compression cost at 67.8 bits, substantially lower than gzip's median of 5,712 bits for losslessly compressing the summary of the target idea. A uniform ensemble of Fable 5.1, Opus 5, and GPT-6 Astra further reduces the median compression cost to 53.8 bits and improves the generation probability lower-bound by .

Primary area:
foundation or frontier models, including LLMs
Keywords:
idea generation, compression, creativity, language model

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_NRtrOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This work aims to measure a language model’s idea-generation capability through the lens of compression, specifically by quantifying how many additional bits are required to guide the model toward the core idea of a target future paper. To this end, the authors propose Priced Guidance, a framework for quantifying the compression cost of ideas. In this framework, a language model, referred to as the generator, generates a predefined idea step by step. At each step, it produces a question and a probability distribution over a finite set of candidate answers. Another language model, referred to as the guide, which has access to the predefined idea, selects an answer. The cost of each step is defined as (-\log_2 p) bits, where (p) is the probability assigned to the selected answer by the generator. The costs across all steps are then aggregated to obtain the overall compression cost.

Critical strengths
  1. The proposed Priced Guidance framework is an interesting approach to quantifying the compression cost of ideas.
  2. The experiment cover several mainstream LLMs.
  3. Overall, the paper is easy to follow.
Critical weaknesses
  1. The research idea is interesting, particularly the goal of measuring a language model’s idea-generation capability. However, the claims appear to be overstated. The proposed framework seems to primarily evaluate how efficiently a language model can be guided toward generating a predefined text, which is not necessarily equivalent to measuring its ability to generate novel and meaningful research ideas. The authors should clarify the scope of the proposed framework and moderate their claims accordingly.
  2. The practical value of this research direction needs to be better established. How could the proposed framework advance research in this community? For example, could the evaluation results provide actionable insights for improving LLMs’ research idea-generation capabilities? The authors should clarify the potential applications of the framework and explain how it could contribute to future LLM development.
  3. The paper does not appear to fully address the question posed by its title, which seems somewhat overstated. Rather than directly investigating whether language models can generate research ideas, the work proposes a framework for quantifying the compression cost of guiding a model toward a predefined idea. The authors should clarify the relationship between the proposed metric and the actual capability of generating research ideas, and ensure that the title and claims accurately reflect the scope of the work.
  4. The motivations behind several core design choices remain unclear. For example, the formulation of the bid and the use of a logarithmic function to calculate the compression cost require stronger justification. The authors should explain the theoretical foundations of these design choices and clarify why they are appropriate for quantifying the difficulty of guiding a model toward a target idea.
  5. There are concerns regarding the experimental settings. The framework employs LLMs as both the generator and the guide, without clear ground-truth labels or an independent evaluation mechanism. This setup makes it difficult to determine whether the measured compression costs reliably reflect the practical quality of idea generation. The authors should consider incorporating human-guided evaluation or other independent assessment methods to validate the proposed framework and its results.
  6. Although the authors claim to investigate LLMs’ ability to generate research ideas, the framework appears to capture only limited aspects of the challenges specific to research idea generation. For example, it is unclear how the proposed approach accounts for the novelty, feasibility, significance, and scientific soundness of a research idea. In its current form, the task resembles a word-guessing game in which one model guides another toward a predefined target. The authors should better characterize the unique challenges of research idea generation and demonstrate how the proposed framework captures these challenges beyond general target-guessing or text-reconstruction tasks.
Questions for authors

Please refer to the above content.

3 comments
AuthorsOpenReview ↗

Response to Reviewer NRtr

We thank the reviewer for the review.

W1 / W3 / W6. Research idea recovery, title and evaluation scope

We respectfully disagree. The generator aims to recreate a research idea rather than the exact text of its summary. This distinction is evident from the compression cost of the summary text, which is ten times higher. The title reflects the research question we investigate: whether models can generate ideas that later appear in published research. Our framework addresses this question by measuring how much guidance a model needs to recover those ideas. Nevertheless, we agree that proposing ideas that are subsequently published captures only one aspect of evaluating idea generation for novelty and value, and we have discussed this in the limitations section.

W2. Practical value and model improvement

The framework provides a graded metric for tracking model development. For example, GLM 5.3 reduces median Directional cost from 100.6 to 53.4 bits compared with GLM 5.2. On a more recent 43-paper subset, Opus 5.5 reduces the 80th-percentile Essence cost from 141.33 to 107.72 bits compared with Opus 5. These comparisons reveal which aspects of recovery improve across model versions. Information cost could also potentially provide a training signal for reinforcement learning, while successful recovery trajectories could supply synthetic training examples. Finally, priced guidance could extend to other domains with defined targets and verifiable success criteria, such as program synthesis. These training applications and extensions remain directions for future work.

W4. Why bids and logarithmic costs?

A bid specifies the next question, the available answers, and their probabilities before the guide selects an answer. This makes the information supplied by the guide explicit and priced, while allowing the generator to choose which uncertainty to resolve. It also defines the corresponding unguided process, which samples those same answers without access to the target.

The logarithmic cost is motivated theoretically. It follows from coding and sampling the same answer sequence. Under fixed generator randomness, the probability of that sequence is the product of its conditional action probabilities. Taking minus log base two makes this product additive in bits. Theorems 2.1 and 2.2 formalize the connection to description length and unguided acceptance, including the terminal judge cost; it is not an arbitrary scoring transform.

W5. Independent human assessment of the evaluation.

Thank you for suggesting human validation. We have now completed an evaluation with four researchers, who were blinded to the automated judge’s labels. Each researcher evaluated 10 ideas, giving 40 assessments across 20 ideas, with two researchers per idea. They assessed Directional match—alignment in the core research problem and contribution type—and the stricter Essence match, which additionally requires agreement in the core method or finding and its defining mechanism.

Among the 14 ideas labeled as Essence matches by the automated judge, 13 (92.9%) received a positive Directional judgment from at least one researcher, and 9 (64.3%) received a positive Essence judgment from at least one researcher. These results provide human support for recovery of the target research direction in this evaluated subset, with a majority of these ideas also receiving at least one positive Essence judgment.

The qualitative feedback adds useful detail: some proposals captured the overall approach while omitting particular mechanisms. We will include the full annotation results and illustrative examples in the revision, distinguishing directional alignment from the stricter requirement of recovering the core mechanism.

Reviewer_NRtrOpenReview ↗

Reply to the rebuttal

Thanks authors for their responses. It is highly suggested to add the detailed human assessment in the revision of this work. My major concerns still retain, such as the overstated novelty, limited practical value and lacking strong motivation for techniques. Thus, I tend to maintain the scores in this time. I will join the discussion with other reviewers before making the final decision.

Thank you for the prompt reply. We have added the detailed human evaluation to the revised paper. While we believe our compression-motivated perspective offers a novel way to evaluate open-ended generation tasks, including idea generation, we appreciate your constructive feedback.

Reviewer_3EzgOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This work provides a compression benchmark for research idea generation. A generator is scored by its compression cost on target ideas from future papers. There are many promising ways to improve the compression cost. For example, enabling experiments on real data and web search over a fixed snapshot of the internet could further reduce compression cost. Beyond idea generation, Priced Guidance provides a tool to quantify the probability of tail events for a family of agent scaffolds as long as the scaffold makes bids.

Critical strengths

This paper evaluates language models’ capability to generate novel research ideas through the lens of compression

Prove that if the model can recover the target idea with at most K bits of guidance in expectation

Critical weaknesses

The proposed lower bound is mathematically valid but extremely loose. Even the best reported compression costs correspond to negligible probabilities of unguided idea generation. Without empirical validation of how well compression cost predicts actual generation success, the practical value of this metric remains unclear.

The compression cost depends heavily on the predefined question strategies, answer options, and recovery mechanisms. It is unclear whether the reported results reflect the underlying models' idea generation capabilities or their performance within this particular scaffold.

Questions for authors

The paper evaluates GLM 5.2 and GLM 5.3 but does not cite their original papers or technical reports. Proper references should be provided for the evaluated models. Please check the paper for all citations

The evaluation relies on an LLM judge to determine whether a generated idea matches the essence of a future paper, without sufficient human expert validation. Since this judgment directly determines success and compression cost, evaluation errors could substantially affect the conclusions.

1 comment
AuthorsOpenReview ↗

Response to Reviewer 3Ezg

We thank the reviewer for the review.

W1. The lower bound is loose, and its relationship to actual generation success lacks empirical validation.

We agree that the lower bound is not necessarily tight. However, a bound that is not tight can still be useful for comparing models and target papers, just as log perplexity is useful for comparing language models without directly measuring their probability of generating good text.

W2. Do the results reflect model capabilities or performance within this particular scaffold?

Our results measure model capabilities within the specified scaffold. However, evaluating models with a harness is standard practice in agentic benchmarks such as ProgramBench. The scaffold is fixed across models, so the rankings compare model capabilities under a common generation protocol. Additionally, removing field-label charges leaves rankings unchanged at both P50 and P80 under both judging criteria.

Directional — Field-label charges removed

ModelOriginal P50Field given P50Original P80Field given P80
Fable 5.152.2240.1072.0461.68
Opus 547.4335.6186.1374.47
Astra57.6046.2793.8481.67
Sol68.7057.45107.4796.79
GLM 5.359.8948.44108.2598.74

Essence — Field-label charges removed

ModelOriginal P50Field given P50Original P80Field given P80
Fable 5.167.8256.65102.0891.87
Opus 568.2856.71138.87127.43
Astra69.4558.19108.7896.82
Sol92.3980.51139.51127.85
GLM 5.3100.2388.89∞∞

Q1. Missing model references

Thank you for pointing this out. We have added the official model references for GLM 5.2 and GLM 5.3, together with references for the other evaluated generators.

Q2. Human expert validation of the idea-equivalence judge.

Thank you for suggesting human validation. We have now completed an evaluation with four researchers, who were blinded to the automated judge’s labels. Each researcher evaluated 10 ideas, giving 40 assessments across 20 ideas, with two researchers per idea. They assessed Directional match—alignment in the core research problem and contribution type—and the stricter Essence match, which additionally requires agreement in the core method or finding and its defining mechanism.

Among the 14 ideas labeled as Essence matches by the automated judge, 13 (92.9%) received a positive Directional judgment from at least one researcher, and 9 (64.3%) received a positive Essence judgment from at least one researcher. These results provide human support for recovery of the target research direction in this evaluated subset, with a majority of these ideas also receiving at least one positive Essence judgment.

The qualitative feedback adds useful detail: some proposals captured the overall approach while omitting particular mechanisms. We will include the full annotation results and illustrative examples in the revision, distinguishing directional alignment from the stricter requirement of recovering the core mechanism.

Reviewer_UFFzOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces Priced Guidance, a framework that models scientific idea generation as twenty-questions-style compression. A generator LLM outputs multiple-choice bids with verbalized probabilities, a knowledgeable guide selects branches, and the cumulative -log2 p cost bounds the unguided generation probability under Theorem 2.2. Experiments on 87 Hugging Face papers show median compression costs of 67.8 bits for Fable 5.1 and 53.8 bits for a three-model ensemble.

Critical strengths
  1. Principled formulation connecting interactive communication cost with an evidence lower bound (ELBO) on rare generation events.
  2. The theoretical proofs rigorously account for recovery backtracking, ensembling, and stochastic multi-round evaluation via an indexed continuation construction.
  3. Detailed reporting of prompt trajectories, failure modes, and run-to-run variance across multiple frontier models.
Critical weaknesses
  1. Scope mismatch of the theoretical guarantee. Theorem 2.2 bounds an artificial system—an unguided scaffold where an external sampler draws strictly from the generator's verbalized distribution pt(a|τt). It does not bound the base model's intrinsic ability to generate complete research ideas end-to-end without the interactive question-answer scaffold.

  2. Sensitivity to verbalized probability calibration and scaffold design. Compression cost directly penalizes uncalibrated verbal probabilities; a model with strong domain knowledge but conservative calibration receives an artificially high bit cost, yet no calibration audit is provided. Furthermore, the bit scale is heavily scaffold-dependent: the synthetic 3,035-node taxonomy adds a fixed ~11.6 bits of overhead, and omitting the "other" branch prevents valid routing for out-of-taxonomy papers. In addition, comparing lossy semantic idea compression (67.8 bits) against lossless verbatim gzip (5,712 bits) in the abstract evaluates incompatible objectives.

  3. Fully LLM-dependent evaluation pipeline without human validation. Gold summaries (Opus 5), the taxonomy (GPT-5.6 Sol), memorization filters (GPT-5.6 Terra), and equivalence judgments (GPT-5.5, n=3) contain zero human ground-truth validation. In addition, using Fable 5 to guide Fable 5.1 leaves potential intra-family preference uncontrolled.

Questions for authors
  1. What is the human agreement rate on the Directional and Essence judge decisions across a representative subset of test papers?
  2. How sensitive are the resulting compression costs to the depth and topology of the GPT-generated taxonomy?
  3. How does the scaffold route papers that fall outside the taxonomy leaves given that the "other" branch is omitted?
  4. Were the ensemble members and weights selected on development-set or test-set performance?
4 comments
AuthorsOpenReview ↗

Response to Reviewer UFFz (1/2)

We thank the reviewer for the review.

W1.1. Theorem 2.2 bounds an unguided scaffold, not the base model’s intrinsic idea-generation ability.

We agree that Theorem 2.2 bounds the success probability of the base LLM with its scaffold. However, evaluating models with a harness is standard practice in agentic benchmarks such as ProgramBench (Yang et al., 2026). In our evaluation, the scaffold is fixed across models, so the rankings compare model capabilities under a common generation protocol.

W1.2. The lower bound is not necessarily tight.

We agree that the lower bound is not necessarily tight. However, a bound that is not tight can still be useful for comparing models and target papers, just as log perplexity is useful for comparing language models without directly measuring their probability of generating good text.

W2.1. Compression cost penalizes uncalibrated verbal probabilities, yet no calibration audit is provided.

The declared probabilities define the answer-sampling distribution in the unguided scaffold, so the theorem does not require them to be calibrated estimates of an external correctness event. However, probability assignment affects compression cost, and the score does not isolate domain knowledge from this ability. To test sensitivity, we applied shared temperature scaling (T=0.5–2) to the declared question probabilities on fixed trajectories. Rankings were largely preserved, with occasional reversals between closely ranked models such as Opus and Fable. The original temperature was within one bit of the best tested P50 cost for each model under both criteria. This is a sensitivity analysis, rather than an empirical calibration audit.

Directional — P50

ModelT=0.5T=0.75T=1 (original)T=1.25T=1.5T=2
Fable 5.159.5552.8352.2252.8553.3053.40
Opus 554.4749.0547.4346.9646.8146.68
Astra66.2857.8857.6058.5359.3159.14
Sol80.3271.8768.7067.8569.1572.26
GLM 5.367.6362.8659.8960.8162.0862.97

Directional — P80

ModelT=0.5T=0.75T=1 (original)T=1.25T=1.5T=2
Fable 5.180.8074.7272.0473.0873.4075.13
Opus 582.5483.3186.1388.7190.7493.55
Astra97.1290.9393.8494.0496.06100.30
Sol122.10106.83107.47109.11111.39114.49
GLM 5.3120.70109.65108.25109.63113.05117.85

Essence — P50

ModelT=0.5T=0.75T=1 (original)T=1.25T=1.5T=2
Fable 5.178.4568.8467.8267.3767.6668.30
Opus 576.2469.4168.2867.5868.0568.91
Astra80.9372.6469.4569.2170.2271.87
Sol104.9096.8292.3991.5492.6792.79
GLM 5.3112.54102.80100.2399.2499.59100.71

Essence — P80

ModelT=0.5T=0.75T=1 (original)T=1.25T=1.5T=2
Fable 5.1114.70102.15102.08103.56104.95108.87
Opus 5144.03134.48138.87144.54148.76154.45
Astra119.35111.72108.78111.39112.66117.22
Sol140.85133.29139.51140.10141.44143.99
GLM 5.3∞∞∞∞∞∞
AuthorsOpenReview ↗

Response to Reviewer UFFz (2/2)

W2.2 / Q2. “The synthetic 3,035-node taxonomy adds a fixed ~11.6 bits of overhead.” How sensitive are the resulting compression costs to the depth and topology of the GPT-generated taxonomy?

In our paper, 11.61 bits is the average taxonomy cost for Opus 5 on the 74 test papers with successful Essence reconstructions, not a fixed charge. It accounts for only 16.6% of the average total Essence cost (69.87 bits) on this subset. To further test whether taxonomy charges drive model comparisons, we recomputed information costs after removing the field-label charges, treating the field label as given while keeping the recorded trajectories fixed. Rankings remained unchanged at both P50 and P80 under both judging criteria.

Directional — Field-label charges removed

ModelOriginal P50Field given P50Original P80Field given P80
Fable 5.152.2240.1072.0461.68
Opus 547.4335.6186.1374.47
Astra57.6046.2793.8481.67
Sol68.7057.45107.4796.79
GLM 5.359.8948.44108.2598.74

Essence — Field-label charges removed

ModelOriginal P50Field given P50Original P80Field given P80
Fable 5.167.8256.65102.0891.87
Opus 568.2856.71138.87127.43
Astra69.4558.19108.7896.82
Sol92.3980.51139.51127.85
GLM 5.3100.2388.89∞∞

W2.3 / Q3. How does the scaffold route papers that fall outside the taxonomy leaves given that the “other” branch is omitted?

Every taxonomy node offers a STOP option. If none of its children fits the target paper, the guide can stop at the current broader field. The generator can then request further information through subsequent questions, without requiring the paper to match a predefined leaf.

W2.4. Comparing lossy semantic idea compression against lossless verbatim gzip evaluates incompatible objectives.

Here, gzip is provided only as a reference and a naive baseline: it guarantees 100% verbatim recovery of the target summary, satisfying both judging criteria by construction. This comparison illustrates that an idea’s compression cost can be substantially lower than the text-encoding cost implied by the perplexity of its description.

W3.1 / Q1. Human validation of Directional and Essence judge decisions.

Thank you for suggesting human validation. We have now completed an evaluation with four AI researchers who have published in top-tier journals. The researchers were blinded to the automated judge’s labels. Each researcher evaluated 10 ideas, giving 40 assessments across 20 ideas, with two researchers per idea. They assessed Directional match—alignment in the core research problem and contribution type—and the stricter Essence match, which additionally requires agreement in the core method or finding and its defining mechanism.

Among the 14 ideas labeled as Essence matches by the automated judge, 13 (92.9%) received a positive Directional judgment from at least one researcher, and 9 (64.3%) received a positive Essence judgment from at least one researcher. These results provide human support for recovery of the target research direction in this evaluated subset, with a majority of these ideas also receiving at least one positive Essence judgment.

The qualitative feedback adds useful detail: some proposals captured the overall approach while omitting particular mechanisms. We will include the full annotation results and illustrative examples in the revision, distinguishing directional alignment from the stricter requirement of recovering the core mechanism.

W3.2. Using Fable 5 to guide Fable 5.1 leaves potential intra-family preference uncontrolled.

Our guide ablation compares a guide from the same family (Opus 5) with one from a different family (Fable 5) for the Opus 5 generator. The median Directional cost is nearly identical: 44.9 versus 45.0 bits on the development set (Figure 5b). This is evidence about the Opus setting; it does not establish the absence of family bias throughout the pipeline.

Q4. Were ensemble members and weights selected on development-set or test-set performance?

The ensemble members—Fable 5.1, Opus 5, and Astra—were selected based on performance on the validation set. We used uniform weights (1/3 each), as specified in Appendix A.2.

Thank you to the authors for their thoughtful and detailed response. I believe the rebuttal has addressed many of my previous concerns, and the additional sensitivity analyses and human evaluations help strengthen the credibility of the results.

However, I still have some reservations about using compression cost as a comparative metric for models' research idea-generation capabilities.

In W1.2, the authors argue that a lower bound can still be useful for comparison even when it is not tight, drawing an analogy to perplexity (PPL). I agree that a proxy metric does not necessarily need to estimate the quantity of interest precisely to be useful. However, I think there is an important distinction worth discussing.

For a fixed text sequence, PPL corresponds to its exact negative log-likelihood (up to normalization). It becomes a proxy requiring additional validation only when we use it to assess higher-level properties, such as text coherence, quality, or the overall probability of generating a particular semantic content.

Similarly, I acknowledge that the compression cost in this paper has a well-defined mathematical interpretation under a specific scaffold and guide policy. However, when using it to compare the probabilities of independently generating different research ideas, or to compare different models, the tightness of the lower bound becomes important.

Specifically, a research idea may be generated through many distinct successful trajectories. Even if the guide identifies a successful trajectory, its probability may not be representative of the total probability mass of all successful trajectories. The extent of this gap may vary across models and target ideas. Consequently, even when two targets have a clear ordering in their true generation probabilities, their lower bounds obtained through Priced Guidance could potentially suggest the opposite ordering.

I appreciate the additional temperature-scaling experiments and taxonomy-cost analysis, which provide useful evidence of the metric's robustness. However, these experiments primarily examine sensitivity to specific aspects of the evaluation protocol. They do not fully establish whether differences in bound tightness could affect comparisons across models or research ideas.

I would therefore encourage the authors to discuss the interpretation and limitations of this metric more carefully in a revised manuscript. In particular, it would be helpful to clarify in what sense the metric measures guided reconstruction efficiency, and under what conditions it can serve as a reliable comparative measure of unguided idea-generation capability.

Despite these remaining technical concerns, I believe the paper addresses an important research question, presents a valuable idea, and makes a timely contribution to the community's ongoing discussion of LLMs' scientific research capabilities.

Given the authors' careful responses to many of my concerns and the valuable additional analyses provided in the rebuttal, I have decided to raise my rating from 2 to 3.

AuthorsOpenReview ↗

Response to Reviewer UFFz: bound tightness

Thank you for the prompt reply. Our ensembling operation combines lower-bound contributions from multiple generators or independent runs, allowing more than one successful trajectory to contribute. However, we agree that this need not capture all successful trajectories, and differences in bound tightness can affect rankings across models and ideas. We have updated the limitations section to clarify that compression cost measures guided reconstruction efficiency and does not guarantee the same ordering of unguided success probabilities.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.