acceptodds
Under review as a conference paper at ICLR 2027

The Novelty Trap: When Repetition Helps LLM Research

Abstract

Novelty is a common objective in automated research, but forbidding repetition can reduce the value of what agents discover. We demonstrate this novelty trap on the Koijen–Levy earnings-announcement benchmark. An LLM proposes economic hypotheses, a second model measures them from event summaries, and hypothesis sets fitted in one quarter are evaluated in two later quarters. At matched proposal budgets, independent restarts improve incremental R² by 23% and 19% relative to a sequential no-repeat prompt (0.63 and 0.57 percentage points). The advantage holds on all six seeds in each quarter, including evaluation of the same runs on historical labels withheld until the analysis protocol was fixed. A memory-preserving prompt that permits repetition also improves value on every seed. We characterise the exploration–measurement tradeoff with a model separating errors in how hypotheses represent constructs from errors in their annotation. Under its assumptions, an informative prior and noisy measurements create a region in which repeated formulations of strong ideas outperform broader coverage. The model also predicts exactly when additional annotation can reverse that ranking. Planted-truth experiments illustrate the value of exploration when prior information is weak. Explanatory value and discovery counts also diverge: counting significant coefficients in joint regressions reverses the policy ranking. The design implication is to broaden construct coverage while preserving opportunities to refine useful mechanisms, and to evaluate both by the joint information they add.

Primary area:
foundation or frontier models, including LLMs
Keywords:
research agents, automated scientific discovery, novelty search, measurement error, hypothesis generation, quantitative finance

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_sBGZOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper studies whether a no-repeat prompt reduces the value of hypothesis generated. On an earnings-announcement benchmark, independent restarts perform better than sequential generation with a no-repeat instruction. The paper also develops a measurement-error model to explain when repeated formulations can be more useful than broader coverage. At a high level this paper examines the classical exploration vs exploitation tradeoff.

Critical strengths
  • The main empirical finding is somewhat interesting. The authors provide robustness checks and withheld-label evaluation to improve validity of the results. Overall the results are reasonably supported for the experiment setting chosen.
  • The paper hints at some interesting ideas, and the distinction between repeated ideas and redundant information makes sense. As in, similar hypothesis may provide different measurements of the same mechanism. The paper further shows that explanatory value and significant hypothesis counts give different policy rankings. These observations could inspire future works.
Critical weaknesses
  • The main evidence comes from one chosen benchmark, and one proposer-annotator pair, while there are various discovery / hypothesis generation benchmarks out there (DiscoveryBench, HypoBench, to name a few). The comparisons also change memory or hisotry framing together with permission to repeat. Therefore, it's difficult to assess how much the results come from the repetition itself, rather than particular prompts and benchmark.
  • The theory and additional experiments do not provide a clear explanation of the main result. The prediction that re-annotation would reduct the restart advantage is not supported. Section 5 studies feedback and selection rather than directly testing the proposed repetition-coverage boundary. From section 3 to section 5, the presentation of the paper moves between these different questions without sufficiently explaining their connection. This gives readers a difficult time parsing the core arguments, and make it difficult to identify what general conclusion can be made beyond the quantitative results and benchmark-specific observation.
Questions for authors
  • What's the main contribution beyond the known tradeoff between exploring new ideas and refining promising ones (exploration vs exploitation)?
  • What properties of the chosen benchmark make repetition useful, and where would you expect the findings to generalize?
  • How would you interpret the prompt comparisons, given that they also seem to change memory or the framing of previous hypothesis?
  • Which theoretical conclusions are supported by the experiments, and which remain untested?
  • In general I would also recommend a re-structuring of the paper's presentation to better surface the key argument, what each experiment examines, what each conclusion supports, to improve the readability of the manuscript.
Reviewer_zL7cOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper argues that a no-repeat instruction can reduce the joint value of hypotheses discovered by LLM research agents (the "novelty trap"). On a real earnings-announcement benchmark, both independent restarts and a memory-preserving prompt that permits repetition outperform the no-repeat prompt out of time. The authors attribute this to an informative prior: under the no-repeat instruction, later proposals drift toward weaker mechanisms. Neither re-annotation nor score feedback closes the gap, and counting significant discoveries reverses the ranking of the policies. A two-layer reliability model then yields a novelty threshold and a phase diagram predicting when repetition beats coverage.

Critical strengths
  • Rigorous protocol. The analysis was frozen before evaluation, test labels were withheld, and prespecified and exploratory analyses are clearly separated, which is rare for LLM-agent papers.
  • Clear theory. The distinctions between semantic similarity and informational redundancy, and between re-annotation and re-proposal, are intuitive and stated under explicit assumptions.
Critical weaknesses
  • The paper never measures novelty, and the task is feature engineering rather than research. The paper never quantifies whether no-repeat actually yields more novel or diverse hypotheses. Appendix A.1 points the other way: about half of the restart advantage (0.330.33 of 0.630.63) remains when only one item per cluster is kept. This suggests the no-repeat prompt may simply degrade quality, rather than trade value for novelty. Moreover, the objective (incremental R2R^2) rewards repeated noisy measurements of a known strong signal. That is closer to LLM feature generation than to research, so the claims about "research agents" seem overly broad.

  • The theory is not turned into a usable method. None of the theoretical results yields a policy that beats simple restarts. The tested threshold policy relies on re-annotation, which Proposition 2 itself identifies as the weaker option. Demonstrating even a simple application of the theory would substantially strengthen the paper.

Questions for authors
  • The proposer and annotator are both GPT-5-family models, so their errors may be correlated and κ\kappa may be model-specific. Does the effect hold with other model families as proposer or annotator, even in a small run?

  • In 2026Q2, excluding guidance-related hypotheses reduces the restart advantage to 0.020.02 ([−0.43,0.48][-0.43, 0.48]), and the prompt lists guidance as its first example mechanism. Does the effect persist (a) if guidance is removed from the prompt or disallowed, and (b) on another dataset without a single dominant signal? The authors' own theory predicts a weaker trap under flat utility profiles (ui∝i−αu_i \propto i^{-\alpha} with α≤1/2\alpha \le 1/2).

Reviewer_G5fcOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

A common design choice in automated LLM research agents is encouraging novelty and avoiding the repetition of earlier ideas. Authors evaluate this design on the Koijen–Levy (2026) earnings-announcement benchmark where gpt-5 proposes economic hypotheses, gpt-5-mini scores them on event summaries, and linear models fitted in 2025Q4 are tested out-of-time in 2026Q1 and 2026Q2. They show that forbidding repetition induces a "novelty trap" that lowers the joint explanatory power of discovered hypothesis sets. Across all 6 paired seeds at matched budgets (6 rounds of 10 proposals), independent restarts outperform a sequential “no-repeat “ prompt by 0.63 percentage points of incremental R^2, while a memory-preserving control permitting repetition beats no-repeat on all 6 seeds by 0.47 percentage points in a specific quarter. To explain this tradeoff, the authors formulate a post hoc two-layer Spearman–Brown reliability model that separates annotation reliability from operationalisation reliability to derive a novelty threshold, a phase boundary between sampling with and without replacement, and a crossover annotation count.

Finally, the paper demonstrates that evaluation metrics can reverse policy rankings: while repetition boosts joint explanatory R^2, the no-repeat prompt yields more individually significant regression coefficients (1.67 and 1.50 per run vs. 0.33 and 0.50 for restarts) because repeated noisy measurements of the same construct share variance and dilute individual t-statistics. These findings are complemented by planted-truth synthetic experiments P1 to P3.

Critical strengths
  • The paper isolates a widespread default in automated research agents i.e. “avoiding repetition to improve novelty” and demonstrates with strict anti-contamination controls (pre-event model cutoffs, frozen protocols, and a sealed 2026Q2 test quarter) that both A) independent restarts and B) a memory-preserving "invite-repeat" prompt consistently outperform a sequential no-repeat prompt across all six paired seeds and extensive robustness checks. This is a clean, practical, and timely insight for the ICLR community

  • Within the same experiment, the authors reconcile conflicting findings in prior literature by showing that no-repeat yields more individually significant hypotheses (1.67 vs. 0.33 per run) while explaining less joint out-of-time return variance. These results in conjunction with additional analysis provide valuable methodological insights.

  • Strong theoretical contribution via the two-layer reliability model, which provides a principled mathematical explanation for why and when repetition is epistemically superior to novelty. Further, to test the two-layer Spearman–Brown reliability mathematical model, authors derive falsifiable predictions and directly confront the model with empirical data, transparently reporting where the model fails.

Critical weaknesses
  • The empirical validation is limited to a single domain (financial econometrics) and a single benchmark (Koijen-Levy), however the paper claims to demonstrate a general "novelty trap" in "LLM research". It is unclear to what extent proposed techniques generalizes to other scientific domains with different prior informativeness and measurement noise characteristics.

  • The paper attributes gains primarily to permitting repetition (restarts and invite-repeat), yet its own OPRO-style raw score-feedback arm which forbids repetition achieves 3.36 pp in 2026Q2, matching invite-repeat (3.37 pp) and approaches restarts (3.52 pp). Furthermore, the paper lacks comparisons against structured tree-search or evolutionary optimizers (e.g., LATS, OPRO) that actively balance exploration and refinement.

  • As acknowledged by the authors, there are potential data leakage and look-ahead bias concerns regarding the unverified provenance of the archived event summaries.

Questions for authors
  1. The Koijen–Levy benchmark is an asset-pricing task where a single well-known feature (management guidance) dominates and measurement noise comes from an LLM-as-judge rater. Why was this chosen over established multi-domain hypothesis-generation benchmarks that you cite (such as HypoBench or MLE-bench), and how would your conclusions hold in true scientific discovery tasks where the strongest mechanism is not already in the LLM's top prior?

  2. Given that raw score feedback largely closes the gap with restarts in 2026Q2 despite forbidding repetition, wouldn't more sophisticated exploration–exploitation algorithms (e.g., LATS or OPRO) mitigate the novelty trap without abandoning novelty search?

  3. To what extent could the apparent decline in later-round no-repeat utility reflect the gpt-5-mini annotator extracting familiar guidance more reliably than genuinely novel constructs and how would that affect the two-layer model's conclusions?

I am open to discussion and reconsideration based on authors addressing these questions/concerns and rescoping their work.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.