acceptodds
Under review as a conference paper at ICLR 2027

IdeaLens: Detecting Idea Provenance In Long-Form Writing

23 views

Abstract

While modern AI detectors identify **_who wrote the words_**, emerging policies on AI use increasingly hinge on a different question: **_who came up with the ideas?_** We introduce _IdeaLens_, a detector that identifies whether a document's ideas came from a human or AI (_idea provenance_), regardless of who wrote its words. To focus IdeaLens on ideas rather than prose, we represent documents as outlines: lists of items that each pair a discourse role with a brief, paraphrased description of the content, minimizing word-level overlap with the raw text. We train IdeaLens on 1M FineWeb documents with silver labels from Pangram, a prose provenance detector. Since the outlines are largely stripped of surface-level information, the labels must be fit mainly through the ideas. In a controlled study, IdeaLens’s AI flag rate drops from 95% to 7% as models write from increasingly detailed human plans, while Pangram 4 still flags 92%; from AI-derived plans, IdeaLens stays above 96%. Conversely, on a new dataset of 50 stories that human authors wrote from AI-generated plans, IdeaLens flags 68% of the stories as AI, compared to 8% for Pangram 4. On a comprehensive suite of 19 existing detection benchmarks, we show that IdeaLens maintains strong detection rates at low false positive rates, suggesting that ideas themselves provide a powerful discriminative signal, and its performance holds across domains, formats, and languages. Finally, we examine 90K predictions from IdeaLens to characterize systematic differences between human and AI ideation. We release our models and labeled datasets to facilitate future research on idea provenance detection.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
idea provenance, ai detection, human-ai collaboration, discourse representation, long-form writing, large language models

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_PxmJOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents IdeaLens. It predicts whether a document’s ideas came from an AI regardless of who wrote the words. Documents are converted to outlines of discourse roles with content descriptions that are paraphrased for training and the model is trained on Pangram’s prose-level labels while ProseLens is trained on raw text as a control. When idea and prose provenance differ the two models disagree and IdeaLens remains competitive on standard detection.

Critical strengths
  1. Only the input changes so the same labels teach a different thing. ProseLens is an appropriate control. In IdeaShift the flag rate of IdeaLens falls as the human plan becomes more detailed while ProseLens and Pangram remain high.
  2. The evaluation is broad and the limitations are clearly acknowledged.
Critical weaknesses
  1. The appendix itself reports that a prose detector still separates paraphrased outlines well above chance and that renaming characters changes the scores. Detection also drops sharply on newer generators.
  2. The key quadrant rests on a small set of stories from one format (creative writing) and one generator. CoCoDet from ACL 2026 is evaluated on the same benchmark and is not compared.
Questions for authors
  1. What is IdeaLens’s accuracy on the documents that Pangram 4 misclassifies when given only the paraphrased outline (Table 9)? What is the TwiceTold flag rate after proper nouns are anonymized?
  2. How does IdeaLens compare with CoCoDet on CoCoNUTS in the binary setting?
Reviewer_z9RLOpenReview ↗
Rating
4
Confidence
4
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

Authors introduce IdeaLens, a detector that identifies whether a document's ideas came from a human or AI (idea provenance), regardless of who wrote its words.

To focus IdeaLens on ideas rather than prose, authors represent documents as outlines which is basically lists of items that each pair a discourse role with a brief, paraphrased description of the content, minimizing word-level overlap with the raw text.

They train IdeaLens on 1M FineWeb documents with silver labels from Pangram, a prose provenance detector.

In a controlled study, IdeaLens’s AI flag rate drops from 95% to 7% as models write from increasingly detailed human plans, while Pangram 4 still flags 92%; from AI-derived plans, IdeaLens stays above 96%.

Conversely, on a new dataset of 50 stories that human authors wrote from AI-generated plans, IdeaLens flags 68% of the stories as AI, compared to 8% for Pangram 4 showing the difficulty.

Finally, authors examine 90K predictions from IdeaLens to characterize systematic differences between human and AI ideation.

Critical strengths
  1. Its a good ambitious problem
  2. Idea is neat to disentagle idealens , proselens
  3. cool way to create outline as a list of items with discourse role
  4. a robust detector trained with distillation signal from pangram which is open sourced. I can see huge impact
  5. Good analyis of how human idea differ from AI idea
Critical weaknesses
  1. No such weakness. Would love to see performance on different domains and a more ambitious benchmark for IdeaLens
Questions for authors

NA

Reviewer_T5sdOpenReview ↗
Rating
1(−1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
1(−1)
Read review
Paper summary

This paper proposes IdeaLens, an AI ideas detector that aims to identify whether the underlying ideas of a document came from a human or an AI, different from most related work that focuses on the language or the prose. Specifically, the authors represent documents as LLM-extracted and paraphrased outlines, and trained classifiers on 1M web documents using silver prose labels from Pangram. They evaluate on both shared and mixed human-AI writing scenarios across 22 benchmarks, including two new diagnostic datasets (IdeaShift and TwiceTold) to test how well the detector tracks idea provenance when wording and planning come from different sources.

Critical strengths
  • The framing of the research problem is timely and well-motivated (especially strong intro!)
  • The evaluation design is thorough and generally rigorous
  • The contributed datasets (IdeaShift and TwiceTold) are novel and could be useful
Critical weaknesses
  • The prompting-based methods in the paper's outline extraction can cleanly isolate ideas. The multi-stage LLM pipeline (Gemini extraction and paraphrasing) likely introduces strong artifacts and confounders (e.g. clear LLM vocabulary or style, formulaic or template outline structures, or length) that the classifier might exploit instead of performing genuine idea provenance.
  • I'm not sure if the reported performance in Section 4 supports the paper's strong claims about what IDEALENS can do: it still fails on 32% of AI-planned human stories in TwiceTold, with an 8.2% false positive rate on LLM-polished human academic abstracts. These limitations should be better foregrounded.
  • The "multilingual" part of this work is superficial, and cross-lingual idea invariance is a clear overclaim. To my understanding, the paper simply translates non-English texts into English outlines by Gemini prior to classification (and this is only disclosed in the appendix, while the "multilingual" claim appears in the main body) instead of directly testing for cross-lingual ideation.
Questions for authors

There are pervasive claud-ish in the paper, likely from the authors’ disclosed use of LLMs to edit the paper. It is fine to use LLMs in that way, but there are many instances where LLMs did not achieve the authors’ intended effect of improving the readability of the paper, including but not limited to the following examples:

  • Formulaic antitheses and rigid/forced dichotomies:

    "Detectors like Pangram answer only who wrote the words (prose provenance), while Gigot’s defense, like many emerging policies on AI use in writing (SpringerNature, 2026; ICLR 2027 Program Chairs, 2026), hinges on who came up with the ideas (idea provenance)." (rows 41–42)

    "The fundamental challenge in training an idea detector is that we have no large-scale labels for who came up with a document’s ideas, only for who wrote its prose" (rows 130–131)

  • “While” and “whereas” are severely overused as conjunctions. The "([Entity A] does X, while [Entity B] does Y)" contrastive structure appears more than 20 times in the paper. E.g., an especially bad case in rows 444–448:

    "Human ideas generally state the limitations of their claims, while AI ideas extend their claims beyond the evidence: the AI idea in the third row of Table 1 predicts that a newly released Kia model will “establish a permanent legacy”, whereas the human idea provides a hedged projection of when a fleet deployment will occur. AI ideas also omit limitations: the AI review in the fourth row describes the MacBook as at an “ideal equilibrium” without any downsides, while human writers explicitly state the limitations that constrain them."

  • Ironic cases of overclaiming:

    "Since they share their backbone, training documents, and labels, any difference in their behavior is a direct result of transforming the input into our outline representation." (rows 221–222)

    Wouldn't claiming "any difference" be far too strong? What about artifacts and stylistic confounds introduced by the multi-stage LLM pipeline used to extract and paraphrase those outlines?

    "We apply the same principle in the opposite direction by keeping the labels of a prose detector but replacing its inputs with paraphrased outlines, so that the labels can only be fit through the ideas." (rows 468–470)

    Is this really an "opposite" direction? And can labels "only be fit through the ideas" when the outlines carry formatting, structural, and vocabulary traits from the LLM paraphraser?

    "...learns from the same cheap labels as prose detectors and can therefore be trained at their scale." (rows 85–86)

    Footnote 5 reveals that extracting and paraphrasing those 1M training outlines cost $66,400 in API credits. Calling a $66k API pipeline "cheap labels" that scale easily is a particularly jarring example of the "Erased Costs" that the authors associate with AI writing in Table 1.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.