acceptodds
Under review as a conference paper at ICLR 2027

Gap Detection: Mitigating Homogenization through Demand and Defensibility

Abstract

Language models often produce semantically similar answers to open-ended questions, even when many plausible alternatives exist. Generating useful alternatives requires identifying both what the query calls for and what existing responses already cover. We introduce Gap Detection, a framework that searches for relevant but unoccupied directions in a shared text embedding space. Its objective, GapScore, combines query relevance (Demand) with dissimilarity to the nearest existing item (Defensibility). The framework optimizes this product on the unit sphere, maps candidate directions to concept words with a language model, and uses task-specific filtering and constrained prompting to turn those concepts into text. On five curated topics and 90 INFINITY-CHAT queries, the reported mean relative reductions in output Collision are 12.8% and 10.8%, respectively. We also instantiate the framework for generative engine optimization over four product markets and for iterative synthetic-data augmentation. In the latter setting, with 100 training examples per method, Gap Detection reduces Collision from 0.324 to 0.298 relative to undirected accumulation and increases the Vendi Score from 27.16 to 29.20; the accuracy difference is one example on a 40-example test set. Component ablations expose the trade-off between semantic separation and relevance, while also showing that better direction scores do not uniformly yield better downstream metrics. These results support explicit gap search as a practical mechanism for diversifying content, with task validity checked separately from geometric novelty.

Primary area:
foundation or frontier models, including LLMs
Keywords:
language model output diversity, semantic search, representation learning, generative engine optimization, model collapse, submodular optimization

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_MT49OpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Gap Detection for reducing repetitive LLM outputs by identifying relevant but unexplored directions in semantic embedding space. The authors introduce a gapscore for combining demand and defensibility, allowing the framework to optimize semantic directions. In addition, map these directions into linguistic concepts using LLMs. They experiment on 90 Infinity-Chat queries, showing a 10.8% relative collision reduction.

Critical strengths
  1. This is a very relevant problem, and the paper addresses LLM output homogenization.
  2. Interpretable dimensions: the Demand vs Defensibility objective provides a clear formulation for balancing relevance and novelty.
  3. Practical: the work uses embedding optimization, retrieval, and LLM validation, allowing approach to identifying under explored content directions.
Critical weaknesses
  1. Novelty: gapscore resembles existing diversity approaches such as MMR. The paper needs to better establish what advantage the proposed formula offers over simpler approaches.
  2. Ablations do not show a consistent story: defensibility only is larger than gapscore, and random directions are comparable. Is the geometric objective providing any meaningful benefit?
  3. Eval only measures geometric diversity but not quality.
  4. Limited stat rigor in lack of CIs, uncertainty estimates. Small sample size.
Questions for authors
  1. What fundamentally distinguishes GapScore from existing diversity objectives such as MMR?
  2. What evidence demonstrates that optimizing GapScore is necessary for the observed diversity improvements?
  3. How sensitive are the reported diversity improvements to the embedding model used for evaluation?
Reviewer_GNyeOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The paper is about homogenization in language model outputs: ask an open-ended question and you tend to get the same few ideas back in different words. Instead of changing the decoding temperature or reranking candidates (as MMR does), the authors try to find ideas that are still missing. Their method, Gap Detection, embeds the existing responses and the query in the same space, and searches for a direction that is both relevant to the query (Demand) and far from the nearest existing response (Defensibility). The score is the product of the two. The search is done with gradient ascent on the unit sphere with several restarts, and it is repeated so that each new direction is pushed away from the earlier ones.

A direction is not text, so the authors add a pipeline to turn it into a sentence. Nearby vocabulary words are retrieved, a small Qwen model picks a few concept words, GPT-4o-mini filters them for plausibility, and then generates one sentence while being told to avoid the most common words and sentence patterns in the existing answers. The result is checked for novelty using embedding similarity. There is also an appendix with some geometry and optimization results. The main one says that a gap always exists in a 1,536-dimensional space when there are only 20 occupied vectors, which the authors themselves point out is not surprising and says nothing about whether that gap can be written as meaningful text.

The method is tested in three settings: open-ended generation, a simulated generative engine optimization (GEO) task on four Amazon product categories, and synthetic data augmentation for SST-2. For open-ended generation, the average pairwise embedding similarity (Collision) drops by 12.8% on five topics and 10.8% on 90 INFINITY-CHAT queries. In GEO, the chosen claims reach an average cited-word share of 0.524. In augmentation, Collision goes from 0.324 to 0.298 against plain data accumulation and test accuracy from 60.0% to 62.5%. The paper reports single runs only and is open about several of its limitations.

Critical strengths
  1. I like the overall design. The method separates the geometric search from the step that writes text, and the concept words in between are easy to inspect. This means that when something goes wrong, one can see whether the search, the concept selection or the sentence generation caused it. The objective itself is simple and easy to understand.

  1. The authors are honest about results that do not favor them. They point out that Defensibility-only reduces Collision more than the full objective, that random directions pass the novelty check more often, that Demand-only gets a higher citation share in GEO, that the accuracy gain is a single example, and that Qwen's Vendi score goes down. This made it much easier for me to judge the results, and I appreciate it.
Critical weaknesses
  1. I don't think the ablation shows that the GapScore objective is needed. GapScore does get the best direction score, but that is what it is optimizing. On the actual generated text (Table 2), it has the lowest novelty pass rate of the four options (84%, against 100% for random directions), and random directions reduce Collision by 10.3%, close to GapScore's 10.7%. The authors suggest themselves that the prompting stages are doing a lot of the work. I agree with that reading, but the paper has no run without direction search at all, so I can't tell how much of the gain comes from the geometry and how much from the avoidance prompts. These are also single runs on five topics, so differences this small could be noise.

  1. The synthetic data comparison is not controlled. According to the appendix, the proposed method uses VADER-based filtering and up to four retries, and Data Accumulation does not. So the better Collision and Vendi numbers could come from the filtering and not from the direction choice. The authors admit this, but it means the main augmentation result doesn't test their claim. On top of that, the test set has 40 examples, so the 2.5-point accuracy gain is one prediction, and the whole table comes from one run.

  1. The evaluation is too small and too circular to support the conclusions. The same embedding model is used to find the directions and to measure Collision and Novelty, there is no human judgment, and there is no comparison with temperature or min-p sampling, which is the obvious alternative. In GEO, the claim is always placed in source position 5, one market has only three brands, and the headline cited-word share is the best of five directions picked using that same simulator, so it is optimistic. There are no standard deviations or significance tests anywhere, and no code is provided. I also have two smaller concerns about the main table. The paper never says what the "after" Collision in Table 1 is computed over, and when I weight the group reductions in the appendix by the number of queries I get about 10.4%, not the reported 10.8%.
Questions for authors
  1. How much of the improvement comes from the direction search and how much from the avoidance prompts? What happens if you remove the avoidance instructions, or use no direction at all?

  1. If Data Accumulation gets the same VADER filter and the same retry budget, does GapScore still win on Collision, Vendi or accuracy?

  1. Can you report means and standard deviations over several seeds for the SST-2 experiments, and a significance test for the accuracy differences?

  1. What exactly are the "before" and "after" sets in Table 1? Why does the weighted mean of the group reductions in the appendix come to about 10.4% instead of 10.8%? For GEO, can you report cited-word share for the selected claims on a fresh simulation, instead of the one used to choose them?
Reviewer_pydEOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper presents a new method Gap Detection for identifying and generating potentially more diverse text, e.g. in response to open-ended prompts. This method works by embedding the relevant query/queries and existing candidate text responses, and calculating a Gap Score, which is the product of two component scores designed to measure similarity between the new candidate embedding to the embedded query/queries (Demand) and dissimilarity to embedded existing candidates (Defensibility). To use the proposed new candidate embedding for generating new candidate text outputs, the method finds tokens/words which are close to the candidate embedding, and puts these words in a pre-written prompt that suggests using these words. The prompt also includes instructions not to use certain words/tokens, which are closest to the centroid of the embeddings of the existing candidates. The prompt is then fed to a generator (an LLM), which actually generates the candidate text. The work extends Gap Detection to GEO and synthetic text generation. Evaluations are conducted on 5 chosen topics and 90 INFINITY-Chat queries and are mostly reported by measuring two metrics: Collision and Novelty, which are both defined based on cosine similarity of the same embeddings used in the design of Gap Detection. The work also evaluates in a GEO setting and a synthetic data generation setting.

Critical strengths

S1. Lack of generation diversity has become a well-documented flaw of even very large scale proprietary models, and some proposed solutions have clear shortcomings (e.g., requiring white- or gray-box access to the generating models). The proposed method does not require access to logits or internal workings of generating models and is based on very simple and interpretable geometric principles.

Critical weaknesses

W1. The evaluations are not convincing. The evaluation is designed circularly with respect to the proposed method. The method is based around the idea of finding new text generations that have high embedded cosine similarity with existing text responses without sacrificing too much cosine similarity to the query, which is also captured by the proposed evaluation. This renders the evaluation relatively meaningless. Further, one issue with most measures of diversity is the difficulty of capturing the complexities of the notion of "diversity" within a simple formula. Thus, even if the authors are not directly making claims around human notions of diversity, but only on geometric diversity, it's important to have some reporting around whether this method can improve human evaluations of diversity, or at the bare minimum, other metrics for diversity. Improving geometric diversity is of little practical interest, whereas improving human or more complex linguistic notions of diversity is of high practicality.

W2. There is an obvious competing method of similar simplicity, which the authors do not address. Specifically, to add to any open-ended prompt an explicit instruction to generate n candidates, making sure that they are distinct from each other. This method works very well for short-form generations (e.g. "give me a metaphor for time"). Where it is unwieldy for generating long-form content (e.g. "write me a creative story about X"), this work also lacks any evaluation, and it seems very likely that this simplistic cosine similarity based approach would not work well.

Questions for authors

Q1. Can you show examples of the generations from your method? I read the appendix and did not see any examples or more qualitative findings recorded there.

Q2. Embeddings of single words/tokens and embeddings of longer text samples are not necessarily comparable. What kinds of words are found when searching near the centroid of existing text generations? Why not just find rare repeated n-grams? Why not find words/tokens for each individual existing text generation rather than the centroid?

Q3. Related to both Q1 and Q2, can you show examples of the words found by searching near the new proposed candidate embeddings in each iterative run through your process, and the resulting final generation from that run?

Q4. W.r.t. W2, I don't think I follow what benefit your method confers over the simple method of asking for multiple diverse generations directly. Can you explain?

Q5. I didn't follow the explanation of the GEO procedure, can you give an example of a set of queries in this setting, as well as examples of what the competing text would be?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.