acceptodds
Under review as a conference paper at ICLR 2027

Doc2LoRA provides decodable representations of scientific idea

Abstract

Representing scientific papers as points in a space lets us search for similar papers and inquire about how fields relate to one another and drive innovation. Beyond search, the vector space of papers invites generation: mixing papers through simple vector operations creates new points, mirroring combinatorial novelty, the recombination of existing ideas into new ones. However, a mixed point often represents an idea no paper has yet realized, with no papers nearby to identify the idea. We propose representing each paper by a LoRA adapter generated by the Doc-to-LoRA hypernetwork. Every point in the space, including mixtures, thus represents a large language model (LLM) open to questions and instructions in natural language. On papers from the American Physical Society (APS), we instruct the LLM at the average of each subfield to name the field in a few words and obtain labels closer to the official names than the labels of five baselines, as judged by word overlap and a panel of five LLM judges. We also ask the LLMs at points between two APS papers to write an abstract and obtain descriptions shifting from one paper to the other in step with the mixing weight. While Doc-to-LoRA is trained for generation, a small invertible transform makes the embeddings competitive for search, on par with SPECTER2 and EmbeddingGemma and close to SBERT. Because the transform is invertible, every point in the transformed space still maps back to an LLM. The embeddings thus serve both search and generation, enabling researchers to question the idea at any point in the space as a starting point for generating new ideas.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
embedding, science of science, hypernetworks

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_hW3oOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Doc2LoRA, a representation framework that addresses the issue of scientific paper embeddings serving search rather than generation, where points created by mixing papers cannot be interpreted or questioned. Specifically, the framework introduces conversational embeddings, where every paper representation expands into a LoRA adapter that turns the base LLM into a model open to natural-language questions about the corresponding idea. It also introduces a small invertible transform trained on citation pairs, serving the purpose of aligning the generation-trained space for retrieval while preserving decodability through its inverse. Evaluations against five baselines on cluster labeling suggest the decoded mixtures shift in step with the mixing weight and remain competitive for search, showing that a single embedding space shares the potential of finding existing papers and generating novel ideas.

Critical strengths
  • Novel definition of conversational embeddings. The paper defines scientific articles as a LoRA component inserted into LLMs that can be interacted with natural-language questions. Arithmetic operations such as averaging a cluster of papers or interpolating between two papers remain well defined, and the resulting points, which correspond to no existing paper, can still be decoded and interrogated. Prior representations do not satisfy this definition, as text encoders support search without decodability, vector-to-text methods reconstruct source text without guarantees for synthetic points, and concept-based vectors restrict interpretation to a predefined vocabulary.

  • Complete ablation and control studies. The experiments are sufficiently designed to rule out alternative explanations in addition to report headline scores. In the interpolation study, the in-context baseline is given both source texts and the exact mixing proportions, an informational advantage the other methods do not have, yet it remains near the midpoint while the proposed embeddings track the mixing weight, and the verbatim copy rate shows that the decoded blends are not assembled by copying the sources. Applying the same invertible transform to dedicated text encoders yields at most +0.009, confirming that the retrieval gain surfaces geometry already present in the Doc-to-LoRA space.

Critical weaknesses
  • Gap between conversational embedding and its contribution towards combinatorial novelty. Averaging and interpolation perform a uniform, untyped blending of whole documents, and the interpolation experiments only measure whether the decoded text tracks the mixing weight and how much wording it copies, which establishes that the blends are legible mixtures of the two sources. Combinatorial novelty, however, is defined by integration [1, 2]: selecting specific notions from the source works and articulating an integration logic that connects them, aligns their structures, and resolves incompatibilities between their assumptions, rather than merely placing their contents side by side [3]. Thus, there remains a gap that explicitly justifies the connection between how the mere arithmetic operations of conversational embeddings contribute to the reasoning in integration, which thereafter leads to combinatorial novelty.

  • Lack of human evaluation in quality judgement. All quality judgements in the paper are delegated to a panel of five LLM judges described by the pairwise win rates behind the cluster-labelling results. Therefore, the lack of human evaluation on the decoded labels or blended abstracts poses a threat to the reliability of the evaluation results.

References:

[1] Uzzi, B., Mukherjee, S., Stringer, M., & Jones, B. (2013). Science 342,468-472(2013).DOI:10.1126/science.1240474

[2] Swanson, Don R. Undiscovered public knowledge. The Library Quarterly 56.2 (1986): 103-118.

[3] Shen, Y., de Sousa, D. X., Nascimento, R. M. A., Guo, H., & Zhu, X. (2026). IDRBench: Understanding the capability of large language models on interdisciplinary research. Forty-third International Conference on Machine Learning

Questions for authors

I would consider raising my score if the authors provide additional evidence to the weakness addressed above.

Reviewer_HG9wOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes to represent scientific papers not as static vectors, but as LoRA adapters generated by a frozen Doc-to-LoRA (D2L) hypernetwork, such that every point in the embedding space—including the average and interpolated points of the paper—can be decoded into a queryable LLM. The main contributions include: (1) Proving that averaging the D2L embeddings of paper clusters and decoding the mean, the resulting field names are closer to the official PACS label than the five baselines as judged by fuzzy overlap and pairwise win-rate analysis; (2) Proving that the abstract generated by interpolating between two paper embeddings transition from one source paper to another with the mixing weight; (3) Introducing a small reversible linear transformation using InfoNCE on OpenAlex citation pairs training which makes the retrieval performance of D2L embeddings comparable to similarity computation methods such as SPECTER2, EmbeddingGemma while maintaining decodability. The code, configuration, and Snakemake workflow are released and facilitates reproducibility.

Critical strengths

This work transforms scientific paper embeddings into an actionable artifact: by utilizing the fact that each point can be decoded into an LLM conditioned via LoRA, arithmetic operations on these embeddings—such as calculating cluster means or performing A–B interpolations—can be queried using natural language. The paper validates this through concrete, well-controlled experiments: decoded cluster labels outperform five baselines when compared against official PACS category names, and interpolated points successfully reflect the specified mixing weights while maintaining a low rate of verbatim reproduction. The idea that the reversible transformation elevates a representation originally trained for generation to a retrieval performance level comparable to specialized encoders.

Furthermore, the release of the code, configurations, and a Snakemake workflow ensures that these empirical claims remain verifiable.

Critical weaknesses

First, the core result of decoding averaged embeddings into domain names mostly demonstrates the capabilities of the pre-existing Doc-to-LoRA (D2L) hypernetwork. The central claim as illustrated in the main table and main figures effectively rests on an off-the-shelf checkpoint combined with empirical averaging, which undermines the paper's purported novelty.

Second, the retrieval evaluation fails to provide a diagnostic assessment of the proposed reversible transformation gg) The comparison is made only against baseline text encoders that have not undergone any in-domain training, lacking crucial baselines where an encoder further trained with in-domain data using objective such as InfoNCE on the same citation pairs. Thus, the performance parity with SOTA methods such as EmbeddingGemma cannot be definitively attributed to the reversible design rather than the additional supervisory signal. The retrieval evaluation also fails to demonstrate that a unified representation for both retrieval and generation offers any tangible advantage over using a robust text encoder for retrieval, followed by an LLM to generate content based on the retrieved documents. This makes the manuscript appear more as a reasonable application of an existing hypernetwork than a contribution that would introduce a novel methods towards representation learning.

Questions for authors

Could the author conduct a head-to-head comparison against the simple pipeline with strong encoder for retrieval and LLM over retrieved docs on a concrete end-to-end task? This would substantially strengthen the unified-representation claim

Reviewer_1VY6OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

Doc2Lora PROVIDES DECODABLE REPRESENTATIONS OF SCIENTIFIC IDEAS investigates the utility of the Doc2Lora hypernetwork as an invertible map between academic papers and parameteric conditioning to LLMs (both as Lora, but also with various other downstream re-projections and learned transformations). They run 3 main experiments on cluster labeling, interpolation, and retrieval.

Critical strengths

Doc2Lora is an interesting idea that I had missed, so I read that paper as well. I think the concept of this work is a critical strength. I'm familiar with the literature on meta-science and the use of AI systems / LLMs for creativity, so I'm predisposed to enjoy this direction.

Another strenght: there is value in selecting a small number of simple experimental conditions, which lends a nice focus to the paper.

Critical weaknesses

The primary critical weaknesses in this paper is that the baselines were unconvincing. I did not leave this work with confidence that this method works better than simple alternatives. Specifics I'll leave in the "questions for authors" section below.

Questions for authors

These are clarification requests and requests that could change my perception of this paper.

  1. "Mixed point… no papers nearby to identify the idea" (abstract, l.015). Did you check this assumption? For example, by measuring how far midpoints are from the nearest real paper compared with real papers. I’d like to see for curves for individual pairs as well, not just averaged.

2.“ This representation isolates what makes that document unique, omitting redundant commonalities present across all documents” (l.093) This feels like an ovreclaim and it contradicts Section 3.1. where averaging a cluster can only decode the shared field name if that shared information is still in the vectors.

  1. I am worried that 3.3 is not fairly comparing to baselines. The in-context baseline gets only the 20 abstracts nearest the centroid, while D2L gets up to 2,000 (Table 3). How was 20 selected? Was this ablated?

  2. Section 3.3 retrieval: wouldn't raw Qwen last layer plus the same linear map do as well? In the appendix you showed ICAE plus the same transform gains +.178 and ends up "virtually equivalent" to D2L plus the transform (l.1421). This makes me think that probably any LLM representation plus a contrastive linear map on citations gets there. I’d like to see mean-pooled Qwen3-4B hidden states plus the transform, on the same 14 benchmarks. A same-size retrieval model such as Qwen3-Embedding-4B would also be a useful comparison. The baselines used here are 110–335M parameters and run zero-shot.

And this one is small, but since it comes early in the paper I thought you should know: “However, a mixed point often represents an idea no paper has yet realized, with no papers nearby to identify the idea.” This is an overclaim and poor writing. It could be that there are papers nearby to identify the idea. More importantly, while I was reading (linearly) through your paper, this sentence confused me. After finishing your paper, I understand what you’re saying, but if I were not reviewing your paper you might have lost me there.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.