FusePRAG: Fuse-then-Project for Parametric Retrieval-Augmented Generation

Parametric retrieval-augmented generation incorporates retrieved evidence into language models through adapter parameters. However, existing document-wise adapter generators produce an adapter independently for each document and combine the resulting updates afterward, so cross-document relationships cannot directly shape what each adapter encodes. We investigate query-conditioned evidence fusion before parameter projection for multi-document question answering. Thus, we propose FusePRAG, a "Fuse-then-Project” framework in which a shallow fusion module jointly processes query and document embeddings, and a parameter translator maps the resulting representation to a single query-specific LoRA adapter. Training proceeds in two phases. First, the fusion module is aligned with embeddings of answer-aware evidence summaries: answer-relevant facts drawn from gold passages using training questions and answers. Second, the parameter translator is optimized for answer generation. These summaries are used only to construct Phase-1 training targets and are neither generated nor required at inference. Across four question-answering benchmarks and three backbone models, FusePRAG improves macro-average F1 over DyPRAG by 3.0–4.7 points with the same retrieved context. A capacity-matched control yields consistent incremental gains from direct document–document attention, while the single-projection design lowers adapter-construction latency at larger retrieval depths.