Grounded without Drifting from Pretraining: Rollout-Free Target Optimization with Implicit Distillation for RAG

Post-training for retrieval-augmented generation (RAG) faces two coupled challenges: human-written supervision can be out-of-distribution with respect to the pretrained model's response distribution, yielding a supervision mismatch that induces undesirable distributional drift, while the learned model must remain faithful to the retrieved context. We propose TIDE (**T**arget Optimization with **I**mplicit **D**istillation for Cont**E**xt-Grounded RAG), a principled and efficient rollout-free approach that retains human supervision while simultaneously encouraging pretrained-distribution compatibility and faithfulness. We first characterize the analytical form of a desired target LLM for RAG by optimizing a novel value-based objective that jointly captures length-normalized likelihood under the pretrained model and faithfulness to retrieved evidence. While the target LLM is intractable due to the unknown human-supervision distribution, we introduce implicit target-LLM distillation, which distills the analytically defined target without explicitly constructing or sampling from it and instead enables training directly on the observed human-written data. This distinguishes our approach from standard teacher–student distillation. Our theoretical analysis characterizes how the optimized target shifts probability mass away from low-compatibility and unfaithful responses and provides a rejection-sampling interpretation of TIDE. Unlike approaches that mitigate supervision mismatch through model-generated demonstrations or constructed preference pairs, TIDE requires no response rollouts or generate-and-filter supervision, reducing the cost of constructing training targets. Across a broad range of RAG benchmarks, TIDE achieves the highest average answer-quality F1, outperforming the strongest trained baseline by 23.2% with Qwen3-4B and 21.7% with Qwen3-14B, while also achieving strong faithfulness to the retrieved context.