acceptodds
Under review as a conference paper at ICLR 2027

Grounded without Drifting from Pretraining: Rollout-Free Target Optimization with Implicit Distillation for RAG

Abstract

Post-training for retrieval-augmented generation (RAG) faces two coupled challenges: human-written supervision can be out-of-distribution with respect to the pretrained model's response distribution, yielding a supervision mismatch that induces undesirable distributional drift, while the learned model must remain faithful to the retrieved context. We propose TIDE (**T**arget Optimization with **I**mplicit **D**istillation for Cont**E**xt-Grounded RAG), a principled and efficient rollout-free approach that retains human supervision while simultaneously encouraging pretrained-distribution compatibility and faithfulness. We first characterize the analytical form of a desired target LLM for RAG by optimizing a novel value-based objective that jointly captures length-normalized likelihood under the pretrained model and faithfulness to retrieved evidence. While the target LLM is intractable due to the unknown human-supervision distribution, we introduce implicit target-LLM distillation, which distills the analytically defined target without explicitly constructing or sampling from it and instead enables training directly on the observed human-written data. This distinguishes our approach from standard teacher–student distillation. Our theoretical analysis characterizes how the optimized target shifts probability mass away from low-compatibility and unfaithful responses and provides a rejection-sampling interpretation of TIDE. Unlike approaches that mitigate supervision mismatch through model-generated demonstrations or constructed preference pairs, TIDE requires no response rollouts or generate-and-filter supervision, reducing the cost of constructing training targets. Across a broad range of RAG benchmarks, TIDE achieves the highest average answer-quality F1, outperforming the strongest trained baseline by 23.2% with Qwen3-4B and 21.7% with Qwen3-14B, while also achieving strong faithfulness to the retrieved context.

Primary area:
learning theory
Keywords:
RAG Post-Training, Faithfulness, Distribution-Drift, Implicit Distillation, Target Optimization

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_hMpxOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes TIDE, a rollout-free RAG post-training framework that derives a closed-form target LLM via KL-regularized value optimization and then trains on fixed human responses through an equivalent weighted-likelihood objective. The value function combines pretrained-model compatibility and context faithfulness. Experiments on five RAG benchmarks with Qwen3-4B/14B show consistent gains over SFT and Self-Demo.

Critical strengths

The problem is well-motivated and practically relevant: jointly preserving pretrained behavior and enforcing faithfulness under fixed human supervision. The theoretical development is clean—Theorem 3.1 gives a closed-form target, and Corollary B.2 establishes the implicit-distillation equivalence. The rejection-sampling interpretation and the acceptance-probability comparison with Self-Demo provide useful intuition. The method is simple and rollout-free, which is a genuine advantage over generate-and-filter pipelines.

Critical weaknesses
  1. Static weighting lacks empirical validation. The weights wi=exp⁡(αV(Xi,gi))w_i=\exp(\alpha V(X_i,g_i)) are precomputed from the frozen pretrained model πθ0\pi_{\theta_0} and a fixed NLI scorer, and remain fixed throughout training. The authors argue in rebuttal that this follows directly from the fixed target distribution πα∗\pi_\alpha^*. While this is theoretically coherent, the paper never empirically verifies whether the static approximation remains valid as πθ\pi_\theta drifts from πθ0\pi_{\theta_0}. The rebuttal promises an ablation with periodically refreshed weights; this would directly address the concern.

  2. Missing comparison with prior weighted-SFT methods. The paper compares only against SFT and Self-Demo. Sample weighting for distribution-aware fine-tuning has been explored before (e.g., FLOW, VAR, ADAPT). Without these baselines, it is unclear whether TIDE’s specific value function V=log⁡sθ0+FV=\log s_{\theta_0}+F adds value beyond perplexity-based weighting, or whether the faithfulness term FF provides additional gains. The rebuttal agrees to add FLOW and a reward-weighted SFT baseline, which would clarify TIDE’s contribution.

  3. No empirical analysis of the weight distribution. The method hinges entirely on wiw_i, yet the paper never shows the distribution of weights, their correlation with sθ0s_{\theta_0} and FF, or qualitative examples of high- and low-weight responses. Without this, it is impossible to tell whether TIDE effectively trains on a small subset, or whether the weights are nearly uniform. The rebuttal agrees to add histograms, scatter plots, and qualitative examples.

Questions for authors

I encourage the authors to further clarify the above concerns and discuss how they plan to address these limitations in future revisions.

5 comments
AuthorsOpenReview ↗

Official Comment by Authors

We are grateful for the reviewer’s constructive feedback and insightful comments. In the following, we provide point-by-point responses to all your comments.

W1["The rebuttal promises an ablation with periodically refreshed weights"]

We would like to clarify that the motivation behind why we keep the weight fixed as the pretrained model (πθ0\pi_{\theta_0}) serves as a fixed reference distribution, and its likelihood addresses one of the aims in our work, i.e. how compatible each response is with the original pretrained distribution. As the trainable model (πθ\pi_\theta) evolves, the pretrained likelihood should remain unchanged, since its purpose is precisely to preserve in-distribution with respect to the original distribution. Therefore, periodically refreshing the weights is not necessary for our proposed objective and would represent a different training procedure.

W2["Missing comparison with prior weighted-SFT methods."]

Thank you for suggesting this comparison. We have added a popular weighted-SFT baseline, i.e. Dynamic Fine-Tuning (DFT) for both Qwen3-4B and Qwen3-14B, each trained separately on MSMARCO and NewsQA and evaluated across five benchmarks. DFT weights supervised token losses using the current model’s token probabilities, with stop-gradient applied to the weights.

Table 1. Comparison of TIDE and DFT across model sizes and training datasets. Scores are rounded to three decimal places. The higher F1F_1 score within each model-size pair is shown in bold. Overall averages are computed across the ten training–evaluation combinations.

TrainEvalQwen3-4B TIDEQwen3-4B DFTQwen3-14B TIDEQwen3-14B DFT
MSMARCOMSMARCO0.5880.4230.5940.638
MSMARCOHotpotQA0.7220.6090.7280.626
MSMARCONQ0.3500.3350.3110.384
MSMARCOWoW0.3020.0130.2900.288
MSMARCOT-REx0.3140.2390.3270.249
NewsQAMSMARCO0.5840.3400.5590.420
NewsQAHotpotQA0.7130.2040.7170.292
NewsQANQ0.3530.1500.3080.210
NewsQAWoW0.3070.2790.3370.191
NewsQAT-REx0.2920.1090.3100.164
Overall Average0.4520.2700.4480.346

The reported results favor TIDE overall. DFT achieves average F1F_1 scores of 0.2700.270 and 0.3460.346 for Qwen3-4B and Qwen3-14B, respectively, compared with 0.4520.452 and 0.4480.448 for TIDE. TIDE achieves higher scores in all ten training–evaluation combinations for Qwen3-4B and eight of ten for Qwen3-14B.

A plausible explanation for this difference is the information used to guide training. DFT emphasizes tokens according to the current model’s confidence, without explicitly assessing whether the complete response is supported by the retrieved evidence. TIDE combines response-level faithfulness with compatibility measured against the frozen pretrained model. These criteria jointly shape its optimized target and enter training through implicit distillation. The results support the usefulness of this RAG-specific design compared with the tested DFT baseline, although this comparison alone does not isolate the contribution of each component.

We will include the additional baseline, results, and discussion in the revised manuscript.

AuthorsOpenReview ↗

Official Comment by Authors

W3[" The method hinges entirely on (wi)(w_i), yet the paper never shows the distribution of weights...."]

We thank the reviewer for raising this point. Following your suggestion, Table 2 below summarizes the distribution of mean-normalized training weights assigned by TIDE. A normalized weight of 1 corresponds to the average training weight, while values below or above 1 indicate examples receiving less or more influence, respectively. The weights span several orders of magnitude. The distribution contains a broad low-weight group and a smaller high-weight group. This indicates that TIDE down-weights less suitable examples while assigning greater importance to responses that are both likely under the pretrained model and faithful to the retrieved context. All examples retain nonzero weight.

Table 2. Summary of the training-weight distribution for Qwen3-14B on MSMARCO with α=0.5\alpha=0.5. The normalized weight is ri=wi/wˉr_i=w_i/\bar w, where wˉ\bar w is the mean raw weight.

StatisticValue
Number of training examples72,553
Mean raw weight wˉ\bar w0.254314
Mean normalized weight1
5th percentile of normalized weight0.001714
Median normalized weight0.037617
95th percentile of normalized weight5.401831
99th percentile of normalized weight6.170301

We sort the training examples by their assigned weights in ascending order and divide them into ten groups of approximately equal size. Each group therefore contains approximately one-tenth of the training examples. The first decile contains the lowest-weight examples, while the last contains the highest-weight examples. Table 3 below shows that the highest-weight decile has greater mean NLI faithfulness than the lowest-weight decile. However, the trend is not monotonic across all deciles. This is consistent with the joint construction of the weights: an example’s weight depends on both its pretrained-model compatibility score and its faithfulness score. Consequently, ordering examples by weight need not produce the same ordering as sorting them by faithfulness alone.

Table 3. Mean NLI faithfulness and mean compatibility across training-weight deciles. Values reproduce the rounded annotations in the figures.

Weight decileMean NLI faithfulnessMean compatibility
1 (lowest weights)0.5361.6×10−71.6 \times 10^{-7}
20.6541.1×10−61.1 \times 10^{-6}
30.7563.5×10−63.5 \times 10^{-6}
40.8449.0×10−69.0 \times 10^{-6}
50.8862.4×10−52.4 \times 10^{-5}
60.9107.7×10−57.7 \times 10^{-5}
70.8936.3×10−46.3 \times 10^{-4}
80.7990.033
90.8300.358
10 (highest weights)0.9720.730
Overall mean0.8080.112

Using the same weight deciles, the compatibility column in Table 3 shows a consistent increase in pretrained model compatibility from the lowest-weight to the highest-weight groups. This indicates that the likelihood-derived component is closely aligned with the empirical weight ordering.

The joint construction of the weights does not require compatibility and faithfulness to exhibit identical trends across these groups. A greater compatibility score can compensate for a lower faithfulness score when determining an example’s weight. Consequently, compatibility can increase consistently across weight deciles while faithfulness varies non-monotonically. The increasing compatibility trend is an empirical observation in this dataset, rather than a general requirement that sorting by the combined weight must also sort either component individually.

AuthorsOpenReview ↗

Official Comment by Authors

W3["qualitative examples of high- and low-weight responses"]

Thank you for this comment. From a shortlist of 100 training examples, we selected two high-weight and two low-weight cases.

Table 4. Raw training weights for the selected examples. Values are rounded.

ExampleWeight
Casement windows (high)1.473577
Neuron structure (high)0.968264
DWIB meaning (low)0.000874
Tooth-filling cost (low)0.000792

Table 5. Selected high- and low-weight training responses. The interpretations are based on the evidence and our qualitative inspection.

ExampleFieldQuestion, evidence, and training response
Casement windows — High weightQuestion:what is a casement windows
Context excerpt:A casement is a window that is attached to its frame by one or more hinges. Casement windows are hinged at the side.
Original training answer:A casement is a window that is attached to its frame by one or more hinges. Casement windows are hinged at the side.
Interpretation:The answer directly addresses the question using information stated in the context. This response receives a high training weight.
Neuron structure — High weightQuestion:structure of a typical neuron
Context excerpt:A typical neuron consists of a cell body (soma), dendrites, and an axon.
Original training answer:A typical neuron consists of a cell body (soma), dendrites, and an axon.
Interpretation:The answer lists the parts of a neuron given in the context and directly answers the question. It receives a high training weight.
DWIB meaning — Low weightQuestion:what does dwib mean
Context excerpt:Acronym Attic: Searching over 3 million acronyms, abbreviations, and initialisms.
Original training answer:The World’s most comprehensive professionally edited abbreviations and acronyms database.
Interpretation:The answer describes a database without explaining what DWIB means. The context also does not support the claims that the database is the world’s most comprehensive or professionally edited. This response receives a low training weight.
Tooth-filling cost — Low weightQuestion:how much does a tooth filling cost
Context excerpt:The cost for amalgam fillings averages approximately $110 to $200 per filling. The cost for resin-based composite fillings, which is determined by the number of tooth surfaces on which the material is placed, ranges on average from $135 to $240 per filling.
Original training answer:$75 to $145 per
Interpretation:The answer gives a price range that is not supported by either range in the context and ends with an incomplete phrase. It receives a low training weight.

The two high-weight responses directly answer their questions using the supplied evidence. The two low-weight responses contain unsupported claims or an unsupported, incomplete answer. These selected cases illustrate how the assigned weights differ. We will include these qualitative examples and their interpretations in the revised manuscript.

Reviewer_EAvAOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper looks at post-training a language model for retrieval-augmented generation (RAG) using fixed human-written answers, where the typical approach is supervised fine-tuning (SFT), i.e. training the model to copy the human answers. However, the main problem with this approach that the authors identify is that human-written answers might either reduce faithfulness (by introducing unrelated information), or cause distribution shift if the human writing style is different from the pretrained model's. In turn, the usual fix for drift is something akin to Self-Demo (the paper's main baseline), which is to let the model rewrite the training answers and keep the ones that still match the human answer, although this introduces additional cost and compute overhead due to its rollouts (i.e. generating and filtering candidate answers for every training question). To combine the best of both worlds, the authors introduce TIDE (Target Optimization with Implicit Distillation for Context-Grounded RAG) which defines an analytic target model that shift the human answers toward ones the base model already finds likely and ones an NLI model finds supported by the context. They show that fitting this target, which they call implicit distillation, comes down to ordinary SFT in which each human answer gets a fixed weight, larger when the base model finds the answer likely and the NLI model finds it supported, so no answers are sampled during training. They compare TIDE with the base model, SFT and Self-Demo on Qwen3-4B and Qwen3-14B, trained on MS MARCO or NewsQA and tested on five benchmarks covering QA, dialogue and fact completion.

Critical strengths

I really liked how the authors presented the trade-off between getting good human responses and matching well to a pre-trained model, and how both SFT and Self-Demo were placed in the same context. What I thought was the best takeaway of the paper was that you could combine the ability of Self-Demo to match to a pre-trained model (i.e., through sampling and filtering the model's responses), using a fixed weight on each human response. Each weight is calculated based on a single forward pass through the base model and an NLI score (Equation 4), where if you set α=0\alpha=0 then it simply returns SFT. TIDE also appears to be fairly efficient compared to Self-Demo, and close to regular SFT. The authors also provide the complete prompt sent to the judges, their decoding options, and their hyper-parameters (in Tables 7, Appendix C-E), which makes the setup easy to reproduce.

Critical weaknesses

First, the training rule (Equation 4) is SFT where each human answer gets a weight, which is a known approach called reward-weighted regression, but the paper does not mention it or compare against simple versions of it. Specifically, since there is typically only one human answer provided per question, the model is still actually trained to copy that one answer, so the weights mostly decide which questions count more. The theory, however, describes moving probability between different answers to the same question (Theorems 3.1 and 4.1), which the training does not directly do. With α = 0.5 the weights also only range between 0 and about 1.65, so I cannot tell whether the gain comes from the paper's target or would come from any reasonable way of weighting examples.

Second, the gains are smaller than they look. The +23.2% and +21.7% are relative gains over Self-Demo, i.e. +8.5 and +8.0 F1 points. Furthermore, all results come from a single run with no variance, and about 75% (4B) and 82% (14B) of the gain over Self-Demo comes from two benchmarks, Wizard of Wikipedia (dialogue) and T-REx (fact completion), where the base model's F1 is close to zero. Finally, all answers are scored by Qwen3 models acting as judges, i.e. the same model family that is being trained, while TIDE favours exactly the answers the base Qwen3 model already finds likely, and there is no check against human labels, which reinforces the result.

Overall, based on the above points, I think the paper overstates it's gains and the results support a simpler claim than the title, i.e. weighting human answers by how likely the base model finds them and by an NLI score improves the judged F1 over SFT and Self-Demo at lower cost than Self-Demo, mostly on dialogue and fact completion. Unless TIDE clearly beats simple weighted SFT baselines with the same compute, across several seeds and with a judge from a different model family, I feel the paper would be more accurately described as a study of weighted SFT for RAG.

Questions for authors

I would like to have additional ablation runs with the same training budget as TIDE (same data, number of steps and learning rate): (i) Using the Likelihood Weights in Weighted SFT, (ii) Using the NLI Weights in Weighted SFT, (iii) Assigning the average weight across all responses of the training set for TIDE to each response in SFT, and (iv) Using SFT with Human Answers which are filtered by their likelihood in the base-model. Also please provide results for at least three seeds along with Standard Deviation, Average and a rescored output from a smaller random sample of outputs by a judge from a different model family compared to actual human annotations on a small number of examples.

6 comments
AuthorsOpenReview ↗

Official Comment by Authors

We are grateful for the reviewer's constructive feedback and insightful comments. In the following, we provide point-by-point responses to all your comments.

W1["First, the training rule (Equation 4) is SFT where each human answer gets a weight..."]

Thank you for this comment. The key distinction is the design principle underlying TIDE: it is developed to address the joint requirements of RAG post-training, namely faithfulness to retrieved evidence and compatibility with the pretrained model.

A general instance-weighted SFT objective accommodates many weighting strategies. Examples include exponentiated reward scores in reward-weighted regression (Hu et al., 2023), sequence-level importance ratios in iw-SFT (Qin and Springenberg, 2025), and binary weights corresponding to reward-based filtering in RAFT (Dong et al., 2023). However, the general objective does not itself determine how these weights should account for the two requirements above.

In TIDE, these requirements guide the construction of the value function and the optimized target distribution. The pretrained-likelihood term favors responses that are compatible with the pretrained model, while the faithfulness term measures their support in the retrieved context. This formulation allows us to incorporate both considerations while retaining human supervision and avoiding the generation of self-demonstrations.

Implicit distillation connects this target distribution to a tractable training objective based on the available human answers. It allows us to transfer the target's preferences for compatibility and faithfulness to the trained model without explicitly estimating the underlying human-response distribution or generating replacement responses. Compatibility therefore informs both the target construction, through the pretrained-likelihood term, and training, through the objective of approximating that target.

The connection to weighted SFT does not therefore fully characterize this procedure. In particular, our analysis provides an explicit upper bound on the probability of responses that have both low pretrained-model compatibility and low faithfulness scores. This guarantee follows from the target construction and distillation analysis; it is not implied by generic instance weighting alone.

Motivated by your suggestion, we additionally evaluate Dynamic Fine-Tuning (DFT) as a probability-weighted fine-tuning baseline. Table 1 compares TIDE and DFT across two model sizes, two training datasets, and five evaluation datasets.

Table 1. Comparison of TIDE and DFT across model sizes and training datasets. Scores are rounded to three decimal places. The higher F1F_1 score within each model-size pair is shown in bold. Overall averages are computed across the ten training--evaluation combinations.

TrainEvalQwen3-4B TIDEQwen3-4B DFTQwen3-14B TIDEQwen3-14B DFT
MSMARCOMSMARCO0.5880.4230.5940.638
MSMARCOHotpotQA0.7220.6090.7280.626
MSMARCONQ0.3500.3350.3110.384
MSMARCOWoW0.3020.0130.2900.288
MSMARCOT-REx0.3140.2390.3270.249
NewsQAMSMARCO0.5840.3400.5590.420
NewsQAHotpotQA0.7130.2040.7170.292
NewsQANQ0.3530.1500.3080.210
NewsQAWoW0.3070.2790.3370.191
NewsQAT-REx0.2920.1090.3100.164
Overall Average0.4520.2700.4480.346

TIDE achieves higher overall F1F_1 at both model sizes: 0.4520.452 versus 0.2700.270 for Qwen3-4B and 0.4480.448 versus 0.3460.346 for Qwen3-14B. It outperforms DFT in all ten training--evaluation combinations for Qwen3-4B and in eight of ten combinations for Qwen3-14B. DFT obtains higher scores on MSMARCO and NQ for the MSMARCO-trained Qwen3-14B model.

These findings support the motivation for incorporating both pretrained-model compatibility and evidence support into RAG post-training. DFT weights individual tokens using the current model's probabilities, but a high token probability does not necessarily indicate that a response is supported by the retrieved evidence. TIDE uses pretrained likelihood as a fixed reference for compatibility and includes a separate faithfulness signal to account for evidence support.

Through implicit distillation, these two signals determine the training emphasis according to the optimized target distribution. This provides a plausible explanation for the stronger overall performance observed in our experiments. The comparison demonstrates the empirical benefit of TIDE over the evaluated DFT baseline and complements the distinction established by our target construction and theoretical analysis. We will incorporate this detailed comparison with weighted SFT into the revised manuscript.

AuthorsOpenReview ↗

Official Comment by Authors

W1["the model is still actually trained to copy that one answer..."]

Thank you for this comment. As in standard SFT, training encourages higher probabilities for the reference tokens within the model's full vocabulary distribution. Thus, even with one reference answer per question, the model learns a conditional probability distribution. In TIDE, the pretrained model compatibility and faithfulness score control how strongly each human answer contributes to training, without explicitly comparing alternative answers to the same question.

Theorem 3.1 characterizes the optimized target distribution, which reweights responses according to pretrained-model compatibility and faithfulness. Section 3.2 connects this distribution-level construction to training: at the population level and under the specified input weighting, forward-KL distillation of the target is equivalent to the value-weighted likelihood objective. The observed training pairs provide an empirical estimate of this objective. Theorem 4.1 then shows that, under its stated assumptions, a model that sufficiently approximates this target inherits its suppression of responses with both low compatibility and low faithfulness, up to an approximation-error term. These results therefore characterize the target and its transfer to the learned model; they do not require explicit comparisons among multiple answers during training.

W2["Second, the gains are smaller than they look..."]

Thank you for your comment. We would like to respectively clarify why we believe TIDE’s improvements are substantial in the field of RAG. TIDE improves average F1 over Self-Demo by 8.5 points for the 4B model and 8.0 points for the 14B model. As RAG is more complex system compared to single LLM evaluation or training, the efforts of unit improvement in RAG is requires more effort. For context, RAG-Critic (ACL 2025) reports overall F1 gains of 2.8–5.3 points over standard RAG, while Reflective RAG (Findings of ACL 2026) reports average gains of 2.65 and 1.47 points over Search-R1. The closely related Self-Demo method (Finlayson et al., 2025) reports aggregate F1F_1 improvements over conventional retrieval-augmented instruction tuning (RA-IT) of 1.7 points for Llama-3-8B (79.6→81.379.6 \rightarrow 81.3) and 3.1 points for Llama-3-70B (76.3→79.476.3 \rightarrow 79.4) using its SFT variant. Its DPO variant achieves gains of 1.9 and 4.4 points, respectively.

We agree with the reviewer that some part of contributions on TIDE’s gains comes from WoW and T-REx. These benchmarks are important as they cover grounded dialogue and factual knowledge use respectively. The great improvement on these benchmark indeed demonstrate unique advantage of our method. Meanwhile, for other benchmarks, we can still observe the performance gain advantage compared to baseline as well.

We hope this clarification helps the reviewer see why we consider the gains substantial, both in their absolute magnitude and in the range of tasks where TIDE improves performance, and they can agree with our point.

AuthorsOpenReview ↗

Official Comment by Authors

Q1[" (i) Using the Likelihood Weights in Weighted SFT..."; "Finally, all answers are scored by Qwen3 models acting as judges, i.e. the same model family that is being trained"]

We greatly appreciate the reviewer's comments! Following your suggestion, we conducted additional experiments with Qwen3-4B trained on MS MARCO to compare TIDE with likelihood-based weighting following DFT, and an NLI-weighted variant that uses NLI faithfulness scores in place of likelihood weights. We report these as Likelihood-weighted loss and NLI-weighted SFT, respectively. We also constructed a High-likelihood-filtered SFT baseline by ranking the training examples according to their pretrained-model likelihood scores, retaining the highest-scoring 50%50\%, and applying standard SFT to this subset. Baseline outputs were evaluated across the five benchmarks using DeepSeek-R1-Distill-Llama-8B. The reported comparison in table 2 favors TIDE's joint criterion: its average F1F_1 is 0.4860.486, compared with 0.4120.412 for NLI-weighted SFT, 0.3240.324 for likelihood-weighted loss, and 0.2670.267 for high-likelihood-filtered SFT. The individual results help explain what each alternative captures and leaves out.

NLI-weighted SFT is the strongest alternative on average and achieves the highest MSMARCO score of 0.6330.633. This shows that faithfulness-only weighting is competitive in that setting. However, it does not account for pretrained-model compatibility: an answer can be supported by the context while having low likelihood under the pretrained model. TIDE includes both criteria, providing a plausible explanation for its stronger average performance across datasets.

Likelihood-weighted loss (DFT) performs relatively well on HotpotQA (0.6090.609), but poorly on WoW (0.0130.013). Its weighting signal reflects current token confidence, which does not by itself establish that a complete response is supported by the retrieved evidence. TIDE adds an explicit faithfulness signal and evaluates compatibility against a fixed pretrained model. These differences offer a possible explanation for the less consistent performance of confidence-only weighting.

High-likelihood-filtered SFT nearly matches TIDE on WoW (0.4140.414 versus 0.4160.416), but has a lower overall average. It selects examples using likelihood alone and then gives the retained responses equal weight. It therefore does not use faithfulness to distinguish among those responses. The hard cutoff also removes half the training examples, so the reduced amount of supervision may contribute to its results.

In TIDE, compatibility and faithfulness jointly shape the target distribution, and implicit distillation carries this criterion into training on the available human answers. The results are consistent with the usefulness of this joint design. These comparisons do not, by themselves, isolate the contribution of each component; our separate component ablations address that question more directly. We will include these comparisons and their interpretation in the revised manuscript.

Table 2. Illustrative comparison across five evaluation datasets. The highest value in each column is shown in bold.

ExperimentMSMARCOHotpotQANQWoWT-RExAverage
TIDE0.6180.7000.3750.4160.3210.486
NLI-weighted SFT0.6330.5700.3660.3450.1480.412
Likelihood-weighted loss0.4230.6090.3350.0130.2390.324
High-likelihood-filtered SFT0.4680.2300.1810.4140.0430.267

Notes. Reported baseline results are for MSMARCO-trained Qwen3-4B, judged by DeepSeek-R1-Distill-Llama-8B.

Reviewer_WwkQOpenReview ↗
Rating
3
Confidence
4
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

The paper addresses the dual challenges of distributional shift and unfaithfulness in RAG post-training. The authors propose TIDE, a rollout-free framework that formulates a value-based objective combining the length-normalized likelihood under the pretrained model and a faithfulness score to the retrieved context. They derive an optimal target LLM and introduce an "implicit distillation" method that reduces the forward-KL distillation objective to a tractable, value-weighted likelihood over observed human data. This avoids the need for expensive response rollouts or generate-and-filter procedures. The method is validated across five RAG benchmarks using Qwen3-4B and Qwen3-14B, demonstrating substantial improvements in answer-quality F1 and faithfulness compared to SFT and Self-Demo baselines.

Critical strengths
  • The formulation of the value-based objective and the derivation of the analytical target LLM provide a principled way to balance human supervision, pretrained distribution compatibility, and context faithfulness. The proof that the optimal target shifts probability mass away from low-compatibility and unfaithful responses (Theorem 4.1) is theoretically satisfying and well-articulated.
  • The evaluation across five diverse RAG benchmarks and two model scales is rigorous. The method shows particularly impressive cross-domain generalization (e.g., massive relative gains on T-REx and WoW compared to baselines) and maintains high faithfulness without relying on train-test alignment. The ablation studies (e.g., removing the faithfulness component, analyzing the scaling parameter α) effectively validate the design choices.
Critical weaknesses
  • While the "implicit distillation" framing and theoretical derivations are neat, the final tractable training objective (Equation 4) is mathematically equivalent to instance-weighted SFT, where the weights are determined by the exponentiated value function. Reward-weighted regression is a well-established technique in RL and alignment. The paper would benefit from a more transparent discussion positioning the core algorithmic contribution as the specific design of the RAG value function and its theoretical guarantees, rather than the "implicit distillation" mechanism itself.
  • The value function relies heavily on F(X, y), which is computed using an external NLI model (DeBERTa) during training. The robustness of TIDE to the noise, biases, or failure modes of this scorer is not evaluated.
Questions for authors
  • Since the final objective (Eq. 4) reduces to instance-weighted SFT, how much of the performance gain is driven by the specific choice of the NLI-based faithfulness scorer versus the pretrained likelihood term? Have you evaluated the sensitivity of the method to the quality or noise of the training-time DeBERTa scorer?
3 comments
AuthorsOpenReview ↗

Official Comment by Authors

We are grateful for the reviewer’s constructive feedback and insightful comments. In the following, we provide point-by-point responses to all your comments.

W1["the final tractable training objective (Equation 4) is mathematically equivalent to instance-weighted SFT.... "]

Thank you for this comment. The main distinction lies in the design principle behind TIDE, which addresses the joint requirements of RAG post-training. A general instance-weighted SFT objective allows different choices of weights, including exponentiated reward scores in reward-weighted regression (Hu et al., 2023), sequence-level importance ratios in iw-SFT (Qin and Springenberg, 2025), and binary weights representing the reward-based filtering used in RAFT (Dong et al., 2023), but does not by itself specify how to balance faithfulness to retrieved evidence and compatibility with the pretrained model.

In TIDE, these RAG requirements guide the design of the value function. We want responses to be faithful to the retrieved context and remain compatible with the pretrained model, while retaining human supervision without generating self-demonstrations. We therefore start by formulating an optimized target distribution. The pretrained-likelihood term favors in-distribution responses, while the faithfulness term measures support in the retrieved context.

Implicit distillation then gives us a way to learn from this target using the available human answers. It serves two purposes: making training tractable without explicitly estimating the underlying human-response distribution, and transferring the target's compatibility and faithfulness preferences to the trained model. Compatibility therefore enters through the likelihood term in the target and through learning to approximate that target.

Our objective thus has some connections to weighted SFT but fundamentally they are different. The distinction lies in the specific demand of the RAG in out-of-distribution and faithfulness problems. As shown below, the weighted SFT method cannot solve the same problem in RAG as our method. This procedure lets us train without explicitly constructing the target distribution or generating replacement responses unlike self-demonstrations. Theoretically, TIDE provides an explicit upper bound on the probability of responses with both low pretrained-model compatibility and low faithfulness scores. Generic instance weighting alone does not imply this RAG-specific guarantee; it follows from our target construction and distillation analysis. We would update this detailed comparison of our method with Weighted-SFT in our manuscript.

Empirically, motivated by your suggestion, we add a popular weighted SFT baseline (Dynamic Fine-Tuning (DFT)) for comparison and we report the results in the following:

Table 1. Comparison of TIDE and DFT across model sizes and training datasets. Scores are rounded to three decimal places. The higher F1F1 within each model-size pair is shown in bold.

TrainEvalQwen3-4B TIDEQwen3-4B DFTQwen3-14B TIDEQwen3-14B DFT
MSMARCOMSMARCO0.5880.4230.5940.638
MSMARCOHotpotQA0.7220.6090.7280.626
MSMARCONQ0.3500.3350.3110.384
MSMARCOWoW0.3020.0130.2900.288
MSMARCOT-REx0.3140.2390.3270.249
NewsQAMSMARCO0.5840.3400.5590.420
NewsQAHotpotQA0.7130.2040.7170.292
NewsQANQ0.3530.1500.3080.210
NewsQAWoW0.3070.2790.3370.191
NewsQAT-REx0.2920.1090.3100.164
Overall Average0.4520.2700.4480.346

TIDE achieves higher overall F1F_1 than DFT at both model sizes: 0.4520.452 versus 0.2700.270 for Qwen3-4B and 0.4480.448 versus 0.3460.346 for Qwen3-14B. It outperforms DFT in all ten training--evaluation combinations for Qwen3-4B and eight of ten for Qwen3-14B. These results are consistent with the motivation for our RAG-specific weighting criterion. DFT weights individual tokens using the current model's probabilities, but high token probability alone does not establish that a response is supported by the retrieved evidence. In TIDE, the likelihood term provides a fixed reference for compatibility, while the faithfulness term adds an explicit signal of evidence support.

Through implicit distillation, these weights connect the training objective to a target that jointly accounts for both RAG requirements. This provides a plausible explanation for TIDE's stronger overall performance: its training emphasis reflects both pretrained-model compatibility and evidence support, rather than current token confidence alone. The comparison supports the usefulness of our method over Weighted-SFT.

AuthorsOpenReview ↗

Official Comment by Authors

W2["TIDE’s reliance on an external NLI faithfulness scorer and whether its performance is robust to the choice of that scorer."]

Thank you for raising this point. Following your suggestion, we evaluated a Qwen3-4B variant initialized through MSMARCO training using ynie/roberta instead of the original DeBERTa-v3 scorer. Table 2 reports the results.

Table 2. Sensitivity to the training-time NLI scorer for TIDE.

Evaluation datasetTIDE (DeBERTa)TIDE (YNIE)
MSMARCO0.5878020.589761
HotpotQA0.7215710.726000
NQ0.3501390.340166
WoW0.3015990.290977
T-REx0.3144290.312429
Average0.4551080.451866

Replacing DeBERTa-v3 with YNIE produces similar results, suggesting that TIDE's gains in this setting are not specific to the original NLI scorer. Both configurations retain higher average F1F_1 than standard SFT.

The average F1F_1 changes from 0.45510.4551 with DeBERTa-v3 to 0.45190.4519 with YNIE, a decrease of approximately 0.320.32 F1 points. Performance improves slightly on MSMARCO and HotpotQA and decreases slightly on NQ, WoW, and T-REx. This comparison supports robustness across the two NLI scorers tested. We will include these results and the accompanying discussion in the revised manuscript. We would reflect this result and discussion in our updated manuscript.

Q1.["Since the final objective (Eq. 4) reduces to instance-weighted SFT, how much of the performance gain is driven by the specific choice of the NLI-based faithfulness scorer versus the pretrained likelihood term?"]

Thank you for pointing this out. Our paper includes separate ablations removing the faithfulness term (Table 3) and the pretrained-likelihood term (Figure 1(a)). We report the results for removing the faithfulness term in the following:

Table 3. Supported faithfulness for Qwen3-14B. The variant without faithfulness removes F(X,y)F(X,y) while retaining pretrained-model compatibility. The higher score within each training setting is shown in bold.

TrainMethodMSMARCOHotpotQANQWoWT-RExAverage
MSMARCOTIDE0.9720.9850.9860.8580.9440.949
MSMARCOTIDE w/o faithfulness0.9870.9380.9740.9110.5880.880
NewsQATIDE0.9720.9800.9820.8460.9340.943
NewsQATIDE w/o faithfulness0.9550.9310.9590.6170.1320.719

Overall average: TIDE: 0.946 TIDE w/o faithfulness: 0.799

Removing the faithfulness term reduces overall supported faithfulness from 0.9460.946 to 0.7990.799, a decrease of approximately 14.714.7 percentage points. Full TIDE performs better in eight of the ten train--evaluation settings. The difference is particularly pronounced on T-REx, where supported faithfulness drops from 0.9440.944 to 0.5880.588 under MSMARCO training and from 0.9340.934 to 0.1320.132 under NewsQA training. These results indicate that pretrained likelihood alone does not provide the same level of grounding in the retrieved evidence.

The pretrained-likelihood term also contributes to performance. In Figure 1(a) in the updated manuscript, removing this term while retaining faithfulness lowers answer-quality F1F_1 from approximately 0.450.45 to 0.360.36 and increases KL divergence from the pretrained model. This supports the role of pretrained compatibility in preserving useful model behavior while improving answer quality. Together, these results support complementary contributions from both terms: faithfulness improves grounding, while pretrained compatibility helps maintain answer quality and limit distributional drift. We will expand the ablation discussion in the revised manuscript to explain the complementary roles of faithfulness and pretrained-model compatibility, supported by these results.

AuthorsOpenReview ↗

Official Comment by Authors

Q2["Have you evaluated the sensitivity of the method to the quality or noise of the training-time DeBERTa scorer?"]

Thank you for raising this point. Following your suggestion, we evaluated TIDE's robustness to noise in the faithfulness scores used during training. We conducted this experiment using Qwen3-4B with an initial MSMARCO training stage. The noisy NLI procedure is described below.

Let Fi∈[0,1]F_i \in [0,1] denote the original DeBERTa-v3 entailment score for training example ii. We independently sampled bi∼Bernoulli⁡(η)b_i \sim \operatorname{Bernoulli}(\eta) and defined

F~i=(1−bi)Fi+bi(1−Fi),η=0.05.\widetilde{F}_i = (1-b_i)F_i+b_i(1-F_i),\qquad \eta=0.05.

This replaces FiF_i with 1−Fi1-F_i for approximately 5% of training examples, leaving the remaining scores unchanged. We then used F~i\widetilde{F}_i in place of FiF_i when computing the training weights. We will include this ablation and its results in the revised manuscript.

Table 4. Training-time scorer-noise sensitivity for TIDE.

Evaluation datasetNLI(DeBERTa)Noisy NLI (η=0.05\eta=0.05)
MSMARCO0.5878020.583720
HotpotQA0.7215710.723571
NQ0.3501390.347139
WoW0.3015990.304198
T-REx0.3144290.309714
Average0.4551080.453669

The noisy-scorer configuration in Table 4 achieves an average F1 of 0.45370.4537, compared with 0.45510.4551 for the original configuration, a negligible decrease of approximately 0.140.14 percentage points. These results suggest that TIDE is robust to the tested 5%5\% random corruption of DeBERTa-v3 faithfulness scores during the initial training stage. We will incorporate these results and the accompanying discussion into the revised manuscript.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.