- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes TIDE, a rollout-free RAG post-training framework that derives a closed-form target LLM via KL-regularized value optimization and then trains on fixed human responses through an equivalent weighted-likelihood objective. The value function combines pretrained-model compatibility and context faithfulness. Experiments on five RAG benchmarks with Qwen3-4B/14B show consistent gains over SFT and Self-Demo.
Critical strengths
The problem is well-motivated and practically relevant: jointly preserving pretrained behavior and enforcing faithfulness under fixed human supervision. The theoretical development is clean—Theorem 3.1 gives a closed-form target, and Corollary B.2 establishes the implicit-distillation equivalence. The rejection-sampling interpretation and the acceptance-probability comparison with Self-Demo provide useful intuition. The method is simple and rollout-free, which is a genuine advantage over generate-and-filter pipelines.
Critical weaknesses
-
Static weighting lacks empirical validation. The weights are precomputed from the frozen pretrained model and a fixed NLI scorer, and remain fixed throughout training. The authors argue in rebuttal that this follows directly from the fixed target distribution . While this is theoretically coherent, the paper never empirically verifies whether the static approximation remains valid as drifts from . The rebuttal promises an ablation with periodically refreshed weights; this would directly address the concern.
-
Missing comparison with prior weighted-SFT methods. The paper compares only against SFT and Self-Demo. Sample weighting for distribution-aware fine-tuning has been explored before (e.g., FLOW, VAR, ADAPT). Without these baselines, it is unclear whether TIDE’s specific value function adds value beyond perplexity-based weighting, or whether the faithfulness term provides additional gains. The rebuttal agrees to add FLOW and a reward-weighted SFT baseline, which would clarify TIDE’s contribution.
-
No empirical analysis of the weight distribution. The method hinges entirely on , yet the paper never shows the distribution of weights, their correlation with and , or qualitative examples of high- and low-weight responses. Without this, it is impossible to tell whether TIDE effectively trains on a small subset, or whether the weights are nearly uniform. The rebuttal agrees to add histograms, scatter plots, and qualitative examples.
Questions for authors
I encourage the authors to further clarify the above concerns and discuss how they plan to address these limitations in future revisions.
5 comments
Official Comment by Authors
W3[" The method hinges entirely on , yet the paper never shows the distribution of weights...."]
We thank the reviewer for raising this point. Following your suggestion, Table 2 below summarizes the distribution of mean-normalized training weights assigned by TIDE. A normalized weight of 1 corresponds to the average training weight, while values below or above 1 indicate examples receiving less or more influence, respectively. The weights span several orders of magnitude. The distribution contains a broad low-weight group and a smaller high-weight group. This indicates that TIDE down-weights less suitable examples while assigning greater importance to responses that are both likely under the pretrained model and faithful to the retrieved context. All examples retain nonzero weight.
Table 2. Summary of the training-weight distribution for Qwen3-14B on MSMARCO with . The normalized weight is , where is the mean raw weight.
| Statistic | Value |
|---|---|
| Number of training examples | 72,553 |
| Mean raw weight | 0.254314 |
| Mean normalized weight | 1 |
| 5th percentile of normalized weight | 0.001714 |
| Median normalized weight | 0.037617 |
| 95th percentile of normalized weight | 5.401831 |
| 99th percentile of normalized weight | 6.170301 |
We sort the training examples by their assigned weights in ascending order and divide them into ten groups of approximately equal size. Each group therefore contains approximately one-tenth of the training examples. The first decile contains the lowest-weight examples, while the last contains the highest-weight examples. Table 3 below shows that the highest-weight decile has greater mean NLI faithfulness than the lowest-weight decile. However, the trend is not monotonic across all deciles. This is consistent with the joint construction of the weights: an example’s weight depends on both its pretrained-model compatibility score and its faithfulness score. Consequently, ordering examples by weight need not produce the same ordering as sorting them by faithfulness alone.
Table 3. Mean NLI faithfulness and mean compatibility across training-weight deciles. Values reproduce the rounded annotations in the figures.
| Weight decile | Mean NLI faithfulness | Mean compatibility |
|---|---|---|
| 1 (lowest weights) | 0.536 | |
| 2 | 0.654 | |
| 3 | 0.756 | |
| 4 | 0.844 | |
| 5 | 0.886 | |
| 6 | 0.910 | |
| 7 | 0.893 | |
| 8 | 0.799 | 0.033 |
| 9 | 0.830 | 0.358 |
| 10 (highest weights) | 0.972 | 0.730 |
| Overall mean | 0.808 | 0.112 |
Using the same weight deciles, the compatibility column in Table 3 shows a consistent increase in pretrained model compatibility from the lowest-weight to the highest-weight groups. This indicates that the likelihood-derived component is closely aligned with the empirical weight ordering.
The joint construction of the weights does not require compatibility and faithfulness to exhibit identical trends across these groups. A greater compatibility score can compensate for a lower faithfulness score when determining an example’s weight. Consequently, compatibility can increase consistently across weight deciles while faithfulness varies non-monotonically. The increasing compatibility trend is an empirical observation in this dataset, rather than a general requirement that sorting by the combined weight must also sort either component individually.
Official Comment by Authors
W3["qualitative examples of high- and low-weight responses"]
Thank you for this comment. From a shortlist of 100 training examples, we selected two high-weight and two low-weight cases.
Table 4. Raw training weights for the selected examples. Values are rounded.
| Example | Weight |
|---|---|
| Casement windows (high) | 1.473577 |
| Neuron structure (high) | 0.968264 |
| DWIB meaning (low) | 0.000874 |
| Tooth-filling cost (low) | 0.000792 |
Table 5. Selected high- and low-weight training responses. The interpretations are based on the evidence and our qualitative inspection.
| Example | Field | Question, evidence, and training response |
|---|---|---|
| Casement windows — High weight | Question: | what is a casement windows |
| Context excerpt: | A casement is a window that is attached to its frame by one or more hinges. Casement windows are hinged at the side. | |
| Original training answer: | A casement is a window that is attached to its frame by one or more hinges. Casement windows are hinged at the side. | |
| Interpretation: | The answer directly addresses the question using information stated in the context. This response receives a high training weight. | |
| Neuron structure — High weight | Question: | structure of a typical neuron |
| Context excerpt: | A typical neuron consists of a cell body (soma), dendrites, and an axon. | |
| Original training answer: | A typical neuron consists of a cell body (soma), dendrites, and an axon. | |
| Interpretation: | The answer lists the parts of a neuron given in the context and directly answers the question. It receives a high training weight. | |
| DWIB meaning — Low weight | Question: | what does dwib mean |
| Context excerpt: | Acronym Attic: Searching over 3 million acronyms, abbreviations, and initialisms. | |
| Original training answer: | The World’s most comprehensive professionally edited abbreviations and acronyms database. | |
| Interpretation: | The answer describes a database without explaining what DWIB means. The context also does not support the claims that the database is the world’s most comprehensive or professionally edited. This response receives a low training weight. | |
| Tooth-filling cost — Low weight | Question: | how much does a tooth filling cost |
| Context excerpt: | The cost for amalgam fillings averages approximately $110 to $200 per filling. The cost for resin-based composite fillings, which is determined by the number of tooth surfaces on which the material is placed, ranges on average from $135 to $240 per filling. | |
| Original training answer: | $75 to $145 per | |
| Interpretation: | The answer gives a price range that is not supported by either range in the context and ends with an incomplete phrase. It receives a low training weight. |
The two high-weight responses directly answer their questions using the supplied evidence. The two low-weight responses contain unsupported claims or an unsupported, incomplete answer. These selected cases illustrate how the assigned weights differ. We will include these qualitative examples and their interpretations in the revised manuscript.
Official Comment by Authors
We are grateful for the reviewer’s constructive feedback and insightful comments. In the following, we provide point-by-point responses to all your comments.
W1["The rebuttal promises an ablation with periodically refreshed weights"]
We would like to clarify that the motivation behind why we keep the weight fixed as the pretrained model (πθ0) serves as a fixed reference distribution, and its likelihood addresses one of the aims in our work, i.e. how compatible each response is with the original pretrained distribution. As the trainable model (πθ) evolves, the pretrained likelihood should remain unchanged, since its purpose is precisely to preserve in-distribution with respect to the original distribution. Therefore, periodically refreshing the weights is not necessary for our proposed objective and would represent a different training procedure.
W2["Missing comparison with prior weighted-SFT methods."]
Thank you for suggesting this comparison. We have added a popular weighted-SFT baseline, i.e. Dynamic Fine-Tuning (DFT) for both Qwen3-4B and Qwen3-14B, each trained separately on MSMARCO and NewsQA and evaluated across five benchmarks. DFT weights supervised token losses using the current model’s token probabilities, with stop-gradient applied to the weights.
Table 1. Comparison of TIDE and DFT across model sizes and training datasets. Scores are rounded to three decimal places. The higher F1 score within each model-size pair is shown in bold. Overall averages are computed across the ten training–evaluation combinations.
The reported results favor TIDE overall. DFT achieves average F1 scores of 0.270 and 0.346 for Qwen3-4B and Qwen3-14B, respectively, compared with 0.452 and 0.448 for TIDE. TIDE achieves higher scores in all ten training–evaluation combinations for Qwen3-4B and eight of ten for Qwen3-14B.
A plausible explanation for this difference is the information used to guide training. DFT emphasizes tokens according to the current model’s confidence, without explicitly assessing whether the complete response is supported by the retrieved evidence. TIDE combines response-level faithfulness with compatibility measured against the frozen pretrained model. These criteria jointly shape its optimized target and enter training through implicit distillation. The results support the usefulness of this RAG-specific design compared with the tested DFT baseline, although this comparison alone does not isolate the contribution of each component.
We will include the additional baseline, results, and discussion in the revised manuscript.