- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes DiSCo, a training-free decoding method that balances parametric and contextual knowledge in RAG. It uses relative confidence to decide whether to increase or reduce the influence of retrieved context, and Jensen–Shannon divergence determines how strongly the logits should be adjusted. These two signals are combined using a nine-rule Takagi–Sugeno fuzzy model. The authors show improvements in accuracy and robustness.
Critical strengths
[S1] Interesting problem setting with a practical approach.
Separating the direction and strength of contextual influence sounds interesting to handle knowledge conflicts in RAG. DiSCo offers a training-free solution that can be applied to existing LLMs without additional training.
[S2] Thorough experiments on knowledge conflicts.
The experiments cover multiple models and QA datasets, with additional settings that vary context reliability and the amount of conflicting information. The ablation studies also show that both relative confidence and distributional disagreement contribute to performance.
Critical weaknesses
[W1] Limited methodological novelty.
I think DiSCo mainly builds on existing ideas, such as adaptive contrastive decoding and fuzzy inference. While separating the direction and strength of contextual influence is a useful idea, it is unclear how much novelty the nine-rule fuzzy model adds. In Table 4, DiSCo-Simple performs similarly to DiSCo even without fuzzy inference. This makes difficult to see whether the additional complexity of the fuzzy model is really necessary.
[W2] Lack of justification for the control mechanism.
The method uses relative confidence to decide which knowledge source is more reliable and distributional disagreement to control the adjustment strength. However, the paper does not directly verify these assumptions or provide a clear theoretical explanation for them. The rule matrix in Eq. (10) also seems to be designed based on heuristics, without explaining why those specific values were chosen. As these components are central to DiSCo, stronger justification is needed to support the proposed design.
Questions for authors
[Q1] What is the main methodological novelty?
- Please explain what is fundamentally new compared to existing adaptive decoding and fuzzy inference methods.
- Also, I wonder if there is a specific setting where the nine-rule fuzzy model has a clear advantage over simpler alternatives, since DiSCo-Simple achieves similar performance in Table 4.
[Q2] Why are these control signals and rules used?
- What is the theoretical or empirical basis for using relative confidence to identify the more reliable knowledge source and distributional disagreement to determine the adjustment strength?
- How were the specific values in the rule matrix of Eq. (10) chosen? A more detailed explanation beyond the current heuristic design and parameter sensitivity analysis need to justify these choices.