acceptodds
Withdrawn from ICLR 2027

DiSCo: Balancing Parametric and Contextual Knowledge by Decoupling the Direction and Strength of Contextual Influence

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Retrieval-augmented generation (RAG) integrates two knowledge sources: parametric knowledge encoded in language models and contextual knowledge provided by retrieved evidence. Neither source is consistently more reliable than the other: parametric knowledge may be incomplete or outdated, while retrieved evidence may be incorrect or noisy. Consequently, decoding methods that consistently favor one source can perform poorly when the other is more reliable. To address this challenge, we propose DiSCo, an adaptive decoding framework that regulates contextual influence by decoupling its direction and strength at each decoding step. DiSCo computes next-token distributions with and without context, derives relative confidence from the difference in their top-1 log probabilities, and measures distributional disagreement using Jensen–Shannon divergence. Relative confidence determines whether the retrieval-induced logit shift should be suppressed or amplified, while distributional disagreement controls the adjustment strength. To implement this decoupled control, DiSCo maps the two signals to soft linguistic states and aggregates them through a nine-rule Takagi–Sugeno fuzzy model, producing a token-wise coefficient that scales the retrieval-induced logit shift relative to the parametric logits. Experiments on four QA datasets and one knowledge-conflict dataset show that DiSCo outperforms both standard and state-of-the-art decoding methods on average while remaining robust under varying context reliability and conflict proportions.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Large Language Models, Retrieval-Augmented Generation, Knowledge Conflict, Adaptive Decoding, Knowledge Reliance

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_vbASOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes DiSCo, a training-free decoding method that balances parametric and contextual knowledge in RAG. It uses relative confidence to decide whether to increase or reduce the influence of retrieved context, and Jensen–Shannon divergence determines how strongly the logits should be adjusted. These two signals are combined using a nine-rule Takagi–Sugeno fuzzy model. The authors show improvements in accuracy and robustness.

Critical strengths

[S1] Interesting problem setting with a practical approach.

Separating the direction and strength of contextual influence sounds interesting to handle knowledge conflicts in RAG. DiSCo offers a training-free solution that can be applied to existing LLMs without additional training.

[S2] Thorough experiments on knowledge conflicts.

The experiments cover multiple models and QA datasets, with additional settings that vary context reliability and the amount of conflicting information. The ablation studies also show that both relative confidence and distributional disagreement contribute to performance.

Critical weaknesses

[W1] Limited methodological novelty.

I think DiSCo mainly builds on existing ideas, such as adaptive contrastive decoding and fuzzy inference. While separating the direction and strength of contextual influence is a useful idea, it is unclear how much novelty the nine-rule fuzzy model adds. In Table 4, DiSCo-Simple performs similarly to DiSCo even without fuzzy inference. This makes difficult to see whether the additional complexity of the fuzzy model is really necessary.

[W2] Lack of justification for the control mechanism.

The method uses relative confidence to decide which knowledge source is more reliable and distributional disagreement to control the adjustment strength. However, the paper does not directly verify these assumptions or provide a clear theoretical explanation for them. The rule matrix in Eq. (10) also seems to be designed based on heuristics, without explaining why those specific values were chosen. As these components are central to DiSCo, stronger justification is needed to support the proposed design.

Questions for authors

[Q1] What is the main methodological novelty?

  • Please explain what is fundamentally new compared to existing adaptive decoding and fuzzy inference methods.
  • Also, I wonder if there is a specific setting where the nine-rule fuzzy model has a clear advantage over simpler alternatives, since DiSCo-Simple achieves similar performance in Table 4.

[Q2] Why are these control signals and rules used?

  • What is the theoretical or empirical basis for using relative confidence to identify the more reliable knowledge source and distributional disagreement to determine the adjustment strength?
  • How were the specific values in the rule matrix of Eq. (10) chosen? A more detailed explanation beyond the current heuristic design and parameter sensitivity analysis need to justify these choices.
Reviewer_UNyCOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This paper is well-motivated as it highlights the possible inaccuracies of retrieved contexts. It also highlights the importance of the balance between parametric and contextual information in retrieval-augmented generation (RAG). The proposed method is simple and intuitive. In particular, the use of relative confidence for directing the influence and distributional disagreement for controlling the scale of such influence makes this method easy to understand. The most interesting part is that the method does not involve any extra training.

Critical strengths
  1. This paper is well-motivated as it highlights the possible inaccuracies of retrieved contexts. It also highlights the importance of the balance between parametric and contextual information in retrieval-augmented generation (RAG).

  2. The proposed method is simple and intuitive. In particular, the use of relative confidence for directing the influence and distributional disagreement for controlling the scale of such influence makes this method easy to understand. The most interesting part is that the method does not involve any extra training.

  3. The experiments include 3 different language models, 4 question answering datasets, and 1 knowledge conflict dataset.

Critical weaknesses
  1. In Section 3.2.1 (Equations 3 and 4), DiSCo exploits the confidence of the top-1 token to determine which knowledge source is more reliable. However, confidence is not synonymous with accuracy; the model could have generated a wrong answer with high confidence due to misleading context. Although Figure 3 and Appendix C illustrate some benefits of relative confidence, these experiments did not test whether relative confidence can determine the correct knowledge source. This problem is vital because the method uses relative confidence as a signal to decide how to adjust.

  2. In Section 3.2.3 (Equation 13), DiSCo combines two logit distributions using dynamic coefficient. The authors do not provide any baseline experiments with fixed weights such as 0.3, 0.5, and 0.7. Sometimes merely lowering the importance of unreliable context could be enough to increase accuracy. Thus, it is not clear how much improvement can be attributed to the proposed method of dynamic coefficients.

  3. In Section 4.2.2 and Appendix A.1, category C1 includes questions for which the parametric model gives right answers and C2 – questions for which the parametric model gives wrong answers. The ratio of C1 to C2 is changed to study the effect of context reliability, but at the same time, it changes the set of questions used in experiments. Therefore, effects of context reliability are mixed with effects of different sets of questions. The results in Table 2 should be considered accordingly. It is necessary to perform the same experiment using the same questions under right and wrong contexts.

  4. Further discussion on the initialization method is needed. The paper discusses calculating the mean and the standard deviation of the control signals based on the question token in Section 3.2.1 (Equation 6). But it is done when the answer is being generated. Question token and answer token distributions may differ from each other. The number of initialization samples is studied in Appendix D. But it does not discuss the difference between them.

Questions for authors
  1. Can the authors test whether the relative confidence can correctly identify the reliability of the knowledge source? Can the model be highly confident to a wrong answer?

  2. Can the authors include DiSCo with fixed weights in the comparison? This can be done under the same experimental settings.

  3. Can the authors report the results of the same questions under both correct and incorrect contexts?

  4. Why does the method use statistics from question tokens? Can the authors report the results using statistics collected from answer tokens?

Reviewer_hww4OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper proposes DiSCo, a decoding method for balancing a language model’s parametric knowledge with retrieved context. At each token, it compares predictions with and without context. The difference in their top-token confidence is used to choose whether to suppress or amplify the contextual logit shift, while Jensen–Shannon divergence controls the strength of that adjustment. A nine-rule fuzzy model combines the two signals. DiSCo achieves the highest average accuracy on three model backbones across four public QA datasets and performs well as context reliability decreases in a synthetic conflict benchmark.

Critical strengths
  • The paper addresses a practical RAG problem: retrieved context can correct an outdated model answer, but it can also introduce a false one.

  • The method is training-free and has a clear token-level formulation. The separate analyses of relative confidence and distributional disagreement help explain the intended roles of the two signals.

  • The controlled ConFiQA experiments expose a meaningful trade-off. On Llama3-8B-Instruct, DiSCo remains at 73.5% accuracy when all supplied contexts conflict with the answer, compared with 58.1% for regular decoding.

Critical weaknesses
  1. The central assumption that higher top-token confidence identifies the more reliable knowledge source is not directly tested. A false passage can produce a confident contextual prediction, while a correct branch can be less confident. The paper reports downstream accuracy but not how often the relative-confidence signal chooses the correct adjustment direction.

  2. Improvements on the four public QA datasets are modest: DiSCo exceeds regular decoding by 0.7–1.2 average accuracy points across the three backbones. Table 1 marks some individual differences as significant, but the paper does not specify the statistical test, report uncertainty for the average gains, or discuss multiple comparisons. This makes the consistency of the main improvement difficult to assess.

  3. The evidence for the fuzzy aggregation mechanism is limited. DiSCo-Simple achieves nearly the same average accuracy and slightly outperforms DiSCo on Mistral. Removing distributional disagreement reduces average accuracy by only 0.3–0.7 points in Table 3. The text also states that DiSCo outperforms this ablation throughout T1–T5, although several entries are ties or favor the ablation.

  4. The strongest robustness evidence comes from ConFiQA scenarios constructed by mixing synthetic correct and conflicting contexts. It remains unclear whether the same signal distinguishes reliable from misleading passages in ordinary retrieval, where evidence may be partially relevant, ambiguous, outdated, or internally inconsistent. DiSCo also gives up accuracy when context is fully reliable: in S1 it scores 76.3%, versus 79.8% for regular decoding and 84.6% for CAD.

  5. DiSCo takes 4.67 seconds per example in the reported compute test, compared with 2.37 seconds for regular decoding. The paper emphasizes its small overhead relative to other two-branch methods, but the approximately doubled latency relative to standard RAG matters when judging the practical value of the small average gains.

Questions for authors
  1. On C1 and C2 examples, how accurately do the sign of relative confidence and the resulting coefficient identify which branch is correct? Can the authors report these diagnostics separately from final-answer accuracy?

  2. Can the authors evaluate naturally occurring retrieval conflicts and partially relevant or outdated passages, rather than relying only on synthetic conflict contexts?

  3. What statistical test produces the significance marks in Table 1? Can the authors provide paired confidence intervals for the average improvements and account for multiple comparisons?

  4. Were baseline coefficients tuned with the same validation budget as DiSCo’s rule table? How does DiSCo compare with a tuned fixed coefficient and simpler two-signal functions?

  5. How does accuracy vary with latency or decoding cost? In particular, does the gain over regular decoding remain worthwhile in a matched-compute comparison?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.