acceptodds
Under review as a conference paper at ICLR 2027

CARD: Calibrated Adaptive Reliance Decoding in LLM-Augmented Search

Abstract

Search engines augmented with large language models (LLMs) synthesize retrieved web content into summarized responses. When handling open-ended queries, LLMs implicitly select and rank candidate items, effectively functioning as recommenders. However, the LLM's reliance on parametric versus retrieved knowledge during this process is difficult to control: over-reliance on parametric knowledge leads to systematic favoritism toward known content, while over-reliance on retrieved content makes it vulnerable to manipulation of the retrieved documents. To investigate this problem, we first define (PRS) to quantify the strength of parametric knowledge reliance in LLM recommendation rankings. We validate this metric through a Latin-square-based controlled experiment across five LLMs, confirming systematic parametric preference. We then propose CARD (alibrated daptive eliance ecoding), a pluggable inference-time method that leverages PRS to enable bidirectional adaptive control over knowledge reliance. CARD measures KL divergence of token probability distributions after context insertion to derive an adaptive intervention strength on external knowledge, and calibrates the anchoring weight of the base predictions using the model-level PRS. Experiments on three open-source LLMs show that CARD mitigates parametric knowledge bias, narrowing the ranking gap between parametric and non-parametric items by 55–60%, and defends against adversarial attacks on retrieved content—suppressing retrieval poisoning to near-random rankings and neutralizing up to 95% of black-hat search engine optimization (SEO) attack effects. Our code is available at https://anonymous.4open.science/r/CARD-1A64.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
LLM-Augmented Search, Parametric Knowledge Reliance, Adversarial Defense, Inference-Time Decoding

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 5 reviews, above 27% of the others.
Reviewer_7ArhOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper looks at how LLM-based search engines rank products, and argues that models favor items they already know from pretraining over unfamiliar ones. To quantify this, the authors propose a simple score (PRS) based on a controlled ranking experiment. They then propose a decoding method, CARD, that contrasts predictions with and without the retrieved context. Depending on the sign of a parameter, it either amplifies the context to reduce this bias, or suppresses it to defend against some poisoning and SEO-style attacks.

Critical strengths

The question is relevant: LLM-based search engines do act as recommenders, and whether they favor well-known brands over new ones, or can be steered by injected content, matters in practice. The controlled ranking setup, which balances descriptions and context positions, is a reasonable way to probe this.

Critical weaknesses
  1. The PRS metric comparison is between the model’s favorite real products and obviously fake names like “S Brand”, so the gap may just reflect that the names look fake(the appendix itself shows the gap shrinks with more realistic fictional names). The main result cannot support the main claims of the paper.

  2. Why preferring a known brand, given the same description, should be seen as a bias.

  3. The method seems to be close to CAD and CK-PLUG with an extra scaling term. The ablation does not re-tune the other parameter, so it does not really show that this term helps.

  4. In the defense results: the attacked item is always a fake-named product, so suppressing the context pushes it down, whether it was attacked or not; The direction is also set by hand, which assumes one already knows an attack is present.

Questions for authors
  1. If in a real deployment, how to decide whether to amplify or suppress the retrieved context, given that it is usually not known in advance whether the content has been attacked?

  2. Why preferring a well-known brand, when the descriptions are the same, should be treated as a bias to remove rather than a reasonable choice from the user’s point of view.

3 comments

We thank the reviewer for their time and insightful feedback. The comments are invaluable for improving our work and clarifying our contributions. We believe there may be some misunderstandings about our paper, and we clarify and respond to each point below.

W1: Placeholder Names

1. PRS under the two name constructions. In Appendix B of the revised version, we add quartile plots of PRS for LLaMA3.1-8B, Qwen3-8B, Gemma3-12B, and Gemini-3.1 when the non-parametric-knowledge items are represented by placeholders and by fictional-but-plausible names, respectively. Except for LLaMA, the interquartile ranges of PRS under the two kinds of names largely overlap, with no notable gap. We conjecture that the difference on LLaMA is related to its earlier release and weaker capability, and is an isolated case. This shows that the gap measured by PRS consistently exists under both kinds of names.

2. CARD under fictional-but-plausible names. We evaluate CARD on fictional-but-plausible names, directly reusing the MM and β\beta from the placeholder setting, without re-tuning. As shown in Table 1, with fictional-but-plausible names, the PRS of Vanilla remains clearly positive on all three models, showing that this gap does not arise merely because the names look fake; CARD's PRS is lower than that of Vanilla and all baselines on all three models, showing that CARD's debiasing effect does not depend on placeholders. Therefore, the main conclusions of the paper still hold under fictional-but-plausible names.

Table 1: PRS under fictional-but-plausible names.

MethodLLaMA3.1-8BQwen3-8BGemma3-12B
Vanilla0.16500.14100.1860
Prompt0.14530.13780.1822
CK-PLUG (Ada.)0.12040.16280.2522
CK-PLUG (Fixed)0.12170.14110.1890
CAD0.09100.10220.0845
CARD (Ours)0.07240.09470.0752

3. Why the main experiments still use placeholders. We use placeholders to represent non-parametric-knowledge items. This is a deliberate design: it guarantees that these items are genuinely absent from the model's parametric knowledge, which is the prerequisite for isolating the factor of "whether an item belongs to parametric knowledge". LLM-generated fictional names that read more naturally cannot achieve this: because LLMs are trained on large-scale web corpora, even low-probability sampling may produce names that coincide with real entities. To quantify this, we prompted GPT-5.5 to generate 400 (50 × 8) natural-sounding fictional item names and searched each of them on the web: 214 (53.5%) of them have real-world counterparts. Therefore, names that look fictional cannot be guaranteed to lie outside the model's parametric knowledge, whereas placeholders provide a clean, verifiable control.

W2 & Q2: Why Favoring Well-Known Items Counts as Bias

The bias in this paper refers to the systematic preference in LLM recommendation rankings caused by over-reliance on parametric knowledge. This reliance is implicit: users see only the final ranking, and can neither know how much of it comes from the model's parametric knowledge nor control it. This amounts to the LLM making the decision to "favor well-known items" on the user's behalf. We agree that, for some users, favoring well-known items can be a reasonable choice. Therefore, CARD does not forcibly eliminate this reliance, but makes it controllable: with M>0M > 0, the ranking relies more on the retrieved content; with M<0M < 0, the ranking relies more on the model's internal knowledge. Whether to favor well-known items is decided by users according to their own needs, rather than implicitly by the model.

W3: Differences from CAD and CK-PLUG, and the Ablation

1. Differences from CAD and CK-PLUG. As stated in the Introduction and Appendix D, CARD is fundamentally different from CK-PLUG. Below, we further explain three core differences, also in relation to CAD. The output of CAD can be written as qbase+α⋅qcontq_{\mathrm{base}} + \alpha \cdot q_{\mathrm{cont}}; the output of CARD is (1−∣βm∣)⋅qbase+α∗(M,κ)⋅qcont(1-|\beta_m|) \cdot q_{\mathrm{base}} + \alpha^{\ast}(M,\kappa) \cdot q_{\mathrm{cont}} (Eq. 9). The three differences are as follows:

  • Adaptive intervention strength. CAD uses a fixed α\alpha throughout; CK-PLUG determines, based on the distribution shift, whether the current decoding step involves a knowledge conflict, and then decides whether to intervene. In contrast, CARD continuously determines the intervention strength from the KL divergence κ\kappa at each step, upper-bounded by ∣M∣|M| (Eq. 8): the greater the influence of the retrieved content on the prediction, the more gently the intervention grows, which avoids over-amplification.
  • PRS calibration. βm\beta_m is exactly the model's PRS, measured in the Latin-square controlled experiment, and is a model-level empirical value. CARD uses (1−∣βm∣)(1-|\beta_m|) to calibrate the anchoring weight of qbaseq_{\mathrm{base}}: the larger the model's reliance bias, the lower the weight of qbaseq_{\mathrm{base}} in the final output. Neither CAD nor CK-PLUG has this model-level calibration.
  • Designed for multi-candidate ranking. CK-PLUG targets knowledge conflicts in factual question answering, aiming to decide whether a single answer should rely on parametric knowledge or external knowledge. In recommendation, multiple candidate items compete simultaneously, and the influence of parametric knowledge manifests as a systematic shift of the entire ranking. This difference is reflected in the results: in Table 2 of the paper, CK-PLUG (Ada.) increases PRS on all three models (e.g., LLaMA 0.2568 → 0.3317), whereas CARD reduces it to 0.1041.

2. Ablation of β\beta. We ablate β\beta by setting it to different constants. As shown in Tables 2 and 3, β>0\beta > 0 clearly outperforms β=0\beta = 0 on both tasks; on both tasks, the optimal region lies around 0.25, and the β\beta measured by PRS falls right in this region, with no tuning required. This demonstrates the effectiveness of PRS calibration (β\beta).

Table 2: Fixed-β\beta ablation for bias mitigation (LLaMA3.1-8B).

β\beta00.1250.250.2568 (PRS)0.375
PRS ↓0.09740.09010.08730.08900.0902

Table 3: Fixed-β\beta ablation for TAP defense (LLaMA3.1-8B). Sˉ\bar{S}: average ranking score of the target item.

MethodSˉ\bar{S} ↓EffDEff_D ↑
Vanilla2.2225–
TAP4.58750%
CARD (β=0\beta = 0)3.032565.8%
CARD (β=0.125\beta = 0.125)2.670081.1%
CARD (β=0.25\beta = 0.25)2.567585.4%
CARD (β=0.2568\beta = 0.2568, PRS)2.592584.4%
CARD (β=0.375\beta = 0.375)2.770076.8%

W4 (first part): Does the Defense Merely Suppress Unfamiliar Item Names?

As shown in Table 4, after replacing the TAP attack target with real items that are familiar to the model, CARD still achieves the highest EffDEff_D on all three models. This demonstrates the effectiveness of CARD's defense: its defense effect comes from suppressing the influence of external attack content, rather than merely from systematically pushing down unfamiliar item names.

Table 4: TAP defense with familiar item names as attack targets. Sˉ\bar{S}: average ranking score of the target item.

LLaMA3.1-8BQwen3-8BGemma3-12B
MethodSˉ\bar{S} ↓EffDEff_D ↑Sˉ\bar{S} ↓EffDEff_D ↑Sˉ\bar{S} ↓EffDEff_D ↑
Vanilla2.3386–2.0531–1.8031–
TAP4.92860%4.95190%4.91560%
CAD3.603951.1%3.806939.5%3.990829.7%
CK-PLUG (Ada.)4.152230.0%4.573113.1%4.093926.4%
CK-PLUG (Fixed)3.201466.7%4.318621.8%3.799235.9%
CARD (Ours)2.847880.3%2.654779.2%3.199755.1%

W4 (second part) & Q1: How to Choose the Direction in Deployment

CARD is designed for bidirectional control over the LLM's knowledge reliance (Introduction), targeting deployment environments where the retrieved content is untrustworthy or, conversely, must be trusted. As stated in Appendix D, the sign of MM is a deployment-time configuration, set by the deployer (user) according to the deployment setting and the retriever trust policy:

  • When the retrieved content is reliable (e.g., using verified private documents): the deployer can use CARD to amplify reliance on the retrieved content, which is especially important when the model's parametric knowledge is known to be outdated.
  • When the retrieved content is unreliable (e.g., retrieval results are noisy), or when the user wishes to rely on the LLM's own judgment: the deployer can suppress the influence of external content and rely on the model's robust parametric knowledge, effectively defending against adversarial manipulation.

Therefore, the choice of direction is not an unstated assumption or weakness; it is precisely CARD's defining feature.


We thank the reviewer again for the thoughtful feedback and would be grateful for further discussion. We hope that the clarifications and additional analyses address the concerns and may support a more positive assessment of our work.

Reviewer_Hh1zOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

this paper studies LLM-augmented search and points out that LLM-augmented searches systematically rank items the model already knows from pretraining above unfamiliar ones. The authors quantify this with Parametric Reliance Score (PRS). It is positive across all five LLMs in a Latin-square controlled experiment.

regarding this issue, the authors proposed Calibrated Adaptive Reliance Decoding (CARD), which is an inference-time decoding method. It contrasts next-token predictions with and without retrieved context. Suppressing context defends against retrieval poisoning, while amplifying context can cut the known-vs-unknown ranking gap (shown on three open models). Which direction to apply has to be chosen externally.

Critical strengths
  1. the paper is well-written and easy to follow.
  2. the motivation is clear; on one end it's defending against retrieval corruption, and on the other end it's the known-vs-unknown gap.
  3. the definition and results are clear and positive.
Critical weaknesses
  1. the models used in the experiments are somewhat outdated and small, mostly 8b experiments, while real recommendation models used in real world are larger ones.
  2. CARD uses token probabilities, which are not accessible when building harnesses.
  3. how to choose the tradeoff between pretrained knowledge and retrieved information has to be determined externally. However, this is apparently a model-specific decision; e.g., each model could have different level biases that show up in different regimes, and the authors should have discussed more concretely how to choose this tradeoff.
Questions for authors
  1. is there an analogue of CARD that does not use token probabilities so this could also work for proprietary models?
  2. can authors provide some discussions how one could concretely choose the tradeoff between pretrained knowledge and retrieved information?
1 comment

We thank the reviewer for their time and insightful feedback. The comments are invaluable for improving our work and clarifying our contributions. We address the primary concerns below.

W1: Model Scale

1. Following the common setting in the field. In the field of LLM-based search and recommendation, other studies [1–6] have all validated the effectiveness of their methods on models of similar scale. We therefore follow this setting and conduct extensive experiments on this basis. In addition, measuring PRS does not depend on logits, and the paper also measures GPT-5-mini and Gemini-3.1-Flash-Lite; both have positive PRS, showing that this bias does not exist only in open-source models.

W2 & Q1: Dependence on Token Probabilities

1. Scope of applicability. We agree with the reviewer. Like decoding methods such as CAD [7] and CK-PLUG [1], CARD requires access to logits and therefore cannot be directly applied to closed-source APIs. For self-deployed open-source models, CARD is plug-and-play and requires no training.

2. Alternatives that do not rely on token probabilities. To the best of our knowledge, no such method for proprietary models exists yet. The most direct alternative is to control the model's reliance on external content via prompting, which the paper already includes as a baseline. As shown in Table 1, prompting has almost no effect, whereas CARD has a clear effect in both directions.

Table 1: Comparison between Prompt and CARD (excerpted from Tables 2 and 4 of the paper). PRS ↓ is reported for bias mitigation, and EffDEff_D ↑ for TAP defense.

LLaMA3.1-8BQwen3-8BGemma3-12B
PRS ↓EffDEff_D ↑PRS ↓EffDEff_D ↑PRS ↓EffDEff_D ↑
Vanilla0.2568-0.1765-0.1994-
Prompt0.2498−0.30%0.17610.14%0.19370.13%
CARD (Ours)0.104193.27%0.079560.02%0.080195.34%

W3 & Q2: How to Concretely Choose the Trade-off

CARD is designed for bidirectional control over the LLM's knowledge reliance (Introduction), targeting deployment environments where the retrieved content is untrustworthy or, conversely, must be trusted. As stated in Appendix D, the sign of MM is a deployment-time configuration, set by the deployer (user) according to the deployment setting and the retriever trust policy:

  • When the retrieved content is reliable (e.g., using verified private documents): the deployer can use CARD to amplify reliance on the retrieved content, which is especially important when the model's parametric knowledge is known to be outdated.
  • When the retrieved content is unreliable (e.g., retrieval results are noisy), or when the user wishes to rely on the LLM's own judgment: the deployer can suppress the influence of external content and rely on the model's robust parametric knowledge, effectively defending against adversarial manipulation.

Regarding the fact that different models exhibit different degrees of bias, CARD introduces the model-level PRS (βm\beta_m) to calibrate each model's anchoring weight, and adaptively adjusts the intervention strength token by token according to the influence of external content, thereby accommodating differences in the degree of bias across models and conditions.

[1] Parameters vs. Context: Fine-Grained Control of Knowledge Reliance in Language Models. ICLR 2026.

[2] StealthRank: LLM Ranking Manipulation via Stealthy Prompt Optimization. ICML 2026 Workshop (Trustworthy AI for Good).

[3] Manipulating Large Language Models to Increase Product Visibility. arXiv:2404.07981, 2024.

[4] "Global is Good, Local is Bad?": Understanding Brand Bias in LLMs. EMNLP 2024.

[5] TALLRec: An Effective and Efficient Tuning Framework to Align Large Language Model with Recommendation. RecSys 2023.

[6] LLaRA: Large Language-Recommendation Assistant. SIGIR 2024.

[7] Trusting Your Evidence: Hallucinate Less with Context-aware Decoding. NAACL 2024.


We thank the reviewer again for the thoughtful feedback and would be grateful for further discussion. We hope that the clarifications and additional analyses address the concerns and may support a more positive assessment of our work.

Reviewer_7cU2OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies an interesting question in LLM-based recommendation: when an LLM-based search engine recommends products, how much does it rely on the retrieved web content versus its inherent (parametric) knowledge?

The authors first propose the Parametric Reliance Score (PRS), which measures how much higher the model ranks products it already knows than products it has never seen. “Known” products are those the model recommends on its own without any retrieved content, and “unknown” products are given made-up names. To control for confounding factors, they adopt a Latin-square design so that the ranking gap can be attributed to the product name rather than to the quality of its description or its position in the prompt. Under this design, all five tested LLMs rank known products significantly higher.

They then propose CARD, a decoding-time method that strengthens or weakens the influence of retrieved content. The direction is set by a user-chosen parameter, while the strength is adjusted automatically based on how much the retrieved content changes the model’s prediction. CARD also uses the measured PRS to down-weight the model’s original prediction. On three open-source models, the paper reports that CARD reduces the ranking gap between known and unknown products by 55–60%, and reduces the effect of two attacks on retrieved content (PoisonedRAG for retrieval poisoning and TAP for black-hat SEO).

Critical strengths
  1. The discussed problem is relevant and timely. Prior work on knowledge conflicts mainly studies question answering, where there is a single correct answer. This paper extends the question to ranking multiple candidates, which is closer to how LLM-based search engines are actually used.

  2. The measurement study is carefully designed. The Latin-square setup partially rules out the effects of description quality and prompt position, and the finding holds consistently across five LLMs, including two closed-source models, with statistical significance tests.

Critical weaknesses
  1. The evaluation does not clearly support the main claims. (1) Fake-looking names weaken both the bias measurement and the defense results. The unknown products are named S_Brand/Z_Model, which look like placeholders to a model. Thus the model may rank them low simply because they seem untrustworthy, not because it prefers products it knows. Appendix B supports this concern: with realistic fictional names, PRS drops by about 36% for LLaMA, and this check covers only two models. All the main CARD experiments still use the placeholder names. The authors may need to provide explanations on this design choice. Also, the promoted item is always a fake-named unknown product, so reducing the influence of retrieved content may push it down simply by bringing back the model’s bias, not by detecting the attack. (2) A lower PRS does not show that the recommendations are better. PRS only shows that product names matter less. The only evidence of recommendation quality is an nDCG@4 score against a human ranking of just four descriptions, and the paper does not report the number of annotators or their agreement. With only four items, even a random ranking gets a high nDCG. The paper also does not discuss whether preferring well-known brands is actually unfair, or simply a reasonable behavior that users might expect.

  2. The two key design choices of CARD are not well justified. (1) The PRS-based calibration does not add a new mechanism. Since the paper uses greedy decoding, multiplying all token scores by the same positive number does not change which token is selected. After this rescaling, Eq. 9 takes the standard Context-Aware Decoding form, with the weight on retrieved content enlarged by a constant factor of 1/(1−|β_m|). So the calibration is equivalent to a stronger intervention. The ablation that removes PRS appears to keep M unchanged and therefore simply uses a weaker intervention. Given the trend in Figure 4, a worse result is expected and does not show that PRS is useful. (2) The claimed bidirectional control requires the user to choose the direction (the sign of M) in advance, because CARD cannot judge whether retrieved content is trustworthy. The bias-mitigation experiments amplify retrieved content, while the defense experiments suppress it. In practice, a system does not know beforehand whether its retrieved content has been attacked, and it is unclear whether any single setting can reduce bias and resist attacks at the same time.

Questions for authors

I am open to raising my score if the authors can address the following questions.

  1. In the “CARD w/o β” ablation, was M kept the same as in the full CARD? Could the authors re-tune M for this variant on LLaMA for the bias-mitigation setting, using the 5-category subset in Section 4.4, and also report results when |β_m| is replaced by a few arbitrary constants? This would show whether the specific PRS value matters.

  2. Could the authors report the bias-mitigation result of CARD on LLaMA using the realistic fictional names from Appendix B?

  3. For TAP, could the authors report the target item’s score when CARD’s defense setting is applied to the input before the attack? If this score is clearly lower than the Vanilla before-attack score, part of the defense effect may come from the bias against unknown products.

  4. For the nDCG@4 evaluation, how many annotators were involved, what was their agreement, which gain function was used, and what score does a random ranking obtain?

5 comments

We thank the reviewer for their time and insightful feedback. The comments are invaluable for improving our work and clarifying our contributions. We address the primary concerns below.

W1 & Q2: Placeholder Names

1. Why we use placeholders. We use symbolic placeholders (e.g., S_Brand / S_Model) to represent non-parametric-knowledge items. This is a deliberate design: it guarantees that these items are genuinely absent from the model's parametric knowledge, which is the prerequisite for isolating the factor of "whether an item belongs to parametric knowledge". LLM-generated fictional names that read more naturally cannot achieve this: because LLMs are trained on large-scale web corpora, even low-probability sampling may produce names that coincide with real entities. To quantify this, we prompted GPT-5.5 to generate 400 (50 × 8) natural-sounding fictional item names and searched each of them on the web: 214 (53.5%) of them have real-world counterparts. Therefore, names that look fictional cannot be guaranteed to lie outside the model's parametric knowledge, whereas placeholders provide a clean, verifiable control.

2. Name construction on more models. In Appendix B of the revised version, we add quartile plots of PRS under the two name constructions for LLaMA3.1-8B, Qwen3-8B, Gemma3-12B, and Gemini-3.1. Except for LLaMA, the interquartile ranges of PRS under the two name constructions largely overlap, with no notable gap. We conjecture that the difference on LLaMA is related to its earlier release and weaker capability, and is an isolated case. This shows that the gap measured by PRS consistently exists under both name constructions.

3. CARD under fictional-but-plausible names (Q2). We evaluate CARD on fictional-but-plausible names, directly reusing the MM and β\beta from the placeholder setting, without re-tuning. As shown in Table 1, CARD's PRS is lower than that of both Vanilla and CAD on all three models; on LLaMA, CARD reduces PRS from 0.1656 to 0.0731. This shows that CARD's debiasing effect does not stem from placeholder names looking untrustworthy, but from regulating the reliance on parametric knowledge itself.

Table 1: PRS under fictional-but-plausible names.

MethodLLaMA3.1-8BQwen3-8BGemma3-12B
Vanilla0.16560.14540.1864
CAD0.09180.10540.0825
CARD (Ours)0.07310.09780.0720

W1 (last sentence) & Q3: Does the Defense Merely Suppress Unfamiliar Item Names?

1. Positioning of the contribution. We propose a method that makes the LLM's knowledge reliance freely controllable: with M>0M > 0, the model relies more on externally retrieved content, which mitigates the bias caused by parametric knowledge; with M<0M < 0, the model relies mainly on its internal knowledge for recommendation ranking, thereby defending against adversarial attacks in the externally retrieved content. Which knowledge to rely on is actively chosen by the user through MM. Choosing M<0M < 0 means that the user chooses to take the model's internal knowledge as authoritative. Unknown items are not in the model's internal knowledge, and information about them can only come from external documents; therefore, the model's selection of unknown items should correspondingly decrease, which is exactly what this choice means. However, CARD's defense effect does not come only from this, as shown below.

2. Familiar item names as attack targets (Q3). We replace the TAP attack target with real items that are familiar to the model. As shown in Table 2, when the attack target is a familiar item name, CARD still achieves the highest EffDEff_D on all three models. This indicates that CARD's defense effect comes from suppressing the influence of external attack content, rather than merely from systematically pushing down unfamiliar item names.

Table 2: TAP defense with familiar item names as attack targets. Sˉ\bar{S}: average ranking score of the target item.

LLaMA3.1-8BQwen3-8BGemma3-12B
MethodSˉ\bar{S} ↓EffDEff_D ↑Sˉ\bar{S} ↓EffDEff_D ↑Sˉ\bar{S} ↓EffDEff_D ↑
Vanilla2.3386–2.0531–1.8031–
TAP4.92860%4.95190%4.91560%
CAD3.603951.1%3.806939.5%3.990829.7%
CK-PLUG (Ada.)4.152230.0%4.573113.1%4.093926.4%
CK-PLUG (Fixed)3.201466.7%4.318621.8%3.799235.9%
CARD (Ours)2.847880.3%2.654779.2%3.199755.1%

W2 & Q4: Does Lower PRS Lead to Better Recommendations?

1. Number of annotators (Q4). 30 annotators.

2. Agreement (Q4). The average Kendall's W is 0.78, and all Friedman tests give p<0.001p < 0.001.

3. Gain function (Q4). The human rank r∈{1,2,3,4}r \in \lbrace 1, 2, 3, 4 \rbrace is mapped to rel=4−rrel = 4 - r; the gain is 2rel−12^{rel} - 1, and the discount is 1/log⁡2(i+1)1/\log_2(i+1).

4. Random baseline (Q4). The expected nDCG@4 of a uniformly random ranking is 0.750. As shown in Table 3, CARD outperforms both Vanilla and the random baseline on all three models.

Table 3: Human nDCG@4.

MethodLLaMA3.1-8BQwen3-8BGemma3-12B
Random0.7500.7500.750
Vanilla0.8110.8260.810
CARD (Ours)0.8210.8440.861

5. Is favoring well-known items unfair? The bias in this paper refers to the systematic preference in LLM recommendation rankings caused by over-reliance on parametric knowledge. This reliance is implicit: users cannot know how much of the ranking comes from parametric knowledge, nor can they control it. Whether favoring well-known items is unfair or a reasonable behavior that users expect should be decided by users according to their own needs; CARD makes this reliance controllable, so users can adjust the model's behavior according to this judgment. The revised version will add this discussion.

W3 & Q1: The Role of PRS Calibration

1. The role of β\beta, and MM in the ablation (Q1, first part). Different models rely on parametric knowledge to different degrees, so the degree of adjustment CARD needs also differs. We therefore introduce the model's PRS into CARD as βm\beta_m: (1−∣βm∣)(1-|\beta_m|) calibrates the anchoring weight of qbaseq_{\mathrm{base}}, so that the degree of adjustment matches the model's own reliance bias. βm\beta_m and MM act jointly: MM determines the direction of the adjustment and the upper bound of its strength, while βm\beta_m calibrates the anchoring weight per model. βm\beta_m is measured in the Latin-square controlled experiment and requires no tuning. The CARD w/o β\beta variant in the paper reuses the MM of the full CARD; below, we directly examine whether this calibration is necessary.

2. Replacing ∣βm∣|\beta_m| with constants (Q1, second part). As shown in Tables 4 and 5, β>0\beta > 0 clearly outperforms β=0\beta = 0 on both tasks; on both tasks, the optimal region lies around 0.25, and the β\beta measured by PRS falls right in this region, with no tuning required.

Table 4: Fixed-β\beta ablation for bias mitigation (LLaMA3.1-8B).

β\beta00.1250.250.2568 (PRS)0.375
PRS ↓0.09740.09010.08730.08900.0902

Table 5: Fixed-β\beta ablation for TAP defense (LLaMA3.1-8B).

MethodSˉ\bar{S} ↓EffDEff_D ↑
Vanilla2.2225–
TAP4.58750%
CARD (β=0\beta = 0)3.032565.8%
CARD (β=0.125\beta = 0.125)2.670081.1%
CARD (β=0.25\beta = 0.25)2.567585.4%
CARD (β=0.2568\beta = 0.2568, PRS)2.592584.4%
CARD (β=0.375\beta = 0.375)2.770076.8%

W4: Choosing the Control Direction

The core value of CARD lies in its controllability, targeting deployment environments where the retrieved content is untrustworthy or, conversely, must be trusted. As stated in Appendix D, the sign of MM is a deployment-time configuration, set by the deployer according to the deployment setting and the retriever trust policy:

  • When the retrieved content is reliable (e.g., using verified private documents): amplify reliance on the retrieved content to mitigate the bias caused by parametric knowledge.
  • When the retrieved content is unreliable (e.g., retrieval results are noisy or potentially manipulated), or when the user wishes to rely on the LLM's own judgment: suppress the influence of external content to defend against adversarial manipulation.

This ability to let the deployer control knowledge reliance according to the specific environment is precisely the core contribution of CARD. Using a single setting to simultaneously reduce bias and resist attacks without knowing in advance whether the retrieved content has been attacked (e.g., automatically selecting the direction in combination with attack detection) is a direction worth exploring in future work.


We thank the reviewer again for the thoughtful feedback and would be grateful for further discussion. We hope that the clarifications and additional analyses address the concerns and may support a more positive assessment of our work.

Thank you for the detailed response and the new experiments. The results with realistic names and with familiar attack targets address my main concern about name construction, and I appreciate the additional annotation details.

However, I would incline to mantain my original evaluation. First, on β: varying β with M fixed is equivalent to varying the intervention strength under greedy decoding, so Tables 4–5 do not separate the effect of β from that of strength; the arbitrary constant 0.25 also performs as well as PRS. I suggest presenting β as a convenient default for setting the strength rather than as a separate calibration mechanism. Second, since the direction of M is set manually, could the authors revise the abstract and introduction to state this clearly?

If the authors agree to these revisions, I would consider raising my score.

Reviewer_1zd6OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper focuses on AI-powered search system. The authors first study how much models favor familiar products, then propose a way to adjust how much they rely on their private knowledge versus retrieved information. Experiments across several models show that the approach can reduce this preference and make product rankings less affected by malicious content.

Critical strengths

[S1] I like the effort put into the controlled experiments. The authors carefully vary product descriptions and their positions to reduce the influence of these factors on the rankings. The results across five models show a solid evidence that the problem is worth studying.

[S2] The method is simple to apply without retraining models, and it is useful that the authors test it for both recommendation bias and adversarial attacks.

[S3] The improvements are consistent across the three evaluated open-source models, especially in reducing the ranking gap and defending against the two attack types. The comparison with existing methods and the ablation study also help support the effectiveness of the approach.

Critical weaknesses

[W1] One concern is that the method needs to know whether to trust the retrieved information in advance. The experiments use positive intervention settings for bias reduction and negative settings for attack defense. However, in real search scenarios, it is often unclear whether the retrieved content is reliable or manipulated. The paper does not show how the method would handle this uncertainty, which limits the practical value of its claimed adaptive control.

[W2] I am also not fully convinced by how the paper measures preference for familiar products. The familiar products are selected from the model's own recommendations, and the unfamiliar products are mostly constructed using artificial names. This makes it difficult to separate the effect of internal knowledge from the model's existing preferences or the credibility of product names. Appendix B partly addresses this with more natural-looking names, but the measured gap becomes noticeably smaller. It is still unclear whether the same improvements would hold for real products with different levels of familiarity, or whether reducing this gap consistently leads to better recommendations.

[W3] The defense evaluation may not fully demonstrate whether the recommendations become more trustworthy. Specifically, in both attack settings, the target is an unfamiliar product, and success is mainly measured by how much its ranking drops. Suppressing retrieved information could push unfamiliar products down, even when they are legitimate. The paper does not sufficiently examine whether the defense also hurts clean recommendations or unfairly suppresses less-known products.

[W4] The proposed rule reduces the adjustment when retrieved information causes a larger change in the model's predictions. However, a strong adversarial attack could also cause a large change, which makes this behavior less intuitive for defense. The paper does not provide much analysis of this relationship under different attack strengths, so it is unclear whether the adaptive rule is generally appropriate.

Questions for authors

[Q1] Can the authors provide results using real lesser-known products and naturally retrieved documents, with independent quality judgments?

[Q2] How would CARD choose the direction of adjustment when trustworthy and manipulated documents appear in the same search results?

[Q3] Can the authors compare PRS calibration against a version that simply retunes the intervention strength to a comparable effective weight?

4 comments

We thank the reviewer for their time and insightful feedback. The comments are invaluable for improving our work and clarifying our contributions. We address the primary concerns below.

W1 & Q2: Choosing the Control Direction

The core contribution of CARD is to let the deployer control the LLM's knowledge reliance according to the specific environment, targeting deployment environments where the retrieved content is untrustworthy or, conversely, must be trusted. Whether the retrieved content is actually reliable or manipulated is a question orthogonal to this paper. As stated in Appendix D, the sign of MM is a deployment-time configuration, set by the deployer according to the deployment setting and the retriever trust policy:

  • When the retrieved content is reliable (e.g., using verified private documents): amplify reliance on the retrieved content to mitigate the bias caused by parametric knowledge.
  • When the retrieved content is unreliable (e.g., retrieval results are noisy or potentially manipulated), or when the user wishes to rely on the LLM's own judgment: suppress the influence of external content to defend against adversarial manipulation.

"Adaptive" refers to the adjustment strength. Once the direction is determined, CARD automatically adjusts the intervention strength token by token according to κ\kappa, without manual tuning. We will make this explicit in the revised version: "adaptive" refers to the adaptivity of the adjustment strength.

W2 & Q1: Measuring Familiarity, and Real Items

1. Familiarity vs. existing preference. PRS measures the model's reliance on parametric knowledge. Whether the model favors an item because it recognizes it or because it considers it better, the preference comes from parametric knowledge, so the two do not need to be separated.

2. The effect of name construction. In Appendix B of the revised version, we add quartile plots of PRS under the two name constructions for LLaMA3.1-8B, Qwen3-8B, Gemma3-12B, and Gemini-3.1. Except for LLaMA, the interquartile ranges of PRS under the two name constructions largely overlap, with no notable gap. We conjecture that the difference on LLaMA is related to its earlier release and weaker capability, and is an isolated case. This shows that the gap measured by PRS consistently exists under both name constructions.

As for whether CARD's improvements still hold, we directly reuse the MM and β\beta from the placeholder setting, without re-tuning, and evaluate on natural, fluent item names. As shown in Table 1, with natural, fluent item names, CARD's PRS is lower than that of both Vanilla and CAD on all three models. This shows that CARD's debiasing effect does not stem from placeholder names looking untrustworthy, but from regulating the reliance on parametric knowledge itself.

Table 1: PRS under natural, fluent item names.

MethodLLaMA3.1-8BQwen3-8BGemma3-12B
Vanilla0.16560.14540.1864
CAD0.09180.10540.0825
CARD (Ours)0.07310.09780.0720

3. Does narrowing the gap lead to better recommendations? The human nDCG@4 evaluation in Appendix D of the paper measures recommendation quality against human quality rankings. As shown in Table 2, on all three models, the interval of CARD relative to Vanilla does not contain 0: recommendation quality also improves while PRS is reduced.

Table 2: 95% confidence intervals of the nDCG@4 difference, CARD − Vanilla. Positive values indicate that CARD is better.

LLaMA3.1-8BQwen3-8BGemma3-12B
CARD − Vanilla[0.001, 0.019][0.008, 0.027][0.038, 0.063]

4. Real lesser-known items and naturally retrieved documents (Q1). To the best of our knowledge, no such public dataset currently exists. If the reviewer is aware of a suitable dataset, we would be happy to add the evaluation.

W3: Does the Defense Merely Suppress Unfamiliar Item Names?

1. Positioning of the contribution. As discussed in W1, the sign of MM is chosen by the deployer according to the retriever trust policy. Choosing M<0M < 0 means choosing to take the model's internal knowledge as authoritative. Unknown items are not in the model's internal knowledge, and information about them can only come from external documents; therefore, the model's selection of unknown items should correspondingly decrease, which is exactly what this choice means. However, CARD's defense effect does not come only from this, as shown below.

2. Familiar item names as attack targets. We replace the TAP attack target with real items that are familiar to the model. As shown in Table 3, when the attack target is a familiar item name, CARD still achieves the highest EffDEff_D on all three models. This indicates that CARD's defense effect comes from suppressing the influence of external attack content, rather than merely from systematically pushing down unfamiliar item names.

Table 3: TAP defense with familiar item names as attack targets. Sˉ\bar{S}: average ranking score of the target item.

LLaMA3.1-8BQwen3-8BGemma3-12B
MethodSˉ\bar{S} ↓EffDEff_D ↑Sˉ\bar{S} ↓EffDEff_D ↑Sˉ\bar{S} ↓EffDEff_D ↑
Vanilla2.3386–2.0531–1.8031–
TAP4.92860%4.95190%4.91560%
CAD3.603951.1%3.806939.5%3.990829.7%
CK-PLUG (Ada.)4.152230.0%4.573113.1%4.093926.4%
CK-PLUG (Fixed)3.201466.7%4.318621.8%3.799235.9%
CARD (Ours)2.847880.3%2.654779.2%3.199755.1%

W4: Why Decrease α∗\alpha^{\ast} When κ\kappa Is Large?

This is an intentional design, and it is equally reasonable for defense. A large κ\kappa indicates that the retrieved content has already substantially changed the model's prediction, in which case qcontq_{\mathrm{cont}} itself is already large; continuing to adjust qcontq_{\mathrm{cont}} with a large coefficient can easily over-adjust the logits and cause the output to collapse. Therefore, ∣α∗(M,κ)∣=1/ln⁡(e1/∣M∣+κ)|\alpha^{\ast}(M,\kappa)| = 1/\ln(e^{1/|M|}+\kappa) decreases monotonically with κ\kappa, and the decrease gradually flattens (Eq. 8). Note that only the coefficient α∗\alpha^{\ast} decreases; the correction actually applied is α∗⋅qcont\alpha^{\ast} \cdot q_{\mathrm{cont}}. The stronger the attack, the larger qcontq_{\mathrm{cont}}, so the absolute magnitude of the correction still increases; only its growth is kept under control.

Q3: PRS Calibration vs. Directly Adjusting the Intervention Strength

1. The role of β\beta. Different models rely on parametric knowledge to different degrees, so the degree of adjustment CARD needs also differs. We therefore introduce the model's PRS into CARD as βm\beta_m: (1−∣βm∣)(1-|\beta_m|) calibrates the anchoring weight of qbaseq_{\mathrm{base}}, so that the degree of adjustment matches the model's own reliance bias. βm\beta_m and MM act jointly: MM determines the direction of the adjustment and the upper bound of its strength, while βm\beta_m calibrates the anchoring weight per model.

2. Replacing βm\beta_m with constants. As shown in Tables 4 and 5, β>0\beta > 0 clearly outperforms β=0\beta = 0 on both tasks; on both tasks, the optimal region lies around 0.25, and the β\beta measured by PRS falls right in this region, with no tuning required. This demonstrates the effectiveness of PRS calibration.

Table 4: Fixed-β\beta ablation for bias mitigation (LLaMA3.1-8B).

β\beta00.1250.250.2568 (PRS)0.375
PRS ↓0.09740.09010.08730.08900.0902

Table 5: Fixed-β\beta ablation for TAP defense (LLaMA3.1-8B).

MethodSˉ\bar{S} ↓EffDEff_D ↑
Vanilla2.2225–
TAP4.58750%
CARD (β=0\beta = 0)3.032565.8%
CARD (β=0.125\beta = 0.125)2.670081.1%
CARD (β=0.25\beta = 0.25)2.567585.4%
CARD (β=0.2568\beta = 0.2568, PRS)2.592584.4%
CARD (β=0.375\beta = 0.375)2.770076.8%

We thank the reviewer again for the thoughtful feedback and would be grateful for further discussion. We hope that the clarifications and additional analyses address the concerns and may support a more positive assessment of our work.

Thanks for the authors' response. My questions about the reliance on retrieved knowledge, performance measures, and attack strength have been largely cleared up. However, some concerns remain, especially regarding the gap between the introductory scenarios and actual evaluations. For example, when evaluating "whether attacking products can really bypass defense mechanisms", the evaluation could include more comparable settings (e.g., attacking familiar/unfamiliar items or with/without defenses) and show the differences in performance, which would provide more insight into the effectiveness of the defenses. Moreover, it would be great to see evaluations on more practical chatbot-based systems. So far, the evaluations are based on three open-source LMs that are not particularly large (8B–12B).

I've also noticed that the manuscript has not yet been revised much to integrate the discussions or additional results, as well as other reviewers' concerns. Therefore, I would recommend refining the manuscript as much as possible (even if offline, since the rebuttal period is almost over). Given those concerns, I will keep my rating for now and would like to see what other reviewers think.

Thank you for your response. We are glad that our previous reply has addressed your questions about the reliance on retrieved knowledge, the performance measures, and the attack strength. We address the remaining concerns below.

1. Familiar / unfamiliar attack targets, with and without defense. We use the same set of candidate items and set the TAP attack target to an unfamiliar item name (with a natural, fluent name) and to a real item familiar to the model, respectively, and compare the target item's ranking with and without defense. The rows of Table 6 are as follows:

  • Vanilla: no attack and no defense, i.e., the ranking of the original LLM, serving as the reference;
  • TAP (without defense): attack without defense;
  • CAD, CK-PLUG, and CARD: attack with defense.

As shown in Table 6, whether the attack target is an unfamiliar or a familiar item name, CARD achieves the lowest Sˉ\bar{S} and the highest EffDEff_D, and it achieves similar effectiveness on both types of targets (EffDEff_D 81.1% vs. 73.7%). This indicates that CARD's defense effect comes from suppressing the influence of external attack content, rather than merely from systematically pushing down unfamiliar item names.

Table 6: TAP defense with unfamiliar / familiar item names as attack targets (LLaMA3.1-8B). Sˉ\bar{S}: average ranking score of the target item.

Unfamiliar item nameFamiliar item name
MethodSˉ\bar{S} ↓EffDEff_D ↑Sˉ\bar{S} ↓EffDEff_D ↑
Vanilla2.3375–2.5125–
TAP (without defense)5.00000%5.00000%
CAD3.425059.2%3.945042.4%
CK-PLUG (Ada.)4.227529.0%4.117535.5%
CK-PLUG (Fixed)2.932577.7%3.525059.3%
CARD (Ours)2.840081.1%3.167573.7%

2. Model scale. In the field of LLM-based search and recommendation, other studies [1–8] have all validated the effectiveness of their methods on models of similar scale. We therefore follow this setting and conduct extensive experiments on this basis. In addition, measuring PRS does not depend on logits, and the paper also measures GPT-5-mini and Gemini-3.1-Flash-Lite; both have positive PRS that is statistically significant in the ANOVA of Appendix C (F(1,798)=887.6F(1,798) = 887.6 and 487.9487.9, p<0.001p < 0.001), showing that this bias does not exist only in open-source models. Due to the limited time of the rebuttal period, we will add experiments on 30B–70B models in the final version.

3. Manuscript revision. Thank you for the suggestion. We have incorporated the main discussions, the additional results, and the concerns raised by other reviewers into the appendix, and uploaded the revised PDF.


We hope these additions address your remaining concerns, and we would be grateful if you would take them into account in your final assessment. Thank you again for your time and constructive suggestions.


[1] Parameters vs. Context: Fine-Grained Control of Knowledge Reliance in Language Models. ICLR 2026.

[2] StealthRank: LLM Ranking Manipulation via Stealthy Prompt Optimization. ICML 2026 Workshop (Trustworthy AI for Good).

[3] Manipulating Large Language Models to Increase Product Visibility. arXiv:2404.07981, 2024.

[4] "Global is Good, Local is Bad?": Understanding Brand Bias in LLMs. EMNLP 2024.

[5] TALLRec: An Effective and Efficient Tuning Framework to Align Large Language Model with Recommendation. RecSys 2023.

[6] LLaRA: Large Language-Recommendation Assistant. SIGIR 2024.

[7] AgenticRec: A Recommendation-Oriented Agentic Framework with Progressive Tool-Integrated Reasoning Optimization. arXiv:2603.21613, 2026.

[8] Group-in-Group Policy Optimization for LLM Agent Training. NeurIPS 2025. Findings of EMNLP 2026.

Reviewer_f5RfOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper uses a Latin-square design to control product names, descriptions, and context positions when measuring ranking differences between familiar and unfamiliar products. It summarizes this preference using PRS. CARD then combines token scores with and without retrieved context, uses KL divergence to adapt a signed contrastive-decoding strength, and weights the base scores using a model-level PRS estimate. The experiments examine both ranking-bias mitigation and suppression of attacker-target promotion under TAP and PoisonedRAG.

Critical strengths

The controlled measurement of name familiarity is a strong aspect of the paper. Separating names, descriptions, and positions addresses important confounding factors, and the experiments span five models. The fluent fictitious-name control also provides evidence that the phenomenon is not restricted to purely symbolic placeholder names.

The decoding method does not require retraining, and the signed parameter offers a common formulation for increasing or decreasing reliance on retrieved information. The evaluation includes ranking preference, attack-target rank, human-assessed nDCG@4, and throughput. The paper therefore provides both task-utility and efficiency evidence, even though some of that evidence needs stronger interpretation.

Critical weaknesses

W1. The defense evaluation confounds resistance to manipulation with suppression of unfamiliar product names.

The defense experiments primarily promote the nonparametric-knowledge product ZBrand/ZModel, while the competing products are familiar to the model. Negative M is designed to reduce the contribution of external information. A lower rank for the attacker’s target could therefore arise partly from suppressing unfamiliar products, rather than specifically resisting malicious content (Section 4.3; Appendix E). The most informative addition would be a small crossed control: attack a familiar product as well, and evaluate legitimate unfamiliar products or new facts under the same negative-M setting. This would separate resistance to attacker promotion from a general loss of responsiveness to unfamiliar external information.

Appendix D explicitly states that the sign of M is selected by a deployment trust policy and that CARD does not itself determine whether a document is trustworthy. I therefore do not criticize the absence of automatic sign selection as a violation of the paper’s stated scope. However, gains obtained with separately chosen positive and negative settings should not be combined into a single deployment benefit without accounting for that external trust policy.

W2. The external validity of PRS and the smaller utility improvements need stronger statistical support.

PRS measures a ranking difference associated with name provenance or familiarity; it is not a direct measurement of the fraction of reasoning attributable to parametric knowledge. The fluent fictitious-name experiment preserves a positive effect, which is supportive evidence. At the same time, Llama’s PRS decreases from 0.2568 to 0.1650 under this control, showing that its scale depends materially on how unfamiliar names are constructed (Table 5).

The mitigation experiments reuse the category collection from the principal measurement, leaving the ability of a model-level β to calibrate new categories insufficiently established. A held-out-category evaluation would test whether PRS provides transferable calibration rather than fitting the measurement distribution. The human nDCG@4 differences also require uncertainty estimates. CARD versus CAD is 0.8207 versus 0.8203 for Llama and 0.8435 versus 0.8412 for Qwen (Table 7). Without paired intervals, these small differences should not be interpreted as robust utility gains simply because one entry is numerically larger. Gemma shows a more substantial difference and should be discussed separately.

Appendix C already averages repeated runs at the category–product-name level rather than treating every run as independent. The remaining concern is dependence among rankings within a category and uncertainty in the final method difference. Category-level paired resampling would be a natural way to address this.

Questions for authors
  1. When α_eff is treated as the actual control quantity and candidate sets and tuning budgets are matched, how much additional benefit comes from measuring PRS rather than ordinary coefficient calibration? What happens with fixed or shuffled β?
  2. Does the defense advantage persist when the attack target is a familiar product? What is the ranking-quality cost for legitimate unfamiliar products under the same negative-M setting?
  3. Does PRS provide useful calibration on held-out categories, and what are the category-level paired uncertainty intervals for the human nDCG@4 differences against CAD?
3 comments

We thank the reviewer for their time and insightful feedback. The comments are invaluable for improving our work and clarifying our contributions. We address the primary concerns below.

W1 & Q2: Does the Defense Merely Suppress Unfamiliar Item Names?

1. Familiar item names as attack targets (Q2a). We replace the TAP attack target with real items that are familiar to the model. As shown in Table 1, when the attack target is a familiar item name, CARD still achieves the highest EffDEff_D on all three models. This indicates that CARD's defense effect comes from suppressing the influence of external attack content, rather than merely from systematically pushing down unfamiliar item names.

Table 1: TAP defense with familiar item names as attack targets. Sˉ\bar{S}: average ranking score of the target item.

LLaMA3.1-8BQwen3-8BGemma3-12B
MethodSˉ\bar{S} ↓EffDEff_D ↑Sˉ\bar{S} ↓EffDEff_D ↑Sˉ\bar{S} ↓EffDEff_D ↑
Vanilla2.3386–2.0531–1.8031–
TAP4.92860%4.95190%4.91560%
CAD3.603951.1%3.806939.5%3.990829.7%
CK-PLUG (Ada.)4.152230.0%4.573113.1%4.093926.4%
CK-PLUG (Fixed)3.201466.7%4.318621.8%3.799235.9%
CARD (Ours)2.847880.3%2.654779.2%3.199755.1%

2. External information under negative MM (Q2b). CARD makes the LLM's knowledge reliance freely controllable: with M>0M > 0, the model relies more on externally retrieved content; with M<0M < 0, it relies more on its internal knowledge. Which knowledge to rely on is actively chosen by the user through MM. Choosing M<0M < 0 means that the user chooses to take the model's internal knowledge as authoritative; the model's adoption of external documents is then correspondingly reduced, which is exactly what this choice means. Therefore, the ranking quality of unfamiliar items is not an evaluation target of the M<0M < 0 setting.

3. Reporting positive and negative MM separately. We agree with the reviewer: bias mitigation (M>0M > 0) and attack defense (M<0M < 0) correspond to different user choices. The revised version will report their benefits separately, rather than combining them into a single deployment benefit.

W2: External Validity and Statistical Support of PRS

1. The meaning of PRS (W2a). We agree with the reviewer's description. PRS is a behavioral metric measured on the ranking output (Eq. 3); it measures the strength of the model's reliance on parametric knowledge, and is not an attribution of the proportion that parametric knowledge contributes to the reasoning process. Directly measuring what proportion of the reasoning process is attributable to parametric knowledge is a worthwhile direction for future research.

2. Variation of PRS with name construction (W2b). In Appendix B of the revised version, we add quartile plots of PRS under the two name constructions for LLaMA3.1-8B, Qwen3-8B, Gemma3-12B, and Gemini-3.1. Except for LLaMA, the interquartile ranges of PRS under the two name constructions largely overlap, with no notable gap. We conjecture that the difference on LLaMA is related to its earlier release and weaker capability, and is an isolated case. This shows that the gap measured by PRS consistently exists under both name constructions.

3. Calibration on held-out categories (W2c & Q3a). MM is selected only on 5 development categories. We estimate β\beta using only these 5 categories and apply it directly to the remaining 45 held-out categories, without re-tuning. As shown in Table 2, CARD's PRS is lower than that of Vanilla and all baselines. This shows that the β\beta measured on a small number of categories can be directly applied to unseen categories.

Table 2: PRS on the 45 held-out categories.

MethodLLaMA3.1-8B
Vanilla0.2587
Prompt0.2510
CK-PLUG (Ada.)0.3297
CK-PLUG (Fixed)0.2475
CAD0.1238
CARD (Ours)0.1046

4. Paired intervals of nDCG@4 relative to CAD (W2d & Q3b). As shown in Table 3, CARD's utility is comparable to that of CAD on LLaMA and Qwen (the intervals contain 0), and significantly better than CAD on Gemma. Together with the PRS results in the paper, this shows that CARD reduces PRS more effectively than CAD without lowering recommendation utility. The revised version will revise the relevant statements accordingly.

Table 3: 95% confidence intervals of the nDCG@4 difference, CARD − CAD.

LLaMA3.1-8BQwen3-8BGemma3-12B
CARD − CAD[−0.006, 0.007][−0.003, 0.007][0.010, 0.022]

Q1: The Value of PRS Calibration

1. The role of β\beta. Different models rely on parametric knowledge to different degrees, so the degree of adjustment CARD needs also differs. We therefore introduce the model's PRS into CARD as βm\beta_m: (1−∣βm∣)(1-|\beta_m|) calibrates the anchoring weight of qbaseq_{\mathrm{base}}, so that the degree of adjustment matches the model's own reliance bias. βm\beta_m and MM act jointly: MM determines the direction of the adjustment and the upper bound of its strength, while βm\beta_m calibrates the anchoring weight per model.

2. Fixed-β\beta ablation. As shown in Tables 4 and 5, β>0\beta > 0 clearly outperforms β=0\beta = 0 on both tasks; on both tasks, the optimal region lies around 0.25, and the β\beta measured by PRS falls right in this region, with no tuning required. This demonstrates the effectiveness of PRS calibration: the calibration coefficient is obtained directly from measurement and requires no additional search.

Table 4: Fixed-β\beta ablation for bias mitigation (LLaMA3.1-8B).

β\beta00.1250.250.2568 (PRS)0.375
PRS ↓0.09740.09010.08730.08900.0902

Table 5: Fixed-β\beta ablation for TAP defense (LLaMA3.1-8B).

MethodSˉ\bar{S} ↓EffDEff_D ↑
Vanilla2.2225–
TAP4.58750%
CARD (β=0\beta = 0)3.032565.8%
CARD (β=0.125\beta = 0.125)2.670081.1%
CARD (β=0.25\beta = 0.25)2.567585.4%
CARD (β=0.2568\beta = 0.2568, PRS)2.592584.4%
CARD (β=0.375\beta = 0.375)2.770076.8%

We thank the reviewer again for the thoughtful feedback and would be grateful for further discussion. We hope that the clarifications and additional analyses address the concerns and may support a more positive assessment of our work.

Thanks for the authors' detailed clarification and the additional experiments provided. The responses have mainly solved my concern. I am happy to raise my score accordingly. Please continue to refine the writing and integrate these new discussions into the final version.

Comments on the paper

5 comments

We thank the reviewer for their time and insightful feedback. The comments are invaluable for improving our work and clarifying our contributions. We address the primary concerns below.

W1 & Q1: The Independent Value of PRS Calibration

1. The role of β\beta. Different models rely on parametric knowledge to different degrees, so the degree of adjustment CARD needs also differs. We therefore introduce the model's PRS into CARD as βm\beta_m: (1−∣βm∣)(1-|\beta_m|) calibrates the anchoring weight of qbaseq_{\mathrm{base}}, so that the degree of adjustment matches the model's own reliance bias. βm\beta_m and MM act jointly: MM determines the direction of the adjustment and the upper bound of its strength, while βm\beta_m calibrates the anchoring weight per model.

2. Replacing βm\beta_m with constants (Q1). As shown in Tables 1 and 2, β>0\beta > 0 clearly outperforms β=0\beta = 0 on both tasks; on both tasks, the optimal region lies around 0.25, and the β\beta measured by PRS falls right in this region, with no tuning required. This demonstrates the effectiveness of PRS calibration: the calibration coefficient is obtained directly from measurement and requires no additional search.

Table 1: Fixed-β\beta ablation for bias mitigation (LLaMA3.1-8B, 5 development categories).

β\beta00.1250.250.2568 (PRS)0.375
PRS ↓0.09740.09010.08730.08900.0902

Table 2: Fixed-β\beta ablation for TAP defense (LLaMA3.1-8B, 5 development categories). Sˉ\bar{S}: average ranking score of the target item.

MethodSˉ\bar{S} ↓EffDEff_D ↑
Vanilla2.2225–
TAP4.58750%
CARD (β=0\beta = 0)3.032565.8%
CARD (β=0.125\beta = 0.125)2.670081.1%
CARD (β=0.25\beta = 0.25)2.567585.4%
CARD (β=0.2568\beta = 0.2568, PRS)2.592584.4%
CARD (β=0.375\beta = 0.375)2.770076.8%

3. Re-tuned CAD. We re-tune CAD using the same candidate set as CARD and compare against it. As shown in Table 3, CARD outperforms the re-tuned CAD on both tasks.

Table 3: Comparison with re-tuned CAD (LLaMA3.1-8B, 5 development categories).

MethodBias mitigation PRS ↓TAP defense EffDEff_D ↑
CAD0.097987.1%
CARD (Ours)0.089093.4%

4. Variation of PRS with name construction. In Appendix B of the revised version, we add quartile plots of PRS under the two name constructions for LLaMA3.1-8B, Qwen3-8B, Gemma3-12B, and Gemini-3.1. Except for LLaMA, the interquartile ranges of PRS under the two name constructions largely overlap, with no notable gap. We conjecture that the difference on LLaMA is related to its earlier release and weaker capability, and is an isolated case. This shows that the gap measured by PRS consistently exists under both name constructions.

W2 & Q3: Does the Defense Merely Suppress Unfamiliar Item Names?

1. Positioning of the contribution (Q3a). The contribution of CARD is to make the LLM's knowledge reliance freely controllable: with M>0M > 0, the model relies more on externally retrieved content, which mitigates the bias caused by parametric knowledge; with M<0M < 0, the model relies more on its internal knowledge, which defends against adversarial attacks. Which knowledge to rely on is actively chosen by the user through MM. Choosing M<0M < 0 means that the user chooses to take the model's internal knowledge as authoritative; the model's adoption of external documents is then correspondingly reduced, which is exactly what this choice means.

2. Familiar item names as attack targets. We replace the TAP attack target with real items that are familiar to the model. As shown in Table 4, when the attack target is a familiar item name, CARD still achieves the highest EffDEff_D on all three models. This indicates that CARD's defense effect comes from suppressing the influence of external attack content, rather than merely from systematically pushing down unfamiliar item names.

Table 4: TAP defense with familiar item names as attack targets.

LLaMA3.1-8BQwen3-8BGemma3-12B
MethodSˉ\bar{S} ↓EffDEff_D ↑Sˉ\bar{S} ↓EffDEff_D ↑Sˉ\bar{S} ↓EffDEff_D ↑
Vanilla2.3386–2.0531–1.8031–
TAP4.92860%4.95190%4.91560%
CAD3.603951.1%3.806939.5%3.990829.7%
CK-PLUG (Ada.)4.152230.0%4.573113.1%4.093926.4%
CK-PLUG (Fixed)3.201466.7%4.318621.8%3.799235.9%
CARD (Ours)2.847880.3%2.654779.2%3.199755.1%

3. Whether TAP is re-optimized (Q3b). No, it is not re-optimized. The TAP attacks are generated on the undefended model and then transferred to each defense method; all methods face the same attack text.

Q2: Transferability, and Familiarity vs. Pre-selected Preference

1. Transfer to held-out categories and fictional-but-plausible names. CARD directly reuses the MM and β\beta from the placeholder setting, without re-tuning. As shown in Table 5, CARD's PRS is lower than that of Vanilla and all baselines on all three models. This shows that, without re-tuning, CARD transfers to fictional-but-plausible names on held-out categories.

Table 5: PRS under fictional-but-plausible names (45 held-out categories).

MethodLLaMA3.1-8BQwen3-8BGemma3-12B
Vanilla0.16560.14540.1864
Prompt0.14680.14190.1827
CK-PLUG (Ada.)0.12020.16750.2532
CK-PLUG (Fixed)0.12240.14560.1887
CAD0.09180.10540.0825
CARD (Ours)0.07310.09780.0720

In addition, we estimate β\beta using only the 5 development categories and apply it directly to the remaining 45 held-out categories, without re-tuning. As shown in Table 10, CARD's PRS is lower than that of Vanilla and all baselines. This shows that, without re-tuning, CARD transfers to held-out categories.

Table 10: PRS on the 45 held-out categories.

MethodLLaMA3.1-8B
Vanilla0.2587
Prompt0.2510
CK-PLUG (Ada.)0.3297
CK-PLUG (Fixed)0.2475
CAD0.1238
CARD (Ours)0.1046

2. Familiarity vs. pre-selected preference. PRS measures the model's reliance on parametric knowledge. Whether the model favors an item because it recognizes it or because it considers it better, the preference comes from parametric knowledge, so the two do not need to be separated.

Q4: Total Ranking Score and Confidence Intervals

1. 1.0575 vs. 1.0625 (Q4a). Following prior work [1], we use fuzzy matching to identify item names in the response. Items that are missing or cannot be parsed receive a ranking score of 0; for items that appear more than once, the ranking score is computed from their first occurrence. Hence, the actual total is slightly lower than the theoretical value. The same rule is applied to both groups of items.

2. Paired confidence intervals (Q4b). As requested by the reviewer, Tables 6–9 report the 95% paired confidence intervals for each improvement reported in the paper, with Table 7 covering nDCG@4. Across all 66 comparisons, CARD is significantly better than the corresponding baseline in 60 and significantly worse in none; the remaining 6 differences are not significant (marked with † in the tables).

Table 6: 95% confidence intervals of the PRS difference, CARD − baseline (category-paired bootstrap). Negative values indicate that CARD is better.

BaselineLLaMA3.1-8BQwen3-8BGemma3-12B
Vanilla[−0.1686, −0.1368][−0.1131, −0.0813][−0.1332, −0.1054]
Prompt[−0.1600, −0.1311][−0.1127, −0.0809][−0.1271, −0.1000]
CAD[−0.0210, −0.0132][−0.0515, −0.0313][−0.0208, −0.0113]
CK-PLUG (Ada.)[−0.2423, −0.2121][−0.1747, −0.1316][−0.2184, −0.1898]
CK-PLUG (Fixed)[−0.1550, −0.1285][−0.1140, −0.0820][−0.1274, −0.0985]

Table 7: 95% confidence intervals of the nDCG@4 difference, CARD − baseline. Positive values indicate that CARD is better. †: the interval contains 0, i.e., the difference is not significant.

BaselineLLaMA3.1-8BQwen3-8BGemma3-12B
Vanilla[0.001, 0.019][0.008, 0.027][0.038, 0.063]
Prompt[0.006, 0.023][0.005, 0.022][0.038, 0.062]
CAD[−0.006, 0.007]†[−0.003, 0.007]†[0.010, 0.022]
CK-PLUG (Ada.)[0.022, 0.049][0.038, 0.068][0.066, 0.100]
CK-PLUG (Fixed)[0.005, 0.022][0.010, 0.029][0.040, 0.065]

Table 8: 95% confidence intervals of the TAP defense EffDEff_D difference, CARD − baseline (percentage points, category-paired bootstrap). Positive values indicate that CARD is better. †: the interval contains 0, i.e., the difference is not significant.

BaselineLLaMA3.1-8BQwen3-8BGemma3-12B
TAP[84.5, 99.6][49.0, 72.8][72.7, 99.7]
Prompt[84.7, 100.0][48.9, 72.6][72.7, 99.5]
PPL[44.0, 86.1][31.6, 67.3][40.7, 89.1]
CAD[13.8, 27.8][18.2, 43.1][−0.9, 41.8]†
CK-PLUG (Ada.)[43.1, 61.7][42.1, 67.0][35.4, 71.9]
CK-PLUG (Fixed)[−0.4, 18.7]†[32.1, 57.2][20.0, 60.3]

Table 9: 95% confidence intervals of the difference in the target item's ranking score Sˉ\bar{S} under PoisonedRAG, CARD − baseline (category-paired bootstrap). Negative values indicate that CARD is better. †: the interval contains 0, i.e., the difference is not significant.

BaselineLLaMA3.1-8BQwen3-8BGemma3-12B
PoisonedRAG[−1.485, −1.252][−1.139, −0.770][−0.657, −0.230]
Prompt[−1.361, −1.127][−1.096, −0.739][−0.675, −0.281]
PPL[−0.439, 0.580]†[−1.371, −0.955][−0.745, −0.288]
CAD[−0.895, −0.686][−0.512, −0.235][−0.138, 0.042]†
CK-PLUG (Ada.)[−1.258, −1.009][−0.792, −0.489][−0.341, −0.043]
CK-PLUG (Fixed)[−0.443, −0.306][−0.653, −0.349][−0.233, −0.002]

[1] Pfrommer et al. Ranking manipulation for conversational search engines. EMNLP 2024.

We thank the reviewer again for the thoughtful feedback and would be grateful for further discussion. We hope that the clarifications and additional analyses address the concerns and may support a more positive assessment of our work.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.