acceptodds
Under review as a conference paper at ICLR 2027

CrowdRAG: Source-Density Defense Against Sybil Poisoning in Multi-User Agent Memory

Abstract

LLM agents increasingly operate over shared memory pools where multiple users both contribute and retrieve, a setting that lets a single user's contribution become evidence that any member's agent retrieves and acts on. This shared write access enables Sybil poisoning: three fake accounts, 1% of users and about 10% of a typical topic's contributors, already fill 53% of the top-5 retrieved passages. Existing content-level defenses systematically fail because honest paraphrases and adversarial claims about the same topic are structurally indistinguishable in embedding space. We propose CrowdRAG, a two-layer defense that exploits the orthogonal signal multi-user settings provide: the number of distinct users independently supporting each claim. CrowdRAG first filters claims that lack support from enough distinct users, then resolves remaining contradictions through a provenance-aware knowledge graph. With three Sybil accounts and GPT-5.4, CrowdRAG reduces attack success from 63.6% to 1.3% at 96.1% accuracy; attack success stays at or below 3.9% across three LLMs and, with a label-free similarity calibration, across four retrievers. We also characterize a density cliff: on topics with no more honest contributors than Sybil accounts, no support threshold can separate the two. Our results show that user provenance is a practical trust signal complementary to content-level defenses.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
retrieval-augmented generation, LLM agents, agent memory, Sybil attack, knowledge poisoning, multi-user systems, provenance, trustworthy AI

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_zoRjOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

CrowdRAG combines a quorum over distinct user identities with provenance-aware triple extraction and contradiction filtering to defend shared agent memory against Sybil poisoning. Experiments cover 154 topics, three answer models and four retrievers, with additional tests of adaptive attacks, flooding, honest disagreement, and topic assignment on Travel StackExchange.

Critical strengths
  1. Counting distinct users across the pool limits the authority of Sybils that dominate retrieval. Results across answer models and retrievers support provenance as a useful defense signal within the stated corruption budget.
  2. The density-cliff analysis identifies when no support threshold can separate honest and adversarial contributors. The security and coverage trade-off is therefore visible.
Critical weaknesses
  1. Section 4.1 filters the pool before retrieval, Algorithm 1 filters a fixed top-five set without backfilling, and Appendix Q uses backfill retrieval. These procedures change the amount of honest evidence and the retrieval budget. The implementation used for Table 1 must be identified to make the central comparison reproducible.
  2. Table 3 tests adaptive attacks through retrieval-level exposure at quorum five, whereas headline end-to-end results use quorum ten. Cosine gaming borrows honest support for contradictory claims, leaving the final defense dependent on triple extraction and entity normalization. Appendix I acknowledges that adversarial synonymy and entity aliasing remain untested, so adaptive end-to-end robustness has not been established.
Questions for authors
  1. Which pipeline produces Table 1: filtering before retrieval, filtering a fixed top-five set, or retrieving replacements after filtering? Please reconcile Section 4.1, Algorithm 1, and Appendix Q, and report the resulting context size and honest-evidence coverage.
  2. At the main quorum of ten, what are end-to-end attack success and answer accuracy under cosine gaming? Can targeted paraphrases or entity aliases that preserve the false assertion bypass the contradiction check?
3 comments
AuthorsOpenReview ↗

Response to Reviewer zoRj

Thank you for the positive assessment and for the two precise questions. We answer both below.

W1 / Q1: which pipeline produces Table 1

A: CrowdRAG in Table 1 follows Section 4.1: the pool is filtered and the top-5 is retrieved from what remains. "Backfill" in Appendix Q is the same procedure seen from the retrieval side, where a filtered position is taken by the next-ranked claim that passes. Algorithm 1 omits the step that continues down the ranking until five claims pass, and we will correct it so that the three places agree. The content-level baselines receive the unfiltered top-5, so the budget is five claims in every case.

As requested, the resulting context size and honest evidence for this pipeline (t=10t=10, 154 topics, mean per topic):

ffClaims in contextHonestAdversarialTopics with at least one honest claim
04.804.800.0097.4%
34.804.690.1097.4%
54.804.610.1996.8%

143 of the 154 topics receive the full five claims.

W2 / Q2: end-to-end adaptive attacks at t=10t=10, paraphrases and entity aliases

A: We agree that Table 3 did not establish this. Following the reviewer's suggestion, we ran the adaptive attacks end to end under the configuration of Table 1 (GPT-5.4, Contriever, f=3f=3, t=10t=10, 154 topics). We also added two attacks that keep the retrieval anchor (the query echo of the original attack): one whose claim mimics honest wording, and one whose claim names the subject by an alias or paraphrase and rewords the attribute, which targets entity normalization.

AttackASR, undefendedACC, undefendedASR, CrowdRAGACC, CrowdRAG
Original (Table 1)63.6%35.7%1.3%96.1%
Cosine gaming (Section 6.2)3.9%94.2%0.0%96.1%
Triple evasion (Section 6.2)0.0%98.1%0.0%96.1%
Anchor + honest wording64.3%35.7%3.2%95.5%
Anchor + canonicalization43.5%55.8%1.3%96.1%

(i) At the main quorum of ten, cosine gaming reaches 3.9% ASR undefended and 0.0% with CrowdRAG, at 96.1% accuracy. With the anchor retained it is as strong as the original attack when undefended (64.3%), and CrowdRAG reduces it to 3.2%.

(ii) On paraphrases and entity aliases: a claim that uses an alias can stay out of the honest (subject, attribute) entry, and Layer 2 then has nothing to compare it with. Such a claim no longer resembles the honest claims, however, so only 48 of 462 pass Layer 1, and the attack reaches 1.3% ASR end to end. Claims that mimic honest wording behave the other way: 138 of 462 pass Layer 1, but they state their value in the honest entry, and Layer 2 blocks 117 of them.

We will add this table to Section 6.2 and update Appendix I, which noted that synonymy and aliasing had not been tested. The construction of the two anchored attacks is described in our response to Reviewer ZYQC (2/2).

Thank you for the reply and the additional experiments. After reading the revised manuscript, I noticed that Appendix E.1 calibrates the threshold with ten Sybil accounts but evaluates it with three. For Contriever, the reported Layer 1 pass counts in the three-account setting imply at most 91.7% coverage, even if every honest claim passes. This falls below the calibration rule’s 95% requirement. I would like to see the results after calibrating on the actual three-account pool, since the current experiment does not establish whether the automatic-calibration result holds under that protocol. The sparse-density experiment also resamples the synthetic benchmark, while the natural Travel data are used only to evaluate topic assignment. A topic can have many contributors without enough independent support for the particular fact being queried. The revised evaluation still leaves unclear how often this requirement holds in natural shared memory and how much useful information the defense retains?

AuthorsOpenReview ↗

Reply to Reviewer zoRj

Thank you for reading the revised paper so carefully and for the detailed check of the calibration. We answer both points below.

1. Calibration on the three-account pool

The table below reports Eq. 9 computed on the three-account pool, on the clean pool and on the ten-account pool, with CrowdRAG run end to end at each threshold (GPT-5.4, Contriever, f=3f=3, t=10t=10, 154 topics). Without defense, ASR is 63.6% and ACC is 35.7%.

Pool used for calibrationτs\tau_sASRACC
Three-account pool0.767.8%92.2%
Clean pool, no Sybil claims0.860.6%98.1%
Ten-account pool (paper)0.851.3%96.1%

On the three-account pool the rule picks a lower threshold, because the Sybil claims are 9.3% of the pool and count as unsupported. More adversarial claims then pass Layer 1 (241 of 462), Layer 2 blocks 160 of them, and CrowdRAG still reduces ASR from 63.6% to 7.8%. On the clean pool the rule picks 0.86, close to the 0.85 in the paper, and ASR is 0.6%. The clean pool also gives nearly the same thresholds as the paper for the other three retrievers (0.91, 0.955 and 0.997), so the automatic calibration does not depend on the ten accounts.

2. Natural shared memory

Thank you for raising this valuable question. Our method targets shared memory in which the same fact is written independently by multiple users. Since, to our knowledge, there is currently no public dataset of real multi-user agent memory with user provenance, we study this setting with a controlled benchmark. In this setting, our benchmark and method address the most basic step: whether a small number of Sybil accounts can be blocked when honest users give a fact enough support, and how the system behaves when the support is insufficient. How common such support is in real shared memory, and how much useful information the defense retains, still needs to be validated on real data at a larger scale. This is the direction of our future work, and we will note it in the paper.

Thank you again for the careful reading.

Reviewer_ZYQCOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies targeted knowledge poisoning in a shared memory pool to which multiple users contribute. A small number of Sybil accounts can obtain a disproportionate share of the retrieved context. CrowdRAG uses pool-wide user provenance in two layers: a cosine-neighborhood filter that counts distinct contributing users, followed by a per-topic knowledge graph that blocks a weakly supported assertion when a conflicting value has quorum support. The paper describes an operating condition in which the adversarial account budget is below the support threshold and honest support is above it, and studies loss of coverage on sparse topics.

The evaluation constructs 154 topics from NQ and three enterprise-style datasets, with 300 synthetic honest users, and reports results for three generation models, four retrievers, ablations, flooding, adaptive attack variants and honest disagreement. At three Sybil accounts, the main GPT-5.4 result reduces end-to-end ASR from 63.6% to 1.3%, with 96.1% answer accuracy. A separate Travel StackExchange experiment evaluates topic assignment rather than end-to-end attack defense.

Critical strengths

S1. A useful defense signal for a consequential threat model. The paper clearly separates the number of users who support a claim in the full pool from the number of passages occupying top-k retrieval. The reported amplification from three Sybil accounts to 2.64 of five retrieved slots makes this distinction concrete. Counting unique users also directly addresses repeated contributions from the same account, and the layer design is interpretable.

S2. A substantial controlled empirical effect with meaningful boundary analyses. At f=3 and t=10, CrowdRAG reports end-to-end ASR of 1.3%, 0.6% and 1.9% on the three generation models, with accuracy of 96.1%, 92.9% and 89.6%. The appendix goes beyond a single main table: it includes retriever calibration, clustering-baseline sweeps, user-level flooding, honest disagreement and a real topic-assignment slice. The explicit discussion of sparse-topic coverage and suppression of legitimate minority variants is especially useful for understanding practical limitations.

Critical weaknesses

W1. The stated operating condition is not sufficient for the implemented semantic pipeline. Section 4.3 describes the topic-level quorum window as the precise effective regime, but the defined density counts honest contributors to a topic, not honest supporters of each assertion. Equation 3 can count factually contradictory honest claims as supporters of an adversarial claim, as the paper itself acknowledges. Equation 7 then blocks an attack only if a conflicting correctly canonicalized assertion reaches t; low-support novel assertions and empty or mismatched extractions can pass. Moreover, unequal values do not always imply contradiction for multi-valued or time-dependent predicates. The paper should state the additional assertion-level and extraction assumptions and narrow its guarantee. The main synthetic construction, in which every topic has at least 12 honest contributors and a shared target fact, makes these assumptions unusually favorable. The real slice tests assignment only, and its full population has median density 3, so it does not validate broad end-to-end applicability.

W2. The experimental specification is not yet consistent enough to establish the comparison and the contribution of Layer 2. Section 4.1 filters before retrieval, Algorithm 1 filters an already retrieved top-k, and Appendix Q uses backfill; these protocols can produce different accuracy and preservation numbers. Appendix L reports an end-to-end change of 14.9% to 13.6%, whereas Table 6 reports 7.8% to 1.3% for the Contriever ablation at f=3 and t=10, without explaining the configuration difference. Appendix A also describes k=10 for RobustRAG while the general setup specifies top-5. Finally, the description of RA-RAG as per-document inference-time voting omits its source-level reliability estimation and reliability-based source selection in the original method; Appendix R does not specify these components. A collapse at two of four selected sources is not a general consequence for weighted voting with learned weights. Together with adaptive attacks evaluated only at the retrieval level and without joint retrieval/evasion optimization, these gaps weaken the claims of comparative and adaptive robustness.

W3. Adaptive attack experiments are valuable, yet they are insufficient to support strong security conclusions. The attacks tested by the authors involve a trade-off between "evading defenses and reducing retrieval visibility". However, the current results cannot prove that this trade-off is unavoidable for attackers, as the attack generation process itself restricts the use of retrieval anchors.

Questions for authors
  • Clarification of the guarantee. What assertion-level honest-support condition and assumptions about extraction, canonicalization and predicate functionality are required for Equation 8 to imply rejection of an adversarial claim? How does the implementation handle empty extractions, two conflicting values each above quorum, and legitimate multi-valued or time-qualified assertions?
  • Clarification of the experimental protocol. Which reported tables use pre-filtering, post-top-k filtering or backfill? Are the end-to-end rates computed over 770 responses, 154 topics, or another aggregation?
  • Clarification of calibration. On which pool is Equation 9 computed: clean claims, the active f-account pool, or the full ten-account pool? Does same-topic support use inferred topics or reference labels?
  • Clarification of the source-aware comparison. Does the RA-RAG baseline include iterative cross-source reliability estimation, user-level source grouping, reliability-based source selection and weighted voting?
  • Evidence that could change the assessment. Do existing results include end-to-end ASR and accuracy for the adaptive attacks under the same t=10 deployment configuration, particularly attacks retaining a retrieval anchor while also mimicking honest wording or targeting triple canonicalization?
3 comments
AuthorsOpenReview ↗

Response to Reviewer ZYQC (1/2)

Thank you for the careful and detailed review. We answer W1, W2 and Q1 to Q4 here. The end-to-end adaptive evaluation requested in W3 and Q5 is in the next comment.

W1 / Q1: the guarantee and its assumptions

A: We agree. Eq. 8 is a topic-level necessary condition, and Section 4.3 should not present it as sufficient. Eq. 7 rejects an adversarial claim asserting v′v' for (s,p)(s,p) when the following hold:

  1. Assertion-level support. The correct value vv has quorum in the knowledge graph, ∣U(s,p,v)∣≥t|U(s,p,v)| \ge t. This is stronger than t≤dTt \le d_T.
  2. Extraction and canonicalization. The adversarial claim yields a triple that is resolved to the same (s,p)(s,p), and honest paraphrases of vv are merged.
  3. Functional predicate. (s,p)(s,p) admits a single value.
  4. Budget. f<tf < t, so v′v' stays below quorum.

When condition 1 or 2 does not hold, the claim is subject to Layer 1 only. We will state these assumptions in Section 4.3 and replace "effective precisely in the window".

On the implementation: (a) Empty extraction. The claim has nothing to compare in Layer 2 and is filtered by Layer 1 alone. None of the 6,040 benchmark claims has an empty extraction. (b) Two conflicting values above quorum. Eq. 7 blocks only values below tt, so both are kept and passed to the model. (c) Multi-valued or time-qualified assertions. A legitimate value with fewer than tt supporters is withheld when another value has quorum. Appendix G measures this with granularity, contested and temporal variants for up to 30% of contributors.

On density, we agree that 12 to 50 contributors is favorable. We ran CrowdRAG end to end with per-topic density resampled from the contributor counts of the Travel StackExchange clusters (median 3, single-contributor topics included) and t=f+1t=f+1. ASR falls from 29.9% to 1.9% at f=1f=1 and from 73.1% to 9.7% at f=3f=3. On topics with t≤dTt \le d_T, accuracy is 88 to 92% with 1 to 6% ASR. The full table is in our response to Reviewer ZPAr (W2).

W2 / Q2 to Q4: experimental protocol

A: Thank you for checking these details. Several settings were not labeled in the paper; we list them here and will label them.

  1. Filtering order (Q2). CrowdRAG in the end-to-end tables (Tables 1, 5, 6) follows Section 4.1: the pool is filtered and the top-5 is retrieved from what remains. "Backfill" in Appendix Q is the same procedure. Algorithm 1 omits the step that continues down the ranking until five claims pass, and we will correct it. The retrieval-level tables (Tables 3, 4, 16, 17) are diagnostics of the filter on the unfiltered top-5, so they do not refill.
  2. Aggregation (Q2). End-to-end rates are over the 154 topics, one response per topic ("Each experimental configuration is evaluated over all 154 topics", Appendix R). The 770 queries are used in the retrieval-level tables.
  3. Appendix L and Table 6. These are two operating points. Appendix L discusses the retrieval-level analysis at t=5t=5 (Table 16), and its example of 14.9% to 13.6% is at t=5t=5. Table 6 is the deployed t=10t=10. Section 5.1 (Operating Points) states this split, but Appendix L did not repeat tt; we will label it.
  4. RobustRAG. All methods receive the same retrieval budget, top-5. For RobustRAG each of the five passages is isolated, which gives m=5m=5 groups and tolerates two corrupted passages, while the top-5 holds 2.64 adversarial passages on average at f=3f=3. Appendix A quoted RobustRAG's default (k=10k=10) when explaining the bound; we will rewrite it with this configuration.
  5. Calibration (Q3). Eq. 9 is computed on the full pool, with all ten adversarial accounts present and no adversarial labels. "Same-topic" used the benchmark topic of each claim. With the topics inferred by Eq. 4, which agree for 6,011 of 6,040 claims, the rule selects the same threshold. Counting supporters over the whole pool with no topic information changes the Layer 1 decision for 0 of 4,962 claims at t=5t=5 and for 11 at t=10t=10.
  6. RA-RAG (Q4). Appendix R should have specified this. Our baseline is RA-RAG's inference procedure with each contributing account as one source. Claims are grouped by user, the κ=4\kappa=4 most relevant sources are selected (κ=4\kappa=4 as in the original paper), each source answers independently, abstentions are discarded, and the answers are aggregated by voting. RA-RAG estimates reliability offline ("Prior to deployment, RA-RAG estimates source reliability"). In shared memory, accounts join after deployment and a Sybil account has no history, so the iterative estimation is not run and all sources have equal reliability. In that case, as the RA-RAG paper notes, weighted voting "reduces to majority voting". We agree that the collapse at two of four sources follows from voting over a small pool with equal weights and is not a general property of learned weights. We will label the baseline accordingly and restate the conclusion in Section 5.2.
AuthorsOpenReview ↗

Response to Reviewer ZYQC (2/2): End-to-end adaptive attacks at t=10 (W3, Q5)

W3 / Q5 (and the last point of W2): end-to-end adaptive attacks at t=10t=10, including attacks that retain the retrieval anchor

A: We agree with the reviewer. The trade-off in Section 6.2 between evading the defense and staying retrievable arose because our attack generation removed the retrieval anchor, so it is not unavoidable for an attacker. Following the reviewer's suggestion, we removed this restriction and evaluated the adaptive attacks end to end under the deployment configuration of Table 1 (GPT-5.4, Contriever, f=3f=3, t=10t=10, 154 topics, same pipeline). In addition to the two attacks of Section 6.2, we added two joint attacks that retain the retrieval anchor, which is the query echo of the original attack:

  1. Anchor + honest wording. The query echo followed by a claim that mimics honest wording (the cosine-gaming text of Section 6.2).
  2. Anchor + canonicalization. The query echo followed by a claim that names the subject by an alias or paraphrase and rewords the attribute, so that its triple is hard to merge with the honest entry.
AttackASR, undefendedACC, undefendedASR, CrowdRAGACC, CrowdRAG
Original (Table 1)63.6%35.7%1.3%96.1%
Cosine gaming (Section 6.2)3.9%94.2%0.0%96.1%
Triple evasion (Section 6.2)0.0%98.1%0.0%96.1%
Anchor + honest wording64.3%35.7%3.2%95.5%
Anchor + canonicalization43.5%55.8%1.3%96.1%

We observe that:

(i) The reviewer's point is confirmed. With the anchor retained, the honest-wording attack is as strong as the original attack when undefended (64.3% vs. 63.6% ASR), so mimicking honest wording no longer costs retrieval visibility.

(ii) CrowdRAG reduces this attack to 3.2% ASR with 95.5% accuracy, and the canonicalization attack to 1.3% ASR with 96.1% accuracy.

(iii) The two layers pull the attacker in opposite directions. Mimicking honest wording raises the number of adversarial claims that pass Layer 1 from 49 to 138 (of 462), but a claim worded like the honest ones states its value in the same (subject, attribute) entry, and Layer 2 blocks 117 of the 138. Using an alias can keep a claim out of that entry, but the claim then no longer resembles the honest claims, and only 48 of 462 pass Layer 1.

We will add this table and the two joint attacks to Section 6.2, restrict the trade-off statement to attacks without an anchor, and update Appendix I, which noted that synonymy and aliasing had not been tested.

Thank you for the clarifications and additional experiments. My concern about end-to-end performance on real shared corpora is only partly addressed, so I maintain my original rating.

Reviewer_ZPArOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies Sybil poisoning in multi-user shared agent memory, where a small number of adversarial accounts can dominate retrieved evidence despite being a minority of contributors. It proposes CrowdRAG, combining distinct-user support counting with knowledge-graph-based contradiction detection, using support statistics from the entire memory pool. The paper characterizes the operating condition (f<t≤d_T) and the failure regime for sparsely supported topics.

Critical strengths

1. A useful threat formulation and defense signal. The paper clearly illustrates how a small adversarial population can become a majority in the retrieved context. Counting distinct contributors across the full pool directly addresses this mismatch and explains why inference-time voting can fail. This is a consequential insight for shared-memory systems with authenticated contributor identities.

2. Substantial evaluation within the controlled setting. The paper evaluates multiple models and retrievers, examines adaptive attacks and flooding, and reports both attack success and answer accuracy. The explicit density-cliff analysis and honest-disagreement experiments also expose meaningful limitations rather than presenting source consensus as an unconditional guarantee.

Critical weaknesses

1. Source Consensus Does Not Necessarily Imply Factual Correctness. CrowdRAG primarily relies on the number of distinct users supporting a claim to assess its trustworthiness. However, majority support does not necessarily imply factual correctness. For example, multiple users may repeat the same outdated or incorrect information, while a small number of more reliable users may provide the correct answer. Although Appendix G considers disagreement among honest users, it still largely assumes that the majority of honest users provide the correct answer. The paper lacks sufficient evaluation of scenarios where most users share incorrect information or where users have different levels of reliability. Therefore, we suggest that the authors include additional analysis of these scenarios.

2. Unrealistic Source-Density Assumptions in the Experimental Evaluation. The main experiments assume 12–50 honest contributors per topic, providing a relatively favorable high-density setting for CrowdRAG. However, in real-world multi-user agent memory systems, users often contribute information about different tasks or personal experiences. Many correct memories may have only one or two contributors and could therefore be incorrectly filtered out. Appendix F also shows that the median number of independent contributors across 929 real-world question clusters is substantially lower than in the main experimental setting. However, the authors only evaluate a selected higher-density subset and do not report end-to-end defense performance under realistic sparse memory distributions. Therefore, the current experiments are insufficient to demonstrate the effectiveness of CrowdRAG in real-world multi-user shared memory environments.

Questions for authors

I would appreciate clarification on the issues raised in the Critical Weaknesses section.

2 comments
AuthorsOpenReview ↗

Response to Reviewer ZPAr

We thank the reviewer for the careful reading and for naming two concrete scenarios. Both points are well taken, and they identify evaluations that were missing from the paper. We ran both end to end with the pipeline of Table 1 (GPT-5.4 for generation and judging, top-5 retrieval). The results are below and will be added to the paper.

W1: Consensus vs. correctness (most users share incorrect information; users differ in reliability)

We agree with the reviewer that support from many users does not make a claim true, and that Appendix G only covered the case where the majority is right. CrowdRAG applies a quorum, not a majority vote. A value is removed only when it lacks tt independent supporters (Eq. 3, Eq. 7). It is never removed because another value has more supporters, so a correct answer held by a few reliable users is kept once they reach tt.

Following the reviewer's suggestion, we evaluated exactly this scenario. In every topic the larger group (5 to 15 users) asserts the same incorrect answer, 4 reliable users give the correct one, and 3 Sybil accounts attack (t=4t=4; 153 topics; mean of two resamples).

SettingASRCorrect answer returned
No attack, undefended0.3%40.5%
Attack, undefended70.9%7.2%
Attack, CrowdRAG7.8%34.6%

(i) The defense does not depend on the honest majority being right: ASR falls from 70.9% to 7.8%. (ii) The reliable minority is not suppressed: under attack, CrowdRAG brings the correct-answer rate back from 7.2% to 34.6%, close to the unattacked level (40.5%). This is also a setting where users differ in reliability (an unreliable majority and a reliable minority); Appendix G covers the converse, where 10 to 30% of users assert a competing value.

On scope: CrowdRAG does not adjudicate truth among honest users. Its claim is that an adversary with ff accounts cannot manufacture the appearance of community support; estimating the reliability of honest users is a complementary problem. We will state this in Section 3.3 and replace "majority" with "quorum" in the captions of Figures 3 and 8, which invited the majority-vote reading.

W2: Source density

We agree that our main benchmark is denser than many real shared memories, and that end-to-end results under a sparse distribution should be reported. The operating condition is f<t≤dTf < t \le d_T. The 12 to 50 range is a property of our benchmark, not a requirement of the method. As requested, we report end-to-end performance under a realistic sparse distribution: the honest contributors of every topic are resampled from the contributor counts of the Travel StackExchange duplicate clusters of Appendix F (median 3; single-contributor topics included), with t=f+1t=f+1 (154 topics; two resamples). Each cell reads undefended → CrowdRAG.

ffttASR, all topicsACC, all topicsASR, topics with dT≥td_T \ge tACC, topics with dT≥td_T \ge t
1229.9% → 1.9%69.8% → 80.2%25.7% → 1.1%73.9% → 90.8%
2351.0% → 3.9%48.4% → 60.1%45.4% → 1.0%54.1% → 92.3%
3473.1% → 9.7%26.3% → 42.2%68.5% → 6.2%30.8% → 87.7%

Three observations. (i) Under sparse density the undefended system is more fragile than in our main benchmark: one fake account already yields 29.9% ASR (17.5% in Table 1). (ii) Over all topics, CrowdRAG lowers attack success and raises accuracy at every attacker budget. (iii) Once a topic has tt honest contributors, which here means as few as 2 to 4, CrowdRAG reaches 88 to 92% accuracy with 1 to 6% ASR. The method therefore needs dT≥t>fd_T \ge t > f, not 12 to 50 contributors.

On memories with one or two contributors: with t=2t=2, topics with exactly two honest contributors go from 36.8% to 3.9% ASR and from 61.8% to 80.3% accuracy. A memory with a single contributor cannot be told apart from a single Sybil claim by any count of independent sources; this is the boundary stated in the abstract and Section 6.1, and such topics are included in the "all topics" columns.

We will add this experiment to Appendix F and state in Section 5.1 that the benchmark density is not a requirement.

We hope this addresses both concerns, and we are happy to run further checks.

Thank you for further clarifying the scope and limitations of the method and for providing additional experimental validation.

These efforts have adequately addressed our main concerns, and we have accordingly raised our score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.