acceptodds
Under review as a conference paper at ICLR 2027

Credit Where Credit Isn't Due: Pricing Reputation in Multi-Agent Shared Memory

Abstract

Shared memory enables multi-agent systems to reuse information across tasks and learn source reputation from task feedback. However, when a reader agent answers correctly from its own knowledge even though the facts it retrieved are false, the success rewards the writer of those facts and raises its reputation. Evidence accumulated while a writer was reliable also outlives a change in its behavior, so a writer that turns unreliable keeps its reputation. Our experiments show that when writers turn from correct to false facts, retrieval ranked by reputation learned from raw task outcomes returns a correct fact on only 20.4% of reads, less than half the rate of random selection. To address these problems, we propose Priced Reputation Learning (PRL) for shared knowledge graph memory. PRL credits the writers of a successful task only in proportion to the probability that the same reader would have failed with no memory. PRL further adds a forgetting mechanism so that reputation can follow writers that change. Across six memory conditions and twelve readers from five model families, PRL selects correct facts where reputation learned from raw outcomes serves almost none, and reaches the writer oracle to within reader variation.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Multi-Agent Systems, Large Language Models, LLM Agent Memory

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_tF1qOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper addresses two failure modes in multi-agent LLM shared memory: misleading credit assignment when a reader correctly answers using parametric knowledge instead of retrieved incorrect facts, and stale reputation that persists after a writer changes behavior. The authors propose Priced Reputation Learning (PRL), which discounts positive credit using the reader's closed-book success probability and applies exponential forgetting to adapt to behavioral shifts. Supported by theoretical analysis on reputation lock-in, extensive empirical evaluations and supplementary experiments demonstrate that PRL consistently improves source selection and approaches oracle performance.

Critical strengths
  1. The paper conceptually distinguishes raw task success from actual memory usefulness, highlighting how parametric knowledge can cause naive systems to reinforce incorrect sources. Coupling this with the temporal issue of stale reputation provides a highly coherent, theoretically grounded motivation for PRL's specific mechanisms, rather than relying on arbitrary heuristics.

  2. The supplementary material bolsters the main paper by comprehensively testing complex threat models like redundant/conflicting memories, multi-step reasoning, colluding writers, and reputation farming. The comparison with counterfactual replay baselines proves that PRL not only improves credit assignment but does so at a fraction of the computational cost, confirming the method is both highly effective and efficient.

Critical weaknesses
  1. The credit signal is a task-level heuristic rather than principled causal attribution. While the closed-book baseline effectively corrects the specific leakage phenomenon identified, it merely discounts expected reader-alone success rather than causally isolating which specific retrieved fact produced the outcome. Although supplementary experiments show this approximation works well empirically even in multi-fact settings, the conceptual limitation remains: PRL is essentially a task-level reputation correction rule, not a fundamental solution to fine-grained, causal memory attribution.

  2. Limited conceptual novelty and broader scope. The core components of PRL—baseline discounting and temporal forgetting—are individually standard concepts in credit assignment and reputation systems. While their specific combination effectively solves the constructed LLM memory benchmarks, the paper lacks a sharp conceptual characterization of what fundamentally new principle is being introduced beyond simply applying these existing ideas to a new domain. Consequently, the theoretical formulation feels somewhat narrow in scope despite its empirical success.

Questions for authors
  1. Is PRL intended primarily as a computationally efficient approximation to counterfactual memory attribution, or does the reader-conditioned pricing rule represent a fundamentally different theoretical formulation? Clarifying this would help assess the method's novelty.

  2. Given that the task-level baseline cannot determine which specific served fact produced a success, do you view this as a fundamental limitation of PRL, or do you expect this pricing principle could be extended to fact-level attribution?

  3. Given that the closed-book probe and the memory-enabled task use different prompts and output formats, is the baseline intended to measure the probability of solving the underlying task generically, or the probability of producing a strictly matched answer?

  4. Where exactly does PRL sit within the emerging design space of shared multi-agent memory? How does PRL conceptually position itself against the rapidly emerging literature on multi-agent reliability memory, particularly frameworks like Σ-Mem?

  5. Is the reader-conditioned pricing mechanism fundamentally specific to "shared memory," or is it a more general framework for outcome-based LLM credit assignment?

3 comments

Thank you for the careful reading and constructive feedback.

Our contribution is identifying a new problem, explaining its theory, and providing its solution. Writer reputation learned from task outcomes fails in two ways. First, a reader that already knows the answer succeeds whatever it is served, so its success rewards the served source even when that source is false. Second, the evidence a writer earned while it was reliable outlives a change in its behavior. We identify both failures, derive the exact conditions under which each occurs, and remove them: PRL credits a source only for what the reader could not achieve without it, and lets old evidence expire.

For W1: The price is a well-defined causal contrast, and it estimates the exact quantity that writer selection requires. Specifically, the credit 1−p^0(xt)1 - \hat p_0(x_t) compares two outcomes of the same reader on the same task: its performance with the served memory versus without it. That contrast measures the net effect of serving memory on task xtx_t. Whether selection favors or filters out a false source depends on the sign of that source's expected reputation drift, and Section 3.4 derives the exact condition under which pricing makes this drift negative. Moreover, whenever a read serves a single fact (97.5% of scored PopMix reads), this task-level price coincides with the leave-one-out effect of that specific fact.

For W2. "Limited conceptual novelty."

Our manuscript introduces a new failure formulation, a theoretical analysis of when it occurs, and a coupled update rule derived directly from that analysis.

1. A previously unexamined failure mode. In shared memory, outcome-based credit is confounded by the reader's own competence: a reader that already knows the answer succeeds regardless of what it retrieves, and that success rewards the served writer even when its fact is false. Existing reliability rules fail under this leak: unpriced Beta reputation, iP-TD, and both Credibility Scoring variants serve the correct fact on only 0.3% of attacked PopMix reads.

2. Exact lock-in conditions. Section 3.4 derives closed-form conditions for when a false source locks in under unpriced credit (q>1/2q > 1/2), under priced credit (q>1/(2−pˉ0)q > 1/(2 - \bar p_0)), and under any constant price. Verified against recorded traces, these thresholds accurately predict both where unpriced rules lock in PopMix experiment and where no lock-in occurs because the reader cannot solve the task alone in TOFU and MQuAKE experiments.

3. Why the update rule differs from a variance baseline. Unlike a variance reducing baseline in policy gradients, reputation here is a ratio of accumulated positive and negative evidence. Pricing therefore flips the sign of the expected drift and shifts the fixed point of source selection. Pricing and forgetting also play non-interchangeable roles: the lock-in threshold is independent of λ\lambda, while pricing alone cannot clear evidence accumulated during an earlier honest phase. In our new four-phase experiment with three reliability reversals (Appendix H.5), only the full PRL rule recovers the newly reliable writer after every reversal, whereas pricing alone and forgetting alone never switch back.

Q1. Approximation of counterfactual attribution, or a different formulation?

It is a fundamentally different formulation rather than an approximation. Post-hoc replay attribution aims to explain a single past outcome fact by fact, whereas PRL is an online learning rule designed to govern future source selection. Accordingly, our theoretical guarantee characterizes the drift of writer reputation under repeated selection (Section 3.4), a property that per-task attribution does not address. Empirically, on the same memory store, PRL is both more accurate and substantially cheaper than counterfactual replay:

PopMix, Qwen3 8BAccuracy (%)Extra model calls
Deletion replay (MemAudit)65.81,152
Substitution replay89.6352
PRL92.9238

Q2. Fundamental limitation, or extensible to fact level?

The pricing principle extends directly to individual writers in multi-writer reads, and we implemented and tested it. Even at the task level, PRL already ranks every reliable writer above every unreliable one across all three multi-fact FEVER benchmarks. When finer attribution is needed on a multi-writer read, we replace the empty-memory probe in Equation 1 with a probe leaving one writer out: writer ww receives cw,t=1−p^0,−w(xt)c_{w,t} = 1 - \hat p_{0,-w}(x_t) on success, where p^0,−w(xt)\hat p_{0,-w}(x_t) probes the reader with the other served writers' facts kept in context. In a controlled co-serving test where colluding writers are co-served on every read and share credit on every success, this per-writer price selects the correct version on 97–100% of attacked reads and matches the writer oracle within paired confidence intervals on every seed (Section 3.2, Table 5, and Appendix H.6).

Q3. What does the baseline measure?

We design our baseline to measure the probability that the same reader produces an answer accepted by the exact same task checker without memory. Because the closed-book probe is graded by the identical checker used for the task with previous memory (zt,j,yt∈{0,1}z_{t,j}, y_t \in \{0,1\}), the baseline and the task outcome lie on the exact same strict-match scale. While the task with memory runs the full multi-agent workflow and the probe queries the reader directly, the two agree closely when memory is empty (63.0% for the single-call probe vs. 60.8% for the full workflow with empty memory; Appendix H.7). This gap is negligible relative to the safety margin in Section 3.4: avoiding lock-in requires a mean baseline above 0.31, and the measured value on PopMix is 0.84–0.86. Increasing the number of probes per task does not improve full PRL either, yielding mean accuracies of 91.8%, 91.4%, and 90.8% for k=1,3,5k = 1, 3, 5 across three seeds (Appendix H.4).

Q4. Position relative to Σ-Mem and the design space.

PRL modifies the credit signal from where reliability is learned, which is orthogonal to how reliability state is parameterized. In the broader design space, agreement-based methods fail under false majorities, raw outcome-based updates suffer from reader-knowledge lock-in, and counterfactual replay requires hundreds of extra model calls. Because pricing operates on the feedback signal itself, it serves as a plug-in improvement across different reliability representations on the same store, including our adaptation of Σ-Mem (Table A2):

PopMix accuracy (%), Qwen3 8BRaw outcome creditPriced creditGain
Beta reputation per writer66.386.7+20.4
Credibility Scoring (multiplicative score per writer)65.490.8+25.4
Σ-Mem adaptation (decayed reliability per writer and relation)80.485.4+5.0

Full details and a structured design-space comparison are provided in Appendix H.7.

Q5. Specific to shared memory?

PRL is a general credit-assignment principle. The same confounding arises whenever a consumer agent is sufficiently competent to succeed on its own and outcome feedback determines which source is selected next, such as retrieval source ranking, tool or peer routing, and multi-agent debate aggregation. Shared memory is where the failure is most severe, because stored sources persist across tasks and selection feedback compounds the leak into permanent lock-in.

Thank you for the detailed rebuttal. It has substantially addressed my concerns. I maintain my original score of 3.

Reviewer_gwRkOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies writer reputation in a shared knowledge-graph memory for multi-agent systems. It identifies two failure modes of outcome-only credit. Priced Reputation Learning (PRL) reduces positive credit according to the same reader’s closed-book performance and discounts old evidence toward a prior. The method is evaluated in PopQA, FEVER, TOFU, and MQuAKE-derived settings, including false-majority, selective-corruption, changing-writer, and reader-knowledge conditions. The main experiments use Qwen3 8B, while PopMix is also tested with twelve readers from five model families.

Critical strengths
  1. The paper separates reader-knowledge leakage from stale reputation after a writer changes behavior. This decomposition makes the failure of raw task-outcome credit understandable and motivates two mechanisms with distinct roles.
  2. The changing-writer, fabricated-name, reputation-farming, and reader-variation experiments test the proposed mechanisms directly, while the component ablation and three-seed replication show that pricing and forgetting each contribute to the improvement.
Critical weaknesses
  1. Task credit is divided among served writers by fact count. If one correct fact is sufficient, a false or irrelevant co-served fact can still earn credit; a reader failure penalizes all served writers. FEVER already includes multi-fact reads, but its aggregate results do not establish how reliably PRL separates useful from harmful writers in these cases.
  2. Only served writers receive updates, including forgetting. A demoted writer can therefore remain excluded after becoming reliable again while another writer stays ranked higher. The changing-writer experiments primarily test deterioration from correct to false contributions. They do not establish recovery after demotion or adaptation to repeated reversals, which limits the evidence for tracking changing reliability.
Questions for authors
  1. In controlled mixed-writer reads where a correct fact is sufficient and an incorrect fact is also served, how do writer reputations evolve? Can the authors report source-selection quality as well as task accuracy?
  2. Can a writer recover its rank after an unreliable phase without explicit exploration? Results for a reliability reversal after demotion would clarify whether feedback-triggered forgetting is sufficient.
7 comments

Thank you for the careful reading and constructive feedback.

We ran both controlled experiments you requested and report source selection quality alongside task accuracy. In both settings, PRL reliably recovers the correct source, whereas unpriced credit stays locked into the false one. All revisions in the updated PDF are highlighted in blue, with the new results summarized in Table 5 (Section 4.5) and detailed in Appendix H.5–H.6 of the supplementary material.

W1 / Q1. Shared credit in mixed-writer reads.

PRL separates useful knowledge from harmful writers even when task credit is shared. In the controlled co-serving test you suggested, task-level PRL already breaks lock-in, and applying the same pricing rule per writer reaches the writer oracle.

Multi-fact reads in the existing benchmarks. In the three FEVER conditions of Table 1, each read already serves three to five facts drawn from multiple writers. Even with task credit split by fact count, PRL ranks every reliable writer above every unreliable one in all three conditions and serves the correct fact on 94.9%, 98.8%, and 95.3% of attacked reads. By contrast, unpriced credit inverts the writer ranking in two of the three conditions.

Controlled co-serving experiment. To isolate co-serving directly, each read in this test serves one version of the queried relation and one fact from a second relation, with equal credit share (120 entities, 240 scored reads; full setup in Appendix H.6). Colluding writers supply a false version of the queried relation after the initial 20% honest phase, but are also co-served on every read through the second relation (either true or false), allowing them to share credit whenever the reader succeeds.

Qwen3 8B, three seedsUnpricedPRL (per-task price)PRL (per-writer price)Writer oracle
Accuracy (%), co-served fact true67.9–68.885.4–86.393.8–95.093.8–97.5
Accuracy (%), co-served fact false66.2–69.684.6–87.192.5–95.093.3–94.2
Correct version served on attacked reads (%)067–8497–100100

Unpriced credit never switches back to the reliable writers (0% on attacked reads). Standard task-level PRL breaks this lock-in and improves accuracy by 15.4 to 29.2 points across all seeds and variants, even though the colluders receive half the credit on every successful read.

Furthermore, the pricing rule in Equation 1 extends naturally from the task level to individual writers. Equation 1 compares the task outcome against the reader's closed-book success without memory. When a read serves multiple writers, we can make the exact same comparison per writer ww by probing the reader without ww's facts while keeping the other served writers' facts in context: cw,t=1−p^0,−w(xt)c_{w,t} = 1 - \hat p_{0,-w}(x_t) on success and −1-1 on failure, which reduces to Equation 1 when only one writer is served. Under this per-writer price, a co-served fact that the reader did not need earns zero credit. As shown in the table, this raises correct source selection to 97–100% and matches the writer oracle within paired confidence intervals on every seed. We keep task-level pricing as the default because the two rules are identical whenever a read serves a single fact, which holds for 97.5% of scored PopMix reads while per-writer pricing adds probes only on successful reads that mix multiple writers.

How writer reputations evolve. The table below shows final reputations when the colluders' co-served fact is true (Qwen3 8B, seed 42; prior 0.50):

RuleReliable writersColluders
Unpricednever served again, stay at 0.500.67, 0.69
PRL (per-task price)0.60, 0.57, 0.600.51, 0.58
PRL (per-writer price)0.64, 0.62, 0.630.49, 0.49

Under unpriced credit, the colluders accumulate credit from tasks the reader would have solved anyway, leaving the reliable writers permanently unserved at 0.50. Under task-level PRL, the reliable writers are recovered and at 0.57–0.60; with per-writer pricing, the colluders are pushed below the 0.50 prior and below every reliable writer.

Shared blame on failed reads. Conversely, although a failed task penalizes all served writers, shared blame does not lower a reliable co-served source. If a separate neutral writer supplies the second relation instead, its reputation at 0.57–0.63 across all three seeds in our experiment, well above both the 0.50 prior and the colluders. Task-level PRL then matches the oracle in accuracy (95.8% vs. 95.4%).

W2 / Q2. Recovery after demotion.

Yes. A demoted writer recovers its top rank after every reliability reversal without any exploration. We can consider two writers, A and B, where A has been demoted below B. Because A is not served, its reputation remains frozen at the value it held when demoted. Meanwhile, the incumbent B is served on every read and continues to receive feedback. If B remains reliable, the reader receives correct facts and no switch is needed. Once B turns unreliable, however, failed tasks add negative credit while forgetting decays the positive evidence B accumulated during its earlier honest phase. As soon as B's reputation drops below A's frozen score, retrieval automatically switches back to A. Thus, rank recovery is driven by the incumbent falling rather than by exploring excluded writers.

We tested this directly using two writers that hold competing versions of every fact and swap roles between correct and false three times across four phases of 60 reads, with no exploration (details in Appendix H.5). Below, switch delay measures the number of reads after each reversal before the newly correct writer is served again ("never returns" indicates 0 reads served in that 60-read phase):

Reader, seedSwitch delay (reads) after reversal 1 / 2 / 3, PRLAccuracy (%), PRLAccuracy (%), priced without forgettingAccuracy (%), unpriced with forgetting
Qwen3 8B, 4217 / 21 / 1987.577.1 (never returns)77.1 (never returns)
Qwen3 8B, 4319 / 21 / 1988.375.8 (never returns)74.6 (never returns)
Qwen3 8B, 4419 / 21 / 1387.575.0 (never returns)75.8 (never returns)
Qwen3 4B, 4223 / 10 / 787.574.6 (never returns)73.3 (never returns)

Across all twelve reversals we test on three seeds on Qwen3 8B and one on Qwen3 4B, PRL switches back to the newly reliable writer within 7–23 reads, and subsequently serves the correct writer on every remaining read of the phase in eleven of the twelve cases. Neither component can achieve this alone: without forgetting, old positive evidence keeps the first writer in place; without pricing, reader-knowledge leakage continues to reward the incumbent even after it turns false. Consequently, both ablations fail to recover in the phases where the first writer turns false, and full PRL outperforms them by 10.4 to 14.2 points across all ten paired comparisons (p≤0.003p \le 0.003).

Thank you for the response and the additional experiments. I remain concerned about credit assignment in mixed-writer reads. The default PRL achieves 67–84% correct-source selection; the near-oracle results come from the new per-writer variant and should be distinguished from those of the original method. A reliable writer’s final reputation remaining above the prior does not show that shared penalties are harmless. The reversal experiment supports recovery in the tested setting, but recovery still requires the incumbent’s score to fall below the excluded writer’s frozen score, which is not guaranteed. Please also clarify which settings yield the reported 15.4–29.2 percentage-point improvement, as I cannot reconcile this range with the displayed results.

Reviewer_bTeCOpenReview ↗
Rating
3
Confidence
1
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

The paper studies writer-reputation learning in shared knowledge-graph memory for multi-agent LLM systems, where several writers store conflicting facts and a reader ranks them by reputation learned from task outcomes. Section 3.4 identifies two failure modes of outcome-based credit: a "leak," where the reader answers correctly from its own knowledge and a false source is credited, and a "lock-in," where evidence from a writer's honest phase keeps a now-false writer favored. The proposed Priced Reputation Learning, or PRL, prices positive credit by 1−p^01-\hat{p}_0, where p^0\hat{p}_0 is estimated from k closed-book probes of the same reader as in Eq. 1, and discounts accumulated evidence with a forgetting factor λ as in Eq. 2. Table 1 covers six memory conditions on Qwen3 8B and Table 2 covers twelve readers on PopMix at h = 20%. For Qwen3 8B on attacked PopMix reads, the served-correct metric is 95.0% for PRL versus 0.3% for unpriced credit, iP-TD, and both Credibility Scoring variants. The paper reports close oracle performance for most readers, with exceptions.

Critical strengths
  1. The failure analysis is sharp, and the small theory is checked against measured traces. Section 3.4 derives the lock-in condition for unpriced credit, q>1/2q > 1/2, for priced credit, q>12−pˉ0q > \frac{1}{2-\bar{p}_0}, and for a constant price. On recorded Qwen3 8B PopMix traces, the measured q lies between 0.59 and 0.64 and pˉ0\bar{p}_0 between 0.84 and 0.86, which falls in the predicted regime where unpriced credit locks in and priced credit does not, matching the observed served-correct rates of under 1% versus about 95%. This tie between theory and measurement is the main reason I find the mechanism convincing.

  2. The evaluation is broad, honest about failure cases, and the method is cheap. Table 2 tests twelve readers from five model families, and Sections 4.3 and 4.6 report cases where PRL loses: on GPT-OSS 20B unpriced credit is the best rule, Gemma 3 4B falls below unfiltered retrieval, and on MQuAKE the reader uses the served edited fact in place of its own knowledge on only about a third of the relevant reads. Table 3 ablates pricing and forgetting separately, and the method adds only k cached closed-book probes, with k = 1 by default.

Critical weaknesses
  1. The evaluation covers two slices rather than a full condition-by-reader grid. All six conditions in Table 1 use Qwen3 8B, whereas Table 2 tests twelve readers only on PopMix at h = 20%. Cross-reader robustness on other corruption conditions remains untested in the paper as submitted. Tables 1 and 2 also omit intervals for the reported oracle comparisons. Figure 5a does include paired intervals against unfiltered retrieval, but those do not verify the separate claim of oracle-level performance. In particular, the 92.9% versus 95.4% Qwen3 8B comparison needs its paired interval and seed information; the paper directs these details to supplementary material.

  2. The default probe estimator and the constant-price comparison deserve a clearer empirical treatment. At k = 1, the cached baseline estimate and positive credit are binary for each task. This remains a valid, potentially noisy estimate of the closed-book success probability, so the issue is robustness to estimator noise rather than the validity of probability pricing itself. Section 4.5 reports 85.4% with three probes versus 86.7% with one probe at h = 20% specifically without forgetting. This does not establish the effect of k in full PRL, and uncertainty is needed before interpreting the difference. The per-task versus constant-price comparison is also summarized only briefly in the main paper and deferred to supplementary material.

Questions for authors
  1. Please report the seeds and paired confidence intervals for Tables 1 and 2, especially the Qwen3 8B difference from the writer oracle, 92.9% versus 95.4%.

  2. Can you include the constant-price comparison for several readers in the main results, showing both a transferred constant and a reader-retuned constant alongside the per-task price?

  3. Can you report probe-count sensitivity for full PRL with forgetting, alongside the 85.4% versus 86.7% no-forgetting comparison, with uncertainty across runs?

  4. Can you test multiple readers on at least one non-PopMix corruption condition?

1 comment

Thank you for the careful reading and the constructive comments.

We ran all four requested analyses, and each one supports the corresponding statement in the paper. The revised PDF marks changes in blue; we report in new tables in Section 4.5 and in Appendix H of the supplementary material.

Q1 (W1). Seeds and paired intervals for Tables 1 and 2. The interval against the writer oracle includes zero in all six conditions of Table 1 and on eleven of the twelve readers of Table 2. Both tables use seed 42. The Qwen3 8B difference, 92.9 versus 95.4, is −2.5 (95% interval −5.8 to +0.4). The one exception is GPT-OSS 20B, which Section 4.3 already discusses. The supplementary material reports three seeds for Qwen3 8B on PopMix in Table A10; over these seeds our method stays 2.5 to 3.3 points from the oracle, 7.5 to 9.6 points above unfiltered retrieval, and 22.1 to 28.3 points above unpriced credit. The oracle is given writer reliability from the first read; PRL learns it from task feedback during the stream.

Q2 (W2). Constant prices on several readers, in the main results. The comparison is now Table 4 in Section 4.5, with a transferred constant (c = 0.7) and a reader-retuned constant on five readers: no constant is significantly ahead of the per-task price in any paired comparison, while transferred and retuned constants each fall significantly behind it. A transferred constant c = 0.7 is behind the per-task price in seven of nine reader-seed rows. A constant retuned to each reader (one minus its own mean closed-book baseline) is 5.8 points behind on Qwen3 8B at seed 42, and retuning needs the same probes the per-task price uses. The per-task price is the only rule with nothing to set per reader. This is the statement of Section 4.5: a constant works only inside a bound that moves with the reader. Paired differences are in Appendix H.3 (Table A22).

Q3 (W2). Probe count in full PRL. With forgetting, three or five probes do not improve on one probe over three seeds. Five of the six paired differences from k = 1 are not significant and the sixth favors k = 1, please see Appendix H.4 for details.

Accuracy (%), Qwen3 8B, PopMixSeed 42Seed 43Seed 44Mean
PRL, k = 192.993.389.291.8
PRL, k = 392.990.490.891.4
PRL, k = 593.888.890.090.8

One probe is enough because the price is linear in the estimate: the k = 1 credit is binary per task but unbiased for 1−p0(xt)1 - p_0(x_t), and reputation sums it over tasks, so the update itself averages the noise. Section 3.2 now states this.

Q4 (W1). Multiple readers on a non-PopMix condition. We tested four readers, Qwen3 4B, 8B, 14B, Llama 3.1 70B, on two conditions. PRL significantly outperform unfiltered retrieval on every reader, above unpriced credit on FEVER, please see Appendix H.2 for details.

Accuracy (%), seed 42FEVER selective corruption: Unfiltered / Unpriced / PRL / OraclePRL − Unfiltered (95% interval)TOFU changing writers: Unfiltered / Unpriced / PRL / OraclePRL − Unfiltered (95% interval)
Qwen3 4B87.9 / 94.2 / 97.1 / 96.7+9.2 (+5.0 to +13.7)66.2 / 97.8 / 97.8 / 98.8+31.6 (+25.9 to +37.2)
Qwen3 8B75.8 / 71.7 / 92.5 / 90.0+16.7 (+11.2 to +22.5)55.3 / 96.6 / 96.9 / 97.5+41.6 (+35.6 to +47.8)
Qwen3 14B72.1 / 87.1 / 90.0 / 91.2+17.9 (+9.2 to +26.3)56.6 / 97.2 / 95.9 / 98.4+39.4 (+32.5 to +46.6)
Llama 3.1 70B51.2 / 56.2 / 61.7 / 63.7+10.4 (+2.1 to +19.2)55.3 / 88.7 / 89.4 / 90.0+34.1 (+26.9 to +41.2)

The two conditions also confirm the analysis across readers: unpriced credit matches PRL on TOFU within 1.3 points on every reader, where the reader knows none of the facts and Section 3.4 predicts no lock-in, and falls 26.7 to 33.3 points below PRL on PopMix for the same four readers.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.