- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper proposes TKC, which stores team knowledge with lifecycle, provenance, authority, and access information. It also introduces TeamKnowledgeBench, where answers depend on both the team’s current knowledge and the requester’s permissions.
Critical strengths
- The paper addresses a distinction between retrieving a statement and establishing whether it is current, accepted, and accessible.
- Separating model-generated proposals from deterministic commits is a sensible design.
- Code-derived benchmark answers improve auditability, and the controlled GroupMemBench comparison provides evidence beyond the authors’ own benchmark.
Critical weaknesses
-
Questions are filtered for retrieval failures, limiting broader comparisons.
-
Privacy testing covers only 24 questions and copied spans, not semantic disclosure.
-
The claimed human audit lacks reported results.
-
Ablations combine type, scope, and lifecycle rather than isolating them.
-
Transfer gains lack uncertainty estimates.
Questions for authors
-
Does the advantage persist on questions sampled without retrieval-failure filtering?
-
What completed human-audit results are available?
-
How does TKC compare with permission-filtered retrieval, and does it prevent paraphrased disclosures?
9 comments
2. Human audit and validation (W3, Q2)
W3. The claimed human audit lacks reported results.
Thank you for highlighting this gap. We agree that our original description of the human audit was insufficiently clear, and we sincerely regret this oversight.
Q2. What completed human-audit results are available?
We have now completed it. Three annotators with no role in building the benchmark or the method each answered all 48 items. The items are stratified over the eight task families in both domains (24 Legal, 24 Finance), and the Finance items are split between retrieval-filtered and unfiltered items. For each item, annotators saw the asker, the question, a plain-language label guide, and the conversation excerpt, with the audience of every message marked. They first answered without seeing the reference answer, then saw it and judged it. The analysis plan and four success criteria were fixed before data collection.
- Human accuracy (blind to gold): 81.9% per annotator and 83.3% by majority vote; 92.9% by majority outside the exact-set C1 format.
- Inter-annotator agreement: Fleiss' κ = 0.59 overall and 0.69 outside C1.
- Error counts: a majority judged the gold correct or defensible on all 48 items. Only 2 of 144 individual verdicts were negative (one "incorrect", one "can't tell"). Counting both items as label errors gives 4.2% (2/48), with an exact 95% upper bound of 14.3%.
| Human study (3 annotators, 48 items) | Legal (24) | Finance (24) | Overall (48) | Overall, excluding C1 (42) |
|---|---|---|---|---|
| Annotator accuracy, blind to gold (%) | 79.2 | 84.7 | 81.9 (73.6, 90.3) | 88.1 (80.2, 94.4) |
| Majority-vote accuracy (%) | 83.3 | 83.3 | 83.3 (70.4, 91.3) | 92.9 (81.0, 97.5) |
| Fleiss' κ | 0.46 | 0.64 | 0.59 (0.36, 0.71) | 0.69 (0.46, 0.82) |
| Gold judged correct by majority (%) | 91.7 | 95.8 | 93.8 (83.2, 97.9) | 100.0 (91.6, 100.0) |
| Gold judged correct or defensible by majority (%) | 100.0 | 100.0 | 100.0 (92.6, 100.0) | 100.0 (91.6, 100.0) |
| Items with any "incorrect" or "can't tell" verdict | 0 | 2 | 2 | 2 |
| Worst case: those items counted as label errors (exact 95% upper bound) | 0/24 (≤ 14.2%) | 2/24 (≤ 27.0%) | 2/48 = 4.2% (≤ 14.3%) | 2/42 (≤ 16.2%) |
| Dialogue naturalness (1–5) | 4.0 | 4.0 | 4.0 (all six ratings were 4) | – |
Because TeamKnowledgeBench's gold answers are computed by code from a ledger fixed before dialogue generation, the remaining question is whether independent readers recover the ledger's labels from the conversations, and these results indicate that they do. We will correct the p.7 claim and add the study, its protocol, the annotation interface and per-item results to the appendix.
3. Permission-filtered retrieval and privacy (W2, Q3)
Q3. How does TKC compare with permission-filtered retrieval [...]?
We particularly appreciate this question, as it lets us clarify where TKC sits relative to access-control work.
Related work on permission-aware memory and retrieval. Collaborative Memory (Rezazadeh et al., 2025) gives multi-user agents private and shared memory tiers, with time-varying read and write policies over a user–agent–resource graph. Bhatt et al. (2025) argue that any content used in retrieval or generation must be authorised for all users in an interaction. Permissioned LLMs (Jayaraman et al., 2025) enforce organisational access control on model responses. These build on classic role-based access control (Sandhu et al., 1996) and lattice-based information flow (Denning, 1976).
Where TKC differs. All of these decide who may read a piece of content, given an access policy. None represents whether a piece of content is a proposal or a team decision, who had the authority to settle it, or whether it was later superseded. TKC takes the deployment's surface audiences as given, as a permission filter would, and focuses on a different question: how information from individual conversations becomes established team knowledge while preserving authority, consensus and access boundaries. A strictly filtered retriever makes protected knowledge inaccessible by construction, so it cannot distinguish knowledge that exists but is protected from knowledge that does not exist; TKB treats these as different correct outcomes. And even within accessible conversations, retrieving conflicting proposals does not establish which one was accepted, by whom, or whether it was superseded. Permission filtering determines what a user can access; TKC additionally represents what the team has established.
A direct comparison from existing runs. A permission filter only changes a retriever's output when that retriever surfaced text the requester may not see. On every question where all twelve turns retrieved by BM25-RAG were already visible to the requester, a permission-filtered BM25-RAG would receive exactly the same context and give the same answer, so its accuracy there is known without new runs:
| Questions where a permission filter changes nothing | n | Permission-filtered retriever | TKC | TKC − filtered (95% CI) |
|---|---|---|---|---|
| Legal, BM25-RAG | 47 | 38.3 | 53.2 | +14.9 (−2.1, +31.9) |
| Finance, BM25-RAG | 43 | 48.8 | 60.5 | +11.6 (−2.3, +25.6) |
| Both domains, BM25-RAG | 90 | 43.3 | 56.7 | +13.3 (+2.2, +24.4) |
This subset is not random, and it contains almost no isolation items, so it tests lifecycle, authority and attribution, not withholding. It supports the point above: even when every retrieved turn is accessible, retrieval alone does not establish what the team accepted.
W2. Privacy testing covers only 24 questions and copied spans, not semantic disclosure.
Beyond the 24-question audit, we audited all 10,375 active knowledge items in the two curated stores: every principal entitled to read an item was in the audience of every source turn it cites, with zero violations.
Q3 (cont.) [...] and does it prevent paraphrased disclosures?
Semantic disclosure is a contextual-integrity problem (Nissenbaum, 2004), and benchmarks such as ConfAIde (Mireshghallah et al., 2024) and PrivacyLens (Shao et al., 2024) show that LLMs leak private information they hold in context, even when instructed to protect it. TKC applies its access check before retrieval results are combined, so the content of an item the requester cannot read never reaches the answering model and cannot be paraphrased from there. This removes the in-context leakage those benchmarks measure for directly protected items. It does not prevent indirect disclosure, where a readable item restates protected content. We therefore limit our privacy claim to what is verified: structural isolation of items the requester cannot read, and no copied protected spans in the audit. We do not claim semantic non-disclosure. The paper already states that the leakage test "does not guarantee semantic privacy" (next to Eq. 5), and we will make this scope equally explicit in the abstract and results.
Dear Reviewer RoUy,
We sincerely appreciate your thoughtful review and constructive feedback. We were particularly encouraged that you recognised the distinction between retrieving information and establishing whether it is current, accepted and accessible. This is precisely the problem that motivated TKC. We agree that your concerns are reasonable, and we have conducted additional analyses to address each of them.
1. Retrieval filtering and generalisability (W1, Q1)
We understand the concern that retrieval-based filtering might favour TKC. Only BM25 and Dense-RAG were used for filtering, following GroupMemBench's criterion; the other nine baselines had no influence on question selection, yet TKC outperforms all of them. TKB's gold answers are computed from a ledger fixed before any dialogue is written, independently of any memory system.
Our intention was not to disadvantage retrieval methods, but to evaluate a problem that existing benchmarks largely overlook: retrieving what someone said is different from determining what a team has actually established. Retrieval is effective for document search, but collaborative agentic memory adds authority, consensus and evolving decisions, and the benchmark was designed to test those distinctions. We see TKC as complementary to RAG, not a replacement for it.
We also revisited our results to investigate your concern. On 144 Legal questions outside the too-easy slots, TKC leads by 12.5 points (CI +2.8 to +22.2). Our estimated unfiltered analysis also suggests a positive advantage (an estimate derived from these slices, not a measured run). Importantly, TKC also leads overall across all four domains of the independently developed GroupMemBench, providing complementary evidence beyond our own benchmark. We hope these results help demonstrate that TKC's improvements are not solely a consequence of our benchmark design. TKB is intended as a targeted test of curation-dependent questions, so its accuracies should not be read as natural-frequency estimates; we will state this explicitly in the revision.