acceptodds
Under review as a conference paper at ICLR 2027

Knowledge Curation for Multi-User Collaboration in AI Workspaces

Abstract

As AI workspaces move from personal, single-user use to shared, team-level projects, they need a memory that captures what a team knows, who can access it, how it was established, and whether it remains valid. Existing memory mechanisms and evaluations largely target storing and retrieving a single user’s long-context history. However, team settings pose a different challenge, as the knowledge is distributed across conversations, meetings, and shared documents, with evolving content and varying visibility. We introduce Team Knowledge Curation (TKC), a framework that represents shared knowledge as scoped items carrying epistemic type, lifecycle state, provenance, and access constraints, curated from raw conversation and maintained through periodic consolidation. To evaluate this setting, which existing benchmarks do not do, we introduce TeamKnowledgeBench (TKB), a benchmark designed to capture team-specific knowledge dynamics and where access denial and abstention are first-class correct answers. Across TKB and an independent GroupMemBench, TKC outperforms existing memory and retrieval systems and is the only system that withholds protected content without leaks, suggesting that reliable collaborative memory requires curating current what the team knows, beyond retrieving what was said. Together, TKC and TKB lay a foundation for AI systems that support shared, evolving team knowledge.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Memory, LLMs, Collaboration

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_RoUyOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes TKC, which stores team knowledge with lifecycle, provenance, authority, and access information. It also introduces TeamKnowledgeBench, where answers depend on both the team’s current knowledge and the requester’s permissions.

Critical strengths
  1. The paper addresses a distinction between retrieving a statement and establishing whether it is current, accepted, and accessible.
  2. Separating model-generated proposals from deterministic commits is a sensible design.
  3. Code-derived benchmark answers improve auditability, and the controlled GroupMemBench comparison provides evidence beyond the authors’ own benchmark.
Critical weaknesses
  1. Questions are filtered for retrieval failures, limiting broader comparisons.

  2. Privacy testing covers only 24 questions and copied spans, not semantic disclosure.

  3. The claimed human audit lacks reported results.

  4. Ablations combine type, scope, and lifecycle rather than isolating them.

  5. Transfer gains lack uncertainty estimates.

Questions for authors
  1. Does the advantage persist on questions sampled without retrieval-failure filtering?

  2. What completed human-audit results are available?

  3. How does TKC compare with permission-filtered retrieval, and does it prevent paraphrased disclosures?

9 comments

Dear Reviewer RoUy,

We sincerely appreciate your thoughtful review and constructive feedback. We were particularly encouraged that you recognised the distinction between retrieving information and establishing whether it is current, accepted and accessible. This is precisely the problem that motivated TKC. We agree that your concerns are reasonable, and we have conducted additional analyses to address each of them.

1. Retrieval filtering and generalisability (W1, Q1)

W1. Questions are filtered for retrieval failures, limiting broader comparisons.

We understand the concern that retrieval-based filtering might favour TKC. Only BM25 and Dense-RAG were used for filtering, following GroupMemBench's criterion; the other nine baselines had no influence on question selection, yet TKC outperforms all of them. TKB's gold answers are computed from a ledger fixed before any dialogue is written, independently of any memory system.

Our intention was not to disadvantage retrieval methods, but to evaluate a problem that existing benchmarks largely overlook: retrieving what someone said is different from determining what a team has actually established. Retrieval is effective for document search, but collaborative agentic memory adds authority, consensus and evolving decisions, and the benchmark was designed to test those distinctions. We see TKC as complementary to RAG, not a replacement for it.

Q1. Does the advantage persist on questions sampled without retrieval-failure filtering?

We also revisited our results to investigate your concern. On 144 Legal questions outside the too-easy slots, TKC leads by 12.5 points (CI +2.8 to +22.2). Our estimated unfiltered analysis also suggests a positive advantage (an estimate derived from these slices, not a measured run). Importantly, TKC also leads overall across all four domains of the independently developed GroupMemBench, providing complementary evidence beyond our own benchmark. We hope these results help demonstrate that TKC's improvements are not solely a consequence of our benchmark design. TKB is intended as a targeted test of curation-dependent questions, so its accuracies should not be read as natural-frequency estimates; we will state this explicitly in the revision.

Slice (existing outputs)nTKCStrongest baselineTKC − strongest (95% CI)BM25-RAGTKC − BM25-RAG (95% CI)
Legal, retrieval-solvable slots7174.6HippoRAG-2 74.60.0 (−9.9, +9.9)69.0+5.6 (−5.6, +16.9)
Legal, all other slots14453.5Mem0 41.0+12.5 (+2.8, +22.2)34.0+19.4 (+9.7, +29.2)

2. Human audit and validation (W3, Q2)

W3. The claimed human audit lacks reported results.

Thank you for highlighting this gap. We agree that our original description of the human audit was insufficiently clear, and we sincerely regret this oversight.

Q2. What completed human-audit results are available?

We have now completed it. Three annotators with no role in building the benchmark or the method each answered all 48 items. The items are stratified over the eight task families in both domains (24 Legal, 24 Finance), and the Finance items are split between retrieval-filtered and unfiltered items. For each item, annotators saw the asker, the question, a plain-language label guide, and the conversation excerpt, with the audience of every message marked. They first answered without seeing the reference answer, then saw it and judged it. The analysis plan and four success criteria were fixed before data collection.

  • Human accuracy (blind to gold): 81.9% per annotator and 83.3% by majority vote; 92.9% by majority outside the exact-set C1 format.
  • Inter-annotator agreement: Fleiss' κ = 0.59 overall and 0.69 outside C1.
  • Error counts: a majority judged the gold correct or defensible on all 48 items. Only 2 of 144 individual verdicts were negative (one "incorrect", one "can't tell"). Counting both items as label errors gives 4.2% (2/48), with an exact 95% upper bound of 14.3%.
Human study (3 annotators, 48 items)Legal (24)Finance (24)Overall (48)Overall, excluding C1 (42)
Annotator accuracy, blind to gold (%)79.284.781.9 (73.6, 90.3)88.1 (80.2, 94.4)
Majority-vote accuracy (%)83.383.383.3 (70.4, 91.3)92.9 (81.0, 97.5)
Fleiss' κ0.460.640.59 (0.36, 0.71)0.69 (0.46, 0.82)
Gold judged correct by majority (%)91.795.893.8 (83.2, 97.9)100.0 (91.6, 100.0)
Gold judged correct or defensible by majority (%)100.0100.0100.0 (92.6, 100.0)100.0 (91.6, 100.0)
Items with any "incorrect" or "can't tell" verdict0222
Worst case: those items counted as label errors (exact 95% upper bound)0/24 (≤ 14.2%)2/24 (≤ 27.0%)2/48 = 4.2% (≤ 14.3%)2/42 (≤ 16.2%)
Dialogue naturalness (1–5)4.04.04.0 (all six ratings were 4)–

Because TeamKnowledgeBench's gold answers are computed by code from a ledger fixed before dialogue generation, the remaining question is whether independent readers recover the ledger's labels from the conversations, and these results indicate that they do. We will correct the p.7 claim and add the study, its protocol, the annotation interface and per-item results to the appendix.

3. Permission-filtered retrieval and privacy (W2, Q3)

Q3. How does TKC compare with permission-filtered retrieval [...]?

We particularly appreciate this question, as it lets us clarify where TKC sits relative to access-control work.

Related work on permission-aware memory and retrieval. Collaborative Memory (Rezazadeh et al., 2025) gives multi-user agents private and shared memory tiers, with time-varying read and write policies over a user–agent–resource graph. Bhatt et al. (2025) argue that any content used in retrieval or generation must be authorised for all users in an interaction. Permissioned LLMs (Jayaraman et al., 2025) enforce organisational access control on model responses. These build on classic role-based access control (Sandhu et al., 1996) and lattice-based information flow (Denning, 1976).

Where TKC differs. All of these decide who may read a piece of content, given an access policy. None represents whether a piece of content is a proposal or a team decision, who had the authority to settle it, or whether it was later superseded. TKC takes the deployment's surface audiences as given, as a permission filter would, and focuses on a different question: how information from individual conversations becomes established team knowledge while preserving authority, consensus and access boundaries. A strictly filtered retriever makes protected knowledge inaccessible by construction, so it cannot distinguish knowledge that exists but is protected from knowledge that does not exist; TKB treats these as different correct outcomes. And even within accessible conversations, retrieving conflicting proposals does not establish which one was accepted, by whom, or whether it was superseded. Permission filtering determines what a user can access; TKC additionally represents what the team has established.

A direct comparison from existing runs. A permission filter only changes a retriever's output when that retriever surfaced text the requester may not see. On every question where all twelve turns retrieved by BM25-RAG were already visible to the requester, a permission-filtered BM25-RAG would receive exactly the same context and give the same answer, so its accuracy there is known without new runs:

Questions where a permission filter changes nothingnPermission-filtered retrieverTKCTKC − filtered (95% CI)
Legal, BM25-RAG4738.353.2+14.9 (−2.1, +31.9)
Finance, BM25-RAG4348.860.5+11.6 (−2.3, +25.6)
Both domains, BM25-RAG9043.356.7+13.3 (+2.2, +24.4)

This subset is not random, and it contains almost no isolation items, so it tests lifecycle, authority and attribution, not withholding. It supports the point above: even when every retrieved turn is accessible, retrieval alone does not establish what the team accepted.

W2. Privacy testing covers only 24 questions and copied spans, not semantic disclosure.

Beyond the 24-question audit, we audited all 10,375 active knowledge items in the two curated stores: every principal entitled to read an item was in the audience of every source turn it cites, with zero violations.

Q3 (cont.) [...] and does it prevent paraphrased disclosures?

Semantic disclosure is a contextual-integrity problem (Nissenbaum, 2004), and benchmarks such as ConfAIde (Mireshghallah et al., 2024) and PrivacyLens (Shao et al., 2024) show that LLMs leak private information they hold in context, even when instructed to protect it. TKC applies its access check before retrieval results are combined, so the content of an item the requester cannot read never reaches the answering model and cannot be paraphrased from there. This removes the in-context leakage those benchmarks measure for directly protected items. It does not prevent indirect disclosure, where a readable item restates protected content. We therefore limit our privacy claim to what is verified: structural isolation of items the requester cannot read, and no copied protected spans in the audit. We do not claim semantic non-disclosure. The paper already states that the leakage test "does not guarantee semantic privacy" (next to Eq. 5), and we will make this scope equally explicit in the abstract and results.

Reviewer_zxGDOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies memory for multi-user AI workspaces and proposes Team Knowledge Curation (TKC). It is a framework for managing shared knowledge in multi-user AI workspaces. TKC represents knowledge using epistemic types, lifecycle states, provenance, relationships, and access permissions. To maintain a structured knowledge graph, TKC uses LLMs to extract and consolidate content. The paper also introduces TeamKnowledgeBench (TKB), which is designed to evaluate team knowledge organization (including how to present and curate knowledge and track provenance). Experimental results show that TKC outperforms several agent memory and retrieval-based baselines on TKB and GroupMemBench.

Critical strengths
  1. The design of the symbolic ledger in TKB is useful. It is constructed based on team knowledge states and events before generating natural language conversations. This can help maintain consistency across knowledge updates and provenance while making the evaluation more auditable. Some additional validation steps are also introduced to check whether conversations faithfully reflect the ledger.
Critical weaknesses
  1. The technical novelty beyond combining many components is limited, and the results make it difficult to determine whether the complexity of these combinations is necessary or whether simpler structured memory approaches could achieve comparable performance. More specifically, the TKB design is closely aligned with the idea of TKC and filters out some questions that can be answered by simple retrieval baselines. Therefore, this may introduce a selection bias in favor of TKC. Though the authors also evaluate on GroupMemBench, the improvements are concentrated in Term Ambiguity and Abstention. On other tasks, such as Temporal Reasoning, TKC shows slight or no improvements.
  2. The evaluation of memory maintenance costs is insufficient. TKC has higher ingestion and per-query costs than lightweight RAG baselines, but the paper doesn't analyze the cost of incremental memory updates and periodic consolidation as new conversations arrive. It is unclear whether it is worthwhile to introduce the system complexity in practical systems.
Questions for authors
  1. Since TKB is constructed around TKC and explicitly filters out some questions solvable by retrieval, could the authors evaluate TKC on an unfiltered set to better assess potential benchmark bias?
  2. Could the authors provide more detailed ablations to clarify which components of TKC are responsible for the gains on Term Ambiguity and Abstention tasks and why there are smaller or no gains on other tasks?
  3. Could the authors provide a more detailed cost analysis of memory maintenance and periodic consolidation?
8 comments

Dear Reviewer zxGD,

We sincerely appreciate the time and effort you have put into reviewing our work. Your concerns are valid and reflect a careful understanding of what we are trying to achieve, and we particularly appreciate your recognition of TeamKnowledgeBench's symbolic ledger and validation procedures.

Because the strengths you highlight concern mainly the benchmark, we think our broader motivation may not have come across as clearly as we intended. The benchmark is a means of evaluating a larger, largely overlooked problem in agentic memory: how to represent and maintain knowledge in collaborative, multi-user settings. Below we clarify this and answer each question with evidence from our existing results.

1. Technical novelty, and whether simpler structure would suffice (W1)

W1. Limited technical novelty beyond combining components.

The problem formulation is part of the contribution. Most agentic memory research targets single-user, long-horizon settings. Collaborative workspaces, where knowledge develops across team meetings, subgroup channels and individual user–assistant conversations, have received little attention, and existing benchmarks do not capture how knowledge is shared and established across these surfaces. We see team memory as complementary to single-user memory, and both are needed for complete memory systems.

Why we built TKC from the ground up. We first tried extending existing memory systems to team collaboration, but the extensions amounted to post-hoc fixes, because those systems were not designed for multi-user, multi-turn interaction. RAG works well for retrieving from documents. In agentic workspaces, however, knowledge evolves through conversations among users, teams and assistants, where authority, consensus and access boundaries matter. A team assistant must not confuse an individual opinion with a team decision, rely on a superseded conclusion, or cross an access boundary. This led us to a ground-up representation, in the way A-Mem and Mem0 approached memory for long-horizon single-user tasks.

What is new. We can see why TKC might look like a combination of familiar ideas: provenance, access control and knowledge types have each been studied before (Appendix A). Our contribution is a common representation of these ideas for team knowledge, grounded in work on team cognition, consensus and cooperative work, and a way to use it when memory is created and updated. In our ontology, authority tells us who can establish a claim, standing whether it is still a proposal or an accepted decision, scope where it applies, and entitlement who can see it. The difficult part is not adding structure; it is deciding which distinctions a team memory must preserve and how they interact as decisions evolve.

W1 (cont.) Results make it hard to tell whether the complexity of these combinations is necessary, or whether simpler structured memory approaches could perform comparably.

A single-user memory may store what people said without retaining who had authority, what the team accepted, or what later changed. If those distinctions are lost at ingestion, the answering model cannot recover them. Three results from our experiments speak to whether simpler structure would suffice:

  • Structure alone is not enough. In our structural ablation, a flat store of extracted claims adds +3.7 points over raw retrieval and entity edges add +1.4, while the typed, scoped, lifecycle-aware representation adds +12.1.
  • The gains are where the representation differs. TKC's Legal margin comes entirely from the families whose answers depend on lifecycle, authority and scope; it ties strong retrievers where finding the right turn suffices (table in §3 below).
  • Consistency. TKC varies by only 1.9 points across five answering models, compared with 12–16 points for retrieval baselines. The answer should not depend heavily on which model interprets the retrieved text.

A permission filter on top of retrieval is the most natural "simpler" alternative. On every question where all twelve turns retrieved by BM25-RAG were already visible to the asker, a permission-filtered BM25-RAG would receive exactly the same context, so its accuracy there is known from existing runs. Across both domains (90 questions), TKC answers 56.7% correctly against 43.3%: +13.3 points (95% CI +2.2 to +24.4; exact McNemar p = 0.04). Even when every retrieved turn is accessible, retrieval does not establish what the team accepted.

2. Benchmark selection and unfiltered evaluation (W1, Q1)

W1 (cont.) Possible selection bias: TKB design is closely aligned with TKC and filters out questions answerable by simple retrieval baselines.

We clarify that only BM25 and dense retrieval were used for filtering, following GroupMemBench's criterion. In fact most agentic memory benchmarks follow undergo filtering strategies to come up with automatic question formulation since the conversations are over 1M context. Hence to increase the complexity of QA pairs beyond retireval based questions, such filters are usually adopted. The other nine evaluated baselines did not influence question selection, yet TKC outperforms all of them.

The filtering was intended to focus TeamKnowledgeBench on questions that test knowledge curation rather than straightforward retrieval. As we explicitly stated in the paper, the goal is to never replace the RAG systems. They are simple and have good performance on documents sources.

Legal (n = 215), baselines not used for filteringAccuracyTKC − baseline (95% CI)p
HippoRAG-251.6+8.8 (+1.9, +16.3)0.027
GraphRAG49.8+10.7 (+2.8, +18.6)0.011
LangMem49.3+11.2–
Hindsight48.4+12.1–
RAPTOR47.4+13.0 (+5.6, +20.0)<0.001
MemoryBank47.0+13.5 (+5.6, +21.4)0.002
MemOS46.5+14.0 (+6.0, +22.3)0.001
Mem044.7+15.8 (+7.9, +23.7)<0.001
Cognee43.7+16.7 (+9.3, +24.2)<0.001

TKC accuracy is 60.5% (Figure 5). Paired bootstrap CIs over questions (10,000 resamples) and exact McNemar tests; bold marks intervals that exclude zero. Per-question judge scores were not stored for LangMem and Hindsight, so for them we report only the difference from Figure 5.

Q1. Since TKB is built around TKC and explicitly filters out questions solvable by retrieval, can TKC be evaluated on an unfiltered set to assess potential benchmark bias?

Our existing outputs provide direct evidence on easier questions, reported by filtering status in Section 4 of the general response. On the Legal questions outside the too-easy slots, TKC leads the strongest baseline by +12.5% . We also estimate accuracy on an unfiltered pool from these slices (general response, Section 4; an estimate, not a measured run). TKC is estimated to lead in every scenario, by +1.8 to +6.6 points over the strongest baseline and +6.4 to +12.5 over BM25-RAG.

Slice (Legal, existing outputs)nTKCBM25-RAGTKC − BM25-RAG (95% CI)
Legal, retrieval-solvable slots7174.669.0+5.6 (−5.6, +16.9)
Legal, all other slots14453.534.0+19.4 (+9.7, +29.2)

3. GroupMemBench results and component ablations (W1, Q2)

W1 (cont.) On GroupMemBench, improvements are concentrated in Term Ambiguity and Abstention; on other tasks such as Temporal Reasoning, TKC shows slight or no improvement.

Q2 (in part). [...] and why gains are smaller or absent on other tasks?

Your observation about concentrated gains is useful for understanding what TKC is designed to improve. GroupMemBench does not explicitly evaluate lifecycle or access permissions, so several of TKC's core capabilities are not directly tested.

Nevertheless, TKC leads overall across all four domains, with particularly strong improvements in Term Ambiguity (39.6 vs. 15.1 for BM25-RAG) and Abstention (90.6). These are precisely the settings where explicit knowledge interpretation and evidence-aware answering are valuable.

GroupMemBench (paired, same answerer and judge)TKCBM25-RAGTKC − BM25-RAG (95% CI)Dense-RAGTKC − Dense-RAG (95% CI)
Overall, user implicit excluded (n = 696, Table 8)51.446.1+5.3 (+1.3, +9.3)41.5+9.9 (+5.7, +14.1)
Overall, all categories (n = 745)50.346.0+4.3 (+0.4, +8.2)42.3+8.1 (+4.2, +12.1)
Term ambiguity (n = 106)39.615.1+24.5 (+15.1, +34.0)18.9+20.8 (+10.4, +31.1)
Term ambiguity, excluding blank baseline answers (n = 102 / 103)39.2 / 38.815.7+23.5 (+13.7, +33.3)19.4+19.4 (+8.7, +30.1)

The smaller improvements in Temporal Reasoning reflect a distinction in objectives: TKC maintains explicit knowledge states and supersession, rather than performing general temporal inference.

Q2 (cont.) Can the authors provide more detailed ablations showing which TKC components drive gains on Term Ambiguity and Abstention [...]?

We appreciate the request for finer-grained ablations of authority, lifecycle, scope, and entitlement. These fields are coupled by design: a proposal moves to settled only when the actor has the authority to settle it, and access is decided jointly by scope and entitlement. Removing a single field therefore also changes how the others behave, which is why our ablation varies the representation as a whole (flat store, entity edges, full item) and we report results by question family. Our per-family results (general response, Section 2) separate the components by the capability each family tests: on Legal, TKC leads on lifecycle (0.515 vs. 0.455), promotion (0.542 vs. 0.458), and abstention (0.676 vs. 0.529). Section 2 also gives an estimate of each field's contribution, not a measured ablation, in which lifecycle and authority account for the largest share on Legal (an estimated 3.3–5.6 points). Together with these, our structural ablation and answerer-robustness results provide evidence for the value of the unified representation.

Field removedTask families whose gold reads itEstimated loss in Legal accuracy (points)
Lifecycle and authorityA2 Lifecycle, B3 Abstention~4
Scope and promotionB1 Promotion~1.5
EntitlementB2 Isolation~1.2
Epistemic typeA1 Epistemic~1
Provenance and attributionC1 Capture, C2 Attribution~2.3
Reviewer_bG2pOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper addresses team-level memory for shared AI workspaces, where knowledge is spread across private chats, subgroup threads and meetings, and has different visibility for different people.

  • TKC (the method). It represents team knowledge as typed items carrying epistemic type, lifecycle state, relations, provenance and access scope. An LLM proposes items and a deterministic writer commits them. Retrieval is gated by permissions, and answers depend on who is asking, with three possible outcomes: an answer, access denied, or not yet established.
  • TeamKnowledgeBench (the benchmark). It is generated from a symbolic plan (a "ledger") fixed before the dialogue is written, so gold answers are computed by code. It covers legal (215 questions) and finance (99 questions).
  • Results. TKC reports about 60% accuracy against at most 56% for 11 memory and retrieval baselines, no leaks on a withholding audit, and gains on the external GroupMemBench.
Critical strengths
  1. A well-motivated formulation of team memory. Separating access denial from "not yet established", and making both first-class correct answers, captures failure modes that single-user memory benchmarks miss.
  2. A benchmark with deterministic gold answers. Gold is computed by code from a fixed ledger and depends on who is asking, which avoids relying on generative judgments for correctness.
Critical weaknesses

Critical Weaknesses

  1. The human audit is claimed but not reported. The main text states that the authors "perform a detailed automated and independent human audit of the generated benchmark" (p.7). But Appendix C.3 says "All of the evidence above comes from code or model judgments", and that the human study was only designed and "ship[ped] as a kit" (p.27). No human results are given: no accuracy, no agreement, no error counts. For a fully synthetic benchmark this validation is essential. The main-text claim must be corrected or the results reported.

  2. The comparison with baselines isn't shown to be fair.

    • Tuning isn't described. The paper describes the shared answering setup (same answerer, guide, budget and "top-12 retrieval by default", p.27), but not how the baselines were configured or tuned.
    • TKC gets method-specific configuration. It has its own constitution.md and domain skill.md (p.5).
    • TKC is adapted for transfer. For GroupMemBench its "extraction instructions, the method-specific QA prompt, and the embedder, chunking, and top-k settings" are changed (p.28). No equivalent effort is described for the baselines.

    Without this, the reported margins can't be attributed to the method itself.

Other comments

  • Naming: the method is called TKC in the text but TKG in Figure 2, Table 1 and the result figures. Please make it consistent.
  • Table 1: the caption should state which domain(s) the results cover.
Questions for authors
  1. (Clarification) Was any human annotation of TeamKnowledgeBench actually carried out? If so, please report human accuracy, inter-annotator agreement and adjudicated error counts. If not, please correct the claim on p.7.
  2. (Clarification) How were the baselines configured and tuned? Did any receive domain- or benchmark-specific adaptation comparable to TKC's constitution and skill files and its GroupMemBench port?
  3. (Clarification) Which domain does Table 1 report?
4 comments

Dear Reviewer bG2p,

First of all, we really appreciate your thoughtful consideration of our work. Your comments and requests for clarification reflect a close understanding of our efforts and contributions. We also appreciate you pointing out drafting inconsistencies, particularly in the table caption and the human audit discussion, which we had either overlooked or failed to explain clearly. Below we report the completed human study, give a full account of how TKC and the baselines were configured, and answer the presentation questions.

1. Human audit and benchmark validation (W1, Q1)

W1. Human audit is claimed but not reported. Main text (p.7) states the authors "perform a detailed automated and independent human audit of the generated benchmark." Appendix C.3 says "All of the evidence above comes from code or model judgments", and that the human study was only designed and "ship[ped] as a kit" (p.27).

Thank you for pointing out the inconsistency between the main text and Appendix C.3. The p.7 sentence overstated what had been done. At submission, our validation consisted of the automated checks in Appendix C and the authors' own inspection; the independent human study had been designed and released as a kit, but not yet run. We regret this. Because Reviewer RoUy raised the same point, we ran the independent study during the rebuttal period, and we will correct p.7.

W1 (cont.) No human results given: no accuracy, no agreement, no error counts. For a fully synthetic benchmark this validation is essential. The main-text claim must be corrected or the results reported.

Q1. Was any human annotation of TeamKnowledgeBench actually carried out? If so, report human accuracy, inter-annotator agreement and adjudicated error counts. If not, correct the claim on p.7.

We have now completed it. Three annotators with no role in building the benchmark or the method each answered all 48 items. The items are stratified over the eight task families in both domains (24 Legal, 24 Finance), and the Finance items are split between retrieval-filtered and unfiltered items. For each item, annotators saw the asker, the question, a plain-language label guide, and the conversation excerpt, with the audience of every message marked. They first answered without seeing the reference answer, then saw it and judged it. The analysis plan and four success criteria were fixed before data collection.

  • Human accuracy (blind to gold): 81.9% per annotator and 83.3% by majority vote; 92.9% by majority outside the exact-set C1 format.
  • Inter-annotator agreement: Fleiss' κ = 0.59 overall and 0.69 outside C1.
  • Error counts: a majority judged the gold correct or defensible on all 48 items. Only 2 of 144 individual verdicts were negative (one "incorrect", one "can't tell"). Counting both items as label errors gives 4.2% (2/48), with an exact 95% upper bound of 14.3%.
Human study (3 annotators, 48 items)Legal (24)Finance (24)Overall (48)Overall, excluding C1 (42)
Annotator accuracy, blind to gold (%)79.284.781.9 (73.6, 90.3)88.1 (80.2, 94.4)
Majority-vote accuracy (%)83.383.383.3 (70.4, 91.3)92.9 (81.0, 97.5)
Fleiss' κ0.460.640.59 (0.36, 0.71)0.69 (0.46, 0.82)
Gold judged correct by majority (%)91.795.893.8 (83.2, 97.9)100.0 (91.6, 100.0)
Gold judged correct or defensible by majority (%)100.0100.0100.0 (92.6, 100.0)100.0 (91.6, 100.0)
Items with any "incorrect" or "can't tell" verdict0222
Worst case: those items counted as label errors (exact 95% upper bound)0/24 (≤ 14.2%)2/24 (≤ 27.0%)2/48 = 4.2% (≤ 14.3%)2/42 (≤ 16.2%)
Dialogue naturalness (1–5)4.04.04.0 (all six ratings were 4)–

The code-derived gold is right. Without seeing it, the annotator majority reproduced it on 92.9% of items outside C1. After seeing it, they endorsed it on every item.

Because TeamKnowledgeBench's gold answers are computed by code from a ledger fixed before dialogue generation, the remaining question is whether independent readers recover the ledger's labels from the conversations, and these results indicate that they do. We will correct the p.7 claim and add the study, its protocol, the annotation interface and per-item results to the appendix.

2. Baseline fairness and configuration (W2, Q2)

Q2. How were the baselines configured and tuned? Did any receive domain- or benchmark-specific adaptation comparable to TKC's constitution and skill files and its GroupMemBench port?

We appreciate this concern, as fair comparison is essential to establishing the value of TKC.

W2. Fairness of baseline comparison not shown. Tuning isn't described. The shared answering setup is described (same answerer, guide, budget, "top-12 retrieval by default", p.27), but not how baselines were configured or tuned.

All evaluated systems share the same ingestion model, answering model, context budget, top-k retrieval setting, and answering guide. The key distinction is how each method represents and maintains knowledge. Each agentic memory has its own way of configuration based on the task they are solving. For example most of our baselines address single user long horizon tasks. We implemented the baselines based on the repo. It is not possible to tune them since we would then not be implementing the same baseline. We have added the details in the paper and made things more explicit.

W2 (cont.) TKC gets method-specific configuration. It has its own constitution.md and domain skill.md (p.5).

TKC's constitution.md and skill.md are not benchmark-specific answer instructions. They operationalise the proposed ontology by specifying how knowledge should be extracted, typed, scoped, and maintained. These are integral to the method being evaluated, rather than additional information supplied to improve answers on a particular benchmark.

W2 (cont.) TKC is adapted for transfer. For GroupMemBench its "extraction instructions, the method-specific QA prompt, and the embedder, chunking, and top-k settings" are changed (p.28). No equivalent effort is described for the baselines.

For GroupMemBench, we adapted TKC's extraction and QA instructions, along with retrieval configurations, to accommodate the external benchmark. We appreciate your point that these adaptations and the corresponding baseline configurations need to be documented more clearly. We will provide the method-specific settings.

W2 (cont.) Without this, the reported margins can't be attributed to the method itself.

We also emphasise that TKC's improvements are supported by more than the overall comparison. Our structural ablation shows that graph edges alone contribute +1.4 points, while the ontology-backed representation yields +12.1 points. TKC also maintains substantially more consistent performance across five answering models (1.9 versus 12–16 points of variation for retrieval baselines). These results provide complementary evidence that the representation itself contributes to the improvements.

3. Presentation corrections (Other comments, Q3)

Thank you for identifying these issues.

Other comments. Naming inconsistency: the method is called TKC in the text but TKG in Figure 2, Table 1 and the result figures.

We will consistently use TKC throughout the text, figures, and tables, replacing the remaining TKG references.

Other comments. Table 1: caption should state which domain(s) the results cover.

Q3. Which domain does Table 1 report?

We will also revise Table 1's caption to explicitly identify the evaluated domain(s).

Final Acknowledgements

We sincerely acknowledge your careful review and the opportunity to address your concerns. We are motivated by your positive assessment of our work and would sincerely appreciate your continued support during the discussion. If our clarifications have further strengthened your confidence in the paper, we would also be grateful if you considered raising your score.

Thank you for the detailed answers to my questions. The human studies adds a required layer of validation. I will maintain my rating.

We thank the reviewer for their acknowledgement and that human studies are providing the required validation for the curated dataset.

We hope that we addressed all of the reviewer's questions. We sincerely request the reviewer to consider increasing their score. Or if any further concerns persist, we request the reviewer to share them so that we could address them during this rebuttal time.

Comments on the paper

1 comment

Dear Reviewers and Area Chair,

We are grateful to all three reviewers for their thoughtful feedback. We acknowledge all the valid concerns raised by all reviewers and have addressed them individually in our responses. That said, we would also like to take this opportunity to clarify our motivation and contributions, as some of the reviewers’ comments on the paper’s strengths suggest that these aspects may not have come across as clearly as we intended.

1.The primary intent of the work is not benchmark design

Addresses zxGD W1 (novelty, and TKB being designed around TKC).

We believe the novelty of our work begins with the problem formulation itself. Despite the growing body of research on agentic memory in recent years, most existing work focuses on single-user settings and long-horizon tasks, with relatively little attention given to collaborative, shared and multi-user environments. We see these as complementary research directions, both of which are necessary to build more complete memory systems.

This gap also extends to evaluation. Since multi-user and team dynamics have received limited attention in the agentic memory literature, existing benchmarks do not adequately capture how knowledge is shared and developed across different communication surfaces, such as team meetings, subgroup Slack channels, and individual user–AI assistant conversations.

We designed Team Knowledge Bench to begin addressing this gap and to provide test cases for evaluating how our method behaves in such settings. We recognise that strong benchmark performance is only a proxy for real-world utility, rather than definitive evidence of it. Nevertheless, we believe it provides a useful starting point for studying and evaluating collaborative memory systems.

2. The problem goes beyond retrieval

Addresses RoUy Q3 (permission-filtered retrieval) and zxGD W1 (whether simpler approaches would suffice).

Our goal is to develop a memory system for team-level AI assistants, a setting largely overlooked by existing work. The team level assistant is supposed to have context managed from team knowledge based on their shared resources and team level conversation, without confusing between individual opinions with team decisions, rely on outdated conclusions, or cross access boundaries.

While prior work has explored access and information-level controls in multi-user settings, these approaches typically rely on fixed access rules rather than supporting the flexible promotion of individual knowledge into shared team knowledge while preserving individual privacy, which is the gap we aim to address.

This is why we see the problem primarily as one of knowledge representation rather than retrieval. Retrieval alone cannot reliably recover distinctions that were never represented in the stored knowledge.

3. Clarifying Our Core Contribution: A Grounded Ontology for Team Knowledge

Addresses zxGD W1 (novelty beyond combining components), RoUy W4 (isolating components) and bG2p W2 (attributing the margins to the method).

We understand why TKC may appear to combine existing mechanisms. While epistemic types, provenance, and access control are not individually new (Appendix A), our contribution is a unified ontology grounded in team cognition, consensus, decision-making, and cooperative work.

We agree that structuring raw information may not sound inherently novel. Our contribution is less about showing why structure matters and more about establishing how team knowledge should be structured. The non-trivial challenge lies in identifying and integrating the necessary concepts into a coherent representation that captures authority, consensus, and the evolution of individual contributions into shared knowledge. Existing single-user memory systems are not designed to capture these dynamics and may lose important distinctions during memory ingestion itself.

Beyond accuracy, we believe consistency across answering models is equally important. Retrieval-based approaches leave the interpretation of team dynamics largely to the downstream model, making performance highly model-dependent. On the other hand, single-user memory systems face a more fundamental limitation: information lost during ingestion cannot be reliably recovered by the answerer. By explicitly preserving these distinctions, TKC varies by only 1.9 points across five answering models, compared with 12–16 points for retrieval baselines.

Together, these findings suggest that the value of our ontology lies not simply in adding structure, but in representing the right distinctions at the right stage, improving both accuracy and consistency while reducing dependence on downstream model reasoning.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.