- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper studies the reliability of LLM-based audits of long-term agent memory. In particular, it investigates whether memory entries contain unsupported claims about users that are incorrectly presented as user-provided information.
The authors compare several LLM-judging protocols, including a three-way origin classifier, a decomposed judgment procedure, and criteria-based coding. They find substantial disagreement between these procedures, with estimated misattribution rates and rankings of memory-writing strategies changing depending on the instrument.
They also study the effect of evaluating entries individually versus in batches, finding that administration mode substantially affects inter-judge agreement. The paper concludes that memory-misattribution prevalence cannot be reliably estimated using the evaluated instruments without human calibration.
Critical strengths
-
Controlled analysis of LLM-judges: I think the experiment comparing per-entry and batched judging is interesting. On the same entries, the authors find a substantial change in inter-judge agreement, with a similar point estimate under a second judge pair. This is a useful reminder that evaluation methodology can affect conclusions even when the underlying data are unchanged.
-
Attention to measurement validity: I appreciate that the authors investigate disagreements between judging protocols instead of reporting one instrument's estimated misattribution rate as ground truth. The paper also provides detailed documentation of its experimental limitations and attempts to distinguish agreement from accuracy.
Critical weaknesses
-
No independently validated labels, despite measurement validity being the main contribution. My main concern is that every label, including the criteria-based reference labels, is generated by an LLM. The paper reports a large discrepancy between the decomposed instrument and criteria-based coding, but there is no independent basis for deciding whether either procedure is correct.
Furthermore, the naive and decomposed instruments operationalize somewhat different constructs. The naive method may classify correctly attributed assistant content as leakage, whereas the decomposed method explicitly excludes it. Different rates and rankings are therefore not necessarily evidence of instability in measuring the same underlying phenomenon.
I am also not entirely convinced by the claim that judges systematically resolve ambiguous entries toward user-stated facts. This interpretation depends on treating the criteria-based coder's classifications as the reference, even though the coder is itself an LLM. The inability to verify that the reconstructed user-turn evidence exactly matches what the judges originally saw adds further uncertainty.
Since the paper is fundamentally about measurement validity, I think a human-annotated subset is particularly important for establishing its conclusions.
-
Limited novelty and generalizability of the main empirical contribution. The administration-mode experiment is useful, but prior work such as BatchEval (Yuan et al., ACL 2024) has already examined batch-wise versus sample-wise LLM evaluation. The claim that this design choice has not been studied seems too broad.
The main administration-mode result also establishes increased agreement under batching, not necessarily more accurate judgments. More generally, the study is limited to synthetic sessions derived from LongMemEval, a small number of writing models, and judges from a single model family. The cross-sample comparisons also change multiple factors simultaneously, limiting what can be attributed to the judging instrument. Finally, the paper does not evaluate whether the misattributed memory entries lead to downstream failures.
Questions for authors
-
On the validity of the labels: Given that the paper's main contribution concerns measurement validity, could the authors independently human-annotate a subset of the memory entries and evaluate how each instrument agrees with these labels? In particular, how confident are the authors that the criteria-based coder's unsupported-specific judgments are correct?
-
On the admin-mode result: Why does batching increase inter-judge agreement? Does it improve consistency with an independently defined reference, or does it simply make the judges more similar in their positive rates and decisions?
-
On novelty: How does the administration-mode analysis differ from prior work such as BatchEval, which already investigates batch-wise LLM evaluation? Is the novelty primarily the application to memory provenance and its effect on inter-judge agreement?
-
On generality: Would the findings hold with judges from different model families or with naturally occurring agent-memory entries? In particular, how much of the disagreement arises from the chosen construct and coding conventions versus the underlying behavior of LLM judges?
3 comments
W2c / Q4. Generality. Other model families. We added a judge from another family, Llama 4 Maverick, on the same 156 entries and the same text. The findings hold, and more strongly. It flags 72% of the entries it labelled, against 45.5% in the gold labels, and agrees with them at , against 0.38–0.50 for the Claude judges and 0.57 for criteria-based coding. Judged one entry at a time, it treats close paraphrases as unsupported: of the 39 entries the gold labels mark as stated, it marks 31 as containing an unsupported specific. In a batch it becomes far stricter, flagging 22% of entries, so its two sets of labels agree at only .
Construct versus judge behaviour. We separate the sources by changing one factor at a time on identical entries and measuring how far the labels move ( between the two sets of labels). Rows use different samples, so compare them roughly:
| Source | What changes, all else fixed | Entries | |
|---|---|---|---|
| Construct definition | three-way vs. decomposed instrument, same judge | 0.47–0.49 | 888 |
| Coding conventions | same definition, with vs. without the written rulebook, same judges and text | 0.11–0.31 | 112–137 |
| Judge, same family | Sonnet 5 vs. Opus 5 or 5.5 | 0.44–0.49 | 156 |
| Judge, other family | Llama 4 Maverick vs. the Claude judges | 0.16–0.20 | 156 |
| Administration | one at a time vs. batched, same judge | 0.04–0.74 | 156 and 887 |
In the coding-conventions row both sets of labels flag few entries (4–16%), which depresses .
The construct definition matters, but least: switching between the two definitions moves the labels about as much as swapping judges within one family ( 0.44–0.49). The rest of the instrument matters more. On the construct side, adding the written coding conventions to the prompt changes the labels substantially ( 0.11–0.31). On the judge side, so do a judge from another family ( 0.16–0.20) and the administration mode ( as low as 0.04 for Llama 4 Maverick). So the disagreement does not come mainly from which construct we chose; it comes from how the construct is written into a prompt, and from which model applies it and how it is called.
Naturally occurring entries. Partly. The entries are written freely by LLM writers under four write objectives that mirror common memory designs: near-verbatim logs, extracted key facts, narrative summaries and task-state notes. The effects hold across all four. At scale, batching raises inter-judge for every write objective, from for key facts to for task-state notes. On the human-labelled entries, Llama 4 Maverick's agreement with itself across modes stays near zero for every write objective ( to , 31–43 entries each). What we have not tested is naturally occurring sessions: every session is built from LongMemEval with a scripted user, and we state this as a limitation.
W2d. Cross-sample comparisons change several factors at once. We agree. For example, the same three-way prompt gives inter-judge on the pilot and on the scale sample, but the two samples also differ in session length, administration mode and instance pool, so the gap cannot be attributed to any one of them. The claims we keep are within-sample: administration mode on identical entries, and all instruments against human labels on identical text. Comparisons across samples are labelled descriptive (§2).
W2e. Downstream failures. We agree; this needs its own experiment, which we plan as follows:
- Take the entries the human labels mark as misattributed.
- Make three versions of each: as written; the same content marked as the assistant's proposal; and removed.
- In a follow-up session, the agent retrieves memory and does a task that depends on the entry's specific.
- Measure task success against what the user actually said, and whether the agent treats the specific as the user's or asks first.
- Over further sessions, check whether later memory writes turn the specific into a user fact.
Because each entry is compared with its own variants, the result does not depend on the overall rate. It also tests the instruments: the one whose labels best predict these failures is the one to trust.
Presentation. We revised the paper for plain wording and clearer definitions, and added the same-text comparison with human labels, the second model family and the corrections above.
We thank the reviewer again; the review led directly to finding the evidence mismatch. We hope the revision addresses the main concerns, and we are happy to answer further questions.
We thank the reviewer for a detailed and fair review. Two of its concerns, the lack of independent labels and the reconstructed evidence, are well founded and we address here blow, we label the parts of each weakness in the order they appear (W1a–W1d, W2a–W2e), this comment answers W1 and Q1, and part 2/2 answers W2–Q4.
W1a / Q1. Independent human labels. Yes. Two coders labelled all 156 pilot entries, blind to every LLM label, and every instrument was re-run on the user turns they saw (study and full table in the General Response). Against the adjudicated gold labels (71 of 156 misattributed), criteria-based coding agrees at (precision 0.80, recall 0.72), with both Claude judges at 0.50 , and a judge from another family, Llama 4 Maverick, at 0.19 . Adjudication kept coder 1's label on all 68 disagreements; on the 88 entries both coders labelled alike, the criteria-based coder's is 0.56. Inter-coder agreement is reported in §4.5.
How confident are we in the criteria-based coder's unsupported-specific judgments? Confident in its positive calls, less so in its negative ones. When it marks an unsupported user-specific, the gold labels agree 92% of the time, but it finds only half of those the gold labels count (recall 0.51). The shortfall is mostly one convention: it rarely separates content the entry marks as the assistant's (gold label B), and labels 38 of the 46 B entries as having no unsupported specific. That leaves its final label unchanged (precision 0.80, recall 0.72); its real misses are 19 of the 71 misattributed entries. On this first step alone, the decomposed Sonnet 5 judge is closer to the gold labels ( vs. 0.27).
W1d. The evidence. We thank the reviewer for raising this; checking it gave one of the clearest results in the revision. The coders' rebuilt turns differ from the original sessions in 8 of 9 cases, and with prompt and entries fixed, the evidence alone moves a judge from 5.8% to 60% flagged. On identical text, the decomposed judges and criteria-based coding agree at (0.011 in the submission), so we no longer present the 2.6% vs. 41.0% gap as an instrument effect. The revised checklist asks studies to log the evidence each judge saw. The human labels compare instruments on fixed text and do not estimate prevalence.
W1b. Different constructs. We agree. The three-way scheme counts content the entry correctly attributes to the assistant as leakage; the decomposed scheme does not. The revision says so, adds a table of what each instrument asks, and treats the rate differences as a construct choice rather than instability: a choice common practice makes silently, and one that changes the answer, since at scale the two instruments rank the write objectives differently.
W1c. Direction of the disagreement. We agree and withdraw the claim: on the same text, the re-run Claude judges flag more entries one at a time than the criteria-based coder (49–62% vs. 41%), not fewer.
W2a / Q3. Novelty relative to BatchEval. We thank the reviewer for the pointer, and agree that our sentence "we have not seen it reported" was too broad. BatchEval (Yuan et al., 2023) already shows that scoring samples in batches changes LLM judgments; the revision cites it and narrows our claim. We use the same manipulation BatchEval studies; what differs is the task, what we measure, and the lesson.
So yes: our contribution here is narrower than the submission implied. It is the effect of administration mode on the reliability evidence of a reference-free audit, and the reporting item that follows, not the observation that batching changes LLM judgments. BatchEval shows that batching can improve a quality evaluator; on our data, in a reference-free audit, it raised agreement between judges but not agreement with human labels, so a higher κ under batching should not be read as better measurement.
W2b / Q2. Why batching raises agreement, and whether it improves accuracy. Both of the reviewer's readings hold in part.
Flags: share of entries marked misattributed (gold labels: 45.5%). Recall: share of the gold-misattributed entries a judge flags. Precision: share of its flags that the gold labels confirm. Llama labelled 151 of the 156 entries one at a time.
The change in κ is +0.02 [−0.24,0.27] for Sonnet 5 and −0.08 [−0.22,0.07] for Llama; with only 9 sessions these intervals are wide, but neither points toward a gain. Batching trades recall for precision: the judges become more cautious, not more discerning. Most entries a judge flags in a batch it had also flagged one at a time (54 of 63 for Sonnet 5, 27 of 35 for Llama), so batching mostly changes how readily a judge says yes, not which entries it picks. For Llama the shift is large because, judged alone, it treats close paraphrases as unsupported (see Q4).