acceptodds
Under review as a conference paper at ICLR 2027

Not Yet a Measurement: Instrument Dependence in LLM-Judge Audits of Agent Memory

Abstract

Lifelong-memory agents write durable records about their users that later sessions read as background fact, without access to the transcript that produced them. Prior work has studied the read side; the write objective, and the provenance of what it admits, has not been a controlled variable. We set out to measure how often such entries assert specifics the user never stated, and report instead that this is not yet a measurement one can trust. On a controlled testbed over LongMemEval we vary the write objective, the elicitation arm and the agent model, then audit the same entries with two LLM judges under two defensible instruments, two administration modes, and criteria-based coding with rules committed before any entry was labelled. The answer depends on the instrument more than on the data. On identical entries, estimated prevalence moves from 2.6% to 41.0% at chance agreement (κ = 0.011). The two instruments rank the write objectives differently, yield different multiplicity-surviving contrasts, and reverse the sign of the elicitation effect, while agreeing at κ ≈ 0.48 within the same judge. The disagreement is directional rather than noise: given the same rules and worked examples in-prompt, judges resolve borderline entries toward "the user said this," the direction that makes a pipeline look cleaner than criteria-based coding says it is. Two things did hold still, and both are contrasts rather than rates. Administration mode — one entry per call versus a shuffled batch, with the evidence and the question unchanged — moves inter-judge κ by +0.209 [0.139, 0.281]; measured again with a different judge pair the difference is +0.213, while the absolute agreement levels move by 0.10 between pairs. We have not seen this design choice reported. The second is the model doing the writing. These results are negative about a measurement, not about the phenomenon. No human validated any label, so our κ values index instrument stability rather than accuracy and we report no prevalence; a human-coded calibration set is the prerequisite we identify. We close with a reporting checklist, each item earned by one of our results.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
agent memory, LLM-as-a-judge, instrument dependence, measurement validity, inter-rater reliability, lifelong agents, memory provenance, agent safety, LongMemEval

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_1ca8OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper studies the reliability of LLM-based audits of long-term agent memory. In particular, it investigates whether memory entries contain unsupported claims about users that are incorrectly presented as user-provided information.

The authors compare several LLM-judging protocols, including a three-way origin classifier, a decomposed judgment procedure, and criteria-based coding. They find substantial disagreement between these procedures, with estimated misattribution rates and rankings of memory-writing strategies changing depending on the instrument.

They also study the effect of evaluating entries individually versus in batches, finding that administration mode substantially affects inter-judge agreement. The paper concludes that memory-misattribution prevalence cannot be reliably estimated using the evaluated instruments without human calibration.

Critical strengths
  1. Controlled analysis of LLM-judges: I think the experiment comparing per-entry and batched judging is interesting. On the same entries, the authors find a substantial change in inter-judge agreement, with a similar point estimate under a second judge pair. This is a useful reminder that evaluation methodology can affect conclusions even when the underlying data are unchanged.

  2. Attention to measurement validity: I appreciate that the authors investigate disagreements between judging protocols instead of reporting one instrument's estimated misattribution rate as ground truth. The paper also provides detailed documentation of its experimental limitations and attempts to distinguish agreement from accuracy.

Critical weaknesses
  1. No independently validated labels, despite measurement validity being the main contribution. My main concern is that every label, including the criteria-based reference labels, is generated by an LLM. The paper reports a large discrepancy between the decomposed instrument and criteria-based coding, but there is no independent basis for deciding whether either procedure is correct.

    Furthermore, the naive and decomposed instruments operationalize somewhat different constructs. The naive method may classify correctly attributed assistant content as leakage, whereas the decomposed method explicitly excludes it. Different rates and rankings are therefore not necessarily evidence of instability in measuring the same underlying phenomenon.

    I am also not entirely convinced by the claim that judges systematically resolve ambiguous entries toward user-stated facts. This interpretation depends on treating the criteria-based coder's classifications as the reference, even though the coder is itself an LLM. The inability to verify that the reconstructed user-turn evidence exactly matches what the judges originally saw adds further uncertainty.

    Since the paper is fundamentally about measurement validity, I think a human-annotated subset is particularly important for establishing its conclusions.

  2. Limited novelty and generalizability of the main empirical contribution. The administration-mode experiment is useful, but prior work such as BatchEval (Yuan et al., ACL 2024) has already examined batch-wise versus sample-wise LLM evaluation. The claim that this design choice has not been studied seems too broad.

    The main administration-mode result also establishes increased agreement under batching, not necessarily more accurate judgments. More generally, the study is limited to synthetic sessions derived from LongMemEval, a small number of writing models, and judges from a single model family. The cross-sample comparisons also change multiple factors simultaneously, limiting what can be attributed to the judging instrument. Finally, the paper does not evaluate whether the misattributed memory entries lead to downstream failures.

Questions for authors
  1. On the validity of the labels: Given that the paper's main contribution concerns measurement validity, could the authors independently human-annotate a subset of the memory entries and evaluate how each instrument agrees with these labels? In particular, how confident are the authors that the criteria-based coder's unsupported-specific judgments are correct?

  2. On the admin-mode result: Why does batching increase inter-judge agreement? Does it improve consistency with an independently defined reference, or does it simply make the judges more similar in their positive rates and decisions?

  3. On novelty: How does the administration-mode analysis differ from prior work such as BatchEval, which already investigates batch-wise LLM evaluation? Is the novelty primarily the application to memory provenance and its effect on inter-judge agreement?

  4. On generality: Would the findings hold with judges from different model families or with naturally occurring agent-memory entries? In particular, how much of the disagreement arises from the chosen construct and coding conventions versus the underlying behavior of LLM judges?

3 comments

W2a / Q3. Novelty relative to BatchEval. We thank the reviewer for the pointer, and agree that our sentence "we have not seen it reported" was too broad. BatchEval (Yuan et al., 2023) already shows that scoring samples in batches changes LLM judgments; the revision cites it and narrows our claim. We use the same manipulation BatchEval studies; what differs is the task, what we measure, and the lesson.

  • Method vs. nuisance. BatchEval proposes batching as a better evaluator of graded text quality: in-batch samples serve as references for one another, and batches are recomposed over several rounds to strengthen that comparison. Our audit labels each memory entry against its own evidence, so a label should not depend on what else is in the call; we shuffle each batch and remove write-objective names to suppress cross-item comparison.
  • Accuracy vs. reliability. BatchEval compares each evaluator's scores with human ratings. We instead measure how consistent the judges are. Between judges, on 887 identical entries, batching raises κ\kappa from 0.32 to 0.53 (+0.21+0.21, 95% CI [0.14,0.28][0.14, 0.28]); in the widely used Landis–Koch bands, that moves the audit's reported reliability from "fair" to "moderate". Within a judge, the two modes give different labels for the same entry: Sonnet 5 agrees with itself at κ=0.39\kappa = 0.39 and Opus 5.5 at 0.740.74, and Llama 4 Maverick, on the 156 human-labelled entries, flags 72% of entries one at a time but 22% in a batch, so its two sets of labels agree at only κ=0.04\kappa = 0.04. BatchEval reports neither kind of agreement: it does not compare two LLM evaluators with each other, or one evaluator's sample-wise and batch-wise scores for the same item.
  • Consequence. For BatchEval, batching is an improvement to adopt. For us, it is a reporting requirement: a κ\kappa quoted without its administration mode cannot be compared across studies.

So yes: our contribution here is narrower than the submission implied. It is the effect of administration mode on the reliability evidence of a reference-free audit, and the reporting item that follows, not the observation that batching changes LLM judgments. BatchEval shows that batching can improve a quality evaluator; on our data, in a reference-free audit, it raised agreement between judges but not agreement with human labels, so a higher κ\kappa under batching should not be read as better measurement.

W2b / Q2. Why batching raises agreement, and whether it improves accuracy. Both of the reviewer's readings hold in part.

  • More alike: yes. In the administration sub-study (887 entries, three-way labels, Sonnet 5 and Opus 5.5), the two judges label very different shares of entries as assistant-originated when they see one entry at a time (34% vs. 21%), but nearly the same share in a batch (24% vs. 27%). They also give the same label to more entries: 64% one at a time, 78% in a batch.
  • More accurate: no evidence. On the 156 entries with human labels, batching makes each judge flag far fewer entries, but agreement with the gold labels does not improve:
Judge and modeFlagsκ\kappa with goldRecallPrecision
Sonnet 5, one at a time60%0.380.820.62
Sonnet 5, batched40%0.400.620.70
Llama 4 Maverick, one at a time72%0.190.830.53
Llama 4 Maverick, batched22%0.110.280.57

Flags: share of entries marked misattributed (gold labels: 45.5%). Recall: share of the gold-misattributed entries a judge flags. Precision: share of its flags that the gold labels confirm. Llama labelled 151 of the 156 entries one at a time.

The change in κ\kappa is +0.02+0.02 [−0.24,0.27][-0.24, 0.27] for Sonnet 5 and −0.08-0.08 [−0.22,0.07][-0.22, 0.07] for Llama; with only 9 sessions these intervals are wide, but neither points toward a gain. Batching trades recall for precision: the judges become more cautious, not more discerning. Most entries a judge flags in a batch it had also flagged one at a time (54 of 63 for Sonnet 5, 27 of 35 for Llama), so batching mostly changes how readily a judge says yes, not which entries it picks. For Llama the shift is large because, judged alone, it treats close paraphrases as unsupported (see Q4).

W2c / Q4. Generality. Other model families. We added a judge from another family, Llama 4 Maverick, on the same 156 entries and the same text. The findings hold, and more strongly. It flags 72% of the entries it labelled, against 45.5% in the gold labels, and agrees with them at κ=0.19\kappa = 0.19 [0.09,0.30][0.09, 0.30], against 0.38–0.50 for the Claude judges and 0.57 for criteria-based coding. Judged one entry at a time, it treats close paraphrases as unsupported: of the 39 entries the gold labels mark as stated, it marks 31 as containing an unsupported specific. In a batch it becomes far stricter, flagging 22% of entries, so its two sets of labels agree at only κ=0.04\kappa = 0.04 [−0.03,0.12][-0.03, 0.12].

Construct versus judge behaviour. We separate the sources by changing one factor at a time on identical entries and measuring how far the labels move (κ\kappa between the two sets of labels). Rows use different samples, so compare them roughly:

SourceWhat changes, all else fixedκ\kappaEntries
Construct definitionthree-way vs. decomposed instrument, same judge0.47–0.49888
Coding conventionssame definition, with vs. without the written rulebook, same judges and text0.11–0.31112–137
Judge, same familySonnet 5 vs. Opus 5 or 5.50.44–0.49156
Judge, other familyLlama 4 Maverick vs. the Claude judges0.16–0.20156
Administrationone at a time vs. batched, same judge0.04–0.74156 and 887

In the coding-conventions row both sets of labels flag few entries (4–16%), which depresses κ\kappa.

The construct definition matters, but least: switching between the two definitions moves the labels about as much as swapping judges within one family (κ\kappa 0.44–0.49). The rest of the instrument matters more. On the construct side, adding the written coding conventions to the prompt changes the labels substantially (κ\kappa 0.11–0.31). On the judge side, so do a judge from another family (κ\kappa 0.16–0.20) and the administration mode (κ\kappa as low as 0.04 for Llama 4 Maverick). So the disagreement does not come mainly from which construct we chose; it comes from how the construct is written into a prompt, and from which model applies it and how it is called.

Naturally occurring entries. Partly. The entries are written freely by LLM writers under four write objectives that mirror common memory designs: near-verbatim logs, extracted key facts, narrative summaries and task-state notes. The effects hold across all four. At scale, batching raises inter-judge κ\kappa for every write objective, from +0.04+0.04 for key facts to +0.42+0.42 for task-state notes. On the human-labelled entries, Llama 4 Maverick's agreement with itself across modes stays near zero for every write objective (κ\kappa −0.11-0.11 to 0.190.19, 31–43 entries each). What we have not tested is naturally occurring sessions: every session is built from LongMemEval with a scripted user, and we state this as a limitation.

W2d. Cross-sample comparisons change several factors at once. We agree. For example, the same three-way prompt gives inter-judge κ=0.066\kappa = 0.066 on the pilot and 0.5910.591 on the scale sample, but the two samples also differ in session length, administration mode and instance pool, so the gap cannot be attributed to any one of them. The claims we keep are within-sample: administration mode on identical entries, and all instruments against human labels on identical text. Comparisons across samples are labelled descriptive (§2).

W2e. Downstream failures. We agree; this needs its own experiment, which we plan as follows:

  1. Take the entries the human labels mark as misattributed.
  2. Make three versions of each: as written; the same content marked as the assistant's proposal; and removed.
  3. In a follow-up session, the agent retrieves memory and does a task that depends on the entry's specific.
  4. Measure task success against what the user actually said, and whether the agent treats the specific as the user's or asks first.
  5. Over further sessions, check whether later memory writes turn the specific into a user fact.

Because each entry is compared with its own variants, the result does not depend on the overall rate. It also tests the instruments: the one whose labels best predict these failures is the one to trust.

Presentation. We revised the paper for plain wording and clearer definitions, and added the same-text comparison with human labels, the second model family and the corrections above.

We thank the reviewer again; the review led directly to finding the evidence mismatch. We hope the revision addresses the main concerns, and we are happy to answer further questions.

We thank the reviewer for a detailed and fair review. Two of its concerns, the lack of independent labels and the reconstructed evidence, are well founded and we address here blow, we label the parts of each weakness in the order they appear (W1a–W1d, W2a–W2e), this comment answers W1 and Q1, and part 2/2 answers W2–Q4.

W1a / Q1. Independent human labels. Yes. Two coders labelled all 156 pilot entries, blind to every LLM label, and every instrument was re-run on the user turns they saw (study and full table in the General Response). Against the adjudicated gold labels (71 of 156 misattributed), criteria-based coding agrees at κ=0.57\kappa = 0.57 [0.43,0.71][0.43, 0.71] (precision 0.80, recall 0.72), Idec\mathrm{I}_{\mathrm{dec}} with both Claude judges at 0.50 [0.38,0.61][0.38, 0.61], and a judge from another family, Llama 4 Maverick, at 0.19 [0.09,0.30][0.09, 0.30]. Adjudication kept coder 1's label on all 68 disagreements; on the 88 entries both coders labelled alike, the criteria-based coder's κ\kappa is 0.56. Inter-coder agreement is reported in §4.5.

How confident are we in the criteria-based coder's unsupported-specific judgments? Confident in its positive calls, less so in its negative ones. When it marks an unsupported user-specific, the gold labels agree 92% of the time, but it finds only half of those the gold labels count (recall 0.51). The shortfall is mostly one convention: it rarely separates content the entry marks as the assistant's (gold label B), and labels 38 of the 46 B entries as having no unsupported specific. That leaves its final label unchanged (precision 0.80, recall 0.72); its real misses are 19 of the 71 misattributed entries. On this first step alone, the decomposed Sonnet 5 judge is closer to the gold labels (κ=0.56\kappa = 0.56 vs. 0.27).

W1d. The evidence. We thank the reviewer for raising this; checking it gave one of the clearest results in the revision. The coders' rebuilt turns differ from the original sessions in 8 of 9 cases, and with prompt and entries fixed, the evidence alone moves a judge from 5.8% to 60% flagged. On identical text, the decomposed judges and criteria-based coding agree at κ=0.61\kappa = 0.61 (0.011 in the submission), so we no longer present the 2.6% vs. 41.0% gap as an instrument effect. The revised checklist asks studies to log the evidence each judge saw. The human labels compare instruments on fixed text and do not estimate prevalence.

W1b. Different constructs. We agree. The three-way scheme counts content the entry correctly attributes to the assistant as leakage; the decomposed scheme does not. The revision says so, adds a table of what each instrument asks, and treats the rate differences as a construct choice rather than instability: a choice common practice makes silently, and one that changes the answer, since at scale the two instruments rank the write objectives differently.

W1c. Direction of the disagreement. We agree and withdraw the claim: on the same text, the re-run Claude judges flag more entries one at a time than the criteria-based coder (49–62% vs. 41%), not fewer.

Reviewer_z2bvOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper sets out to measure how often lifelong-memory agents write down “misattributed” entries. These are claims about the user, such as a preference, deadline or constraint, that the user never stated and that the entry does not mark as the agent’s own suggestion. Sessions are built from LongMemEval records under two elicitation styles (all at once vs. spread over several turns) and two agent models, and memory is written under four write objectives. The entries are then audited with three LLM-based instruments: a three-way origin label, a two-question decomposed judgment, and a rule-based coding pass by a separate LLM. The central finding is negative: estimated prevalence depends heavily on the instrument, from 2.6% to 41.0% on the same entries. Instrument choice also changes the ranking of write objectives and the sign of the elicitation effect. Two contrasts are stable across instruments. Judging entries one at a time rather than in a batch raises inter-judge agreement by about 0.21 κ, and one agent model produces more flagged entries than the other. The authors close with a reporting checklist for LLM-based memory audits.

Critical strengths
  1. A well-defined construct and a useful practical contribution. Misattribution is clearly separated from staleness, conflict and injection. Treating administration mode (single vs. batched) as part of the instrument is a simple, actionable point that practitioners likely overlook. The checklist ties each item to a result in the paper.
  2. An important, under-explored problem with a clear definition. Most work on agent memory looks at retrieval: whether the right entry is found and whether it’s still true. This paper looks at the write side, asking whether what gets stored actually came from the user. That’s a real risk in deployed systems, since a later session will treat an invented “user preference” as fact. The definition of misattribution is precise and usefully separated from staleness, conflict and injection, and the distinction between unsupported content and content the entry openly attributes to the assistant is a thoughtful one.
Critical weaknesses
  1. No human ground truth. Every label, including the 41% “reference”, comes from an LLM. The reference coder applies rules it wrote itself and shares a model family with one judge. The claim that judges lean toward “the user said this” is therefore only a claim relative to another LLM, and is equally consistent with the coder over flagging. With sessions of only 1–6 user turns, double human coding of the 156 entry subset seems feasible.
  2. The headline disagreement may be partly an artifact. The user turns the judges saw were not saved, and the coder’s evidence was reconstructed afterwards. So it is unverified that the coder and judges labelled against the same text. In addition, κ near zero partly follows mechanically from very different flag rates (about 3% vs. 41%) despite 59% raw agreement. More broadly, all models come from one provider, the samples differ in session length and administration mode at once, and the scale run reached 76 of 150 planned instances. The replication of the administration effect also rests on a 59-entry probe whose interval includes zero.
Questions for authors
  1. Could you have humans code the 156-entry subset, ideally two coders using your released criteria, and report which instrument their labels agree with? This would show whether the 41% or the 2.6% is closer to the truth.
  2. Would the instruments still disagree if the judge were set up more carefully, with reasoning enabled, several samples combined by majority vote, and a rubric refined on a small human-labelled set? If they converge, the problem may be the setup rather than LLM-judge audits in general.
  3. Can you re-run the judges on the reconstructed user turns the coder used, so both are shown to see identical text? How much of the 2.6% vs. 41% gap remains?
  4. Batching makes judges agree more, but does it make them more correct, or just more alike? And does the effect hold for a judge from a different model family?
1 comment

We thank the reviewer for a careful and constructive review, and especially for the concrete tests it proposes. We ran all of them, and together they give the paper a stronger evidential basis. With human labels and every instrument on identical text, we can now say which instruments track human judgment (W1/Q1). We also show that two setup choices commonly used to make LLM judges more reliable, batching and careful reasoning, raise agreement between LLM instruments without raising agreement with humans (Q2, Q4). The Q3 re-run also led us to revise one comparison in the submission, as described below.

W1 / Q1. Human labels on the 156 entries. Two coders labelled all 156 entries blind to every LLM label, and every instrument below was re-run on the user turns they saw. Adjudication kept coder 1's label on all 68 disagreements, so the last column uses only the 88 entries both coders labelled alike. Against the gold labels (71 of 156 misattributed, 45.5%):

InstrumentFlaggedκ\kappa vs. gold [95% CI]PrecisionRecallRecall (agreed entries)
Icrit\mathrm{I}_{\mathrm{crit}} (criteria-based coding)41%0.57 [0.43, 0.71]0.800.720.83
Idec\mathrm{I}_{\mathrm{dec}}, both Claude judges49%0.50 [0.38, 0.61]0.710.760.78
Idec\mathrm{I}_{\mathrm{dec}}, Llama 4 Maverick72%0.19 [0.09, 0.30]0.530.830.83

*CIs resample the 9 sessions. *

Which number is closer to the truth? On identical text, the decomposed judges flag 49% and criteria-based coding 41%, against 45.5% in the gold labels. The coders worked from the paper's definition rather than the criteria-based coder's rulebook, so that their agreement with Icrit\mathrm{I}_{\mathrm{crit}} is not built in. Because these turns are not the original sessions, the labels compare instruments on fixed text and do not support a prevalence estimate. Inter-coder agreement is reported in §4.5.

Q3 and W2. Re-running the judges on the coder's text. We thank the reviewer for this test; it produced one of the clearest results in the revision. Little of the gap remains: on the user turns the criteria-based coder used, the decomposed judges flag 49% of the 156 entries and criteria-based coding 41%, and the two agree at κ=0.61\kappa = 0.61. What moves the result is the evidence the judge sees. The same judge, with the same prompt and the same entries, flags 5.8% on the original session turns and 60% on the rebuilt ones, which, as the reviewer suspected, differ from the original sessions in saved facts. The revision adds "log the exact evidence each judge saw" to the reporting checklist. §4.5 now reports the same-text comparison. (Judge 2 in the re-run is Claude Opus 5.5)

W2, other points.

  • Near-zero κ\kappa. We agree. With flag rates of 2.6% and 41.0%, the largest attainable κ\kappa was 0.07, so κ=0.011\kappa = 0.011 mostly restated the gap, and the gap itself came from the evidence (above).
  • One provider. We added a judge from another family, Llama 4 Maverick, on the same entries (Q4). It agrees with the human labels at κ=0.19\kappa = 0.19, against 0.38–0.57 for the Claude instruments.
  • Samples that differ in session length and administration mode; 76 of 150 instances. We agree. The claims we keep are within-sample: administration mode on identical entries, and all instruments against human labels on identical text. Comparisons across samples are labelled descriptive (§2).
  • Replication of the administration effect. We agree that "replicated" overstated it. The main estimate (+0.209+0.209, 95% CI [0.139,0.281][0.139, 0.281], n=887n = 887) comes from one judge pair; a second pair gives a similar point estimate (+0.213+0.213) on 59 entries, with an interval that includes zero. The revision now says “with a similar point estimate under a second judge pair”.
Reviewer_Aq1vOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper addresses the problem of agent memory, specifically trying to detect whether memory contains assertions that were not made by the user. In trying to evaluate this, the paper creates a dataset derived from an available dataset with varying conditions for how the user interactions are structured (across multiple interactions vs. a single turn, with multiple write objectives). The measurement instruments used for the evaluation are also varied into what is called per entry vs. batched administration mode and different rubrics specifying attributes of interest. The paper finds that judge models in these settings do not agree across the different conditions and hence are not reliable evaluators of whether the user said what was present in the memory.

Critical strengths

This work addresses an important aspect of Agent memory which has not been studied in depth - whether the memory consists of assertions that were not made by the user. Evaluating this with high accuracy would lead to better systems, especially as systems start personalizing to users more and more by relying on memory. The dataset construction protocol is also interesting and sound.

Critical weaknesses

Although the authors repeatedly call out the lack of human labels in this work, I think it would significantly strengthen the work to show which of the instruments/modes are actually capturing some notion of correctness. Otherwise, the paper's main argument is equivalent to "LLM-judges do not agree and are not reliable" which is well known.

The paper reads very much as AI-written (the authors disclose AI use) which makes it difficult to read. Simpler language with more clear descriptions of experimental conditions and the dataset construction process would make the paper stronger.

Questions for authors

N/A

1 comment

We thank the reviewer for the careful reading, and for recognising both the importance of write-side memory provenance and the soundness of the dataset construction. We think both weaknesses are fair. We have addressed them with a new human-labelled study and a substantial revision of the text.

W1. Which instruments actually capture correctness?

We agree this was the main gap in the submission. Two coders (one author, one non-author) independently labelled all 156 pilot entries, blind to every LLM label, using the paper's definition:

A = no unstated user-specific;

B = an unstated specific that the entry marks as the assistant's;

C = an unstated specific presented as the user's (misattributed).

Disagreements were adjudicated; because adjudication kept coder 1's label on all 68 of them, we also checked the 88 entries both coders labelled alike. We re-ran every judge on the text the coders saw. Against the gold labels (71 of 156 misattributed, 45.5%):

Instrument (same 156 entries, same text)Flagged, one at a time / batchedκ\kappa vs. gold, one at a timeκ\kappa vs. gold, batched
Icrit\mathrm{I}_{\mathrm{crit}} (criteria-based coding)41%0.57 [0.43, 0.71]—
Idec\mathrm{I}_{\mathrm{dec}}, both Claude judges must flag49% / 37%0.50 [0.38, 0.61]0.47 [0.27, 0.66]
Idec\mathrm{I}_{\mathrm{dec}}, Opus 5.562% / 54%0.49 [0.31, 0.65]0.52 [0.33, 0.69]
Idec\mathrm{I}_{\mathrm{dec}}, Sonnet 560% / 40%0.38 [0.30, 0.47]0.40 [0.17, 0.61]
Idec\mathrm{I}_{\mathrm{dec}}, Llama 4 Maverick72% / 22%0.19 [0.09, 0.30]0.11 [−0.04, 0.24]

CIs resample the 9 sessions.

On identical text, the Claude instruments are moderately close to human judgment (κ\kappa 0.38–0.57), criteria-based coding the closest; a judge from another family is not (κ=0.19\kappa = 0.19). No administration mode or reasoning setting we tried brings an instrument closer to the human labels: batching changes κ\kappa with gold by −0.08 to +0.03, and a careful setup (high-effort reasoning, majority of three samples) changes Sonnet 5's by +0.02.

This is where we think the contribution goes beyond "LLM judges disagree". The same choices do make the instruments agree more with each other: batching raises inter-judge κ\kappa by +0.21 [0.14, 0.28] on 887 identical entries, and careful reasoning raises Sonnet 5's agreement with criteria-based coding from 0.41 to 0.62. Agreement between LLM instruments, the reliability evidence an audit usually reports, can therefore rise with no gain in agreement with humans. The revision says this explicitly (§4.3, §4.5). Because these turns are not the original sessions, the human labels compare instruments on fixed text and do not support a prevalence estimate; a human-labelled subset of the scale sample, where sessions and evidence both come from saved instances, is the next step.

W2. Readability.

We thank the reviewer for this comment and agree that the submission was harder to read than it should have been. We revised the abstract, the introduction and §2–3 sentence by sentence, aiming for plain wording, every term defined where it first appears. In the revised PDF:

• Dataset construction (§2). a step-by-step account of how each LongMemEval record becomes a multi-turn task and how memory is written, under new paragraph headings.

• Experimental conditions (§2–3). each condition and each instrument is defined where it is introduced.

• Abstract and introduction. shorthand and slogan-style sentences removed; The account of refuted hypotheses, which readers previously met before the setup, is now a one-line pointer to §4.4.

If any passage still reads unclearly, we would be glad to revise it further. We hope the human-labelled results and the clearer presentation address the reviewer's concerns. If so, we would be grateful if the reviewer would consider updating the score. We are happy to answer any further questions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.