acceptodds
Under review as a conference paper at ICLR 2027

What Social Memory Forgets: Measuring What Decision Accuracy Cannot Detect

Abstract

Large language models increasingly operate as communities of interacting agents that divide responsibilities, delegate tasks, and establish leadership roles. As these communities exchange findings and build on one another's work, understanding how they form and maintain collective memory becomes essential to understanding their collective behavior. We introduce ORALMEM, a framework for separating memory fidelity, the decision relevance of memory errors, and agents' use of remembered evidence. ORALMEM tracks source relationships through compression and evaluates whether an observed distortion changes the optimal decision, making it possible to distinguish errors that matter for a task from those that its outcome cannot expose. In our controlled competing-incident task, correct decisions coexist with distorted source relationships. The three observed contradictions leave the optimal allocation unchanged, so correct actions do not establish that coordinators detected or compensated for the loss. A prospective allocation cohort reveals a complementary failure. Most memories preserve source relationships under outcome-blind human annotation, yet one coordinator makes correct allocations in only 34 of 60 cases with untouched memory. Explicit source-attribution notes raise this to 55 of 60, including benefits when the relation was already preserved in memory. A matched reminder can also change decisions without adding source evidence. Together, these findings expose a blind spot in outcome-only evaluation. It can reward successful coordination while leaving the integrity of collective memory and its use untested. ORALMEM provides a structured way to determine what decision success actually tells us about what a team remembers.

Primary area:
foundation or frontier models, including LLMs
Keywords:
large language models, multi-agent systems, collective memory, source attribution, memory compression, evaluation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_susYOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2(+1)
Soundness
2
Read review
Paper summary

This paper introduces ORALMEM, a framework that separately evaluates the fidelity of handover memories, whether memory errors change the optimal decision, and coordinators' actual decisions. Through several sets of author-designed resource allocation scenarios, the paper finds that correct decisions can conceal distortions in source relationships, while preserving those relationships does not guarantee correct coordinator decisions.

Critical strengths

Separating memory fidelity, the decision consequences of memory errors, and actual coordinator behavior is a useful evaluation principle. It helps avoid treating final decision accuracy as sufficient evidence that intermediate information has been faithfully preserved and appropriately used.

Critical weaknesses
  1. The presentation substantially impedes understanding and assessment of the core experimental setting. Terms such as "handover," "shared memory," "bounded memories," and "capsules" are used interchangeably or in close succession without clearly establishing their relationships. S1/D3/I3 are not adequately defined when first introduced, and transitions between the illustrative task and the primary task involve changes in parameters and budgets without sufficient guidance. Source lineage, Bayesian updating, and decision loss are introduced before readers have a concrete understanding of the complete task. Consequently, readers must repeatedly backtrack and reconstruct the experimental workflow themselves. The main text requires substantial restructuring: necessary background and consistent terminology should be established, and a complete worked example should precede the formal definitions.

  2. The incremental contribution and significance are insufficiently established. Separately evaluating memory fidelity, decision consequences, and actual actions is reasonable, but the paper does not clearly establish what new capability this framework provides beyond existing approaches to source tracking, summarization faithfulness, and decision analysis. The evidence primarily comes from artificially constructed reports, tightly constrained summaries, and fixed decision rules, leaving its applicability to broader multi-agent settings unclear. The competing-incident experiment identifies only three explicit source-relation contradictions, none of which changes the optimal action. These cases demonstrate that decision accuracy can miss memory errors, but do not establish the frequency or broader scope of this phenomenon. Overall, the current framework and empirical evidence do not sufficiently establish a substantial contribution to multi-agent memory research.

Questions for authors

What motivates the choice of a 40-word handover budget? What evidence do the existing experiments provide that the main findings are not highly dependent on this restrictive length constraint?

4 comments
AuthorsOpenReview ↗

Response to susY

Thank you. We reorganized the exposition in the order you asked for and added evidence for the consequence step.

W1. Presentation ($\S$2)

The section now opens with terms, moves to a full worked example, and only then gives the definitions.

  • Terms. A report is a single firsthand or relayed message. A handover is the compressor's short replacement for the reports, and it serves as the shared memory. The appendix protocols call the same text a capsule.
  • Example. Two crews, four sites. Three sites offer 60, 40 and 15 points of expected prevented harm. The target site prevents 100 if its hazard is real. We state the prior and warning probabilities before the table.
  • Table. S1 is one inspection with one report, D3 is one inspection with three reports, and I3 is three separate inspections with three reports. S1 and D3 imply a 20.59% risk and favor the 60- and 40-point sites. I3 implies 58.53% and favors the 60-point site and the target.
  • Consequence. Suppose an I3 handover presents two inspectors as relays. Following it costs 58.53−40=18.5358.53 - 40 = 18.53 points, and that loss is Δ\Delta. If the alternative were worth 18 or 65 instead, the same error would leave the optimal action unchanged.
  • Transition. A short bridge marks Figure 1 as the separate competing-incident task, which has its own priors, costs and budget. The equations that follow refer back to the worked primary example.

W2. Added evidence and scope ($\S$4.3)

The new test computes Δ\Delta before any action is taken, on naturally generated errors. On the same seven model-consensus errors, each of two coordinators gets 0/7 correct at Δ>0\Delta>0 and 14/14 at Δ=0\Delta=0, and in the Δ=0\Delta=0 cells neither reports a correct count. So the test shows both when an outcome score hides an error and when a different use of the same text exposes it. A separate repair experiment gives 26/27 correct at Δ>0\Delta>0 versus 0/27 for a matched sham, across three coordinators (Q.2).

The seven errors come from 80 handovers across ten new event families (seven of the 40 I3 cases). Two blind human annotators flag 18 and 13 errors. All 13 of the second annotator's errors are also among the first annotator's 18, and on those 13 texts both report the same number of independent observations. They confirm 7/7 and 5/7 of the primary cases. Eight more jointly identified errors show the same Δ>0\Delta>0 pattern in a separate frozen decision run. These rates are descriptive: they come from a synthetic pipeline with a continuation gate and incomplete model coding, and real-world prevalence remains open. The external CPB-Live material adds a different decision task, and its positive-Δ\Delta errors are controlled injections.

$\S$6 now compares our contribution with LongMemEval's stage analysis and FactGraph's factuality evaluation. Generic faithfulness judges usually detect our relation errors (Q.4). What we add is the task-specific decision consequence.

Q. Why 40 words?

Forty words is a controlled compression stress test. The checks we have show both that it is feasible and where it breaks:

  • Two source-faithful reference formats fit within 40 words and support 60/60 correct decisions.
  • In the six-event 40/80-word branch check, all three pairs that comply at both caps keep the same count error and missed dispatch (O.4). This is a small result tied to one configuration.
  • On external packets, preservation of the target proposition and lineage is 21/24 at 40 words and 24/24 at 80. All actions are correct, because the queries supply the proposition context that the three short records lack (P.1).
  • In the new prospective test, 49/80 outputs go over the cap, including four of the seven model-coded errors. Three errors comply with 40 words. The test applies a requested budget, and compliance is partial.

Together these show that 40 words can be enough, and that a larger cap still leaves some observed errors in place. The definitions apply at any length. We hope the restructured task description and the new decision-consequence evidence support a reconsideration of the presentation and informativeness assessments.

Dear Reviewer,

As the response deadline approaches, please let us know if there are any points we can clarify. If our response has addressed your concerns, we would appreciate hearing whether you would consider raising your score.

Thank you again for your time and feedback.

Best regards, The Authors

Reviewer_susYOpenReview ↗

Response to the Authors' Comments

While I still have concerns regarding the novelty, significance, and empirical generalizability of the work, I will maintain my overall rating. Nevertheless, I appreciate the authors' efforts to improve the manuscript. In recognition of the improved exposition and clearer experimental descriptions, I have increased my Presentation score from 1 to 2.

Thank you for revisiting the paper and recognizing the improved presentation. We would like to briefly clarify our contribution in light of your remaining concerns about novelty, significance, and generalizability.

  • Novelty. LongMemEval examines stages of memory use, and FactGraph evaluates summary faithfulness. ORALMEM connects a specific error about who observed what to a downstream decision. Before the coordinator acts, we can determine whether relying on the altered record would change the optimal action.
  • Significance. On the same seven naturally corrupted handovers, each coordinator made 0/7 correct decisions when the error changed the optimal action and 14/14 when it left the optimal action unchanged, as predicted (§4.3). This distinction helps prioritize corrections when several decisions rely on the same handover.
  • Generalizability. The seven errors arose in four of ten new event families, and both coordinators followed the predicted pattern. We also observed this pattern across three coordinators in a different decision task using externally authored CPB-Live material with controlled errors.

We would be glad to clarify any of these points. Thank you again.

Reviewer_xVj8OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces ORALMEM, an evaluation framework that separates source-relation fidelity, the decision consequence of a determinate memory error, and the coordinator’s observed decision. It uses controlled allocation tasks with known observation lineage and stipulated Bayesian utilities. Compression reduces sensitivity to independent corroboration in the primary experiment. In a competing-incident cohort, three explicit attribution errors change reported risks but leave the optimal allocation unchanged, illustrating why correct actions cannot certify memory fidelity. Attribution-note experiments further show that explicit source information can improve decisions, including on memories annotated as faithful, while source-free reminders can themselves help or harm. Validation includes repeated generations, multiple coordinators, a prospective cohort, human annotations, and several supplementary controls.

Critical strengths
  1. A useful evaluation decomposition. ORALMEM distinguishes an inaccurate record from a consequential error and from a poor downstream decision (§2). The three corrupted memories with unchanged optimal actions provide concrete examples of why action accuracy alone is insufficient. This is consequential because outcome-only evaluation can overlook defects in evidence that may matter under different future decisions.
  2. Careful treatment of controls and unsuccessful experiments. The paper compares attribution notes against both untouched memory and matched reminders, revealing that reminder harm can inflate apparent gains (§4.4–4.5). It also preserves null results, missingness, and ineligible restoration experiments rather than presenting them as successful repair. This transparency strengthens the credibility of the reported findings.
Critical weaknesses
  1. Limited empirical coverage of the framework’s distinctive consequence audit. The determinate-error analysis rests on only three memories, all with Δ=0\Delta=0; no observed determinate corruption has Δ>0\Delta>0 (Appendix B.6, Table 10). The hypothetical perturbation enumeration does not fill this empirical gap. These results establish that correct actions can coexist with corrupted provenance, but provide limited evidence that the complete fidelity–consequence–behavior audit yields broadly useful diagnoses beyond this existence demonstration. Shared synthetic templates and stipulated independence further restrict its demonstrated significance.
  2. The attribution gains do not isolate a memory-specific failure or remedy. Notes supply evaluator-known source multiplicity and alter explicitness and salience. Benefits also occur with faithful-labeled memories, and separate reports do not eliminate the prospective coordinator’s errors (§4.5). The conflicting-note and wording controls are helpful, but leave parsing, attention, Bayesian reasoning, and cue-following unresolved. Although the authors acknowledge this, much of the intervention evidence therefore supports sensitivity to input presentation rather than a specifically identified collective-memory mechanism.
Questions for authors
  1. Clarification: What diagnoses or intervention choices would ORALMEM change beyond a combined fidelity-and-action audit on the existing data? Please distinguish the demonstrated contribution of Δ\Delta from the hypothetical sensitivity enumeration.
  2. Clarification: In the prospective faithful-memory cases where attribution changes an incorrect allocation to a correct one, do existing outputs or controls distinguish failure to extract source multiplicity from failure to apply the Bayesian or allocation rule? This distinction materially affects the interpretation of “use of remembered evidence.”
  3. Clarification: Are the exact prompts, generated memories, annotation rationales, frozen analysis plans/amendments, and scoring code available to reviewers? These are important for assessing the provenance-sensitive labels and the distinction between prespecified and adaptive analyses.
3 comments
AuthorsOpenReview ↗

Response to xVj8 (1/2)

Thank you for emphasizing the distinction between a useful decomposition and demonstrated empirical capability. We have revised the paper to clarify three practical applications of the consequence audit, supported by new experiments.

Q1 / W1. What Δ\Delta adds beyond fidelity and action correctness

(a) Predicting which errors affect decisions.

In ten new event families, we fixed text coding and Δ\Delta values before collecting coordinator decisions (§4.3, Q.1). On the same seven model-consensus errors, both Sol and Opus achieve 0/7 correct decisions at Δ>0\Delta>0 and 14/14 at Δ=0\Delta=0. Neither coordinator recovers the true source count in the latter condition (0/14 each).

These results demonstrate the prospective role of Δ\Delta: it identifies which task utilities make an existing memory error consequential before the coordinator acts. Fidelity auditing identifies the altered relation, while observed action accuracy measures the resulting behavior. Δ\Delta adds a task-conditioned consequence estimate that distinguishes different uses of the same defective record.

Two blinded human annotators independently confirm 7/7 and 5/7 primary errors. They also jointly identify eight additional errors. A separately frozen evaluation on these cases yields 0/8 correct decisions at Δ>0\Delta>0 for each coordinator and 31/32 at Δ=0\Delta=0 across both. We report the model-coded, human-checked, and human-selected populations separately.

(b) Evaluating the consequences of corrections.

On nine archived naturally occurring errors, untouched memories yield 0/9 correct decisions at Δ>0\Delta>0 for each coordinator. We then tested a correction explicitly stating both the inspection and its positive finding, alongside a matched sham. Across Sol, Opus, and Sonnet, the corrections achieve 26/27 correct decisions, compared with 0/27 for the sham (Q.2).

At Δ=0\Delta=0, the specified evidence error carries zero normative decision loss, although introducing a correction can still affect model behavior. One Sonnet correction produces a new error, and additional repair failures are documented in App. O. These observations help distinguish the consequence of an identified memory error from the effectiveness and possible side effects of a particular correction.

(c) Prioritizing corrections under limited resources.

We also examined how Δ\Delta can guide repair allocation. For each of nine handovers, we considered three downstream uses with only one corrected copy available. The retrospective calculation gives the following equal-family mean total regret (App. P.2):

  • 18.53 without repair;
  • 12.80 when assigning the correction uniformly among fidelity-tied uses;
  • 1.24 with Δ\Delta-guided assignment.

Both allocation strategies have access to the identified contradiction. The Δ\Delta-guided strategy additionally accounts for the utility associated with each downstream use, allowing the available correction to be directed toward the most consequential case.

This analysis assumes known source links and an available correction, and reuses previously measured responses. Its findings describe retrospective allocation under these conditions; evaluating an online repair policy remains future work.

The new prospective cohort contains naturally arising positive-Δ\Delta errors across four event families, while the archived branch and clean-relay sets cover five families. External CPB-Live packets extend the evaluation to another decision task using controlled errors in both directions (App. P, Q.2). Naturally occurring external error rates and deployment-scale generalization remain open questions.

Together, these experiments provide empirical evidence for the prospective, corrective, and allocation uses of the consequence audit. We hope they help clarify the practical contribution of Δ\Delta and address the concerns about empirical added value.

AuthorsOpenReview ↗

Response to xVj8 (2/2)

Q2. Extracting multiplicity versus applying the rule

To examine where decision errors arise, we extended the earlier 12-case selected probe to all 57 I3 memories classified as faithful by the archived pre-decision model screen (§4.6, Q.3). Each memory was evaluated with one fresh call per condition, independently of its archived decision outcome. The overlapping prompts are identical to those used in the earlier probe, byte-for-byte.

RequestAll 5723 archived misses34 archived successes
Action only29524
Count, risk, action481632
Same readout + supplied count552332

Among the nine failures in the count-risk-action condition, seven report an incorrect observation count and select the action optimal for that reported count. The remaining two correctly report a count of three but choose the wrong action. Both errors persist when the correct count is supplied.

Providing the count resolves all 23 archived misses, while overall accuracy reaches 55/57. This provides direct evidence of a count-extraction bottleneck in the new diagnostic calls and shows that coordinators can generally apply the decision rule when given the correct count.

The increase from 29/57 to 48/57 when requesting an explicit readout also demonstrates sensitivity to output format. Since the archived action-only calls lack intermediate count reports, their underlying failure mechanisms remain unresolved. We have made this distinction explicit in the revised main text.

W2. Is attribution a memory-specific remedy?

We agree that improvements on intact memories leave the mechanism of attribution open. Parsing, attention, explicitness, and cue-following may all contribute to the observed gains.

We therefore treat the attribution note as a diagnostic probe. The new repair experiments address a separate question by targeting independently identified memory errors and measuring the effects of supplied corrections. They include both successful repairs and cases where corrections introduce additional errors, providing a more direct assessment of repair behavior.

Q3. Inspectable artifacts

Original 120-memory human annotation (§4.6; App. K). A human annotator reviewed all 120 memories after the coordinator decision runs. The annotator received only the source messages and corresponding memory, presented under opaque identifiers, with decision outcomes withheld.

Each memory was classified as faithful, unclear, or contradictory, with source and memory observation counts recorded whenever determinate. The annotation yielded 115 faithful and 5 unclear memories, with zero determinate count errors.

This original annotation is reported separately from the new 80-handover evaluation in App. Q.1, which uses two independent human annotators.

Reproducibility package. The updated Supplementary Material attached to this submission includes:

  • Exact prompts and response schemas;
  • Generated handovers, model responses, call manifests, and retained failed attempts;
  • Recorded protocols, frozen schedules, hashes, and amendments;
  • Model-based coding outputs and rationales;
  • Both human annotators' labels for the new 80-handover evaluation, together with blinded annotation materials, annotator metadata, and comparison code;
  • Scripts for reproducing the offline analyses.

The paired comparisons in App. N.2 and the diagnostic experiments in App. Q.3 use the archived pre-decision model screen. We preserve this screen separately from the new human annotations, and the package README identifies the corresponding files and analysis populations.

PROTOCOLS.md summarizes the experimental designs. Detailed protocol records, prompts, and execution logs are indexed in data/records.jsonl, together with their file paths and hashes.

Running python reproduce.py restores the expected directory structure and reproduces 17 offline analyses, checking the resulting numerical outputs against the stored results.

The documented amendments cover two operational cases: replacing an unavailable generator before any successful outputs were produced, and allowing one additional attempt after a provider session-limit rejection that returned no output. Other rejected calls remain recorded as missing.

These materials allow the experimental populations, coding decisions, protocol changes, and reported results to be inspected and reproduced. We hope the prospective evidence, expanded diagnostics, and explicit treatment of their limitations address the concerns underlying the soundness and significance assessments.

Dear Reviewer,

As the response deadline approaches, please let us know if there are any points we can clarify. If our response has addressed your concerns, we would appreciate hearing whether you would consider raising your score.

Thank you again for your time and feedback.

Best regards, The Authors

Reviewer_XKvDOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

ORALMEM separates three aspects of multi-agent handover evaluation: source-relation fidelity, whether a distortion changes the optimal action (Δ), and actual coordinator performance. In authored allocation tasks, compression reduces correct independent-evidence responses from 88/90 to 64/90. Three explicit provenance distortions leave actions correct because Δ=0. In a new cohort, 115/120 memories are labeled faithful, yet Opus achieves only 34/60 correct independent-evidence allocations; attribution notes increase this to 55/60. Matched reminders can also affect performance. The contribution is a diagnostic framework showing why memory fidelity, evidence use, and intervention benefits require separate evaluation.

Critical strengths

The framework provides an operational diagnostic distinction. Separating source annotation, utility sensitivity Δ, and observed actions makes clear why correct allocations cannot certify memory integrity. The three distortions change risk estimates without changing optimal actions, and ambiguous memories are appropriately left without invented counterfactual states.

The controls and reporting strengthen credibility. Comparing attribution against both untouched memory and matched reminders exposes reminder-induced harm. The authors retain unsuccessful repairs, null transfer results, and annotation disagreements. Benefits within faithful-labeled memories further support separating evidence preservation from evidence use.

Critical weaknesses

The incremental contribution over prior work is insufficiently established and narrowly validated. The paper omits LongMemEval’s staged memory analysis (ICLR 2025) and FactGraph’s structured semantic factuality evaluation (NAACL 2022), leaving its positioning beyond outcome-only evaluation incomplete. Its more distinctive component is utility sensitivity Δ, yet the principal audit cohorts contain only three determinate distortions, all with Δ=0. The evidence therefore does not sufficiently establish the framework’s added value over existing fidelity and stage-level evaluations.

Intervention benefits remain difficult to interpret and generalize. Attribution notes may add information, increase salience, or shift action preferences, while reminders are not neutral controls. Prospective Opus accuracy is 48/60 with original reports versus 55/60 with supplemented memory. Single completions, reused templates, and simultaneous changes to events and compressor further limit a generalizable evidence-use interpretation.

Questions for authors

Please compare ORALMEM with LongMemEval (ICLR 2025) and FactGraph (NAACL 2022).

Please confirm whether the principal audit cohorts lack natural, determinate Δ>0 distortions, and clarify whether comparable source-relation annotations are available for Appendix I’s branch experiments to broaden the supporting evidence.

Please provide paired intervention effects for faithful I3 cases by source template, distinguish explicit independence statements from collective wording.

3 comments
AuthorsOpenReview ↗

Response to XKvD (1/2)

Thank you for highlighting the empirical gap around Δ\Delta and requesting a comparable branch-level audit. We have addressed this with a retrospective audit under a common rubric and a prospective evaluation on newly generated events.

Q1. LongMemEval and FactGraph ($\S$6)

LongMemEval evaluates long-term conversational memory, focusing on indexing, retrieval, and reading. FactGraph assesses summary factuality by comparing meaning graphs extracted from source documents and summaries. We now discuss both in $\S$6.

ORALMEM extends this line of work to downstream decision consequences. Given an identified source-relation error, Δ\Delta measures the loss incurred when a coordinator acts optimally on the altered record.

Our comparison with FactGraph is conceptual. To provide additional empirical context, we evaluated three prompt-based generic faithfulness methods on 30 minimal pairs (App. Q.4). All three detect most relation errors, while also flagging issues in 32–48% of relation-faithful texts. Some of these flags may be valid under their broader factuality criteria. Pairwise discrimination ranges from 0.74 to 0.88, compared with 1.00 for our targeted relation rubric, whose labels follow the same relation criterion. These results clarify the scope of relation-specific auditing. The corresponding Δ\Delta further quantifies how an altered relation affects the coordinator's optimal decision.

W1 / Q2. Natural determinate Δ>0\Delta>0 errors

We agree that the principal audit cohorts in the original submission contained no naturally occurring, determinate errors with Δ>0\Delta>0. We have since conducted two complementary evaluations.

Retrospective audit (App. N.1). We applied a common explicit-lineage rubric to all 36 archived 40-word I3 branch handovers. Six incorrectly collapse three observations into one, each yielding Δ=18.53\Delta=18.53. In all six cases, the previously recorded coordinator decisions follow the optimum implied by the distorted record. Of the remaining handovers, two are faithful and 28 are unclear. This audit relies on model-based text coding; the supplement provides coding rationales and intermediate relays for inspection.

Prospective evaluation ($\S$4.3 / App. Q.1). Before generation, we fixed ten new event families, a pilot gate, coding rules, and predictions. Across 80 handovers, we identified seven model-consensus errors spanning four families. All seven incorrectly report two observations where three occurred. We finalized the labels and Δ\Delta values before running the coordinator decisions.

The results on these seven handovers are consistent across both coordinators:

  • Δ>0\Delta>0: Sol and Opus each achieve 0/7 correct decisions. Every choice is optimal for the distorted state.
  • Δ=0\Delta=0: Both achieve 14/14 correct decisions, although neither recovers the true observation count (0/14 each).
  • Condition-matched interaction: The prespecified estimate is 0.96 for Sol (event-bootstrap CI [0.88, 1]) and 0.86 for Opus ([0.73, 1]).

The detected error rate is 7/80 (descriptive Wilson interval: 4–17%). This estimate applies to the tested synthetic pipeline under its recorded continuation gate. Five failed coder calls remain uncoded, and 21 handovers were classified as unclear.

Independent human validation. Two annotators outside the study independently reviewed the shuffled handovers. They received only the target issue and text, without access to sources, model labels, Δ\Delta values, or coordinator decisions.

They agreed on 75/80 count classifications (κ=0.90\kappa=0.90). The first annotator confirmed all seven model-coded errors, while the second confirmed five. Across the full set, they identified 18 and 13 contradictions, respectively. We retain the two disputed primary cases in the reported analysis.

Both annotators also identified eight additional errors that the model-based audit had classified as unclear. We evaluated these cases in a separate run with labels and Δ\Delta values frozen before the decisions:

  • Δ>0\Delta>0: Both coordinators achieve 0/8 correct decisions.
  • Δ=0\Delta=0: Sol achieves 16/16 and Opus 15/16, with neither recovering the true count.

Combining these eight cases with the five primary errors confirmed by both annotators yields 13 jointly confirmed errors, with 0/13 correct decisions at Δ>0\Delta>0 for each coordinator.

AuthorsOpenReview ↗

Response to XKvD (2/2)

W2. Intervention interpretation and generalization

We agree that the effects of added information, salience, and wording deserve separate examination. We have expanded the coordinator comparisons and introduced an additional sham control.

Coordinator replication and external evaluation (App. P, Q.2). We replayed the same nine archived errors with Opus and Sonnet, extending the original Sol results. At Δ>0\Delta>0, both untouched and sham texts yield 0/9 correct decisions for every coordinator. At Δ=0\Delta=0, the untouched Claude calls achieve 34/36. We also evaluated external CPB-Live packets with controlled errors in both directions. Each coordinator achieves 0/12 at Δ>0\Delta>0 and 12/12 at Δ=0\Delta=0.

Correction controls (App. Q.2). The original corrections yield 8/9, 4/9, and 5/9 correct decisions at Δ>0\Delta>0 for Sol, Opus, and Sonnet, respectively. Several corrections mentioned every inspection while omitting its positive finding. We therefore designed a follow-up that explicitly states both, alongside a sham containing the same finding but retaining a single root. The design was frozen before the new calls.

The revised corrections achieve 26/27 correct decisions at Δ>0\Delta>0, compared with 0/27 for the matched sham. Across utility settings, correction accuracy reaches 79/81. Sonnet still fails two prespecified criteria, including one repair-induced error at Δ=0\Delta=0. Both failures are included in our analysis.

Decision diagnostics ($\S$4.6 / App. Q.3). On the full cohort, accuracy is 29/57 with action-only prompts, 48/57 with count-risk-action prompts, and 55/57 when the count is supplied. Of the nine failures in the count-risk-action condition, seven report an incorrect count and choose an action consistent with it. The remaining two report the correct count of three but select the wrong action.

These results suggest that count extraction is a major source of error in the new diagnostic calls. Accuracy also varies with the requested output format. The intermediate reasoning behind the archived action-only decisions remains unavailable.

Q3. Paired effects by template and wording (App. N.2)

We used the archived pre-decision model screen to examine how attribution performs across source templates. Among I3 texts screened as faithful, Opus attribution shows the following paired wins and losses relative to untouched memory:

Source templateFaithful I3 textsWins / losses
T1: Third-person inspection185 / 0
T2: Quoted first-person account157 / 0
T3: Warning with evidential basis247 / 0

Improvements appear across all three templates. The wording-stratified analysis shows 18 wins and no losses among the 54 texts that already state independence explicitly, and one win with no losses among the three collectively worded texts. Thus, the observed gains extend to cases where independence is already explicit in the source.

Across all 57 texts, Opus attribution yields 38 wins and no losses against the reminder condition, and seven wins against two losses when compared with separate reports. Sonnet shows eight wins and no losses against untouched memory across 56 complete pairs; one untouched call is missing. These comparisons are paired and descriptive.

The expanded evaluation now includes additional events, utility settings, and coordinators. Repeated use of some texts and reliance on single completions continue to limit generalization. Together, the prospective Δ\Delta results, controlled correction experiments, and paired template analyses provide further evidence of ORALMEM's empirical contribution. We hope these additions address the reviewer's concerns and support reconsideration of the assessment.

Dear Reviewer,

As the response deadline approaches, please let us know if there are any points we can clarify. If our response has addressed your concerns, we would appreciate hearing whether you would consider raising your score.

Thank you again for your time and feedback.

Best regards, The Authors

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of revisions: prospective consequence tests and clearer exposition

Thank you to the reviewers. In this revision we respond to three concerns: whether Δ\Delta has real empirical value, how to read the attribution gains, and how the paper is presented. The rest of the submitted text is unchanged, and every new or edited sentence is in red. Our strongest new evidence is a test where we fixed the source-count labels and Δ\Delta before any coordinator made a decision ($\S$4.3, App. Q.1).

1. Natural errors, with decisions predicted in advance. We collected 80 handovers from ten new event families. Two model coders agreed that 47 texts were faithful and that seven contained a determinate 3→23 \to 2 error, spread over four families. Another 21 were unclear, and five have no label because one coder's call was rejected. On those seven errors, GPT-5.6 Sol and Claude Opus 5 each chose the correct action 0/7 times where Δ>0\Delta > 0, picking the optimum for the distorted record. Where Δ=0\Delta = 0 they were correct 14/14, yet neither reported the true count in any of those cases (0/14). We stated this prediction before running any decisions.

2. Independent human check and follow-up. Two annotators outside the study labeled all 80 texts, seeing only the target issue and the handover. They agree on the count category for 75/80 texts and confirm 7/7 and 5/7 of the model-coded errors. We then ran a separately frozen decision run on eight further errors that both humans agree on. Each coordinator gets 0/8 correct at Δ>0\Delta>0. At Δ=0\Delta=0, Sol gets 16/16 and Opus 15/16, giving 31/32 across the two coordinators. All eight texts come from the same 80, so this is not a fresh corpus.

3. Intervention and external transfer. On nine archived natural errors, a correction that states both what was inspected and what it found gives 26/27 correct actions at Δ>0\Delta > 0, against 0/27 for the matched sham, across Sol, Opus and Claude Sonnet 5 (App. Q.2). Over all three utilities it gives 79/81. The two misses are one at Δ>0\Delta > 0 and one repair-induced at Δ=0\Delta = 0. On externally authored CPB-Live packets with controlled relation errors, each coordinator gets 0/12 correct at Δ>0\Delta > 0 and 12/12 at Δ=0\Delta = 0 (App. P, Q.2). The errors in these packets are injected, since first-pass external compression produced no determinate count error.

4. Coordinator diagnostic. Take the 57 memories that the archived model screen labeled faithful. Opus gets 29/57 right with an action-only prompt, 48/57 when asked to report count, risk and action, and 55/57 when given the count ($\S$4.6, App. Q.3). Of the nine readout failures, seven report a wrong count and then act consistently with it. This localizes the failures under the diagnostic prompt. It cannot tell us what the model was reasoning in the earlier action-only calls.

5. Presentation and positioning. $\S$2 now defines the terms and walks through the full primary task before the equations. $\S$6 compares ORALMEM with LongMemEval and FactGraph. A prompt-based faithfulness probe (App. Q.4) shows that generic judges usually catch relation errors, and Δ\Delta adds the consequence for the specific task.

We fixed each protocol before that experiment's own calls. We report all adaptive follow-ups, failed predictions, missing calls, a generator-model substitution and over-length outputs. The updated Supplementary Material contains the code, recorded protocols, prompts, annotations and outputs we refer to, and python reproduce.py runs 17 offline analyses.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.