- Rating
- 2
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 2(+1)
- Soundness
- 2
Read review
Paper summary
This paper introduces ORALMEM, a framework that separately evaluates the fidelity of handover memories, whether memory errors change the optimal decision, and coordinators' actual decisions. Through several sets of author-designed resource allocation scenarios, the paper finds that correct decisions can conceal distortions in source relationships, while preserving those relationships does not guarantee correct coordinator decisions.
Critical strengths
Separating memory fidelity, the decision consequences of memory errors, and actual coordinator behavior is a useful evaluation principle. It helps avoid treating final decision accuracy as sufficient evidence that intermediate information has been faithfully preserved and appropriately used.
Critical weaknesses
-
The presentation substantially impedes understanding and assessment of the core experimental setting. Terms such as "handover," "shared memory," "bounded memories," and "capsules" are used interchangeably or in close succession without clearly establishing their relationships. S1/D3/I3 are not adequately defined when first introduced, and transitions between the illustrative task and the primary task involve changes in parameters and budgets without sufficient guidance. Source lineage, Bayesian updating, and decision loss are introduced before readers have a concrete understanding of the complete task. Consequently, readers must repeatedly backtrack and reconstruct the experimental workflow themselves. The main text requires substantial restructuring: necessary background and consistent terminology should be established, and a complete worked example should precede the formal definitions.
-
The incremental contribution and significance are insufficiently established. Separately evaluating memory fidelity, decision consequences, and actual actions is reasonable, but the paper does not clearly establish what new capability this framework provides beyond existing approaches to source tracking, summarization faithfulness, and decision analysis. The evidence primarily comes from artificially constructed reports, tightly constrained summaries, and fixed decision rules, leaving its applicability to broader multi-agent settings unclear. The competing-incident experiment identifies only three explicit source-relation contradictions, none of which changes the optimal action. These cases demonstrate that decision accuracy can miss memory errors, but do not establish the frequency or broader scope of this phenomenon. Overall, the current framework and empirical evidence do not sufficiently establish a substantial contribution to multi-agent memory research.
Questions for authors
What motivates the choice of a 40-word handover budget? What evidence do the existing experiments provide that the main findings are not highly dependent on this restrictive length constraint?
4 comments
Dear Reviewer,
As the response deadline approaches, please let us know if there are any points we can clarify. If our response has addressed your concerns, we would appreciate hearing whether you would consider raising your score.
Thank you again for your time and feedback.
Best regards, The Authors
Response to the Authors' Comments
While I still have concerns regarding the novelty, significance, and empirical generalizability of the work, I will maintain my overall rating. Nevertheless, I appreciate the authors' efforts to improve the manuscript. In recognition of the improved exposition and clearer experimental descriptions, I have increased my Presentation score from 1 to 2.
Thank you for revisiting the paper and recognizing the improved presentation. We would like to briefly clarify our contribution in light of your remaining concerns about novelty, significance, and generalizability.
- Novelty. LongMemEval examines stages of memory use, and FactGraph evaluates summary faithfulness. ORALMEM connects a specific error about who observed what to a downstream decision. Before the coordinator acts, we can determine whether relying on the altered record would change the optimal action.
- Significance. On the same seven naturally corrupted handovers, each coordinator made 0/7 correct decisions when the error changed the optimal action and 14/14 when it left the optimal action unchanged, as predicted (§4.3). This distinction helps prioritize corrections when several decisions rely on the same handover.
- Generalizability. The seven errors arose in four of ten new event families, and both coordinators followed the predicted pattern. We also observed this pattern across three coordinators in a different decision task using externally authored CPB-Live material with controlled errors.
We would be glad to clarify any of these points. Thank you again.
Response to susY
Thank you. We reorganized the exposition in the order you asked for and added evidence for the consequence step.
W1. Presentation ($\S$2)
The section now opens with terms, moves to a full worked example, and only then gives the definitions.
W2. Added evidence and scope ($\S$4.3)
The new test computes Δ before any action is taken, on naturally generated errors. On the same seven model-consensus errors, each of two coordinators gets 0/7 correct at Δ>0 and 14/14 at Δ=0, and in the Δ=0 cells neither reports a correct count. So the test shows both when an outcome score hides an error and when a different use of the same text exposes it. A separate repair experiment gives 26/27 correct at Δ>0 versus 0/27 for a matched sham, across three coordinators (Q.2).
The seven errors come from 80 handovers across ten new event families (seven of the 40 I3 cases). Two blind human annotators flag 18 and 13 errors. All 13 of the second annotator's errors are also among the first annotator's 18, and on those 13 texts both report the same number of independent observations. They confirm 7/7 and 5/7 of the primary cases. Eight more jointly identified errors show the same Δ>0 pattern in a separate frozen decision run. These rates are descriptive: they come from a synthetic pipeline with a continuation gate and incomplete model coding, and real-world prevalence remains open. The external CPB-Live material adds a different decision task, and its positive-Δ errors are controlled injections.
$\S$6 now compares our contribution with LongMemEval's stage analysis and FactGraph's factuality evaluation. Generic faithfulness judges usually detect our relation errors (Q.4). What we add is the task-specific decision consequence.
Q. Why 40 words?
Forty words is a controlled compression stress test. The checks we have show both that it is feasible and where it breaks:
Together these show that 40 words can be enough, and that a larger cap still leaves some observed errors in place. The definitions apply at any length. We hope the restructured task description and the new decision-consequence evidence support a reconsideration of the presentation and informativeness assessments.