- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper investigates whether consolidating stored experiences can change downstream decisions even when aggregate retrieval success and task accuracy remain stable. It constructs paired narrative and risk-attribute memories from 394 occupational-incident records, holds the target narratives and outcome labels fixed, excludes self-retrieval, and evaluates nine-class accident classification. The primary experiment reports identical Hit@3 of 87.06%, but a 5.20-percentage-point excess of cross-condition decision disagreement over same-memory repeat disagreement. Additional experiments substitute the consumer or retriever and examine recurrence of vulnerable targets.
Critical strengths
-
The paired intervention is easy to interpret: the incident, target query and historical outcome remain fixed while the stored representation changes. Excluding the target from retrieval avoids a direct self-retrieval shortcut.
-
The distinction between aggregate accuracy and instance-level decision stability is useful. The repeat control and paired bootstrap make the reported disagreement more informative than a simple comparison of two stochastic model outputs. The paper also reports both harmful and beneficial decision changes rather than equating every change with an error.
-
The retrieval analysis goes beyond a single embedding model, and the concentration near the caught-between/struck-by boundary gives the study an interpretable observation beyond an aggregate score.
Critical weaknesses
-
The experiment does not establish how much the task depends on memory. The consumer receives the complete target accident narrative, which may already contain most of the information needed for classification. Without a query-only baseline, it is difficult to distinguish disruption of genuinely necessary memory use from sensitivity to optional in-context examples. The paper acknowledges the single-corpus and non-sequential setting, but the central memory interpretation still needs this control. Moreover, decision preservation is a diagnostic, not automatically a desirable objective: the paper itself reports 25 incorrect-to-correct changes alongside 30 correct-to-incorrect changes across the two repeats.
-
The intervention identifies a total effect, not which part of the representation–retrieval–decision pathway causes it. The retrieved record identities and the content shown for those records change together. A crossed control that independently fixes retrieved IDs and switches their displayed representations would distinguish retrieval-mediated effects from direct effects of compressed evidence. Also, label purity already drops from 66.92% to 57.70%, so the result is specifically a limitation of aggregate Hit@k and accuracy, not evidence that retrieval metrics in general fail to reveal a change.
-
The cross-component localization null does not account for outcome-category difficulty. Table 4 shows that 12 of the 15 targets shared across consumers are caught-between cases. Using the category-specific vulnerable-set sizes in that table gives an expected overlap of approximately 4.30 under within-category independence, compared with 1.59 under the pooled null. The observed overlap of 15 is still substantially larger; this does not dismiss the finding, but it changes the meaning of the headline 9.43-fold enrichment. A stratified analysis is needed to distinguish transferable instance susceptibility from shared category composition. Two repeats also provide only a limited basis for labeling individual targets reproducibly vulnerable.
Questions for authors
-
What is the query-only classification accuracy and repeat disagreement? Can the authors cross retrieved record identity with full versus consolidated evidence presentation to identify the main source of the decision changes?
-
Does vulnerable-set overlap remain significant under a label-stratified permutation test, and how stable are the identified targets when more independent repeats are used?
-
Can the authors provide the paired-record construction specification, complete prompts/category definitions and exact inference settings, together with the per-target predictions underlying the bootstrap and overlap analyses?
2 comments
Follow-up on our response
Dear Reviewer,
Thank you again for your careful and constructive review. We understand this is a busy period and we would welcome any further questions or comments on our response before the discussion period closes.
Best regards,
Response to Reviewer 3
We thank the reviewer for the careful reading and constructive suggestions. We appreciate the recognition of the paired design and instance-level findings. The review helped us strengthen the controls and interpretation. Changes are marked in blue.
Summary of revisions: We added a query-only baseline, crossed retrieval/representation conditions, and five independent repeats across five conditions and two consumers (19,700 predictions). We added a second task, category-adjusted overlap tests, and Appendices A-D.
Responses to Critical Weaknesses:
W1: We agree that the complete target narrative may already support classification. The new query-only baseline makes this clear: without retrieved memory, accuracy is 86.95% for Sol and 86.60% for Astra, compared with 88.02% and 88.68% under Full memory (Table B.1). Thus, memory improves aggregate accuracy only modestly. It does, however, affect individual predictions: adding Full memory produces repeat-controlled excess disagreement of +4.52 pp [2.89, 6.32], comparable to +5.40 pp [3.63, 7.35] for Full versus Consolidated memory (Table B.2). We have narrowed the claim to how memory changes decisions, not how much classification depends on memory. We also agree that preserving a decision is not necessarily desirable. Across five repeats, consolidation changed 16 majority decisions from correct to incorrect and 14 in the other direction under Sol (Astra: 13 and 14; Table B.3). Sections 5.3 and 6 now explicitly treat decision preservation as a diagnostic, not an accuracy objective.
W2: We implemented the crossed controls the reviewer suggested. FC displays the Full-retrieved records in consolidated form; CF displays Consolidated-retrieved records in full form. This lets us compare retrieval identity while holding presentation fixed, and presentation while holding retrieved IDs fixed. Changing retrieved IDs produces larger contrasts (+5.24 and +4.37 pp) than changing only displayed content (+2.33 and +0.40 pp; Section 5.3, Table B.2). This is evidence that retrieval identity contributes more strongly in our setting, not proof of a universal causal mechanism. The reviewer is also right about retrieval metrics: label purity falls from 66.92% to 57.70%, and diversity rises from 1.57 to 1.86. Our revised claim is specifically about the limitations of aggregate Hit@k and accuracy, not all retrieval measures. Section 6 now recommends tracking neighbor identity and label composition.
W3: We agree that the original pooled null overstated the enrichment. Therefore, we replaced it with a category-adjusted null preserving vulnerable-target counts within each outcome class, with exact p-values from the convolution of per-category hypergeometric distributions (Sections 3.3 and 4.5). With five repeats, 13 targets are shared across consumers versus 3.35 expected (3.88×, p = 9.35 × 10⁻⁹). The enrichment persists at thresholds 0.3 and 0.7 (4.00× and 3.28×). Within caught-between cases alone, 9 are shared versus 2.80 expected (p = 2.25 × 10⁻⁵). Under retriever substitution, 11 are shared versus 2.01 expected (5.48×, p = 1.01 × 10⁻⁸); only three are caught-between. All 13 consumer-shared targets have the same modal Full-to-Consolidated transition under both models, with each mode supported by at least four of five repeats. We retain and explicitly identify the two-repeat protocol for retriever substitution (Section 5.4).
Responses to Questions for Authors:
Query-only within-condition repeat disagreement is 1.68% for Sol and 0.81% for Astra. The accuracy and crossed-condition contrasts are reported above and in Tables B.1–B.2.
Yes. We use an exact category-stratified overlap test rather than relying on the pooled null. Results remain enriched across vulnerability thresholds, and all 13 consumer-shared targets retain the same modal decision transition across models (Section 5.4; Tables C.1–C.2).
Appendix A specifies paired-record construction, all five conditions, classification prompts and category definitions, inference settings, normalization, and the repeat/bootstrap/overlap procedures. Appendix D details the second task; Appendices B-C provide additional results. The 19,700 per-target prediction records were used to verify the reported statistics. Individual records are not included in the appendix.
We appreciate the reviewer’s suggestions. They led to more informative controls, a fairer statistical comparison and a more precise statement of what the evidence shows.