acceptodds
Under review as a conference paper at ICLR 2027

PathMem: Toward Cognition-Aligned Memory Transformation for Pathology MLLMs

Abstract

Computational pathology demands both visual pattern recognition and dynamic integration of structured domain knowledge, including taxonomy, grading criteria, and clinical evidence. In practice, diagnostic reasoning requires linking morphological evidence with formal diagnostic and grading criteria. Although multimodal large language models (MLLMs) demonstrate strong vision–language reasoning capabilities, they lack explicit mechanisms for structured knowledge integration and more reliable memory control. As a result, existing models struggle to consistently incorporate pathology-specific diagnostic standards during reasoning. Inspired by the hierarchical memory process of human pathologists, we propose **PathMem**, a memory-centric multimodal framework for pathology MLLMs. **PathMem** organizes structured pathology knowledge as a long-term memory (LTM) and introduces a **Memory Transformer** that models the dynamic transition from LTM to working memory (WM) through multimodal memory activation and context-aware knowledge grounding, enabling context-aware memory refinement for downstream reasoning. **PathMem** achieves SOTA performance across benchmarks, improving WSI-Bench report generation (**+13.0%** WSI-Precision, **+11.1%** WSI-Relevance) and open-ended diagnosis by **+9.7%** and **+8.9%** over prior WSI-based models. Code: [PathMem](https://anonymous.4open.science/r/PathMem-BB4C/)

Primary area:
foundation or frontier models, including LLMs
Keywords:
Computational Pathology, Memory-Centric Multimodal Learning

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_kEivOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces PathMem, a memory-centric multimodal framework for pathology MLLMs that integrates structured domain knowledge as a long-term memory graph and uses a novel Memory Transformer to dynamically transition it into working memory for context-aware reasoning. Extensive evaluations demonstrate that PathMem achieves sota performance across various pathological benchmarks, significantly improving both report generation and open-ended diagnosis.

Critical strengths

Effectively mimics the hierarchical memory retrieval and diagnostic reasoning process of human pathologists by linking visual patterns with structured medical knowledge graphs.

Consistently achieves superior performance across multiple benchmarks (such as WSI-Bench) and diverse downstream tasks like report generation, diagnosis, and treatment planning.

Employs an efficient hashing-based deduplication and probabilistic multi-evidence fusion mechanism to maintain a high-precision, non-redundant long-term memory bank.

Critical weaknesses

The overall reasoning capability is heavily bounded by the completeness and precision of the initial biomedical literature retrieval and LLM-based triple extraction, which may still carry noise or missing relations despite filtering thresholds.

The sparse top-kk selection and token-level integration within the Memory Transformer introduce additional computational complexity during inference, potentially impacting scalability for ultra-large-scale slide sets.

While evaluated on standard tumor types within benchmark suites, the framework's adaptability to rare or atypical pathological variants—where literature evidence is scarce—remains underexplored.

Questions for authors

See weakness

9 comments
AuthorsOpenReview ↗

Response to Reviewer kEiv on knowledge reliability, inference cost, and rare variants.

Weakness 1 :

The overall reasoning capability is heavily bounded by the completeness and precision of the initial biomedical literature retrieval and LLM-based triple extraction, which may still carry noise or missing relations despite filtering thresholds.

Authors’ Response:

Thank you for highlighting the importance of knowledge quality. Our KG construction combines source-grounded extraction, explicit quality controls, and expert supervision.

  • 1. Rigorous construction.

    • The pipeline performs disease-targeted PubMed retrieval, PMID and normalized-abstract deduplication, and schema-constrained extraction with GPT-4o at temperature 0. The extraction prompt requires explicitly stated information and prohibits guessing.
      • Subsequent processing applies confidence thresholds, evidence-aware confidence caps for hedged/speculative statements, relation normalization, and filtering of experimental/drug-related diagnostic clues.
        • Raw extractions and source PMIDs are retained, together with available evidence spans, enabling traceability and review.
  • 2. Pathologist-supervised validation.

    • Construction and validation were supervised by four pathologists from three medical centers across two countries.

      • They performed article-level screening to determine the included literature and reviewed each constructed KG for the validity and pathological plausibility of its knowledge and relations.

        • The curated resource includes 1,998 accepted articles, 490 disease entities, and 19 relation types (As shown in the table below), providing substantial coverage with domain-expert oversight.

        • CategoryMetricValue
          Graph sizeUnique nodes9,974
          Graph sizeUnique directed edges10,748
          Graph sizeRelation types19
          Source coverageAccepted pathology articles1,998
          Source coverageUnique PMIDs in the initial pool2,914
          ConnectivityLargest component93.9%
          ConnectivityApproximate diameter11
          ConnectivityAvg. path length4.78
          Disease complexityDisease nodes490
          Disease complexityMean feature facts per disease22.36
          Disease complexityMean relation types per disease5.63
          Expert supervisionPathologists4
          Expert supervisionParticipating centers3
          Expert supervisionCountries2
  • 3. Demonstrated downstream utility. Table as listed below shows that activating this curated knowledge improves report generation:

  • ConfigurationBLEU-4WSI-PWSI-R
    External-knowledge baseline0.2410.4040.445
    Static activation0.2840.4920.516
    Dynamic activation0.2780.4890.510
    Full PathMem0.3110.5100.540
    • The full model improves WSI-P/R by 0.106/0.095 over the external-knowledge baseline. These gains support the practical value of the curated KG and its activation mechanisms.
      • Although literature coverage remains finite, the risk is actively mitigated through both automated controls and human validation.
        • We will make these safeguards and the experts’ roles more explicit in the revised manuscript.

Weakness 2 :

The sparse top-k selection and token-level integration within the Memory Transformer introduce additional computational complexity during inference, potentially impacting scalability for ultra-large-scale slide sets.

Authors’ Response:

Thank you for raising this important concern.

  • Our efficiency analysis demonstrates that this overhead is well controlled.

    • Specifically, PathMem achieves 0.69 s/WSI (Generation latency, excluding WSI feature extraction), (1.45 WSIs/s) on a single GPU, matching SlideChat (0.69 s/WSI) and outperforming Quilt-LLaVA (0.75 s/WSI) and WSI-LLaVA (0.93 s/WSI).
    • In terms of GPU memory, PathMem requires 16.3 GB, which is higher than lightweight baselines such as SlideChat (11.58 GB), but remains lower than WSI-LLaVA (16.70 GB) and MLLM-HWSI (25.82 GB).
AuthorsOpenReview ↗

Response to Reviewer kEiv on knowledge reliability, inference cost, and rare variants.

    • Table 1. Inference efficiency analysis on WSI-Bench.

      • MetricsQuilt-LLaVAWSI-LLaVASlideChatMLLM-HWSIPathMem
        Time (s/WSI) ↓0.750.930.692940.69
        GPU Memory (GB) ↓12.6216.7011.5825.8216.3
    • Table 2. Efficiency and practicality analysis of PathMem.

      • MetricValue
        Inference Time~1.45 WSIs/s
        GPU Utilization1 × GPU
        GPU Memory Usage16.3 GB
        Image Feature Size1728 × 4096 tokens
        KG Construction CostOffline embedding + indexing
        KG Update CostRe-embedding + re-indexing
  • Importantly, our sparse top-kk design restricts memory integration to selected relevant entries,

    • while KG embedding and indexing are performed offline, avoiding additional knowledge construction costs during inference.
      • Overall, PathMem achieves competitive inference efficiency with a manageable memory overhead, demonstrating a favorable trade-off between computational cost and memory-enhanced representation.

Weakness 3 :

While evaluated on standard tumor types within benchmark suites, the framework's adaptability to rare or atypical pathological variants—where literature evidence is scarce—remains underexplored.

Authors’ Response:

  • We would like to clarify the scope of PathMem: a pan-cancer, multi-organ memory-augmented MLLM for general whole-slide question answering, pathological analysis, and report generation.

    • Its central contribution is context-dependent activation of structured long-term knowledge for these tasks.

    • The available dataset metadata shows the following breadth:

    • TaskTumor categoriesOrgan/tissue-site groups
      Morphology understanding3025
      Diagnostic assessment3025
      Treatment-related reasoning1212
      Report generation3025
      • The 25 site groups include adrenal/paraganglionic, bladder, breast, cervix, bile duct, colon, lymphoid tissue, esophagus, head and neck, kidney, brain, liver, lung, mesothelium, ovary, prostate, rectum, soft tissue, skin, stomach, testis, thyroid, thymus, uterus, and uvea.
        • These counts describe train/test metadata coverage using the stated organ/tissue grouping.
          • Our KG additionally organizes 490 disease entities.
  • Dedicated validation on rare or atypical variants is outside the present experimental scope.

    • It requires subtype-defined cohorts and targeted evaluation of literature-scarce conditions, addressing a more specific generalization question than the cross-organ, multi-task objective studied here.
      • Broad anatomical coverage and rare-variant robustness are distinct evaluation dimensions.
  • Accordingly, we will explicitly delimit our empirical claims to the reported tasks and cohorts,

    • and identify rare/atypical-variant validation as a subsequent extension.

Dear Reviewer kEiv,

We sincerely appreciate your time and effort in providing such thoughtful reviews and insightful comments.

Could you please kindly let us know if our rebuttal has addressed your concerns?

If you have any further questions regarding our work, we would be delighted to address them.

If our responses have addressed your concerns, we would greatly appreciate your reconsideration of your overall assessment.

Best regards,

Authors

Reviewer_yoA3OpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes PathMem, which is a memory-augmented multimodal LLM for whole-slide image pathology. In Methodology, the authors build a graph from PubMed abstracts by using a GPT-4-class extractor with confidence filtering and noisy-or evidence fusion, and treat it as long-term memory. Then the authors use a memory Transformer to select knowledge embeddings for current slide and question by using static and a dynamic path to scale the selected embeddings by their relevance, and then prepends them to the multimodal input as working memory.

For the experiment section, the paper reports gains in WSI-P and WSI-R On WSI-Bench for report generation and open-ended diagnosis over WSI-LLaVA. Also, the author reports zero-shot results on external datasets with ablations on static and dynamic retrieval, and a sensitivity study, and an inference-cost comparison.

Critical strengths
  1. The motivation is natural and the problem is well-chosen. Pathological diagnosis is a knowledge-intensive task, and the lack of structured knowledge control is indeed a pain point. The result shows that PathMem outperforms WSI-LLaVA on report generation, open-ended diagnosis, and the zero-shot sets.

  2. Training details and code links are provided.

Critical weaknesses
  1. The core mechanisms are not fully defined. The Introduction (Contribution 3) and Figure 3 both emphasize dynamic activation and adaptive selection,but 3.3 formalizes only the single-step top-k selection, which seems to correspond to a static path. The dynamic activation and the criteria for adaptive selection are not formalized. The cognition-aligned in the title and cognitively grounded in Contribution 1 both exceed the scope of existing evidence. There is no experimental verification linking memory pathways to the pathologists' reasoning process; therefore, I think maybe the authors can use terms like inspiration or analogy. In general, The core contribution lies in this mechanism. Without a clear definition of the mechanism, the central claims cannot be verified or reproduced.

  2. There lacks a direct comparison against a baseline using the same KG with standard RAG and no retrieval-augmented baseline appears in any table. The only agentic baseline, PathAgent, is compared in Table 3 (report generation) but not in Tables 2 and 4, and it is not stated whether baseline numbers were reproduced under the same settings or taken from the original papers. The authors should demonstrate that the performance gains are from the memory control mechanism rather than from providing a larger number of knowledge tokens. The table 2 to 4 lack confidence interval or std so it is hard to determine whether the gain exceeds the fluctuations. Without these controls and variances, the gain cannot be attributed to the proposed mechanism, as opposed to the additional knowledge tokens or the KG itself, so the claim is not validated.

Questions for authors
  1. What are the complete algorithms/formulas for dynamic activation and adaptive selection as described in the Introduction and Figure 3? How are the criteria for adaptive selection determined?

  2. Is the difference between the averages (0.768 vs. 0.754 in total average reported in 4.1) statistically significant? Does the zero-shot gain exceed run-to-run variation? I think it could be better if you add such thing as std?

7 comments
AuthorsOpenReview ↗

Response to Reviewer yoA3 on mechanism scope, controlled comparisons, and statistical reliability.

Question 1 :

What are the complete algorithms/formulas for dynamic activation and adaptive selection as described in the Introduction and Figure 3? How are the criteria for adaptive selection determined?

Authors’ Response:

Thank you for your comment.

  • 1. Retrieval and weighting. Let XX denote the multimodal input sequence, ei=PEie_i=PE_i the projected memory vectors, and vˉ=norm⁡(v)\bar v=\operatorname{norm}(v). We compute

    • $ q=\overline{\operatorname{Pool}(X)},\quad c_i=\langle\bar e_i,q\rangle,\quad s_i=c_i/\sqrt d. $

    • The two groups are selected without replacement:

      • I_d=\operatorname{TopK}_{i\notin I_s}(c,k'_d).$$ The gated refinement is: $$z'_i=\operatorname{LN}\left( z_i+\sum_{j\in\{X,a,K\}}g_{i,j}h_{j,i} \right).$$ The effective retrieval counts are: $$k'_s=\min(k_s,N,B),\qquad k'_d=\min(k_d,N-k'_s,B-k'_s).$$
      • Their tokens are separately softmax-weighted using ss and cc, respectively. Since ss and cc have identical rankings, the second group is not an independent learned ranker.
  • 2. Context-dependent activation. A global anchor is computed as

  • $ a=G\left(\sum_i\operatorname{softmax}(s)_i e_i\right), $

    • where GG is a two-layer GELU MLP. For each weighted group ZbZ_b, three attention operations extract information from the input, global anchor, and selected memory:
    • $ H_X=\operatorname{MHA}_X(Z_b,X,X),\quad H_a=\operatorname{MHA}_a(Z_b,a,a),\quad H_K=\operatorname{MHA}_K(Z_b,Z_b,Z_b). $
    • Their contributions are combined through token-specific gates:
      • $ g_i=\operatorname{softmax} \left(\operatorname{MLP}([z_i;h_{X,i};h_{a,i};h_{K,i}])\right), $
      • $ z'i=\operatorname{LN}\left( z_i+\sum{j\in{X,a,K}}g_{i,j}h_{j,i} \right). $
      • The refined groups are concatenated, scaled, and appended to the input:
        • $ X'=\operatorname{Concat} \left(X,\lambda\operatorname{Concat}(Z'_s,Z'_d)\right). $
      • Thus, dynamic activation refers to input-conditioned attention and adaptive gating during memory refinement.
  • 3. Selection criteria. Given a memory bank of NN entries and an optional total retrieval budget BB, the effective retrieval counts are

    • $ k'_s=\min(k_s,N,B),\quad k'_d=\min(k_d,N-k'_s,B-k'_s). $
    • Without an additional cap, B=NB=N. Importantly, ksk_s and kdk_d are predefined configuration hyperparameters, not dynamically inferred from score gaps or a learned stopping criterion.

Question 2 :

Is the difference between the averages (0.768 vs. 0.754 in total average reported in 4.1) statistically significant? Does the zero-shot gain exceed run-to-run variation? I think it could be better if you add such thing as std?

Authors’ Response:

  • We have added sample-level SDs to Table below; for example, morphological WSI-P is 0.535 ± 0.172 for PathMem versus 0.488 ± 0.193 for WSI-LLaVA.
    • These quantify variability across test samples.
ModelInputMorphological Analysis (Open WSI-P)Morphological Analysis (Open WSI-R)Morphological Analysis (Close Acc)Morphological Analysis (Average)Diagnosis (Open WSI-P)Diagnosis (Open WSI-R)Diagnosis (Close Acc)Diagnosis (Average)Treatment Planning (Open WSI-P)Treatment Planning (Open WSI-R)Treatment Planning (Close Acc)Treatment Planning (Average)
WSI-LLaVAWSI0.488±0.1930.610±0.1660.9510.6830.610±0.1630.612±0.1600.8630.6950.810±0.1310.845±0.1271.000±0.0000.885
PathMem (LLaMA)WSI0.535±0.1720.542±0.1720.9720.6830.707±0.1440.701±0.1460.8680.7580.796±0.1370.792±0.1381.000±0.0000.863

AuthorsOpenReview ↗

Response to Reviewer yoA3 on mechanism scope, controlled comparisons, and statistical reliability.

Weakness 1 :

The core mechanisms are not fully defined. The Introduction (Contribution 3) and Figure 3 both emphasize dynamic activation and adaptive selection,but 3.3 formalizes only the single-step top-k selection, which seems to correspond to a static path. The dynamic activation and the criteria for adaptive selection are not formalized. The cognition-aligned in the title and cognitively grounded in Contribution 1 both exceed the scope of existing evidence. There is no experimental verification linking memory pathways to the pathologists' reasoning process; therefore, I think maybe the authors can use terms like inspiration or analogy. In general, The core contribution lies in this mechanism. Without a clear definition of the mechanism, the central claims cannot be verified or reproduced.

Authors’ Response:

Thank you for this insightful comment.

  • We agree with the reviewer that the dynamic activation and adaptive selection mechanisms are insufficiently formalized, making the central architectural contribution difficult to verify and reproduce.

    • We also acknowledge that cognition-aligned and cognitively grounded overstate our current evidence,

      • We will revise Section 3.3 and Figure 3 to accurately reflect the implemented mechanisms,

        • replace cognitive alignment claims with more appropriate terms such as inspired by or motivated by an analogy to human memory processes.

        • We will also revise the title and contributions accordingly.


Weakness 2 :

There lacks a direct comparison against a baseline using the same KG with standard RAG and no retrieval-augmented baseline appears in any table. The only agentic baseline, PathAgent, is compared in Table 3 (report generation) but not in Tables 2 and 4, and it is not stated whether baseline numbers were reproduced under the same settings or taken from the original papers. The authors should demonstrate that the performance gains are from the memory control mechanism rather than from providing a larger number of knowledge tokens. The table 2 to 4 lack confidence interval or std so it is hard to determine whether the gain exceeds the fluctuations. Without these controls and variances, the gain cannot be attributed to the proposed mechanism, as opposed to the additional knowledge tokens or the KG itself, so the claim is not validated.

Authors’ Response:

We thank the reviewer for pointing out the insufficient baseline comparisons and statistical analysis.

1、Compare with RAG-Based Method

  • We have added DOS-RAG [1], a representative RAG-based method, as an additional baseline.

    • Unlike PathMem, DOS-RAG relies on conventional retrieval without the proposed memory control mechanism.

      • As shown in the revised comparison, PathMem consistently outperforms DOS-RAG across all eight evaluation metrics, demonstrating the benefit of our approach beyond conventional RAG.

      • VersionExternal KnowledgeStatic Ret.Dynamic Ret.BLEU-1BLEU-2BLEU-3BLEU-4ROUGE-LMETEORWSI-PWSI-R
        a (Baseline)✓0.4810.3570.2870.2410.4770.4640.4040.445
        b✓✓0.5280.4050.3320.2840.5180.5080.4920.516
        c✓✓0.5210.3960.3240.2780.5100.5110.4890.510
        d (Ours)✓✓✓0.5610.4330.3600.3110.5560.5300.5100.540
        DOS-RAG [1]✓--0.5310.3820.3070.2530.4960.4860.4420.479
    • The results of PathAgent reported in Table 3 (for report generation) were obtained from our own experiments, rather than directly adopted from its original paper. We independently ran PathAgent in our experimental environment to ensure a consistent basis for comparison with PathMem, and acknowledge the missing evaluations as a limitation.

2、Between-sample standard deviation in Table 2, 3, 4, 5

  • We have added between-sample standard deviations to Table 2, 3, 4, 5, reporting results in the form of mean ± standard deviation.
    • These statistics provide a more comprehensive characterization of performance variability and allow readers to better assess the consistency of the observed improvements.
      • For baselines whose results are adopted from prior work, we retain the originally reported scores when sample-level results are unavailable.
AuthorsOpenReview ↗

Response to Reviewer yoA3 on mechanism scope, controlled comparisons, and statistical reliability.

  • Table 2:

    • ModelInputMorphological Analysis (Open WSI-P)Morphological Analysis (Open WSI-R)Morphological Analysis (Close Acc)Morphological Analysis (Average)Diagnosis (Open WSI-P)Diagnosis (Open WSI-R)Diagnosis (Close Acc)Diagnosis (Average)Treatment Planning (Open WSI-P)Treatment Planning (Open WSI-R)Treatment Planning (Close Acc)Treatment Planning (Average)
      WSI-LLaVAWSI0.488±0.1930.610±0.1660.9510.6830.610±0.1630.612±0.1600.8630.6950.810±0.1310.845±0.1271.000±0.0000.885
      PathMem (LLaMA)WSI0.535±0.1720.542±0.1720.9720.6830.707±0.1440.701±0.1460.8680.7580.796±0.1370.792±0.1381.000±0.0000.863
  • Table 3:

    • ModelsBLEU-1BLEU-2BLEU-3BLEU-4ROUGE-LMETEORWSI-PrecisionWSI-Relevance
      GPT-4o0.2020.0690.0300.0160.1320.1670.0670.138
      Quilt-LLaVA0.4740.3510.2820.2360.4750.4600.3240.333
      MI-Gen (WSICaption)0.4030.3060.2480.2090.4460.4070.3100.377
      Hist-Gen0.4060.3070.2480.2080.4480.4160.3000.397
      WSI-LLaVA0.4800.3580.2870.2400.4900.4650.3800.429
      PathMem (Ours)0.561 ± 0.1490.433 ± 0.1700.360 ± 0.1870.311 ± 0.1940.556 ± 0.1530.530 ± 0.1560.510 ± 0.1430.540 ± 0.141
  • Table 4:

    • Table. Zero-shot validation on external benchmarks.

      ModelInputWSI-VQATumorGradeSubtypeAvgCPTAC
      Quilt-LLaVAThumb0.1320.1130.1940.1810.1630.603
      WSI-VQAWSI0.4690.0390.3000.0000.1130.000
      SlideChatWSI–0.9020.2310.1750.436–
      WSI-LLaVAWSI0.5460.9050.4630.2920.5530.721
      PathMem (Ours)WSI0.572 ± 0.1700.9180.482 ± 0.1530.313 ± 0.2090.571 ± 0.1440.754
Reviewer_EhaQOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2(+1)
Read review
Paper summary

PathMem augments a pathology multimodal language model with literature-derived structured knowledge. A pathology knowledge graph serves as long-term memory, while query-conditioned selection produces a smaller working-memory block that is concatenated with the multimodal input. The proposed controller combines static and dynamic activation, with an adaptive selection mechanism. Evaluation covers WSI-Bench and three external datasets. For report generation, the LLaMA variant reports WSI-P/WSI-R of 0.510/0.540, compared with 0.380/0.429 for WSI-LLaVA and 0.502/0.518 for MLLM-HWSI. Component ablations examine static and dynamic retrieval.

Critical strengths

The knowledge graph contains 9,974 nodes and 10,748 directed edges, and its construction and validation involve four pathologists from three medical centers. Explicitly representing this knowledge creates a useful opportunity for inspecting provenance and updating knowledge separately from the language model. The expert involvement is a meaningful strength, although it does not by itself establish the graph’s error rate. The external evaluations report improvements over WSI-LLaVA on WSI-VQA, SlideBench-VQA, and CPTAC. The static-only, dynamic-only, and combined ablations also provide evidence that the proposed components affect performance.

Critical weaknesses

Section 3.3 formalizes query–memory similarity, masking, top-(k) selection, relevance weighting, and concatenation. However, I could not reconstruct from this specification how the static and learned dynamic branches are combined, how the adaptive score-gap cutoff is implemented, or how the “graph-aware” memory embeddings are learned. These details matter because they constitute the claimed architectural contribution. Table 5 is useful, but every row already includes external knowledge; it does not establish the advantage over a conventional retrieval-and-concatenation pipeline matched for knowledge, backbone, training, and token budget.

Questions for authors

Can you provide a complete specification of memory embedding construction, dynamic scoring, static–dynamic fusion, adaptive cutoff selection, and controller training? Does the advantage remain against retrieval using the same backbone, knowledge bank, and retrieved-token budget? What are the exact claim sources, denominators, judge model, prompts, and exclusion rules for WSI-P and WSI-R? Were all compared outputs evaluated using the same implementation? How are unsupported findings absent from the reference report assessed? Where are the case-disjoint results mentioned in the reproducibility statement?

7 comments
AuthorsOpenReview ↗

Response to Reviewer EhaQ on reproducible memory control, matched RAG, and evaluation.

Question 1 :

Can you provide a complete specification of memory embedding construction, dynamic scoring, static–dynamic fusion, adaptive cutoff selection, and controller training? Does the advantage remain against retrieval using the same backbone, knowledge bank, and retrieved-token budget? What are the exact claim sources, denominators, judge model, prompts, and exclusion rules for WSI-P and WSI-R? Were all compared outputs evaluated using the same implementation? How are unsupported findings absent from the reference report assessed? Where are the case-disjoint results mentioned in the reproducibility statement?

Authors’ Response:

Thank you for these important questions.

1. Memory construction, retrieval, fusion, and training.

  • KG edges are serialized as head–relation–tail text, optionally including confidence, provenance, and evidence.
    • A text encoder extracts final-layer representations using non-padding mean pooling, with optional normalization. The resulting bank is frozen and projected into the model space. Given multimodal input (X), retrieval uses normalized cosine similarity:
      • $ s_i=\frac{\langle\operatorname{norm}(PE_i),q\rangle}{\sqrt d}, \quad q=\operatorname{norm}(\operatorname{Pool}(X)). $
      • The static group selects the top-ksk_s entries, while the dynamic group selects the top-kdk_d entries from the remaining bank. Both groups use the same underlying similarity ranking and are independently softmax-weighted.
      • The learned controller operates in the subsequent refiner. It combines attention outputs from the multimodal input, global KG anchor, and selected memory group through context-dependent gates:
        • $ g=\operatorname{softmax}(\operatorname{MLP}([z;h_X;h_a;h_K])), $
        • $ z'=\operatorname{LN}(z+g_Xh_X+g_ah_a+g_Kh_K). $
        • The refined groups are concatenated and injected into the original input:
          • $ X'=\operatorname{Concat}(X,\lambda\operatorname{Concat}(Z'_s,Z'_d)). $
        • The memory bank is frozen, whereas the KG projector, global projector, and refiner are trainable under supervised causal-language-model loss, with knowledge-token labels masked.
          • Adaptive selection. We acknowledge that the implementation uses configured ksk_s and kdk_d, subject to bank size and any total selection cap.

2. Matched conventional retrieval comparison.

we added a matched comparison with conventional retrieval baselines, DOS-RAG [1].

  • Specifically, we control for the backbone, external knowledge bank, and total retrieved-token budget, ensuring that the comparison isolates the effect of the retrieval strategy.

  • The results demonstrate that

    • our method consistently outperforms both static-only retrieval (b) and dynamic-only retrieval (c) across all eight evaluation metrics.

    • Compared with the stronger of these two baselines on each metric, our method achieves absolute improvements of 0.027–0.033 on BLEU, 0.038 on ROUGE-L, 0.019 on METEOR, and 0.018/0.024 on WSI-P/R, respectively.

    • VersionExternal KnowledgeStatic Ret.Dynamic Ret.BLEU-1BLEU-2BLEU-3BLEU-4ROUGE-LMETEORWSI-PWSI-R
      a (Baseline)✓0.4810.3570.2870.2410.4770.4640.4040.445
      b✓✓0.5280.4050.3320.2840.5180.5080.4920.516
      c✓✓0.5210.3960.3240.2780.5100.5110.4890.510
      d (Ours)✓✓✓0.5610.4330.3600.3110.5560.5300.5100.540
      DOS-RAG [1]✓--0.5310.3820.3070.2530.4960.4860.4420.479
  • These results suggest that the improvement does not simply come from using a stronger backbone, a larger knowledge bank, or more retrieved tokens.

    • We will add these results and the corresponding discussion to the revised manuscript.
AuthorsOpenReview ↗

Response to Reviewer EhaQ on reproducible memory control, matched RAG, and evaluation.

3. WSI-P and WSI-R evaluation definitions.

Following [2], we use GPT-4o for claim extraction and scoring.

MetricClaim sourceEvaluation
WSI-PReference-answer claimsCorrectness of the model response against each claim
WSI-RModel-generated claimsSupport/relevance of the reference answer to each claim

The scoring prompts assign 0, 0.3, 0.7, or 1.

  • Denominators and exclusions. Each sample’s score is the mean of its valid numerical claim scores, rounded to three decimals. The overall score averages valid sample scores, rather than pooling all claims.
    • Nonnumerical scores and samples without valid scores are excluded; we will report parsing failures and exclusion counts for each model.
      • For WSI-R, claims unmentioned in the reference are ignored, so this metric does not fully penalize unsupported additions.

4、Judge and prompts.

Following [2], we use GPT-4o for claim extraction and scoring.

WSI-P: Reference-claim extraction

  • You are an AI assistant specialized in processing pathological diagnosis Q&A pairs. I will provide you with a pathology diagnosis question and its corresponding answer.

  • Your task is to: Claims Extraction: • Carefully analyze the answer and remove any unnecessary information that is not directly relevant to the question. • Only extract claims that directly address the question. Discard any information that does not directly answer or pertain to the question. • Divide the refined answer into several distinct and granular claims. • Keep closely related information together in the same claim to preserve context and meaning. Do not split sentences or ideas that are logically connected. • Break down complex sentences into smaller, individual claims only if it does not disrupt the logical flow or separate connected ideas. • Ensure there is no omission or repetition among the claims.

  • Guidelines: • Only output the claims without including any additional text or explanations. • Each claim should be concise and represent a single fact or point directly related to the question. • Maintain the integrity of statements that are contextually connected.

  • Output Format: Present the extracted claims as a list in the following format: ["claim1", "claim2", "claim3", ...]

WSI-P: Claim scoring

  • Please act as an impartial judge and evaluate the correctness of the AI assistant's pathology dialogue for each claim based on the following scoring criteria. Provide an explanation for each evaluation and assign a score.

  • Scoring Criteria:

    • 1: The information in the pathology dialogue is completely correct regarding the claim.
    • 0.7: The information is mostly correct and closely aligns with the claim.
    • 0.3: The claim is mentioned but contains errors in the core content (e.g., mistakes in differentiation degree or malignancy).
    • 0: The information in the pathology dialogue is completely incorrect regarding the claim.
  • Output Requirements:

  • Please output your evaluations as a list of dictionaries in plain text format (not JSON). The format should be as follows:

  • [ { "claim": "Original claim1", "explanation": "Explanation for the score", "score": 1 or 0.7 or 0.3 or 0 }, { "claim": "Original claim2", "explanation": "Explanation for the score", "score": 1 or 0.7 or 0.3 or 0 }, ... ]

WSI-R: Model-claim extraction

  • You are an AI assistant specialized in processing pathological diagnosis texts. I will provide you with a pathology diagnosis text.

  • Your task is to:

  • Claims Extraction: • Divide the text into several distinct and granular claims. • Keep closely related information together in the same claim to preserve context and meaning. Do not split sentences or ideas that are logically connected. • Break down complex sentences into smaller, individual claims only if it does not disrupt the logical flow or separate connected ideas. • Ensure there is no omission or repetition among the claims.

  • Guidelines: • Only output the claims without including any additional text or explanations. • Each claim should be concise and represent a single fact or point. • Maintain the integrity of statements that are contextually connected.

  • Output Format: Present the extracted claims as a list in the following format: ["claim1", "claim2", "claim3", ...]

AuthorsOpenReview ↗

Response to Reviewer EhaQ on reproducible memory control, matched RAG, and evaluation.

WSI-R: Claim scoring

  • Please act as an impartial judge and evaluate the relevance of the original ground truth answer to each claim derived from the model's answer. Provide an explanation for each evaluation and assign a score based on the following criteria.

  • Scoring Criteria:

    • 1: The content in the ground truth answer is completely relevant to the claim.
    • 0.7: The content is mostly relevant but has minor omissions or deviations.
    • 0.3: The content is partially relevant with significant omissions or irrelevant information.
    • 0: The content in the ground truth answer is not relevant to the claim.
  • Evaluation Guidelines: • Focus on Relevance: Assess how well the content in the ground truth answer pertains to the claim derived from the model's answer. • Use Partial Credit: Utilize intermediate scores (0.7 and 0.3) to reflect varying degrees of relevance. • Ignore Unmentioned Claims: Do not evaluate claims that are not addressed in the ground truth answer.

  • Output Requirements: Please output your evaluations as a list of dictionaries in plain text format (not JSON). The format should be as follows: [ { "claim": "Original claim1", "explanation": "Explanation for the score", "score": 1 or 0.7 or 0.3 or 0 }, { "claim": "Original claim2", "explanation": "Explanation for the score", "score": 1 or 0.7 or 0.3 or 0 }, ... ]

5、Other Concerns

  • ** Evaluator consistency.** For a fair comparison, we re-evaluate all methods with accessible outputs using the same GPT-4o evaluator and identical prompts, reference claims, test IDs, parsing, aggregation, and exclusion rules.
    • Following [2], we report valid sample counts and parsing failures for each method. Published results without accessible outputs are retained for context but excluded from this controlled comparison.
      • Thus, all methods in our quantitative comparison are evaluated under an identical protocol.
  • Validation of findings beyond references. Absence from a reference report does not necessarily imply hallucination, as pathology reports may not capture all slide-level findings.
    • Following [2], accordingly, WSI-P/R measure reference agreement, not factual validity. To assess additional findings, we conduct blinded, slide-grounded pathological review, classifying them as supported, contradicted, or unverifiable.
      • This distinguishes genuine additional pathological findings from unsupported generations.
  • Case-disjoint evaluation. Thank you for identifying this omission. A patient-disjoint split has been prepared by excluding training examples from test patients, identified by their TCGA participant barcodes. We verified zero patient overlap in this split.

Referenece

[1] Laitenberger, Alex, Christopher D. Manning, and Nelson F. Liu. "Stronger baselines for retrieval-augmented generation with long-context language models." Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing. 2025.

[2] Liang, Yuci, et al. "Wsi-llava: A multimodal large language model for whole slide image." 2025 IEEE/CVF International Conference on Computer Vision (ICCV). IEEE, 2025.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.