acceptodds
Under review as a conference paper at ICLR 2027

Zero-Shot Context-Aware Medical Case Retrieval via Token-Set to Token-Set Similarity

Abstract

In healthcare, model-predicted outcome labels are, in practice, unhelpful, especially when clinicians are concerned about the uncertainty of these predictions. In contrast, references to past similar cases provide valuable guidance. Therefore, in this work, when presented with a new patient, our objective is to retrieve similar past medical cases to support treatment planning within a specific clinical context of interest. Previous methods rely on semantic searches and content-based image retrieval using vector databases of EHR and medical image embeddings. Although efficient, these approaches often lack contextual relevance, leading to lower retrieval performance. Recent multimodal foundation models can enable more effective context-aware cross-modal retrieval. While these models provide deeper context and richer embeddings, they are computationally expensive. Considering this, we propose a two-stage retrieval system that combines the efficiency of vector-based searches with the contextual depth of token embeddings provided by foundation models without further training. More importantly, we propose a token-set to token-set similarity for context-aware retrieval using rich token embeddings. Our extended experiments demonstrate the efficiency contributed by Stage 1 and the effectiveness contributed by the context-aware re-ranking in Stage 2.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
medical AI, multimodal medical LLMs, zero-shot, token-set to token-set, medical case retrieval

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_tPRxOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper focuses on context-aware medical case retrieval. The method itself is a standard two-stage pipeline. Stage1 retrieval candidates via FAISS; Stage2 feeds context and images into CheXagent, takes hidden stages at the context-question token positions, and reranks candidates. Experiments run on 482 queries across 6 pathologies, with extensive ablations. The authors report consistent improvements over stage 1 baselines and honestly note regressions on a few pathologies.

Critical strengths
  1. Well motivated with zero-shot settings, matching real deployment where no labeled data exists per contexts.
  2. Detailed in-domain evaluation. The evaluation, ablations, latency msesurements and negative results are detailed and efficient.
Critical weaknesses
  1. The paper claims "similar cases for treatment planning", but relevance is just same pathology labeling. Other relevance signals such as severity are ignored.
  2. MaxSim and mean-pooled cosine are never compared with the core design of the method. This makes the core claim untested, weakening the contribution of the paper.
  3. How does a zero-shot label-matching baseline ( retrieve by predicted-label agreement ) perform?
Questions for authors

See Weaknesses.

1 comment

We thank the reviewer for the insightful comments. We will add the following comparisons to the revision.

Q1A & Q2A: In the original evaluation, a retrieved case is relevant if its binary label for the queried finding matches the query's label. Therefore, we add more experiments on MS-CXR which contains bounding boxes for additional evaluation. Concretely, we first average over queries within each finding and then equally across six findings from the label agreement perspective. Both query cohorts (the original 482 queries and MS-CXR test queries) search the same 720-image report-available MS-CXR train+validation corpus. CheXagent reranks the visual RadIR top 50 with image and report inputs. we added the following comparison.

We can observe that Sinkhorn as a distribution to distribution measure would be more beneficial when the involved number of tokens are higher. They are similar with small number of tokens under expectation. Therefore, Sinkhorn is generally more beneficial.

Scorer and token set, image + reportMS-CXR tokens (query / candidate)MS-CXR nDCG@5 (94 positive queries)Original cohort tokens (query / candidate)Original cohort nDCG@5 (593 positive queries)
Visual RadIR, before reranking1 global vector per case.4291 global vector per case.359
Sinkhorn / MaxSim, all tokens1,177–1,435 / 1,146–1,656.434 / .3871,145–1,686 / 1,145–1,656.393 / .386
Sinkhorn / MaxSim, image tokens1,024 / 1,024.449 / .4181,024 / 1,024.387 / .371
Sinkhorn / MaxSim, report tokens88–346 / 59–568.422 / .40857–597 / 59–568.422 / .417
Sinkhorn / MaxSim, question tokens11–14 / 11–14.499 / .49911–14 / 11–14.476 / .476
Mean-pooled cosine, question tokens11–14 / 11–14.46311–14 / 11–14.432

Moreover, on the same 94 MS-CXR queries, box coverage@5 is the fraction of the top five cases with a reference box for the target finding. Region cosine@5 compares features from the query and retrieved annotated regions. The proposed method is further demonstrated. External clinician validation is costly and will be our future work.

MethodBox coverage@5BioMedCLIP region cosine@5
RadIR visual28.0%.795
Question-token Sinkhorn30.8%.840

Q3A: Hard zero-shot predicted-label agreement achieved .415 nDCG@5 and 26.2% reference-box coverage@5 on MS-CXR. The corresponding results were .402 and 25.1% for visual RadIR and .466 and 33.4% for question-token Sinkhorn.

All methods use image-only case inputs and the same visual RadIR top 50. The corpus is the 892-image MS-CXR train+validation split. We evaluate 170 positive test image–finding queries from 128 patients and average equally over findings. Box coverage@5 is the fraction of the top five cases with a reference box for the queried finding. This further demonstrated the proposed method.

Image-only rerankerHit@1Precision@5nDCG@5Box coverage@5
RadIR visual.403.399.40225.1%
Hard predicted-label agreement.409.413.41526.2%
Soft predicted-label agreement.422.386.38824.2%
Question-token Sinkhorn.507.455.46633.4%
Reviewer_DxVqOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a zero-shot, context-aware medical case retrieval framework that combines coarse-grained vector retrieval with token-level re-ranking. The first stage uses pretrained embeddings to retrieve candidate cases, while the second stage extracts context-conditioned representations from CheXagent and applies Sinkhorn-based optimal transport to measure token-set similarity. Experiments on MIMIC-CXR across six pathologies show improvements in label-based retrieval metrics over the evaluated baselines.

Critical strengths
  1. The two-stage framework combines efficient candidate retrieval with fine-grained re-ranking using pretrained medical foundation models. It does not require additional training, which is useful for adapting to different diagnostic contexts.
  2. The experiments compare several medical retrieval encoders across six pathologies. The reported improvements in top-ranked retrieval performance are encouraging, and the ablation studies provide some evidence for the effectiveness of the proposed framework.
Critical weaknesses
  1. It is unclear how much the proposed Sinkhorn-based similarity measure contributes to the retrieval performance. Although MaxSim and optimal transport are discussed in Section 2.4, the paper does not provide a direct comparison using the same token representations and candidate pools. Also, the token-filtering ablation changes both the number and type of tokens, so it is difficult to tell which component is responsible for the improvement. I suggest adding a more controlled comparison to better understand the contribution of each component.
  2. The experiments do not provide sufficient evidence for the claimed context awareness. The evaluation covers six pathologies, but it would be useful to examine whether the retrieval rankings change appropriately when different diagnostic questions are given for the same patient. In addition, Section 4.1 uses pathology-label agreement to evaluate retrieval relevance. However, cases with the same pathology may differ in severity, anatomical findings, and clinical presentation. This limitation should be discussed more clearly.
Questions for authors
  1. How does the proposed Sinkhorn-based similarity measure compare with MaxSim when using the same CheXagent token representations and candidate pools? This comparison would help clarify the contribution of the proposed similarity measure.
  2. Can the authors provide an experiment where the diagnostic question is changed while the patient record remains the same? This would help demonstrate whether the retrieval results actually adapt to different diagnostic contexts.
  3. Does the latency reported in Table 2 include CheXagent feature extraction for the candidate cases? Also, are the context-conditioned representations recomputed for each query, or can they be reused? It would be helpful to clarify the computational cost of the second stage.
  4. Could the authors clarify whether Figure 3 reports positive-class or macro-averaged results? There also seem to be some inconsistencies between Equation (2), Equation (4), and Algorithm 1. In addition, were any statistical significance tests conducted to support the claims in Section 4.2?
3 comments

We thank the reviewer for the insightful comments. We will add the following comparisons and clarifications to the revision.

Q1A & Weakness: In the original evaluation, a retrieved case is relevant if its binary label for the queried finding matches the query's label. Therefore, we add more positive-query experiments on MS-CXR which contains bounding boxes for additional evaluation. Concretely, we first average over queries within each finding and then equally across six findings from the label agreement perspective. Both query cohorts (the original 482 queries and MS-CXR test queries) search the same 720-image report-available MS-CXR train+validation corpus. CheXagent reranks the visual RadIR top 50 with image and report inputs. we added the following comparison.

We can observe that Sinkhorn as a distribution to distribution measure would be more beneficial when the involved number of tokens are higher. They are similar with small number of tokens under expectation. Therefore, Sinkhorn is generally more beneficial.

Scorer and token set, image + reportMS-CXR tokens (query / candidate)MS-CXR nDCG@5 (94 positive queries)Original cohort tokens (query / candidate)Original cohort nDCG@5 (593 positive queries)
Visual RadIR, before reranking1 global vector per case.4291 global vector per case.359
Sinkhorn / MaxSim, all tokens1,177–1,435 / 1,146–1,656.434 / .3871,145–1,686 / 1,145–1,656.393 / .386
Sinkhorn / MaxSim, image tokens1,024 / 1,024.449 / .4181,024 / 1,024.387 / .371
Sinkhorn / MaxSim, report tokens88–346 / 59–568.422 / .40857–597 / 59–568.422 / .417
Sinkhorn / MaxSim, question tokens11–14 / 11–14.499 / .49911–14 / 11–14.476 / .476
Mean-pooled cosine, question tokens11–14 / 11–14.46311–14 / 11–14.432

Moreover, on the same 94 MS-CXR queries, box coverage@5 is the fraction of the top five cases with a reference box for the target finding. Region cosine@5 compares features from the query and retrieved annotated regions. The proposed method is further demonstrated. External clinician validation is costly and will be our future work.

MethodBox coverage@5BioMedCLIP region cosine@5
RadIR visual28.0%.795
Question-token Sinkhorn30.8%.840

Q2A: We used the simplest context. In fact, the reranker can accept any natural-language instruction. We compared the following two instructions and observe that more complicated instructions can potentially improve the performance.

Direct question — Does this chest X-ray contain a {finding}?

Describe and answer — Please describe the observations of this case relevant to {finding} and answer Does this case have {finding}?

Positive-query cohortHit@1, directHit@1, describe and answernDCG@5, directnDCG@5, describe and answer
MS-CXR test, 94 queries.526.586.499.506

Q3A: The 19.4 s in Table 2 is the submitted Stage 2 mean for a 100-candidate shortlist and includes candidate feature extraction. At the paper's main setting of 50 candidates, Stage 2 took 9.419 s.

We profiled disk-cached versus freshly encoded candidate question tokens on eight MS-CXR queries, one per finding, with image and report inputs and resident models on one RTX PRO 6000 Blackwell Max-Q. The follow-up columns report medians of component sums, including Stage 1, query encoding, candidate loading or encoding, and reranking. Model startup and cache construction are outside these per-query totals.

CandidatesSubmitted Stage 2 mean ± SDFollow-up, fresh candidatesFollow-up, cached candidates
509.419 ± 0.251 s9.161 s0.337 s
10019.408 ± 0.386 s17.974 s0.983 s

The cache is reusable for different cases using the same prompt, and model, which can significantly reduce the time requirement as shown.

Q4A: Figure 3 reports performance on positive-class queries, averaged across the six findings at k=3k=3. We will state the subset and averaging explicitly in the caption. Moreover, we will carefully review all equations.

The submitted Section 4.2 did not report a formal test for its significance claim. We will replace that statement with paired patient-cluster results. We use 10,000 bootstrap draws for pointwise 95% intervals, retaining draws with all six findings, and one million sign flips for two-sided tests with Holm adjustment across the four exploratory comparisons below.

nDCG@5 comparisonMS-CXR difference [95% CI]Adjusted ppOriginal cohort difference [95% CI]Adjusted pp
Question-token Sinkhorn minus visual RadIR+.070 [−.002, +.148].198+.117 [+.085, +.151]< .00001
All-token Sinkhorn minus MaxSim+.047 [−.002, +.097].198+.007 [−.011, +.026].460

Thanks for the author's response and additional experiments. They address most of my concerns, so I am raising my score to 3.

Thank you for taking the time to review our response and the additional experiments. Your feedback helped us clarify when Sinkhorn adds value and how the rankings respond to a change in the diagnostic question. We will incorporate these results and the clarifications on latency, notation, and statistical testing into the revision.

Reviewer_XQL8OpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes a two-stage, training-free framework for medical case retrieval. In Stage 1, a coarse retrieval step is performed using cosine similarity—computed from fixed embeddings generated by a pre-trained medical vision-language encoder—facilitated by FAISS to narrow down a large database to a small set of candidates. In Stage 2, the images and reports of the query patient and the candidate records, along with clinical context-related questions, are fed into a decoder-style medical vision-language model (VLM) (e.g., CheXagent). The final-layer hidden states at the positions of the context tokens are extracted as "context-aware token sets," which are then used for fine-grained re-ranking via the Sinkhorn algorithm (optimal transport).

The contributions include: a two-stage zero-shot retrieval framework; a context-aware token-set similarity metric based on medical VLM token representations; and experimental evaluations conducted on six pathological conditions using the MIMIC-CXR dataset.

Critical strengths

This method offers a concise and effective way to leverage pre-trained medical vision-language models (VLMs). By skipping additional training for "Stage 2" and instead utilizing context-token representations—reordered via optimal transport (OT)—the approach yields significant performance gains on both positive cases and challenging pathological findings (e.g., increasing the Recall@3 for "consolidation" from 0.240 to 0.680). This demonstrates that token-level, context-aware representations capture fine-grained signals often lost in the global embeddings produced during "Stage 1"—a key insight that stands as the paper’s most valuable contribution.

The two-stage design and the accompanying ablation studies offer practical guidance: Stage 1 ensures efficiency, while Stage 2 focuses on fine-grained detail. The ablation results highlight the "context question" and the "visual token filter" as critical components (removing the filter causes a drop of 3.0–4.1 percentage points in Recall@1). These findings validate the "coarse retrieval + lightweight re-ranking" architectural paradigm, rather than simply stacking models.

Critical weaknesses

There is insufficient validation regarding clinical relevance and context. Relevance is defined as a binary match based on "identical pathology labels," a setup where retrieval for negative classes is effectively saturated. Furthermore, the "context" experiments are limited to simple, single-pathology queries—essentially asking, "Does this patient have condition X?"—rather than the "clinician free-text diagnostic context" claimed by the authors. Consequently, the study fails to demonstrate that the method remains effective within the complex, real-world clinical context (encompassing factors such as severity, anatomical location, patient background, and comorbidities). Moreover, the macro-averaged results are artificially inflated due to this saturation in the negative classes. This represents a critical gap in the evidence supporting the paper's core contributions.

The study lacks robust baselines to isolate the specific contribution of Optimal Transport (OT) or token-set similarity. There is no comparison against established re-ranking baselines—such as ColBERT/ColPali-style MaxSim, mean/CLS pooling, or basic token-level similarity—making it impossible to justify the necessity of using Sinkhorn OT. Additionally, the computational efficiency is questionable, with Stage 2 requiring approximately 19.4 seconds per query; this raises doubts regarding its practical clinical utility. Finally, the evaluation is restricted to a single dataset from a single center, lacking confidence intervals, significance testing, and external validation. These limitations undermine the claim that "context-aware token-set similarity" offers a meaningful advantage over simpler re-ranking approaches.

Questions for authors

What is the precise definition of "relevance" used for retrieval? Does it require an exact match of the pathology label, or does it also account for clinical similarities such as severity, anatomical location, and patient background? Could the saturation of negative samples skew the macro-averaged results? Would it be possible to report performance metrics based exclusively on positive samples or on relevance scores provided by clinicians?

Is the "context" in your experiments limited solely to pathology-related queries (e.g., "Does this patient have X?")? Could you include experiments involving free-text or complex, multi-constraint clinical contexts to better substantiate the claim that your approach is "context-aware"?

Could you incorporate additional re-ranking baselines—such as MaxSim, mean/CLS pooling, or simple token-wise cosine similarity—to provide a direct comparison with the gains achieved by Sinkhorn Optimal Transport (OT)? This would help clarify whether the use of token-set OT is a necessary contribution.

Is a latency of approximately 19.4 seconds per query in Stage 2 (with m=50m=50) considered acceptable? Could this latency be reduced through techniques such as caching candidate tokens, batch processing, or using an approximation of the Sinkhorn algorithm? Furthermore, could you provide results based on multiple random seeds, include confidence intervals, and perform validation on an external dataset?

2 comments

We thank the reviewer for the insightful comments and we will add the following discussion to the revision.

Q1A: In the original evaluation, a retrieved case is relevant if its binary label for the queried finding matches the query's label. Therefore, we add more positive-query experiments on MS-CXR which contains bounding boxes for further evaluation. Concretely, we first average over queries within each finding and then equally across six findings from the label agreement perspective. Both query cohorts (the original 482 queries and MS-CXR test queries) search the same 720-image report-available MS-CXR train+validation corpus. CheXagent reranks the visual RadIR top 50 with image and report inputs. We can easily observe the benefit of the proposed method.

Positive-query cohortRadIR Hit@1Sinkhorn Hit@1RadIR nDCG@5Sinkhorn nDCG@5
MS-CXR test, 94 queries.397.526.429.499
Original 482-image cohort, 593 queries.355.499.359.476

Moreover, on the same 94 MS-CXR queries, box coverage@5 is the fraction of the top five cases with a reference box for the target finding. Region cosine@5 compares features from the query and retrieved annotated regions. The proposed method is further demonstrated. External clinician validation is costly and will be our future work.

MethodBox coverage@5BioMedCLIP region cosine@5
RadIR visual28.0%.795
Question-token Sinkhorn30.8%.840

Q2A: Yes, we used the simplest context. In fact, the reranker can accept any natural-language instruction. We compared the following two instructions and observe that more complicated instructions can potentially improve the performance.

Direct question — Does this chest X-ray contain a {finding}?

Describe and answer — Please describe the observations of this case relevant to {finding} and answer Does this case have {finding}?

Positive-query cohortHit@1, directHit@1, describe and answernDCG@5, directnDCG@5, describe and answer
MS-CXR test, 94 queries.526.586.499.506

Q3A: Yes, we added the following comparison. We can observe that Sinkhorn as a distribution to distribution measure would be more beneficial when the involved number of tokens are higher. They are similar with small number of tokens under expectation. Therefore, Sinkhorn is generally more beneficial.

Scorer and token set, image + reportMS-CXR tokens (query / candidate)MS-CXR nDCG@5 (94 positive queries)Original cohort tokens (query / candidate)Original cohort nDCG@5 (593 positive queries)
Visual RadIR, before reranking1 global vector per case.4291 global vector per case.359
Sinkhorn / MaxSim, all tokens1,177–1,435 / 1,146–1,656.434 / .3871,145–1,686 / 1,145–1,656.393 / .386
Sinkhorn / MaxSim, image tokens1,024 / 1,024.449 / .4181,024 / 1,024.387 / .371
Sinkhorn / MaxSim, report tokens88–346 / 59–568.422 / .40857–597 / 59–568.422 / .417
Sinkhorn / MaxSim, question tokens11–14 / 11–14.499 / .49911–14 / 11–14.476 / .476
Mean-pooled cosine, question tokens11–14 / 11–14.46311–14 / 11–14.432

Q4A: The 19.4 s in Table 2 is the submitted Stage 2 mean for a 100-candidate shortlist and includes candidate feature extraction. At the paper's main setting of 50 candidates, Stage 2 took 9.419 s. A morning round discussion for each patient would take about 30 mins. Therefore, either one is acceptable.

We profiled disk-cached versus freshly encoded candidate question tokens on eight MS-CXR queries, one per finding, with image and report inputs and resident models on one RTX PRO 6000 Blackwell Max-Q. The follow-up columns report medians of component sums, including Stage 1, query encoding, candidate loading or encoding, and reranking. Model startup and cache construction are outside these per-query totals.

CandidatesSubmitted Stage 2 mean ± SDFollow-up, fresh candidatesFollow-up, cached candidates
509.419 ± 0.251 s9.161 s0.337 s
10019.408 ± 0.386 s17.974 s0.983 s

The cache is reusable for different cases using the same prompt, and model, which can significantly reduce the time requirement as shown.

We added an evaluation on the official MS-CXR test split. The intervals below come from 10,000 paired patient-cluster bootstrap draws, retaining draws in which all six findings are represented. These are exploratory, pointwise 95% intervals.

Query cohortPositive queries / patientsRadIR nDCG@5Question-token Sinkhorn nDCG@5Paired difference [95% CI]
MS-CXR test94 / 82.429.499+.070 [−.002, +.148]
Original 482-image cohort593 / 158.359.476+.117 [+.085, +.151]

Thanks for your rebuttal. The arguments presented above—whether they concern proxy metrics, non-primary experimental setups, or idealized caching scenarios—fail to address the paper’s core claim. I maintain my original score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.