- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper focuses on context-aware medical case retrieval. The method itself is a standard two-stage pipeline. Stage1 retrieval candidates via FAISS; Stage2 feeds context and images into CheXagent, takes hidden stages at the context-question token positions, and reranks candidates. Experiments run on 482 queries across 6 pathologies, with extensive ablations. The authors report consistent improvements over stage 1 baselines and honestly note regressions on a few pathologies.
Critical strengths
- Well motivated with zero-shot settings, matching real deployment where no labeled data exists per contexts.
- Detailed in-domain evaluation. The evaluation, ablations, latency msesurements and negative results are detailed and efficient.
Critical weaknesses
- The paper claims "similar cases for treatment planning", but relevance is just same pathology labeling. Other relevance signals such as severity are ignored.
- MaxSim and mean-pooled cosine are never compared with the core design of the method. This makes the core claim untested, weakening the contribution of the paper.
- How does a zero-shot label-matching baseline ( retrieve by predicted-label agreement ) perform?
Questions for authors
See Weaknesses.
We thank the reviewer for the insightful comments. We will add the following comparisons to the revision.
Q1A & Q2A: In the original evaluation, a retrieved case is relevant if its binary label for the queried finding matches the query's label. Therefore, we add more experiments on MS-CXR which contains bounding boxes for additional evaluation. Concretely, we first average over queries within each finding and then equally across six findings from the label agreement perspective. Both query cohorts (the original 482 queries and MS-CXR test queries) search the same 720-image report-available MS-CXR train+validation corpus. CheXagent reranks the visual RadIR top 50 with image and report inputs. we added the following comparison.
We can observe that Sinkhorn as a distribution to distribution measure would be more beneficial when the involved number of tokens are higher. They are similar with small number of tokens under expectation. Therefore, Sinkhorn is generally more beneficial.
Moreover, on the same 94 MS-CXR queries, box coverage@5 is the fraction of the top five cases with a reference box for the target finding. Region cosine@5 compares features from the query and retrieved annotated regions. The proposed method is further demonstrated. External clinician validation is costly and will be our future work.
Q3A: Hard zero-shot predicted-label agreement achieved .415 nDCG@5 and 26.2% reference-box coverage@5 on MS-CXR. The corresponding results were .402 and 25.1% for visual RadIR and .466 and 33.4% for question-token Sinkhorn.
All methods use image-only case inputs and the same visual RadIR top 50. The corpus is the 892-image MS-CXR train+validation split. We evaluate 170 positive test image–finding queries from 128 patients and average equally over findings. Box coverage@5 is the fraction of the top five cases with a reference box for the queried finding. This further demonstrated the proposed method.