- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes a prompt optimization method, Ranking-PE for multimodal medical tasks. It replaces the accuracy metric with AUC in the prompt optimization method GEPA. By reflective prompt search based on paired scores, Ranking-PE selects the Pareto optimality in candidate prompts. The method is evaluated on the MIMIC dataset with multi-label classification tasks. Ranking-PE can outperform the baselines on AUC and balanced ACC.
Critical strengths
- The motivation of using AUC instead of ACC for medical tasks makes sense.
Critical weaknesses
- Limited novelty: the method is basically replacing the evaluation metric in the existing prompt optimization method GEPA, with no further design.
- Poor evaluation: Ranking-PE is tested on the MIMIC dataset with multimodal multi-label classification tasks. However, the setting is meaningless. Using existing MLLMs for discriminative tasks is meaningless, as their performance is worse than multimodal fusion methods based on discriminative models. In fact, even early works [1] can achieve near 90 AUC on CheXpert, significantly outperforming Ranking-PE.
- There are 14 labels in the CheXpert dataset, why does the paper just use 3 of them?
[1] Multi-Granularity Cross-modal Alignment for Generalized Medical Visual Representation Learning, NeurIPS 2022
Questions for authors
- The writing is poor. Many presuppositions are not clearly stated. For example, the 'reflector' in the second paragraph. It is difficult to understand for researchers not familiar with prompt evolution (or simply just not familiar with GEPA). Another example is 'parent selection' in the third paragraph.
- Using QA&VQA tasks would be better for a prompt optimization method
Response to Reviewer jyXN
We thank the reviewer for recognizing the motivation for ranking-based evaluation. We clarify the method's contribution, the purpose of the diagnostic experiments, and the broader evaluation below.
W1 - Ranking-PE changes how prompts are compared and revised during search. In accuracy-directed GEPA, prompts are compared through instance-level correctness. Selecting the final prompt by AUROC leaves that search process unchanged.
Pair-level Pareto scoring instead records whether each prompt ranks a positive example above a negative one: 1 for correct ordering, 0.5 for a tie, and 0 for reversed ordering. The full-pair column mean equals empirical AUROC, while individual rows retain information that a scalar AUROC loses. Two prompts with the same AUROC can order different pairs correctly; these rows let search retain prompts with complementary strengths. They reuse validation predictions without additional task-model inference.
Ranking-shaped feedback then uses clinical error types, confidence magnitude, and cross-example rank context to guide prompt revision. Final selection returns the prompt with the highest validation AUROC. The ablation separates these contributions:
Changing final selection alone gives 61.3 AUROC; adding pair-level scoring raises it to 70.7, and ranking feedback further improves it to 73.8. With ranking feedback and final selection held fixed, pair-level rows also outperform instance-level rows by 8.0 points (73.8 versus 65.8). The benefit therefore extends beyond changing the final selection metric.
W2 - MLLMs combine medical images, patient-specific clinical context, and task instructions in one model. High AUROC from a dedicated image classifier does not answer how to improve diagnostic ranking in this image–EHR, instruction-based setting. Our experiments test this directly: Ranking-PE improves average AUROC over Accuracy-PE by 5.8 points on Qwen3-VL-8B and 16.2 points on MedGemma-4B, using the same model checkpoints and image–EHR inputs.
MGCA [1] reports its near-90 AUROC on CheXpert after medical image–report pretraining and supervised training of a linear classifier on frozen image features. We evaluate MLLM Yes/No scores on a linked MIMIC-CXR/MIMIC-IV cohort. Even where disease labels overlap, these are different datasets, inputs, and training protocols, so the absolute AUROCs are not directly comparable.
W3 - Atelectasis, Cardiomegaly, and Consolidation are among the five CheXpert competition tasks [2] and are also evaluated in MGCA [1]. They cover different visual findings: lung-field changes, the cardiac silhouette, and pulmonary opacity. Some other findings were not included because very few usable examples remain after excluding uncertain and missing labels.
Our experiments also cover Pneumothorax, Edema, and Pneumonia. On MedGemma, Ranking-PE improves AUROC on all three:
The average gain is 9.11 points, extending our CXR evaluation to six findings. We further evaluate glioma grading on BraTS 2020 brain MRI:
Ranking-PE improves AUROC by 11.57 points over Accuracy-PE and 6.09 points over Balanced-Accuracy-PE on this MRI task.
Questions - The reflector reads evaluated examples and feedback and proposes revised prompt instructions. Parent selection chooses the candidate prompt to revise next. The pair-level matrix supplies the objectives for Pareto-based search. We will define these terms at first use and clarify their roles in the optimization loop.
QA/VQA is a valuable application of prompt optimization. Binary diagnosis lets us directly compare decision accuracy with the ordering of continuous diagnostic scores. A prompt can improve one without improving the other. Our experiments test whether guiding search with pairwise ordering improves AUROC under the same model and data conditions.
[1] Multi-Granularity Cross-modal Alignment for Generalized Medical Visual Representation Learning, NeurIPS 2022
[2] CheXpert: A Large Chest Radiograph Dataset with Uncertainty Labels and Expert Comparison, AAAI 2019