acceptodds
Withdrawn from ICLR 2027

Ranking-Aware Prompt Optimization for Multimodal Clinical Diagnosis

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Multimodal large language models (MLLMs) are rapidly advancing clinical diagnosis, yet their adaptation pipelines remain anchored to accuracy-based objectives. Clinical data are heavily class-imbalanced: a constant-majority predictor can score above accuracy while being clinically useless. We therefore evaluate and optimize for AUROC, a threshold-free score that ranks positives above negatives and is invariant to class balance. We focus on prompt optimization in MLLMs. Reflective methods such as GEPA use a binary *scores matrix* with one row per evaluation instance and one column per candidate prompt; cells record per-instance correctness, so the column average is accuracy and drives candidate selection. We introduce *pair-level Pareto prompt evolution* (Ranking-PE), which replaces each correctness row with a pairwise-ordering row over (positive, negative) instance pairs: the cell is if the candidate scores the positive higher than the paired negative. The column average then equals empirical AUROC (by the Wilcoxon–Mann–Whitney identity). We apply this swap at all three layers the prompt evolution search reads from—the scores matrix that decides Pareto dominance, the per-example feedback to the reflection LM, and final candidate selection—at no extra model calls and with no surrogate loss. Across three diseases on MIMIC, accuracy-based prompt evolution can degrade ranking; Ranking-PE reverses this, beating the accuracy-based recipe by AUROC pp on fine-tuned Qwen3-VL-8B and pp on MedGemma-4B. Ablations examine each design component and show that a medical-grade visual backbone—via vision-encoder-tuned SFT or medical pretraining—is a prerequisite that prompt search cannot replace—our recipe extends reflective prompt evolution from text-only data to multimodal clinical decision-making.

Primary area:
foundation or frontier models, including LLMs
Keywords:
agents, evolving agents, multimodal, prompt evolution, clinical diagnosis

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_jyXNOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a prompt optimization method, Ranking-PE for multimodal medical tasks. It replaces the accuracy metric with AUC in the prompt optimization method GEPA. By reflective prompt search based on paired scores, Ranking-PE selects the Pareto optimality in candidate prompts. The method is evaluated on the MIMIC dataset with multi-label classification tasks. Ranking-PE can outperform the baselines on AUC and balanced ACC.

Critical strengths
  • The motivation of using AUC instead of ACC for medical tasks makes sense.
Critical weaknesses
  • Limited novelty: the method is basically replacing the evaluation metric in the existing prompt optimization method GEPA, with no further design.
  • Poor evaluation: Ranking-PE is tested on the MIMIC dataset with multimodal multi-label classification tasks. However, the setting is meaningless. Using existing MLLMs for discriminative tasks is meaningless, as their performance is worse than multimodal fusion methods based on discriminative models. In fact, even early works [1] can achieve near 90 AUC on CheXpert, significantly outperforming Ranking-PE.
  • There are 14 labels in the CheXpert dataset, why does the paper just use 3 of them?

[1] Multi-Granularity Cross-modal Alignment for Generalized Medical Visual Representation Learning, NeurIPS 2022

Questions for authors
  • The writing is poor. Many presuppositions are not clearly stated. For example, the 'reflector' in the second paragraph. It is difficult to understand for researchers not familiar with prompt evolution (or simply just not familiar with GEPA). Another example is 'parent selection' in the third paragraph.
  • Using QA&VQA tasks would be better for a prompt optimization method
1 comment
AuthorsOpenReview ↗

Response to Reviewer jyXN

We thank the reviewer for recognizing the motivation for ranking-based evaluation. We clarify the method's contribution, the purpose of the diagnostic experiments, and the broader evaluation below.

W1 - Ranking-PE changes how prompts are compared and revised during search. In accuracy-directed GEPA, prompts are compared through instance-level correctness. Selecting the final prompt by AUROC leaves that search process unchanged.

Pair-level Pareto scoring instead records whether each prompt ranks a positive example above a negative one: 1 for correct ordering, 0.5 for a tie, and 0 for reversed ordering. The full-pair column mean equals empirical AUROC, while individual rows retain information that a scalar AUROC loses. Two prompts with the same AUROC can order different pairs correctly; these rows let search retain prompts with complementary strengths. They reuse validation predictions without additional task-model inference.

Ranking-shaped feedback then uses clinical error types, confidence magnitude, and cross-example rank context to guide prompt revision. Final selection returns the prompt with the highest validation AUROC. The ablation separates these contributions:

Qwen + SFT configurationAverage test AUROC
Accuracy-PE68.0
AUROC final selection only61.3
+ pair-level Pareto, correctness feedback70.7
+ ranking feedback, full method73.8

Changing final selection alone gives 61.3 AUROC; adding pair-level scoring raises it to 70.7, and ranking feedback further improves it to 73.8. With ranking feedback and final selection held fixed, pair-level rows also outperform instance-level rows by 8.0 points (73.8 versus 65.8). The benefit therefore extends beyond changing the final selection metric.

W2 - MLLMs combine medical images, patient-specific clinical context, and task instructions in one model. High AUROC from a dedicated image classifier does not answer how to improve diagnostic ranking in this image–EHR, instruction-based setting. Our experiments test this directly: Ranking-PE improves average AUROC over Accuracy-PE by 5.8 points on Qwen3-VL-8B and 16.2 points on MedGemma-4B, using the same model checkpoints and image–EHR inputs.

MGCA [1] reports its near-90 AUROC on CheXpert after medical image–report pretraining and supervised training of a linear classifier on frozen image features. We evaluate MLLM Yes/No scores on a linked MIMIC-CXR/MIMIC-IV cohort. Even where disease labels overlap, these are different datasets, inputs, and training protocols, so the absolute AUROCs are not directly comparable.

W3 - Atelectasis, Cardiomegaly, and Consolidation are among the five CheXpert competition tasks [2] and are also evaluated in MGCA [1]. They cover different visual findings: lung-field changes, the cardiac silhouette, and pulmonary opacity. Some other findings were not included because very few usable examples remain after excluding uncertain and missing labels.

Our experiments also cover Pneumothorax, Edema, and Pneumonia. On MedGemma, Ranking-PE improves AUROC on all three:

Additional findingAccuracy-PE AUROCRanking-PE AUROCGain, percentage points
Pneumothorax55.8868.10+12.22
Edema69.2974.46+5.17
Pneumonia54.5864.52+9.94

The average gain is 9.11 points, extending our CXR evaluation to six findings. We further evaluate glioma grading on BraTS 2020 brain MRI:

MethodMean test AUROC (%)
Accuracy-PE57.78
Balanced-Accuracy-PE63.26
Ranking-PE69.35

Ranking-PE improves AUROC by 11.57 points over Accuracy-PE and 6.09 points over Balanced-Accuracy-PE on this MRI task.

Questions - The reflector reads evaluated examples and feedback and proposes revised prompt instructions. Parent selection chooses the candidate prompt to revise next. The pair-level matrix supplies the objectives for Pareto-based search. We will define these terms at first use and clarify their roles in the optimization loop.

QA/VQA is a valuable application of prompt optimization. Binary diagnosis lets us directly compare decision accuracy with the ordering of continuous diagnostic scores. A prompt can improve one without improving the other. Our experiments test whether guiding search with pairwise ordering improves AUROC under the same model and data conditions.

[1] Multi-Granularity Cross-modal Alignment for Generalized Medical Visual Representation Learning, NeurIPS 2022

[2] CheXpert: A Large Chest Radiograph Dataset with Uncertainty Labels and Expert Comparison, AAAI 2019

Reviewer_vGgLOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper proposes Ranking-PE, an extension of GEPA for multimodal clinical diagnosis that combines pair-level Pareto scoring, AUROC-based prompt selection, and ranking-aware feedback. It evaluates the approach on three disease classification tasks using linked MIMIC-CXR and MIMIC-IV data, primarily with Qwen3-VL-8B and MedGemma-4B. The authors report improvements over accuracy-oriented prompt optimization and provide ablations examining the method’s components, SFT configurations, and reflector choices.

Critical strengths
  1. The paper addresses a relevant mismatch between accuracy-oriented prompt optimization and ranking-based clinical evaluation. Adapting the search to positive–negative pair ordering is a concrete methodological idea that could be useful for diagnostic tasks evaluated by AUROC.
Critical weaknesses
  1. The experimental controls do not sufficiently isolate the source of improvement. The method extends GEPA with pair-level scoring, AUROC-based selection, and ranking-aware feedback. Although the results and ablations support the combined approach, the Scalar-AUROC baseline receives correctness-only feedback, whereas the full method receives richer feedback. This limits attribution of the gains to the pair-level search mechanism. Claims concerning visual foundations and calibration also lack direct validation, and parts of the qualitative analysis appear inconsistent with the quoted prompts.

  2. Dataset size and statistical reliability are incompletely reported. The appendix reports means and standard deviations for several experiments, but does not clearly specify the number of runs, random seeds, or whether SFT was repeated. Split sizes and class counts are also missing, alongside test-set confidence intervals. These omissions are particularly consequential for Atelectasis, where negative examples constitute only 3.4% of the test set, making the stability of the reported gains difficult to assess.

Questions for authors
  1. GEPA already supports custom evaluation metrics. Could you clarify the key methodological advance over GEPA and distinguish the contributions of using AUROC, changing the unit of Pareto scoring, and introducing ranking-aware feedback?

  2. Ranking-PE also improves average performance without SFT. What evidence establishes a strong visual foundation as necessary, rather than beneficial? Additionally, are there direct calibration measurements supporting the paper’s calibration-related claims?

  3. How many runs underlie the reported means and standard deviations? Was SFT retrained for each run, or was only prompt optimization repeated? Which sources of randomness varied across runs?

  4. Please provide the patient counts and positive/negative class counts for each disease in the training, validation, and test splits, together with confidence intervals for the main performance differences, particularly for Atelectasis.

  5. Figure 4 describes Accuracy-PE as permitting “No” on unclear images, whereas the quoted prompt requires clear visualization and confirmation of normality before answering “No.” Could you clarify this interpretation and provide supporting evidence from error patterns or prediction scores?

Additionally, the presentation would benefit from a more consistent ordering of the results and table references (such as line 407 and 412), with the main comparisons and component ablations grouped together. Shortening long sentences and reducing the frequent use of em dashes would also improve readability.

1 comment
AuthorsOpenReview ↗

Response to Reviewer vGgL

We thank the reviewer for recognizing the relevance of metric alignment and the concrete pair-ordering formulation. We clarify the methodological contribution, the experimental controls, and the interpretation of the visual and qualitative results.

W1 / Q1 - GEPA supports custom metrics, our contribution is to change the units of Pareto comparison. A scalar AUROC records overall ranking quality but discards which positive–negative pairs a candidate orders correctly. Pair-level rows preserve this information, so prompts with complementary ranking behavior remain distinguishable during search. The full-pair column mean equals empirical AUROC with half credit for ties, reflective search consumes these discrete events without a differentiable surrogate or additional task-model calls for constructing the pairs.

The following comparison holds both ranking feedback and AUROC final selection fixed:

Pareto representationAtelectasisCardiomegalyConsolidationAverage AUROC
Instance rows56.068.273.365.8
Pair rows, Ranking-PE70.771.379.373.8

Pair-level scoring improves AUROC on every disease, by 8.0 points on average, with the same ranking feedback and final selection. The pair matrix preserves which ordering events each prompt handles well, allowing search to retain prompts with complementary ranking behavior. This information is lost when all events are reduced to one scalar AUROC, even though that scalar is suitable for selecting the final prompt.

The simpler objective baselines provide additional context:

Search objectiveQwen + SFT average AUROCMedGemma average AUROC
Accuracy-PE68.053.5
Balanced-accuracy selection67.659.0
Class-weighted PE68.055.7
Scalar-AUROC PE69.657.6
Ranking-PE73.869.7

Ranking-PE achieves the highest average AUROC on both backbones, outperforming balanced-accuracy selection, class-weighted accuracy, and scalar-AUROC selection.

W1 / Q2 - Under Ranking-PE, Qwen reaches 68.4 average AUROC without SFT, 70.3 with frozen-encoder SFT, and 73.8 with tuned-encoder SFT. Tuning the vision encoder improves every disease over freezing it, adding 3.5 points on average. The visual-adaptation comparison thus measures a contribution beyond language-side SFT and prompt optimization. We agree that “beneficial” describes this evidence more precisely than “prerequisite,” and will revise that wording while retaining the measured advantage of visual adaptation.

The prompt examples show how search changes the instructions for handling confidence and ambiguous evidence. For example, some instructions caution the model against overconfidence or ask it to account for uncertainty when visual findings are unclear. These examples help explain how the prompts evolve. In the experiments, we use AUROC to measure whether the resulting prompts improve the ordering of diagnostic scores across patients. We will revise the wording to distinguish these confidence-related instructions from the ranking improvements measured in our experiments.

W2 / Q3–Q4 - We repeated prompt optimization three times with different random seeds, keeping the SFT checkpoint fixed. The table below reports mean AUROC and 95% Student-t confidence intervals across runs for Qwen3-VL-8B with vision-encoder-tuned SFT.

DiseaseAccuracy-PE mean [95% CI]Ranking-PE mean [95% CI]
Atelectasis63.60 [62.31, 64.89]70.70 [68.33, 73.07]
Cardiomegaly68.07 [64.92, 71.21]71.30 [68.85, 73.75]
Consolidation72.43 [71.92, 72.95]79.27 [76.20, 82.33]

The paired DeLong test gives p = 0.025 for Ranking-PE versus Accuracy-PE.

After excluding uncertain and missing labels, the training/validation/test sets contain 901/230/146 examples for Atelectasis, 4,232/349/194 for Cardiomegaly, and 1,071/82/51 for Consolidation.

There is zero patient overlap between training, validation, and test. We retain the Symile-MIMIC patient-disjoint splits and apply disease-specific label filtering within each split. Both stages use the same validation split for checkpoint or prompt selection. Stage 2 caps the seed-shuffled training pool at 500 examples; validation and test are used in full. Test is reserved for final evaluation.

W1 / Q5 - Thank you for catching this annotation error: the quoted instruction requires clear visualization and confirmation of normality before answering “No.” We will correct the figure annotation and the accompanying description; the numerical results are unchanged.

The prompt examples show how search changes the instructions, and the ablation experiments measure the contribution of each component. We will make these roles explicit, define “reflector” and “parent selection” when they first appear, and organize the ablation results around the components being compared.

Reviewer_yX9bOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper addresses metric misalignment in adapting MLLMs to clinical diagnosis: under class imbalance, accuracy misleads, whereas screening and triage require ranking quality (AUROC). Its key insight is that reflective prompt evolution optimizes whatever event fills the rows of its scores matrix; replacing instance correctness with positive-negative ordering makes the column average exactly equal to empirical AUROC (via the Wilcoxon–Mann–Whitney identity), redirecting the entire search toward ranking with no surrogate loss or extra rollouts. Built on vision-encoder-tuned SFT, this Ranking-PE outperforms accuracy-based evolution by +5.8 to +16.2 AUROC points on three MIMIC diseases across two open-source MLLMs, with ablations confirming the pair-level matrix as the main gain source.

Critical strengths

The paper’s core insight—that the row event of the scores matrix determines what the column mean and Pareto dominance actually optimize, and that replacing instance correctness with positive–negative ordering makes the column mean equal empirical AUROC under full pair enumeration and the stated tie convention (via the Wilcoxon–Mann–Whitney identity)—is mathematically clean, requires no surrogate loss and no additional task-model rollouts, and yields a reusable design principle (“choose the row event”) that transfers beyond this setup.

The cumulative ablation (Table 3) shows that changing only final selection from accuracy to AUROC degrades AUROC (68.0% → 61.3%), while the pair-level matrix is what reverses this—demonstrating the gain comes from redirecting the search trajectory, not relabeling the metric. The headline gains (+5.8 to +16.2 AUROC points over accuracy-based evolution) are correspondingly well-supported, and the incidental finding that accuracy-based evolution erodes the base model's ranking (MedGemma 59.3% → 53.5%) gives the metric-misalignment warning rare empirical teeth.

Critical weaknesses

Ranking-PE confounds objective alignment with richer feedback. In Table 3, AUROC-only selection drops average AUROC from 68.0 to 61.3; the pair-level matrix then reaches 70.7 (+2.7 vs. Accuracy-PE), and ranking feedback adds +3.1, to 73.8. On Atelectasis the matrix only restores 43.7 to 63.8 (+0.2 vs. Accuracy-PE); feedback supplies the reported +7.1. Removing the matrix while keeping ranking feedback yields 65.8, and that row still selects by AUROC. The missing control is Accuracy-PE with the same three signals. Table 12 (Consolidation, cumulative) moves AUROC from 79.8 to 79.3 when rank context is added, inside the ±1.2 SD (Table 6), while balanced accuracy rises from 63.8 to 68.5. These results do not isolate the three-layer claim in §4.2.3.

The task is framed as multimodal diagnosis, but no image-only, EHR/lab-only, or image-removed ablation is reported. CheXpert labels are extracted from reports (Appendix A.4), and the Ranking-PE Cardiomegaly prompt tells the model to use lab context to support or refute the finding (Figure 5). Vision-encoder tuning and the MedGemma–Gemma3 gap concern the backbone. Without these controls for the base, Accuracy-PE, and Ranking-PE, dependence on the radiograph rather than EHR/lab correlates remains unchecked.

Questions for authors

Could you add Accuracy-PE with the same three feedback signals, and extend the Table 12 cumulative ablation to all three diseases? The 65.8 row in Table 3 keeps ranking feedback but still selects by AUROC. If that cell approaches Ranking-PE, please narrow §4.2.1; if full feedback does not beat the pre-rank-context row beyond Consolidation, please narrow §4.2.2.

The paper claims multimodal clinical diagnosis, but no image-only, EHR/lab-only, or image-removed ablation is reported. Could you report these for base and Ranking-PE to verify that gains come from visual evidence rather than EHR/lab shortcuts?

Are Stage 1 checkpoint selection and Stage 2 validation/test strictly patient-disjoint? What are the test size and absolute negative count per disease, especially Atelectasis (96.6% test prevalence), and how is the test set kept out of prompt selection?

How many runs per configuration, and are headline AUROC differences supported by confidence intervals or paired tests? Given MedGemma’s ±3.1–4.5 SD, how should the +16.2-point gain be interpreted? Also, why is Table 4 non-monotonic across pair caps (300 pairs higher than 1,000)?

AUROC is symmetric and threshold-free, but deployment needs a threshold and the paper stresses that false negatives are clinically dangerous. Could you report sensitivity at fixed specificity or a cost-weighted utility? Does Ranking-PE’s gain persist at clinically relevant operating points?

2 comments
AuthorsOpenReview ↗

Response to Reviewer yX9b (Part 1/2)

We thank the reviewer for recognizing the mathematical clarity of the pair-level formulation, the computational efficiency, and the evidence that redirecting search matters beyond changing final selection. We address the component, modality, and statistical questions below.

W1 / Q1 - To clarify the contribution of pair-level scoring, we compare instance-level and pair-level representations while holding feedback and final selection fixed. All four configurations use AUROC-based final selection; entries are average test AUROC (%) across the three diseases.

Pareto representationCorrectness-only feedbackRanking-shaped feedback
Instance-level61.365.8
Pair-level70.773.8

Pair-level scoring improves average test AUROC by 9.4 points with correctness-only feedback and 8.0 points with ranking-shaped feedback. In both comparisons, the feedback and final-selection rule are unchanged; the difference is whether Pareto search compares prompts through individual correctness events or positive–negative ordering events. This shows that the matrix contributes under both feedback settings, rather than the gain arising solely from richer feedback.

The cumulative feedback ablation now covers all three diseases. Full feedback improves average AUROC from 70.7 to 73.8, a 3.1-point gain over correctness-only feedback. Adding cross-example rank context to clinical-error and confidence feedback raises the average from 71.2 to 73.8. This context lets the reflector consider how examples are ordered relative to one another, in addition to whether each prediction is correct. The results show an average benefit from complete ranking feedback across the three tasks.

Feedback, added cumulativelyAverage AUROC (%)
Correctness only70.7
+ clinical error type70.8
+ confidence magnitude71.2
+ cross-example rank context (full)73.8

W2 / Q2 - The optimizer comparison uses identical image–EHR inputs and the same task-model checkpoint. The visual-adaptation comparison provides complementary evidence about the importance of adapting the image encoder:

Qwen3-VL-8B + Ranking-PEAtelectasis AUROCCardiomegaly AUROCConsolidation AUROCAverage
No SFT63.565.876.068.4
SFT, vision encoder frozen65.967.377.670.3
SFT, vision encoder tuned70.771.379.373.8

This comparison addresses the contribution of the visual branch while retaining the same EHR information. With EHR inputs, the language-side SFT configuration, and prompt optimization held fixed, tuning the vision encoder improves AUROC on all three diseases, by 3.5 points on average over frozen-encoder SFT. The gain therefore comes from allowing the visual encoder to adapt within the joint image–EHR system. Together, the two comparisons show that the optimizer gains are obtained under matched multimodal inputs, and that adapting the visual branch further improves the ranking signal used by prompt search.

Q3 - We use patient-disjoint training, validation, and test splits, with zero patient overlap between any two splits. Stage 1 selects the SFT checkpoint on validation, and Stage 2 uses the same validation split to evaluate and select prompts. Test data are excluded from training, checkpoint selection, prompt feedback, and prompt selection, and are used only to evaluate the baseline and selected prompt.

The test sets contain 146, 194, and 51 examples for Atelectasis, Cardiomegaly, and Consolidation, respectively. Each task retains explicit positive and negative labels, excluding uncertain and missing labels.

AuthorsOpenReview ↗

Response to Reviewer yX9b (Part 2/2)

Q4 - The main PE results summarize three runs, with per-disease SDs reported in the appendix. We additionally provide 95% Student-t confidence intervals for the across-run mean on the Qwen pipeline.

DiseaseAccuracy-PE mean AUROC95% CI across runsRanking-PE mean AUROC95% CI across runs
Atelectasis63.60[62.31, 64.89]70.70[68.33, 73.07]
Cardiomegaly68.07[64.92, 71.21]71.30[68.85, 73.75]
Consolidation72.43[71.92, 72.95]79.27[76.20, 82.33]

A paired DeLong test supports the AUROC improvement of Ranking-PE over Accuracy-PE (p = 0.025).

For MedGemma, ±3.1–4.5 denotes the run-to-run SD for individual diseases, while +16.2 points is the average AUROC gain over Accuracy-PE. The gains are +22.7, +9.6, and +16.4 points for Atelectasis, Cardiomegaly, and Consolidation, respectively, with Ranking-PE SDs of 3.1, 4.5, and 3.8 points. The improvement is therefore present in all three disease-level means, alongside the reported run-to-run variability.

As discussed in the robustness analysis, a larger pair cap does not guarantee better test performance after adaptive prompt search. Across the tested caps, Cardiomegaly AUROC varies by only 0.7 percentage points, and every configuration exceeds Accuracy-PE on both AUROC and balanced accuracy.

Q5 - Alongside AUROC, we report balanced accuracy at the model’s default Yes/No decision:

BackboneAccuracy-PE AUROC / BAccRanking-PE AUROC / BAcc
Qwen3-VL-8B + SFT68.0 / 55.173.8 / 68.6
MedGemma-4B53.5 / 49.869.7 / 60.6

The balanced-accuracy gains are 13.5 points on Qwen and 10.8 points on MedGemma. Balanced accuracy weights sensitivity and specificity equally, so it is informative here despite the unequal class frequencies. Ranking-PE improves both AUROC and balanced accuracy on both backbones, with balanced accuracy evaluated using the same default Yes/No decision rule for both methods.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.