- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies how noisy demonstration labels and demographic information affect medical in LLM prediction. The authors analysed two medical imaging datasets using GPT and Qwen models. They then proposed FANDS, which trains a surrogate model to predict how different demonstration sets will affect classification performance. The method combines image similarity, demographic information, estimated label noise, and demonstration order. The experiments report improvements in F1 and accuracy.
Critical strengths
The controlled diagnostic experiments are useful. The authors change demonstration labels while keeping the query, image content, and prompt position fixed. This helps isolate how incorrect demonstrations affect predictions.
The paper evaluates 2 medical datasets and 2 LLM families. It also includes component ablations and experiments with increased label noise.
Critical weaknesses
Overall, it was felt that demonstration selection and fairness in in-context learning are well-studied areas. Label perturbation has also been explored in many previous works. However, the current related work section does not clarify the fundamental differences between this work and previous ones. Also our biggest concern is that the number of datasets and models evaluated in the experiments is very limited (two each) and we would recommend adding more experiments and further revisions.
The detailed comments are as follows.
[Major] Limited number of datasets and models. Currently the paper only considers two datasets and two LLMs. This is insufficient to support the generalizability of the proposed method FANDS. The experiments with three datasets and four models would be ideal.
[Major] The other main concern is how to determine if a perturbed label is actually incorrect rather than representing a counterfactual example. At line 1188, the method treats disagreement between nearby examples as evidence of label noise. However, nearby patients can have different diagnoses (which is actually a common topic discussed in counterfactual studies). This score may reflect clinical variation rather than incorrect labels. Additional experiments and clarification are needed to validate the noise estimate as a measure of actual label errors.
[Minors] More clarification on the baselines is needed. For example, Full-pool relevance-risk retrieval and Same-group relevance-risk retrieval are associated with multiple references. Which specific paper's implementation did the authors reproduce? Or did the authors follow the general ideas and implement the algorithms themselves?
[Minors] Inconsistent experimental settings. The proposed FANDS method can use 1~3 demonstrations, whereas all baselines use three demonstrations. Although intuitively, using more shots may lead to better performance, allowing a dynamic number of demonstrations gives FANDS more flexibility. It would be helpful to present an additional experiment comparing FANDS with a fixed number of three demonstrations, consistent with the baseline settings (i.e., an equal search space).
[Small Typo] Inconsistent notation. From lines 1543 to 1546, the random perturbation variable is introduced as in line 1543 and reused in line 1546, but the equation in line 1545 uses , which is not defined.
Questions for authors
[For major concern2] Can the authors evaluate the noise estimator using independently reviewed labels and report its precision and recall separately for each demographic group?
Can the authors report paired confidence intervals for the difference between FANDS and the strongest baseline in each setting, including worst group F1?
Can the authors report changes in demographic false positive and false negative rate gaps, rather than relying only on worst group F1?