acceptodds
Withdrawn from ICLR 2027

Noise-Aware Demonstration Selection for Fairness-Sensitive In-Context Prediction

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Few-shot prediction with large language models is often treated as a retrieval problem where the goal is to choose relevant demonstrations. This overlooks a key failure mode in fairness-sensitive settings because same-group, different-group, and desensitized demonstrations can all carry noisy labels. We study the relationship between label noise, group identity, and demonstration selection for in-context prediction, with sex and race as sensitive attributes and prompt-only access to black-box large language models. Regression analyses across two prediction tasks quantify how demonstration label noise interacts with group concordance, desensitization, prompt length, and query-group identity. A central finding is that sensitive-attribute concordance does not protect a prompt from label noise (i.e., same-group demonstrations with noisy labels are still associated with reduced target-label correctness). Motivated by these findings, we introduce Fairness- and Noise-Aware Demonstration Selection (FANDS), which combines estimated noise risk with group relation, similarity, and demonstration-order information to improve few-shot prediction. At selection time, a surrogate model estimates the expected performance of candidate demonstration sets without requiring internal large language model scores or repeated exhaustive prompting. Held-out evaluations show that FANDS improves overall predictive performance and fairness metrics relative to baselines. These results show that demonstration label noise is an important factor in fairness-sensitive in-context prediction and that accounting for both noise and group context can improve few-shot predictive performance and fairness.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
In-context learning, Fairness, Demonstration selection, Label noise, Large language models

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_ucrUOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies how noisy demonstration labels and demographic information affect medical in LLM prediction. The authors analysed two medical imaging datasets using GPT and Qwen models. They then proposed FANDS, which trains a surrogate model to predict how different demonstration sets will affect classification performance. The method combines image similarity, demographic information, estimated label noise, and demonstration order. The experiments report improvements in F1 and accuracy.

Critical strengths

The controlled diagnostic experiments are useful. The authors change demonstration labels while keeping the query, image content, and prompt position fixed. This helps isolate how incorrect demonstrations affect predictions.

The paper evaluates 2 medical datasets and 2 LLM families. It also includes component ablations and experiments with increased label noise.

Critical weaknesses

Overall, it was felt that demonstration selection and fairness in in-context learning are well-studied areas. Label perturbation has also been explored in many previous works. However, the current related work section does not clarify the fundamental differences between this work and previous ones. Also our biggest concern is that the number of datasets and models evaluated in the experiments is very limited (two each) and we would recommend adding more experiments and further revisions.

The detailed comments are as follows.

[Major] Limited number of datasets and models. Currently the paper only considers two datasets and two LLMs. This is insufficient to support the generalizability of the proposed method FANDS. The experiments with three datasets and four models would be ideal.

[Major] The other main concern is how to determine if a perturbed label is actually incorrect rather than representing a counterfactual example. At line 1188, the method treats disagreement between nearby examples as evidence of label noise. However, nearby patients can have different diagnoses (which is actually a common topic discussed in counterfactual studies). This score may reflect clinical variation rather than incorrect labels. Additional experiments and clarification are needed to validate the noise estimate as a measure of actual label errors.

[Minors] More clarification on the baselines is needed. For example, Full-pool relevance-risk retrieval and Same-group relevance-risk retrieval are associated with multiple references. Which specific paper's implementation did the authors reproduce? Or did the authors follow the general ideas and implement the algorithms themselves?

[Minors] Inconsistent experimental settings. The proposed FANDS method can use 1~3 demonstrations, whereas all baselines use three demonstrations. Although intuitively, using more shots may lead to better performance, allowing a dynamic number of demonstrations gives FANDS more flexibility. It would be helpful to present an additional experiment comparing FANDS with a fixed number of three demonstrations, consistent with the baseline settings (i.e., an equal search space).

[Small Typo] Inconsistent notation. From lines 1543 to 1546, the random perturbation variable is introduced as ξi\xi_i in line 1543 and reused in line 1546, but the equation in line 1545 uses uiu_i, which is not defined.

Questions for authors

[For major concern2] Can the authors evaluate the noise estimator using independently reviewed labels and report its precision and recall separately for each demographic group?

Can the authors report paired confidence intervals for the difference between FANDS and the strongest baseline in each setting, including worst group F1?

Can the authors report changes in demographic false positive and false negative rate gaps, rather than relying only on worst group F1?

Reviewer_Zu6EOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies how to choose in-context demonstrations for medical image classification with black-box models (GPT-5, Qwen3-VL) when demonstration labels may be wrong, and demonstrations carry a sex or race cue. Mixed-effects analyses of controlled label flips (CheXpert and Harvard-FairVLMed, 8 fits) show that incorrect labels hurt in concordant, discordant, and desensitized conditions, with incorrect concordant demonstrations having the lowest main-effect odds ratios. FANDS then trains an MLP surrogate on 70,000 logged LLM outcomes per setting to predict sensitivity and specificity for each of 41,750 candidate prompts, and sends the top-scoring prompt. On 100 held-out subjects per setting, it reports the best overall F1.

Critical strengths
  1. The paper presents a relevant and important problem with a sound diagnostic design.

  2. A strict information boundary is deployed at selection time with no held-out labels, responses or logits and subject-disjoint candidate pools shared by all methods.

  3. The reporting is detailed and insightful.

Critical weaknesses
  1. The evaluation is limited to medical datasets.

  2. The "incorrect concordant is most harmful" finding holds only for privileged queries. Incorrect concordant is the least harmful in two fits and tied in a third. Contribution 1 still holds, but this claim needs qualifying, since the fairness motivation concerns disadvantaged groups.

  3. The fairness claims are thin. The objective has no group term (CheXpert utility weights are (0,1,0,0), Table 7), and the only fairness evidence is per-group and worst-group F1 (50 subjects per group in CheXpert). In FairVLMed race with Qwen, FANDS matches full-pool retrieval on Black queries (.63 vs .62) but drops on White queries (.63 vs .75), which is leveling down.

  4. The evaluation is underpowered, and some claims overstate it. With 100 test subjects, the F1 difference to the best baseline is within FANDS's own 95% margin in 7 of 8 settings, and the paired intervals described in Appendix I are not reported. Ablations are single-seed and GPT-only.

  5. Missing baselines and an unequal search space: There is no zero-shot, random, or kNN/linear-probe baseline on the same embeddings.

Questions for authors
  1. Can you report, per fit, the unprivileged-query contrasts (main effect times interaction) with CIs, including incorrect concordant vs. incorrect discordant?

  2. Can you report paired bootstrap CIs for FANDS minus the best baseline, ideally on a larger test set?

  3. What are the zero-shot, random 3-shot, kNN-vote, and FANDS-restricted-to-3-cued results, and how often does FANDS pick 1 or 2 demonstrations or desensitized prompts?

  4. How were CheXpert uncertain labels handled, and are FANDS and the baselines retrained on the noise-injected pools in Section 7.3?

  5. Section 6 points to Appendix G for the prompts, but none are given. Please add the templates, state whether the query's own sex or race is shown, and say whether code will be released.

Reviewer_Zj3GOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper investigates the impact of demonstration label noise on fairness-sensitive in-context learning (ICL), focusing on medical prediction tasks involving sensitive attributes such as sex and race. Through regression analyses, the authors find that noisy demonstrations can negatively affect prediction performance, even when demonstrations belong to the same demographic group or sensitive attributes are removed. Based on these observations, the authors propose Fairness- and Noise-Aware Demonstration Selection (FANDS), which trains a surrogate model to estimate the utility of candidate demonstration sets using label noise risk, demographic relations, similarity, and demonstration order. Experiments on two medical imaging datasets show improvements in overall predictive performance and worst-group F1 compared with several demonstration selection baselines.

Critical strengths
  1. Interesting research direction. The paper explores an interesting research direction by investigating label noise in fairness-sensitive in-context learning.

  2. Clear presentation. The paper is well-written, easy to follow, and clearly presents the motivation and proposed method.

  3. Extensive experiments. The authors conduct a relatively large number of experiments, including baseline comparisons, ablation studies, and robustness analyses.

Critical weaknesses
  1. Limited datasets and generalizability. The evaluation is restricted to two medical imaging datasets, which is insufficient to support the broad claims of fairness-sensitive in-context prediction. Common fairness-sensitive NLP tasks, such as toxicity detection, are not explored. Moreover, the experiments focus on binary demographic group comparisons without investigating more diverse or intersectional groups. Therefore, I suggest narrowing the scope of the paper to fairness-sensitive in-context prediction in medical imaging.

  2. Insufficient evidence for fairness-specific effects of label noise. The preliminary analysis does not adequately distinguish the general negative effects of label noise from its interactions with sensitive attributes. For example, the authors observe that removing explicit sensitive attributes does not eliminate the performance degradation caused by noisy demonstrations. However, this may simply indicate that incorrect demonstration labels harm prediction regardless of demographic information. To establish a fairness-specific effect, the authors should examine whether label noise causes significantly different performance degradation across demographic groups and whether removing sensitive attributes changes this effect. Without such analysis, the motivation for fairness-specific noise modeling remains insufficiently supported.

  3. Inadequate fairness evaluation. Although the authors report subgroup F1 and worst-group F1, these metrics alone cannot establish that demographic disparities have been reduced. Improvements in worst-group performance may simply result from better overall prediction accuracy. The paper lacks a systematic analysis of performance gaps between demographic groups and how these gaps change under different noise conditions. Furthermore, since the proposed method explicitly incorporates sensitive-attribute relations, it would be valuable to investigate whether changing only the sensitive attribute of a query (e.g., male to female) affects demonstration selection and subsequent predictions, where such interventions are clinically meaningful.

  4. Limited methodological novelty. FANDS essentially follows a conventional surrogate-based demonstration selection framework, where a model is trained to predict the utility of candidate demonstrations and select the most promising ones. Although fairness-related features are incorporated, the overall methodology remains similar to existing learned demonstration selection approaches. More importantly, the surrogate primarily predicts conventional classification performance rather than explicitly optimizing fairness. Therefore, it remains unclear whether the improvements arise from fairness-specific modeling or simply from better general-purpose demonstration selection. Stronger comparisons and fairness-focused ablations are needed to establish the necessity and effectiveness of the proposed design.

Questions for authors
  1. Why is the evaluation limited to two medical imaging datasets? Can the authors provide additional experiments on common fairness-sensitive NLP tasks, such as toxicity detection, or clarify the generalizability of FANDS beyond medical imaging?
  2. How do the authors distinguish the general negative effects of noisy demonstrations from their interactions with sensitive attributes? For example, if removing sensitive attributes does not eliminate the negative effects of label noise, how does this observation support the necessity of fairness-specific modeling?
  3. Does FANDS actually reduce performance disparities between demographic groups, rather than simply improving overall prediction accuracy? Can the authors report group performance gaps and investigate whether changing sensitive attributes affects demonstration selection and predictions?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.