- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper focuses on bias mitigation in LLMs for medical decision support. It proposes MedSteer, an inference-time representation steering method for reducing demographic bias in medical LLMs while preserving clinically relevant information. It combines diagnosis-conditioned representation differences, semantic-stability and clinical-orthogonality weighting, and an auxiliary stigma–neutral lexical direction. Evaluations across four models cover medical question answering, clinical entity retention, and demographic sensitivity. Compared with unweighted MeanDiff steering, the authors report an average QA accuracy improvement of 2.6 percentage points and higher entity retention, with fairness outcomes varying across models.
Critical strengths
- The paper highlights the significance of separating sensitive attributes from diagnosis-relevant clinical information.
- MedSteer intervenes on hidden representations at inference time while keeping model parameters frozen, enabling post-deployment debiasing without fine-tuning the underlying LLM.
- The paper conducts diverse experiments in 12 model–dataset combinations.
Critical weaknesses
-
This paper does not sufficiently position MedSteer relative to existing work on mitigating demographic bias while preserving clinically relevant information. Prior studies, including [1], [2], [3], [4], and [5], have investigated related objectives and reported improvements in medical fairness and performance via diverse strategies, including model fine-tuning, reasoning steering, and prompt engineering. However, the paper states that existing steering methods “often fail to separate sensitive attributes from diagnosis-relevant clinical information” without identifying which methods exhibit this limitation, under what clinical conditions, or according to which evaluation criteria. In particular, the paper has cited and discussed [5], which is similar to MedSteer, but [5] has not been included as a baseline. Additionally, the paper has not positioned inference-time steering methods, such as MedSteer, among the medical debiasing approaches set to demonstrate the superiority of inference-time steering methods. For example, because both MedSteer and prompt engineering can mitigate bias without extra model training or tuning, the advantages of the proposed methods are not clear and need more justification.
-
The presentation of motivation can be misleading, and there is a gap between claim and evidence. The claims, such as "MedSteer strictly targets ungrounded sensitivity without disrupting valid diagnostic pathways, " and "The missing capability is therefore a selective intervention: identifying which components of an attribute-associated representation shift can be attenuated without perturbing the clinical evidence underlying the model’s response...," may mislead readers into thinking that MedSteer can distinguish whether a sensitive attribute, such as sex, age, or race, is a clinically relevant attribute or a non-clinically relevant attribute for specific medical diagnosis. For instance, some diseases, such as Systemic Lupus Erythematosus (SLE) are strongly sex-related. So, sex is a clinically relevant attribute in SLE diagnosis.
However, the evaluation of the paper does not directly distinguish clinically decisive from non-decisive demographic effects based on a specific medical question. Instead, the paper uses changes in diagnostic metrics, such as accuracy and F1, as a proxy for measurement, whose reliability needs more solid evidence. Aggregate medical QA accuracy, entity retention, and bias metrics do not establish that unjustified demographic effects are reduced while clinically necessary effects are preserved. Direct, clinically grounded evaluation of both cases is needed to substantiate this selectivity.
The definition and measurement of bias need more justification. The paper defines an attribute as “clinically non-decisive” when changing it should not alter the medically justified conclusion (Section 3.1). However, it does not specify how this judgment is established for individual cases, including the clinical criteria, annotation procedure, or expert validation used. Without this operational definition, it is unclear how the evaluation distinguishes unjustified demographic sensitivity from legitimate clinical variation, limiting its ability to assess the stated conditional-invariance objective.
-
Table 1 and Table 2 demonstrate that the results among methods are very close, and sometimes MeanDiff is better than MedSteer. Therefore, it is difficult to recognize that 2.6 percentage points is a significant improvement without statistical analysis in these experimental tasks. The paper has not reported the results of repeating the experiment and the standard deviation across different runs of the same experimental settings. Thus, we cannot determine whether the 2.6 percentage points of improvement is a benefit from the proposed approach or just uncertainty of LLMs.
-
MeanDiff seems to be an ablation of MedSteer and not an external baseline to demonstrate the superiority of MedSteer against existing methods. Therefore, diverse types of existing methods may be more suitable as baselines, such as [1], [4], and [5].
Questions for authors
-
What is the novelty of MedSteer and its advantages over existing bias-mitigating strategies, such as prompt engineering?
-
What direct, clinically grounded evidence shows that MedSteer suppresses unjustified demographic sensitivity while preserving demographic effects necessary for correct clinical decisions, beyond improvements in aggregate QA accuracy, entity retention, and bias metrics?
-
Is the reported 2.6-percentage-point gain statistically reliable, and can you provide confidence intervals or paired significance tests, along with repeated-run variability where applicable?
-
How does MedSteer compare with existing approaches such as [1], [4], and [5] under comparable evaluation settings?
2 comments
Thank you for the response, which partially addresses my concerns. Two main concerns remain.
-
The new diagnostic evaluation is helpful, but both case groups test whether predictions remain consistent when race is changed while clinical evidence is held fixed. This demonstrates improved invariance, but does not establish whether MedSteer preserves clinically justified demographic sensitivity when changing an attribute should legitimately affect the medical decision. This distinction is central to the paper's motivation and claimed contribution.
-
While the authors clarify how MedSteer differs from prompt engineering and other debiasing strategies, the response does not provide empirical evidence demonstrating its advantages over these alternatives. The evaluation still relies primarily on MeanDiff, making it difficult to assess the practical significance of the proposed method relative to existing approaches.
Regarding Q4, I apologize for inadvertently omitting the reference list from my original review, which made the cited works ambiguous. This was an oversight on my part, and I do not expect the authors to address that question.
Overall, the additional evidence addresses some of my concerns, but I remain unconvinced that the central claim of clinically selective debiasing has been sufficiently validated. I therefore maintain my current rating.
Thank you for the detailed review. We clarify the contribution and provide additional experimental evidence below.
Q1. Contribution and relationship to existing approaches.
MedSteer's contribution is diagnosis-conditioned direction construction and weighted aggregation. Each demographic contrast matches admissions on diagnosis and the other recorded demographic/social covariates. An exhaustive audit verified these matching constraints and source-field/template consistency for every pair. Semantic weighting favors pairs with similar representations, while centroid-based weighting reduces the contribution of shifts aligned with diagnosis representations:
wi=max(0,cos(hi,1,hi,2))(1−∣cos(Δhi,vc)∣).
An auxiliary stigma–neutral branch adds a framing-related signal using semantic weighting. MedSteer then applies layer-wise gated interventions during inference. This introduces a clinical prior into direction construction and provides a lightweight mechanism for balancing demographic sensitivity and medical utility.
Relative to prompt engineering, MedSteer acts directly on hidden states without adding a debiasing instruction to each input. Model parameters remain frozen, and the intervention strength is selected on validation data. The distinction is the clinical information used to construct and weight the steering directions.
Q2. Direct behavioral evidence under fixed clinical evidence.
We additionally evaluated two sets of 100 cases, changing only the Black/White descriptor while keeping each case's clinical evidence fixed. The ancestry-associated set includes sickle-cell disease (60), thalassemia (20), and G6PD deficiency (20). The case-level non-decisive set includes appendicitis, cholecystitis, pancreatitis, bowel obstruction, and epilepsy (20 each).
Joint correctness increases by 13 and 8 percentage points, respectively, alongside improved agreement. Measuring correctness under both descriptors distinguishes successful invariance from two identical but incorrect outputs. This evaluates diagnostic behavior directly, complementing the representation projections and aggregate QA results.
The operational criterion is case-specific: given the fixed diagnostic evidence in these cases, changing only the race descriptor should preserve the diagnosis. Both cohorts assess this form of invariance, including cases involving ancestry-associated conditions; the evaluation does not treat race as a substitute for genotype or assume equal prevalence across populations.
Q3. Accuracy gains and uncertainty evidence.
The 12 model–dataset differences in Table 1 average 2.57 percentage points, reported as approximately 2.6. MedSteer improves observed QA accuracy over MeanDiff in 11 of the 12 combinations. The reported confidence intervals are entirely above zero for eight configurations, spanning all four backbones. For example, Qwen's gains are +3.00 points [+1.00, +5.00] on MedQA and +4.00 [+2.00, +6.00] on MedMCQA; Internist improves by +4.66 [+2.66, +6.66] on MMLU, and Meditron by +3.38 [+1.38, +5.38] on MMLU. These intervals provide setting-level uncertainty information alongside the descriptive average.
Entity-retention F1 also improves on all four backbones: absolute increases over MeanDiff are 0.0786 for Internist, 0.0453 for Qwen, 0.0034 for Meerkat, and 0.0104 for Meditron.