- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper presents an interesting finding that positional bias can persist even when LLM decisions remain unchanged, and proposes a framework to localize and mitigate it while preserving evidence sensitivity. Their integration provides useful empirical insights. Further analysis of the mean-centered intervention and fallback cases could strengthen the mechanistic understanding of why positional bias is easier to control in some models than others.
Critical strengths
- The motivation is clear. The paper identifies an important distinction between behavioral consistency and latent positional bias, showing that substantial positional effects can persist even when LLM predictions remain unchanged.
- The empirical part is systematic. The paper integrates margin-based measurement, activation patching, and constrained attention-head interventions into a coherent framework. Experiments across six models and multiple comparison tasks demonstrate substantial bias reduction while largely preserving evidence sensitivity.
Critical weaknesses
- The proposed framework primarily combines established techniques, including logit-margin regression, activation patching, and attention-head scaling.
- The mean-centered intervention scales entire attention-head outputs without explicitly separating positional information from task-relevant evidence. Consequently, it remains unclear whether the observed bias reduction reflects selective control of positional representations or compensating changes across attention heads. Comparisons with alternative intervention strategies would strengthen the mechanistic claims.
- The fallback procedure relies on heuristic head selection and performance thresholds, with limited explanation of why some models permit near-complete bias removal while others do not.
Questions for authors
-
The intervention parameterization in Eq. (4) appears inconsistent with the attenuation procedure, since α = 0 corresponds to no intervention. The authors should clarify the implementation and verify the effects across different gains.
-
Comparing mean-centered scaling with direct head scaling, mean ablation, and position-specific interventions under matched performance constraints would help establish whether the proposed method selectively controls positional bias.
-
The fallback cases deserve further investigation through bias-evidence trade-off curves, particularly for Phi-3 and Mistral-7B, to clarify whether limited controllability arises from performance constraints or the restricted intervention search.
-
The stability of head selection and gain calibration should be evaluated across multiple development seeds, including the consistency of selected heads, held-out bias reduction, and evidence preservation.
-
The authors could examine conditional positional effects across different score gaps, , to determine whether reductions in average positional bias conceal substantial effects at specific comparison difficulties.