- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper studies how much intervention effectiveness is retained when prompt-specific attribution is compressed into one fixed semantic routing decision per deployment regime. Block Prior (BP) decomposes prompts into Subject, Argument, Action, and Setting, computes exact four-player Shapley values under a specified input-side lexical risk function, and selects the role with the highest mean attribution on a calibration corpus. At inference time, recognized lexical evidence within that role is neutralized.
On three BBQ regimes, BP achieves 54.5% pooled reduction of the specified input-side risk score, compared with 43.4% for Subject-only and 70.7% for prompt-adaptive Shapley routing, retaining approximately 77% of the latter’s measured reduction. The paper also examines preprocessing, role granularity, lexical coverage, and structural perturbations. In particular, relocating recognized evidence outside the fixed role defeats BP, while adaptive routing recovers substantial risk reduction. Additional validation includes a four-target/two-judge output evaluation, attribution-profile agreement on a selected DiscrimEval subset, and a synthetic security-domain study of the four-role interface. The central contribution is an empirical characterization of the capabilities and limitations of an attribution-derived fixed routing policy.
Critical strengths
-
The controlled comparison makes the routing question interpretable. BP, Subject-only, and Prompt-Shapley share the semantic representation and lexical neutralization operator, so the comparison in Table 3 isolates the consequences of changing the routing policy within the specified input-side objective. The substantial Age improvement over Subject-only and the exact coincidence of the two policies for Gender Identity provide useful contrasting cases. This is a concrete and understandable way to study what a regime-level decision retains from instance-level attribution.
-
The structural failure analysis provides useful insight into the fixed-policy assumption. The relocation experiment is particularly informative: recognized evidence remains present and actionable, but moving it outside the selected role produces 100% attack success against BP, while recomputing attribution restores substantial reduction (Appendix G.7). This connects the observed failure to a specific design choice. The paper also clearly distinguishes its internal risk metric from independent behavioral evidence and explicitly limits its efficiency, transfer, and security claims (Section 5 and Appendix I). These qualifications make the study more useful as a diagnosis of when fixed semantic routing is appropriate.
Critical weaknesses
-
The experiments do not establish what Shapley calibration contributes beyond simpler fixed routing rules. The paper acknowledges that a route selected from aggregate per-role lexical evidence is missing (Section 5; Appendix I.5), but this comparison remains central because the attribution game itself is constructed from lexical evidence. A second necessary comparison is a fixed role selected by directly maximizing calibration-set mean Bias Reduction over the four candidate roles, using the same intervention operator. Maximizing mean Shapley attribution is not generally equivalent to maximizing the reported mean relative Bias Reduction. Consequently, the current comparison with Subject-only does not establish that Shapley calibration is responsible for the improvement, and the gap to Prompt-Shapley may reflect both the restriction to a fixed role and the particular criterion used to select that role. These inexpensive, matched comparisons would substantially clarify the contribution.
-
The independent evaluation does not yet establish the practical significance of the central routing results. The headline reduction and 77% retention figure use the same characteristic function that defines attribution and routing. This is informative about the specified objective, but it does not independently establish reduced output bias or preserved task utility, as the authors acknowledge in Appendix I.1. Appendix F provides independent output-side evidence, but compares only Original against BP; it does not evaluate Subject-only and Prompt-Shapley under the same protocol. Thus, it does not test whether the principal routing advantage or compression gap persists outside the internal metric. The substantial evaluator disagreement on DiscrimEval further limits the conclusion: GPT-4o-mini-as-judge reports improvement for all four targets, whereas Gemini-as-judge reports zero or negative changes. Embedding similarity and decision stability do not resolve whether useful task information or correctness is preserved. Even within the internal evaluation, the Physical Appearance improvement remains uncertain: its BP-minus-Subject-only interval is [-4.07, 15.11] percentage points (Appendix D.5).
These two gaps drive my weak-rejection recommendation. The study is clearly scoped and contains useful diagnostics, but stronger evidence is needed to establish the value of the proposed calibration criterion and the practical meaning of the effectiveness it retains.
Questions for authors
-
Additional evidence: What does the Shapley-based routing criterion add? On the same calibration and held-out data, how does BP compare with fixed routes selected by (a) mean per-role lexical evidence and (b) directly maximizing calibration-set mean BR over the four possible roles? Please report the selected roles, route agreement, and held-out outcomes under the same intervention operator. These comparisons would help distinguish the effect of restricting the policy to one role from the effect of the criterion used to choose that role.
-
Additional evidence: Do the central comparisons persist under independent evaluation? Can the existing output-side protocol include Subject-only and Prompt-Shapley alongside BP, with a focused measure of task correctness or independently assessed intervention appropriateness? I am particularly interested in whether BP’s advantage over Subject-only and its gap to adaptive routing remain visible, and how those conclusions depend on the judge. A matched comparison on the existing evaluation setting would materially affect my assessment.
-
Clarification: How are calibration and repeated evaluation separated? Section 4 and Appendix D.1 describe five 80/20 partitions while keeping the calibrated BP profile fixed across partitions. Is the calibration corpus disjoint from every evaluation partition? Please specify the number of unique prompts and the calibration/evaluation overlaps for each seed, and clarify whether the fixed object is the full BP vector or only its selected role. This information is necessary to interpret the reported results as held-out performance.
-
Clarification: What is the complete coalition scoring function? Please specify W_r(S), including signed lexical weights and any coalition-dependent normalization. For an additive game with nonnegative role weights and an increasing link function, within-prompt Shapley rankings coincide with raw role-weight rankings regardless of curvature, although this equivalence need not survive averaging across prompts. How should the convexity-based distinction in Section 5 and Appendix I.5 be understood under the actual implementation, and what accounts for the negative social-bias Shapley values reported in Appendix B.4?
Thank you for your response. I understand your point to some extent, but much of what you wrote is difficult to follow; therefore, I will keep my rating as it is. If you could provide a clearer explanation, the outcome might be more favorable.