acceptodds
Under review as a conference paper at ICLR 2027

Shapley-Calibrated Semantic Routing for Black-Box LLM Guardrails: What a Fixed Route Retains

Abstract

Input attribution localizes model behavior to specific prompt components, turning debiasing from untargeted rewriting into targeted intervention. A deployed guardrail, however, must commit in advance to the component it will act on. We study what is lost when prompt-level attribution is compressed into a single fixed routing decision. We introduce Block Prior (BP), which decomposes prompts into four interpretable roles (Subject, Argument, Action, and Setting). BP computes exact four-player Shapley attributions offline under an auditable input-side bias function and aggregates them into one regime-level route applied unchanged to every incoming prompt. Exact attribution costs coalition evaluations per prompt and is tractable both offline and online. The question we answer is how much localization a single stored coordinate can carry. On social-bias regimes from BBQ, the four-role calibration issues a falsifiable prediction about which regimes should depart from a fixed Subject-only policy, and held-out intervention confirms it: +39.2 points of bias reduction for Age and +6.5 for Physical Appearance, with exact coincidence for Gender Identity, where the two policies are identical by construction. The compressed route recovers 77% of the bias reduction obtained by recomputing exact Shapley attribution for every prompt (54.5% versus 70.7% pooled) while improving on the fixed Subject-only policy by 11.2 points, and it preserves high prompt similarity (0.929). We then map where the compression works and where it fails: it is near-insensitive to additive padding (<1% attack success), degrades under out-of-vocabulary lexical substitution (≈40%), and fails completely when bias-bearing evidence is relocated to a non-routed role (100%), a perturbation against which prompt-adaptive attribution remains effective. Cross-dataset and cross-regime studies establish, respectively, ordinal profile agreement on a target-conditioned subset and representational portability of the four-role interface under a different risk specification. Our results quantify the price of a fixed attribution-derived policy

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Large Language Models, Black-Box Models, Shapley Values, Semantic Routing, LLM Guardrails

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_1jCAOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies how much intervention effectiveness is retained when prompt-specific attribution is compressed into one fixed semantic routing decision per deployment regime. Block Prior (BP) decomposes prompts into Subject, Argument, Action, and Setting, computes exact four-player Shapley values under a specified input-side lexical risk function, and selects the role with the highest mean attribution on a calibration corpus. At inference time, recognized lexical evidence within that role is neutralized.

On three BBQ regimes, BP achieves 54.5% pooled reduction of the specified input-side risk score, compared with 43.4% for Subject-only and 70.7% for prompt-adaptive Shapley routing, retaining approximately 77% of the latter’s measured reduction. The paper also examines preprocessing, role granularity, lexical coverage, and structural perturbations. In particular, relocating recognized evidence outside the fixed role defeats BP, while adaptive routing recovers substantial risk reduction. Additional validation includes a four-target/two-judge output evaluation, attribution-profile agreement on a selected DiscrimEval subset, and a synthetic security-domain study of the four-role interface. The central contribution is an empirical characterization of the capabilities and limitations of an attribution-derived fixed routing policy.

Critical strengths
  1. The controlled comparison makes the routing question interpretable. BP, Subject-only, and Prompt-Shapley share the semantic representation and lexical neutralization operator, so the comparison in Table 3 isolates the consequences of changing the routing policy within the specified input-side objective. The substantial Age improvement over Subject-only and the exact coincidence of the two policies for Gender Identity provide useful contrasting cases. This is a concrete and understandable way to study what a regime-level decision retains from instance-level attribution.

  2. The structural failure analysis provides useful insight into the fixed-policy assumption. The relocation experiment is particularly informative: recognized evidence remains present and actionable, but moving it outside the selected role produces 100% attack success against BP, while recomputing attribution restores substantial reduction (Appendix G.7). This connects the observed failure to a specific design choice. The paper also clearly distinguishes its internal risk metric from independent behavioral evidence and explicitly limits its efficiency, transfer, and security claims (Section 5 and Appendix I). These qualifications make the study more useful as a diagnosis of when fixed semantic routing is appropriate.

Critical weaknesses
  1. The experiments do not establish what Shapley calibration contributes beyond simpler fixed routing rules. The paper acknowledges that a route selected from aggregate per-role lexical evidence is missing (Section 5; Appendix I.5), but this comparison remains central because the attribution game itself is constructed from lexical evidence. A second necessary comparison is a fixed role selected by directly maximizing calibration-set mean Bias Reduction over the four candidate roles, using the same intervention operator. Maximizing mean Shapley attribution is not generally equivalent to maximizing the reported mean relative Bias Reduction. Consequently, the current comparison with Subject-only does not establish that Shapley calibration is responsible for the improvement, and the gap to Prompt-Shapley may reflect both the restriction to a fixed role and the particular criterion used to select that role. These inexpensive, matched comparisons would substantially clarify the contribution.

  2. The independent evaluation does not yet establish the practical significance of the central routing results. The headline reduction and 77% retention figure use the same characteristic function that defines attribution and routing. This is informative about the specified objective, but it does not independently establish reduced output bias or preserved task utility, as the authors acknowledge in Appendix I.1. Appendix F provides independent output-side evidence, but compares only Original against BP; it does not evaluate Subject-only and Prompt-Shapley under the same protocol. Thus, it does not test whether the principal routing advantage or compression gap persists outside the internal metric. The substantial evaluator disagreement on DiscrimEval further limits the conclusion: GPT-4o-mini-as-judge reports improvement for all four targets, whereas Gemini-as-judge reports zero or negative changes. Embedding similarity and decision stability do not resolve whether useful task information or correctness is preserved. Even within the internal evaluation, the Physical Appearance improvement remains uncertain: its BP-minus-Subject-only interval is [-4.07, 15.11] percentage points (Appendix D.5).

These two gaps drive my weak-rejection recommendation. The study is clearly scoped and contains useful diagnostics, but stronger evidence is needed to establish the value of the proposed calibration criterion and the practical meaning of the effectiveness it retains.

Questions for authors
  1. Additional evidence: What does the Shapley-based routing criterion add? On the same calibration and held-out data, how does BP compare with fixed routes selected by (a) mean per-role lexical evidence and (b) directly maximizing calibration-set mean BR over the four possible roles? Please report the selected roles, route agreement, and held-out outcomes under the same intervention operator. These comparisons would help distinguish the effect of restricting the policy to one role from the effect of the criterion used to choose that role.

  2. Additional evidence: Do the central comparisons persist under independent evaluation? Can the existing output-side protocol include Subject-only and Prompt-Shapley alongside BP, with a focused measure of task correctness or independently assessed intervention appropriateness? I am particularly interested in whether BP’s advantage over Subject-only and its gap to adaptive routing remain visible, and how those conclusions depend on the judge. A matched comparison on the existing evaluation setting would materially affect my assessment.

  3. Clarification: How are calibration and repeated evaluation separated? Section 4 and Appendix D.1 describe five 80/20 partitions while keeping the calibrated BP profile fixed across partitions. Is the calibration corpus disjoint from every evaluation partition? Please specify the number of unique prompts and the calibration/evaluation overlaps for each seed, and clarify whether the fixed object is the full BP vector or only its selected role. This information is necessary to interpret the reported results as held-out performance.

  4. Clarification: What is the complete coalition scoring function? Please specify W_r(S), including signed lexical weights and any coalition-dependent normalization. For an additive game with nonnegative role weights and an increasing link function, within-prompt Shapley rankings coincide with raw role-weight rankings regardless of curvature, although this equivalence need not survive averaging across prompts. How should the convexity-based distinction in Section 5 and Appendix I.5 be understood under the actual implementation, and what accounts for the negative social-bias Shapley values reported in Appendix B.4?

1 comment

Thank you for your response. I understand your point to some extent, but much of what you wrote is difficult to follow; therefore, I will keep my rating as it is. If you could provide a clearer explanation, the outcome might be more favorable.

Reviewer_KJPGOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The work investigates what is lost when prompt-level input-risk attribution is compressed into a fixed semantic route. Block Prior (BP) averages exact Shapley attributions over four prompt roles during calibration, then neutralizes recognized lexical evidence within the selected role at runtime. On three BBQ regimes, BP retains 77% of the input-side risk reduction achieved by per-prompt attribution (54.5% versus 70.7%). Further experiments examine output behavior, transfer, and robustness. OOV substitutions degrade the intervention, while relocating evidence outside the selected role defeats it.

Critical strengths
  1. The work makes the trade-off between fixed and prompt-adaptive routing measurable. Comparisons using the same lexical intervention isolate the effect of routing, with BP retaining about 77% of the prompt-adaptive input-risk reduction.

  2. The ablation study supports separating Subject from Argument in the tested regimes.

Critical weaknesses
  1. Evidence of bias reduction is unclear. BR uses the same input-side function that defines the Shapley game and selects the route, so the held-out results establish performance on that objective rather than independently measured bias reduction. The output evaluation provides some support, with lower judged bias in 23 of 32 settings, but results depend strongly on the judge. It also compares only Original and BP, leaving unclear whether BP's routing improves downstream behavior beyond Subject-only or other simple interventions. Task correctness after neutralization is not included.

  2. It's unclear whether the routing design is generalizable. There is no comparison with a fixed route selected from mean per-role lexical evidence, so the benefit of Shapley calibration is unresolved. Since per-prompt attribution requires only 16 input-side evaluations, the practical benefit of fixing the route also needs stronger justification.

Questions for authors
  1. Can you compare BP, Subject-only, and Prompt-Shapley on independently measured output bias and task accuracy?

  2. How does BP compare with a fixed route selected from mean per-role lexical evidence? What practical benefit motivates fixing the route when per-prompt attribution requires only 16 input-side evaluations?

  3. Does the source prior preserve the attribution ordering on all 75 DiscrimEval prompts, without selecting profiles based on similarity to that prior?

Reviewer_DSQsOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces Block Prior, an input-side framework for mitigating bias in black-box LLMs. It decomposes prompts into four semantic roles and uses Shapley attribution on calibration data to select a fixed intervention target for each deployment regime. At inference time, it modifies recognized risk-bearing expressions within the selected role before passing the prompt to the target model. The evaluation covers social-bias benchmarks, downstream model responses, and controlled transfer.

Critical strengths

1. The paper offers an interpretable framework connecting attribution to intervention. Its shared semantic representation gives attribution results directly editable counterparts in the input, making the relationship between risk scoring, target selection, and modification explicit. This provides a coherent framework for studying input-side interventions without access to the protected model’s internals.

2. The paper contributes an empirical characterization of fixed attribution-guided policies and their limitations. Across effectiveness, downstream behavior, transfer, and robustness experiments, it examines how much prompt-specific attribution can be compressed into a reusable policy and where that policy breaks down.

Critical weaknesses

1. Missing comparisons with existing methods. The paper primarily compares BP with two internal routing baselines, Subject-only and Prompt-Shapley, without experimental comparisons against existing guardrail or debiasing methods. This limits assessment of its practical advantages relative to prior work.

2. The evaluation does not adequately establish response correctness or usefulness. The output-side evaluation compares responses before and after intervention primarily through model-judged bias scores. This comparison can indicate reduced perceived bias, but does not establish whether the resulting answers are more accurate or useful. Consequently, it remains unclear whether debiasing corrects erroneous responses while preserving the information and reasoning needed to complete the task.

3. Applicability to realistic deployment remains unclear. The method assumes that the input’s deployment regime is known and uses the corresponding lexical resources and fixed route. It does not establish how to handle unknown regimes or overlapping bias categories. These dependencies limit its applicability as a general-purpose guardrail.

Questions for authors

I would appreciate clarification on the issues raised in the Critical Weaknesses section.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.