- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper addresses how to identify which components of an LLM agentic system cause it to treat matched inputs differently. Inputs are paired such that they differ only in a synthetic sensitive attribute. The authors present an agent as a graph of components (rule clauses, demonstrations, retrieval triggers, etc.) and pre-write an interface-preserving “neutral” replacement for every candidate component. They then run randomized experiments that swap several components at once, and fit a sparse statistical model estimating each component’s effect, along with effects of pairs of connected components. Separate models estimate loss of usefulness, and the smallest set of patches that confidently meets preset thresholds on disparity reduction, utility loss and regression is selected and then re-tested on held-out data. Experiments use 3 synthetic environments run with Llama-3.1-8B, with planted biases of several kinds.
Critical strengths
S1. The paper brings causal, intervention-based attribution to the components practitioners have access to and can edit (individual rule clauses, retrieval triggers, routing edges, etc.), rather than intervening on model internals or purely observing agentic trajectories. Its design also accounts for components that only matter in combination or that back each other up, which trace-based and one-at-a-time analyses can miss.
Critical weaknesses
W1. The experiments suffer from limited realism. Planting biases is reasonable for establishing ground truth, but the setup differs from practice in two ways: in every example shown in Appendix E, the biased component explicitly conditions on the attribute (e.g. “if group_tag is violet, retrieve…”), and the neutral replacements are written with privileged knowledge of the mechanism to remove (Appendix D.7.2). In a real audit, neither holds: the bias may not be written into any single component, and the auditor does not know which component is biased or what its neutral version should be. The experiments therefore show that, given correct neutral versions of every component, the statistical design can identify which swaps matter. This is narrower than the Conclusion’s claim that the findings confirm a “rigorous and effective basis” for identifying causal components.
Questions for authors
Q1. At least two citations do not seem to support the claims they are attached to: de Aragao et al. (2022) and Zhu et al. (2022). The former is a chemical physics paper about electron charge transfer, and the latter is about multi-party empathetic dialogue generation that is unrelated to what it was cited for, large-scale intersectional templates, in the paper. Which works did the authors intend to cite, and could they cross-check the remaining references?
Q2. Section 4.2 reports that PFSIR exceeds leave-one-out by 0.018 AUPRC on isolated carriers, 0.214 on pure interactions, and 0.137 on redundant carriers, but these terms are not defined and the values do not appear to match Table 4. For example, Table 4 reports a redundant-pair node AUPRC of .836 for PFSIR vs .636 for leave-one-out (a gap of .200), and the node AUPRC gaps for the three carrier types consisting of a single component (rule clause, demonstration, retrieval trigger) range from .043 to .107, so no average of them could equal .018. Could the authors define these categories and clarify which experiments the numbers come from?