- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
The authors introduce DeepSift, a framework that uses Shapley values on hierarchically partitioned context for post-hoc provenance localization.
Critical strengths
I definitely think the high-level idea is interesting and deserves some merit. The paper introduces a way to localize malicious inputs using Shapley values that I think will be more valuable as we move to longer and longer horizon tasks. The wide range of empirical results also help the case for it being useful in various settings, although there are parts I am not confident about.
Critical weaknesses
The main problem I have with the paper is that it seems to be framed in a more theoretic way than it actually is, and I am not sure if the parts that are formalized actually hold up as strongly as the paper suggests. I list my main concerns below and leave more in the questions below.
-
The Shapley values and all associated with it are defined in terms of a surrogate LLM, and to my understanding there's no theoretical aspect that suggests this surrogate is connected to the actual model. This means even with empirical evidence, there's no actual theoretical guarantee that makes this a causal attribution. Thus the main contribution as I stated above become more of a way to localize malicious inputs using Shapley values on a surrogate model rather than anything causal or theoretical. In the same way, my understanding of Theorem 1 is that if malicious elements score much higher than benign ones, the gap will separate them. But the values it uses seem to depend on things you would not actually know during the attribution (I may be wrong on this). Finally, Shapley values are generally as meaningful as the function over the coalitions, but with the feature-removals in the counterfactual experiments the method may be sound mathematically but in practice it may be too OOD to the surrogate to avoid having a caveat written about it.
-
The empirical validation also seems a little circular, as Equation 5 holds at 93% when it holds, but the theorem itself should already guarantee the exact selection whenever Equation 5 holds. In addition, this requires knowing which children are malicious and copmuting the exact Shapley values and the bound over all the subsets, meaning that this is not doing much theoretical validation.
Separate from the two points, I do generally think that the paper was not easy to follow, following the earlier comment about the paper maybe being more formalized than it needs to be. Note that this is more of a personal comment as a reviewer, not something that speaks for the contributions of the paper.
Questions for authors
-
[Clarification] The paper defines as the spans written by the attacker but also motivates DeepSift as something that identifies the texts that are "responsible for"/"cause" this. But benign task context can be causally necessary for an attack to succeed, while injected spans may end up being redundant. Could the authors clarify what the actual intended task is?
-
[Clarification] The theoretical analysis heavily uses Shapley values of , where is a surrogate model. Is the assumption that there is some sort of formal relationship between the surrogate and the actual agent? Without this, it seems like what Theorems 1 and 2 are saying is that they can only recover when the malicious spans already show different scores under the surrogate, rather than establishing some sort of causal guarantee of the agent's actions. These seem like two notably different claims to be making, and would be worth addressing.
-
[Clarification] DeepSift computes Shapley values on counterfactual contexts that remove portions of the agent context. But why would we expect the likelihoods on these potentially OOD contexts to provide causal attribution for the original execution? It seems like this would only create semantically incoherent contexts in the subset and thus would be difficult to maintain reliability under these conditions.
-
[Clarification] The post-hoc certificate in Theorem 3 requires parameters and . Is it possible to have these values without access to malicious-span ground truth?