- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
GIF models local information flow from an input span to the next-token distribution using the model Jacobian and softmax Fisher geometry. It derives a local Gaussian-channel information bound, uses the Fisher-pullback trace as an influence proxy, and estimates that trace with randomized matrix-vector operations. Experiments test source localization and the use of selected spans for policy decisions on three agent-security benchmarks, including approximations and surrogate analysis models.
Critical strengths
The local information-theoretic result gives a precise interpretation to the influence score. The fixed-context, small-perturbation assumptions are explicit, and Appendix B develops the argument from expected KL to the Gaussian-channel proxy. Connecting this result to an estimator that avoids materializing the full Jacobian is a useful contribution for studying information flow in language models.
The empirical evaluation connects the estimator to practical source selection. Table 2 compares sketched and exact influence, and the same-model comparisons in Figure 2 show that compact attributed evidence can preserve or improve policy decisions while reducing the judge's input.
Critical weaknesses
-
The headline claim of matching GPT-5.5 xhigh at more than 81 times lower cost is not supported by a comparison on a common evaluation population. Section 4.2 and Figure 2 compare open-model rows evaluated on each model's own trajectories with a GPT-5.5 row pooled across all six agent models, as the caption states. Differences in trace difficulty and violation prevalence therefore enter the quality comparison. The shared-sample attribution experiment in Appendix F.2 concerns a different task and does not repair this policy-decision comparison. Report both systems on the same trajectories, actions, and labels, and include GIF's gradient and ranking costs in the total. The paper should distinguish the cost of the policy judge from the complete attribution-and-decision pipeline.
-
The incremental contribution of the Fisher-based selector needs a more precise account. JacobianScopes/FisherScope already defines the softmax Fisher pullback and its trace as token influence (January 2026 version, Section 3.2 and Appendix A). The additional contribution here is the local information bound, estimation procedure, and policy application. For the application, Figure 2 does not compare GIF-selected evidence with BM25 or GIF-minus evidence supplied to the same judge at the same length and format. This matters because Table 1 shows that BM25 exceeds GIF on average MSB localization at @15: AUROC 0.92 versus 0.89 and AUPRC 0.94 versus 0.91. A matched downstream comparison would establish whether the geometric selection itself improves policy decisions beyond compact evidence selection.
Questions for authors
-
What are the policy-decision results and full costs when the Qwen-GIF and GPT-5.5 systems receive exactly the same trajectory/action examples? Please distinguish attribution computation from judge inference in the reported cost factor.
-
With the judge, evidence length, and prompt format fixed, how do GIF, GIF-minus, and BM25 span selection compare? Please also state precisely which mathematical result is new relative to the Fisher-pullback construction in JacobianScopes/FisherScope.
3 comments
Thank you for clarifying the cost figure and the relation to Liu et al. I am keeping my rating at 3. The local information-theoretic result and matrix-free estimation remain the main reasons for my recommendation.
The acknowledgment that the 81x comparison uses different trajectory populations is important. Restricting the number to downstream inference cost does not resolve the quality-comparison issue. Please describe this comparison as illustrative throughout the paper, including the abstract, and distinguish it from the same-model results. A backward-pass count and surrogate-transfer results also do not quantify the attribution overhead of the evaluated pipeline. A matched comparison needs the same trajectories, actions, and labels, with attribution and judging costs reported separately and in total.
The downstream selector question remains unanswered. Section 4.1 measures localization; it does not establish what GIF's ranking adds to policy decisions. A focused comparison of GIF, GIF-minus, and BM25 with the same judge, trajectories, evidence length, and prompt format would distinguish the contribution of the geometric ranking from the benefit of shortening the context. This is a targeted control for the application claim.
We thank the reviewer for their follow-up and continued recognition of GIF's theoretical and methodological contributions.
Regarding the GPT-5.5 comparison, we appreciate the reviewer's suggestion to control for differences in trajectory populations. We have disaggregated our results to compare GIF-guided Qwen3-8B against GPT-5.5 xhigh on identical trajectories, actions, and ground-truth labels across all six trajectory-generating models on AgentDojo and MSB, the two benchmarks underlying our 81× comparison.
The table below reports F1 (%) for GIF-3 with Qwen3-8B as the downstream declassifier versus GPT-5.5 xhigh performing full-trajectory judging.
| Model | AgentDojo | AgentDojo | MSB | MSB |
|---|---|---|---|---|
| GIF-3 + Qwen3-8B | GPT-5.5 xhigh | GIF-3 + Qwen3-8B | GPT-5.5 xhigh | |
| Qwen3-8B | 74.0 | 76.1 | 83.1 | 84.0 |
| Qwen3-14B | 89.7 | 90.5 | 91.5 | 92.4 |
| Qwen3-32B | 88.1 | 88.4 | 82.2 | 84.2 |
| Gemma-4-31B | 86.6 | 90.1 | 61.2 | 83.6 |
| GPT-OSS-20B | 72.7 | 75.5 | 76.8 | 77.8 |
| GPT-OSS-120B | 77.3 | 74.9 | 87.4 | 89.2 |
| Macro-average | 81.4 | 82.6 | 80.4 | 85.2 |
These matched results demonstrate that GIF-guided Qwen3-8B achieves comparable detection performance on AgentDojo (81.4% vs. 82.6% F1) and MSB (80.4% vs. 85.2% F1). On MSB, the difference is predominantly driven by the Gemma-4-31B configuration; across the other five models, performance differs by approximately 1–2 percentage points. We will include this disaggregated comparison in the revised manuscript and clarify that the 81× ratio refers specifically to downstream declassifier inference cost, rather than total pipeline cost.
Regarding computational overhead, we agree that detailed end-to-end cost analysis would be valuable for practical deployment. Our current work characterizes GIF's computational requirements, while further implementation optimization and deployment-level benchmarking are natural directions for future work.
Regarding downstream evidence-selection controls, we thank the reviewer for this constructive suggestion. A systematic comparison of alternative evidence selectors under a fixed declassifier would provide additional insight into their relative effectiveness in downstream policy decisions. While our current evaluation focuses on information-flow measurement and demonstrating its applicability to semantic declassification, we agree that exploring alternative evidence-selection and policy-decision mechanisms is a valuable direction for future work.
We thank the reviewer for their thoughtful assessment and recognition of GIF's theoretical and practical contributions. We address both concerns below.
Q1: GPT-5.5 comparison and computational cost
We clarify that the 81× figure explicitly refers to downstream model inference cost, not total pipeline cost, as stated in Section 4.2. It illustrates that GIF enables a substantially smaller declassifier (Qwen3-8B) to achieve detection performance comparable to GPT-5.5 xhigh, rather than claiming an 81× end-to-end cost reduction.
We acknowledge that the cross-model comparison uses different trajectory populations and is therefore illustrative rather than a strictly paired comparison. Importantly, Figure 2 also provides same-model, same-trajectory comparisons, demonstrating that GIF-guided declassification preserves or improves policy-decision quality while reducing downstream token consumption by 59–73% on AgentDojo and MSB.
GIF's computational overhead is modest, requiring a single backward pass per analyzed token through the model being analyzed. Only a small number of output tokens need to be analyzed, corresponding to security-relevant actions such as generated tool-call names and parameters. Moreover, Figure 3 demonstrates that a lightweight 0.6B surrogate preserves downstream detection performance even for trajectory-generating models up to 200× larger, enabling efficient information-flow analysis even for substantially larger trajectory-generating models.
Q2: Incremental contribution and downstream evidence-selection baselines
We acknowledge that Liu et al. (2026) also employ the Fisher pullback and its trace for token-level attribution. However, GIF's primary contribution is not the Fisher pullback itself, but the information-theoretic framework and formal guarantees constructed around it.
Specifically, we introduce a local Gaussian surrogate channel whose KL geometry (Section 2.2) matches that of the autoregressive model, yielding an explicit mutual-information measure: CapT(τ;x)=21logdet(I+τMT(x)).
We establish direction-uniform second-order faithfulness (Theorem 2.2) and prove that this measure upper-bounds true perturbation-to-output mutual information to first order (Theorem 2.3): I(UT;Y)≤CapT(τ;x)+o(τ).
We further develop scalable matrix-free estimation using Hutch++ and connect its estimation guarantees to our soundness theorem. Neither the information-channel construction nor these soundness results are provided by Liu et al., whose objective is token-level attribution rather than quantitative information-flow semantics. We already cite Liu et al. as closely related work and will further clarify these distinctions in the revised paper.
Regarding downstream baselines, our evaluation deliberately separates attribution quality (Section 4.1) from policy-decision quality (Section 4.2). Section 4.1 directly compares GIF against GIF−, RTBAS-LLA, BM25, and random selection without an LLM judge, isolating the attribution signal. Section 4.2 establishes that GIF-selected evidence supports accurate semantic policy decisions with substantially reduced context.
Evaluating every alternative selector with the same downstream judge would constitute an additional end-to-end systems comparison. Our claims concern GIF's information-flow measurement and its effectiveness in supporting downstream policy decisions, not superiority over every possible evidence-selection pipeline.