- Rating
- 2
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes Counterfactual Influence Distillation (CID), a method that uses LoRA adapters trained with GRPO to predict input attribution scores that reflect the importance of different input elements to an LLM's outputs. The authors propose using pre-computed labels derived from costly leave-one-out estimation (LOO, requiring one model call per label) as targets for reinforcement learning from verifiable rewards. At inference, CID masks top-ranked inputs and replaces their scores with measured LOO values, finally rescaling all remaining scores.
CID is evaluated using correlation between scores predicted by the learned model and LOO over all inputs, and compared with ContextCite and SelfCite for Qwen3.5-9B and Gemma-4-12B. The authors show that 1) with a little number of masks, CID outperforms ContextCite-64 on every tested task; and 2) training helps most when few masks are used, and it adds little for Gemma-4-12B.
Critical strengths
- The paper controls well for possible confounds by applying the same verification step to prompted, random, lexical and CID rankings, making differences between methods depend only on the set of masked inputs selected by each method. The authors report that the verified prompted report already beats ContextCite-64 within four masks, so most of the fidelity gains can be associated with the masking itself rather than training. Even against this stricter reference, CID outperforms other methods with two masks on all tasks for Qwen3.5-9B, suggesting that ad-hoc training can further improve the identification of salient inputs.
Critical weaknesses
-
Most of the reported gain comes from verification rather than the added training procedure, and the ContextCite comparison does not show a meaningful efficiency gain. DFS contexts have at most 23 inputs, so ContextCite-64 costs more than exact LOO for each task, and the statement "ten times fewer calls than ContextCite" is misleading at this scale. On DFS-R and DFS-A (the tasks where masking saves calls over exact LOO), CID beats the prompted report by a very small margin. The current evidence seem to support a modest benefit of training at small budgets, but is not sufficient to justify the broader claim that CID makes faithful attribution cheap enough to enable its use at each step of agentic execution.
-
As pointed out in the limitations of Appendix G, fidelity scores may not reflect the decision the agent actually makes, since the labels are computed on a different prompt from the one the model uses to decide its answer (without the chat template, as shown in Appendix H), and on that prompt the model rarely gives its own answer. When the model writes its report, it also sometimes gives a different answer from the one being explained (authors report that this happens about half the time for the prompted model on DFS-R and DFS-A). CID’s advantage may therefore come from giving the expected answer more often rather than simply ranking inputs better.
Questions for authors
- What is the fidelity of the prompted and CID reports on the examples where both reports’ decisions match the gold one? This would help me understand how much of CID’s gain over the prompted report comes from better ranking rather than simply preserving the original decision.
Author's response to Reviewer L6Qr
Thank you for your feedback. We answer both questions with new controls.
W1 — Most of the gain comes from verification; the efficiency claim.
Verification is half of CID by design; learning decides where it measures. section 5.2 states that most of the fidelity at four or more masks comes from masking itself, and the abstract will say so. Verification makes any report exact on the inputs it masks; learning decides which inputs those are, and therefore how many masks a target fidelity costs. That matters most when masks are scarce, which is when attribution is cheap enough to run at every step. With intervals, CID minus the prompted report at B=0 and 1 is +0.164 and +0.125 on Qwen DFS-0, and +0.197 and +0.149 on DFS-1. On the real-text tasks the gain is smaller but significant at B≤2 (DFS-R +0.032 [0.018, 0.045] and DFS-A +0.024 [0.007, 0.043] at B=2). Trained Qwen reaches CC-64's fidelity with fewer masks than the prompted report (point estimates): 0 vs 2 on DFS-1, 1 vs 2 on DFS-R. The gain reproduces on a freshly generated DFS-0 split never used in development, with matched configurations and two seeds: +0.157 [0.131, 0.184] at B=0 and +0.067 [0.047, 0.086] at B=2 for the Rank reward, against +0.065 and +0.014 for SFT on the same data (our response to Pyj9, part 3).
Efficiency. The call comparison against CC-64 is accurate, and with ≤23 inputs exact LOO is the tighter reference. The saving holds against it too, in tokens:
The saving grows with the number of inputs: at 100 paragraphs, exact LOO needs 101 scoring calls and CID with four masks one generation and 5. The revised abstract states the saving against both references.
W2 / Q — Does CID win by keeping the expected answer? This is the right control, and we computed it. On the examples where both the prompted and the CID report give the labeled decision y⋆, CID minus prompted is:
The gain persists, at about the same size as on all examples (+0.164 / +0.075, +0.197 / +0.058, +0.029 / +0.032, +0.075 / +0.024). It holds when both answers match the gold answer too: +0.159 and +0.184 at B=0 on DFS-0 and DFS-1. So CID's advantage comes from better ranking, not from preserving the decision. For Gemma on the paper's labels, the gain on DFS-1 persists on its 278 decision-matched examples (+0.026 [0.011, 0.041] at B=2), while the Gemma DFS-R and DFS-A subsets are small (37 and 86 examples) and not significant.
The direct test: both reports must give y⋆. We evaluated decision-conditioned reports, whose answer field is fixed to y⋆, so that report, verification and labels explain the same decision. With the decision held equal by construction, CID still beats the identically conditioned prompted report at B=0 on every Qwen task. With the chat template used throughout, the margins are +0.171 [0.148, 0.195], +0.113 [0.082, 0.143], +0.044 [0.025, 0.063] and +0.097 [0.073, 0.120] on DFS-0/1/R/A; under the paper's labels they are +0.193, +0.203, +0.029 and +0.089, all significant. Without any masks, conditioned CID also exceeds CC-64 on DFS-1, DFS-R and DFS-A under the chat template, and on DFS-0 and DFS-1 under the paper's labels (0.698 vs 0.667; 0.668 vs 0.595).
The scoring prompt. For Qwen3.5-9B we now evaluate the deployed tuple end to end under the chat template in which the decision is generated: labels, verification, and CC-64 rerun under the same prompt. CID never trained on these labels, which differ from the stored ones (ρ = 0.38–0.64), yet every conclusion holds:
Details are in our response to Pyj9, part 2.
The decision-matched controls show that CID's advantage comes from better ranking, not from keeping the answer, and the token counts show that the saving holds against exact LOO. We would be grateful if you would reconsider your score.