acceptodds
Under review as a conference paper at ICLR 2027

Text-Enhanced Mask Probability Optimization for Long-Horizon Span-to-Span Attribution

Abstract

Auditing language models that answer from text far earlier in the sequence requires tracing a generated span to the earlier text it depends on, which may lie in a long prompt or in the response itself, far from the span being explained. We formulate this as long-horizon span-to-span attribution: the effect of intervening on selected source spans on a fixed later target span, with all other recorded text held unchanged. Propagation-based methods score sources from internal computation and need differentiable or architecture-specific access, while perturbation-based methods measure the target directly but face a combinatorial search whose cost grows with the source. We propose TEMPO (Text-Enhanced Mask Probability Optimization), a black-box method that optimizes a probabilistic mask over source units against the target's likelihood under sampled interventions and decodes the result into a complete ranking across deletion budgets. The decoder uses word and clause structure together with the learned mask probabilities, preserves the best searched set at the search budget, and issues no further model queries, so the number of forward passes is fixed regardless of input length or source–target distance. On four task families spanning retrieval, multi-hop reasoning, mathematical continuation with injected errors and counterfactual multiple choice, TEMPO attains the highest deletion AUC on every task and leads all macro-average faithfulness metrics against seven baselines, at lower cost than the strongest of them. The experiments also show that faithfulness and recovery of annotated evidence diverge: the recall leader changes by task, and a query-matching variant of our pipeline raises reference recall while lowering deletion AUC, indicating that behavioral dependence and annotation agreement must be evaluated separately.

Primary area:
interpretability and explainable AI
Keywords:
Attribution, Interpretation, LLM

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_iw3yOpenReview ↗
Rating
3(+1)
Confidence
3(+1)
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies long-horizon span-to-span attribution, aiming to identify which earlier text influences a later generated span. The authors propose TEMPO, a black-box attribution method that uses probabilistic mask optimization to identify influential source tokens and then combines the learned mask probabilities with word and clause structure to produce a complete attribution ranking without additional model queries. Experiments on four task families compare TEMPO with seven attribution baselines in terms of deletion faithfulness, computational efficiency, and reference-evidence recovery.

Critical strengths

(i) The paper focuses on the problem of tracing how earlier text influences later generated content, which is important for understanding and evaluating language model behavior in long-context and multi-step generation.

(ii) The experiments cover four task families and compare seven attribution baselines, with additional analyses of computational efficiency and decoder design. The paper also evaluates deletion faithfulness and reference-evidence recovery separately, providing a clearer picture of what the attribution methods actually capture.

Critical weaknesses

(i) The experiments do not fully cover the long-horizon span-to-span attribution setting emphasized in the paper. Section 3.1 defines attribution by intervening on earlier source text while keeping the intermediate text and target fixed. This measures dependence under a fixed continuation, but does not capture how changes to the source might affect the target by changing intermediate generated reasoning. Moreover, the main experiments primarily attribute input or supplied reasoning prefixes to generated answers or entire continuations. The paper does not directly evaluate attribution between distant spans within a model's own generated reasoning, nor analyze how attribution performance changes with source–target distance.

(ii) Section 2 does not clearly distinguish the methodological contribution of TEMPO from existing approaches. The paper reviews prior work on mask optimization and hierarchical text explanations, and describes TEMPO as connecting these two directions. It identifies the main difference from TRACE as converting a searched mask into a complete ranking without additional model queries. However, the discussion does not explain how the proposed word- and clause-based decoding differs from earlier methods that use linguistic structure for attribution. Although the paper demonstrates improvements over probability-based ordering, it is unclear what new methodological contribution TEMPO provides beyond adapting mask search and using linguistic structure to organize the resulting ranking.

Questions for authors

(i) Please clarify how the current experiments support the claimed long-horizon span-to-span attribution setting, particularly attribution between distant spans within model-generated reasoning.

(ii) Please explain how TEMPO differs from existing methods that combine mask optimization or linguistic structure for attribution, and what additional methodological contribution the proposed combination provides.

After Rebuttal

4 comments
AuthorsOpenReview ↗

Response to Reviewer iw3y

Thank you for recognizing the importance of the problem and our separate evaluation of faithfulness and reference-evidence recovery.

W1 / Q1: Token attribution, evidence recovery, and regeneration

The distinction is between model influence under an intervention and textual evidence supporting an answer. Our token attribution measures how replacing selected source tokens changes the likelihood of the same recorded target. Reference-evidence recovery measures agreement with supporting spans. These questions need not have the same answer, as our experiments demonstrate.

Regenerating intermediate reasoning changes the conditioning history and may produce a different answer. Evidence supporting that new answer does not itself measure the source's influence on the original target. Quantifying such influence requires a separately defined outcome and comparison criterion. Our protocol holds the recorded text and target fixed so that all interventions share the same target score, while recomputing hidden states (Section 3.1; Appendix A.1).

The experiments evaluate earlier prompt text or supplied reasoning prefixes against later generated targets; generated-reasoning-to-generated-reasoning attribution and a distance sweep are not separately evaluated.

W2 / Q2: The specific methodological contribution

The contribution is the conversion from fixed-budget mask search to a more faithful complete ranking at essentially no additional cost. We adapt TRACE's visual mask search to text units, position-preserving padding, and a fixed-target necessity objective. The decoder splits word/clause bundles at the searched-set boundary and orders their parts using proposal density. It uses text structure to decode existing search artifacts, preserving the searched set at 20% without additional model scoring.

The same-search ablation directly evaluates this contribution: compared with the anchor-preserving probability ordering, TEMPO increases AUC from 8.54 to 9.50 (+11.2%), with gains on all four tasks, using zero additional model queries and approximately 9 ms of CPU time. The BM25 query-matching control, which releases the anchor constraint, instead lowers AUC to 7.51 (-12.0% versus probability ordering) despite improving evidence recovery. Thus, the benefit comes from the particular decoding procedure; adding text relevance does not automatically improve attribution faithfulness. These results are in Appendix C.8, Table 16.

Following Reviewer 1's aggregation question, the revised means give each eligible input equal weight. The original equal-task comparison was 9.00 to 10.00; task-level results are unchanged. Appendix F also reports relative within-task gains.

I appreciate the authors' detailed response and the additional explanations and experimental evidence. However, my main concern about the experimental evaluation remains. Although the fixed-target attribution setting is now better explained, the experiments still do not directly evaluate attribution between distant spans within a model's own generated reasoning or examine how performance changes with source–target distance. These are important aspects of the long-horizon span-to-span attribution setting emphasized in the paper. I appreciate the authors' efforts to address my concerns, but given the remaining gap between the setting emphasized in the paper and the experimental evaluation, I will maintain my original rating.

AuthorsOpenReview ↗

Clarifications and revisions regarding "long-horizon."

Thank you for clarifying this concern. We have revised the terminology to state our contribution more precisely.

1. Origin of the framing and evaluation choices.

Our use of "long-horizon" followed the framing of FlashTrace (ICML 2026), which we include as a baseline. Our evaluation builds on its retrieval, multi-hop reasoning, and code-generation task settings, while adding misleading-prefix Math and counterfactual MCQ diagnostics. These additions probe error-origin and decisive-value dependence and broaden the comparison between intervention faithfulness and reference-evidence recovery.

2. Targeted terminology revision and the core contribution.

Before this revision, excluding the bibliography, "long-horizon" appeared seven times: six as a modifier in "long-horizon span-to-span attribution" and once as a paragraph heading. We have removed all seven occurrences and reworded one additional sentence using "long horizons". The reference to FlashTrace retains its original published title. Our revised title is "Text-Enhanced Mask Probability Optimization for Span-to-Span Attribution."

Thus, the revision primarily removes a modifier from the span-to-span formulation. The core contribution remains black-box span-to-span attribution through probabilistic mask optimization and query-free text-enhanced decoding into a complete ranking. Source and target spans may be nonadjacent within the model's context window; their selection and the fixed-target intervention objective define the attribution problem.

Section 4.1 now states the standard token-attribution setting explicitly: attributing a fixed generated target to preceding source tokens, with rankings evaluated under a common token-replacement protocol. The revised wording avoids implying a dedicated evaluation of distance-dependent performance. These changes are limited to the title and explanatory text; the formulation, algorithm, experiments, and numerical results are unchanged.

I appreciate the authors' further clarification and the revisions to the terminology. I think these changes are reasonable and help better align the paper's claims with its actual contributions and experimental scope. Given these revisions and the authors' previous responses, I believe my main concerns have been mostly addressed. I raise my rating and would be glad to see the paper accepted.

Reviewer_x6YiOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
3(+1)
Read review
Paper summary

The paper proposes TEMPO (Text-Enhanced Mask Probability Optimization) and formulates long-horizon span-to-span attribution as intervening on source text while holding the surrounding context and target output fixed, then measuring the resulting change in the target span’s mean log-probability. Inspired by TRACE, TEMPO first performs probabilistic mask search using Gumbel sampling and the cross-entropy method at a 20% deletion budget, producing a best set A0A_0 and unit-level probabilities. It then uses word boundaries, clause structure, and probability density to decode a ranking without additional model queries. This yields a nested ranking across deletion budgets while preserving A0A_0 as the prefix at the 20% budget. The experiments compare TEMPO with seven baselines on four task families using Qwen3-8B: NIAH, multi-hop QA, mathematics problems with misleading prefixes, and counterfactual multiple-choice questions. TEMPO achieves the highest macro-average results on signed-drop AUC and Drop@10/20/30% in the main experiments. Finally, it shows that faithfulness and reference-evidence localization are not equivalent: TEMPO produces stronger deletion effects, while FlashTrace or AttriBoT achieves better evidence recall on some tasks; a BM25 query-matching control improves evidence recall but lowers deletion AUC.

Critical strengths
  1. The problem formulation and method implementation are clearly specified. The paper provides substantial technical detail, making it relatively straightforward for readers to reproduce the proposed method.
  2. The experimental setup is reasonably broad. In addition to the main results, the paper provides clear ablation analyses and examines the distinction between faithfulness and localization.
Critical weaknesses
  1. The claimed contributions may exceed what is substantiated by the paper. First, regarding methodological novelty, the paper states that the Search stage is inspired by TRACE but does not clearly explain the specific differences between the two methods. On comparison, the Search stage appears to closely follow TRACE, with nearly identical hyperparameter settings; it should therefore be understood as an adaptation of TRACE to text attribution. The methodological novelty of this paper appears to lie primarily in the second stage, Decoding. Accordingly, the improvement attributable to the proposed method should be assessed by the gain from Search Only (9.00) to TEMPO (10.00) in Appendix C.8, rather than by the comparison between AttriBoT (6.84) and TEMPO (10.00) in Table 1. Second, the paper’s first stated contribution is to “formulate long-horizon span-to-span attribution.” However, prior work has studied attribution between text spans. In the LongCite task [1], sufficiently long contexts can place a substantial distance between source and target spans. The paper should clarify the principal differences between its proposed task and the task introduced by LongCite.

Reference

[1] Zhang, J., Bai, Y., Lv, X., Gu, W., Liu, D., Zou, M., ... & Li, J. (2025, July). Longcite: Enabling llms to generate fine-grained citations in long-context qa. In Findings of the Association for Computational Linguistics: ACL 2025 (pp. 5098-5122).

  1. The experimental scope does not sufficiently validate the paper’s broad framing of span-to-span attribution. The task definition states that “The prefix may contain prompt text or earlier generated text, and the target may be a selected output span or the entire continuation.” However, all experiments use a single-turn “prompt → generated continuation” structure. Attribution in multi-turn dialogue is a challenging and worthwhile setting that the paper does not investigate; the paper should acknowledge this limitation to avoid giving a misleading impression of its scope. In addition, the paper does not report the context lengths of the datasets, making it difficult to assess whether the experiments substantively instantiate the proposed “long-horizon span-to-span attribution” task.
Questions for authors
  1. Could the authors clarify the specific algorithmic differences between TEMPO’s Search stage and TRACE?
  2. Could the authors clarify how their “long-horizon span-to-span attribution” task differs from LongCite, and report the context lengths and source–target distances in the experiments? Do any experiments evaluate attribution from an earlier generated span to a later output span, or do all of them use a single-turn prompt-to-continuation setup?
3 comments
AuthorsOpenReview ↗

Response to Reviewer x6Yi

Thank you for recognizing the clarity of the implementation and the breadth of the experiments. We clarify the methodological contribution and experimental scope below.

W1 / Q1: TRACE adaptation and the decoder's contribution

Section 3.2 explicitly credits TRACE for probabilistic mask search and CEM updates; we do not claim novelty in the inherited optimizer or hyperparameters. However, TEMPO's search is a text adaptation, not native TRACE:

  • Units and budget: TRACE searches fixed-cardinality image-superpixel masks. TEMPO searches text units under token-cost constraints, including indivisible multi-token atoms.
  • Intervention and objective: TRACE uses image masking and, in its classification setting, visual sufficiency. TEMPO uses position-preserving padding and maximizes the decrease in a fixed target's mean log-probability, with intervening text fixed.
  • Complete ranking: TRACE uses additional Greedy or PhaseWin search to extend a mask into a full ranking. TEMPO decodes the searched set and proposal weights using word/clause structure without further model queries.

Thus, Search Only is an ablation of TEMPO's text-adapted pipeline, not unchanged visual TRACE. Table 1 evaluates the complete method against external baselines; Appendix C.8 isolates the decoder using the same search output.

The reviewer's 9.00 → 10.00 comparison is precisely this decoder ablation under the original equal-task summary. Following Reviewer bZ6R's question about aggregate weighting and task scales, we now give each eligible sample equal weight. This changes the summary to 8.54 → 9.50 (+11.2%), without changing task-level results. The aggregate is a convenient summary; Appendix F also reports relative within-task gains because sample weighting does not remove scale differences.

The decoder improves AUC on all four tasks: 4.4% on NIAH, 3.7% on multi-hop, 1.5% on Math, and 22.3% on MCQ. It takes approximately 9 ms on the CPU and zero additional model queries, while preserving the searched set at 20%.

The BM25 query-matching control reuses the same search output but releases the anchor constraint. It lowers pooled AUC to 7.51, a 12.0% decrease from Search Only and 20.9% below TEMPO, despite improving evidence recovery. The decrease occurs on every task. These existing comparisons show that adding text-based ordering does not automatically improve faithfulness; the specific decoder supplies a measurable gain at negligible additional cost.

W1 / Q2: Distinction from LongCite

We have added LongCite to Appendix G. LongCite generates answers with citations to supporting context snippets. Our task instead fixes a recorded target and measures how source-token interventions change its likelihood, producing a ranking evaluated through nested interventions.

Supporting evidence and model dependence are different relations: a passage can support an answer without being the passage on which the model's output most strongly depends. This distinction is demonstrated by our faithfulness-versus-localization analysis. Our formulation specifies source and target selection, fixed intervening text, and the intervention score; we do not claim that relating earlier and later text spans is itself new. Consistent with the approach used by the paper FlashTrace (ICML2026).

W2 / Q2: Evaluated scope and lengths

Every experiment attributes earlier text to a later generated target using a prompt/prefix-to-continuation setup. The Math prefix is supplied as input. The experiments do not separately evaluate sources sampled from the model's own earlier generation, multi-turn dialogue, or a source–target-distance sweep. These are supported source/target configurations of the formulation, beyond the present empirical coverage.

Appendix B.1 reports generation limits of 64/96/1,024/32 tokens for NIAH/multi-hop/Math/MCQ and 2,048 for code. Appendix D specifies NIAH needle placement at 10–90% of document length. These are generation and construction settings; they are not measured full-context lengths or aggregate source–target distances, which are not reported in the current manuscript.

Our evaluation builds on FlashTrace's retrieval, multi-hop, and code-generation settings. Code evaluation follows FlashTrace's protocol, applied to our Exercism Python pool. Our additions are the misleading-prefix Math and counterfactual MCQ diagnostics, designed to probe error-origin and decisive-value dependence. Their contribution is these controlled attribution settings and annotations, not the introduction of mathematical reasoning as a task.

References

Gu et al. (2026). A Good Initialization Is All You Need for Faithful Visual Attribution.

Zhang et al. (2025). LongCite: Enabling LLMs to Generate Fine-grained Citations in Long-Context QA. 、

Pan et al. (2026). Towards Long-Horizon Interpretability: Efficient and Faithful Multi-Token Attribution for Reasoning LLMs.

Thank you for your response. The authors’ clarifications address most of my concerns. My remaining concern is the adapted search stage appears to account for most of TEMPO’s gains over the external baselines, while the proposed decoder yields comparatively modest improvements over the search-only variant on most tasks. I therefore raise my soundness score and reaffirm my overall rating.

Thank you for your time and consideration. Regarding your current concerns, our explanation is as follows: the search method itself is a contribution of ours. Since the search process accounts for the entirety of the computational cost, it is only logical that it yields greater benefits; meanwhile, the cost of decoding—in terms of CPU time—is virtually negligible (on the order of milliseconds).

We present TEMPO as a single two-stage method. Its first stage adapts probabilistic-mask search from TRACE’s setting of 64 image superpixels to span-to-span token attribution. This adaptation is part of our contribution. The search incurs the model-query cost and accounts for most of the gain over external baselines on most tasks. The second stage uses textual priors to turn that same search output into a ranking, with zero additional model queries and negligible additional runtime. Given these different computational costs and roles, it is reasonable for the two stages to yield gains of different magnitudes. The search-only ablation measures the decoder’s incremental value within TEMPO. Our contribution claim concerns the combined method.

Reviewer_bZ6ROpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors of the paper generalize the problem of input attribution. From finding the inputs that contribute to the generation of a particular output; to any text that appeared before the portion of the output the user is interested in (possibly reasoning tokens as well). The approach they propose falls under the category of perturbation based methods which perturb the input to see how it influences the output. If there are n input tokens, a naive implementation would take 2^n forward passes which is infeasible and therefore methods in this family attempt to minimize the number of forward passes required to estimate the influence of input perturbations. The specific problem they study is: how to get an ordering over all input tokens such that replacing the top k tokens, at any fraction k, with padding tokens leads to the highest drop in probability of the selected output tokens. Their method TEMPO has two parts — first finding a good set of tokens at a budget 20% of tokens through efficient mask search influenced by another work TRACE and the second is using word/clause boundaries to obtain an ordering over all tokens without additional model queries. They find that their method has the highest faithfulness (faithful to the objective they consider) scores when they compare against 7 other baselines on 4 task families for Qwen3-8B. They also find that faithfulness is not aligned with the annotated evidence.

Critical strengths
  1. The analysis showing that the annotation type differs across tasks, and determines the method that best aligns with annotated evidence, was interesting. On tasks like multi-hop and needle in a haystack where annotation is sentence-level, methods that score like tokens similarly do well. While, in math and MCQ where the answer is often dependent on a local span, AttriBot that checks the effect of leaving a token out does well. This finding possibly implies that different families of methods might be needed for different tasks based on the expected granularity of attribution, and should guide future evals for attribution.
Critical weaknesses
  1. The evaluation metric and what the proposed approach TEMPO optimizes for are similar: the drop in probability of selected output tokens when some input tokens are replaced with a padding token (difference being evaluation is over multiple budgets). In contrast, propagation-based methods simply assign an importance score to input tokens and don’t account for the deletion metric (i.e. they are not designed for the paper’s faithfulness evaluation metric). Similarly, CAMAB was designed for identifying which sentences when included best explain the output, and is different from the paper’s baseline implementation of operating at a token-level and padding the unselected tokens. So while the results make it seem like TEMPO outperforms baselines on faithfulness, the comparison is not exactly fair. Additionally, I am not convinced that drop in output token probability on selected input tokens being replaced with padding is a useful objective. While I find the sentence/passage level granularity used in prior work well motivated — it asks the counterfactual question: what would happen if these sentences were not present. The replacement of tokens with padding results in inputs unlike those the model encounters in practice.

  2. The major contribution of the paper seems to be the second step of TEMPO, which uses the best set of tokens found using the search step to construct a complete ordering over tokens with the help of word/clause boundary priors — the first step being largely similar to TRACE and swapping the objective. I don’t think this contribution is enough especially given that the baselines are all beaten by the search-only step without the decoder using AUC metric on all task families (table 16 and table 1 together). The macro AUC gains largely come from MCQ only, ~22% task-level gain compared to < ~4% for the other categories. D@10 has a CI that includes 0 and D@30 is 0.22[.03, .44] which is small.

Questions for authors

I don't agree with the decision to equally weight the 4 task families for the Macro AUC because they have different scales; for example Math and MCQ. I am curious to know what the magnitude of results look like with a scale-free alternative such as relative per-task gain.

4 comments
AuthorsOpenReview ↗

Response to Reviewer bZ6R I

Thank you for recognizing the analysis of task-dependent agreement with reference evidence. We address the intervention protocol, the decoder's contribution, and the aggregation question below. The manuscript changes are targeted and marked in blue; detailed clarifications are in the appendix. All numerical comparisons use existing results.

W1: Intervention protocol and comparison across attribution methods

Token replacement and sentence removal ask different counterfactual questions. Our task ranks source tokens by their effect on a fixed target under position-preserving replacement. Padding changes selected token identities while retaining sequence length, position identifiers, and attention visibility; physically deleting and repacking tokens also changes positional relationships. Keeping the recorded intervening tokens fixed allows the same target to be evaluated under teacher forcing, with hidden states recomputed. Appendix A.1 now explains these choices. Reference-evidence coverage is a separate criterion: an annotation may cover a whole expression even when disrupting a few tokens substantially changes its effect on the target.

The alignment between TEMPO's objective and deletion faithfulness is deliberate. This behavioral criterion can evaluate both perturbation-based and internal-score estimators. Propagation methods also use target information and perturbation evaluation: FlashTrace's deletion test explicitly allows replacement by a padding token (Appendix F of FlashTrace). DePASS (NeurIPS 2025) projects decomposed states onto the target's LM-head direction (Eq. 17) and compares heuristic MLP allocation rules using removal-based comprehensiveness and sufficiency (Appendix A). We have added this discussion to Appendix G. The comparison measures faithfulness under the declared intervention, with attribution costs reported alongside it.

CAMAB is explicitly identified as an adapted implementation in Appendix B.2. Its original formulation permits units from tokens to paragraphs (Section 2.1 of CAMAB). We retain its linear Thompson-sampling estimator and fixed-response log-probability objective, using token retention and padding of unselected positions under the common protocol. Thus the comparison evaluates this estimator in our token-level setting. AttriBoT likewise uses its prefix-cached token leave-one-out component. The implementation details distinguish these adaptations from the original sentence-level evaluations.

W2: Contribution beyond mask search

We credit TRACE for the probabilistic-mask search idea. Our algorithmic contribution is the query-free text-enhanced decoder, which converts the searched set and proposal weights into a complete, budget-nested token ordering while preserving the searched set at 20%. The decoder uses word/clause structure, takes approximately 9 ms on the CPU, and requires zero additional model queries. Fixed-budget subset search alone leaves the ordering within and beyond the selected set unresolved.

The existing same-search ablation isolates this contribution: all decoding variants reuse the same search output. TEMPO's decoder improves AUC in all four tasks, by 4.4% on NIAH, 3.7% on multi-hop, 1.5% on Math, and 22.3% on MCQ. The conventional query-matching decoder substantially reduces faithfulness:

Decoding of the same search outputPooled AUCChange vs. search only
Probability ordering (search only)8.54Reference
BM25 query-matching control7.51-12.0%
TEMPO text-enhanced decoding9.50+11.2%

The query-matching control lowers AUC on every task and is 20.9% below TEMPO overall, despite improving reference recall. This comparison demonstrates that a conventional text-based decoding rule can degrade the search result substantially; the gains depend on how the search output is decoded. TEMPO improves the complete curve while keeping D@20 identical to search only by construction. These existing comparisons are reported in Table 16 and Appendix C.8, where we have made the control's decrease explicit.

The paper also contributes the span-to-span formulation, the analysis separating intervention faithfulness from evidence recovery, and four attribution datasets with reference annotations. Math and counterfactual MCQ are self-constructed diagnostics. The contribution statement and planned public release have been clarified concisely.

References

Hong et al. (2025). DePass: Unified Feature Attributing by Simple Decomposed Forward Pass. NeurIPS 2025. Eq. 17 and Appendix A.

Pan et al. (2026). Towards Long-Horizon Interpretability: Efficient and Faithful Multi-Token Attribution for Reasoning LLMs. Appendix F.

Pan et al. (2026). Context Attribution with Multi-Armed Bandit Optimization. Findings of ACL 2026. Section 2.1.

AuthorsOpenReview ↗

Response to Reviewer bZ6R II

Question: Aggregation and scale-free effect sizes

Thank you for drawing attention to the aggregation and task-scale issue. We have changed the primary aggregate from an equal-task macro to a pooled sample mean:

qpool=∑tnt qt∑tnt.q_{\mathrm{pool}}=\frac{\sum_t n_t\,q_t}{\sum_t n_t}.

Here ntn_t counts inputs eligible for the relevant metric. For faithfulness, the counts are 179, 51, 79, and 94 across NIAH, multi-hop, Math, and MCQ (403 total). Section 4.2 now explains that an aggregate is a convenient descriptive summary; the taskwise comparisons establish consistency. Figure 1 retains its original macro illustration, identified as an example in the caption. Sample weighting does not remove task-scale differences, so Appendix F also reports relative within-task AUC gains over the strongest external baseline:

TaskTEMPO AUCStrongest baseline (AUC)Relative gain
NIAH8.00DePASS (7.06)13.3%
Multi-hop14.04IG (12.63)11.2%
Math0.629DePASS (0.605)4.0%
MCQ17.33AttriBoT (11.08)56.4%

TEMPO leads on every task. The mean of these dimensionless relative gains is 21.2%; the pooled AUC is approximately 9.50 versus 6.80 for AttriBoT, the strongest pooled baseline, a 39.7% relative gain. Table 19 provides these taskwise comparisons. The reaggregations use the existing rounded task means; Appendix F specifies the cohorts and distinguishes the pooled summaries from the original task-balanced statistics.

I appreciate the authors’ response, particularly the addition of scale-free per-task relative gains. However, my two primary concerns remain:

  1. Intervention protocol. While you agree that “Token replacement and sentence removal ask different counterfactual questions”, what remains unanswered is why replacing tokens with padding tokens is a good evaluation metric for any task that a user would care about. Sentence removal asks the counterfactual “what would happen if these sentences were not present”, which does map to a reasonable question. I can’t say the same about your replacement strategy, because the perturbed text no longer resembles something natural for both users and the model. The constraints that are mentioned, such as “Keeping the recorded intervening tokens fixed allows the same target to be evaluated under teacher forcing”, are implementation advantages of your strategy, but the metric still has to be proven to correlate with something real. You also call the alignment between TEMPO’s objective and the evaluation metric deliberate: while the baselines may be evaluated with the same metric in their own papers, none of them is optimized for it, and they perform the way you measured out of the box. If you have optimized for some evaluation metric, it either has to be shown that the metric is correlated with some useful downstream user experience, or you have to show results on other metrics, such as sufficiency (which DePASS itself reports).

  2. Contribution beyond mask search.

TEMPO's decoder improves AUC in all four tasks, by 4.4% on NIAH, 3.7% on multi-hop, 1.5% on Math, and 22.3% on MCQ

I am aware of these numbers and have previously mentioned that the gains seem to come from MCQ primarily.

The conventional query-matching decoder substantially reduces faithfulness

Fixed-budget subset search leaves the ordering unresolved

My concern was that the search-only baseline, or the first step, already beats the other baselines, and thus the contribution of the second step (where large gains are only in MCQ) is limited. The fact that conventional query-matching is worse than the second step is irrelevant, since it is also worse than the search-only baseline.

I will maintain my original rating.

AuthorsOpenReview ↗

Further Clarification on Baselines and Details Regarding Ablation Studies

Thank you for reading the revision and for your feedback on the per-task comparisons. We would like to clarify the choice of intervention and the contribution of each stage of TEMPO.

  1. Intervention protocol.

Your sentence-removal example asks how the model responds when an entire statement is absent. For token attribution, the question is finer: which parts of the source affect the score the model assigns to the particular words it generated? We replace selected source tokens with padding so that the surrounding text, its positions, and the target stay fixed. This lets us examine dependence within a statement, including for generated spans with no predefined answer choices. Section 4.1, lines 309–312, specifies this protocol; FlashTrace also uses token replacement in its deletion evaluation. We interpret the score as sensitivity under that intervention, not as a prediction of how a user would edit the text. The interventions and objectives you mentioned are suited for a different category of interpretability tasks—specifically, those involving the attribution of evidence to a set of discrete, predefined outcomes. Examples include the fine-grained image classification task (upon which the "trace" approach relies) or tasks requiring selection based on evidence (as in the case of CAMAB). In our specific task setting, however, there is no such finite set of target outcomes; therefore, those approaches are not applicable.

As you note, TEMPO optimizes a deletion score, while several baselines were designed for other objectives. Using target responses to score perturbed inputs is also part of RISE and CAMAB. The computational question for TEMPO is how to find an influential subset among exponentially many masks with a limited number of queries. We spend those queries at one deletion budget (20%); the ranking at other budgets must be constructed from the same search output.

The sufficiency test you suggest, as reported by DePASS, asks how well the retained tokens support the target on their own. For a distant generated span, restoring only a few source tokens can leave too little context to distinguish their contributions. Deletion starts from the recorded context in which the target was generated, which is why we use it here. Gu et al. discuss this difficulty with output-level recovery in Text Attribution as Continuous Trajectories (Findings of EMNLP 2026).

  1. Contribution beyond mask search.

TEMPO builds on TRACE's probabilistic search over image regions. Our first stage adapts that search to text units with token costs and scores their effect on a fixed generated span. The search-only result shows what this text adaptation achieves before decoding.

We agree with you that search-only is the right comparison for measuring what the decoder adds. Both variants use exactly the same searched set and proposal weights. The BM25 control serves a separate purpose: it shows how favoring query-matching text can increase evidence recall while reducing deletion AUC.

Against search-only, the decoder improves deletion AUC on all four tasks without an additional model query. The gain is especially pronounced on MCQ (+22.3%), where the answer turns on discrete values. The gains on NIAH, multi-hop QA, and Math are +4.4%, +3.7%, and +1.5%, respectively (Table 19). With each eligible input weighted equally, pooled AUC rises from 8.54 to 9.50 (Table 16).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.