acceptodds
Under review as a conference paper at ICLR 2027

CASI: Credit Assignment via Set Intervention for Agentic Retrieval

Abstract

Large language models increasingly generate multiple retrieval actions jointly, while downstream systems often provide only one reward for the resulting set. Broadcasting this reward provides a valid score-function gradient estimator for the expected set-level objective, but does not determine a coalition-dependent within-set attribution. We introduce **CASI (Credit Assignment via Set Intervention)**, which constructs the missing counterfactual information through snapshot-grounded set intervention: generate and retrieve once, hold replayable snapshot components fixed, and evaluate alternative subsets under common conditions. Shapley aggregation converts these marginals into within-set credit, which bounded zero-sum advantage shaping incorporates into policy optimization. For \(m \ge 4\), singleton, leave-one-out, and full-set endpoints do not generally determine the Shapley allocation; under additive set values, extra coalition evaluations are redundant. In a five-action sponsored-search setting, exhaustive auction replay over 1,200 requests reveals substantial coalition dependence. Under the training-time evaluator, CASI achieves the highest median cosine similarity to a replay-defined Shapley reference among evaluated credit rules; under a sensitivity backend, CASI does not outperform Singleton. Across five independent training runs, CASI-trained policies achieve higher held-out auction-replay set value than Singleton- or Broadcast-trained policies. In a randomized seven-day production A/B test on 10% of traffic, the CASI-trained policy yields an observed **3.51%** lift in the primary advertiser-value metric over production control.

Primary area:
reinforcement learning
Keywords:
Reinforcement Learning, Credit Assignment, Counterfactual Credit Assignment, Set-Valued Actions, Shapley Value

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_qw6YOpenReview ↗
Rating
2
Confidence
2
Informativeness
3
Presentation
2
Soundness
1
Read review
Paper summary

The paper studies credit assignment when an LLM jointly generates multiple retrieval actions, specifically in sponsored search, but receives feedback for the resulting set as a whole. It proposes CASI, which caches each retrieval action's results, evaluates alternative subsets under a fixed snapshot, and aggregates their marginal values into per-action Shapley credit. Bounded, zero-sum adjustments then use this credit to shape the actions’ training advantages. The paper shows that singleton, leave-one-out, and full-set evaluations generally do not determine Shapley credit for sets of at least four actions. Validation includes an auction-replay audit of subset interactions and credit alignment with Shapley reference, comparisons of policies trained with different credit rules, and an online A/B test in which the CASI-trained policy improves the reported advertiser-value metric.

Critical strengths
  1. The method - The paper addresses the challenge of assigning credit to individual retrieval actions when only their joint outcome is observed, and each action’s contribution depends on the others. A novel method is presented: snapshot-grounded set intervention evaluates different combinations of actions, and Shapley-based aggregation assigns these evaluations to action-level credit. This may provide a more informative training signal for the paper’s central problem.

  2. Evidence from a production deployment - The seven-day randomized A/B test reports increased advertiser value for the CASI-trained policy over the production control. Evaluating the policy in a deployed sponsored-search system is consequential because it provides evidence of practical value beyond the theoretical analysis and offline auction replay evaluation.

Critical weaknesses
  1. Insufficient experimental setting details - The paper omits essential details about the models used and SFT initialization, training and evaluation data, RL algorithm, and key hyperparameters. It also gives only a few concrete details about the definition of the evaluation measures (e.g., advertiser value), retrieval, and the auction production system, or the frozen evaluator used to score coalitions. These omissions make the method difficult to reproduce and limit the reader’s ability to assess the validity of the results and conclusions.

  2. Limited and unclear policy-baseline comparison - The held-out policy experiment trains Broadcast, Singleton, and CASI, but not leave-one-out or Banzhaf, although both are evaluated as credit assignment methods. The paper does not explain how hyperparameters were selected or whether the baselines received comparable tuning, making the policy comparison difficult to assess. Moreover, the SFT initialization scores 60.32, above Broadcast (43.3) and Singleton (55) in Mean replay value. The decline after training for both baselines raises a concrete question about whether they were trained effectively, and thus how much of CASI’s advantage is due to its credit-assignment method.

Questions for authors
  1. Section 3.5 compares methods by the number of evaluator scores, but does not report the actual effects on training. What is the training time and/or compute cost for CASI and each baseline? How do they differ?

  2. See weaknesses.

1 comment

Dear Reviewer qw6Y,

We hope you are doing well. We just wanted to briefly follow up and thank you again for your thoughtful feedback. We hope our responses and additional experimental results have helped clarify the points raised in your review.

If any part of our explanations or experimental details remains unclear, please feel free to leave a follow-up question. We would be very happy to provide further clarification.

Thank you again for your time and consideration!

Reviewer_Y1J6OpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies credit assignment when LLM generates several retrieval queries that share a downstream reward. It proposes CASI that reconstructs query subsets from cached retrievals, compute a shapley-based credit with a frozen evaluator, and incorporates it into policy training. Authors use sponsored-search experiments to examine coalition effects and report improvements in held-out replay value and a production A/B test.

Critical strengths

The paper addresses a practical problem, and the combination of repeated coalition replay, independent training runs, and production evaluation provides meaningful evidence of usefulness.

Critical weaknesses

The experimental description needs more detail to assess the learning results. I could not find the model identity, essential training settings, shaping parameter values, or an explicit numerical definition of the cached-filter evaluator in the paper and appendices. The evaluator is particularly important because it determines the subset values and hence the credits used for training. The sensitivity experiment also shows that changing this evaluator removes CASI’s credit-alignment advantage over Singleton, making its precise construction consequential to understanding the method.

The policy comparisons leave uncertainty about the benefit of the particular credit rule. LOO and Banzhaf are evaluated for agreement with a Shapley-based reference, but neither is included in the policy-training comparison. Agreement with that reference does not establish which credit rule produces better learning. Banzhaf would be especially informative because it uses the same subset evaluations with different aggregation weights, while LOO would test whether a cheaper interaction-aware signal suffices. The existing results establish an advantage over the tested policies, but do not resolve these comparisons.

Questions for authors

See above weaknesses.

(1) more information about model, training procedure, how settings were selected, and mathematical structure of evaluator helps

(2) Are policy level results available for LOO or Banzhaf under a comparable training setup?

Reviewer_1KsyOpenReview ↗
Rating
3
Confidence
1
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies credit assignment when an LLM jointly generates multiple retrieval actions but receives only a set-level reward. It proposes CASI, which evaluates cached action subsets under a frozen snapshot, aggregates contextual marginal contributions using Shapley values, and applies zero-sum action-level advantage shaping. The method is evaluated through replay audits, offline policy training, and a production sponsored-search A/B test.

Critical strengths
  • The paper clearly distinguishes optimizing a set-level objective from the harder problem of attributing credit within a set. Proposition 1 (p. 5) shows that singleton, leave-one-out, and full-set observations do not identify the Shapley allocation when m≥4m \ge 4, while the additive special case clarifies when additional coalition evaluations are unnecessary. This framing avoids treating a scalar rollout reward as sufficient action-level supervision.

  • The evaluation is concrete and appropriately cautious about the surrogate’s limits. Figure 2 and Tables 1–2 show substantial coalition interaction and quantify agreement with replay-derived credits. Table 3 reports results for five independently trained policies, and Section 4.5 adds a randomized production A/B test. The authors acknowledge that coalition-rerun sensitivity does not favor CASI and distinguish replay outcomes from production counterfactual utility, strengthening the empirical interpretation.

Critical weaknesses
  • The policy-level results do not isolate whether the improvement comes from Shapley aggregation, access to more coalition values, or the particular cached-filter evaluator. This distinction matters because the paper’s practical contribution is not simply coalition evaluation, but the use of Shapley-based CASI credits.

  • Section 3.5 states that exact CASI and Banzhaf use the same 2m−1=312^m-1=31 evaluator scores when m=5m=5, and Table 2 shows that Banzhaf is a close coalition-aware comparator in the credit-alignment audit. Yet Table 3 trains policies only with Broadcast, Singleton, and CASI; it includes no Banzhaf-trained policy. In addition, under the coalition-rerun sensitivity evaluator, CASI performs slightly worse than Singleton on the primary alignment metric (Section 4.3 and Appendix G), suggesting that its benefit depends on the evaluator.

  • Thus, Table 3 shows that the specific CASI-plus-filter-evaluator training configuration performs well, but does not establish that Shapley aggregation drives the policy improvement or is preferable to an equally costly all-coalition rule. Training Banzhaf under the same configuration and reporting its held-out replay value would address this gap. A smaller controlled comparison would still be informative if full retraining is infeasible.

Questions for authors
  1. Was exact Banzhaf credit evaluated for policy training using the same model, β\beta, κ\kappa, training budget, and five-seed protocol as in Table 3? If so, please report the results. If not, why was it omitted despite having the same coalition-scoring cost as CASI? A Banzhaf result close to or better than CASI would materially change my assessment of the value added by Shapley aggregation.

  2. How were the shaping hyperparameters—including β\beta, κ\kappa, and any optimizer or reward-normalization settings—selected for the three methods in Section 4.4? Were they chosen using a validation set and tuned under equivalent budgets for CASI, Singleton, and Broadcast? Confirmation of a shared, prespecified protocol or equally intensive tuning would increase my confidence in the Table 3 comparison.

1 comment

Dear Reviewer 1Ksy,

We hope you are doing well. We just wanted to briefly follow up and thank you again for your thoughtful feedback. We hope our responses and additional experimental results have helped clarify the points raised in your review.

If any part of our explanations or experimental details remains unclear, please feel free to leave a follow-up question. We would be very happy to provide further clarification.

Thank you again for your time and consideration!

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.