- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
The paper formulates multi-step information acquisition plans, for active perception, using a joint diffusion policy. By contrast, existing work tends to represent an action-preference over the next acquisition. Unlike existing joint policy methods, which evaluate sampled action sequences at the trajectory level, the authors also introduce Counterfactual Taylor Attribution (CAT) to estimate the contributions of individual acquisitions within the plans to the utility differences between different plans. They formalize CAT by providing a action-based decomposition of the utility difference between two plans. They estimate the action-level contribution to the utility difference to first-order by fitting a linear model to estimate utility differences as a function of the differences in the sampled plans. Experiments on APPLE active perception tasks and ALINE sequential Bayesian experimental design demonstrate the approach.
Critical strengths
- The work is well-motivated both in objective and formulation. Credit assignment is an important problem, especially in sequential decision making, where it is typically difficult to attribute task success to specific decisions along a trajectory. The author's proposed counterfactual acquisition decomposition is theoretically sound and motivated.
- The chosen DPPO baseline clearly separates the contribution of CTA, and the experiments demonstrates that CTA generally helps across various active perception baselines.
Critical weaknesses
- The authors only compare to non-counterfactual baselines (DPPO, APPLE, ALINE) which demonstrates that their method provides meaningful credit assignment but does not place this in context of existing credit assignment methods, of which many already exist. The related work section should also mention and discuss return-decomposition and counterfactual credit-assignment methods in RL.
In this comparison setting, the experiments demonstrate that CTA helps but the improvement is relatively small over non-CTA baselines and it is not clear whether CTA is using more samples to estimate the counterfactual attributions for actions, in which case the comparison is not fair. The ablation in Table 6 is also confounded since large action differences will result in large CTA attribution in (13), but the goal is to characterize whether the slope g is meaningful. It would help if the authors provide a control/comparison against slot replacement using only the relative differences between actions, i.e., .
- Novelty is not clear. The authors do not cite prior work for Definition 1 or their decomposition of the utility difference in Section 3.2, while this seems to be very similar to the integrated gradients approach of [R1]. The CTA approximation seems to be novel, but diverges from the theory with no analysis on the error resulting from the approximation.
[R1] Mukund Sundararajan, Ankur Taly, & Qiqi Yan (2017). Axiomatic Attribution for Deep Networks. ICML.
Questions for authors
Please see weaknesses. I am willing to increase my score if these are addressed sufficiently.
- The authors re-iterate that existing work accounts for the future consequences of a decision through long-term returns while parameterizing their policies one acquisition at a time. This is true for the vast majority of RL methods, and is appropriate assuming the environment is Markov. The authors seem to consider this a limitation, but it would be useful for the paper to explain why? Are acquisition problems Non-Markov? Can the authors offer an example?
- L199: It is not clear to me by a "receding-horizon" scheme combining joint multi-step planning with closed-loop adaptation. Receding horizon usually means replanning at every step, but I'm not sure how that connects to the prior sentence of executing E acquisitions prior to planning. Or do the authors mean they plan, execute E actions, and then plan again? And nothing here seems to be closed loop, there is no feedback in terms of the objective to guide the planning.
- Can the authors comment on whether/how CTA models/accounts for joint dependencies across actions in a plan? For instance, suppose actions and are only good together for a sampled plan . They will plausibly be poor if sampled alone (one without the other). Does CTA capture such dependencies?
- Can the authors explain how are chosen in (13)
- Can the authors explain how the resulting weights from (14) are used in the policy update for DPPO?
- Is CTA specific to diffusion policies (DPPO) or can it be adapted to other model-based RL methods? It seems to require a policy that proposed joint, multi-step actions? Is that the only requirement?
- Can the authors provide a quantification of the sample efficiency of CTA compared to the baselines? How does CTA-DPPO compare to DPPO when both use the same number of samples to learn?
- Shapley values are also often used for explainability/feature attribution, and are often estimated using regression [R2, R3]. Can the authors baseline their method against Shapley based attribution for acquisitions, and explain the difference of their approximation to Shapley attributions?
[R1] Lundberg, Scott M., and Su-In Lee. "A unified approach to interpreting model predictions." Advances in neural information processing systems 30 (2017).
[R2] Olsen, Lars Henry Berge, and Martin Jullum. "Improving the weighting strategy in KernelSHAP." World Conference on Explainable Artificial Intelligence. Cham: Springer Nature Switzerland, 2025.
8 comments
Official Rebuttal to Reviewer Gsqb (2/5)
Weakness 1 & Question 8 (Part 2): Attribution Comparisons and Evaluation Fairness
We compare attribution quality on CIFAR-10 using three metrics: MAE against Exact Shapley, Sens.-1 measured by the Pearson correlation between attribution scores and utility decreases from single-acquisition replacement, and Ranking accuracy, which measures agreement with advantage-signed intervention effects. The budget column reports additional coalition evaluations per reference, with denoting CSCA samples and denoting KernelSHAP evaluations. CTA requires no additional coalition evaluations.
| Method | Budget | MAE | Sens.-1 | Ranking (%) |
|---|---|---|---|---|
| CTA | 0 | 0.03475 | 0.3052 | 58.05 |
| CSCA Fixed-plan | M1 | 0.22187 | 0.1317 | 53.79 |
| CSCA Fixed-plan | M4 | 0.11581 | 0.2593 | 58.26 |
| CSCA Fixed-plan | M8 | 0.08164 | 0.3364 | 60.54 |
| CSCA Fixed-plan | M32 | 0.04105 | 0.5059 | 65.93 |
| Standard KernelSHAP | B32 | 0.01924 | 0.6194 | 70.77 |
| Olsen KernelSHAP | B32 | 0.01638 | 0.6445 | 72.45 |
| Exact Shapley | 256 | 0 | 0.6960 | 75.88 |
CTA achieves lower MAE than CSCA-M1 through CSCA-M32 without additional coalition evaluations. As the sampling budget increases, CSCA achieves better attribution accuracy, while KernelSHAP provides stronger sensitivity and intervention-ranking results. These comparisons highlight the different attribution objectives and motivate evaluating their effects on DPPO training.
Policy Training Comparison.
We further compare CTA-DPPO with Olsen KernelSHAP-B32-DPPO under the same diffusion-policy backbone, credit weighting, and DPPO optimization. Given the high computational cost of KernelSHAP's coalition evaluations, we conduct a matched comparison over 512 real-environment steps on CIFAR-10, reporting Native accuracy, Top-1 accuracy, and policy-training time (mean$\pm$SD across three seeds).
| Method | Native Acc. (%) | Top-1 (%) | Time (min) |
|---|---|---|---|
| CTA-DPPO | |||
| Olsen-B32-DPPO |
CTA achieves comparable Native and Top-1 accuracy while requiring approximately less training time, demonstrating the computational advantage of reusing existing counterfactual plan evaluations over coalition-based attribution.
Extended Policy Training Comparison.
We further compare CTA-DPPO and CSCA Fixed-plan M1-DPPO over 20k real-environment steps on CIFAR-10. Time measures policy training and predictor adaptation up to the selected checkpoint.
| Method | Best Native (%) | Time to checkpoint (min) | Native at 20k (%) |
|---|---|---|---|
| CTA-DPPO | 79.00 | 247.44 | 77.00 |
| CSCA Fixed-plan M1-DPPO | 77.00 | 881.68 | 77.00 |
Both methods reach their best checkpoint after 15k steps and one predictor update. CTA achieves higher peak Native accuracy (79.00% vs. 77.00%) in approximately less time, while both methods obtain 77.00% at the final 20k checkpoint.
Comparison Fairness and Action-Difference Confounding.
We clarify the sampling budget and introduce an action-difference control to isolate the contribution of the estimated utility gradient. We have added Appendix H to provide detailed comparisons under matched sampling budgets, including an action-difference control and additional policy-learning experiments.
Sampling Fairness.
CTA-DPPO and standard DPPO use the same diffusion-policy initialization, counterfactual plans per observation history, and the same policy--predictor training schedule. CTA estimates local utility gradients from these existing plan--utility evaluations and introduces acquisition-level credit weighting without additional counterfactual rollouts.
Action-Difference Control.
To determine whether the estimated gradient provides information beyond action displacement, we compare CTA with an action-difference control using the same sampled plans and reference actions:
Both methods use the same score normalization and credit-weighting procedure. Here and denote the normalized scores. CTA uses to incorporate the estimated utility gradient and advantage direction, while the action-difference control uses , prioritizing larger action displacements without gradient information.
We evaluate both methods using identical slot-replacement interventions, replacing each acquisition with its reference action and measuring the resulting utility change , averaged over references.
Official Rebuttal to Reviewer Gsqb (3/5)
Weakness 1 (Part 3): Action-Difference Control Results and Policy Learning
We evaluate both methods using the same slot-replacement interventions. For each acquisition, we replace its action with the reference action and measure the downstream utility change , averaged over references. We compare credit-weight rankings against the advantage-signed intervention effects . Pairwise ranking accuracy assigns half credit to ties and excludes indistinguishable intervention effects.
| Method | Macro (%) | Micro (%) |
|---|---|---|
| Uniform | 50.00 | 50.00 |
| Action-difference control | 56.20 | 56.10 |
| CTA | 58.95 | 58.77 |
| CTA improvement | +2.75 | +2.67 |
Across three seeds, 64 shared images, 1,528 target plans, and 41,186 decisive acquisition pairs, CTA improves macro ranking accuracy by 2.75 percentage points (paired 95% CI: [1.30, 4.21] pp, bootstrapped over shared images). This demonstrates that the estimated utility gradient provides directional information beyond action differences alone.
Policy Learning Comparison.
We further compare standard DPPO, the action-difference control, and CTA-DPPO under the same policy--predictor training schedule. We report CIFAR-10 train-split Native accuracy (%), averaged over three seeds, across training checkpoints up to 20k real-environment steps.
| Stage | DPPO | Action-difference control | CTA-DPPO |
|---|---|---|---|
| 5k | 70.50 | 71.83 | 71.50 |
| 10k | 70.33 | 71.17 | 71.67 |
| 15k | 71.00 | 71.00 | 71.33 |
| Predictor update 1 | 76.83 | 76.67 | 77.50 |
| 20k | 78.67 | 75.83 | 79.67 |
At 20k real-environment steps, CTA achieves 79.67% Native accuracy, exceeding the action-difference control by 3.84 percentage points and standard DPPO by 1.00 percentage point.
Weakness 2 (Part 1): Novelty and Approximation Error
We thank the reviewer for pointing out the connection to Integrated Gradients. We have revised the method section to clarify this relationship, added the reference to Sundararajan et al. (2017), and provided theoretical and empirical analyses of CTA's first-order approximation error.
Novelty and Relationship to Integrated Gradients.
Definition 1 adapts the path-integrated attribution framework of Sundararajan et al. (2017) to sequential acquisition, attributing differences in expected downstream utility to individual acquisition positions. For a fixed reference plan and a straight-line path, the formulation corresponds to Integrated Gradients applied to the plan's action coordinates. We have explicitly acknowledged this connection and clarified that the completeness property follows from the established path-attribution framework.
Our algorithmic contribution is CTA-DPPO, designed for sequential information-acquisition problems such as those studied in APPLE and ALINE. These settings allow alternative acquisition plans to be evaluated from the same observation history, providing the counterfactual plan--utility data needed for acquisition-level credit assignment.
CTA estimates local utility sensitivity through weighted regression over these existing plan--utility pairs, constructs first-order acquisition attributions relative to sampled reference plans, and incorporates them into DPPO through acquisition-level credit weights. This enables efficient policy optimization using black-box downstream utility evaluations without additional attribution rollouts.
Approximation Error.
We characterize the error between the path-integrated acquisition contribution and CTA's first-order estimate. Consider a target plan and reference plan , with and . For a straight-line path, the contribution of acquisition to the expected downstream utility difference is
where denotes expected downstream utility. CTA approximates this contribution using the locally estimated gradient, .
If has an -Lipschitz continuous gradient along the path, with expressed on the same utility scale, the approximation error satisfies
The bound separates gradient estimation error from first-order truncation error. The latter depends on counterfactual distance and utility curvature, motivating CTA's use of local counterfactual neighborhoods for gradient estimation. For multiple references, the corresponding errors are averaged using the reference weights.
Official Rebuttal to Reviewer Gsqb (1/5)
Weakness 1 & Question 8 (Part 1): Comparison with Existing Credit-Assignment Methods
We thank the reviewer for suggesting comparisons with existing credit-assignment methods. We expand our discussion of related approaches and evaluate CTA against Standard KernelSHAP, Olsen KernelSHAP, Exact Shapley, and the importance-sampling estimator proposed in Counterfactual Shapley Credit Assignment (CSCA; Li et al., 2026). We have expanded the Related Work section to discuss existing return-decomposition and counterfactual credit-assignment methods. We have also added Appendix G, which provides detailed comparisons with Shapley-based methods, including offline attribution evaluations and policy-training experiments.
Methodological Motivation and Novelty.
Our work develops CTA-DPPO for sequential information-acquisition problems, such as those studied in APPLE and ALINE. These problems provide three key properties for counterfactual policy learning: alternative acquisition plans can be evaluated from the same observation history, their downstream utilities can be measured through prediction performance or information gain, and variations among sampled plans provide information for estimating local utility sensitivity.
Building on these properties, we introduce a joint diffusion policy that generates alternative multi-step acquisition plans and a counterfactual attribution method that assigns learning credit to individual acquisitions. Standard DPPO uses a single advantage to optimize each sampled plan, whereas CTA exploits the action–utility differences among plans to estimate acquisition-level contributions. These contributions are incorporated into DPPO through acquisition-level credit weights.
The main algorithmic contribution is an evaluation-efficient credit-assignment mechanism that extracts structured learning signals from existing counterfactual plan evaluations. This connects joint multi-step planning, local utility estimation, and acquisition-level policy optimization without additional attribution rollouts.
Relationship to Existing Credit-Assignment Methods.
We thank the reviewer for suggesting comparisons with existing credit-assignment approaches. RUDDER (Arjona-Medina et al., 2019) redistributes delayed rewards across time steps through return decomposition, while COMA (Foerster et al., 2018) uses counterfactual baselines to compute agent-specific advantages in multi-agent RL. Shapley-based methods provide another approach by assigning credit through coalition-averaged marginal utility contributions. We will expand the related-work discussion to include these methods.
Shapley-based methods can also provide acquisition-level credit for DPPO. Standard KernelSHAP and Olsen and Jullum (2025) estimate Shapley values through weighted regression over coalition evaluations, while CSCA (Li et al., 2026) uses importance sampling within a causal RL framework.
CTA instead estimates local directional utility contributions from existing plan–utility evaluations using weighted regression and first-order Taylor attribution, providing acquisition-level credit without additional coalition evaluations.
We evaluate these alternative credit-assignment methods both in terms of attribution quality and their effect on DPPO training.
Offline Attribution Comparison.
We compare CTA with Standard KernelSHAP, Olsen KernelSHAP, Exact Shapley, and the CSCA importance-sampling estimator on CIFAR-10, using 138 target plans from 69 observation histories (23 images across three training seeds).
For a consistent comparison, all methods use the same recorded observation histories and target–reference pairs selected by CTA. We treat the eight acquisition positions as Shapley players. Each coalition defines a plan that retains the target actions at selected positions and uses reference actions elsewhere. We restore the original environment state using the recorded reset seed and preceding actions, then evaluate the resulting plan with the frozen predictor and original discounted inference-gain reward. Recorded returns are verified by replay, and identical plans share cached evaluations.
Exact Shapley uses all 28=256 coalition values. Standard and Olsen KernelSHAP estimate Shapley contributions through efficiency-constrained weighted regression using B=32 coalition evaluations per reference, including the empty and full coalitions. Olsen further uses paired sampling of middle-sized coalitions with corrected sampling weights. For CSCA, we adapt its importance-sampling estimator to the fixed-plan setting, sampling M coalitions with replacement per reference and averaging importance-weighted target-minus-hybrid utility differences. We repeat CSCA attribution estimation 100 times and average the resulting evaluation metrics.