acceptodds
Under review as a conference paper at ICLR 2027

Counterfactual Taylor Attribution for Sequential Policy Learning

Abstract

Existing reinforcement-learning approaches have shown strong performance in sequential active perception by optimizing acquisition policies from long-term utility. However, these methods typically model only the next acquisition conditioned on the current observation history, leaving alternative multi-step acquisition strategies implicit in the policy. We introduce a diffusion policy with Counterfactual Taylor Attribution (CTA) to explicitly exploit such future plan variation. The diffusion policy models a joint distribution over multi-step acquisitions, enabling diverse candidate plans to be sampled and counterfactually evaluated from the same decision state. CTA uses the resulting utility variation to estimate action-level contributions and redistribute within-plan credit during policy optimization, without introducing an additional reward model or changing the original task objective. Our framework provides a general mechanism for exploiting counterfactual plan variation in active information acquisition, including active perception and sequential Bayesian experimental design.

Primary area:
generative models
Keywords:
Sequential Policy Design, Active Perception, Adaptive Data Acquisition, Scientific Discovery

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_GsqbOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

The paper formulates multi-step information acquisition plans, for active perception, using a joint diffusion policy. By contrast, existing work tends to represent an action-preference over the next acquisition. Unlike existing joint policy methods, which evaluate sampled action sequences at the trajectory level, the authors also introduce Counterfactual Taylor Attribution (CAT) to estimate the contributions of individual acquisitions within the plans to the utility differences between different plans. They formalize CAT by providing a action-based decomposition of the utility difference between two plans. They estimate the action-level contribution to the utility difference to first-order by fitting a linear model to estimate utility differences as a function of the differences in the sampled plans. Experiments on APPLE active perception tasks and ALINE sequential Bayesian experimental design demonstrate the approach.

Critical strengths
  1. The work is well-motivated both in objective and formulation. Credit assignment is an important problem, especially in sequential decision making, where it is typically difficult to attribute task success to specific decisions along a trajectory. The author's proposed counterfactual acquisition decomposition is theoretically sound and motivated.
  2. The chosen DPPO baseline clearly separates the contribution of CTA, and the experiments demonstrates that CTA generally helps across various active perception baselines.
Critical weaknesses
  1. The authors only compare to non-counterfactual baselines (DPPO, APPLE, ALINE) which demonstrates that their method provides meaningful credit assignment but does not place this in context of existing credit assignment methods, of which many already exist. The related work section should also mention and discuss return-decomposition and counterfactual credit-assignment methods in RL.

In this comparison setting, the experiments demonstrate that CTA helps but the improvement is relatively small over non-CTA baselines and it is not clear whether CTA is using more samples to estimate the counterfactual attributions for actions, in which case the comparison is not fair. The ablation in Table 6 is also confounded since large action differences will result in large CTA attribution in (13), but the goal is to characterize whether the slope g is meaningful. It would help if the authors provide a control/comparison against slot replacement using only the relative differences between actions, i.e., ∣at+j(k)−at+j(i)∣|a_{t + j}^{(k)} - a_{t + j}^{(i)}|.

  1. Novelty is not clear. The authors do not cite prior work for Definition 1 or their decomposition of the utility difference in Section 3.2, while this seems to be very similar to the integrated gradients approach of [R1]. The CTA approximation seems to be novel, but diverges from the theory with no analysis on the error resulting from the approximation.

[R1] Mukund Sundararajan, Ankur Taly, & Qiqi Yan (2017). Axiomatic Attribution for Deep Networks. ICML.

Questions for authors

Please see weaknesses. I am willing to increase my score if these are addressed sufficiently.

  1. The authors re-iterate that existing work accounts for the future consequences of a decision through long-term returns while parameterizing their policies one acquisition at a time. This is true for the vast majority of RL methods, and is appropriate assuming the environment is Markov. The authors seem to consider this a limitation, but it would be useful for the paper to explain why? Are acquisition problems Non-Markov? Can the authors offer an example?
  2. L199: It is not clear to me by a "receding-horizon" scheme combining joint multi-step planning with closed-loop adaptation. Receding horizon usually means replanning at every step, but I'm not sure how that connects to the prior sentence of executing E acquisitions prior to planning. Or do the authors mean they plan, execute E actions, and then plan again? And nothing here seems to be closed loop, there is no feedback in terms of the objective to guide the planning.
  3. Can the authors comment on whether/how CTA models/accounts for joint dependencies across actions in a plan? For instance, suppose actions at+1a_{t + 1} and at+2a_{t + 2} are only good together for a sampled plan At(k)A_t^{(k)}. They will plausibly be poor if sampled alone (one without the other). Does CTA capture such dependencies?
  4. Can the authors explain how ωki\omega_{ki} are chosen in (13)
  5. Can the authors explain how the resulting weights from (14) are used in the policy update for DPPO?
  6. Is CTA specific to diffusion policies (DPPO) or can it be adapted to other model-based RL methods? It seems to require a policy that proposed joint, multi-step actions? Is that the only requirement?
  7. Can the authors provide a quantification of the sample efficiency of CTA compared to the baselines? How does CTA-DPPO compare to DPPO when both use the same number of samples to learn?
  8. Shapley values are also often used for explainability/feature attribution, and are often estimated using regression [R2, R3]. Can the authors baseline their method against Shapley based attribution for acquisitions, and explain the difference of their approximation to Shapley attributions?

[R1] Lundberg, Scott M., and Su-In Lee. "A unified approach to interpreting model predictions." Advances in neural information processing systems 30 (2017).

[R2] Olsen, Lars Henry Berge, and Martin Jullum. "Improving the weighting strategy in KernelSHAP." World Conference on Explainable Artificial Intelligence. Cham: Springer Nature Switzerland, 2025.

8 comments
AuthorsOpenReview ↗

Official Rebuttal to Reviewer Gsqb (1/5)

Weakness 1 & Question 8 (Part 1): Comparison with Existing Credit-Assignment Methods

We thank the reviewer for suggesting comparisons with existing credit-assignment methods. We expand our discussion of related approaches and evaluate CTA against Standard KernelSHAP, Olsen KernelSHAP, Exact Shapley, and the importance-sampling estimator proposed in Counterfactual Shapley Credit Assignment (CSCA; Li et al., 2026). We have expanded the Related Work section to discuss existing return-decomposition and counterfactual credit-assignment methods. We have also added Appendix G, which provides detailed comparisons with Shapley-based methods, including offline attribution evaluations and policy-training experiments.

Methodological Motivation and Novelty.

Our work develops CTA-DPPO for sequential information-acquisition problems, such as those studied in APPLE and ALINE. These problems provide three key properties for counterfactual policy learning: alternative acquisition plans can be evaluated from the same observation history, their downstream utilities can be measured through prediction performance or information gain, and variations among sampled plans provide information for estimating local utility sensitivity.

Building on these properties, we introduce a joint diffusion policy that generates alternative multi-step acquisition plans and a counterfactual attribution method that assigns learning credit to individual acquisitions. Standard DPPO uses a single advantage to optimize each sampled plan, whereas CTA exploits the action–utility differences among plans to estimate acquisition-level contributions. These contributions are incorporated into DPPO through acquisition-level credit weights.

The main algorithmic contribution is an evaluation-efficient credit-assignment mechanism that extracts structured learning signals from existing counterfactual plan evaluations. This connects joint multi-step planning, local utility estimation, and acquisition-level policy optimization without additional attribution rollouts.

Relationship to Existing Credit-Assignment Methods.

We thank the reviewer for suggesting comparisons with existing credit-assignment approaches. RUDDER (Arjona-Medina et al., 2019) redistributes delayed rewards across time steps through return decomposition, while COMA (Foerster et al., 2018) uses counterfactual baselines to compute agent-specific advantages in multi-agent RL. Shapley-based methods provide another approach by assigning credit through coalition-averaged marginal utility contributions. We will expand the related-work discussion to include these methods.

Shapley-based methods can also provide acquisition-level credit for DPPO. Standard KernelSHAP and Olsen and Jullum (2025) estimate Shapley values through weighted regression over coalition evaluations, while CSCA (Li et al., 2026) uses importance sampling within a causal RL framework.

CTA instead estimates local directional utility contributions from existing plan–utility evaluations using weighted regression and first-order Taylor attribution, providing acquisition-level credit without additional coalition evaluations.

We evaluate these alternative credit-assignment methods both in terms of attribution quality and their effect on DPPO training.

Offline Attribution Comparison.

We compare CTA with Standard KernelSHAP, Olsen KernelSHAP, Exact Shapley, and the CSCA importance-sampling estimator on CIFAR-10, using 138 target plans from 69 observation histories (23 images across three training seeds).

For a consistent comparison, all methods use the same recorded observation histories and target–reference pairs selected by CTA. We treat the eight acquisition positions as Shapley players. Each coalition defines a plan that retains the target actions at selected positions and uses reference actions elsewhere. We restore the original environment state using the recorded reset seed and preceding actions, then evaluate the resulting plan with the frozen predictor and original discounted inference-gain reward. Recorded returns are verified by replay, and identical plans share cached evaluations.

Exact Shapley uses all 28=2562^8=256 coalition values. Standard and Olsen KernelSHAP estimate Shapley contributions through efficiency-constrained weighted regression using B=32B=32 coalition evaluations per reference, including the empty and full coalitions. Olsen further uses paired sampling of middle-sized coalitions with corrected sampling weights. For CSCA, we adapt its importance-sampling estimator to the fixed-plan setting, sampling MM coalitions with replacement per reference and averaging importance-weighted target-minus-hybrid utility differences. We repeat CSCA attribution estimation 100 times and average the resulting evaluation metrics.

AuthorsOpenReview ↗

Official Rebuttal to Reviewer Gsqb (2/5)

Weakness 1 & Question 8 (Part 2): Attribution Comparisons and Evaluation Fairness

We compare attribution quality on CIFAR-10 using three metrics: MAE against Exact Shapley, Sens.-1 measured by the Pearson correlation between attribution scores and utility decreases from single-acquisition replacement, and Ranking accuracy, which measures agreement with advantage-signed intervention effects. The budget column reports additional coalition evaluations per reference, with MM denoting CSCA samples and BB denoting KernelSHAP evaluations. CTA requires no additional coalition evaluations.

MethodBudgetMAE ↓\downarrowSens.-1 ↑\uparrowRanking (%) ↑\uparrow
CTA00.034750.305258.05
CSCA Fixed-planM10.221870.131753.79
CSCA Fixed-planM40.115810.259358.26
CSCA Fixed-planM80.081640.336460.54
CSCA Fixed-planM320.041050.505965.93
Standard KernelSHAPB320.019240.619470.77
Olsen KernelSHAPB320.016380.644572.45
Exact Shapley25600.696075.88

CTA achieves lower MAE than CSCA-M1 through CSCA-M32 without additional coalition evaluations. As the sampling budget increases, CSCA achieves better attribution accuracy, while KernelSHAP provides stronger sensitivity and intervention-ranking results. These comparisons highlight the different attribution objectives and motivate evaluating their effects on DPPO training.

Policy Training Comparison.

We further compare CTA-DPPO with Olsen KernelSHAP-B32-DPPO under the same diffusion-policy backbone, credit weighting, and DPPO optimization. Given the high computational cost of KernelSHAP's coalition evaluations, we conduct a matched comparison over 512 real-environment steps on CIFAR-10, reporting Native accuracy, Top-1 accuracy, and policy-training time (mean$\pm$SD across three seeds).

MethodNative Acc. (%)Top-1 (%)Time (min)
CTA-DPPO71.83±1.1571.83\pm1.1544.00±1.8044.00\pm1.8012.86±1.4512.86\pm1.45
Olsen-B32-DPPO71.33±0.2971.33\pm0.2944.83±0.7644.83\pm0.76520.08±21.24520.08\pm21.24

CTA achieves comparable Native and Top-1 accuracy while requiring approximately 40.4×40.4\times less training time, demonstrating the computational advantage of reusing existing counterfactual plan evaluations over coalition-based attribution.

Extended Policy Training Comparison.

We further compare CTA-DPPO and CSCA Fixed-plan M1-DPPO over 20k real-environment steps on CIFAR-10. Time measures policy training and predictor adaptation up to the selected checkpoint.

MethodBest Native (%) ↑\uparrowTime to checkpoint (min) ↓\downarrowNative at 20k (%)
CTA-DPPO79.00247.4477.00
CSCA Fixed-plan M1-DPPO77.00881.6877.00

Both methods reach their best checkpoint after 15k steps and one predictor update. CTA achieves higher peak Native accuracy (79.00% vs. 77.00%) in approximately 3.6×3.6\times less time, while both methods obtain 77.00% at the final 20k checkpoint.

Comparison Fairness and Action-Difference Confounding.

We clarify the sampling budget and introduce an action-difference control to isolate the contribution of the estimated utility gradient. We have added Appendix H to provide detailed comparisons under matched sampling budgets, including an action-difference control and additional policy-learning experiments.

Sampling Fairness.

CTA-DPPO and standard DPPO use the same diffusion-policy initialization, K=32K=32 counterfactual plans per observation history, and the same policy--predictor training schedule. CTA estimates local utility gradients from these existing plan--utility evaluations and introduces acquisition-level credit weighting without additional counterfactual rollouts.

Action-Difference Control.

To determine whether the estimated gradient provides information beyond action displacement, we compare CTA with an action-difference control using the same sampled plans and reference actions:

Δak,j=ak,j−1∣Rk∣∑r∈Rkar,j\Delta a_{k,j}=a_{k,j}-\frac{1}{|\mathcal R_k|}\sum_{r\in\mathcal R_k}a_{r,j}

Φk,j=g^k,j⊤Δak,j,Dk,j=∣Δak,j∣2\Phi_{k,j}=\widehat g_{k,j}^{\top}\Delta a_{k,j},\qquad D_{k,j}=\\|\Delta a_{k,j}\\|_2

Both methods use the same score normalization and credit-weighting procedure. Here Φ^k,j\widehat\Phi_{k,j} and D^k,j\widehat D_{k,j} denote the normalized scores. CTA uses sgn⁡(Ak)Φ^k,j\operatorname{sgn}(\mathcal A_k)\widehat\Phi_{k,j} to incorporate the estimated utility gradient and advantage direction, while the action-difference control uses D^k,j\widehat D_{k,j}, prioritizing larger action displacements without gradient information.

We evaluate both methods using identical slot-replacement interventions, replacing each acquisition with its reference action and measuring the resulting utility change ΔJk,j\Delta J_{k,j}, averaged over references.

AuthorsOpenReview ↗

Official Rebuttal to Reviewer Gsqb (3/5)

Weakness 1 (Part 3): Action-Difference Control Results and Policy Learning

We evaluate both methods using the same slot-replacement interventions. For each acquisition, we replace its action with the reference action and measure the downstream utility change ΔJk,j\Delta J_{k,j}, averaged over references. We compare credit-weight rankings against the advantage-signed intervention effects Tk,j=sgn⁡(Ak)ΔJk,jT_{k,j}=\operatorname{sgn}(\mathcal A_k)\Delta J_{k,j}. Pairwise ranking accuracy assigns half credit to ties and excludes indistinguishable intervention effects.

MethodMacro (%)Micro (%)
Uniform50.0050.00
Action-difference control56.2056.10
CTA58.9558.77
CTA improvement+2.75+2.67

Across three seeds, 64 shared images, 1,528 target plans, and 41,186 decisive acquisition pairs, CTA improves macro ranking accuracy by 2.75 percentage points (paired 95% CI: [1.30, 4.21] pp, bootstrapped over shared images). This demonstrates that the estimated utility gradient provides directional information beyond action differences alone.

Policy Learning Comparison.

We further compare standard DPPO, the action-difference control, and CTA-DPPO under the same policy--predictor training schedule. We report CIFAR-10 train-split Native accuracy (%), averaged over three seeds, across training checkpoints up to 20k real-environment steps.

StageDPPOAction-difference controlCTA-DPPO
5k70.5071.8371.50
10k70.3371.1771.67
15k71.0071.0071.33
Predictor update 176.8376.6777.50
20k78.6775.8379.67

At 20k real-environment steps, CTA achieves 79.67% Native accuracy, exceeding the action-difference control by 3.84 percentage points and standard DPPO by 1.00 percentage point.

Weakness 2 (Part 1): Novelty and Approximation Error

We thank the reviewer for pointing out the connection to Integrated Gradients. We have revised the method section to clarify this relationship, added the reference to Sundararajan et al. (2017), and provided theoretical and empirical analyses of CTA's first-order approximation error.

Novelty and Relationship to Integrated Gradients.

Definition 1 adapts the path-integrated attribution framework of Sundararajan et al. (2017) to sequential acquisition, attributing differences in expected downstream utility to individual acquisition positions. For a fixed reference plan and a straight-line path, the formulation corresponds to Integrated Gradients applied to the plan's action coordinates. We have explicitly acknowledged this connection and clarified that the completeness property follows from the established path-attribution framework.

Our algorithmic contribution is CTA-DPPO, designed for sequential information-acquisition problems such as those studied in APPLE and ALINE. These settings allow alternative acquisition plans to be evaluated from the same observation history, providing the counterfactual plan--utility data needed for acquisition-level credit assignment.

CTA estimates local utility sensitivity through weighted regression over these existing plan--utility pairs, constructs first-order acquisition attributions relative to sampled reference plans, and incorporates them into DPPO through acquisition-level credit weights. This enables efficient policy optimization using black-box downstream utility evaluations without additional attribution rollouts.

Approximation Error.

We characterize the error between the path-integrated acquisition contribution and CTA's first-order estimate. Consider a target plan AkA_k and reference plan ArA_r, with ΔA=Ak−Ar\Delta A=A_k-A_r and Δak,j=ak,j−ar,j\Delta a_{k,j}=a_{k,j}-a_{r,j}. For a straight-line path, the contribution of acquisition jj to the expected downstream utility difference is

Φk,jpath=Δak,j⊤∫01∇jU(Ar+tΔA),dt\Phi_{k,j}^{\mathrm{path}}=\Delta a_{k,j}^{\top}\int_0^1\nabla_j U(A_r+t\Delta A)\\,dt

where UU denotes expected downstream utility. CTA approximates this contribution using the locally estimated gradient, Φ^k,j=g^k,j⊤Δak,j\widehat\Phi_{k,j}=\widehat g_{k,j}^{\top}\Delta a_{k,j}.

If UU has an LL-Lipschitz continuous gradient along the path, with g^k,j\widehat g_{k,j} expressed on the same utility scale, the approximation error satisfies

∣Φk,jpath−Φ^k,j∣≤∣Δak,j∣(εk,j+L2∣ΔA∣)|\Phi_{k,j}^{\mathrm{path}}-\widehat\Phi_{k,j}|\leq\\|\Delta a_{k,j}\\|\left(\varepsilon_{k,j}+\frac{L}{2}\\|\Delta A\\|\right)

εk,j=∣∇jU(Ak)−g^k,j∣\varepsilon_{k,j}=\\|\nabla_j U(A_k)-\widehat g_{k,j}\\|

The bound separates gradient estimation error from first-order truncation error. The latter depends on counterfactual distance and utility curvature, motivating CTA's use of local counterfactual neighborhoods for gradient estimation. For multiple references, the corresponding errors are averaged using the reference weights.

Reviewer_MPDAOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The main motivation for this paper is the credit assignment problem. In DPPO total trajectory-level advantage is used for policy learning disregarding the contribution of each individual action. This paper proposes an algorithm CTA-DPPO to fix this problem. It fits a linear regression that estimates the local utility gradient of an action using similar counterfactual trajectories. It then uses a first order Taylor attribution to get each actions contribution which is then used for computing DPPO update weights at the action level.

Critical strengths
  1. The credit assignment problem is real and well motivated.
  2. The proposed solution is simple, lightweight, intuitie and seems easy to implement.
Critical weaknesses
  1. Main concern is regarding the recoverability of the utility gradients using local linear regression. For example, does the counterfactual trajectories need to satisfy certain conditions? How much data is needed for recovery? And how does this scale? What about sensitivity to noise? The paper also relies on getting multiple similar counterfactual trajectories from each state which may be difficult for more general RL environments. The chosen experiments are simple and perfectly given counterfactual trajectories and requires less data for estimation, so it doesn't sufficiently answer when this method will work and when it will not.
  2. The first order Taylor approximation might leads to some approximation errors. Are you able to characterize what this theoretically or empirically?
Questions for authors

All questions in the weakness section.

4 comments
AuthorsOpenReview ↗

Official Rebuttal to Reviewer MPDA (1/3)

Weakness 1 (Part 1): Local Gradient Estimation and Reference Selection

We thank the reviewer for raising these important questions about the reliability and applicability of CTA.

CTA estimates acquisition-level credit from existing counterfactual plan–utility evaluations and uses this credit to guide DPPO optimization. Its effectiveness depends on three key conditions. First, the policy must generate diverse counterfactual plans conditioned on the same observation history, providing alternative acquisition sequences for comparison. Second, the task must allow the downstream utility of these plans to be evaluated through a simulator, reward model, or predictor. Third, the utility should exhibit sufficiently smooth local variation to support a first-order approximation.

Within this setting, three parameters govern CTA's counterfactual estimation and attribution. KK determines the number of sampled candidate plans, MM controls the local neighborhood used for utility gradient estimation, and RR specifies the reference plans used for acquisition-level attribution. We analyze these factors through our existing ablations and further examine sensitivity to utility noise.

Local Gradient Estimation.

For each target plan, CTA selects the MM nearest counterfactual plans generated from the same observation history and estimates a local utility slope from their action–utility differences. The choice of MM balances the number of comparisons available for estimation against the locality of the fitted approximation.

In the ablation study in Section 4.3, we fix K=32K=32 and R=2R=2 and vary the neighborhood size MM. On CIFAR-10, each plan contains eight two-dimensional acquisitions. With M=8M=8, CTA achieves 0.763±0.0150.763\pm0.015 accuracy, comparable to standard DPPO (0.762±0.0380.762\pm0.038). Increasing MM to 1616 improves accuracy to 0.802±0.0180.802\pm0.018, while further increasing it to 2424 reduces accuracy to 0.755±0.0250.755\pm0.025.

These results reflect a trade-off between the amount of counterfactual information and the locality of the linear approximation. A smaller neighborhood provides fewer plan-level utility comparisons, whereas M=24M=24 includes 24 of the 31 available alternative plans, extending the regression beyond the closest counterfactuals. The results therefore suggest that effective gradient estimation benefits from sufficient local comparisons without unnecessarily broadening the fitting neighborhood.

Counterfactual Reference Selection.

In CTA, a reference is an alternative acquisition plan sampled from the same observation history. It serves as a baseline for attributing the utility difference between the target plan and alternative acquisition choices. While MM determines the neighboring plans used to estimate the local utility gradient, RR specifies the number of references used to compute acquisition-level attribution.

We select references according to their standardized advantages, which measure each plan's utility relative to the average utility of the sampled population. For R=1R=1, we select the plan whose advantage is closest to zero. For R=2R=2, we select one positive-advantage and one negative-advantage plan closest to zero, with equal weights. This provides a comparison against plans near the average-utility baseline and reduces dependence on a single reference.

Our reference-count ablation, reported in the table "Effect of reference count RR on CIFAR-10" in Appendix C, shows that increasing RR from 11 to 22 improves native accuracy from 0.7800.780 to 0.8020.802, supporting the use of multiple references for a more representative attribution baseline.

AuthorsOpenReview ↗

Official Rebuttal to Reviewer MPDA (2/3)

Weakness 1 (Part 2): Noise Sensitivity and Computational Scalability

Sensitivity to Utility Noise.

We further investigate how noise in counterfactual utility evaluations affects CTA's local gradient estimation.

We evaluate the final CTA-DPPO policies and adapted predictors on 23 shared CIFAR-10 images across three training seeds, yielding 69 observation histories. Each history contains 32 sampled acquisition plans and their evaluated utilities. We construct three fixed splits per history, each with 28 fitting plans and four held-out plans. For each split, we select four fitting plans as anchors, estimate their local gradients using weighted ridge regression with 16 nearest neighbors, and evaluate predictions against the four held-out plans, yielding 3,312 comparisons.

To simulate noisy utility evaluations, we add Gaussian perturbations to the standardized utility labels used in regression:

\qquad \epsilon_i \sim \mathcal N(0,1),$$ where $y_i$ denotes the standardized discounted return, $s_y$ is the standard deviation of the clean standardized labels, and $\sigma$ controls the relative noise level. We then re-estimate the local gradients using the perturbed labels without changing the sampled plans or their original returns. The estimated gradients are used to predict utility differences between fitting and held-out plans, which are compared against the original utility differences. We report Pearson correlation, sign accuracy, and normalized RMSE (NRMSE) over the held-out comparisons, with 20 independent repetitions for each nonzero noise level. **Sensitivity of CTA's local gradient estimation to utility noise on CIFAR-10.** Results are averaged across three training seeds. | Noise $\sigma$ | Pearson $\uparrow$ | Sign accuracy (%) $\uparrow$ | NRMSE $\downarrow$ | |---|---|---|---| | 0 | 0.673 | 74.00 | 0.766 | | 0.1 | 0.670 | 73.61 | 0.770 | | 0.25 | 0.653 | 72.50 | 0.794 | | 0.5 | 0.603 | 70.17 | 0.870 | | 1.0 | 0.479 | 65.53 | 1.117 | The results show that CTA's local utility estimates are stable under small perturbations. At $\sigma=0.1$, Pearson correlation remains nearly unchanged (0.673 vs. 0.670), while sign accuracy decreases only slightly from 74.00% to 73.61%. As noise increases, prediction accuracy gradually declines, but the estimated gradients retain meaningful directional information, achieving 70.17% sign accuracy at $\sigma=0.5$ and 65.53% at $\sigma=1.0$. These results demonstrate the robustness of CTA's local utility estimation to moderate noise and characterize its sensitivity to stronger perturbations. **Computational Scalability and Applicability.** We thank the reviewer for raising the question of scalability. CTA estimates acquisition-level credit using counterfactual plans already evaluated during policy optimization, avoiding the additional coalition evaluations required by Shapley-based methods. To evaluate its computational efficiency, we compare CTA-DPPO with Olsen KernelSHAP-B32-DPPO on CIFAR-10. Both methods use the same diffusion-policy backbone and are evaluated after 512 executed acquisition steps. **Accuracy and training time after 512 executed acquisitions on CIFAR-10 (mean\$\pm\$SD over three training seeds).** | Method | Native accuracy (%) | Training time (min) | |---|---|---| | CTA-DPPO | $71.83\pm1.15$ | **$12.86\pm1.45$** | | Olsen-B32-DPPO | $71.33\pm0.29$ | $520.08\pm21.24$ | CTA achieves comparable prediction accuracy, while Olsen KernelSHAP requires approximately 40.4 times as much training time. This demonstrates the computational benefit of reusing existing plan evaluations rather than performing additional coalition evaluations. For broader applications, CTA requires alternative action sequences to be generated and evaluated from the same observation history. Our active perception and BOED tasks provide this capability through their evaluation environments. In other sequential decision problems, such as robotic manipulation, a simulator or learned dynamics and reward model could provide the corresponding counterfactual evaluations.
AuthorsOpenReview ↗

Official Rebuttal to Reviewer MPDA (3/3)

Weakness 2: Taylor Approximation Error

We thank the reviewer for this question. We characterize the approximation error theoretically and evaluate the accuracy of CTA's local utility estimates using held-out counterfactual plans.

Theoretical Characterization.

CTA uses a locally estimated utility gradient to approximate the acquisition contributions defined in Definition 1. Let AkA_k denote a target plan and ArA_r a counterfactual reference, with ΔA=Ak−Ar\Delta A=A_k-A_r and Δaj=ak,j−ar,j\Delta a_j=a_{k,j}-a_{r,j}.

For a straight-line path connecting these plans, the contribution of acquisition jj to their difference in expected downstream utility is

Φk,jpath=Δaj⊤∫01∇jU(Ar+tΔA),dt\Phi_{k,j}^{\mathrm{path}}=\Delta a_j^{\top}\int_0^1\nabla_j U(A_r+t\Delta A)\\,dt

whereas CTA uses the first-order estimate Φ^k,j=g^k,j⊤Δaj\widehat\Phi_{k,j}=\widehat g_{k,j}^{\top}\Delta a_j.

Assuming that the estimated gradient and expected utility are expressed on the same scale, and that UU has an LL-Lipschitz continuous gradient along the path, the approximation error is bounded by

∣Φk,jpath−Φ^k,j∣≤∣Δaj∣(εk,j+L2∣ΔA∣)|\Phi_{k,j}^{\mathrm{path}}-\widehat\Phi_{k,j}|\leq \\|\Delta a_j\\|\left(\varepsilon_{k,j}+\frac{L}{2}\\|\Delta A\\|\right)

εk,j=∣∇jU(Ak)−g^k,j∣\varepsilon_{k,j}=\\|\nabla_j U(A_k)-\widehat g_{k,j}\\|

where εk,j\varepsilon_{k,j} denotes the gradient estimation error. The bound separates the error from estimating the local gradient and the error from the first-order approximation. The latter depends on the distance between the two plans and the curvature of the utility function. This motivates CTA's use of nearby counterfactual plans for gradient estimation.

Empirical Evaluation.

We evaluate CTA's first-order utility predictions on CIFAR-10 using 23 shared images across three training seeds, yielding 69 observation histories with 32 sampled plans each. For each history, we construct three fixed splits with 28 fitting plans and four held-out plans. We select four fitting plans as anchors per split, estimate their local gradients using 16 nearest neighbors, and predict utility differences to the held-out plans, yielding 3,312 comparisons.

To examine the effect of counterfactual distance, we divide the comparisons into near, medium, and far groups according to the distance between their acquisition plans.

Held-out first-order approximation accuracy on CIFAR-10 across counterfactual distances. Relative MAE is normalized by the mean absolute utility difference within each group.

DistanceMean distanceMAE ↓\downarrowRelative MAE ↓\downarrowSign accuracy (%) ↑\uparrow
Near1.3560.1310.76571.44
Medium2.0240.1510.69976.18
Far2.7760.1900.74774.37

As counterfactual distance increases, absolute prediction error grows alongside the magnitude of the observed utility differences. However, relative error does not increase monotonically with distance.

The estimated gradients also retain useful directional information across all three groups. Even for distant counterfactual plans, the first-order estimate achieves 74.37% sign accuracy, indicating that it can still identify the direction of utility changes.

Acquisition-Level Attribution.

We further evaluate whether CTA's attributions reflect the effects of individual acquisitions. In the slot-replacement study reported in the manuscript, mean absolute utility change increases monotonically with CTA attribution rank, from 0.042 to 0.104.

To examine whether this relationship is driven simply by action differences, we introduce a magnitude-only control in Appendix H. CTA achieves 58.95% attribution ranking accuracy, compared with 56.20% for the control. Together with the held-out prediction results, these findings support the effectiveness of CTA's estimated utility gradients for acquisition-level credit assignment.

AuthorsOpenReview ↗

Follow-up on Our Rebuttal

Dear Reviewer MPDA,

Thank you again for your thoughtful review. We have provided detailed responses to your concerns, including additional analyses of local gradient estimation, noise sensitivity, Taylor approximation error, and computational scalability.

As the discussion period is approaching its end, we would appreciate your thoughts on whether these analyses have addressed your concerns. We would be happy to clarify any remaining questions.

Thank you for your time and consideration.

Reviewer_7tc4OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Summary: This paper proposes an approach for training diffusion policy models for sequential active perception problems including sequential Bayesian experimental design. They propose a conterfactual Taylor attribution method, converting cross-state cross-plan utility variation into a within-plan credit assignment using only data that would have been collected regardless.

Critical strengths

This paper is clearly written and proposes a well defined and well-specified experimental setup. The idea of using diffusion policies for active perception and Bayesian experimental design is well motivates given the success of diffusion policies in the rest of the field. The method for attribution appears to me to be well motivated, in that it does not require more data, and works to decrease variance of the gradient estimates.

Critical weaknesses

My concerns with this work are only minor, but would help to clarify the degree to which this approach works in a robust and scalable way. Specifically, there are many approaches in the diffusion polices space that can work well even when the diffusion policy is not well-trained, or does not adapt much to the task, just given the better inductive biases in these policies. As such, it would be useful to know to what degree the performance is a result of a solid diffusion base-model, or from behaviours that the base model was able to learn in a addition with the help of the CTA method.

One point of data that would be helpful in this respect would be to simply show the qualitative behaviours of the base dataset, the trained diffusion model, and the fine-tuned diffusion model after CTA. It is often visually clear which behaviours existed in the underlying dataset, and which were discovered in fine-tuning.

A second data point would be to have a quantitative measure of how well the diffusion mode fit the prior data, and how far it has deviated from it in fine-tuning.

Questions for authors

Could you please help me understand the qualitative behaviours of the diffusion model before and after fine-tuning?

3 comments
AuthorsOpenReview ↗

Official Rebuttal to Reviewer 7tc4

Weakness 1: Base Policy vs. CTA Fine-Tuning: Performance and Acquisition Behavior

We thank the reviewer for this suggestion. We compare four acquisition policies on CIFAR-10: APPLE, which provides the original acquisition demonstrations; BC, a diffusion policy pretrained through behavior cloning of these demonstrations; standard DPPO, which fine-tunes BC using policy optimization; and CTA-DPPO, which additionally incorporates acquisition-level credit weighting.

Contribution of Policy Fine-tuning.

Our existing results show that APPLE achieves 70.50% native accuracy, while the pretrained BC policy achieves 73.00%. Standard DPPO fine-tuning improves accuracy to 76.17±3.7976.17\pm3.79%, and CTA-DPPO further increases it to 80.17±1.7680.17\pm1.76%.

DPPO and CTA-DPPO share the same behavior-cloned initialization, policy architecture, training budget, and alternating policy–predictor schedule. Their controlled comparison isolates the contribution of CTA's acquisition-level credit weighting.

Qualitative Acquisition Behavior.

We have added a new qualitative comparison in Figure 5 of the revised manuscript, showing acquisition trajectories from APPLE, BC, DPPO, and CTA-DPPO on the same CIFAR-10 images under matched initial observation conditions.

In the deer example, BC and DPPO focus on the lower-right image border and misclassify the object as a frog, whereas CTA-DPPO explores the deer's body and neck and correctly predicts deer. Similarly, for the horse image, CTA-DPPO follows the horse's torso and hindquarters and predicts correctly, while DPPO focuses on the upper-left region and predicts truck.

These examples illustrate how CTA fine-tuning can redirect acquisitions toward informative object regions, complementing the performance improvements observed in our quantitative comparison.

Weakness 2: Fit to Pretraining Data and Distribution Shift After Fine-Tuning

We thank the reviewer for this suggestion. We conduct two quantitative analyses to measure the fit to the original APPLE demonstrations and the acquisition distribution changes introduced by fine-tuning.

Evaluation Setup.

We evaluate the final BC, DPPO, and CTA-DPPO checkpoints using 200 observation histories from distinct CIFAR-10 images in the APPLE demonstration validation set. DPPO and CTA-DPPO are evaluated across three training seeds.

First, we evaluate each policy's denoising MSE on the original APPLE demonstration plans, using identical Gaussian noise across all 20 diffusion steps and the corresponding inference configurations. This measures how well each policy models the demonstrated acquisition behavior.

Second, we quantify changes in the acquisition distribution. For each fixed observation history, we independently sample 32 acquisition plans from each policy and compute the RBF-kernel MMD$^2$ between the fine-tuned policies and BC, using diagonal-excluded within-policy similarity estimates. We average the results across the 200 histories and consider both the complete plans (H=8H=8) and the executed prefixes (E=2E=2) before replanning.

Fit to APPLE demonstrations and policy distribution shift on CIFAR-10. Results are mean$\pm$SD across three fine-tuning seeds, with BC as the reference policy.

PolicyDenoising MSE ↓\downarrowMMD$^2$ (H=8H=8)MMD$^2$ (E=2E=2)
BC0.205610 (ref.)0 (ref.)
DPPO0.22746±0.007310.22746\pm0.007310.01445±0.005280.01445\pm0.005280.01142±0.008700.01142\pm0.00870
CTA-DPPO0.23236±0.004750.23236\pm0.004750.03137±0.029840.03137\pm0.029840.03024±0.018370.03024\pm0.01837

Results and Discussion.

BC achieves the lowest denoising error on the original demonstrations, while DPPO and CTA-DPPO exhibit higher errors after fine-tuning.

The MMD$^2$ results further show that both fine-tuned policies differ from BC in their generated acquisition distributions, including the actions executed before replanning. CTA-DPPO exhibits a larger mean distribution shift than DPPO, with considerable variation across seeds.

Together with the prediction accuracy comparison and qualitative acquisition trajectories, these findings show that fine-tuning changes the pretrained acquisition behavior and is accompanied by improved downstream native accuracy.

Question 1: Qualitative Behavior Before and After Fine-tuning

We thank the reviewer for this question. Our four-policy acquisition trajectory comparison on CIFAR-10 illustrates the behavioral changes introduced by fine-tuning.

For the deer, BC and DPPO focus on the image border, whereas CTA-DPPO explores the body and neck. For the horse image, CTA-DPPO follows the torso and hindquarters, while DPPO concentrates on the upper-left region. CTA-DPPO correctly classifies both examples.

Denoising-MSE and state-conditioned MMD$^2$ analyses further quantify the changes from the pretrained diffusion policy, including the acquisitions executed before replanning. These results provide both qualitative and quantitative evidence of how fine-tuning changes acquisition behavior.

Reviewer_7tc4OpenReview ↗

Response to Rebuttal

I thank the authors for their in-depth response to my review. I appreciate the clarification on several points. Overall given my evaluation of the paper's contribution and the other reviewer's comments my score remains unchanged.

AuthorsOpenReview ↗

Thank You for Your Feedback

We thank the reviewer for taking the time to consider our rebuttal. We will ensure that the clarifications and additional analyses are clearly reflected in the revised manuscript.

Comments on the paper

5 comments
~Mingxuan_Li1OpenReview ↗

The submission cited Counterfactual Shapley Credit Assignment but never discuss in more details how similar they are

We are the authors of Counterfactual Shapley Credit Assignment (CSCA; Li et al., RLJ 2026), which this submission mentioned. Upon reading the proposed method, contribution 2 is not clear to us as whether it can be claimed as novel. The detailed per-step credit assignment/attribution under counterfactuals has been discussed carefully in CSCA with proven attribution desiderata. Our original algorithm proposes the exact design that the way to do the counterfactual credit assignment is through comparing the factual trajectory against alternative counterfactual plans. Additionally, we provide careful theoretical treatment on how to implement the counterfactual simulation via common random numbers/states.

In short, we ask that the authors discuss properly their method (sec. 3.2, 3.3) alongside CSCA, rescope the contributions properly and add CSCA to the baselines. Grounding the "counterfactuals" with structural causal models will also provide a solid theoretical foundation to the work. There are also many other important credit assignment methods in the related work section of CSCA paper and we highly recommend the authors to also discuss those.

AuthorsOpenReview ↗

Response to the Authors of CSCA

We thank the authors of CSCA for their comments and suggestions. We agree that a more explicit comparison with existing counterfactual credit-assignment methods helps clarify the motivation and contributions of our work. We have expanded our Related Work section to discuss CSCA and clarify its relationship to CTA. While CSCA estimates Shapley-based credit through counterfactual evaluations, CTA uses local utility regression over existing sampled plan–utility pairs to construct acquisition-level credit without additional attribution rollouts. We believe this provides a computational advantage for sequential information-acquisition problems, where counterfactual plan utilities are already available during policy optimization.

We have also added Appendix G, which includes comparisons with CSCA, KernelSHAP, and Exact Shapley, covering both attribution quality and policy training performance. Our experiments further demonstrate the computational efficiency of CTA while achieving competitive downstream performance. For CSCA, we implement its importance-sampling estimator adapted to our fixed-plan acquisition setting.

Thank you again for your constructive feedback.

Thanks for your reply. However, the updated draft doesn't reveal clearly enough the strong correlations of the proposed method with CSCA or Shapley value in general.

In sec 3.2 and 3.3, eq7 the counterfactual utility margin is pretty close to eq 4 in CSCA and eq 5 natural total effect in the first paper proposing counterfactual shapley values. And line 253-261 share an interesting similarity to Shapley efficiency (the draft calls it completeness in line 248) and other inherent shapley value properties (see https://www.rand.org/content/dam/rand/pubs/papers/2021/P295.pdf). I think a more careful comparison would be very fruitful for readers to connect with the highly relevant broader literature.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.