acceptodds
Under review as a conference paper at ICLR 2027

Towards Adaptive Temporal Credit Assignment via Bidirectional Fractional Operators

Abstract

In continuous control and sequential decision-making tasks, agents may receive only an episodic return, with no intermediate rewards. Learning from such feedback requires connecting decisions to the terminal outcome. However, recurrent compression can weaken early contextual information, while fixed exponential weighting can attenuate distant learning signals. Moreover, history encoding and credit propagation may require different temporal scales. To address these challenges, we propose Fractional Credit Assignment (FCA), a framework based on two oppositely directed Caputo fractional operators. First, the left-sided operator introduces nonlocal history encoding through a power-law kernel. This allows early trajectory information to directly contribute to later representations. Second, the right-sided operator provides power-law temporal weights for backward credit assignment. Terminal-to-history attention combines content matching with these weights to redistribute the episodic return across steps. The fractional orders α1 and α2 are learned independently, allowing the two temporal scales to adapt to each task. We evaluate FCA under episodic-return-only feedback on seven MuJoCo tasks and four DeepMind Control Suite tasks. The latter use fixed 1,000-step episodes. FCA achieves the highest final average return on nine of the eleven tasks. On Swimmer, Ant, and Walker2d, it improves average returns over the strongest baseline in the same feedback setting by 90.7%, 53.3%, and 42.8%, respectively. It also outperforms SAC trained with dense rewards on five MuJoCo tasks. Ablation studies support the benefits of nonlocal history encoding and independently learned temporal scales for long-horizon credit assignment. The code is available at https://anonymous.4open.science/r/FCA-FBEC.

Primary area:
reinforcement learning
Keywords:
Credit Assignment, Reinforcement Learning, Episodic Return, Reward Decomposition, Fractional-Order Dynamics, Sparse Reward

Reviews

Among:
Shape:
1234this paper, Bottom 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.25 over 4 reviews, above 49% of the others.
Reviewer_feGPOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies temporal credit assignment from episodic returns. It learns two independent fractional orders to control temporal weighting in history encoding and return allocation. Content matching between terminal and historical representations produces stepwise rewards, which are then distilled into a state–action reward model for policy training. The paper evaluates the approach on delayed-reward control tasks and examines the effects of learning separate fractional orders.

Critical strengths

Learning separate temporal scales for history encoding and return allocation is a well-motivated design. The shared-order and fixed-order ablations support this choice, while the overall results demonstrate its practical potential for learning from episodic returns.

Critical weaknesses

The experiments leave the benefit of the fractional kernels over simpler temporal weighting insufficiently established.

Figures 5 and 6 provide useful comparisons with shared or fixed fractional orders and a GRU replacement. These results support learning the two orders independently within the proposed design. However, they do not directly establish how the fractional weighting compares with a simpler learnable temporal kernel. The GRU replacement changes the representation architecture as well, making it difficult to assess the temporal kernel separately.

A focused comparison with a learnable exponential kernel, while retaining the same encoder, content matching, and training objective, would address this concern. Uniform temporal weights with content matching retained would also be informative. This latter control differs from assigning the episodic return uniformly across steps, as described in Appendix B.3.

Questions for authors

With the encoder, content matching, and training objective held fixed, how does the fractional kernel compare with a learnable exponential kernel or uniform temporal weights on existing tasks such as Hopper and Ant? Do any existing experiments cover this comparison?

Are results available for the uniform reward-allocation control described in Appendix B.3?

I’d be happy to raise the score if you can answer these questions.

5 comments
AuthorsOpenReview ↗

Official Comment by Authors

We thank you for the valuable feedback and for your support of our work. The fractional kernel achieves higher mean returns than both matched alternatives on Hopper and Ant.

1. Exponential kernel and uniform temporal weights. We replace only the backward temporal weighting rule. The encoder architecture, content matching, normalization, losses, and distillation remain unchanged. Each variant is trained separately under the same protocol.

Table A. Temporal-weighting controls with content matching retained.

Temporal weightingHopperAnt
Learnable fractional kernel, full FCA3574.4 ± 454395.7 ± 860
Learnable exponential kernel3380 ± 1604010 ± 750
Uniform temporal weights3420 ± 1204120 ± 740

The gains over uniform temporal weights are 4.5% on Hopper and 6.7% on Ant. The gains over the exponential kernel are 5.8% and 9.6%, respectively.

The exponential control learns a time constant in absolute steps. The uniform-time control sets ωk=1\omega_k = 1 and retains content matching. Its credits are

r~k=R(τ)ek∑jej.\tilde r_k = R(\tau)\frac{e_k}{\sum_j e_j}.

This is the α2=1\alpha_2 = 1 boundary case of the temporal kernel.

Figures 5 and 6 examined encoder replacement and fractional-order choices. They did not include these two direct kernel controls. Table A now provides the requested comparison.

2. Uniform return allocation in Appendix B.3. This is a separate control. It assigns

r~k=R(τ)T\tilde r_k = \frac{R(\tau)}{T}

before distillation. It does not use content matching.

Its returns are 3210±2003210 \pm 200 on Hopper and 3380±7003380 \pm 700 on Ant. These numerical results were omitted from the original Appendix B.3. We have included them in Table B.

We have added the kernel definitions and matched comparisons to Appendix B.5. We have also clarified the roles of the different ablations in §5.3.

Revised-PDF locations: Table A (lines 855–863) and Table B(a) (lines 890–902). Kernel definitions and matched comparisons are in Appendix B.5 (lines 931–945). The roles of the ablations are clarified in Section 5.3 (lines 472–485).

We welcome further discussion if any concerns remain.

AuthorsOpenReview ↗

Summary of rebuttal

Hi Reviewer feGP, thank you again for your time and thoughtful review! We have carefully addressed all your concerns in our rebuttal and provided additional experimental results and clarifications where needed.

Could you please let us know if any concerns remain? If our responses have adequately addressed your questions, we would greatly appreciate it if you could reconsider your rating.

Thank you again for your time, consideration, and support of our work!

AuthorsOpenReview ↗

Official Comment by Authors

Hi Reviewer feGP,

Thank you again for your constructive feedback. We have addressed your main concern about the fractional temporal kernel. We now provide direct comparisons with a learnable exponential kernel and uniform temporal weights. All other components and training settings remain unchanged. The results show consistent gains from the fractional kernel on both Hopper and Ant. We have also clarified the uniform reward-allocation results.

As the discussion period closes at 8:00 PM today, could you please let us know if any concerns remain? We hope these clarifications have addressed your questions. If so, we would be very grateful if you could reconsider your rating.

Thank you again for your time and valuable feedback!

Reviewer_4XuxOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Fractional Credit Assignment (FCA) for reinforcement learning with episodic-return-only feedback. It combines a fractional trajectory encoder with terminal-to-history content matching and a fractional temporal weighting kernel to redistribute episodic returns. Two independently learned orders control history encoding and temporal credit weighting. The redistributed rewards are distilled into a state–action reward model for SAC training. Experiments cover seven MuJoCo tasks and four DeepMind Control Suite tasks.

Critical strengths
  1. The separation of history encoding and backward credit weighting is a useful design choice, allowing the two temporal scales to be adapted independently.

  2. The empirical evaluation covers eleven continuous-control tasks, including fixed 1,000-step episodes, with results reported across five seeds. FCA achieves the highest final mean return among terminal-feedback methods on nine tasks.

Critical weaknesses
  1. The evidence explaining the encoder’s temporal behavior and its advantage over gated recurrent alternatives is incomplete. Equal explicit weights on historical dynamics responses do not necessarily imply equal effective influence of the original inputs, because those responses depend recursively on previous hidden states. The existing order ablations do not include the important boundary case of α1=1\alpha_1=1. Furthermore, although the parameter-matched GRU comparison is informative, the reported configuration and tuning details are insufficient to establish a robust advantage over well-tuned gated recurrent encoders.

  2. The contribution of the fractional temporal prior and the practical computational tradeoff are insufficiently isolated. The reported fixed-order and shared-order ablations do not directly establish whether the α2\alpha_2-controlled temporal prior improves upon content-based attention alone. Although hardware and asymptotic complexity are provided, comparative measurements of training time and memory consumption are missing. These measurements are particularly relevant given the encoder’s quadratic complexity in trajectory length.

Questions for authors
  1. Effective temporal influence and the α1=1\alpha_1=1 boundary. In Eq. (4), setting α1=1\alpha_1=1 gives equal explicit weights to the dynamics responses and yields hk+1=hk+fϕ(xk,hk).h_{k+1}=h_k+f_\phi(x_k,h_k). However, each response depends on the accumulated hidden state. Earlier inputs can therefore influence subsequent responses through additional computational paths, so equal explicit weights do not establish equal effective influence across input positions. Could the authors include an α1=1\alpha_1=1 ablation alongside the existing fixed-order experiments and directly examine sensitivity to early versus late inputs, for example through controlled perturbations or gradient-based diagnostics? This would clarify when the encoder preserves, attenuates, or amplifies information from different temporal positions.

  2. Interpretation and tuning of the GRU/LSTM comparisons. GRUs and LSTMs use gating to facilitate retention of long-range information. This does not guarantee better downstream performance, but it makes the lower returns of the GRU replacement worth investigating. Could the authors explain which properties of the fractional encoder account for this advantage and provide controlled comparisons with both tuned GRU and LSTM encoders? Please report the hidden dimensions, layer counts, learning rates, hyperparameter search ranges, tuning budgets, and results across seeds, with comparable tuning effort and clearly stated parameter counts.

  3. Empirical computational cost. The paper reports O(T2)O(T^2) trajectory encoding, O(T)O(T) credit allocation at fixed network width, and use of an NVIDIA A800 GPU. Could the authors additionally report training wall-clock time or GPU-hours and peak GPU memory relative to the baselines under matched experimental settings? Measurements across trajectory lengths, including the 1,000-step setting, would help assess whether the performance gains justify the additional computational cost.

  4. Necessity of the α2\alpha_2-controlled temporal prior. The credit allocation combines content-based attention scores with a fractional temporal prior. Could the authors remove this prior by setting ωk≡1\omega_k\equiv1, equivalently using the α2=1\alpha_2=1 boundary, while keeping the encoder, normalization, learning objectives, and distillation procedure unchanged? This gives an attention-only allocation, r~k=R(τ)ek∑jej.\tilde r_k=R(\tau)\frac{e_k}{\sum_j e_j}. Comparing this variant with full FCA would directly establish the incremental benefit of the fractional temporal prior. The existing fixed-order comparisons do not fully isolate this contribution.

4 comments
AuthorsOpenReview ↗

【1/2】Official Comment by Authors

Thank you for the precise questions.

To Q1 and W1: α1=1\alpha_1=1 is a boundary case of our encoder. It is not the encoder we use. The learned order converges to 0.6 from three initialisations (Fig. 4). It gives a higher return than α1\alpha_1 fixed at 1.

  1. What α1\alpha_1 controls. Our encoder is a family with two ends. At α1=1\alpha_1=1, it is the recursion you describe: hk+1=hk+fϕ(xk,hk)h_{k+1}=h_k+f_\phi(x_k,h_k) (Appendix A.1, lines 678–679). At α1→0\alpha_1\to0, it is the memoryless update hk+1=h0+fkh_{k+1}=h_0+f_k. The order α1\alpha_1 sets the memory span between these two ends. This order is learned.

  2. Effective influence. From Eq. (4), ∂hT/∂fj=wT−1,j(α1)Γ(1+α1)I+(paths through later states)\partial h_T/\partial f_j=\frac{w_{T-1,j}(\alpha_1)}{\Gamma(1+\alpha_1)}I+\text{(paths through later states)}. The first term does not depend on the Jacobians of fϕf_\phi. It decays as (T−j)α1−1(T-j)^{\alpha_1-1}. This decay is polynomial in the lag (Appendix A.1, lines 678–687).

  3. Boundary experiment. With α1\alpha_1 fixed at 1, Hopper reaches 3340 ± 160. Ant reaches 3930 ± 730. The learned order gives 3574.4 on Hopper and 4395.7 on Ant (Table A, lines 855–863).

  4. Measured influence. Equal explicit weights do not imply equal effective influence. We therefore measure effective influence directly on held-out 1000-step Hopper trajectories. We compute the ratio of ∥∂gψ(hT)/∂xj∥\|\partial g_\psi(h_T)/\partial x_j\| over the first 10% of steps to the last 10% (Appendix B.8, lines 1036–1044; Table F, lines 1047–1050).

EncoderFCAα1=1\alpha_1=1GRULSTM
Early / late sensitivity0.381.30.050.07

Gated encoders attenuate early inputs by more than an order of magnitude. α1=1\alpha_1=1 does not attenuate them. The learned order attenuates them polynomially. It also gives the highest return.

To Q2 and W1 (GRU and LSTM). The advantage over gated encoders remains after tuning them. On Hopper, tuned GRU reaches 3450 ± 130. Tuned LSTM reaches 3430 ± 140. FCA reaches 3574.4 (Table E, lines 1027–1034).

  1. Source of the advantage. It is the additive power-law path above. Gated encoders instead use a product-of-Jacobians path.

  2. FCA is not tuned per task. The orders α1\alpha_1 and α2\alpha_2 are learned. FCA uses the single configuration of Appendix B.1 (lines 799–817) on all eleven tasks.

  3. Tuned comparison. We tested hidden dimensions {128, 256}. We tested {1, 2} layers and learning rates {1e-4, 3e-4, 1e-3}. Each configuration used three seeds on Hopper. We selected the best configuration for each encoder on Hopper. We then evaluated it on Hopper and Ant with five seeds. We did not retune on Ant (Appendix B.8, lines 1022–1025).

EncoderHopperAnt
Fractional (FCA)3574.44395.7
GRU, reported3400 ± 1504000 ± 700
GRU, tuned3450 ± 1304080 ± 690
LSTM, tuned3430 ± 1404050 ± 700
  1. Independent evidence. FracEncoder (Wang et al., NeurIPS 2026) studies context encoding for partially observable control. This is a different setting. It uses an encoder of the same Caputo type. Its reported results match or exceed those of recurrent encoders GRU-ODE, RESeL and LRU on most of its tasks. It also reports that a learned order outperforms fixed orders of 0.3, 0.6 and 0.9. This agrees with our Fig. 5 and Fig. 6. FracEncoder is now cited in Sec. 2 (lines 141–143) and Sec. 4.1 (lines 230–231) of the revised PDF.

To Q3 and W2 (cost). FCA takes 3.4 h for 1M steps on Hopper. This is 13% more time than RBT. FCA uses less memory than RBT. No credit-model computation is needed at deployment (lines 375–377). The comparison uses matched settings and one A800 (Table C(a), lines 973–983):

MethodWall-clockPeak allocated GPU memory
SAC (terminal)1.9 h0.9 GB
RRD2.2 h1.0 GB
Diaster2.6 h1.2 GB
RBT3.0 h2.6 GB
GP-LRR3.6 h1.8 GB
FCA (parameter-matched GRU encoder)2.2 h1.0 GB
FCA3.4 h1.6 GB

Training time covers environment interaction, credit updates, label refresh, reward-model training, and SAC updates. Evaluation and one-time startup and compilation costs are excluded (Appendix B.6, lines 949–953).

We time one FCA credit update with a batch of 8 trajectories. The measurement includes the forward pass, backward pass, and optimizer step (Table C(b), lines 973–975 and 984–987):

TT2505001000
Time0.09 s0.20 s0.45 s
Peak allocated GPU memory1.0 GB1.2 GB1.6 GB

The quadratic term re-weights stored responses with scalars. Within each forward pass, fϕf_\phi is evaluated once per position. On Hopper, the fractional encoder costs 1.2 h more than the parameter-matched GRU variant. It gives a higher mean return on Hopper and Ant (Fig. 5; Appendix B.6, lines 954–959).

AuthorsOpenReview ↗

【2/2】Official Comment by Authors

To Q4 and W2 (temporal prior). The learned temporal prior gives higher mean returns than attention-only allocation on both tasks. Attention-only allocation is the continuous boundary case α2=1\alpha_2=1. All temporal weights are equal here (Appendix A.2, lines 770–772). Content matching remains active. We ran this control with the encoder, normalisation, objectives and distillation unchanged (Table A, lines 855–863; Table B(a), lines 890–902):

AllocationHopperAnt
Uniform (ωk≡1\omega_k\equiv1, ek≡1e_k\equiv1)3210 ± 2003380 ± 700
Attention only (ωk≡1\omega_k\equiv1)3420 ± 1204120 ± 740
FCA (learned α2\alpha_2)3574.44395.7
  1. The explicit temporal prior is useful under our parameterization and training budget. We fixed α2\alpha_2 at 0.3 and trained content matching normally. Hopper’s mean return falls to 3050.1 (Section 5.3, lines 472–478).

  2. The backward temporal prior adds one learned scalar and O(T)O(T) work. The quadratic cost comes from the forward encoder (Appendix B.6, lines 954–957).

We welcome further discussion if any concerns remain.

AuthorsOpenReview ↗

Summary of rebuttal

Hi Reviewer 4Xux, thank you again for your time and thoughtful review! We have carefully addressed all your concerns in our rebuttal and provided additional experimental results and clarifications where needed.

Could you please let us know if any concerns remain? If our responses have adequately addressed your questions, we would greatly appreciate it if you could reconsider your rating.

Thank you again for your time and consideration!

AuthorsOpenReview ↗

Official Comment by Authors

Hi Reviewer 4Xux,

Thank you again for your detailed feedback. We have addressed your concerns about temporal influence and encoder comparisons. We clarified the α₁ = 1 boundary case and reported early-versus-late input sensitivity. We also provided tuned GRU/LSTM comparisons. Finally, we clarified the computational costs and the role of the fractional temporal prior.

As the discussion period closes at 8:00 PM today, could you please let us know if any concerns remain? If our responses have adequately addressed your questions, we would be very grateful if you could reconsider your rating.

Thank you again for your time and consideration!

Reviewer_Ci9QOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper tackles the following problem: an RL agent only receives a single total return at the end of an episode, with no rewards in between — how can it tell which steps were good and which were bad?

The paper points out two difficulties: (1) recurrent networks gradually forget early information when compressing history; (2) traditional exponential decay makes very old signals almost useless. Besides, "remembering history" and "assigning credit" may need different time scales.

The authors propose FCA: a left-sided Caputo operator encodes history so early information can directly influence later representations; a right-sided Caputo operator assigns weights to each step, combined with terminal-to-history attention, to split the total return into per-step rewards — and it is proven that these rewards always sum to the total return. The two orders α₁ and α₂ are learned independently to adapt to each task.

Validation: on seven MuJoCo (terminal-only feedback) and four DMC (fixed 1,000-step) tasks, FCA ranks first on nine of eleven; on DMC it beats the strongest baseline RBT by 2.5%–11.2%. Ablations show clear drops when the encoder is replaced by a parameter-matched GRU, when the two orders are tied or fixed, or when the cut point changes.

Critical strengths

Elegant idea: two oppositely directed Caputo operators with independently learned orders explicitly separate "remembering history" from "assigning credit", using power-law kernels for nonlocal encoding and allocation, with return conservation rigorously proven — a principled, verifiable answer to the two-temporal-scale problem.

Critical weaknesses

Statistics and experiments are not rigorous enough: only five seeds, no significance tests or confidence intervals, and the advantages on Humanoid and most DMC tasks fall within seed variation, weakening the "best overall" conclusion; the benchmarks are limited to dense continuous-control tasks like MuJoCo/DMC, with no scenario that truly requires long-delayed credit (e.g., sparse or delayed rewards, longer horizons).

Questions for authors
  1. Are the DMC gains of 2.5%–11.2% over RBT and the Humanoid gap of 5580.5 vs 5565.9 actually significant with only five seeds? Could you add seeds, provide confidence intervals or paired significance tests, and plot curves with variance bands?

  2. Why no comparison with FracTrace, the most directly related fractional method?

4 comments
AuthorsOpenReview ↗

【1/2】Official Comment by Authors

Thank you for the detailed feedback.

To Q1 and W1. We have added taskwise two-sided Welch tests (Table D, lines 998–1010). FCA achieves the highest final mean return on nine of eleven tasks. Eight comparisons favor FCA at unadjusted p<0.05p<0.05 (Appendix B.7, lines 992–995).

  1. Number of seeds. Five seeds is the protocol of the compared methods. Diaster and GP-LRR both report five random seeds. All methods in our paper are run with the same five seeds (Section 5.1, lines 419–421).

  2. Variance bands. They are already in the paper. In Fig. 3 (caption, lines 303–304), solid lines show the means. Shaded areas show the standard deviation over the five seeds. This follows the presentation in Diaster / GP-LRR and RBT.

  3. Significance tests. We have added Welch's t-tests for each task. These compare FCA with the strongest terminal-feedback baseline on that task (Table D, lines 998–1010):

TaskStrongest baselineFCABaselinep
HopperDiaster3574.4 ± 453000 ± 2600.007
Walker2dGP-LRR5211.3 ± 9003650.4 ± 4500.014
AntGP-LRR4395.7 ± 8602867.3 ± 7000.016
SwimmerDiaster288.1 ± 70151.1 ± 400.008
HalfCheetahDiaster9900 ± 2507300 ± 300<0.001
HumanoidGP-LRR5565.9 ± 125580.5 ± 130.095
HumanoidStandupGP-LRR127k ± 3k130k ± 4k0.22
cheetah-runRBT820 ± 15780 ± 190.006
quadruped-walkRBT830 ± 12800 ± 140.007
fish-uprightRBT830 ± 35810 ± 400.43
cartpole-swingupRBT845 ± 15760 ± 25<0.001

Finally, consider a method that consistently outperforms all others on every task. This may indicate a problem with the comparison setup rather than genuine universality. Our claim is not universal superiority. It is a conclusion with a clear mechanistic boundary. FCA's bidirectional fractional mechanism introduces a power-law memory bias. This bias brings the largest and most consistent gains when long-horizon credit assignment is important. These settings include MuJoCo tasks such as Swimmer, Ant, and Walker2d. They also include all four DMC tasks with fixed 1,000-step episodes.

On HumanoidStandup, policy optimization in a high-dimensional control space may become the bottleneck. GP-LRR and Diaster perform better on this task. GP-LRR models state-action reward correlations. Diaster uses return relationships among complementary sub-trajectories. Their stronger performance may be related to these features. We have added this discussion to the revised manuscript (Appendix B.7, lines 1013–1018).

Honestly reporting weaker results on individual tasks should not be viewed as a flaw. It clarifies where FCA has its largest and most consistent advantages. These settings have long horizons and episodic-return-only feedback. They also require credit assignment across time.

To W1 (benchmarks). The underlying MuJoCo and DMC environments provide dense rewards. However, our terminal-feedback wrapper sets every intermediate reward to zero. It gives the agent only the accumulated episodic return at the end of each episode (Section 5.1, lines 402–404). The agent therefore trains with episode-level feedback only. It has no access to any intermediate reward. This is the extreme case of delayed and sparse reward. There is one feedback signal per episode, delayed by the whole episode. Under terminal-only feedback, SAC's return drops sharply on Hopper (Fig. 3). As for longer horizons, 1000 steps is the episode limit of the MuJoCo and DMC tasks. Existing return-decomposition methods are evaluated at this horizon. These include RRD, Diaster, GP-LRR and RBT. Our experiments cover this standard setting on eleven widely used tasks. The four DMC tasks always run the full 1000 steps (Section 5.2, lines 456–459).

AuthorsOpenReview ↗

【2/2】Official Comment by Authors

To Q2 (FracTrace). We do not include FracTrace as a baseline for two reasons. First, it is concurrent work. Second, it addresses a different problem under a different training protocol. A controlled comparison is therefore not possible. The differences are as follows.

  1. Concurrent work. FracTrace appeared online on 2 September 2026 (Neurocomputing, Vol. 706, 134924). This was about three weeks before the submission deadline. It is cited in Section 2, lines 138–141.

  2. Different problem. The methods in our benchmark decompose episodic returns. They receive only R(τ)R(\tau) (Section 5.1, lines 402–404). Their step-wise proxy rewards sum to R(τ)R(\tau) (Eq. 1, Lemma 4.3). FracTrace is an advantage estimator. It replaces GAE's λk\lambda^k weights with fractional weights on future TD errors (its Eq. 20). It does not decompose R(τ)R(\tau) or output proxy rewards.

  3. Different protocol. All methods in our paper share the same off-policy SAC learner. They also share a 1M-step budget. Only the credit-assignment module therefore differs (Section 5.1, lines 407–421; Appendix B.1, lines 799–806). FracTrace is built on on-policy PPO. It uses a 51-quantile distributional critic and a 10M-step budget (its Sec. 5.1). A direct comparison would mix credit-assignment effects with those of the policy optimiser and the budget.

  4. Different operator. FracTrace computes its traces with a discrete convolution truncated at Jmax⁡=512J_{\max}=512 steps. Beyond this window, it relies on critic bootstrapping (its Secs. 3.2 and 5.3). FCA solves two Caputo equations. One defines a forward state–action encoder (Eq. 2). The other defines a backward terminal-value problem (Eq. 5). FCA assigns credit from hTh_T over the whole 1000-step trajectory without truncation.

  5. Shared and independent orders. FracTrace applies one order to both past-feature pooling and future-TD weighting (its Alg. 1). FCA learns two independent orders. Our ablation tests this choice. A shared order gives a lower mean return on both Hopper and Ant (Fig. 5; Section 5.3, lines 479–481).

We welcome further discussion if any concerns remain.

AuthorsOpenReview ↗

Summary of rebuttal

Hi Reviewer Ci9Q, thank you again for your time and thoughtful review! We have carefully addressed all your concerns in our rebuttal and provided additional experimental results and clarifications where needed.

Could you please let us know if any concerns remain? If our responses have adequately addressed your questions, we would greatly appreciate it if you could reconsider your rating.

Thank you again for your time and consideration!

AuthorsOpenReview ↗

Official Comment by Authors

Hi Reviewer Cj9Q,

Thank you again for your careful review. We have addressed your concerns about statistical significance and benchmark settings. We now report task-wise Welch's t-tests based on the same five seeds. We have also clarified the results on tasks where the differences are not significant. In addition, we explained the methodological differences between FCA and FracTrace.

As the discussion period closes at 8:00 PM today, could you please let us know if any concerns remain? If our responses have addressed your questions, we would sincerely appreciate your reconsideration of the rating.

Thank you again for your time and consideration!

Reviewer_FPyaOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper addresses credit assignment with only episodic returns and no intermediate rewards. FCA uses a left-sided fractional encoder to aggregate history, and a right-sided fractional operator to allocate the return using temporal weights and content matching. The two fractional orders are learned independently. The model is trained via return prediction and a midpoint prefix soft target, then distills trajectory-level credits into a step-wise reward network for SAC.

Critical strengths
  1. Demonstrates cross-task gains under matched feedback conditions. The experiments compare with similar methods. FCA shows large improvements in final mean return on several tasks. Evaluations on four DMC tasks with fixed 1000 steps and no early termination also alleviate the concern that gains only come from short episodes. These results support the practical effectiveness of the overall method.

  2. Decouples the temporal scales of forward history encoding and backward credit assignment by learning separate fractional orders. This avoids the suboptimality of forcing both processes to share a single scale. Ablations on shared or fixed orders provide initial support.

  3. Uses distillation to decouple training from deployment. The complex trajectory-level credit assignment (training phase) is decoupled from the lightweight Markovian state-action mapping (deployment phase), ensuring no additional computational overhead during inference.

Critical weaknesses
  1. The core allocator relies on an unvalidated prefix soft target. The prediction head is supervised on the true total return only on the complete trajectory, but its output at the midpoint prefix is used as the target for the first half's credit. The paper does not explain why this prefix prediction is reliable, nor does it show an ablation experiment without this prefix loss. Non-unique allocation does not mean the algorithm is valid, and direct evidence is needed to prove that this soft target stably improves credit and policy learning.

  2. The experiments have not answered the key question regarding the objective bias of the step-wise reward network. The authors acknowledge that reward distillation, discounting, and entropy regularization make the final optimization objective different from the original return. Appendix B.3 also mentions and describes several setups, but lacks the corresponding results. More theoretical analysis and experimental evidence are needed to determine whether this distillation is truly reliable.

  3. Mechanism benefits versus training costs need to be reported. The O(T^{2}) forward encoding could incur huge time and memory costs in long-horizon tasks. The authors should demonstrate that the additional computational overhead is worth the performance gains. Specifically, in longer episodes, while baseline methods degrade significantly, your method's additional compute cost should yield a pronounced performance advantage, proving the non-local mechanism is worth the expense.

Questions for authors
  1. Since the prediction head is only supervised on the true total return on complete trajectories, why can its prefix output serve as a reliable target? How does the policy perform if the prefix loss is removed?

  2. Where are the actual experimental results mentioned in Appendix B.3?

  3. If the episode length increases further (e.g., to 10,000 steps), will the significantly increased time and memory consumption of the O({\mathrm{T}}^{2}) encoding still worth the mechanism's benefits?

4 comments
AuthorsOpenReview ↗

【1/2】Official Comment by Authors

Thank you for the detailed feedback.

To W1 and Q1

  1. Why the target is meaningful. The encoder is causal (Eq. 4). Thus, hkch_{k_c} uses only prefix information. The head uu learns from observed complete-episode returns. Early termination provides episodes of different lengths in the relevant MuJoCo tasks (Appendix B.2, lines 835–838). This motivates applying the same head to prefixes.

The target constrains the total prefix credit. The kernel and content scores determine its internal allocation (lines 321–322).

  1. Existing evidence that the target is useful. Only the prefix term provides training gradients to α2\alpha_2 (Eq. 12; Appendix B.2, lines 835–839). On Hopper, learning α2\alpha_2 achieves 3574.4. The fixed-order results are 3050.1 at 0.3, 3400 at 0.6, and 3450 at 0.9 (Fig. 6; Section 5.3, lines 472–478). Three initializations also yield similar final orders, 0.80–0.87 (Fig. 4, lines 378–392). These results support learning the temporal prior.

  2. Ablation. Removing the entire Lsub\mathcal L_{\mathrm{sub}} reduces Hopper’s return from 3574.4 to 3150±2203150\pm220. Ant’s return falls from 4395.7 to 3320±7403320\pm740. This control removes both terms of Eq. (12).

The additional control removed only the prefix term. It retained terminal supervision:

Lcontrol=Lret+λsub(u(hT)−R(τ))2.\mathcal L_{\mathrm{control}} = \mathcal L_{\mathrm{ret}} +\lambda_{\mathrm{sub}}\bigl(u(h_T)-R(\tau)\bigr)^2.

Hopper = 3150; Ant = 3320 (Appendix B.2, lines 839–845; Table A, lines 855–860).

  1. Direct check of the target. We evaluate u(hkc)u(h_{k_c}) on held-out trajectories. We use the same midpoint as in training. The reference is the sum of original environment rewards before that midpoint.

The Pearson correlations are 0.90 on Hopper and 0.83 on Ant. They show a strong association with accumulated prefix reward. Original stepwise rewards are used only for this evaluation (Appendix B.2, lines 846–849).

To W2 and Q2. Appendix B.3 described the controls but omitted their numerical results. Table B provides these results and the corresponding diagnostics.

Table B(a). Control experiments on Hopper and Ant.

VariantHopperAnt
Full FCA3574.4 ± 454395.7 ± 860
No distillation: direct use of trajectory-level credit labels3390 ± 1704010 ± 720
Credit model frozen at 300k steps3430 ± 1403890 ± 700
Proxy reward model frozen at 300k steps3310 ± 1903560 ± 680
Allow distillation gradients into the credit model3260 ± 2403720 ± 760
Uniform redistribution + distillation3210 ± 2003380 ± 700
FCA: γ=1\gamma=1, no bootstrapping across episode boundaries2850 ± 4203100 ± 800
FCA: fixed entropy temperature of 0.23480 ± 1104150 ± 780
SAC (terminal): γ=1\gamma=1340 ± 15030 ± 70

Values are final environment returns, reported as mean ± sample standard deviation across training seeds. In the gradient-flow control, the distillation loss updates both the credit model Θ\Theta and the reward model θ\theta.

Table B(b). Prediction and policy-ranking diagnostics.

DiagnosticHopperAnt
Distillation MSE / credit-label variance0.210.33
Normalised mean absolute return discrepancy5.8%9.4%
Policy-ranking Spearman ρ\rho: true return vs. undiscounted proxy return0.940.90
Policy-ranking Spearman ρ\rho: true return vs. discounted proxy return0.890.85

Prediction errors are measured on held-out trajectories. Let R^i=∑krθ(sik,aik)\hat R_i=\sum_k r_\theta(s_{ik},a_{ik}). The normalised return discrepancy is

1N∑i∣R^i−Ri∣1N∑i∣Ri∣\frac{\frac{1}{N}\sum_i|\hat R_i-R_i|}{\frac{1}{N}\sum_i|R_i|}

The denominator is shared across all diagnostic trajectories. Ranking correlations compare mean returns across a fixed set of policies. Discounted proxy returns exclude the entropy term.

1. Evidence. All performance results use the original environment returns (§5.1). Full FCA has a higher mean return than the no-distillation control on both tasks. The undiscounted finite-horizon control also has lower mean returns. SAC with terminal feedback remains low-performing in this setting. The diagnostics directly measure alignment with the original return. The normalised return discrepancies are 5.8% on Hopper and 9.4% on Ant. Policy-ranking correlations are 0.94 and 0.90 before discounting. They remain 0.89 and 0.85 after discounting.

2. Analysis. Appendix A.3 considers a fixed distillation distribution. The population optimum is

rθ∗(s,a)=EDstep[r~∣s,a].r_{\theta^{\ast}}(s,a)=\mathbb{E}_{D_{\mathrm{step}}}[\tilde{r}\mid s,a].

It therefore preserves mean step credit under that distribution. This result does not establish policy invariance for the discounted, entropy-regularized objective. We assess the practical effects through the controls and diagnostics above.

We have added Table B to Appendix B.3. Appendix B.4 now specifies the metric definitions and evaluation protocol. We have also added a direct reference to these results in §4.4.

AuthorsOpenReview ↗

【2/2】Official Comment by Authors

To W3 and Q3 .

FCA takes 3.4 hours and 1.6 GB on Hopper for 1M interactions. Compared with RBT, this is about 13% more training time and 38% less peak allocated memory. Table C reports all comparisons.

Table C(a). Training cost on Hopper. All methods use one A800 GPU and 1M environment interactions.

MethodTraining time (h)Peak allocated GPU memory (GB)
SAC (terminal)1.90.9
RRD2.21.0
Diaster2.61.2
RBT3.02.6
GP-LRR3.61.8
FCA with the parameter-matched GRU encoder2.21.0
FCA3.41.6

Training time covers the full training loop through 1M interactions. It includes environment interaction, credit updates, label refresh, reward-model training, and SAC updates. Evaluation and one-time startup and compilation costs are excluded.

Table C(b). One FCA credit update with a batch of eight trajectories.

Trajectory length TT2505001000
Update time (s)0.090.200.45
Peak allocated GPU memory (GB)1.01.21.6

Each update includes the forward pass, backward pass, and optimizer step. Memory measurements use allocated memory.

Revised-PDF locations: Table C (lines 973–987). The measurement protocol is in Appendix B.6 (lines 949–953).

  1. Source of the cost. The direct encoder has O(T2)O(T^{2}) complexity (§4.4, lines 375–377; Appendix A.1). Within each forward pass, fϕf_{\phi} is evaluated once per position. The quadratic work aggregates stored response vectors with temporal weights. These computations are used during training and label generation. Execution uses only the actor and the current observation.

  2. Performance benefit. On Hopper, FCA adds 1.2 hours over the parameter-matched GRU variant. Its mean return increases from 3400 to 3574.4. On Ant, the mean return increases from 4000 to 4395.7 (Fig. 5; Appendix B.6, lines 957–959). These comparisons use the same interaction budget.

  3. On 10,000-step episodes. 1000 steps is the episode limit of the MuJoCo and DMC tasks, so many published papers, including the methods we compare with, use trajectories of at most 1000 steps. We therefore do not evaluate 10,000-step episodes (Appendix B.6, lines 959–961). Our experiments cover this standard setting on eleven widely used tasks, and we believe our method has strong potential for tasks with longer horizons.

We welcome further discussion if any concerns remain.

AuthorsOpenReview ↗

Summary of rebuttal

Hi Reviewer FPya, thank you again for your time and thoughtful review! We have carefully addressed all your concerns in our rebuttal and provided additional experimental results and clarifications where needed.

Could you please let us know if any concerns remain? If our responses have adequately addressed your questions, we would greatly appreciate it if you could reconsider your rating.

Thank you again for your time and consideration!

AuthorsOpenReview ↗

Official Comment by Authors

Hi Reviewer FPya,

Thank you again for your thoughtful review. We have carefully addressed all your concerns. We clarified the prefix target and its reliability. We also reported the relevant ablations and trajectory diagnostics. Further clarifications on reward distillation and computational costs are included in the revised PDF.

As the discussion period closes at 8:00 PM today, could you please let us know if any concerns remain? If our responses have addressed your questions, we would greatly appreciate it if you could reconsider your rating.

Thank you again for your time and consideration!

Comments on the paper

2 comments
AuthorsOpenReview ↗

Brief Summary of the Revised PDF

We thank all reviewers for their time and constructive feedback.

Revisions are marked in blue. Changes to the main text are brief. They clarify related work, methodological details, and error-bar definitions. We also added pointers to the supplementary results.

Most additions are in the appendices. Appendix A provides theoretical clarifications. Appendix B includes six new tables:

  1. Table A reports prefix-loss ablations, the integer-order boundary, and temporal-kernel comparisons.
  2. Table B reports diagnostics for reward distillation and the optimization objective.
  3. Table C reports training time, peak GPU memory, and update costs at different trajectory lengths.
  4. Table D reports final returns across five training seeds and taskwise two-sided Welch tests.
  5. Table E reports comparisons with tuned GRU and LSTM encoders.
  6. Table F reports the ratio of sensitivity to early versus late inputs.
AuthorsOpenReview ↗

Summary of the Rebuttal and Discussion

Dear Area Chair,

Thank you for organizing the review and coordinating the discussion. As the discussion period draws to a close, we would like to summarize our main contributions and responses to the reviewers. We hope this overview will assist your final assessment.

1. Main Contributions

We study temporal credit assignment in reinforcement learning with episodic-return-only feedback. The agent receives only the total return at the end of each episode. It receives no intermediate rewards.

We propose Fractional Credit Assignment (FCA). It uses two oppositely directed Caputo operators with independently learned fractional orders. The left-sided operator uses a power-law kernel to encode history and retain information from earlier steps. The right-sided operator combines terminal-to-history content attention with power-law temporal weights. This mechanism identifies key steps based on both their semantic relevance to the terminal state and their temporal distance.

This design separates history encoding from credit assignment. It allows them to operate on different temporal scales. Before distillation, the assigned step-wise credits sum exactly to the episodic return. The distilled state-action reward model enables lightweight deployment without additional inference-time overhead.

Across the reviews, reviewers recognized several strengths of our work. These include the separation of temporal scales, the theoretical guarantee of return conservation, and the empirical effectiveness across tasks.

We evaluated FCA on seven MuJoCo tasks and four DMC tasks, including four fixed 1,000-step DMC tasks. FCA achieved the highest final mean return among the compared terminal-feedback methods on nine of the eleven tasks.

2. Main Reviewer Concerns and Our Responses

2.1. Contribution of the fractional temporal kernel

To address questions from feGP and 4Xux, we varied only the temporal weighting rule. The encoder architecture, content-based attention, and training objective remained unchanged.

Compared with a learnable exponential kernel, the fractional kernel improved final mean return by 5.8% on Hopper and 9.6% on Ant. Compared with uniform temporal weights, the gains were 4.5% and 6.7%, respectively (Appendix B.5 and Table A).

2.2. Reliability of prefix targets and distilled rewards

To address FPya's concern, we evaluated prefix predictions at the midpoint of held-out trajectories. We compared them with the accumulated environment rewards up to that point. The Pearson correlations were 0.90 on Hopper and 0.83 on Ant.

We also removed the prefix loss while retaining terminal supervision. Mean returns decreased from 3574.4 ± 45 to 3150 ± 220 on Hopper, and from 4395.7 ± 860 to 3320 ± 740 on Ant.

For the distilled reward model, the normalized mean absolute return discrepancies were 5.8% on Hopper and 9.4% on Ant. The Spearman correlations between true-return rankings and discounted proxy-return rankings were 0.89 and 0.85, respectively (Appendices B.2–B.4).

2.3. Other revisions and additional analyses

We examined the α₁ = 1 boundary case. We also added comparisons with tuned GRU and LSTM encoders, together with early-versus-late input sensitivity diagnostics (Appendices B.7–B.8 and Tables E–F).

We added task-wise Welch's t-tests using the original five random seeds. Eight of eleven comparisons reached the unadjusted significance threshold of p < 0.05. We also explicitly reported the tasks without significant differences (Appendix B.7 and Table D).

We further measured training time and GPU memory under matched settings. On Hopper, FCA took 13% longer than RBT for one million environment steps. However, it used 38% less peak allocated GPU memory (Appendix B.6 and Table C).

Finally, Section 2 clarifies the distinction between FCA and FracTrace. FracTrace estimates advantages using future TD errors. FCA decomposes episodic returns into step-wise credits. They also use different training protocols.

3. Discussion Update

Reviewer feGP was the only reviewer to provide further feedback during the discussion. They welcomed our new comparisons with the exponential kernel and uniform temporal weights. They subsequently raised their rating from 2 to 3.

As of this writing, the other three reviewers have not provided further comments.

We sincerely thank Reviewer feGP for reconsidering their assessment. We also thank all reviewers for their time and constructive feedback.

We hope this summary and our detailed rebuttal will help with your final assessment. Thank you again for your time and consideration.

Sincerely,
The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.