- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies temporal credit assignment from episodic returns. It learns two independent fractional orders to control temporal weighting in history encoding and return allocation. Content matching between terminal and historical representations produces stepwise rewards, which are then distilled into a state–action reward model for policy training. The paper evaluates the approach on delayed-reward control tasks and examines the effects of learning separate fractional orders.
Critical strengths
Learning separate temporal scales for history encoding and return allocation is a well-motivated design. The shared-order and fixed-order ablations support this choice, while the overall results demonstrate its practical potential for learning from episodic returns.
Critical weaknesses
The experiments leave the benefit of the fractional kernels over simpler temporal weighting insufficiently established.
Figures 5 and 6 provide useful comparisons with shared or fixed fractional orders and a GRU replacement. These results support learning the two orders independently within the proposed design. However, they do not directly establish how the fractional weighting compares with a simpler learnable temporal kernel. The GRU replacement changes the representation architecture as well, making it difficult to assess the temporal kernel separately.
A focused comparison with a learnable exponential kernel, while retaining the same encoder, content matching, and training objective, would address this concern. Uniform temporal weights with content matching retained would also be informative. This latter control differs from assigning the episodic return uniformly across steps, as described in Appendix B.3.
Questions for authors
With the encoder, content matching, and training objective held fixed, how does the fractional kernel compare with a learnable exponential kernel or uniform temporal weights on existing tasks such as Hopper and Ant? Do any existing experiments cover this comparison?
Are results available for the uniform reward-allocation control described in Appendix B.3?
I’d be happy to raise the score if you can answer these questions.
5 comments
Summary of rebuttal
Hi Reviewer feGP, thank you again for your time and thoughtful review! We have carefully addressed all your concerns in our rebuttal and provided additional experimental results and clarifications where needed.
Could you please let us know if any concerns remain? If our responses have adequately addressed your questions, we would greatly appreciate it if you could reconsider your rating.
Thank you again for your time, consideration, and support of our work!
Official Comment by Authors
Hi Reviewer feGP,
Thank you again for your constructive feedback. We have addressed your main concern about the fractional temporal kernel. We now provide direct comparisons with a learnable exponential kernel and uniform temporal weights. All other components and training settings remain unchanged. The results show consistent gains from the fractional kernel on both Hopper and Ant. We have also clarified the uniform reward-allocation results.
As the discussion period closes at 8:00 PM today, could you please let us know if any concerns remain? We hope these clarifications have addressed your questions. If so, we would be very grateful if you could reconsider your rating.
Thank you again for your time and valuable feedback!
Official Comment by Authors
We thank you for the valuable feedback and for your support of our work. The fractional kernel achieves higher mean returns than both matched alternatives on Hopper and Ant.
1. Exponential kernel and uniform temporal weights. We replace only the backward temporal weighting rule. The encoder architecture, content matching, normalization, losses, and distillation remain unchanged. Each variant is trained separately under the same protocol.
Table A. Temporal-weighting controls with content matching retained.
The gains over uniform temporal weights are 4.5% on Hopper and 6.7% on Ant. The gains over the exponential kernel are 5.8% and 9.6%, respectively.
The exponential control learns a time constant in absolute steps. The uniform-time control sets ωk=1 and retains content matching. Its credits are
r~k=R(τ)∑jejek.
This is the α2=1 boundary case of the temporal kernel.
Figures 5 and 6 examined encoder replacement and fractional-order choices. They did not include these two direct kernel controls. Table A now provides the requested comparison.
2. Uniform return allocation in Appendix B.3. This is a separate control. It assigns
r~k=TR(τ)
before distillation. It does not use content matching.
Its returns are 3210±200 on Hopper and 3380±700 on Ant. These numerical results were omitted from the original Appendix B.3. We have included them in Table B.
We have added the kernel definitions and matched comparisons to Appendix B.5. We have also clarified the roles of the different ablations in §5.3.
Revised-PDF locations: Table A (lines 855–863) and Table B(a) (lines 890–902). Kernel definitions and matched comparisons are in Appendix B.5 (lines 931–945). The roles of the ablations are clarified in Section 5.3 (lines 472–485).
We welcome further discussion if any concerns remain.