acceptodds
Under review as a conference paper at ICLR 2027

Off-policy RL Framework for Learning from Diverse Temporal Perspectives

Abstract

Reinforcement learning typically trains a policy with a fixed discount factor, assigning exponentially less weight to rewards further in the future. We introduce Temporal Perspective Learning (TPL), an off-policy actor–critic framework that jointly learns policies under different temporal weightings of the same rewards to improve learning for the standard discounted objective. TPL refers to these temporal weightings as temporal perspectives and jointly trains a shared actor and a shared critic conditioned on these perspectives. Unlike prior approaches that decompose a single policy’s value across multiple horizons, TPL learns perspective-specific policies and evaluates the consequences of following each policy. This shared architecture allows learning across diverse temporal preferences to influence the policy associated with the standard discounted objective. Experiments on 24 continuous-control tasks across three benchmark suites demonstrate improved performance over the evaluated baselines. Ablations further support the benefits of jointly learning perspective-specific policies and values. The source code is included in the supplementary materials.

Primary area:
reinforcement learning
Keywords:
Reinforcement Learning, Timescales

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_TSh1OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces Temporal Perspective Learning (TPL), an off-policy actor-critic framework that conditions both the actor and critic on a temporal weighting of future rewards. The central premise is that learning a family of policies under diverse temporal perspectives—each emphasizing different reward horizons—can improve the performance of a designated reference policy under standard exponential discounting. To render this tractable, the authors construct a residual-basis mixture family grounded in the residual decomposition of TD(Δ), where each perspective is parameterized by a simplex-valued vector α that recombines short-horizon and incremental-horizon value components. The reference perspective corresponds to the simplex centroid, which exactly recovers the exponential kernel. Training proceeds off-policy: only the reference-conditioned policy interacts with the environment, while all perspectives are trained from a shared replay buffer. Experiments span 24 continuous-control tasks from AdroitHand, DMControl-Hard, and HumanoidBench, with additional integrations into Simba and TD-MPC2, and a preliminary Atari 100k evaluation. Ablations examine the roles of perspective conditioning, mixture family design, and perspective diversity.

Critical strengths
  1. The paper identifies a genuine limitation of standard discounting—its inability to simultaneously prioritize short-term reactivity and long-term foresight—and proposes a principled mechanism to address it. Conditioning both actor and critic on the temporal weighting, rather than merely reweighting returns under a fixed policy, is a subtle but consequential distinction. This design allows auxiliary perspectives to shape the reference policy through shared parameters, and the residual-basis construction ensures that the reference objective is recovered exactly at the centroid. The theoretical scaffolding in Appendix I, particularly the reference performance bound under policy switching, provides useful intuition even if not fully operationalized.
  2. The evaluation covers three diverse continuous-control suites with 24 tasks, four seeds per task, and comparisons against SAC, SAC+TD(Δ), CompQ, and MR.Q. The integration experiments with Simba and TD-MPC2 demonstrate that the framework is not tied to a single backbone. The ablation results, while not uniformly conclusive, do indicate that the full TPL configuration outperforms variants lacking perspective conditioning or residual diversity. The reported improvements, particularly on HBench-Periodic and several HumanoidBench tasks, are non-trivial and suggest that the approach captures something beyond noise.
Critical weaknesses
  1. The paper states that exact prediction of Q_{w_α}^{π_θ^α} does not guarantee monotonic policy improvement when the update is used for non-exponential perspectives, and Appendix I.2 provides a counterexample where greedy improvement fails. This is a material gap: the central mechanism—using recombined residual values to update the actor—lacks a general improvement guarantee, and the paper does not offer an alternative surrogate with theoretical backing.
  2. Table 1 shows that Full TPL substantially outperforms No conditioning (0.55 vs. 0.30), but this comparison conflates the presence of auxiliary perspectives with the presence of a residual critic architecture and additional parameters. The Homogeneous horizon basis variant, which retains conditioning but removes temporal distinctions among components, scores 0.24—lower than No conditioning—suggesting that the mere existence of multiple components can be detrimental without meaningful temporal separation. However, the paper does not isolate whether the gains stem from temporal diversity per se or from increased model capacity and auxiliary gradient signals.
  3. MR.Q is included, but other contemporary off-policy algorithms with strong performance on continuous control—such as BRO, QSM, or even well-tuned REDQ variants—are absent. More importantly, the baseline SAC+TD(Δ) is implemented as an off-policy variant by the authors, which may not reflect the original method’s best configuration. The paper reports that TPL uses separate critic heads and adaptive residual bases, and while Table 2 shows these contribute only marginally on HBench-Periodic, the interaction between these design choices and the baseline implementations is not fully disentangled.
  4. Table 2 indicates that removing both mechanisms reduces HBench-Periodic performance from 0.55 to 0.54—a negligible difference given the confidence intervals. Yet these components add architectural complexity and hyperparameters (e.g., horizon update frequency, dual gradient descent).
  5. The reported aggregate metrics (Mean HNS 0.793 vs. 0.553 for SAC) are based on a single variant, four seeds, and omit comparison to modern sample-efficient discrete-control methods.
Questions for authors
  1. The paper acknowledges that the actor surrogate lacks a monotonic improvement guarantee for non-exponential perspectives. Could the authors provide a more detailed characterization of the conditions under which the joint update remains stable? Specifically, does the shared-parameter architecture impose an implicit regularization that mitigates the failure mode in Example I.2, and can this be quantified empirically or theoretically?
  2. In the ablation of perspective diversity (Figure 4b), performance increases monotonically with λ, the simplex contraction parameter. Does this trend continue beyond λ = 1, i.e., would training with perspectives sampled from a broader distribution than Dirichlet(1) yield further gains, or does the simplex constraint impose a natural ceiling?
  3. The residual-basis construction requires a sequence of discount factors γ_1 < … < γ_L. The adaptive horizon mechanism adjusts these during training, but the initialization and update frequency are fixed (G = 500). How sensitive is performance to these hyperparameters, and does the adaptive mechanism ever converge to a degenerate grid where one component dominates?
  4. The comparison against SAC+TD(Δ) uses an off-policy implementation authored by the present authors. Could the authors provide a sanity check against the original TD(Δ) results on a subset of tasks, or at minimum, clarify whether the off-policy adaptation required any hyperparameter tuning that might disadvantage the baseline?
6 comments
AuthorsOpenReview ↗

Response to Reviewer TSh1 (1/3)

We sincerely thank you for your thoughtful and constructive feedback. Below, we address your key concerns through additional experiments and analyses that further clarify the strengths and limitations of TPL.


TSh1-1. Learning Stability of Non-Exponential Temporal Perspectives

TPL aims to improve reference policy learning by jointly learning diverse perspectives within a shared model, rather than guaranteeing individual policy improvement for each non-exponential temporal perspective. Our current analysis has not established a useful theoretical property showing that this shared structure alleviates the policy improvement issue for individual perspectives. Instead, we empirically analyzed whether the joint update used in practice is compatible with improving the reference objective.

In Response to Reviewer YTjr-1, we analyzed gradient alignment for individual perspectives and their one-step transfer to the reference actor loss. Although negative transfer was observed for some perspectives far from the reference, the gradient of the full P(α)P(α) mixture had cosine similarities of 0.78 and 0.90 with the reference gradient at the 200k and 1M checkpoints, respectively, with a conflict rate of 0% at both checkpoints. Furthermore, after applying actor updates of the same magnitude, the reference actor loss measured on a separate evaluation batch decreased by 1.01 [0.71, 1.30] and 0.71 [0.37, 0.99].

These results do not guarantee monotonic policy improvement for individual non-exponential perspectives or the general stability of joint training. However, they provide empirical evidence that, even when some perspectives produce negative gradients, the overall update combined according to the actual training distribution acts to reduce the reference actor loss. This supports TPL's design, which targets improved reference performance through joint learning rather than individual improvement of auxiliary perspectives.


TSh1-2. A Modified Objective with a Policy Improvement Guarantee for Non-Exponential Temporal Perspectives

Original TPL jointly learns policies with each perspective held fixed. Under a non-exponential kernel, the relative weights assigned to the remaining rewards change over time, so an actor update based on value evaluation at a fixed perspective may not yield policy improvement. As an alternative that addresses this issue, we present a modified objective in which the perspective shifts over time, together with a policy improvement guarantee.

Shifted-perspective variant. Motivated by the reviewer's question, we revisited the structure of the residual kernel and found that a kernel shifted by one step can also be represented within the same family of residual basis mixtures: wα(t+1)=s(α) wF(α)(t).w_α(t+1)=s(α)\,w_{F(α)}(t). Here, F(α)F(α) is the perspective shifted by one step, and s(α)s(α) is a scale factor. Following this transformation, we change the conditioning of the bootstrapped next Q-value in the residual TD target of Eq. (14) from

\quad\longrightarrow\quad \widehat Q_{\bar{ψ}}^{(ℓ)}(s',a',F(α)),$$ and sample the next action from $a'∼π_θ^{F(α)}(·∣ s')$. The critic therefore evaluates a policy whose perspective shifts over time as follows, and the modified objective is formulated to maximize the return of this shifting policy: $$α_{t+1}=F(α_t), \qquad a_t∼π_θ^{α_t}(·∣ s_t).$$ **Policy improvement guarantee.** Under the reward-only formulation in the main text, combining an exact critic with exact greedy improvement at every state–perspective pair ensures that the shifted-policy return evaluated under the initial kernel does not decrease. Consequently, the mixture objective obtained by averaging this return over initial states and perspectives is also non-decreasing. **Experimental results.** We compared the two methods on the three HBench-Periodic tasks using the same training budget. (4 seeds) | Method | HBench-Periodic (3 tasks) | | --- | ---: | | Original TPL | 0.55 [0.44, 0.66] | | Shifted-perspective variant | 0.48 [0.40, 0.57] | The shifted variant had a lower observed mean, and the reported intervals overlapped. This variant provides a construction with a policy improvement guarantee under the conditions stated above and may facilitate extensions to deterministic policies or discrete action spaces. TPL focuses on jointly learning policies for **fixed** temporal perspectives to capture the differences among perspectives. In contrast, this variant evaluates and improves policies whose internal perspectives gradually converge toward the reference perspective over time. Analyzing how this difference in objectives affects the final reference policy, and exploring ways to combine the two formulations for more effective transfer to the reference, are interesting directions for future work. We have added the shifted-perspective objective, policy improvement theorem, proof, and experimental results to **Appendix N** of the revised manuscript.
AuthorsOpenReview ↗

Response to Reviewer TSh1 (2/3)

TSh1-3. Controlled Setup for No Conditioning and Comparison with TD(Δ) Implementations

Controlled setup for No conditioning. No conditioning is an ablation that fixes the training perspective to the reference condition while retaining TPL's architecture and all other training settings. Thus, the performance difference from Full TPL is attributable to auxiliary perspective learning, with other factors controlled.

Configuration of SAC+TD(Δ). SAC+TD(Δ) applies the residual decomposition of the original TD(Δ) to SAC's one-step target. It does not incorporate the design components in Appendix A of TPL and uses shared critic heads and a fixed basis, as in the original TD(Δ). To improve learning, we added layer normalization, as in the baseline SAC.

The specific configurations of each baseline and ablation are as follows:

ComponentTPLNo conditioningSAC+TD(Δ)PPO+TD(Δ)
Perspective trainingDiverse conditionsReference onlyReference onlyReference only
Residual criticYesYesYesYes
Critic architectureSeparate headsSeparate headsShared headsShared heads
Residual CDQ bias correctionYesYesNoNot applicable
Adaptive basisYesYesNoNo
Training regimeOff-policyOff-policyOff-policyOn-policy
Number of parameters2.5M2.5M0.7M0.02M

Comparison with the original on-policy configuration. As a sanity check against the original TD(Δ) configuration, we evaluated PPO+TD(Δ), based on the original authors' code, under the same budget of 1M transitions. (4 seeds)

MethodAdroitHandDMControl-HardHumanoidBench
SAC+TD(Δ)0.76 [0.75, 0.78]0.55 [0.53, 0.57]0.54 [0.52, 0.55]
PPO+TD(Δ)0.09 [0.06, 0.13]0.05 [0.05, 0.06]0.05 [0.05, 0.06]
PPO+TD(Δ) (0.38M parameters)0.07 [0.03, 0.11]0.04 [0.04, 0.04]0.03 [0.03, 0.04]

PPO+TD(Δ) performed worse than SAC+TD(Δ) under the same transition budget.


TSh1-4. Effect of Expanding Perspective Diversity with λ>1\lambda > 1

Following your suggestion, we expanded the perspective sampling range around the reference beyond the simplex (λ>1\lambda > 1):

αλ=αref+λ(α−αref).α_λ=α_{\mathrm{ref}}+λ(α-α_{\mathrm{ref}}).

In this case, some perspective coefficients can become negative, and the kernel can also take negative values over some intervals.

λ\lambdaHBench-Periodic (3 tasks, 4 seeds)
00.30 [0.28, 0.31]
0.250.40 [0.37, 0.43]
0.50.45 [0.37, 0.53]
0.750.48 [0.41, 0.56]
1.0 (default)0.55 [0.44, 0.66]
1.250.55 [0.47, 0.61]
1.50.60 [0.49, 0.71]
1.750.37 [0.22, 0.52]
2.00.27 [0.12, 0.44]

At λ=1.25\lambda=1.25 and 1.51.5, mean performance was 0.55 and 0.60, respectively, matching or exceeding the original value of 0.55 at λ=1\lambda=1. In contrast, performance decreased substantially at λ≥1.75\lambda\ge1.75. This shows that perspectives outside the simplex, which allow negative w(t)w(t), can still be useful for learning, but excessively broad sampling ranges can degrade performance.

We have added the definition of the perspective expansion range and the experimental results to Appendix M of the revised manuscript.

AuthorsOpenReview ↗

Response to Reviewer TSh1 (3/3)

TSh1-5. Sensitivity of Adaptive Horizons to Update Interval and Initialization

To assess the hyperparameter sensitivity of the adaptive horizons, we varied the update interval and initialization separately.

Update interval. We evaluated GG values ranging from 10 to 1000. (4 seeds)

Update interval GGHBench-Periodic (3 tasks)
100.51 [0.45, 0.58]
1000.54 [0.49, 0.60]
500 (default)0.55 [0.44, 0.66]
10000.54 [0.47, 0.60]

Mean performance remained within the range of 0.51–0.55. This indicates that the adaptive horizons are not highly sensitive to the update interval.

Basis initialization. We replaced the default Doubling initialization (γℓ−1=γℓ2γ_{ℓ-1}=γ_ℓ^2) with Tripling (γℓ−1=γℓ3γ_{ℓ-1}=γ_ℓ^3) or Linear initialization (γℓ=1−Lℓ(1−γL)γ_ℓ=1-\frac{L}{ℓ}(1-γ_L)) and also compared configurations without the adaptive basis.

Basis initializationAdaptive basisHBench-Periodic (3 tasks)
Doubling (default)Yes0.55 [0.44, 0.66]
TriplingYes0.54 [0.46, 0.62]
TriplingNo0.41 [0.38, 0.45]
LinearYes0.46 [0.40, 0.51]
LinearNo0.42 [0.34, 0.50]

Tripling achieved a mean performance of 0.54, similar to the default Doubling result of 0.55, while Linear achieved a lower mean of 0.46. Using the adaptive basis increased mean performance from 0.41 to 0.54 for Tripling and from 0.42 to 0.46 for Linear.

Grid convergence. The grids converged to different configurations depending on the environment, and no degenerate grids were observed. These differences across environments are also reflected in the kernel mass results for one-hot perspectives in Response to Reviewer nBbP-1.

The adaptive basis is designed to reduce the need for manual horizon tuning and regulate learning signals across residual components. The results above on update intervals and grid convergence support the practical stability of this adaptive configuration.

We have incorporated the update interval and initialization experiments into Appendix J.4, and the grid convergence and kernel mass diagnostics into Appendix J.3 of the revised manuscript.


TSh1-6. Comparisons with Recent Algorithms

The original continuous control comparisons focused on multi-horizon prediction and auxiliary learning methods. In Section 5.6, we instead evaluated the benefits of incorporating TPL into recent methods, namely Simba and TD-MPC2.

Direct comparisons on continuous control. To strengthen the direct comparisons with recent algorithms, we present additional results for BRO[1] and QSM[2], obtained using their official code, alongside the Simba and TD-MPC2 results included in the manuscript. (4 seeds)

MethodHBench-Periodic (3 tasks)
TPL0.55 [0.44, 0.66]
Mr.Q0.32 [0.30, 0.33]
Simba0.43 [0.41, 0.45]
TD-MPC20.40 [0.36, 0.45]
BRO0.31 [0.29, 0.34]
QSM0.07 [0.06, 0.09]
  • For BRO, we evaluated the version with 2.6M total trainable parameters (the 1.14M-parameter critic version in the paper) and the default replay ratio of 10.
  • For QSM, we followed the official implementation and hyperparameters.

TPL achieved higher mean performance than the additional baselines, demonstrating its competitiveness in direct comparisons with recent algorithms.

Atari-100k comparisons. The Atari-100k results in the original Appendix K are a preliminary evaluation of applicability to environments with image observations and discrete actions, and their performance is lower than that of recent methods. The table below presents the aggregate human-normalized scores (HNS) from our existing evaluation, together with the published results for SPR, SR-SPR, and BBF reported in Table A.1 of the BBF paper [3].

MethodMean HNSMedian HNSIQM HNS
SAC0.5530.1860.237
SAC+TPL0.7930.2320.390
SPR0.6160.3960.337
SR-SPR1.2720.6850.631
BBF2.2470.9171.045

In Appendix O of the revised manuscript, we have added the published results for SPR, SR-SPR, and BBF to enable comparisons of both aggregate metrics and per-game results.


We thank you again for your valuable feedback. We would be happy to provide further clarification or address any additional questions you may have.


References

[1] Michal Nauman, Mateusz Ostaszewski, Krzysztof Jankowski, Piotr Miłoś, and Marek Cygan. “Bigger, Regularized, Optimistic: Scaling for Compute and Sample-Efficient Continuous Control.” NeurIPS, 2024.

[2] Michael Psenka, Alejandro Escontrela, Pieter Abbeel, and Yi Ma. “Learning a Diffusion Model Policy from Rewards via Q-Score Matching.” ICML, 2024.

[3] Max Schwarzer, Johan Samir Obando Ceron, Aaron Courville, Marc G. Bellemare, Rishabh Agarwal, and Pablo Samuel Castro. “Bigger, Better, Faster: Human-level Atari with human-level efficiency.” ICML, 2023.

Reviewer_nBbPOpenReview ↗
Rating
4(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes Temporal Perspective Learning (TPL), an off-policy actor–critic framework that learns policies under different temporal reward weightings to improve a fixed reference policy. Both actor and critic are perspective-conditioned, and a residual basis derived from TD(Δ) supports one-step value updates while recovering the reference discount through uniform recombination. Only the reference policy collects experience and is evaluated. Experiments cover 24 continuous-control tasks, with ablations, approximately parameter-matched comparisons, and extensions to Simba and TD-MPC2.

Critical strengths

1.A useful extension from multi-horizon prediction to auxiliary control. Conditioning both actor and critic allows each temporal perspective to learn and evaluate its own policy. The residual construction makes this practical with one-step TD updates. Table 1 provides evidence that the full formulation contributes beyond the residual architecture alone.

2.Broad empirical support. Improvements are reported across three task suites and extend to Simba and TD-MPC2. Comparisons against larger SAC and SAC+TD(Δ) critics also reduce the concern that additional capacity alone explains the gains.

Critical weaknesses

1.The source of the benefit could be better isolated. Equation (7) does not generally preserve kernel mass across perspectives, and the adaptive basis in Appendix A.3 balances policy-gradient magnitudes rather than explicitly equalizing kernel mass. The contraction experiment preserves the mean kernel, but can change both temporal-shape diversity and per-perspective mass variation. Such variation may interact with shared optimization and the entropy-regularized surrogate in Appendix G. Its practical importance remains unclear, so the evidence supports the complete construction more strongly than an attribution specifically to temporal-shape diversity.

2.The value diagnostic has a limited scope. Appendix J reasonably compares conditioned policies using a common reference kernel. However, this summed residual readout differs from the weighted readout used by auxiliary actors; accuracy of the sum does not establish accuracy of other combinations. The effect of truncating Monte Carlo returns at 200 steps is also unquantified. These leave uncertainty about the auxiliary value estimates, without establishing that they are inaccurate.

Questions for authors

1.How much does kernel mass vary across perspectives during training? If the variation is substantial, can existing evidence or a small control on HBench-Periodic help separate temporal-shape effects from scale effects?

2.Can you evaluate the auxiliary actors’ perspective-weighted soft-value readouts against Monte Carlo targets with matching temporal weights and entropy terms, report unnormalized RMSEs, and check sensitivity to rollout length? This would clarify whether the diagnostic conclusions extend to the values driving auxiliary updates.

4 comments
AuthorsOpenReview ↗

Response to Reviewer nBbP (1/2)

We sincerely thank you for your thoughtful and constructive feedback. Below, we address your key concerns through additional experiments and analyses that further clarify the strengths and limitations of TPL.


nBbP-1. Differences in Kernel Mass Across Perspectives and Performance After Normalization

Although kernel mass varied across perspectives, TPL maintained its performance under two variants that normalize these differences.

Kernel mass and learning signals across perspectives. Because the critic learns values for each residual horizon, differences in kernel mass across perspectives do not directly affect the critic loss. In contrast, the actor uses Q^wαπθα(s,a)=L∑ℓ=1LαℓQ^ψ(ℓ)(s,a,α)\widehat Q_{w_\alpha}^{\pi_\theta^\alpha}(s,a)=L\sum_{\ell=1}^{L}\alpha_\ell\widehat Q_\psi^{(\ell)}(s,a,\alpha), so kernel mass can affect the relative weighting of the policy gradient and entropy regularization.

We measured kernel mass, action-gradient norm, and entropy per action dimension for each one-hot perspective on DMControl humanoid-walk and HBench-slide. (4 seeds)

DMControl humanoid-walkKernel massAction-gradient normEntropy/action dim
Reference100.00±0.000.88±0.12−0.51±0.01
α=(1,0,0,0)\alpha=(1,0,0,0)25.64±0.871.61±0.18−0.50±0.05
α=(0,1,0,0)\alpha=(0,1,0,0)27.53±1.351.56±0.22−0.61±0.03
α=(0,0,1,0)\alpha=(0,0,1,0)87.56±3.821.47±0.21−0.65±0.02
α=(0,0,0,1)\alpha=(0,0,0,1)259.27±3.271.00±0.20−0.29±0.08
HBench-slideKernel massAction-gradient normEntropy/action dim
Reference100.00±0.000.54±0.06−0.49±0.05
α=(1,0,0,0)\alpha=(1,0,0,0)74.06±1.710.76±0.06−0.66±0.07
α=(0,1,0,0)\alpha=(0,1,0,0)98.51±5.080.79±0.10−0.41±0.06
α=(0,0,1,0)\alpha=(0,0,1,0)107.97±2.780.89±0.05−0.73±0.13
α=(0,0,0,1)\alpha=(0,0,0,1)119.46±7.420.94±0.08−0.81±0.06

Larger kernel mass did not consistently correspond to a larger action-gradient norm or lower entropy. To account for these differences, TPL's adaptive basis uses the distribution of action gradients rather than equalizing kernel mass.

Kernel mass normalization experiments. To isolate the effect of kernel mass, we examined whether performance was maintained after normalization. We also evaluated two variants using the following coefficients to compare different treatments of entropy. (4 seeds)

\qquad c_\alpha=\frac{M_{\mathrm{ref}}}{M_\alpha}.$$ - **A: Full-loss scaling.** Multiply the entire actor loss by $c_\alpha$, while keeping the critic's entropy coefficient unchanged. - **B: Q scaling with entropy correction.** Multiply only the value term in the actor loss by $c_\alpha$, and adjust the entropy coefficient in the critic target to $\eta/c_\alpha$. The actor losses in the table below correspond to the integrand in Eq. (28) of Appendix G, and the critic entropy coefficients correspond to the entropy term in Eq. (27). | Method | Actor loss | Critic entropy coefficient | Performance | | --- | --- | --- | ---: | | Original TPL | $\eta\log\pi_\theta^\alpha-\widehat Q_{w_\alpha}^{\pi_\theta^\alpha}$ | $\eta$ | 0.55 [0.44, 0.66] | | A: Full-loss scaling | $c_\alpha\left(\eta\log\pi_\theta^\alpha-\widehat Q_{w_\alpha}^{\pi_\theta^\alpha}\right)$ | $\eta$ | 0.53 [0.46, 0.60] | | B: Q scaling with entropy correction | $\eta\log\pi_\theta^\alpha-c_\alpha\widehat Q_{w_\alpha}^{\pi_\theta^\alpha}$ | $\eta/c_\alpha$ | 0.57 [0.48, 0.64] | Both variants, which normalize kernel mass while treating entropy differently, maintained performance comparable to Original TPL. This supports the conclusion that TPL's benefits do not depend solely on differences in kernel mass across perspectives. We have added the definitions and results of the kernel mass, action-gradient norm, and entropy diagnostics across perspectives to **Appendix J.3** of the revised manuscript.
AuthorsOpenReview ↗

Response to Reviewer nBbP (2/2)

nBbP-2. Accuracy of the Value Estimates Used by Auxiliary Actors

In the original Section 5.4, we analyzed the stability of value learning by evaluating each conditioned critic using the common reference kernel. To assess the accuracy of the value estimates actually used by auxiliary actors, we added perspective-weighted Q diagnostics. We also conducted a comparison in which data were collected directly using auxiliary policies, finding that estimation error decreased while reference performance declined.

Perspective-weighted Q diagnostics. We constructed Monte Carlo targets matching the temporal weights and entropy terms of each output and measured unnormalized RMSE.

  • Reference Q: Q^wrefπθα(s,a)=∑ℓ=1LQ^ψ(ℓ)(s,a,α)\widehat Q_{w_{\mathrm{ref}}}^{\pi_\theta^\alpha}(s,a)=\sum_{\ell=1}^{L}\widehat Q_\psi^{(\ell)}(s,a,\alpha).
  • Perspective-weighted Q: Q^wαπθα(s,a)=L∑ℓ=1LαℓQ^ψ(ℓ)(s,a,α)\widehat Q_{w_\alpha}^{\pi_\theta^\alpha}(s,a)=L\sum_{\ell=1}^{L}\alpha_\ell\widehat Q_\psi^{(\ell)}(s,a,\alpha).

The Monte Carlo target corresponding to the perspective-weighted Q is

\sum_{t=1}^{H-1}w_\alpha(t) \left[r_t-\eta\log\pi_\theta^\alpha(a_t\mid s_t)\right].$$ We conducted the evaluation on HBench-Periodic (3 tasks, 4 seeds). To reduce truncation effects from finite rollouts, we increased the MC rollout length from $H=200$ to $H=500$. This reduces the fraction of the reference kernel's total mass excluded beyond the rollout from 13.40% to 0.66%. The theoretical upper bound on this fraction across all perspectives and discount grids also decreases from 40.46% to 3.98%. The table below reports the RMSEs measured at $H=500$. The distance bins are based on the normalized distance from the reference condition defined in Section 5.4. | Perspective distance bin | Reference RMSE | Weighted RMSE | | --- | ---: | ---: | | Reference | 9.24 [8.67, 9.82] | 9.24 [8.67, 9.82] | | [0.0, 0.2] | 9.10 [8.55, 9.65] | 9.05 [8.51, 9.61] | | [0.2, 0.4] | 9.34 [8.79, 9.87] | 10.68 [10.09, 11.28] | | [0.4, 0.6] | 9.48 [8.70, 10.38] | 11.06 [10.17, 12.13] | | [0.6, 0.8] | 9.79 [8.93, 10.81] | 12.27 [11.09, 13.69] | | [0.8, 1.0] | 10.67 [9.56, 12.16] | 16.52 [14.27, 19.44] | The RMSE of the perspective-weighted Q increased with distance from the reference. In the most distant bin, its RMSE was 16.52, higher than the reference Q RMSE of 10.67. **Auxiliary data collection experiments.** To examine whether the estimation error arises from reference-only replay, we varied the proportion of data collected using auxiliary policies. We also evaluated whether auxiliary data collection improves reference performance. For each episode, we selected a random auxiliary perspective with a specified probability and kept that perspective fixed throughout the episode while collecting data (HBench-Periodic, 3 tasks, 4 seeds). | Auxiliary collection rate | Reference performance | Weighted RMSE (distance 0.8–1.0) | | --- | ---: | ---: | | 0% (reference-only) | 0.55 [0.44, 0.66] | 16.52 [14.27, 19.44] | | 50% | 0.45 [0.37, 0.52] | 14.07 [12.67, 15.41] | | 100% | 0.46 [0.40, 0.53] | 12.54 [11.84, 13.26] | Auxiliary data collection improved weighted Q accuracy, but this improvement did not translate into better final reference performance. We have expanded and reorganized the original value diagnostics appendix (Appendix J) into **Appendix K** of the revised manuscript, incorporating the evaluation definitions and methodology, rollout truncation analysis, and additional experiments. --- We thank you again for your valuable feedback. We would be happy to provide further clarification or address any additional questions you may have.

Thank you for the additional experiments and clarifications. The normalization results substantially address my concern about kernel-scale effects. The perspective-weighted value diagnostics and longer rollouts also address my concern about the scope of the original value evaluation. The auxiliary data-collection results provide useful context for interpreting the relationship between value accuracy and reference-policy performance.

Both concerns are now largely resolved, and I have no further experimental requests. I will raise my score to 4,

We are pleased to hear that your concerns have been addressed. We sincerely appreciate your valuable feedback, which has helped us improve our manuscript.

Reviewer_YTjrOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes Temporal Perspective Learning (TPL), which applies multiple temporal weightings to the same reward function, allowing a shared actor and critic to learn policies and value functions with different temporal preferences.

Critical strengths

The paper studies whether learning under multiple temporal weightings of the same reward can improve performance on a standard discounted objective, an interesting question for off-policy reinforcement learning. And propose TPL that combines a perspective-conditioned actor with a residual critic, providing a structured way to represent policies and value functions associated with different temporal preferences. In experiments, the paper provides broad empirical evaluation, which covers 24 continuous-control tasks across three benchmark suites and include comparisons with several model-free and model-based methods.

Critical weaknesses

The paper defines the training objective as the average of the objectives for the different temporal perspectives (Equation 4), while the ultimate goal is the reference discounted objective. Because the parameters are shared, gradients from the auxiliary perspectives affect the reference policy. However, this does not by itself guarantee improved performance on the reference objective; the auxiliary objectives may conflict with it and even cause negative transfer. The non-reference policies are trained solely on replay data collected by the reference policy. In Equation 14, the target samples an action from the current conditioned policy at the next state, while the states in the data come from the reference policy's state distribution. Its effectiveness may depend on policy divergence, data coverage, and function approximation error. The adaptive residual basis in Appendix A.3 updates the discount grid every G=500 actor–critic updates, whereas Theorem I.1 assumes a fixed grid. Consequently, the theorem's fixed-policy evaluation and convergence guarantees do not directly apply to the algorithm as implemented: the residual Bellman operator changes during training, and the paper provides no analysis of how the critic tracks this moving target or remains stable after a grid update.

Questions for authors

Refer to the Weaknesses.

3 comments
AuthorsOpenReview ↗

Response to Reviewer YTjr (1/2)

We sincerely thank you for your thoughtful and constructive feedback. Below, we address your key concerns through additional experiments and analyses that further clarify the strengths and limitations of TPL.


YTjr-1. Contribution of Auxiliary Perspectives to the Reference Objective

To quantitatively assess how learning from auxiliary perspectives affects the reference objective, we analyzed gradient alignment and changes in the reference actor loss at the 200k and 1M checkpoints. The full P(α)P(\alpha) mixture exhibited positive transfer at both checkpoints.

Evaluation Protocol. We divided auxiliary perspectives into five bins according to their normalized distance from the reference. We computed the reference and auxiliary gradients on the same replay batch and measured the following metrics:

  • Alignment: Cosine similarity between the two gradients.
  • Conflict rate: The proportion of cases in which the inner product of the two gradients is negative.
  • Norm ratio: The ratio of the auxiliary gradient norm to the reference gradient norm.

One-step transfer gain[1] is the reduction in the reference actor loss measured on a separate evaluation batch after a single hypothetical update. A positive value indicates a decrease in loss, and a larger value indicates a greater decrease. With the critic, discount grid, and entropy temperature held fixed, we matched the update magnitude along each gradient direction to ∥Δθ∥2=0.005\lVert\Delta\theta\rVert_2=0.005. For the full P(α)P(\alpha) mixture, we averaged the gradients according to the original sampling distribution and evaluated the resulting update using the same procedure.

The experiments were conducted on HBench-Periodic (3 tasks, 4 seeds).

200k checkpoint

CategoryAlignmentConflict rateNorm ratioOne-step transfer gain
Reference1.000.00%1.001.63 [1.37, 1.84]
Full P(α)P(\alpha) mixture0.780.00%0.771.01 [0.71, 1.30]
Distance 0–0.20.830.00%1.021.30 [1.02, 1.54]
Distance 0.2–0.40.600.00%1.160.78 [0.57, 0.98]
Distance 0.4–0.60.400.00%1.260.43 [0.25, 0.63]
Distance 0.6–0.80.250.00%1.480.22 [0.04, 0.45]
Distance 0.8–1.00.202.47%1.680.11 [−0.05, 0.29]

1M checkpoint

CategoryAlignmentConflict rateNorm ratioOne-step transfer gain
Reference1.000.00%1.001.12 [0.65, 1.52]
Full P(α)P(\alpha) mixture0.900.00%0.870.71 [0.37, 0.99]
Distance 0–0.20.910.00%1.031.02 [0.61, 1.35]
Distance 0.2–0.40.760.00%1.110.61 [0.28, 0.86]
Distance 0.4–0.60.590.13%1.240.37 [0.10, 0.59]
Distance 0.6–0.80.410.26%1.420.13 [−0.09, 0.35]
Distance 0.8–1.00.331.43%1.72−0.26 [−0.48, −0.07]

Gradient alignment and transfer decreased as individual perspectives became more distant from the reference, and negative transfer was observed in the most distant bin at the 1M checkpoint. In contrast, the full P(α)P(\alpha) mixture used in training aligned with the reference gradient and improved the reference actor loss at both the 200k and 1M checkpoints. This shows that, even when not every perspective individually provides positive transfer, the overall update combining their gradients reduced the reference actor loss.

We have added the detailed methodology and results for each checkpoint and distance bin to Appendix L of the revised manuscript.


YTjr-2. Effect of the Reference-Only Replay Distribution on Auxiliary Policy Learning

In Section 5.4, behavioral differences among the conditioned policies increased with distance from the reference, while their Q-value RMSEs evaluated using the common reference kernel remained similar. This suggests that behavioral diversity among policies can coexist with stable value estimation.

In addition, in Response to Reviewer nBbP-2, we measured the estimation error of the perspective-weighted Q-values actually used by the auxiliary actors. To examine the effect of coverage under reference-only replay, we also compared settings in which data were collected directly using auxiliary policies. Auxiliary data collection reduced the weighted Q error for perspectives far from the reference (RMSE: 16.52→12.54), but decreased average reference performance (0.55→0.46). These results support retaining reference-only data collection in the settings evaluated.

We have incorporated the perspective-weighted Q diagnostics and auxiliary data collection experiments into Appendix K of the revised manuscript.

AuthorsOpenReview ↗

Response to Reviewer YTjr (2/2)

YTjr-3. Fixed-Policy Evaluation and Tracking Error with an Adaptive Residual Basis

Extending Theorem I.1 to a changing grid shows that, under a fixed policy, the additional tracking error term introduced by grid changes is proportional to the change in the effective horizons.

Tracking error analysis. Consider a fixed policy π\pi, bounded rewards ∣r∣≤Rmax⁡\lvert r\rvert\le R_{\max}, and a grid Γt=(γ1,t,…,γL,t)\Gamma_t=(\gamma_{1,t},\ldots,\gamma_{L,t}) satisfying 0≤γ1,t<⋯<γL,t=γ<10\le\gamma_{1,t} < \cdots < \gamma_{L,t}=\gamma < 1. We define the effective horizons and their change as

\qquad \Delta H_t=\max_\ell\lvert H_{\ell,t+1}-H_{\ell,t}\rvert.$$ Summing the first $\ell$ components of the residual fixed point yields $Q^\pi_{\gamma_{\ell,t}}$. Therefore, under the cumulative-sum norm $\lVert q\rVert_C=\max_\ell\lVert\sum_{i\le\ell}q_i\rVert_\infty$, the change in the residual fixed point $q_t^\star$ is bounded by $R_{\max}\Delta H_t$. We define the tracking error and the approximation error in the Bellman update, respectively, as $$e_t=\lVert\hat q_t-q_t^\star\rVert_C, \qquad \varepsilon_t=\lVert\hat q_{t+1}-T^\pi_{\Gamma_t}\hat q_t\rVert_C.$$ This yields the following tracking error bound: $$e_{t+1}\le\gamma e_t+\varepsilon_t+R_{\max}\Delta H_t.$$ **Observed grid changes.** In our implementation, the horizons are parameterized to preserve their ordering and maximum value, and the grid is updated every 500 critic updates. To assess the magnitude of the additional error term due to grid changes, we measured $\Delta H_t$ at each grid update on HBench-Periodic (3 tasks, 4 seeds). | Metric | Median | 95th percentile | Maximum | | --- | ---: | ---: | ---: | | $\Delta H_t$ per grid update | $4.97\times10^{-3}$ | $1.35\times10^{-2}$ | $1.89\times10^{-2}$ | Under a fixed-policy analysis, the fixed-point shift induced by grid changes can be bounded in terms of the change in horizon. The small observed horizon changes support the conclusion that the additional non-stationarity introduced by grid adaptation is limited. However, this analysis does not guarantee the stability of the overall learning process, which also involves changes in the policy and temperature, as well as function approximation errors. We have added the tracking error analysis and proof and the observed horizon changes to **Appendix J** of the revised manuscript. --- We thank you again for your valuable feedback. We would be happy to provide further clarification or address any additional questions you may have. --- ## References [1] Christopher Fifty, Ehsan Amid, Zhe Zhao, Tianhe Yu, Rohan Anil, and Chelsea Finn. “Efficiently Identifying Task Groupings for Multi-Task Learning.” NeurIPS, 2021.

Thank you again for your constructive feedback. We would appreciate your thoughts on whether the additional experiments and clarifications in our rebuttal address your main concerns. Please let us know if any concerns remain or if further clarification would be helpful.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.