acceptodds
Under review as a conference paper at ICLR 2027

Provably Variance-Reduced Federated Categorical Distributional TD Learning with Linear Function Approximation

Abstract

Federated distributional policy evaluation enables multiple clients to estimate the entire return distribution from locally generated interaction samples without pooling raw data, thereby combining uncertainty-aware evaluation with distributed computation. However, finite-sample analyses of linear categorical distributional TD have largely been restricted to centralized settings, while theoretical studies of federated TD primarily address expected returns. This leaves a gap in the understanding of variance reduction for distributed return-distribution evaluation with multiple local updates. We study federated distributional policy evaluation under linear categorical function approximation and propose FedVR-DistTD, which combines a global control direction aggregated from distributed control batches with same-sample TD differences. Under homogeneous clients, full participation, and independent generative-model sampling, We show that the final server iterate , obtained after communication rounds, converges geometrically in mean square to a statistical neighborhood determined by the finite control batch With a fixed per-round control batch, the tail-averaged server iterate converges to the projected fixed point in mean square at rate , where is the per-client local-iteration budget. This result yields a corresponding expected squared Cram\'er-error guarantee, and is supported by experiments on a four-state Markov reward process.

Primary area:
reinforcement learning
Keywords:
Federated reinforcement learning, Distributional policy evaluation, Categorical distributional TD

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_hghJOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies the problem of federated reinforcement learning. The authors propose FedVR-DistTD algorithm based on variance-reduced federated categorical distributional TD learning with linear function approximation. Meanwhile, the authors establish its finite-time convergence guarantees, indicating that the proposed method can converge geometrically and achieves the root-error convergence convergence rate of O(1/T)\mathcal{O}(1/\sqrt{T}).

Critical strengths

The variance-reduced mechanism is considered in FedVR-DistTD, which is efficient to reduce the variance without sharing the private training trajectories, and this is crucial in decentralized setting. Meanwhile, The authors provide a solid finite-time guarantees and the corresponding analysis, and the ablation studies are conducted to evaluate the performance of each component of FedVR-DistTD.

Critical weaknesses
  1. The experiments are conducted under full participation and i.i.d. data sampling strategy, which is ideal and can not reflect the practical decentralized setting.

  2. In the contribution statement (line 110), the authors state that the finite-time guarantees are established under Markovian sampling. However, Assumption 1 (line 244) requires that the drawn samples for each client are i.i.d. from the distribution μπ\mu_\pi. The statement and the assumption are contradictory. Meanwhile, authors state that the Markovian sampling still remains open, which is also contradictory with the contribution stated in line 110.

  3. The manuscript lacks numerical comparisons of communication costs of the proposed method and corresponding baselines, which is important to evaluate the method's efficiency under federated setting.

Questions for authors
  1. Could authors clarify the theoretical difference between the claimed theoretical results under Markovian sampling and the i.i.d. sampling strategy in Assumption 1? Do the results still hold under these two conditions?

  2. Could authors provide more detailed numerical comparisons of communication costs of different methods?

1 comment
AuthorsOpenReview ↗

Thank you for your detailed summary and constructive suggestions. We have some issues with the theory being not rigorous and the experiments being insufficient. We have already made supplementary revisions.

We thank the reviewer for identifying this inconsistency. The original contribution statement incorrectly described our guarantees as applying to Markovian sampling. We have corrected the introduction and contribution statement and clarified the assumptions in Section3.3: our finite-time results hold for homogeneous clients with full participation under independent generative-model sampling, subject to the stated stability conditions. The additional analysis in Appendices D.4-D.6 also operates under Assumption1; it does not establish a Markovian guarantee.

The distinction is that our sampler draws a fresh state from the stationary distribution for each transition and then draws its successor from the Markov kernel. Consecutive samples along a Markov trajectory may have the same stationary marginal distribution, but they are temporally dependent. In that case, the conditional centering and independence arguments used for the control batches and local updates, including the 1/(NB)1/(NB) variance calculation, do not follow from Assumption1. Thus the stated guarantees hold under the independent-sampling protocol, but their validity and rates under continuous Markov trajectories have not been established. We have made this limitation explicit and leave a mixing-dependent extension for future work. AppendixG reports Markovian results as empirical evidence only.

We have added numerical communication accounting in Appendix G.1. With 16 features and 51 categorical intervals, the per-client payloads are 13,056, 19,584, and 26,112 bytes per round for FedDistTD-NoVR, SCAFFLSA-Linear-CTD, and FedVR-DistTD, respectively. These counts include parameter transfers in both directions. Under the same message protocol and representation, the per-round byte counts do not depend on whether transitions are sampled independently or along Markovian trajectories.

The table reports completed-run costs at nearly equal per-client transition budgets. FedVR-DistTD's transition count includes samples used to construct its control direction.

MethodBytes/roundRoundsTransitions/clientTotal MB/client
NoVR13,0563,375135,00044.06
SCAFFLSA19,5843,375135,00066.10
FedVR26,112562134,88014.67

FedVR-DistTD sends more bytes per round but completes the nearly sample-matched run with fewer rounds and less total communication. Appendix G.4 additionally compares error trajectories under matched transition and communication budgets , showing lower tail-averaged error for FedVR-DistTD under the tested configurations.

Reviewer_JJApOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies federated distributional policy evaluation with linear function approximation. The authors propose FedVR-DistTD, which combines a globally aggregated control direction with same-sample TD differences to reduce variance during local updates. They establish geometric convergence of the last server iterate to a statistical neighborhood and an O(E/(TNB))O(\sqrt{E/(TNB)}) root-mean-square convergence rate for the tail-averaged estimator. Experiments on a four-state Markov reward process illustrate the theoretical findings.

Critical strengths
  • The problem of federated distributional policy evaluation with function approximation is interesting and theoretically relevant.

  • The proposed algorithm is relatively simple, and the combination of a global control direction with same-sample TD differences provides an interpretable variance-reduction mechanism.

  • The analysis distinguishes geometric last-iterate convergence from the statistical improvements obtained through tail averaging.

  • The connection between parameter convergence and the stationary-weighted Cramér distance provides a meaningful distributional interpretation of the guarantees.

Critical weaknesses
  • The main concern is the originality of the proposed approach. Section 4.1 shows that the projected distributional Bellman equation reduces to an affine stochastic approximation problem. Consequently, the proposed algorithm closely resembles existing variance-reduced federated linear stochastic approximation methods. In particular, [1] already studies control variates for federated linear stochastic approximation and establishes finite-sample guarantees under heterogeneous environments, while [2] develops variance-reduced distributional TD methods with linear function approximation. Although both works are cited, the authors do not adequately explain which technical difficulties are genuinely new beyond combining these existing frameworks. A more precise comparison is needed to establish the novelty of the theoretical contributions.

  • All clients are assumed to share the same transition kernel, reward distribution, policy, and feature representation. This excludes statistical heterogeneity, one of the main challenges in federated reinforcement learning. Client disagreement consequently arises from stochastic fluctuations rather than differences between local fixed points. This assumption is particularly restrictive given that [1] already provides guarantees for heterogeneous federated linear stochastic approximation. Extending the analysis to heterogeneous reward functions or transition kernels would substantially strengthen the contribution.

  • Assumption 1 requires independent samples from the stationary distribution, rather than observations generated along Markovian trajectories. This eliminates temporal dependence and mixing-time considerations, considerably simplifying the analysis. The emphasis on Markovian sampling and practical federated RL is therefore somewhat misleading. The authors should either extend the guarantees to trajectory-based sampling or explicitly acknowledge this limitation.

  • All experiments are conducted on a four-state Markov reward process with homogeneous clients and independent sampling. This setting is too simple to convincingly demonstrate the advantages of the proposed algorithm. Experiments on larger environments with nontrivial feature representations are needed. The authors should also investigate the dependence on N, B, E, the discount factor, and the categorical support size, and include experiments with Markovian trajectories. Comparisons against variance-reduced distributional TD methods and federated linear stochastic approximation baselines under matched sample and communication budgets would be particularly valuable.

  • The proposed linear representation is defined over signed measures, meaning that the learned categorical coefficients need not be nonnegative. Consequently, the estimated return distributions may not be valid probability distributions. This limitation is important given the motivation concerning uncertainty quantification and risk-sensitive decision-making. The authors should discuss whether nonnegativity can be guaranteed or enforced without affecting convergence. Furthermore, the guarantees concern the projected fixed point, and the effects of approximation and discretization errors are not investigated experimentally.

  • The main results hide potentially important dependencies on the discount factor, feature covariance, categorical support size, and number of local updates in constants defined in the appendix. Explicit step-size conditions and sample-complexity bounds would make the theoretical contribution easier to assess. Additionally, Table 1 compares methods under different sampling assumptions, representations, and error criteria, making the comparison difficult to interpret.

[1] Mangold et al., SCAFFLSA: Taming Heterogeneity in Federated Linear Stochastic Approximation and TD Learning. NeurIPS,2024

[2] Jin et al., Accelerated Distributional Temporal Difference Learning with Linear Function Approximation

Questions for authors
  • Which technical arguments are fundamentally new compared with the federated linear stochastic approximation analysis of [1] and the variance-reduced distributional TD framework of [2]?

  • Can the method handle heterogeneous clients with different reward distributions or transition kernels?

  • Can the guarantees be extended to Markovian trajectory sampling?

  • Can the authors provide experiments on larger environments and comparisons under matched computational budgets?

3 comments
AuthorsOpenReview ↗

Thank you for your detailed assessment and constructive suggestions. We acknowledge that the original manuscript did not sufficiently clarify the theoretical contribution and its scope, and that the empirical evaluation was limited. We have revised the analysis and expanded the experiments accordingly.

We thank the reviewer for asking us to clarify the technical contribution. We agree that writing the projected distributional Bellman equation as an affine stochastic-approximation problem is not itself new. Our analysis concerns three coupled features of FedVR-DistTD: a single empirical control direction shared across clients within each round, reuse of its estimation error through EE local updates, and the interaction between that shared error and client-specific propagation noise after endpoint aggregation. Because clients share the control estimate, their endpoints are not independent even when their local samples are independent.

Appendix D.4 conditions on the shared control direction and derives an exact round-level variance decomposition. It identifies shared-control, local-propagation, and interaction terms with factors $ 1/(NB),, 1/N$, and 1/(N2B)1/(N^2B), respectively, retaining their dependence on the step size and local-update count. Appendices D.5 and D.6 develop scalar and matrix stability criteria and finite-time bounds for the server iterates and tail average. This analysis of shared control-batch reuse under repeated federated local updates is the technical distinction we claim relative to [1] and [2].

Our guarantees assume homogeneous clients, full participation, and independent sampling. The algorithm can be implemented with different client reward distributions or transition kernels, but the present guarantees do not cover such heterogeneity. Nor do they directly extend to Markovian trajectories: that extension would require controlling temporal dependence within control batches and local updates, as well as dependence across rounds. We regard heterogeneous clients and Markovian sampling as priorities for future theoretical work. The continuous-trajectory experiments in Appendix G provide empirical evidence in the latter setting, without invoking the independent-sampling guarantees.

To broaden the evaluation, Appendix G uses a 20×2020\times20 grid with 30 traps, 16 fixed radial-basis features, and continuous Markov trajectories. It varies NN, EE, γ\gamma, the support size K+1K+1, and the control-batch size BB. Appendix G.4 further compares the methods under matched transition-sample and communication budgets. The following table summarizes the resource comparisons of the algorithms.

Table 1. Common communication cap: 6.528 MB per client

MethodBytes/roundRoundsSamples/clientNetwork TD evalsTail MSE
FedDistTD-NoVR13,05650020,000200,0001.22e-2
SCAFFLSA-Linear-CTD19,58433313,320133,2001.50e-2
FedVR-DistTD26,11225060,000700,0006.74e-4

Table 2. Completed runs at nearly equal per-client transition budgets

MethodRoundsSamples/clientMB/clientNetwork TD evalsFinal tail MSE
FedDistTD-NoVR3,375135,00044.061,350,0001.16e-2
SCAFFLSA-Linear-CTD3,375135,00066.101,350,0009.95e-4
FedVR-DistTD562134,88014.671,573,6002.82e-4

The transition counts include samples used to construct FedVR-DistTD's control direction. At the common communication cap, the methods use different numbers of transitions and TD evaluations. At nearly equal transition budgets, FedVR-DistTD achieves the lowest final tail MSE with less communication, while requiring more TD evaluations. These results provide evidence of improved transition and communication efficiency in the tested Markovian setting. Sensitivity to EE, NN, and γ\gamma is reported in Appendix G.2; support size K+1K+1 and control-batch size BB are examined in Appendix G.3. Appendix G.4 further confirms the lower tail-averaged error of FedVR-DistTD under matched transition and communication budgets.

Thank you for your detailed response and for the additional experiments, which significantly strengthen the empirical evaluation of the manuscript.

However, I am still struggling to identify the main theoretical novelty of the paper.

As you explain in Section 4.1, the projected distributional Bellman equation can be reduced to a linear stochastic approximation problem. Moreover, the convergence rates you establish appear to be essentially the same as those already obtained in the federated LSA literature.

I understand that your analysis accounts for the dependence induced by reusing a shared stochastic control direction across multiple local updates. However, it is not clear to me why this requires fundamentally new techniques compared with existing analyses of variance-reduced federated LSA, particularly SCAFFLSA.

Could you clarify precisely what prevents us from applying existing federated LSA results to your setting? More specifically, which part of your theoretical analysis cannot be recovered through a direct adaptation of existing techniques?

This question is particularly important given that your theoretical analysis assumes homogeneous clients and independent sampling, and therefore does not address two major challenges in federated reinforcement learning: client heterogeneity and Markovian sampling. While the additional experiments with Markovian trajectories are appreciated, they do not extend the theoretical guarantees to this setting.

At present, the main contribution seems to be the application of existing federated LSA methodology to categorical distributional TD, rather than a genuinely new theoretical result. I would appreciate a clearer explanation of what the analysis brings beyond this reduction, especially considering that your current analysis does not extend to heterogeneous environments or Markovian sampling.

AuthorsOpenReview ↗

Thank you very much for your valuable response.

Thank you for this important question. Our aim is to establish a federated distributional policy-evaluation framework and characterize the convergence and limitations of its variance-reduction mechanism.

Algorithmic motivation. Starting from federated distributional TD, we consider same-sample direction differences. For h(θ;ξ)=A(ξ)θ−b(ξ)h(\theta;\xi)=A(\xi)\theta-b(\xi),

h(θi,tr;ξ)−h(θr;ξ)=A(ξ)(θi,tr−θr).h(\theta_{i,t}^{r};\xi)-h(\theta^r;\xi)=A(\xi)(\theta_{i,t}^{r}-\theta^r).

This cancels the additive term b(ξ)b(\xi). However, each client starts every round from the global reference, θi,0r=θr\theta_{i,0}^{r}=\theta^r, so the difference alone would produce zero updates throughout the round. Inspired by control-variate approaches such as SCAFFLSA, we introduce a shared empirical reference direction,

cr=1NB∑i=1N∑b=1Bh(θr;ζi,br),c^r=\frac1{NB}\sum_{i=1}^{N}\sum_{b=1}^{B}h(\theta^r;\zeta_{i,b}^{r}),

giving

gi,tr=A(ξi,tr)(θi,tr−θr)+cr.g_{i,t}^r=A(\xi_{i,t}^r)(\theta_{i,t}^r-\theta^r)+c^r.

Here, crc^r supplies the reference-point direction, while the difference adjusts it as the local parameter changes. Batch averaging reduces reference-direction estimation noise, but requires additional samples and introduces a shared error reused throughout the round.

Why the SCAFFLSA theorem does not apply directly. SCAFFLSA recursively updates client-specific controls across rounds; our method combines same-sample differences with a freshly estimated common direction. Writing

cr=hˉ(θr)+εcr,c^r=\bar h(\theta^r)+\varepsilon_c^r,

we have

E[εcr∣Fr]=0,E[gi,tr∣Gi,tr]=hˉ(θi,tr)+εcr,\mathbb E[\varepsilon_c^r\mid\mathcal F_r]=0,\qquad\mathbb E[g_{i,t}^r\mid\mathcal G_{i,t}^r]=\bar h(\theta_{i,t}^r)+\varepsilon_c^r,

where Fr\mathcal F_r is the history before generating the control batch, and Gi,tr\mathcal G_{i,t}^r includes that control and the local history before the next transition. Thus, the control error is centered before the round but is not fresh martingale-difference noise at each local step. Client endpoints also share this error and are independent conditional on Fr\mathcal F_r and crc^r, rather than on Fr\mathcal F_r alone.

What the analysis adds beyond the LSA reduction. We derive the round recursion

θr+1=θr−αHrcr,Hr=1N∑iHi,r,\theta^{r+1}=\theta^r-\alpha H_rc^r,\qquad H_r=\frac1N\sum_iH_{i,r},

where Hi,rH_{i,r} describes local propagation. Define

HˉE=E[Hi,r∣Fr],VE=E[(Hi,r−HˉE)⊤J(Hi,r−HˉE)∣Fr].\bar H_E=\mathbb E[H_{i,r}\mid\mathcal F_r],\qquad V_E=\mathbb E[(H_{i,r}-\bar H_E)^\top J(H_{i,r}-\bar H_E)\mid\mathcal F_r].

For the centered round innovation Δr+1=θr+1−E[θr+1∣Fr]\Delta^{r+1}=\theta^{r+1}-\mathbb E[\theta^{r+1}\mid\mathcal F_r], Appendix D.4 establishes

E[∣Δr+1∣J2∣Fr]=α2Nhˉ(θr)⊤VEhˉ(θr)⏟local propagation+α2NBtr⁡(HˉE⊤JHˉES(θr))⏟shared-control noise+α2N2Btr⁡(VES(θr))⏟interaction,\begin{aligned}\mathbb E[\\|\Delta^{r+1}\\|_J^2\mid\mathcal F_r]={}&\underbrace{\frac{\alpha^2}{N}\bar h(\theta^r)^\top V_E\bar h(\theta^r)}_{\text{local propagation}}\\\\&+\underbrace{\frac{\alpha^2}{NB}\operatorname{tr}(\bar H_E^\top J\bar H_E S(\theta^r))}_{\text{shared-control noise}}\\\\&+\underbrace{\frac{\alpha^2}{N^2B}\operatorname{tr}(V_ES(\theta^r))}_{\text{interaction}},\end{aligned}

where S(θ)=Cov⁡(h(θ;ξ))S(\theta)=\operatorname{Cov}(h(\theta;\xi)). This exact identity retains the dependence on the step size and local-update length and separates independent local variability from shared noise. Moreover,

S(θr)=Cov⁡([A(ξ)−Aˉ](θr−θ⋆)+h(θ⋆;ξ)),S(\theta^r)=\operatorname{Cov}(\lbrack A(\xi)-\bar A\rbrack(\theta^r-\theta^\star)+h(\theta^\star;\xi)),

so the control variance depends on the current parameter and must enter the mean-square stability analysis.

This analysis gives stability conditions, establishes a nonzero asymptotic last-iterate error lower bound under the stated conditions when solution noise is nonzero and the step size and batch size are fixed, and proves convergence of the server tail average. These results do not follow directly from the SCAFFLSA theorem, although our proofs build on established LSA techniques.

Finally, homogeneous clients and independent sampling provide a controlled starting point, rather than a complete model of practical federated reinforcement learning. Heterogeneous environments and Markovian sampling require specifying the evaluation target and accounting for client-dependent TD operators, stationary distributions, mixing rates, and additional bias and covariance terms. This requires revisiting the round recursion and stability analysis. We first establish the framework under controlled conditions as a foundation for these necessary extensions. Appendix G.4 provides preliminary empirical evidence under Markovian trajectories, but does not extend the theoretical guarantees. Our contribution is therefore an algorithm-specific variance, stability, and finite-time analysis, rather than a fundamentally new general LSA methodology.

Reviewer_bub3OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies federated distributional policy evaluation, motivated by the need to capture return variability and tail risk that expected-value estimates obscure, while allowing clients to collaborate without sharing raw interaction data. Multiple clients jointly estimate the distribution of discounted returns under a fixed policy, using a fixed categorical support and linear function approximation of cumulative masses. The learning objective is the fixed point of the projected distributional Bellman operator.

The authors proposed method, FedVR-DistTD, combines a globally aggregated control direction with same-sample differences between local and shared-model TD updates to reduce sampling noise and client drift. Clients perform multiple local updates before the server averages their parameters. Analysis establishes geometric convergence of the final server iterate to a finite-batch error neighborhood and a statistical convergence rate for the average of later server iterates.

Critical strengths
  1. The authors provide a principled formulation of federated distributional policy evaluation. This worke xtends federated TD learning from expected returns to a categorical representation of return distributions with linear function approximation. This is an intersting idea and nice contribution because it provides a theoretically grounded way to retain information about return variability while allowing clients to keep their interaction data local.

  2. The convergence analysis accounts for the algorithm’s coupled errors. The analysis explicitly handles both the control-batch error shared across local steps and the parameter-dependent noise remaining after same-sample differencing. It distinguishes geometric convergence of the final server iterate to a statistical neighborhood from convergence of the tail-averaged output to the projected fixed point. This distinction explains the role of averaging and supplies theoretical support for the proposed method.

Critical weaknesses

I have three main concerns about the paper:

  1. The scope of the guarantees needs clearer alignment with the motivation. The introduction claims guarantees under Markovian sampling, whereas the analysis assumes fresh independent samples, homogeneous clients, and full participation. The result presented is still useful, but these restrictions narrow its applicability to practical federated settings. In my opinion the authors should clarify the scope of the results.

  2. The empirical evaluation provides limited evidence of broader benefits. The four-state experiment does a good job illustrating the proposed mechanisms, but, it offers limited insight into behavior with larger state spaces or more demanding approximation problems. Also, the reported update budget excludes control-batch samples, making the overall sample-efficiency benefit difficult to assess. A broader evaluation with matched total sampling budgets would make the practical contribution more convincing.

  3. Still on the topic of the experiments: The convergence experiments fall outside the stated theoretical guarantees. Specifically, Figure 1(a) uses 10 clients and a control-batch size of 500, giving 5,000 control samples per round; Figure 1(b), which illustrates the tail-averaged rate, uses a batch size of 200, giving 2,000 control samples per round. With a discount factor of 0.95, the contraction condition in Appendix C.2 (Lemma c.1) necessarily requires more than approximately 12,000 control samples per round. Thus, neither experiment satisfies the sufficient conditions for its corresponding convergence guarantee. Is it possible to sharpen the bound or provide an experiemt that lies within the theoretical bounds?

Questions for authors

Please clarify whether the convergence guarantees extend to correlated Markovian trajectories, as suggested in the introduction, or are restricted to the independent generative-model sampling assumed in Appendix A.3? What additional challenges would arise in extending the analysis?

How does FedVR-DistTD compare with the baselines when the total number of sampled transitions—including those used to construct the control direction—is held fixed? This would help clarify whether the variance reduction translates into improved overall sample efficiency.

1 comment
AuthorsOpenReview ↗

Thank you for your careful summary and constructive suggestions. We have expanded the empirical evaluation and clarified that our convergence guarantees apply to independent generative-model sampling, while the Markovian results are empirical.

Thank you for these questions. We have clarified that our finite-time guarantees assume homogeneous clients, full participation, and independent generative-model sampling with states drawn from the stationary distribution (Assumption 1 and Section 3.3). They do not cover continuous Markov trajectories. Extending the analysis would require controlling temporal dependence in the control batches and local updates, including dependence propagated across communication rounds. We now state this scope in the theoretical discussion and conclusion.

We agree that Figure 1(a) and (b) use 5,000 and 2,000 control samples per round, respectively, and do not satisfy the sufficient condition of Lemma C.1 at γ=0.95\gamma=0.95. We therefore present Figure 1 as an empirical result, rather than an instance covered by that condition. Appendix F.5 provides a separate numerical comparison with the finite-time bound expressions under its stated configurations. We have also added Appendices D.4--D.6: D.4 separates shared-control error, local propagation, and their interaction in the round-level variance, while D.5 and D.6 develop scalar and matrix stability criteria under Assumption 1. These additions refine the analysis but do not extend it to Markovian sampling.

To assess behavior beyond the independent-sampling setting, we added the Markovian grid experiments in Appendix G. In the transition-matched comparison of Appendix G.1, all sampled transitions—including FedVR-DistTD control batches—are counted. At nearly equal per-client budgets, FedVR-DistTD achieves the lowest final tail MSE. This provides empirical evidence that variance reduction can improve sample efficiency at the tested budget, although FedVR-DistTD requires more TD evaluations. We do not infer a general sample-complexity guarantee from this Markovian result.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.