- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper addresses the problem of combining scarce high-fidelity (HF) data with cheaper low-fidelity (LF) data to reduce the variance of policy gradient estimates. The author(s) propose MFPG-PPO, a stable multi-fidelity policy gradient (MFPG) algorithm tailored to PPO. Compared to existing works, which apply the control variate (CV) directly to the REINFORCE or PPO objectives, the authors add three techniques to reduce variance and control reliance on LF data: (1) synchronizing the LF rollouts with the HF rollouts every T steps to increase the correlation between LF and HF data; (2) estimating the CV coefficient from episode-level sums instead of per-time-step values, which captures cross-step covariance; and (3) a confidence-based gate that drops the LF correction, falling back to HF-only PPO, when the estimated correlation is not significant or the variance is not reduced.
The paper also proposes a budget-aware allocation of LF and HF data. Results show that these techniques improve vanilla MFPG-PPO and that it compares favorably with existing multifidelity RL schemes, including the off-dynamics methods PAR (Lyu et al., 2024) and DARC (Eysenbach et al., 2021).
Critical strengths
-
The three proposed techniques are principled: (1) periodically synchronizing LF rollouts with HF rollouts, (2) estimating the CV coefficient from episode-level statistics, and (3) confidence-based gating of the LF correction. Ablation studies validate them empirically.
-
The experiments show clear improvements over vanilla MFPG-PPO and more consistent gains than the other multi-fidelity baselines. The method is also demonstrated on a real robot.
-
The method is robust to a poor simulator. The LF data is used only as a control variate, and the confidence gate falls back to HF-only PPO when the LF data does not help. As a result, a poorly matched simulator mainly reduces the benefit rather than biasing learning.
Critical weaknesses
-
Although the proposed techniques stabilize the CV estimates for MFPG-PPO, the core components of MFPG are already established in prior work. These include the multi-fidelity CV estimator for policy gradients, coupled LF rollouts, and the application of this estimator to the PPO objective. This limits the novelty of the contribution, which lies mainly in the stabilization techniques and in scaling the evaluation to GPU-parallel simulation and physical robots.
-
The synchronization technique reduces variance at the cost of introducing bias. The appendix discusses this tradeoff, but the paper does not quantify it. In the sim-to-sim experiments, a large-batch HF gradient is available as a reference, so an ablation over the synchronization interval T reporting bias and variance (and/or MSE) against this reference would clarify whether the variance reduction outweighs the added bias.
-
The benefit of the budget-aware variant (MFPG-PPO-BA) is not clearly demonstrated. It is evaluated only in the cross-task comparison, where its performance is close to the fixed-ratio variant, and there is no direct comparison or ablation isolating the effect of the allocation.
Questions for authors
-
The paper cites DARAIL (Guo et al., 2024) but does not compare against it. This comparison would be informative because DARAIL was designed to fix DARC’s performance degradation when LF-to-HF transfer is hard.
-
The episode-sum coefficient fitting is presented as a contribution but is used only in the real-robot experiment, while all simulation experiments use per-step fitting. Its benefit is supported only by a higher estimated correlation per iteration, without a direct performance comparison with per-step fitting. An ablation comparing the two fitting choices in terms of performance would strengthen the conclusions.
5 comments
Response to Reviewer 7t5b (2/2)
Weakness 3. The budget-aware variant (MFPG-PPO-BA).
We have added a side-by-side comparison between the budget-aware variant (MFPG-PPO-BA) and the fixed-ratio variant (MFPG-PPO-FR) (Appendix A.10, Figure 19) and an experiment that trains MFPG-PPO-BA at different HF-to-LF cost ratios to study the effect of cost allocation (Appendix A.10, Figure 20).
In Figure 19, we show the difference in mean HF evaluation return between MFPG-PPO-BA and MFPG-PPO-FR in 19 task and HF data budget settings. MFPG-PPO-BA outperforms MFPG-PPO-FR in 16 of the 19 settings, and the two are not distinguishable in the other 3. The gain is generally larger at smaller nominal HF data budgets.
In Figure 20, we train MFPG-PPO-BA at HF-to-LF cost ratios = 3, 10, and 30. In the tested setting, the return of MFPG-PPO-BA remains relatively stable across , with a slight decrease at , i.e., when HF data are more expensive than at = 3 and 10. As HF data become increasingly expensive (larger ), MFPG-PPO-BA allocates fewer HF and more uncoupled LF environments. The allocation in Figure 20 also suggests a possible reason for the higher return of MFPG-PPO-BA than MFPG-PPO-FR in Figure 19: within its cost budget, MFPG-PPO-BA collects more HF data at every than MFPG-PPO-FR.
Question 1. Comparison to DARAIL.
We added a comparison to DARAIL and DARC, both with the SAC backbone used in their original publication (Appendix A.11, Figures 21–22).
Among the four methods (the two added baselines DARAIL and SAC-DARC, SAC HF-Only, and our MFPG-PPO-FR), MFPG-PPO-FR yields among the best mean returns in all 8 tested settings and the most consistent performance across tasks and HF budgets, cf. Figure 21. In Figure 22, we analyze the failure modes of SAC-DARC and DARAIL.
Question 2. Comparison to per-step coefficient fitting.
We agree with the reviewer's point, and we are happy to add an ablation with per-step fitting.
We would like to clarify that we present the coefficient fitting unit as a tradeoff. Per-step fitting multiplies the sample size but misses cross-step covariance, cf. Lemma 1. We choose the unit by the measured correlation. In simulation, per-step correlations are already high, so we fit per step. On the robot, grasps occur at different steps in the two fidelities, and on the same samples the per-step correlation is only about 0.17 (240 pairs), against about 0.6 for episode sums (4 pairs, but noisier).
We hope these changes and answers address the reviewer's concerns. We are happy to answer any further questions.
Follow-up on Question 2: per-step vs. episode-sum coefficient fitting
"An ablation comparing the two fitting choices in terms of performance would strengthen the conclusions."
As promised in our earlier response, we ran this ablation on the real robot (Appendix A.12, Figure 23). We trained MFPG-PPO-FR on the Franka cube-picking task with the control variate coefficient fitted per step and otherwise the same training configuration as in Figure 4. With per-step fitting, MFPG-PPO-FR reaches a final real-robot success rate of 70.0%, below the 91.7% of MFPG-PPO-FR with episode-sum fitting and above the 55.0% of PPO HF-Only. So fitting the coefficient on episode sums also gives better real-robot performance on this task, in addition to the higher correlation shown in Figure 4(c).
We hope this addresses the reviewer's concern. We are happy to answer any further questions.
Response to Reviewer 7t5b (1/2)
We thank the reviewer for the detailed review and insightful suggestions, which have helped us improve the manuscript. All changes in the revised manuscript are marked in blue. We first summarize the changes that address the reviewer's points and then respond to each point.
Summary of manuscript changes:
Weakness 1. Novelty.
We agree that the machinery of the multi-fidelity control variate estimator and coupled low-fidelity (LF) rollouts, and its use in the PPO objective, come from prior work, which we cite. The stabilization and the larger-scale evaluation are indeed part of our contribution. Beyond them, our contributions are the following.
Identifying the failure modes of naive MFPG constructions. A naive construction of MFPG for PPO fails. Through analysis and experiments, we identify its failure modes: loss of cross-fidelity correlation and variance inflation.
Analysis. Although the multi-fidelity control variate estimator takes a simple form, designing an RL algorithm around it involves many design choices. We analyze the effect of the unit at which the control variate coefficient is fitted (Lemma 1), the connection between scalar surrogate and policy gradient variance reduction (the new Lemma 2), and the effect of periodically re-synchronizing the coupled, but not the uncoupled, LF rollouts (Lemma 3). Prior MFPG work [1] does not study these aspects.
Although we focus on PPO, the identified failure modes and analysis are not limited to it. In particular, Lemmas 1–3 hold, under their stated assumptions, for any per-step policy gradient surrogate used as the paired statistic.
Algorithm. MFPG-PPO is the result of the identification and analysis steps and addresses the identified failures. We also propose a budget-aware MFPG-PPO variant.
[1] Liu, Xinjie, Cyrus Neary, Kushagra Gupta, Wesley A. Suttle, Christian Ellis, Ufuk Topcu, and David Fridovich-Keil. "A multi-fidelity control variate approach for policy gradient estimation." 2026.
Weakness 2. Measuring gradient error and bias.
We thank the reviewer for the constructive feedback. We have added a gradient study that measures the policy gradient mean squared error and bias of MFPG-PPO at synchronization intervals T = 12, 24, and 48 (Section 5.6, Figure 6), using the policy gradient computed from a large batch of high-fidelity (HF) data as the reference.
MFPG-PPO yields a lower policy gradient error than the policy gradient estimated from all the HF data alone in nearly all 18 settings. In 3 of the 18 settings, the 95% confidence interval of the gradient bias excludes 0. Even in these settings, the benefit of MFPG-PPO outweighs the bias.