acceptodds
Under review as a conference paper at ICLR 2027

Multi-Fidelity Policy Gradients Stabilize Data-Scarce Reinforcement Learning

Abstract

Policy gradient methods for on-policy reinforcement learning (RL) can become unstable when expensive, scarce target-domain data yield noisy gradient estimates. We address this challenge by complementing limited high-fidelity (HF) target-domain data with abundant, cheap, but biased low-fidelity (LF) data, e.g., from a simplified simulator. Most existing methods directly optimize biased objectives based on LF data. In contrast, the recently introduced multi-fidelity policy gradient (MFPG) framework uses LF data solely to construct a control variate that reduces variance and improves HF data efficiency without biasing the policy gradient estimator. However, published work on MFPG is limited to REINFORCE on small-scale simulation tasks. We develop MFPG for modern actor-critic learning in GPU-parallel simulation and on a physical robot. Our analysis and experiments show that naive extensions to proximal policy optimization (PPO) can lose cross-fidelity correlation or inflate variance. Our MFPG-PPO addresses these failures by redesigning the sampling, advantage estimation, and control variate construction to preserve cross-fidelity correlation, and by monitoring estimator uncertainty to prevent variance inflation. We also introduce a budget-aware MFPG-PPO to divide a fixed sampling budget among high- and low-fidelity data sources. Across simulated robot locomotion tasks of varying LF-to-HF transfer difficulty and HF data budgets, MFPG-PPO improves upon PPO trained on HF data alone in nearly all settings, and consistently matches the performance of PPO trained with 16× more HF data on the hardest task at the smallest HF budgets. In contrast, most baselines that use LF data perform well only where direct LF-to-HF transfer succeeds. MFPG-PPO enables stable learning on a physical Franka arm using only 4 real-robot episodes per update and no human demonstrations.

Primary area:
reinforcement learning
Keywords:
reinforcement learning, sample efficiency, policy gradients, robot learning, sim-to-real, control variates, variance reduction

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_7t5bOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper addresses the problem of combining scarce high-fidelity (HF) data with cheaper low-fidelity (LF) data to reduce the variance of policy gradient estimates. The author(s) propose MFPG-PPO, a stable multi-fidelity policy gradient (MFPG) algorithm tailored to PPO. Compared to existing works, which apply the control variate (CV) directly to the REINFORCE or PPO objectives, the authors add three techniques to reduce variance and control reliance on LF data: (1) synchronizing the LF rollouts with the HF rollouts every T steps to increase the correlation between LF and HF data; (2) estimating the CV coefficient from episode-level sums instead of per-time-step values, which captures cross-step covariance; and (3) a confidence-based gate that drops the LF correction, falling back to HF-only PPO, when the estimated correlation is not significant or the variance is not reduced.

The paper also proposes a budget-aware allocation of LF and HF data. Results show that these techniques improve vanilla MFPG-PPO and that it compares favorably with existing multifidelity RL schemes, including the off-dynamics methods PAR (Lyu et al., 2024) and DARC (Eysenbach et al., 2021).

Critical strengths
  • The three proposed techniques are principled: (1) periodically synchronizing LF rollouts with HF rollouts, (2) estimating the CV coefficient from episode-level statistics, and (3) confidence-based gating of the LF correction. Ablation studies validate them empirically.

  • The experiments show clear improvements over vanilla MFPG-PPO and more consistent gains than the other multi-fidelity baselines. The method is also demonstrated on a real robot.

  • The method is robust to a poor simulator. The LF data is used only as a control variate, and the confidence gate falls back to HF-only PPO when the LF data does not help. As a result, a poorly matched simulator mainly reduces the benefit rather than biasing learning.

Critical weaknesses
  • Although the proposed techniques stabilize the CV estimates for MFPG-PPO, the core components of MFPG are already established in prior work. These include the multi-fidelity CV estimator for policy gradients, coupled LF rollouts, and the application of this estimator to the PPO objective. This limits the novelty of the contribution, which lies mainly in the stabilization techniques and in scaling the evaluation to GPU-parallel simulation and physical robots.

  • The synchronization technique reduces variance at the cost of introducing bias. The appendix discusses this tradeoff, but the paper does not quantify it. In the sim-to-sim experiments, a large-batch HF gradient is available as a reference, so an ablation over the synchronization interval T reporting bias and variance (and/or MSE) against this reference would clarify whether the variance reduction outweighs the added bias.

  • The benefit of the budget-aware variant (MFPG-PPO-BA) is not clearly demonstrated. It is evaluated only in the cross-task comparison, where its performance is close to the fixed-ratio variant, and there is no direct comparison or ablation isolating the effect of the allocation.

Questions for authors
  • The paper cites DARAIL (Guo et al., 2024) but does not compare against it. This comparison would be informative because DARAIL was designed to fix DARC’s performance degradation when LF-to-HF transfer is hard.

  • The episode-sum coefficient fitting is presented as a contribution but is used only in the real-robot experiment, while all simulation experiments use per-step fitting. Its benefit is supported only by a higher estimated correlation per iteration, without a direct performance comparison with per-step fitting. An ablation comparing the two fitting choices in terms of performance would strengthen the conclusions.

5 comments
AuthorsOpenReview ↗

Response to Reviewer 7t5b (1/2)

We thank the reviewer for the detailed review and insightful suggestions, which have helped us improve the manuscript. All changes in the revised manuscript are marked in blue. We first summarize the changes that address the reviewer's points and then respond to each point.


Summary of manuscript changes:

  1. Weakness 1: Lemma 2 (Section 4.1; proof in Appendix A.2).
  2. Weakness 2: A policy gradient study at TT = 12, 24, and 48 (Section 5.6, Figure 6), with details, the effect of TT, and training runs at each TT in Appendix A.9 (Figures 16–18).
  3. Weakness 3: A direct comparison of MFPG-PPO-BA with MFPG-PPO-FR, and MFPG-PPO-BA at cost ratios rr = 3, 10, and 30 (Appendix A.10, Figures 19 and 20).
  4. Question 1: SAC-DARC and DARAIL baselines (Appendix A.11, Figures 21–22).

Weakness 1. Novelty.

We agree that the machinery of the multi-fidelity control variate estimator and coupled low-fidelity (LF) rollouts, and its use in the PPO objective, come from prior work, which we cite. The stabilization and the larger-scale evaluation are indeed part of our contribution. Beyond them, our contributions are the following.

  • Identifying the failure modes of naive MFPG constructions. A naive construction of MFPG for PPO fails. Through analysis and experiments, we identify its failure modes: loss of cross-fidelity correlation and variance inflation.

  • Analysis. Although the multi-fidelity control variate estimator takes a simple form, designing an RL algorithm around it involves many design choices. We analyze the effect of the unit at which the control variate coefficient is fitted (Lemma 1), the connection between scalar surrogate and policy gradient variance reduction (the new Lemma 2), and the effect of periodically re-synchronizing the coupled, but not the uncoupled, LF rollouts (Lemma 3). Prior MFPG work [1] does not study these aspects.

    Although we focus on PPO, the identified failure modes and analysis are not limited to it. In particular, Lemmas 1–3 hold, under their stated assumptions, for any per-step policy gradient surrogate used as the paired statistic.

  • Algorithm. MFPG-PPO is the result of the identification and analysis steps and addresses the identified failures. We also propose a budget-aware MFPG-PPO variant.

[1] Liu, Xinjie, Cyrus Neary, Kushagra Gupta, Wesley A. Suttle, Christian Ellis, Ufuk Topcu, and David Fridovich-Keil. "A multi-fidelity control variate approach for policy gradient estimation." 2026.


Weakness 2. Measuring gradient error and bias.

We thank the reviewer for the constructive feedback. We have added a gradient study that measures the policy gradient mean squared error and bias of MFPG-PPO at synchronization intervals TT = 12, 24, and 48 (Section 5.6, Figure 6), using the policy gradient computed from a large batch of high-fidelity (HF) data as the reference.

MFPG-PPO yields a lower policy gradient error than the policy gradient estimated from all the HF data alone in nearly all 18 settings. In 3 of the 18 settings, the 95% confidence interval of the gradient bias excludes 0. Even in these settings, the benefit of MFPG-PPO outweighs the bias.

AuthorsOpenReview ↗

Response to Reviewer 7t5b (2/2)

Weakness 3. The budget-aware variant (MFPG-PPO-BA).

We have added a side-by-side comparison between the budget-aware variant (MFPG-PPO-BA) and the fixed-ratio variant (MFPG-PPO-FR) (Appendix A.10, Figure 19) and an experiment that trains MFPG-PPO-BA at different HF-to-LF cost ratios to study the effect of cost allocation (Appendix A.10, Figure 20).

In Figure 19, we show the difference in mean HF evaluation return between MFPG-PPO-BA and MFPG-PPO-FR in 19 task and HF data budget settings. MFPG-PPO-BA outperforms MFPG-PPO-FR in 16 of the 19 settings, and the two are not distinguishable in the other 3. The gain is generally larger at smaller nominal HF data budgets.

In Figure 20, we train MFPG-PPO-BA at HF-to-LF cost ratios rr = 3, 10, and 30. In the tested setting, the return of MFPG-PPO-BA remains relatively stable across rr, with a slight decrease at r=30r = 30, i.e., when HF data are more expensive than at rr = 3 and 10. As HF data become increasingly expensive (larger rr), MFPG-PPO-BA allocates fewer HF and more uncoupled LF environments. The allocation in Figure 20 also suggests a possible reason for the higher return of MFPG-PPO-BA than MFPG-PPO-FR in Figure 19: within its cost budget, MFPG-PPO-BA collects more HF data at every rr than MFPG-PPO-FR.


Question 1. Comparison to DARAIL.

We added a comparison to DARAIL and DARC, both with the SAC backbone used in their original publication (Appendix A.11, Figures 21–22).

Among the four methods (the two added baselines DARAIL and SAC-DARC, SAC HF-Only, and our MFPG-PPO-FR), MFPG-PPO-FR yields among the best mean returns in all 8 tested settings and the most consistent performance across tasks and HF budgets, cf. Figure 21. In Figure 22, we analyze the failure modes of SAC-DARC and DARAIL.


Question 2. Comparison to per-step coefficient fitting.

We agree with the reviewer's point, and we are happy to add an ablation with per-step fitting.

We would like to clarify that we present the coefficient fitting unit as a tradeoff. Per-step fitting multiplies the sample size but misses cross-step covariance, cf. Lemma 1. We choose the unit by the measured correlation. In simulation, per-step correlations are already high, so we fit per step. On the robot, grasps occur at different steps in the two fidelities, and on the same samples the per-step correlation is only about 0.17 (240 pairs), against about 0.6 for episode sums (4 pairs, but noisier).


We hope these changes and answers address the reviewer's concerns. We are happy to answer any further questions.

AuthorsOpenReview ↗

Follow-up on Question 2: per-step vs. episode-sum coefficient fitting

"An ablation comparing the two fitting choices in terms of performance would strengthen the conclusions."

As promised in our earlier response, we ran this ablation on the real robot (Appendix A.12, Figure 23). We trained MFPG-PPO-FR on the Franka cube-picking task with the control variate coefficient cc fitted per step and otherwise the same training configuration as in Figure 4. With per-step fitting, MFPG-PPO-FR reaches a final real-robot success rate of 70.0%, below the 91.7% of MFPG-PPO-FR with episode-sum fitting and above the 55.0% of PPO HF-Only. So fitting the coefficient on episode sums also gives better real-robot performance on this task, in addition to the higher correlation shown in Figure 4(c).

We hope this addresses the reviewer's concern. We are happy to answer any further questions.

Reviewer_j48HOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies on-policy RL with limited target-domain data, where high-fidelity (HF) interactions are expensive, and low-fidelity (LF) simulation data are relatively cheap, abundant, but biased. Building on the multi-fidelity policy gradient (MFPG) framework, the authors propose MFPG-PPO, extending MFPG to modern actor-critic learning. The proposed method redesigns the sampling, advantage estimation, and control variate construction to preserve cross-fidelity correlation, and monitors estimator uncertainty to prevent variance inflation. A budget-aware variant further adapts HF and LF sample allocation. Experiments across simulated tasks and a physical robot task demonstrate improvements over HF-only PPO.

Critical strengths
  • The paper addresses an important practical problem of maintaining reliable learning when target-domain interactions are scarce in on-policy RL and proposes a well-motivated extension of MFPG to PPO, including periodic re-synchronisation, separate critics, coefficient re-fitting, and monitoring.
  • The experiments cover different HF budgets, LF-to-HF transfer difficulties, and a physical robot, demonstrating promising improvements over existing baselines.
Critical weaknesses

The paper motivates MFPG-PPO by the instability caused by noisy policy-gradient estimates, but the variance-reduction analysis and diagnostics focus on PPO surrogate statistics before differentiation. Lower surrogate variance does not necessarily imply lower policy-gradient variance. While the empirical performance improvements are encouraging, the extent to which they can be attributed to improved gradient estimation remains unclear.

Questions for authors

Gradient estimation improvement.

Could the authors clarify the connection between surrogate variance reduction and gradient estimation quality?

Coupling design.

A key challenge identified by the paper is preserving cross-fidelity correlation despite differences in HF and LF dynamics. The re-synchronisation interval TT influenced not only the correlation between rollouts but also the PPO horizon in simulation. The simulation experiments use a fixed rollout length, while the real-robot experiments report tuning the LF segment length without presenting the corresponding sensitivity results. Could the authors provide further evidence of how TT affects cross-fidelity correlation and learning performance? An ablation analysis of TT would help understand the proposed data-collection design.

Budget allocation.

The budget-aware experiments consider only one assumed HF and LF cost ratio. Its sensitivity to different cost ratios remains unclear. Moreover, could the authors clarify why the MFPG-PPO-BA sampling budget p is measured as 10Nrefhigh+409610N_{\mathrm{ref}}^{\mathrm{high}}+4096 instead of 11Nrefhigh+409611N_{\mathrm{ref}}^{\mathrm{high}}+4096?

Separate critics and Stitched GAE.

The proposed method trains separate critics for three branches. Could the authors clarify what critic configuration is used in the “w/o separate critics” ablation in Figure 5? Specifically, does it use a single critic shared across all three branches, or a shared critic only for the two LF branches?

In the real-robot experiments, coupled LF rollouts are independently re-synchronised and then stitched together for GAE computation. Could the authors clarify how this affects the interpretation of the resulting advantage estimates?

4 comments
AuthorsOpenReview ↗

Response to Reviewer j48H (1/2)

We thank the reviewer for the thoughtful review and comments, which have helped us strengthen the manuscript. All changes in the revised manuscript are marked in blue. Below, we first summarize the changes that address the reviewer's points and then answer each question.


Summary of manuscript changes:

  1. Weakness, Question 1: Lemma 2 (Section 4.1; proof in Appendix A.2) and a policy gradient study (Section 5.6, Figure 6; Appendix A.9.1, Figure 16).
  2. Question 2: Gradient correlation and learning performance across rollout lengths TT (Appendix A.9.2, Figures 17 and 18).
  3. Question 3: MFPG-PPO-BA at cost ratios rr = 3, 10, and 30 (Appendix A.10, Figure 20).
  4. Question 4: Clarifications (Sections 5.5 and 4.1).

Weakness, Question 1. Surrogate variance reduction and gradient estimation quality.

Lower surrogate variance does not guarantee lower gradient variance in general. We have added an analysis that gives the condition under which a control variate coefficient also lowers the gradient variance (Section 4.1, Lemma 2), and we measured the gradient estimation error and bias directly (Section 5.6, Figure 6; Appendix A.9.1, Figure 16).

Analysis (Section 4.1, Lemma 2; proof in Appendix A.2). We show that although the surrogate variance-minimizing coefficient c∗c^{\ast} need not equal the gradient variance minimizer c∇∗c_{\nabla}^{\ast}, any coefficient cc strictly between 00 and 2c∇∗2c_{\nabla}^{\ast} yields a lower multi-fidelity policy gradient variance than the policy gradient estimated from the coupled high-fidelity (HF) data alone, i.e., c=0c = 0.

Measurement (Section 5.6, Figure 6; Appendix A.9.1, Figure 16). Using the policy gradient computed from a large batch of HF data as the reference, we measure the policy gradient mean squared error of MFPG-PPO and its bias at rollout lengths TT = 12, 24, and 48. MFPG-PPO yields a lower policy gradient error than the policy gradient estimated from all the HF data alone in nearly all 18 settings. In 3 of the 18 settings, the 95% confidence interval of the gradient bias excludes 0. Even in these settings, the benefit of MFPG-PPO outweighs the bias.

Figure 16 in Appendix A.9.1 further compares the applied control variate coefficient cc with the interval 0<c/c∇∗<20 < c/c_{\nabla}^{\ast} < 2 of Lemma 2. The applied cc lies in this gradient-variance-reducing interval in 16 of the 18 settings. MFPG-PPO has a lower policy gradient error than the policy gradient estimated from the coupled HF data alone (c=0c = 0) in all 18 settings. The 2 settings outside the interval can still have a lower error because MFPG-PPO sets c=0c = 0 on minibatches that fail the monitoring checks (Section 4.1).


Question 2. Coupling design.

Besides the gradient study (Section 5.6, Figure 6; Appendix A.9.1, Figure 16), which measures the gradient error of MFPG-PPO at rollout lengths TT = 12, 24, and 48, we added Appendix A.9.2 (Figures 17 and 18). It studies how the policy gradient correlation between the coupled HF and low-fidelity (LF) data, ρ∇\rho_{\nabla}, and the learning performance change with TT.

As TT grows, the gradient correlation ρ∇\rho_{\nabla} falls monotonically, and the gradient error reduction also weakens monotonically in roughly all the settings. With an equal HF data budget, the learning performance is relatively stable across TT, possibly because TT sets both the GAE horizon and the re-synchronization interval. A short TT limits the divergence between the coupled HF and LF rollouts but also shortens the GAE horizon.

AuthorsOpenReview ↗

Response to Reviewer j48H (2/2)

Question 3. Budget allocation.

We added experiments (Appendix A.10, Figure 20) that train MFPG-PPO-BA at HF-to-LF cost ratios rr = 3, 10, and 30, with budgets equal to the per-iteration cost of MFPG-PPO-FR, p=(r+1)Nrefhigh+4096p = (r+1) N_{\mathrm{ref}}^{\mathrm{high}} + 4096.

In the tested setting, the return of MFPG-PPO-BA remains relatively stable across rr, with a slight decrease at r=30r = 30, i.e., when HF data become more expensive. Figure 20 also shows the budget allocation of MFPG-PPO-BA across rr. As HF data become more expensive (larger rr), MFPG-PPO-BA allocates fewer HF and more uncoupled LF environments.

In the experiments of Figures 3 and 19, the budget of MFPG-PPO-BA was p=10Nrefhigh+4096p = 10N_{\mathrm{ref}}^{\mathrm{high}} + 4096 instead of 11Nrefhigh+409611N_{\mathrm{ref}}^{\mathrm{high}} + 4096, since we did not count the cost of the NrefhighN_{\mathrm{ref}}^{\mathrm{high}} coupled LF environments. Under our cost model, this budget is 0.2 to 5.3% below the per-iteration cost of MFPG-PPO-FR, so the gap does not favor MFPG-PPO-BA. In Figure 19, MFPG-PPO-BA still outperforms MFPG-PPO-FR in 16 of 19 settings and is not distinguishable in the other 3 (Appendix A.10). Appendix A.10 notes this budget difference explicitly, and we have also revised its description in Appendix A.7.2. Figure 20 uses the matching budget.


Question 4. Separate critics and stitched GAE.

In the "w/o separate critics" ablation (Figure 5), the ablated variant uses a single critic, trained on HF data only and shared across all three branches. We thank the reviewer for the careful reading, and we have clarified this point in the manuscript (Section 5.5).

In the real-robot experiments, the coupled LF rollout segments within an HF episode are stitched into one episode for GAE computation. The advantage computed along the stitched coupled LF segments is not an advantage along a single LF trajectory, but a statistic aligned with the HF episode and constructed to correlate with the HF advantage. We have clarified this point in the manuscript (Section 4.1).


We hope these changes and answers address the reviewer's concerns. We are happy to answer any further questions.

Thank you for the detailed response and for providing the revised manuscript. These revisions have adequately addressed my main concerns. I appreciate the authors' efforts in strengthening the theoretical analysis and experimental evaluation.

AuthorsOpenReview ↗

Thank you for the follow-up

We thank the reviewer for reading our response and the revised manuscript. We are glad that the revisions have addressed the reviewer's main concerns. If the reviewer feels it is appropriate, we would be grateful if this could be reflected in the rating. We are happy to answer any further questions.

Reviewer_4cMGOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes MFPG-PPO, which uses low-fidelity (LF) simulations as control variates to reduce the variance of high-fidelity (HF) policy gradient estimation. The method extends multi-fidelity policy gradients to PPO and introduces several mechanisms to improve estimator stability. Experiments cover four simulated locomotion tasks and one real-world Franka manipulation task.

Critical strengths
  1. The paper addresses an important problem: improving RL data efficiency when target-domain interactions are expensive.

  2. The proposed sampling and variance-monitoring mechanisms are technically reasonable, with encouraging results in simulation.

Critical weaknesses
  1. Limited practical applicability. Most experiments are conducted on simulated locomotion tasks, where simulation-based training and sim-to-real transfer are already well established. The real-world evaluation is limited to a single Franka cube-picking task with low-dimensional state observations. It remains unclear whether the proposed method provides meaningful benefits for more challenging real-world RL problems, such as vision-based manipulation or complex multi-stage tasks. Consequently, the practical scope of the proposed approach appears limited.

  2. Limited methodological novelty. The core idea of using low-fidelity data as control variates builds directly on existing multi-fidelity policy gradient methods. The main contributions are adaptations to PPO, including coupled sampling, separate critics, coefficient re-estimation, and variance monitoring. While these modifications are useful, the methodological advance appears incremental, and the paper does not sufficiently establish a broader conceptual contribution beyond this specific algorithmic setting.

  3. Insufficient analysis of estimator bias. The authors acknowledge that differences between the distributions of coupled and uncoupled LF samples can introduce bias into the control variate estimator, but its practical impact on policy optimization is not sufficiently quantified.

Questions for authors
  1. Can the authors demonstrate the method's effectiveness on more challenging real-world manipulation tasks or visual policies?

  2. Can they directly measure policy gradient variance and quantify the bias introduced by the LF sampling design?

  3. How would the method compare with off-policy algorithms that leverage both HF and LF data?

4 comments
AuthorsOpenReview ↗

Response to Reviewer 4cMG (1/2)

We thank the reviewer for the careful review and constructive feedback, which have helped us improve the manuscript. All changes in the revised manuscript are marked in blue. We first summarize the changes that address the reviewer's points and then respond to each point.


Summary of manuscript changes:

  1. Weakness 2: We added Lemma 2 (Section 4.1; proof in Appendix A.2), which states when the coefficient fitted for the scalar surrogate also lowers the policy gradient variance.
  2. Weakness 3, Question 2: We added a policy gradient study that measures the gradient error and bias of MFPG-PPO (Section 5.6, Figure 6; Appendix A.9, Figures 16–18).
  3. Question 3: We added two off-policy baselines that use both high-fidelity (HF) and low-fidelity (LF) data, SAC-DARC and DARAIL (Appendix A.11, Figures 21–22).

Weakness 1, Question 1. Practical scope.

We agree that the main evaluation is on simulated locomotion tasks and that the real-robot evaluation is a state-based manipulation task.

Our focus. We do not claim to solve vision-based manipulation or complex multi-stage tasks. Rather, we focus on the reliability and scaling of multi-fidelity methods across HF data scarcity regimes and across tasks of differing LF-to-HF transfer difficulty, since a practitioner rarely knows in advance how informative LF data will be. Our evaluation emphasizes depth over breadth: for each task, we evaluate at least 4 HF data budgets, and 7 on the most challenging task. Across 16 simulated task settings, our results show that MFPG-PPO is the most reliable use of LF data in our comparison and scales monotonically with more HF data.

More complex real-world tasks. This is an exciting future direction, but it also raises new research questions, e.g., which visual representations preserve cross-fidelity correlation, and how to learn them. Understanding the fundamental properties of multi-fidelity visual representations is an important topic [1] but is beyond the scope of this work.

[1] Lei, Yu, Minghuan Liu, Abhiram Maddukuri, Zhenyu Jiang, and Yuke Zhu. "A mechanistic analysis of sim-and-real co-training in generative robot policies." 2026.


Weakness 2. Methodological novelty.

We agree that the multi-fidelity control variate estimator and coupled LF rollouts come from prior work, which we cite. Our contributions are the following.

  • Identifying the failure modes of naive MFPG constructions. A naive construction of MFPG for PPO fails. Through analysis and experiments, we identify its failure modes: loss of cross-fidelity correlation and variance inflation.

  • Analysis. Although the multi-fidelity control variate estimator takes a simple form, designing an RL algorithm around it involves many design choices. We analyze the effect of the unit at which the control variate coefficient is fitted (Lemma 1), the connection between scalar surrogate and policy gradient variance reduction (the new Lemma 2), and the effect of periodically re-synchronizing the coupled, but not the uncoupled, LF rollouts (Lemma 3). Prior MFPG work [2] does not study these aspects.

    Although we focus on PPO, the identified failure modes and analysis are not limited to it. In particular, Lemmas 1–3 hold, under their stated assumptions, for any per-step policy gradient surrogate used as the paired statistic.

  • Algorithm. MFPG-PPO is the result of the identification and analysis steps and addresses the identified failures. We also propose a budget-aware MFPG-PPO variant.

[2] Liu, Xinjie, Cyrus Neary, Kushagra Gupta, Wesley A. Suttle, Christian Ellis, Ufuk Topcu, and David Fridovich-Keil. "A multi-fidelity control variate approach for policy gradient estimation." 2026.

AuthorsOpenReview ↗

Response to Reviewer 4cMG (2/2)

Weakness 3, Question 2. Gradient variance and bias.

The revised manuscript now measures both (Section 5.6, Figure 6; Appendix A.9).

Using the policy gradient computed from a large batch of HF data as the reference, we measure the policy gradient mean squared error of MFPG-PPO and its bias. MFPG-PPO yields a lower policy gradient error than its HF-only counterpart in nearly all 18 settings. In 3 of the 18 settings, the 95% confidence interval of the gradient bias excludes 0. Even in these settings, the benefit of MFPG-PPO outweighs the bias.


Question 3. Off-policy methods that use HF and LF data.

The revised manuscript adds two off-policy baselines that use both HF and LF data (Appendix A.11, Figures 21–22).

These are DARC with a SAC backbone (SAC-DARC) [3] and DARAIL [4], which trains a policy to imitate a DARC teacher. Of the four methods (the two added off-policy multi-fidelity baselines, SAC HF-Only, and our MFPG-PPO-FR), MFPG-PPO-FR yields among the best mean returns in all 8 tested settings and the most consistent performance across tasks and HF budgets.

[3] Eysenbach, Benjamin, Swapnil Asawa, Shreyas Chaudhari, Sergey Levine, and Ruslan Salakhutdinov. "Off-dynamics reinforcement learning: Training for transfer with domain classifiers." 2021.

[4] Guo, Yihong, Yixuan Wang, Yuanyuan Shi, Pan Xu, and Anqi Liu. "Off-dynamics reinforcement learning via domain adaptation and reward augmented imitation." 2024.


We hope these changes and answers address the reviewer's concerns. We are happy to answer any further questions.

Thank you for the response, which has addressed some of my concerns. However, my main concern remains the practical relevance of this work. As a robotics researcher, I see two important settings for real-world applications: zero-shot sim-to-real transfer, as in humanoid robotics, where training relies entirely on low-fidelity simulation; and RL post-training of vision-language-action (VLA) models or world models that incorporate visual and language understanding. In the latter setting, low-fidelity simulation data is generally of very limited use because of the visual sim-to-real gap. This is why I find it difficult to see how this work would be useful in practice, and it is the main reason for my recommendation to reject the paper.

AuthorsOpenReview ↗

Response to Reviewer 4cMG's follow-up

We thank the reviewer for the follow-up. We are glad that our response addressed some of the concerns, and we appreciate the clear statement of the remaining one.

Our work targets a third setting between the two that the reviewer names. In this setting, low-fidelity (LF) simulation data alone may be too coarse for zero-shot transfer but still correlate with the high-fidelity (HF) target domain, and HF data are too scarce to train on alone. This setting is also studied by off-dynamics RL [1, 2] and sim-and-real co-training [3, 4]. In our experiments, MFPG-PPO provides the most reliable way to combine HF and LF data and scales monotonically with more HF data.

We agree that RL post-training of vision-language-action (VLA) models or world models with real-world and simulation data is an important direction. We believe that it calls for further research in several areas, e.g., real-to-sim tools [5] that build simulators from real scenes and could provide LF data that correlate more strongly with the real world, and understanding the fundamental properties of multi-fidelity visual representations [4].

We hope this clarifies the setting that our work targets. We are happy to answer any further questions.

[1] Eysenbach, Benjamin, Swapnil Asawa, Shreyas Chaudhari, Sergey Levine, and Ruslan Salakhutdinov. "Off-dynamics reinforcement learning: Training for transfer with domain classifiers." ICLR 2021.

[2] Lyu, Jiafei, Chenjia Bai, Jingwen Yang, Zongqing Lu, and Xiu Li. "Cross-domain policy adaptation by capturing representation mismatch." ICML 2024.

[3] Wei, Adam, Abhinav Agarwal, Boyuan Chen, Rohan Bosworth, Nicholas Pfaff, and Russ Tedrake. "Empirical analysis of sim-and-real cotraining of diffusion policies for planar pushing from pixels." IROS 2025.

[4] Lei, Yu, Minghuan Liu, Abhiram Maddukuri, Zhenyu Jiang, and Yuke Zhu. "A mechanistic analysis of sim-and-real co-training in generative robot policies." ICML 2026.

[5] Jain, Arhan, Mingtong Zhang, Kanav Arora, William Chen, Marcel Torne, Muhammad Zubair Irshad, Sergey Zakharov et al. "PolaRiS: Scalable real-to-sim evaluations for generalist robot policies." RSS 2026.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response: Summary of Revisions

We thank all reviewers for their thoughtful reviews and constructive feedback, which have helped us strengthen the manuscript. All changes in the revised manuscript are marked in blue. The main changes are:

  1. Gradient variance analysis (Section 4.1, Lemma 2; proof in Appendix A.2). The condition under which a fixed control variate coefficient cc, such as one fitted for the scalar surrogate, also lowers the policy gradient variance. (Reviewers j48H, 4cMG, 7t5b)
  2. Policy gradient study (Section 5.6, Figure 6; Appendix A.9, Figures 16–18). We measure the policy gradient error and bias of MFPG-PPO against a policy gradient reference computed from a large batch of high-fidelity (HF) data at rollout lengths TT = 12, 24, and 48. We also test the learning performance of MFPG-PPO at each TT with equal HF data. (Reviewers j48H, 4cMG, 7t5b)
  3. Budget-aware variant (Appendix A.10, Figures 19–20). A direct comparison of MFPG-PPO-BA with MFPG-PPO-FR in 19 settings, and MFPG-PPO-BA at HF-to-low-fidelity (LF) cost ratios rr = 3, 10, and 30. (Reviewers j48H, 7t5b)
  4. Off-policy baselines that use HF and LF data (Appendix A.11, Figures 21–22). SAC-DARC and DARAIL on H1 Rough and ANYmal-D Rough tasks at four HF data budgets. (Reviewers 4cMG, 7t5b)
  5. Ablation study with per-step control variate coefficient fitting on the real robot (Appendix A.12, Figure 23). MFPG-PPO-FR on the Franka robot task with the coefficient cc fitted per step instead of on episode sums. (Reviewer 7t5b)
  6. Clarifications and corrections. The details of the "w/o separate critics" ablation (Section 5.5), the advantage along stitched LF rollout segments (Section 4.1), and the budget of MFPG-PPO-BA in Figures 3 and 19 (Appendices A.7.2 and A.10). (Reviewer j48H)

Our individual responses point to these sections.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.