acceptodds
Under review as a conference paper at ICLR 2027

Model-Free Umbrella Reinforcement Learning

Abstract

We study hard reinforcement learning problems that combine sparse reward, state traps, and the absence of a terminal state. After an agent enters a trap, the trajectories it collects provide little evidence about an escape route. Umbrella Reinforcement Learning (URL) tackles this setting with an ensemble over the state space. Its objective combines return with the entropy of the ensemble's joint state and action distribution. The entropy encourages broad state coverage, while the return term favors rewarding behavior. The original URL formulation is written in continuous time and requires analytical state dynamics and their divergence. We introduce Model-Free Umbrella Reinforcement Learning (MF URL), a discrete-time instantiation trained from independent one-step queries to a resettable simulator, without analytical dynamics or policy trajectories, at the cost of resets to arbitrary states and many simulator queries. MF URL adapts the DualDICE objective by using a known proposal density as the denominator of the learned density ratio, so that the ratio determines the discounted state occupancy density. A second saddle point objective trains a critic with separate value and advantage heads. On Multi-Valley Mountain Car and StandUp, MF URL learns policies that reach and remain in the reward regions, with returns comparable to Model Based URL. Standard SAC and PPO remain well below both umbrella methods, even when PPO uses task-specific reward shaping. The learned densities recover the main structure of the Monte Carlo reference state distributions under the learned policies. A diagnostic comparison with MF URL SAC, which uses a temporal difference critic, finds that the saddle point critic agrees more closely with Monte Carlo reference returns under the policy learned by each method.

Primary area:
reinforcement learning
Keywords:
Reinforcement Learning, Model-Free Reinforcement Learning, Umbrella Reinforcement Learning, Sparse Reward Exploration, State Occupancy Estimation, Saddle-Point Optimization, State-Distribution Entropy, Hard Reinforcement Learning Problems

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_LVPnOpenReview ↗
Rating
2
Confidence
1
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This paper proposes Model-Free Umbrella Reinforcement Learning to address sparse-reward RL problems. The method extends Umbrella RL by replacing analytical dynamics with one-step simulator transitions. It combines state-distribution entropy regularization, DualDICE-based occupancy density estimation. Experiments on Multi Valley Mountain Car and StandUp show performance comparable to model-based URL and substantially better than standard baselines.

Critical strengths

The paper provides a mathematically grounded extension of Umbrella RL using discrete-time occupancy estimation. The interpretation is simple and illustration with the toy task is straightforward.

Critical weaknesses

The evaluation focuses on simple tasks, how is the performance on more complicated deep RL control suite is unclear. It is questionable on the extension of the framework to high-dimensional tasks, given that the method is quite complicated and require multiple optimization schemes. The baseline comparison is also limited, how about ask Astra to write a reward function and iterate? These simple tasks should be solvable in this way and basic RL algo.

Questions for authors
  1. how is performance to high-dim control problems with this approach?
1 comment
AuthorsOpenReview ↗

Response to Reviewer LVPn

1. Simple tasks, limited baselines, and a reward designed by a language model plus basic RL.

Thank you for these notes. We study hard problems with a sparse reward, state traps and no terminal state, where MF URL learns directly from the sparse reward. Reward shaping, whether written by hand or by a language model, is environment dependent and needs task knowledge and iteration for every new task; SAC with our Multi Valley Mountain Car shaping still reaches only 0.013 (Table 10). The revised paper compares against more baselines at equal training queries (Sec. 5.2, Table 1): PPO and A2C with the sparse reward reach a return of 0 on both tasks, while MF URL solves both. Designed rewards are represented by PPO with shaped rewards, which also solves both tasks, and SAC with shaped rewards (Table 10). We added planning methods with the same simulator access, Grid VI and fitted Q-iteration (App. A.13). A reward written and refined by a language model is another form of reward design; we could not run it in the discussion period.

Q1. Performance on high-dimensional control problems.

Thank you for this question. Shared queries lower the cost per iteration by a factor of 3.85, and the queries of one iteration can run in parallel (App. A.8). For larger state spaces we are extending MF URL with adaptive proposals (App. D), and on StandUp the first one reaches a return above 0.9 within 50,000 iterations, where the uniform proposal is still near 0 (Fig. 12). Our plan is to show that each proposal works on simple environments and then move step by step to high-dimensional tasks (App. D.5).

Reviewer_dPsbOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper extends Umbrella RL (Nuzhin & Brilliantov, 2025) to sparse-reward problems with traps and no terminal state, training on states sampled from a fixed proposal distribution instead of trajectories. Its main change is to drop the need for the analytical dynamics v(s,a) and their divergence grad v, using one-step queries to a simulator that can be reset to any state. The occupancy density is learned with a DualDICE-style min-max problem. The critic is learned with the SBEED min-max form of the Bellman error, plus a value/advantage head centred on the current policy. On two low-dimensional tasks with discrete actions and deterministic dynamics, the method matches the model-based version, while SAC, PPO and A2C fail.

Critical strengths

MF URL removes the original Umbrella RL’s need for analytical dynamics and their divergence by using only one-step queries to a resettable simulator, through a clean combination of a DualDICE density estimator referenced to the known proposal and a policy-centred saddle-point critic, both supported by correct idealized saddle-point proofs.

Critical weaknesses
  • Originality seems limited: the objective, the benchmarks, the gradient argument and the training scheme come from a previous 2025 paper. The new part basically combines known estimators.

  • The fair baseline is missing: with the same simulator access, grid value iteration does better, and the method uses about 10⁵ times more simulator calls than the RL baselines.

  • The policy-gradient proof weights states by dπ, but the actor weights them by m, so the update used is not the gradient of the stated objective (this really is a drawback in the current exposition)....

  • The theory also assumes dπ is absolutely continuous and positive on the proposal support which breaks in many settings. In particular, state clipping and unreachable regions break both assumptions, which fits the unstable density results on StandUp.

  • Finally, the critic comparison and the ablation are not rigorous enough (different networks and clamps, iterations counted instead of simulator queries), and measured in queries the ablation favours removing the density term.

Questions for authors
  • The C.2 policy-gradient argument does not justify the actor the paper actually uses. the actor (eq. 23) weights states by m, not dπ. Under m, the analogous term E_m[∇ log dπ] has no reason to vanish. So C.2 says nothing about the update being performed. Eq. 23 is not the gradient of J_MFURL, and with shared parameters it need not be an ascent direction.

  • Absolute continuity dπ ≪ m and a bounded ratio fail on the benchmarks. In StandUp, gravity drives φ1 toward 0 or π, and d0 starts right next to those boundaries. So dπ almost certainly carries mass on lower-dimensional boundary sets, which is singular with respect to the uniform m. Hence dπ/m does not exist, so eq. 13 has no saddle point. “Every saddle point has w = dπ/µ” becomes vacuous rather than false. The same issue affects the critic. Bellman holding µ-a.e. (B.4) does not pin down Q on boundary sets that are µ-null but reached with positive probability. Q* = Qπ then does not follow, even in the idealized unrestricted-function setting

  • The assumption “dπ > 0 on the support of m” (B.3) is contradicted by Figure 3 ? Figure 3 shows large regions of zero occupancy under the learned policy, and with deterministic dynamics from a narrow d0 these are expected. So the objective is ill-defined on a large part of the proposal support. What is actually optimized is defined by the clamp ζ ≥ 10⁻¹² (A.4). The MF URL SAC baseline uses a different clamp, 10⁻⁸ (A.6), and a different density network (width 512, 5 layers, frequencies 30/4 vs 256, 6 layers, 5/1). So the two critics in the “critic comparison” are not even estimating the same shaped value

  • Section 4.3’s rationale for the saddle-point critic seems unfitted for the setting studied? Var(δ | s,a) = 0 on both benchmarks, as the authors acknowledge. And with a generative model, double sampling costs one extra query. The paper already spends |A| queries per state per call, so “unbiased without a second simulator query” saves about 2× in a regime where cost is dominated elsewhere.

  • Grid VI is described as “known dynamics.” But equation 36 only needs f(s_i, a) at grid points, i.e. one-step queries of a resettable deterministic simulator. That is exactly MF URL’s access assumption. On a 401×401 grid with 2 actions, one sweep costs ~3×10⁵ queries. MF URL uses ~1.9×10⁵ queries per outer iteration × 1.2×10⁶ iterations ≈ 2.3×10¹¹ queries on MVMC. Under equal access, the natural baselines are fitted value/Q-iteration on proposal states, not SAC/PPO on 1.2×10⁶ transitions.

3 comments
AuthorsOpenReview ↗

Response to Reviewer dPsb (1/2)

0. Originality.

Thank you for this note. The objective and the benchmarks follow Umbrella RL, and the contribution is a model-free realization of it for hard problems with a sparse reward, state traps and no terminal state. With the sparse reward, PPO, A2C and SAC do not solve these tasks, while MF URL solves them directly from the sparse reward, without environment-dependent reward shaping (Figure 2, Table 1). Using the known proposal as the reference measure turns a DualDICE ratio into the absolute discounted occupancy density, which the ensemble entropy requires and which cannot be read from a single transition. The critic combines the Fenchel dual form of the Bellman residual with a value and advantage head centred under the current policy, which supplies the successor term without a successor action sample.

1. Appendix C.2 does not justify the actor of Eq. 23, which weights states by mm instead of dπd^\pi.

Thank you for noting this. We took it into account in Sec. 4.4 and App. C.2. Eq. 23 is a soft policy improvement step weighted by the proposal, and it coincides with the policy gradient when m=dπm=d^\pi. With qk=Qk+τlog⁡πkq_k=Q_k+\tau\log\pi_k and τ=λΔt\tau=\lambda\Delta t,

Fk(π)=Es∼m[∑aπ(a∣s)qk(s,a)+τH(π(⋅∣s))]=−τEs∼m[KL(π(⋅∣s)∥softmax(qk(s,⋅)/τ))]+Ck,F_k(\pi)=\mathbb E_{s\sim m}\Big[\sum_a\pi(a|s)q_k(s,a)+\tau H(\pi(\cdot|s))\Big]=-\tau\mathbb E_{s\sim m}\big[\mathrm{KL}\big(\pi(\cdot|s)\Vert\mathrm{softmax}(q_k(s,\cdot)/\tau)\big)\big]+C_k,

and ∇ωFk=−Δt∇ωLπ\nabla_\omega F_k=-\Delta t\nabla_\omega\mathcal L_\pi at the current policy. The KL term is minimized state by state, so the target policy is the same for any proposal, and mm weights where it is fitted. Weighting by mm also improves the policy at states it does not yet visit. The learned ratio gives the dπd^\pi-weighted direction when it is needed, Em[ζξ∑a∇ωπωA^]\mathbb E_m[\zeta_\xi\sum_a\nabla_\omega\pi_\omega\hat A].

2. Absolute continuity dπ≪md^\pi\ll m with clipping, and the critic statement in B.4.

Thank you for noting this. Both simulators clip to the valid region, so a policy can stay exactly on its boundary, which the uniform proposal samples with probability zero. The ratio identity and the saddle point statements describe the occupancy inside the region (App. A.9). We measured the occupancy on the boundary (Table 7). The untrained StandUp policy places about half of it there. Trained policies move away from the boundary, with a mean of 3.75% at the end of training, and the final Multi Valley Mountain Car (MVMC) policies do not reach it. Adding the boundary to the reference measure, or reflecting instead of clipping, covers the full occupancy (App. A.9).

AuthorsOpenReview ↗

Response to Reviewer dPsb (2/2)

3. Positivity dπ>0d^\pi>0, the clamps, and the critic comparison with MF URL SAC.

Thank you for noting this. States that the policy never reaches carry no weight in JMFURLJ_{\mathrm{MFURL}}. Their reward term uses the learned ratio, which is positive by construction, and the floor where the ratio is very small (App. A.10). Table 9 shows how often the floor is active: for MF URL the learned ratio sets the reward on more than 97% of proposal states, while for MF URL SAC the floor is active on a larger share of StandUp states. Each critic is therefore compared with a Monte Carlo reference of its own shaped value, and Sec. 5.2 and App. A.10 describe the comparison as one between two complete implementations.

4. Rationale of the saddle-point critic: Var(δ∣s,a)=0\mathrm{Var}(\delta|s,a)=0 on both tasks, and double sampling with a generative model.

Thank you for noting this. We took it into account in Sec. 4.3 and App. B.2. On the deterministic benchmarks with exact action sums, direct residual minimization has the same solution as the saddle form. We keep the saddle form because MF URL is meant to extend to stochastic transitions, and we now present double sampling as an equally valid option there; in our implementation it would use the same number of queries. In both cases the critic minimizes a Bellman residual on proposal states, an off-policy distribution, where residual objectives are preferred for stability with neural networks (Baird, 1995; Zhang et al., 2020; Saleh and Jiang, 2019). We also tried a double-sampled squared-residual objective for the occupancy density (App. B.5): a weak-form residual over random Fourier test functions, with an unbiased gradient from two independent batches. On MVMC with a fixed policy, after 3.4×1063.4\times10^6 iterations it recovered the two main occupancy modes and not yet the paths between them (Fig. 11), because the variance of the double-sampled gradient needs very large sampling. The saddle-point estimator recovers the modes and the paths connecting them on MVMC (Sec. 5.2, Fig. 3).

5. Grid VI uses the same access as MF URL; fitted value or Q-iteration on proposal states as the natural baseline.

Thank you for noting this. Grid VI evaluates one-step queries at grid points, the same access as MF URL (Sec. 5.1, App. A.5). We also ran fitted Q-iteration on uniform proposal states with a kk-nearest-neighbour approximator. Both are in Table 1 and in App. A.13 (Table 13) at several budgets. On these two-dimensional tasks the planners need fewer queries; their cost grows with the state dimension, as rd∣A∣r^d|\mathcal A| for the grid and through the stored transitions for fitted Q-iteration, while MF URL keeps only network parameters. The trajectory baselines address a different question, whether learning from the agent's own experience solves these sparse-reward tasks at the same budget (Sec. 5.2).

AuthorsOpenReview ↗

Further revision on Q2 and Q3

Dear Reviewer dPsb, following your questions Q2 and Q3 we have revised the theory statements once more in the current PDF. Section 4.1 now states that the estimator targets the part of dπd^\pi absolutely continuous with respect to mm, and that clipping places the remainder on the region boundary. Appendix B.4 now states that equation 13 has a saddle point only when dπd^\pi is absolutely continuous with respect to μ\mu, so that otherwise the statement is vacuous rather than false, and that the Bellman condition determines Q∗Q^\ast only where the backup stays in the support of μ\mu, which clipping violates. Appendix B.3 now states that the floor ε\varepsilon enters the objective, with its activity reported in Table 9. Appendix A.9 shows that the boundary mass, singular with respect to mm, bounds the density error from below, accounting for about one third of the StandUp distance in Table 6 and for none on Multi Valley Mountain Car. We would be glad to clarify anything further.

Reviewer_6sZNOpenReview ↗
Rating
4(+2)
Confidence
3
Informativeness
3(+1)
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes MF-URL, a model-free extension of Umbrella RL that addresses hard RL problems combining sparse rewards, state traps, and the absence of a terminal state (i.e., a target region that must be maintained rather than reached). Whereas the existing MB-URL requires an analytical model of the dynamics (the rate of state change v and its divergence ∇·v), MF-URL replaces this requirement with one-step queries to a resettable simulator and learns by solving two saddle-point problems: occupancy density estimation (based on DualDICE) and a saddle-point critic (based on SBEED).

Critical strengths
  • The paper effectively extends MB-URL to a model-free approach by proposing optimization methods that address its limited applicability: MB-URL can only be applied to a very restricted class of environments.
Critical weaknesses
  • The poor sample efficiency appears to be a serious issue. The additional loops introduced by the proposed optimization procedures are not lightweight, which could make the method highly vulnerable to the curse of dimensionality as the state space grows.

  • The experimental evaluation seems insufficient. It consists only of a simple comparison on two tasks, and even there, the comparison is based on outer iterations, which leads to a mismatch in the actual amount of training. At the very least, the baselines should be given a training budget equal to the amount of MF-URL's inner-loop training.

  • The ablation study was conducted only on MVMC, and essential experimental results, such as the returns of MF-URL SAC and SAC+reward shaping, are not included.

Questions for authors
  • In the ablation, β = 0.01 appears to perform better than β = 0 when compared by outer iterations. However, in terms of inner-loop training, the training amounts are [49152 × 456000, 49152 × 658000] for β = 0 and [189152 × 254000, 189152 × 284000] for β = 0.01, which suggests that the method may actually be better off without the β term (β = 0). What would the results be under the same amount of training? A comparison on the same scale seems necessary.

  • As mentioned in the weaknesses, I am curious about the returns of MF-URL SAC and SAC+reward shaping. In particular, I would like to know how much the difference in critic RMSE for MF-URL SAC affects the return.

  • It seems necessary to validate the method on environments with more complex state spaces. Please show to what extent the method is robust to the state-space explosion problem.

5 comments
AuthorsOpenReview ↗

Response to Reviewer 6sZN

1. Sample efficiency; the inner loops are not lightweight and may suffer from the curse of dimensionality.

Thank you for noting this. We took it into account in three ways. First, the density updates now reuse the critic's queries, which lowers the queries per iteration by a factor of 3.85 (App. A.8, Table 5). Second, the queries of one iteration are independent one-step transitions, so they can be computed in one batch or spread over workers, while trajectory methods advance each environment step by step (Sec. 5.2, App. A.8). Third, we tested a lighter weak-form density objective with a single loss over random Fourier test functions. It is unbiased, but its variance needs very large sampling: after 3.4×1063.4\times10^6 iterations it recovered the two main occupancy modes and not yet the paths between them (App. B.5, Fig. 11). For larger state spaces we are developing adaptive proposals (point 4).

2. The comparison by outer iterations hides the amount of training; baselines and the β\beta ablation should be on the same scale.

Thank you for noting this. We now compare all methods at equal training queries (Sec. 5.2, Table 1; App. A.12, Fig. 10). At 1.97×10101.97\times10^{10} queries on Multi Valley Mountain Car (MVMC) and 2.95×10102.95\times10^{10} on StandUp, PPO and A2C with the sparse reward reach a return of 0, and MF URL reaches 0.456 and 1.015. With shared queries, β=0.01\beta=0.01 and β=0\beta=0 use the same queries per iteration, so the ablation is now at equal queries (App. A.11, Table 12, Fig. 9). On MVMC the density term reaches G≥0.40G\ge0.40 with 1.33×10101.33\times10^{10} queries against 2.75×10102.75\times10^{10}, and it matches with 1.97×10101.97\times10^{10} queries the return that β=0\beta=0 reaches with 5.9×10105.9\times10^{10}. On StandUp both variants reach the same return.

3. Returns of MF URL SAC and SAC with reward shaping; how much does the critic error of MF URL SAC affect the return?

Thank you for this question. App. A.10 and Table 10 now report the returns. Across methods, the critic with the larger error also gives the lower return: MF URL reaches 0.487 and 1.107, MF URL SAC 0.379 and 0.595. Within each method, seeds with different critic errors reach similar returns (Spearman correlation between normalized RMSE and return from −0.10 to 0.24, all p>0.5p>0.5). The two implementations also use different density networks and floors (App. A.10), and both differences contribute to the gap. SAC with the shaped reward reaches 0.013 on MVMC and 1.131 on StandUp.

4. Validation on more complex state spaces; robustness to state-space explosion.

Thank you for this note, which we share. At a fixed batch size the cost of one iteration does not grow with the state dimension, while the number of iterations does, because a uniform proposal covers the visited states more thinly. We are extending MF URL with adaptive proposals for this (App. D). The first one draws training states from a mixture of the uniform proposal, the start distribution and a tempered occupancy estimate, sampled with Langevin and global moves, and reweights the density loss by m/q≤2m/q\le2 without extra queries (Eqs. 53 to 55). On StandUp it reaches a return above 0.9 within 50,000 iterations, where the uniform proposal is still near 0, and ends at 1.15 against 1.02 (App. D.4, Fig. 12). We are also working on residual-based sampling, which places states where the critic's Bellman residual is large (App. D.5). Our plan is to show that each proposal works on simple environments and then move step by step to higher-dimensional tasks.

Reviewer_6sZNOpenReview ↗

Thank you for the revision; raising my score

Dear Authors,

Thank you for the detailed response and the revision, which addresses most of my concerns. The equal-query comparison with PPO and A2C (Table 1, Figure 10) and the per-query benefit of the density term with shared queries (Table 12) make the results much more convincing. They also clarify where MF URL stands: it solves these tasks from the sparse reward alone at a budget where sparse-reward PPO and A2C do not, while shaped PPO, and the planners that share its query access, do better on these low-dimensional tasks. It is also reasonable that some experiments could not be completed in the discussion period. I have raised my score from 2 to 3. Two points remain. An explanation would be enough for now, but I would appreciate as much detail as you can give, since my further assessment will depend on these points.

1. Lower return of MF URL SAC (Table 10). Besides the density network and floor that you mention, the critic training setup also differs (one sampled action vs. a sum over actions, a different batch, a target network). How much of the gap do you attribute to the critic objective itself?

2. Low return of SAC with the shaped reward on MVMC.

  • SAC uses 1.2 × 10⁶ transitions, while shaped PPO in Table 1 uses 1.97 × 10¹⁰, and at 1.2 × 10⁶ shaped PPO is also low (Figure 2). Could this be a budget effect?
  • The energy term of Eq. 39 is clipped by max(0, ·) and is not potential-based, and E(s) omits the factor g of Eq. 25. Could a policy collect it by swinging inside a valley instead of staying in the reward region?
  • On StandUp, shaped SAC reaches a return comparable to MF URL with far fewer queries, which would be worth discussing.

Minor. The abstract, introduction (line 093) and Sec. 5.2 (lines 410–411) say that shaped PPO stays well below both umbrella methods. At equal queries it is above MF URL on both tasks (Table 1), so please state the budget there.

Future directions. I like the idea of this work and hope it continues toward practical use. In the final version, I would encourage a short limitations and future-work paragraph in the main text that brings together the next steps now spread across the conclusion and Appendix, including the access requirement and how it might be relaxed.

Directions such as resetting only to visited states, combining MF URL with trajectory data, and tasks where shaping is hard to design seem promising, and if you see better directions, I would be glad to hear them.

AuthorsOpenReview ↗

Reply to Reviewer 6sZN, shaped SAC analysis and remaining points

Dear Reviewer 6sZN,

Thank you for the careful reading and for raising your score. We have uploaded a revised PDF with the changes below.

1. Lower return of MF URL SAC

The current runs do not separate the critic objective from the other differences, so we do not attribute the gap to it. Two results point elsewhere. Within each method, seeds with a lower critic error do not reach a higher return (Spearman correlations all close to zero, at most 0.24 in absolute value). The gap is largest on StandUp, where the density floor of MF URL SAC is active on 41% of proposal states against 0.8% for MF URL (Table 9). The controlled test, MF URL with only its critic replaced by a temporal difference critic, is the next experiment we will run.

2. Shaped SAC on MVMC

Energy term. Yes, the policies collect it by swinging in their start valley. We rolled out the three shaped SAC policies from the 50 evaluation starts.

Seed 0Seed 1Seed 2
Rollouts reaching the goal region140
Rollouts swinging in the start valley454250
Energy term collected per episode0.861.030.88
Net energy gain (what a potential based term pays)0.110.210.03

The term pays 5 to 25 times the net gain, because max⁡(0,⋅)\max(0,\cdot) rewards every upswing and never charges the downswing, and EE omits gg, which weights height 400 times more. With the same shaping, PPO at a larger budget reaches the goal while SAC swings. This is the dependence on reward design that MF URL avoids by learning from the sparse reward. App. A.7 now reports this analysis, and the final version will add a potential based variant of the shaping.

Budget. It contributes as well. Shaped PPO is also low at 1.2×1061.2\times10^6 transitions and reaches the goal only with many more (Fig. 10). SAC at equal queries is out of reach, at 7.4 GPU hours per 1.2×1061.2\times10^6 transitions.

StandUp. When a dense and well aligned shaping exists, as there, trajectory methods are more query efficient, and MF URL is meant for tasks without one. App. A.10 now states this.

3. Budget of the shaped PPO statement

The introduction and Sec. 5.2 now state the budget and point to Table 1 for equal queries.

Future directions

We will add a limitations and future work paragraph to the main text, including the access requirement.

Resetting only to visited states is an interesting generalization where arbitrary resets are impossible in principle. In our setting, however, the bottleneck is often the cost of a single simulator query rather than the ability to reset, so reusing a buffer of transitions is even more attractive, since it reduces queries regardless of where the states come from.

Combining MF URL with trajectory data is a substantive problem, but we rank it lower, because when the simulator can be modified, resets to arbitrary states appear to us a more efficient way to solve the control problem than restructuring the method around sequential data.

We consider tasks where shaping is hard to design the most important direction, since most genuinely hard RL problems are of this kind. Our next step is truly high dimensional problems with adaptive sampling, where the proposal adapts during training, with quadrotor control as a concrete target.

We would be glad to clarify any remaining point.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General response and summary of revisions

We thank all reviewers for their careful reading and their notes. We took every note into account and uploaded a revised paper; all changes are marked in blue. The abstract and the main results in outer iterations (Figure 2) are unchanged. The main text now uses the tenth page allowed during the discussion, with a new paragraph and a new Table 1 in Section 5.2 that compare all methods at equal training queries.

MF URL targets hard problems that combine a sparse reward, state traps and no terminal state. On these tasks PPO, A2C and SAC with the sparse reward do not reach the reward region, while MF URL solves both tasks directly from the sparse reward, without reward shaping (Figure 2, Table 1). Reward shaping is environment dependent: it needs task knowledge, such as energy shaping for Multi Valley Mountain Car (MVMC) and goal-progress shaping for StandUp, and a shaping that helps one method may not help another (SAC with the MVMC shaping reaches 0.013, Table 10). MF URL uses the same objective on both tasks.

The main change in the method is shared queries: the density updates now reuse the transitions that the critic has already queried. The density objective and the number of updates stay the same, and one outer iteration needs 49,152 queries on MVMC and 98,304 on StandUp instead of 189,152 and 378,304 (App. A.8, Table 5). All new MF URL results use shared queries.

Changes in the revised paper

ChangeReviewersWhere
All methods at equal training queries6sZN, LVPnSec. 5.1, Sec. 5.2, Table 1; App. A.12, Fig. 10
Shared queries (3.85 times fewer queries per iteration)6sZNSec. 5.2; App. A.8, Table 5
Queries of one iteration can run in parallel6sZNSec. 5.2; App. A.8
State density ablation at equal queries6sZNSec. 5.2; App. A.11, Table 12, Fig. 9
Returns of MF URL SAC and SAC with a shaped reward6sZNSec. 5.2; App. A.10, Table 10
Adaptive proposal (extension in progress)6sZN, LVPnApp. D.1 to D.4, Fig. 12
Residual-based adaptive sampling, step-by-step plan6sZN, LVPnApp. D.5
Weak-form density objective that we tested6sZNApp. B.5, Fig. 11
Actor as a proposal-weighted soft policy improvement stepdPsbSec. 4.4; App. C.2
Occupancy on the clipping boundarydPsbSec. 4.1; App. A.9, Table 7; App. B.4
Unreached states and density floorsdPsbSec. 4.1; App. A.10, Table 9
Saddle form and double samplingdPsbSec. 4.3; App. B.2
Grid VI uses the same access; fitted Q-iteration addeddPsb, LVPnSec. 5.1, Table 1; App. A.5; App. A.13, Table 13
Access requirement and planners in the conclusionallSec. 6

Points we could not complete in the discussion period

PointReviewersStatus
High-dimensional control suites6sZN, LVPnWe first want each adaptive proposal to work on simple environments, then scale up step by step (App. D). StandUp results in App. D.4.
Reward written by a language modelLVPnNot run in the time available. Designed rewards are represented by PPO and SAC with shaped rewards (Table 1, Table 10).
Boundary in the reference measure, or reflecting dynamicsdPsbNamed in App. A.9; runs need more time.
Figure 2 retrained with shared queries for 1.2M iterations6sZNFigure 2 keeps the submitted runs; the equal-query comparison is in Table 1 and App. A.12.

Table 1 of the revised paper (same 50 start states, greedy actions, horizon 1000, sparse reward; one query is one successor state for one state action pair, or one environment transition for trajectory methods)

MethodDataMVMC queriesMVMC returnStandUp queriesStandUp return
MF URL, sparse rewardproposal states1.97e100.456 ± 0.0052.95e101.015 ± 0.089
PPO, sparse rewardown trajectories1.97e100.000 ± 0.0002.95e100.000 ± 0.000
A2C, sparse rewardown trajectories1.97e100.000 ± 0.0002.95e100.000 ± 0.000
PPO, shaped rewardown trajectories1.97e100.493 ± 0.0082.95e101.238 ± 0.009
Fitted Q-iterationproposal states1.2e70.513 ± 0.0041.2e71.140 ± 0.061
Grid VIgrid states3.2e50.5052.6e61.473

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.