- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper studies first-order bilevel reinforcement learning with continuous states and finite actions. It combines paired policy responses, shared-trajectory estimation, and persistent paired critic training to obtain interaction complexity for expected squared upper-gradient norm. The guarantee concerns a selected local response under fixed KL regularization, initialization identifiability, and exact value realizability in a fixed feature span. Validation includes a constructed continuous-state problem and adapted SAC experiments on Walker Walk and Highway-fast.
Critical strengths
-
The analysis controls normalized differences between critic errors, rather than bounding endpoint errors separately. Together with shared-trajectory sampling, this addresses an important source of variance and bias amplification as the response perturbation shrinks.
-
The paper explicitly analyzes local containment, persistent critic tracking, and finite actor solves, including their sampling costs. These results strengthen the guarantee beyond an analysis assuming accurate critics or exact lower-level optimization.
Critical weaknesses
-
The contribution needs clearer positioning. The guarantee relies on strong fixed KL regularization, a restricted actor chart, and exact fixed-feature value realizability; the critic also approaches a linear model as accuracy increases. Related coupled first-order bilevel methods already achieve the same accuracy exponent under stochastic smoothness, notably Kwon, Kwon, and Lyu (ICML 2024). The RL-specific advances over this work should be clarified.
-
Empirical validation is limited primarily to an adapted Gaur baseline. Missing comparisons include the same pipeline without response correction and standard preference-based reward learning, making the source of improvement difficult to establish. Known-reward SAC is mentioned but not shown in the benchmark figures. Additionally, the Walker curves appear inconsistent with the stated evaluation checkpoints and total-interaction accounting. These issues weaken the practical evidence.
Questions for authors
- Clarification: Which technical contributions go beyond the coupled estimation and tracking analysis of Kwon, Kwon, and Lyu (ICML 2024)?
- Clarification: Can you reconcile the Walker figures with the four stated evaluation checkpoints and the additional 9,216 interactions, using per-checkpoint logs?
- Evidence request: Under matched interaction and label budgets, how does the method compare with the same pipeline without response correction and with standard preference-based reward learning?
3 comments
Dear Reviewer,
Thank you again for your review. As the discussion period closes soon, we wanted to check whether our response and the revised PDF (changes in blue) address your concerns. We would be glad to clarify anything that remains unclear. If our response resolves your concerns, we would be grateful if you would consider updating your assessment.
In brief, the revision explains what bilevel RL requires beyond Kwon, Kwon & Lyu (ICML 2024), adds the matched-budget coefficient-zero control and known-reward SAC (Appendix E, Figure 6, Tables 10–11), and corrects the evaluation schedule and interaction axis.
Thank you for the detailed response and for running the additional ablation. These revisions have helped address my main concerns, and I have raised my rating accordingly.
We thank the reviewer for a detailed reading, including the interaction accounting. Please also see the general response and Table 1.
Q1: beyond Kwon, Kwon & Lyu (ICML 2024). Thank you; we now cite this work (our existing "Kwon et al., 2024" is the ICLR 2024 penalty paper). We agree it attains the same accuracy exponent under stochastic smoothness, with a matching lower bound, and we claim no new exponent; its dichotomy explains the gap between Gaur et al. (2025) and ours. Relative to prior BRL guarantees (Table 1; general response), ours is the only one that is single-loop, fully first-order, for continuous states, free of lower-level PL, and covers a trained nonlinear critic. KKL treat generic bilevel optimization with a globally strongly convex lower level. The RL-specific work is establishing their conditions where they fail:
Restrictions. Section 4 now lists them: fixed τ large enough for curvature, the retained actor chart, exact realizability (C2; verified on a continuous-state instance, Lemma C.70), finite actions. On the critic, we agree: with mQ≳ε−1/2 it is a vanishing perturbation of a linear model in fixed features, and we now say we claim no feature learning. The paired tracking of the divided coordinate v is what carries over to the linear case.
Q3: matched budgets. We ran the same pipeline with correction coefficient 0 on both tasks and the same five seeds, with the same 500 labels and schedule. Its response branches still collect their auxiliary interactions (9,216 on Walker), which enter no update, so the total budget is matched. As the stack follows PEBBLE (unsupervised warm-up, disagreement sampling, replay relabeling, critic reset), this control is also the standard preference-based baseline at a matched interaction and label budget; known-reward SAC is added as an upper reference. Final returns with bootstrap intervals are in Table 10, per-seed paired differences against the control in Table 11, and full curves in Figure 6. The Gaur et al. (2025) baseline is the upper-level objective Gaur et al. use in their own experiments (their Section F), G+Vbase/σ, run in the same pipeline; its σ was tuned on one seed (Table 13), so we draw no conclusion from that comparison. Appendix E now lists the adaptation's departures from the analyzed algorithm, including that its perturbed branch is tilted toward the task reward, so these runs do not test the theorem. In light of these results, the revision presents Walker Walk as an exploratory adaptation rather than as evidence of practical gains, and the empirical claims rest on the exact-instance mechanism study (Section 5, Appendix D). On Highway-fast all four methods reach the same return.
Q2: Walker figures vs. checkpoints. The text and figures disagreed; the logs resolve it. Both preference methods were evaluated at step 0 and every 5k main steps, and the actors saved at 25k/50k/75k/100k were reloaded and re-evaluated to validate the checkpoints; the text described only the saved actors. The submitted curves also ended at 100k, although the final checkpoint costs 109,216 interactions: nine correction blocks, from 15k to 55k main steps, each add 1,024 auxiliary interactions. Appendix E now states the schedule, and revised Figure 6 places every measured checkpoint at main plus cumulative auxiliary interactions, with markers only where evaluations exist (the control and known-reward SAC at step 0 and the four saved checkpoints). Per-seed, per-checkpoint records with all ten held-out episodes are in the updated supplementary material, and the Highway counts in Table 9 match them.