- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies follower-response ambiguity in Stackelberg / bilevel reinforcement learning. Existing methods typically optimize the leader against a particular entropy-regularized best response of the follower. However, when several follower behaviors are nearly optimal for the follower but have substantially different consequences for the leader, such a nominal-response assumption may lead to brittle leader policies.
The paper proposes RBC-HG, which first computes the nominal entropy-regularized follower response and then constructs a locally pessimistic follower response
\propto \pi_F^*(b\mid s,a)\exp(-\alpha Q_L(s,a,b)).$$ This response shifts probability mass toward follower actions that are less favorable to the leader while keeping the follower close to its nominal best response. The authors derive an off-policy hypergradient that differentiates through both the nominal follower adaptation and the pessimistic response transformation. The paper also provides theoretical guarantees showing that the proposed response is worst-case within an induced local KL neighborhood, remains $O(\alpha^2)$-optimal for the follower, and incurs at most an $O(\alpha)$ loss in nominal leader performance. Experiments are conducted on two three-state tabular Stackelberg Markov games. The results show that RBC-HG behaves similarly to BC-HG when follower-response ambiguity is benign, while sacrificing a small amount of nominal performance in exchange for substantially improved robustness in a response-sensitive benchmark. The paper additionally compares the qualitative behavior of the learned solution against the exact $\delta$-robust Stackelberg equilibrium in the small benchmark.Critical strengths
1. A simple and well-motivated differentiable surrogate for pessimistic follower responses.
The main conceptual contribution is not robust Stackelberg optimization itself, nor exponential tilting, but the introduction of a closed-form, differentiable, locally pessimistic response around an endogenous entropy-regularized follower best response. This provides a clean way to incorporate unfavorable near-optimal follower behavior into a bilevel RL objective without explicitly solving an additional global worst-response optimization problem at every leader update. I find this formulation elegant and potentially useful.
2. The theoretical analysis is closely aligned with the proposed construction.
The paper does more than introduce a heuristic adversarial follower. It provides a local KL worst-case interpretation and establishes that the pessimistic response remains near-optimal for the follower while controlling the nominal performance loss of the leader. The exact -RSE analysis in the small benchmark also provides a useful sanity check that the proposed mechanism induces the expected qualitative shift from a risky nominally optimal solution to a safer robust solution.
Critical weaknesses
1. The main tractability / scalability motivation is not convincingly validated by the current experiments.
RBC-HG is presented as a robust extension of BC-HG. However, the original BC-HG already supports trajectory-based hypergradient estimation and has been demonstrated in substantially larger and continuous settings. In contrast, RBC-HG is evaluated only on two three-state tabular Markov games.
More importantly, the reported implementation relies on the known transition model to compute the state-occupancy ratio and the value sensitivities
\qquad G_L=\nabla_{\theta_L}Q_L^v$$ by differentiating Bellman fixed points exactly. In particular, even the $G_F$ term inherited from BC-HG is evaluated using exact tabular sensitivities rather than the trajectory-based approximation used by the original BC-HG method. Therefore, while the proposed pessimistic response removes the need to solve an additional global robust-response optimization, it is not yet clear that the resulting method remains tractable in the more realistic unknown-dynamics or function-approximation regimes that motivate trajectory-based bilevel RL in the first place. At present, the practical tractability claim appears stronger at the formulation level than at the demonstrated algorithmic level. **2. The current experiments do not sufficiently disentangle the benefit of the robustness objective from the benefit of the proposed hypergradient implementation.** The pessimistic response is constructed using a learned nominal critic $\hat Q_L^v$, while the corresponding sensitivity $G_L$ is computed exactly using the known transition model. The paper does not systematically study how errors in $\hat Q_L^v$ affect the learned pessimistic response. This seems particularly important because the critic enters through the exponential transformation $$\pi_F^\alpha \propto \pi_F^*\exp(-\alpha \hat Q_L^v),$$ so relative action-value errors may be amplified, especially for larger $\alpha$. In addition, the paper does not include a simple stop-gradient baseline that uses the same pessimistic response $\pi_F^\alpha$ but omits differentiation through the pessimistic transformation. Such a baseline would clarify whether the new $-\alpha \nabla_{\theta_L}Q_L^v$ term is important in practice, or whether most of the observed gain comes simply from training the leader against a pessimistically perturbed follower. Since the benchmark is extremely small and the transition model is known, it should also be feasible to compute or closely approximate the exact optimum of the proposed RBC objective. Comparing RBC-HG against this oracle would help separate objective approximation from optimization error.Questions for authors
- Scalability and tractability.
BC-HG already provides a trajectory-based hypergradient estimator and has been evaluated in larger discrete and continuous settings. In the current work, RBC-HG instead relies on exact model-based value sensitivities and exact state-occupancy ratios in the reported experiments. Can the authors replace these quantities with trajectory-based or learned estimators under unknown dynamics and demonstrate RBC-HG on at least one of the larger or continuous settings used by BC-HG? If not, what is the main empirical evidence supporting the claim that the proposed robust extension remains tractable beyond the current tabular setting?
- Sensitivity to critic approximation.
The pessimistic response is constructed using the learned nominal critic , whereas the corresponding sensitivity is computed exactly from the known transition model. How sensitive is RBC-HG to approximation error in ? In particular, since the critic enters through , relative errors may alter or even reverse the ranking of follower actions. An ablation comparing exact-, learned-, and noisy- constructions would be very informative.
- Necessity of the new pessimistic hypergradient term.
What happens if the same pessimistic follower response is used during training, but the transformation is treated with stop-gradient, i.e., the term is removed from the leader hypergradient? This would help isolate the benefit of the proposed hypergradient derivation from the more basic effect of training against a pessimistically perturbed follower.
- Exact optimization of the proposed objective.
Given the very small tabular benchmarks and known transition model, can the authors compute the exact or near-exact optimum of
and compare it against the solution found by RBC-HG? This would clarify whether any remaining gap is due to the local robustness surrogate itself or to the learning algorithm used to optimize it.
4 comments
Continuation of the previous comment
3. Necessity of the pessimistic hypergradient term
We agree that the original benchmarks did not isolate the necessity of differentiating through the pessimistic transformation. In particular, when follower actions induce identical successor states, the relevant action-dependent continuation sensitivities cancel.
To address this concern directly, we introduced a new response-coupled three-state Markov game, documented in §5.5 and Appendix C.5, Figure 10. In this environment, follower actions affect subsequent state transitions, making the indirect response derivatives nonzero.
We compare full RBC-HG against two variants that use the same pessimistic response construction but omit either:
- The additional pessimistic derivative .
- All indirect response derivatives, retaining only the direct leader policy-gradient contribution.
Across 10 matched seeds at , full RBC-HG achieves a mean improvement in 150-step undiscounted leader return under the pessimistic follower of 3.73 over the first ablation (95% bootstrap CI: [3.49, 3.91]) and 2.95 over the direct-only variant (95% CI: [2.83, 3.04]).
Both improvements are positive across all 10 seeds.
The new Figure 11 (Appendix C.5) also reports gradient diagnostics, confirming that the pessimistic derivative contributes nontrivially during training before gradually decreasing.
Additional post-training stress tests, summarized in §5.5 and Figure 3, show that full RBC-HG sacrifices nominal performance and can perform worse under mild follower deviations, but provides improved robustness under certain stronger pessimistic, temperature-scaled, and near-best adversarial responses. The advantage is not uniform across perturbation types or strengths.
Since all three methods optimize against the same pessimistic response rule, while differing in which response derivatives are retained, these ablations help separate the benefit of the robustness objective from the benefit of the complete hypergradient.
The results therefore provide evidence that differentiating through the pessimistic transformation can meaningfully improve leader optimization in response-coupled environments.
4. Exact optimization of the proposed objective
We additionally evaluated the final policies from the derivative-ablation study against a numerical optimization reference for the proposed RBC objective.
This analysis is presented in §5.5 and the new Appendix C.6.
In the response-coupled benchmark, only the leader's action at state affects its policy-dependent decisions. The relevant leader policy can therefore be parameterized by .
For each candidate , we recompute the nominal entropy-regularized follower response, construct the pessimistic transformation using exact nominal leader action values, and evaluate the infinite-horizon discounted RBC objective through tabular Bellman solutions.
We then numerically maximize the resulting one-dimensional objective.
For , the numerical reference is attained at , with value 1038.174. The learned policies achieve the following mean values across 10 matched seeds:
| Method | RBC objective | Gap to reference |
|---|---|---|
| Full RBC-HG | 1038.105 | 0.069 |
| No pessimistic derivative | 1036.464 | 1.710 |
| No indirect derivatives | 1036.819 | 1.355 |
Full RBC-HG therefore reduces the observed numerical-reference optimization gap by approximately 96% and 95%, respectively, relative to the two ablations.
The corresponding paired improvements are 1.642 (95% CI: [1.518, 1.732]) and 1.286 (95% CI: [1.224, 1.330]), both positive across all 10 seeds.
This result indicates that the complete hypergradient approaches the numerical optimizer of the proposed RBC objective more closely than either ablation in this benchmark.
We emphasize, however, that the reference is numerical rather than a certified global optimum. It is also the optimum of the proposed local pessimistic-response objective, not the global robust Stackelberg objective. The experiment therefore helps distinguish optimization error from the properties of the response construction, without establishing a general convergence or global robustness guarantee.
We hope that the additional experiments and the corresponding revisions to §5.5 and Appendices C.5–C.7, together with the clarified limitations in §4.3 and §6, address the main uncertainties raised in the review.
Official Review
Thank you for the detailed response and the additional experiments.
The new results address my Questions 2–4. The exact-/learned-/noisy-critic comparison (Appendix C.7), the derivative ablations on the response-coupled game (§5.5, Appendix C.5), and the comparison against the numerical reference for the RBC objective (Appendix C.6) are the analyses I asked for, and I appreciate that their limitations are stated clearly.
However, my main concern (Weakness 1 / Question 1) remains unresolved. RBC-HG is positioned as a robust extension of BC-HG, whose central appeal is trajectory-based hypergradient estimation under unknown dynamics, demonstrated in larger and continuous settings. The response confirms that the reported implementation requires the known transition model, exact Bellman sensitivity solves, and exact occupancy ratios, and that it is evaluated only on three-state tabular games. No trajectory-based or learned estimator of these quantities is provided, and no larger setting is tested.
I appreciate that the tractability claims have been narrowed in §4.3 and §6. But narrowing the claim does not supply the missing evidence: whether the method remains practical in the regime that motivates trajectory-based bilevel RL is still open, and this was the primary basis for my score.
I therefore keep my rating unchanged. A demonstration with estimated sensitivities and occupancy ratios under unknown dynamics, on at least one of the larger settings used by BC-HG, would change my assessment.
We thank the reviewer for their reassessment and for acknowledging that Questions 2–4 have been addressed.
We agree that scalability remains an important limitation. However, our primary contribution is the locally pessimistic response formulation, its theoretical robustness guarantees, and the associated hypergradient, rather than scalable hypergradient estimation.
BC-HG's continuous Markov-game benchmark uses a Gaussian follower response obtained through a Riccati solve with also known dynamics. While our robustness formulation could in principle extend to this setting, the benchmark was primarily designed to evaluate bilevel optimization, not sensitivity to unfavorable near-optimal follower responses. Our controlled benchmarks deliberately isolate this latter phenomenon , which is central to our contribution.
We acknowledge that trajectory-based implementation and larger-scale validation remain future work, but hope the reviewer will assess the theoretical contribution and its empirical validation within their intended scope.
We thank the reviewer for their constructive assessment and for recognizing the motivation and theoretical properties of the proposed response mechanism. We agree that the original experimental evaluation left important questions about scalability, critic sensitivity, and the necessity of the additional hypergradient term unanswered.
We have revised the manuscript and conducted additional experiments to address these concerns, as detailed below in the order raised. For ease of review, all revisions made to the manuscript are highlighted in yellow, allowing the changes discussed below to be located more easily.
1. Scalability and tractability
We agree that the original experiments do not establish scalability to large or continuous state-action spaces or unknown dynamics.
The implementation reported in the paper uses exact tabular Bellman sensitivities and occupancy corrections to isolate the proposed response mechanism from additional estimation errors. In particular, although the hypergradient identity in Theorem 4.1 admits off-policy trajectory-based estimation, the experiments compute its value sensitivities using the known transition model.
This distinction is now made explicit in §4.3 and Appendix B.4–B.6.
The pessimistic response transformation itself only requires the nominal follower response and estimated leader action values. Its tabular construction costs O(∣S∣∣AL∣∣AF∣). However, the complete implementation also relies on model-based Bellman sensitivity solves and occupancy-ratio calculations, whose computational costs increase with the size of the state space.
Thus, while the local response transformation avoids an additional global worst-response optimization, the present implementation does not establish that RBC-HG scales comparably to trajectory-based BC-HG.
We have clarified this limitation in §4.3 and §6. Future work will investigate scalable estimation of the required sensitivities and occupancy ratios under unknown dynamics, extensions to larger and continuous state-action spaces, and games with multiple followers.
We therefore restrict our demonstrated tractability claims to the local response construction and its tabular implementation.
2. Sensitivity to critic approximation
We conducted an additional experiment to investigate how approximation errors in the nominal leader critic affect the pessimistic follower response.
The results are presented in the new Appendix C.7, Figure 12.
For each of 10 matched training seeds, we fixed the learned leader policy and nominal follower response, then compared pessimistic responses constructed using the learned nominal critic Q^Lf and the exact nominal leader action values obtained from the tabular Bellman equations.
For full RBC-HG, the mean total variation distance between these responses is approximately 7×10−4, indicating little distortion from the learned critic in the evaluated setting. The two derivative ablations exhibit similarly small discrepancies.
We additionally introduced controlled Gaussian noise into the exact nominal critic, using 100 noise realizations per seed. Response distortion increases with noise magnitude. At σ=2, full RBC-HG exhibits a mean TV distance of approximately 0.095, compared with 0.22–0.23 for the two derivative ablations.
We also included a short analytical observation in Appendix C.7. If ∥Q^L−QL∥∞≤ε, then the perturbation of any pairwise pessimistic-response logit difference is at most 2αε. Consequently, action rankings cannot reverse when their original logit separation exceeds this bound, although near-tied actions may remain sensitive.
Importantly, these experiments evaluate response distortion at fixed learned policies. The differences between methods therefore reflect the sensitivity of their respective learned policies, rather than an intrinsic stability guarantee for the full hypergradient estimator. They do not establish training convergence under corrupted critics.
The revised Appendix C.7 explicitly distinguishes these conclusions and their limitations.