acceptodds
Under review as a conference paper at ICLR 2027

Pessimistic Follower Responses for Robust Stackelberg Reinforcement Learning

Abstract

Stackelberg and bilevel reinforcement learning optimize a leader while accounting for the adaptation of a follower, but typically assume that a particular optimal or entropy-regularized follower response will be realized. This can be fragile when approximately optimal follower behaviors have substantially different consequences for the leader. We introduce a locally pessimistic response model for two-player Stackelberg Markov games that exponentially tilts the follower's entropy-regularized best response toward actions that are less favorable to the leader, without modifying the follower's learning objective. We derive an off-policy hypergradient that differentiates through both follower adaptation and the pessimistic response transformation. We show that the proposed response is exactly worst-case within an induced local KL neighborhood, remains -optimal for the follower, and that optimizing the leader against this pessimistic response sacrifices at most nominal leader performance.

Primary area:
reinforcement learning
Keywords:
Bilevel Reinforcement Learning, Stackelberg Games, Game Theory

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_2McAOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies follower-response ambiguity in Stackelberg / bilevel reinforcement learning. Existing methods typically optimize the leader against a particular entropy-regularized best response of the follower. However, when several follower behaviors are nearly optimal for the follower but have substantially different consequences for the leader, such a nominal-response assumption may lead to brittle leader policies.

The paper proposes RBC-HG, which first computes the nominal entropy-regularized follower response πF∗\pi_F^* and then constructs a locally pessimistic follower response

\propto \pi_F^*(b\mid s,a)\exp(-\alpha Q_L(s,a,b)).$$ This response shifts probability mass toward follower actions that are less favorable to the leader while keeping the follower close to its nominal best response. The authors derive an off-policy hypergradient that differentiates through both the nominal follower adaptation and the pessimistic response transformation. The paper also provides theoretical guarantees showing that the proposed response is worst-case within an induced local KL neighborhood, remains $O(\alpha^2)$-optimal for the follower, and incurs at most an $O(\alpha)$ loss in nominal leader performance. Experiments are conducted on two three-state tabular Stackelberg Markov games. The results show that RBC-HG behaves similarly to BC-HG when follower-response ambiguity is benign, while sacrificing a small amount of nominal performance in exchange for substantially improved robustness in a response-sensitive benchmark. The paper additionally compares the qualitative behavior of the learned solution against the exact $\delta$-robust Stackelberg equilibrium in the small benchmark.
Critical strengths

1. A simple and well-motivated differentiable surrogate for pessimistic follower responses.

The main conceptual contribution is not robust Stackelberg optimization itself, nor exponential tilting, but the introduction of a closed-form, differentiable, locally pessimistic response around an endogenous entropy-regularized follower best response. This provides a clean way to incorporate unfavorable near-optimal follower behavior into a bilevel RL objective without explicitly solving an additional global worst-response optimization problem at every leader update. I find this formulation elegant and potentially useful.

2. The theoretical analysis is closely aligned with the proposed construction.

The paper does more than introduce a heuristic adversarial follower. It provides a local KL worst-case interpretation and establishes that the pessimistic response remains near-optimal for the follower while controlling the nominal performance loss of the leader. The exact δ\delta-RSE analysis in the small benchmark also provides a useful sanity check that the proposed mechanism induces the expected qualitative shift from a risky nominally optimal solution to a safer robust solution.

Critical weaknesses

1. The main tractability / scalability motivation is not convincingly validated by the current experiments.

RBC-HG is presented as a robust extension of BC-HG. However, the original BC-HG already supports trajectory-based hypergradient estimation and has been demonstrated in substantially larger and continuous settings. In contrast, RBC-HG is evaluated only on two three-state tabular Markov games.

More importantly, the reported implementation relies on the known transition model to compute the state-occupancy ratio and the value sensitivities

\qquad G_L=\nabla_{\theta_L}Q_L^v$$ by differentiating Bellman fixed points exactly. In particular, even the $G_F$ term inherited from BC-HG is evaluated using exact tabular sensitivities rather than the trajectory-based approximation used by the original BC-HG method. Therefore, while the proposed pessimistic response removes the need to solve an additional global robust-response optimization, it is not yet clear that the resulting method remains tractable in the more realistic unknown-dynamics or function-approximation regimes that motivate trajectory-based bilevel RL in the first place. At present, the practical tractability claim appears stronger at the formulation level than at the demonstrated algorithmic level. **2. The current experiments do not sufficiently disentangle the benefit of the robustness objective from the benefit of the proposed hypergradient implementation.** The pessimistic response is constructed using a learned nominal critic $\hat Q_L^v$, while the corresponding sensitivity $G_L$ is computed exactly using the known transition model. The paper does not systematically study how errors in $\hat Q_L^v$ affect the learned pessimistic response. This seems particularly important because the critic enters through the exponential transformation $$\pi_F^\alpha \propto \pi_F^*\exp(-\alpha \hat Q_L^v),$$ so relative action-value errors may be amplified, especially for larger $\alpha$. In addition, the paper does not include a simple stop-gradient baseline that uses the same pessimistic response $\pi_F^\alpha$ but omits differentiation through the pessimistic transformation. Such a baseline would clarify whether the new $-\alpha \nabla_{\theta_L}Q_L^v$ term is important in practice, or whether most of the observed gain comes simply from training the leader against a pessimistically perturbed follower. Since the benchmark is extremely small and the transition model is known, it should also be feasible to compute or closely approximate the exact optimum of the proposed RBC objective. Comparing RBC-HG against this oracle would help separate objective approximation from optimization error.
Questions for authors
  1. Scalability and tractability.

BC-HG already provides a trajectory-based hypergradient estimator and has been evaluated in larger discrete and continuous settings. In the current work, RBC-HG instead relies on exact model-based value sensitivities and exact state-occupancy ratios in the reported experiments. Can the authors replace these quantities with trajectory-based or learned estimators under unknown dynamics and demonstrate RBC-HG on at least one of the larger or continuous settings used by BC-HG? If not, what is the main empirical evidence supporting the claim that the proposed robust extension remains tractable beyond the current tabular setting?

  1. Sensitivity to critic approximation.

The pessimistic response is constructed using the learned nominal critic Q^Lv\hat Q_L^v, whereas the corresponding sensitivity GLG_L is computed exactly from the known transition model. How sensitive is RBC-HG to approximation error in Q^Lv\hat Q_L^v? In particular, since the critic enters through exp⁡(−αQ^Lv)\exp(-\alpha \hat Q_L^v), relative errors may alter or even reverse the ranking of follower actions. An ablation comparing exact-QQ, learned-QQ, and noisy-QQ constructions would be very informative.

  1. Necessity of the new pessimistic hypergradient term.

What happens if the same pessimistic follower response πFα\pi_F^\alpha is used during training, but the transformation is treated with stop-gradient, i.e., the −αGL-\alpha G_L term is removed from the leader hypergradient? This would help isolate the benefit of the proposed hypergradient derivation from the more basic effect of training against a pessimistically perturbed follower.

  1. Exact optimization of the proposed objective.

Given the very small tabular benchmarks and known transition model, can the authors compute the exact or near-exact optimum of

JL(πL,πFα(πL))J_L(\pi_L,\pi_F^\alpha(\pi_L))

and compare it against the solution found by RBC-HG? This would clarify whether any remaining gap is due to the local robustness surrogate itself or to the learning algorithm used to optimize it.

4 comments

We thank the reviewer for their constructive assessment and for recognizing the motivation and theoretical properties of the proposed response mechanism. We agree that the original experimental evaluation left important questions about scalability, critic sensitivity, and the necessity of the additional hypergradient term unanswered.

We have revised the manuscript and conducted additional experiments to address these concerns, as detailed below in the order raised. For ease of review, all revisions made to the manuscript are highlighted in yellow, allowing the changes discussed below to be located more easily.

1. Scalability and tractability

We agree that the original experiments do not establish scalability to large or continuous state-action spaces or unknown dynamics.

The implementation reported in the paper uses exact tabular Bellman sensitivities and occupancy corrections to isolate the proposed response mechanism from additional estimation errors. In particular, although the hypergradient identity in Theorem 4.1 admits off-policy trajectory-based estimation, the experiments compute its value sensitivities using the known transition model.

This distinction is now made explicit in §4.3 and Appendix B.4–B.6.

The pessimistic response transformation itself only requires the nominal follower response and estimated leader action values. Its tabular construction costs O(∣S∣∣AL∣∣AF∣)O(|\mathcal S||\mathcal A_L||\mathcal A_F|). However, the complete implementation also relies on model-based Bellman sensitivity solves and occupancy-ratio calculations, whose computational costs increase with the size of the state space.

Thus, while the local response transformation avoids an additional global worst-response optimization, the present implementation does not establish that RBC-HG scales comparably to trajectory-based BC-HG.

We have clarified this limitation in §4.3 and §6. Future work will investigate scalable estimation of the required sensitivities and occupancy ratios under unknown dynamics, extensions to larger and continuous state-action spaces, and games with multiple followers.

We therefore restrict our demonstrated tractability claims to the local response construction and its tabular implementation.

2. Sensitivity to critic approximation

We conducted an additional experiment to investigate how approximation errors in the nominal leader critic affect the pessimistic follower response.

The results are presented in the new Appendix C.7, Figure 12.

For each of 10 matched training seeds, we fixed the learned leader policy and nominal follower response, then compared pessimistic responses constructed using the learned nominal critic Q^Lf\hat Q_L^f and the exact nominal leader action values obtained from the tabular Bellman equations.

For full RBC-HG, the mean total variation distance between these responses is approximately 7×10−47\times10^{-4}, indicating little distortion from the learned critic in the evaluated setting. The two derivative ablations exhibit similarly small discrepancies.

We additionally introduced controlled Gaussian noise into the exact nominal critic, using 100 noise realizations per seed. Response distortion increases with noise magnitude. At σ=2\sigma=2, full RBC-HG exhibits a mean TV distance of approximately 0.095, compared with 0.22–0.23 for the two derivative ablations.

We also included a short analytical observation in Appendix C.7. If ∥Q^L−QL∥∞≤ε\|\hat Q_L-Q_L\|_\infty\leq\varepsilon, then the perturbation of any pairwise pessimistic-response logit difference is at most 2αε2\alpha\varepsilon. Consequently, action rankings cannot reverse when their original logit separation exceeds this bound, although near-tied actions may remain sensitive.

Importantly, these experiments evaluate response distortion at fixed learned policies. The differences between methods therefore reflect the sensitivity of their respective learned policies, rather than an intrinsic stability guarantee for the full hypergradient estimator. They do not establish training convergence under corrupted critics.

The revised Appendix C.7 explicitly distinguishes these conclusions and their limitations.

AuthorsOpenReview ↗

Continuation of the previous comment

3. Necessity of the pessimistic hypergradient term

We agree that the original benchmarks did not isolate the necessity of differentiating through the pessimistic transformation. In particular, when follower actions induce identical successor states, the relevant action-dependent continuation sensitivities cancel.

To address this concern directly, we introduced a new response-coupled three-state Markov game, documented in §5.5 and Appendix C.5, Figure 10. In this environment, follower actions affect subsequent state transitions, making the indirect response derivatives nonzero.

We compare full RBC-HG against two variants that use the same pessimistic response construction but omit either:

  1. The additional pessimistic derivative −α∇LQLπL,πF∗-\alpha\nabla_L Q_L^{\pi_L,\pi_F^*}.
  2. All indirect response derivatives, retaining only the direct leader policy-gradient contribution.

Across 10 matched seeds at α=0.5\alpha=0.5, full RBC-HG achieves a mean improvement in 150-step undiscounted leader return under the pessimistic follower of 3.73 over the first ablation (95% bootstrap CI: [3.49, 3.91]) and 2.95 over the direct-only variant (95% CI: [2.83, 3.04]).

Both improvements are positive across all 10 seeds.

The new Figure 11 (Appendix C.5) also reports gradient diagnostics, confirming that the pessimistic derivative contributes nontrivially during training before gradually decreasing.

Additional post-training stress tests, summarized in §5.5 and Figure 3, show that full RBC-HG sacrifices nominal performance and can perform worse under mild follower deviations, but provides improved robustness under certain stronger pessimistic, temperature-scaled, and near-best adversarial responses. The advantage is not uniform across perturbation types or strengths.

Since all three methods optimize against the same pessimistic response rule, while differing in which response derivatives are retained, these ablations help separate the benefit of the robustness objective from the benefit of the complete hypergradient.

The results therefore provide evidence that differentiating through the pessimistic transformation can meaningfully improve leader optimization in response-coupled environments.

4. Exact optimization of the proposed objective

We additionally evaluated the final policies from the derivative-ablation study against a numerical optimization reference for the proposed RBC objective.

This analysis is presented in §5.5 and the new Appendix C.6.

In the response-coupled benchmark, only the leader's action at state SS affects its policy-dependent decisions. The relevant leader policy can therefore be parameterized by p=πL(1∣S)∈[0,1]p=\pi_L(1\mid S)\in[0,1].

For each candidate pp, we recompute the nominal entropy-regularized follower response, construct the pessimistic transformation using exact nominal leader action values, and evaluate the infinite-horizon discounted RBC objective through tabular Bellman solutions.

We then numerically maximize the resulting one-dimensional objective.

For α=0.5\alpha=0.5, the numerical reference is attained at p=1p=1, with value 1038.174. The learned policies achieve the following mean values across 10 matched seeds:

MethodRBC objectiveGap to reference
Full RBC-HG1038.1050.069
No pessimistic derivative1036.4641.710
No indirect derivatives1036.8191.355

Full RBC-HG therefore reduces the observed numerical-reference optimization gap by approximately 96% and 95%, respectively, relative to the two ablations.

The corresponding paired improvements are 1.642 (95% CI: [1.518, 1.732]) and 1.286 (95% CI: [1.224, 1.330]), both positive across all 10 seeds.

This result indicates that the complete hypergradient approaches the numerical optimizer of the proposed RBC objective more closely than either ablation in this benchmark.

We emphasize, however, that the reference is numerical rather than a certified global optimum. It is also the optimum of the proposed local pessimistic-response objective, not the global robust Stackelberg objective. The experiment therefore helps distinguish optimization error from the properties of the response construction, without establishing a general convergence or global robustness guarantee.

We hope that the additional experiments and the corresponding revisions to §5.5 and Appendices C.5–C.7, together with the clarified limitations in §4.3 and §6, address the main uncertainties raised in the review.

Reviewer_2McAOpenReview ↗

Official Review

Thank you for the detailed response and the additional experiments.

The new results address my Questions 2–4. The exact-/learned-/noisy-critic comparison (Appendix C.7), the derivative ablations on the response-coupled game (§5.5, Appendix C.5), and the comparison against the numerical reference for the RBC objective (Appendix C.6) are the analyses I asked for, and I appreciate that their limitations are stated clearly.

However, my main concern (Weakness 1 / Question 1) remains unresolved. RBC-HG is positioned as a robust extension of BC-HG, whose central appeal is trajectory-based hypergradient estimation under unknown dynamics, demonstrated in larger and continuous settings. The response confirms that the reported implementation requires the known transition model, exact Bellman sensitivity solves, and exact occupancy ratios, and that it is evaluated only on three-state tabular games. No trajectory-based or learned estimator of these quantities is provided, and no larger setting is tested.

I appreciate that the tractability claims have been narrowed in §4.3 and §6. But narrowing the claim does not supply the missing evidence: whether the method remains practical in the regime that motivates trajectory-based bilevel RL is still open, and this was the primary basis for my score.

I therefore keep my rating unchanged. A demonstration with estimated sensitivities and occupancy ratios under unknown dynamics, on at least one of the larger settings used by BC-HG, would change my assessment.

We thank the reviewer for their reassessment and for acknowledging that Questions 2–4 have been addressed.

We agree that scalability remains an important limitation. However, our primary contribution is the locally pessimistic response formulation, its theoretical robustness guarantees, and the associated hypergradient, rather than scalable hypergradient estimation.

BC-HG's continuous Markov-game benchmark uses a Gaussian follower response obtained through a Riccati solve with also known dynamics. While our robustness formulation could in principle extend to this setting, the benchmark was primarily designed to evaluate bilevel optimization, not sensitivity to unfavorable near-optimal follower responses. Our controlled benchmarks deliberately isolate this latter phenomenon , which is central to our contribution.

We acknowledge that trajectory-based implementation and larger-scale validation remain future work, but hope the reviewer will assess the theoretical contribution and its empirical validation within their intended scope.

Reviewer_H35yOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper proposes a method for training robust leader policies in Stackelberg games with a single follower. The key idea is to train the leader against an alpha-pessimistic follower that deviates from best response by exponentially tilting its policy away from actions favorable to the leader. The authors derive hypergradient estimators that account for how the follower's policy changes in response to leader updates, and demonstrate improved robustness on small synthetic grid-world benchmarks compared to naive policy gradient and actor-critic baselines.

Critical strengths
  1. The mathematical derivation correctly accounts for the indirect effect of leader updates on follower policy through the chain rule. The use of importance sampling and occupancy ratios to make the gradient tractable is well-executed
  2. Robustness to follower deviations is important in real applications like pricing, security games, and human-AI interaction where followers may not perfectly best-respond due to bounded rationality, noise, or misaligned incentives.
Critical weaknesses
  1. The alpha-pessimistic follower seems a specific adversarial deviation that appears designed to make the hypergradient method tractable rather than to capture realistic behavior, and the reasoning feels circular: "we need robustness to follower deviations, so we train against this specific deviation"
  2. The evaluation is restricted to tiny synthetic environments that do not demonstrate practical applicability or scalability. All experiments involve one leader and one follower in small gridworlds with discrete actions (five-by-five grids, four actions each).
Questions for authors
  1. What is the empirical or theoretical justification for the alpha-pessimistic follower model?
  2. What is the wall-clock runtime of your method compared to baselines, and how does it scale with problem dimensions (state space size, action space size, horizon length, number of followers)?
3 comments

We thank the reviewer for their constructive assessment, particularly their recognition of the hypergradient derivation and the importance of robustness to follower deviations. We address the two questions in order.

1. Justification for the α\alpha-pessimistic follower model

The proposed response is not an adversarial perturbation selected solely for differentiability. Its motivation is to account for unfavorable follower behavior while remaining close to the response generated by the follower's own entropy-regularized objective.

As established in Proposition 4.2 (§4.4), for a fixed leader policy and state-action pair, πFα\pi_F^\alpha is the exact minimizer of the leader's nominal action value within an induced local KL neighborhood of πF∗\pi_F^*. Equivalently, it solves the KL-regularized optimization problem \min_{q\in\Delta(\mathcal{A}_F)}\left\\{\mathbb{E}_{b\sim q}[Q_L^{\pi_L,\pi^*\_F}(s,a,b)]+\frac{1}{\alpha}D_{\mathrm{KL}}(q\\|\pi^*\_F)\right\\}, where the distributions are conditioned on (s,a)(s,a)

This provides a precise interpretation: the follower response is shifted toward unfavorable actions while departures from its nominal behavior are penalized. The transformation preserves the probabilistic nature of the response and its full support in our finite-action setting.

Moreover, Theorem 4.3 (§4.4) establishes that the response remains O(α2)O(\alpha^2)-optimal for the follower's entropy-regularized objective, while Corollary 4.4 bounds the nominal leader-performance loss after optimizing against the modified response by O(α)O(\alpha), under the stated assumptions.

The experimental motivation is complementary. §5.4 and Appendix C.3–C.4 evaluate trained policies against epsilon-random, temperature-scaled, and near-best adversarial follower responses, rather than only against the perturbation used during training. The results demonstrate improvements under certain adverse deviations, with an explicit nominal-performance trade-off.

We have clarified this interpretation in §4, §4.4, and the revised introduction and related-work discussion (§1–2). We do not claim that the exponential tilt is a universal behavioral model, nor that local KL robustness is equivalent to solving the global worst-response problem. Its purpose is to provide a tractable local robustness mechanism with explicit approximation guarantees.

2. Computational runtime and scalability

We agree that the scalability of the complete algorithm must be distinguished from the computational cost of constructing the pessimistic response itself.

Given the nominal follower response and the leader action values, the exponential reweighting and normalization require O(∣S∣∣AL∣∣AF∣)O(|\mathcal S||\mathcal A_L||\mathcal A_F|) operations in the tabular setting. Importantly, the transformation does not require solving an additional global worst-response optimization problem.

However, our reported implementation also uses exact Bellman value-sensitivity solves and state-occupancy corrections from known transition dynamics. Dense Bellman-system factorizations can require O(∣S∣3)O(|\mathcal S|^3) operations, while finite-horizon occupancy propagation costs O(H∣S∣2)O(H|\mathcal S|^2) using dense transition matrices. These costs constrain the scalability of the complete implementation.

We have added this complexity discussion to §4.3, with implementation details in Appendix B.4–B.6. The revised §6 (Limitations and Future Work) also explicitly identifies larger and continuous state-action spaces, unknown dynamics, and multiple followers as future research directions.

Concerning the reported evaluation scale, we would also like to clarify that the benchmarks in the manuscript are three-state games with two leader actions and three follower actions, rather than the 5×55\times5 gridworlds with four actions mentioned in the review (Appendix C.1). Nevertheless, the underlying scalability criticism is valid: our experiments remain small, tabular, and limited to one leader and one follower.

We do not currently report a matched wall-clock comparison sufficient to establish an empirical efficiency advantage over BC-HG. We therefore distinguish the low cost of the local response construction from the substantially greater cost of the model-based estimation backend. In particular, our present results do not demonstrate scalability comparable to trajectory-based BC-HG in larger or unknown-dynamics environments.

We have narrowed the practical claims accordingly in §4.3, Appendix B, and §6, rather than extrapolating beyond the demonstrated setting.

For ease of review, all revisions made to the manuscript are highlighted in yellow, allowing the changes discussed below to be located more easily.

Thank you for clarifying the KL robustness interpretation and correcting my description of the experiments. I understand the mathematical rationale, but remain uncertain how well this approach protects against practically relevant follower deviations. Moreover, the evaluation is limited to three-state games. I maintain my concerns about applicability and scalability.

We thank the reviewer for their reassessment and acknowledge the remaining concerns regarding practical applicability and scalability.

While our experiments are limited to controlled tabular games, the robustness evaluations deliberately include follower deviations different from the exponential tilt used during training, namely epsilon-random, temperature-scaled, and near-best adversarial responses. These represent distinct mechanisms of follower suboptimality, and the observed improvements under several perturbation regimes provide preliminary evidence that the benefits are not restricted to the specific adversarial response used for training.

We agree that these evaluations do not establish robustness to empirically observed follower behavior or scalability to larger environments. Our objective is to establish and characterize a locally pessimistic response mechanism theoretically, with controlled experiments demonstrating its principal properties. We hope the reviewer will consider this contribution within its explicitly stated scope.

Reviewer_3zUhOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper considers situations in which approximately optimal follower behaviors can have substantially different consequences for the leader. The authors propose a pessimistic response model for two-player Stackelberg Markov games that exponentially reweights the follower's entropy-regularized best response. The algorithm is evaluated on two tabular games.

Critical strengths

The problem of robustness to unfavorable approximately optimal follower responses is interesting.

Critical weaknesses

Poor writing makes this work hard to follow and its theoretical contribution difficult to assess.

  1. Key parameters are used before being explained. α\alpha appears repeatedly in the abstract, introduction, and related work, but its meaning is only explained in Section 4. Section 5.3 introduces δ\delta-RSE without clearly defining what loss δ\delta measures in the main text.

  2. The notation is confusing and introduces unnecessary reading overhead. (1) The paper introduces vv and β\beta as joint policies, although neither symbol is a conventional choice for denoting a policy (Section 4.1). (2) The probability ratio ww is incorrectly introduced as a joint policy (Section 4.1). (3) Some shorthand notation offers limited simplification while making the text harder to understand, such as κα\kappa_\alpha for a KL divergence, ζ(s)\zeta(s) for an importance weight, and ρα\rho_\alpha for the previously stated constant. (4) The paper introduces p,qαp,q_\alpha and then pπL,qπLαp_{\pi_L},q_{\pi_L}^{\alpha} instead of consistently using the established policy notation.

  3. The mathematical presentation does not convey information efficiently. Proposition 4.2 and Corollary 4.4 mix auxiliary definitions and intermediate comparisons with the main conclusions. The explanations following the formulas are not sufficiently tied to their individual terms, making the results difficult to interpret and check.

The main robustness experiment does not establish the usefulness of the indirect hypergradient terms. In this experiment, all follower actions at either branch return to state SS. Consequently, the follower responses are independent of the leader policy. The indirect terms in Theorem 4.1, which capture how changes in the leader policy affect the follower response and thereby the leader's return, vanish in expectation.

Overall, the topic is interesting, but the poor writing prevents me from adequately checking the theoretical results and assessing the paper's theoretical contribution. Therefore, I recommend rejection.

Questions for authors

See the weaknesses above.

3 comments

We thank the reviewer for their detailed feedback. We acknowledge that the original presentation made the theoretical contributions unnecessarily difficult to assess. We have substantially revised the manuscript to clarify the motivation, notation, guarantees, and experimental evidence. For ease of review, all revisions made to the manuscript are highlighted in yellow, allowing the changes discussed below to be located more easily. We address the concerns in the order raised.

1. Parameter definitions: α\alpha and δ\delta-RSE

We agree that these parameters should be defined before their interpretation is needed.

Regarding α\alpha: Section 4 now explicitly introduces α≥0\alpha\geq 0 as the pessimism strength controlling departure from the nominal entropy-regularized follower response πF∗\pi_F^*, immediately before defining the pessimistic transformation πFα\pi_F^\alpha. The revised introduction (§1) also presents the theoretical contributions and their dependence on α\alpha more directly.

Regarding δ\delta-RSE: We have added its definition at the beginning of §5.3. Here, δ\delta measures the allowable loss in the follower's unregularized discounted return relative to its optimal response. Appendix C.2 provides the corresponding stochastic sequential derivation, following the robustness criterion of Gan et al. (2025). We now explicitly distinguish this reference from the entropy-regularized follower optimality criterion used in our RBC analysis. The exact δ\delta-RSE comparison is therefore a qualitative robust-equilibrium reference, not an optimizer of our proposed RBC objective.

2. Notation and unnecessary reading overhead

We agree that the original notation introduced unnecessary complexity and have revised the relevant definitions.

(1) Joint policies: Section 4.1 now denotes the nominal and modified joint action distributions by ff and gg, respectively, instead of vv and β\beta. This also makes their respective roles in off-policy estimation more transparent.

(2) Importance ratio: We have separated the action-level likelihood ratio w=πFα/πF∗w=\pi_F^\alpha/\pi_F^* from the joint-policy definitions in §4.1. The state-occupancy correction is explicitly identified as ζ(s)=dg(s)/df(s)\zeta(s)=d^g(s)/d^f(s). Their roles are explained in §4.2, with implementation details in Appendix B.3–B.4.

(3) Auxiliary notation: The revised Proposition 4.2 explicitly defines κα\kappa_\alpha as the KL radius induced by the pessimistic response, rather than leaving its interpretation implicit. We have also removed redundant shorthands from the main statement of Corollary 4.4, making the nominal-performance bound directly visible.

(4) Follower-policy notation: Proposition 4.2 and Corollary 4.4 now state their principal conclusions using πF∗\pi_F^* and πFα\pi_F^\alpha. Local auxiliary distributions remain useful in the proofs, particularly Appendix A.5–A.7, where the dependence on a fixed leader policy or state-action pair can be suppressed without ambiguity.

3. Presentation of the theoretical contributions

We agree that Proposition 4.2 and Corollary 4.4 previously mixed their main conclusions with intermediate definitions and comparisons. Both statements have been reorganized in §4.4, retaining the complete derivations in Appendix A.5–A.7.

To make the theoretical development easier to assess, the revised manuscript now follows a more explicit progression:

  • Construction (§4): A locally pessimistic follower response obtained by tilting the nominal entropy-regularized best response without modifying the follower's objective.
  • Hypergradient (§4.1, Theorem 4.1): Differentiation through both nominal follower adaptation and the pessimistic response transformation; proof in Appendix A.4.
  • Local robustness (§4.4, Proposition 4.2): Exact worst-case selection within an induced state-action-local KL neighborhood, evaluated using nominal leader action values.
  • Approximation guarantees (§4.4, Theorem 4.3 and Corollary 4.4): An O(α2)O(\alpha^2) follower optimality gap and an O(α)O(\alpha) nominal leader-performance bound, including after optimizing the leader against the modified response.

The introduction (§1) now summarizes these contributions explicitly, and §4.4 provides a short explanation of the guarantees before stating them.

We emphasize that the contribution is a differentiable, locally pessimistic response mechanism with explicit approximation guarantees, not a solver for the unrestricted global robust Stackelberg equilibrium problem.

AuthorsOpenReview ↗

Continuation of the previous comment

4. Vanishing indirect hypergradient in the original response-sensitive benchmark

We thank the reviewer for identifying this important limitation. The observation is correct for the original response-sensitive benchmark shown in Appendix C.1, Figure 4(b). At both AA and BB, every follower action returns to state SS. The action-dependent continuation sensitivities therefore cancel when differentiating the follower-response distributions, and the indirect contributions in Theorem 4.1 vanish in expectation.

That benchmark was designed to test robustness to unfavorable follower deviations, not to demonstrate the necessity of differentiating through the pessimistic response. We now make this distinction explicit in §5.5.

To address the concern directly, we introduced a separate three-state response-coupled benchmark, in which follower action 1 switches between states AA and BB, while actions 0 and 2 return to SS. Consequently, follower responses affect subsequent state transitions, allowing nonzero indirect hypergradient contributions.

The new environment and transition structure are documented in Appendix C.5, Figure 10. We compare full RBC-HG against two ablations that retain the same pessimistic response but omit either the additional pessimistic derivative or all indirect derivatives.

Across 10 matched seeds at α=0.5\alpha=0.5, full RBC-HG achieves a mean improvement in 150-step undiscounted leader return under the pessimistic follower of:

  • 3.73 over the variant without the pessimistic derivative (95% bootstrap CI: [3.49, 3.91]).
  • 2.95 over the variant without any indirect derivatives (95% CI: [2.83, 3.04]).

Both improvements are positive across all 10 seeds. The gradient diagnostics in Appendix C.5, Figure 11 additionally demonstrate that the pessimistic indirect contribution is nonzero during training and decreases as optimization progresses.

These results are summarized in the new §5.5, with an additional numerical optimization comparison in Appendix C.6. They provide separate evidence for the usefulness of the additional derivative, while preserving the original benchmarks for their intended robustness evaluation.

We hope these revisions make both the theoretical contribution and the distinct purposes of the experiments clearer and easier to assess.

Thank you for your response, but I still think the paper requires a careful revision. I will maintain my original score.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of the rebuttal and remaining concerns

We thank all the reviewers for their feedback and would like to briefly summarize the progress made during the discussion. Our revised manuscript, with changes highlighted in yellow, addresses the principal technical questions raised during review:

  • Theoretical contribution: We clarified the locally pessimistic response formulation, its exact local KL-robustness interpretation, the follower-optimality and leader-performance guarantees, and the associated hypergradient.
  • Experimental validation: We introduced a response-coupled benchmark demonstrating the necessity of indirect and pessimistic derivatives, alongside numerical-oracle comparisons and critic-sensitivity analyses. Reviewer 3 acknowledged that these experiments address three of their four questions.
  • Robustness: We clarified that our evaluations include multiple types of follower deviations distinct from the training response. Reviewer 2 acknowledged the mathematical rationale and experimental clarification

The remaining concerns primarily relate to scalability and applicability beyond controlled tabular environments. We recognize these limitations and have explicitly narrowed our claims accordingly.

We wish to emphasize that our primary contribution is a theoretically characterized robustness formulation for bilevel reinforcement learning, rather than a new scalable hypergradient estimation technique. While larger-scale implementation is an important and foreseen direction for future work, we believe the theoretical results and additional empirical validation constitute meaningful contributions independently of this remaining limitation.

We hope that these contributions, the substantial revisions and the reviewers' acknowledgments of resolved concerns would be taken into account in the final assessment.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.