acceptodds
Under review as a conference paper at ICLR 2027

Leader-Follower Games with Policy Recommendation: Reinforcement Learning under a Preservation Constraint

Abstract

In leader-follower games (LFGs), policy recommendations provide a self-interested leader with an additional channel, beyond her committed policy, to influence a boundedly rational follower who has a recommendation-sensitive response. This steering effect can lead to outcomes that depart from the quantal Stackelberg equilibrium benchmark and potentially harm the follower's interests. To protect the follower's interests, an external preservation constraint is designed to restrict the leader's recommendations so that the follower's executed response is close to her quantal response, without requiring disclosure of her reward function. We study online reinforcement learning (RL) under this constraint in episodic LFGs with linear function approximation, where the leader cannot observe the follower's reward function or her realized rewards. We propose an algorithm that efficiently infers the follower's response from observed actions and operates under an estimated preservation constraint. We establish complementary positive and negative results on follower protection with and without the preservation constraint. Under the preservation constraint, we establish simultaneous high-probability bounds on absolute leader regret and three measures of the recommendation's effect on the follower. For horizon length , feature dimension , and episodes, the bounds are for absolute leader regret, for cumulative preservation violation and reward gap, and for cumulative criterion gap, with other problem parameters fixed. Without the preservation constraint, we construct an explicit game in which any leader achieving sublinear regret against the unconstrained benchmark induces linear cumulative follower losses. Together, these results demonstrate the importance of the preservation constraint in protecting the follower from long-term losses caused by the leader's recommendations.

Primary area:
reinforcement learning
Keywords:
reinforcement learning, Stackelberg game, leader-follower game, policy recommendation, quantal response

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_gvC7OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper studies episodic leader-follower games where a recommendation influences a myopic follower through a KL refernce response model. It proposes a preservation constraints that make the executed response equal the recommendation-free quantal response, retaining the original QSE benchmark.

With hidden followers reward, PROM combines action based max likelihood estimation with optimistic value iteration. Under the specificed linear model and an exact planning oracle, it establishes simutaneous bound on absolte leader regret, behavioral deviation, reward gap, and entropy regularized criterion gap. A complementary construction shows that approaching the unconstrained or relaxed leader benchmark can force linear follower losses. Experiments evaluate three constraint tolerances on one synthetic instance.

Critical strengths
  • Joint learning and protection guarantees. The paper controls both absolute leader regret and recommendation-induced deviations while observing follower actions but no follower rewards. Absolute regret addresses episodes where steering raises the leader's value above the constrained benchmark.
  • A useful response identity. Under estimated preservation, νθ∗π,μθ^π=μ(1−κ)θ∗+κθ^π\nu_{\theta^{\ast}}^{\pi,\mu_{\hat{\theta}}^{\pi}} = \mu_{(1-\kappa)\theta^{\ast}+\kappa\hat{\theta}}^{\pi}. This connects executed-response observations to the preservation target and supports first-order TV bounds and sharper squared-error KL bounds
Critical weaknesses
  • The methodological advance over prior work appears incremental. Chen et al. already combine action-based MLE, fixed-estimate planning, covariance-based response bonuses and optimistic value iteration, with the same leading standard-regret order O~(H2d3/2T)\widetilde{O}(H^2 d^{3/2}\sqrt{T}). The recommendation-conditioned analysis and joint protection results are legitimate additions, but their technical significance beyond extending that framework needs stronger justification. The paper acknowledges the connection; the concern is the magnitude for the advance.
  • The numerical evidence is insufficiently reproducible. Section T uses one synthetic instance and five seeds, but does not specify the feature/reward/transition construction, planning grid, solver settings, or numerical confidence-bonus choices. Evaluation uses a grid-restricted benchmark, while the theorem assumes global continuous optimization, Consequently, the experiments provide limited evidence about the proposed algorithm's practical implementation.
Questions for authors
  • Which central technical step requires a substantially new argument beyond extending Chen et al.'s fixed-MLE scheme with recommendation-dependent likelihood offsets and the effective temperature? Please identify the precise contribution that supports the methodological significance claim.
  • Please specify the synthetic instance construction, planning grid, solver tolerances, and bonus settings. How does planning approximation affect the reported benchmark and regret measurements?
4 comments
AuthorsOpenReview ↗

Response to Reviewer gvC7, Part 1

Thank you for recognizing the strengths. We address your questions and concerns below.

W1/Q1: Technical advance and theoretical significance

W1: We disagree that the methodological advance is incremental. The advance is a joint analysis of learning and preservation when the observed response differs from the response defining the benchmark. We build on Chen et al.'s MLE and OVI framework. Thm.6.1 retains its leading O~(H2d3/2T)\tilde O(H^2d^{3/2}\sqrt T) order for absolute leader regret, simultaneously with O~(HdT)\tilde O(Hd\sqrt T) CPV/RG and O(Hd2log⁡2(eTH/δ))O(Hd^2\log^2(eTH/\delta)) CG. In Eq.37, Bellman estimation still contributes the leading leader term; the additional response analysis establishes these stronger joint guarantees. The shared leading order does not measure the additional proof obligations or the protection achieved.

Q1: The central additional argument transfers ER-likelihood confidence to true-QR optimism and second-order preservation, within the adaptive Bellman recursion.

1. Confidence transfer from ER observations to QR targets

Recommendation offsets and the effective temperature specify the correct ER likelihood for the announced commitment–recommendation pair (π,y)(\pi,y). The further task is to certify the different QR target and control the loss between the two responses. The ER likelihood's Fisher geometry is ε2Covν(ϕp)\varepsilon^2\mathrm{Cov}_\nu(\phi^p), whereas the QSE and preservation targets use true QR and η\eta.

At a fixed step, state and commitment pp, put qz=μθ^+z(θ∗−θ^)pq_z=\mu^p_{\hat\theta+z(\theta^*-\hat\theta)}. Prop. M.4 identifies y=q0y=q_0, ν=q1−κ\nu=q_{1-\kappa} and μ=q1\mu=q_1, with κ=ητ/(1+ητ)\kappa=\eta\tau/(1+\eta\tau). These are the fitted recommendation, actual execution and true-QR target, which can differ during learning. The fitted ER=QR identity (Eq.17) supplies a common covariance geometry. Lemmas M.2–M.3 transfer adaptive likelihood confidence to the historical fitted-covariance norm, and Prop. M.4 controls three comparisons through the same width: fitted-to-actual ER for execution, fitted-to-true QR for optimism, and actual ER-to-true QR for preservation. Centering removes response-invariant reward shifts: covariance seminorms measure follower-action reward contrasts, so these bounds control response errors without requiring unique recovery of the hidden reward parameter (App. G).

2. Second-order control yields polylogarithmic criterion loss

The decisive quantitative step is direct second-order control of the final response segment. With confidence radius RΘR_\Theta and ρ2=Tr(V−1Covy(ϕp))\rho^2=\mathrm{Tr}(V^{-1}\mathrm{Cov}_y(\phi^p)), where VV is Eq.16's historical fitted-covariance matrix, Lemma L.6 gives η−1KL(ν∥μ)≤ηκ2e2ηRΘ2ρ2/2\eta^{-1}\mathrm{KL}(\nu\Vert\mu)\le\eta\kappa^2e^{2\eta}R_\Theta^2\rho^2/2. This retains both squared estimation error and the squared influence factor κ2\kappa^2. Lemma P.5 uses a multiplicative supermartingale to transfer the log-determinant bound to conditional expected squared widths under actual ER occupancies, without a T\sqrt T concentration remainder. Together with RΘ2=O~(d)R_\Theta^2=\tilde O(d) and Eq.7, this yields the polylogarithmic loss in the follower's original criterion. Specifying offsets or a first-order response bound does not establish this conclusion.

3. Uniform Bellman confidence for adaptive value learning

The transfer also has to close inside value learning: the fitted-QR bonus changes the data-dependent continuation functions. Prop. N.1 and Lemma O.1 establish the deterministic value-class cover and uniform Bellman confidence for this recursion. These steps connect action-based response estimation, protection and optimistic planning in one analysis.

4. Direct preservation certificates control absolute regret

Preservation also controls the extra direction required by absolute regret. Under ordinary QSE execution, episode suboptimality is nonnegative; estimated preservation can produce returns above the unchanged QSE optimum. Lemma Q.1 and Thm. E.7 control these gains by remaining-horizon-weighted preservation deviations, giving Regabs(T)≤Reg(T)+2CPVw(T)\mathrm{Reg}_{\mathrm{abs}}(T)\le\mathrm{Reg}(T)+2\mathrm{CPV}_{\mathrm w}(T). The weights account for subsequent state-occupancy changes. To control this bridge statistically, Prop. M.4 (Eq.97) bounds ER-to-QR deviation directly by bPb^P. Thm. E.7 thereby avoids the extra action-cardinality-dependent factor LdevL_{\mathrm{dev}} in Thm. E.9's generic inverse-map conversion from RRD. This supplies the two-sided leader guarantee without cancellation between positive and negative suboptimalities, together with follower protection at the stated leading order.

5. Structural characterization, learning and incentive conflict

The results also answer a substantive design question: can recommendations preserve QSE while the leader learns from actions without observing follower rewards? Thm.3.3 converts the outcome constraint ν=μ\nu=\mu into the unique rule y=μθpy=\mu_\theta^p, removing an independent recommendation search for each commitment.

AuthorsOpenReview ↗

Response to Reviewer gvC7, Part 2

With θ∗\theta^* hidden, Eq.12 implements this rule at θ^\hat\theta, so control of true preservation error remains a learning task. Eq.8 retains the QSE optimum and full commitment class; Thm.6.1 establishes joint online learning and protection.

Conversely, on Thm.6.2's full-information game, for each fixed α∈(0,1]\alpha\in(0,1], every feasible sequence with sublinear regret against the corresponding relaxed benchmark incurs linear signed follower reward loss, CPV, RG and CG. This includes the unconstrained case. The structural conflict persists without estimation error, while the positive result shows how to control recommendation-induced losses during learning. We will foreground this design–learning connection and the proof chain above in §6.1/App. I.4.

W2/Q2: Reproducibility and planning approximation

W2: The controlled experiment tests the learning–preservation mechanism. Figs.2–3 vary the preservation tolerance α\alpha in a fixed game. This isolates how greater leader returns can accompany larger absolute regret and follower deviations, illustrating Thm. E.19's first-/second-order tolerance terms.

The grid implementation retains conditional MLE, optimistic backward planning and the recommendation constraint. Finite candidate enumeration makes planning computable. Using the same commitment grid for the learner and true-QR reference gives comparisons across tolerances a common decision class and benchmark; the benchmark-comparison bounds below relate reported regret to the continuous QSE benchmark.

Q2: Further implementation details.

Instance. With zero-based n=ABs+Ba+bn=ABs+Ba+b, use step-independent ϕ=en\phi=e_n for n<dn<d, else (ej+ek)/2(e_j+e_k)/2, where j=(n−d) mod dj=(n-d)\bmod d, k=(j+1+(j mod 4)) mod dk=(j+1+(j\bmod4))\bmod d. Rewards are uh=ϕ⊤ϑhu_h=\phi^\top\vartheta_h, rh=ϕ⊤θhr_h=\phi^\top\theta_h, with fixed i.i.d. coefficient draws from Unif[.1,.9]\mathrm{Unif}[.1,.9]. Transitions use pj=.9zj+.118/8p_j=.9z_j+.1\mathbf1_8/8, zj∼Dir(18)z_j\sim\mathrm{Dir}(\mathbf1_8) and Ph(s′∣s,a,b)=∑jϕj(s,a,b)pj(s′)P_h(s'|s,a,b)=\sum_j\phi_j(s,a,b)p_j(s') for all hh.

Grid. Backward planning enumerates 260 commitments per state/step: simplex vertices and edge mixtures at spacing .1; the true-QR benchmark uses the same grid. At α=0\alpha=0, yy is fitted QR (Eq.12). For α>0\alpha>0, retain fitted QR plus the furthest TV-feasible point (Eq.70) toward each vb=(1−8ϵy)eb+ϵy18v_b=(1-8\epsilon_y)e_b+\epsilon_y\mathbf1_8, ϵy=10−4\epsilon_y=10^{-4}, and its midpoint with fitted QR (17 candidates per commitment).

Solver. L-BFGS-B minimizes stepwise summed conditional NLL over [0,1]d[0,1]^d with analytic gradients and warm starts (initially .51d.5\mathbf1_d). Use ftol/gtol=10−10/10−610^{-10}/10^{-6} and maxiter/maxls=100/30. On reported failure or a convex first-order NLL-gap certificate >10−3>10^{-3}, retry with 10−12/10−710^{-12}/10^{-7} and 300/100, respectively.

Bonuses. Initialize Λh=Vh=Id\Lambda_h=V_h=I_d (λ=1\lambda=1). With Mh=H−h+1M_h=H-h+1, use fixed empirical multipliers in bU=.25Mhϕ⊤Λh−1ϕb_U=.25M_h\sqrt{\phi^\top\Lambda_h^{-1}\phi} and bQ=min⁡1,.10ηρb_Q=\min\\{1,.10\eta\rho\\}, where ρ2=Tr(Vh−1Covν^(ϕp))\rho^2=\mathrm{Tr}(V_h^{-1}\mathrm{Cov}_{\hat\nu}(\phi^p)) for the candidate pair's fitted ER ν^\hat\nu. Add MhbQM_hb_Q in planning; clip optimistic leader estimates and continuations to [0,Mh][0,M_h].

We will add the above details to §T.1.

Q2: Benchmark approximation and measured regret. Eq.153 uses the grid QSE reference Jg∗J_g^*. Put Δg=J∗−Jg∗≥0\Delta_g=J^*-J_g^*\ge0. For the same executed sequence, Regfull(T)=Regg(T)+TΔg\mathrm{Reg}_{\mathrm{full}}(T)=\mathrm{Reg}_g(T)+T\Delta_g, ∣Regabs,full(T)−Regabs,g(T)∣≤TΔg|\mathrm{Reg}_{\mathrm{abs,full}}(T)-\mathrm{Reg}_{\mathrm{abs},g}(T)|\le T\Delta_g. With certified reference error ϵref\epsilon_{\mathrm{ref}} and episode-value errors ϵt\epsilon_t, the absolute-regret comparison bound becomes T(Δg+ϵref)+∑tϵtT(\Delta_g+\epsilon_{\mathrm{ref}})+\sum_t\epsilon_t. Our additional grid analysis gives Δg≤(1/2+η/4)∑h=1H(H−h+1)rh\Delta_g\le(1/2+\eta/4)\sum_{h=1}^H(H-h+1)r_h for fixed statewise grids with uniform ℓ1\ell_1 covering radii rhr_h. Thus covering radii control the continuous-benchmark correction.

Q2: Additional planning-approximation analysis. Our extension retains Eq.37's statistical bound against Jg∗J_g^* under α=0\alpha=0, exact planning on fixed rectangular statewise grids, matching regression continuations and the paper's estimation/confidence calibration. For a predictable approximate planner with certified global objective gaps bounded uniformly in state by predetermined at,h≥0a_{t,h}\ge0, recalibrating Bellman radii and backward covers for Eq.21's clipped selected-objective continuations gives Regabs(T)≤BT(β~,RΘ)+∑t,hat,h\mathrm{Reg}_{\mathrm{abs}}(T)\le B_T(\tilde\beta,R_\Theta)+\sum_{t,h}a_{t,h}. Here BTB_T is Eq.37 with recalibrated radii; at,h≤c/Ta_{t,h}\le c/\sqrt T with cc independent of T,H,dT,H,d preserves Thm.6.1's leading rate under its fixed-parameter conditions. Follower protection requires no exact planner: arbitrary predictable commitments suffice under the stated estimation/recommendation conditions. We will add these results and proofs.

We hope our responses resolve your questions and concerns. If any of them remain, please let us know.

Thanks for the detailed response.

W1/Q1: The breakdown helps locate the contribution. I agree the interpolation identity (98) and the direct certificate bPb^P for absolute regret are the new pieces. Most of the remaining steps, though, are adaptations of known tools: the hellinger-to-covariance transfer, the self-normailized bellman confidence with a value-class cover, the polylog CG bound follows fairly directly. So my view is unchanged: the results are correct and useful, but the technical step beyond Chen et al. is modest.

W2/Q2: This resolves my reproducibility concern. The instance construction, planning grid, solver settings, and bonus choices are what I was asking for, and I'd like them in section T.1. One remaining point: the experiment use fixed empirical bonus multipliers (bU=0.25Mhϕ⊤Λh−1ϕb_U =0.25 M_h\sqrt{\phi^\top\Lambda_h^{-1}\phi}, bQ=min⁡{1,0.10 ηρ}b_Q=\min\{1, 0.10\, \eta\rho\}) rather than the theoretical radii in (30) and (33), and they plan over a finite grid. So the experiments illustrate the preservation mechanism rather than validate Theorem 6.1. Please say this explicitly. The new grid-gap bound and the planning-approximation result should go in the appendix with full proofs. I am keeping my score. Reproducibility is addressed but my concern about the size of the methodological contribution remains.

AuthorsOpenReview ↗

Response to Reviewer gvC7, Part 3

Thank you for recognizing the results as correct and useful, identifying the interpolation identity and direct bPb^P certificate as novel, and confirming that reproducibility is addressed. We have uploaded a revised PDF with a new full-class, all-budget minimax theorem (§6.3, p.10), sharper protection bounds, and their proofs in App. R. The new result characterizes the unavoidable statistical cost of preservation and attains it with an estimator compatible with leader learning.

Technical advance and significance

1. Optimal protection cost. Theorem 6.4 establishes, for the full linear-game class of §R.7 (d≥2d\ge2, known η,τ>0\eta,\tau>0),

inf⁡Asup⁡M∈MH,dEMA[CG(T)]≍Hητ2Φd(ε2T),ε=η/(1+ητ),\inf_{\mathsf A}\sup_{M\in\mathfrak M_{H,d}}\mathbb E_M^{\mathsf A}[\mathrm{CG}(T)]\asymp H\eta\tau^2\Phi_d(\varepsilon^2T),\qquad\varepsilon=\eta/(1+\eta\tau),

where Φd(x)=x\Phi_d(x)=x for x≤1x\le1, x\sqrt x for 1<x≤d21<x\le d^2, and d[1+log⁡(x/d2)]d[1+\log(x/d^2)] for x>d2x>d^2. Constants are uniform in H,d,T,τH,d,T,\tau for fixed η\eta, action counts and class bounds. The lower bound covers every adaptive recommendation learner under our protocol. Its hard games have zero leader rewards, so all commitments have optimal leader return and leader regret is identically zero. Preservation therefore has an intrinsic learning cost even when QSE policy learning is trivial. A regularized conditional MLE attains the upper bound on general action-dependent linear games.

2. What yields the advance. On the interpolation path you identified, the fitted QR yy is the fitted ER, the true ER ν\nu generates observations, and the true QR μ\mu defines protection. Lemma R.9 proves

e−2ηητ2KL(ν∥y)≤η−1KL(ν∥μ)≤e2ηητ2KL(ν∥y).e^{-2\eta}\eta\tau^2\mathrm{KL}(\nu\|y)\le\eta^{-1}\mathrm{KL}(\nu\|\mu)\le e^{2\eta}\eta\tau^2\mathrm{KL}(\nu\|y).

Thus predictive log loss from observed ER actions controls loss in the follower's original reward-plus-entropy criterion relative to the unobserved QR. Its deterministic bound yields the small-budget regime. Lemma R.10 uses increasing regularization capped at dd for the other two regimes; the cap also keeps the unregularized NLL gap at O(d)O(d), making the same estimator admissible for PROM's response and Bellman-confidence analysis. In §R.7, an adaptive-information lower bound, optimized over hard-instance dimensions, matches all three regimes. These connections establish a new optimal statistical result, not merely an ER likelihood adjustment.

For fixed behavioral parameters, high-probability CG improves from O~(Hd2)\tilde O(Hd^2) to O~(Hd)\tilde O(Hd), and CPV/RG from O~(HdT)\tilde O(Hd\sqrt T) to O~(HdT)\tilde O(H\sqrt{dT}) (§R.6). Coupled to calibrated PROM, the same estimator retains high-probability absolute leader regret O~(H2d3/2T)\tilde O(H^2d^{3/2}\sqrt T), at the leading order of Chen et al.'s linear-myopic signed-regret bound. The direct bPb^P certificate you noted controls recommendation-induced gains above QSE, which signed regret alone does not exclude.

3. Why this contribution matters. Benchmark equivalence does not give learning equivalence: ER actions generate the data, while QR defines the benchmark and preservation target. The paper establishes the unique response-preserving recommendation without reducing the leader's commitment class (Theorem 3.3, Eq. (8)); joint leader learning and follower protection with hidden follower rewards (Theorem 6.1, §R.6); linear follower reward losses for every relaxed-benchmark sublinear-regret sequence in its full-information game (Theorem 6.2); and the optimal statistical cost of preservation (Theorem 6.4). Exact preservation retains both players' expected total rewards under each commitment; Theorem E.7 also controls complete trajectory distributions and follower total-return differences during learning. Together these results show why preservation can matter, what it guarantees, and what it costs to learn.

Experimental and planning clarifications

Appendix V.1 (formerly T.1) now gives the instance construction, grids, solver settings and empirical bonuses. It explicitly distinguishes the finite-grid experiments illustrating preservation from Theorem 6.1's theoretically calibrated, full-simplex guarantee (V.1, lines 4975–4978). Theorem S.5 and Proposition S.6 give the complete certified approximate-planning and grid-gap proofs, including Bellman-confidence recalibration and the TΔGT\Delta_{\mathcal G} continuous-benchmark correction (§V.2).

We hope the revised paper and the above response resolve your concerns. We would appreciate your reconsideration of the overall assessment in light of the revised paper's new minimax theorem and combined technical and theoretical contribution.

Reviewer_yTREOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper studies reinforcement learning in episodic linear leader–follower games where the leader can also influence a boundedly rational follower through policy recommendations. The authors introduce a preservation constraint that requires the recommendation to match the follower's quantal response. Since the follower's reward is unobserved, the online algorithm PROM implements an estimated version of this constraint, combining conditional MLE of the follower's response with optimistic LSVI. The authors establish sublinear absolute leader regret together with follower-side protection guarantees, including a polylogarithmic criterion-gap bound. They also construct a game in which any leader with sublinear regret against the unconstrained or relaxed benchmark incurs linear follower-side losses.

Critical strengths
  1. The preservation constraint provides an interesting formulation for regulating recommendation-induced behavior while retaining the original QSE benchmark. The structural characterization connecting recommendations, QR, and executed responses is clean and intuitive.

  2. The paper establishes simultaneous guarantees for absolute leader regret and follower-side protection. In particular, the interpolation identity provides a useful connection between response estimation errors and preservation violations, leading to a polylogarithmic criterion-gap bound. The main theoretical analysis appears technically sound.

Critical weaknesses
  1. Limited technical novelty. The learning framework closely follows Chen et al. (2023), combining conditional MLE with optimistic LSVI. As noted in Appendix H, the constrained full-information problem reduces to the standard QSE. While the ER-generated observations introduce additional challenges, the new analysis mainly relies on the interpolation identity (98) and covariance comparisons. The technical advances over existing QSE learning methods seem somewhat limited.

  2. Enforcement of the preservation constraint. The guarantees assume that the self-interested leader follows the prescribed MLE procedure. However, constraint (12) can be satisfied with any θ^∈Θ\widehat{\theta}\in\Theta, potentially allowing the leader to steer the follower through a biased estimate. Appendix H acknowledges that enforcement is not modeled, and the audit certificates mentioned in Appendix D.3 are not developed into a formal verification mechanism.

  3. Strong modeling assumptions. The follower is assumed to be myopic and to condition on the full recommendation distribution rather than a sampled recommendation. The main guarantees also require known behavioral parameters. Under behavioral misspecification, Section R only provides regret and RRD bounds, which contain terms linear in TT for a fixed calibration error, and no CPV or CG guarantees are given in this regime.

  4. Limited scope of follower protection. The three follower-side guarantees are closely related: RG is bounded by CPV, and the CPV rate also follows from the CG bound via Pinsker's inequality and Cauchy–Schwarz. These metrics do not directly measure follower welfare; Figure 3b even shows that relaxing the constraint can increase the follower's raw expected reward. The negative result is based on a simple 2×22\times2 game and does not establish matching lower bounds for the learning rates.

  5. Computational and empirical limitations. The algorithm assumes an exact global planning oracle for a generally nonconcave objective. The experiments use a grid-based planner on a single synthetic environment with five seeds, without comparisons to QSE learning methods or an unconstrained leader baseline.

Questions for authors
  1. What are the main technical challenges beyond Chen et al. (2023)? Could similar guarantees be obtained by extending their algorithm with recommendations set to the estimated QR?

  2. Can the preservation constraint be verified through an auditable likelihood-based condition, such as a bounded NLL gap? Would the guarantees still hold if the leader strategically selects an estimate satisfying this condition?

  3. The protection bounds (Corollaries E.3 and E.4) appear to hold for any predictable commitment sequence, so the exact planning oracle seems needed only for the regret bound. Can the authors confirm this and quantify the effect of approximate planning on regret?

4 comments
AuthorsOpenReview ↗

Response to Reviewer yTRE, Part 1

Thank you for your review. We address your questions and concerns below.

W1/Q1: Technical novelty beyond QSE learning

Our technical advance is joint learning and protection under ER feedback. For Q1, PROM uses estimated QR (Eq.12); this rule alone does not establish the joint guarantees.

The constraint preserves the QSE optimum and full commitment space (Thm. 3.3; Eq.8); benchmark equivalence does not give learning equivalence. During learning, the fitted recommendation, ER observation law ν\nu and true QR μ\mu can differ (Prop. M.4).

ER feedback requires adapting the QR likelihood. Historical recommendation offsets give Fisher geometry ε2Covν(ϕ)\varepsilon^2\mathrm{Cov}_\nu(\phi), where ε=η/(1+ητ)\varepsilon=\eta/(1+\eta\tau), while preservation and QSE targets use true QR and η\eta. Using the fitted ER=QR identity (Eq.17), Lemmas M.2–M.3 and Prop. M.4 transfer likelihood confidence through one fitted-covariance width to ER prediction, optimism for the QSE benchmark and preservation. Prop. N.1 controls the resulting data-dependent Bellman recursion through a deterministic value-function cover.

Second-order control yields polylogarithmic criterion loss: Lemma L.6 retains κ2\kappa^2, where κ=ητ/(1+ητ)\kappa=\eta\tau/(1+\eta\tau); Lemma P.5's predictable squared-width bound then gives O(Hd2log⁡2(eTH/δ))O(Hd^2\log^2(eTH/\delta)) cumulative criterion loss, beyond guarantees for leader regret.

Absolute regret must also control returns above the QSE benchmark, which estimated preservation can permit. The bridge in line 3692 controls this excess by remaining-horizon-weighted preservation deviations (CPVw\rm{CPV}_{\rm w}) for every admissible policy-pair sequence. PROM matches Chen et al's linear-myopic O~(H2d3/2T)\tilde O(H^2d^{3/2}\sqrt T) signed-regret upper-bound order, now for absolute regret, together with behavioral protection and polylogarithmic CG, with other parameters fixed.

We will highlight this chain in §6.1 (App. I.4).

W2/Q2: Auditing & strategic estimation

Protection can be certified even when estimates are selected strategically. Both answers to Q2 are YES under this audit.

Corollary E.5 already permits near-MLE estimates; we make this auditable. Before episode tt, the leader discloses θ~ht∈Θh\tilde\theta_h^t\in\Theta_h. The auditor checks Eq.12 with this estimate and verifies a conditional NLL gap at most a predeclared γ\gamma. Using authenticated historical states, commitments, recommendations and follower actions, convexity (Prop. M.1) gives ℓht(θ~)−min⁡θ∈Θhℓht(θ)≤Ght(θ~):=max⁡v∈Θh⟨∇ℓht(θ~),θ~−v⟩.\ell_h^t(\tilde\theta)-\min_{\theta\in\Theta_h}\ell_h^t(\theta)\le G_h^t(\tilde\theta):=\max_{v\in\Theta_h}\langle\nabla\ell_h^t(\tilde\theta),\tilde\theta-v\rangle. It suffices to certify that GhtG_h^t is at most γ\gamma by linear optimization over Θh\Theta_h. Neither follower rewards nor θh∗\theta_h^* are needed. Exact MLE passes with zero gap; App. D.3 bounds true preservation deviation with high probability.

Self-interested selection is covered. Lemma M.2 is uniform over parameters, covering any episode-start predictable certified near-MLE selector. With historical covariances frozen at their episode's disclosed estimates, arbitrary predictable commitments satisfy RG(T)≤CPV(T)=O~(HdT)\mathrm{RG}(T)\le\mathrm{CPV}(T)=\tilde O(Hd\sqrt T), CG(T)=O~(Hd2)\mathrm{CG}(T)=\tilde O(Hd^2) with high probability for γ=O~(d)\gamma=\tilde O(d) and fixed remaining parameters. Summing the bridge in lines 3733–3738 gives a O~(H2dT)\tilde O(H^2d\sqrt T) bound on cumulative positive gains above QSE, excluding linear gains. Using the same audited estimate in PROM's recalibrated optimistic planner retains its absolute-regret rate and QSE benchmark. We will add the audit and proof.

W3: Modeling & calibration

Myopia isolates recommendation effects from continuation-value incentives, retaining HH-step leader learning under unknown, action-dependent dynamics as in Zhong et al (2023) and Chen et al (2023).

The recommendation is a public policy, like the leader's commitment. Both (π,y)(\pi,y) are announced (§2.1); the follower uses the full yh(⋅∣s)y_h(\cdot\mid s) as the reference in −τKL(q∣yh)-\tau\mathrm{KL}(q\\|y_h) (Eq.3). A sampled action alone does not specify that reference (App. A).

Known η\eta is shared with the QSE baseline (Chen et al, Algorithm 3). The preservation rule itself is independent of τ>0\tau>0 (Thm. 3.3). Our added CPV/CG bounds quantify errors in both behavioral coefficients. For fitted (ηˉ,τˉ)(\bar\eta,\bar\tau) with calibration error bound ζ0\zeta_0, extending App. R gives, under Assump. R.1 and Dbeh=Cbehζ0≤1D_{\mathrm{beh}}=C_{\mathrm{beh}}\zeta_0\le1, RG(T)≤CPV(T)=O~(HdT+Hζ0Td),CG(T)=O~(Hd2+Hdζ02T)\mathrm{RG}(T)\le\mathrm{CPV}(T)=\tilde O(Hd\sqrt T+H\zeta_0T\sqrt d),\quad\mathrm{CG}(T)=\tilde O(Hd^2+Hd\zeta_0^2T) with high probability and other parameters fixed. As in misspecified linear RL (Jin et al 2020), linear terms reflect persistent model bias: first order in ζ0\zeta_0 for CPV and second order for CG. With ζ0=O(T−1/2)\zeta_0=O(T^{-1/2}), the leading protection rates persist. We will add the proof to App. R and clarify Eq.3's protocol.

AuthorsOpenReview ↗

Response to Reviewer yTRE, Part 2

W4: Follower welfare & learning costs

CG directly measures loss in the follower's original criterion (Eq.1), expressed as KL divergence in Eq.7. CPV measures behavioral deviation and RG absolute gaps in expected raw reward at the same contexts; Pinsker and Cauchy–Schwarz transfer CG's bound to both. Fig. 3b's same-context raw-reward increase can coexist with criterion loss (Remark 3.7; App. T.4), so it does not contradict this protection.

New results protect trajectories and episodic rewards. Under the same commitment, initial law and transition kernel, conditional ER/QR trajectory laws satisfy KL(Ktν∣Ktμ)=ηCt,∑tCt=CG(T),\mathrm{KL}(K_t^\nu\\|K_t^\mu)=\eta C_t,\quad\sum_t C_t=\mathrm{CG}(T), where CtC_t is the expected episode criterion gap. Pinsker controls event-probability differences despite different state occupancies; with weighted performance difference, it bounds expected episodic raw-return deviations: \sum_t|J_{\mathrm F}(\pi^t,\nu^t)-J_{\mathrm F}(\pi^t,\mu^t)|\le\min\left\\{\mathrm{CPV}_{\mathrm w}(T),H\sqrt{\eta T\mathrm{CG}(T)/2}\right\\}=\tilde O(H^{3/2}d\sqrt T) with high probability for PROM under Thm.6.1's fixed-parameter conditions.

Thm.6.2's full-information 2×22\times2 construction shows that relaxed optimization can harm the follower even without learning error: for each fixed admissible positive relaxation, every feasible sequence with sublinear relaxed-benchmark regret incurs linear CPV, RG, CG and signed follower raw-reward loss.

New matching bounds quantify preservation's statistical cost. On a fixed linear-game family (KK states, HH steps, d=K+1d=K+1, zero leader rewards), we prove expected minimax orders for arbitrary adaptive recommendation learners using the paper's feedback: Θ(HKT)\Theta(H\sqrt{KT}) for CPV/RG and Θ(HKlog⁡T)\Theta(HK\log T) for CG, for fixed positive parameters and large TT. Conditional MLE attains these orders with zero leader regret. We will add these results and proofs.

W5/Q3: Protection & approximate planning

The oracle isolates the statistical cost of joint learning and preservation. This scope has a direct precedent in linear leader–follower RL: Zhong et al (2023, Assump. 4) likewise analyze sample complexity using an equilibrium oracle. Q3: protection requires no exact planner.

The proofs of Corollaries E.3–E.4 don't use Eq.20 maximization; W2 covers any episode-start predictable commitment sequence under its audit and covariance conditions.

Let a predictable planner certify global gaps for the recalibrated optimistic objective, with deterministic budgets at,h≥0a_{t,h}\ge0 fixed before interaction: 0≤sup⁡p∈Δ(A)Q~ht(s,p)−Q~ht(s,p~ht(s))≤at,h0\le\sup_{p\in\Delta(\mathcal A)}\tilde Q_h^t(s,p)-\tilde Q_h^t(s,\tilde p_h^t(s))\le a_{t,h} for every ss. Use the selected objective clipped per Eq.21 as the regression continuation. With aˉh=max⁡tat,h\bar a_h=\max_ta_{t,h} and aˉH+1=0\bar a_{H+1}=0, enlarge Eq.33's Bellman radius by (T+T+1BP)aˉh+1(\sqrt T+\sqrt{T+1}B_P)\bar a_{h+1} and recompute Eq.34's backward covers. With high probability, Regabs(T)≤BT(β~,RΘ,γ)+∑t,hat,h,\mathrm{Reg}_{\mathrm{abs}}(T)\le B_T(\tilde\beta,R_{\Theta,\gamma})+\sum_{t,h}a_{t,h}, where BTB_T is Eq.37's statistical bound with recalibrated Bellman/near-MLE radii. For aˉh≤c/T\bar a_h\le c/\sqrt T, with cc independent of T,H,dT,H,d, and γ=O~(d)\gamma=\tilde O(d), the leading O~(H2d3/2T)\tilde O(H^2d^{3/2}\sqrt T) absolute-regret rate persists under Thm.6.1's fixed-parameter conditions.

With matching continuations and bonuses, exact planning on predetermined rectangular statewise grids retains the statistical bound against their QSE benchmark JG∗J_{\mathcal G}^*. For the same execution, switching to J∗J^* changes absolute regret by at most TΔGT\Delta_{\mathcal G}, where ΔG=J∗−JG∗≥0\Delta_{\mathcal G}=J^*-J_{\mathcal G}^*\ge0. We will add the results and proofs.

W5: Experimental comparisons

The experiments isolate the effect of modest preservation relaxations. Fig. 2–3 compare α=0,0.1,0.15\alpha=0,0.1,0.15 on the same game over 5 seeds per condition, illustrating Thm. E.19's ∑tαt\sum_t\alpha_t (CPV/RG) and ∑tαt2\sum_t\alpha_t^2 (CG) terms.

We tested α=1\alpha=1 (unconstrained, same τ>0\tau>0 and ER feedback): absolute regret was substantially larger, with a steeper linear trend than at α=0.15\alpha=0.15. We omitted this curve because α=0.15\alpha=0.15 already shows near-linear growth, while α=1\alpha=1 on the same scale would compress the three plotted curves and obscure their differences.

We will compare Chen et al's linear-myopic method with PROM at τ=0\tau=0, matching QR feedback, games, commitment classes, grids and evaluation, and report seed variability and reproducibility settings.

Postsubmission diagnostics broaden the empirical check to a second H=S=A=B=3H=S=A=B=3 instance, examining recommendation sensitivity, relaxation and a τ=0\tau=0 QR control. QSE grid refinement (0.1 to 0.025) changes its value by about 5.7×10−65.7\times10^{-6}, supporting stability under this refinement. We will include these diagnostics.

We hope our responses resolve your concerns. If any remain, please let us know. Thank you.

Reviewer_yTREOpenReview ↗

Response to the Rebuttal

Thank you for the detailed response. The rebuttal addresses several of my specific concerns, particularly regarding auditability of the preservation constraint, follower-side guarantees, and the role of approximate planning. However, my main concern remains the overall technical novelty relative to Chen et al. (2023): the learning framework still closely follows conditional MLE with optimistic LSVI, while the new contributions are primarily analytical extensions needed to handle ER feedback and preservation. I also remain concerned about the restrictive behavioral assumptions and the reliance on a global planning oracle. Overall, the rebuttal improves the paper substantially, but does not change my overall assessment, so I will keep my score unchanged.

AuthorsOpenReview ↗

Response to Reviewer yTRE, Part 3

Thank you for recognizing progress on auditability, follower protection and approximate planning. We have uploaded a revised PDF adding a full-class, all-budget minimax protection theorem (§6.3, p.10), sharper guarantees and proofs in Apps. R–V. The new estimator attains optimal expected protection while retaining leader learning.

Technical contribution and theoretical significance

Benchmark equivalence does not give learning equivalence. Chen et al. (2023) observe actions from the QR to the commitment; here actions follow the recommendation-dependent ER, while the true QR defines the benchmark and protection target. Signed leader regret alone does not establish follower protection or its learning cost.

Beyond the fixed-family bounds in our first response, Theorem 6.4 gives the minimax expected cumulative criterion gap (CG) for §R.7's full nondegenerate linear-game class (d≥2d\ge2, known η,τ>0\eta,\tau>0) at every budget:

inf⁡Asup⁡M∈MH,dEMA[CG(T)]≍Hητ2Φd(ε2T).\inf_{\mathsf A}\sup_{M\in\mathfrak M_{H,d}}\mathbb E_M^{\mathsf A}[\mathrm{CG}(T)]\asymp H\eta\tau^2\Phi_d(\varepsilon^2T).

Here ε=η/(1+ητ)\varepsilon=\eta/(1+\eta\tau) and Φd(x)=x\Phi_d(x)=x, x\sqrt x, or d[1+log⁡(x/d2)]d[1+\log(x/d^2)] for x≤1x\le1, 1<x≤d21<x\le d^2, or x>d2x>d^2, respectively. Constants are uniform in H,d,T,τH,d,T,\tau for fixed η\eta, action counts and class bounds. The lower bound covers all adaptive learners even when leader rewards and regret are zero: protection has its own learning cost. Corollary R.13 also identifies a sharp threshold: for fixed η\eta and bounds, normalized minimax CG vanishes iff known τT=o(T)\tau_T=o(\sqrt T), even as dT≥2d_T\ge2 grows.

New analysis and joint guarantees. Lemma R.9 relates predictive loss from ER actions to loss in the follower's original reward-plus-entropy criterion relative to the unobserved QR. Lemma R.10 controls sequential log loss with capped increasing regularization; §R.7 supplies the matching adaptive-information lower bound. The cap keeps the unregularized NLL gap at O(d)O(d), so the same estimator in calibrated PROM retains high-probability absolute leader regret O~(H2d3/2T)\tilde O(H^2d^{3/2}\sqrt T), at the leading order of Chen et al.'s signed-regret bound. High-probability CG improves from O~(Hd2)\tilde O(Hd^2) to O~(Hd)\tilde O(Hd) (§R.6).

Independent theoretical contribution. Theorem 3.3 and Eq. (8) retain the full QSE commitment class and benchmark; Theorem 6.1 and §R.6 jointly bound leader regret and follower losses; Theorem 6.2 shows that relaxed-benchmark sublinear regret forces linear follower reward loss in its full-information game; and Theorem 6.4 determines protection's optimal learning cost. Theorem E.7 extends CG protection to complete trajectories and follower total return under the same commitment. Separately, Theorem S.3 protects against strategic selection among likelihood-certified estimates. Together they show why protection can be needed, how it can be learned, and what it costs without follower-reward feedback.

Planning

The minimax protection theorem does not require a global leader-planning oracle. Proposition R.11 covers arbitrary episode-start predictable Markov commitments using regularized-MLE recommendations; the matching lower bound has trivial leader optimization. PROM's oracle optimizes an estimated leader objective: it neither reveals hidden follower rewards nor removes the response-learning problem. Zhong et al. (2023, Assump. 4) likewise use an equilibrium oracle to isolate statistical learning in Stackelberg RL.

For leader regret, Theorem S.5 quantifies predeclared, state-uniform certified global planning gaps: budgets c/Tc/\sqrt T (fixed cc), with γ=O~(d)\gamma=\tilde O(d), retain the absolute-regret order after Bellman-radius and cover recalibration. Proposition S.6 gives the calibrated fixed-grid guarantee and bounds transfer to the continuous benchmark by TΔGT\Delta_{\mathcal G}, where ΔG=J∗−JG∗\Delta_{\mathcal G}=J^*-J_{\mathcal G}^*. Thus the optimal protection rate is oracle-independent, while planning error in leader learning is quantified.

Behavioral scope and calibration

Myopia isolates recommendation effects on current choices from follower continuation incentives, while the leader still learns an HH-step problem with unknown, action-dependent dynamics. The recommendation is a public distribution, not a sampled action: in the inventory example (§I.6), it supplies the retailer's KL reference (Eq. (3)). Chen et al.'s Algorithm 3 also requires known η\eta; the exact preserving rule is independent of τ>0\tau>0. With a supplied error bound ζ0\zeta_0 for both coefficients, Theorem T.5 adds high-probability terms O~(Hζ0Td)\tilde O(H\zeta_0T\sqrt d) for CPV and O~(Hdζ02T)\tilde O(Hd\zeta_0^2T) for CG under its stated local calibration condition.

We hope the revised paper and the above response resolve your concerns. We would appreciate your reconsideration in light of the new minimax result and the revised paper's combined contributions.

Reviewer_5CYDOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
1
Soundness
3
Read review
Paper summary

The paper studies Leader-Follower Games, and proposes an external preservation constraint designed to restrict the leader’s recommendations, to protect the follower’s interests. Their constraint enforces equality between the recommendation and the QR. Under their preservation constraint, their algorithm achieves sublinear leader regret and sublinear preservation violation and reward gap for the follower. They also construct a negative example in which every leader that achieves sublinear regret leads to linear cumulative follower losses.

Critical strengths
  1. The mathematical results under the preservation constraint (the leader regret bound and the preservation violation bound for the follower) are interesting and non-trivial.
  2. The negative result leading to linear cumulative follower losses is useful.
Critical weaknesses
  1. The implication of the preservation constraint seem to be misaligned, since a recommendation that increases a follower's expected reward can still be considered harmful according to the constraint.
  2. The paper is very dense and hard to follow. I suggest that the authors make the writing clearer and add more intuition.
Questions for authors

As noted in the Weaknesses section above, can the authors discuss the implications of the preservation constraint and how it relates to protection of the follower's interests?

2 comments
AuthorsOpenReview ↗

Response to Reviewer 5CYD, Part 1

Dear Reviewer 5CYD,

Thank you for your review and for recognizing the nontrivial learning and protection guarantees and the value of the negative result. We clarify the protection objective and how we will make it easier to follow.

W1/Q1: Preservation and the follower's interests

W1: A raw-reward increase does not invalidate the protection objective. Remark 3.7 and Fig.3(b) already allow this possibility. Preservation protects the follower's recommendation-free response under each leader commitment; a violation need not mean a raw-reward decrease. The regulatory goal is a uniform safeguard against the recommendation channel's additional influence when follower rewards are hidden and the players' objectives may conflict. A gain in a particular instance is compatible with this goal.

Q1: Exact preservation retains the follower's total raw reward at the corresponding QSE outcome. Thm.3.3 shows that recommending a commitment's true QR leaves the executed response equal to that QR at every state and step. Under the same commitment, the entire trajectory law, including future state visitation, therefore stays unchanged. Both players' expected total raw rewards are preserved for every admissible game and commitment. Optimizing the commitment retains the full QSE outcome set (Eq.8; §3.3).

For each admissible game M∈MM\in\mathcal M, let V0(M)V_0(M) and V∗(M)V^*(M) denote follower returns at corresponding preserving and QSE optima. Exact preservation gives V0(M)=V∗(M)V_0(M)=V^*(M) for every MM. Let V1,ϵ(M)V_{1,\epsilon}(M) denote the follower's total raw return under an arbitrarily selected unrestricted policy pair whose leader return is within ϵ\epsilon of the unrestricted leader benchmark.

The negative result explains why this safeguard matters. In Thm.6.2's full-information construction, the follower prefers one action while the leader earns reward only from the other. Every sufficiently near-optimal unrestricted leader choice gives the follower a lower raw return than QSE. Consequently, for every sufficiently small ϵ>0\epsilon>0,

With preservation:inf⁡M∈M[V0(M)−V∗(M)]=0,\text{With preservation:}\quad \inf_{M\in\mathcal M}[V_0(M)-V^{\ast}(M)]=0,

Without preservation:inf⁡M∈M[V1,ϵ(M)−V∗(M)]<0.\text{Without preservation:}\quad \inf_{M\in\mathcal M}[V_{1,\epsilon}(M)-V^{\ast}(M)] < 0.

Thus preservation guarantees that the follower's return does not fall below the corresponding QSE benchmark in any game. Recommendations that improve raw reward in some instances do not contradict this uniform guarantee. Cor.6.3 further shows that every sequence with sublinear regret against the unrestricted leader benchmark incurs linear follower raw-reward loss in this game.

Why a raw-reward gain can still violate preservation. At a fixed state and commitment, the QR μ\mu maximizes the follower's recommendation-free criterion in Eq.1. For the executed response ν\nu, Prop.3.6 gives F(q)=⟨rπ,q⟩+η−1H(q),F(μ)−F(ν)=η−1KL(ν∥μ).F(q)=\langle r^\pi,q\rangle+\eta^{-1}\mathcal H(q),\qquad F(\mu)-F(\nu)=\eta^{-1}\mathrm{KL}(\nu\Vert\mu). For rewards (0,1)(0,1) and η=1\eta=1, raising action two's probability from its QR value 0.7310.731 to 0.90.9 increases raw reward but lowers FF from 1.3131.313 to 1.2251.225: the entropy decrease outweighs the reward gain. This explains how Fig.3(b)'s reward improvement at the same visited states and commitments can coexist with criterion loss (App. T.4). We will add this example beside Remark 3.7.

During learning, protection also controls the entire episode's raw return. Let JF(π,ξ)J_{\mathrm F}(\pi,\xi) denote expected total follower raw reward under commitment π\pi and response ξ\xi. For the executed response νt\nu^t and true QR μt\mu^t under the same commitment πt\pi^t, we have derived ∑t=1T∣JF(πt,νt)−JF(πt,μt)∣≤HηTCG(T)/2=O~(H3/2dT)\sum_{t=1}^T|J_{\mathrm F}(\pi^t,\nu^t)-J_{\mathrm F}(\pi^t,\mu^t)|\le H\sqrt{\eta T\mathrm{CG}(T)/2}=\tilde O(H^{3/2}d\sqrt T) with high probability under Thm.6.1's fixed-parameter conditions. Here CG is cumulative loss in the original criterion. Each return uses its own induced trajectory, so this includes downstream effects and bounds cumulative harmful changes; their per-episode average vanishes. The proof combines the trajectory KL chain rule, Pinsker and Cauchy–Schwarz. We will include this corollary and proof.

AuthorsOpenReview ↗

Response to Reviewer 5CYD, Part 2

W2: Clearer writing and more intuition

We will make the main text clearer and easier to follow by explaining the protection rationale and algorithmic intuition alongside the relevant concepts and results. Our original manuscript included these explanations in the main text, but we moved some of them to the appendix to meet the nine-page limit. Relevant explanations remain in App. H (the regulatory objective), App. I.6 (dynamic examples), and App. J (the proof mechanisms). As one additional main-text page is allowed in the revision, we will use it to restore these explanations near their first use and make three focused improvements:

  1. In §3: introduce commitment, recommendation and executed response through the inventory example before Definition 3.1; explain the fixed-commitment comparison; add the reward/criterion example beside Remark 3.7 and clarify the corresponding-QSE raw-return guarantee after Eq.8.
  2. At the start of §5: explain estimation, the fitted-QR recommendation and optimistic planning before introducing the matrix notation.
  3. Around Thm.6.1: explain why first-order response errors control behavioral deviation, why squared errors control criterion loss, and how preservation and Bellman optimism jointly support absolute leader regret. State the episodic raw-return consequence briefly and place its detailed proof in the appendix.

These changes will make the model, protection objective, algorithm and guarantees easier to follow for a broader audience.

To illustrate the algorithmic intuition we will bring into the main text, PROM first fits the follower's response model from observed actions. It then selects a commitment through optimistic planning, with the recommendation for each candidate commitment fixed to that model's QR. With the true model, this recommendation leaves the follower's response unchanged. During learning, the analysis connects the fitted recommendation, actual execution and true QR to control the recommendation's additional effects while the leader explores.

If a particular passage remains hard to follow, please point us to it. We would be glad to clarify it during the discussion.

We hope our responses resolve your questions and concerns. If so, we would appreciate it if you could consider raising the overall evaluation. If any remain, please let us know so that we can address them fully.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.