Leader-Follower Games with Policy Recommendation: Reinforcement Learning under a Preservation Constraint

In leader-follower games (LFGs), policy recommendations provide a self-interested leader with an additional channel, beyond her committed policy, to influence a boundedly rational follower who has a recommendation-sensitive response. This steering effect can lead to outcomes that depart from the quantal Stackelberg equilibrium benchmark and potentially harm the follower's interests. To protect the follower's interests, an external preservation constraint is designed to restrict the leader's recommendations so that the follower's executed response is close to her quantal response, without requiring disclosure of her reward function. We study online reinforcement learning (RL) under this constraint in episodic LFGs with linear function approximation, where the leader cannot observe the follower's reward function or her realized rewards. We propose an algorithm that efficiently infers the follower's response from observed actions and operates under an estimated preservation constraint. We establish complementary positive and negative results on follower protection with and without the preservation constraint. Under the preservation constraint, we establish simultaneous high-probability bounds on absolute leader regret and three measures of the recommendation's effect on the follower. For horizon length $H$, feature dimension $d$, and $T$ episodes, the bounds are $\tilde{O}(H^2{d}^{3/2}\sqrt{T})$ for absolute leader regret, $\tilde{O}(Hd\sqrt{T})$ for cumulative preservation violation and reward gap, and $O(H{d}^2\mathrm{polylog}(T))$ for cumulative criterion gap, with other problem parameters fixed. Without the preservation constraint, we construct an explicit game in which any leader achieving sublinear regret against the unconstrained benchmark induces linear cumulative follower losses. Together, these results demonstrate the importance of the preservation constraint in protecting the follower from long-term losses caused by the leader's recommendations.