Should Humans Be in the Loop? Human-AI Collaboration Paradox and Automation Cliffs

Human-in-the-loop AI systems are widely viewed as a safe and effective design for high-stakes decision-making, yet human oversight may weaken as AI becomes more capable. We study how strategic human adaptation shapes organizational human–AI collaboration. We develop a game-theoretic framework in which a coordinator allocates tasks across AI-only, human-only, and AI-assisted workflows while human reviewers endogenously adjust oversight effort. We characterize equilibrium coordination structures and extend the analysis to dynamic learning and heterogeneous multi-task settings. We establish a Collaboration Paradox: human–AI collaboration may be suboptimal even when it nominally dominates human-only and AI-only operation, because humans strategically reduce oversight effort as AI becomes more reliable. We also identify Automation Cliffs, where small improvements in AI capability trigger abrupt transitions between collaboration, independent operation, and near-full automation. Our results show that the organizational impact of AI depends jointly on technical capability and endogenous human oversight.