acceptodds
Under review as a conference paper at ICLR 2027

Should Humans Be in the Loop? Human-AI Collaboration Paradox and Automation Cliffs

Abstract

Human-in-the-loop AI systems are widely viewed as a safe and effective design for high-stakes decision-making, yet human oversight may weaken as AI becomes more capable. We study how strategic human adaptation shapes organizational human–AI collaboration. We develop a game-theoretic framework in which a coordinator allocates tasks across AI-only, human-only, and AI-assisted workflows while human reviewers endogenously adjust oversight effort. We characterize equilibrium coordination structures and extend the analysis to dynamic learning and heterogeneous multi-task settings. We establish a Collaboration Paradox: human–AI collaboration may be suboptimal even when it nominally dominates human-only and AI-only operation, because humans strategically reduce oversight effort as AI becomes more reliable. We also identify Automation Cliffs, where small improvements in AI capability trigger abrupt transitions between collaboration, independent operation, and near-full automation. Our results show that the organizational impact of AI depends jointly on technical capability and endogenous human oversight.

Primary area:
optimization
Keywords:
Human-AI Collaboration, Generalized Nash Equilibrium, Dynamic AI Learning

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_RcwrOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies the question of when an organization is faced with deciding between AI-only operation, human-only, or collaboration, which option benefits organizational decision making in the long run? The authors model this problem game-theoretically first with a static model in which the AI does not improve, and then on their main focus of an improving AI system. They find a "collaboration paradox" in equilibrium in which even if the human-AI assisted solution apepars optimal compared to AI or human only it is suboptimal. They term "automation cliffs" for situations in which slight improvements in AI capabilities will lead to full automation via AI in their model. They demonstrate their theory empirically on synthetic data for radiographic diagnosis.

Critical strengths

The paper provides a clean and insightful model of a critical problem in AI and its integration to society. While stylized, the model is tractable and the results are interesting. The result that that scarce oversight should not automatically go to the highest-stakes tasks is surprising.

Critical weaknesses

I don't feel the experiments add much but they are a nice illustration of the theory on data even if synthetic. I think a better experiment could have been possible, maybe using a weaker LLM as a stand-in for human, etc.

Questions for authors

How much does the Collaboration Paradox hold if the human values performance directly, instead of only meeting a threshold at minimum effort?

Are the automation cliffs anything beyond the boundaries of the LP?

Reviewer_pcTCOpenReview ↗
Rating
3
Confidence
1
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates whether human involvement in AI-assisted decision-making is always beneficial. The authors propose a game-theoretic framework that considers different task allocation strategies and how humans adjust their oversight effort. The paper introduces two interesting phenomena, the Collaboration Paradox and Automation Cliffs, highlighting how AI capabilities and organizational constraints can influence human–AI collaboration. The research question is relevant, and the consideration of adaptive human behavior is particularly interesting. However, the experimental evaluation relies mainly on synthetic simulations, while the radiographic case study is illustrative rather than a direct empirical validation. More comparisons with alternative coordination strategies and a clearer distinction between theoretical predictions and real-world evidence would strengthen the paper.

Critical strengths
  • The paper addresses an important question about whether human involvement always improves AI-assisted decision-making. In particular, considering how humans adapt their oversight effort as AI capabilities improve provides an interesting perspective on human–AI collaboration.

  • The paper considers organizational constraints such as human capacity and performance requirements, making the framework relevant to real-world applications. The Collaboration Paradox and Automation Cliffs offer interesting insights into how these factors can influence task allocation and the role of human oversight.

Critical weaknesses
  1. There is something I do not fully understand about the assumptions of Proposition 1. In particular, assumption (ii) requires the human's optimization problem to be feasible for every coordinator policy that satisfies the capacity constraint. However, consider the case where (pA=1p_A=1), meaning that all tasks are assigned to AI. This policy always satisfies the human capacity constraint since no human effort is required. Nevertheless, if (rA<δr_A<\delta), the human's optimization problem becomes infeasible because the performance requirement cannot be satisfied, regardless of the choice of (qRq_R). Does this imply that Proposition 1's assumptions exclude the case (rA<δr_A<\delta)? If so, how does this align with the equilibrium characterization in Theorem 1, which also considers regimes where AI performance is below the required threshold?

  2. While the behavioral assumptions are reasonable and the synthetic experiments effectively illustrate the proposed phenomena, the radiographic case study remains an illustrative calibration rather than a direct empirical validation. The paper could better distinguish between theoretically predicted organizational behaviors and phenomena observed in real-world human–AI collaboration.

Questions for authors

Please explain 1.

Reviewer_y9MVOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
3
Read review
Paper summary

Terminology clarification: I'm going to write this review using the standard terminology of principal-agent mechanism design. In this terminology both the principal and the agent are humans. There is also an AI in the model, however, the AI is not strategic. I will not refer to the AI as an agent.

This paper considers a model for human-AI collaboration where a principal chooses the rates of assigning tasks to human-alone, AI-alone, and AI-human-combined review. The human (a.k.a. agent) in the AI-human combined review chooses how much effort to put into the monitoring of the AI. Key parameters are the allowed level of human effort bb and the accuracy δ\delta (note: large δ\delta is more accurate). In this model bb and δ\delta are constraints. The principal and the (human) agent choose their strategies in Nash equilibrium. The agent aims to minimize their effort, while still ensuring that the quality threshold δ\delta is met.

The paper considers a static model and a dynamic learning model. In the static model the equilibrium concept between the principal and agent is Nash. In the dynamic learning model, human labels, when the human is more skilled than the AI, are used to improve the AI and the AI's performance is monotonic in the human labels collected. The equilibrium concept is Stackelberg. (The dynamic model has an oddity that I will discuss more later which is that human labels stop being helpful for training the AI when the AI alone exceeds the skill of the human alone. This is odd because the human+AI collaboration is considered always better than the AI alone which means the human always has something to offer.)

A main analysis of the paper is to characterize the equilibria in the b,δb,\delta plane by partitioning it to regions that are infeasible and where certain modes of task solving are not used (e.g., no collaboration, full collaboration, partial collaboration, human only). They interpret aspects of this analysis as identifying a collaboration paradox and an automation cliff.

I struggled to find meaning in their partition of the b,δb,\delta. Unfortunately these figures first appear as Figure 2 and 3 with essentially no discussion. (That discussion is left to Section 4 of the paper.) First, usually when an optimization problem is solved with constraints, those constraints are binding. Assuming the constraint on the human budget bb and the accuracy δ\delta are binding, then I'm not sure what else we care about. I find a different perspective on the b,δb,\delta plane to be more understandable. The principal should aim to minimize bb and maximize δ\delta. That gives a Pareto frontier. For any linear tradeoff between the cost of bb and the benefit of δ\delta the principal will choose a vertex of the Pareto frontier. These vertices tell a very different story as to what is relevant. In all of the figures the only vertices are at the b=0b = 0 corner (which is AI only, by definition) and the corner at the top of the full collaboration region. Nothing else in the plane seems relevant to the principal. So why is the paper discussing it in detail?

Their modeling assumptions have a few issues, and I would very much like to know whether any of the result are robust to these issues.

Issue 1: I don't understand why in the human alone condition, the human cannot adjust their effort to lower accuracy if the δ\delta bound is not tight. I expect "fixing" this issue does not change the results of the model.

Issue 2: I don't understand why in the dynamic model that the human-only data is only useful when the human is better than the AI. Given that the human-AI team is always better than the AI alone, this means that the human has complementary abilities/information, and this means that the AI can still be trained with the human data to be better. I don't know if "fixing" this issue changes the results of the model.

Comments on exposition:

  • I recommend that you do your analysis of the static model in Section 2 (not Section 4). Figure 2 unfortunately had no discussion at the point it appeared in the paper and this created a lot of confusion as to how I should interpret it.

  • I recommend referring to the static model as "the static model" not "the core model". In this context "core" has no helpful meaning.

  • I don't know how to read Figure 3 and understand how current performance is sacrificed for future outcomes. Again there is no interpretation of Figure 3 where it occurs. I recommend changing this.

  • In the machine learning literature, I am most used to δ\delta being error (small is good) not accuracy.

  • Can you give some justification for switching solution concepts from Nash to Stackelberg in the static and dyamic models?

Critical strengths
  • understanding strategic aspects of human-AI collaboration is an important question.
Critical weaknesses
  • there are a few issues with the model (hopefully the authors will address in their response)

  • it's not clear why the authors' analysis is relevant (and not just the Pareto frontier, which is much simpler to understand)

Questions for authors

Explain why interior regions of Figures 2 and 3 are interesting, and why we do not only care about the Pareto frontier.

Explain why the AI cannot be trained on the complementary information from the human that must exist for the human-AI team to be superior to AI alone. How does changing this modeling assumption change the results of the paper?

Explain why the human cannot better optimize their effort in the human alone condition if the accuracy parameter δ\delta is not binding. How does changing this modeling assumption change the results of the paper?

Comments on the paper

1 comment
Reviewer_y9MVOpenReview ↗

Why study interior points rather than only the Pareto frontier?

[My apologies for the slow response, I didn't realize the ICLR author-response phase was interactive. Thanks to the authors to alerting me to this.]

The authors' answer about Pareto frontier does not resolve my non-understanding of the paper. Let me spell this out: Suppose the principal is given constraints (b,δ)(b,\delta) and this is in the interior of the feasible set. Then this principal can either:

a) Save money, and spend b′<bb' < b to obtain the exact same δ\delta. I do not understand why we care how to spend more than we need to spend to achieve a given δ\delta.

b) Improve the accuracy, and achieve δ′>δ\delta' > \delta at the exact same cost bb. I do not understand why we care how to produce less accurate answers than we could for the exact same budget bb.

Once you realize we always want to be on the Pareto frontier---and there is no reasonable reason we cannot be---the analysis of the paper is mostly not so relevant.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.