acceptodds
Under review as a conference paper at ICLR 2027

Human Modeling Meets AI Metacognition: Toward Coupled Cognitive Alignment for Interactive Agents

Abstract

AI agents increasingly participate in complex human workflows, where failures arise not only from incorrect outputs but also from inappropriate interactions. An agent may answer when it should ask for clarification, personalize despite high uncertainty about the user, or express confidence in ways that distort human trust and reliance. We argue that these failures stem from inaccurate human modeling that aims to infer humans' latent cognitive states, and unreliable AI metacognition which estimates the agent's own uncertainty, capability boundaries, and risk, and more importantly, misalignment between the two in the interaction policy. A primary bottleneck is that the latent states of humans and AI have to be inferred from partial observations. This position paper advocates Coupled Cognitive Alignment (CCA), a framework that unifies human modeling and AI metacognition as a coupled problem of interaction-level decision making under uncertainty. In CCA, agents maintain uncertainty-aware dynamics models of human states while simultaneously reasoning about their own uncertainty and capability limits. These coupled estimates determine not only what the agent outputs, but whether it should answer, ask, explain, retrieve, decompose tasks, defer, or escalate. We formalize it as a partially observable interactive system, identify its key research challenges, and discuss potential strategies for its learning, evaluation, and deployment.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
human-AI interaction/cooperation, human-in-the-loop, human-centered evaluation, user-centered design, human factors in NLP

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_3jQkOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper addresses an interesting and important problem. The argument that AI agents need to consider both the user’s cognitive state and their own capabilities and limitations when deciding how to interact is useful. The distinction between producing the correct answer and providing the right kind of assistance is also important. Another strength is that the framework treats this as a dynamic process, where the agent updates its understanding of the user and its own capabilities as the interaction progresses, rather than relying on static estimates.

Critical strengths

Overall, the paper does a good job of bringing together two important research areas and highlighting why both matter for human-AI interaction. The emphasis on recursively updating the human and AI models based on interaction feedback is also a useful aspect of the proposed framework.

Critical weaknesses

That said, there are a few concerns:

  1. Considerable work already exists on shared mental models, common ground, adaptive automation, interactive POMDPs and understanding teammates' capabilities and limitations in human-agent teaming. The authors need to do a better job of positioning this work against that literature and explaining what is fundamentally new here.

  2. Although the authors provide a mathematical formulation, they provide little detail on how the framework could be implemented. For example, how would the joint distribution over human and AI states be estimated? How is the Joint Epistemic State (JES) constructed, and what information does it contain? The mapping, loss function, and update mechanism are all left fairly abstract.

  3. The application examples illustrate the motivation, but they remain largely hypothetical. Even a relatively simple implementation comparing human modelling alone, metacognition alone, and the proposed coupled approach would help demonstrate the value of the framework.

Questions for authors

Please see some indicative ones in the weaknesses section.

1 comment

We thank the reviewer for pointing out these connections and asking for a clearer account of JES and its evaluation.

Q1: What does CCA contribute beyond shared mental models, common ground, adaptive automation, and interactive POMDPs?

A1: Thank you. CCA focuses on how human modeling and AI metacognition jointly constrain the next interaction act. We build on the frameworks you identify and discuss these connections in Section 1 of the General Response. As clarified in Section 3, JES is the agent’s decision-relevant summary of the coupled posterior. It does not, by itself, establish common ground between the human and the agent.

Q2: How are the joint distribution, JES, mapping, loss function, and update mechanism specified?

A2: Thank you for requesting these details. DCSM maintains beliefs about the human cognitive state, while MSM maintains beliefs about the agent’s metacognitive state. JES summarizes their coupled posterior for policy selection. The two marginal beliefs alone do not determine the joint posterior, so a concrete implementation must supply the joint model and inference assumptions.

Table 2 in Section 3 of the General Response explains the roles of Φ\Phi, the update operator UU, and the mismatch loss LCCAL_{CCA}. The tutoring implementation in Section 2 uses fitted estimates to select actions. Its results support this use of the estimates, while recovery of the full coupled posterior remains open.

Q3: Could a simple implementation compare human modeling alone, metacognition alone, and the coupled approach?

A3: Thank you for suggesting this comparison. Our controlled tutoring experiment includes all three conditions: human modeling alone, AI metacognition alone, and coupled modeling under CCA. Table 1 in Section 2 of the General Response reports the results. The coupled condition performs better on both held-out problem solving and online misconception resolution.

Reviewer_oZm5OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper presents that interactive AI agents should leverage both the state of human (e.g. goals) and AI metacognition (e.g. confidence) and introduce coupled cognitive alignment (CCA) that maintain uncertainty-aware dynamic models of human while reasoning about its own uncertainty. The paper discuses its applications in tutoring, healthcare, etc.

Critical strengths

Overall, I found that the paper is easy to follow along with good motivating examples and the motivation to model both human and AI metacognition. Also, the paper includes synthesis of related work to present the proposed approach.

Critical weaknesses

Although the topic is interesting, the paper does not include any technical framework or empirical validation. Instead , the paper is limited to provide an abstract modeling through literature review. Given that there is no implementation, evaluation or results, overall proposal is illustrative instead of a validated approach.

Questions for authors

Could you provide explanations on how the implementation of the approach can be specified?

1 comment

We thank the reviewer for requesting a clearer account of implementation and empirical support.

Q1: Could you explain how the implementation of the approach can be specified?

A1: Thank you for requesting these details. Our tutoring implementation estimates student responsiveness and tutor execution reliability, then uses their estimated joint effect to select teaching actions. These operationalize selected aspects of human modeling and MSM, with a decision-relevant summary serving as an operational JES representation. Section 2 of the General Response reports the implementation and results; Section 3 explains the broader specification through tutoring.

Reviewer_TUEXOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

This is a position paper. It proposes the position that the best way to provide an AI agent to support humans in non-trivial tasks is to combine meta-cognition and human modelling. The human modelling is motivated by the need to understand sufficient details about the task context to ensure that answers provided by the agent are suitable (e.g. providing guidance rather than a full solution in a tutoring task) and meta-cognition is for the AI agent to understand it’s own capabilities and task context estimate to better judge it’s ability to support the human.

Critical strengths

The argument for combining meta-cognition and human modelling is a compelling one. The examples provided are convincing that an advance in agentic human support will come from the position espoused by the paper.

Critical weaknesses
  1. Whilst citations are provided for individual methods, e.g. for AI meta-cognition, or human modelling, very little effort is spent relating the position to frameworks/positions that exist in the literature. Table 1 lists a collection of areas where related work is done, but that work is not unpacked enough. E.g. the human/AI teaming, and HRI/HAI literature can be quite closely related to this position. Equally human studies (e.g. the dyadic meta-cognition tasks “Two Heads are Better Than One”) could provide stronger evidence for this posiition, or may possibly contain contradictory results. Either way, for a position paper I expect the position to be connect to the research landscape (of positions) better. Some papers that I found when looking for similar things, that seem related enough to warrant at least a mention here, but there are many, many others
    1. Collins et al., “Building machines that learn and think with people”, Nature Human Behaviour 2024
    2. Shen et al., “Position: Towards Bidirectional Human-AI Alignment”, NeurIPS 2025 position track.
  2. Formalisation doesn’t really add to the paper. Eqns (2) (4) just appear to be long lists of things associated with agent-internal states in general, without justification or more detailed understanding. As such the use of these symbols is just window dressing, and does’t really help us understand the commitments the framework makes (since without defining the symbols, defining a loss or blelief over them is meaningless).
  3. Lack of technical evidence/contribution. ICLR does not have an explicit position paper track or keyword (that I could see), so this paper really needs to make a technical contribution, but there is not supporting experimentation or even a proof-of-concept demo.
Questions for authors
  1. What related positions exist, and how does your differ significantly from them.
2 comments

We thank the reviewer for the careful assessment and the suggested connections to prior research. Our specific replies below complement the General Response.

Q1: What related positions exist, and how does CCA differ from them?

A1: Thank you for highlighting this question. CCA emphasizes the reciprocal constraints between human modeling and AI metacognition in interaction-level decision making. Section 1 of the General Response positions this framing within thought partnership, bidirectional alignment, and existing coordination frameworks, acknowledging their substantial overlap.

Q2: What do the state variables and equations contribute beyond lists of potentially relevant factors?

A2: Thank you for identifying this presentation issue. The equations specify how latent states, maintained beliefs, JES, interaction decisions, and feedback updates relate. Section 3 of the General Response, particularly Table 2, connects these quantities to tutoring and identifies the state dimensions as task-dependent candidates.

Q3: What contribution and technical evidence support the paper?

A3: Thank you for requesting clarification. The intended contribution is the CCA policy-level framing and its specification, supported by published findings and a preliminary controlled implementation. Section 1 of the General Response presents the scope and prior evidence; Section 2 reports the implementation and comparisons with single-sided modeling.

Thanks for the response and the experimental results. It's good to see the tutoring implementation, which provides useful preliminary evidence for the benefits of jointly modelling human and agent states.

That said, my key weakness (Q1) is the positioning and distinctiveness of the proposed position relative to the literature. This remains largely unaddressed. Your response acknowledges the two position papers I mentioned in the review but does not explain how CCA differs from them in a substantive way. I also expect there are other position papers out there beyond these two. Doing this work is what makes a position paper strong and unique, and motivates its publication.

So, whilst I recognise the improvements made in the rebuttal, my central concern about the paper's distinct contribution remains.

Comments on the paper

4 comments
AuthorsOpenReview ↗

General Response

We thank all three reviewers for highlighting the need for clearer positioning, concrete implementation guidance, and a more informative specification of Coupled Cognitive Alignment (CCA). We address these shared concerns below.

Section 1. Scope of the position and relationship to existing research

Developing and comprehensively validating a general implementation is outside the main scope of this position paper. CCA is a policy-level framing that couples human modeling and AI metacognition for interaction-level decision making under uncertainty. Estimates of evolving human cognitive states and the agent’s uncertainty, capability boundaries, and risk should jointly determine the next interaction act. Human modeling constrains how metacognitive uncertainty is communicated, while AI metacognition constrains when and how the agent acts on its human model.

CCA builds on Collins et al.’s (2024) thought-partner agenda, including models of people and the world, misconception-sensitive explanations, evolving human models, machine reliability, and task routing. It also connects to Shen et al.’s (2025) reciprocal-alignment perspective, including human understanding, critical assessment, and adaptation to AI. CCA develops the implications of these agendas for interaction-level policy calibration. Collins et al. · Shen et al.

Grounding theory, mixed-initiative systems, adaptive automation, and I-POMDPs provide foundations for coordination, uncertain user goals, reliability-sensitive assistance, and belief updating. We will expand these connections and revise the manuscript’s positioning table to reflect overlapping assumptions and capabilities. Clark and Brennan, 1991 · Horvitz, 1999 · Parasuraman et al., 2000 · Gmytrasiewicz and Doshi, 2005

Published empirical findings provide a partial basis for the concerns motivating CCA.

  • Complementarity and reliability. Bahrami et al. (2010) found that communicating dyads could outperform their stronger member when perceptual sensitivities were similar, but perform worse when sensitivities differed markedly. Steyvers et al. (2022) improved human–AI prediction by modeling human confidence, machine confidence, and class-specific errors. Bahrami et al., 2010 · Steyvers et al., 2022

  • Communication and reliance. Zhang et al. (2020) found that confidence displays calibrated reliance without significantly improving assisted accuracy, highlighting the importance of complementary human knowledge. Bansal et al. (2021) found that explanations increased acceptance of correct and incorrect AI recommendations without further improving complementary team performance. Zhang et al., 2020 · Bansal et al., 2021

  • Human characteristics and adaptation. Buçinca et al. (2021) found that cognitive-forcing interventions improved decisions when AI recommendations were wrong, with benefits varying by cognitive motivation and accompanied by acceptability costs. Nikolaidis and Shah (2013) found improved mental-model similarity through human–robot cross-training and better coordination fluency within a matched-preference subset. Buçinca et al., 2021 · Nikolaidis and Shah, 2013

  • Sequential intervention. Rafferty et al. (2016) used POMDP planning over latent learner knowledge to balance instructional benefit and diagnostic information, accelerating concept learning relative to random teaching in behavioral experiments. Rafferty et al., 2016

Section 2. Concrete implementation and preliminary results

A controlled tutoring implementation illustrates CCA’s coupled decision principle. An LLM identifies recurring misconceptions from observed errors. A structured latent-variable model estimates student responsiveness and tutor execution reliability by maximizing the likelihood of response-correctness histories while marginalizing over hidden learning trajectories. These estimates jointly determine teaching-action values, and an LLM generates the selected intervention. Evidence about tutor execution reliability is shared across students.

Student responsiveness operationalizes one aspect of human modeling, while tutor execution reliability operationalizes one aspect of Metacognitive Self Modeling (MSM). The learned estimates and resulting action values form an operational Joint Epistemic State (JES) representation for teaching-action selection.

We evaluate offline problem solving on held-out questions, using a frozen learned policy, and online misconception resolution within a fixed teaching budget. Table 1 aligns the comparisons with the manuscript’s terminology and retains the source experiment labels in parentheses.

Table 1. Preliminary GPT-4.1-mini results in the controlled tutoring simulation.

Condition and source experiment labelQuantities explicitly modeledOffline Success@5Online Misconceptions Resolved per Student
No ModelingNeither student responsiveness nor tutor execution reliability is explicitly estimated0.3385.51
Human Modeling Only (HumanState Only)Student responsiveness, without a separate tutor execution-reliability model0.5746.33
AI Metacognition Only (AgentState Only)Tutor execution reliability pooled across students, without a student-specific responsiveness model0.5476.20
Coupled Modeling under CCA (HumanState + AgentState)Student responsiveness and tutor execution reliability jointly inform action selection0.6577.25
Oracle Student ProfileOracle reference condition0.7148.31

Coupled modeling improves both evaluated outcomes over either single-sided condition. These comparisons support the decision value of jointly using the learned estimates under the simulator’s assumptions. They concern a restricted tutoring implementation with explicitly specified student responsiveness and tutor reliability; effectiveness for real learners and broader interaction settings remains future work.

Section 3. Specification of CCA/JES through a concrete setting

The formalization distinguishes latent states, beliefs about those states, and the summary used for policy selection. Dynamic Cognitive State Modeling (DCSM) maintains a belief μtH\mu_t^H over the latent human cognitive state hth_t. Metacognitive Self Modeling (MSM) maintains a belief μtM\mu_t^M over the agent’s metacognitive state mtm_t.

The coupled state is zt=(ht,mt)z_t=(h_t,m_t), with coupled posterior p^t(z)\hat{p}_t(z). Its marginals are μtH\mu_t^H and μtM\mu_t^M. The Joint Epistemic State, J^t=Φ(p^t,τt)\hat{J}_t=\Phi(\hat{p}_t,\tau_t), preserves decision-relevant information for the JES-conditioned interaction policy.

Table 2 connects this specification to the illustrative tutoring dialogue in Appendix B, where the tutor must decide whether to request the student’s reasoning or provide a hint.

Table 2. Interpretation of the manuscript’s quantities in the tutoring setting.

QuantityInterpretation in the tutoring setting
Task context xtx_t, interaction observation oto_t, internal evidence ete_t, and history τt\tau_tThe problem, observed student work and dialogue, evidence about the tutor’s uncertainty and reliability, and accumulated interaction history.
Latent human cognitive state hth_tTask-relevant beliefs and knowledge (btHb_t^H, ktHk_t^H), goals (gtHg_t^H), preferences (θtH\theta_t^H), uncertainty (utHu_t^H), cognitive load (ℓtH\ell_t^H), reliance or trust (rtHr_t^H), and interaction strategy (σtH\sigma_t^H). In tutoring, these can concern understanding the power rule and seeking limited guidance.
Agent’s metacognitive state mtm_tTask uncertainty (utAIu_t^{AI}), confidence or expected quality (qtAIq_t^{AI}), capability boundary information (κtAI\kappa_t^{AI}), risk (ρtAI\rho_t^{AI}), expected success (stAIs_t^{AI}), and uncertainty over the human-state estimate (ηtH∣AI\eta_t^{H\mid AI}). In tutoring, solution confidence can be high while diagnostic confidence remains low.
Coupled posterior p^t(z)\hat{p}_t(z)A joint belief about the human cognitive state and the agent’s metacognitive state. Its construction requires domain-specific modeling and inference assumptions.
JES J^t\hat{J}_t and mapping Φ\PhiA summary preserving information relevant to the next interaction act, such as a suspected misconception together with insufficient diagnostic evidence.
Structured interaction action atAI=(yt,δt,ct)a_t^{AI}=(y_t,\delta_t,c_t) and JES-conditioned policy πAI\pi^{AI}Task-level content, interaction mode such as clarification or direct assistance, and the communication of uncertainty, rationale, limitations, or risk.
Feedback ft+1f_{t+1} and Recursive Coupled Update UUThe student’s reply and other new evidence revise the coupled belief, after which JES is recomputed for the next turn.
Policy-level mismatch loss LCCAL_{CCA}The cost of an interaction act that mismatches inferred human needs and agent capabilities, such as giving a complete solution when diagnostic clarification or a limited hint would be appropriate. Its implementation requires domain-specific observable objectives.

The dimensions in Eqs. (2) and (4) are task-dependent candidates. Their relevance and measurement must be specified for an implementation. HumanState and AgentState in Appendix B label qualitative estimates used in the example, rather than directly observed values of hth_t and mtm_t.

The tutor can know the correct mathematical answer while remaining uncertain about the student’s reasoning. This favors a diagnostic question; the student’s reply may then support a particular diagnosis and make a targeted hint appropriate. This illustrates how the human model and metacognitive model jointly constrain the interaction policy.

We have implemented the controlled tutoring instance described in Section 2, where the preliminary comparisons provide initial support for CCA’s coupled decision principle.

Hi everyone, please acknowledge the authors’ rebuttal with a brief reply, even if your assessment is unchanged (if you already did or are about to do so, please ignore this message).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.