acceptodds
Published as a conference paper at NeurIPS 2026

The AI Observatory: A Public Measure of Real-World AI Use

Shayne Longpre, Anka Reuel-Lamparth, Dayeon Ki, Zhiping Zhang, Cathy Fang, Chuanyang Jin, Jennifer Mickel, Cedric Whitney, Niloofar Mireshghallah, Victor Ojewale, Megan Richards, Ariel N. Lee, Alex Pentland, Tianshi Li, Yuntian Deng, Sara Hooker, Mykel J Kochenderfer, Sanmi Koyejo

Abstract

Understanding the benefits, risks, and impacts of general-purpose AI assistants requires looking at real conversations—not curated benchmarks or surveys. Yet AI use narratives rely on limited proprietary reports, or on few data sources, fragmented across platforms, models, and time. We introduce the AI Observatory, a public measurement platform that aggregates the most comprehensive set of real AI conversation sources, and develop a common taxonomy of 145 features, spanning function, topic, sensitive use, interaction style, multi-turn dynamics, and conversation structure. Across 23,158 conversations and 85,633 turns, we find that real AI use is highly heterogeneous: sources differ substantially in tasks, structures, and sensitive-use distributions, with no single source that generalizes. We further show that occupational summaries such as Anthropic's Clio capture an important but incomplete slice of use: 48% of the conversations we analyze would be filtered out for being non-occupational, omitting substantial personal, social, cultural, and safety-relevant interaction. Real AI use is also not static: we find tokens and turns per conversation all rise sharply between 2023 and 2025, and we expose how model variants within the same developer support distinct usage regimes. The Observatory provides a new taxonomy, reusable annotation tools, and a public platform for reproducible measurement of how AI assistants are really used and evolving in the wild: https://project-ai-observatory.vercel.app/.

Related papers

See more related submissions when the market opens.