acceptodds
Published as a conference paper at NeurIPS 2026

Multimodal Foundation Agents Should Use Brain Data as Privileged Supervision

Richard Csaky

Abstract

This position paper argues that multimodal foundation models should use brain recordings as privileged internal-state supervision during training. External behavior alone leaves human-like cognition, robustness, and reasoning weakly constrained. Neural recordings can serve as a training-time modality that shapes representations, latent-state dynamics, private computation policies such as reasoning tokens, and robustness before distillation into agents that operate from ordinary sensory, language, and action streams. We develop this position through Foundational Human Modeling (FHM), a program centered on scaling generative brain models, aligning heterogeneous neural modalities through shared latent geometry, inserting brain tokens into unified multimodal token streams, and testing whether the privileged channel improves distilled agents after removal. The argument is grounded in privileged-information theory and prior work showing that neural or human-derived targets can improve robustness, semantic transfer, data efficiency, and behavioral similarity after auxiliary signals are removed.

Related papers

See more related submissions when the market opens.