Resolving Agent Uncertainty And Handoff Errors Under Scarce Human Attention

When an agent in a multi-agent system (MAS) encounters uncertainty, requesting clarification is preferable to silently guessing, particularly in cases where human expertise is required. However, an agent's uncertainty reflects information missing from its local view, not necessarily a system-wide need to query a human. Consequently, clarification requests can escalate system-resolvable uncertainty to humans, while silent errors in inter-agent handoffs may go unreported. A MAS can scale faster than available human attention, amplifying avoidable human queries and costly downstream revisions. We introduce Arbiter, a lightweight plug-in meta-agent that jointly handles agent-raised clarification requests and terminal handoffs at which errors may otherwise remain silent. Its policy decides whether and how to resolve them within the system or by querying a human while treating human attention as scarce. We train the policy by modeling the trade-off between end-to-end task success and human-query count across feasible resolution choices. Because multiple choices can achieve the same outcome, Arbiter retains this flexibility during training rather than imposing a unique target. We evaluate a 4B Arbiter on math task across MAS configurations, and 30B agent models. Our experiments show that Arbiter reduces human queries by up to 9.1% compared with strong prompt-only baselines while maintaining end-to-end task success. Specifically, we find that 21.4% of baseline human queries occur even when a non-human resolution succeeds. Overall, Arbiter reduces avoidable human queries by 8.7%, enabling more efficient allocation of human attention without parametric updates to the underlying MAS agents.