EventLedger: Typed Counterfactual Credit for Multi-Agent LLM Orchestration

Multi-agent LLM systems solve a task by orchestrating typed events: an orchestrator spawns roles, delegates subgoals, issues tool calls, requests critiques, aggregates candidates, and decides to stop. Supervision still arrives once at the end, so the outcome reward is broadcast to every event and a decisive tool call is credited the same as a wasted search or a premature stop. Recent counterfactual credit methods sharpen this signal, but they assume a single homogeneous unit (a message or an agent), are evaluated only with a programmatic verifier, and do not score the stopping decision that governs cost. We present EventLedger, which treats one execution as an orchestration event graph and assigns each event an interventional, auditable process reward. Typed counterfactual operators define a type-compatible intervention for every event type, including a force-continue/force-stop pair that makes termination a scored decision. A perturb-rollout engine estimates the marginal effect of every replayed event without bias, pairs factual and counterfactual continuations with common random numbers, and spends its replay budget on the highest-leverage events. For tasks without a verifier, a cross-family judge committee is calibrated isotonically on verifiable anchors and abstains by a conformal rule, which bounds the credit bias under an explicit transfer assumption. The rewards are combined with potential-based shaping and distilled into a graph-aware student that makes no API calls at inference. Across coding, math, text-to-SQL and open-ended question answering, an orchestrator trained with EventLedger rewards reaches an average score of $87.75$ against $85.57$ for the strongest baseline; the budgeted estimator needs $39.8\%$ of the replay jobs of the full estimator; the oracle pipeline lowers a single judge's expected calibration error from $0.44$ to $0.12$ at $93.33\%$ retention; and the student reaches $0.80$ agreement with the teacher at $0.05\times$ its scoring cost while matching the end-task scores of the teacher-trained orchestrator.