From System Performance to Decision Attribution in Agent Evaluation

In agent evaluation, how much of an apparent statistical advantage comes from the decision policy, and how much from the human-designed system around it? An expert shortlist can remove difficult choices before the statistical policy makes a decision, while the LLM may have to propose actions on its own. End-to-end scores combine these contributions. We introduce Matched Policy Intervention (MPI), a framework for component-level decision attribution in agent evaluation. MPI holds the decision affordance (the external information, action space, budget, and execution rules) fixed and randomly preassigns which policy's recommendation is executed. Paired recommendations reveal whether similar outcomes arise from similar decisions. Across CausalGame, CARE, and a discrete-action adaptation of CausaLab, we find that some apparent end-to-end advantages disappear under matched affordances. On CARE's ChemLex task, a +5.72-point system gain accompanies a matched gate-minus-LLM effect of -0.09 points. In CausalGame, changing the affordance alone raises benchmark success from 25% to 62.5% under a fixed statistical policy. MPI distinguishes the contribution of a decision policy within an agent from the support supplied by its decision affordance.