From Information Value to Intervention Value in Language Models

Additional information can improve language model performance, but measuring that potential does not tell us what system change will realize it. We study the gap between the value of information and the value of interventions. We separate four questions: whether oracle information improves task utility (*opportunity*), whether the channel used to supply it preserves that gain (*instrument validity*), whether the remaining shortfall can be reliably assigned to information acquisition or use (*attribution*), and whether acting on that assignment improves utility on held out data (*decision validity*). Across eight benchmarks, oracle information value is the most robust quantity in our audits, while attribution can change under subsampling, reference construction, and reader family. On BigCodeBench, the injection channel itself destroys the oracle gain, making recovery measurements uninterpretable. We then test the final question prospectively on FEVER and QASPER. With 56 diagnostic items per system, the procedure produced stable shortfall attributions under subsampling and threshold perturbations, yet the corresponding frozen interventions reduced utility on untouched test sets by 0.078 verdict accuracy on FEVER and 0.048 QASPER score on QASPER. In both cases, another declared action performed better. These results distinguish two problems: locating where information value is lost and determining which intervention has value. They motivate a simple workflow: use the diagnostic to form intervention hypotheses, then measure intervention value directly on held out outcomes.