Consequence Audits for Regression Diagnosis in Learned Dynamics

A learned dynamics update can reduce prediction error while introducing errors that have larger effects on a downstream controller. We evaluate these effects by inserting a model prediction into a reference simulator and comparing paired finite-horizon continuations under a fixed controller. Each violation yields a reproducible consequence witness; comparing labels before and after an update identifies regressions. We also study how to obtain these witnesses from a finite model bank under a simulation budget, using the signed responses of queried predictions to prioritize subsequent queries. In a post-hoc analysis of a six-task repair study, 3,931 of 4,221 regressions under uniform repair have lower one-step mean squared error (MSE). A separate fresh-source experiment finds 7,286 failure sources versus 6,569 for MSE round-robin at the same simulation budget. This discovery advantage is not consistent across three new tasks, and witness weighting does not establish an incremental repair benefit. In a repeated-use test, consequence-based selection yields lower return than cost-matched selection rules. The audit thus exposes model-update regressions that prediction error alone can miss, while its selection performance depends on how the model is used.