Beyond Pass Rates: Attributing Relative-Spatial Failures in Generated Animation Code

Language models are increasingly utilized to generate animation programs for explanatory videos, yet relative-spatial instructions remain substantially harder than absolute motion or temporal constraints. Existing work establishes this performance gap but largely treats spatial failure as a single phenomenon and evaluates interventions through aggregate pass rates. In this paper, we introduce **RefProbe**, a diagnostic framework that attributes relative-spatial failures to distinct underlying causes using two signals exposed by program generation: (i) the observed placement behavior in the execution trace; (ii) the geometric quantities consulted by the generated program. Our framework assigns failures under a fixed decision procedure and enables interventions to be analyzed by the causes they repair and the behaviors they regress, rather than only by their net effect on pass rate. Across seven open-weight generators and 42,000 generated programs, we find that relative-spatial failures are strongly heterogeneous across both causes and models, with a substantial fraction arising from object-extent errors, reference anchoring failures, and verifier-boundary mismatches. Moreover, interventions with modest aggregate effects often produce large but opposing repair and regression effects at the mechanism level. These results suggest that aggregate pass rates are insufficient for diagnosing spatial generation failures and motivate cause-aware evaluation of program-generating models.