Diagnosing and Repairing Long-Horizon Failure in Transformer Queueing Simulators

Neural event simulators are evaluated on short test histories but deployed over much longer rollouts, where accurate one-step predictions need not ensure faithful trajectories. Standard test loss leaves predictions beyond the training length unexamined. We investigate whether predictions on shared long reference histories can diagnose these failures and guide corrective supervision. In an M/M/1 queue, we evaluate $46$ transformers trained on $400$-event sequences using $256$ rollouts of $40,000$ events per model, and compute deep-prefix drift (DPD), the mean predicted queue increment beyond the training length, on $1,024$ simulator-generated reference histories. Across $6$ failing checkpoints and three fine-tuning seeds, we compare deep supervision with matched shallow supervision and continued training, and on three of them compare exact conditional targets with fresh simulator draws and recorded continuations. Despite near-oracle short-history test loss, models cross a fixed queue boundary on $0$–$255$ of $256$ rollouts, with $99.4\%$ of first crossings occurring beyond the training length. On $36$ evaluation models, DPD ranks crossing counts at Spearman $+0.82$ and flags $11$ of $12$ severe failures with $1$ false positive. DPD requires neither candidate-model rollouts nor exact conditional probabilities, using model predictions on shared simulator-generated reference histories. Deep supervision produces fewer crossings than both controls in all $18$ model-by-seed comparisons, by at least $37$ of $256$ rollouts, and fresh simulator targets outperform both controls in all $9$ tested comparisons, while recorded continuations leave at least $199$ more crossings than exact targets. These findings support shared long-history evaluation as a parallel diagnostic for learned event simulators and show that supervision beyond the training length can reduce failures hidden by short-history test loss.