acceptodds
Under review as a conference paper at ICLR 2027

Diagnosing and Repairing Long-Horizon Failure in Transformer Queueing Simulators

Abstract

Neural event simulators are evaluated on short test histories but deployed over much longer rollouts, where accurate one-step predictions need not ensure faithful trajectories. Standard test loss leaves predictions beyond the training length unexamined. We investigate whether predictions on shared long reference histories can diagnose these failures and guide corrective supervision. In an M/M/1 queue, we evaluate transformers trained on -event sequences using rollouts of events per model, and compute deep-prefix drift (DPD), the mean predicted queue increment beyond the training length, on simulator-generated reference histories. Across failing checkpoints and three fine-tuning seeds, we compare deep supervision with matched shallow supervision and continued training, and on three of them compare exact conditional targets with fresh simulator draws and recorded continuations. Despite near-oracle short-history test loss, models cross a fixed queue boundary on – of rollouts, with of first crossings occurring beyond the training length. On evaluation models, DPD ranks crossing counts at Spearman and flags of severe failures with false positive. DPD requires neither candidate-model rollouts nor exact conditional probabilities, using model predictions on shared simulator-generated reference histories. Deep supervision produces fewer crossings than both controls in all model-by-seed comparisons, by at least of rollouts, and fresh simulator targets outperform both controls in all tested comparisons, while recorded continuations leave at least more crossings than exact targets. These findings support shared long-history evaluation as a parallel diagnostic for learned event simulators and show that supervision beyond the training length can reduce failures hidden by short-history test loss.

Primary area:
generative models
Keywords:
autoregressive models, long-horizon generation, length generalization, neural simulators, queueing systems, transformers

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_7x19OpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper studies long-horizon failures in transformer-based neural event simulators using an M/M/1 queueing system. It shows that models with similar short-history test losses can exhibit substantially different behaviors during long rollouts. To diagnose these failures, the paper proposes Deep-Prefix Drift (DPD), which measures predicted queue increments on shared reference histories beyond the training length. It further introduces a supervision approach that fine-tunes models on longer contexts using oracle or simulator-sampled targets. Experiments evaluate the effectiveness of DPD in identifying severe failures and compare the proposed repair against several training baselines.

Critical strengths
  1. The paper identifies and empirically demonstrates a discrepancy between short-history prediction accuracy and long-horizon simulation behavior.
Critical weaknesses
  1. Motivation and Necessity: The failure scenario studied by this paper appears to be largely induced by its model design. The M/M/1 queue's transition law depends only on the current queue length, making absolute event positions irrelevant to its underlying dynamics. However, the paper adopts sinusoidal absolute positional encodings and evaluates models at positions beyond their training length, where extrapolation failures are already well known. The authors' own experiments also show that none of the 30 models using ALiBi, NoPE, or mixed-position training exhibits severe failures, compared with 18 of the 46 models trained with sinusoidal absolute positional encodings. These results raise a fundamental question about the necessity of the proposed approach: why is it important to diagnose and repair failures that can largely be avoided by using other positional encoding techniques?

  2. Validity of the DPD Diagnostic: It's unclear why DPD should reliably characterize long-horizon failure. DPD averages predicted queue increments across different states and positions, whereas the probability of crossing a queue boundary depends on the state-dependent transition dynamics throughout a rollout. As a result, a positive (or negative) average drift does not necessarily imply a high (or low) crossing risk. Counterexamples for both cases are also shown in the paper's results: model 53 has positive DPD but crosses the boundary in only 1 of 256 rollouts, whereas model 104 has negative DPD but crosses in 210 rollouts. This limitation also extends to the alternative positional encoding models, all of which have negative DPD despite some exhibiting up to 98 crossings, compared with the oracle's expected 2.86. To my understanding, DPD primarily detects a particular pattern of drift reversal; it cannot reliably characterize long-horizon failure in general.

  3. Novelty of the Repair: The proposed repair method seems to be a straightforward application of long-context fine-tuning where the preceding diagnosis is almost not involved. Although DPD identifies potentially failing checkpoints, the repair applies additional supervision at predetermined deep positions without using model-specific diagnostic results to determine where or how to intervene. Additionally, the repair's comparisons against shallow supervision and continued training mainly demonstrate that training on longer contexts is more effective than additional training within the original context length. Given that the identified failures arise beyond the training length, this result is largely expected. Table 9 in the supplementary material also shows that ordinary long-sequence fine-tuning substantially reduces failures, from 255, 245, and 240 to 5, 71, and 7, respectively.

Questions for authors
  1. Given that ALiBi, NoPE, and mixed-position training largely avoid the severe failures observed with sinusoidal positional encodings, what is the significance of the studied failure scenario beyond this particular model configuration?

  2. Given the counterexamples in models 53 and 104 and the substantial crossing rates observed in models with negative DPD, why should average predicted drift be used to characterize long-horizon failure risk?

  3. What specific role does DPD play in guiding the repair beyond identifying failing checkpoints, and what advantages does the proposed repair offer over ordinary long-sequence fine-tuning (which already substantially reduces failures in Table 9)?

1 comment
AuthorsOpenReview ↗

## Response to Reviewer 7x19

We thank Reviewer 7x19 for the detailed technical critique, which motivated our comparison against ordinary long-sequence fine-tuning and our within-configuration ranking analyses.

1. Motivation and necessity given ALiBi, NoPE, and mixed positions (W1, Q1):

  • We agree that sinusoidal drift reversal comes from extrapolating absolute positions past the training length, and that ALiBi, NoPE, and mixed positions avoid severe failure (00 of 3030 models); the Introduction and Section 4.1 state this upfront.
  • Diagnosis and repair are still needed for two reasons: (i) 1616 of the 3030 alternative-encoding models still exceed the oracle's 97.5%97.5\% crossing point (77 crossings, up to 9898), and doubling the sinusoidal training length to 800800 events still leaves 33 of 88 models severe (Table 6); (ii) we study the published simulator of Mittal et al. (2025), where crossings range from 00 to 255255 with the training seed alone while test CE varies by at most 1.53×10−41.53 \times 10^{-4} nats. Because the true M/M/1 law is position-invariant, any position-dependent shift is a model error; our protocol measures that error before rollout.

2. Validity of DPD, counterexamples (models 53 and 104), and negative DPD (W2, Q2):

  • We agree that an average drift across states and positions does not uniquely determine crossing probability, and that DPD>0\mathrm{DPD} > 0 specifically detects net drift reversal. Models 53 and 104 are analyzed where the threshold is introduced (Section 4.2, Appendix A.7): model 53 drifts upward below q=20q = 20 (+0.405+0.405) but pulls downward at q≥35q \ge 35 (−0.170-0.170), turning back before q=50q = 50; model 104's upward drift is confined to q≤9q \le 9 and diluted by averaging (DPD=−0.033\mathrm{DPD} = -0.033), with crossings beginning at a median depth of 1,0491{,}049 events (beyond the 800800-event bank).
  • The revision restricts the zero threshold to sinusoidal M/M/1 (all 3030 alternative-encoding models have negative DPD, and the zero threshold over-flags in M/M/5) and uses DPD primarily for continuous within-configuration ranking:
    • Within ALiBi, NoPE, and mixed positions (no severe failures), DPD's Spearman correlations with crossings are +0.84+0.84, +0.53+0.53, and +0.42+0.42 (mean +0.60+0.60, p=.001p = .001; the two weakest are unresolved alone at 1010 models each; Table 5).
    • Among the 2828 non-severe sinusoidal M/M/1 models, DPD ranks crossings at +0.58+0.58 (p=.001p = .001), and on M/M/5 (where true drift changes sign across states) at +0.86+0.86 (p<10−4p < 10^{-4}).
    • In the selection analysis (Table 4), picking the lowest-DPD of 55 candidates averages 6.56.5 crossings vs. 82.182.1 for random on sinusoidal M/M/1, 1.41.4 vs. 14.314.3 on ALiBi, and 2.82.8 vs. 67.467.4 on M/M/5 (resolved in 33 of 55 configurations).

3. Role of DPD in repair and comparison with ordinary long-sequence fine-tuning (W3, Q3):

  • Role of DPD: DPD selects which checkpoints to repair; the supervised positions (400400--799799) and states (11--1919) are fixed by protocol (Section 4.3).
  • Comparison with ordinary long-sequence fine-tuning (Section 4.3, Tables 11--12): We compared deep supervision (using single-step simulator draws) against ordinary long fine-tuning on unconditioned 800800-event trajectories across all 1818 severe failures at fine-tuning seed 0 on one machine, matched in updates (3,1203{,}120), optimizer, anchor data, and targets per update (50%50\% beyond position 400400 for ordinary long vs. 100%100\% for deep supervision). Deep supervision left fewer crossings on 1414, more on 33, and tied on 11 (121121 vs. 508508 total; Table 12), exceeding the oracle's 97.5%97.5\% point (77) on 55 of 1818 models versus 1111 of 1818 for ordinary long fine-tuning. Across 33 checkpoints run at 33 fine-tuning seeds each (Table 11), the winner changed with the seed on 22 of 33 checkpoints (66 wins vs. 33 losses over 99 cells; 1818 wins, 55 losses, 11 tie across all 2424 comparisons). Both recipes remove most crossings, with a larger overall reduction under deep supervision that varies across checkpoints and seeds.
  • Additional findings: (i) Single-step simulator draws suffice as targets, and (ii) training on the survival-selected bank's own recorded labels fails (≥199\ge 199 more crossings per cell due to survival bias, while test CE changes by less than its spread across models; Table 10). In two repaired models (seeds 8 and 105), 86%86\%--94%94\% of remaining crossings start at position ≥800\ge 800, beyond both recipes' supervised window (Appendix A.4); prediction quality beyond position 800800 remains open.
Reviewer_Cq3GOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper investigates long-horizon failures in transformer-based neural event simulators despite accurate one-step predictions. Using an M/M/1 queue, the authors evaluate 46 transformers trained on 400-event sequences over 40,000-event rollouts. They seem to use "deep-prefix drift" which is unclear to me is a novel contribution or not. Also the main contributions of the paper is not clear. Overall this paper is not a good fit for a top-tier conference like ICLR.

Critical strengths
  1. Studies an important problem: accurate one-step predictions do not guarantee reliable long-horizon rollouts in transformers.
Critical weaknesses
  1. The paper is difficult to follow, with a number-heavy abstract and insufficient intuitive explanation of M/M/1 queueing for a new reader. The abstract or the methodology doesn't talk about why certain design choices both in the modelling approach and selection of hyperparmeters made, making it very hard to follow the paper.

  2. Experiments are limited to one queueing system and transformer architecture, with unclear generalisability to broader sequence-modeling tasks like forecasting, next token prediction etc. Also no baseline methods were compared against.

  3. Reasoning behind choice of inductive biases is unclear nor is the reasoning behind the evaluation dataset selection.

Questions for authors

Q1: How do the findings generalize to time-series forecasting, LLM next-token prediction, or other sequential modeling tasks?

Q2: Would similar failures occur with alternative architectures such as state-space models, CNNs, RNNs, or linear models?

Q3: What motivates the choice of "46" models and the "40,000"-event rollouts? More specifically, how are these hyperparmeters or setup selected? What motivates the hyperparameter choices(e.g. number of models) , and how sensitive are the results to them?

Q4:What motivates the choice of the training datasets and reference histories generated and selected?

Q5: What is the core contribution of the paper and its novelty ?

1 comment
AuthorsOpenReview ↗

## Response to Reviewer Cq3G

We thank Reviewer Cq3G for pointing out where presentation and design rationales needed clearer explanation.

W1 (Readability, abstract density, and intuitive explanation of M/M/1):

  • We rewrote the abstract and introduction to reduce number density and explain the queue in plain terms: customers arrive randomly and are served one at a time; each arrival (+1+1) or departure (−1-1) updates the queue length, which directly accumulates per-step prediction errors. At 80%80\% server load, departures are slightly more likely than arrivals (0.55560.5556 vs. 0.44440.4444), creating a negative drift (−0.111-0.111 per event) that pulls the queue back to empty.

W2, Q1 & Q2 (Generalizability across systems, architectures, and tasks; baselines):

  • Second system & positional schemes: We added a five-server M/M/5 queue (1616 models; Section 4.4, Table 16), where DPD ranks crossing risk at Spearman +0.86+0.86 and repair leaves 00 crossings in all 66 cells, alongside 3838 M/M/1 models across ALiBi, NoPE, mixed positions, and training length 800800 (Tables 5--6). Non-transformer architectures (RNNs, SSMs, CNNs) and non-queueing tasks (forecasting, LLMs) were not tested and are stated as limitations in Section 5; the protocol applies wherever sequences accumulate a state and reference histories can be simulated.
  • Baselines: Figure 2b compares DPD against 1010 baselines on the 3636 evaluation models: 44 within-training-length metrics (test CE, waiting-time NLL, joint NLL, shallow drift; all near chance, AUROC 0.510.51--0.590.59), 33 deep-position bank metrics (deep CE, unsigned drift error, oracle-informed deep KL), and 33 metrics on unconditioned 800800-event histories. For repair, we compare deep supervision against shallow supervision, continued training, own-label training, and ordinary long-sequence fine-tuning (Section 4.3, Tables 2, 10--12).

W3, Q3 & Q4 (Motivation and sensitivity for 46 models, 40,000-event rollouts, training length 400, and reference bank):

  • Appendix A.5 documents every design choice, and Table 15 tests sensitivity across boundaries (40,50,6040, 50, 60) and horizons (10,000,20,000,40,00010{,}000, 20{,}000, 40{,}000):
    1. Why L=400L=400 and 40,00040{,}000-event rollouts? At 80%80\% load, 99.95%99.95\% of busy periods end within 400400 events, so L=400L=400 covers nearly all busy periods. Over 40,00040{,}000 events (100L100L), a typical rollout encounters ≈1.99\approx 1.99 busy periods that outlast 400400 events, probing unseen deep positions. At qesc=50q_{\rm esc} = 50, the exact queue crosses on only 1.13%1.13\% of rollouts (2.862.86 of 256256), so excess crossings stand out. Across all 99 horizon-by-boundary pairs (Table 15), DPD's Spearman correlation stays at +0.74+0.74 to +0.88+0.88 (AUROC 0.880.88--0.990.99), while test CE stays at chance (−0.07-0.07 to +0.04+0.04).
    2. Why 4646 models? Because outcomes under one recipe range from 00 to 255255 crossings with the training seed alone, evaluating a checkpoint diagnostic requires a multi-seed cohort: 1010 development models (seeds 0--9) and 3636 held-out evaluation models (1212 severe failures).
    3. Why survival-selected reference histories? Unselected busy periods outlast 400400 events with probability 5×10−45 \times 10^{-4}. Selecting 1,0241{,}024 busy periods of length ≥800\ge 800 supplies 400400 deep positions per history; even 3232 histories achieve a median AUROC of 0.990.99 (Table 8).

Q5 (Core contribution and novelty):

  • Please see Item 1 ("Clarified Core Claims") in the General Response above.
Reviewer_NXwQOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper studies the long-horizon failures in Transformer-based queueing simulators. Althought the Transformer can achieve near-perfect one-step prediction accuracy, the prediction drift may be severe for long horizon scenarios. The authors show that the failures mainly occur beyond the training sequence length, and small prediction biases may accumulate and lead to unrealistic queue growth.

To detect the failures, the authors propose DPD to predict long-horizon instability without requiring expensive model rollouts. They further show that supervision beyond the training length reduces these failures, outperforming standard continued training and shallow supervision. The paper highlights the importance of length generalization and correction for reliable long-horizon simulation.

Critical strengths
  • The failure mode is convincing. The authors show that the test loss of 46 Transformers is consistent, but the long-horizon performance can vary significantly.
  • The DPD detection and deep supervision repair make the paper's story and contribution complete.
  • The evaluation is comprehensive.
Critical weaknesses
  • The generalizability seems limited. The core experiment is based on M/M/1 queue, with a single load setting. It is natural to define the drift here, but for more complex stochastic system, it might be tricky to define the scalar drift.
  • The novelty of DPD is limited. The DPD's methodological novelty seems to be limited. It's a bit unclear if the real contribution is the DPD metric or evaluating the model beyond the training length.
  • The existing positional encoding methods can also significantly fix the problem, which weakens the necessity of the proposed repair method.
Questions for authors
  • Can the authors discuss generalizing the methods to more complex stochastic systems?
  • Can the authors clarify the contribution of DPD and the necessity of the proposed repair?
  • Is it possible to generalize the methods to LLM or even agentic progress?
1 comment
AuthorsOpenReview ↗

## Response to Reviewer NXwQ

We thank Reviewer NXwQ for the positive assessment of our problem formulation and experimental thoroughness.

W1 & Q1 (Generalizability to complex stochastic systems and scalar drift):

  • Multi-server M/M/5 queue (Section 4.4, Table 16): Unlike M/M/1 (constant drift −0.111-0.111 at all q>0q > 0), M/M/5 has state-dependent drift that is positive at 11--22 customers, zero at 33, and negative from 44 onward. Even though DPD averages across states, it ranks the 1616 M/M/5 checkpoints at Spearman +0.86+0.86 (p<10−4p < 10^{-4}) and separates the 55 severe failures at AUROC 1.001.00, while our repair leaves 00 crossings in all 66 repaired cells (vs. 112112--240240 for controls).
  • Queueing networks (Section 5 Limitations): In a single-station queue, the scalar increment 1−2p^1 - 2\hat{p} tracks the state that defines boundary crossing. In a network, summing occupancy across nodes can cancel opposite errors at different stations, so a node- or bottleneck-resolved readout of the shared-history pass is needed. Section 5 states this explicitly.

W2 & Q2 (Contribution of DPD vs. evaluating beyond the training length):

  • We agree that the DPD formula itself is simple. The main contribution is the evaluation protocol: scoring all candidates in one parallel teacher-forced pass on shared long simulator histories at positions beyond the training length (10s10\text{s} per model after a one-time 38s38\text{s} bank build; Section 4.2).
  • DPD is the direct summary of this pass for queues. Its main use is within-configuration ranking: across all 55 tested configurations (Table 5), higher DPD tracks higher crossing risk, reducing mean crossings from 82.182.1 (random) to 6.56.5 when picking the lowest-DPD of 55 sinusoidal M/M/1 candidates (Table 4). Its zero threshold is validated only on sinusoidal M/M/1 and over-flags in M/M/5 (77 of 1111 non-severe models, though 55 still exceed the oracle's 97.5%97.5\% point); the revision restricts the sign rule accordingly.

W3 & Q2 (Necessity of repair given alternative positional encodings):

  • ALiBi, NoPE, and mixed positions avoid severe failure (00 of 3030 models vs. 1818 of 4646 sinusoidal), but moderate long-horizon errors remain: 1616 of the 3030 exceed the oracle's 97.5%97.5\% crossing point (77 crossings, up to 9898), and training the sinusoidal recipe on 800800-event sequences still leaves 33 of 88 models severe (Section 4.1, Table 6).
  • In addition, for the published simulator of Mittal et al. (2025), crossings range from 00 to 255255 across training seeds at the same test loss, so trained checkpoints still need screening and repair.

Q3 (Generalization to LLMs or agentic workflows):

  • We make no empirical claim about LLMs. The protocol applies wherever generated sequences accumulate an observable state and deployment contexts exceed the training length; testing analogous shared-prefix diagnostics in language or agentic settings is future work.

Comments on the paper

1 comment
AuthorsOpenReview ↗

## General Response (to All Reviewers)

We thank the reviewers for their constructive feedback. Four questions recur across the reviews: (1) generalizability beyond one M/M/1 queue, (2) whether the contribution is the shared-history evaluation protocol or the DPD formula, (3) how targeted deep supervision compares against ordinary long-sequence fine-tuning and alternative positional encodings, and (4) presentation clarity and motivation of design choices. We have rewritten the manuscript for clarity and added three sets of experiments.

1. Clarified Core Claims

  • Diagnostic protocol (shared-history evaluation beyond the training length): Scoring candidate checkpoints in one parallel teacher-forced pass on a shared bank of long simulator-generated busy periods exposes length-extrapolation errors invisible to short-history test loss, without generating rollouts or using the oracle.
  • Checkpoint screening (DPD as a within-configuration ranker): Deep-prefix drift (DPD), the mean predicted queue increment beyond the training length, summarizes this pass in one scalar. Its zero threshold applies only to sinusoidal M/M/1, where drift reverses sign, but its continuous value ranks checkpoints by crossing risk within all 55 model configurations tested (Spearman +0.42+0.42 to +0.86+0.86).
  • Targeted repair (deep supervision): Fine-tuning at positions beyond the training length removes most crossings in both queues, even when targets are single-step simulator draws, and leaves fewer crossings than matched ordinary long-sequence fine-tuning on 1414 of all 1818 severe M/M/1 failures at one fine-tuning seed each (121121 vs. 508508 total crossings).

2. New Experiments & Revisions

  • Second system with state-dependent drift (M/M/5; Section 4.4, Table 16): Across 1616 new transformers on a five-server M/M/5 queue (utilization 0.60.6), where true drift changes sign across states, DPD ranks negative mean first-crossing time at Spearman +0.86+0.86 (p<10−4p < 10^{-4}; test CE: +0.35+0.35) and separates the 55 severe failures at AUROC 1.001.00. The unchanged repair recipe reduces crossings from 187187 and 182182 to 00 in all 66 checkpoint-by-seed cells (vs. 112112--240240 for controls).
  • Comparison with ordinary long-sequence fine-tuning (Section 4.3, Tables 11--12): Over all 1818 severe M/M/1 failures at fine-tuning seed 0 (one machine, matched updates, optimizer, anchor data, and targets per update), deep supervision with simulator targets left fewer crossings on 1414, more on 33, and tied on 11 (121121 vs. 508508 total; 55 vs. 1111 models exceeding the oracle's 97.5%97.5\% point of 77 crossings). Both recipes remove most crossings; the per-checkpoint winner can vary across fine-tuning seeds (Table 11).
  • Within-configuration ranking & selection (Section 4.2, Tables 4--5): DPD positively ranks crossing risk within sinusoidal M/M/1 (+0.82+0.82), ALiBi (+0.84+0.84), NoPE (+0.53+0.53), mixed positions (+0.42+0.42; mean +0.60+0.60, p=.001p = .001), and M/M/5 (+0.86+0.86). Picking the lowest-DPD of 55 candidates averages 6.56.5 crossings on sinusoidal M/M/1 (vs. 82.182.1 random, 89.889.8 lowest test CE) and 2.82.8 on M/M/5 (vs. 67.467.4 random, 64.464.4 test CE).
  • Presentation and design rationale (Abstract, Sections 1--5, Appendix A.5, Table 15): We rewrote the abstract and main text, added plain-language queueing intuition, and documented all setup choices and horizon/boundary sensitivities in Appendix A.5.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.