acceptodds
Withdrawn from ICLR 2027

RESYNC: Residual Latent-State Synchronization for Deadline-Bound Streaming Inference

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Predicting every observation in a high-rate stream requires a model to finish before the next arrival. A large model may offer better predictions but miss that deadline. We propose RESYNC: the large model runs asynchronously, while a fast path updates its latest exact but stale latent state using the observations received since and applies its unchanged prediction head. The updater learns from the frozen model's representations without task labels. Across one real and two synthetic streams and five architecture families, RESYNC beats the best compact or distilled alternative with a paired 95% interval above zero in all twelve dataset-architecture pairs with a material teacher advantage at anchor ages of 4-12 observations, and in eleven pairs at older tested ages up to 32. The share of the teacher's gain preserved remains complete on the real stream but is lower and generally declines with anchor age on the synthetic streams. In live replay with 1 ms arrivals, the synchronous large models cannot deliver any predictions on time while RESYNC serves at least 99.4% of arrivals within the interval, with 0.35-0.62 ms median latency on one CPU thread: around the same speed as or faster than the compact model, and 7.9-17.7 faster than synchronous large-model inference. RESYNC targets settings with background compute available but a strict per-observation response deadline.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
real-time inference, streaming inference, latent representations, time-series forecasting

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_zm8dOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

RESYNC is a slow-fast system. It accelerates deadline-bound streaming inference by combining asynchronous teacher computation with a lightweight latent updater. At each arrival, the updater uses the latest exact but stale teacher state, subsequent observations, and anchor age to reconstruct the current latent, which is decoded by the frozen teacher head.

Critical strengths
  1. The paper distinguishes per-observation response latency, coverage, and prediction quality.
  2. Direct reconstruction from exact anchors avoids recursively propagating approximate teacher states. Incremental suffix encoding and active/candidate states support efficient asynchronous execution.
  3. Experiments demonstrate good fixed-gap prediction quality.
Critical weaknesses
  1. Prediction quality is measured at fixed anchor gaps, separately from live replay. The accuracy of predictions actually delivered within the deadline remains unmeasured.
  2. Insufficient comparison with close alternatives: Slow–fast execution and feature reuse have substantial precedents. The recursive control does not establish superiority over strong anchor-conditioned students or recursively updated models trained through multiple steps.
  3. Limited application validation: The evaluation includes one real stream and two synthetic streams. It remains unclear when every observation requires an independent prediction, rather than aggregation, reduced-rate inference, or event-triggered updates.
Questions for authors
  1. Can you evaluate the exact predictions produced during concurrent replay, jointly reporting on-time quality, coverage, and latency?
  2. What is the best predictive quality attainable by models optimized for the actual deadline, rather than RESYNC’s shorter runtime?
  3. Which real application requires the proposed per-observation service contract, and what downstream benefit does it provide over aggregation or event-triggered inference?
Reviewer_3FVGOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper targets streams where every observation needs its own prediction before the next arrives, but the best model (the teacher) is slower than the arrival rate. RESYNC runs the teacher body asynchronously and, at each arrival, a small GRU+MLP updater maps the newest exact but stale latent, the observations since and the gap nn to the current latent, then applies the frozen teacher head. The updater trains on teacher latents only, and each estimate restarts from the exact anchor. Experiments use FI-2010 and two synthetic streams with five architecture families, with offline macro-F1 at fixed gaps and live replay at 1 to 3 ms arrivals.

Critical strengths

The service contract (one prediction per arrival, D=ΔD = \Delta) is clear and the live replay tests it directly. At 1 ms no synchronous teacher is ever on time, while RESYNC keeps p99≤0.71p_{99} \leq 0.71 ms (Table 2), and Table 3 reports the extra CPU and GPU cost openly. The matched recursive updater (Section 6.3, Table 15) makes a convincing case for restarting from the exact anchor, with latent error 0.27 vs 0.69 at n=32n = 32.

Critical weaknesses

I am not sure the gain comes from the anchor. The compact models have 0.8k to 76k parameters against 0.20 to 0.68M for the updater (Table 6) and were not chosen as the largest models meeting the deadline. Capacity matters a lot here, since in Table 17 an 18k Continual Transformer reaches 0.919 on Synthetic-1. On FI-2010 the stale anchor scores about 0.39, yet RESYNC is at teacher level from n=4n = 4 and flat to n=32n = 32 (Table 8). That suggests the GRU over recent observations does much of the work. I would want an anchor-free updater and a fast model of the updater's size before crediting the anchor.

The evidence is also thin. There is one real stream, the synthetic ones were built so long context matters, there is one seed, and quality is scored offline at fixed gaps rather than on served predictions (Section 7). Everything is frozen, so drift is untested. Gap-conditioned latent evolution (Rubanova et al., 2019), latency-compensating forecasting (Yang et al., 2022) and delay compensation for stale state (Zheng et al., 2017) should be discussed. Next to Delta Distillation and slow-fast systems, the new part is narrow.

Questions for authors
  1. Can you report the same GRU+MLP without zaz_a, fed the recent observations only, on FI-2010 and Synthetic-1?
  2. What does the largest windowed model per family that meets 1 ms on one CPU thread score?
  3. Why is FI-2010 InceptionTime below the stale anchor at n=1n = 1 when n=4n = 4 already matches the teacher?
  4. Under drift, can the updater be refit online from new anchors without labels, and how fast does it recover?
Reviewer_pGnJOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper aims to tackle streaming classification where every observation needs a prediction before a deadline that a large model cannot meet. RESYNC runs the frozen teacher asynchronously in the background. At each arrival, a small GRU-based updater adjusts the teacher’s latest stale latent using the observations received since, and the teacher’s head makes the prediction. The updater is trained without labels to match the teacher’s fresh latents. On one real and two synthetic streams across five architectures, RESYNC outperforms compact and distilled models where the teacher has a clear advantage. In live replay at 1 ms arrivals, it meets almost every deadline on one cpu thread.

Critical strengths
  • The problem framing is clear and practically motivated. A per-observation service contract, measured with live asynchronous replay, coverage, deadline misses, and resource use, is a more honest evaluation than offline accuracy alone. The paper reports failure cases and limitations openly.

  • The comparison between direct re-synchronization from the newest exact anchor and a matched recursive updater is clean and convincing. It shows that recursive updates accumulate latent error, sometimes ending worse than not updating at all, while direct updates stay bounded across anchor ages.

Critical weaknesses
  • The compact baselines are much smaller than the updater and much faster than needed. The compact models have about 1k to 76k parameters and run well under the deadline, while the updater has 0.2M to 0.68M parameters. The relevant baseline is the strongest model that still meets the deadline, not a fixed tiny one. The paper’s own incremental study shows that models one size up, with only 18k to 32k parameters, already match or exceed RESYNC on Synthetic-1.

  • A key ablation is missing: the updater without the anchor. On FI-2010, the stale teacher collapses within a few observations while RESYNC stays flat from age 4 to 32, and RESYNC is weak at age 1 when the suffix holds only one observation. Both patterns suggest that much of the performance may come from the GRU encoding of recent observations rather than from the stale teacher latent.

  • A streaming GRU predictor of the same size as the updater is also missing. Such a model is constant cost per step, meets the deadline without a GPU, and can be trained with labels or distilled from the teacher. Without it, it is unclear whether the asynchronous teacher adds value beyond a well-sized recurrent model.

  • The comparison of compute and hardware is asymmetric. RESYNC uses a CPU thread plus a GPU that is busy 85 to 93 percent of the time running the teacher, whereas the compact model uses only a CPU thread. The synchronous teacher latencies of 4 to 8 ms at batch size one also come from plain PyTorch implementations, including a quadratic Mamba-2 scan. An optimized teacher could plausibly close much of the gap, so the deadline regime appears partly constructed.

  • The evidence rests largely on synthetic streams designed to require long context, with only one real stream. On that real stream, RESYNC exceeds the teacher because of a class-frequency shift between training and test days. This makes it harder to judge how often the targeted regime occurs in practice.

  • Quality is measured offline at fixed anchor ages, separately from live replay, so the paper never reports the quality of the predictions actually delivered on time. In addition, slower streams mostly produce an anchor age of two, which the main fixed-age evaluation does not cover.

  • Gains are uneven on the synthetic streams. The share of the teacher’s advantage retained declines with anchor age, falling to 0.16 for the Synthetic-2 Transformer at age 32. RESYNC also falls below the distilled compact model on the Synthetic-1 Transformer from age 18 onward.

  • Novelty is limited. Slow-fast execution, head reuse, and learning teacher feature differences are established in prior work, as the paper acknowledges. The recursive baseline is a simplified reimplementation at the same split rather than Delta Distillation itself.

Questions for authors

please see questions outlined in weakness

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.