acceptodds
Under review as a conference paper at ICLR 2027

Learned Prefetching with Delayed Utility

Abstract

While learned sequence models have proven highly effective at capturing complex memory access patterns, they are conventionally framed as short-horizon predictors and evaluated under idealized, instant-fill assumptions. The rapid shift toward disaggregated memory topologies—where memory is pooled across high-latency fabrics—fundamentally breaks this paradigm. In these environments, higher and variable latencies mean that timeliness dictates system performance: a highly accurate prediction that arrives too late fails to hide latency, while an early prediction wastes limited bandwidth and pollutes the cache. To address these disaggregated constraints, we argue that prefetching must be recast from next-step prediction to a delayed-utility learning problem. We propose a multi-horizon sequence model that predicts across several future lead times, exposing a richer set of candidate accesses tailored to the deployment's specific latency regime. Evaluated under a rigorous, stall-aware framework using real-world memory access traces, we demonstrate that standard single-horizon models frequently fail in high-latency environments, whereas our multi-horizon approach successfully captures the deployment-dependent utility of different lead times at a modest inference cost.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
Prefetching, Caching, Delayed Utility, CXL, LSTM, GNN

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_fB4qOpenReview ↗
Rating
4
Confidence
2
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

The paper's target is learned memory request prediction, where a neural network with small size is trained to predict future addresses that will be requested in a program. The paper's main claim is the fact that the accuracy of the prediction is not the only metric to care for; the timeliness of the prediction is just as important if not more, for if a prefetch request lands in the cache too early it will lead to premature cache eviction, and if it lands too late it may not be useful (because the actual memory request has already stalled).

The paper mentions that most learned based predictors target a single horizon: they predict the address requested by a fixed number of instructions ahead. This limited horizon of prediction breaks the timeliness of the prediction, a feature that early heuristic-based cache prefetching methods emphasized on. As a solution, the authors propose to predict multiple horizons of future requested addresses. To enable an efficient implementation, the authors propose a shared LSTM trunk which takes in a sequence of past memory access patterns as input, and generates a hidden representation vector as output. The hidden representation is then utilized by multiple linear layers (heads), each of which predicts the memory address accessed in a certain time horizon.

Critical strengths
  • I sincerely appreciate the inclusion of section 2, preliminaries, in the paper. It helped me a lot in understanding the scope of the paper. In addition, the writing of the paper is really good, with a clear structure that follows a purpose: to help the reader understand the problem at hand. Reading the paper I realize that the authors respect the readers time. I appreciate and encourage the authors to continue this style of writing.

  • The authors distinguish betweeen the long horizon cost minimization (J(pi)) and individual horizon prefetch predictors, which helps formalize the proper learning objective. They also explain clearly why the overall task is not a supervised task.

  • Experimental results are solid, the multi horizon prefetching method shows superior hit rate and lower delay compared to the related work.

  • The design of the LSTM input/output streams are reasonable

Critical weaknesses
  • I could not understand how the authors connect prefetch delay to the cost function J(pi). More specifically, does the delay estimation require the authors to know which cache layer the requested memory block resides in? Or the delay is simply simulated through the dataset/simulator mentioned in the experiment section? Please clarify this in the paper.
  • I could not connect the learning objective to the optimization of the cost function J(pi). To my understanding, the LSTM + heads training learns a number of prefetch address predictors. How is J(pi) backpropagated through this setup?
Questions for authors
  • Is the set of requested addresses (offseted by the current program counter) dynamically updated during inference? if yes how? If no, please indicate that in the paper.
  • Have you entertained using a transformer block with limited-horizon causal attention rather than the LSTM?
3 comments

We thank the reviewer for reviewing our paper and providing valuable feedback.

Q1: No, we do not dynamically update the offsets during inference. We will indicate it in the paper.

Q2: We will include the Transformer based results in the paper. The transformer based model gave roughly the same performance as multihead while being a bulkier model and requiring more inference time.

Area_Chair_D1g8OpenReview ↗

Author-reviewer discussion period is ending soon

Dear Reviewer fB4q,

If you haven't done so, please read through authors' response and let authors know if you need further clarification or have more questions. Beginning 11/19, authors are not allowed to respond any more, so this is good time to review the response and ask more.

Thank you.

-AC

Reviewer_fB4qOpenReview ↗

keeping my rating

authors provided sufficient response. I am keeping my score unchanged.

Reviewer_p28QOpenReview ↗
Rating
2
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes modeling cache prefetching as a delayed-utility problem instead of a simple next-access (or next-delta) prediction problem. This formulation assumes that prediction accuracy alone is not enough for a prefetcher to perform well in real-world scenarios, as it does not account for delays, especially in CXL-backed memory systems. A good prediction must be both timely and correct to be useful in practice.

The authors introduce a new objective function to evaluate prefetching policies: it considers the total cost of each access, which increases when a miss or a prefetch results in addresses not yet available in the cache. The objective of a policy is the sum of the total cost across each timestep during runtime. Ideally, an optimal prefetching policy should minimize this objective.

The authors then propose an LSTM-based architecture with a multi-head (shared-trunk) design to predict the cumulative delta n steps ahead across different lead times, one per head. The heads share a single hidden state. Each head of the model is trained by minimizing the cross-entropy between the prediction at that specific lead time and the true cumulative delta. The per-head prediction is then aggregated and scored over the lead time horizon using exponential discounting, and the top-k scores are extracted. The final result is a candidate set that will be prefetched.

The proposed architecture is then compared with several baselines across five different workloads, measuring both the hit rate in a simplified delay model, with delay counted in accesses, and the stall-aware runtime, which considers the full objective. The results of the evaluation show that the shared-trunk approach achieves a higher or comparable hit rate and lower total cycles compared to the single-head LSTM and the traditional baselines on most traces.

Critical strengths
  • S1. The idea of modeling cache prefetching as a delayed-utility problem is interesting; the evaluation gap illustrated in this paper is relevant, and the proposed objective is valuable for future work in this area.

  • S2. The evaluation is broad and very detailed, making reproducibility easier.

  • S3. The preliminaries section is very well written, even for readers who are not familiar with that specific research area.

Critical weaknesses
  • W1. The architecture discussed in section 3.2 and then used in all the experiments is based on the LSTM architecture. Recent work demonstrates that transformers can achieve higher performance on sequential data. The authors acknowledge that some transformer-based solutions do exist (last paragraph of the appendix "Machine learning based prefetchers"), but there are no experiments or baselines with this specific architecture, nor a discussion of why the LSTM has been chosen instead.

  • W2. The inference cost of the multi-head model is not discussed or empirically analyzed in the experimental section of the work.

  • W3. The architecture described in 3.2 doesn't take into account the objective discussed in 3.1: neither delay nor cadence is considered. The authors could perhaps expand the section that acknowledges this (the paragraph "Why this is not the supervised task"). Even in the scoring method, none of these constraints are taken into account to obtain the candidate set. It is not very clear how the model described in 3.2 is related to the objective discussed in 3.1, rather than to the evaluation.

Questions for authors
  • Q1. Is there a specific reason why the authors didn't choose a transformer-based architecture or didn't use TransforMAP as a baseline? If yes, this must be justified; otherwise, it must be included in the comparison.

  • Q2. What is the cost of the model in terms of latency and parameter count?

4 comments

We thank the reviewer for reviewing our paper and providing valuable feedback.

Q1: We will include the Transformer based results in the paper. The transformer based model gave roughly the same performance as multihead while being a bulkier model and required more inference time.

Q2: The inference latency is fairly high compared to the amount of time required to serve a cache miss. We observed in our simulations that plain LRU without any prefetching is still faster than any of the machine learning based models.

Area_Chair_D1g8OpenReview ↗

Author-reviewer discussion period is ending soon

Dear Reviewer p28Q,

If you haven't done so, please read through authors' response and let authors know if you need further clarification or have more questions. Beginning 11/19, authors are not allowed to respond any more, so this is good time to review the response and ask more.

Thank you.

-AC

Reviewer_p28QOpenReview ↗

Comment to the Authors' Response

I thank the authors for their response, but, essentially, my concerns have not been addressed. Therefore, I keep my score unchanged.

We did test a Transformer based model, but decided against using it as LSTM was observed to be the more robust of the two across traces and is cheaper per decision. We will add this comparison to the appendix. We are happy to address any concerns the reviewer might have.

Mean stall-aware cycles over all ten traces (×103\times 10^3, lower is better)

Prefetcherd=10d=10d=30d=30d=50d=50d=100d=100d=150d=150d=200d=200d=300d=300
LRU (no prefetch)150.1390.2630.41230.81831.22431.63632.4
Stride68.0145.4224.4422.3620.2818.01213.8
LSTM-Step5089.3208.9329.8633.1936.41239.91849.0
Multihead 50–70, LSTM (ours)46.780.2113.9208.3304.4400.4592.4
Multihead 50–70, Transformer54.9104.9155.0290.9428.6566.3841.7

Per trace (×103\times 10^3 cycles)

TraceLSTM, d=10d=10Transformer, d=10d=10LSTM, d=300d=300Transformer, d=300d=300
blender ASIC33.232.6153.8138.4
blender FPGA32.932.9143.1148.0
imagick ASIC99.6136.12119.33213.9
imagick FPGA90.4151.41856.83670.8
lbm ASIC30.330.2130.6129.9
lbm FPGA30.330.4131.9131.7
parest ASIC36.337.3326.3342.6
parest FPGA35.834.7310.4275.6
povray ASIC35.931.8292.2180.8
povray FPGA42.632.1459.5185.5

We measured the cost of one prefetch decision for every learned model in the paper on all ten traces. For reference, we also timed an ensemble of 20 independent single-horizon LSTMs (horizons 51–70): this is the obvious way to get the same horizon coverage without a shared trunk.

Latency per prefetch decision (mean over the ten traces; range across traces in parentheses)

ModelHeadsParameters (mean)GPU (ms)CPU (ms)GPU cost vs. Step50
LSTM-Step50122.7M0.71 (0.67–0.74)3.4 (2.4–4.2)1.0×
Multihead 50–55 (ours)648.1M1.23 (1.05–1.32)13.2 (7.4–17.3)1.7×
Multihead 30–50 (ours)21123.1M2.92 (2.46–3.17)40.4 (22.0–52.5)4.1×
Multihead 50–70 (ours)21125.0M2.95 (2.48–3.25)41.2 (21.6–53.0)4.1×
Multihead 50–70, Transformer trunk21125.1M3.32 (3.14–3.46)41.6 (22.0–52.8)4.7×
Ensemble of 20 single-horizon LSTMs (51–70)20 × 1455.0M11.70 (11.11–12.15)66.7 (48.9–77.5)16.5×
Reviewer_4BWCOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper argues that learned prefetchers should be judged by whether correct predictions arrive in time, not by next-step accuracy. It defines a trace-driven evaluation with fill delay dd, decision cadence τ\tau (decisions at t∈{τ,2τ,… }t \in \lbrace \tau, 2\tau, \dots \rbrace, at most kk prefetches each), and a fully associative 1000-entry LRU cache. The per-access cost c(t)c(t) is 1 on a hit, 1+d1+d on a clean miss, and 1+rt1+r_t if the line is still in flight; the objective is J(π)=∑t=1Tc(t)J(\pi) = \sum_{t=1}^{T} c(t).

The model extends the LSTM delta predictor of Hashemi et al. (2018) to a shared trunk with one softmax head per horizon n∈{50,…,70}n \in \lbrace 50, \dots, 70 \rbrace. Each head contributes its top-10 deltas, candidates are scored as score(a)=∑nγrank(n)pn(a−at)\mathrm{score}(a) = \sum_n \gamma^{\mathrm{rank}(n)} p_n(a - a_t), and the top k=20k=20 are issued. Evaluation uses five SPEC CPU2017 traces × two CXL configurations from CXL-DMSim, against LRU, Stride, Berti-like, Best-Offset, a single-horizon LSTM-Step50 and an SGDP-style GNN. Multihead 50–70 reports 47.7–67.1% fewer simulated cycles than LSTM-Step50, and a single-head ablation shows each head alone costs roughly 22–3×3\times the pooled model.

Critical strengths
  1. The question is relevant: with higher, variable CXL latency, timeliness matters as much as correctness, and reporting stall-aware cost alongside hit rate is a reasonable step.
  2. The single-head ablation is budget-matched (m=k=20m=k=20), and the paper is open about the traces where the method loses (imagick, parest).
Critical weaknesses
  1. The central comparison is biased by construction. At cadence τ\tau, LSTM-Step50 targets one future position at+50a_{t+50} per decision, so it can cover at most a 1/τ1/\tau fraction of demand accesses (1/10 or 1/30 here). Multihead 50–70 targets 21 positions at+50,…,at+70a_{t+50}, \dots, a_{t+70} per decision. The reported gain, and the head ablation, mainly measure how many future positions are targeted, not the value of lead-time diversity. The learned baseline is also weak: plain Stride beats LSTM-Step50 at every delay (Table 1). Fair baselines would include a predictor invoked every access or rolled out over a window, direct set prediction of the next WW accesses, and classical prefetchers with tuned degree and distance (Best-Offset and Berti-like are run but not shown).
  2. The simulator cannot support the CXL and systems claims. There are no cache lines (every distinct byte address is its own "line"), a hit costs one cycle, misses never overlap, and bandwidth, queueing and memory-level parallelism are not modeled, so the motivating costs (wasted bandwidth, pollution) are absent. dd is swept over 10–300 abstract cycles unrelated to the 154/244 ns trace configurations. Results are single deterministic runs on five benchmarks with 30k–50k evaluation accesses, each model is trained on the trace it is tested on, and no inference latency or storage cost is reported despite the "modest inference cost" claim. There is no comparison with established learned prefetchers (Voyager, TransforMAP, Pythia) or a standard simulator such as ChampSim. On the ML side, adding horizon-specific heads is a small change for ICLR.
Questions for authors
  1. If the single-horizon model is invoked every access, or spreads its kk candidates over horizons so both models target the same number of future positions, does the multi-horizon advantage remain? (Directly affects the main claim.)
  2. How do the results change with 64B line granularity, a set-associative cache and overlapping misses, e.g. in ChampSim? (Affects soundness.)
  3. Where would this model run (core, OS, CXL controller), and what is its per-decision latency relative to dd? (Affects significance.)
  4. Section 5.2 states a 70/30 split, but footnote 3 says all policies are evaluated on the final 10% of each trace. Which is correct? The A.2 aggregate (46.0 / 571.4 ×103\times 10^3 cycles) also does not match Table 1 (46.7 / 592.4). (Clarification.)
  5. Please report Best-Offset and Berti-like in Figures 3–5 and Table 1. (Clarification.)
4 comments

We thank the reviewer for reviewing our paper and providing valuable feedback.

We agree that TransforMAP, Pythia, and Voyager are important reference points for learned prefetching. Our study asks whether predicting several future memory accesses helps a prefetcher choose data that will be ready when the program needs it. We compare predictors on the same traces, at the same decision points, with the same limit on how much data they can request.

The reviewer raises an important concern. Our multi-horizon model predicts more future positions than the original single-horizon baseline. To test how much that matters, we compared our model with a group of single-horizon predictors that together cover the same future positions. We will clarify that our results measure performance under a modeled memory delay. Establishing an end-to-end speedup on CXL hardware requires a separate systems evaluation.

Q1: Yes, multi-horizon still beats the ensemble single horizon model targeting the same number of future positions by 3-4% on every trace other than parest ASIC where it loses by 1.5%. The ensemble model also uses 3.6x the parameters and 4x more inference time per decision compared to multihead.

Q2: With 64B granularity, the advantage of multihead remains with respect to non learning based prefetchers while increasing the advantage over Stride, but the advantage decreases slightly when compared to Step50 LSTM.

We agree that our evaluation is a trace-driven model, not a full-system simulation, and that it leaves out several real effects. We will say so clearly in the paper. These simplifications are deliberate. The paper's question is narrower than a full CXL systems evaluation.

Q3: It is not so straightforward to model the delay latency. Which is why we chose a wide range of delay values to show our model's robustness.

Q4: All policies are evaluated on the final 10% of each trace, but only the first 70% of the trace was used for training. The A2 aggregate (46.0 / 571.4 cycles) is a typo and will be fixed.

Q5: Best Offset and Berti-like always trailed Stride, which is why they were not included in the tables. We will add them to the appendix.

Area_Chair_D1g8OpenReview ↗

Author-reviewer discussion period is ending soon

Dear Reviewer 4BWC,

If you haven't done so, please read through authors' response and let authors know if you need further clarification or have more questions. Beginning 11/19, authors are not allowed to respond any more, so this is good time to review the response and ask more.

Thank you.

-AC

I thank the authors for the matched-coverage ensemble experiment, which is the right control. However, it shows that with coverage equalized, the multi-horizon gain falls to roughly 3–4% (and is negative on parest ASIC).

This indicates that most of the 47.7–67.1% improvement reported in the paper comes from targeting more future positions rather than from lead-time diversity.

The headline claims and abstract would need substantial revision, and the contribution becomes mainly an efficiency argument (similar accuracy to an ensemble at lower cost).

Remaining questions which need the paper address them in future in clearer version:

(1) Report the per-trace numbers for the ensemble comparison and the 64B-line experiment, including inference latency per decision in absolute terms. (2) Q3 is unanswered: where would the model run, and how does its inference latency compare with the delay range considered? Without this, the timeliness claims cannot be assessed. (3) Were the ensemble and multihead results averaged over multiple seeds? Given a 3–4% margin, variance matters. I am raising my score from 1 to 2 for the added control and clarifications, but the core claims still need revision and the evaluation remains too simplified for acceptance.

Overall, the paper addresses an important and practical problem, but the experiments do not reflect the impact of the idea, and some experimental designs need rethinking. I suggest filling these gaps and writing a clearer version with a clearer setup for submission to a future venue.

We thank the reviewer for their feedback.

The model is meant to run in host-side software, as part of the memory-tiering runtime that decides which pages to bring from CXL-attached memory into local DRAM. It is not meant to run in the processor's cache hierarchy as a hardware line prefetcher. A second option is to run the model on the CXL device, in a near-memory controller with compute that observes the device's access stream directly. This would need a smaller model than our prototype.

What our current numbers do and don't show. Our latency measurements (W2) come from an unoptimized PyTorch prototype at batch size 1: 2.95 ms per decision on a GPU, 41 ms on a CPU. They show the relative cost of the design choices: 4.1× a single-horizon LSTM, against 16.5× for an equivalent ensemble. They do not show the latency of a deployed implementation. Most of the cost is in the 21 output heads (softmaxes over up to 50k deltas), not the trunk. The main ways to shrink it are therefore smaller per-head vocabularies, fewer heads (6 heads already cost 1.23 ms and still beat the single-horizon baseline at every delay), quantization, and distillation. We do not claim the current prototype runs at line rate. Our contribution is the training objective and the delay-aware evaluation, and both carry over to a compressed model.

We measured the cost of one prefetch decision for every learned model in the paper on all ten traces. For reference, we also timed an ensemble of 20 independent single-horizon LSTMs (horizons 51–70): this is the obvious way to get the same horizon coverage without a shared trunk.

Latency per prefetch decision (mean over the ten traces; range across traces in parentheses)

ModelHeadsParameters (mean)GPU (ms)CPU (ms)GPU cost vs. Step50
LSTM-Step50122.7M0.71 (0.67–0.74)3.4 (2.4–4.2)1.0×
Multihead 50–55 (ours)648.1M1.23 (1.05–1.32)13.2 (7.4–17.3)1.7×
Multihead 30–50 (ours)21123.1M2.92 (2.46–3.17)40.4 (22.0–52.5)4.1×
Multihead 50–70 (ours)21125.0M2.95 (2.48–3.25)41.2 (21.6–53.0)4.1×
Multihead 50–70, Transformer trunk21125.1M3.32 (3.14–3.46)41.6 (22.0–52.8)4.7×
Ensemble of 20 single-horizon LSTMs (51–70)20 × 1455.0M11.70 (11.11–12.15)66.7 (48.9–77.5)16.5×

We did run multiple seeds (5) for the learned prefetcher models. The variance between runs was always withing 0.01% which is why we report single seed results.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.