acceptodds
Under review as a conference paper at ICLR 2027

Remember or Recite? Pricing State Restoration for On-Device Multi-Agent LLM Inference

Abstract

Multi-agent large language model (LLM) workflows are moving onto personal devices, where several roles share one base model through low-rank adaptation (LoRA) adapters and take turns, each with its own key–value (KV) cache. Device memory holds only a few such sessions, so evicted state must be written to flash or recomputed later, a cost that dominates the workflow's latency and energy. Existing serving systems price this choice as data centers do, assuming symmetric memory and independent requests. Our measurements show that both assumptions fail on a device: flash writes over 130× slower than it reads, and the workflow graph already says which turns are ready, mergeable, and finished, so much state need never be written. We propose Recite, an on-device runtime that turns these two facts into three closed-form rules: a per-tier save-or-recompute threshold in which session length cancels, liveness read off the workflow graph, and batched decoding of ready same-role turns under a memory-traffic ceiling. From traces and device constants alone, the rules predict opposite decisions on the two flash tiers of one board and where batching hurts. On Jetson boards with two model families, Recite cuts debate and voting makespan 2.1–2.5× and energy 2.3–2.7× at equivalent accuracy against the strongest of five reference configurations, and runs up to 6.8× faster than the stateless ones across all workloads. Where both arms of a decision are measured, the rule picks the faster arm every time, and the cost model predicts the restore share of held-out configurations with R² = 0.99.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
on-device LLM inference, multi-agent LLM workflows, KV cache, state restoration, LoRA serving, edge devices, batched decoding, energy efficiency

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_1WJbOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper presents RECITE, a runtime for multi-agent LLM workflows on memory-limited devices. It decides whether evicted KV state should be saved or recomputed, uses the workflow DAG to identify finished sessions, and batches ready turns that share the same role. Experiments on recorded traces and live workloads show substantial improvements in makespan and energy for parallel workflows, but little benefit when batching is not possible. I think the main contribution is how these mechanisms work together under the high cost of flash writes, rather than a new agent model or training method.

NB: The below is not really a summary anymore but rather comments on the work for the authors; I don't know where else to put them but here.

The most interesting contribution is the interaction between batching and slow flash writes. RECITE's device-specific integration is useful, but the underlying ideas of KV recomputation and workflow-aware management have clear precedents. I see it more as a systems integration contribution than a fundamentally new scheduling method, and would like stronger evidence for the restoration rule's independent value.

It was surprising to see a throughput optimization create more blocking writes, and the experiments explain this quite well for batching on eMMC. However, Cake has already studied loading versus recomputation, while KVFlow uses workflow information for cache management. I am still not fully sure how much of RECITE is new compared with these earlier systems. The paper could explain this difference more clearly. The headline speedups are difficult to interpret. As the restore-capable baselines already use adaptive restoration, these numbers mainly reflect batching and liveness. I found the capability-matched experiments more useful. The tier rule improves performance on the GUI workload using eMMC, but slightly hurts performance on the debate workload with the same storage. This distinction deserves more attention.

In the debate experiment, the cost model predicts an improvement, but the actual result is a slight regression. The authors explain that page-cache buffering makes small writes cheaper than expected, which seems reasonable. Table 8 reports that the rule chooses the faster option in every paired comparison there, but the debate result suggests this may not always hold. I wonder whether different write sizes or cache conditions would lead to more such cases.

The experiments reserve four KV slots, each sized for 32K tokens. Shorter sessions may leave some of this reserved memory unused. A byte-budgeted pool could reduce the number of evictions, and I am curious whether RECITE would still achieve similar improvements under this allocation strategy.

I would also like to see a complete phone workflow rather than only storage microbenchmarks. The live Jetson experiments help, but most main tests still use recorded traces and shape-only adapters. Testing with trained role-specific adapters on a phone would make the practical contribution more convincing. Wider batching also increases tail latency, which may be a problem for interactive agents even when the overall makespan improves.

Critical strengths

Batching alone makes the system slower, but becomes beneficial when combined with liveness and the tier rule. This interaction is well demonstrated.

I like how RECITE decides whether to save or recompute an evicted KV cache by comparing their estimated costs. The workflow DAG also helps identify finished sessions whose KV caches can be discarded directly. Both mechanisms are simple and do not require learned predictors.

On GSM8K, restoring KV or sharing a prefix has little effect on accuracy. HumanEval+ shows similar results. The authors also include the GUI and linear workloads, even though RECITE offers almost no speedup in these cases.

Critical weaknesses

The headline speedups are measured against baselines that already use adaptive restoration, so they do not isolate the new tier rule. Its independent benefit also varies considerably between GUI and debate workloads.

The experiments reserve four KV slots, each sized for 32K tokens. When sessions are shorter, some of this reserved memory may go unused. A more flexible allocation could reduce evictions and potentially change RECITE's measured benefits.

The restoration rule does not always make the better choice. For the debate workload on eMMC, its predicted gain becomes a slight regression because of page-cache buffering. The authors report correct decisions in every paired comparison in Table 8, but these tests do not cover this case. The high R² also measures restoration-share prediction rather than decision optimality.

Questions for authors

Not applicable

4 comments
AuthorsOpenReview ↗

Response to Reviewer 1WJb (1/3)

Thank you for the constructive review. Each point below led to a new experiment or a correction in the revision. The revised PDF marks all substantive changes in blue. The experiments are in Appendix J, and for most of them we registered predictions before the runs. Table 21 lists every registered prediction with its verdict, including the misses. Analyses made after seeing data are labeled post hoc. They are the burst-aware variant of J.1, the batching fix of J.4, and the latency budgets of J.6.

W1. Against a capability-matched reference the rule alone gives 1.46–1.65× on gui-5, 1.42× on gui-7, and 0.98× on debate. We agree that the headline factors over Hot-swap combine batching, liveness, and the rule. A capability-matched reference isolates the rule. It batches, drops finished sessions, and saves every live eviction. The table below gives three runs per arm on eMMC (Appendix J.1, Table 15).

WorkloadSave reference (s)Recite (s)Rule alone
gui-5371.7255.11.46× (1.65× in the run of Section 4.3)
gui-7510.9359.91.42×
debate688.7704.00.98×

On NVMe the rule always saves, so it coincides with the reference by construction. Neither gui-3/eMMC nor voting evicts a live session. The rule's own gain is therefore on eMMC cells that evict long sessions, while the 2.1–2.5× on voting and debate comes from batching and liveness. On gui-5 the two factors differ in the save arm, which took 426.5 s in the run of Section 4.3 and takes 371.7 ± 15.4 s in the rerun.

The GUI–debate difference comes from the size of each write. On gui-5 one live eviction writes 2362 MB. That write is synchronous and takes 127–160 s, so replay is cheaper. On debate each eviction writes about 53 MB. The page cache absorbs it, so saving is nearly free and the device-rate rule over-prices it.

W2. With a byte-budgeted pool the rule matters when memory is tight, and batching and liveness carry the gain when it is not. We replaced the four slots with a pool on the same board (Appendix J.2, Table 16).

  • At 18 GiB no workload evicts a live session and save-everything writes nothing, so the rule never fires. Hot-swap takes 2.90× as long as Recite on voting, 2.48× on debate, and 1.00× on gui-5.
  • As the pool tightens, writes return and the rule matters again. At 4 GiB the capability-matched reference takes 2.67× as long as Recite on gui-5 and 2.56× on gui-7, and at 8 GiB it takes 1.12× on gui-7.
  • With a pool, batching alone is 1.56× faster than Hot-swap at 4 GiB although it still writes 12.0 GiB. The 0.76× slowdown of batching alone is therefore specific to slot allocation, and Section 4.3 now says so. On this runtime the pool is also slower than slots in absolute terms. The same arm takes 1.23–2.12× as long, because a unified KV buffer makes each decode step attend over the whole occupied span. We therefore keep slots as the default and report the pool as a sensitivity result.
AuthorsOpenReview ↗

Response to Reviewer 1WJb (2/3)

W3. Decision quality is now counted separately from R². The rule is right in 15 of 16 cells and at 110 of 184 sweep points. We agree with both remarks. R² measures only how well the model predicts restore share. The abstract now says that where both arms of a decision are measured the rule picks the faster arm in 15 of 16 cells, and the one miss costs 1.8%. It also states that a sweep of 184 single evictions maps the region where the rule is conservative.

Different write sizes and cache states do produce more cases like debate. In the sweep we timed both arms at each eviction on eMMC (Appendix J.1, Table 14). The grid covers bursts of 53 MiB to 4.4 GiB and dirty-page budgets D of 64 MiB, 256 MiB, 1 GiB, and the default, which absorbs 980 MiB on our board. Evictions are 0, 8, or 23 s apart.

  • When the burst plus the existing backlog is at most D/2, saving was faster at 54 of 56 points. When the burst is at least 2D, recomputing was faster at all 42.
  • The device-rate rule always recomputes on eMMC. It picks the faster action at 110 points and errs at 74. Every error recomputes where saving was faster, and none occurs once the burst reaches 2D.
  • A burst-aware variant charges only the part of the burst above the cache's headroom. It picks the faster action at 150 points and errs at 34. Of these 34 errors, 33 save where recomputing was faster, because the measured crossover arrives at 0.29–0.71 of the burst it predicts.
  • The interval between evictions matters because it lets the backlog drain. Over the 16 settings measured at all three intervals, saving was faster at 38%, 44%, and 73% of points for gaps of 0, 8, and 23 s.

End to end against the capability-matched reference (Table 15), the burst-aware variant removes the debate loss at 1.002×, keeps gui-5 at 1.46×, and trails on gui-7 at 1.36× against 1.42×. We proposed the variant after seeing debate and registered its predictions only for debate and gui-5. A single wrong eviction costs more than these end-to-end numbers suggest. Its median penalty is 6.25× for the device-rate rule, but one eviction is a small part of a workflow.

C1. Cake decides how to read saved state, and KVFlow manages state in memory. Recite decides whether to write it, and on eMMC it chooses the opposite action. Cake decides on the read side how to divide a saved prefix between loading and recomputing. It never decides whether to write state at eviction. KVFlow uses the agent step graph to set eviction priority and to prefetch. It keeps state in GPU or CPU memory, where both directions of a swap cost about the same. It does not drop state unwritten. Recite prices the write through the tier's write–read asymmetry α = r_r/r_w and takes the return probability q from terminal events.

In the limit α → 0 with q = 1, the write costs nothing and Proposition 2 reduces to the read-side comparison. Our rule therefore agrees with a read-side policy on NVMe (α = 2.8) and on the phone's UFS (α = 1.01), where both save. On the 32 GB board's eMMC (α = 132) the write term decides. A read-side policy cannot close that gap even with free reads. Save-everything on gui-5/eMMC spends only 2.8 s of its 543.1 s reading, so it needs at least 540.3 s against 259.0 s for Recite, a factor of 2.09×. On debate/eMMC at width four the bound is 2266.5 s against 727.8 s, a factor of 3.11×. These two save-everything arms also write finished sessions, so their gaps are wider than the rule's own factors in W1. The revised Section 2 states this lineage.

Each ingredient has precedents, and Section 3.4 says that our batching is standard continuous batching. Neither Cake nor KVFlow prices the write, and that term is what reverses the decision on eMMC. The paper also measures how the ingredients interact on a device whose storage tier is two orders of magnitude asymmetric, with rules simple enough to predict the outcome from traces and device constants. Our answers to W1 and W3 give the evidence for the rule's own value.

AuthorsOpenReview ↗

Response to Reviewer 1WJb (3/3)

C2. On a phone with trained role adapters, Recite is 1.27× and 1.18× faster on the CPU backend and slower on the GPU backend. We ran voting and debate end to end on a OnePlus 15T with a Snapdragon 8 Elite Gen 5 and UFS storage (Appendix J.5). The phone runs Qwen3-4B with the trained planner, critic, and coder adapters as voter, judge, and debater. Inference, KV state, and storage stay on the phone, and the control plane runs on a laptop over USB.

  • On the CPU backend Recite finishes five voting questions in 5574 s against 7053 s for Hot-swap, a factor of 1.27×. It finishes five debate questions in 9323 s against 10979 s, a factor of 1.18×. The registered point estimates were 1.26× and 1.17×. Restoration takes under 0.02% of turn time on UFS.
  • The phone's GPU backend decodes four merged streams at only 0.70× the aggregate rate of one in the current llama.cpp OpenCL kernels. End to end there, Hot-swap takes only 0.87× as long as Recite on voting and 0.77× on debate, inside the registered ranges.
  • Limits: one seed per cell, five CPU questions, and three GPU questions for voting and two for debate. Energy was not measured.

The main matrix uses shape-only adapters because it measures time, bytes, and energy, and these depend on adapter shape. Table 4 measures the base model's accuracy. The phone runs, the code-repair loop (J.4), and the fused-adapter study (J.3) all use trained adapters.

C3. A P95 turn budget now maps to a batch width, and we corrected how the paper counts P95. Appendix J.6 maps P95 turn budgets to widths on debate over NVMe. A budget of 15 s admits width one, 20 s admits width two at 1.62× the speed, and 30 s admits width four at 2.10×. Energy per token falls from 545 to 329 and 240 mJ. Per decode step, the merged step time fits Proposition 3's linear form as 35.4 + 9.72k ms within 6%. While doing this we found that our P95 had counted merged members at zero, because the runtime logs a group's time on its lead turn. The revision now charges every member its group's service time. Width four on debate/NVMe has a P95 of 29.9 s, up from 29.1 s. Table 5's width-four cells rise from 29.5 s and 59.4 s to 30.3 s and 77.4 s. The P95 cost of width two over Hot-swap rises from 3.2 s to 4.1 s. Figures 4c and 9c are redrawn with the corrected P95. On the phone's CPU the P95 turn service time rises 4.38× on voting and 3.23× on debate. The runtime still takes the width as a fixed setting, and an online policy that sets it from a budget is future work.

We hope the isolated rule, the decision counts and their failure region, the pool, the phone runs, and the corrected latency numbers address the soundness concern. We would welcome any further check you think the rule should pass.

AuthorsOpenReview ↗

Follow-up on our response

Dear Reviewer 1WJb,

Thank you again for your review. We posted our response and a revised PDF earlier in the discussion period, and we would like to check whether they address your concerns before the discussion closes.

In short, the revision isolates the rule against a capability-matched reference (W1), replaces the fixed slots with a byte-budgeted pool (W2), and counts decision quality separately from R² (W3). The new experiments are in Appendix J.

If any point remains unclear, we are glad to answer it right away. If the response resolves your concerns, we would be grateful if you could update your assessment.

Best regards,

The authors

Reviewer_a5akOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies multi-agent LLM inference on memory-constrained edge devices, focusing on the cost of saving and recomputing session KV caches after eviction. Its motivation comes from asymmetric flash read/write speeds, session-lifetime information available from workflows, and the interaction between batching and KV eviction.

The authors propose RECITE, which combines three mechanisms: choosing whether to save or recompute KV states based on storage read/write and prefill speeds; discarding completed session states using workflow liveness; and batching ready turns that use the same role adapter to improve decoding efficiency.

The main evaluation uses a Jetson AGX Orin 32 GB and covers voting, debate, and concurrent GUI workflows, measuring completion time, energy consumption, and state-restoration overhead. The authors also provide mechanism ablations, cost-model validation, and quality evaluation, with additional experiments on other Jetson devices, a second model family, live workflows, and a phone storage microbenchmark. In the main matrix, voting and debate achieve approximately 2.47× and 2.10× speedups over the strongest reference configuration, respectively.

Critical strengths
  1. The paper provides useful analysis of the underlying mechanisms. Beyond end-to-end speedups, the ablations show that batching alone can hurt performance on slow flash: its relative speed is 0.76×, increasing to 2.39× when combined with liveness and the restoration rule. This helps explain the interaction between compute scheduling and storage behavior in edge inference.

  2. The evaluation is broad and discusses the method's limitations. In addition to the main performance matrix, the authors examine quality effects, live execution, different devices and models, and cases where predictions fail or mechanisms provide little benefit. These analyses help interpret the measurements.

Critical weaknesses
  1. The applicability of the tier rule under buffered, overlapping I/O remains insufficiently quantified. Section 4.3 explains the debate/eMMC counterexample: page-cache absorption makes saving cheaper on the critical path than the device-rate model predicts, yielding about a 1.8% slowdown. The 40/40 result in Table 11 supports the rule on the measured pairs, but does not establish when it transfers to workflows with different write-burst sizes and compute gaps. A practical criterion for this boundary would strengthen the deployment guidance.
Questions for authors
  1. Regarding Weakness 1: Beyond the reported debate/eMMC case, when do page-cache absorption and write–compute overlap change the preferred action, in terms of write-burst size, dirty-page capacity, and time to reuse? Across such conditions, how often is the tier rule suboptimal, and what is the end-to-end penalty relative to the capability-matched save baseline?
3 comments
AuthorsOpenReview ↗

Response to Reviewer a5ak

Thank you for stating the gap so precisely. The revised PDF marks all substantive changes in blue. New experiments are in Appendix J, and for most of them we registered predictions before the runs. Table 21 lists every registered prediction with its verdict, including the misses. Analyses made after seeing data are labeled post hoc.

We swept the three coordinates you name, and Appendix J.1 gives the details (Tables 14 and 15). We timed both arms at 184 single evictions on eMMC. The grid covers bursts of 53 MiB to 4.4 GiB and dirty-page budgets of 64 MiB, 256 MiB, 1 GiB, and the default. Evictions are 0, 8, or 23 s apart.

Q1, when the preferred action changes. The burst relative to the dirty-page headroom sets the boundary, not the device write rate. On our board the default budget absorbs 980 MiB. When the burst plus the existing backlog is at most half of the budget, saving was faster at 54 of 56 points. When the burst is at least twice the budget, recomputing was faster at all 42. The two time coordinates act differently.

  • The interval between evictions lets the backlog drain. Over the 16 burst–budget settings measured at all three intervals, saving was faster at 38%, 44%, and 73% of points for gaps of 0, 8, and 23 s. The faster action flipped with the interval in 5 of the 16 settings.
  • Time to reuse acts on the read side. After the page cache is dropped, saving costs 0.68–0.70 of recomputing at 377 tokens, 0.73–1.41 at 2K tokens, and 2.98–8.52 at 8K tokens and above.

Q1, how often the rule is wrong and at what cost. The device-rate rule errs at 74 of 184 points, and every error recomputes where saving was faster. A burst-aware variant charges only the part of the burst above the headroom. It errs at 34 of 184 points. Of these 34 errors, 33 save where recomputing was faster, because the measured crossover arrives at 0.29–0.71 of the burst it predicts. End to end against the capability-matched save baseline, the device-rate rule is 2.2% slower on debate, 1.46× faster on gui-5, and 1.42× faster on gui-7. The burst-aware variant removes the debate loss at 1.002×, keeps gui-5 at 1.46×, and trails on gui-7 at 1.36×. We proposed it after seeing debate and registered only its debate and gui-5 predictions. On NVMe the rule always saves and coincides with the baseline. A single wrong eviction costs more than these end-to-end numbers suggest. Its median penalty is 6.25× for the device-rate rule, but one eviction is a small part of a workflow.

W1, a practical criterion. Measure the dirty-page budget once per device, alongside the read and write rates. The two boundaries of Q1 give the criterion. Save when the burst plus the backlog fits in half of the budget, and recompute when the burst is at least twice the budget. Between the two, the device-rate rule is safe and costs at most the replay of a small session, which was 2.2% on debate in our cells. The burst-aware variant gains more but can err toward saving. Appendix J.1 states this boundary, and Section 4.3 points to it.

We would be glad to add any further operating point you consider important.

AuthorsOpenReview ↗

Follow-up on our response

Dear Reviewer a5ak,

Thank you again for your review. We posted our response and a revised PDF earlier in the discussion period, and we would like to check whether they answer your question before the discussion closes.

In short, a sweep of 184 single evictions now maps when the preferred action changes, how often the rule errs, and what each error costs (Q1). It also yields a practical criterion that needs one extra measurement per device (W1). The details are in Appendix J.1.

If any point remains unclear, we are glad to answer it right away. If the response resolves your concerns, we would be grateful if you could update your assessment.

Best regards,

The authors

Thank you for the additional experiments. The sweep and error analysis largely address my concern about buffered I/O and write–compute overlap. The practical criterion also clarifies the rule’s applicability in the tested settings.

Reviewer_TUJMOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper addresses the problem of memory management (KV, base weights, LoRA adapters and session info) for multi-agent on-device agents. They consider two points: write latency of flash memory when KV and other details needs to be moved in and out, and the nature of the workflow. The formalization of the problem is given in Section 3.1.

  • SS sessions on a single device, with DAG as workflows, its nodes are turns.
  • Each DAG consists of turns assigned to a role (=same base model + specific LoRA)
  • Each turn of a session has a memory footprint of: base weight + adapter + bsb_s bytes for total KV cache
  • ϕ\phi is the fraction of turns with their session evicted under the policy
  • reuse distance DtD_t, the number of distinct other sessions run between a session’s consecutive turns

The authors propose RECITE (Figure 1) as a solution.

  • They define a threshold based on which they decide what to do with the session info when it is evicted. This is dependent on type of storage technologies ("tier" in paper's terminology). One of the contributions of the paper is that the threshold. This is captured in Proposition 2 based on comparing α\alpha (cost of writing) and α⋆\alpha^\star (cost of recomputing).
  • Then, RECITE checks the workflow DAG to see whether an evicted session's KV cache will ever be used again. Three actions are defined: save the session, replay it (which is recomputing it as I understand), and fully drop it. This is discussed in 3.3 part of "Liveness first".
  • Finally, when several turns that use the same LoRA role are ready at the same time according to the workflow DAG, they run as one batched decode. This is where the memory-bandwidth creates a bottleneck (memory traffic ceiling). This is discussed in Section 3.4. The first two is about evicted sessions, however the third one is about scheduling the incoming sessions. The method is detailed in Section 3 of the paper accompanied by Lemma 1 which connects the eviction rate to the memory budget and DAG of sessions.

Experiments are given in Section 4. The chosen platform and tiers are explained, the LLM models that were used, and the workloads (debate, vote, gui, linear, GSM8k, etc).

Overall, the paper's contributions are intuitive: for eviction, compare the cost of memory movement and recompute to decide; for decode, see how much one can batch the data by using the same model.

Critical strengths

Grounding the decision for evicted sessions on α\alpha (measuring how much slower this storage writes than it reads) and α⋆\alpha^\star (measuring how much faster reading the KV back is than recomputing it) is simple and intuitive.

Critical weaknesses

In general, the paper is really about optimizing workflows in memory and bandwidth bound scenarios, mainly limited to single shot answer generation tasks with know DAGs a priori. The main ideas for such optimization are not new, and the propositions and lemmas are straightforward. I also think that the writing can be significantly improved. Please see more comments below.

  • The batching idea is straightforward and not novel. Many workflows and providers try to use such idea in their pipeline: see for example multi-LoRA serving.
  • The threshold based on comparing memory movement cost and recompute cost is elegant, but also used in prior works:
  • The tasks are one shot tasks without environment and human interaction, therefore the workflow is static and hard coded. Liveness in real agentic tasks are more difficult to verify. Besides, environment delays, for instance in code generation tasks, create idle sessions that makes the decision on recompute and storage and batching trickier.
  • The paper is difficult to read, although the original idea is not complex. To me, many phrases were convoluted in this paper.
  • Many notations are not properly introduced: Tturn(s)T_{\text{turn}}(s), π\pi , rwr_w, rrr_r , Cπ(s)C_\pi(s), Cadapt(s)C_{\text{adapt}}(s) - equation (1) which is an important part of the paper is not understandable.
  • With the definitions above, Lemma 1 is straight-forward, the eviction ratio ϕ\phi of a session is the frequency of the event that the number of sessions DtD_t exceed the memory budget. This is a simple LRU style, and it is not clear what the novelty is.
  • I feel that the paper can be written in a way that is more understandable. For example, I had difficulty to get the abstract without the full context of the paper. The expressions like "per-tier save-or-recompute threshold", "liveness read off the workflow graph", or "batched decoding of ready same-role turns under a memory-traffic ceiling" are difficult to parse to me, although can ultimately be understood. This pattern repeats throughout the paper. I am not sure if the paper at this stage is presentable.
Questions for authors
  • The recompute decision only applies to KV cache. How can we recompute the other session information? Especially in multi-turn (without user interaction), the cost of environment interaction needs to be included. And I do not think it is possible for episodes with user interaction.
  • How does the finding generalize to other platforms beside Jetson?
  • Please clarify the notations.
  • In the computation the role LoRA is accounted separately. What is the impact if we use fused version of the model?
  • DAG of the workflow is in general not known in advance, and depends on the particular task. Therefore, how can one look at the DAG see if the KV will be reused or not?
6 comments
AuthorsOpenReview ↗

Response to Reviewer TUJM (1/3)

Thank you for the review. We took the writing concerns seriously and changed the revision accordingly. Several points about scope came from what the paper failed to say clearly, not from what the system needs. We answer them with the paper's own evidence and with the new experiments of Appendix J. For most of these we registered predictions before the runs. Table 21 lists every registered prediction with its verdict, including the misses. Analyses made after seeing data are labeled post hoc. The revised PDF marks all substantive changes in blue.

W1 / Q5. The runtime does not need the DAG in advance, and its evaluation already includes online workflows.

  • It reads only ready turns and terminal events. It takes the turns that are ready now and a terminal event when a role's session ends, and it treats any session without a terminal event as live (q = 1). Section 3.1 ("What the Rules Need") and a Scope paragraph in Appendix I now state this, and we removed the phrases that implied looking ahead. In replayed traces the terminal event is the session's last recorded turn, the point at which an orchestrator would retire it. In the live runs the orchestrator emits it.
  • The submission already ran three online settings. The first is live voting and debate on GSM8K on the 64 GB board, with gains of 2.35× and 1.85× at accuracy within ±5 pp (Figure 3c). The second is a live three-agent Android GUI loop with real observation and actuation delays (Figures 4d and 11). The third is a set of GUI traces recorded from real Android episodes.
  • A new code-repair loop decides its shape at run time (Appendix J.4). Planner, coder, and critic adapters solve 48 HumanEval+ problems. Generated code runs against the tests, and failures trigger up to three repairs.
  • The registered run missed because of a batching defect, and a post-hoc fix recovers 1.42–1.47×. In the registered run Hot-swap took 1.04× as long as Recite on eMMC and 1.01× on NVMe, against a registered 1.30–1.85×. Our engine ordered each batch so that llama.cpp split most merged decode steps into several passes. The fix sorts each batch by sequence id. We registered six predictions before rerunning, and the post-hoc factor became 1.42× on eMMC and 1.47× on NVMe, inside the registered 1.30–1.65×. The main matrix is barely affected, at under 1% per cell.
  • Terminal events make no measurable difference in that loop. On eMMC neither arm writes, so they change only which session is evicted. After the fix the arm without them takes 1110.4 s, between the two Recite runs of 1031.9 and 1264.8 s. The two runs differ because sampling changes how many repair rounds a problem needs. Same-seed reruns land within 4.2% of the registered runs.
  • Wrong terminal events are safe. In a test arm the orchestrator reopened problems it had declared finished. Over two runs 9 follow-up turns reached dropped sessions, and all 9 were rebuilt from their text history with no engine error.

W4. As you expected, delay lowers the batching gain. It changes no save-or-recompute decision. In the registered runs on eMMC with 30 s of injected delay between turns, Recite's mean batch width falls from 2.27 to 2.07 (Appendix J.4, Table 20). Hot-swap then takes 0.87× as long as Recite. The save-or-recompute decisions do not change with delay, because the session length cancels in Proposition 2. All 2506 live evictions on eMMC recompute and all 975 on NVMe save.

AuthorsOpenReview ↗

Response to Reviewer TUJM (2/3)

Q1. Only the KV cache needs pricing, and recomputation never re-executes a tool or re-asks a user. The KV cache is the only large state. A session's KV reaches 4.5 GiB at 32K tokens (144 KiB per token on Qwen3-8B), its adapter is read-only and resident, and its metadata is kilobytes. Recomputation replays the session's recorded token history through prefill. Tool and user outputs are part of that history, so an episode with user or environment interaction is recomputable in the same way. In the code-repair loop the orchestrator keeps each session's history as text, including test outputs. A human-in-the-loop session stays live until the user ends it and loses only the drop-without-write optimization.

W3. The three papers compare reading with recomputing. Proposition 2 adds the write, and the write decides on eMMC. Thank you for the three pointers. Cake (Jin et al., 2025), Meng et al. (2026), and the concurrent Zadeh et al. (HotStorage '26) compare reading a saved KV cache back with recomputing it, on servers whose storage paths are close to symmetric. None of them prices the write that saving requires. Zadeh et al. treat stores as background traffic off the critical path. Our Proposition 2 adds two terms: the tier's write–read asymmetry α = r_r/r_w and the probability q that the session returns. In the limit α → 0 with q = 1, the write costs nothing and Proposition 2 reduces to their read-side comparison. Our rule therefore agrees with them on NVMe (α = 2.8) and on the phone's UFS (α = 1.01), where both save. On the 32 GB board's eMMC (α = 132) the write term decides. A read-side policy cannot close that gap even with free reads. Save-everything on gui-5/eMMC spends only 2.8 s of its 543.1 s reading, so it needs at least 540.3 s against 259.0 s for Recite, a factor of 2.09×. On debate/eMMC at width four the bound is 2266.5 s against 727.8 s, a factor of 3.11×. The revised Section 2 cites all three and states this lineage.

W2. We do not claim batching. We claim its memory-traffic ceiling and the case where it hurts. Section 3.4 says that merging is standard continuous batching. First, Proposition 3 bounds the speedup by memory traffic. The bound is 1.99, 2.96, and 3.91× at widths two to four, against a measured 1.94, 2.35, and 2.52×. Second, on slow flash with fixed slots, batching alone hurts. Merging four turns evicts more sessions at once and turns hidden write-back into synchronous writes, so the cell runs at 0.76× of width one. With liveness and the rule it reaches 2.39× (Table 5). The new pool experiment shows that this slowdown is specific to slot allocation (Appendix J.2), and the revision says so.

W5 / W8. The abstract no longer uses the three phrases you quote. We agree that they were hard to parse. The revised wording is:

  • "per-tier save-or-recompute threshold" now reads "one threshold per storage tier that decides whether to save or recompute an evicted session regardless of its length";
  • "liveness read off the workflow graph" now reads "dropping a session's state once the orchestrator reports it finished";
  • "batched decoding of ready same-role turns under a memory-traffic ceiling" now reads "decoding ready turns of the same role together up to a width set by memory bandwidth".

The abstract also no longer claims that the rule always picks the faster arm. It now says 15 of 16 cells. We would be grateful for any further passage you find hard to parse.

AuthorsOpenReview ↗

Response to Reviewer TUJM (3/3)

W6 / Q3. A symbol table now defines every notation, and Equation 1 is a share of turn time. Appendix K (Table 23) lists every symbol with its meaning and where it is defined, and Section 3.1 points to it. Equation 1 reads as follows. R_π is the share of a turn's wall clock spent restoring state under policy π. φ(K) is the fraction of turns that find their session evicted when K sessions fit in memory. C_π(s) is the time to make session s resident again. Saving costs b_s/r_w + b_s/r_r, the time to write its b_s bytes now and to read them later. Recomputing costs n_s/r_p, a prefill of its n_s tokens. T_turn is the turn's own compute. The numerator is the expected restore time per turn and the denominator adds the turn's compute, so R_π is a share between 0 and 1.

W7. We do not claim Lemma 1 as new. It is now Fact 1 and cites Mattson et al. (1970). We agree that it is the stack-distance property of LRU, and the contribution list no longer presents it as a lemma. It is in the paper because it lets the restore count at K slots be read off a trace before a cell runs. Four slots should miss exactly twice on gui-5 and never on gui-3, and they do (Table 6).

Q2. The rule needs two numbers per platform, and a Snapdragon phone tests it outside Jetson. The numbers are the tier's asymmetry α and the device–model constant α* computed from the read rate, the KV bytes per token, and the prefill rate. On the 32 GB board alone α is 2.8 on NVMe and 132 on eMMC, and the rule chooses opposite actions on the two tiers. The phone's UFS has α = 1.01, so the rule saves. There a cold save-and-load of an 8K-token session beats CPU replay by 74×. End to end on the phone, batching pays 1.18–1.27× on the CPU backend but not on the GPU backend, whose current kernels slow merged decoding (Appendix J.5 and our reply to 1WJb, C2). We did not add another device in the rebuttal window.

Q4. Fused weights decode 1.18× faster, but with role switches a debate takes 2.08× as long as under Recite. We measured fused adapters on an 8 GB Orin Nano with Qwen3-4B and three trained adapters, reading models from NVMe (Appendix J.3, Table 17). Fused weights decode 1.18× faster at width one. A fused role switch restarts the engine and loads a 2.33 GiB model, which takes 7.95 s warm. On a shortened debate with 49 role changes, the fused system runs one turn at a time. It takes 2.08× as long as Recite and 1.12× as long as Hot-swap. Its reloads take 22% of the makespan, and removing all reload time still leaves 1.62×. Fused weights do not by themselves prevent same-role batching, but we did not measure that combination. The save-or-recompute rule is unaffected, since it prices only the KV cache.

We hope the stated inputs, the explicit lineage, the rewritten abstract and symbol table, and the new experiments on dynamic workflows, delay, and fused adapters address the concerns behind the scores. We would be grateful if you would take another look at the revised version.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of the discussion and the revision

Dear Area Chair and Reviewers,

Thank you for the reviews and the discussion. This comment summarizes where each concern stands.

The revision. Appendix J reports the experiments the reviews asked for. For most of them we registered predictions before the runs, and Table 21 lists each prediction with its verdict. The revision also rewrites the abstract phrases that were hard to parse, adds a symbol table (Appendix K), positions the rule against the related work the reviewers named (Section 2), and states what the method needs from a platform (Appendix I). All substantive changes are marked in blue.

Reviewer a5ak. The question was when page-cache buffering changes the preferred action. We timed both actions at 184 single evictions over burst size, dirty-page budget, and eviction interval, and we derived a criterion that needs one extra measurement per device (Appendix J.1). The reviewer replies that the sweep and error analysis "largely address my concern" and that the criterion clarifies where the rule applies in the tested settings.

Reviewer TUJM. The reviewer raised the score. The reviewer states that the concern about a DAG known in advance is addressed and that the effect of environment delay is clarified. The reviewer also agrees with the differences from the three cited papers and notes that the writing has improved. The remaining concern is whether other platforms already make this decision and how far the rule generalizes. Our follow-up answers it with evidence in the paper.

  • The three cited papers compare reading with recomputing. The rule adds the write, and the write decides on eMMC.
  • The rule has four measured inputs, so porting it to another platform means measuring these four. The right action then follows from one ratio per storage tier.
  • The flash tier's write–read asymmetry spans 1.01 to 132 on the tiers we measure. A policy that prices only the read misjudges the 32 GB board's eMMC. The 64 GB board's eMMC has an asymmetry of only 2.2, and there the same policy picks the slower action in all 21 paired measurements (Appendix E.3).
  • From device constants alone the rule predicts opposite actions on the two tiers of the 32 GB board. It picks the faster action in 15 of 16 measured cells, and the one miss costs 1.8%.
  • The llama.cpp server leaves the save-or-recompute decision to the caller, and EdgeLoRA keeps no KV across an eviction (Section 4.5). We invited the reviewer to name a platform that makes this decision from the tier's write rate.
  • Only voting is single-turn. The rule prices one evicted session, so it applies with one agent as well as with several.

Reviewer 1WJb. We have not yet received a reply to this response. Each of the three weaknesses led to a new experiment.

  • W1 asks for the rule's own value. Against a reference that batches, drops finished sessions, and saves every live eviction, the rule alone gives 1.46–1.65× on gui-5, 1.42× on gui-7, and 0.98× on debate (Appendix J.1, Table 15). The rule's own gain is on eMMC cells that evict long sessions.
  • W2 asks about the four fixed slots. With a byte-budgeted pool the rule matters when memory is tight, and batching and liveness carry the gain when it is not (Appendix J.2).
  • W3 asks whether the rule always chooses well. Decision quality is now counted separately from R². The rule picks the faster action in 15 of 16 measured cells, and the sweep of 184 evictions maps the region where it is conservative. This is the question that Reviewer a5ak raised.
  • The further comments led to the positioning against Cake and KVFlow, an end-to-end phone run with trained role adapters (Appendix J.5), and a map from P95 latency budgets to batch widths (Appendix J.6).

We thank the reviewers and the Area Chair again for their time.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.