- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper presents RECITE, a runtime for multi-agent LLM workflows on memory-limited devices. It decides whether evicted KV state should be saved or recomputed, uses the workflow DAG to identify finished sessions, and batches ready turns that share the same role. Experiments on recorded traces and live workloads show substantial improvements in makespan and energy for parallel workflows, but little benefit when batching is not possible. I think the main contribution is how these mechanisms work together under the high cost of flash writes, rather than a new agent model or training method.
NB: The below is not really a summary anymore but rather comments on the work for the authors; I don't know where else to put them but here.
The most interesting contribution is the interaction between batching and slow flash writes. RECITE's device-specific integration is useful, but the underlying ideas of KV recomputation and workflow-aware management have clear precedents. I see it more as a systems integration contribution than a fundamentally new scheduling method, and would like stronger evidence for the restoration rule's independent value.
It was surprising to see a throughput optimization create more blocking writes, and the experiments explain this quite well for batching on eMMC. However, Cake has already studied loading versus recomputation, while KVFlow uses workflow information for cache management. I am still not fully sure how much of RECITE is new compared with these earlier systems. The paper could explain this difference more clearly. The headline speedups are difficult to interpret. As the restore-capable baselines already use adaptive restoration, these numbers mainly reflect batching and liveness. I found the capability-matched experiments more useful. The tier rule improves performance on the GUI workload using eMMC, but slightly hurts performance on the debate workload with the same storage. This distinction deserves more attention.
In the debate experiment, the cost model predicts an improvement, but the actual result is a slight regression. The authors explain that page-cache buffering makes small writes cheaper than expected, which seems reasonable. Table 8 reports that the rule chooses the faster option in every paired comparison there, but the debate result suggests this may not always hold. I wonder whether different write sizes or cache conditions would lead to more such cases.
The experiments reserve four KV slots, each sized for 32K tokens. Shorter sessions may leave some of this reserved memory unused. A byte-budgeted pool could reduce the number of evictions, and I am curious whether RECITE would still achieve similar improvements under this allocation strategy.
I would also like to see a complete phone workflow rather than only storage microbenchmarks. The live Jetson experiments help, but most main tests still use recorded traces and shape-only adapters. Testing with trained role-specific adapters on a phone would make the practical contribution more convincing. Wider batching also increases tail latency, which may be a problem for interactive agents even when the overall makespan improves.
Critical strengths
Batching alone makes the system slower, but becomes beneficial when combined with liveness and the tier rule. This interaction is well demonstrated.
I like how RECITE decides whether to save or recompute an evicted KV cache by comparing their estimated costs. The workflow DAG also helps identify finished sessions whose KV caches can be discarded directly. Both mechanisms are simple and do not require learned predictors.
On GSM8K, restoring KV or sharing a prefix has little effect on accuracy. HumanEval+ shows similar results. The authors also include the GUI and linear workloads, even though RECITE offers almost no speedup in these cases.
Critical weaknesses
The headline speedups are measured against baselines that already use adaptive restoration, so they do not isolate the new tier rule. Its independent benefit also varies considerably between GUI and debate workloads.
The experiments reserve four KV slots, each sized for 32K tokens. When sessions are shorter, some of this reserved memory may go unused. A more flexible allocation could reduce evictions and potentially change RECITE's measured benefits.
The restoration rule does not always make the better choice. For the debate workload on eMMC, its predicted gain becomes a slight regression because of page-cache buffering. The authors report correct decisions in every paired comparison in Table 8, but these tests do not cover this case. The high R² also measures restoration-share prediction rather than decision optimality.
Questions for authors
Not applicable
4 comments
Response to Reviewer 1WJb (2/3)
W3. Decision quality is now counted separately from R². The rule is right in 15 of 16 cells and at 110 of 184 sweep points. We agree with both remarks. R² measures only how well the model predicts restore share. The abstract now says that where both arms of a decision are measured the rule picks the faster arm in 15 of 16 cells, and the one miss costs 1.8%. It also states that a sweep of 184 single evictions maps the region where the rule is conservative.
Different write sizes and cache states do produce more cases like debate. In the sweep we timed both arms at each eviction on eMMC (Appendix J.1, Table 14). The grid covers bursts of 53 MiB to 4.4 GiB and dirty-page budgets D of 64 MiB, 256 MiB, 1 GiB, and the default, which absorbs 980 MiB on our board. Evictions are 0, 8, or 23 s apart.
- When the burst plus the existing backlog is at most D/2, saving was faster at 54 of 56 points. When the burst is at least 2D, recomputing was faster at all 42.
- The device-rate rule always recomputes on eMMC. It picks the faster action at 110 points and errs at 74. Every error recomputes where saving was faster, and none occurs once the burst reaches 2D.
- A burst-aware variant charges only the part of the burst above the cache's headroom. It picks the faster action at 150 points and errs at 34. Of these 34 errors, 33 save where recomputing was faster, because the measured crossover arrives at 0.29–0.71 of the burst it predicts.
- The interval between evictions matters because it lets the backlog drain. Over the 16 settings measured at all three intervals, saving was faster at 38%, 44%, and 73% of points for gaps of 0, 8, and 23 s.
End to end against the capability-matched reference (Table 15), the burst-aware variant removes the debate loss at 1.002×, keeps gui-5 at 1.46×, and trails on gui-7 at 1.36× against 1.42×. We proposed the variant after seeing debate and registered its predictions only for debate and gui-5. A single wrong eviction costs more than these end-to-end numbers suggest. Its median penalty is 6.25× for the device-rate rule, but one eviction is a small part of a workflow.
C1. Cake decides how to read saved state, and KVFlow manages state in memory. Recite decides whether to write it, and on eMMC it chooses the opposite action. Cake decides on the read side how to divide a saved prefix between loading and recomputing. It never decides whether to write state at eviction. KVFlow uses the agent step graph to set eviction priority and to prefetch. It keeps state in GPU or CPU memory, where both directions of a swap cost about the same. It does not drop state unwritten. Recite prices the write through the tier's write–read asymmetry α = r_r/r_w and takes the return probability q from terminal events.
In the limit α → 0 with q = 1, the write costs nothing and Proposition 2 reduces to the read-side comparison. Our rule therefore agrees with a read-side policy on NVMe (α = 2.8) and on the phone's UFS (α = 1.01), where both save. On the 32 GB board's eMMC (α = 132) the write term decides. A read-side policy cannot close that gap even with free reads. Save-everything on gui-5/eMMC spends only 2.8 s of its 543.1 s reading, so it needs at least 540.3 s against 259.0 s for Recite, a factor of 2.09×. On debate/eMMC at width four the bound is 2266.5 s against 727.8 s, a factor of 3.11×. These two save-everything arms also write finished sessions, so their gaps are wider than the rule's own factors in W1. The revised Section 2 states this lineage.
Each ingredient has precedents, and Section 3.4 says that our batching is standard continuous batching. Neither Cake nor KVFlow prices the write, and that term is what reverses the decision on eMMC. The paper also measures how the ingredients interact on a device whose storage tier is two orders of magnitude asymmetric, with rules simple enough to predict the outcome from traces and device constants. Our answers to W1 and W3 give the evidence for the rule's own value.
Response to Reviewer 1WJb (3/3)
C2. On a phone with trained role adapters, Recite is 1.27× and 1.18× faster on the CPU backend and slower on the GPU backend. We ran voting and debate end to end on a OnePlus 15T with a Snapdragon 8 Elite Gen 5 and UFS storage (Appendix J.5). The phone runs Qwen3-4B with the trained planner, critic, and coder adapters as voter, judge, and debater. Inference, KV state, and storage stay on the phone, and the control plane runs on a laptop over USB.
- On the CPU backend Recite finishes five voting questions in 5574 s against 7053 s for Hot-swap, a factor of 1.27×. It finishes five debate questions in 9323 s against 10979 s, a factor of 1.18×. The registered point estimates were 1.26× and 1.17×. Restoration takes under 0.02% of turn time on UFS.
- The phone's GPU backend decodes four merged streams at only 0.70× the aggregate rate of one in the current llama.cpp OpenCL kernels. End to end there, Hot-swap takes only 0.87× as long as Recite on voting and 0.77× on debate, inside the registered ranges.
- Limits: one seed per cell, five CPU questions, and three GPU questions for voting and two for debate. Energy was not measured.
The main matrix uses shape-only adapters because it measures time, bytes, and energy, and these depend on adapter shape. Table 4 measures the base model's accuracy. The phone runs, the code-repair loop (J.4), and the fused-adapter study (J.3) all use trained adapters.
C3. A P95 turn budget now maps to a batch width, and we corrected how the paper counts P95. Appendix J.6 maps P95 turn budgets to widths on debate over NVMe. A budget of 15 s admits width one, 20 s admits width two at 1.62× the speed, and 30 s admits width four at 2.10×. Energy per token falls from 545 to 329 and 240 mJ. Per decode step, the merged step time fits Proposition 3's linear form as 35.4 + 9.72k ms within 6%. While doing this we found that our P95 had counted merged members at zero, because the runtime logs a group's time on its lead turn. The revision now charges every member its group's service time. Width four on debate/NVMe has a P95 of 29.9 s, up from 29.1 s. Table 5's width-four cells rise from 29.5 s and 59.4 s to 30.3 s and 77.4 s. The P95 cost of width two over Hot-swap rises from 3.2 s to 4.1 s. Figures 4c and 9c are redrawn with the corrected P95. On the phone's CPU the P95 turn service time rises 4.38× on voting and 3.23× on debate. The runtime still takes the width as a fixed setting, and an online policy that sets it from a budget is future work.
We hope the isolated rule, the decision counts and their failure region, the pool, the phone runs, and the corrected latency numbers address the soundness concern. We would welcome any further check you think the rule should pass.
Follow-up on our response
Dear Reviewer 1WJb,
Thank you again for your review. We posted our response and a revised PDF earlier in the discussion period, and we would like to check whether they address your concerns before the discussion closes.
In short, the revision isolates the rule against a capability-matched reference (W1), replaces the fixed slots with a byte-budgeted pool (W2), and counts decision quality separately from R² (W3). The new experiments are in Appendix J.
If any point remains unclear, we are glad to answer it right away. If the response resolves your concerns, we would be grateful if you could update your assessment.
Best regards,
The authors
Response to Reviewer 1WJb (1/3)
Thank you for the constructive review. Each point below led to a new experiment or a correction in the revision. The revised PDF marks all substantive changes in blue. The experiments are in Appendix J, and for most of them we registered predictions before the runs. Table 21 lists every registered prediction with its verdict, including the misses. Analyses made after seeing data are labeled post hoc. They are the burst-aware variant of J.1, the batching fix of J.4, and the latency budgets of J.6.
W1. Against a capability-matched reference the rule alone gives 1.46–1.65× on gui-5, 1.42× on gui-7, and 0.98× on debate. We agree that the headline factors over Hot-swap combine batching, liveness, and the rule. A capability-matched reference isolates the rule. It batches, drops finished sessions, and saves every live eviction. The table below gives three runs per arm on eMMC (Appendix J.1, Table 15).
On NVMe the rule always saves, so it coincides with the reference by construction. Neither gui-3/eMMC nor voting evicts a live session. The rule's own gain is therefore on eMMC cells that evict long sessions, while the 2.1–2.5× on voting and debate comes from batching and liveness. On gui-5 the two factors differ in the save arm, which took 426.5 s in the run of Section 4.3 and takes 371.7 ± 15.4 s in the rerun.
The GUI–debate difference comes from the size of each write. On gui-5 one live eviction writes 2362 MB. That write is synchronous and takes 127–160 s, so replay is cheaper. On debate each eviction writes about 53 MB. The page cache absorbs it, so saving is nearly free and the device-rate rule over-prices it.
W2. With a byte-budgeted pool the rule matters when memory is tight, and batching and liveness carry the gain when it is not. We replaced the four slots with a pool on the same board (Appendix J.2, Table 16).