acceptodds
Withdrawn from ICLR 2027

SOMA: Efficient Multi-turn LLM Serving via Small Language Model

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Large Language Models (LLMs) are increasingly deployed in multi-turn dialogue settings where preserving conversational context across turns is essential. A standard serving practice concatenates the full dialogue history at every turn, which reliably maintains coherence but incurs substantial cost in latency, memory, and API expenditure, especially when queries are routed to large proprietary models. Existing approaches often struggle to balance the trade-off between response quality and efficiency. We propose a framework that exploits the early turns of a session to estimate a local response manifold and then adapt a smaller surrogate model to this local region for the remainder of the conversation. Concretely, we train soft prompts that push the small surrogate away from the large model's responses, with an anti-degeneration term for stability, and score each early turn by how much agreement survives. Localized LoRA fine-tuning then distills the turns with robust agreement, so the surrogate runs without prompts at inference. A gate then hands the session to the surrogate and rolls back on drift. We further provide a theoretical analysis for key components in SOMA. Experiments on six multi-turn benchmarks suggest that SOMA largely preserves the original model's responses and lowers serving cost for medium-to-long sessions that stay on topic. The source code is provided at: https://anonymous.4open.science/r/SOMA-4175 .

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM, multi-turn conversations, manifold approximation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_bbniOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes SOMA, a framework for efficient multi-turn LLM serving that adapts a smaller surrogate model to the behavior of a larger model within a conversation-specific region. SOMA uses adversarial soft prompts to derive an anchoring score for each warm-start turn, performs score-weighted LoRA fine-tuning on the large model's responses, and then serves later turns with a compressed context containing a fixed-budget summary and recent turns. A response-fidelity gate controls the initial switch to the surrogate, while a query-locality gate detects drift and triggers rollback to the large model.

The paper evaluates SOMA on six datasets and two model families, measuring response similarity, task-grounded accuracy on MATH, token usage, throughput, and end-to-end latency. The results suggest that SOMA can preserve more of the large model's behavior than the evaluated baselines while reducing serving costs for sufficiently long and locally coherent sessions. The paper also presents a theoretical analysis of switching and amortized efficiency.

Critical strengths
  1. The paper addresses a practical and important problem. Session-specific adaptation is a plausible way to trade one-time computation for lower serving costs over subsequent turns, especially when a session is sufficiently long.
  2. The evaluation covers multiple dialogue domains and two model families. The paper compares SOMA with several relevant baselines, including direct surrogate serving, fine-tuning, context compression, and model routing. It also reports end-to-end latency, token usage, and throughput rather than relying only on response similarity. Running all methods with vLLM automatic prefix caching is a useful control for comparing the methods under a shared serving setup.
  3. The system includes a mechanism for handling distribution shift. The fidelity and locality checks, together with rollback, acknowledge that a session-local surrogate may cease to be appropriate as a conversation evolves. The paper also formalizes assumptions behind its switching and cost analyses, making the intended scope of the method relatively explicit.
Critical weaknesses

CW1. The evidence for the motivating workload pattern is narrower than the paper's general MLaaS framing suggests.

The paper uses the observed decline in average tokens per turn as a central motivation for its approach and presents this as a long-tail pattern in real-world multi-turn dialogue. However, the four datasets used for this observation—ShareGPT, ReMeDi, Craigslist Bargain, and Multi-Character—represent substantially different settings. They include user–LLM conversations, medical dialogues, human–human negotiation, and multi-character role-play; they do not constitute a representative sample of general MLaaS traffic.

There is also a mismatch between the strength of the claim and the statistic presented. Figure 1 primarily shows average token counts by turn, normalized relative to the first turn. This illustrates a decline in average turn length, but does not by itself characterize a long-tail distribution of session-level workloads or establish how common the pattern is among MLaaS users. The same datasets are also used in the evaluation, so the results do not independently establish that this workload pattern is broadly representative or that it predicts the method's benefits on unseen service traffic. This matters because SOMA's amortization depends not only on later turns being shorter, but also on sessions being long enough and remaining locally coherent for long enough to recover the warm-start and adaptation costs.

The authors should either narrow the empirical claim to the studied datasets or provide stronger evidence from independently collected and more representative user–LLM sessions. Reporting session-level length distributions, variation across domains, the fraction of sessions long enough to amortize adaptation, and the relationship between these statistics and SOMA's measured savings would better connect the initial observation to the claimed MLaaS use case.

CW2. The central anchoring-score weighting strategy is not sufficiently validated against simpler alternatives.

The main algorithmic contribution is not merely the use of soft prompts or LoRA, but the claim that adversarial prompt probing identifies warm-start responses that are particularly useful for adapting the surrogate to future turns. However, the current experiments do not isolate the value of this weighting strategy from the effects of fine-tuning itself or from other training configurations.

Specifically, SOMA weights each teacher response by an anchoring score derived from the residual mining loss after adversarial perturbation. The paper argues that responses whose agreement with the teacher survives this perturbation are more likely to encode persistent conversational information, such as topic, constraints, and style. Yet this does not directly establish that the score predicts future adaptation utility: robustness to an input perturbation and usefulness for reproducing future teacher responses are different properties. The reported ablations remove terms from the prompt-mining objective, but do not compare anchoring-score weighting with uniform weighting or weighting by the unperturbed teacher–student output divergence. Moreover, the History-FT baseline uses LoRA on attention projections, whereas SOMA adapts both attention and MLP projections; consequently, their performance difference does not cleanly identify the contribution of anchoring-score weighting.

This issue is central because the proposed score is the paper's main mechanism for selecting which parts of the warm-start dialogue should influence the adapter. A controlled comparison should use identical trainable modules, data, and adaptation budgets, and compare at least uniform weighting, raw-output-divergence weighting, and anchoring-score weighting. An additional analysis testing whether the score predicts imitation quality on subsequent turns would help establish the intended interpretation. Without such evidence, the results support session-local fine-tuning as a potentially effective strategy more strongly than they support the specific novelty and necessity of the anchoring score.

CW3. The efficiency results do not sufficiently isolate SOMA's incremental contribution from context compression and serving-system optimizations.

The paper does measure end-to-end latency and explicitly includes soft-prompt mining, LoRA adaptation, and gating costs. Therefore, the concern is not that the paper ignores one-time overhead. Instead, the current analysis does not sufficiently separate the sources of the reported gains or establish the incremental advantage of SOMA's adaptation-and-switching strategy over alternative approaches to multi-turn serving.

SOMA's post-switch path combines a surrogate model with a fixed-budget summary and recent turns, while the system also benefits from shared vLLM automatic prefix caching. The reduction in token usage may therefore reflect both the change of model and the compressed-context policy. Although LLMLingua-2 and several serving-system methods are discussed or evaluated, these do not provide a fully controlled comparison that holds the context representation and serving protocol fixed while removing only SOMA's session-local adaptation. The related-work discussion includes relevant work on multi-turn cache reuse and resource management, including CachedAttention, efficient resource management for multi-turn serving, FlowKV, and LoopServe; the paper should more precisely explain the remaining gap that SOMA addresses beyond these approaches.

The cost model in Section 4.2 is useful for expressing the amortization logic, but it remains an accounting framework rather than a practical rule for selecting the warm-start length or deciding whether adaptation should begin for a session. In addition, the paper does not provide a sufficiently detailed breakdown of the costs of prompt mining, LoRA adaptation, the response-fidelity checks, summary creation and refresh, locality monitoring, and rollback. The authors should report a component-level cost breakdown and provide a controlled ablation that retains SOMA's summary-plus-recent-turn context policy while removing local adaptation. They should also clarify the cost of generating and updating the summary and how this cost enters the reported end-to-end measurements. These analyses are necessary to determine how much of the reported benefit comes from the paper's core algorithm rather than from context compression or other serving choices.

Questions for authors
  1. Could you report matched-budget LoRA experiments comparing anchoring-score weighting with uniform weighting and raw teacher-student output-divergence weighting, using identical trainable modules?
  2. How is the fixed-budget summary generated and refreshed, and are both operations fully included in the end-to-end latency measurements?
  3. How are the MATH examples converted into context-dependent multi-turn sequences, and do subsequent turns use each method's own generated responses or a fixed teacher-generated trajectory?
  4. How does SOMA's session-local adaptation differ from long-horizon multi-turn routing, such as DialRouter (Zhang et al., 2026), and would such routing be a complementary baseline or a fundamentally different comparison?

References

  1. Gao, B., He, Z., Sharma, P., Kang, Q., Jevdjic, D., Deng, J., Yang, X., Yu, Z., & Zuo, P. (2024). Cost-Efficient Large Language Model Serving for Multi-Turn Conversations with CachedAttention. USENIX Annual Technical Conference (USENIX ATC 2024).
  2. Jeong, J., & Ahn, J. (2025). Accelerating LLM Serving for Multi-Turn Dialogues with Efficient Resource Management. Proceedings of the ACM International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS).
  3. Liu, X., Chen, H., Hu, X., & Chu, X. (2025). FlowKV: Enhancing Multi-Turn Conversational Coherence in LLMs via Isolated Key-Value Cache Management. arXiv:2505.15347.
  4. Li, H., Lan, T., Fu, Z., Cai, D., Liu, L., Collier, N., Watanabe, T., & Su, Y. (2025). LoopServe: An Adaptive Dual-Phase LLM Inference Acceleration System for Multi-Turn Dialogues. arXiv:2507.13681.
  5. Zhang, J., Liu, X., Hu, Y., Niu, C., Zeng, H., Tang, S., Wu, F., & Chen, G. (2026). From Myopic Selection to Long-Horizon Awareness: Sequential LLM Routing for Multi-Turn Dialogue. arXiv:2604.12385. This is a preprint and is included only as relevant contextual work, not as a required peer-reviewed baseline.
Reviewer_8ax9OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

SOMA serves a multi-turn dialogue with a small model instead of a large one but only after the session has "warmed up". The large model handles the first few turns, and its replies are cached. The authors then train a soft prompt on the frozen small model that actively pushes it away from those cached replies... whatever agreement survives this adversarial probe becomes a per-turn "anchoring score". A session-specific lora distills the large model's replies weighted by those scores, and a semantic gate switches the session over to the adapted small model, with a drift monitor that rolls back to the large model on a topic shift.

Critical strengths

The idea is creative and clearly motivated by something real. Using the adversarial soft prompt not to improve the small model directly, but as a probe that tells you which early turns are worth distilling, is a nice reframing, and it is the part of the paper I found most interesting. Six benchmarks and two model families, five baselines, three judges, and MATH exact match as a task-grounded signal on top of the similarity numbers. The paper reports its own failure cases, which makes me trust the rest of it more.

Critical weaknesses

The central methodological choice is not isolated, and this is my main reason for rejecting at this stage. Everything hinges on the adversarial soft-prompt mining being the right way to pick which warm-start turns to distil, but the paper never compares it against the simple alternatives: weighting turns by raw disagreement with the teacher, or sampling them uniformly at the same lora budget. Appendix Gand I both say these controls are missing, so the case for mining rests on a component ablation that only removes terms of the mining objective. The comparison to History-FT is also not architecturally matched: SOMA puts lora on attention and mlp, the baseline on attention only (Appendix C.3). So the gap to History-FT mixes turn selection with adapter capacity. I would want a plain Lora with alternative turn weighting, identical architecture and identical compute budget. The efficiency claim, which is the headline, lacks a compute-quality analysis: no flop counts, no controlled surrogate-size comparison ( the pairs are fixed at 70B to 7B and 8B to 0.6B), and no account of how the one-off mining and lora cost should be traded against per turn savings. One adapter per conversation also raises a scalability question the paper leaves open, since the reported latencies are for the single-session path.

Questions for authors

Will you compare SOMA against divergence-weighted and uniformly sampled turn selection at the same lorabudget, with the same adapter architecture and rank? Will you re-run the History-FT baseline with lora on attention and mal as in SOMA, so that the only difference is the turn weighting? Can you give a flop, token and wall-clock breakdown of soft-prompt mining, lora adaptation and post-switch serving, and a sweep over the surrogate size? What is the cost of serving many sessions with their own adapters?

Reviewer_Sje3OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

Efficient Multi-turn LLM is a important task as it requires to preserve previous rounds of memories for recall to produce better next round conversation. Soft-prompts for lOcal Manifold Approximation (SOMA) try to solve this research problem. SOMA is designed as an efficient multi-turn LLM serving framework that combines 1.adversarial soft-prompt mining, 2.session-local LoRA adaptation, and 3.dynamic model routing to mitigate inference lagging and cost while preserving response quality.

Critical strengths

1.Interesting Research Problem has been studied which is efficient multi-turn LLM. It serves as an important and practically relevant research problem in the area of LLM.

2.Results are promising and the paper evaluates on six benchmarks across two model families and reports improvements in response similarity, latency, and inference efficiency.

Critical weaknesses

Limited Methodological Novelty

Novelty is a relatively abstract concept, and I believe the authors should more clearly articulate the fundamental methodological contribution of SOMA to the multi-turn LLM research community. In particular, what important research challenges have been overlooked or remain insufficiently addressed by existing approaches, and how does the proposed method specifically address these challenges in a way that prior methods cannot?

Efficient multi-turn LLM serving has already been extensively investigated through history-aware model routing, temporary model adaptation, knowledge distillation, and context compression. Relevant works include MTRouter, Temp-LoRA, StreamAdapter, LoopServe, and A2D2 [1-8]. SOMA appears to integrate several established techniques, including soft-prompt optimization, knowledge distillation, LoRA adaptation, and model routing, with adversarial soft-prompt mining.

While combining existing techniques can constitute a meaningful contribution, the paper does not sufficiently explain what fundamental limitations of previous methods motivate this particular combination, why these components are necessary, or what genuinely new methodological insights emerge from their integration. In particular, it remains unclear whether adversarial soft-prompt mining provides a significant advantage over simpler sample-selection and weighting strategies.

I encourage the authors to strengthen the motivation and methodological justification, clearly distinguish SOMA from closely related approaches, and provide controlled experiments demonstrating that the proposed mechanism addresses a previously unresolved problem rather than merely combining established techniques.

Unfair Experimental Comparison:

History-based Fine-Tuning ( using earlier conversion as context and response pair for finetuning the small model) applies LoRA to attention projections, but SOMA uses both attention and MLP projections. The authors need to ensure identical adaptation configurations and computational budgets for fair comparisons.

Theoretical and Implementation inconsistency:

The theoretical switching guarantee assumes an independent acceptance set, whereas the actual implementation reuses training examples for acceptance. As result, the theoretical guarantee is not consistent to the implementaiton.

How does this discrepancy impact the legibility and practical significance of the theoretical analysis? The authors need to show experiments using an independent acceptance set and clarify the limitations of the theoretical guarantees.

References

[1] MTRouter: Cost-Aware Multi-Turn LLM Routing with History–Model Joint Embeddings. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (ACL 2026).

[2] With Greater Text Comes Greater Necessity: Inference-Time Training Helps Long Text Generation. Conference on Language Modeling (COLM 2024).

[3] StreamAdapter: Efficient Test Time Adaptation from Contextual Streams.(2024)

[4] LoopServe: An Adaptive Dual-phase LLM Inference Acceleration System for Multi-Turn Dialogues. (2025)

[5] The Power of Scale for Parameter-Efficient Prompt Tuning. Proceedings of EMNLP( 2021)

[6] Distilling the Knowledge in a Neural Network.(2015)

[7] LoRA: Low-Rank Adaptation of Large Language Models. International Conference on Learning Representations (ICLR 2022).

[8] RouteLLM: Learning to Route LLMs from Preference Data. International Conference on Learning Representations (ICLR 2025).

Questions for authors

Questions for the Authors

  1. How does SOMA differ methodologically from existing approaches such as MTRouter [1], Temp-LoRA [2], StreamAdapter [3], and LoopServe [4],, beyond combining established techniques?

  2. what advantages does adversarial soft-prompt mining provide over simpler sample-selection and weighting methods? could the authors to show controlled experiments to illusrate these advantages ?

[1] MTRouter: Cost-Aware Multi-Turn LLM Routing with History–Model Joint Embeddings. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (ACL 2026),

[2] With Greater Text Comes Greater Necessity: Inference-Time Training Helps Long Text Generation. Conference on Language Modeling (COLM 2024).

[3] StreamAdapter: Efficient Test Time Adaptation from Contextual Streams.

[4] LoopServe: An Adaptive Dual-phase LLM Inference Acceleration System for Multi-Turn Dialogues.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.