acceptodds
Under review as a conference paper at ICLR 2027

Training is Serving: Unifying Inference and Adaptation for Multimodal Extension

Abstract

Extending large language models (LLMs) to multiple modalities is a key step toward general-purpose multimodal intelligence. As foundation models scale, conventional fine-tuning increasingly assumes homogeneous clusters and pays model-scale communication across ranks. We present Training is Serving (TiS), an API-driven compute-sharing framework that decouples the heavy computation of a frozen LLM from lightweight multimodal adaptation. Specifically, we enable the LLM to support training and return gradients in the form of an API. An edge client trains only its modality encoder and adapter, sends embeddings to the service, and receives gradients; raw data and trainable parameters remain at the edge, while the LLM weights remain at the cloud server. TiS decouples edge-side data parallel training from provider-side model parallel serving, which has four core advantages: (1) compute sharing, where edge clients can perform training by accessing the API without acquiring server control; (2) hardware decoupling, where heterogeneous accelerators are unified behind a standard API backend; (3) low-bandwidth communication, enabling multi-node MLLM training over only 1 Gbps links; and (4) data locality, where raw training data remains on the edge while the service receives only embeddings and labels. Using shared compute, we extend Gemma-4-31B-it to the speech modality and support 70 languages.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Split Learning, Multimodal Extension, Speech-to-Text Translation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_Q3egOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
1
Soundness
3
Read review
Paper summary

To address challenges in scaling multimodal large models—such as fragmented computing resources, hardware heterogeneity, and the reliance of traditional distributed training on centralized management and high-bandwidth communication—the authors propose TiS, an API-based computation-sharing framework. TiS decouples local multimodal adaptation from the computation performed by the frozen remote large model; it retains raw data, trainable modules, and optimizers locally, transmits embeddings and labels to the remote end, and updates the adapter using the returned input gradients. Through dynamic scheduling, the framework assigns training requests to independent service nodes hosting the same model, enabling diverse hardware to participate collaboratively in training while supporting both standard inference and gradient computation. In my view, this work offers a practical solution for leveraging distributed computing resources to scale multimodal models—characterized by clear computational boundaries and a service-oriented design—and demonstrates robust multi-node scalability.

Critical strengths
  1. The author proposes a scheme for sharing computing resources via service interfaces, designed to address complex conditions characterized by independent management, hardware heterogeneity, and limited network bandwidth; this approach holds practical value for deployment.

2.The authors provide large-scale validation and demonstrate near-linear throughput scaling across four nodes with a fixed global batch size.

Critical weaknesses
  1. TiS bears a strong resemblance to existing architectures for split learning, remote automatic differentiation systems, and multimodal federated large models (such as Uniflow and FedNano); the authors failed to adequately distinguish their work from these existing approaches in the "Related Work" section.

  2. Although the authors conducted experiments with large-scale models, there is a significant lack of experiments regarding module ablation and parameter validation.

Questions for authors

The author has not considered complex operational scenarios—such as node departures, retry mechanisms, protection for low-resource languages, and data locality. Overall, while the author's idea is promising, the experimental analysis fails to substantiate the claims; I will lower the score if the author does not address my questions.

3 comments

[W1] TiS bears a strong resemblance to existing architectures for split learning, remote automatic differentiation systems, and multimodal federated large models (such as Uniflow and FedNano); the authors failed to adequately distinguish their work from these existing approaches in the "Related Work" section.

We thank the reviewer for pressing on this point. The review describes the data path—embeddings out, input gradients back—but the data path is the interface, not the contribution. The paper's claim concerns what the service side is allowed to be.

Classical split learning places the server inside the training system: the server holds trainable parameters, runs a training process for the duration of the job, and its parallelism is fixed when the job starts.

TiS makes a different assumption: the edge and service sides are administratively isolated, the training party needs no SSH access or server control, and the two sides interact only through the API. The process resident on the service side is an inference process, and it holds no training state between steps. A TiS worker is an inference worker plus a gradient-return path, serving standard inference and training-gradient-return requests from the same resident checkpoint. This changes the deployment unit from "a training rank" to "an interchangeable worker."

This is the difference the paper measures. The following are the ways TiS differs from split learning:

  • The framework supports multiple modality adapters sharing one frozen LLM backbone, with no aggregation and no global model state; split learning instead trains a single split model.
  • Each worker can join or leave within one step: 8→12→4 workers at 35.7/46.2/20.4 samples/s, with no restart and no change to other workers' layouts (Fig. 5a).
  • Training and production inference share one checkpoint; co-running reduces inference throughput by only 8.3% (1229→1127 tokens/s) while training sustains 8.27 samples/s (Fig. 4c).
  • A GPU–NPU pool serves one checkpoint without a spanning communicator: 20.4 + 5.0 samples/s separately versus 25.1 samples/s pooled (Fig. 4b).

For FedNano and UniFL specifically:

  • Aggregation: federated methods merge client updates into one global model and need a coordinator that owns model state, whereas TiS never aggregates or updates the backbone, so workers stay stateless between steps and the provider keeps its weights and parallel layout.
  • Software stack: federated aggregation assumes clients run backward through the same architecture, whereas TiS exposes only a gradient-return API, so clients may differ in accelerators, collective libraries, and parallel layouts.
  • Unit of communication: federated methods move parameters or low-rank factors, scaling with the adapter, whereas TiS moves fixed-width embeddings and their gradients, set by K, D, and the batch, and independent of the number of trainable parameters.

TiS is therefore a pool of independently managed inference services rather than a federation of training clients.

[W2] Although the authors conducted experiments with large-scale models, there is a significant lack of experiments regarding module ablation and parameter validation.

Thank you for the comment. The revision adds the following ablations and system measurements:

  • Worker scale-out (Fig. 4(a)). 1→4→8→16 workers: 5.3 / 20.4 / 35.5 / 60.7 samples/s, 11.5× from 1 to 16.
  • Heterogeneous pool (Fig. 4(b)). Four 910C workers alone sustain 20.4 samples/s and two RTX 5880 Ada workers alone 5.0 samples/s; pooled, the six workers reach 25.1 samples/s. The scheduler assigns samples in proportion to measured throughput, so the slow workers do not set the step time.
  • Training–inference co-running (Fig. 4(c)). Training alone 20.40 samples/s; inference alone 1229 tokens/s; running both, training sustains 8.27 samples/s and inference 1127 tokens/s (−8.3%).
  • Elastic scaling (Fig. 5(a)). 8→12→4 workers: 35.7 / 46.2 / 20.4 samples/s; membership changes bind within one step without restarting training, and timed-out samples return to the queue by sample ID.
  • Communication (Fig. 5(b)). TiS carries 1.76 GB per step, independent of trainable parameters; on the same four nodes ZeRO++ carries 1.02 GB (adapter only) and 8.67 GB (encoder + adapter).
  • Bandwidth (Fig. 5(c)). At 1 Gbps one direction takes 9.4 s against a ~17 s compute window, with a crossover near 555 Mbit/s.
  • Gradient fidelity (Table 6). The same embedding batch sent to 910C and RTX 5880 Ada backends returns gradients with cosine similarity 1.000.

[Q1] The author has not considered complex operational scenarios—such as node departures, retry mechanisms, protection for low-resource languages, and data locality. Overall, while the author's idea is promising, the experimental analysis fails to substantiate the claims; I will lower the score if the author does not address my questions.

We thank the reviewer for this suggestion. TiS originated as a training framework, so the initial draft emphasized model quality and did not clearly define which system properties should be measured or how far system validation should go. The 70-language run shows that the framework can carry a real-scale training job, but it is not itself a system validation of TiS. The reviewer's comment helped us clarify this. The revision adds 8–9 pages and dedicates Figures 4–5 to system throughput, heterogeneous pooling, elasticity, communication cost, and ablation, reporting system validation separately from model quality.

  • Node departure and retry. Elasticity and failure recovery share one liveness rule: at step boundaries the scheduler probes whether a worker is online and estimates its throughput, then adds or removes members and re-dispatches samples. Elasticity is measured in Fig. 5(a): the pool grows from 8 to 12 and shrinks to 4, holding at 35.7, 46.2, and 20.4 samples/s; each membership change requires no restart of the training process. Failure re-dispatch uses the same check: timed-out samples return to the queue by sample ID.
  • Low-resource languages.
    • Finding. The appendix is renamed Detailed Results. §4.1 now analyzes the low-resource sources: only five source languages (Burmese, Khmer, Assamese, Amharic, Georgian) trail the best baseline by more than 5 COMET, while the reverse directions are normal (English→Burmese 89.1 vs 87.2 for Gemini, English→Khmer 83.1 vs 81.4). The gap is therefore on the input side, not in target decoding.
    • Cause. The source of the gap is the frozen Whisper encoder on the input side. These five languages are underrepresented in Whisper's pretraining data, so the encoder has not learned features that separate their phonemes and acoustic cues. With the encoder frozen, the adapter can only adapt on top of this fixed acoustic representation and cannot compensate for the missing coverage.
    • Unfreezing the encoder. We also tested unfreezing the encoder (636.97M), and it brings no clear quality gain: held-out ASR and S2TT quality shows no material change, while step time rises substantially, so the revision keeps the encoder frozen. Encoder capacity is therefore not the binding constraint.
    • Release. The full 70×69 per-direction results will be released with the code.
  • Data locality. Raw audio, trainable parameters, and optimizer state stay at the edge; the service receives only embeddings and label token IDs.

Thank you for your response. I have seen your revisions; I am maintaining my score, as I feel that 3/2 is sufficient.

Reviewer_S2YkOpenReview ↗
Rating
2
Confidence
1
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

TiS places a split-learning boundary between a frozen cloud LLM and locally trainable modality modules. Clients send embeddings and labels, receive input-embedding gradients, and perform local backpropagation and optimizer updates. Independent service replicas and heterogeneous scheduling avoid model-parallel collectives across services; the paper also describes chunked transfers and failure redispatch. The framework is used to extend Gemma-4-31B-it to speech, evaluated on 70 FLEURS source languages and 4,830 non-identity translation directions, with throughput measurements from one to four service nodes.

Critical strengths
  1. The deployment problem is relevant and the interface boundary is sensible. Independently managed GPU/NPU services cannot simply be treated as one coordinated training cluster. Returning input gradients from frozen LLM replicas while keeping trainable parameters and optimizer state local enables reuse of compatible services. Section 3.2 usefully distinguishes the absence of inter-service collectives from tensor parallelism within each service.

  2. The evaluation goes beyond a purely conceptual demonstration. At a fixed global batch, throughput increases from 16.2 samples/s on one node to 60.7 on four nodes, a 3.75× improvement (Table 2). Table 5 reports 18.8 spBLEU/77.8 COMET across 70 source languages, competitive with MCAT’s 17.4/77.3. The combination of service scaling and downstream quality supports feasibility at a meaningful adaptation scale.

Critical weaknesses
  1. The evaluation does not separate TiS-specific systems benefits from adding hardware and using established split learning. Table 2 compares different node counts within TiS, but lacks a simple remote-gradient baseline under matched hardware, frozen model, and training settings, or matched controls for scheduling and chunking. Section 2.3’s DP estimate assumes 1B trainable parameters, whereas the main experiment freezes Whisper and trains the adapter; that estimate is not a direct baseline for the actual workload. Inference interference under mixed load, dynamic scheduling, and failure redispatch are largely described rather than quantified. The incremental value over a simpler implementation therefore remains unclear.

  2. Gradient fidelity and the claim of no convergence degradation are insufficiently supported. Table 2 gives median cosine similarities of 0.945/0.966 without clearly defining the reference gradient, sampling/aggregation unit, or why same-machine comparisons are substantially below one. Section 5.1 infers no heterogeneous-training degradation from six paired measurements, without matched monolithic-training loss curves or final-quality differences. A gradient-returning service needs to rule out systematic effects from precision, loss/token normalization, chunking, and recomputation. Strong downstream scores alone do not establish fidelity to the intended training computation.

Questions for authors
  1. With device count, global batch, frozen model, and actual trainable adapter size matched, what are TiS’s step time and communication volume relative to a simple remote split-learning implementation? Please use at least a fixed-assignment or unchunked control to identify the benefit of the proposed scheduling or transfer mechanism.

  2. What implementation and precision produce the reference gradients in Table 2? Can you report relative gradient error, directional error, and matched short training curves or final adaptation quality for a synchronous monolithic reference and TiS using the same data and seed?

  3. How many 1 Gbps links does the edge use in the four-node experiment, and what are its aggregate bandwidth and measured utilization? Please also specify the actual trainable modules, parameter count, optimizer/learning rate, and training-data construction so the communication analysis can be connected to the three-day run.

4 comments

[W1] The evaluation does not separate TiS-specific systems benefits from adding hardware and using established split learning. Table 2 compares different node counts within TiS, but lacks a simple remote-gradient baseline under matched hardware, frozen model, and training settings, or matched controls for scheduling and chunking. Section 2.3's DP estimate assumes 1B trainable parameters, whereas the main experiment freezes Whisper and trains the adapter; that estimate is not a direct baseline for the actual workload. Inference interference under mixed load, dynamic scheduling, and failure redispatch are largely described rather than quantified. The incremental value over a simpler implementation therefore remains unclear.

We agree with the reviewer's observation: the value of TiS is a different way of using compute—putting the inference pool (vLLM) to work for training and letting multiple modality adapters share one frozen backbone. The deployment unit therefore changes from a training rank to an interchangeable inference worker: it is stateless between steps and reachable only through the API, which is what allows throughput-proportional assignment, membership changes within one step, and one checkpoint serving both training and production inference. The measurements below show how much compute this deployment can schedule.

  • Scale-out and pooling. Worker scale-out (1→16) is measured in Fig. 4(a), and heterogeneous pooling in Fig. 4(b): four 910C workers alone sustain 20.4 samples/s, two RTX 5880 Ada workers alone 5.0 samples/s, and the six pooled workers reach 25.1 samples/s against 25.4 for the two pools run separately; the small gap comes from pooling and transfer overhead.
  • Communication. On communication, Appendix A.3 states that the 1B/DP estimate is not a baseline for this workload, and the same-four-node comparison is in Fig. 5(b); TiS has a fixed cost that does not grow with trainable parameters.
  • Scheduling. On scheduling, the scheduler follows the same principle as throughput-based scheduling and is not a new scheduling algorithm: at each step boundary it estimates each worker's sustainable throughput and assigns samples in proportion, so a slow worker does not become the straggler for the whole step. Eq. (2) balances token time rather than sample count and includes communication time in the makespan.
  • Training–inference co-running. The revision adds this experiment (Fig. 4(c)) on the same 4-worker 910C pool, with 256 in-flight requests per worker on the inference side: training alone sustains 20.40 samples/s; inference alone 1229 tokens/s; running both, training sustains 8.27 samples/s while inference sustains 1127 tokens/s, a drop of only 8.3% in inference throughput.
  • KV-cache. On the KV-cache point, there is a misunderstanding: the inference path always uses its KV cache. The two modes share the same checkpoint resident in GPU memory; only the training path runs a single parallel prefill without allocating an autoregressive KV cache. Training therefore neither disables nor contends for the inference KV cache.
  • Elasticity and failure recovery. Elasticity and failure recovery share one liveness rule: at step boundaries the scheduler probes whether a worker is online and estimates its throughput, then adds or removes members and re-dispatches samples. Elasticity is measured in Fig. 5(a): the pool grows from 8 to 12 and shrinks to 4, holding at 35.7, 46.2, and 20.4 samples/s; each membership change requires no restart of the training process. Failure re-dispatch uses the same check: timed-out samples return to the queue by sample ID.
WorkloadTraining (samples/s)Inference (tokens/s)
Training only20.40—
Inference only—1229
Both concurrently8.27 (−59.5%)1127 (−8.3%)

[W2] Gradient fidelity and the claim of no convergence degradation are insufficiently supported. Table 2 gives median cosine similarities of 0.945/0.966 without clearly defining the reference gradient, sampling/aggregation unit, or why same-machine comparisons are substantially below one. Section 5.1 infers no heterogeneous-training degradation from six paired measurements, without matched monolithic-training loss curves or final-quality differences. A gradient-returning service needs to rule out systematic effects from precision, loss/token normalization, chunking, and recomputation. Strong downstream scores alone do not establish fidelity to the intended training computation.

  • Reference gradient. We accept the criticism that the reference gradient was not clearly defined, and the revision gives a clear definition: the reference gradient is the gradient computed with respect to the input embeddings of a fixed batch, on the same frozen checkpoint. To test whether the returned gradient is faithful, we send the same batch of embeddings to vLLM backends running on Ascend 910C and on NVIDIA RTX 5880 Ada, and compare the returned gradients by cosine similarity (table below).
  • Cross-vendor agreement. All pairs agree at 1.000, so the return path is insensitive to accelerator class and parallel layout; for identical embedding inputs, different vendors return identical gradients.
  • Original comparison. The invalid comparison result in the original draft has been removed from the main text, and Appendix C provides the corresponding explanation.
SettingBackend pairCosine similarity
Same vendor, repeated910C vs 910C1.000
Same vendor, repeatedRTX 5880 Ada vs RTX 5880 Ada1.000
Cross vendor910C vs RTX 5880 Ada1.000

[Q1] With device count, global batch, frozen model, and actual trainable adapter size matched, what are TiS's step time and communication volume relative to a simple remote split-learning implementation? Please use at least a fixed-assignment or unchunked control to identify the benefit of the proposed scheduling or transfer mechanism.

On communication volume:

  • Accounting. We agree that the 1B/DP figure is an order-of-magnitude reference, not a baseline, and Appendix A.3 now states that the two accountings are not directly comparable. The replacement is a same-four-node, same-checkpoint comparison (Fig. 5(b)); the accompanying table reports the result.
  • Claim. The claim is therefore adjusted to "TiS has a fixed communication cost that does not grow with trainable parameters," not that it is smallest at every scale.
  • Overlap. On communication–computation overlap: with batch 1024, K=80, D=5376, BF16, TiS streams the transfer in chunks of 32 samples. The bidirectional payload is 1.76 GB; upload and download run concurrently, so one direction carries 0.88 GB raw (1.18 GB on the wire after base64), which takes 9.4 s at 1 Gbps and is below the 17 s compute window. The link is therefore not the bottleneck.

On scheduling and the allocation comparison:

  • Rule. The scheduler addresses a data-assignment problem: it checks which workers are online, estimates each worker's sustainable throughput (samples/s or tokens/s), and assigns the current step's samples in proportion to that throughput—slower workers receive fewer or shorter samples, faster workers more. It follows the same principle as throughput-based scheduling and is not a new scheduling algorithm.
  • Example. Fig. 4(b) is a concrete scheduling case: the pool has four 910C workers and two RTX 5880 Ada workers. The two classes differ substantially in compute—20.4 samples/s for the 910C pool alone and 5.0 samples/s for the 5880 Ada pool alone. If samples were split by worker count, each 910C worker would finish its share early and then wait for the two RTX workers to process the same number of samples, so the step time would be set by the two RTX workers and the 910C workers would sit idle for most of the step. At each step boundary the scheduler measures each worker's sustainable throughput and assigns the step's samples in proportion to it: 910C workers receive more (or longer) samples and 5880 Ada workers fewer (or shorter) ones, equalizing their compute time. The six-worker pool then reaches 25.1 samples/s.
MethodPer-step traffic, 4 nodes
TiS, adapter only (85.12M)1.76 GB
TiS, encoder + adapter (722.09M)1.76 GB
ZeRO++, adapter only (85.12M)1.02 GB
ZeRO++, encoder + adapter (722.09M)8.67 GB (4.9× TiS)

[Q2] What implementation and precision produce the reference gradients in Table 2? Can you report relative gradient error, directional error, and matched short training curves or final adaptation quality for a synchronous monolithic reference and TiS using the same data and seed?

  • Implementation and precision. All runs use BF16 training: the reference is a single-process PyTorch forward/backward on the same frozen checkpoint, and TiS returns BF16 embeddings and gradients.
  • Reference gradient. We accept the criticism that the reference gradient was not clearly defined, and the revision gives a clear definition: the reference gradient is the gradient computed with respect to the input embeddings of a fixed batch, on the same frozen checkpoint. To test whether the returned gradient is faithful, we send the same batch of embeddings to vLLM backends running on Ascend 910C and on NVIDIA RTX 5880 Ada, and compare the returned gradients by cosine similarity (table below).
  • Cross-vendor agreement. All pairs agree at 1.000, so the return path is insensitive to accelerator class and parallel layout; for identical embedding inputs, different vendors return identical gradients.
  • Original comparison. The invalid comparison result in the original draft has been removed from the main text, and Appendix C provides the corresponding explanation.
SettingBackend pairCosine similarity
Same vendor, repeated910C vs 910C1.000
Same vendor, repeatedRTX 5880 Ada vs RTX 5880 Ada1.000
Cross vendor910C vs RTX 5880 Ada1.000

[Q3] How many 1 Gbps links does the edge use in the four-node experiment, and what are its aggregate bandwidth and measured utilization? Please also specify the actual trainable modules, parameter count, optimizer/learning rate, and training-data construction so the communication analysis can be connected to the three-day run.

Links and utilization. Each cloud–edge pair is connected by a 1 Gbps link, so the four-node experiment uses four such links, and all experiments run at 1 Gbps (Fig. 5(c)), whose simulation also covers other access rates. The per-step payload is 1.76 GB bidirectional—0.88 GB one way raw, 1.18 GB with base64 (Table 1). Across the four links that is 0.29 GB per link one way, 2.3 s at 1 Gbps; aggregate traffic is about 4 Gbps, or roughly 13–14% of the 17 s compute window.

Modules and data. The trainable modules are the Q-Former (69.37M) and MLP (15.75M), 85.12M in total; Whisper-large-v3 (636.97M) and the 31B backbone are frozen. K=80, D=5376, BF16, global batch 8×128=1024; the data is Common Voice ASR followed by FLEURS S2TT (Table 3).

ItemValue
Links1 Gbps between each cloud and edge; 4 links in total
Traffic per link0.29 GB one way (base64), 2.3 s at 1 Gbps
Aggregate≈4 Gbps, 13–14% of the 17 s compute window
Trainable modulesQ-Former 69.37M + MLP 15.75M = 85.12M; Whisper 636.97M and 31B frozen
DataCommon Voice ASR → FLEURS S2TT (Table 3)
Reviewer_okd4OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents Training is Serving (TiS), a framework for using distributed LLM inference services to support multimodal adaptation. The main idea is to keep the pretrained LLM frozen on remote servers while training the modality encoder and adapter locally. During training, the client sends embeddings and labels to the server and receives gradients for updating its local modules. This allows training across independently managed machines without synchronizing model parameters between servers.

TiS also introduces a scheduler that distributes training workloads based on available compute and network bandwidth, supporting heterogeneous accelerators and elastic workers. The authors evaluate the framework by extending Gemma-4-31B-it with speech capabilities, covering 70 languages and 4,830 translation directions. The resulting model achieves competitive translation quality, while the system reports 3.75× throughput scaling across four service nodes.

Critical strengths
  • The paper addresses a practical problem. Sharing idle GPUs and NPUs across different machines and teams is useful, especially when these machines cannot be managed as a single cluster.

  • I like the idea of using existing LLM services for training. Keeping the LLM frozen and only sending embeddings and gradients makes the design relatively simple, and avoids expensive model synchronization.

  • The experiments use a fairly large 31B model and cover 70 languages, which is a good scale for demonstrating the idea. The reported 3.75× throughput scaling across four nodes is also promising.

Critical weaknesses
  • I think the novelty is quite limited. The main idea is basically split learning: keeping a frozen LLM remotely, sending embeddings, and getting gradients back. This is not new, and the paper doesn't really explain what is fundamentally different from existing split learning or collaborative training methods. The related work discussion is also quite weak here.

  • One thing that really concerns me is Figure 1 (Page 2, Lines 066–070), where the authors explicitly call it "A hypothetical TiS deployment scenario." They repeat this in Lines 079–085. But heterogeneous compute sharing is supposed to be one of the main contributions. The actual scaling experiments in Table 2 only use homogeneous 910C nodes. So where is the real heterogeneous deployment that the paper keeps talking about?

  • Despite the title "Training is Serving", I don't see any real experiments showing training and inference running together. In §3.2, the authors even mention disabling the inference KV-cache path during gradient computation. There is no inference latency, interference analysis, or training throughput under concurrent inference workloads. This is probably the most important experiment for this paper, but it is completely missing.

  • The paper introduces a heterogeneous scheduler in §3.2, but I don't see a proper evaluation of it. There is no comparison with simple load balancing, equal batch allocation, or throughput-based scheduling. The algorithm itself is also not explained in much detail. I really can't tell how much the scheduler contributes to the reported performance.

  • A lot of other system features are described but barely evaluated. For example, §3.2 talks about elastic workers, fault recovery, cross-region training, chunk streaming, and compute credits. But there is no failure injection, dynamic worker joining/leaving experiment, WAN evaluation, or meaningful ablation. I am not even sure how much of the complete system was actually tested.

  • The gradient consistency results in Table 2 (Page 7) are also concerning. The median cosine similarity is only 0.945 on 910C, with some values as low as 0.802. What exactly is the reference gradient? Why are the differences this large? The authors then claim heterogeneous training converges without degradation based on six paired measurements, but provide no detailed convergence curves or final accuracy comparisons. I find this claim pretty hard to accept with so little evidence.

  • The communication comparison in §2.3 doesn't seem fair. The authors compare synchronizing 1B trainable parameters using DP against TiS transferring embeddings and gradients. But these are different training setups. In the actual experiment, the Whisper encoder and LLM are frozen, and only the adapter is trained. The claimed communication advantage mostly comes from the split-learning setup, rather than a new communication mechanism. Also, with the actual batch size of 1024, the embedding traffic is about 1.76 GB per step. The paper should explain the network bottleneck and how much communication really overlaps with computation.

  • The multilingual results look good, but they don't really validate TiS itself. Training a speech adapter on Gemma without TiS could potentially achieve the same quality. There is no direct comparison with conventional local training or standard split learning. Also, §4 gives very limited training details. It is not clear how the translation supervision for 4,830 directions was constructed, how many training steps were used, or what the actual training data distribution looks like. For such a large multilingual experiment, I would expect much more information.

  • The appendix is also not as complete as the paper suggests. It is called "Full Results", but Table 4 only provides detailed English-to-69 results, while Table 5 gives averages over 69 targets for each source language. These are not the full results for all 4,830 translation directions. Some low-resource languages also perform quite poorly, but the paper barely discusses these cases.

  • There are also a few things that make me question the maturity of the implementation. On Page 2, Lines 090–091, the authors say they will release the code and models. On Page 9, Lines 477–479, they again promise to release the implementation, configurations, checkpoints, and prompts. On Page 9, Lines 466–468, they say service-side parameter-efficient updates are a next step, so the current system is still limited to input-side adaptation. None of these are necessarily wrong, but together with the missing experiments, it becomes difficult to verify how complete the system really is.

  • Overall, the paper feels a bit over-engineered to me. It has a very complete and polished system story, covering almost everything from heterogeneous scheduling to fault tolerance and compute credits. But the actual evaluation is mostly multilingual model quality and homogeneous throughput scaling. I didn't find explicit "not implemented" or "simulated" experiments, and I'm not saying the results are fabricated. Still, the writing has some AI-slop-like feeling, where the story sounds much more complete than the supporting evidence. I think the authors need to demonstrate the key system mechanisms rather than just describe them.

Some AI signs:

Page 2, Figure 1 (Lines 066–070) explicitly says "A hypothetical TiS deployment scenario." The authors repeat this in Lines 079–085. This is quite concerning since heterogeneous compute sharing is one of the main selling points. The figure looks like a complete system deployment, but it is actually hypothetical.

The paper repeatedly emphasizes four major advantages, but the experiments mostly show multilingual translation quality and homogeneous throughput scaling. The story is much more complete than the actual evidence, which gives me some AI-slop-like feeling.

Page 2 (Lines 090–091) and Page 9 (Lines 477–479) promise future code, model, and checkpoint releases. Page 9 (Lines 466–468) also says service-side parameter-efficient updates are future work. These are not necessarily problems by themselves, but they make it harder to verify the implementation. I didn't find explicit "not implemented" or "simulated" experiments, though.

Clear Technical and Experimental Problems

The novelty is quite limited. Sending embeddings to a frozen remote LLM and returning gradients is basically split learning. I don't see a strong enough technical difference from existing approaches, and there is no proper comparison with conventional split learning.

There is no actual joint training-serving experiment. For a paper called "Training is Serving", this is a major problem. The authors even disable the inference KV-cache path during gradient computation (§3.2), but provide no inference latency, interference, or concurrent workload results. I don't think the central claim is properly demonstrated.

The heterogeneous scheduling claim is not well supported. Table 2 only reports scaling on homogeneous 910C nodes. There is no actual heterogeneous throughput comparison, and no scheduler ablation against simple load balancing. So it's hard to tell whether the proposed scheduler is useful at all.

The gradient consistency results are questionable. Table 2 reports median cosine similarity of 0.945 on 910C, with values as low as 0.802. The reference gradient is not clearly explained. The authors also claim no convergence degradation, but there are no detailed convergence curves or convincing accuracy comparisons.

The communication comparison in §2.3 is not really fair. The authors compare DP synchronization of 1B trainable parameters against transferring embeddings and gradients in TiS. But the actual training setup freezes the LLM and Whisper encoder. The communication advantage largely comes from changing the training setup, not necessarily from a new system mechanism.

The multilingual training details are surprisingly limited. It is unclear how the supervision for all 4,830 translation directions was constructed, how much training data was used, and what the full optimization setup looks like. The appendix is called "Full Results", but it doesn't actually provide individual results for all 4,830 directions.

Several important system claims have almost no experimental support. Fault recovery, elastic worker joining/leaving, cross-region training, and chunk streaming are all discussed in §3.2, but there are no corresponding experiments or meaningful ablations.

Questions for authors

The title is "Training is Serving", but I couldn't find experiments with training and inference running together. Can the authors show how training affects inference latency and throughput? This seems pretty important for the main claim.

How much does the proposed scheduler actually help? Could the authors compare it with simple load balancing or throughput-based scheduling?

I'm a bit confused by the gradient consistency results in Table 2. What is the reference gradient? Why is the cosine similarity sometimes as low as 0.802? Could the authors provide actual convergence curves for mixed-hardware training?

6 comments

[W1] I think the novelty is quite limited. The main idea is basically split learning: keeping a frozen LLM remotely, sending embeddings, and getting gradients back. This is not new, and the paper doesn't really explain what is fundamentally different from existing split learning or collaborative training methods. The related work discussion is also quite weak here.

We thank the reviewer for pressing on this point. The review describes the data path—embeddings out, input gradients back—but the data path is the interface, not the contribution. The paper's claim concerns what the service side is allowed to be.

Classical split learning places the server inside the training system: the server holds trainable parameters, runs a training process for the duration of the job, and its parallelism is fixed when the job starts.

TiS makes a different assumption: the edge and service sides are administratively isolated, the training party needs no SSH access or server control, and the two sides interact only through the API. The process resident on the service side is an inference process, and it holds no training state between steps. A TiS worker is an inference worker plus a gradient-return path, serving standard inference and training-gradient-return requests from the same resident checkpoint. This changes the deployment unit from "a training rank" to "an interchangeable worker."

This is the difference the paper measures. The following are the ways TiS differs from split learning:

  • The framework supports multiple modality adapters sharing one frozen LLM backbone, with no aggregation and no global model state; split learning instead trains a single split model.
  • Each worker can join or leave within one step: 8→12→4 workers at 35.7/46.2/20.4 samples/s, with no restart and no change to other workers' layouts (Fig. 5a).
  • Training and production inference share one checkpoint; co-running reduces inference throughput by only 8.3% (1229→1127 tokens/s) while training sustains 8.27 samples/s (Fig. 4c).
  • A GPU–NPU pool serves one checkpoint without a spanning communicator: 20.4 + 5.0 samples/s separately versus 25.1 samples/s pooled (Fig. 4b).

For FedNano and UniFL specifically:

  • Aggregation: federated methods merge client updates into one global model and need a coordinator that owns model state, whereas TiS never aggregates or updates the backbone, so workers stay stateless between steps and the provider keeps its weights and parallel layout.
  • Software stack: federated aggregation assumes clients run backward through the same architecture, whereas TiS exposes only a gradient-return API, so clients may differ in accelerators, collective libraries, and parallel layouts.
  • Unit of communication: federated methods move parameters or low-rank factors, scaling with the adapter, whereas TiS moves fixed-width embeddings and their gradients, set by K, D, and the batch, and independent of the number of trainable parameters.

TiS is therefore a pool of independently managed inference services rather than a federation of training clients.

[W2] One thing that really concerns me is Figure 1 (Page 2, Lines 066–070), where the authors explicitly call it "A hypothetical TiS deployment scenario." They repeat this in Lines 079–085. But heterogeneous compute sharing is supposed to be one of the main contributions. The actual scaling experiments in Table 2 only use homogeneous 910C nodes. So where is the real heterogeneous deployment that the paper keeps talking about?

Figure 1 illustrates the application scenario of the framework; it is not experimental evidence. The heterogeneous measurement added in Fig. 4(b) is: four 910C workers alone sustain 20.4 samples/s, two RTX 5880 Ada workers alone 5.0 samples/s, and the six pooled workers reach 25.1 samples/s, against a sum of 25.4 samples/s for the two single-vendor pools.

ConfigurationThroughput (samples/s)
910C, 4 workers, alone20.4
RTX 5880 Ada, 2 workers, alone5.0
Pooled (910C 4 workers + 5880 Ada 2 workers)25.1

[W3] Despite the title "Training is Serving", I don't see any real experiments showing training and inference running together. In §3.2, the authors even mention disabling the inference KV-cache path during gradient computation. There is no inference latency, interference analysis, or training throughput under concurrent inference workloads. This is probably the most important experiment for this paper, but it is completely missing.

The revision adds this experiment (Fig. 4(c)) on the same 4-worker 910C pool, with 256 in-flight requests per worker on the inference side: training alone sustains 20.40 samples/s; inference alone 1229 tokens/s; running both, training sustains 8.27 samples/s while inference sustains 1127 tokens/s, a drop of only 8.3% in inference throughput.

On the KV-cache point, there is a misunderstanding: the inference path always uses its KV cache. The two modes share the same checkpoint resident in GPU memory; only the training path runs a single parallel prefill without allocating an autoregressive KV cache. Training therefore neither disables nor contends for the inference KV cache.

WorkloadTraining (samples/s)Inference (tokens/s)
Training only20.40—
Inference only—1229
Both concurrently8.27 (−59.5%)1127 (−8.3%)

[W4] The paper introduces a heterogeneous scheduler in §3.2, but I don't see a proper evaluation of it. There is no comparison with simple load balancing, equal batch allocation, or throughput-based scheduling. The algorithm itself is also not explained in much detail. I really can't tell how much the scheduler contributes to the reported performance.

The scheduler addresses a data-assignment problem: it checks which workers are online, estimates each worker's sustainable throughput (samples/s or tokens/s), and assigns the current step's samples in proportion to that throughput—slower workers receive fewer or shorter samples, faster workers more. It follows the same principle as throughput-based scheduling and is not a new scheduling algorithm.

Fig. 4(b) is a concrete scheduling case: the pool has four 910C workers and two RTX 5880 Ada workers. The two classes differ substantially in compute—20.4 samples/s for the 910C pool alone and 5.0 samples/s for the 5880 Ada pool alone. If samples were split by worker count, each 910C worker would finish its share early and then wait for the two RTX workers to process the same number of samples, so the step time would be set by the two RTX workers and the 910C workers would sit idle for most of the step. At each step boundary the scheduler measures each worker's sustainable throughput and assigns the step's samples in proportion to it: 910C workers receive more (or longer) samples and 5880 Ada workers fewer (or shorter) ones, equalizing their compute time. The six-worker pool then reaches 25.1 samples/s.

[W5] A lot of other system features are described but barely evaluated. For example, §3.2 talks about elastic workers, fault recovery, cross-region training, chunk streaming, and compute credits. But there is no failure injection, dynamic worker joining/leaving experiment, WAN evaluation, or meaningful ablation. I am not even sure how much of the complete system was actually tested.

  • Elasticity and failure recovery share one liveness rule: at step boundaries the scheduler probes whether a worker is online and estimates its throughput, then adds or removes members and re-dispatches samples. Elasticity is measured in Fig. 5(a): the pool grows from 8 to 12 and shrinks to 4, holding at 35.7, 46.2, and 20.4 samples/s; each membership change requires no restart of the training process. Failure re-dispatch uses the same check: timed-out samples return to the queue by sample ID.
  • Cross-region training is not limited by the software stack: both sides interact only through the API, so deployability depends only on link bandwidth. Fig. 5(c) gives the bandwidth requirement: at 1 Gbps one direction takes 9.4 s against a ~17 s compute window, with a crossover near 555 Mbit/s; we recommend at least 1 Gbps.
  • Chunk streaming also follows from the API boundary: a 1,024-sample step can be split into smaller batches (e.g., 32) and sent to the vLLM endpoint separately, which reduces the time compute waits on transfer and overlaps transfer with remote computation.
  • Compute credits are an incentive layer for compute sharing within a LAN: contributing idle capacity earns credits, and requesting service spends them.

[W6] The gradient consistency results in Table 2 (Page 7) are also concerning. The median cosine similarity is only 0.945 on 910C, with some values as low as 0.802. What exactly is the reference gradient? Why are the differences this large? The authors then claim heterogeneous training converges without degradation based on six paired measurements, but provide no detailed convergence curves or final accuracy comparisons. I find this claim pretty hard to accept with so little evidence.

  • Reference gradient. We accept the criticism that the reference gradient was not clearly defined, and the revision gives a clear definition: the reference gradient is the gradient computed with respect to the input embeddings of a fixed batch, on the same frozen checkpoint. To test whether the returned gradient is faithful, we send the same batch of embeddings to vLLM backends running on Ascend 910C and on NVIDIA RTX 5880 Ada, and compare the returned gradients by cosine similarity (table below).
  • Cross-vendor agreement. All pairs agree at 1.000, so the return path is insensitive to accelerator class and parallel layout; for identical embedding inputs, different vendors return identical gradients.
  • Original comparison. The invalid comparison result in the original draft has been removed from the main text, and Appendix C provides the corresponding explanation.
SettingBackend pairCosine similarity
Same vendor, repeated910C vs 910C1.000
Same vendor, repeatedRTX 5880 Ada vs RTX 5880 Ada1.000
Cross vendor910C vs RTX 5880 Ada1.000

[W7] The communication comparison in §2.3 doesn't seem fair. The authors compare synchronizing 1B trainable parameters using DP against TiS transferring embeddings and gradients. But these are different training setups. In the actual experiment, the Whisper encoder and LLM are frozen, and only the adapter is trained. The claimed communication advantage mostly comes from the split-learning setup, rather than a new communication mechanism. Also, with the actual batch size of 1024, the embedding traffic is about 1.76 GB per step. The paper should explain the network bottleneck and how much communication really overlaps with computation.

  • Accounting. We agree that the 1B/DP figure is an order-of-magnitude reference, not a baseline, and Appendix A.3 now states that the two accountings are not directly comparable. The replacement is a same-four-node, same-checkpoint comparison (Fig. 5(b)); the accompanying table reports the result.
  • Claim. The claim is therefore adjusted to "TiS has a fixed communication cost that does not grow with trainable parameters," not that it is smallest at every scale.
  • Overlap. On communication–computation overlap: with batch 1024, K=80, D=5376, BF16, TiS streams the transfer in chunks of 32 samples. The bidirectional payload is 1.76 GB; upload and download run concurrently, so one direction carries 0.88 GB raw (1.18 GB on the wire after base64), which takes 9.4 s at 1 Gbps and is below the 17 s compute window. The link is therefore not the bottleneck.
MethodPer-step traffic, 4 nodes
TiS, adapter only (85.12M)1.76 GB
TiS, encoder + adapter (722.09M)1.76 GB
ZeRO++, adapter only (85.12M)1.02 GB
ZeRO++, encoder + adapter (722.09M)8.67 GB (4.9× TiS)

[W8] The multilingual results look good, but they don't really validate TiS itself. Training a speech adapter on Gemma without TiS could potentially achieve the same quality. There is no direct comparison with conventional local training or standard split learning. Also, §4 gives very limited training details. It is not clear how the translation supervision for 4,830 directions was constructed, how many training steps were used, or what the actual training data distribution looks like. For such a large multilingual experiment, I would expect much more information.

We agree the multilingual results validate that the framework can carry a real 31B, 70-language training run, not that TiS trains better than training without it.

Training details are now in Table 3, which has three columns including a Detail column: an 85.12M adapter (Q-Former 69.37M + MLP 15.75M), K=80, D=5376, BF16, global batch 8×128=1024, Common Voice ASR adaptation followed by FLEURS S2TT SFT, and three days on four nodes with 16 workers.

ComponentValue
Adapter (trainable)85.12M = Q-Former 69.37M + MLP 15.75M
EncoderWhisper-large-v3, 636.97M, frozen
BackboneGemma-4-31B-it, frozen
Adapter tokens / widthK=80 / D=5376
PrecisionBF16
Global batch8×128 = 1024
DataCommon Voice ASR → FLEURS S2TT
Runtime3 days on 4 nodes, 16 workers

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.