acceptodds
Under review as a conference paper at ICLR 2027

Early-Exit-Aware Partitioning for Distributed DNN Inference

Abstract

An early-exit strategy allows samples to exit when a certain confidence threshold is reached at intermediate layers of the model. However, distributing early-exit networks across multiple machines introduces unique complications. Applying naive layer-wise partitioning can negate the advantages of early exits: unlike conventional model parallelism, where all inputs traverse every partition, the volume of data reaching downstream machines varies. This produces an uneven workload across the cluster. Addressing this imbalance requires jointly deciding where to place early exits and how to partition and distribute the network across nodes. In this paper, we propose a load-balanced, distributed, model-parallel inference strategy that allocates compute capacity in proportion to the inference load at each stage. We evaluate our approach across multiple CNN architectures (DenseNet-121, ResNet-50, and AlexNet) on standard benchmark datasets (ImageNet, CIFAR-100, and CIFAR-10). Experimental results demonstrate that our architecture significantly reduces inference latency and outperforms conventional model partitioning.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
Early-Exit DNN, Distributed Inference

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_WtAHOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper looks at distributed inference for early-exit neural networks. Since some samples exit at intermediate classifiers, different stages receive different amounts of work, leading to load imbalance. The proposed system combines asynchronous execution, stage replication, and Dynamic Stage In-lining. The latter limits remote transfers and handles excess downstream work locally by swapping model weights. The evaluation covers AlexNet/CIFAR-10, DenseNet/CIFAR-100, and ResNet-50/ImageNet on one to five GPU machines connected over 10 Gbps Ethernet.

The main argument is that reducing computation through early exits does not automatically improve system performance. Execution, communication, and downstream capacity also need to be coordinated. The paper reports throughput gains of up to 6.23× over sequential distributed baselines, with a separate 3.98× improvement attributed to stage in-lining. My main concern is that the evaluation does not clearly establish whether these gains hold under comparable hardware budgets and accuracy constraints. The claimed improvements in request latency are also not adequately supported by the experiments. I find the novelty of the core contribution limited relative to prior work(SPINN/Apparate), as discussed in the weaknesses below.

Score updated from Weak Reject to Weak Accept during Rebuttal

Critical strengths
  1. The problem is practically relevant and well motivated. Early exits change downstream traffic as well as computation. The paper correctly emphasizes that reducing FLOPs does not guarantee faster distributed execution when serialization, communication, and scheduling dominate.

  2. The evaluation uses physical machines connected via interconnect and exposes meaningful failure cases. Testing multiple CNN workloads across actual GPU nodes is more informative than relying exclusively on simulated compute savings. Reporting cases where replication makes performance worse, particularly ResNet-50 at moderate exit rates, helps identify the system’s operating limits.

  3. The proposed local execution fallback is a plausible systems mechanism. Retaining excess activations locally can reduce pressure on a congested interconnect. Combining this with stage replication could provide useful flexibility when network bandwidth and compute capacity are imbalanced.

  4. The paper examines several interacting deployment choices. Batch size, exit rate, model architecture, and execution topology all affect performance. The results show that a configuration beneficial for one workload may be ineffective for another.

Critical weaknesses

The paper would be stronger with the following clarifications and improvements in the paper:

  1. The comparisons do not isolate the benefit of the proposed scheduling and partitioning strategy making it challenging to grasp the core benefit.

The principal comparisons change both execution strategy and hardware allocation: Sequential-2 versus Multi-Stream-3, and Sequential-3 versus Multi-Stream-5. Consequently, the reported improvements combine additional compute capacity with concurrency and scheduling changes. In Table 2, the 6.23× comparison is numerically correct, but the five-node configuration is also 11.6% slower than the three-node multi-stream configuration at this operating point. This limits what the headline establishes about scaling. The evaluation needs an asynchronous pipeline using the same number of nodes, and full-model replication across the same hardware where feasible. Additionally, the single-node non-exiting baseline evaluates auxiliary classifiers even though it never uses their predictions. A backbone-only baseline would reveal the overhead introduced by those classifiers.

  1. Accuracy losses substantially qualify the reported speedups diminishing the core contribution

Table 1 reports ResNet-50 accuracy falling from 83% to 76% Top-1 at the designated primary operating point of approximately 50% exits. At approximately 90% exits, the operating regime used for the three-stage comparisons, accuracy falls to 51%, a 32-percentage-point reduction. Comparing execution strategies at the same exit policy remains informative. However, these results do not establish that the resulting configurations provide an attractive trade-off against full-model inference. Calling an operating point “optimal” requires an explicit accuracy constraint or application objective.

The paper should report accuracy–throughput and accuracy–latency frontiers, with comparisons at common accuracy targets. The statement that stage in-lining improves performance “without sacrificing accuracy” should specify whether it means preserving the selected early-exit policy’s accuracy or preserving full-model accuracy.

  1. Throughput improvements are repeatedly interpreted as latency improvements. This also makes the paper hard to follow.

Table 2 reports samples per second, while the abstract and discussion emphasize inference latency. Equation 2 describes an idealized pipeline completion interval under overlapping execution, rather than the response time experienced by an individual request. Higher throughput can coexist with longer request latency, particularly when batches wait in queues or local inference is deferred until a later weight-swapping phase. The current measurements therefore do not establish the claimed latency benefits. The authors should distinguish request latency, batch completion time, and steady-state throughput, and report median and tail request latency at specified arrival rates. This distinction is especially important for evaluating Dynamic Stage In-lining.

  1. The core contribution 3.98× ResNet-50 improvement needs additional experimental results.

Dynamic Stage In-lining is central to the paper’s novelty and conclusion, but its reported gains, including the 3.98× ResNet-50 improvement, appear in prose rather than a reproducible results table. The note following Table 2 explicitly excludes in-lining from those measurements and attributes its evaluation to dynamic traffic bursts. Adding the burst workload, the baseline denominator for each speedup, the relevant batch size, or the contributions of queue draining and weight-transfer overhead will improve the paper. It also remains unclear how the remote-transfer threshold is selected and whether it adapts to measured conditions.

  1. The mathematical framework needs more precise assumptions and additional clarifications.

Equation 3 equates downstream arrivals with the upstream stage’s maximum processing capacity multiplied by its non-exit fraction. This equality assumes a saturated upstream stage and consistent units. More generally, downstream arrivals depend on the actual departure rate, which can be lower because of limited input traffic or backpressure.

Similarly, Equation 5’s approximate capacity equality is a useful sizing heuristic, but does not establish queue stability under variable arrivals and service times. Operating near full utilization can produce substantial queueing delays. The model also needs to clarify whether communication links are shared across replicas and how computation time changes with the dynamically shrinking batch sizes.

These limitations matter because the equations are presented as supporting broad claims about preventing backpressure and eliminating stalls. The authors should state the assumptions explicitly and validate the model against measured arrival rates, queue occupancy, and throughput.

  1. Low novelty over prior work.

SPINN already jointly adapts early-exit policies and model partitioning to changing execution conditions. Ebrahimi et al.’s Combining DNN Partitioning and Early Exit also develops a joint performance model for partitioned early-exit execution. The manuscript cites these works but does not empirically establish what its approach adds beyond them. arxiv.org

The comparison with Apparate also needs qualification. Apparate deliberately continues computation after returning early predictions to obtain feedback for accuracy monitoring and adaptation, while retaining compatibility with batching. Removing this computation introduces a different trade-off; it does not by itself establish a superior serving design. arxiv.org

Furthermore, although the abstract motivates jointly selecting exit locations and partitions, the method fixes partition boundaries at exit locations and evaluates prescribed topologies. An explicit optimization procedure for selecting exit placement, replication, and routing is not provided. The contribution should therefore be framed more narrowly unless such a procedure and its benefits are demonstrated.

Questions for authors

Please address each of the weakness points -

Reviewer_qrHPOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper studies distributed, model-parallel inference of early-exit CNNs, with two exit heads added to AlexNet, DenseNet and ResNet-50 . Model cuts are placed at the exit boundaries. The testing-bed setup has 2–5 Quadro RTX 4000 GPUs connected by 10 Gbps Ethernet. Six execution paradigms are compared. Reported results show that the proposed method speedups of Multi-Stream over Sequential of up to 6.23× and Multi-Stream-3 is slower than Sequential-2.

Critical strengths
  1. A practical problem with real-hardware measurements. The paper studies a problem that matters in edge and cluster inference.

  2. Capacity matching gives a sensible design heuristic. Eq. 5 states that stage replication should be matched to the non-exiting arrival rate. This is intuitive.

Critical weaknesses
  1. The evaluation has no data-parallel baseline. All three models fit easily in the 8 GB of a single RTX 4000. With 3–5 GPUs, the natural baseline is data-parallel replication of the full early-exit model, i.e., the Single-Node Early Exit configuration run independently on each GPU. This baseline has no inter-node activation traffic and roughly scales linearly. It is never reported.

  2. No single-node throughput. Single-node numbers are not shown in Table 2, so readers cannot tell whether any distributed configuration beats one GPU.

  3. Strawman serial baseline. The "Sequential" baseline is a non-pipelined serial chain with idle stages. Standard pipelined execution with micro-batching on the same nodes is missing.

  4. Unequal hardware in comparisons. Multi-Stream-5 (5 GPUs) is compared with Sequential-3 (3 GPUs) without normalizing throughput per GPU.

Questions for authors

Q1: As mentioned above, the headline speedups (2.18×, 3.70×, 6.23×) are relative to a weak baseline and may not exceed simple data parallelism, or even a single GPU.

Q2: The claimed method and theory are not delivered or validated, and the main mechanism's results are missing.

Q3: The author should discuss with partitioning or placement method. Nothing jointly optimizes exit placement, partition points or replication.

Q4: The capacity-matching condition is never used to choose R_k, and predicted throughput is never compared with measurements.

Q5: The claimed contributions (joint partitioning, formal capacity matching, in-lining) are not supported by an algorithm or by evidence.

Q6: The units of Tcomp (per batch or per sample) are unspecified, which matters for Eq. 3.

Q7: Please report, under those settings: 1) single-node throughput (with and without early exit); 2) data-parallel replication of the full early-exit model on the same number of GPUs; 3) a properly pipelined (micro-batched, asynchronous) Sequential baseline; 4) per-GPU normalized throughput for every configuration.

Reviewer_xJU5OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper addresses the systems problem of running early-exit neural networks across multiple machines. While early exits reduce computation by letting confident samples terminate at intermediate classifiers, naive layer-wise partitioning can erase these gains: unlike conventional model parallelism, the volume of activations reaching downstream nodes varies with each sample, producing uneven workloads, queueing, and communication stalls. The authors formalize the dynamics of distributed early-exit inference, deriving latency and flow relations that relate per-stage exit rates, worker replication factors, and compute/communication times, and establishing capacity-matching conditions needed to prevent interconnect backpressure. They also propose Dynamic Stage In-lining, a runtime mechanism that caps the fraction of samples offloaded to remote nodes and executes the overflow locally by swapping model weights in VRAM, effectively converting network stalls into local compute.

Critical strengths
  • Clear problem framing and motivation.

  • Experiments show that the proposed Dynamic Stage In-lining mechanism is a pragmatic, deployable fix that converts a 0.66× communication-bound slowdown into a 3.98× net speedup without new hardware.

Critical weaknesses
  • Narrow workload scope and limited accuracy analysis. Only three small CNN vision models are tested

  • There is no comparison against closely related baselines such as Apparate or SPNN under matched conditions

Questions for authors

Table 1 shows ResNet-50 dropping from 83%/96% (top-1/top-5) at η = 0% to 51%/77% at η ≈ 90%, and AlexNet from 84% to 77%. A latency win obtained by discarding a third of the accuracy is not a favorable trade-off. How about the accuracy–latency Pareto analysis, and at what point would you claim the system is actually useful?

The abstract claim "inference latency" reduction, but all results are steady-state throughput (samples/sec). Do you have per-request latency measurements, including warm-up, the first-batch cost of Dynamic Stage In-lining, and tail latency?

How dependent are the conclusions on the 10 Gbps Ethernet fabric? You acknowledge InfiniBand would shift optimal exit rates to η ≈ 30–50%. Since the entire motivation for Dynamic Stage In-lining is link saturation, what remains of the contribution on a modern interconnect?

Does the framework extend beyond static CNNs? You propose LLMs as future work, but autoregressive decoding has sequential dependencies and different memory profiles. Is there any reason to expect the capacity-matching conditions to carry over?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.