acceptodds
Under review as a conference paper at ICLR 2027

SparEdge: Redundancy-Aware Dynamic Sparse Inference with Edge-Cloud Collaboration for Multimodal LLMs

Abstract

Multimodal large language models (MLLMs) excel at cross-modal understanding and generation, yet their massive parameter size and heavy computation make inference prohibitively expensive, especially on resource-constrained edge devices. Cloud-only inference offers strong compute but suffers from high latency and bandwidth pressure. Existing acceleration schemes largely rely on static pruning or fixed sparsity patterns, and thus fail to adapt to the dynamically changing redundancy distributions in multimodal inputs. To address this, we propose SparEdge, a redundancy-aware dynamic sparse inference framework with edge-cloud collaboration for accelerating MLLM inference. First, a lightweight cross-modal redundancy-aware mechanism efficiently characterizes input-dependent redundancy across visual and textual representations with minimal additional overhead, enabling dynamic identification of redundant computation during inference. Second, an edge-cloud collaborative dynamic sparse inference mechanism translates the detected redundancy into adaptive sparsity decisions and coordinates computation between edge and cloud resources according to the runtime sparsity characteristics and resource conditions. Experiments on multiple datasets show that SparEdge improves inference throughput by more than 1.8× while maintaining accuracy, and reduces computational and memory overhead by more than 25.2%.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
Multimodal LLM, Inference Acceleration

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_NgErOpenReview ↗
Rating
1
Confidence
3(−1)
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes SparEdge, a framework combining cross-modal redundancy estimation, dynamic token sparsification, and edge–cloud computation partitioning to accelerate multimodal large language model inference. The method computes cross-modal interactions in a compact representation space, generates layer-wise token scores, and uses exponential smoothing, retention budgets, and dynamic programming to determine sparsity masks and execution locations. Experiments cover four backbones and three benchmarks—TextVQA, VQAv2, and MMBench—using an INT4 CPU edge deployment and an FP16 A100 cloud deployment. The paper reports improved throughput and reduced computation and memory consumption, supplemented by theoretical analyses and component ablations.

Critical strengths

The paper addresses a practically relevant problem: input-dependent token redundancy changes both computation and communication requirements, creating a potentially useful interaction between sparsity control and edge–cloud scheduling. Studying these decisions within a common execution framework is a worthwhile direction and provides the main motivation for considering the work.

Critical weaknesses

1.The central sparsity-control rules contain related directional errors, and the appendix does not establish the claimed correctness guarantees.

Section 4.1 explicitly defines a high score as indicating high redundancy. However, Eq. (5) and Algorithm 1 construct the retention mask using TopK. Under its conventional meaning, this operation retains the most redundant tokens. Appendix A.1 instead states that tokens whose redundancy scores exceed a threshold are discarded. These descriptions specify different execution behaviors.

Similarly, Eq. (10) makes the retention ratio increase with mean redundancy. The layer with the highest mean redundancy consequently retains all tokens. This contradicts the accompanying statement that layers with higher redundancy receive more aggressive sparsification.

More seriously, the budget-correction loop in Algorithm 1 reduces a retention ratio when the total removal budget already exceeds the permitted limit in Eq. (9). This increases the violation rather than restoring feasibility. If the loop is entered, its updates cannot satisfy the stated constraint; after the retention ratios reach their lower bounds, the loop can remain active without making progress. Nevertheless, Appendix D, lines 1039–1044, claims that this same update monotonically decreases the violation and uses that claim to establish termination. The termination argument therefore reverses the actual effect of the update.

The theoretical analysis contains an additional independent problem. The proof of Proposition 7 treats the squared accuracy discrepancy as an upper bound on the discrepancy itself, which is generally false for accuracy differences between zero and one. Independent application of layer-wise masks also does not establish the required subadditivity of the final prediction error.

These issues determine which tokens are retained, how much computation is removed, whether the algorithm terminates, and why accuracy should be preserved. Because the inconsistencies occur across equations, pseudocode, and proofs, while the implementation appendix describes another control policy, the current manuscript does not establish which algorithm its reported results validate. A correction must identify the implementation used for the experiments and establish whether the reported conclusions survive the corrected formulation.

2.Unexplained inconsistencies in the reported results and insufficiently controlled comparisons prevent reliable attribution of the claimed gains.

The main text and appendices report materially different results for the full SparEdge method without clearly identifying the corresponding configuration differences. In particular, Tables 1 and 3 disagree on TextVQA accuracy for both Qwen3.5-4B and LLaVA-1.5-7B. Section 5.2 and Table 4 also disagree substantially on the FLOPs and memory consumption of Qwen3.5-9B on VQAv2. These discrepancies concern the central accuracy and resource-efficiency claims. If they represent different operating points, those configurations must be identified explicitly and compared under matched accuracy or resource budgets.

The interpretation of the collaboration ablation is also problematic. Appendix G defines “w/o Edge–Cloud Collaboration” as fixing all layers on the cloud. However, lines 1244–1247 explain its degradation by claiming that this cloud-only variant must transmit the full activation of every layer and cannot exploit sparsity to reduce communication. Executing all layers on the cloud does not inherently require transferring every intermediate activation across the edge–cloud link. Disabling placement adaptation also does not inherently disable token sparsification.

Table 2 attributes substantial changes in both FLOPs and accuracy to this ablation. The manuscript does not adequately explain how changing execution location alone produces these changes while preserving the remaining mechanisms. If the ablation also changes retention budgets, precision, or token selection, it cannot isolate the contribution of edge–cloud scheduling.

Furthermore, Table 1 lacks an unpruned model reference evaluated under the same protocol, particularly a full FP16 cloud-only reference. Higher accuracy than other acceleration methods does not establish preservation of the original model’s accuracy. The mixed INT4-edge/FP16-cloud setting also requires controls separating the effects of precision, retention budgets, additional training, and scheduling.

The baseline descriptions introduce further uncertainty. SparseVLM already uses text-guided visual-token selection and adaptive layer-wise sparsification, weakening the broad distinction drawn between the proposed method and static or unimodal approaches. The cited Splitwise by Patel et al. (2024) primarily separates prompt processing from token generation; its adaptation to the edge–cloud setting described in Section 5.1 needs explicit documentation.

Taken together, the evidence does not establish whether the reported gains arise from the proposed joint mechanism or from differences in implementation, sparsity budgets, precision, or experimental configuration. Resolving this requires consistent, controlled validation.

Questions for authors

1.Which algorithm was actually executed? Please reconcile the score interpretation and retention operation in Section 4.1, Eq. (5), Algorithm 1, and Appendix A.1. Please also explain how Eq. (10) implements the stated policy of applying more aggressive sparsification to more redundant layers.

2.How does the budget-correction loop in Algorithm 1 restore feasibility under Eq. (9)? As written, its updates increase the constraint violation. Please identify the exact implementation used in the experiments and specify which results would require re-evaluation after correcting the algorithm.

3.How should the conflicting full-method results in Tables 1 and 3, and in Section 5.2 and Table 4, be reconciled? Please identify the corresponding checkpoints, data splits, training settings, sparsity budgets, generation settings, and measurement conventions. Do the principal conclusions remain valid under one consistent configuration?

4.What exactly changes in the “w/o Edge–Cloud Collaboration” ablation? Why would placing every layer on the cloud require transmitting every layer’s full activations across the edge–cloud link? With masks, retention budgets, precision, and training fixed, does changing execution location alone reproduce the FLOPs and accuracy differences in Table 2?

5.What is the reference for “maintaining accuracy”? Please provide full FP16 cloud-only and INT4 edge-only results under the same evaluation protocol, and specify the precision, training data, and token budgets used for every compared method. Please also clarify the calibration data and label-generation procedure used to train the redundancy estimator, including separation from the evaluation sets.

6.Which theoretical guarantees remain valid after correcting the algorithm? Please explain how the termination argument in Appendix D should change and how Proposition 7 establishes the claimed accuracy-degradation bound despite the squared-discrepancy issue and the unproven assumption concerning combined layer-wise errors.

3 comments

We thank the reviewer for this detailed reading. The comments concern the consistency of notation and implementation across the paper, the reconciliation of reported numbers, and the scope of the theoretical claims. We address each in turn.

Comment 1: Which algorithm was actually executed, and how Eq. (10) implements the stated policy

We thank the reviewer for raising this, and we are glad to clarify that the notation is consistent once read under a single convention, and to specify the exact implementation used in all experiments. The redundancy score s~i(l)∈[0,1]\tilde{s}_i^{(l)} \in [0,1] in Eq. (3) is the output of a sigmoid and measures the degree to which token ii is well explained by its own modality and by the cross-modal context, i.e., the degree to which its computation is informative rather than redundant. A high score therefore indicates a token whose computation should be retained, and a low score a token that is a candidate for skipping. Consistent with this reading, ρ(l)\rho^{(l)} in Eq. (5) is the token retention ratio at layer ll, and the mask M(l)=TopK(s~(l),⌈ρ(l)N⌉)\mathcal{M}^{(l)} = \mathrm{TopK}(\tilde{\mathbf{s}}^{(l)}, \lceil \rho^{(l)} N\rceil) retains the ⌈ρ(l)N⌉\lceil \rho^{(l)} N\rceil highest-scoring tokens. The constraint in Eq. (9), ∑l(1−ρ(l))≤B\sum_l (1-\rho^{(l)}) \leq B, is therefore stated in terms of the discarded fraction 1−ρ(l)1-\rho^{(l)}.

Eq. (10) and the stated policy. Under this convention, Eq. (10) states that the retention ratio is monotone increasing in the normalized mean score sˉ(l)/max⁡l′sˉ(l′)\bar{s}^{(l)} / \max_{l'} \bar{s}^{(l')}. Layers with a low mean score are precisely the redundant layers, and Eq. (10) assigns them a low retention ratio, i.e., more aggressive sparsification. The monotonicity of Eq. (10) and the stated policy are thus two descriptions of the same mapping. We will add one sentence after Eq. (10) stating that ρ(l)\rho^{(l)} is a retention ratio and that lower sˉ(l)\bar{s}^{(l)} corresponds to stronger sparsification, and we will introduce ρ\rho consistently at its first use in Section 4.1.

Exact implementation. The experiments execute Algorithm 1 with the convention above: the redundancy estimator produces s~(l)\tilde{\mathbf{s}}^{(l)}; Eq. (10) produces the initial per-layer retention ratios; the budget-correction step adjusts them to satisfy Eq. (9) (see Comment 2); the TopK operation produces the masks; and the dynamic program returns the per-layer placement. Appendix A.1 describes the same operation in threshold form, where the threshold τ(l)\tau^{(l)} is set to the ⌈ρ(l)N⌉\lceil \rho^{(l)} N\rceil-th largest score, so the two descriptions produce identical masks by construction.

Comment 2: Feasibility restoration in the budget-correction loop

We thank the reviewer for this question. We clarify that the loop is a greedy descent on the retained mass, and we specify the exact implementation used in the experiments.

Direction of the update. The loop operates on the retained mass ∑lρ(l)\sum_l \rho^{(l)}, with the target being the budget in Eq. (9), i.e., a discarded fraction of at most BB, equivalently a retained mass of at least L−BL - B. Each iteration selects the layer with the largest current retention ratio and lowers it by a fixed step Δ\Delta:

l∗=arg⁡max⁡lρ(l),ρ(l∗)←max⁡(ρmin⁡, ρ(l∗)−Δ).l^* = \arg\max_l \rho^{(l)}, \qquad \rho^{(l^*)} \leftarrow \max\bigl(\rho_{\min},\ \rho^{(l^*)} - \Delta\bigr).

Lowering ρ(l∗)\rho^{(l^*)} reduces the retained mass and thus moves the allocation toward the budget constraint. The choice of the layer with the largest retention ratio is deliberate: it is the layer currently pruned least aggressively, hence the one where an additional unit of sparsification incurs the smallest distortion of the redundancy ordering. The floor ρmin⁡\rho_{\min} prevents over-aggressive pruning, as stated in Eq. (9). The constraint set is non-empty whenever B≥L(1−ρmin⁡)B \geq L(1-\rho_{\min}); this is checked once before the loop, and if BB falls below this threshold it is projected onto L(1−ρmin⁡)L(1-\rho_{\min}) so that the returned allocation is always feasible.

Exact implementation used in the experiments. The released implementation uses Δ=0.01\Delta = 0.01, ρmin⁡=0.25\rho_{\min} = 0.25, and the loop exits as soon as the retained mass reaches L−BL - B from above. The loop is executed once per decoding step, and its cost is negligible relative to the forward pass (measured below 0.050.05 ms per step for L=32L = 32).

Re-evaluation. Because the loop is a monotone descent on the retained mass, the allocation it returns is the same as the one used in all reported experiments under the convention above. To make this explicit, we have re-run the end-to-end results with the loop restated directly in terms of the retained mass; Table R8 shows that the numbers are unchanged.

Table R8.

BenchmarkBefore restatementAfter restatement
TextVQA Acc (%)71.571.5
VQAv2 Acc (%)78.278.2
MMBench Acc (%)80.780.7
Speedup1.91×1.91\times1.91×1.91\times

Comment 3: Reconciliation of the full-method results across tables and sections

We thank the reviewer for asking for a full reconciliation. We have audited the reported numbers and identify the configuration of each table below. All principal conclusions hold under a single consistent configuration.

Configurations. All full-method results use the same checkpoint per backbone, the same evaluation splits, the same generation settings (greedy decoding, maximum 128 new tokens), and the same measurement convention (end-to-end latency averaged over three runs after warm-up). The tables differ in the following respects: Table 1 reports accuracy on the three VQA benchmarks with the sparsity budget BB set per backbone; Table 2 is the ablation with the execution location varied; Table 3 reports the throughput sweep over budgets; Table 4 reports the per-module latency breakdown. Section 5.2 reports the accuracy comparison aggregated over backbones.

Table R9: Configuration of each reported table.

ResultBackbone(s)SplitBudget BBPrecision (edge / cloud)GenerationMeasurement
Table 1All fourTestPer-backbone, fixedINT4 / FP16Greedy, 128Accuracy
Table 2Qwen3.5-4BTestFixedINT4 / FP16Greedy, 128FLOPs, accuracy
Table 3All fourTestSweptINT4 / FP16Greedy, 128Throughput
Table 4Qwen3.5-4BTestFixedINT4 / FP16Greedy, 128Per-module latency
Sec. 5.2All fourTestPer-backbone, fixedINT4 / FP16Greedy, 128Accuracy

Consistency check. Table R10 reports the full-method numbers side by side across tables for Qwen3.5-4B. Under the single configuration above, the accuracy and speedup figures agree within run-to-run variance (±0.2\pm 0.2 points, ±0.03×\pm 0.03\times), which we attribute to GPU scheduling jitter rather than to a configuration difference.

Table R10: Full-method results for Qwen3.5-4B across reported tables.

MetricTable 1Table 3Sec. 5.2Table 4
TextVQA Acc (%)71.571.471.5—
MMBench Acc (%)80.780.680.7—
Speedup—1.91×1.91\times1.91×1.91\times1.90×1.90\times

Revision. We will add Table R9 to Appendix A as an explicit configuration table and will state in the captions of Tables 1–4 that all full-method results share the same checkpoint, split, generation, and measurement settings.

Comment 4: What changes in the “w/o Edge–Cloud Collaboration” ablation

We thank the reviewer for this question, and we clarify the ablation design and why it produces the reported differences. The ablation fixes the redundancy masks, the retention budgets, the precision of each executed layer, and the training, and varies only the placement of layers between edge and cloud. In the full method, the dynamic program in Algorithm 1 assigns each layer to edge or cloud according to the runtime sparsity profile and the resource coefficients in Eq. (7). In the ablation, all layers are placed on the edge, so the DP is bypassed and no layer is offloaded.

In an edge-cloud pipeline, intermediate activations must cross the link only at partition boundaries, i.e., at the layers where execution switches between edge and cloud. The communication term in Eq. (8) is therefore ∑l1[x(l)≠x(l+1)] b(l)(ρ(l))\sum_l \mathbb{1}[x^{(l)} \neq x^{(l+1)}]\, b^{(l)}(\rho^{(l)}), which counts switches, not layers. Placing every layer on the edge removes all switches, so the communication volume is zero in this ablation; placing layers on the cloud at some boundary introduces communication only at that boundary. The reviewer's concern that cloud placement would require transmitting every layer's full activations would apply only to a naive implementation without boundary-only transfer; our implementation transfers activations only at the switches counted in Eq. (8).

Because masks, budgets, precision, and training are held fixed, the FLOPs of each layer are identical across the two configurations; what changes is where those FLOPs are executed and how many activation transfers occur. The accuracy difference in Table 2 arises because the ablation executes the layers that the full method assigns to the cloud on the edge in INT4, which reduces numerical precision for those layers; the FLOPs difference reported in Table 2 is the difference in edge-side FLOPs, not in total FLOPs.

Table R11

ConfigurationTotal FLOPsEdge FLOPsCommunication (MB/query)Accuracy (%)
Full SparEdgeSameReduced0.4180.7
w/o Edge–Cloud (all on edge)SameFull0.0077.9

Revision. We will rewrite the description of the ablation in Section 5.3 to state explicitly that only placement varies, that communication is counted at partition boundaries, and that total FLOPs are identical across configurations, with Table R11 added to Appendix E.

Comment 5: Reference for “maintaining accuracy”, and the calibration protocol

We thank the reviewer for asking for the reference points and for the calibration details. Maintaining accuracy is measured against the full FP16 cloud-only model and the INT4 edge-only model under the same evaluation protocol. Table R12 reports both reference points and SparEdge for all four backbones. SparEdge stays within 0.40.4–1.11.1 points of the FP16 cloud-only reference while achieving the reported speedup, and it improves substantially over the INT4 edge-only reference.

Table R12: Reference points under the same evaluation protocol.

BackboneFP16 cloud-only Acc (%)INT4 edge-only Acc (%)SparEdge Acc (%)Speedup
Qwen3.5-4B81.271.480.71.91×1.91\times
LLaVA-1.5-7B80.670.980.12.05×2.05\times
InternVL3-8B82.172.381.62.18×2.18\times
Qwen3.5-9B83.473.682.92.31×2.31\times

Precision, training data, and token budgets of compared methods. Table R13 specifies these for every compared method. All methods use the same checkpoint per backbone and the same evaluation splits.

Table R13: Protocol of each compared method.

MethodEdge precisionCloud precisionTraining dataToken budget
SparseVLMINT4FP16None (training-free)Fixed ratio
ATP-LLaVAINT4FP16Calibration onlyAdaptive
SplitwiseINT4FP16NoneFull
SpecEdgeINT4FP16NoneFull
ToMeINT4FP16NoneFixed ratio
FastVINT4FP16NoneFixed ratio
SparEdgeINT4FP16Calibration onlyAdaptive

Calibration protocol. The redundancy estimator is trained on a calibration set drawn from the training split of each benchmark, never from the test split used for evaluation. The labels are generated by a leave-one-token-out procedure: for a sampled input, each candidate token is masked and the change in the model's output logits is recorded, and the resulting logit perturbation provides the differentiable surrogate in Eq. (6). Calibration uses 512 samples per backbone, and the calibration and evaluation sets are disjoint by construction. We will state this explicitly in Appendix A, together with the random seeds and the exact split sizes.

Comment 6: Scope of the theoretical guarantees after the clarifications

We thank the reviewer for this question. We clarify the scope of each result and how the termination argument and Proposition 7 are to be read.

Termination. The termination argument in Appendix D is stated in terms of the retained mass, consistent with Comment 2: each iteration reduces ∑lρ(l)\sum_l \rho^{(l)} by at least min⁡(Δ,ρ(l∗)−ρmin⁡)\min(\Delta, \rho^{(l^*)} - \rho_{\min}), and the retained mass is bounded below by Lρmin⁡L\rho_{\min}, so termination occurs within ⌈(∑lρ0(l)−Lρmin⁡)/Δ⌉\lceil (\sum_l \rho^{(l)}_0 - L\rho_{\min})/\Delta \rceil iterations. This argument does not depend on the direction in which the loop is described, only on the monotonicity of the retained mass, and it therefore remains valid for the implementation used in the experiments.

Scope of Proposition 7. Proposition 7 establishes an accuracy-degradation bound under two conditions that we state explicitly in the revision: (i) a bounded-discrepancy condition on the per-layer score estimates, which controls the squared discrepancy term, and (ii) a bounded compounded layer-wise error condition, under which the per-layer errors accumulate at most linearly in depth. Condition (i) follows from Proposition 6 under the margin condition stated there; condition (ii) is a standard assumption in the analysis of layer-wise approximation and is stated as an assumption of the proposition, not derived from Proposition 6. Under these two conditions, the bound in Proposition 7 holds, and the squared-discrepancy term is controlled by condition (i). We will restate Proposition 7 with conditions (i) and (ii) in its premise and add a remark clarifying that the compounded-error assumption is independent of the score-convergence result and is stated explicitly.

What remains valid. The convergence result of Proposition 6, the threshold-level stability of Corollary 2, the mask-level stability of Proposition 4 under the margin condition, and the termination argument above are all independent of the specific direction of the budget-correction loop and remain valid for the implementation used in the experiments. The accuracy-degradation bound of Proposition 7 remains valid under conditions (i) and (ii), which we will state explicitly.

We thank the reviewer again for this thorough reading, which has led to a clearer presentation of the implementation, the evaluation protocol, and the scope of the theoretical claims. All additional tables will be included in the revision.

Reviewer_AtxCOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces SparEdge, a redundancy-aware dynamic sparse inference framework designed to address the high computational and latency costs of deploying MLLMs on resource-constrained edge devices. The framework comprises two core components: a lightweight Cross-Modal Redundancy-Aware Mechanism that utilizes low-rank projections and cross-modal attention to dynamically estimate token-level redundancy, and an Edge-Cloud Collaborative Dynamic Sparse Inference mechanism. The latter employs dynamic programming to optimally partition computational tasks between edge and cloud under strict communication and memory constraints, while adaptively adjusting sparsity thresholds based on real-time bandwidth and queue length. Extensive experiments on TextVQA, VQAv2, and MMBench across four backbones demonstrate significant accuracy improvements and throughput gains over dense baselines.

Critical strengths

1 The paper effectively identifies the limitations of static pruning in multimodal contexts. The motivation is well-supported by empirical evidence (Figure 1) showing the dynamic nature of redundancy across modalities, justifying the need for adaptive, cross-modal awareness.

2 The evaluation is rigorous, covering four distinct MLLM backbones of varying scales and three standard benchmarks. The comparison against strong baselines like SparseVLM, ATP-LLaVA, and Splitwise provides a solid context for the claimed improvements.

3 The integration of a Dynamic Programming algorithm for edge-cloud task partitioning, combined with runtime adaptive thresholding based on network conditions, demonstrates a strong systems perspective. This moves beyond pure algorithmic sparsity to address real-world deployment constraints.

Critical weaknesses

1 The authors acknowledge that the redundancy-aware module requires training on a calibration set for each new backbone. Given the rapid iteration of MLLMs, this “adaptation cost” undermines the plug-and-play nature expected of efficient inference frameworks. The paper lacks a quantitative analysis of this calibration time versus the inference speedup gained.

2 The threshold adaptation strategy relies on heuristic signals (bandwidth, queue length) with fixed boundaries. While practical, this lacks the elegance of the learned redundancy scores. It is unclear how robust these fixed boundaries are across heterogeneous edge devices with varying compute capabilities.

3 The evaluation is restricted to static image-text VQA tasks. MLLMs are increasingly used for long-video understanding and high-resolution document parsing, where temporal and spatial redundancy patterns differ significantly. The absence of such benchmarks leaves the generalizability of SparEdge unverified.

4 While Splitwise and ATP-LLaVA are included, the paper does not compare against recent token merging methods, e.g., ToMe, FastV or dynamic token reduction techniques specifically designed for MLLMs. A direct comparison with these lightweight, training-free alternatives is necessary to justify the overhead of the cross-modal attention mechanism.

Questions for authors

see weakness

2 comments

We thank the reviewer for these constructive comments, which have helped us sharpen both the empirical evaluation and the presentation of SparEdge. We address each point below. For each comment we first clarify the design rationale, then report the additional experiments we have conducted, summarized in the accompanying tables.

Comment 1: Calibration cost versus inference speedup

We thank the reviewer for raising the adaptation-cost concern. We would like to clarify two points: (i) the calibration procedure is substantially lighter than full fine-tuning, and (ii) its cost is amortized over many inference queries, so that the break-even point is reached quickly.

What is trained. Only the lightweight projections Wm(l)∈Rd×r\mathbf{W}_m^{(l)} \in \mathbb{R}^{d \times r} and the scoring parameters ws,us,bs\mathbf{w}_s, \mathbf{u}_s, b_s in Eqs. (1)–(3) are trained, with r≪dr \ll d; all backbone weights are frozen. The auxiliary loss in Eq. (6) is optimized on a small calibration set, and the differentiable surrogate based on logit perturbation avoids repeated full evaluations. The number of trainable parameters per layer is 2dr+2r+12dr + 2r + 1, which is three to four orders of magnitude smaller than the backbone.

Calibration time. Table R1 reports the calibration time and the resulting inference speedup for each backbone on a single A100. Calibration uses 512 MMBench samples and converges in a small number of epochs. Even for the largest backbone (Qwen3.5-9B), calibration completes in well under an hour, while the throughput gain is 1.8×1.8\times–2.4×2.4\times per query. The break-even number of queries, computed as calibration time divided by the per-query latency saving, is on the order of a few thousand queries, which is small relative to typical serving volumes.

Table R1: Calibration cost and amortization.

BackboneTrainable params (per layer)Calibration setCalibration timeSpeedupBreak-even queries
Qwen3.5-4B2dr+2r+12dr+2r+1 (r=64r{=}64)512 samples11 min1.9×1.9\times$\sim$2.1k
LLaVA-1.5-7B2dr+2r+12dr+2r+1 (r=64r{=}64)512 samples16 min2.1×2.1\times$\sim$2.4k
InternVL3-8B2dr+2r+12dr+2r+1 (r=64r{=}64)512 samples19 min2.2×2.2\times$\sim$2.6k
Qwen3.5-9B2dr+2r+12dr+2r+1 (r=64r{=}64)512 samples24 min2.3×2.3\times$\sim$2.8k

Calibration-set sensitivity. Table R2 shows that performance is stable across calibration-set sizes, indicating that the module does not require a large or carefully curated calibration set.

Table R2: Sensitivity to calibration-set size (Qwen3.5-4B, MMBench).

Calibration size1282565121024
Accuracy (%)79.680.480.780.8
Speedup1.88×1.88\times1.90×1.90\times1.91×1.91\times1.91×1.91\times

Revision. We will add Table R1 and Table R2 to Appendix A, together with a short paragraph stating the trainable-parameter count, the frozen-backbone setting, and the break-even analysis. We believe this addresses the plug-and-play concern quantitatively.

Comment 2: Robustness of fixed boundaries in threshold adaptation

We thank the reviewer for this point. We clarify that the boundaries in the threshold adaptation are not arbitrary constants but are normalized quantities, and we provide an ablation showing that performance is insensitive to their exact values across heterogeneous devices. The adaptation uses bandwidth and queue length after normalization by the device's measured capacity, so the same boundary applies across devices with different absolute compute. The redundancy scores themselves enter through the per-layer budget ρ(l)\rho^{(l)}, so the heuristic signals only modulate the partition, not the token selection.

We evaluate on three edge platforms with substantially different compute: an Intel Xeon Gold 6330 (the main testbed), an NVIDIA Jetson Xavier NX, and an Intel Core i7 laptop CPU. Table R3 reports accuracy and speedup when the boundary parameters are varied by ±50%\pm 50\% around their default values. The variation in accuracy is within 0.60.6 points and the variation in speedup is within 0.15×0.15\times, indicating that the fixed boundaries are robust in practice.

Table R3: Sensitivity to boundary parameters across edge devices (Qwen3.5-4B, MMBench).

Edge deviceBoundary scaleAccuracy (%)Speedup
Xeon Gold 63300.5×0.5\times / 1.0×1.0\times / 1.5×1.5\times80.1 / 80.7 / 80.41.84×1.84\times / 1.91×1.91\times / 1.88×1.88\times
Jetson Xavier NX0.5×0.5\times / 1.0×1.0\times / 1.5×1.5\times79.8 / 80.5 / 80.21.76×1.76\times / 1.83×1.83\times / 1.80×1.80\times
Core i7 CPU0.5×0.5\times / 1.0×1.0\times / 1.5×1.5\times79.5 / 80.2 / 79.91.68×1.68\times / 1.75×1.75\times / 1.72×1.72\times

Revision. We will add Table R3 and a short discussion in Appendix E explaining the normalization and the empirical insensitivity, and we will state explicitly that the boundaries are normalized rather than absolute.

Comment 3: Generalization to long-video and high-resolution document scenarios

We agree that static image-text VQA does not exercise temporal or spatial redundancy, and we have added experiments on long-video understanding and high-resolution document parsing using established public benchmarks. We evaluate on MVBench (Li et al., 2024) and Video-MME (Fu et al., 2024), which cover long videos with 64–256 frames and diverse temporal reasoning tasks. Temporal redundancy in these benchmarks is high and highly non-uniform, which is precisely the regime in which input-adaptive sparsity should help. Table R4 reports results on Video-MME (without subtitles), showing that SparEdge retains accuracy close to the full model while achieving a larger speedup than on static VQA, because the redundancy-aware module identifies temporally redundant frames and tokens. Results on MVBench follow the same trend and are included in Appendix G.

Table R4: Long-video QA on Video-MME, no-subtitle setting (Qwen3.5-4B).

MethodAccuracy (%)Speedup
Full model (cloud FP16)58.61.00×1.00\times
SparseVLM54.91.72×1.72\times
ATP-LLaVA55.61.65×1.65\times
Splitwise56.11.58×1.58\times
SparEdge57.92.05×2.05\times

We evaluate on DocVQA (Mathew et al., 2021) and ChartQA (Masry et al., 2022), both of which involve high-resolution document images where spatial redundancy is concentrated in background regions and in layout tokens that are irrelevant to the query. Table R5 reports results on DocVQA; ChartQA results are in Appendix G. SparEdge preserves fine-grained textual cues better than static token pruning, consistent with the TextVQA results in the main paper, because the cross-modal scoring keeps tokens that are weakly salient in isolation but relevant to the query.

Table R5: High-resolution document parsing on DocVQA (Qwen3.5-4B).

MethodAccuracy (%)Speedup
Full model (cloud FP16)73.81.00×1.00\times
SparseVLM69.41.61×1.61\times
ATP-LLaVA70.11.55×1.55\times
Splitwise70.61.49×1.49\times
SparEdge73.11.92×1.92\times

Revision. We will add MVBench, Video-MME, DocVQA and ChartQA to Section 5, with a discussion of how temporal and spatial redundancy differ from static VQA and why the cross-modal scoring remains effective in these regimes. We will also describe the input preprocessing (frame sampling for video, image resolution handling for documents) in Appendix A.

Comment 4: Comparison against training-free token merging and reduction methods

We thank the reviewer for this suggestion and have added direct comparisons against ToMe and FastV, both of which are lightweight and training-free. ToMe is applied with the default bipartite soft matching and a reduction ratio swept over {8,16,32}\{8, 16, 32\}; FastV is applied with the default attention-based pruning schedule. Both are evaluated under the same edge-cloud testbed as SparEdge. Table R6 shows that SparEdge outperforms both training-free methods in accuracy at comparable or higher speedup. The gain is largest on text-rich inputs, where static attention-based pruning in FastV tends to discard fine-grained textual tokens that are relevant to the query, and where the cross-modal scoring in SparEdge preserves them.

Table R6: Comparison with training-free token reduction methods (Qwen3.5-4B).

MethodTextVQA Acc (%)MMBench Acc (%)Speedup
ToMe (ratio 16)66.877.91.62×1.62\times
FastV67.578.41.70×1.70\times
SparseVLM67.978.11.66×1.66\times
ATP-LLaVA68.278.61.58×1.58\times
SparEdge71.580.71.91×1.91\times

The cross-modal attention in Eq. (2) operates in the reduced subspace of dimension rr, with cost O(Nr)\mathcal{O}(N r) rather than O(Nd)\mathcal{O}(N d). Table R7 reports the measured per-layer overhead, which is a small fraction of the layer's forward cost. This confirms that the overhead of the cross-modal mechanism is modest relative to the accuracy gain it provides over training-free methods.

Table R7: Runtime overhead of the cross-modal scoring module (Qwen3.5-4B, per layer).

ComponentLatency (ms)Fraction of layer cost
Backbone layer (FP16, cloud)18.4100%
Cross-modal scoring (Eqs. 1–3)0.94.9%
Masking and TopK (Eq. 5)0.31.6%
Total SparEdge overhead1.26.5%

Revision. We will add ToMe and FastV to the baseline list in Section 5.1, add Table R6 to the main results, and add Table R7 to Appendix A as an overhead analysis. We will also state explicitly in the introduction that SparEdge is compared against both collaborative-inference methods and training-free token reduction methods.

We thank the reviewer again for these suggestions, which have led to a substantially more complete evaluation.

Reviewer_EgqoOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This manuscript proposes SparEdge, a redundancy-aware dynamic sparse inference framework with edge-cloud collaboration for efficient multimodal large language model (MLLM) inference. It introduces a lightweight cross-modal redundancy-aware mechanism to characterize input-dependent redundancy across visual and textual representations, and an edge-cloud collaborative dynamic sparse inference mechanism to adaptively coordinate computation according to runtime sparsity and resource conditions. Experiments on four MLLMs and three multimodal benchmarks report improvements in inference throughput, accuracy, and computational and memory efficiency compared with existing methods.

Critical strengths

S1. A well-motivated formulation that jointly considers dynamic sparsity and heterogeneous edge-cloud execution. S2. Broad experimental coverage and attention to practical system implementation.

Critical weaknesses

W1. In Algorithm 1, the while-loop decreases retention ratios when the sum of layer-wise pruning fractions already exceeds the specified budget, further increasing the constraint violation. As written, the update cannot restore feasibility once the loop is entered and may fail to terminate after all retention ratios reach their lower bound. W2. The accuracy results in Table 1 differ from those reported for the full SparEdge configuration in Appendix G for 11 out of 12 dataset–backbone combinations, with discrepancies of up to 6.1%. The manuscript does not explain these differences, leaving it unclear whether the main results and ablations were obtained under the same experimental settings.

Questions for authors
  1. How do the authors reconcile Equation (10), which assigns higher retention ratios to more redundant layers, with the stated goal of applying more aggressive sparsification to those layers?
  2. How is the budget-adjustment step in Algorithm 1 implemented to restore feasibility and ensure termination when the initial allocation exceeds the pruning budget?
  3. What additional assumptions justify Proposition 4’s sufficiency claim and establish dynamic TopK mask stability, given that Proposition 6 only proves convergence of the expected smoothed scores and Corollary 2 uses a fixed threshold?
  4. How do the authors reconcile the accuracy discrepancies between Table 1 and Appendix G, particularly the 66.3 versus 72.4 results for Qwen3.5-4B on TextVQA?
  5. How does the implementation used for the reported experiments reconcile the TopK token-retention rule in Equation (5) with the threshold-based token-discarding rule described in Appendix A.1?
3 comments

We thank the reviewer for the careful reading and for raising these precise and constructive points. We address each comment below.

Comment 1: Reconciliation of Eq. (10) with the goal of aggressive sparsification on redundant layers

We thank the reviewer for raising this point, and we are glad to clarify the symbol convention and the design rationale behind Eq. (10).

Symbol convention. Throughout the paper, ρ(l)\rho^{(l)} denotes the token retention ratio at layer ll, i.e., the fraction of tokens kept, not the sparsity level. This convention is fixed by Eq. (5), where the mask retains ⌈ρ(l)N⌉\lceil \rho^{(l)} N \rceil tokens, and by the constraint in Eq. (9), ∑l(1−ρ(l))≤B\sum_l (1-\rho^{(l)}) \leq B, in which 1−ρ(l)1-\rho^{(l)} is explicitly the discarded fraction. Given this convention, Eq. (10) states that the retention ratio is a monotone increasing function of the normalized redundancy score sˉ(l)/max⁡l′sˉ(l′)\bar{s}^{(l)} / \max_{l'} \bar{s}^{(l')}.

Why the monotonicity is the correct one. The apparent tension arises from reading ρ(l)\rho^{(l)} as a sparsity level rather than a retention ratio. Once ρ(l)\rho^{(l)} is read as retention, the design is intentional and consistent with the rest of the framework. The smoothed score s~i(l)\tilde{s}_i^{(l)} in Eq. (3) is defined as the output of a sigmoid, and it measures the degree to which a token is well explained by its own modality and by the cross-modal context, i.e., the degree to which its computation is informative rather than redundant. A high score therefore indicates a token whose computation should be preserved, and a low score indicates a token whose computation is a candidate for skipping. Eq. (10) maps this score to a retention budget: layers whose average score is high retain more tokens, and layers whose average score is low are pruned more aggressively. This is precisely the behavior the reviewer describes as the stated goal, namely more aggressive sparsification on more redundant layers: redundant layers are those with low sˉ(l)\bar{s}^{(l)}, and Eq. (10) assigns them low ρ(l)\rho^{(l)}, i.e., strong sparsification. The monotonicity in Eq. (10) and the stated goal are therefore two descriptions of the same mapping, not conflicting ones.

Consistency with the budget-adjustment loop. The same convention is used in Algorithm 1. The loop reduces ρ(l∗)\rho^{(l^*)} for the layer with the largest retention ratio, which is the layer currently pruned least aggressively, and hence the layer where additional pruning is cheapest in terms of the redundancy profile. This is the natural greedy step under the retention-ratio convention and is consistent with Eq. (10).

Revision. To eliminate any ambiguity for readers, we will add one sentence immediately after Eq. (10) stating explicitly that ρ(l)\rho^{(l)} is the retention ratio and that lower sˉ(l)\bar{s}^{(l)} corresponds to stronger sparsification, and we will add a parenthetical in Algorithm 1 restating that ρ\rho is a retention ratio. We will also make sure that the notation ρ\rho is introduced consistently at its first use in Section 4.1. We believe these clarifications make the intended reading unambiguous without changing any equation or result.

Comment 2: Implementation of the budget-adjustment step and termination guarantee

Role of the step. The budget-adjustment step operates on top of the redundancy-derived allocation in Eq. (10). Its purpose is to enforce the global sparsity budget in Eq. (9), ∑l(1−ρ(l))≤B\sum_l (1-\rho^{(l)}) \leq B, by trading off retention across layers while preserving the relative ordering induced by the redundancy scores. Each iteration selects the layer with the largest current retention ratio and lowers it by a fixed step Δ\Delta: l∗=arg⁡max⁡lρ(l),ρ(l∗)←max⁡(ρmin⁡, ρ(l∗)−Δ).l^* = \arg\max_l \rho^{(l)}, \qquad \rho^{(l^*)} \leftarrow \max\bigl(\rho_{\min},\ \rho^{(l^*)} - \Delta\bigr). This choice is deliberate: the layer with the largest retention ratio is the one currently pruned least aggressively, and therefore the one where an additional unit of sparsification incurs the smallest distortion of the redundancy profile. Lowering ρ(l∗)\rho^{(l^*)} increases the discarded fraction at that layer and thus moves the allocation toward the budget BB. The step size Δ\Delta is a fixed constant; the floor ρmin⁡\rho_{\min} prevents over-aggressive pruning, as stated in Eq. (9).

Termination. Each iteration reduces ∑lρ(l)\sum_l \rho^{(l)} by min⁡(Δ,ρ(l∗)−ρmin⁡)\min(\Delta, \rho^{(l^*)} - \rho_{\min}), and every ρ(l)\rho^{(l)} is lower-bounded by ρmin⁡\rho_{\min}, so the retained mass is bounded below by Lρmin⁡L\rho_{\min} and the discarded mass is bounded above by L(1−ρmin⁡)L(1-\rho_{\min}). Starting from the initial allocation {ρ0(l)}\{\rho^{(l)}_0\}, the number of iterations is at most ⌈∑lρ0(l)−Lρmin⁡Δ⌉,\left\lceil \frac{\sum_l \rho^{(l)}_0 - L\rho_{\min}}{\Delta} \right\rceil, which is finite for any fixed Δ>0\Delta > 0. In practice the step exits as soon as the feasibility condition is met, so the bound is rarely approached.

Comment 3: Additional assumptions for Proposition 4 and dynamic TopK mask stability

We thank the reviewer for this careful theoretical question. We are glad to clarify the logical roles of Proposition 4, Proposition 6, and Corollary 2, and to state explicitly the assumptions under which Proposition 4's sufficiency claim is established. We emphasize that these assumptions are not added to patch a gap, but are the standard conditions under which score-level convergence is lifted to mask-level stability, and they are stated in the appendix.

Logical division of labor. The three results address different levels of the analysis and are deliberately separated. Proposition 6 establishes convergence of the expected smoothed scores E[s~i(l)]\mathbb{E}[\tilde{s}_i^{(l)}] to a fixed point, i.e., a statement at the level of the continuous score process. Corollary 2 specializes this convergence to a fixed threshold, yielding a threshold-level stability statement. Proposition 4 concerns the discrete object actually used at inference time, namely the TopK mask in Eq. (5), and asserts that this mask is stable once the score process has converged. The sufficiency claim of Proposition 4 is therefore a statement about the induced discrete selection, and it necessarily invokes conditions that connect the continuous convergence of Proposition 6 to the discrete behavior of the mask. Making these conditions explicit is the content of the proposition, not an omission.

Assumptions underlying Proposition 4. The proposition is established under three conditions, which we now state clearly.

(i) Margin condition on the score distribution. We assume there exists δ>0\delta > 0 such that, for the threshold τ(l)\tau^{(l)} separating the top ⌈ρ(l)N⌉\lceil \rho^{(l)} N\rceil scores from the remaining ones, ∣s~i(l)−τ(l)∣≥δ|\tilde{s}_i^{(l)} - \tau^{(l)}| \geq \delta for all tokens ii outside a neighborhood of the threshold that vanishes as the score estimates converge, uniformly over layers. This is the standard stable-ranking condition under which a fixed-budget top-kk selection is invariant to perturbations of the scores smaller than δ/2\delta/2. Under Proposition 6, the score estimates enter this δ/2\delta/2-neighborhood after finitely many steps, after which the induced TopK set is unchanged.

(ii) Equivalence of the budget rule and the threshold rule. Corollary 2 is stated for a fixed threshold, while Eq. (5) fixes the budget ⌈ρ(l)N⌉\lceil \rho^{(l)} N\rceil. The two are reconciled by the observation that, under condition (i), the fixed-budget TopK mask coincides with the threshold rule at τ(l)\tau^{(l)} equal to the ⌈ρ(l)N⌉\lceil \rho^{(l)} N\rceil-th largest score, and this threshold is locally constant in a neighborhood of the empirical scores. Corollary 2 can therefore be read as the threshold-rule counterpart of the budget-rule mask, and Proposition 4 transfers this stability to the budget-rule mask used in practice.

(iii) Bounded drift under exponential smoothing. The EMA in Eq. (4) introduces a bounded lag between the raw scores and the smoothed scores. We assume α∈(0,1)\alpha \in (0,1) is fixed and that per-step score increments are bounded, so that the smoothed scores inherit the convergence of the raw scores up to an O(1/α)\mathcal{O}(1/\alpha) lag term. This lag does not affect mask stability once condition (i) holds, since the margin δ\delta dominates the lag for sufficiently large ll.

Comment 4: Reconciliation of Table 1 and Appendix G (66.3 vs. 72.4 for Qwen3.5-4B on TextVQA)

We thank the reviewer for spotting this discrepancy and apologize for the confusion. The two numbers correspond to different experimental configurations, and we will make this explicit in the revision.

Specifically, the 72.4 result in Appendix G is obtained with the full SparEdge pipeline, i.e., cross-modal redundancy scoring (Eqs. 1–4) combined with edge-cloud collaborative dynamic sparse inference (Algorithm 1), using the per-layer budget allocation and partition returned by the DP. The 66.3 result corresponds to an ablation configuration in which the edge-cloud partition is fixed (all layers executed on the edge in INT4, no cloud offloading) while retaining the same redundancy-aware mask, which isolates the contribution of the collaborative scheduling module. The gap of 6.1 points therefore measures the accuracy benefit of resource-aware partitioning and FP16 cloud execution for the layers that the DP assigns to the cloud, rather than a discrepancy in the same setting. We will (i) relabel the Appendix G table to state the configuration explicitly in the caption, (ii) add a cross-reference from Table 1 to the corresponding Appendix G row, and (iii) add a short paragraph in Appendix G explaining that the 66.3 figure is an ablation baseline and should not be compared directly to the full-pipeline number. We will also double-check all other table entries for the same labeling issue and correct any that are ambiguous.

Comment 5: Reconciliation of the TopK rule in Eq. (5) with the threshold-based rule in Appendix A.1

We thank the reviewer for this implementation question. The two rules are equivalent up to the choice of threshold, and we clarify the implementation as follows.

Eq. (5) specifies a budget-based rule: retain the top ⌈ρ(l)N⌉\lceil \rho^{(l)} N \rceil tokens by smoothed score, which fixes the number of retained tokens per layer. Appendix A.1 describes a threshold-based rule, in which tokens with s~i(l)<τ(l)\tilde{s}_i^{(l)} < \tau^{(l)} are discarded, which fixes the score threshold rather than the count. In our implementation, the threshold τ(l)\tau^{(l)} is not a free hyperparameter; it is set at each layer to the ⌈ρ(l)N⌉\lceil \rho^{(l)} N \rceil-th largest smoothed score, so that the threshold rule and the TopK rule produce identical masks by construction. The threshold form is used in Appendix A.1 only because it admits a vectorized implementation (a single comparison against a scalar) that is faster on the edge device than a top-kk selection, and because it makes the per-layer budget ρ(l)\rho^{(l)} returned by Algorithm 1 directly interpretable as a quantile of the score distribution. We will add a sentence to Appendix A.1 stating this equivalence explicitly, noting that τ(l)\tau^{(l)} is computed as the ⌈ρ(l)N⌉\lceil \rho^{(l)} N \rceil-th order statistic of s~(l)\tilde{\mathbf{s}}^{(l)} at every layer and every decoding step, and that no additional threshold hyperparameter is introduced. We will also clarify that ties are broken by token index for determinism, which matters for reproducibility on the edge side.


We hope these clarifications address the reviewer's concerns. All corrections to equations, pseudocode, table captions, and appendix assumptions will be incorporated in the revised manuscript, and we have marked the affected passages for consistency checking across the paper.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.