- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes DegreeSpar, a structured compression framework for efficient secure Transformer inference. The central idea is to use the secure nonlinear evaluation degree as a shared, computation-aware optimization variable, rather than optimizing polynomial approximation, token pruning, and model-dimension pruning independently. DegreeSpar represents Softmax and GeLU evaluations through structured degree matrices and induces sparsity at three granularities: entry-wise degree reduction to lower nonlinear cryptographic cost, token-aligned sparsity to remove sequence-dependent computation, and dimension-aligned sparsity to eliminate redundant FFN dimensions. To make aggressive low-degree approximation trainable, the method introduces approximation-aware training, including adaptive noise injection for Softmax and soft-boundary training for segmented GeLU approximations. Token sparsity is learned in a canonical rank space and instantiated securely with fixed per-layer retention budgets, while dimension sparsity is compacted into the deployed model offline.
Critical strengths
-
A principled and well-motivated unification of multiple compression dimensions through secure evaluation degree.
The strongest aspect of the paper is the formulation of polynomial evaluation degree as a shared, computation-aware optimization variable that jointly connects nonlinear approximation, token removal, and model-dimension pruning. This is more compelling than simply stacking existing compression techniques, since the degree variable is directly tied to the cryptographic cost of secure nonlinear evaluation and provides a common space in which different forms of redundancy can be coordinated. The matched-compression comparison, where DegreeSpar retains 80.12% ViT-S accuracy versus 76.41% for independently stacked approximation, token pruning, and model pruning, provides concrete evidence that this joint formulation mitigates the error accumulation caused by independently optimized compression decisions. This conceptual unification is the main reason I view the paper as more than an incremental combination of existing techniques. -
Strong end-to-end validation demonstrating meaningful accuracy--latency improvements in secure inference.
The paper evaluates DegreeSpar across multiple Transformer scales, vision and language tasks, sequence lengths, and network conditions using an actual secure-inference backend, rather than relying only on FLOPs or proxy cost estimates. DegreeSpar achieves 2.29×--6.63× end-to-end speedups while largely preserving task performance, and it also improves substantially over the closest hybrid baseline, CipherPrune: on BERT/SST-2, DegreeSpar achieves 92.68% accuracy in 110.55 s, compared with 92.66% in 167.26 s for CipherPrune. These results make the proposed optimization framework practically consequential and substantially strengthen the case that the method delivers real system-level gains rather than only theoretical savings.
Critical weaknesses
-
The evaluation does not cover modern large language models or autoregressive generation settings.
The empirical study is limited to ViT-Small/Base/Large and BERT-Base, with sequence lengths up to 512 tokens. While this is sufficient to demonstrate the core mechanism, it leaves open whether DegreeSpar remains effective in the settings that are increasingly central to secure Transformer inference, such as decoder-only LLMs with autoregressive decoding, KV-cache reuse, much longer contexts, and substantially larger model scales. These settings have different computational bottlenecks and may change the relative benefit of nonlinear degree reduction, token pruning, and dimension pruning. As a result, the current experiments establish the effectiveness of DegreeSpar for ViT/BERT-style workloads, but provide limited evidence for its broader applicability to modern secure LLM inference. -
The empirical contribution of dimension-aligned sparsity appears relatively limited compared with the paper's unified three-way compression narrative.
A central claim is that the shared degree space jointly enables nonlinear degree reduction, token-level computation removal, and model-dimension removal. However, in the reported ViT-S/ImageNet cost breakdown, entry-wise degree reduction and token removal account for 42% and 54% of the total cost reduction, respectively, whereas dimension-level removal contributes only 4% under the stated sequential accounting. The authors correctly note that these numbers are order-dependent and should not be interpreted as fully independent contributions, but the result still suggests that most of the practical gain in the representative setting comes from degree reduction and token sparsification. Stronger ablations across models and tasks would be useful to demonstrate that dimension-aligned sparsity is a consistently important component rather than a comparatively minor addition to the overall framework. -
The comparison against the independently stacked baseline could be substantially stronger.
The paper uses the large accuracy gap between DegreeSpar and a modular combination of approximation, token pruning, and model pruning as key evidence for the benefit of joint optimization in the degree space. However, the stacked baseline retains the original hyperparameter settings of the component methods while matching their approximation degrees, token-retention budgets, and dimension-pruning ratios to DegreeSpar. This comparison convincingly shows that naive composition of independently designed methods can lead to severe error accumulation, but it does not fully isolate whether the advantage comes specifically from the proposed degree-space formulation or more generally from performing coordinated joint optimization. A stronger baseline with jointly tuned compression strengths, end-to-end fine-tuning, or other coordinated optimization strategies would make the central claim considerably more convincing.
Questions for authors
-
Clarification regarding applicability to modern LLM inference.
The current evaluation is restricted to ViT and BERT-style models, with sequence lengths up to 512. Could the authors clarify whether the DegreeSpar formulation is expected to remain effective for decoder-only LLMs under autoregressive generation, where KV-cache reuse, causal attention, and token-by-token decoding substantially alter the computation profile? In particular, which parts of the current token-aligned and dimension-aligned sparsification mechanisms would remain unchanged, and which would require modification? This would materially affect how broadly the paper's claims about secure Transformer inference should be interpreted. -
Clarification and additional evidence on the role of dimension-aligned sparsity.
In the reported ViT-S/ImageNet breakdown, dimension-level removal accounts for only 4% of the total cost reduction under the paper's sequential accounting, compared with 42% from entry-wise degree reduction and 54% from token removal. Could the authors provide additional ablations showing the effect of disabling dimension-aligned sparsity, ideally across several models/tasks, and report the resulting accuracy and end-to-end latency? This would help determine whether dimension-aligned sparsity is a substantively important part of the proposed unified framework or primarily a secondary component in the evaluated settings. -
Clarification regarding the strength of the modular baseline.
The matched-compression comparison is central evidence for the benefit of the shared degree-space formulation, but the modular baseline appears to retain the original hyperparameter settings of the constituent approximation, token-pruning, and model-pruning methods. Could the authors report a stronger jointly tuned baseline, for example by jointly optimizing compression strengths and performing end-to-end fine-tuning after combining the three mechanisms? If the performance gap remains large under such a baseline, it would provide substantially stronger evidence that the gain comes specifically from the proposed degree-space coordination rather than from comparing a jointly optimized method against a relatively naive composition of independently configured techniques.
Thank you to the authors for their detailed responses and additional experiments. I apologize for my delayed follow-up due to a busy schedule recently.
The rebuttal has substantially addressed several of my concerns, particularly regarding the contribution of dimension-aligned sparsification (Q2) and the stronger jointly optimized baseline (Q3).
I still have two minor concerns. First, the response to Q1 remains largely qualitative, without concrete end-to-end quantitative results on decoder-only LLMs. Second, regarding Q3, the authors mention that Joint-Independent retains more tokens and produces fewer fully sparse degree rows and columns, but the actual sparsity ratios are not reported. I would appreciate seeing these statistics to better understand the benefits of the shared degree-space formulation.
Overall, I appreciate the authors' efforts and find the additional evidence convincing. I am willing to raise my overall rating.