acceptodds
Under review as a conference paper at ICLR 2027

DegreeSpar: Structured Degree Sparsity for Efficient Secure Transformer Inference

Abstract

Secure Transformer inference protects sensitive inputs but incurs substantial cryptographic overhead, with nonlinear operations such as Softmax and GeLU becoming major bottlenecks. Existing compression methods reduce nonlinear complexity, sequence-dependent computation, or model structure through separately defined compression variables. Under aggressive compression, however, these independently optimized perturbations can accumulate: at a matched compression level, stacking representative approximation, token-pruning, and model-pruning methods reduces ViT-S accuracy from 80.20% to 76.41%. We introduce DegreeSpar, which formulates secure Transformer compression as structured sparsification over nonlinear polynomial degrees. Polynomial degree directly controls the cost of secure nonlinear evaluation, while computation-aligned zero-degree structures expose token-level and model-dimension computation as removable within the same optimization space. DegreeSpar further incorporates approximation-aware training for low-degree Softmax and GeLU, enabling aggressive degree reduction and creating the optimization headroom required for structured computation removal. Across vision and language Transformers, DegreeSpar consistently improves the accuracy–latency trade-off across model scales, tasks, and sequence lengths, achieving speedups from to over the corresponding baselines. Under the same network setting, DegreeSpar achieves 92.68% accuracy on BERT/SST-2 in 110.55 s, compared with 92.66% in 167.26 s for CipherPrune, the closest prior hybrid secure-inference approach. These results establish structured polynomial degree as an effective shared optimization space for secure Transformer compression.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Secure Transformer Inference, Privacy-Preserving Machine Learning, Transformer Compression, Structured Sparsity, Polynomial Approximation, Token Pruning

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 2 reviews, above 85% of the others.
Reviewer_QR5VOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes DegreeSpar, a structured compression framework for efficient secure Transformer inference. The central idea is to use the secure nonlinear evaluation degree as a shared, computation-aware optimization variable, rather than optimizing polynomial approximation, token pruning, and model-dimension pruning independently. DegreeSpar represents Softmax and GeLU evaluations through structured degree matrices and induces sparsity at three granularities: entry-wise degree reduction to lower nonlinear cryptographic cost, token-aligned sparsity to remove sequence-dependent computation, and dimension-aligned sparsity to eliminate redundant FFN dimensions. To make aggressive low-degree approximation trainable, the method introduces approximation-aware training, including adaptive noise injection for Softmax and soft-boundary training for segmented GeLU approximations. Token sparsity is learned in a canonical rank space and instantiated securely with fixed per-layer retention budgets, while dimension sparsity is compacted into the deployed model offline.

Critical strengths
  • A principled and well-motivated unification of multiple compression dimensions through secure evaluation degree.
    The strongest aspect of the paper is the formulation of polynomial evaluation degree as a shared, computation-aware optimization variable that jointly connects nonlinear approximation, token removal, and model-dimension pruning. This is more compelling than simply stacking existing compression techniques, since the degree variable is directly tied to the cryptographic cost of secure nonlinear evaluation and provides a common space in which different forms of redundancy can be coordinated. The matched-compression comparison, where DegreeSpar retains 80.12% ViT-S accuracy versus 76.41% for independently stacked approximation, token pruning, and model pruning, provides concrete evidence that this joint formulation mitigates the error accumulation caused by independently optimized compression decisions. This conceptual unification is the main reason I view the paper as more than an incremental combination of existing techniques.

  • Strong end-to-end validation demonstrating meaningful accuracy--latency improvements in secure inference.
    The paper evaluates DegreeSpar across multiple Transformer scales, vision and language tasks, sequence lengths, and network conditions using an actual secure-inference backend, rather than relying only on FLOPs or proxy cost estimates. DegreeSpar achieves 2.29×--6.63× end-to-end speedups while largely preserving task performance, and it also improves substantially over the closest hybrid baseline, CipherPrune: on BERT/SST-2, DegreeSpar achieves 92.68% accuracy in 110.55 s, compared with 92.66% in 167.26 s for CipherPrune. These results make the proposed optimization framework practically consequential and substantially strengthen the case that the method delivers real system-level gains rather than only theoretical savings.

Critical weaknesses
  • The evaluation does not cover modern large language models or autoregressive generation settings.
    The empirical study is limited to ViT-Small/Base/Large and BERT-Base, with sequence lengths up to 512 tokens. While this is sufficient to demonstrate the core mechanism, it leaves open whether DegreeSpar remains effective in the settings that are increasingly central to secure Transformer inference, such as decoder-only LLMs with autoregressive decoding, KV-cache reuse, much longer contexts, and substantially larger model scales. These settings have different computational bottlenecks and may change the relative benefit of nonlinear degree reduction, token pruning, and dimension pruning. As a result, the current experiments establish the effectiveness of DegreeSpar for ViT/BERT-style workloads, but provide limited evidence for its broader applicability to modern secure LLM inference.

  • The empirical contribution of dimension-aligned sparsity appears relatively limited compared with the paper's unified three-way compression narrative.
    A central claim is that the shared degree space jointly enables nonlinear degree reduction, token-level computation removal, and model-dimension removal. However, in the reported ViT-S/ImageNet cost breakdown, entry-wise degree reduction and token removal account for 42% and 54% of the total cost reduction, respectively, whereas dimension-level removal contributes only 4% under the stated sequential accounting. The authors correctly note that these numbers are order-dependent and should not be interpreted as fully independent contributions, but the result still suggests that most of the practical gain in the representative setting comes from degree reduction and token sparsification. Stronger ablations across models and tasks would be useful to demonstrate that dimension-aligned sparsity is a consistently important component rather than a comparatively minor addition to the overall framework.

  • The comparison against the independently stacked baseline could be substantially stronger.
    The paper uses the large accuracy gap between DegreeSpar and a modular combination of approximation, token pruning, and model pruning as key evidence for the benefit of joint optimization in the degree space. However, the stacked baseline retains the original hyperparameter settings of the component methods while matching their approximation degrees, token-retention budgets, and dimension-pruning ratios to DegreeSpar. This comparison convincingly shows that naive composition of independently designed methods can lead to severe error accumulation, but it does not fully isolate whether the advantage comes specifically from the proposed degree-space formulation or more generally from performing coordinated joint optimization. A stronger baseline with jointly tuned compression strengths, end-to-end fine-tuning, or other coordinated optimization strategies would make the central claim considerably more convincing.

Questions for authors
  • Clarification regarding applicability to modern LLM inference.
    The current evaluation is restricted to ViT and BERT-style models, with sequence lengths up to 512. Could the authors clarify whether the DegreeSpar formulation is expected to remain effective for decoder-only LLMs under autoregressive generation, where KV-cache reuse, causal attention, and token-by-token decoding substantially alter the computation profile? In particular, which parts of the current token-aligned and dimension-aligned sparsification mechanisms would remain unchanged, and which would require modification? This would materially affect how broadly the paper's claims about secure Transformer inference should be interpreted.

  • Clarification and additional evidence on the role of dimension-aligned sparsity.
    In the reported ViT-S/ImageNet breakdown, dimension-level removal accounts for only 4% of the total cost reduction under the paper's sequential accounting, compared with 42% from entry-wise degree reduction and 54% from token removal. Could the authors provide additional ablations showing the effect of disabling dimension-aligned sparsity, ideally across several models/tasks, and report the resulting accuracy and end-to-end latency? This would help determine whether dimension-aligned sparsity is a substantively important part of the proposed unified framework or primarily a secondary component in the evaluated settings.

  • Clarification regarding the strength of the modular baseline.
    The matched-compression comparison is central evidence for the benefit of the shared degree-space formulation, but the modular baseline appears to retain the original hyperparameter settings of the constituent approximation, token-pruning, and model-pruning methods. Could the authors report a stronger jointly tuned baseline, for example by jointly optimizing compression strengths and performing end-to-end fine-tuning after combining the three mechanisms? If the performance gap remains large under such a baseline, it would provide substantially stronger evidence that the gain comes specifically from the proposed degree-space coordination rather than from comparing a jointly optimized method against a relatively naive composition of independently configured techniques.

Reviewer_mi3gOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes DegreeSpar for efficient secure Transformer inference. The main idea is to use the degree of secure polynomial evaluation as a shared and computation-aware optimization space.The degree reduction reduces the cost of nonlinear operations, while token-aligned and dimension-aligned degree sparsity enables removal of larger computational structures.

Critical strengths
  1. The paper presents a motivated unification of several secure-inference compression mechanisms. Using polynomial evaluation degree as a shared optimization coordinate is intuitive because degree directly affects secure nonlinear evaluation cost. The structured formulation then connects nonlinear approximation with token- and dimension-level computation removal. This gives the paper a clear technical core.
  2. The evaluation experiment is strong. They cover both vision and language Transformers, multiple model scales, five datasets, sequence lengths from 64 to 512, and several network conditions. The reported latency is measured on an actual secure-inference backend.
Critical weaknesses
  1. I think the contribution of individual components is not fully isolated. DegreeSpar contains several ideas: approximation-aware training, entry-wise degree reduction, token-aligned sparsity, dimension-aligned sparsity, and VCA-based token ranking. The paper provides evidence for some of these components separately, but I could not find a ablation comparing degree-only, degree+token, degree+dimension, and the full method under matched settings. A cleaner component analysis would make it easier to understand which parts of the full framework are necessary.

  2. The breadth of the comparison against prior methods is narrower than the breadth of the overall evaluation. DegreeSpar is evaluated across many models and datasets, but the main comparisons with existing compression approaches are concentrated on ViT-Base/ImageNet and BERT/SST-2. The remaining settings mainly compare DegreeSpar with the uncompressed baseline and therefore the claim of superiority over existing methods across all settings is not directly established.

Questions for authors
  1. Can the authors provide a clean component ablation and anlysis, e.g., degree reduction only, degree+token sparsity, degree+dimension sparsity, and the full DegreeSpar method?

  2. Can the authors clarify the extent to which the comparison with prior methods generalizes beyond ViT-Base/ImageNet and BERT/SST-2? Are comparable results available for the other model/task settings in Table 3?

Comments on the paper

1 comment

Thank you to the authors for their detailed responses and additional experiments. I apologize for my delayed follow-up due to a busy schedule recently.

The rebuttal has substantially addressed several of my concerns, particularly regarding the contribution of dimension-aligned sparsification (Q2) and the stronger jointly optimized baseline (Q3).

I still have two minor concerns. First, the response to Q1 remains largely qualitative, without concrete end-to-end quantitative results on decoder-only LLMs. Second, regarding Q3, the authors mention that Joint-Independent retains more tokens and produces fewer fully sparse degree rows and columns, but the actual sparsity ratios are not reported. I would appreciate seeing these statistics to better understand the benefits of the shared degree-space formulation.

Overall, I appreciate the authors' efforts and find the additional evidence convincing. I am willing to raise my overall rating.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.