acceptodds
Under review as a conference paper at ICLR 2027

ZonoGPT: Towards An Abstract Domain for Verifying Large GPT Models

Abstract

Transformer-based models are widely used for reasoning, coding, and multimodal agentic tasks. To provide formal assurance of desirable behaviors, such as robustness, safety, and fairness, neural network verification techniques prove required properties and provide auditable guarantees before deployment. However, prior work remains limited to small or restricted Transformers, and maintaining precision across deep models remains challenging. In this work, we introduce ZonoGPT, an abstract domain for verifying large transformers that maintains a space complexity independent of network depth. ZonoGPT uses a structured zonotope and a generator reduction mechanism to efficiently preserve correlations. To maintain precision, it introduces block-specific fused transformations for Attention and LayerNorm that retain feature relations, along with an affine transform for GELU that preserves generator relations. These mechanisms enable ZonoGPT to be the first approach to verify standard architectures, scaling to official HuggingFace models up to GPT-2 Medium (24 blocks, 300M+ parameters) and successfully verifying 1,339 instances across text and vision tasks.

Primary area:
interpretability and explainable AI
Keywords:
neural network verification, certified robustness, transformer verification, transformer, abstract interpretation

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_Rg7LOpenReview ↗
Rating
4
Confidence
4
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

The paper analyzes the formal verification of tranformer-based language models. It exploits joint fusion mechanisms to enclose the output of individual layers in language models with zonotopes, in particular for attention layers and their respective skip connection. This allows the authors to verify models of the size of GPT-2, a significant improvement over the current state of the art.

Critical strengths
  • The topic is highly relevant to formally ensure robustness of language models
  • The applied fusion mechanism trough residual Taylor expansion is novel and seems to work well
  • It outperforms the state of the art by verifying significantly larger models.
  • Proofs are formalized in Lean.
Critical weaknesses
  • I have found the write-up to be a little difficult to follow. I am very familiar with zonotope-based neural network verification, but I think for non-experts it is difficult to understand. I would suggest revising the running example and instead first give a short introduction in zonotopes and verification with zonotopes generally (i.e., what is known), before introducing your new contributions. Maybe the current running example -- which is nice overall, just not necessarily at this position -- could be moved to the appendix. In Fig. 2, one could maybe show the output in one axis or at least have the same axis limits to better illustrate the smaller output set.
  • The perturbation radii and which tokens are perturbed seem to be chosen somewhat randomly, making it difficult to assess what properties one tries to verify.
  • Related work is only presented in the appendix
Questions for authors
  • How are the relation between shared and local generators balanced? I assume there has to be some trade-off between the two.
  • Could one also store the local generators of the structured zonotope as sparse matrices? This might simplify the balancing.
  • Also, is added the interval bound b sometimes absorbed back into the generators to avoid a wrapping effect?
  • How is the perturbation radius chosen? For image data as typically considered in nnv, adding some bounded noise is sensible, but how does this translate to text input?
  • The text says up to three token embeddings are perturbed. How were these chosen? What limits more perturbations? I would assume that after one attention layer, the perturbation is distributed across all tokens anyway?
Reviewer_ihoGOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

Neural Network Verification has made significant advances in the last couple of years, but still struggles to deal with transformers due to the fact that linearly bounding the nonlinear attention heads leads to significant overapproximation errors. This paper proposes new relaxations for the different components of transformers by deriving zonotope abstractions for their components which jointly relax multiple components rather than proceedings layer by layer. The resulting "abstract domain" introduced by the authors can be used to verify transformers at a much larger scale than was possible before, and the approach is shown to scale to GPT-2 Medium while other existing verifiers fail to tackle transformers of that size.

Critical strengths
  • The lack of verifiers that scale to transformers is clearly an issue in the domain, so this is a topic that is worth investigating. To the best of my knowledge, this is also the first paper to scale to GPT-2 blocks at Medium scale, previous papers focused only on some components of transformers or on verifying smaller toy models.
  • The paper's proofs were formalised in Lean and then checked which, assuming a correct formalisation, provides additional trust in the theory.
Critical weaknesses
  • The gap between the paper's motivation and initial examples and the setting which the algorithm is actually applied in seems to be substantial. Verification is run at tiny radii from 10−510^{-5} to 10−310^{-3} and only 68 out of 576 instances are verified at 5×10−45 \times 10^{-4}. This is well below the radii usually considered for MNIST, and in the unnormalised space would be equivalent to a perturbation size of ~0.13/255. It makes sense to me that a linearisation of the network's behaviour based on the centre would work well, but I assume this would not transfer to larger perturbation sizes where the nonlinear effect is stronger (this seems supported by the fact that performance collapses when increasing perturbation sizes between 10−410^{-4} and 10^{-30}$. The authors also mention using "official HuggingFace models" in the abstract but then do not actually use such official model checkpoints and instead train models from scratch. Section 3 also makes a memory argument using 512 tokens as the sequence length, but the actual verification queries are run with 28-48 tokens. It would be important for the paper to actually demonstrate scaling to larger sequence lengths. The experiments also seem to be run on rather small benchmarks, as far as I understand, only four MNIST images are used in the experiments.
  • The experimental evaluation does not enable a comparison with previous work since all baselines just achieve zero performance with some of them running out of memory on the small GPUs used for the experimentation. I am also very surprised that α,β\alpha, \beta-CROWN and DeepZ fail even on models with just two blocks and that IBP isn't able to verify anything on the IBP-trained (and therefore suitably regularised) SST models. No comparison with the DeepT verifier (probably the closest work) is run because it doesn't support GeLUs. This seems like a weak justification, a comparison could simply be run on the benchmarks from the prior work by [1, 2, 3].
  • The paper has multiple inconsistencies, needs better contextualisation with existing work and despite being familiar with the topic I found it extremely hard to follow given the current organisation of contents as well as the amount of notation and symbols. For example, the shared generators are initialised to zero in Algorithm 1 and never seem to be created, I could not find where they are created, so the algorithm seems to contradict the textual description. The main idea of linearising a function at the zonotope's centre and then bounding the remainder seems to be closely related to existing work on zonotopes such as [4], the paper's main contributions of applying this to transformers should be clarified and more of the existing zonotope literature discussed. Lastly, the notation needs to be significantly simplified. Right now, the core ideas which seem relatively simple are buried under dense and overloaded notation. To only name a few examples, L is used for both local generators and the number of blocks, m is used for #shared generators and the GELU offset, d is used for hidden size/GeLU half-width/the diag generators in Eq. 28. There are a number of other clashes like this, so a thorough revision is needed. The motivating example then also uses a lot of notation and references things that haven't been introduced yet, so Section 3 is basically impossible to understand without reading Section 4 first.

References

[1] Zhouxing Shi, Huan Zhang, Kai-Wei Chang, Minlie Huang, and Cho-Jui Hsieh. Robustness verification for transformers. arXiv preprint arXiv:2002.06622, 2020.

[2] Gregory Bonaert, Dimitar I Dimitrov, Maximilian Baader, and Martin Vechev. Fast and precise certification of transformers. In Proceedings of the 42nd ACM SIGPLAN international conference on programming language design and implementation, pp. 466–481, 2021.

[3] Pei Huang, Dennis Wei, Omri Isac, Haoze Wu, Min Wu, and Clark Barrett. Parameterized abstract interpretation for transformer verification. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 40, pp. 35500–35508, 2026.

[4] Althoff, M., Stursberg, O., & Buss, M. (2008, December). Reachability analysis of nonlinear systems with uncertain parameters using conservative linearization. In 2008 47th IEEE Conference on Decision and Control (pp. 4042-4048). IEEE.

Questions for authors
  • Where are the shared generators created and how many of them are used in the experiments?
  • How is the interval in the example in section 3 computed, could some additional intermediate steps for this be provided in the appendix?
  • What clean accuracies do the self-trained models achieve? Why does IBP not verify any models that were trained using IBP?
  • Could the experimental results be expanded to use more than just four MNIST images?
  • How do the authors ensure that the Lean proof formalisation is consistent with the proofs in the paper?
  • How does the method scale to longer sequence lengths like the 512 used in the prose text?
  • How does the method perform on pretrained GPT-2 models fine-tuned for classification?
Reviewer_1oLQOpenReview ↗
Rating
4(+1)
Confidence
4(+1)
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper presents ZonoGPT, a verification method for Transformers. It combines established verification techniques with Transformer-specific designs to control memory use and keep bounds tight. The experiments cover GPT-2-based classifiers with ~100M parameters.

Critical strengths
  1. Scaling verification to standard GPT-2 architectures is a useful contribution. The method is clear and fits the Transformer structure well.

  2. The ablation results support the main design choices, especially the benefit of fused Attention.

Critical weaknesses

The evaluation covers GPT-2-based classifiers trained on MNIST and SST, with a small number of input examples per setting. Results on pretrained GPT-2 models and a wider range of inputs would help show how broadly the method applies.

Questions for authors

Questions:

  1. Are there any other existing verification methods designed for Transformers or LLMs that can directly handle the standard GPT-2 architecture? If so, how do they compare with ZonoGPT?

  2. Were the MNIST and SST models trained with any methods designed to improve robustness? trained with any method to imporve verifiability? Table 3 mentions IBP for SST; could you clarify how it was used during training?

Minor Comments / Typos:

  1. Page 1, lines 46-48: "recent techniques ... and does not" should use "do not".
  2. Page 1, near line 53: "memory footprint remain bounded" should use "remains".
  3. Page 2, line 71: "the GPT models ... is still very small" should use "are".

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.