- Rating
- 4
- Confidence
- 4
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
The paper analyzes the formal verification of tranformer-based language models. It exploits joint fusion mechanisms to enclose the output of individual layers in language models with zonotopes, in particular for attention layers and their respective skip connection. This allows the authors to verify models of the size of GPT-2, a significant improvement over the current state of the art.
Critical strengths
- The topic is highly relevant to formally ensure robustness of language models
- The applied fusion mechanism trough residual Taylor expansion is novel and seems to work well
- It outperforms the state of the art by verifying significantly larger models.
- Proofs are formalized in Lean.
Critical weaknesses
- I have found the write-up to be a little difficult to follow. I am very familiar with zonotope-based neural network verification, but I think for non-experts it is difficult to understand. I would suggest revising the running example and instead first give a short introduction in zonotopes and verification with zonotopes generally (i.e., what is known), before introducing your new contributions. Maybe the current running example -- which is nice overall, just not necessarily at this position -- could be moved to the appendix. In Fig. 2, one could maybe show the output in one axis or at least have the same axis limits to better illustrate the smaller output set.
- The perturbation radii and which tokens are perturbed seem to be chosen somewhat randomly, making it difficult to assess what properties one tries to verify.
- Related work is only presented in the appendix
Questions for authors
- How are the relation between shared and local generators balanced? I assume there has to be some trade-off between the two.
- Could one also store the local generators of the structured zonotope as sparse matrices? This might simplify the balancing.
- Also, is added the interval bound b sometimes absorbed back into the generators to avoid a wrapping effect?
- How is the perturbation radius chosen? For image data as typically considered in nnv, adding some bounded noise is sensible, but how does this translate to text input?
- The text says up to three token embeddings are perturbed. How were these chosen? What limits more perturbations? I would assume that after one attention layer, the perturbation is distributed across all tokens anyway?