- Rating
- 1
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies input-relational verification for Transformers. It considers pairs of nearby inputs and requires their outputs to remain close. It first introduces a self-composition baseline. It then proposes ReVerT, which explicitly propagates differences between two executions through the Transformer. The main technical step is to reformulate differences in attention operations so that cross-execution dependencies are preserved, and then apply McCormick relaxations. Two variants are proposed: ReVerT-F for faster bound propagation and ReVerT-P for tighter LP-based bounds. The method is evaluated on CCPP, Yelp, and SST. The reported results show tighter bounds than self-composition and naive difference propagation.
Critical strengths
- The paper identifies a useful precision issue in naive self-composition. Tracking cross-execution differences inside the network is more informative than abstracting two executions independently and only relating them at the output. The experiments show that this can substantially tighten certified bounds.
- The attention-specific reformulation is simple and practically useful. Rewriting the difference of bilinear attention terms before applying McCormick relaxation preserves dependencies that would otherwise be lost. The empirical results suggest that this improves verification precision.
Critical weaknesses
- The novelty relative to existing input-relational verification is not sufficiently established. The paper cites prior work on input-relational DNN verification, including Banerjee et al. (2024), but does not meaningfully compare ReVerT with these methods. The central idea of tracking relations across two executions is already the main principle of this literature. The paper mainly argues that no prior method targets Transformers. I do not think this is enough to establish a strong methodological contribution. The paper should explain what is fundamentally new beyond adapting existing relational verification ideas to Transformer attention.
- The problem formulation and evaluation are weaker than the paper’s framing suggests. Definition 1 is a fixed-scale pairwise output-sensitivity condition over a bounded domain. It is closely related to standard continuity or robustness notions, so I do not see a strong conceptual contribution in introducing it as “global robustness.” At the same time, the introduction gives broad safety-critical motivations, while the experiments use only CCPP, Yelp, and SST. The main certification-success experiment is based on only five CCPP tasks. This is not enough to support the broad claims made in the paper.
Questions for authors
-
The paper cites prior work on input-relational verification of DNN, which is very close to the current topic. What is the precise technical difference between ReVerT and applying or extending those relational verification methods to Transformer attention? Which part of the proposed abstraction is fundamentally new?
-
What is the conceptual significance of Definition 1? It appears to be a fixed-, fixed- pairwise output-sensitivity condition over a bounded domain, closely related to standard continuity notions. Why should this be viewed as a new “global robustness” formulation?
-
Do the evaluated models use standard LayerNorm, or the no-variance LayerNorm described in the paper?
-
Can the authors report bound comparisons on exactly the same set of tasks for all methods? The current averages may use different valid task subsets, which makes the comparison difficult to interpret.