- Rating
- 2
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The authors propose a new architecture for NLI called SMIC. The idea is to construct a single representation from the premise-hypothesis input pair, which is then fed directly into each Transformer layer. The representation is computed through a set of convolutions with different kernels using a method called “complex-vector token representation” (CVTR) published last year. The authors demonstrate gains across different model families.
Critical strengths
Good evaluation practice reporting FLOPs and results with confidence intervals from different random seeds.
The paper aims to give more color to the empirical results by running sensitivity diagnostics that circle back to their initial motivation of NLI models being overly sensitive to local change.
Critical weaknesses
My biggest concern is clarity. The paper is very tiring to read: Section 3 is swamped with details that distract from the actual contributions. Fig. 1 is not much of a help either. My impression is that the AI writing assistance was a bit over-used here: the text may contain the actual substance of the paper, but it is buried under many unnecessary details. A good example is the related work paragraph L045-L071: Most of these papers are not about NLI at all. They may be related to the proposed method in some particular aspect, but that aspect is often so general (“model sensitivity”, “convolutional sequence models”) that I don’t get much out of the entire paragraph - not what I’d expect in the introduction.
The evaluation section 4 is better in terms of clarity. Unfortunately, only small-ish models are tested, and the gains in the apples-to-apples comparison (Table 3) seem rather modest, so I’m not sure whether this justifies the increased system complexity.
Finally, IIUC the novelty in this paper is feeding the input representation directly to each Transformer layer. But there are many other ways to represent the input premise-hypothesis pair besides CVTR. I would have wished for a target ablation that justifies the use of CVTR+TCA.
Questions for authors
-
Have you tested larger models?
-
Have you tried other ways to represent the input and feed them to each layer?