acceptodds
Withdrawn from ICLR 2027

SMIC: Shared Multiscale Input-Reference Conditioning for Natural Language Inference

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Natural Language Inference models can be sensitive to localized lexical changes. We introduce Shared Multiscale Input-Reference Conditioning (SMIC), which constructs a single input-dependent multiscale reference and reuses it across Transformer layers. Each layer applies its own multiscale transformation to the shared reference and injects the resulting conditioning feature before the attention update. We instantiate SMIC in a token–context Transformer (TCT) and conduct from-scratch comparisons against decoder-based and MLP baselines across all three ANLI rounds, with additional encoder comparisons on ANLI-R1. TCT-SMIC attains the highest observed ten-seed mean accuracy among the evaluated configurations. The ablations further support the contribution of SMIC. We also evaluate a real-valued Llama-style plug-in and extend deletion/insertion diagnostics to pretrained ModernBERT encoder and Llama decoder models with and without SMIC. Across the evaluated perturbation protocols, TCT-SMIC exhibits comparatively low pooled-attention redistribution and smaller signed accuracy changes. We further establish a finite-sample bound on average subset-mass redistribution in terms of the mean Jensen–Shannon divergence between the restricted and renormalized attention distributions.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Natural language inference, Transformer architectures, Multiscale representations, Feature conditioning, Perturbation sensitivity

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_L1Q7OpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The authors propose a new architecture for NLI called SMIC. The idea is to construct a single representation from the premise-hypothesis input pair, which is then fed directly into each Transformer layer. The representation is computed through a set of convolutions with different kernels using a method called “complex-vector token representation” (CVTR) published last year. The authors demonstrate gains across different model families.

Critical strengths

Good evaluation practice reporting FLOPs and results with confidence intervals from different random seeds.

The paper aims to give more color to the empirical results by running sensitivity diagnostics that circle back to their initial motivation of NLI models being overly sensitive to local change.

Critical weaknesses

My biggest concern is clarity. The paper is very tiring to read: Section 3 is swamped with details that distract from the actual contributions. Fig. 1 is not much of a help either. My impression is that the AI writing assistance was a bit over-used here: the text may contain the actual substance of the paper, but it is buried under many unnecessary details. A good example is the related work paragraph L045-L071: Most of these papers are not about NLI at all. They may be related to the proposed method in some particular aspect, but that aspect is often so general (“model sensitivity”, “convolutional sequence models”) that I don’t get much out of the entire paragraph - not what I’d expect in the introduction.

The evaluation section 4 is better in terms of clarity. Unfortunately, only small-ish models are tested, and the gains in the apples-to-apples comparison (Table 3) seem rather modest, so I’m not sure whether this justifies the increased system complexity.

Finally, IIUC the novelty in this paper is feeding the input representation directly to each Transformer layer. But there are many other ways to represent the input premise-hypothesis pair besides CVTR. I would have wished for a target ablation that justifies the use of CVTR+TCA.

Questions for authors
  • Have you tested larger models?

  • Have you tried other ways to represent the input and feed them to each layer?

Reviewer_wgQnOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a shared multiscale input-reference conditioning method for the NLI task to improve accuracy on natural language inference (NLI) and reduce the model’s sensitivity to local lexical perturbations. The method extracts multiscale features from the original input through a convolutional module and constructs a reference representation shared across different Transformer layers. Each layer independently applies multiscale transformations to the shared reference representation and, before the attention update, injects the resulting features into the Token and Context representations. The authors achieve excellent results across multiple evaluation datasets and different model families. They also conduct extensive analyses of how the method works.

Critical strengths

1、The experimental results are strong. The paper achieves good performance on multiple ANLI datasets, and the use of multiple random seeds and statistical analyses makes the results relatively reliable.

2、The ablation studies are relatively systematic. The paper investigates the number of learnable windows, whether SMIC is used, reference construction, shared reference versus layer-local reference, and the effects of model depth and tokenizer.

3、The experiments focus not only on accuracy but also analyze the model’s sensitivity to lexical perturbations.

Critical weaknesses

1、The method’s novelty is limited, and the paper does not clearly discuss how it differs from existing methods. The related works discussed in the paper all involve different architectures. For example, Sentence Embeddings in NLI with Iterative Refinement Encoders and Shortcut-Stacked Sentence Encoders for Multi-Domain Inference both study ways of using original input information across layers in NLI tasks. Although these methods differ from SMIC in their specific implementations, the paper does not sufficiently discuss their connections or clearly explain whether SMIC’s core innovation comes from a new architectural mechanism or a combination of existing techniques.

2、The paper’s core motivation is that Transformers are relatively sensitive to local lexical changes, and that these changes may propagate through successive Transformer layers and affect the final prediction. However, the authors do not sufficiently analyze why local lexical perturbations cause prediction errors, or whether these errors are indeed related to the propagation of information across layers.

3、Although the experiments are extensive, they do not sufficiently demonstrate that the method solves the stated problem. While the proposed method achieves good performance in the main ANLI comparisons, the ablation studies show that SMIC’s gains are inconsistent across datasets. Could the authors further analyze the conditions under which the method is applicable and some boundary cases?

Questions for authors

1、Local lexical changes may propagate through successive Transformer layers and affect the final NLI prediction. What exactly is the source of this sensitivity? Is the influence of a local token amplified layer by layer, is important semantic information lost during inter-layer propagation, or is the model itself inadequate at modeling lexical-semantic relations? Could the authors provide a more fine-grained analysis to clarify the mechanism of the problem that SMIC aims to address?

2、SMIC combines several design elements, including multiscale feature extraction, shared reference representations, and layer-specific conditioning. However, it remains unclear which component plays the most critical role in improving model performance. Could the authors further analyze the independent contributions of the different components and clearly identify which design is the primary source of SMIC’s performance gains, as well as why that design can effectively improve NLI accuracy and sensitivity to local lexical perturbations?

3、Table 3 shows that SMIC improves accuracy on ANLI-R2 and R3, but on R1, removing SMIC instead yields higher average accuracy, although the difference is not significant. Could the authors further explain why SMIC has inconsistent effects across datasets? For example, could they analyze which examples have better predictions and which have worse predictions after adding SMIC, and whether these changes are related to the examples’ local lexical features or semantic information?

Reviewer_v1g9OpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper introduces Shared Multiscale Input-Reference Conditioning (SMIC) that creates a representation from convolutional features of the original input at multiple scales and injects layer-specific transformations of this representation into successive transformer layers. The main implementation is built on the existing Token–Context Transformer architecture. Experiments evaluate ANLI accuracy, attention redistribution, and prediction changes under lexical perturbations, and include additional diagnostic evaluations on pretrained backbones.

Critical strengths
  1. The idea of architecture is comprehensible and perhaps even useful. Providing successive layers with features directly based on the original input is a plausible means to retain access to the lexical information at varying scales.

  2. Ten seeds main accuracy comparisons and confidence intervals. The within-backbone ablation shows evidence of modest improvements on ANLI-R2 and ANLI-R3.

  3. The paper notes acknowledges important limitations. It clearly separates the signed accuracy changes from the prediction-flip rates, and acknowledges that its perturbations may alter the ground truth NLI label. The real-valued implementation also shows that SMIC is not intrinsically limited to the complex-valued backbone.

Critical weaknesses
  1. The main comparisons show little evidence of practical task-performance advantages. Tables 1 and 2 train large architectures using random initialization on single rounds of ANLI. Several encoder baselines perform near chance, making it difficult to interpret the observed advantage as either a general useful architecture or greater suitability for this specific training regime. The pretrained experiments provide deletion/insertion diagnostics but not the corresponding NLI accuracy values. Thus, the paper fails to demonstrate that SMIC improves task performance when combined with a competitive pretrained model.

  2. The improvement attributable to SMIC is small and erratic across rounds. Indeed, comparisons between architectures provide less information about the proposed component than Table 3. It shows improvements of 1.17 and 1.39 percentage points in R2 and R3, but a decrease of 1.19 points in R1, with the latter confidence interval containing 0. This is significant because the large perturbation analysis focuses on R1, where no clean-accuracy benefit is shown. The findings support a more modest conclusion than the headline comparisons indicate.

  3. When compared to simpler alternatives, the overhead of architecture does not appear to be worth the complexity. The token-context representation and attention mechanism are based on previous work [1]. SMIC uses several convolution banks, learned scale mixtures, and layer-specific reference transformations. Existing ablations are useful, but they do not show that this machinery outperforms simpler input reinjection or fixed multiscale convolutional conditioning with similar parameter and compute budgets. The theoretical result does not give an answer, where proposition 1 follows from standard divergence inequalities and applies to any attention distribution, rather than proving a SMIC-specific property.

  4. Perturbation metrics don't improve robustness. If the rates at which incorrect predictions become correct and correct predictions become incorrect are comparable, and the signed change in accuracy is minor. It can thus conceal a large amount of prediction instability. Furthermore, the modified instances retain their original labels, even if the replacements alter the correct relationship. Furthermore, low attention JSD does not imply that predictions are still correct and respond appropriately to edits that alter the meaning. The paper mentions these limitations, but they are not adequately validated in the paper's primary motivation. Separate break, recovery, and flip rates, as well as semantically verified perturbations, would provide more direct evidence.

  5. Better controls are required for deletion and insertion diagnostics. Insertion starts with most tokens replaced but leaves the pooling token intact, and its score measures confidence relative to the original prediction without subtracting the initial masked-input confidence. A model that remains confident despite receiving little useful information can therefore obtain a high insertion score. However, the distinction is obscured when this is combined with deletion in a single summary. Reporting the two curves separately, combined with random-ranking controls and masked-input confidence, would improve the results' interpretability.

  6. The presentation makes the method difficult to follow. Figure 1 does not clearly communicate how the shared reference is constructed, transformed, and injected into successive layers. The method section contains dense paragraphs with extensive notation and implementation details before establishing the overall intuition. A clearer architecture diagram, a concise overview of the computation, and shorter paragraphs would substantially improve readability.

[1] Zhang, X., and Sheng, V. S. Token-Context Attention for NLI: An Alternative to Self-Attention. AAAI, 2026.

Questions for authors
  1. What are the clean ANLI accuracies of the pretrained ModernBERT and Llama models with and without SMIC under the same fine-tuning protocol?

  2. Can the perturbation results report break, recovery, and prediction-flip rates separately, and distinguish label-preserving edits from edits that should change the answer?

  3. How do deletion and insertion perform separately relative to random token rankings? What confidence does each model assign to its original prediction before any tokens are restored?

  4. Does SMIC outperform simpler input-reinjection and fixed-window convolutional baselines with comparable parameter counts?

  5. Could the authors provide the optimizer, batch size, hyperparameter search ranges, selected configurations? Also the FLOP estimation procedure, and pretrained plug-in training details would be helpful to understand the implementation. These are not sufficiently specified in the supplied manuscript.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.