acceptodds
Withdrawn from ICLR 2027

: Input-Conditioned Complex Convolution for Natural Language Inference

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

We propose Input-Conditioned Complex Convolution (IC^3), an attention-free method for natural language inference. Building on a complex-valued representation that separates token and contextual information, IC^3 generates both a value sequence and a convolution kernel from each premise-hypothesis pair. The kernel determines how the values are combined across the sequence, and the real part of the complex convolution is used to update the token representations. Zero-padded Fourier transforms compute this convolution efficiently. Our analysis characterizes the resulting input-dependent operators, establishes limits on their approximation using fixed operators, and separates the effects of changes in the values and the kernel. With all models trained from scratch under a deterministic training protocol over ten random seeds, IC^3 achieves higher mean accuracy than baseline decoder architectures with comparable parameter counts on ANLI and FOLIO, with consistent gains on SNLI using reduced training data. Controlled operator comparisons and interventions further support the contributions of input conditioning and the imaginary component to predictive performance.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Natural Language Inference, Attention Free Sequence Modeling

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_G4DaOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies the interaction between token and contextual information in natural language inference and proposes IC³ (Input-Conditioned Complex Convolution), an attention-free method. Building on existing complex-valued token–context representations, the method generates a complex value sequence and a dynamic convolution kernel from the input, performs convolutional propagation using zero-padded Fourier transforms, and uses the real part of the convolution output to update token representations. The theoretical analysis characterizes the structure and affine dimension of the conditional convolution operators, their approximation error using fixed operators, and the effects of changes in the values and operators on the output. Experiments on ANLI, FOLIO, and a reduced-data SNLI setting compare models of similar size trained from scratch and show that IC³ achieves higher mean accuracy than the selected baselines. Further operator comparisons, ablations, and response interventions indicate that input-conditioned propagation and the imaginary response contribute to prediction.

Critical strengths
  1. IC³ achieves higher mean accuracy than the selected baseline models of comparable size across five evaluation settings on ANLI, FOLIO, and reduced-data SNLI. The authors assess the stability of the results using ten random seeds and paired confidence intervals. Comparisons with alternative propagation operators and component ablations further demonstrate the contributions of input-conditioned propagation, complex-valued representations, and the imaginary response to model performance.
  2. The paper provides a detailed mathematical analysis of the conditional propagation operators in IC³, including their structure, affine dimension, fixed-operator approximation error, and the decomposition of output changes into value and operator variation. Response-spectrum analysis, low-dimensional projections, and interventions such as response swapping further examine how the input-conditioned operators vary and how this variation affects prediction.
Critical weaknesses
  1. IC³ directly inherits the Complex–Vector Token Representations proposed by Zhang and Sheng [1] and replaces the original Token–Context Attention with input-conditioned complex convolution. However, the only direct comparison with Token–Context Attention appears in the ANLI-R2 ablation in Table 4. In contrast, Table 3 compares other propagation operators on ANLI-R1, R2, and R3. Given that the main contribution is a new propagation mechanism, the lack of a systematic comparison with the most directly related prior method across multiple evaluation settings leaves insufficient evidence that the new operator offers a consistent performance advantage over the original method.
  2. Although the authors analyze the theoretical properties and internal mechanisms of IC³ in considerable detail, the experiments primarily focus on classification accuracy and the effects of the operators on prediction. In contrast, the directly preceding Token–Context Attention work also evaluates parameter efficiency, computational cost, adversarial robustness, and cross-dataset transfer. This paper does not establish whether these properties are preserved or improved after replacing the propagation operator. In particular, given the use of FFT-based convolution and the emphasis on computational efficiency, the absence of direct comparisons of runtime, FLOPs, and GPU memory usage leaves the tradeoff between performance and computational cost relative to the original method unclear.
Questions for authors
  1. Could the authors explain why the direct comparison with Token–Context Attention is limited to ANLI-R2 and provide further evidence that the performance advantage of IC³ extends to the other main evaluation settings? Please also clarify whether architectural factors other than the propagation operator were strictly controlled in this comparison.
  2. The paper implements input-conditioned complex convolution using FFTs and analyzes the theoretical complexity of the propagation stage, but does not compare actual computational costs with the original Token–Context Attention method. Could the authors report runtime, FLOPs, and GPU memory usage to help assess the tradeoff between performance and computational cost?
  3. The experiments in Section 5.1 show that replacing the dynamic response in Layer 0 with the mean response reduces accuracy by approximately 9.58 percentage points, whereas analogous interventions applied individually to Layers 1–11 produce absolute mean accuracy changes of no more than 0.06 percentage points. Meanwhile, Figure 2(a) shows substantial structure in response variation in later layers. How do the authors explain this pronounced difference across layers? Does it suggest that the predictive contribution of input conditioning is concentrated in the first layer? Do the existing experiments adequately support the need for dynamic input-conditioned responses in all 12 layers?
Reviewer_J28hOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper presents Input-Conditioned Complex Convolution (IC3) an architecture specifically targeting Natural Language Inference (Entailment / Contradiction / Neutral classification).

Critical strengths

here), specific architectures (like IC3) can provide positive inductive biases leading to better accuracy.

Critical weaknesses

The paper does not compare against fine-tuned methods at all (at least not explicitly stated) - these will be quite a lot higher. The ANLI leaderboard from 5 years ago reports higher numbers than what is shown here: https://github.com/facebookresearch/anli/ The theorems are not relevant to solve the task or provide any insight.

Questions for authors

Why don't you compare against a fine-tuned baseline?

Reviewer_zWB7OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

The paper proposes Input-Conditioned Complex Convolution (IC3), an attention-free architecture for natural language inference. It separates token and contextual information in complex-valued representations, generates an input-dependent value sequence and convolution kernel, and uses the real component of their convolution to update token representations. Fourier transforms implement the convolution. The paper analyzes the resulting operator family and evaluates models trained from scratch over ten seeds on ANLI, FOLIO, and a reduced-data SNLI setting. It reports gains over the evaluated baselines and uses response projection, response swapping, and imaginary-component interventions to investigate the learned mechanism.

Critical strengths

None

Critical weaknesses
  1. The practical value of the method is not established. The efficiency argument relies on theoretical complexity without measured latency or GPU memory comparisons under matched conditions. Comparisons with pretrained Transformer-based NLI models are also needed to establish the accuracy-efficiency trade-off. Jev System One is another comparison worth examining to clarify IC3's advantages in prediction quality, processing cost, and local deployment. Without such evidence or a clear large-scale application, I cannot determine which practical problem IC3 solves better than available approaches.

  2. The empirical comparisons and mechanism analysis leave important questions unresolved. Two encoder baselines both obtain 33.30% accuracy, approximately the uniform-guessing level for three classes, with identical reported confidence intervals for their accuracy differences from IC3. This raises concerns about whether the baselines were sufficiently trained. Robustness to the selected SNLI subset and meaning-preserving position shifts is unclear. Response swapping may be affected by length differences, and removing an imaginary component that directly contributes to the real update does not by itself explain the mechanism's value. The meaning of "Original" in Figure 2 is also unclear, making the analysis difficult to follow.

Questions for authors
  1. Beyond the theoretical complexity in lines 219-220, does IC3 offer a better trade-off among accuracy, latency, and GPU memory usage than pretrained Transformer-based NLI models on the same hardware and input lengths? Could you report measured running speed and resource consumption, and identify which large-scale text-pair judgment scenarios would actually benefit from this classifier?

  2. How does IC3 compare with Jev System One, which also targets fast, structured judgment tasks, in prediction quality, processing cost, and local deployment? A comparison on the same NLI test sets would help clarify IC3's practical value. Please report API call latency separately from local inference time.

  3. In Table 2 (lines 364-368), why do DeBERTa-v2-xlarge and RoBERTa-large both obtain 33.30% accuracy, with identical reported confidence intervals for their accuracy differences from IC3? These accuracies are close to uniform random guessing for three classes. Could you provide training curves and evidence of convergence to show whether these baselines were sufficiently trained?

  4. How was the 5% SNLI training subset in lines 353-355 selected, and do the gains persist with other sampled subsets? For the improvements discussed in lines 357-362, which phenomena, such as negation, quantitative relations, or lexical relations, account for the gains? Could you report accuracy by phenomenon in a table and give successful and failed examples?

  5. Given the relationship between input positions and propagation distances in lines 135-138, how sensitive are predictions to adding a meaning-preserving word at the beginning of a sentence, which shifts the positions of the following words?

  6. What does "Original" denote in Figure 2 and the discussion in lines 420-424?

  7. What is the purpose of response swapping in lines 428-431, and what does it establish? Does the accuracy drop persist when the examples involved have both the same label and similar sentence lengths, so that length differences can be ruled out as an explanation?

  8. Given that the imaginary response already contributes directly to the real-valued update through complex multiplication, with i2=−1i^2 = -1, what additional conclusion follows from removing it or reversing its sign in lines 432-438? How does this intervention establish more than dependence on a component already used in the computation?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.