acceptodds
Withdrawn from ICLR 2027

A Context Alignment Mechanism and Recurrent Neural Networks For Information Retrieval

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

To overcome the failure of bag-of-words approach in Information Retrieval (IR), recently IR researchers lead toward the use of deep learning models to learn the query and document representations. The relevance score of a document given a query is defined as the distance between the two representations, it can be binary: relevant or not relevant, or on a scale from very relevant to not at all relevant. In this paper, we use the Context-Aligned Recurrent Neural Networks model (CA-RNN) proposed by Qin et al. (2018) to learn low dimension representations of queries and documents. The CA-RNN model is based on a context alignment mechanism and Recurrent Neural Networks (RNN) model. Given a query, documents are ranked using the relevance score. The evaluation results on two small datasets extracted from TREC Desk4 and Desk 5 show a net outperformance of CA-RNN over traditional models like BM25, LSA, BERT and BiLSTM. The context alignment mechanism used in CA-RNN can improve significantly the representations learning to rank IR systems.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Information Retrieval, Context Alignment Mechanism, Recurrent Neural Networks, Deep Learning.

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 3 reviews, above 0% of the others.
Reviewer_DvQoOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates the application of Context-Aligned Recurrent Neural Networks (CA-RNN), originally proposed by Qin et al. (2018), to information retrieval. The model learns representations of queries and document titles using BiLSTM and a context alignment mechanism that incorporates contextual information from aligned query-document terms. Relevance is computed based on the Manhattan distance between the learned representations, and documents are ranked accordingly. The method is evaluated on two subsets constructed from TREC Disk 4 and Disk 5 and compared against BM25, LSA, BiLSTM, and BERT-base using MAP and nDCG metrics. The reported results show that CA-RNN generally performs favorably against these baselines, including improvements over the BiLSTM variant without context alignment.

Critical strengths

The empirical results are generally positive. In particular, CA-RNN improves over the BiLSTM baseline without context alignment on most reported metrics, providing some empirical evidence that incorporating context alignment can benefit query-document relevance modeling. The method is also evaluated against several conventional retrieval baselines, including BM25, LSA, and BERT-base.

Critical weaknesses
  1. Limited novelty and unclear contribution. The main methodological component, CA-RNN and its context alignment mechanism, was introduced by Qin et al. (2018). The present work primarily applies this existing architecture to information retrieval, without introducing a clear new architectural component, learning objective, or alignment mechanism. Moreover, the original CA-RNN was already evaluated on answer-selection tasks such as TREC-QA and WikiQA, which are closely related ranking settings. Therefore, it is unclear what substantial methodological or scientific contribution is introduced beyond applying an existing model to a different retrieval setting. This is my primary concern.

  2. Limited empirical validation of the proposed application. Although the reported results are generally positive, the evaluation is conducted on two small subsets containing only 8 and 5 queries, respectively. In addition, the comparisons do not include stronger or more recent neural IR approaches, despite several such methods being discussed in the related work. Consequently, the current experiments make it difficult to assess whether applying CA-RNN provides a meaningful advantage over established neural retrieval approaches.

Questions for authors
  1. Could the authors clarify what the main methodological or scientific contribution of this work is beyond applying the existing CA-RNN architecture of Qin et al. (2018) to information retrieval? In particular, are there any substantive modifications to the original CA-RNN formulation that are specific to the IR setting?

  2. Could the authors provide additional evidence demonstrating the effectiveness of CA-RNN in a broader IR setting, for example by evaluating on larger or standard retrieval benchmarks and comparing against stronger neural IR baselines? Such results would help clarify whether the proposed application provides advantages beyond the current experimental setting.

Reviewer_mr8tOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper applies an existing context-aligned recurrent neural network model to information retrieval, using queries and document titles to compute relevance scores. It evaluates the model on two small subsets of TREC disks 4 and 5 and compares it against BM25, LSA, BERT, and BiLSTM.

Critical strengths

I could not identify a substantive strength that supports acceptance in the current version. The paper first needs clearer presentation, accurate references, and evaluation against modern retrieval baselines before I can meaningfully assess its strengths.

Critical weaknesses
  1. The paper has substantial presentation problems and citation errors. Several references contain incorrect author names or inconsistent details, and TwinBERT appears twice with conflicting citation details. These issues raise concerns about the accuracy of attribution and how carefully the authors checked the bibliography. The figures are also extremely small and remain difficult to interpret even at 300% zoom, with some unclear symbols.

  2. The experimental evaluation does not provide sufficient support for the claims. The latest references date to 2020, and the experiments do not compare against recent retrieval baselines. The two datasets contain only eight and five queries, respectively, with the same queries appearing in training and testing. This leaves both generalization to unseen queries and performance relative to current approaches unclear.

Questions for authors

I would appreciate more care in preparing the submission, out of respect for both the authors’ and reviewers’ time. Accurate references, readable figures, and an up-to-date evaluation are basic requirements for a useful discussion of the contribution and how to improve it.

Reviewer_DZAQOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper applies the existing Context-Aligned Recurrent Neural Networks (CA-RNN) architecture to information retrieval. It learns query and document representations using recurrent neural networks and context alignment, and evaluates ranking effectiveness on two small subsets derived from TREC Disks 4 and 5.

Critical strengths

The paper provides a straightforward empirical comparison of an established neural matching model against several traditional retrieval baselines. The reported nDCG improvements suggest that contextual alignment can be useful for query–title matching.

Critical weaknesses
  1. Lack of methodological novelty.

The paper directly adopts the CA-RNN architecture proposed by Qin et al. (2018), without introducing a substantive architectural extension, learning objective, or new alignment mechanism. The original work already evaluated relevance-oriented answer selection, making the independent contribution of the current application limited.

  1. Serious limitations in the evaluation protocol.

Section 7.1 and Table 1 show that the datasets contain only 8 and 5 unique queries, respectively. The training and testing sets are randomly divided by query–document records rather than by queries, so the test queries overlap with those used for training.

Consequently, the evaluation does not establish generalization to unseen queries. The extremely small number of distinct test queries also undermines the reliability of ranking metrics and statistical significance claims.

The evaluation additionally relies on outdated baselines and document titles rather than full document content.

Questions for authors
  1. What is the specific methodological contribution beyond the original CA-RNN architecture?

  2. Why were the datasets split by query–document records rather than by queries?

  3. Can the authors provide query-disjoint evaluation and explain how statistical significance was assessed with only 8 and 5 distinct queries?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.