acceptodds
Under review as a conference paper at ICLR 2027

SLMR: Supervised Sequential Long-Context Multi-Step Retriever

Abstract

Multi-step retrieval over long contexts is a key capability for retrieval-augmented generation, but strong existing approaches are costly and complex to train. Supervised retrievers are far simpler and cheaper, yet they fall well behind in this setting. We show that this gap stems not from supervised training itself, but from three misalignments between training and inference: easy training negatives, noisy retrieval prefixes, and position-agnostic scoring. Guided by this analysis, we introduce SLMR, a simple yet effective supervised sequential retriever that models multi-step retrieval as the evolution of an explicit textual state encoded by a single shared Transformer. On long-context reasoning, multi-hop QA, and needle-in-a-haystack benchmarks with contexts from 4K to 10M tokens, SLMR matches or outperforms state-of-the-art methods and far surpasses prior supervised retrievers, while training over 20 faster and remaining stable across a wide range of hyperparameters.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
RAG, Long Context, Multi-Hop Reasoning

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_WkUxOpenReview ↗
Rating
2
Confidence
4
Informativeness
2(+1)
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates whether a purely supervised retriever can match Q-RAG on long-context, multi-step retrieval. The authors attribute the shortcomings of supervised methods such as MDR and Beam Retriever to three factors: negatives that are too easy, later hops being trained on incorrect predicted prefixes, and failure to use a chunk’s position relative to what has already been retrieved. S2LMR addresses these issues with a single shared bi-encoder, a per-hop InfoNCE loss over all chunks from the same context and reference prefixes during training.

Critical strengths

The method is simple and easy to implement. When chunk-level supervision is available, removing the value network, the target network and the TD targets is a real practical simplification.

The analysis of negatives in Section 3.1 is the most interesting part for me. Table 6, Figures 3, 7 show quite convincingly that MDR-style in-batch training ends up separating the wrong distribution.

The evaluation covers both long synthetic tasks and standard open-domain multi-hop QA.

Critical weaknesses
  1. The positional encoding is not new. Q-RAG also recomputes positions after each hop. The real novelty is replacing TD learning with teacher-forced InfoNCE over the whole context.

  2. The training-time and stability comparison is unreliable. Table 4 reports ~54h for Q-RAG on QA3, but the Q-RAG paper says at most 12 hours on one A100, with curves flattening in about two hours. The Q-RAG run in Figure 6a also only gets to 0.876 on QA3, while Figure 4 in this paper shows Q-RAG at about 0.98. So the 24x speedup and the whole of Section 4.3 seem to be based on a Q-RAG run that wasn't trained properly.

  3. The reproduced Q-RAG numbers are lower than the original and change the conclusions. On RULER QA, HotpotQA, and MuSiQue, the original Q-RAG numbers are higher and beat S2LMR at every length.

  4. The method needs fine-grained labeling. It requires gold chunks in a fixed hop order and K at test time. HotpotQA has no natural order, and document order is only a convention. Q-RAG only checks whether all support facts were found, could use an answer-based reward, does not penalize other valid paths, and can use Q-values to stop. None of this works for S2LMR, and the paper should discuss it.

  5. Minor: CCR and label smoothing are used but never defined. Delta is 200 for QA3 and 20 for other tasks, with no explanation. Table 8 has identical numbers to three decimals for all N. Chunk-axis encoding is disabled on RULER, HotpotQA, and MuSiQue, but this is not stated in the main text.

Questions for authors
  1. How does S2LMR behave when K is not known at test time, for example with a larger fixed budget or with a learned stopping rule?
  2. Can S2LMR be used when only answer-level supervision is available, for example exact match or an LLM judge on the final answer? A small experiment, for example on HotpotQA without supporting-fact labels, would help a lot.
  3. How are CCR and label smoothing defined, and how do the results change without them?
  4. Q-RAG reports out-of-distribution results on MuSiQue after training only on HotpotQA. How does S2LMR do in the same out-of-distribution setting?
5 comments
AuthorsOpenReview ↗

Response to Reviewer WkUx

We sincerely thank the reviewer for an exceptionally careful, detailed and helpful review.

W1: Positional encoding.

Yes, we agree with the reviewer's comments on the positional encoding and the teacher-forced InfoNCE. As we wrote in the paper, the chunk-axis encoding, including its per-hop recomputation, is inherited from Q-RAG. S²LMR's only modification is to compute positions from the gold prefix during training.

One important part of our contribution is showing why the key components such as positional encoding and teacher-forced InfoNCE are needed and that they suffice: supervised retrievers fall behind RL not because supervision is too weak, but because their training is misaligned with inference. With these misalignments removed, a plain supervised bi-encoder matches or outperforms Q-RAG from 4K to 10M tokens, without value networks or TD updates.

W2: Training time and stability.

We thank the reviewer for this careful check.

Training time. The 54.4 h in Table 4 is the wall time of the full schedule of Q-RAG's released configuration on one A6000. We will revise the speedup claim to: "On BabiLong QA3, S²LMR reaches its peak performance in 1.4 GPU hours and completes training in 2.25, compared with 3 and 54.4 for Q-RAG."

"0.876 vs. 0.98." The gap comes from the evaluation protocol. Fig. 4 follows the Q-RAG paper: a budget of T=6T=6 retrieval steps, with credit whenever all gold facts are retrieved. Fig. 6a evaluates both methods under the same stricter protocol, with T=K=3T=K=3 steps that must all be correct. We will state both protocols explicitly.

Other parts of Section 4.3. The robustness results (Fig. 5) do not depend on this run: they come from separate learning-rate and batch-size sweeps, and the specific budget (TT) for each run is reported.

W3: Reproduced Q-RAG numbers.

We thank the reviewer for pointing this out! In the revision, we report Q-RAG's original numbers together with results of the authors' released checkpoints. Under matched settings, our conclusions hold on all three benchmarks.

RULER QA. In submission we use the authors' released Q-RAG checkpoint, without retraining.

  • Why the numbers differ. Q-RAG's paper uses T=4T = 4 retrieval steps (Appendix G), while its released evaluation commands use T=1T = 1 (SH) and T=3T = 3 (MH). T=4T=4 reproduces the published SH numbers within 2 points but not MH.
  • Matched comparison. We report both settings. In each, the two methods share the training data, backbone, test data, reader (Qwen3-4B, no thinking) and answer evaluation.

EM at 4K / 16K / 32K / 128K / 1M:

SettingMethodSHMH
Code (T=1/3T=1/3)Q-RAG55/53/52/50/5064/60/63/61/56
S²LMR56/51/50/47/4664/62/63/62/59
Paper (T=4T=4)Q-RAG61/59/59/57/5362/60/58/60/57
S²LMR63/57/57/55/4565/61/63/63/61
PublishedQ-RAG62/59/59/55/5267/64/65/65/61

The two settings agree: on MH, S²LMR matches or outperforms Q-RAG at every length; on SH, it is better at 4K and behind at longer contexts.

HotpotQA and MuSiQue. Our submission retrained Q-RAG with the official repository, whose configuration differs from that of the released checkpoints. We now compare against Q-RAG's original numbers and its released checkpoints.

  • HotpotQA. Both methods retrieve exactly two chunks; S²LMR is better on all four metrics.
MethodSourceFact F1Fact EMAns F1Ans EM
Q-RAGOriginal0.930.890.760.59
S²LMROurs0.950.900.780.62
  • MuSiQue. The two methods use different protocols. Q-RAG retrieves up to 4 steps and stops once all gold facts are found; S²LMR retrieves exactly K=∣gold∣K = \lvert\text{gold}\rvert chunks. We evaluate both under both protocols, using Q-RAG's first ∣gold∣\lvert\text{gold}\rvert chunks for the latter. All runs use the QwQ-32B reader on the full dev set.
ProtocolMethodSourceFact F1Fact EMAns F1Ans EM
Up to 4 steps, stop when all gold foundQ-RAGOriginal0.810.720.590.43
Q-RAGReleased ckpt0.810.720.600.45
S²LMROurs0.860.770.620.47
K=∣gold∣K = \lvert\text{gold}\rvertQ-RAGReleased ckpt0.780.570.550.41
S²LMROurs0.790.670.560.42
AuthorsOpenReview ↗

Response to Reviewer WkUx

S²LMR is better under both protocols. Even with 4 steps and no stop, it reaches 0.77 Fact EM and 0.62/0.47 Ans F1/EM, still above Q-RAG with the stop. We will state all settings and protocols in the paper.

W4: Labeling requirements.

This is a valuable point, and we will add this discussion to the paper. None of these requirements is intrinsic to S²LMR:

  • Known KK. S²LMR can run with a fixed budget, or with a learned stopping rule on its own retrieval scores. The latter plays the same role as Q-RAG's Q-value stopping (see reply for Q1).
  • Fixed hop order. The per-hop loss can take a soft minimum over all orderings of the gold set. Training then only requires the full set to be retrieved, and no ordering of it is penalized. Without any order labels, this matches the fixed-order model in distribution (Fact EM: HotpotQA 0.90 vs. 0.90; MuSiQue 0.69 vs. 0.67) and transfers better across datasets (see reply for Q4).
  • Supporting-fact labels. These can be replaced by pseudo-labels derived from the answer alone (see reply for Q2).

We also note that the results Q-RAG reports are likewise trained with rewards computed from gold supporting facts.

W5: Minor issues.

We will fix all four.

(i) CCR and label smoothing. Both regularize the per-hop InfoNCE:

Lt=CEαLS({score(st,cj)/τ}j∈At,it⋆)−λCCRH(softmaxj∈At(score(st,cj)/τ)),\mathcal{L}_ t = \mathrm{CE}_ {\alpha_ {\mathrm{LS}}}\Big(\{\mathrm{score}(s_ t,c_ j)/\tau\}_ {j\in\mathcal{A}_ t}, i_ t^\star\Big) - \lambda_ {\mathrm{CCR}} H\Big(\mathrm{softmax}_ {j\in\mathcal{A}_ t}\big(\mathrm{score}(s_ t,c_ j)/\tau\big)\Big),

where αLS\alpha_{\mathrm{LS}} is label smoothing over At\mathcal{A}_t and the entropy term is a confidence penalty (Pereyra et al., 2017). Removing either changes Fact EM only marginally (Fig. 8–9).

(ii) Δ\Delta. Performance is stable for Δ∈[100,400]\Delta \in [100, 400] on QA3 (0.96–0.99 Fact EM) and drops at Δ=50\Delta = 50 (0.94–0.95). QA3 needs this finer positional resolution because its gold facts are tightly clustered (Table 7).

(iii) Table 8. The identical values are expected: a position-agnostic encoder scores text only, and both the gold and the strongest distractor are bAbI facts present at every NN, while the added PG-19 filler never outranks them. The failure is therefore semantic, not caused by pool size.

(iv) Chunk-axis PE. We will state in §4.1 that it is disabled on RULER, HotpotQA, and MuSiQue, where chunk order carries no cross-hop signal (shuffled paragraphs or randomly inserted needles).

Q1: Unknown KK at test time.

Thanks for this great question! We study this on MuSiQue, where questions need 2–4 paragraphs and KK is not given. Without changing the trained retriever, we compare three rules:

  1. larger fixed budgets;
  2. a training-free stop that halts when the next pick's score falls below a threshold;
  3. a lightweight learned stop: a logistic regression on S²LMR's own per-hop scores (pick score, top-1/top-2 margin, score drop) that predicts whether the evidence is already complete.

Thresholds are calibrated on held-out data (2-fold on dev). Results on the full dev set:

Retrieval ruleFact EMAvg. #picks
Known K=∣gold∣K = \lvert\text{gold}\rvert0.672.65
Fixed T=2T = 2 / 3 / 40.39 / 0.65 / 0.772 / 3 / 4
Score threshold0.632.60
Learned stop0.652.70

A larger fixed budget is a safe default. With T=4T = 4, evidence recall rises to 0.77 and answer quality also improves (QwQ-32B Ans F1/EM 0.62/0.47, vs. 0.56/0.42 with known KK), as the reader tolerates a few extra paragraphs. When a compact evidence set is preferred, the learned stop recovers most of the known-KK recall (0.65 vs. 0.67) at nearly the same cost (2.70 vs. 2.65 picks), without retraining the encoder.

Q2: Answer-level supervision.

Great question! On HotpotQA, we discard all supporting-fact annotations and derive training targets from the answer alone. An LLM (Qwen3-8B) reads the question, the gold answer and the context paragraphs, and names the two paragraphs needed, in reasoning order; this order is used for teacher forcing. Supporting-fact labels are used only for evaluation. The pseudo-labels are imperfect: both paragraphs are correct for 77% of questions. Even so, with the same training recipe, S²LMR trained on them reaches 0.88 Fact EM on the full dev set, close to 0.90 with gold labels and well above the zero-shot 0.65. Answer-derived labels therefore suffice to train S²LMR.

Q3: Definition of CCR and label smoothing.

See reply for W5.

AuthorsOpenReview ↗

Response to Reviewer WkUx

Q4: Out-of-distribution transfer.

We follow Q-RAG's protocol (up to 4 steps on MuSiQue, stopping once all gold paragraphs are found; 2 steps on HotpotQA). S²LMR matches Q-RAG on HotpotQA → MuSiQue and is ahead on MuSiQue → HotpotQA. With the order-agnostic objective from W4, it is ahead in both directions:

Fact EMHotpotQA → MuSiQueMuSiQue → HotpotQA
Q-RAG0.550.68
S²LMR0.540.76
S²LMR, order-agnostic0.600.77

We will add these results to the revision.

Reviewer_Bod9OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

S²LMR is a supervised multi-hop retriever combining same-context negatives, gold-prefix supervision, and history-adaptive positional encoding. Experiments on BabiLong, RULER, HotpotQA, and MuSiQue show performance broadly comparable to Q-RAG, with approximately 24× lower training cost on BabiLong QA3. Ablations and stability analyses support the design, although gains on natural-language QA are mixed.

Critical strengths
  • Simple and efficient: achieves performance comparable to Q-RAG with approximately 24× lower training cost on BabiLong QA3.
  • Targeted ablations and stability analyses provide useful evidence for the design choices.
Critical weaknesses
  • Generalization evidence is limited: gains on natural-language QA are mixed, and history-adaptive positioning is only enabled on BabiLong.
  • The claimed 3.1% retrieval ceiling is unsupported: multiplying marginal per-hop success rates does not establish a strict upper bound.
Questions for authors
  1. Why is history-adaptive positional encoding disabled outside BabiLong, and what evidence supports its generalization to real-world long-context multi-hop retrieval?

  2. How do you justify the claimed 3.1% strict upper bound, given that multiplying marginal per-hop success rates under oracle prefixes does not generally bound joint retrieval success?

3 comments
AuthorsOpenReview ↗

Response to Reviewer Bod9

We thank the reviewer for the careful and valuable review, which greatly helps to polish our paper.

W1: Generalization.

The core objective generalizes without PE. Same-context negatives and teacher forcing alone give the gains on natural-language QA:

  • HotpotQA: above Q-RAG on all four metrics, including its published numbers.
  • MuSiQue: In the submission's table, S²LMR retrieves exactly K=∣gold∣K = \lvert\text{gold}\rvert chunks, whereas Q-RAG's numbers come from its own protocol, which stops once all gold facts are found. Under matched protocols, S²LMR is above Q-RAG's released checkpoint (Fact EM 0.67 vs. 0.57 at K=∣gold∣K = \lvert\text{gold}\rvert; 0.77 vs. 0.72 with stop-when-found).
  • RULER QA: equal or better on MH, and better at 4K and behind at longer contexts on SH, against Q-RAG's released checkpoint.

PE is enabled only on BabiLong because it is the only benchmark whose evidence has positional structure (see reply for Q1).

W2: 3.1% ceiling.

We thank the reviewer for catching this error. Per-hop successes are positively correlated (e.g., hop-3 success rises from 20% to 55% given correct hops 1–2), so the product of marginals is not an upper bound. We will remove the bound and report measured quantities instead. Under oracle prefixes, the position-agnostic encoder retrieves all three facts in only 8.5% of examples, versus 98–99% with chunk-axis PE. End-to-end Fact EM is 0.11 versus 0.99 (Table 12). The conclusion is unchanged: position-agnostic scoring is a primary bottleneck.

Q1: Why is PE disabled outside BabiLong?

Following Q-RAG, PE is enabled only where evidence positions are related. Q-RAG likewise applies its chunk-axis encoding only on BabiLong. The encoding tells the model which gap between retrieved evidence a candidate lies in and how far along it, so it helps whenever this position is informative about the next hop, whether the next fact is near or far.

  • BabiLong: in 69–79% of cases the next gold fact lies within 10% of the context from the previous one. PE raises Fact EM from 0.11 to 0.99, even though the remaining 21–31% of next hops lie farther away.
  • HotpotQA, MuSiQue and RULER: evidence comes from independent documents, so positions carry no cross-hop signal. PE helps little there, so we disable it to avoid unnecessary complexity.

Real-world applicability. PE helps whenever context order follows temporal order, as in narratives, logs and dialogue histories, and can be switched off otherwise. We will state the per-benchmark PE setting in §4.1.

Q2: 3.1% ceiling.

See reply for W2.

Reviewer_Bod9OpenReview ↗

Reply to Authors

I appreciate the authors' rebuttal, which generally resolves my questions. Since I am not an expert in this domain, I will maintain my current score 3-Weak acceptance.

We thank the reviewer for the further response! We appreciate it!

Best, Authors

Reviewer_3zbPOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces S$^2$LMR, a supervised sequential retriever that addresses three training-inference misalignments in prior supervised methods through (i) same-context contrastive learning, (ii) teacher-forced supervision using gold evidence, and (iii) history-adaptive relative positional encoding. The authors report competitive performance against the RL-based Q-RAG and substantial improvements over prior supervised retrievers across long-context retrieval and multi-hop QA benchmarks, while achieving significantly lower training costs and greater robustness to hyperparameter choices.

Critical strengths

Clear writing and intuitive methodology. The paper clearly motivates the problem and explains the proposed method. In particular, teacher-forced supervision using gold evidence is an intuitive approach to improving multi-step retrieval over prior methods such as Beam Retriever.

Efficient training with competitive performance. According to the reported results, S$^2$LMR is substantially cheaper to train than the reproduced Q-RAG baseline (2.25 vs. 54.4 GPU hours on BabiLong QA3) and more robust to learning rates and batch sizes, while remaining competitive across the evaluated context lengths.

Critical weaknesses

Limited justifications for the generality of Issues 1 and 3.

  • Issue 1 (easy training negatives): hard-negative mining is already widely adopted in modern embedding models, including but not limited to Qwen3-Embedding, Gemini-Embedding, and NV-Embed. Although the authors demonstrate the advantages of same-context negatives over MDR-style in-batch negatives, it remains unclear whether the identified limitation persists with stronger contemporary embedding models trained using hard negatives.
  • Issue 3 (position-agnostic scoring): the motivation appears highly dependent on the construction of BabiLong, where supporting facts exhibit strong positional correlation. In more general long-context QA settings, document order does not necessarily correspond to temporal order, and relevant evidence may be distributed across distant chunks. It remains unclear to me whether the reported positional correlation is representative of broader retrieval tasks these days.

Limited evaluation and unexplained discrepancies in Q-RAG results.

  • Limited evaluation with contemporary embedding models. The experiments use Contriever and multilingual-e5-large, without evaluating stronger modern embedding backbones such as Qwen3-Embedding.
  • Discrepancies in reproduced Q-RAG results. The Q-RAG results in Table 2 are consistently lower than those originally reported by Sorokin et al. (2026), by 2-7 points on SH QA and 2-5 points on MH QA. For example, the published SH/MH QA scores are 62/67 at 4K and 52/61 at 1M, compared with 55/64 and 50/56 in this paper. Using the originally published results, Q-RAG outperforms S$^2$LMR on both QA subtasks at every evaluated context length. Since Q-RAG is the main baseline supporting the paper’s performance claims, these reproduction discrepancies require explanation.
  • Limited long-context comparison on RULER. Table 2 includes relatively few baselines beyond 16K tokens, with only Q-RAG compared at 1M. Although Figure 4 provides additional long-context comparisons on BabiLong, these are limited to quite antiquated methods and the RULER results offer limited independent evidence for the claimed long-context competitiveness.
Questions for authors

Why do authors use contriver for BabiLong but e5-large for other benchmarks?

7 comments
AuthorsOpenReview ↗

Response to Reviewer 3zbP

We thank the reviewer for the constructive and specific feedback, which greatly helps to polish our paper.

W1: Issue 1 with modern embedders.

Hard negatives in Qwen3-Embedding, NV-Embed, and Gemini-Embedding are mined offline per query for single-step retrieval. In multi-step retrieval, however, a chunk's role depends on the hop: relevant facts, including later gold facts, act as negatives until their turn. To test whether this limitation persists, we repeated the Table 12 ablation with a Qwen3-Embedding-0.6B backbone, using the same recipe and with chunk-axis PE in both arms. Same-context negatives reach 0.97 Fact EM on QA3, versus 0.36 with in-batch negatives (Contriever in Table 12: 0.99 vs. 0.28). The gap persists with a hard-negative-pretrained backbone, which indicates that the issue lies in the training signal rather than in backbone strength. We will add this result to appendix.

QA3 Fact EMSame-context neg.In-batch neg.
Contriever (Table 12)0.990.28
Qwen3-Embedding-0.6B0.970.36

W2: Issue 3.

Great question! We agree that the positional signal is task-dependent.

  • Distant evidence. Chunk-axis PE encodes which gap between retrieved evidence a candidate lies in and how far along it (Eq. 2), not its distance to the previous hop, so it does not assume locality. On BabiLong QA3, 21–31% of next hops lie more than 10% of the context away, yet Fact EM reaches 0.99.
  • Unordered contexts. On HotpotQA and MuSiQue, gold positions are at chance level. Enabling PE on MuSiQue neither helps nor hurts (Fact EM 0.67 vs. 0.67).

We will scope Issue 3 to ordered contexts, such as narratives and logs.

W3: Contemporary embedding backbones.

We use the same backbones as Q-RAG for comparability. To test whether S²LMR transfers to modern embedders, we re-trained it with Qwen3-Embedding-0.6B (last-token pooling, task instruction), using each benchmark's original training budget and keeping all other hyperparameters fixed:

Fact EMPaper backbone: zero-shotPaper backbone: S²LMRQwen3-Emb-0.6B: zero-shotQwen3-Emb-0.6B: S²LMR
BabiLong QA30.00 (Contriever)0.990.000.97
HotpotQA0.65 (e5-large)0.900.540.90

S²LMR transfers to the modern backbone without adaptation. A stronger off-the-shelf embedder alone does not help: zero-shot Qwen3-Embedding is no better than the paper backbones. With in-batch instead of same-context negatives, the Qwen3 backbone reaches only 0.36 on QA3, versus 0.97 with S²LMR. S²LMR's gains therefore come from its training objective rather than from the choice of encoder. We will add these results to the appendix.

W4: Q-RAG reproduction on RULER QA.

We thank the reviewer for spotting this discrepancy. All Q-RAG results use the authors' released checkpoint without retraining. The discrepancy comes from the evaluation setting. Q-RAG's paper (App. G) uses T=4T=4 retrieval steps, whereas its released evaluation commands use T=1T=1 (SH) and T=3T=3 (MH); our submission followed the code. T=4T=4 reproduces the published SH numbers within 2 points but not MH. Under both settings, the two methods are evaluated identically, with the same training data, backbone, evaluation script and Qwen3-4B reader.

EM at 4K / 16K / 32K / 128K / 1M:

SettingMethodSHMH
Code (T=1/3T=1/3)Q-RAG55/53/52/50/5064/60/63/61/56
S²LMR56/51/50/47/4664/62/63/62/59
Paper (T=4T=4)Q-RAG61/59/59/57/5362/60/58/60/57
S²LMR63/57/57/55/4565/61/63/63/61
PublishedQ-RAG62/59/59/55/5267/64/65/65/61

Under either setting, S²LMR matches or exceeds Q-RAG on MH at every length. On SH, it is better at 4K and behind at longer contexts. We will report both settings in the paper.

AuthorsOpenReview ↗

Response to Reviewer 3zbP

W5: Limited long-context comparison on RULER.

Why few baselines beyond 32K. Following Q-RAG, the non-retrieval rows in Table 2 report published numbers, and none of these methods can be run at 1M:

  • Titans and Atlas report results only up to 16K, and no official implementation is available.
  • LongRoPE2-8B targets a 128K context window, and its extended weights are not publicly released.
  • Beam-Retriever rescores every chunk with a cross-encoder at each hop, which takes ~20 min per 1M-token sample (Table 4).

New baseline. We evaluated Mamba2-Hybrid-8B ourselves on the same RULER data, using NVIDIA's released 128K-context checkpoint:

Method128K: NIAH avg128K: SH / MH1M: NIAH avg1M: SH / MH
Mamba2-Hybrid-8B-128K80.836 / 260.0†–
Q-RAG10050 / 6199.750 / 56
S²LMR10047 / 6299.946 / 59

† At 1M the model continues the haystack text instead of answering.

Mamba2-Hybrid already degrades at 128K and fails at 1M, whereas both retrieval methods remain near-perfect on NIAH and well ahead on QA. We will add these rows to the appendix.

Q1: Choice of backbone.

For each benchmark we use the same backbone as Q-RAG, our main baseline. This makes our results directly comparable to prior work and easy to reproduce, and differences reflect the training method rather than the encoder.

AuthorsOpenReview ↗

Follow-up on Our Responses

Dear Reviewer 3zbP,

Thank you so much for your time and efforts! As the discussion period ends soon, we would appreciate it if you could review our responses and additional experimental results. We’re happy to clarify anything further.

Best regards, The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.