acceptodds
Withdrawn from ICLR 2027

When Does Recursive Access Help? Context-Native Recursive Language Models for Medical Evidence Tasks

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Medical evidence tasks often require locating a small amount of decision-relevant information within a collection too large for a single prompt, where the needed evidence rarely shares surface form with the query. Recursive Language Models (RLMs) address this by treating the evidence collection as a persistent, programmatically addressable environment that a controller can search and revisit across many steps, rather than compressing it into one prompt or relying on one-shot retrieval. We evaluate this access pattern on MIRAGE medical question answering (6,574 questions, four task types) and R2MED reasoning-driven retrieval (876 queries, 357,080 corpus documents, three task families), where RLMs are competitive with prior published methods and current leaderboard results on both medical benchmarks, and outperform standard retrieval-augmented generation and agentic search baselines under matched backbones and compute for R2MED. To understand the source of remaining errors in retrieval tasks, we introduce a trace-level diagnostic protocol that separates candidate misses, where relevant evidence is never encountered during search, from ordering misses, where evidence is found but subsequently lost or ranked too low to count, finding that ordering misses are the more common of the two. Motivated by this diagnosis, we show that a lightweight, weight-frozen self-improvement mechanism, which distills recurring trajectory failures into reusable retrieval policies, improves retrieval on the subsets it targets without any change to model parameters. Together, these results characterize not only how well recursive access performs on medical evidence tasks, but specifically where it succeeds and fails, and why, thus identifying evidence retention, rather than discovery, as the central bottleneck for future systems.

Primary area:
datasets and benchmarks
Keywords:
Recursive Language Models, Retrieval-Augmented Generation, Medical Question Answering, Evidence Retrieval, LLM Agents, Information Retrieval, Clinical NLP

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_Nye9OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Sorry for the mismatched review. I apologize for mistakenly submitting an incorrect review. I have now updated it with the correct version.

This paper investigates Recursive Language Models (RLMs) for medical question answering and evidence retrieval, evaluating their performance on MIRAGE and R2MED. Beyond benchmark comparisons, the authors introduce a trajectory-level diagnostic protocol that distinguishes candidate misses from ordering misses, and explore a weight-frozen, trace-guided self-improvement mechanism.

The results demonstrate competitive performance on medical benchmarks, substantial improvements over the selected controlled baselines, and potentially useful observations about evidence handling in recursive retrieval systems.

Critical strengths

Meaningful empirical evaluation. The paper evaluates two medical benchmark families and includes controlled comparisons across two model backbones and multiple recursion depths.

Useful trajectory-level diagnostics. Distinguishing candidate discovery, evidence exposure, and final ranking provides a more informative perspective than aggregate retrieval metrics alone.

Interesting non-monotonic recursion results. The degradation from depth 1 to depth 2 suggests that additional delegation can introduce substantial coordination or aggregation difficulties.

Critical weaknesses

W1. The claimed causal benefit of recursive access is not sufficiently isolated.

The paper attributes the improvements over RAG and Vanilla Pi to the RLM access pattern. However, the RLM configuration simultaneously introduces persistent execution state, programmatic corpus operations, candidate aggregation, recursive delegation, and retrieval-specific instructions.

Importantly, depth-0 RLM already substantially outperforms the controlled baselines. Although depth-1 further improves performance, the current experiments do not isolate whether this additional gain arises from recursion itself, additional semantic evaluation, or a more effective evidence-processing workflow.

A stronger controlled baseline should preserve the same tools, persistent candidate state, semantic subcalls, and actual inference budget while removing only recursive delegation.

W2. The central conclusion that retention, rather than discovery, is the primary bottleneck is not adequately established.

The authors emphasize that ordering misses exceed candidate misses for RLM (17.6% versus 14.0%). However, the comparison against RAG reveals that candidate misses decrease from 42.9% to 14.0%, whereas ordering misses decrease only from 19.7% to 17.6%.

Thus, the dominant improvement introduced by RLM appears to be candidate discovery, even though ordering misses constitute a slightly larger fraction of its remaining failures.

These findings are not necessarily contradictory, but they support different conclusions. The paper should distinguish the largest residual error category from the most consequential or actionable performance bottleneck.

The relative frequency of failure categories alone does not establish which intervention would produce the greatest improvement.

W3. Ordering misses are not equivalent to evidence-retention failures.

The proposed outcome classification identifies ordering misses based on the presence of relevant documents in the final top-100 but not the top-10.

However, this outcome can arise from inaccurate relevance estimation even when the evidence is correctly preserved. The exposure analysis further complicates the interpretation: exact-ID exposure reaches 72.7% for ordering misses, but semantic-judgment exposure is only 29.9%.

Consequently, the evidence does not establish that most ordering misses involve information that was meaningfully inspected and subsequently lost.

A stage-wise analysis separating document discovery, content inspection, relevance assessment, state preservation, and final ranking would make the diagnostic claims substantially more convincing.

Questions for authors
  1. Does trace-guided self-improvement outperform a matched generic retrieval recipe that does not use failure trajectories?

  2. How do the methods compare in actual token consumption, model calls, latency, and invalid-execution rates?

Reviewer_wxAoOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates Recursive Language Models (RLMs) for medical question answering and evidence retrieval, focusing on persistent and programmatic access to external evidence collections. The authors evaluate RLMs on four MIRAGE datasets (6,574 questions) and eight R2MED subsets (876 queries), including comparisons against published systems and controlled RAG and agentic baselines. The results demonstrate competitive performance on both benchmarks. Furthermore, the paper introduces a trace-level diagnostic protocol to distinguish candidate misses from ordering misses and analyzes recurring retrieval failure mechanisms. Based on these observations, the authors evaluate a weight-frozen, trace-guided self-improvement approach that improves retrieval performance on held-out queries within three R2MED subsets.

Critical strengths
  1. Broad empirical evaluation with meaningful performance improvements.

The paper evaluates RLMs across diverse medical QA and retrieval tasks, including controlled comparisons on R2MED using two backbone models and different recursion depths. The substantial improvements over the implemented RAG and agentic baselines, particularly on clinically inferential retrieval tasks, provide useful empirical evidence for the potential of persistent, programmatic evidence access.

  1. Informative trace-level analysis of retrieval failures.

Rather than relying solely on aggregate retrieval metrics, the paper examines execution trajectories to characterize candidate coverage and evidence ranking failures. This analysis provides useful insights into the limitations of recursive retrieval, while the held-out evaluation of trace-guided retrieval recipes offers preliminary evidence that these observations can inform practical improvements.

Critical weaknesses
  1. The experimental comparisons do not fully isolate the sources of improvement within the RLM framework.

Although the paper reports substantial improvements, its controlled comparisons against non-RLM methods are limited to BM25-based RAG and Vanilla Pi, without stronger iterative retrieval baselines under matched conditions. The depth-0 ablation provides useful evidence for the benefits of persistent, programmatic corpus access, while the depth-1 results demonstrate additional gains from enabling delegation. However, it remains unclear to what extent these additional gains arise from the recursive delegation structure itself rather than increased semantic evaluation or computational resources. More detailed cost measurements and stronger controlled baselines would help clarify this distinction.

  1. The interpretation and generalizability of the trace-level diagnostic findings remain insufficiently established.

The candidate-miss and ordering-miss categories are defined by the final top-100 and top-10 rankings, which do not necessarily distinguish failures in evidence discovery from those in evidence retention. Although the authors complement this classification with exposure analysis, the connection between intermediate evidence handling and the final failure categories remains insufficiently established. Furthermore, the reported failure mechanisms are characterized using only 50 sampled traces from 87 locally available runs, limiting conclusions about their prevalence across the full benchmark. The trace-guided improvement results are also difficult to attribute specifically to learned failure patterns without a generic retrieval-instruction control.

Questions for authors
  1. Baseline fairness and computational cost.

Could the authors provide comparisons against a stronger iterative retrieval or agentic RAG baseline under matched backbone, corpus, and inference-budget conditions? Additionally, reporting actual token consumption, delegated model calls, and inference latency would help clarify whether the improvements over depth-0 RLM can be attributed to recursive delegation rather than additional semantic computation.

  1. Interpretation of the failure diagnosis.

Given that candidate misses are defined by the final top-100 ranking, relevant documents may have been encountered or evaluated during intermediate steps but subsequently discarded. Although Table 9 reports exact-identifier and semantic-judgment exposure separately, could the authors provide a finer-grained analysis of how relevant documents transition through candidate generation, semantic evaluation, and final ranking? This would help establish whether the observed failures primarily reflect evidence discovery, retention, or ranking.

  1. Effectiveness of trace-guided self-improvement.

Could the authors compare the trace-derived recipes against generic retrieval instructions of comparable length and computational cost to isolate the contribution of trajectory-derived knowledge? Paired confidence intervals over evaluation queries would also help assess the reliability of the reported gains. Additionally, while the paper already provides matched comparisons over completed queries, a supplementary failure-aware evaluation over all held-out queries would provide a more complete assessment of the approach.

Reviewer_GMkeOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

Medical questions often require finding a few relevant passages in a collection too large to fit in a prompt, and the right passage often uses different words from the question. The paper studies Recursive Language Models (RLMs) for this setting. An RLM does not read the collection directly. Instead, a controller model writes Python code to search the collection, inspect documents, and keep candidates in variables across steps. It can also send selected documents to further model calls, called sub-calls, for judgment. The authors evaluate this approach with the open model Qwen3.6-27B, and repeat part of the experiments with Gemma 4 A4B. They use two benchmarks. MIRAGE contains 6,574 medical questions with multiple-choice or yes/no answers, scored by accuracy. R2MED contains 876 queries; for each, the system must rank 100 documents from a collection of up to about 61,000, scored by nDCG@10. (i) On MIRAGE, the RLM reaches 87.1% average accuracy, higher than the published leaderboard entries, which use other models. (ii) On R2MED, it scores 43.08, close to the best leaderboard entry (43.18). With the same model, it outperforms by about 20 points both a RAG baseline based on BM25 and a general coding agent with access to the full collection. One level of sub-calls works best, and two levels work worse. (iii) The authors then analyze the recorded steps of each run. When the RLM fails on a query, the relevant documents are more often found but ranked too low (17.6% of queries) than never found (14.0%). In most of the first cases, the relevant document had appeared during the search. An analysis of 50 runs, by a model and checked by a human, lists recurring errors. Examples are keeping documents that only share words with the query, or losing the judgments of sub-calls before the final ranking. (iv) Finally, the authors turn lessons from runs on 12 to 17 queries per subset into fixed instructions. These instructions improve nDCG@10 on the remaining queries of three low-scoring subsets, without changing the model's weights.

Critical strengths

Without any training, the RLM with an open 27B model reaches 43.08 nDCG@10 on R2MED. This would place it second on the public leaderboard, after an embedding model trained for reasoning-intensive retrieval (43.18). On the clinical evidence queries, its average (45.01) is slightly above the best leaderboard entry for this group of datasets (43.84), although it scores lower on two of the three datasets. The approach is therefore competitive on this benchmark without task-specific training.

The analysis of the logs describes concrete ways in which such systems fail, each illustrated with quoted passages. For example, the controller keeps documents that only share words with the query, loses the judgments of its sub-calls before the final ranking, or cites results of code it never ran. These observations are useful for building systems of this kind, and the paper reports them openly rather than showing only successes.

Critical weaknesses
  • All main results use one model, Qwen3.6-27B. The second model, Gemma 4 A4B, is used only in the R2MED comparison with the controlled baselines. It does not appear on MIRAGE, in the comparison with the leaderboards, in the error analysis, or in the self-improvement experiment. The conclusions about recursive access therefore rest on a single model.
  • On MIRAGE, the RLM is compared only with leaderboard systems that use other models, mostly older ones such as GPT-4-32k and Llama-3-70B (Table 3). The paper does not report how well Qwen3.6-27B does on its own, without any documents, or with standard retrieval. A recent 27B model may already answer these public exam questions well, so the 87.1% cannot be credited to recursive access. MIRAGE also gets no error analysis, so the question-answering part of the paper adds only one table of accuracies.
  • On MIRAGE, the RLM searches only a fixed part of the Textbooks corpus, the same for every question and of unreported size, and the paper gives no reason for this choice. The MedRAG toolkit used with MIRAGE also provides PubMed, StatPearls and Wikipedia, and their combination, MedCorp (Xiong et al., 2024). The compared systems use different sources: PubMed, MedCorp, or Textbooks with StatPearls. The RLM is never tested with these sources, so the comparison also mixes different collections of documents. It also remains unclear whether the result holds with larger collections, where searching is harder.
  • The paper leaves out MMLU-Med, one of the five MIRAGE datasets, because it has the same format as two other datasets and comes from general MMLU. This is not a valid reason to drop it. The claim that the RLM beats the published MIRAGE results is therefore made on a reduced benchmark.
  • On R2MED, the RAG baseline is described only as "BM25 retrieval", without saying how it produces its ranking of 100 documents. The RLM uses the language model to judge and reorder documents, so the fair baseline is BM25 or a dense retriever followed by reordering with the same model. This baseline is missing. Simple published baselines are also stronger than the paper's RAG. On the public leaderboard, BM25 used with an answer first drafted by Qwen2.5-7B (a method known as HyDE) reaches 26.38, and with an answer drafted by o3-mini 41.01. The paper's RAG with Qwen3.6-27B reaches 22.33. The coding agent Vanilla Pi scores barely above RAG with Qwen and below it with Gemma, and its setup is not described. Despite this, the paper concludes that the gain comes from the recursive access pattern itself.
  • The paper says several times that all methods use the same compute budget, but it gives no cost for any method. The only numbers are the RLM's limits: up to 30 controller steps and 100 sub-calls per query, with a median of 10 steps. The best leaderboard system is an 8B embedding model that needs a single pass per query once the documents are indexed. Without tokens or time per query, the claim of equal compute cannot be checked, and the cost of the method is unknown.
  • Each result comes from a single run, and the paper reports no confidence interval or statistical test. Results also jump a lot between settings: with Gemma, Bioinformatics goes from 9.37 at depth 0 to 32.78 at depth 1. Without repeated runs, it is unclear which differences are real. Failed runs were repeated and only the accepted run was scored, but the paper does not say how many.
  • In the self-improvement experiment, the improved runs use a different system, the Prime Agent harness, with "standard retrieval tools". The main RLM has only standard Python libraries and sub-calls. The paper does not say whether the baseline in Table 10 uses the same system and tools. The three subsets were chosen because they scored low, and the improved results come from a new run compared with the old one. Showing that the learned instructions cause the gain requires a new run with the same system but without the instructions. The model used in this experiment is not stated.
  • In 15 of the 50 analyzed runs, the controller's reasoning cites the output of code that was never run. In Run 10, it writes 50 sub-call judgments before any sub-call happens and uses them for its final ranking. In Run 07, it replaces 34 scores of 1/10 with an invented 3. For a system meant to find medical evidence, this is a serious reliability problem. The paper describes it, but does not measure it on the whole benchmark or propose a way to prevent it.
  • The error catalog is produced by the same model that generated the runs, Qwen3.6-27B. A single human analyst then reviewed the labels, rejecting one and downgrading three. The paper does not give the prompt used for this labeling, the analyst's background, or any agreement between independent annotators. The 50 runs come from only 87 runs described as "locally available", out of 876 queries, with no explanation.
  • The method is the Recursive Language Model of Zhang et al. (2025a), and the self-improvement step adapts the Continual Harness of Prime Agent (Karten et al., 2026). The paper applies these existing systems to two medical benchmarks, with mostly standard analysis tools. It proposes no new method and no insight that extends beyond these benchmarks, which falls short of ICLR standards.
  • Several details needed to reproduce the results are missing: how the RAG baseline produces its ranking, and how many documents it retrieves; how Vanilla Pi is configured and how it accesses the corpus; the size of the Textbooks part used for MIRAGE; the prompts used for MIRAGE (Appendix E shows only retrieval prompts); the limits on observation length, generation length and time, which Appendix D mentions without values; how many runs were repeated; the model and tools used with Prime Agent.
  • Figures 1, 3 and 4 are embedded as bitmap images rather than vector graphics.
Questions for authors
  • What accuracy does Qwen3.6-27B reach on MIRAGE without documents, and with standard retrieval?
  • Why does the RLM use only a part of the Textbooks corpus on MIRAGE, how large is this part, and what are the results with PubMed or MedCorp?
  • What are the results on MMLU-Med?
  • How does the RAG baseline produce its ranking on R2MED, and how does the RLM compare with BM25 or a dense retriever followed by reordering with the same model, at the same cost?
  • How many tokens and how much time per query does each method use?
  • How much do the results vary over repeated runs, and how many failed runs were repeated?
  • Does a run with the Prime Agent harness and tools, but without the learned instructions, improve over the baseline in Table 10? Which model is used in this experiment?
  • Why were only 87 runs available for the error analysis, and how much do independent annotators agree on the labels?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.