- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
Sorry for the mismatched review. I apologize for mistakenly submitting an incorrect review. I have now updated it with the correct version.
This paper investigates Recursive Language Models (RLMs) for medical question answering and evidence retrieval, evaluating their performance on MIRAGE and R2MED. Beyond benchmark comparisons, the authors introduce a trajectory-level diagnostic protocol that distinguishes candidate misses from ordering misses, and explore a weight-frozen, trace-guided self-improvement mechanism.
The results demonstrate competitive performance on medical benchmarks, substantial improvements over the selected controlled baselines, and potentially useful observations about evidence handling in recursive retrieval systems.
Critical strengths
Meaningful empirical evaluation. The paper evaluates two medical benchmark families and includes controlled comparisons across two model backbones and multiple recursion depths.
Useful trajectory-level diagnostics. Distinguishing candidate discovery, evidence exposure, and final ranking provides a more informative perspective than aggregate retrieval metrics alone.
Interesting non-monotonic recursion results. The degradation from depth 1 to depth 2 suggests that additional delegation can introduce substantial coordination or aggregation difficulties.
Critical weaknesses
W1. The claimed causal benefit of recursive access is not sufficiently isolated.
The paper attributes the improvements over RAG and Vanilla Pi to the RLM access pattern. However, the RLM configuration simultaneously introduces persistent execution state, programmatic corpus operations, candidate aggregation, recursive delegation, and retrieval-specific instructions.
Importantly, depth-0 RLM already substantially outperforms the controlled baselines. Although depth-1 further improves performance, the current experiments do not isolate whether this additional gain arises from recursion itself, additional semantic evaluation, or a more effective evidence-processing workflow.
A stronger controlled baseline should preserve the same tools, persistent candidate state, semantic subcalls, and actual inference budget while removing only recursive delegation.
W2. The central conclusion that retention, rather than discovery, is the primary bottleneck is not adequately established.
The authors emphasize that ordering misses exceed candidate misses for RLM (17.6% versus 14.0%). However, the comparison against RAG reveals that candidate misses decrease from 42.9% to 14.0%, whereas ordering misses decrease only from 19.7% to 17.6%.
Thus, the dominant improvement introduced by RLM appears to be candidate discovery, even though ordering misses constitute a slightly larger fraction of its remaining failures.
These findings are not necessarily contradictory, but they support different conclusions. The paper should distinguish the largest residual error category from the most consequential or actionable performance bottleneck.
The relative frequency of failure categories alone does not establish which intervention would produce the greatest improvement.
W3. Ordering misses are not equivalent to evidence-retention failures.
The proposed outcome classification identifies ordering misses based on the presence of relevant documents in the final top-100 but not the top-10.
However, this outcome can arise from inaccurate relevance estimation even when the evidence is correctly preserved. The exposure analysis further complicates the interpretation: exact-ID exposure reaches 72.7% for ordering misses, but semantic-judgment exposure is only 29.9%.
Consequently, the evidence does not establish that most ordering misses involve information that was meaningfully inspected and subsequently lost.
A stage-wise analysis separating document discovery, content inspection, relevance assessment, state preservation, and final ranking would make the diagnostic claims substantially more convincing.
Questions for authors
-
Does trace-guided self-improvement outperform a matched generic retrieval recipe that does not use failure trajectories?
-
How do the methods compare in actual token consumption, model calls, latency, and invalid-execution rates?