acceptodds
Under review as a conference paper at ICLR 2027

ACARE: An Agent-Context-Aware Reranker for iterative search

Abstract

In agentic search, conventional rerankers score candidate documents against an agent-generated sub-query, without considering the original question or the agent’s reasoning trace. We introduce the Agent-Context-Aware Reranker (ACARe), a 4B decoder-only LLM that reranks documents using this context when available. Across three benchmarks, ACARe improves dataset-averaged nDCG@10 by 0.9–2.0 percentage points over the base reranker, with fewer searches for all five agents, each from a different LLM family. In answer generation across two agents on ClimbMix, ACARe raises accuracy by 8.2–9.1 percentage points and reduces searches relative to no reranking, while matching or exceeding the base reranker’s mean accuracy with higher evidence precision. On BrowseComp-Plus, ACARe helps smaller agents close more of the nDCG@10 gap to larger agents without reranking: 51–74%, compared with 26–46% using the base reranker. We will release the model, training data, and code upon acceptance.

Primary area:
foundation or frontier models, including LLMs
Keywords:
agentic search, reranking, cross-encoder, deep research agents, information retrieval, context-aware retrieval, LLM agents, retrieval-augmented generation

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_xmhfOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces the agent-context-aware re-ranking model for agentic search tasks. Conventional rerankers score candidate documents against an agent-generated sub-query, without considering the original question or the agent’s reasoning trace. To bridge this gap, during reranker training, the authors provide the reranker with the full context from the agent's trace. By integrating more context and the full trace, the reranker is better trained for agentic tasks and returns more relevant documents when the agent receives a sub-query. Experiments on several widely used datasets, such as BRIGHT, validate the effectiveness of this paper.

Critical strengths
  1. Good Research Question: Making the ranking model more consistent with the agent's trace is a timely and practical research question.
  2. Easy to implement: The authors provide a simple and straightforward solution for training a context-aware ranking model.
Critical weaknesses
  1. Novelty: One of the biggest concerns is the novelty. Training a ranking model with more agents' traces seems to be an incremental contribution. And several prior (or maybe contemporary) studies like AgentIR and Agent-R have similar ideas. This makes me more concerned about the positioning and contribution of this work. I suggest the authors provide more explanation of how their approach differs from previous work.

  2. What do the "harness A" and "harness B" mean in this paper? I tried to find a definition or explanation for these two terms, but found nothing. Are these words placeholders? I suggest the author clarify these terms.

Questions for authors

See the weaknesses above.

4 comments

We thank the reviewer for recognizing the importance of making document ranking consistent with a search agent's context. We clarify two points that affect the novelty assessment and the reading of the experiments.

What ACARe receives. The summary describes the full trace, but ACARe uses the reasoning text immediately preceding the current search, not an accumulated trajectory (§3.2, lines 143–150). In both evaluation harnesses, the available trace is capped at 512 tokens (Appendix B, lines 1054–1056). The trace is optional: when an agent does not expose it, ACARe can rerank using the original question and sub-query alone.

Difference from AgentIR and Agentic-R. These works train first-stage retrievers. AgentIR embeds the agent's reasoning with its query; Agentic-R trains a retriever using local passage relevance and final-answer correctness. ACARe instead scores candidate documents after retrieval, independently and with one 4B checkpoint supporting four combinations of sub-query, original question, and available current trace. It therefore works with lexical or dense retrieval and with a search API that returns document text but does not expose its corpus or index. Section 2 already distinguishes first-stage retrieval from this setting (lines 76–79, 91–99). The methods are also complementary: on TRQA with AgentIR as the first stage, ACARe improves mean nDCG@10 for all five agents.

The new controls further separate prompt context from training. The unchanged Qwen3 reranker improves mean nDCG@10 from 69.7 to 72.2 when given our M3 context prompt. With identical M3 input and inference, ACARe improves all five reported means over Qwen3 and reduces agent API cost by 13.1% (BY-adjusted p=0.016); only model weights differ. The context formulation and the trained checkpoint thus make distinct, measurable contributions (expanded Table 3 in our response to Reviewer GZWq).

Harnesses. They are evaluation configurations, not placeholders. Section 4 explicitly defines “Harness A for iterative document retrieval” and “Harness B for end-to-end answer generation” (line 199), then specifies their respective measures (lines 203–205). Harness A's search and final-ranking procedure is explained at lines 244–247; Harness B's answer-generation setup is detailed in Appendix J. We will repeat the descriptive names alongside the labels in the camera-ready version for easier navigation.

Thanks for the explanation.

Let me try to figure out the positioning of this work. Does the author target the re-ranking stage instead of the initial retrieval stage, which uses a ranking model to sort the documents?

Thank you, Reviewer xmhf, for the follow-up.

Yes, that is correct. ACARe targets the second (reranking) stage, not first-stage retrieval. A first-stage retriever (dense, e.g., Agent-IR or Agentic-R, or lexical, e.g., BM25) retrieves candidates from the corpus, and ACARe reorders them so that the most relevant ones are at the top.

This stage is especially important in agentic iterative search. Context cost accumulates across iterations, so only a few candidates are passed to the agent at each step, and the reranker decides which ones.

Two results support this positioning:

  • Complementary to context-aware retrieval (Harness A). On TRQA, ACARe further improves the results of Agent-IR for all 5 agents, whereas its off-the-shelf backbone, Qwen3-Reranker, hurts them for 4 of the 5.
  • Applicable when the corpus is not accessible (Harness B). With live web search or API-based search, a context-aware retriever cannot be deployed, but a context-aware reranker still can since all it needs is access to the candidate documents regardless of where they are coming from.

Thanks for your response.

I think the overall paper is solid while the novelty is somehow limited. I choose to keep my score.

Reviewer_GZWqOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes a pointwise reranker for agentic search loops. Rather than scoring documents against the sub-query alone, the reranker is further conditioned on the original question and, when available, the agent’s reasoning trace. A single model is trained to support whichever of these inputs is available (e.g., proprietary models may not surface reasoning tokens). Evaluation on challenging benchmarks such as BrowseComp-Plus and BRIGHT shows improved retrieval quality over the base reranker with fewer search calls.

Critical strengths
  • Thorough experimental setup across agents from multiple LLM families and benchmarks, with repeated runs and significance testing. It shows that proper re-ranking can improve agentic search quality, while reducing search calls.
  • Informative analysis of the impact of each piece of information the re-ranker can be conditioned on (sub-query only; + original query; + context).
  • Extended DR-Synth to generate more fine-grained training data, which the authors commit to releasing alongside code and model.
Critical weaknesses

W1:

The authors should review the novelty positioning. The paper cites CoSearch and Agentic-R which already condition the ranker/retriever on the original question. AgentIR conditions a retriever on reasoning traces. The claimed “one checkpoint for four context modes” property is also shared by base Qwen3-Reranker-4B, which handles all modes zero-shot via the prompt (Table 19), and the context dropout (which allows for a single checkpoint) is not ablated.

The major difference between ACARe and prior work is that ACARe is a pointwise ranker, rather than a listwise ranker (CoSearch, RubicRanker), or a retriever (Agentic-R, AgentIR), and the empirical analysis across agents and benchmarks. Comparison against a list wise rankers trained in similar a setting is not provided (only Jina-v3, one-shot w/o contex), which does not allow to test the efficiency-effectiveness motivation.

W2:

In section 4, the authors write “Unless noted, Qwen3-Reranker-4B receives only the sub-query in its native prompt”, i.e., in table 2 the authors compare base model without training and M0 prompt, against trained model and M2/M3 prompt.

Therefore, the reported gains relate to two changes, the prompt, and the fine-tuning. The thin experiment separating them (one agent, one benchmark), in appendix at Table 19, suggests that the majority of the gains arise from the prompt. Table 20 further shows that the trained ACARe model does not attain statistically significant gains over untrained Qwen3-Reranker-4B with the final prompt (M3).

The resulting information is still useful (i.e., for agentic search, context-aware re-ranking should be considered), but to solidify the proposed methodology, the authors should report Qwen3-m2/m3 vs acare-m2/acare-m3 on the main table 2.

W3:

The authors evaluate their re-ranker on top of multiple downstream agents. However, the generalization of the training recipe across re-ranker backbones is not studied. Qwen3-Reranker is an “Instruction aware” ranker, which can explain the strong zero-shot M3 prompt performance discussed in W2. The authors could consider applying the recipe to a backbone that is not already a reranker (e.g., Llama-3.2-3B-Instruct).

Questions for authors

ACARe does not condition on previously surfaced documents. The harness filters already-seen documents (“the retriever returns ten unseen documents by default”), but two different documents can still surface the same information (i.e., be redundant, and clutter context). Since ACARe is trained, it could learn to favor documents that add new information relative to those the agent has already seen. Have the authors considered conditioning on previously surfaced documents (e.g., titles or short summaries) to increase document diversity in the agentic’s trajectory?

3 comments

We thank the reviewer for the detailed review and insightful comments. Addressing them has improved the quality of the paper.

W1: Novelty and positioning

ACARe is, to our knowledge, the first pointwise reranker trained for search-agent context. One 4B checkpoint supports all four context modes without retraining the agent, works with search APIs, and complements context-aware retrieval. Our prompt formulation is itself a contribution, validated by Qwen3's gains without fine-tuning. Please see our general response, “Novelty and positioning,” for the full distinctions from AgentIR/Agentic-R, CoSearch, and concurrent RubricRanker.

W2: Context versus fine-tuning

We expanded Table 3 to separate the contributions of context and fine-tuning. Rows b1–b2 evaluate the out-of-the-box Qwen3-Reranker-4B in M0 and M3; rows c1–c2 evaluate one unified ACARe checkpoint in the same modes. Within each inference mode, the architecture, prompt, and reranking procedure are identical; only model weights differ.

Expanded Table 3 (M0/M3): BCP/Nemotron-3 Ultra/BM25; five runs; mean (±std). †: vs. no reranking; ∗: vs. same-mode Qwen3; BY-adjusted p<0.05. Bold: best mean shown.

RerankerModenDCG@10 ↑Recall@10 ↑Searches ↓LLM turns ↓Cost (¢) ↓
(a1) No reranker–66.9 (±1.3)68.6 (±1.0)14.9 (±0.4)16.0 (±0.4)35.0 (±1.8)
(b1) Qwen3-Reranker-4BM069.7 (±1.1)†72.2 (±1.5)†13.1 (±0.3)†14.1 (±0.3)†30.8 (±0.8)†
new (b2) Qwen3-Reranker-4BM372.2 (±0.5)†74.7 (±0.4)†12.1 (±0.3)†13.2 (±0.3)†29.8 (±0.9)†
(c1) ACARe-4BM070.8 (±0.9)†73.4 (±1.3)†12.7 (±0.4)†13.8 (±0.4)†33.0 (±2.1)
(c2) ACARe-4BM372.6 (±1.2)†75.2 (±0.9)†11.6 (±0.3)†12.6 (±0.3)†25.9 (±1.4)†∗

With the same M3 input, unified ACARe-4B (c2) improves all five means over Qwen3-Reranker-4B (b2). The 13.1% reduction in agent API cost (29.8→25.9¢ per query) is statistically significant (BY-adjusted p=0.016). Unified ACARe in M3 achieves the best mean on all five metrics among the configurations shown. Furthermore, Harness B's matched-M3 comparison favors ACARe on all four means, without significance across six runs (Table 5). With identical M0 prompts, ACARe improves both quality means in all twelve retriever–dataset combinations (Table 1); added context on its own cannot explain these gains.

Our context controls show that the prompt construction contributes much of the agentic gain (Tables 3 and 5). Unfortunately, adding Qwen3 with M2/M3 inputs throughout Table 2 exceeds our budget. However, for the camera-ready version, we will select two additional settings with different datasets and agent model families so that, together with the new row in Table 3, these controls cover M2 and M3 context modes for Qwen3, all three datasets, all three first-stage retrieval methods, and three of the five agent families.

The training mode ablation, comparing unified ACARe with four separately trained specialists, appears in review Z2Gs, Part 1, W1/Q1.

W3: Other backbones

With 16,601 instances, we judged the data better spent adapting an established ranking ability than teaching ranking to a general instruction model such as Llama-3.2-3B-Instruct, since instruction awareness alone is insufficient for reranking. For reference, our starting checkpoint, Qwen3-Reranker-4B, was supervised fine-tuned using approximately 7 million labeled examples and 12 million high-quality synthetic relevance pairs (Qwen3 technical report, Table 6), orders of magnitude more data than our context adaptation dataset. Open-weight decision models Kev, Clef, and pplx-decider, reported as strong out-of-the-box rerankers, are better-fit candidates for extending our recipe through independent document scoring. We leave their context-aware adaptation to future work.

Question: Previously surfaced documents

We thank the reviewer for this insightful suggestion. Filtering previously seen document IDs still allows different documents to repeat the same information, especially in a large corpus. Conditioning ACARe on the titles or short summaries of previously surfaced documents could help it favor evidence that adds something new. Training should then account for retrieval history: a relevant document may become redundant once its information is already known. History-conditioned supervision or reinforcement learning could reward the additional evidence a document contributes. We will discuss this promising direction and the tradeoff between relevance, diversity, and added context cost as future work.

AuthorsOpenReview ↗

Comparison results with the concurrent RubricReranker work

RubricReranker-8B(Liu et al., 2026) is a listwise context-aware reranker, arxived on Aug 4.

Setup. 423 questions from four datasets in the CMASS collection over the ClimbMix corpus. The agent is Nemotron-3.5 Lightning (the paper's Harness B); each search retrieves the BM25 top 100, and the reranker returns its top 5 to the agent. RubricRanker-8B's original prompt is used with listwise reranking with window=20, stride=10. Both rerankers see the same candidates and passages, and the trace is capped at its first 1,024 tokens for both. Six runs per reranker.

Table 1: Answer quality and retrieval

Dataset (questions)RerankerAccuracy ↑Recall@all ↑Precision@all ↑Searches ↓
BrowseComp-Plus (80)ACARe-4B (M3)30.6 (±5.3)3.4 (±0.4)14.6 (±1.5)16.7 (±1.0)
BrowseComp-Plus (80)RubricRanker-8B26.5 (±1.8)2.9 (±0.2)13.0 (±0.4)16.4 (±0.9)
TotalRecall-QA (68)ACARe-4B (M3)67.9 (±2.7)7.9 (±0.4)65.7 (±1.0)3.5 (±0.3)
TotalRecall-QA (68)RubricRanker-8B70.6 (±3.1)8.1 (±0.3)66.3 (±1.3)3.6 (±0.2)
DeepSearchQA (71)ACARe-4B (M3)48.8 (±3.6)10.0 (±0.9)42.9 (±1.6)8.8 (±1.5)
DeepSearchQA (71)RubricRanker-8B49.1 (±2.7)11.3 (±0.5)42.8 (±0.8)8.0 (±0.6)
HotpotQA (204)ACARe-4B (M3)76.4 (±1.3)11.1 (±0.2)53.8 (±0.6)2.7 (±0.2)
HotpotQA (204)RubricRanker-8B76.0 (±1.4)11.3 (±0.4)53.7 (±0.7)2.8 (±0.2)
All datasets (423)ACARe-4B (M3)61.7 (±1.1)9.0 (±0.3)46.5 (±0.7)6.5 (±0.3)
All datasets (423)RubricRanker-8B61.2 (±0.9)9.2 (±0.2)46.2 (±0.5)6.4 (±0.1)

No difference in Table 1 is significant after correction. Question-level accuracy difference, ACARe − RubricRanker (per-question accuracy averaged over the six runs; 95% bootstrap CI over questions, 10,000 resamples, stratified by dataset for All; uncorrected): BrowseComp-Plus +4.2 pp [+0.6, +7.7]; TotalRecall-QA −2.7 pp [−6.9, +1.2]; DeepSearchQA −0.2 pp [−4.9, +4.5]; HotpotQA +0.4 pp [−1.6, +2.5]; All datasets +0.5 pp [−1.0, +2.1].

Table 2: Serving cost and latency

Uses Effective token cost (ETC) = non-cached input + α·cached input + β·output tokens, in thousands per question. Agent: α = 0.5, β = 2.5, from Lightning's prices ($0.08 input, $0.04 cached input, $0.20 output per million tokens). Rerankers (served locally): α = 0, since a cached prompt prefix is not recomputed, and β = 1. Reranker ETC is shown before and after scaling by γ; Total ETC = Agent ETC + γ·Reranker ETC. Latency: end-to-end seconds per question, measured in the runs. The agent, Nemotron-3.5 Lightning, is a mixture-of-experts model with about 3B of its 30B parameters active and attention in only 6 of its 52 layers; served at a 262,144-token context, it needs 14.2 GFLOPs per token. ACARe-4B and RubricRanker-8B are dense models with attention in all 36 layers. Served at 8,192 and 40,960 tokens, they need 8.1 and 26.0 GFLOPs per token, giving γ = 0.57 and γ = 1.83.

Dataset (questions)RerankerAgent ETC ↓Reranker ETC ↓γ·Reranker ETC ↓Total ETC ↓Latency (s) ↓
All datasets (423)ACARe-4B (M3), γ = 0.57153 (±26)283 (±8)∗161 (±4)∗314 (±24)∗151.9 (±5.8)∗
All datasets (423)RubricRanker-8B, γ = 1.83139 (±10)512 (±10)939 (±18)1,078 (±25)188.4 (±6.5)

Per reranked search. ACARe reads 106.0K prompt tokens (45.0K not cached) and emits one token per candidate; RubricRanker reads 88.5K (81.2K not cached) and generates 90. Over all datasets, RubricRanker costs 5.8× ACARe's γ·Reranker ETC, 3.4× its total ETC, and 24% more latency.

Latency setup. Both rerankers ran with the same agent server configuration and load (four concurrent sessions per agent server). ACARe's server shared one GPU with two other reranker servers, while each RubricRanker session had a GPU of its own, which favors RubricRanker's latency.

Dear Reviewer GZWq, a brief pointer in case it was missed: on Nov 15 we posted the listwise comparison raised in W1 (“Comparison results with the concurrent RubricReranker work”). On 423 questions with the same agent and candidates, ACARe-4B and RubricRanker-8B do not differ significantly in accuracy (61.7 vs. 61.2), while RubricRanker costs 3.4× the total effective token cost and takes 24% longer. Together with the matched-context control (expanded Table 3) and the context-dropout ablation (response to Z2Gs, Part 1), this covers W1 and the core of W2. We would be grateful to hear whether any concern remains while there is still time to address it.

Reviewer_Z2GsOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

ACARe is a pointwise reranker for agentic search, fine-tuned from Qwen3-Reranker-4B on 16,601 labeled instances. It scores candidate documents against the sub-query, using the original question and reasoning trace when available. Training uses independent context dropout so that one checkpoint supports four input modes. The paper evaluates retrieval quality, search effort, and end-to-end answer generation across several agents and retrieval settings.

Critical strengths
  1. The method addresses a practical mismatch between an agent-generated sub-query and the information needed for the original question. As a second-stage reranker, ACARe can use returned document text without local corpus access or additional trace generation.
  2. The retrieval evaluation covers three benchmarks and five LLM families. In Table 2, nDCG@10, averaged equally across the benchmarks, improves by 0.9–2.0 percentage points over the base reranker, with fewer searches for every agent. These results support applicability across the tested agents.
  3. The paper documents training, input truncation, harness behavior, and statistical procedures in detail. Run-level standard deviations and multiple-testing corrections help interpret the results. Appendix J.2 also provides a useful control using the unchanged base reranker with ACARe’s context inputs.
Critical weaknesses
  1. The contribution of training across context modes remains unclear. Table 1 establishes fine-tuning gains in one-shot retrieval with identical context-free M0 inputs. The main agentic comparisons also give ACARe more context than the base reranker. The control with the same M3 context fields and prompt templates in Appendix J.2 covers only Nemotron-3.5 Lightning on BrowseComp-Plus (BCP); the differences between ACARe and Qwen3-Reranker-4B in answer accuracy, Recall@all, Precision@all, and searches are not significant after correction. The paper does not compare training with M0, training with all available context, and training with independent context dropout under fixed training data and budget.
  2. The ClimbMix transfer experiment does not establish an advantage in answer accuracy over the base reranker. Across six runs on all 57 BrowseComp-Plus$_{\mathrm{CM}}$ questions, both rerankers obtain 71.1% with GPT-5.6 Luna; Lightning improves from 18.7% to 19.3%. Neither accuracy difference is significant, although Luna shows significantly higher Precision@all with ACARe. The significant BCP gain over the sub-query-only base does not settle this transfer question. Moreover, the run-level tests quantify variation on fixed questions, leaving uncertainty across independently sampled questions unmeasured.
  3. The main cost comparisons cover agent API charges and exclude retrieval and reranker serving costs. Table 14 does show a significant end-to-end latency improvement over the base reranker at depth 50, from 90.8 to 84.1 seconds; no reranking takes 71.7 seconds. That study uses only Nemotron-3 Ultra, BCP, BM25, and one A100. For the smaller agents, the paper has not established whether the savings remain after including the additional 4B reranker’s serving cost.
Questions for authors
  1. Can the authors give the base reranker the same context in the main agentic retrieval settings and compare the three training strategies using the same training data and budget? How much of the gain remains in each comparison?
  2. Can the authors provide paired confidence intervals across questions for differences in answer accuracy and evaluate additional independent questions? Which errors does ACARe correct beyond the base reranker, and which improvements only reduce search effort?
  3. For the smaller agents, what are the total serving cost and end-to-end latency under matched hardware, load, and caching conditions, relative to both the base reranker and no reranking?
3 comments
AuthorsOpenReview ↗

Part 1/2: Training controls and serving cost

We thank the reviewer for the careful review and insightful questions. This is Part 1 addressing the per context mode separate trainings and serving cost/latency for small agetns. Part 2 continues with ClimbMix transfer, confidence intervals, and error analysis.

W1/Q1: Contribution of training under matched context

Expanded Table 3 (response to Reviewer GZWq) evaluates unchanged Qwen3 in the same modes as ACARe, adding Nemotron-3 Ultra and a second harness to Appendix J.2's Lightning control. Within each mode, architecture, prompt, and inference procedure are identical; only weights differ.

In M3, ACARe improves all five means and significantly reduces agent cost by 13.1% (29.8→25.9¢; BY-adjusted p=0.016). Fewer searches (11.6 vs. 12.1) also require fewer reranking calls, with no extra context, parameters, or passes. Identical-M0 gains for ACARe vs Qwen3 also hold for both quality means in all twelve one-shot settings (Table 1).

Unfortunately, full Table 2 context controls exceed our budget. However, in addition to the table 3 expansion above, we will add two more diverse settings for Qwen 3 in M2/M3 mode so that the controls span all three datasets, all three first-stage methods, and three of five agent families.

To ablate context dropout, we compare the unified checkpoint with four specialists, each trained and evaluated solely in its own mode, with fixed data and budget, including M0-only and full-context-only training.

Training mode ablation (to be added to the paper): BCP/Nemotron-3 Ultra/BM25; five runs; mean (±std). u1–u4: original ACARe, one checkpoint with context dropout; s1–s4: separate checkpoints trained and evaluated in their named mode. Bold: best displayed mean.

RerankerInference modenDCG@10 ↑Recall@10 ↑Searches ↓LLM turns ↓Cost (¢) ↓
(u1) ACARe-4B (unified)M070.8 (±0.9)73.4 (±1.3)12.7 (±0.4)13.8 (±0.4)33.0 (±2.1)
(u2) ACARe-4B (unified)M171.9 (±0.6)74.9 (±0.6)12.2 (±0.3)13.2 (±0.3)31.9 (±1.3)
(u3) ACARe-4B (unified)M272.1 (±1.6)74.8 (±1.8)12.0 (±0.3)13.0 (±0.3)31.3 (±1.2)
(u4) ACARe-4B (unified)M372.6 (±1.2)75.2 (±0.9)11.6 (±0.3)12.6 (±0.3)25.9 (±1.4)
(s1) ACARe-M0-4BM071.1 (±0.7)73.4 (±1.1)12.9 (±0.3)13.9 (±0.3)31.6 (±2.9)
(s2) ACARe-M1-4BM172.0 (±1.8)74.4 (±1.9)12.2 (±0.3)13.2 (±0.3)28.8 (±1.9)
(s3) ACARe-M2-4BM272.6 (±1.3)74.9 (±1.3)11.7 (±0.4)12.8 (±0.4)26.4 (±2.2)
(s4) ACARe-M3-4BM372.2 (±1.4)74.7 (±1.6)12.1 (±0.2)13.2 (±0.2)27.4 (±1.4)

Unified ACARe stays within 0.5 nDCG@10 points of every corresponding specialist. Unified M3 has the best mean on all five metrics in this ablation. No matched unified–specialist difference is significant after correction. One checkpoint serves all four modes competitively, avoiding four separate models.

W3/Q3: Total serving cost and latency for small agents

Lightning is our only open source small agent with 30B total parameters and about 3B active parameters. We report effective token cost (ETC; Rerank Before You Reason), accounting for cached input, non-cached input, and output tokens. We scale reranker usage by its FLOPs per token relative to Lightning: γ=0.57 at their respective context limits (8K tokens for the reranker and 262K for Lightning). ETC estimates compute cost; end-to-end latency is measured. All arms shared the same agent server, and the two reranking arms shared one GPU under matched load and caching settings.

CMASS, all 423 questions: Lightning; six runs; ETC: thousands of tokens per question; latency: end-to-end per question.

RerankerAgent ETC ↓γ·Reranker ETC ↓Total ETC ↓Latency (s) ↓
No reranker194 (±11)0194 (±11)121.7 (±5.4)
Qwen3-Reranker-4B (M0)172 (±19)179 (±5)351 (±18)156.1 (±4.0)
ACARe-4B (M3)153 (±26)161 (±4)314 (±24)151.9 (±5.8)

Reranking costs more ETC than the agent tokens it saves: compared with no reranking, ACARe raises mean accuracy by 6.1 percentage points (55.6%→61.7%), while total ETC rises by 62% (194K→314K) and latency by 25%. Between the two reranking arms, ACARe has higher mean accuracy and uses 10.6% less total ETC (351K→314K) and 2.7% less time than Qwen3. Caching reuses the question and reasoning trace across candidates; ACARe makes 11% fewer reranking calls, reducing reranking ETC by 10.2%.

Other small agents' undisclosed parameter counts prevent computing γ. Lightning is our cheapest agent: the others' list prices are 2.5–25× its uncached-input rate and 6–60× its output rate (Table 7), suggesting a smaller relative reranking cost, without establishing it. We will add this accounting and caveat to the cost appendix.

AuthorsOpenReview ↗

Part 2/2: ClimbMix transfer and error analysis

W2/Q2: Transfer to ClimbMix on additional independent questions

At submission, the ClimbMix study covered 57 BrowseComp-Plus (BCP) questions. The CMASS collection, expanded on arXiv on October 26, 2026, now provides 423 questions over ClimbMix from four datasets; 368 of them are new to our evaluation (all but 57 of the BCP questions). We re-ran Harness B with Nemotron-3.5 Lightning, six runs per configuration, comparing no reranking, Qwen3-Reranker-4B, and ACARe in M3. We will add GPT-5.6 Luna results to the camera-ready version.

CMASS (Harness B): Nemotron-3.5 Lightning; six runs; mean (±std). †: vs. no reranking; ∗: vs. Qwen3; BY-adjusted p<0.05 (40 tests), improvements only. Bold: best per dataset.

Dataset (questions)RerankerAccuracy ↑Recall@all ↑Precision@all ↑Searches ↓
BrowseComp-Plus (80)No reranker21.7 (±3.8)2.9 (±0.2)9.9 (±0.7)19.7 (±0.7)
BrowseComp-Plus (80)Qwen3-Reranker-4B28.8 (±4.0)3.1 (±0.2)11.2 (±1.1)18.9 (±1.2)
BrowseComp-Plus (80)ACARe-4B (M3)30.6 (±5.3)3.4 (±0.4)14.6 (±1.5)†∗16.7 (±1.0)†∗
TotalRecall-QA (68)No reranker64.5 (±3.5)8.8 (±0.3)62.3 (±1.3)4.3 (±0.4)
TotalRecall-QA (68)Qwen3-Reranker-4B67.9 (±2.2)7.2 (±0.2)57.6 (±0.8)3.6 (±0.2)
TotalRecall-QA (68)ACARe-4B (M3)67.9 (±2.7)7.9 (±0.4)∗65.7 (±1.0)†∗3.5 (±0.3)†
DeepSearchQA (71)No reranker41.5 (±5.0)10.7 (±0.7)36.6 (±1.5)10.6 (±0.5)
DeepSearchQA (71)Qwen3-Reranker-4B44.8 (±4.1)10.0 (±0.5)33.6 (±1.7)10.1 (±2.3)
DeepSearchQA (71)ACARe-4B (M3)48.8 (±3.6)10.0 (±0.9)42.9 (±1.6)†∗8.8 (±1.5)
HotpotQA (204)No reranker70.8 (±1.8)11.4 (±0.3)47.2 (±0.7)3.8 (±0.2)
HotpotQA (204)Qwen3-Reranker-4B76.3 (±1.4)10.3 (±0.3)46.3 (±0.6)3.1 (±0.1)
HotpotQA (204)ACARe-4B (M3)76.4 (±1.3)†11.1 (±0.2)∗53.8 (±0.6)†∗2.7 (±0.2)†∗
All datasets (423)No reranker55.6 (±1.1)9.2 (±0.2)40.8 (±0.6)8.0 (±0.2)
All datasets (423)Qwen3-Reranker-4B60.7 (±0.8)8.4 (±0.2)39.4 (±0.4)7.3 (±0.2)
All datasets (423)ACARe-4B (M3)61.7 (±1.1)†9.0 (±0.3)∗46.5 (±0.7)†∗6.5 (±0.3)†∗

ACARe significantly improves Precision@all over both baselines on every dataset (46.5% vs. 39.4% overall) and Recall@all over Qwen3 overall, with 11% fewer searches than Qwen3 (6.5 vs. 7.3). As in the paper, the accuracy difference from Qwen3 is not significant (question-level CIs below); ACARe's consistent transfer benefit is more precise evidence with less search.

Q2: Question-level confidence intervals and error analysis

For each question, we average correctness over the six runs and compute paired differences with 95% bootstrap CIs (10,000 resamples of questions, stratified by dataset). These assess uncertainty across questions; table markers retain the corrected run-level tests:

  • ACARe vs. no reranking: +6.1 pp [+3.9, +8.5] over all 423 questions; the CI excludes zero on BCP (+9.0), DSQA (+7.3), and HotpotQA (+5.6), but not TRQA (+3.4 [−1.5, +8.6]).
  • ACARe vs. Qwen3: Recall@all (9.0 vs. 8.4), Precision@all (46.5 vs. 39.4), and searches (6.5 vs. 7.3) all improve significantly overall after correction. Accuracy also rises by 1.1 pp [−0.6, +2.7], with dataset differences from 0.0 pp on TRQA to +4.0 pp on DSQA [−1.2, +9.2]; the accuracy difference alone is not significant.

Calling a question solved if it is correct in at least four of six runs and failed if it is correct in at most two, ACARe solves 13 questions that Qwen3 fails and fails 7 that Qwen3 solves.

Dear Reviewer Z2Gs, thank you again for the careful review. In brief, against your three questions: (Q1) a matched-context control in Harness A and a unified-vs-specialist training ablation under fixed data and budget; (Q2) 368 additional independent questions, paired bootstrap confidence intervals across questions, and a solved/failed error analysis; (Q3) total effective token cost and measured latency for the small agent against both the base reranker and no reranking. We have stated plainly where differences are not significant. We would be grateful to know whether these address your concerns, or what remains.

Comments on the paper

2 comments
AuthorsOpenReview ↗

General Response: New Ablations, Novelty, and Evaluation Protocol

New evidence and results

We thank all reviewers for their time and constructive feedback. We add two missing ablations both using BCP/Nemotron-3 Ultra/BM25 with five runs. See Review GZWq Qwen3 eval in M3 mode expanding Table 3, and Review Z2Gs, Part 1, for the training mode ablation, to be added to the paper. Z2Gs also reports ClimbMix transfer on 423 questions, paired confidence intervals, error analysis, and total cost.

  • Fine-tuning lowers cost at matched context. With the global question and trace (M3), ACARe improves all five means over Qwen3 and reduces agent cost by 13.1% (29.8→25.9¢; adjusted p=0.016). Architecture, prompt, and inference are identical; only weights differ. The gain needs no extra context, parameters, or reranking passes.
  • Our context formulation helps without fine-tuning. Qwen3's mean nDCG@10 rises significantly from 69.7 in M0 to 72.2 in M3. We deliberately put agent context in Qwen3's native <query> field, so 16k examples teach context use rather than a new format. The prompt design is itself a contribution: Qwen3's zero-shot gains validate it independently of fine-tuning (Tables 3/5). We will make this explicit in the camera-ready version and respectfully ask that this intended compatibility not be interpreted as a lack of novelty.
  • One shared checkpoint covers four modes. Context dropout avoids training and maintaining a separate model for each context setting. Unified ACARe stays within 0.5 nDCG@10 points of the matching specialist in every mode. In M3, it achieves the best mean on all five metrics in the training ablation. No matched unified–specialist difference is significant after correction. This supports one model across modes, with competitive quality and the best results with full context.

Novelty and positioning

To our knowledge, ACARe is the first pointwise LLM reranker trained for search-agent sub-queries using agent context and, alongside concurrent RubricRanker, the first reranker conditioned on the agent's preceding reasoning trace. One 4B checkpoint supports the question, trace, both, or neither, without retraining the agent; question-only input supports agents with hidden reasoning.

  • AgentIR/Agentic-R are first-stage retrievers requiring corpus access. ACARe works with search APIs without corpus access (Harness B), and complements context-aware retrieval. With AgentIR on TRQA, ACARe improves mean nDCG@10 over no reranking for all five agents; native-prompt Qwen3 lowers it for four. ACARe significantly outperforms Qwen3 for all five (Table 14).
  • CoSearch and RubricRanker are, to our knowledge, arXiv-only; RubricRanker is concurrent. CoSearch appeared on April 19, 2026; RubricRanker on August 4. The ICLR 2027 Reviewer Guide sets July 17 as the contemporaneous peer-reviewed-work cutoff and states that comparisons to arXiv-only work are not required. We therefore respectfully ask that these two papers not be counted against the novelty of this submission. We cite both for completeness.
  • Their inputs, training, and scoring differ. CoSearch jointly trains its agent and a 7B listwise ranker receiving the question and sub-query, without the preceding trace. RubricRanker's 8B listwise selector receives the sub-query and trace, without a separate original-question field. ACARe independently scores documents in one forward pass, without generating ranking sequences. Scoring can run in parallel, avoiding sequential sliding-window passes. CoSearch evaluates Wikipedia QA, not our deep-search benchmarks and we found no released model checkpoint to evaluate directly. We are running a controlled RubricRanker comparison for the camera-ready version and will post it during this discussion if ready.
  • Published evidence motivates pointwise scoring. Without a direct CoSearch/RubricRanker comparison, our design draws on Rerank Before You Reason: pointwise Qwen3-0.6B achieves 52.9% accuracy versus 56.0% for listwise gpt-oss-20b-A4b with the same agent at depth 50. Its reported token counts and relative FLOPs per token (0.32×), accounting for uncached input, cached input, and output, imply 57% lower pipeline effective token cost (ETC) and 85% lower reranking ETC.

Experimental rigor

Every configuration uses five or six runs, with means and standard deviations; significance requires correction across the full test family. Evaluation spans five main agent families, three benchmarks, four first-stage retrievers plus API search, and two harnesses. This combination of repetition, reported variability, and corrected testing remains uncommon, including among the vast majority of related studies we cite. We would be grateful if reviewers considered this rigor when assessing soundness and consistent improvements that do not reach significance.

Area_Chair_auPoOpenReview ↗

Author-Reviewer Discussion

Dear Reviewers,

The authors have provided the rebuttal. Please have a look and let the authors know if you need further clarification or still have follow-up questions.

Thanks, Your AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.