- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces the agent-context-aware re-ranking model for agentic search tasks. Conventional rerankers score candidate documents against an agent-generated sub-query, without considering the original question or the agent’s reasoning trace. To bridge this gap, during reranker training, the authors provide the reranker with the full context from the agent's trace. By integrating more context and the full trace, the reranker is better trained for agentic tasks and returns more relevant documents when the agent receives a sub-query. Experiments on several widely used datasets, such as BRIGHT, validate the effectiveness of this paper.
Critical strengths
- Good Research Question: Making the ranking model more consistent with the agent's trace is a timely and practical research question.
- Easy to implement: The authors provide a simple and straightforward solution for training a context-aware ranking model.
Critical weaknesses
-
Novelty: One of the biggest concerns is the novelty. Training a ranking model with more agents' traces seems to be an incremental contribution. And several prior (or maybe contemporary) studies like AgentIR and Agent-R have similar ideas. This makes me more concerned about the positioning and contribution of this work. I suggest the authors provide more explanation of how their approach differs from previous work.
-
What do the "harness A" and "harness B" mean in this paper? I tried to find a definition or explanation for these two terms, but found nothing. Are these words placeholders? I suggest the author clarify these terms.
Questions for authors
See the weaknesses above.
4 comments
Thanks for the explanation.
Let me try to figure out the positioning of this work. Does the author target the re-ranking stage instead of the initial retrieval stage, which uses a ranking model to sort the documents?
Thank you, Reviewer xmhf, for the follow-up.
Yes, that is correct. ACARe targets the second (reranking) stage, not first-stage retrieval. A first-stage retriever (dense, e.g., Agent-IR or Agentic-R, or lexical, e.g., BM25) retrieves candidates from the corpus, and ACARe reorders them so that the most relevant ones are at the top.
This stage is especially important in agentic iterative search. Context cost accumulates across iterations, so only a few candidates are passed to the agent at each step, and the reranker decides which ones.
Two results support this positioning:
- Complementary to context-aware retrieval (Harness A). On TRQA, ACARe further improves the results of Agent-IR for all 5 agents, whereas its off-the-shelf backbone, Qwen3-Reranker, hurts them for 4 of the 5.
- Applicable when the corpus is not accessible (Harness B). With live web search or API-based search, a context-aware retriever cannot be deployed, but a context-aware reranker still can since all it needs is access to the candidate documents regardless of where they are coming from.
Thanks for your response.
I think the overall paper is solid while the novelty is somehow limited. I choose to keep my score.
We thank the reviewer for recognizing the importance of making document ranking consistent with a search agent's context. We clarify two points that affect the novelty assessment and the reading of the experiments.
What ACARe receives. The summary describes the full trace, but ACARe uses the reasoning text immediately preceding the current search, not an accumulated trajectory (§3.2, lines 143–150). In both evaluation harnesses, the available trace is capped at 512 tokens (Appendix B, lines 1054–1056). The trace is optional: when an agent does not expose it, ACARe can rerank using the original question and sub-query alone.
Difference from AgentIR and Agentic-R. These works train first-stage retrievers. AgentIR embeds the agent's reasoning with its query; Agentic-R trains a retriever using local passage relevance and final-answer correctness. ACARe instead scores candidate documents after retrieval, independently and with one 4B checkpoint supporting four combinations of sub-query, original question, and available current trace. It therefore works with lexical or dense retrieval and with a search API that returns document text but does not expose its corpus or index. Section 2 already distinguishes first-stage retrieval from this setting (lines 76–79, 91–99). The methods are also complementary: on TRQA with AgentIR as the first stage, ACARe improves mean nDCG@10 for all five agents.
The new controls further separate prompt context from training. The unchanged Qwen3 reranker improves mean nDCG@10 from 69.7 to 72.2 when given our M3 context prompt. With identical M3 input and inference, ACARe improves all five reported means over Qwen3 and reduces agent API cost by 13.1% (BY-adjusted p=0.016); only model weights differ. The context formulation and the trained checkpoint thus make distinct, measurable contributions (expanded Table 3 in our response to Reviewer GZWq).
Harnesses. They are evaluation configurations, not placeholders. Section 4 explicitly defines “Harness A for iterative document retrieval” and “Harness B for end-to-end answer generation” (line 199), then specifies their respective measures (lines 203–205). Harness A's search and final-ranking procedure is explained at lines 244–247; Harness B's answer-generation setup is detailed in Appendix J. We will repeat the descriptive names alongside the labels in the camera-ready version for easier navigation.