- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper presents Iris-mini and Iris-pro, two search agents trained at the 35B-A3B and 397B-A17B scales, respectively, using an end-to-end training pipeline for long-horizon information seeking. The proposed pipeline consists of three main components. First, the authors construct challenging multi-hop search tasks from web hyperlink structures by extracting entity graphs, generating reasoning chains, abstracting explicit entity references, and filtering questions based on difficulty and solvability. Second, teacher-generated search trajectories are refined through trajectory-level filtering and turn-level loss masking before supervised fine-tuning. Third, the agents undergo reinforcement learning with live web search, followed by iterative SFT–RL climbing, where successful and efficient trajectories from RL are incorporated into subsequent SFT rounds. The authors evaluate both models on BrowseComp, BrowseComp-ZH, DeepSearchQA, and Humanity's Last Exam. They additionally investigate the impact of inference-time context management, comparing settings without context management against discard-all, retry, and their combination. The proposed systems achieve strong performance across the four benchmarks, with particularly notable improvements on BrowseComp. The results also highlight the substantial influence of context management on the measured capabilities of long-horizon search agents.
Critical strengths
- Strong empirical performance. Iris-mini and Iris-pro achieve competitive results across four challenging search benchmarks, outperforming most comparable open-source agents at their respective model scales.
- Insightful context management analysis. The paper evaluates performance with and without CM, revealing substantial gains (e.g., +17.5 points on BrowseComp for Iris-mini) and highlighting the importance of inference-time strategies.
- Practical iterative training framework. The proposed SFT–RL climbing strategy incorporates successful RL trajectories into subsequent SFT rounds, offering an intuitive approach to progressively improving search capabilities.
Critical weaknesses
- Missing ablations for key components. No controlled experiments isolate the contributions of data synthesis, turn-level filtering, or iterative SFT–RL climbing. In particular, the advantage over standard SFT→RL remains unverified.
- Insufficient reproducibility details. Important information is missing, including training data sizes, teacher model identities, filtering thresholds, climbing iterations, and RL hyperparameters.
- Incomplete inference-cost analysis. CM provides substantial gains, but token consumption, tool-call counts, and compute-matched comparisons are absent. The role and cost of the auxiliary 397B summarizer during evaluation also remain unclear.
Questions for authors
- Effectiveness of iterative climbing. Could the authors report performance across existing SFT–RL climbing rounds or intermediate checkpoints? This would help clarify whether iterative climbing provides consistent improvements over the training process.
- Data quality and verification. How reliable is the proposed dual-criteria verification in ensuring genuinely challenging and unambiguous multi-hop questions? Have the authors conducted any quality assessment of the synthesized data?
- Training transparency. Could the authors provide more details on the training data scale, teacher models, number of climbing iterations, and trajectory selection statistics? These details would help clarify the practical training recipe without requiring additional experiments.
- Inference efficiency and fairness. Could the authors clarify the effective inference budgets across different CM strategies, including token consumption and tool calls?
3 comments
Thank you for your rebuttal. After reading the response carefully, I think my questions remain partially resolved. Therefore, I will maintain my rating.
Dear Reviewer 6Huz:
Thank you very much for your follow-up and for taking the time to consider our responses. The revised manuscript has been submitted. Thank you again for your thoughtful feedback and consideration.
Best regards, The Authors
Thank you very much for your careful review. We greatly value your comments and suggestions.
W1 + Q1
We apologize that we cannot disclose all training details at this stage. However, we can provide a clearer account of the climbing schedule and the role of each stage.
We perform two SFT–RL climbing rounds: SFT → RL → SFT → RL. The first SFT stage builds the foundational capabilities needed for search-agent tasks and contributes the largest improvement in benchmark scores. The following RL stage further develops these capabilities through exploration. In the second SFT stage, we reinforce useful patterns found in successful RL rollouts. The immediate benefit is greater efficiency rather than higher benchmark scores. This improved efficiency then supports exploration of more challenging questions during the second RL stage, leading to further performance gains. Under evaluation without any context management, completing both climbing rounds consistently improves performance by more than 2% on BrowseComp, BrowseComp-ZH, DeepSearchQA, and HLE compared with stopping after the first SFT–RL round.
Q2
This is a particularly important question. We agree that assessing the quality of synthetic data is essential.
Regarding difficulty, when Kimi-K2.6 attempts questions generated by our pipeline, successful attempts require more than 50 tool-call turns on average, while unsuccessful attempts involve around 150 turns. These figures illustrate the substantial search effort involved (note that this does not mean the teacher model used in training is Kimi-K2.6).
Regarding ambiguity, we distinguish between incorrectly constructed questions and questions with multiple valid answers. Dual-criteria verification helps filter out incorrect questions, but it does not guarantee that every retained question has a unique answer. Therefore, before RL training, we generate three rollouts per question using a strong model and retain only questions for which at least two rollouts are judged correct and produce the same answer. This provides an additional check on answer consistency, although it is not a guarantee of uniqueness.
We will release further details about the synthetic data, the complete data construction pipeline, and a subset of the QA pairs and trajectories upon acceptance.
W2 + Q3
We fully understand your point that these details would help readers understand the training procedure without requiring additional experiments. This is a completely reasonable request, and we sincerely apologize that we cannot disclose all of this information at present.
We have clarified the number of climbing rounds above. Details concerning the training data and teacher models remain unavailable for disclosure at this stage, but we remain committed to releasing the complete data construction pipeline and training recipe.
W3 + Q4
Thank you for highlighting the need for precise budget accounting. Although detailed cost and configuration information is not available for every comparison model, we can clarify our own evaluation settings:
The 397B summarization model serves two main purposes: filtering irrelevant webpage content, such as advertisements, and summarizing retrieved pages to reduce the amount of context consumed.
Our evaluation primarily focuses on tool-call turns. We are also willing to release all evaluation trajectories to support a more detailed analysis of token consumption.
These settings describe our own evaluation protocol. Different teams use different resource allocations, API configurations, and inference budgets, and these details are not always fully disclosed. We therefore cannot claim that all comparisons are matched for total token consumption. However, we take responsibility for the accuracy of every result we report for our own models.
We will revise the manuscript to address your suggestions and notify you of the specific changes and their locations as soon as the revision is ready. We welcome any further questions or discussion.