- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper proposes three tricks to improve interleaved search and visual reasoning, including an evidence-dependent synthesis pipeline to generate training data, reference-guided credit assignment to improve RL training, and a Multimodal Evidence Interface (MEI) that uses a separate model to process retrieved text and images. With these tricks, Neos-9B reports much better performance than several competing baselines.
Critical strengths
- Good performance in Table 1
Critical weaknesses
- Severe clarity issue - The paper uses complicated terminology to describe relatively simple ideas without clearly explaining them. For example, MEI seems to be just another model that processes retrieved content before passing it to the main agent, with the two models trained alternately. Why not just explain it this way? Instead, the paper introduces confusing terms like "goal-conditioned observations" and "agent–environment co-evolution," making the method unnecessarily difficult to follow. Figures 2 and 3 are also unclear and require going back and forth between the figures and main text to understand them, which defeats the purpose of having figures. Please improve the clarity throughout the paper and avoid using unnecessary, confusing, or misleading terminology.
- Unfair experimental setup - Neos use different training data, search pipeline, tools and training compute compared to all the methods in Table 1. I don't see any experimental setup that compare these methods fairly. How do we know the improvements are from the proposed tricks, rather than simply different training data or a better search pipeline? In fact, Table 13 already shows that changing the workflow alone can make a huge difference. The data ablation in Table 2 only compares different data sources within Neos, which does not address this issue. It is already well known that better training data and more sophisticated search pipelines can improve performance. If that is where most of the improvement comes from, then what new insight does this paper actually provide? Without a fair comparison, it is hard to assess the real contribution of the proposed methods.
Questions for authors
- Most of the methods in Table 1 are open-source, e.g. SenseNova-MARS, OpenSearch-VL etc. Why not reevaluate these models on the benchmarks with missing results in table 1?
- Table 1 only show frontier models like GPT and Gemini under Direct Answer and Agentic Workflow settings. What about frontier models like GPT and Claude running in existing deep search agent frameworks? Why are these comparisons missing?
- How many search turns can Neos sustain? How well does it perform on longer tasks that require hundreds or even thousands of steps, instead of only 50 turns?
- The gap between RL-only and SFT->RL in Table 11 seem quite small. Several prior works suggest that RL can generalize better in OOD settings. Are there any evaluations on OOD tasks? Could the RL-only model actually perform better in these settings? Since we are training foundation models, we should care more about generalization rather than just getting better performance on a few benchmarks.
- Why do we need a separate model to process retrieved information? Why cant the main agent do this itself? Please provide an ablation comparing the two setups.
5 comments
Q1: Most of the methods in Table 1 are open-source, e.g. SenseNova-MARS, OpenSearch-VL etc. Why not reevaluate these models on the benchmarks with missing results in table 1?
For multimodal long-horizon search agents with open-source evaluation frameworks and models like SenseNova-MARS and OpenSearch-VL, we provide additional evaluations on benchmarks to make a more comprehensive comparison. Non-bold values indicate previously reported results.
| Model | MMSearch+ | MM-BC | BC-V3 | VisBrowse | BC-VL | VDR | Avg. |
|---|---|---|---|---|---|---|---|
| SenseNova-MARS-32B | 18.3 | 16.5 | 14.8 | 24.9 | 39.4 | 25.8 | 23.3 |
| OpenSearch-VL-30B | 17.4 | 18.3 | 15.5 | 27.8 | 41.1 | 33.5 | 25.6 |
| Neos-9B | 47.6 | 39.5 | 37.0 | 53.3 | 63.2 | 31.4 | 45.3 |
Neos-9B achieves an average score of 45.3 across the six benchmarks, outperforming SenseNova-MARS-32B and OpenSearch-VL-30B by 22.0 and 19.7 points. These substantial gains demonstrate stronger overall multimodal long-horizon search performance despite its smaller model size.
Q2: Table 1 only show frontier models like GPT and Gemini under Direct Answer and Agentic Workflow settings. What about frontier models like GPT and Claude running in existing deep search agent frameworks? Why are these comparisons missing?
We fully agree that directly comparing agent workflows using frontier models provides a more intuitive assessment. In Table 13 of the appendix, we already compare REDSearcher-MM, a representative workflow that returns all retrieved information to the context, with our MEI-based workflow, highlighting the performance differences between these designs. Furthermore, within our limited available resources, we have added comparisons with other workflows featuring distinct designs, as shown below:
| Policy | Workflow | BC-VL | MM-BC | MMSearch+ |
|---|---|---|---|---|
| GPT-5.6 | SenseNova-MARS | 61.3 | 33.8 | 48.9 |
| REDSearcher-MM | 64.2 | 35.5 | 53.5 | |
| OpenSearch-VL | 62.1 | 34.3 | 50.2 | |
| Neos | 66.4 | 59.5 | 65.9 | |
| Qwen3.5-35B-A3B | SenseNova-MARS | 42.6 | 14.8 | 20.9 |
| REDSearcher-MM | 45.4 | 14.5 | 21.6 | |
| OpenSearch-VL | 44.6 | 16.3 | 20.3 | |
| Neos | 51.4 | 29.0 | 28.1 | |
| Qwen3.5-9B | SenseNova-MARS | 38.1 | 8.8 | 18.6 |
| REDSearcher-MM | 38.6 | 9.8 | 19.3 | |
| OpenSearch-VL | 37.1 | 9.5 | 17.7 | |
| Neos | 43.6 | 22.8 | 24.8 |
Holding the policy model fixed, Neos consistently outperforms all three alternative workflows across all benchmarks. With GPT-5.6, it exceeds the strongest competing result on BC-VL, MM-BC, and MMSearch+ by 2.2, 24.0, and 12.4 points, respectively; corresponding gains are 6.0, 12.7, and 6.5 points with Qwen3.5-35B-A3B, and 5.0, 13.0, and 5.5 points with Qwen3.5-9B. These consistent improvements across frontier and open-source policies support the advantage of the Neos workflow beyond the capabilities of the underlying policy model alone.
Q3: How many search turns can Neos sustain? How well does it perform on longer tasks that require hundreds or even thousands of steps, instead of only 50 turns?
Thank you for providing the insight that industrial-grade agents may handle long-horizon tasks with more than hundreds of interaction turns. We have added an evaluation with a larger turn budget.
| Maximum turns | BC-VL | BC-V3 | FVQA |
|---|---|---|---|
| 50 | 63.2 | 37.0 | 83.3 |
| 80 | 62.6 | 37.3 | 82.7 |
| 100 | 63.6 | 37.8 | 83.3 |
| 150 | 63.4 | 37.5 | 83.7 |
We found that excessively long interactions frequently exceed the configured context window. Since our method does not incorporate context compression, and considering both comparison fairness and the limited rebuttal period, we conducted an extended evaluation allowing up to 150 turns. The results show that increasing the turn budget yields modest performance gains for Neos. However, for long-tail samples, additional reasoning turns do not substantially improve accuracy.
Q4: The gap between RL-only and SFT->RL in Table 11 seem quite small. Several prior works suggest that RL can generalize better in OOD settings. Are there any evaluations on OOD tasks? Could the RL-only model actually perform better in these settings? Since we are training foundation models, we should care more about generalization rather than just getting better performance on a few benchmarks.
High-quality SFT trajectories provide a strong initialization, while agentic RL further develops the model’s ability to tackle challenging, long-horizon search tasks. As shown in Table 5, vanilla GRPO yields limited improvements, whereas our agentic RL approach effectively improves upon SFT, highlighting the complementary roles of the two training stages.
Thank you for your insightful suggestion. Although we train a specialized agent for multimodal deep search, investigating the out-of-distribution (OOD) generalization of RL-trained agents is also a valuable research direction. We have added experiments on several OOD benchmarks, including multimodal reasoning benchmarks.
| Training | WeMath | MathVista | LogicVista | VisuLogic | CharXiv-RQ | VisualToolBench APR | VisualToolBench ARS |
|---|---|---|---|---|---|---|---|
| Qwen3.5-9B base | 82.48 | 85.60 | 73.32 | 33.60 | 71.90 | 4.32 | 24.83 |
| SFT only | 82.91 | 85.40 | 72.54 | 34.80 | 72.20 | 5.03 | 23.27 |
| RL only | 82.77 | 85.70 | 74.11 | 34.90 | 72.35 | 5.38 | 21.70 |
| SFT→RL (Neos-9B) | 83.05 | 85.60 | 74.89 | 35.20 | 72.50 | 5.73 | 24.05 |
Compared with RL-only training, Neos-9B achieves higher scores on six of the seven OOD metrics, including gains of 0.78 points on LogicVista and 0.30 points on VisuLogic. On VisualToolBench, SFT→RL improves APR from 5.38 to 5.73 and ARS from 21.70 to 24.05, yielding gains of 0.35 and 2.35 points.
The results suggest that our training approach offers better overall generalization than RL alone. Although such specialized training may not produce OOD gains as substantial as those achieved by MixRL, we attribute the observed improvements partly to the task’s role in developing the model’s reasoning abilities during search, which can benefit generalization. The SFT-based cold start may also contribute to these gains.
Q5: Why do we need a separate model to process retrieved information? Why cant the main agent do this itself? Please provide an ablation comparing the two setups.
We compared policy-side summarization, policy-side image processing, their combination, and the standard MEI setup. MEI performs best across all three benchmarks, supporting the benefit of delegating evidence processing to a separate interface in our evaluated configuration. Moving this processing to the policy side does not eliminate its overhead: deploying an auxiliary agent entails a setup comparable to MEI, whereas having the main agent perform it directly adds latency. Our results also suggest that Neos’s specialized training can interfere with its self-summarization ability. Methods such as SimpleSearch-VL[1] instead seek to integrate this ability into the policy through explicit training, which introduces additional training requirements.
| Evidence processing | BC-VL | MM-BC | SimpleVQA |
|---|---|---|---|
| Policy summarizes pages | 59.9 | 36.0 | 78.7 |
| Policy processes image candidates | 61.7 | 37.8 | 80.0 |
| Policy handles both | 60.2 | 35.8 | 79.3 |
| MEI handles both (standard Neos) | 63.2 | 39.5 | 81.3 |
[1] Dai, Ming, et al. "SimpleSearch-VL: A Simple Recipe for Multimodal Agentic Deep Search." arXiv preprint arXiv:2606.31504 (2026).
Thank you for your thorough reading of our manuscript and your insightful analysis of our methodology. We appreciate your rigorous approach to scientific evaluation and have done our best to address your concerns below, as well as to present the additional experiments we have conducted, which are also incorporated in Appendix.E highlighted in purple for ease of reference.
W1: Severe clarity issue - The paper uses complicated terminology to describe relatively simple ideas without clearly explaining them. For example, MEI seems to be just another model that processes retrieved content before passing it to the main agent, with the two models trained alternately. Why not just explain it this way? Instead, the paper introduces confusing terms like "goal-conditioned observations" and "agent–environment co-evolution," making the method unnecessarily difficult to follow...
We sincerely thank the reviewer for this constructive feedback and acknowledge that the core mechanism of MEI can be simply understood as a preprocessing model for retrieved content that is trained alternately with the main agent; however, we would like to respectfully clarify that the specific terminology used is technically precise rather than unnecessarily complex, as "goal-conditioned observations" accurately captures that MEI explicitly filters and returns information conditioned on the search goal passed by the agent—a nuance that normal terms like "process retrieved information" would fail to convey—and "agent–environment co-evolution" precisely describes the coupled optimization dynamics arising from alternating training with MEI embedded in the environment, which aligns with established concepts in reinforcement learning literature. That said, we fully agree that our current exposition could be more accessible, and in the revised manuscript we will introduce these terms with clearer plain-language explanations upfront and fine-tune the figures in our paper, thereby preserving technical rigor while significantly improving readability.
W2: Unfair experimental setup - Neos use different training data, search pipeline, tools and training compute compared to all the methods in Table 1. I don't see any experimental setup that compare these methods fairly. How do we know the improvements are from the proposed tricks, rather than simply different training data or a better search pipeline? In fact, Table 13 already shows that changing the workflow alone can make a huge difference. The data ablation in Table 2 only compares different data sources within Neos, which does not address this issue...
Regarding the comparison with other methods in the main table: because online search evaluation is inherently sensitive to temporal shifts—leading to discrepancies when reproducing results via search APIs—and given that some baseline methods are not fully open-sourced, we followed the common practice in the search research community by reporting the original numbers from prior work. Nevertheless, we fully recognize the importance of comprehensive ablation studies for ensuring the rigor of our paper. To address your concerns, we have supplemented alignment experiments to the extent permitted by our available resources.
First, we conducted experiments to verify data fairness. Our training data includes questions from RedSearcher-MM as well as self-constructed questions; however, these were sampled as trajectories within the Neos workflow and used for SFT training. To ensure a fair comparison, we applied the same rejection sampling model to these questions under the RedSearcher-MM workflow, obtaining an SFT dataset of equivalent size for training and evaluation. Building upon the results in Table 2, we obtained the following ablation experiment results:
Using our synthetic data instead of REDSearcher-MM questions improves BC-VL, MMSearch+, and FVQA scores by 6.3, 10.0, and 5.0 points under the REDSearcher-MM workflow, and by 1.5, 4.3, and 1.0 points under Neos, respectively. Holding the question source fixed, Neos outperforms REDSearcher-MM by 10.0–14.8, 10.4–16.1, and 2.0–6.0 points on the three benchmarks.
We appreciate your suggestion. These experiments clearly and independently demonstrate that both our data construction approach and search pipeline make unique contributions to enhancing multimodal long-horizon search capabilities. Specifically, they provide support from two perspectives—task design and search workflow—for shaping the agent's ability to capture critical visual information during long-horizon search processes and guide subsequent reasoning steps.