- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The paper presents a harness for multimodal agentic search. It implements a file system for retrieved images and supports runtime recovery. The authors also performed sft and rl training. The harness and sft training both yield noticeable improvements. They also proposed some practical tricks for gspo and demonstrated their rl works.
Critical strengths
- The experiments are solid, and the benchmark improvements are real.
- The harness design is clever: appropriate caching for search, auto-repair from the harness, maintaining a persistent multimodal state, etc.
Critical weaknesses
- The individual components in their methodology might not be novel: people usually use caching to save search cost, and previous works have presented similar data collection processes.
- The proposed fa-gspo training is strongly coupled with their harness. A model trained using fa-gspo would probably not work very well on a harness without runtime recovery. I'd consider this a relatively minor weakness.
Questions for authors
- The comparison with Hermes is presented in Section 7.3. The average benchmark improvement is about 2-3%. Can the authors clarify the differences between the proposed harness and Hermes, and potentially ablate which component in the proposed harness matters the most?
- I am interested in knowing the contribution of the in-house data to the post-training. For example, the authors mentioned using 56.2k open-source and 3.3k in-house SFT trajectories. Training the model using only the 56.2k open-source or only the 3.3k in-house trajectories, but not both, might clarify the effectiveness of the proposed data collection pipeline.
Dear reviewers,
Will you please check the authors' response and reply it?
Best, AC