- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
M-ASK divides multi-turn search into five roles that share an LLM and communicate through a structured knowledge state. The roles are jointly trained with PPO, using absolute answer F1 and per-turn F1 improvements as rewards. Experiments on seven QA benchmarks report higher average F1 and more stable training than the compared baselines, with additional evaluations across Qwen model sizes and an exploratory web-search setting.
Critical strengths
-
Useful knowledge management and intermediate supervision. Removing knowledge management or turn-level rewards reduces multi-hop average F1 by 2.86 and 15.23 points, respectively (Table 4), supporting their value within this framework.
-
Improved performance and training stability. M-ASK achieves 50.09 average F1 versus 46.79 for the strongest overall baseline (Table 1). In ten runs, it records no collapses through 1,000 steps, versus nine for Search-R1 under the reported setup (Table 3).
Critical weaknesses
-
Limited evidence for generality and statistical reliability. All evaluated backbones are Qwen models between 0.5B and 8B. Despite the separate ten-run stability study, Tables 1 and 4 lack uncertainty estimates and clear run counts. Gains over the strongest baselines are small on AmbigQA (+0.31) and Musique (+0.72), with no gains on PopQA or Bamboogle. Cross-family evaluation and uncertainty estimates would strengthen confidence in the findings.
-
Missing end-to-end latency comparison. Appendix K reports token counts and throughput, but not average task completion time; scheduling also differs between methods. It remains unclear whether sequential agent calls make M-ASK slower per query than Search-R1. A matched comparison including retrieval and all agent calls would clarify whether the accuracy gains require additional waiting time.
Questions for authors
-
How many independent runs underlie Tables 1 and 4, and how were checkpoints selected? Please provide available uncertainty estimates, especially for the small performance margins.
-
Does M-ASK take longer per query than Search-R1 under matched hardware and serving conditions? Please report available mean end-to-end times, including retrieval and all agent calls, or clarify whether this requires new measurements.