- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper presents RubricAtlas, a framework for training generalist deep-research agents. Its main components are taxonomy-guided task synthesis, Decoupled Rubric Discovery and Validation (DRDV), a Memo Context Manager, and rubric-wise reward normalization through GDPO. The authors construct 8,000 query-rubric pairs and train a Qwen3-14B model using cold-start SFT followed by RL. The resulting model performs strongly on DeepResearch Bench, DeepConsult, and ResearchRubrics, reaching a 46.67 RACE score, an 82.60% DeepConsult win rate, and 62.15% rubric compliance.
Critical strengths
- Separating rubric discovery from factual verification is well motivated: criteria inferred from model outputs can otherwise reward verbosity or unsupported claims. Combining DRDV with a trajectory-grounded evidence reward gives the framework a coherent design beyond simply optimizing report style.
- RubricAtlas-RL-14B consistently improves over its Qwen3-14B backbone and outperforms substantially larger open-source models across all three reported benchmarks. The component ablations also provide useful evidence that rubric validation, memory management, and criterion-wise normalization each contribute to the final performance. RL is effective.
Critical weaknesses
- All three benchmarks focus mainly on long-form report generation and rely heavily on model-based judgments. It remains unclear whether the method also improves factual retrieval, search planning, and answer correctness on QA-oriented deep-search tasks such as BrowseComp, GAIA, or DeepSearchQA.
- The paper reports strong downstream curves, but provides little direct evidence about the quality of the generated rubrics—for example, human audit results, validator agreement, difficulty. Moreover, the DRDV variants contain different numbers of rubrics, making it difficult to separate the effect of factual validation from rubric quantity or density.
Questions for authors
- Can the authors report results on at least one fact-centric deep-search benchmark, such as BrowseComp, GAIA, or DeepSearchQA (if expensive, subset is ok)? This would help determine whether the gains extend beyond long-form rubric compliance to retrieval and reasoning accuracy.
- Can the authors provide a human audit of the constructed rubrics, including factual correctness, relevance, atomicity, and agreement with the automatic validators? A rubric-count-matched comparison between unvalidated discovery and DRDV would also make the claimed mechanism substantially more convincing.