- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies how to train a utility-aware retriever for document deep research, placing evidence carried by different information carriers such as text, tables, and figures, as well as evidence at different granularities such as chunk, page, and summary, into a unified retrieval pool. The method builds on BGE-M3 and uses four types of supervision signals—LR, GAC, ECC, and LOC—to select positive samples and perform multi-positive contrastive learning. Among these, tables and figures participate in retrieval mainly through text descriptions generated by a VLM. On BlueprintSynth-Eval, the full method improves Full-chain@20 from 0.588 to 0.674, but simple positive-sample rules also achieve comparable or higher results, and the gains on external retrieval and answering tasks are small.
Critical strengths
The problem is practically meaningful. Document retrieval needs to handle different information carriers, different granularities, and citation locations; local relevance alone may not reflect how much evidence helps the final answer. Constructing the supervision signal in an offline stage and using only the trained retriever at deployment is also a practically valuable design.
Critical weaknesses
-
I am still not sure whether the existing experiments clearly demonstrate the advantage of the proposed mixed retrieval design. The full method achieves Full-chain@20 of 0.674, while simple positive-sample rules on the same mixed pool reach 0.678–0.681. External gains are also small, for example MMDocIR Page R@5 improves from 0.705 to 0.710. These comparisons do not answer whether unified retrieval is more effective than retrieving separately by information carrier or granularity and then fusing the results. The coverage of existing benchmarks and baselines is also insufficient to show under which conditions this design helps.
-
It is still not clear to me why the four signals are combined in this way. The paper gives some priority and gating rules, but does not sufficiently explain the basis for choosing these rules, or how conflicts between signals are handled concretely. For example, chain units with GAC≤0 are demoted to weak positives, while the training section also says that at most three positives passing the GAC gate are selected. How these weak positives ultimately participate in training remains unclear, making the positive-sample selection process difficult to reproduce.
Questions for authors
- Under the same encoder, corpus, and context budget, how does unified mixed retrieval compare with "retrieve separately, then fuse"? Which types of questions benefit the most?
- How are the signal priorities and gating rules determined? How do weak positive samples with GAC ≤ 0 participate in the training loss?
- In the external benchmarks, which subsets directly require evidence retrieval across information carriers and across granularities? Can comparative results for the relevant retrieval strategies be reported on one such subset?