- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces WorkSurface-Bench, a benchmark to evaluate the ability of knowledge surface routing. These agents usually need several types of knowledge when performing tasks, and they need to select the correct knowledge source. The proposed benchmark projects Workspace-Bench-Lite workspaces onto various types of knowledge and create 1000+ tasks to evaluate the agents' surface routing capability. With various LLM backbone and agent settings, it is found that good routing does not guarantee better answers, providing some insights on future work in this direction.
Critical strengths
- This paper introduces a dataset and a corresponding creating pipeline, which will be useful for the direction of retrieval from different types of knowledge sources. Moreover, the annotations in the dataset provide reference answers with strong grounding.
- The experiments are comprehensive, covering various models and agent settings, with detailed analysis.
Critical weaknesses
- I'm not sure what the surface routing can indicate. In many cases, the agent might need to check multiple resources, and combine multiple pieces of information. Maybe ultimately we are still doing RAG (although it is not only constrained to documents, but also retrieval acoording to other knowledge types). Then it is unclear the difference between retrieval and surface routing.
- It is also unclear if there is redundant information across different sources, how will the evaluation be adjusted.
Questions for authors
- Some of the agent settings are not very straight-forward to understand, how about listing some examples for easier comprehension?
- According to the paragraph at Line 300, it seems that routing is mainly evaluated at the surface-type level, will it be too coarse-grained?
- Only three surface types are included, have the authors consider other types of knowledge sources?