acceptodds
Withdrawn from ICLR 2027

WorkSurface-Bench: Benchmarking Enterprise Agents on Multi-Surface Knowledge Routing

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Enterprise agents often need several kinds of knowledge at once: documents for narrative facts, tables for calculations, and dependency graphs for file relationships. Existing benchmarks usually test retrieval or tool use, but do not separate choosing the right knowledge form from using it correctly. We introduce WorkSurface-Bench, which evaluates this choice as surface routing. The benchmark projects five persona-scoped Workspace-Bench-Lite workspaces onto document, table, and graph surfaces and contains 1,151 atomic tasks. Its reference answers are auditable: table answers come from executed DuckDB queries, document answers are checked against verbatim spans, and graph answers derive from source annotations. We evaluate four backbones under six agent settings, yielding 27,624 retained trajectories with no protocol errors. Gold-constrained agents reach 98.7–99.8 Route F1, yet Answer remains 56.1–75.3%. A matched intervention shows that surface hints improve Answer for three of four models, whereas removing irrelevant tools mainly improves routing and efficiency. In an independent three-annotator audit, all 200 sampled tasks pass all six criteria by majority vote. We release the construction pipeline, scoring code, and agent harness at https://anonymous.4open.science/r/WorkSurface-Bench-Anonymous-D097.

Primary area:
datasets and benchmarks
Keywords:
AI Agents, Benchmark Evaluation, Tool Routing, Retrieval-Augmented Generation, Multi-Surface Reasoning

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_mqV7OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces WorkSurface-Bench, a benchmark to evaluate the ability of knowledge surface routing. These agents usually need several types of knowledge when performing tasks, and they need to select the correct knowledge source. The proposed benchmark projects Workspace-Bench-Lite workspaces onto various types of knowledge and create 1000+ tasks to evaluate the agents' surface routing capability. With various LLM backbone and agent settings, it is found that good routing does not guarantee better answers, providing some insights on future work in this direction.

Critical strengths
  • This paper introduces a dataset and a corresponding creating pipeline, which will be useful for the direction of retrieval from different types of knowledge sources. Moreover, the annotations in the dataset provide reference answers with strong grounding.
  • The experiments are comprehensive, covering various models and agent settings, with detailed analysis.
Critical weaknesses
  • I'm not sure what the surface routing can indicate. In many cases, the agent might need to check multiple resources, and combine multiple pieces of information. Maybe ultimately we are still doing RAG (although it is not only constrained to documents, but also retrieval acoording to other knowledge types). Then it is unclear the difference between retrieval and surface routing.
  • It is also unclear if there is redundant information across different sources, how will the evaluation be adjusted.
Questions for authors
  • Some of the agent settings are not very straight-forward to understand, how about listing some examples for easier comprehension?
  • According to the paragraph at Line 300, it seems that routing is mainly evaluated at the surface-type level, will it be too coarse-grained?
  • Only three surface types are included, have the authors consider other types of knowledge sources?
Reviewer_3y8hOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper treats documents, tables, and file-dependency graphs in enterprise workspaces as three knowledge surfaces. It constructs 1,151 atomic questions and scores routing, evidence access, answers, and efficiency separately. Reference answers come from executed queries, verified text spans, and graph annotations. Results across four models and six settings show that selecting the required surfaces does not guarantee a correct answer. Surface hints help some models, whereas removing irrelevant tools improves routing and efficiency more consistently. Human audits and off-the-shelf agent frameworks provide additional validation.

Critical strengths
  1. Documents, tables, and dependency graphs are evaluated within shared enterprise workspaces. Reference answers trace to executed queries, source text, and graph annotations, supported by full automated checks and a sampled human audit.
  2. Gold-hint/all and Gold-constrained retain the same surface hints while changing tool exposure, separating information from restriction. Together with separate routing and answer scores, this provides convincing evidence that selecting the right sources does not guarantee correct answers.
Critical weaknesses
  1. Route treats the annotated surface set as the correct target, although valid answers may follow different evidence paths. If accessing every annotated surface is unnecessary, the score may penalize effective source selection. This affects whether routing scores accurately reflect the ability to choose information sources.
  2. Evidence measures access to required files, views, or nodes, not whether the correct evidence was extracted and used. High Route with low Answer reveals a performance gap but does not adequately distinguish acquisition from reasoning failures, limiting the depth of the proposed diagnosis.
Questions for authors
  1. How is the need to combine all annotated sources in cross-surface tasks validated? How does the benchmark handle alternative paths that yield sufficient evidence and a correct answer without accessing every annotated surface?
  2. In existing traces with high Route and Evidence but incorrect answers, is the main problem failure to obtain the right evidence or failure to reason from it? Could representative tasks illustrate the distinction?
Reviewer_FT7fOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

WorkSurface-Bench is a benchmark of 1151 atomic tasks from workspace-bench-lite spread over three surfaces: documents (RAG), DuckDB tables, and a file-dependency graph. Each task gets four scores: Route, Evidence, Answer, and Efficiency. The paper evaluates 4 models in 6 settings and reports that gold-constructed agents reach about 99 route F1 while answer stays 56-75%.

Critical strengths

1- Gold answers can be audited: table answers come from executed SQL, document answers from verified spans, and graph answers from source annotations, with no LLM judge for answers 2- Scoring routing, evidence, and answers separately is a useful idea for diagnosing failures. 3- Construction, scoring code, and traces are released, and the S4/S5/S6 settings form a controlled comparison.

Critical weaknesses

1- The results section lacks analysis. The point of the experiments and results section is to analyze the results not just repeating the results from the tables. 2- Table The paper switches between setting names and labels 3- Table 31 claims on SOP/skill dimension that is never evaluated, and Safety if defined and then dropped. This is clarity problem. 4- The intro motivates multi-part analyst question, but every task is atomic. I don't see why we need this benchmark and workspace-bench-lite is not enough. 5- The headline results is close to true by design. In S5 the agent can only use the required surfaces, so near-perfect route F1 is expected and route precision is 100% by construction. The "routing is not answering" finding then mostly shows that retrieval and reasoning after routing are hard which is already known. 6- There is a possible selection bias because the candidate tasks were LLM-generated and then filtered by GPT 5.5, DeepSeek, and Gemini, the same models that were evaluated. 7-The evaluation setup is small for a benchmark. There should be multi run with multiple seeds. 8-turn cap limits the model performances. The model set is narrow.

Questions for authors

1- Can you give a per-model break down of failures into selection, acquisition and synthesis, using actual counts. 2- Why does GPT-4o-min i's RAG answer fall from 70 to 19.7 under ReAcl-all? 3- How do results change with multiple runs and a larger turn budget? 4- Do the rankings still hold if efficiency uses a budget that doesn't depend on any model?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.