- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes SearchOS, a multi-agent search framework that maintains shared tasks, evidence, coverage, and failure records outside agent conversations. It organizes search around relational tables, uses middleware to record evidence and detect stalled searches, and generates reusable skills for accessing websites. Experiments report improvements over single-agent and multi-agent baselines on four benchmarks.
Critical strengths
The paper addresses a practical problem. Maintaining shared search progress and failure records can help agents avoid repeated searches and coordinate their work more effectively.
The experiments cover both structured information gathering and general search tasks. The ablations show benefits from shared context and failure memory, while the scheduling experiment shows reduced execution time.
Critical weaknesses
The evaluation does not clearly establish performance on unseen tasks. Section 4.1 states that SearchOS first runs each case, generates access skills from that trajectory, and then reruns the same case for evaluation. This gives the system task-specific preparation, while no comparable procedure is described for the baselines. Even without using reference answers, this additional experience could contribute to the reported gains. Evaluation with a skill library built on separate tasks is needed for a fairer comparison.
The evidence for robust search and grounding is incomplete. The middleware analysis mainly presents three successful intervention examples, without a controlled evaluation of how often interventions help or stop useful searches. Similarly, linking a value to a source excerpt does not by itself establish that the excerpt supports the value. Since reliable progress tracking and grounded evidence are central claims, direct evaluation of these components is important.
Questions for authors
How does SearchOS perform on its first attempt, before generating skills from the same test case?
Are the initial run, skill generation, and evidence-extraction calls included in the reported costs? Do the baselines receive the same total resource budget?
Do the generated skills improve performance on different questions involving the same websites, or on entirely unseen tasks?
Can you report citation-support accuracy and a controlled ablation of the stall-detection middleware?
6 comments
For runtime control, we additionally audited 100 existing trajectories:
| Observed intervention behavior | Result |
|---|---|
| Strategy-switch prompts | 55 across 16 cases |
| Prompts followed by a further tool call from the same agent | 52/55 |
| Explicit forced stops attributed to budget exhaustion | 69/69 |
These audits extend the evidence beyond the intervention examples in §5.4 (L421-L454) and will be incorporated into the main-text analysis.
Q1. “How does SearchOS perform on its first attempt, before generating skills from the same test case?”
These are the first-attempt outputs, before access skills are generated from the current question. The first attempt runs without the skill hierarchy; its results are the no-skill condition in §5.5, Table 5 (L456-L466). We reproduce WideSearch and GISA below:
| Setting | WideSearch Item F1 | WideSearch Row F1 | GISA Table Item F1 | GISA Table Row F1 | GISA Set F1 | GISA List F1 | GISA Item EM |
|---|---|---|---|---|---|---|---|
| First attempt (without skills) | 69.8 | 41.4 | 68.17 | 44.72 | 62.22 | 54.93 | 33.33 |
| Full SearchOS | 73.2 | 47.6 | 68.89 | 48.23 | 63.24 | 59.35 | 27.78 |
Without skills, SearchOS still exceeds the highest baseline WideSearch Item F1 of 68.7 and the strongest baseline on four of five GISA metrics: Table Item F1, Set F1, List F1, and Item EM. Enabling skills further improves WideSearch Item F1 by 3.4 points and Row F1 by 6.2 points.
Q2. “Are the initial run, skill generation, and evidence-extraction calls included in the reported costs? Do the baselines receive the same total resource budget?”
All methods use GLM-5 for search and coordination (§4.1, L354-L355). SearchOS assigns evidence extraction to Qwen3.5-35B-A3B and persists the resulting records programmatically (§3.3.2, L261-L274). Our baseline reproductions use GLM-5 to process observations and produce structured outputs:
| Method | Agent backbone | Evidence or structured-output processing |
|---|---|---|
| ReAct | GLM-5 | GLM-5 reasoning and answer generation |
| Plan-and-Solve | GLM-5 | GLM-5 planning, search, and answer generation |
| MindSearch | GLM-5 | GLM-5 search coordination and synthesis |
| Table-as-Search | GLM-5 | GLM-5 webpage summarization and database updates |
| A-MapReduce | GLM-5 | GLM-5 summarization and aggregation |
| Web2BigTable | GLM-5 | GLM-5 workers and shared-workboard updates |
| SearchOS | GLM-5 | Qwen3.5-35B-A3B extraction and programmatic persistence |
All methods share the evaluation-time table-reformatting pipeline on WideSearch and GISA. The common execution limits apply per session: 50 orchestrator iterations, 8 parallel sub-agents, and 1,800 seconds (§4.1, L357-L360). Figure 4 reports scored-session time and search/Jina API usage, with reductions of 36.6%, 39.1%, and 42.7% from enabling skills (L438-L448, §5.5, L467-L468). Extraction operates within search execution. The initial attempt and skill construction are preparation stages specified in L354-L356. The first-attempt results in Q1 already establish the WideSearch Item-F1 advantage before current-question skill preparation. This makes the preparation-free performance and the skill-enabled performance explicit.
Q3. “Do the generated skills improve performance on different questions involving the same websites, or on entirely unseen tasks?”
Access skills are reused across different questions. An execution analysis covering all 100 questions in a separate WideSearch run gives:
| Measure | Result |
|---|---|
| Questions invoking access skills | 87/100 |
| Distinct skills invoked | 153 |
| Total skill calls | 2,560 |
| Skills used on multiple questions | 29/153 (18.95%) |
For an unseen source, SearchOS uses general search and browsing tools, guided by source-independent strategy skills. The resulting trajectory can seed a source-specific access skill, admitted after interface checks and execution probes (§3.4, L305-L315).
Q1 measures the skill-enhancement gain over the first attempt, and these execution records establish cross-question reuse. Website-access skills expose reusable source operations, allowing different questions about the same source to invoke the same skill. The evaluated protocol permits online library updates as new sources are encountered (§3.4, L305-L315). A frozen library generated on disjoint tasks would assess transfer without these updates. We report the measured enhancement gains and observed reuse under the stated protocol. The completed reuse and source-coverage analyses will be incorporated into §5.5 of the revised main text.
Q4. “Can you report citation-support accuracy and a controlled ablation of the stall-detection middleware?”
We quantify the Sensor Middleware contribution through final-answer quality on 100 WideSearch questions:
| Setting | Item P | Item R | Item F1 | Row P | Row R | Row F1 |
|---|---|---|---|---|---|---|
| w/o Sensor Middleware | 73.18 | 71.13 | 71.46 | 41.12 | 39.88 | 40.18 |
| Full SearchOS | 79.09 | 71.24 | 73.18 | 51.34 | 46.30 | 47.63 |
Removing Sensor Middleware lowers Item F1 by 1.72 points and Row F1 by 7.45 points. Row precision falls by 10.22 points. The much larger row-level loss supports the role of runtime intervention in maintaining complete, correct records over long searches. The 100-trajectory audit in W2 describes intervention behavior across a full run; the ablation measures the resulting difference in answer quality. Section 5.4 (L421-L454) illustrates how these interventions redirect stalled searches.
W2 reports 74.48% fully supported and 12.69% partially supported source-linked facts across the 50-case audit, totaling 87.18%. Saved material permits a support judgment for 95.25%. Across 100 trajectories, 52 of 55 strategy-switch prompts were followed by another tool call from the same agent, and all 69 explicit forced stops were attributed to budget exhaustion. This aggregate sensor analysis extends the runtime evidence beyond the three intervention examples in §5.4 (L421-L454).
The first-attempt results establish a WideSearch Item-F1 advantage before current-question skill generation, and the middleware ablations quantify losses in complete-record quality when components are removed. The source-support audit further examines whether retained sources substantiate the answers. We ask the reviewer to reconsider the recommendation in light of this evidence and the clarified evaluation protocol.
We thank the reviewer for highlighting task-specific preparation and evidence reliability. The responses below present first-attempt performance, direct middleware comparisons, and a source-support audit of 50 cases, together with intervention statistics from 100 trajectories. These results allow the preparation protocol and the reliability claims to be assessed directly.
The assessment must account for the existing no-skill ablation and baseline preparation. Skill generation and use are explicit methodological contributions (§3.4, L294-L315). Section 4.1 specifies trajectory-based access-skill generation followed by a scored attempt (L354-L356). Our reproductions retain A-MapReduce’s experience-guided planning and Web2BigTable’s learned skill banks:
insights.json) and positive/negative examples guiding task decomposition, output templates, and batchingtask-routeranddecompose-*skills for decomposition by entity, time, category, rank, or dependency. Worker skills for retrieval, evidence verification, and synthesisTable 5 already evaluates SearchOS with the entire skill hierarchy disabled (§5.5, L456-L466). Without skills, SearchOS exceeds the strongest baseline on WideSearch Item F1 and four of five GISA metrics, as reproduced in Q1. The performance advantage persists with all skills disabled.
Q1 reports these first-attempt outputs, before skill generation from the current question. The evaluated protocol addresses online acquisition and use of source-access skills. A library frozen on disjoint tasks would assess transfer without further library updates; our claims concern the first-attempt performance, measured skill-enhancement gains, and cross-question reuse documented in Q3.
Middleware ablations on 100 WideSearch questions (Context: Table 2, L378-L385; full system: Table 1, L324-L342). Context and full-system results are three-run means; extraction and sensor results are single runs.
The context ablation produces the largest Item-F1 loss, from 73.18 to 66.51, together with a 6.85-point loss in Item recall. This supports the motivation in §1 (L044-L049): long-horizon collection benefits from explicit search state that tells agents which records and required attributes still need work. The existing SOCM control keeps the extractor fixed, directly testing the contribution of this context interface.
Evidence Extraction and Sensor Middleware have a particularly strong effect on complete records. Removing them costs 6.91 and 7.45 Row F1 points, respectively, compared with 2.27 and 1.72 Item F1 points. WideSearch counts a row as correct only when all required fields are correct. The larger row-level losses suggest that persistent evidence capture and runtime feedback help agents assemble complete, correct entity records over long searches.
We evaluated source support across 50 WideSearch cases, checking source-linked final-table facts against archived webpages and structured source records for entity, attribute, time, and unit consistency.
Percentages refer to source-linked substantive cells; the last row includes supported, partially supported, and unsupported judgments. Over 90% had sufficient saved source material for verification, and 87.18% were fully or partially supported.