acceptodds
Under review as a conference paper at ICLR 2027

SearchOS: Towards Robust Open-Domain Information-Seeking Agent Collaboration

Abstract

Recent advances in tool-integrated large language models have made web search a core capability of information-seeking agents. However, as interaction histories grow, agents increasingly struggle to track task progress. When search attempts fail to yield useful evidence, current single- and multi-agent systems can become trapped in repetitive loops, wasting search budgets and ultimately compromising the quality and completeness of the final output. We introduce , a system-level multi-agent framework that turns fragile, implicit search progress into explicit, persistent, and shared state. formulates open-domain information seeking as relational schema completion with grounded citations, where agents discover entities, record attributes across linked tables, and anchor each value to source evidence. Inspired by this paradigm, we design three core modules of : (1) Search-Oriented Context Management (SOCM), which externalizes and shares evolving search state across sub-agents and supports pipeline-parallel scheduling to improve utilization and throughput; (2) a Search Tool Middleware Harness, which intercepts model and tool interactions to record grounded evidence and respond to stalls or budget exhaustion; and (3) a reusable hierarchical skill system, which provides strategy and access skills to guide task-level search and source-specific retrieval. On WideSearch and GISA, leads all metrics among the evaluated single- and multi-agent baselines, paving the way toward robust information-seeking collaboration.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
Search Agent, Multi-Agent System

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_PV4COpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes SearchOS, a multi-agent search framework that maintains shared tasks, evidence, coverage, and failure records outside agent conversations. It organizes search around relational tables, uses middleware to record evidence and detect stalled searches, and generates reusable skills for accessing websites. Experiments report improvements over single-agent and multi-agent baselines on four benchmarks.

Critical strengths

The paper addresses a practical problem. Maintaining shared search progress and failure records can help agents avoid repeated searches and coordinate their work more effectively.

The experiments cover both structured information gathering and general search tasks. The ablations show benefits from shared context and failure memory, while the scheduling experiment shows reduced execution time.

Critical weaknesses

The evaluation does not clearly establish performance on unseen tasks. Section 4.1 states that SearchOS first runs each case, generates access skills from that trajectory, and then reruns the same case for evaluation. This gives the system task-specific preparation, while no comparable procedure is described for the baselines. Even without using reference answers, this additional experience could contribute to the reported gains. Evaluation with a skill library built on separate tasks is needed for a fairer comparison.

The evidence for robust search and grounding is incomplete. The middleware analysis mainly presents three successful intervention examples, without a controlled evaluation of how often interventions help or stop useful searches. Similarly, linking a value to a source excerpt does not by itself establish that the excerpt supports the value. Since reliable progress tracking and grounded evidence are central claims, direct evaluation of these components is important.

Questions for authors

How does SearchOS perform on its first attempt, before generating skills from the same test case?

Are the initial run, skill generation, and evidence-extraction calls included in the reported costs? Do the baselines receive the same total resource budget?

Do the generated skills improve performance on different questions involving the same websites, or on entirely unseen tasks?

Can you report citation-support accuracy and a controlled ablation of the stall-detection middleware?

6 comments

We thank the reviewer for highlighting task-specific preparation and evidence reliability. The responses below present first-attempt performance, direct middleware comparisons, and a source-support audit of 50 cases, together with intervention statistics from 100 trajectories. These results allow the preparation protocol and the reliability claims to be assessed directly.

W1. “The evaluation does not clearly establish performance on unseen tasks. … Evaluation with a skill library built on separate tasks is needed for a fairer comparison.”

The assessment must account for the existing no-skill ablation and baseline preparation. Skill generation and use are explicit methodological contributions (§3.4, L294-L315). Section 4.1 specifies trajectory-based access-skill generation followed by a scored attempt (L354-L356). Our reproductions retain A-MapReduce’s experience-guided planning and Web2BigTable’s learned skill banks:

MethodReusable skills or experienceRetained mechanism
A-MapReduceExperience-memory insights (insights.json) and positive/negative examples guiding task decomposition, output templates, and batchingRetrieval of previously collected experience during planning
Web2BigTabletask-router and decompose-* skills for decomposition by entity, time, category, rank, or dependency. Worker skills for retrieval, evidence verification, and synthesisPre-generated orchestrator and worker skill banks with skill retrieval
SearchOSOrchestrator playbooks, search-strategy skills, and executable website-access skillsStrategy initialization from Web2BigTable, plus trajectory-based generation, validation, and use of access skills

Table 5 already evaluates SearchOS with the entire skill hierarchy disabled (§5.5, L456-L466). Without skills, SearchOS exceeds the strongest baseline on WideSearch Item F1 and four of five GISA metrics, as reproduced in Q1. The performance advantage persists with all skills disabled.

Q1 reports these first-attempt outputs, before skill generation from the current question. The evaluated protocol addresses online acquisition and use of source-access skills. A library frozen on disjoint tasks would assess transfer without further library updates; our claims concern the first-attempt performance, measured skill-enhancement gains, and cross-question reuse documented in Q3.

W2. “The evidence for robust search and grounding is incomplete.”

Middleware ablations on 100 WideSearch questions (Context: Table 2, L378-L385; full system: Table 1, L324-L342). Context and full-system results are three-run means; extraction and sensor results are single runs.

SettingItem PItem RItem F1Row PRow RRow F1
w/o Evidence Extraction Middleware73.2169.9170.9142.0040.1840.72
w/o Sensor Middleware73.1871.1371.4641.1239.8840.18
w/o Context Middleware73.2764.3966.5143.6538.7239.99
Full SearchOS79.0971.2473.1851.3446.3047.63

The context ablation produces the largest Item-F1 loss, from 73.18 to 66.51, together with a 6.85-point loss in Item recall. This supports the motivation in §1 (L044-L049): long-horizon collection benefits from explicit search state that tells agents which records and required attributes still need work. The existing SOCM control keeps the extractor fixed, directly testing the contribution of this context interface.

Evidence Extraction and Sensor Middleware have a particularly strong effect on complete records. Removing them costs 6.91 and 7.45 Row F1 points, respectively, compared with 2.27 and 1.72 Item F1 points. WideSearch counts a row as correct only when all required fields are correct. The larger row-level losses suggest that persistent evidence capture and runtime feedback help agents assemble complete, correct entity records over long searches.

We evaluated source support across 50 WideSearch cases, checking source-linked final-table facts against archived webpages and structured source records for entity, attribute, time, and unit consistency.

Source-support evaluationResult
Cases audited50
Fully supported74.48%
Partially supported12.69%
Unsupported8.07%
Insufficient material4.75%
Fully or partially supported87.18%
Sufficient material for a support judgment95.25%

Percentages refer to source-linked substantive cells; the last row includes supported, partially supported, and unsupported judgments. Over 90% had sufficient saved source material for verification, and 87.18% were fully or partially supported.

For runtime control, we additionally audited 100 existing trajectories:

Observed intervention behaviorResult
Strategy-switch prompts55 across 16 cases
Prompts followed by a further tool call from the same agent52/55
Explicit forced stops attributed to budget exhaustion69/69

These audits extend the evidence beyond the intervention examples in §5.4 (L421-L454) and will be incorporated into the main-text analysis.

Q1. “How does SearchOS perform on its first attempt, before generating skills from the same test case?”

These are the first-attempt outputs, before access skills are generated from the current question. The first attempt runs without the skill hierarchy; its results are the no-skill condition in §5.5, Table 5 (L456-L466). We reproduce WideSearch and GISA below:

SettingWideSearch Item F1WideSearch Row F1GISA Table Item F1GISA Table Row F1GISA Set F1GISA List F1GISA Item EM
First attempt (without skills)69.841.468.1744.7262.2254.9333.33
Full SearchOS73.247.668.8948.2363.2459.3527.78

Without skills, SearchOS still exceeds the highest baseline WideSearch Item F1 of 68.7 and the strongest baseline on four of five GISA metrics: Table Item F1, Set F1, List F1, and Item EM. Enabling skills further improves WideSearch Item F1 by 3.4 points and Row F1 by 6.2 points.

Q2. “Are the initial run, skill generation, and evidence-extraction calls included in the reported costs? Do the baselines receive the same total resource budget?”

All methods use GLM-5 for search and coordination (§4.1, L354-L355). SearchOS assigns evidence extraction to Qwen3.5-35B-A3B and persists the resulting records programmatically (§3.3.2, L261-L274). Our baseline reproductions use GLM-5 to process observations and produce structured outputs:

MethodAgent backboneEvidence or structured-output processing
ReActGLM-5GLM-5 reasoning and answer generation
Plan-and-SolveGLM-5GLM-5 planning, search, and answer generation
MindSearchGLM-5GLM-5 search coordination and synthesis
Table-as-SearchGLM-5GLM-5 webpage summarization and database updates
A-MapReduceGLM-5GLM-5 summarization and aggregation
Web2BigTableGLM-5GLM-5 workers and shared-workboard updates
SearchOSGLM-5Qwen3.5-35B-A3B extraction and programmatic persistence

All methods share the evaluation-time table-reformatting pipeline on WideSearch and GISA. The common execution limits apply per session: 50 orchestrator iterations, 8 parallel sub-agents, and 1,800 seconds (§4.1, L357-L360). Figure 4 reports scored-session time and search/Jina API usage, with reductions of 36.6%, 39.1%, and 42.7% from enabling skills (L438-L448, §5.5, L467-L468). Extraction operates within search execution. The initial attempt and skill construction are preparation stages specified in L354-L356. The first-attempt results in Q1 already establish the WideSearch Item-F1 advantage before current-question skill preparation. This makes the preparation-free performance and the skill-enabled performance explicit.

Q3. “Do the generated skills improve performance on different questions involving the same websites, or on entirely unseen tasks?”

Access skills are reused across different questions. An execution analysis covering all 100 questions in a separate WideSearch run gives:

MeasureResult
Questions invoking access skills87/100
Distinct skills invoked153
Total skill calls2,560
Skills used on multiple questions29/153 (18.95%)

For an unseen source, SearchOS uses general search and browsing tools, guided by source-independent strategy skills. The resulting trajectory can seed a source-specific access skill, admitted after interface checks and execution probes (§3.4, L305-L315).

Q1 measures the skill-enhancement gain over the first attempt, and these execution records establish cross-question reuse. Website-access skills expose reusable source operations, allowing different questions about the same source to invoke the same skill. The evaluated protocol permits online library updates as new sources are encountered (§3.4, L305-L315). A frozen library generated on disjoint tasks would assess transfer without these updates. We report the measured enhancement gains and observed reuse under the stated protocol. The completed reuse and source-coverage analyses will be incorporated into §5.5 of the revised main text.

Q4. “Can you report citation-support accuracy and a controlled ablation of the stall-detection middleware?”

We quantify the Sensor Middleware contribution through final-answer quality on 100 WideSearch questions:

SettingItem PItem RItem F1Row PRow RRow F1
w/o Sensor Middleware73.1871.1371.4641.1239.8840.18
Full SearchOS79.0971.2473.1851.3446.3047.63

Removing Sensor Middleware lowers Item F1 by 1.72 points and Row F1 by 7.45 points. Row precision falls by 10.22 points. The much larger row-level loss supports the role of runtime intervention in maintaining complete, correct records over long searches. The 100-trajectory audit in W2 describes intervention behavior across a full run; the ablation measures the resulting difference in answer quality. Section 5.4 (L421-L454) illustrates how these interventions redirect stalled searches.

W2 reports 74.48% fully supported and 12.69% partially supported source-linked facts across the 50-case audit, totaling 87.18%. Saved material permits a support judgment for 95.25%. Across 100 trajectories, 52 of 55 strategy-switch prompts were followed by another tool call from the same agent, and all 69 explicit forced stops were attributed to budget exhaustion. This aggregate sensor analysis extends the runtime evidence beyond the three intervention examples in §5.4 (L421-L454).

The first-attempt results establish a WideSearch Item-F1 advantage before current-question skill generation, and the middleware ablations quantify losses in complete-record quality when components are removed. The source-support audit further examines whether retained sources substantiate the answers. We ask the reviewer to reconsider the recommendation in light of this evidence and the clarified evaluation protocol.

Reviewer_jekWOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes SearchOS, a multi-agent framework for long-horizon, open-domain information seeking. It addresses challenges such as lost search progress, redundant retrieval, and unproductive search loops. SearchOS formulates information seeking as relational schema completion and introduces three main components: Search-Oriented Context Management (SOCM), Search Tool Middleware Harness, and a Hierarchical Search Skill System. Pipeline-parallel scheduling is further introduced to improve coordination efficiency. Experiments on WideSearch, GISA, XBench, and GAIA demonstrate improvements in retrieval accuracy, coverage, and efficiency over existing baselines.

Critical strengths
  1. Practical system design for long-horizon information seeking. The paper addresses important challenges in multi-agent search, including progress tracking, redundant retrieval, and execution stagnation. Explicitly maintaining shared tasks, evidence, and failure records provides a practical approach to managing long-horizon search workflows.

  2. Comprehensive evaluation across multiple benchmarks. The evaluation covers four benchmarks and compares SearchOS with several single-agent and multi-agent baselines. Component ablations and efficiency analyses further strengthen the empirical evaluation.

Critical weaknesses
  1. Potentially unfair evaluation due to test-instance-specific skill generation.

Section 4.1 indicates that SearchOS first runs on each test question to generate website access skills, then reruns the same question for final evaluation. This introduces test-instance-specific preparation that may encode relevant websites, navigation paths, and retrieval strategies. Without equivalent preparation for baselines, the reported improvements may not reflect a fair comparison.

  1. Limited novelty beyond existing agent orchestration frameworks.

Most components of SearchOS, including shared state management, task scheduling, evidence tracking, runtime monitoring, and reusable skills, have been explored in existing multi-agent and agent harness frameworks. The contribution appears primarily to be system integration, while the technical novelty of the proposed "Operating System" abstraction remains unclear.

  1. Insufficient control over computational budgets and additional model usage.

Although GLM-5 is used as the backbone model, SearchOS additionally employs Qwen3.5-35B-A3B for evidence extraction and relies on preconstructed search skills. The evaluation does not sufficiently account for these additional costs, including test-instance-specific skill generation and repeated execution. It remains unclear whether the improvements hold under matched end-to-end computational budgets.

  1. Need more ablations of middleware and system-level interventions.

While several components are ablated, the independent contributions of key middleware mechanisms remain unclear. In particular, the effectiveness of Evidence Extraction Middleware and Sensor Middleware is not systematically quantified. The qualitative examples in Section 5.4 are insufficient to establish their overall impact.

Questions for authors
  1. Could the evaluation clarify how test-instance-specific access skills are generated and whether baselines receive equivalent preparation? How does SearchOS perform without such preparation and under matched end-to-end budgets, including skill generation, additional model calls, and repeated execution?

  2. What are the key technical innovations of SearchOS compared with Table-as-Search, Web2BigTable, and existing agent harness frameworks? Could additional ablations isolate the contributions of Evidence Extraction Middleware and Sensor Middleware?

  3. How does SearchOS identify undiscovered entities when Coverage only measures already discovered records? Could the effectiveness of scope auditing be quantified through false-completion rates, entity recall, and final answer completeness?

5 comments

We appreciate the reviewer's recognition of the practical problem and the evaluation across four benchmarks.

We strongly disagree with the characterization of our contribution as primarily system integration. Several criticisms reflect an insufficiently careful reading of the submission and give inadequate consideration to the authors’ completed experimental work. The paper evaluates six baselines across four benchmarks (§4.1, L322-L352) and includes dedicated analyses of SOCM, Failure Memory, schema planning, scheduling, and skills (§5.1–§5.5, L373-L468). We formally ask the reviewer to assess the paper’s motivation, problem formulation, and methodological design together with the reported experiments. Selectively overlooking these foundations and reducing the assessment to experimental fault-finding misrepresents the scope of the research contribution, and is neither professional nor reasonable.

W1. “Potentially unfair evaluation due to test-instance-specific skill generation.”:

Skill generation and use are explicit contributions of the method (§3.4, L294-L315). Access skills extract information from repeatedly visited websites, reducing repeated search–open loops. Prior work also constructs reusable knowledge from execution experience. Our reproductions retain A-MapReduce’s experience-guided planning and Web2BigTable’s learned skill banks:

MethodReusable skills or experienceRetained mechanism
A-MapReduceExperience-memory insights (insights.json) and positive/negative examples guiding task decomposition, output templates, and batchingRetrieval of previously collected experience during planning
Web2BigTabletask-router and decompose-* skills for decomposition by entity, time, category, rank, or dependency. Worker skills for retrieval, evidence verification, and synthesisPre-generated orchestrator and worker skill banks with skill retrieval
SearchOSOrchestrator playbooks, search-strategy skills, and executable website-access skillsStrategy initialization from Web2BigTable, plus trajectory-based generation, validation, and use of access skills

The fairness assessment must account for the first-attempt results, obtained before generating skills from the current question. These are the no-skill results in Table 5 (L456-L466), reproduced alongside GISA below:

SettingWideSearch Item F1WideSearch Row F1GISA Table Item F1GISA Table Row F1GISA Set F1GISA List F1GISA Item EM
First attempt (without skills)69.841.468.1744.7262.2254.9333.33
Full SearchOS73.247.668.8948.2363.2459.3527.78

Without skills, SearchOS still exceeds the highest baseline WideSearch Item F1 of 68.7 and the strongest baseline on four of five GISA metrics: Table Item F1, Set F1, List F1, and Item EM.

The WideSearch Item-F1 advantage is therefore already present on the first attempt. Our protocol evaluates the complete online skill-acquisition method. The reproductions retain each baseline's own reusable-experience workflow, as detailed above.

W2. “Limited novelty beyond existing agent orchestration frameworks.”

The contribution is a stateful execution design for collaborating search agents in open-world information collection (§1–§3, L044-L068, L137-L147). We generalize Table-as-Search’s single-table storage into task-adaptive relational schemas with primary and foreign keys, supporting multiple linked entity types (§2, L090-L098, §5.2, L399-L405). The orchestrator dispatches unresolved cells, middleware commits evidence and updates coverage, and role-specific projections expose the new state to subsequent agent decisions. Table 2 (L378-L385) measures the effect of these state interfaces while retaining the extractor.

SettingWideSearch Item PItem RItem F1Row F1
Without SOCM projection73.364.466.540.0
Without Failure Memory74.765.067.639.6
Full SearchOS79.171.273.247.6

SOCM projections improve Item F1 by 6.7 points and Row F1 by 7.6 points with the extractor fixed. The paper also tests schema modeling on 40 questions, using semantically matched fixed schemas with the same required attributes (§5.2, L399-L410). Table 3 (L387-L395) reports:

Schema settingItem F1Row F1
Fixed Single-Table47.929.2
Fixed Multi-Table58.334.2
Oracle Single/Multi62.441.2
SearchOS search-time schema planning70.648.9

The oracle selects the better fixed schema for each question by Item F1 (L407-L410). Search-time planning exceeds this oracle by 8.2 Item F1 and 7.7 Row F1 points. Table 4 (L432-L437) separately tests batch and continuous scheduling (§5.3, L414-L417).

W3. “Insufficient control over computational budgets and additional model usage.”

All methods use GLM-5 for search and coordination (§4.1, L354-L355).

We totally disagree with this review. SearchOS assigns evidence extraction to Qwen3.5-35B-A3B and persists records programmatically (§3.3.2, L261-L274). Our baseline reproductions use GLM-5 calls to process observations and produce structured outputs:

MethodAgent backboneEvidence or structured-output processing
ReActGLM-5GLM-5 reasoning and answer generation
Plan-and-SolveGLM-5GLM-5 planning, search, and answer generation
MindSearchGLM-5GLM-5 search coordination and synthesis
Table-as-SearchGLM-5GLM-5 webpage summarization and database updates
A-MapReduceGLM-5GLM-5 summarization and aggregation
Web2BigTableGLM-5GLM-5 workers and shared-workboard updates
SearchOSGLM-5Qwen3.5-35B-A3B extraction and programmatic persistence

All methods use the same evaluation-time table-reformatting pipeline on WideSearch and GISA. The execution limits are per session: 50 orchestrator iterations, 8 parallel sub-agents, and 1,800 seconds (§4.1, L357-L360). Figure 4 reports 36.6% lower scored-session time, 39.1% fewer search calls, and 42.7% fewer Jina API calls from enabling skills (L438-L448, §5.5, L467-L468). These measurements cover the scored execution. The initial search and skill construction are preparation stages (L354-L356); the first-attempt results in W1 quantify performance before this preparation. That first attempt already exceeds the strongest baseline on WideSearch Item F1.

W4. “Need more ablations of middleware and system-level interventions.”

The SOCM ablations already hold automatic extraction fixed while varying shared-state access (§5.1, L373-L385). Section 3.3 specifies the middleware mechanisms (L237-L290).

Middleware ablations on 100 WideSearch questions (Context: Table 2, L378-L385; full system: Table 1, L324-L342).

SettingItem PItem RItem F1Row PRow RRow F1
w/o Evidence Extraction Middleware73.2169.9170.9142.0040.1840.72
w/o Sensor Middleware73.1871.1371.4641.1239.8840.18
w/o Context Middleware73.2764.3966.5143.6538.7239.99
Full SearchOS79.0971.2473.1851.3446.3047.63

The context ablation produces the largest Item-F1 loss, from 73.18 to 66.51, together with a 6.85-point loss in Item recall. This supports the motivation in §1 (L044-L049): long-horizon collection benefits from explicit search state that tells agents which records and required attributes still need work. The existing SOCM control keeps the extractor fixed, directly testing the contribution of this context interface.

Evidence Extraction and Sensor Middleware have a particularly strong effect on complete records. Removing them costs 6.91 and 7.45 Row F1 points, respectively, compared with 2.27 and 1.72 Item F1 points. WideSearch counts a row as correct only when all required fields are correct. The larger row-level losses suggest that persistent evidence capture and runtime feedback help agents assemble complete, correct entity records over long searches.

We additionally examined all 100 trajectories from the run used in §5.4 (L421-L454):

Observed intervention behaviorCount
Strategy-switch prompts55 across 16 questions
Prompts followed by a further tool call from the same agent52/55
Explicit forced stops attributed to budget exhaustion69/69

We also evaluated source support across 50 WideSearch cases, checking source-linked final-table facts against archived webpages and structured source records for entity, attribute, time, and unit consistency.

Source-support auditResult
Cases audited50
Fully supported74.48%
Partially supported12.69%
Unsupported8.07%
Insufficient material4.75%
Fully or partially supported87.18%
Sufficient material for a support judgment95.25%

Percentages refer to source-linked substantive cells; the last row includes supported, partially supported, and unsupported judgments. Over 90% had sufficient saved source material for verification, and 87.18% were fully or partially supported. These completed audits will be included in the main-text analysis.

Q1. “Could the evaluation clarify how test-instance-specific access skills are generated and whether baselines receive equivalent preparation? How does SearchOS perform without such preparation and under matched end-to-end budgets …?”

The code generates access skills from completed search trajectories (§3.4, L305-L310). It groups visited URLs and tool observations by website, selects sources with repeated visits or poor extraction, probes representative pages, and builds callable retrieval code. Candidates pass interface checks and execution probes before entering the library. The evaluation then reruns the question with skills generated for up to three websites (§4.1, L354-L356).

W1 reports the first attempt, before current-question skill generation, as well as the skill-enhanced attempt (Table 5, §5.5, L456-L466). Our reproductions retain A-MapReduce's experience memory and Web2BigTable's skill banks. W3 specifies the model allocation, per-session limits, and the stages covered by the reported efficiency metrics.

Q2. “What are the key technical innovations of SearchOS compared with Table-as-Search, Web2BigTable, and existing agent harness frameworks? Could additional ablations isolate the contributions of Evidence Extraction Middleware and Sensor Middleware?”

Our core contribution is system-maintained search state and execution infrastructure for open-world information collection (§2–§3, L090-L107, L137-L147). Three mechanisms implement this principle:

  1. Relational task modeling. SearchOS generalizes Table-as-Search's single table to task-adaptive schemas with linked entity types, primary keys, and foreign keys. Section 5.2 (L399-L410) evaluates this extension against fixed single-table, multi-table, and per-question oracle schemas.
  2. Shared state that drives execution. SOCM maintains pending work, anchored evidence, cell coverage, and failed attempts. Each agent receives a role-specific view, and unresolved cells drive continuous dispatch (§3.1–§3.2, L137-L215). Web2BigTable's strategy skills are retained as initialization (§3.4, L301-L303).
  3. Middleware-enforced state transitions. Evidence Extraction Middleware commits schema-bound, source-anchored findings and updates coverage after tool execution. Sensor Middleware uses evidence and coverage changes to detect stalls and enforces resource limits (§3.3, L261-L290).

Table 2 already tests the shared-state contribution while holding the extractor fixed (§5.1, L373-L385), with results reproduced in W2. W4 additionally reports the extraction and sensor ablations: removing Evidence Extraction Middleware lowers Item/Row F1 by 2.27/6.91 points; removing Sensor Middleware lowers them by 1.72/7.45 points. The much larger row-level losses connect these mechanisms to the task's requirement for complete entity records. Section 5.4 (L421-L454) analyzes sensor interventions; W4 adds the 100-trajectory intervention analysis and 50-case source-support audit.

Q3. “How does SearchOS identify undiscovered entities when Coverage only measures already discovered records? Could the effectiveness of scope auditing be quantified through false-completion rates, entity recall, and final answer completeness?”:

Explore estimates the entity scope, and the orchestrator audits it before completion. Explore supplies schema priors (§3.2, L204-L207), including candidate entities and an estimated entity count with its evidence basis. The coverage denominator adds a term for declared empty tables (§3.1.3, L178-L184), keeping them unfinished. Newly discovered entities expand the set of schema cells.

The orchestrator prompt explicitly compares collected rows with Explore's estimate and the query's implied dimensions, including years, regions, and categories. Before synthesis, missing scope triggers further discovery tasks. The appendix already illustrates this mechanism in the Phelps case (Appendix H.2, L1130-L1138):

CheckpointKnown rowsAction
Initial cell saturation16Compare with approximately 31 eligible events implied by agent reports, audit competition and year coverage, and dispatch targeted backfill
Final result35Verify records, remove a duplicate, and correct erroneous entries

This case directly demonstrates continued entity discovery after the known cells are saturated. The orchestrator uses the query scope and Explore's estimate to identify further collection work. The final table's completeness is evaluated through the benchmark's Item and Row recall (Table 1, L324-L342). The appendix exposes the intermediate audit decision: saturation of the 16 known rows prompts a scope check, further discovery, and verification of 35 final records. This connects the completion rule to an observable search decision and the final answer evaluated by the benchmark.

Reviewer_SFAhOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper addresses long-horizon open-domain information seeking with multi-agent search systems. The authors argue that existing agentic search methods keep intermediate state implicit in the interaction trajectory, so agents lose track of established evidence, unresolved gaps, and failed attempts as trajectories grow. They reformulate the task as relational schema completion with grounded citations: the system builds a relational search schema with tables, primary keys, and foreign keys, and must populate it with cited evidence.

SearchOS has three components. Search-Oriented Context Management (SOCM) externalizes execution state into a Frontier Task, an Evidence Graph, a Coverage Map, and Failure Memory, and gives each agent a role-specific projection of this state. A Search Tool Middleware Harness intercepts model and tool calls to compose context, extract and anchor evidence into the graph, and detect stalls and budget exhaustion. A Hierarchical Search Skill System organizes orchestrator, strategy, and executable source-access skills, with access skills generated from execution trajectories. Scheduling is pipeline-parallel and event-driven rather than batched. On WideSearch, GISA, XBench-DeepSearch, and GAIA, SearchOS achieves the best or tied-best score on every reported metric, with Item F1 of 73.2 on WideSearch versus 68.7 for the strongest baseline, and 59.8 ACC on GAIA versus 56.9.

Critical strengths
  1. The relational schema formulation is a clean and useful abstraction. Making the search target explicit, with per-cell citation matrices, gives a verifiable objective and makes coverage measurable. This is a more principled framing than free-form report generation or flat table completion.
  2. The three modules address distinct failure modes, and the division is sensible. SOCM handles state sharing and staleness, the middleware harness handles grounding and control that prompt instructions cannot reliably enforce, and the skill system handles reuse across sources. The middleware design in particular, where evidence is accepted only if it both binds to the schema and anchors to a span, is a good mechanism for keeping provenance trustworthy.
  3. The ablations are informative. Removing SOCM projections drops WideSearch Item F1 from 73.2 to 66.5, and removing Failure Memory drops it to 67.6, which supports the claim that shared state is doing real work rather than being a representational flourish.
  4. The skill system has measurable benefit. The efficiency analysis shows 36.6 percent reduction in session time, 39.1 percent in search calls, and 42.7 percent in Jina calls, which is a practical result beyond accuracy.
Critical weaknesses
  1. The gains are small and unevenly distributed. On WideSearch, the improvement is 2.0 PR, 5.4 recall, and 4.5 F1, and on GISA item-level the improvement is 2.1 and 2.3 points. Given the size of the system, three modules, a skill library, and a middleware harness, the margin over a strong baseline like Web2BigTable is modest. The paper should report variance and clarify which differences are significant.
  2. No comparison to a strong open deep-research agent under equal budget. Baselines are ReAct, Plan-and-Solve, MindSearch, Table-as-Search, A-MapReduce, and Web2BigTable. Recent trained search agents, such as WebSailor, WebThinker, and Search-R1, are cited in related work but not evaluated. Since the paper's claim is that explicit state beats trajectory-based search, a trained agent with a long context is the most informative comparison.
  3. The skill library is built from the system's own trajectories, and skills are validated by interface checks and execution probes. There is no analysis of skill quality, of failure rates of generated skills, or of how the library behaves on sources it has not seen. The claim of reusable search knowledge is central and under-supported.
Questions for authors
  1. How much of the reported gain comes from the second pass with generated access skills? Can you report a protocol in which skills are generated on a disjoint task set and evaluated on held-out tasks, with baselines given an equal number of passes?
  2. Do baselines have access to an auxiliary extraction model comparable to Qwen3.5-35B-A3B? Please state the compute and model configuration for each arm.
  3. What is the run-to-run variance on GAIA and XBench? Can you report at least three runs for these benchmarks?
  4. What is the quality of generated access skills? Please report the fraction that pass validation, the fraction that later fail at execution, and the behavior of the library on unseen sources.
4 comments

We thank the reviewer for recognizing the relational formulation and the value of shared state and skills. Below, we clarify the first-attempt protocol and provide three-run variability, a paired significance test, and skill execution statistics to address the remaining concerns.

W1. “The gains are small and unevenly distributed. … The paper should report variance and clarify which differences are significant.”

WideSearch and GISA already use three-run means (§4.1, L357-L360). Below, we supplement Table 1 (L324-L342) with population standard deviations and three-run SearchOS results on GAIA and XBench. Baseline GAIA/XBench accuracies retain their reported single-run scores.

MethodWideSearch Item PItem RItem F1Row PRow RRow F1
ReAct73.79 ± 1.3263.80 ± 1.1666.27 ± 1.1448.22 ± 1.3241.64 ± 1.4743.30 ± 1.37
Plan-and-Solve77.08 ± 2.4565.83 ± 2.4868.69 ± 2.5148.65 ± 1.4841.46 ± 1.8643.27 ± 1.82
MindSearch68.52 ± 1.6852.42 ± 1.2056.06 ± 1.2137.50 ± 2.2528.33 ± 1.9530.38 ± 1.92
Table-as-Search74.21 ± 1.1764.83 ± 1.8267.01 ± 1.4245.06 ± 2.0640.30 ± 1.4641.48 ± 1.46
A-MapReduce74.14 ± 1.8965.15 ± 2.4367.25 ± 2.4643.84 ± 2.7338.75 ± 2.5340.06 ± 2.65
Web2BigTable69.33 ± 0.4064.39 ± 0.1865.20 ± 0.4048.88 ± 0.9645.57 ± 0.4646.31 ± 0.38
SearchOS79.09 ± 0.1571.24 ± 1.9573.18 ± 1.5651.34 ± 0.9646.30 ± 2.1447.63 ± 1.84
MethodGISA Table Item F1Table Row F1Set F1List F1Item EMXBench ACCGAIA ACC
ReAct66.79 ± 1.5445.92 ± 2.0849.12 ± 2.7152.27 ± 6.030.0041.045.1
Plan-and-Solve66.22 ± 1.3742.94 ± 2.4251.09 ± 1.7550.28 ± 2.3011.1141.050.0
MindSearch52.36 ± 5.9928.88 ± 5.9715.15 ± 7.4217.94 ± 7.8311.1130.041.2
Table-as-Search63.91 ± 2.8642.69 ± 4.2043.28 ± 4.8541.82 ± 6.0216.6744.053.9
A-MapReduce63.29 ± 1.5241.58 ± 1.5049.88 ± 3.2144.62 ± 4.7411.1125.041.2
Web2BigTable57.93 ± 0.7935.58 ± 0.8145.42 ± 2.2451.28 ± 3.3427.7842.056.9
SearchOS68.89 ± 2.2648.23 ± 3.5163.24 ± 1.4559.35 ± 2.6827.7849.00 ± 0.8262.09 ± 3.24

SearchOS leads in WideSearch Item F1 and Row F1 and all five GISA metrics, including one tie. Its gains over the strongest baselines reach 4.49 points in WideSearch Item F1, 12.15 in GISA Set F1, and 7.08 in GISA List F1. Q3 additionally shows that every SearchOS run on GAIA and XBench exceeds the strongest reported baseline.

The WideSearch Item-F1 improvement over Web2BigTable is statistically significant (two-sided paired permutation test, p = 0.0003). We pair the same 100 questions, average the three runs per question for each method, and use 99,999 permutations.

W2. “No comparison to a strong open deep-research agent under equal budget.”:

We disagree that the proposed trained agents offer a controlled comparison of our contribution. Open-world information collection requires persistent search-state management and system-level infrastructure (§1, L044-L068). SearchOS tracks entities, evidence, unresolved gaps, and failed attempts across concurrent workers, schedules searches around coverage gaps, and handles stalled execution with model weights held fixed.

We therefore compare against ReAct, Plan-and-Solve, MindSearch, Table-as-Search, A-MapReduce, and Web2BigTable, all using GLM-5 (§4.1, L349-L355). Together with component ablations, this design directly examines the contribution of our infrastructure. Switching to the suggested agents would simultaneously vary model scale, search-specific training, and tool protocols. Equal inference budgets leave these confounds intact. This review is not professional.

W3 & Q4. “There is no analysis of skill quality, of failure rates of generated skills, or of how the library behaves on sources it has not seen.”:

Generated access skills must pass interface checks and execution probes before library admission (§3.4, L305-L315). Candidates failing either gate are excluded. Section 5.5 already quantifies downstream utility (L456-L468), with gains of 3.4 Item F1 and 6.2 Row F1 on WideSearch. We additionally analyze all 100 questions in a separate WideSearch run:

MeasureResult
Questions invoking access skills87/100
Distinct skills invoked153
Total skill calls2,560
Calls returning an explicit error328/2,560 (12.81%)
Skills used on multiple questions29/153 (18.95%)
Returned structured candidate row occurrences2,444

The execution-error denominator is 2,560 actual invocations of admitted skills. Validation describes the pre-admission qualification procedure; the 12.81% figure measures errors observed after admission. These statistics cover the retained library and its executions.

The access-skill library spans 248 skills across six source categories:

Access-skill categorySkillsMean operations per skill
Commercial1383.70
Government364.47
Organization334.06
Education144.14
Other domains203.90
Generic/API71.00
Total2483.82

For an unseen source, SearchOS uses general search and browsing tools, guided by source-independent strategy skills. The resulting trajectory can seed a source-specific access skill, which enters the library after validation.

These completed analyses will be incorporated into §5.5 of the revised main text.

Q1. “How much of the reported gain comes from the second pass with generated access skills? … skills are generated on a disjoint task set and evaluated on held-out tasks, with baselines given an equal number of passes?”

First, skill generation and use are themselves part of SearchOS’s methodological contribution (§3.4, L294-L315). Prior work also constructs reusable knowledge from execution experience. Our reproductions retain A-MapReduce’s experience-guided planning and Web2BigTable’s learned skill banks, as summarized below:

MethodReusable skills or experienceRetained mechanism
A-MapReduceExperience-memory insights (insights.json) and positive/negative examples guiding task decomposition, output templates, and batchingRetrieval of previously collected experience during planning
Web2BigTabletask-router and decompose-* skills for decomposition by entity, time, category, rank, or dependency. Worker skills for retrieval, evidence verification, and synthesisPre-generated orchestrator and worker skill banks with skill retrieval
SearchOSOrchestrator playbooks, search-strategy skills, and executable website-access skillsStrategy initialization from Web2BigTable, plus trajectory-based generation, validation, and use of access skills

Second, the no-skill results below are the first-attempt outputs, before generating access skills from the current question. The skill-enhanced attempt is reported alongside them (§5.5, Table 5, L456-L466). This existing ablation should be taken into account; overlooking it reflects insufficiently careful reading.

WideSearch settingItem PItem RItem F1Row PRow RRow F1
First attempt (without skills)73.268.869.843.440.641.4
Full SearchOS79.171.273.251.346.347.6

These results show gains of 3.4 Item F1 and 6.2 Row F1 from enabling the skill hierarchy. SearchOS without skills already exceeds the strongest baseline’s WideSearch Item F1 of 68.7.

The additional GISA evaluation shows:

GISA settingTable Item F1Table Row F1Set F1List F1Item EM
First attempt (without skills)68.1744.7262.2254.9333.33
Full SearchOS68.8948.2363.2459.3527.78

On GISA, the no-skill variant also outperforms the strongest baseline on four of five metrics, covering Table Item F1, Set F1, List F1, and Item EM.

The evaluated protocol measures online acquisition and use of website-access procedures (§4.1, L354-L356). The proposed disjoint-task, frozen-library protocol would assess transfer without further library updates. The first-attempt comparison above directly measures performance before current-question skill preparation, while the baseline reproductions retain their own experience and skill mechanisms.

Q2. “Do baselines have access to an auxiliary extraction model comparable to Qwen3.5-35B-A3B? Please state the compute and model configuration for each arm.”

Evidence extraction and persistence are explicit responsibilities of our middleware (§3.3.2, L261-L274). Qwen3.5-35B-A3B extracts schema-bound evidence; middleware commits it to persistent shared state (§4.1, L354-L355). This implements system-maintained search progress across agent interactions (§1, L044-L049). Our baseline reproductions use GLM-5 for evidence processing and structured outputs:

MethodAgent backboneEvidence or structured-output processing
ReActGLM-5GLM-5 reasoning and answer generation
Plan-and-SolveGLM-5GLM-5 planning, search, and answer generation
MindSearchGLM-5GLM-5 search coordination and synthesis
Table-as-SearchGLM-5GLM-5 webpage summarization and database updates
A-MapReduceGLM-5GLM-5 summarization and result aggregation
Web2BigTableGLM-5GLM-5 workers and shared-workboard updates
SearchOSGLM-5Qwen3.5-35B-A3B extraction and programmatic persistence

All methods use the same evaluation-time table-reformatting pipeline on WideSearch and GISA. Assigning extraction to Qwen frees the search agents from repeated database-update calls.

With the extractor fixed, SOCM improves precision by 5.8 points and recall by 6.8 points (§5.1, Table 2, L373-L385):

SettingItem precisionItem recall
Without SOCM projection73.364.4
Full SearchOS79.171.2

This controlled ablation directly supports the contribution of shared-state access.

The experimental limits apply per search session (§4.1, L357-L360). Figure 4 reports scored-session time and search/Jina API usage (L438-L448). Evidence extraction runs within search execution; the initial attempt and skill construction are preparation stages specified in L354-L356. The first-attempt results in Q1 establish the WideSearch Item-F1 advantage before that preparation.

Q3. “What is the run-to-run variance on GAIA and XBench? Can you report at least three runs for these benchmarks?”

We now report three runs on both benchmarks. Accuracy is reported in percent; SD uses the population definition (ddof = 0), consistent with W1.

BenchmarkRun 1Run 2Run 3Mean ± SD
GAIA59.8059.8066.6762.09 ± 3.24
XBench48.0049.0050.0049.00 ± 0.82

Every run exceeds the strongest reported baseline on its benchmark (GAIA: 56.9%; XBench: 44.0%).

We ask the reviewer to reconsider the assessment in light of the first-attempt results, the statistically significant WideSearch Item-F1 improvement over Web2BigTable, and the component evidence above. These findings support the paper's central contribution: system-maintained search state and execution infrastructure for open-world information collection.

AuthorsOpenReview ↗

A Kindly Reminder

Dear Reviewer,

With only one day remaining in the discussion period, we would sincerely appreciate your feedback on the additional evidence in our rebuttal.

We understand that this is a busy period and appreciate the time and effort you have devoted to our work. In response to your concerns, we have provided three-run GAIA and XBench results, variance and significance analyses, and skill execution and reuse statistics. We have also clarified first-attempt performance, baseline preparation, and model configurations.

Even a brief response indicating which concerns these results address and which remain would be very helpful. This would give us an opportunity to clarify unresolved points before the discussion closes.

We remain available for further questions and would greatly value your assessment. Thank you again for your time and consideration.

Comments on the paper

3 comments

We thank the reviewers for recognizing several strengths of SearchOS. SFAh highlights the principled relational formulation, complementary modules, informative state ablations, and measurable skill efficiency. jekW recognizes the practical shared-state design and comprehensive evaluation across four benchmarks. PV4C recognizes the value of progress and failure tracking, evaluation spanning structured collection and general search, and the benefits of shared context, failure memory, and scheduling.

Our core contribution is a stateful execution design for open-world information collection: relational schemas define the search objective, persistent shared state drives agent coordination, middleware commits grounded evidence and handles stalls, and hierarchical skills supply reusable search knowledge.

To address the remaining concerns, we supplement the submission with the following results and analyses:

ReviewerSupplementary experiments and analyses
SFAh3 runs each on GAIA and XBench (6 reported runs); three-run variability on WideSearch/GISA; 1 paired significance test (p = 0.0003 against Web2BigTable on WideSearch Item F1); skill analysis covering 100 questions and 2,560 calls.
jekW3 middleware ablation settings covering 100 WideSearch questions each, compared with full SearchOS; clarification of the relational formulation, preparation protocol, and model allocation.
PV4CFirst-attempt results on 2 benchmarks; a 50-case source-support audit; a 100-trajectory intervention analysis; and the Sensor Middleware comparison.

These completed analyses will be incorporated into the revised experimental and analysis sections, together with clearer descriptions of skill preparation, model allocation, and the scope of efficiency measurements.

As the rebuttal deadline is approaching, we would appreciate the reviewers' replies while time remains for further discussion. We are happy to address any new questions or remaining concerns.

AuthorsOpenReview ↗

A Kindly Reminder

Dear Reviewers,

With only two hours remaining before the discussion closes, we are posting one final follow-up. We have not received feedback from any reviewer on our rebuttal, additional experiments, or revised manuscript, despite our previous messages.

We have added several experiments and clarification during the rebuttal period. We would sincerely appreciate your assessment of these evidences.

Even a brief reply indicating which concerns have been addressed and which remain would give us an opportunity to clarify unresolved points before the deadline.

We understand that everyone is under considerable time pressure and appreciate the effort involved in reviewing. Your participation in these final hours would be very valuable to us and would help ensure that the discussion informs the final assessment.

Thank you again for your time and consideration.

AuthorsOpenReview ↗

Final Summary: All Review Points Addressed

The problem and contribution. Long searches lose track of evidence, gaps, and failed attempts as conversations grow. SearchOS formulates open-world information collection as relational schema completion and makes persistent search state a system responsibility. Shared state drives coordination; middleware grounds evidence and controls execution; hierarchical skills enable reuse.

Our point-by-point rebuttal responds to every original review point through experiments, analyses, and protocol clarifications.

Where reviewers stand.

  • SFAh recognized the formulation, complementary modules, ablations, and skill efficiency. We supplied variability, significance, and skill analyses.
  • jekW recognized the shared-state design and four-benchmark evaluation. We clarified the contribution and supplied middleware comparisons and preparation details.
  • PV4C recognized progress tracking, context ablations, and scheduling efficiency. We supplied first-attempt and grounding/runtime results. The reviewer acknowledged our response and maintained the score without identifying remaining concerns or discussing the evidence.

Concerns and how we addressed them. ✓ = raised by the reviewer. References point to our rebuttal.

ConcernSFAhjekWPV4COur evidence and clarifications
Novelty: what does the stateful design contribute?—✓—Linked relational tables, gap-driven scheduling, and middleware-enforced updates. Existing schema controls yield 70.6/48.9 Item/Row F1, versus 62.4/41.2 with per-question oracle schemas; fixed-extractor SOCM ablations test the state interface. jekW W2/Q2.
Trained agents: comparison scope?✓——Six baselines share GLM-5 with SearchOS. Comparisons and ablations examine search infrastructure with model weights fixed. Trained agents additionally vary model scale, training, and tool protocols. SFAh W2.
Preparation: does the advantage persist before current-question skill generation?✓✓✓First-attempt results on 2 benchmarks: WideSearch Item F1 69.8 vs. 68.7 for the strongest baseline; SearchOS also leads on four of five GISA metrics. Our reproductions retain A-MapReduce's experience memory and Web2BigTable's skill banks. SFAh Q1; PV4C Q1.
Skills: execution quality, reuse, and unseen sources?✓—✓Across 100 questions, 153 admitted skills receive 2,560 calls; 12.81% return explicit errors, and 29 skills are used across questions. We summarize 248 skills in six source categories and explain validation gates and general browsing on unseen sources. SFAh W3/Q4; PV4C Q3.
Middleware: independent component effects?—✓✓3 ablation settings on 100 WideSearch questions each. Removing Context, Evidence Extraction, and Sensor Middleware lowers Item/Row F1 by 6.67/7.64, 2.27/6.91, and 1.72/7.45 points. Context/full-system results are three-run means; extraction/sensor results are single runs. jekW W4; PV4C W2/Q4.
Grounding and interventions: source support and runtime behavior?—✓✓A 50-case audit finds 74.48% fully and 12.69% partially supported source-linked substantive cells. Across 100 trajectories, 52/55 strategy-switch prompts are followed by another tool call; all 69 explicit forced stops are attributed to budget exhaustion. jekW W4; PV4C W2/Q4.
Scope: how are undiscovered entities pursued?—✓—Explore estimates entity scope; the coverage denominator keeps declared empty tables unfinished. The orchestrator compares collected rows with the estimate and query dimensions before synthesis. The appendix documents continued discovery after known-cell saturation. jekW Q3.
Variance: are gains consistent and significant?✓——We add WideSearch/GISA standard deviations and 3 runs each on GAIA/XBench: 62.09 ± 3.24 / 49.00 ± 0.82. Every run exceeds the strongest reported baseline on these two benchmarks. WideSearch Item-F1 gains over Web2BigTable are significant: paired permutation p = 0.0003. SFAh W1/Q3.
Costs: models, budgets, and accounting scope?✓✓✓We specify backbone and extraction models, baseline evidence processing, and common per-session limits. Efficiency measurements cover scored sessions; initial attempts and skill generation are described separately. SFAh Q2; jekW W3; PV4C Q2.

These results support first-attempt competitiveness and quantify the contribution of persistent state, middleware, and skills. These analyses will be consolidated in the revised manuscript.

We respectfully ask the AC, SAC, and Program Chairs to independently assess this evidence in the final evaluation. We welcome further questions and thank all reviewers and chairs for their time and effort.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.