acceptodds
Under review as a conference paper at ICLR 2027

OmniMaster: Global Relations as the Unit of Verification in Deep Research

Abstract

Deep Research increasingly relies on iterative search, evidence verification, and cross-source synthesis. Yet locally correct evidence does not guarantee a valid research result, since relations across evidence, sources, and outputs may still fail. We identify a failure mode, Relational Verification Collapse (RVC), in which relational requirements are weakened during decomposition, judgment, or aggregation, allowing local checks to pass while the underlying relation remains invalid. To study this problem, we introduce OmniMaster-Bench, which evaluates seven global relation types in two complementary settings, long-horizon open-web research and controlled multi-hop retrieval. Evaluations of frontier models reveal substantial remaining headroom, while strong performance on existing Deep Research benchmarks does not reliably transfer to global relational capability. We further develop relation-centered data synthesis and agent training. Relation-aware mid-training (MT) learns atomic capabilities for tracking relational states, while relation-preserving supervised fine-tuning (SFT) composes these capabilities into complete research trajectories. Without reinforcement learning, the resulting agents enter the frontier range on OmniMaster-Closed, narrow the gap on OmniMaster-Open, and remain competitive on established Deep Research benchmarks, showing that global relations can be explicitly learned through relation-centered supervision.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
deep research

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_J2gAOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The paper focuses on the importance of global relations in deep research workflows. It claims to identify a phenomenon it calls ‘Relational Verification Collapse’, where checking only individual (local) facts can misrepresent a deep research process as correct even though relations between individual facts or entities are in fact invalid. The paper then introduces a benchmark with two components, OmniMaster-Closed and OmniMaster-Open, to measure to what extent current frontier models maintain correct relations. It identifies significant room for improvement even in frontier models and proposes two training strategies to close this gap: relation aware mid-training and relation-preserving supervised fine-tuning. Together, these two training stages allow an open source model to become competitive with frontier models on the newly introduced benchmark and to perform well on five additional deep research benchmarks.

Critical strengths
  • The problem is interesting and potentially impactful for several research areas, i.e. can current agents/ reasoning models handle multiple (complex) relations, especially during long-horizon tasks such as deep research. Formalizing this problem and measuring its prevalence with a purpose-built benchmark is a potentially significant and useful contribution.
  • Investigating to what extent existing mid- and post-training techniques can potentially improve model performance in these settings is also useful and could help distinguish between tasks that are just naturally hard because of the (number of) relations involved and tasks that even current medium-sized models can eventually learn to solve.
Critical weaknesses
  • Structure and presentation: The paper in its current form is very hard to read and follow. New terminology is regularly introduced without explanation or formal definition and it is often not clear what constitutes a new concept and what should simply be seen as a synonym for a previously used term (see examples under ‘Questions For Authors’). This isn’t helped by the fact that the paper tries to do a lot: it introduces a new benchmark to measure ‘Relational Verification Collapse’, although it never really becomes clear why this is called a ‘collapse’, it introduces two particular instances of continued pre-training and supervised fine-tuning targeted specifically at correctly handling relations during deep research, and it finally evaluates both general frontier models and purpose-trained models on its own benchmark and existing deep research tasks. This leads to the paper not covering several key details in the main text, including the motivation behind the atomic supervision presented in section 4.1 and the notion of ‘relational complexity’ in section 3.3. It also leads to multiple statements appearing without (sufficient) context or justification. For example, in section 3.2, “simply adding several independent constraints does not require global relational reasoning” Why not? A few examples or an earlier definition of ‘relational reasoning’ would go a long way here. Similarly in section 5.2: “The main difficulty therefore lies in how relations interact with the evidence and retrieval structure of a task” Why? The two sentences directly prior to this do not seem to provide sufficient evidence for this kind of conclusion. Finally, I would encourage the authors to revisit figures 1 and 2: at least to me, these figures are so crowded with information that it becomes difficult to identify a key message, and several components (e.g. the loss functions in figure 2) seem to effectively duplicate the main text.
  • Empirical evidence: several key details and evaluations seem to be missing. For example, what are the ‘held-out atomic tasks’ presented in table 2? How is accuracy measured here? The accompanying text refers the reader to section 4.1, which in turn refers to appendix B.1, but that section only describes “local prediction problem[s]” without any examples or information on what options the model has for these problems. Further, in table 3, the paper does not report the effectiveness of SFT applied directly to the base model. That makes it difficult to understand to what extent mid-training really contributes to the model’s success here. The limited choice of backbones trained here also does not help - how large/ strong does the base model need to be for relation-preserving SFT to be effective? Finally, uncertainty measures seem to be (almost?) entirely absent from the paper. Training multiple seeds for tables 2 and 3 or providing mean and standard deviations over multiple evaluations for figure C.1 would have been valuable (especially for the ‘Open’ part of the benchmark in figure C.1, where clear trends are often hard to identify).
Questions for authors
  • Section 3.1: how are research units defined? What makes them ‘heterogeneous’?
  • Section 3.2: “multiple relations must form genuine dependencies within the same research process” What makes dependencies ‘genuine’? Can you provide examples with dependencies that are genuine and dependencies that are not?
  • Section 3.3: what are multi-hop dependencies here and how are they related to relational constraints?
  • Also section 3.3: what is ‘relational complexity’ here? How is it defined/ what makes some relations more ‘complex’ than others? The paper’s central measure of task difficulty seems to be the number of relations. Can the authors explain how their work relates to existing notions of relational reasoning and complexity, e.g. in [1]? The existing section on related works largely ignores this entire field of research.

[1] Fesser, L., Ektefaie, Y., Fang, A., Kakade, S.M. & Zitnik, M. (2026). Evaluating Relational Reasoning in LLMs with REL. Proceedings of the 43rd International Conference on Machine Learning.

9 comments
AuthorsOpenReview ↗

Response to Reviewer J2gA (Part 1/4)

Thank you for your kind support and valuable comments. We sincerely hope our rebuttal adequately address your concerns. If so, we would deeply appreciate it if you could consider raising your score. If not, please let us know your further concerns, and we will continue actively responding to your comments.

W1. Structure and presentation

We appreciate this concern. We agree that the presentation should be easier to follow, and address each point below in the order raised.

1. A single, clearer research narrative

  • The paper asks one question: do Deep Research agents preserve requirements that hold jointly across evidence? OmniMaster-Bench measures this capability, relation-centered synthesis builds tasks that require it, and MT and SFT test whether it can be learned. We have stated this chain in the Introduction and removed overlapping material in Sections 3–5.

2. Define new terms, especially Relational Verification Collapse

  • We call it a “collapse” because a joint requirement is reduced to separate checks that can all pass while the requirement itself fails. For example, two revenue figures can each be correctly sourced yet refer to different fiscal years. The effect is measurable: given the same gold evidence, local checks accept 32.0% of relation-invalid answers, versus 0.68% under joint checks (Table E2). RVC is described in the Introduction (Page 1) and formalized in Equation (3), Section 3.1 (Page 4); we have added this explicit definition at first use. Table J0 lists every term and marks which are distinct concepts rather than synonyms.

3. Explain why the four atomic MT tasks are needed

  • MT is needed because a full trajectory supervises all decisions at once and gives no direct signal for the individual state updates that relational research depends on. The four tasks are exactly these updates: IE extracts grounded evidence, CS preserves it over a long context, GR identifies the relations a task requires, and NS selects the unresolved relation that drives the next search. The results support this design. MT raises all four held-out atomic accuracies on both backbones (on 35B, NS 23.6 → 74.9 and CS 6.7 → 63.6; Table J1), and with identical SFT trajectories MT + SFT reaches 62.7 on Closed versus 51.0 for SFT alone (+11.7; see W2, point 3). We will move this motivation to the start of Section 4.1 (Pages 6–7).

4. Clarify “relational complexity”

  • In Section 3.3, “relational complexity” means the number and composition of relation requirements in a task (one to seven of the seven types), not an intrinsic complexity score. We do not claim that all relation types are equally hard; per-type differences are moderate (Appendix C.1). We have replaced the term with “relation count and composition” in Section 3.3 (Page 6) and distinguished it from the arity-based complexity of REL (see Q4).

5. Justify the claim about independent constraints

  • Independent constraints can each be checked in isolation: satisfying one never changes the evidence needed for another. “Headquartered in Germany” and “founded after 2000” are two separate lookups. By contrast, fixing the fiscal year determines which reports are admissible before two revenue figures can be compared, so the second check cannot be done correctly without the first. This cross-step dependence is what we mean by global relational reasoning. Our original sentence was too categorical; we have revised it in Section 3.2 (Page 5) and added both examples.
AuthorsOpenReview ↗

Response to Reviewer J2gA (Part 2/4)

6. Correct the interpretation of task difficulty

  • We agree that the sentence overstated the evidence, and we have narrowed it. The data show that relation count alone does not predict performance: Open has no monotonic trend and Closed shows no consistent decrease with k (Figure C.1). They do not isolate the interaction with retrieval structure as the cause. The new evidence-access analysis is consistent with a retrieval-structure effect, since 98.5% of incorrect Closed answers miss at least one gold document (Table E3), which we present as supporting evidence only. The revised Section 5.2 (Page 9) states the narrower observation.

7. Simplify Figures 1 and 2

  • Each figure will carry one message. Figure 1 (Page 2) will show only locally correct evidence yielding an invalid relation; Figure 2 (Page 4) will show only the construction and training workflow. The loss formulas duplicated in Figure 2 and minor implementation details will be removed from the figures.

Table J0. Definitions and distinctions to make explicit in the revised manuscript.

TermDefinitionExample
Research unitClaim, evidence passage, source, entity, or output itemA numerical claim and the report that supports it
Global relationJoint requirement over two or more research unitsTwo values must share a comparable reporting basis
Relational Verification Collapse (RVC)Local checks pass although a required joint relation is invalidBoth values are sourced, but refer to different fiscal years
Genuine dependencyResolving one requirement changes a later retrieval, interpretation, or decisionFixing a reporting year determines which source to compare
Independent constraintsConditions that can be checked without one result determining the otherA firm is in Germany and was founded after 2000
Multi-hop dependencyA retrieval path where an intermediate finding determines a subsequent lookupAn intermediate entity found in document A identifies document B
Relation countNumber of instantiated relation requirements in a taskA task requiring R1, R2, and R5 has count 3
Relational complexityUsed descriptively in our paper; not the higher-arity metric of RELReport relation count and composition, rather than asserting a universal complexity score
Evidence / relation stateEvidence gathered so far and relations already resolved or still openIE and CS track evidence; GR and NS track relations

W2. Empirical evidence

Thank you for identifying the missing details and controls. We have added the task specifications, an SFT-only control, and repeated evaluations, and answer each point in order.

1. What are the held-out atomic tasks?

  • The held-out atomic tasks are 196 single-step questions, disjoint by question ID from the MT data: 39 GR, 54 NS, 53 IE, and 50 CS. Each is open-ended generation under a fixed JSON schema, not multiple choice. GR receives a research request and returns the required relation types and checks. NS receives the request with its resolved and unresolved relations and returns the next relation to address plus one search query. IE receives a document and returns a verbatim supporting sentence with its provenance, or states that the evidence is absent. CS receives a multi-turn evidence stream and returns sourced confirmed, rejected, and pending items.
AuthorsOpenReview ↗

Response to Reviewer J2gA (Part 3/4)

2. Exactly how is atomic accuracy measured?

  • Accuracy is the fraction of outputs that satisfy task-specific reference criteria. Each question is sampled three times at temperature 0.8 and judged by Claude Opus 4.8 at temperature 0. GR is correct if it covers the required relation types with actionable checks; NS if it selects the correct unresolved relation and a useful query without answering the question; IE if it quotes the supporting source text, or reports “not found” on a negative case; CS if it retains the grounded key facts with sources and invents nothing. Judge abstentions are excluded from the denominator. The inputs, targets, and scoring criteria of each task have been added to Appendix B.1 (Page 18), with a summary in Section 5.3 (Page 9); one full input–output example per task will be added.

3. Does MT add value beyond identical SFT trajectories?

  • Setup. We follow the Table 3 experiment and change one factor. Both runs start from the same Qwen3.5-35B-A3B backbone and are fine-tuned on the same 10,914 OmniMaster trajectories with the same SFT hyperparameters; the only difference is whether the 1,040-example MT stage precedes SFT. Evaluation follows Section 5.1: Open is scored by the acceptance-item judge, and Closed by exact match on 150 tasks answered twice. Result. MT raises Closed from 51.0 to 62.7 (+11.7), well above the evaluation SD of 1.4; Open is unchanged within noise (46.1 vs. 46.7; Table E4). Why. MT trains the evidence- and relation-state updates separately before SFT composes them, and Closed tasks depend most on tracking these states along a multi-hop evidence chain. Implication. The gain is not explained by the trajectories alone, which supports the role of relation-aware MT. We have added the SFT-only row to Table 3.

4. What do the backbone experiments establish?

  • The method is effective at every scale we tested. MT raises all four atomic accuracies on both 9B and 35B (Table J1); for example, NS improves from 25.2 to 61.3 on 9B and from 23.6 to 74.9 on 35B. End to end, OmniMaster-27B and OmniMaster-35B both reach the frontier range on Closed (61.7 and 62.7 without RL; Tables R1 and R2). We did not run end-to-end SFT below 27B, so we do not claim a minimum effective size, and we state this scope in Section 5.3.

5. How stable are the reported gains?

  • We now report mean and sample standard deviation over repeated evaluations. Table J1 adds ± to the original Table 2 (five runs), Table J2 adds ± to the original Table 3 (five runs), and Table J3 reports three runs of the final checkpoints. The gains are far larger than the variation: for example, Closed rises from 21.1 ± 1.6 (Base) to 67.3 ± 1.8. These are repeated evaluations of the same checkpoints; retraining with multiple seeds was not feasible within the rebuttal period. Several entries differ slightly from the single-run values in the submission, and the direction of every comparison is unchanged. Figure C.1 now reports mean ± SD over three evaluations, with 21–22 tasks per relation count. On Open, differences between adjacent relation counts are within evaluation variation, which matches the reviewer’s observation; we will show the error bars and avoid reading a trend into adjacent points.

Table J1. Held-out atomic task accuracy (%).

ModelGRNSIECS
Qwen3.5-9B16.4 ± 2.425.2 ± 2.333.9 ± 2.617.1 ± 2.2
+ MT26.5 ± 2.961.3 ± 3.189.3 ± 1.840.7 ± 3.0
Qwen3.5-35B-A3B9.4 ± 1.623.6 ± 2.314.0 ± 2.06.7 ± 1.4
+ MT24.8 ± 2.874.9 ± 2.793.1 ± 1.563.6 ± 1.6
Reviewer_EgicOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies Relational Verification Collapse (RVC), where individual facts can be correct while the relationships between them are invalid. It introduces OmniMaster-Bench, with open-web and controlled retrieval tasks covering seven relation types. It also proposes relation-aware mid-training followed by trajectory-level supervised fine-tuning.

Critical strengths
  1. Correct facts do not always support a correct conclusion. The paper makes this distinction explicit and studies practical issues such as mismatched dates, measurement units, scopes, and sources.
  2. The Open and Closed settings provide complementary tests. Evidence checks, trajectory filtering, and human spot checks strengthen the benchmark. The paper also reports useful training results and failure analyses.
Critical weaknesses
  1. The Open scoring prompt allows claims that cannot be checked against sources to be judged by completeness, URLs, and internal consistency. A plausible answer could therefore receive credit even when its sources do not support it. Agreement between two judges does not resolve this issue. The paper also does not directly measure how often existing verifiers accept locally correct but globally invalid answers. Many reported failures involve missing evidence, making it difficult to separate relational failures from general retrieval failures.

  2. Table 3 lacks an SFT-only baseline using the same trajectories. Without this control, the gains could come mainly from high-quality research trajectories and better task completion, rather than relation-aware mid-training. A matched generic-supervision baseline would also help establish the value of explicit relational supervision.

Questions for authors
  1. Does Open scoring verify claims against retrieved evidence beyond the published prompt? Can a source-grounded audit show whether it reliably detects relation violations?
  2. Can you compare MT+SFT with SFT-only using the same trajectories and SFT budget?
  3. Can you directly test existing verifiers on answers whose local facts are correct but whose global relationships are invalid?
6 comments
AuthorsOpenReview ↗

Response to Reviewer Egic (Part 1/3)

Thank you for your kind support and valuable comments. We sincerely hope our rebuttal adequately address your concerns. If so, we would deeply appreciate it if you could consider raising your score. If not, please let us know your further concerns, and we will continue actively responding to your comments.

W1. Verification reliability and failure attribution (Q1 and Q3)

We appreciate this concern. We answer in three parts: (1) why Open is scored on the final deliverable, (2) how our verification procedure compares with alternatives, and (3) the empirical evidence on judge stability, source support, and failure attribution.

1. Evaluation method: Open scoring checks the final state of the deliverable (Q1)

Thank you for raising this point. We first clarify the design of Open scoring and its scope.

  • Open is scored on the final state that the agent reaches, which is the general practice for tasks that admit many valid solutions. Each of the 150 Open tasks carries task-specific acceptance items written as binary predicates, and the judge labels every item PASS, FAIL, or NA on the final deliverable (Section 5.1, Page 7; Appendix E). The score therefore depends on whether the required outcome is achieved, not on which search path or which sources the agent used. The scoring is reproducible: a second judge using the same prompt agrees on 84.4% of 1,292 acceptance items (Cohen’s κ\kappa = 0.68) and yields a model ranking with a Spearman correlation of 0.943 (part 3).
  • Scoring against one reference solution would create false negatives. An open-web research request can be satisfied by different sources, entities, and report structures, so a fixed gold answer or gold source list would reject valid deliverables. For this reason Open uses acceptance items, while Closed, where every task has a unique answer, is scored by exact match over 300 responses (150 tasks, each answered twice) without an LLM judge. The two settings are complementary, and the central claim does not rest on the Open judge alone.
  • To answer Q1 directly: the judge does not reopen cited pages beyond the published prompt. It requires every claim to carry a concrete source URL and fails items that are missing, unsourced, or self-contradictory. Open credit therefore certifies a complete, sourced, and internally consistent deliverable; it is not a certificate of source support. Part 3 tests how the judge responds to relation violations: it accepts 90.9% of the original deliverables and 45.5% of relation-invalid versions.

2. Verification procedure: comparison with alternative checks (Q3)

Thank you for suggesting this direct test. We agree that the question is important, and the manuscript already shows the problem in practice: relation and calibration failures remain in 7.3%–14.3% of evaluable Open outputs for OmniMaster, GPT-5.6 Sol, and DeepSeek-V4-Pro (Table D.2). We therefore ran a controlled test that holds the evidence fixed and varies only the checking rule.

  • Setup. For each Closed task we pair the correct answer with an answer whose individual facts are supported but which violates one required relation (150 tasks). The same verifier (Claude Opus 4.8) receives the same gold evidence under two instructions: local-only, which checks each fact against its evidence, and joint, which additionally checks the required relations. Result. Local checks accept 32.0% of relation-invalid answers; joint checks accept 0.68%. Acceptance of correct answers changes from 91.8% to 87.0% (Table E2).
AuthorsOpenReview ↗

Response to Reviewer Egic (Part 2/3)

  • Trend. The gap holds for every relation type: local checks accept 16.7%–39.3% of invalid answers per type, while joint checks accept none in six of seven types, with one R7 case remaining (Table E2b). Why. Every fact in an invalid answer is individually supported, so item-wise checks have nothing to reject; the error exists only in the relation between facts. Because the model and the evidence are fixed, the difference comes from the verification rule itself. Implication. This is direct evidence for RVC. We have added this experiment to Section 5.2.
  • The result holds across verifier models. We repeated the test with GPT-5.6 Sol and DeepSeek-V4-Pro on the same tasks, evidence, and prompts. Local checks accept 18.6% and 41.2% of relation-invalid answers; joint checks accept 0.0% and 0.8%. Acceptance of correct answers stays high under joint checks (92.7% and 91.4%, versus 94.8% and 92.6% under local checks). All three verifiers therefore show the same failure under local checking and the same correction under joint checking.

Table E2. Acceptance of correct and relation-invalid answers (%).

Answer typeLocal checksJoint checks
Correct answers91.887.0
Globally invalid answers32.00.68

Table E2b. Acceptance of relation-invalid answers by type (%).

RelationLocal checksJoint checks
R1 Temporal Alignment39.20.0
R2 Measurement Comparability16.70.0
R3 Scope Alignment18.20.0
R4 Source Authority39.30.0
R5 Evidence Traceability16.70.0
R6 Absence Verification38.10.0
R7 Cross-Item Distinctness16.78.3

3. Empirical evidence: judge stability, sensitivity to relation violations, and failure attribution

We agree that agreement between two judges does not by itself establish source correctness, and that a wrong final answer is not by itself a relational error. We therefore report three kinds of evidence.

  • The Open ranking is stable across judges. Rescoring the six frontier models with DeepSeek-V4 gives a Spearman correlation of 0.943 with the ranking produced by Claude Opus 4.8 (Section 5.1; Appendix B.2). At the level of individual acceptance items, DeepSeek-V4-Pro with the same judge prompt agrees with Claude Opus 4.8 on 84.4% of 1,292 items (Cohen’s κ\kappa = 0.68), which is substantial agreement.
  • The Open judge distinguishes valid from relation-invalid deliverables. We built 33 matched pairs from credited Open deliverables: the invalid version differs from the original by one to three string-level edits that break exactly one required relation, with all other text and URLs unchanged; GPT-5.6 Sol independently checked each construction. The original Open judge accepts 90.9% of the original deliverables and 45.5% of the relation-invalid versions, so its acceptance drops by half even for these minimal edits. Strict relational validity is measured by Closed and by the joint checks in part 2, and we will state this division of roles in Section 5.1.
  • The relational claims do not rest on the Open judge. Closed is scored by exact match against unique gold answers, and the paired tests in part 2 are run on gold evidence.
  • Retrieval failures are separated from relational failures. Among 199 incorrect Closed answers, 80 (40.2%) opened no gold document, 116 (58.3%) opened some but not all, and 3 (1.5%) opened all (Table E3). Most errors therefore involve incomplete evidence, and we do not count them as relational failures.
AuthorsOpenReview ↗

Response to Reviewer Egic (Part 3/3)

  • Relational failure is measured by the paired test in part 2, where gold evidence removes retrieval: local checks still accept 32.0% of relation-invalid answers. We report both analyses in Section 5.2 (Page 9) and Appendix D.2, and have revised the text so that the two failure sources are not conflated.

Table E3. Evidence-access categories among 199 incorrect Closed answers.

Error typeDefinitionShare
No gold document openedNo required gold evidence document opened before answer40.2%
Some, but not all, openedAt least one gold document opened; others still missing58.3%
All gold documents openedThe entire required evidence set opened before wrong answer1.5%

W2. Contribution of MT beyond SFT trajectories (Q2)

Thank you for requesting these two controls. We agree that they are necessary: without them, the gains could come from high-quality trajectories rather than from relational supervision. We ran both, keeping every other factor fixed.

1. Same-trajectory control: MT versus SFT only

  • Setup. We follow the Table 3 experiment and change one factor. Both runs start from the same Qwen3.5-35B-A3B backbone and are fine-tuned on the same 10,914 OmniMaster trajectories with the same SFT hyperparameters; the only difference is whether the 1,040-example MT stage precedes SFT. Evaluation follows Section 5.1: Open is scored by the acceptance-item judge, and Closed by exact match on 150 tasks answered twice. Result. MT raises Closed from 51.0 to 62.7 (+11.7), well above the evaluation SD of 1.4; Open is unchanged within noise (46.1 vs. 46.7; Table E4). Why. MT trains the evidence- and relation-state updates separately before SFT composes them, and Closed tasks depend most on tracking these states along a multi-hop evidence chain. Implication. The gain is not explained by the trajectories alone, which supports the role of relation-aware MT.

2. Generic trajectories: test relation-centered data

  • Setup. We replace the OmniMaster trajectories with the same number (10,914) of generic research trajectories sampled from the public Step-3.5-Flash-SFT collection and train the same backbone with SFT only with the same SFT hyperparameters; the evaluation is identical. Result. Generic SFT reaches 37.1 on Open and 26.0 on Closed, versus 46.1 and 51.0 for OmniMaster SFT (+9.0 and +25.0; Table R1). Why. Generic trajectories teach search and report writing but do not supervise the relations that must hold across evidence; the gap is largest on Closed, where one violated relation makes the answer wrong. The two sets differ in content and difficulty, so this is an equal-count control. Implication. Explicit relational supervision, not trajectory volume, accounts for the gains.

Table E4. MT ablation using the same OmniMaster SFT trajectories (%).

MetricSFT onlyMT + SFTDifferenceEvaluation std
Open46.146.7+0.61.8
Closed51.062.7+11.71.4
Reviewer_ReZYOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces OmniMaster, a framework for studying and improving global relational reasoning in Deep Research. It proposes OmniMaster-Bench, consisting of open-web and controlled-retrieval tasks covering seven relation types, and trains research agents through relation-aware mid-training followed by supervised fine-tuning on verified trajectories. It is always a hard work to generate high-quality data trajetories.

Critical strengths
  1. The paper addresses a practically relevant problem: individually correct facts can produce invalid conclusions when temporal, measurement, scope, or provenance requirements are not preserved.
  2. The data construction pipeline includes evidence auditing, trajectory filtering, shortcut checks, and human spot checks.
  3. Evaluation involves actual search and browsing, with shared tools and interaction limits on OmniMaster-Bench.
  4. The paper also transparently reports performance trade-offs and residual failures.
Critical weaknesses
  1. Table 3 omits SFT-only and matched generic research-trajectory training baselines.
  2. Table 1 lacks essential comparisons. None of the comparable open research agents is evaluated on OmniMaster-Bench, while extensive missing frontier-model scores limit cross-benchmark conclusions.
  3. The final training-data mixture is insufficiently documented. How about the data quality and data distributions?
Questions for authors

I have no further question.

5 comments
AuthorsOpenReview ↗

Response to Reviewer ReZY (Part 1/3)

Thank you for your kind support and valuable comments. We sincerely hope our rebuttal adequately address your concerns. If so, we would deeply appreciate it if you could consider raising your score. If not, please let us know your further concerns, and we will continue actively responding to your comments.

W1. SFT-only and generic research-trajectory baselines

Thank you for identifying these missing controls. We agree that both are necessary to show where the gains come from, and we added them in the setting of Table 3 (Table R1).

1. Effect of relation-centered SFT data

  • Setup. We replace the OmniMaster trajectories with the same number (10,914) of generic research trajectories sampled from the public Step-3.5-Flash-SFT collection and train the same backbone with SFT only with the same SFT hyperparameters; the evaluation is identical. Result. Generic SFT reaches 37.1 on Open and 26.0 on Closed, versus 46.1 and 51.0 for OmniMaster SFT (+9.0 and +25.0; Table R1). Why. Generic trajectories teach search and report writing but do not supervise the relations that must hold across evidence; the gap is largest on Closed, where one violated relation makes the answer wrong. The two sets differ in content and difficulty, so this is an equal-count control. Implication. Explicit relational supervision, not trajectory volume, accounts for the gains.

2. Incremental effect of atomic MT

  • Setup. We follow the Table 3 experiment and change one factor. Both runs start from the same Qwen3.5-35B-A3B backbone and are fine-tuned on the same 10,914 OmniMaster trajectories with the same SFT hyperparameters; the only difference is whether the 1,040-example MT stage precedes SFT. Evaluation follows Section 5.1: Open is scored by the acceptance-item judge, and Closed by exact match on 150 tasks answered twice. Result. MT raises Closed from 51.0 to 62.7 (+11.7), well above the evaluation SD of 1.4; Open is unchanged within noise (46.1 vs. 46.7; Table R1). Why. MT trains the evidence- and relation-state updates separately before SFT composes them, and Closed tasks depend most on tracking these states along a multi-hop evidence chain. Implication. The gain is not explained by the trajectories alone, which supports the role of relation-aware MT. Both rows have been added to Table 3 in Section 5.3.

Table R1. SFT-only and generic research-trajectory comparisons (%).

ModelSFT dataOpenClosed
OmniMaster (SFT)OmniMaster / 10,91446.151.0
OmniMaster (MT + SFT)OmniMaster / 10,91446.762.7
OmniMaster (SFT)General / 10,91437.126.0

Both SFT conditions use 10,914 trajectories. The general baseline is sampled from the public Step-3.5-Flash-SFT search-trajectory collection.

W2. Open research-agent comparisons and missing scores

Thank you for requesting these baselines. We agree that open research agents must be evaluated on OmniMaster-Bench for the comparison to be complete, and that missing frontier scores limit cross-benchmark conclusions. We added both (Table R2).

AuthorsOpenReview ↗

Response to Reviewer ReZY (Part 2/3)

1. Open research-agent baselines on OmniMaster-Bench

  • Setup. We ran QUEST-35B, Tongyi DeepResearch-30B-A3B, and DR Tulu-8B on OmniMaster-Open and OmniMaster-Closed with the same tools, tool-call format, interaction limits, and scoring as the Table 1 evaluation (Section 5.1). For each agent we wrote an adapter that maps its native output format to the format required by the benchmark, so that no agent is penalized for formatting. Result. Their best scores are 17.1 on Open and 25.3 on Closed. OmniMaster-27B without RL reaches 39.3 and 61.7 (+22.2 and +36.4); with RL, OmniMaster-35B reaches 53.3 on Open (GRPO, step 120) and 70.7 on Closed (GSPO, step 120). Why. These agents are strong on the tasks they were developed for: QUEST-35B scores 69.5 on BC-Plus and 80.8 on GAIA. OmniMaster tasks are selected by a strict funnel (Closed keeps 1,469 of 5,183 candidate tasks; Table A.3), and every task requires relations to hold jointly across evidence. On such tasks these agents also fail to preserve global relations. Implication. This supports our claim that performance on existing Deep Research benchmarks does not transfer to global relational capability.

2. Unified reruns and remaining external-score limitations

  • Setup. We filled every missing frontier-model cell and updated the entries that we reran, all under one scaffold with each benchmark’s official scoring. Result. Table R2 has no missing frontier score. For open agents, a dash remains only where neither a published score nor a comparable rerun is available. Implication. Cross-benchmark conclusions now rest on complete rows. For example, GPT-5.6 Sol reaches 70.0 on GAIA and 71.4 on WideSearch but 40.5 on Open and 49.7 on Closed.

Table R2. Expanded OmniMaster-Bench and Deep Research results (%).

ModelOpenClosedBC-PlusGAIAHLEDRBWideSearch
Frontier Models
Claude Opus 550.976.320.064.356.354.167.3
GPT-5.6 Sol40.549.731.970.044.536.471.4
Kimi K359.264.011.176.569.246.358.7
Qwen3.8-Max54.060.748.070.255.336.055.2
GLM-5.350.163.78.030.042.052.782.7
DeepSeek-V4-Pro43.652.344.958.042.736.547.6
Open Deep Research Agents
QUEST-35B12.725.369.580.837.248.260.6
Tongyi DeepResearch-30B-A3B17.17.7—70.932.9——
DR Tulu-8B13.22.3———45.4—
OmniMaster
OmniMaster-27B39.361.770.876.735.446.048.1
OmniMaster-35B (MT + SFT + GRPO, step 120)53.367.368.274.834.647.240.7
OmniMaster-35B (MT + SFT + GSPO, step 120)46.370.766.573.935.946.847.6

DRB reports RACE and WideSearch reports item-level F1; other columns report accuracy. Published external open-agent results may use different protocols.

AuthorsOpenReview ↗

Response to Reviewer ReZY (Part 3/3)

W3. Final training-data mixture, quality, and distribution

Thank you for asking for the full training-data picture. Table R3 gives the composition, distribution, and quality controls of every stage.

1. Training composition and evidence quality

  • Each stage uses separate data that pass explicit quality gates. MT uses 1,040 atomic and replay examples, SFT uses 10,914 complete trajectories, and RL uses 1,000 tasks. Open candidates pass an evidence-support gate (7,805 of 8,911 retained, 87.6%). Closed trajectories pass nine programmatic filters, an independent evidence audit, a two-vote gold review, and adversarial screening. Human spot checks confirm 18 of 20 relation labels and 13 of 15 evidence chains.

2. Final-mixture distribution

  • The final SFT set has 10,914 examples: 5,696 Open and 5,218 Closed. It consists of 5,137 base trajectories upsampled to 9,619 examples, plus 1,295 additional trajectories. Within the 9,619 examples, Open has 700 for each k = 1–6 and 1,200 for k = 7, and Closed has 820 / 819 / 1,819 / 338 / 423 for k = 1–5. In Closed, the seven relation types are balanced (1,467–1,573 occurrences each) and each of the 14 academic fields contributes 260–342 examples; Open tasks spread over 764 fine-grained professional domains. We will add this distribution and the selection funnel to the appendix.

Table R3. Training stages, distribution, and quality controls.

StageSizeSource / domain distributionRelation-count distributionQuality control
MT1,040GR, NS, IE, CS: 250 each; 40 general replayAtomic supervision of the four capabilitiesDerived from evidence-verified tasks and trajectories
SFT10,9149,619 expanded + 1,295 additional; seed: Open 2,700 / Closed 2,437, Closed spanning 14 fieldsSeed Open: k = 1–6, 350 each; k = 7, 600. Seed Closed: k = 1–5, 460 / 488 / 945 / 194 / 350Open evidence gate 7,805/8,911 (87.6%); Closed nine trajectory gates, independent audit, two-vote gold review and red-team; spot checks: relation 18/20, chain 13/15
RL1,000854 combined (Closed: 14 fields; Open: persona and skill synthesis) + 146 pass@4 top-ups (124 Closed / 22 Open)pass@4 difficulty tiers k1 / k2 / k3; top-ups 49 / 50 / 47pass@4 filter in a 20-turn frozen environment (kept tasks solved in 1–3 of 4 attempts); inherits the SFT evidence and trajectory gates

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Response

Dear Area Chair and Reviewers,

Thank you for the thorough and constructive review process, which has been invaluable in strengthening our paper.

Based on the reviewers' feedback, the key strengths are as follows.

  • Reviewers recognized the novelty and importance of the problem.
    • "Formalizing this problem and measuring its prevalence with a purpose-built benchmark is a potentially significant and useful contribution," and the problem is "interesting and potentially impactful for several research areas." (Reviewer J2gA)
    • "Correct facts do not always support a correct conclusion. The paper makes this distinction explicit." (Reviewer Egic)
    • The paper "addresses a practically relevant problem." (Reviewer ReZY)
  • The benchmark and data construction were recognized as rigorous.
    • "The Open and Closed settings provide complementary tests. Evidence checks, trajectory filtering, and human spot checks strengthen the benchmark." (Reviewer Egic)
    • "The data construction pipeline includes evidence auditing, trajectory filtering, shortcut checks, and human spot checks." (Reviewer ReZY)
  • The evaluation was recognized as realistic and transparent.
    • "Evaluation involves actual search and browsing," and the paper "transparently reports performance trade-offs and residual failures." (Reviewer ReZY)
    • The paper "reports useful training results and failure analyses." (Reviewer Egic)
  • The clarity of the paper was also commended.
    • The presentation was rated "Clear and easy to follow." (Reviewers ReZY and Egic)

In response, we went beyond clarification by conducting extensive new experiments that address every major point, all incorporated in the revised manuscript.

  • Direct evidence for Relational Verification Collapse. (for Reviewers Egic and J2gA)
    • With the verifier and gold evidence fixed, local checks accept 32.0% of relation-invalid answers, versus 0.68% under joint checks; the pattern holds for three verifiers (Section 5.2).
    • The Open judge is reliable (84.4% agreement, κ=0.68\kappa = 0.68) and sensitive to relation errors (90.9% to 45.5% acceptance after one relation is broken); retrieval and relational failures are reported separately (Appendices B.2, D.2).
  • Clear evidence for relation-aware MT and relation-centered data. (for all reviewers)
    • With identical trajectories, MT raises Closed from 51.0 to 62.7 (+11.7); relation-centered data outperform generic trajectories by +9.0 on Open and +25.0 on Closed (Table 3).
    • MT lifts held-out atomic accuracy by large margins, e.g., IE from 14.0 to 93.1 and NS from 23.6 to 74.9 (Table 2).
    • All gains hold with mean ± SD over repeated evaluations (Tables 2 and 3, Figure C.1).
  • Complete comparisons and documentation. (for Reviewer ReZY)
    • Open research agents reach at most 17.1 on Open and 25.3 on Closed, versus 53.3 and 72.7 for OmniMaster; all frontier scores are filled (Table 1).
    • The composition, distribution, and quality gates of every training stage are reported (Appendix B.1).
  • Clearer presentation. (for Reviewer J2gA)
    • New terms are defined at first use, the motivation and examples of the atomic tasks are added, and REL is discussed.

After the discussion, Reviewer ReZY thanked us for "the detailed experiments and responses" and maintained a positive rating, and Reviewer J2gA thanked us for our "efforts in addressing my concerns."

We believe our comprehensive rebuttals and the significant new experimental results have addressed the reviewers' concerns. Thank you again for your time and effort.

Best,

Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.