- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The paper focuses on the importance of global relations in deep research workflows. It claims to identify a phenomenon it calls ‘Relational Verification Collapse’, where checking only individual (local) facts can misrepresent a deep research process as correct even though relations between individual facts or entities are in fact invalid. The paper then introduces a benchmark with two components, OmniMaster-Closed and OmniMaster-Open, to measure to what extent current frontier models maintain correct relations. It identifies significant room for improvement even in frontier models and proposes two training strategies to close this gap: relation aware mid-training and relation-preserving supervised fine-tuning. Together, these two training stages allow an open source model to become competitive with frontier models on the newly introduced benchmark and to perform well on five additional deep research benchmarks.
Critical strengths
- The problem is interesting and potentially impactful for several research areas, i.e. can current agents/ reasoning models handle multiple (complex) relations, especially during long-horizon tasks such as deep research. Formalizing this problem and measuring its prevalence with a purpose-built benchmark is a potentially significant and useful contribution.
- Investigating to what extent existing mid- and post-training techniques can potentially improve model performance in these settings is also useful and could help distinguish between tasks that are just naturally hard because of the (number of) relations involved and tasks that even current medium-sized models can eventually learn to solve.
Critical weaknesses
- Structure and presentation: The paper in its current form is very hard to read and follow. New terminology is regularly introduced without explanation or formal definition and it is often not clear what constitutes a new concept and what should simply be seen as a synonym for a previously used term (see examples under ‘Questions For Authors’). This isn’t helped by the fact that the paper tries to do a lot: it introduces a new benchmark to measure ‘Relational Verification Collapse’, although it never really becomes clear why this is called a ‘collapse’, it introduces two particular instances of continued pre-training and supervised fine-tuning targeted specifically at correctly handling relations during deep research, and it finally evaluates both general frontier models and purpose-trained models on its own benchmark and existing deep research tasks. This leads to the paper not covering several key details in the main text, including the motivation behind the atomic supervision presented in section 4.1 and the notion of ‘relational complexity’ in section 3.3. It also leads to multiple statements appearing without (sufficient) context or justification. For example, in section 3.2, “simply adding several independent constraints does not require global relational reasoning” Why not? A few examples or an earlier definition of ‘relational reasoning’ would go a long way here. Similarly in section 5.2: “The main difficulty therefore lies in how relations interact with the evidence and retrieval structure of a task” Why? The two sentences directly prior to this do not seem to provide sufficient evidence for this kind of conclusion. Finally, I would encourage the authors to revisit figures 1 and 2: at least to me, these figures are so crowded with information that it becomes difficult to identify a key message, and several components (e.g. the loss functions in figure 2) seem to effectively duplicate the main text.
- Empirical evidence: several key details and evaluations seem to be missing. For example, what are the ‘held-out atomic tasks’ presented in table 2? How is accuracy measured here? The accompanying text refers the reader to section 4.1, which in turn refers to appendix B.1, but that section only describes “local prediction problem[s]” without any examples or information on what options the model has for these problems. Further, in table 3, the paper does not report the effectiveness of SFT applied directly to the base model. That makes it difficult to understand to what extent mid-training really contributes to the model’s success here. The limited choice of backbones trained here also does not help - how large/ strong does the base model need to be for relation-preserving SFT to be effective? Finally, uncertainty measures seem to be (almost?) entirely absent from the paper. Training multiple seeds for tables 2 and 3 or providing mean and standard deviations over multiple evaluations for figure C.1 would have been valuable (especially for the ‘Open’ part of the benchmark in figure C.1, where clear trends are often hard to identify).
Questions for authors
- Section 3.1: how are research units defined? What makes them ‘heterogeneous’?
- Section 3.2: “multiple relations must form genuine dependencies within the same research process” What makes dependencies ‘genuine’? Can you provide examples with dependencies that are genuine and dependencies that are not?
- Section 3.3: what are multi-hop dependencies here and how are they related to relational constraints?
- Also section 3.3: what is ‘relational complexity’ here? How is it defined/ what makes some relations more ‘complex’ than others? The paper’s central measure of task difficulty seems to be the number of relations. Can the authors explain how their work relates to existing notions of relational reasoning and complexity, e.g. in [1]? The existing section on related works largely ignores this entire field of research.
[1] Fesser, L., Ektefaie, Y., Fang, A., Kakade, S.M. & Zitnik, M. (2026). Evaluating Relational Reasoning in LLMs with REL. Proceedings of the 43rd International Conference on Machine Learning.
9 comments
Response to Reviewer J2gA (Part 2/4)
6. Correct the interpretation of task difficulty
- We agree that the sentence overstated the evidence, and we have narrowed it. The data show that relation count alone does not predict performance: Open has no monotonic trend and Closed shows no consistent decrease with k (Figure C.1). They do not isolate the interaction with retrieval structure as the cause. The new evidence-access analysis is consistent with a retrieval-structure effect, since 98.5% of incorrect Closed answers miss at least one gold document (Table E3), which we present as supporting evidence only. The revised Section 5.2 (Page 9) states the narrower observation.
7. Simplify Figures 1 and 2
- Each figure will carry one message. Figure 1 (Page 2) will show only locally correct evidence yielding an invalid relation; Figure 2 (Page 4) will show only the construction and training workflow. The loss formulas duplicated in Figure 2 and minor implementation details will be removed from the figures.
Table J0. Definitions and distinctions to make explicit in the revised manuscript.
| Term | Definition | Example |
|---|---|---|
| Research unit | Claim, evidence passage, source, entity, or output item | A numerical claim and the report that supports it |
| Global relation | Joint requirement over two or more research units | Two values must share a comparable reporting basis |
| Relational Verification Collapse (RVC) | Local checks pass although a required joint relation is invalid | Both values are sourced, but refer to different fiscal years |
| Genuine dependency | Resolving one requirement changes a later retrieval, interpretation, or decision | Fixing a reporting year determines which source to compare |
| Independent constraints | Conditions that can be checked without one result determining the other | A firm is in Germany and was founded after 2000 |
| Multi-hop dependency | A retrieval path where an intermediate finding determines a subsequent lookup | An intermediate entity found in document A identifies document B |
| Relation count | Number of instantiated relation requirements in a task | A task requiring R1, R2, and R5 has count 3 |
| Relational complexity | Used descriptively in our paper; not the higher-arity metric of REL | Report relation count and composition, rather than asserting a universal complexity score |
| Evidence / relation state | Evidence gathered so far and relations already resolved or still open | IE and CS track evidence; GR and NS track relations |
W2. Empirical evidence
Thank you for identifying the missing details and controls. We have added the task specifications, an SFT-only control, and repeated evaluations, and answer each point in order.
1. What are the held-out atomic tasks?
- The held-out atomic tasks are 196 single-step questions, disjoint by question ID from the MT data: 39 GR, 54 NS, 53 IE, and 50 CS. Each is open-ended generation under a fixed JSON schema, not multiple choice. GR receives a research request and returns the required relation types and checks. NS receives the request with its resolved and unresolved relations and returns the next relation to address plus one search query. IE receives a document and returns a verbatim supporting sentence with its provenance, or states that the evidence is absent. CS receives a multi-turn evidence stream and returns sourced confirmed, rejected, and pending items.
Response to Reviewer J2gA (Part 3/4)
2. Exactly how is atomic accuracy measured?
- Accuracy is the fraction of outputs that satisfy task-specific reference criteria. Each question is sampled three times at temperature 0.8 and judged by Claude Opus 4.8 at temperature 0. GR is correct if it covers the required relation types with actionable checks; NS if it selects the correct unresolved relation and a useful query without answering the question; IE if it quotes the supporting source text, or reports “not found” on a negative case; CS if it retains the grounded key facts with sources and invents nothing. Judge abstentions are excluded from the denominator. The inputs, targets, and scoring criteria of each task have been added to Appendix B.1 (Page 18), with a summary in Section 5.3 (Page 9); one full input–output example per task will be added.
3. Does MT add value beyond identical SFT trajectories?
- Setup. We follow the Table 3 experiment and change one factor. Both runs start from the same Qwen3.5-35B-A3B backbone and are fine-tuned on the same 10,914 OmniMaster trajectories with the same SFT hyperparameters; the only difference is whether the 1,040-example MT stage precedes SFT. Evaluation follows Section 5.1: Open is scored by the acceptance-item judge, and Closed by exact match on 150 tasks answered twice. Result. MT raises Closed from 51.0 to 62.7 (+11.7), well above the evaluation SD of 1.4; Open is unchanged within noise (46.1 vs. 46.7; Table E4). Why. MT trains the evidence- and relation-state updates separately before SFT composes them, and Closed tasks depend most on tracking these states along a multi-hop evidence chain. Implication. The gain is not explained by the trajectories alone, which supports the role of relation-aware MT. We have added the SFT-only row to Table 3.
4. What do the backbone experiments establish?
- The method is effective at every scale we tested. MT raises all four atomic accuracies on both 9B and 35B (Table J1); for example, NS improves from 25.2 to 61.3 on 9B and from 23.6 to 74.9 on 35B. End to end, OmniMaster-27B and OmniMaster-35B both reach the frontier range on Closed (61.7 and 62.7 without RL; Tables R1 and R2). We did not run end-to-end SFT below 27B, so we do not claim a minimum effective size, and we state this scope in Section 5.3.
5. How stable are the reported gains?
- We now report mean and sample standard deviation over repeated evaluations. Table J1 adds ± to the original Table 2 (five runs), Table J2 adds ± to the original Table 3 (five runs), and Table J3 reports three runs of the final checkpoints. The gains are far larger than the variation: for example, Closed rises from 21.1 ± 1.6 (Base) to 67.3 ± 1.8. These are repeated evaluations of the same checkpoints; retraining with multiple seeds was not feasible within the rebuttal period. Several entries differ slightly from the single-run values in the submission, and the direction of every comparison is unchanged. Figure C.1 now reports mean ± SD over three evaluations, with 21–22 tasks per relation count. On Open, differences between adjacent relation counts are within evaluation variation, which matches the reviewer’s observation; we will show the error bars and avoid reading a trend into adjacent points.
Table J1. Held-out atomic task accuracy (%).
| Model | GR | NS | IE | CS |
|---|---|---|---|---|
| Qwen3.5-9B | 16.4 ± 2.4 | 25.2 ± 2.3 | 33.9 ± 2.6 | 17.1 ± 2.2 |
| + MT | 26.5 ± 2.9 | 61.3 ± 3.1 | 89.3 ± 1.8 | 40.7 ± 3.0 |
| Qwen3.5-35B-A3B | 9.4 ± 1.6 | 23.6 ± 2.3 | 14.0 ± 2.0 | 6.7 ± 1.4 |
| + MT | 24.8 ± 2.8 | 74.9 ± 2.7 | 93.1 ± 1.5 | 63.6 ± 1.6 |
Response to Reviewer J2gA (Part 1/4)
Thank you for your kind support and valuable comments. We sincerely hope our rebuttal adequately address your concerns. If so, we would deeply appreciate it if you could consider raising your score. If not, please let us know your further concerns, and we will continue actively responding to your comments.
W1. Structure and presentation
We appreciate this concern. We agree that the presentation should be easier to follow, and address each point below in the order raised.
1. A single, clearer research narrative
2. Define new terms, especially Relational Verification Collapse
3. Explain why the four atomic MT tasks are needed
4. Clarify “relational complexity”
5. Justify the claim about independent constraints