- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper studies deep-research agents that must carry evidence between a SQL database and web search within one reasoning chain. It builds hybrid training items by extending validated single-source seeds in two directions. Text-to-SQL tasks receive a web leg made of coined facts, with fabricated gold documents and hard negatives. Hard web-search questions receive a synthetic SQLite database that produces or consumes the bridge entity. Gates are meant to ensure that neither source alone suffices. Qwen3.5-4B is fine-tuned on these items plus existing HyDRA and text-to-SQL data, and then trained with RL using a five-vote LLM judge. On the 120-question hard subset of HybridDeepResearch, run in the authors' local corpus and harness, pass@8 rises from 30.83 to 60.83, above Qwen3.5-397B (52.50) and below Claude Sonnet 4.6 (65.83). Single-source performance on BrowseComp, BrowseComp-Plus and Spider 2.0-lite also improves.
Critical strengths
- Extending validated single-source seeds across the boundary, with coined facts and anti-shortcut gates, is a practical way to obtain hard, checkable hybrid data.
- The gains are large for a 4B model and do not hurt single-source ability: BrowseComp and BrowseComp-Plus improve by 8–12 pass@8 points (Table 2).
- The paper is clearly written, the construction is documented in enough detail to reproduce, and the per-rollout failure analysis (App. I) is more informative than what such papers usually provide.
Critical weaknesses
- All hybrid results come from a single 120-question benchmark run in the authors' own local corpus and harness. That is the same environment, including ~2.4M fabricated documents, in which the 4B model was trained, while the larger baselines see it zero-shot.
- There is no variance. With single runs, the "beats Qwen3.5-397B" result (27.19 vs. 26.77 avg@8), the RL gain over SFT (+1.7 pass@8) and the RL ablations (1.6–2.0 points) are hard to separate from noise.
- It is unclear how much of the gain comes from the proposed construction. Existing HyDRA plus raw text-to-SQL data already lifts pass@8 from 30.83 to 47.50, leaving about 12 points for the new data.
- The anti-shortcut gates, the paper's core idea, are never ablated against ungated hybrid synthesis, and they are not uniform. For search-originated items, nothing checks that the web side is unsolvable without the bridge, and the App. D.3 example looks answerable from the shark-sequel clue alone.
- Possible overlap between the HDR test set and the training sources (HyDRA, BIRD/Spider seed databases) is not discussed.
Questions for authors
- What are the HDR hard-subset results with live web search, for your model and for the baselines?
- At fixed data volume, how does SFT on hybrid items built without the gates compare to the full recipe?
- How was overlap between the HDR hard subset and HyDRA or the BIRD/Spider seed databases ruled out, and how much do results vary across training seeds?
3 comments
Response to Reviewer 4dKj (Part 2)
W4 + Q2. Cross-Source Necessity and the Anti-Shortcut Gates
The gates are applied during construction, and rejected items, along with their documents and trajectories, were not retained. Rebuilding an ungated set at matched volume would require rerunning the full synthesis and teacher rollouts, which is not feasible within the rebuttal period. We instead compare trajectories with and without the gates. Without the gates, 79.5% of accepted teacher trajectories never execute a substantive SQL query (Section 3.1). With the gates, nearly all trajectories do, and most carry an entity returned by SQL into a later web search:
| Subset | Execute non-schema SQL (%) | Use an SQL-returned entity in a later search (%) |
|---|---|---|
| SQL-originated SQL2S | 99.99 | 83.90 |
| SQL-originated S2SQL | 100.00 | 80.30 |
| Search-originated SQL2S | 100.00 | 70.33 |
| Search-originated S2SQL | 100.00 | 68.30 |
Since single-source trajectories do not improve hybrid performance at a matched budget (Section 3.1), an ungated set would be unlikely to teach the cross-source step. For S2SQL items, the answer is the result table of a database that exists only within the item, so it cannot be obtained without SQL, and the hidden anchor must be resolved on the web before the correct query can be written. For SQL-originated SQL2S items, the web facts are coined and pass a no-document probe and a mask test, so they cannot be found without the bridge returned by SQL, while the answer itself lies in the web documents. For search-originated SQL2S items, the answer also lies on the web, and the gate guarantees that the bridge can only be obtained through SQL.
W5 + Q3 : No Overlap Between Training Data and the Evaluation Set
When constructing our training data, we ensured that there is zero overlap between it and HybridDeepResearch (HDR). We also ensured that the training data and HDR use disjoint databases. HDR is built on LiveSQLBench, a benchmark designed to be contamination-free, whose databases do not appear in BIRD, Spider, or any other training source. The databases behind HyDRA come from BIRD and Spider, and our SQL-originated seeds come from the sources in Table 7, none of which include LiveSQLBench. Each search-originated item ships with its own newly synthesized database.
Repeating the full SFT and RL pipeline across seeds was beyond our compute budget. However, several results indicate that the gains are not due to training noise. The gain over the base model (+19.48 Avg@8) is about seven times the run-to-run standard deviation. The SFT ablation (Table 13) and the matched-size comparison (response to W3) each come from separately trained models, and adding our data improves performance in every case. The gains also hold under live web retrieval (response to W1), and three single-source benchmarks (Table 2).
Thanks for the rebuttal, which addressed most of the concerns. I will raise my score. Please ensure to include these results and discussions into the revised paper.
Response to Reviewer 4dKj (Part 1)
Thank you for your review and for appreciating that "gains are large", our methods has "enough detail to reproduce" and our analysis "is more informative than what such papers usually provide". Please refer to our response to your comments below:
We evaluated on the HybridDeepResearch hard subset with live web search, replacing the local retrieval service with the same live search and page-fetching tools used for BrowseComp (Appendix H.1). The live web contains none of our fabricated documents, so every model faces this environment zero-shot.
As shown in the table above, with live web search, SFT and RL improve over the base model by 17.50 and 20.84 points in Pass@8 and 11.98 and 14.59 points in Avg@8 (p<0.001). These results show that the gains are not an artifact of the local corpus or the fabricated documents and transfer to the open web.
Thank you for this point. We evaluated our models across 8 independent runs on the 120-question HybridDeepResearch benchmark:
As shown in the table above, both SFT and RL outperform the base model by more than 16.0 points in Avg@8 (p<0.001) and more than 28.3 points in Pass@8. RL further improves over SFT by 3.44 points in Avg@8 (Welch's t-test, p=0.02). With these error bars, HybridResearcher-4B remains comparable to much larger frontier models such as Claude Sonnet 4.6 (65.83 Pass@8). We will include these results in the final version.
To our data’s contribution, we compare against HyDRA-based mixtures at matched training-set sizes. For each comparison, we sample the same number of rows from our SFT mixture while keeping the final recipe's per-source proportions. All other settings follow the final recipe.
At the same size, our synthesized data alone outperforms HyDRA + raw SQL by 9.17 Pass@8 and 7.39 Avg@8, and it is better on all three task types. HyDRA + raw SQL barely improves SQL2S (20.00 → 22.50), the direction where the web search must start from a SQL result, while our data targets this bridging with multi-hop constraints and hard negatives. When combined with existing data, our data adds 20.0 / 12.5 Pass@8 points on S2SQL / SQL2S and raises Avg@8 from 13.96 to 23.75 (+70% relative). In addition, all RL data is constructed by our pipeline, so the RL gains also come from our data. We will include this experiment in the final version.