- Rating
- 1
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
Scope2Rubric (S2R) is a training-free pipeline that builds query-specific rubrics for long-form deep research (LFDR) reports. It has two stages:
Stage I (scope). An LLM pulls requirements from the query (explicit and implicit) and from reference reports (factual). It groups them into weighted “evaluation directions”, each with semantics, boundaries and structure, which together form a fixed evaluation scope. Rubrics are then generated inside each direction and refined at three levels: single item, within direction, and across directions. Direction weights stay fixed, and rubric weights are normalized within each direction. Stage II (decomposition). During RL, rubrics that fail to separate the current rollouts are split into finer criteria. Splits stay inside the fixed scope and keep the parent rubric’s weight.
The rubrics serve as GRPO rewards, with Gemini-2.5-Pro as the reward judge. The paper reports:
Human-preference accuracy: 76.39% on DeepResearch Bench (DRB) and 53.15% on ScholarQA-CS2 (SQAv2), against 74.16% and 48.72% for RubricEM. Downstream RL: a Qwen3-8B-SFT policy reaches a 67.7 average over five LFDR benchmarks. The best evolving-rubric baseline gets 59.8, and the paper claims the 8B policy beats OpenAI Deep Research and GPT-5+Search. Further analyses: scope and rubric quality, alignment with expert rubrics, ablations, and transfer to short-form QA. The appendix adds propositions on coverage, refinement gain and scope drift.
Critical strengths
- A sensible decomposition of the rubric-construction problem. The paper separates what a rubric set should cover (a fixed, weighted task scope) from how finely it should discriminate (decomposition driven by responses, kept inside the scope). This is a useful way to think about rubric rewards. Fixing direction weights during decomposition directly addresses a known failure of evolving-rubric methods: the effective task priorities drift as rubrics grow.
- Broad diagnostics of rubric quality. The paper evaluates rubrics at several levels: direction correctness, coverage and redundancy; semantic and behavioral agreement with expert rubrics (Table 4); and individual-item quality against human rubrics (Table 9). This goes beyond reporting downstream scores, and the scope-quality metrics could be reused by others.
Critical weaknesses
-
The headline RL results are unsupported, and the paper’s own figures contradict them. Training curves vs final numbers. The S2R training curves in Fig. 4 peak at about 47.9 (DRB), 47.4 (DRB2), 45.2 (HealthBench), 73.8 (ResearchQA) and 68.4 (SQAv2). Table 2 reports the same model at 54.1, 48.4, 65.2, 80.5 and 90.4. Table 3 gives S2R 80.50 on ResearchQA, while its Fig. 4 curve never passes about 74. Training setup. The RL setup is essentially unspecified: no training queries; no source for the reference reports used in training; no optimizer, learning rate or rollout settings; no search or tool environment; no number of steps (“at least 75 steps”, while the plots run to 800); no Stage II frequency, and no API cost. Appendix G says these settings “are required” and that Table 6 records them, but Table 6 lists only the policy initialization, the models used, the GPUs and “≥75 steps”. Shared judge. The reward judge (Gemini-2.5-Pro) is also the standard RACE judge for DRB, so part of the reported gain could be the policy learning to please that judge. Baseline numbers. Many baseline cells are copied from papers with different judges and protocols. The unstarred cells have no stated provenance. The 8B claim. An 8B policy beating OpenAI Deep Research by 7 points on DRB and 11 on SQAv2 is an extraordinary claim. It would need the fully reproducible setup that is missing here.
-
The evaluation protocol is incomplete, and the appendix reads as an unfinished draft. Unresolved notes. The appendix contains drafting notes that were never resolved: “the exact ScholarQA release represented by the inherited SQAv2 column requires confirmation”; “The SQAv2 column requires a confirmed release and component aggregation before this formula can be assigned to its reported values”; the 2Wiki column “needs its actual evaluator identified”; “its variance divisor is not specified and must be recorded”; “Per-task training curves and run-level uncertainty are still needed to determine whether the relative component effects agree” (I.5). The SQAv2 column carries the paper’s largest claimed margins. Placeholder sections. I.6 (cost) and J (tool and domain analysis) describe analyses but report no results. Table 7 errors. In the DeepResearch Gen. row, the DRB preference-accuracy cell holds “49.3∗”, which is that system’s downstream DRB score. The row average of 56.60 implies a different value (about 68.1). S2R is labeled “Hybrid” in Table 1 but “Co-evolving (Agent–Generator)” in Table 7, although the method is described as training-free. Preference accuracy on SQAv2 is near chance. Accuracy is barely above 50%: S2R gets 53.15, QUEST 50.60, and most baselines fall below 50 (GPT-5.4 45.05, RubricRAG 40.73). Yet the paper reports no number of pairs, no handling of ties, and no confidence intervals. Ablations. All ablations are single runs. Figure 1. The motivating figures (“22% uncovered, 37% task-irrelevant”, “+8%/+13%”) have no stated method. Together these make it impossible to check the empirical claims.
Questions for authors
- Please reconcile Fig. 4 with Tables 2 and 3. Which checkpoint, scorer and evaluation subset produced each set of numbers? For example, HealthBench is 45.2 at most in Fig. 4 but 65.2 in Table 2.
- Please give the complete RL setup: training queries and their source; the reference reports used in Stage I for training queries; optimizer, learning rate, batch size, rollouts per prompt and total steps; the search tools; how often Stage II runs; total judge and generator API cost compared with the baselines.
- Which ScholarQA release and scoring setup produced the SQAv2 column? Were the unstarred baseline cells in Table 2 re-run by you, under the same judge and protocol as S2R?
- Is the DRB downstream judge the same model as the RL reward judge (Gemini-2.5-Pro)? Can you report DRB and DRB2 under a different judge to rule out the policy simply learning that judge’s preferences?
- For the preference-alignment evaluation:
- How many pairs are there per benchmark, and how are ties handled?
- Does Stage II use the two responses being judged to decompose rubrics?
- Do baselines get the same reference reports S2R uses in Stage I?
- Can you correct the DeepResearch Gen. row in Table 7 and the inconsistent method labels?
7 comments
Response to Reviewer GAtU (2/7)
W-a2 & Q2. About the training setup: the complete RL setup and API cost.
A: The revision replaces the original Table 6 with Appendix Tables G1 (SFT) and G2 (RL), which list every setting the reviewer asks about; Appendix Table I4d gives the API cost alongside the baselines. In the reviewer's order:
(1) Training queries: the 8,881 RL prompts released by DR Tulu, comprising 4,881 from SearchArena and OpenScholar, 3,000 from RaR-Science, and 1,000 from RaR-Medicine (Table H1). (2) Stage I reference reports: 10 per training query, generated independently by four deep research systems (Gemini, OpenAI and Perplexity Deep Research, and Grok Deeper Search) and six models equipped with the same search, browsing and paper-search tools (Appendix H.1.2). (3) Optimization: GRPO with AdamW (β = (0.9, 0.95), weight decay 0), a constant learning rate of 5 × 10⁻⁷, and a KL coefficient of 0.001; each step uses 32 prompts with 8 rollouts per prompt. (4) Total steps: the full method and every ablation train for 2,080 steps (7.5 epochs), and we evaluate the final checkpoint; each setting uses three training seeds, and each run takes about 2,770 GPU hours on eight H20 GPUs. (5) Search tools: google_search (Serper), web_browse (Crawl4AI), and paper_search (Semantic Scholar), with at most 10 tool calls per rollout. (6) Stage II: runs once per query at every training step; it takes the five rubrics in R_q^0 with the lowest score standard deviation over the eight rollouts as candidates and splits each into at most five finer criteria (Appendix H.5, Algorithm 1).
Steps. The “≥75 steps” row has been removed and replaced by the 2,080 steps above. Figures 4 and 8 plot only the first 800 steps, and their captions now say so; W-a1 & Q1 reconciles the curves with Tables 2 and 3.
API cost. Table I4d estimates the API cost of rubric generation and reward judging for one training seed through step 2,080, from the number and token counts of each request type at the standard GPT-5 and Gemini-2.5-Pro rates. All methods are trained under the controlled setting of Table 2:
| Method (controlled, one seed) | Rubric generation (USD) | Reward judging (USD) | Total (USD) | Table 2 Avg. |
|---|---|---|---|---|
| Tournament-GRPO | 480 | 6,700 | 7,180 | 62.51 |
| OpenRS (PAMR) | 5,160 | 5,490 | 10,650 | 60.10 |
| RubricEM | 3,450 | 9,720 | 13,170 | 62.85 |
| S2R (GPT-5) | 6,760 | 11,950 | 18,710 | 66.01 |
S2R costs about USD 5,540 more per run than RubricEM, USD 3,310 for rubric generation and USD 2,230 for reward judging, and its five-benchmark average is 3.16 points higher. See the training-details paragraph of Sec. 4.1; Appendix G (Tables G1, G2), H.1.1 (Table H1), H.1.2, H.5, and I.8 (Table I4d); and the captions of Figures 4 and 8.
W-a3 & Q4. About the shared judge, and DRB/DRB2 under a different judge.
A: Yes. Both DRB and DRB2 are scored by Gemini-2.5-Pro, the same model as the RL reward judge. We have rescored both with DeepSeek-V3.2, and S2R still leads.
We rescore the same DRB and DRB2 reports of every method in the controlled block of Table 2 with DeepSeek-V3.2, a model from which the policy never received reward during training. DRB keeps its RACE aggregation and DRB2 its fraction of passed criteria; the reports, tasks, and scoring prompts are unchanged. As in Table 2, each value is the mean over three training seeds.
| Controlled block (Qwen3-8B-SFT start) | DRB, Gemini-2.5-Pro | DRB, DeepSeek-V3.2 | DRB2, Gemini-2.5-Pro | DRB2, DeepSeek-V3.2 |
|---|---|---|---|---|
| Qwen3-8B-SFT | 38.67 | 33.68 | 30.04 | 24.79 |
| Tournament-GRPO | 51.47 | 41.62 | 41.12 | 31.17 |
| OpenRS (PAMR) | 47.83 | 39.03 | 37.72 | 28.96 |
| RubricEM | 51.47 | 41.62 | 42.31 | 33.79 |
| S2R (GPT-5) | 53.16 | 46.27 | 45.36 | 38.42 |
Under DeepSeek-V3.2, S2R (GPT-5) exceeds the strongest controlled baseline by 4.65 points on DRB and 4.63 points on DRB2, compared with 1.69 and 3.05 points under Gemini-2.5-Pro. Its advantage over the controlled baselines therefore does not arise only under the reward judge. Its gains over Qwen3-8B-SFT are 1.90 and 1.69 points smaller under DeepSeek-V3.2 (12.59 and 13.63 vs. 14.49 and 15.32 points). DeepSeek-V3.2 gives every method lower absolute scores, so we do not attribute this difference solely to the policy fitting Gemini-2.5-Pro, but we cannot rule that out either.
In the revised paper, Section 4.1 now states which judge scores each benchmark and that DRB and DRB2 share their judge with the reward judge, and Section 4.2 reports the rescoring results. The last two columns of Appendix Table 8 give the full scores, and Appendix H.4 gives the setup. The limitations in Appendix K state that we test only one alternative judge, DeepSeek-V3.2.
Response to Reviewer GAtU (3/7)
W-a4. About the baseline numbers. A: Every cell of Table 2 now states its source, and our claims about the reward construction now rest on a controlled comparison in which only the reward differs.
Starred cells in Table 2 are published scores whose samples, judge, metric and system version match our setup. All other cells are obtained by us under the protocol of Sec. 4.1, running each system through its official API, released code or released weights. The new Appendix Table H6 lists, row by row, the source of the scores (with the paper and table for every starred cell) and the configuration of each original system.
The controlled comparison retrains the strongest baseline of each rubric type in Table 1 from the same Qwen3-8B-SFT checkpoint, with the same queries, tools, training budget, rubric generator (GPT-5) and reward judge (Gemini-2.5-Pro) as S2R. Values are means over three training seeds:
| Controlled block | DRB | DRB2 | HB | RQA | SQAv2 | Avg. |
|---|---|---|---|---|---|---|
| Qwen3-8B-SFT | 38.67 | 30.04 | 37.92 | 68.84 | 72.82 | 49.66 |
| Tournament-GRPO | 51.47 | 41.12 | 54.68 | 78.54 | 86.76 | 62.51 |
| OpenRS (PAMR) | 47.83 | 37.72 | 53.84 | 77.21 | 83.92 | 60.10 |
| RubricEM | 51.47 | 42.31 | 54.08 | 78.78 | 87.63 | 62.85 |
| S2R (GPT-5) | 53.16 | 45.36 | 59.46 | 82.46 | 89.63 | 66.01 |
S2R exceeds the strongest controlled baseline on all five benchmarks, by 1.69 to 4.78 points, and by 3.16 points on average. The revised paper bases its claims about the reward construction on this block only; the other groups serve as system-level context. See the Table 2 caption and controlled block, Sec. 4.2, and Appendix Table H6 (H.3); seed standard deviations are in Appendix I.8.
W-a5. About the 8B claim.
A: The revised paper makes no claim about this cross-system comparison; its claims rest on the controlled comparison below.
The submitted text did not state this conclusion directly, but the abstract and introduction led with the 67.72 average, and Table 2 assigned OpenAI Deep Research an average of 61.0, which invites the comparison. OpenAI Deep Research is a closed system whose retrieval tools and inference budget are not disclosed, so we cannot compare it with the 8B policy under the same setup.
The revised abstract and introduction report only the controlled block of Table 2. All methods in this block start from the same Qwen3-8B-SFT checkpoint and share the 8,881 RL queries, the training and tool budgets, the GPT-5 rubric generator, and the Gemini-2.5-Pro reward judge; only the reward construction differs, and each value is the mean over three training seeds. Under this setup, S2R (GPT-5) averages 66.01 across the five benchmarks, 3.16 points above the strongest controlled baseline, RubricEM (62.85). Appendix G gives the full SFT and RL settings.
Table 2 keeps the proprietary rows as cross-system references only, as Section 4.2 now states. For OpenAI Deep Research, DRB 46.98 is the official DeepResearch Bench score, DRB2 45.40 is from Table 3 of DeepResearch Bench II, and SQAv2 79.60 is from Table 1 of DR Tulu; these three cells match our samples, judges, and metrics and are marked ∗. We evaluated HB and RQA under our evaluation setup. Because the five scores come from different sources without a common system-version mapping, the row's Avg. is now –.
Response to Reviewer GAtU (1/7)
We thank the reviewer for the precise review; reading Figure 4 against Table 2 and checking the Table 7 row average identified the curve–table mismatch and the Table 7 cell error, both now corrected. The reviewer views the scope/granularity separation as "a useful way to think about rubric rewards"; this response is about making the results behind it checkable. Changes are marked in blue in the revised PDF.
W-a1 & Q1. About the training curves vs. the final numbers, and which checkpoint, scorer and subset produced each.
A: Figure 4 and Tables 2–3 report different checkpoints, and in the submission they also used different evaluation settings. Figure 4 stops at step 800, whereas Tables 2 and 3 report the final checkpoint at step 2,080. We have re-evaluated every curve under the Table 2 setting and rerun Tables 2 and 3 with three training seeds.
The submitted curves were scored during training with the training-time tools (local Crawl4AI browsing with truncated pages); the DRB2 curve reported the equal-weight mean of three dimension-level pass rates; and the HealthBench curve used a different sample of 1,000 examples from Table 2. In the revision, the curves, Table 2 and the downstream columns of Table 3 share one setting: Serper search and Jina browsing; DRB (100 tasks, Gemini-2.5-Pro), DRB2 (132 tasks, Gemini-2.5-Pro, official overall score), HealthBench (1,000 examples, GPT-4.1), ResearchQA (776 questions, GPT-4.1-mini) and SQAv2 (100 questions, Gemini-2.5-Flash). Each number averages three training seeds (42, 123, 456), each evaluated three times.
Under the unified setting, the gap between the curves and the tables reflects only the training after step 800. Step 800 is 38% of the 2,080-step schedule; at that point S2R has reached 37–41% of its total gain on DRB, DRB2, HealthBench and ResearchQA, and 66% on SQAv2. On HealthBench, the reviewer's example, the score rises from 48.29 at step 800 to 53.97 at step 1,200, 59.62 at step 1,600 and 64.92 at step 2,080. After the rerun, the S2R (Gemini 3) row of Table 2 changes by at most 0.7 points.
ResearchQA 80.50 in Table 3. Figure 4 and Table 3 come from the same training runs. The ResearchQA curve of S2R is at 73.16 at step 800 and 80.58 at step 2,080, which is the value in the revised Table 3. The two ablations reach 77.78 (w/o Fixed Scope) and 77.12 (w/o Scope Constraint) at step 2,080, identical to their rows in Table 3.
Revision: Figures 4 and 8 now show curves evaluated under the Table 2 setting; Section 4.3 states that Tables 2 and 3 report the step-2,080 checkpoint; Appendix H.3 gives the evaluation setting of both figures and the scores at steps 1,200, 1,600 and 2,080.