acceptodds
Under review as a conference paper at ICLR 2027

Scope2Rubric: Recovering Query-Specific Evaluation Scope before Rubric Generation for Long-Form Deep Research

Abstract

Long-form deep research (LFDR) tasks lack definitive answers for RL with verifiable rewards (RLVR), motivating rubric-based rewards for training deep research agents. Rubric effectiveness hinges on jointly covering full task requirements and distinguishing task-relevant quality differences among reports. Yet existing methods often miss implicit requirements or capture task-irrelevant response differences. We attribute both failures to an overlooked constraint: full task requirements are not explicitly elicited to constrain what rubrics should cover and which response differences they should capture. Accordingly, given the multifaceted nature of LFDR tasks, our key insight is to group requirements sharing an evaluation goal into directions that jointly form a task-defined evaluation scope, constraining both rubric generation and decomposition. We propose Scope2Rubric (S2R), a two-stage rubric construction framework. Stage I extracts full task requirements from query and reference analyses, organizes them into weighted directions, and reviews their coverage to form the evaluation scope. The scope constrains rubric generation within each direction, followed by individual, intra-direction, and cross-direction refinement to ensure the rubric set’s necessity, completeness, and non-redundancy. Stage II identifies rubrics that fail to distinguish candidate responses and decomposes them by contrasting task-relevant differences within the fixed Stage I scope, preventing salient differences from inducing task-irrelevant rubrics. Extensive experiments show S2R achieves the best human-preference alignment over strong baselines. When used as rewards to train Qwen3-8B, it yields a superior 67.72 average score across five LFDR benchmarks. Further analyses confirm these gains stem from structured task-defined scope constraints.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
rubric as reward, rl, deepresearch

Reviews

Among:
Shape:
1234this paper, Bottom 9%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.40 over 5 reviews, above 9% of the others.
Reviewer_GAtUOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

Scope2Rubric (S2R) is a training-free pipeline that builds query-specific rubrics for long-form deep research (LFDR) reports. It has two stages:

Stage I (scope). An LLM pulls requirements from the query (explicit and implicit) and from reference reports (factual). It groups them into weighted “evaluation directions”, each with semantics, boundaries and structure, which together form a fixed evaluation scope. Rubrics are then generated inside each direction and refined at three levels: single item, within direction, and across directions. Direction weights stay fixed, and rubric weights are normalized within each direction. Stage II (decomposition). During RL, rubrics that fail to separate the current rollouts are split into finer criteria. Splits stay inside the fixed scope and keep the parent rubric’s weight.

The rubrics serve as GRPO rewards, with Gemini-2.5-Pro as the reward judge. The paper reports:

Human-preference accuracy: 76.39% on DeepResearch Bench (DRB) and 53.15% on ScholarQA-CS2 (SQAv2), against 74.16% and 48.72% for RubricEM. Downstream RL: a Qwen3-8B-SFT policy reaches a 67.7 average over five LFDR benchmarks. The best evolving-rubric baseline gets 59.8, and the paper claims the 8B policy beats OpenAI Deep Research and GPT-5+Search. Further analyses: scope and rubric quality, alignment with expert rubrics, ablations, and transfer to short-form QA. The appendix adds propositions on coverage, refinement gain and scope drift.

Critical strengths
  • A sensible decomposition of the rubric-construction problem. The paper separates what a rubric set should cover (a fixed, weighted task scope) from how finely it should discriminate (decomposition driven by responses, kept inside the scope). This is a useful way to think about rubric rewards. Fixing direction weights during decomposition directly addresses a known failure of evolving-rubric methods: the effective task priorities drift as rubrics grow.
  • Broad diagnostics of rubric quality. The paper evaluates rubrics at several levels: direction correctness, coverage and redundancy; semantic and behavioral agreement with expert rubrics (Table 4); and individual-item quality against human rubrics (Table 9). This goes beyond reporting downstream scores, and the scope-quality metrics could be reused by others.
Critical weaknesses
  1. The headline RL results are unsupported, and the paper’s own figures contradict them. Training curves vs final numbers. The S2R training curves in Fig. 4 peak at about 47.9 (DRB), 47.4 (DRB2), 45.2 (HealthBench), 73.8 (ResearchQA) and 68.4 (SQAv2). Table 2 reports the same model at 54.1, 48.4, 65.2, 80.5 and 90.4. Table 3 gives S2R 80.50 on ResearchQA, while its Fig. 4 curve never passes about 74. Training setup. The RL setup is essentially unspecified: no training queries; no source for the reference reports used in training; no optimizer, learning rate or rollout settings; no search or tool environment; no number of steps (“at least 75 steps”, while the plots run to 800); no Stage II frequency, and no API cost. Appendix G says these settings “are required” and that Table 6 records them, but Table 6 lists only the policy initialization, the models used, the GPUs and “≥75 steps”. Shared judge. The reward judge (Gemini-2.5-Pro) is also the standard RACE judge for DRB, so part of the reported gain could be the policy learning to please that judge. Baseline numbers. Many baseline cells are copied from papers with different judges and protocols. The unstarred cells have no stated provenance. The 8B claim. An 8B policy beating OpenAI Deep Research by 7 points on DRB and 11 on SQAv2 is an extraordinary claim. It would need the fully reproducible setup that is missing here.

  2. The evaluation protocol is incomplete, and the appendix reads as an unfinished draft. Unresolved notes. The appendix contains drafting notes that were never resolved: “the exact ScholarQA release represented by the inherited SQAv2 column requires confirmation”; “The SQAv2 column requires a confirmed release and component aggregation before this formula can be assigned to its reported values”; the 2Wiki column “needs its actual evaluator identified”; “its variance divisor is not specified and must be recorded”; “Per-task training curves and run-level uncertainty are still needed to determine whether the relative component effects agree” (I.5). The SQAv2 column carries the paper’s largest claimed margins. Placeholder sections. I.6 (cost) and J (tool and domain analysis) describe analyses but report no results. Table 7 errors. In the DeepResearch Gen. row, the DRB preference-accuracy cell holds “49.3∗”, which is that system’s downstream DRB score. The row average of 56.60 implies a different value (about 68.1). S2R is labeled “Hybrid” in Table 1 but “Co-evolving (Agent–Generator)” in Table 7, although the method is described as training-free. Preference accuracy on SQAv2 is near chance. Accuracy is barely above 50%: S2R gets 53.15, QUEST 50.60, and most baselines fall below 50 (GPT-5.4 45.05, RubricRAG 40.73). Yet the paper reports no number of pairs, no handling of ties, and no confidence intervals. Ablations. All ablations are single runs. Figure 1. The motivating figures (“22% uncovered, 37% task-irrelevant”, “+8%/+13%”) have no stated method. Together these make it impossible to check the empirical claims.

Questions for authors
  • Please reconcile Fig. 4 with Tables 2 and 3. Which checkpoint, scorer and evaluation subset produced each set of numbers? For example, HealthBench is 45.2 at most in Fig. 4 but 65.2 in Table 2.
  • Please give the complete RL setup: training queries and their source; the reference reports used in Stage I for training queries; optimizer, learning rate, batch size, rollouts per prompt and total steps; the search tools; how often Stage II runs; total judge and generator API cost compared with the baselines.
  • Which ScholarQA release and scoring setup produced the SQAv2 column? Were the unstarred baseline cells in Table 2 re-run by you, under the same judge and protocol as S2R?
  • Is the DRB downstream judge the same model as the RL reward judge (Gemini-2.5-Pro)? Can you report DRB and DRB2 under a different judge to rule out the policy simply learning that judge’s preferences?
  • For the preference-alignment evaluation:
  • How many pairs are there per benchmark, and how are ties handled?
  • Does Stage II use the two responses being judged to decompose rubrics?
  • Do baselines get the same reference reports S2R uses in Stage I?
  • Can you correct the DeepResearch Gen. row in Table 7 and the inconsistent method labels?
7 comments
AuthorsOpenReview ↗

Response to Reviewer GAtU (1/7)

We thank the reviewer for the precise review; reading Figure 4 against Table 2 and checking the Table 7 row average identified the curve–table mismatch and the Table 7 cell error, both now corrected. The reviewer views the scope/granularity separation as "a useful way to think about rubric rewards"; this response is about making the results behind it checkable. Changes are marked in blue in the revised PDF.


W-a1 & Q1. About the training curves vs. the final numbers, and which checkpoint, scorer and subset produced each.

A: Figure 4 and Tables 2–3 report different checkpoints, and in the submission they also used different evaluation settings. Figure 4 stops at step 800, whereas Tables 2 and 3 report the final checkpoint at step 2,080. We have re-evaluated every curve under the Table 2 setting and rerun Tables 2 and 3 with three training seeds.

The submitted curves were scored during training with the training-time tools (local Crawl4AI browsing with truncated pages); the DRB2 curve reported the equal-weight mean of three dimension-level pass rates; and the HealthBench curve used a different sample of 1,000 examples from Table 2. In the revision, the curves, Table 2 and the downstream columns of Table 3 share one setting: Serper search and Jina browsing; DRB (100 tasks, Gemini-2.5-Pro), DRB2 (132 tasks, Gemini-2.5-Pro, official overall score), HealthBench (1,000 examples, GPT-4.1), ResearchQA (776 questions, GPT-4.1-mini) and SQAv2 (100 questions, Gemini-2.5-Flash). Each number averages three training seeds (42, 123, 456), each evaluated three times.

S2R (Gemini 3)DRBDRB2HealthBenchResearchQASQAv2
Fig. 4 peak in the submission (as cited)47.947.445.273.868.4
Step 0 (= Qwen3-8B-SFT row, Table 2)38.6730.0437.9268.8472.82
Step 800 (last point in revised Fig. 4)45.3137.7648.2973.1684.30
Step 1,20048.9641.6053.9776.5086.68
Step 1,60052.4445.7459.6278.8088.76
Step 2,080 (= revised Table 2)54.6949.0664.9280.5890.21
Table 2 in the submission54.148.465.280.590.4

Under the unified setting, the gap between the curves and the tables reflects only the training after step 800. Step 800 is 38% of the 2,080-step schedule; at that point S2R has reached 37–41% of its total gain on DRB, DRB2, HealthBench and ResearchQA, and 66% on SQAv2. On HealthBench, the reviewer's example, the score rises from 48.29 at step 800 to 53.97 at step 1,200, 59.62 at step 1,600 and 64.92 at step 2,080. After the rerun, the S2R (Gemini 3) row of Table 2 changes by at most 0.7 points.

ResearchQA 80.50 in Table 3. Figure 4 and Table 3 come from the same training runs. The ResearchQA curve of S2R is at 73.16 at step 800 and 80.58 at step 2,080, which is the value in the revised Table 3. The two ablations reach 77.78 (w/o Fixed Scope) and 77.12 (w/o Scope Constraint) at step 2,080, identical to their rows in Table 3.

Revision: Figures 4 and 8 now show curves evaluated under the Table 2 setting; Section 4.3 states that Tables 2 and 3 report the step-2,080 checkpoint; Appendix H.3 gives the evaluation setting of both figures and the scores at steps 1,200, 1,600 and 2,080.

AuthorsOpenReview ↗

Response to Reviewer GAtU (2/7)

W-a2 & Q2. About the training setup: the complete RL setup and API cost.

A: The revision replaces the original Table 6 with Appendix Tables G1 (SFT) and G2 (RL), which list every setting the reviewer asks about; Appendix Table I4d gives the API cost alongside the baselines. In the reviewer's order:

(1) Training queries: the 8,881 RL prompts released by DR Tulu, comprising 4,881 from SearchArena and OpenScholar, 3,000 from RaR-Science, and 1,000 from RaR-Medicine (Table H1). (2) Stage I reference reports: 10 per training query, generated independently by four deep research systems (Gemini, OpenAI and Perplexity Deep Research, and Grok Deeper Search) and six models equipped with the same search, browsing and paper-search tools (Appendix H.1.2). (3) Optimization: GRPO with AdamW (β = (0.9, 0.95), weight decay 0), a constant learning rate of 5 × 10⁻⁷, and a KL coefficient of 0.001; each step uses 32 prompts with 8 rollouts per prompt. (4) Total steps: the full method and every ablation train for 2,080 steps (7.5 epochs), and we evaluate the final checkpoint; each setting uses three training seeds, and each run takes about 2,770 GPU hours on eight H20 GPUs. (5) Search tools: google_search (Serper), web_browse (Crawl4AI), and paper_search (Semantic Scholar), with at most 10 tool calls per rollout. (6) Stage II: runs once per query at every training step; it takes the five rubrics in R_q^0 with the lowest score standard deviation over the eight rollouts as candidates and splits each into at most five finer criteria (Appendix H.5, Algorithm 1).

Steps. The “≥75 steps” row has been removed and replaced by the 2,080 steps above. Figures 4 and 8 plot only the first 800 steps, and their captions now say so; W-a1 & Q1 reconciles the curves with Tables 2 and 3.

API cost. Table I4d estimates the API cost of rubric generation and reward judging for one training seed through step 2,080, from the number and token counts of each request type at the standard GPT-5 and Gemini-2.5-Pro rates. All methods are trained under the controlled setting of Table 2:

Method (controlled, one seed)Rubric generation (USD)Reward judging (USD)Total (USD)Table 2 Avg.
Tournament-GRPO4806,7007,18062.51
OpenRS (PAMR)5,1605,49010,65060.10
RubricEM3,4509,72013,17062.85
S2R (GPT-5)6,76011,95018,71066.01

S2R costs about USD 5,540 more per run than RubricEM, USD 3,310 for rubric generation and USD 2,230 for reward judging, and its five-benchmark average is 3.16 points higher. See the training-details paragraph of Sec. 4.1; Appendix G (Tables G1, G2), H.1.1 (Table H1), H.1.2, H.5, and I.8 (Table I4d); and the captions of Figures 4 and 8.


W-a3 & Q4. About the shared judge, and DRB/DRB2 under a different judge.

A: Yes. Both DRB and DRB2 are scored by Gemini-2.5-Pro, the same model as the RL reward judge. We have rescored both with DeepSeek-V3.2, and S2R still leads.

We rescore the same DRB and DRB2 reports of every method in the controlled block of Table 2 with DeepSeek-V3.2, a model from which the policy never received reward during training. DRB keeps its RACE aggregation and DRB2 its fraction of passed criteria; the reports, tasks, and scoring prompts are unchanged. As in Table 2, each value is the mean over three training seeds.

Controlled block (Qwen3-8B-SFT start)DRB, Gemini-2.5-ProDRB, DeepSeek-V3.2DRB2, Gemini-2.5-ProDRB2, DeepSeek-V3.2
Qwen3-8B-SFT38.6733.6830.0424.79
Tournament-GRPO51.4741.6241.1231.17
OpenRS (PAMR)47.8339.0337.7228.96
RubricEM51.4741.6242.3133.79
S2R (GPT-5)53.1646.2745.3638.42

Under DeepSeek-V3.2, S2R (GPT-5) exceeds the strongest controlled baseline by 4.65 points on DRB and 4.63 points on DRB2, compared with 1.69 and 3.05 points under Gemini-2.5-Pro. Its advantage over the controlled baselines therefore does not arise only under the reward judge. Its gains over Qwen3-8B-SFT are 1.90 and 1.69 points smaller under DeepSeek-V3.2 (12.59 and 13.63 vs. 14.49 and 15.32 points). DeepSeek-V3.2 gives every method lower absolute scores, so we do not attribute this difference solely to the policy fitting Gemini-2.5-Pro, but we cannot rule that out either.

In the revised paper, Section 4.1 now states which judge scores each benchmark and that DRB and DRB2 share their judge with the reward judge, and Section 4.2 reports the rescoring results. The last two columns of Appendix Table 8 give the full scores, and Appendix H.4 gives the setup. The limitations in Appendix K state that we test only one alternative judge, DeepSeek-V3.2.

AuthorsOpenReview ↗

Response to Reviewer GAtU (3/7)

W-a4. About the baseline numbers. A: Every cell of Table 2 now states its source, and our claims about the reward construction now rest on a controlled comparison in which only the reward differs.

Starred cells in Table 2 are published scores whose samples, judge, metric and system version match our setup. All other cells are obtained by us under the protocol of Sec. 4.1, running each system through its official API, released code or released weights. The new Appendix Table H6 lists, row by row, the source of the scores (with the paper and table for every starred cell) and the configuration of each original system.

The controlled comparison retrains the strongest baseline of each rubric type in Table 1 from the same Qwen3-8B-SFT checkpoint, with the same queries, tools, training budget, rubric generator (GPT-5) and reward judge (Gemini-2.5-Pro) as S2R. Values are means over three training seeds:

Controlled blockDRBDRB2HBRQASQAv2Avg.
Qwen3-8B-SFT38.6730.0437.9268.8472.8249.66
Tournament-GRPO51.4741.1254.6878.5486.7662.51
OpenRS (PAMR)47.8337.7253.8477.2183.9260.10
RubricEM51.4742.3154.0878.7887.6362.85
S2R (GPT-5)53.1645.3659.4682.4689.6366.01

S2R exceeds the strongest controlled baseline on all five benchmarks, by 1.69 to 4.78 points, and by 3.16 points on average. The revised paper bases its claims about the reward construction on this block only; the other groups serve as system-level context. See the Table 2 caption and controlled block, Sec. 4.2, and Appendix Table H6 (H.3); seed standard deviations are in Appendix I.8.


W-a5. About the 8B claim.

A: The revised paper makes no claim about this cross-system comparison; its claims rest on the controlled comparison below.

The submitted text did not state this conclusion directly, but the abstract and introduction led with the 67.72 average, and Table 2 assigned OpenAI Deep Research an average of 61.0, which invites the comparison. OpenAI Deep Research is a closed system whose retrieval tools and inference budget are not disclosed, so we cannot compare it with the 8B policy under the same setup.

The revised abstract and introduction report only the controlled block of Table 2. All methods in this block start from the same Qwen3-8B-SFT checkpoint and share the 8,881 RL queries, the training and tool budgets, the GPT-5 rubric generator, and the Gemini-2.5-Pro reward judge; only the reward construction differs, and each value is the mean over three training seeds. Under this setup, S2R (GPT-5) averages 66.01 across the five benchmarks, 3.16 points above the strongest controlled baseline, RubricEM (62.85). Appendix G gives the full SFT and RL settings.

Table 2 keeps the proprietary rows as cross-system references only, as Section 4.2 now states. For OpenAI Deep Research, DRB 46.98 is the official DeepResearch Bench score, DRB2 45.40 is from Table 3 of DeepResearch Bench II, and SQAv2 79.60 is from Table 1 of DR Tulu; these three cells match our samples, judges, and metrics and are marked ∗. We evaluated HB and RQA under our evaluation setup. Because the five scores come from different sources without a common system-version mapping, the row's Avg. is now –.

Reviewer_xVvxOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

Overall, Scope2Rubric makes evaluation criteria for long research reports to address two problems: omitted task requirements and criteria that cannot distinguish response quality, fixing the task scope before it adapts rubric detail to current responses. To be more specific, Stage I recovers requirements from the query, task context, and reference reports, grouping them into evaluation directions with boundaries and weights, and then makes rubric items within each direction. Checks at the item, direction, and cross-direction levels address omissions and duplicate criteria. Stage II divides an item when current responses contain task-related differences that the item cannot identify, and the child items keep the parent item's total weight, within the fixed direction.

Critical strengths

1. The method gives clear controls on rubric changes.

Equations 3, 5, and 7 keep each direction's total weight constant when an item becomes multiple items. Stage II also uses task-related differences that the current item cannot identify. New criteria must stay within the task scope. These controls show how rubric changes preserve direction weights.

2. The rubric scores agree more often with human preferences.

In Table 1, S2R (Gemini3) has the highest accuracy on both preference sets, exceeding the best baseline by 2.23 and 2.55 percentage points, respectively. All rubric-guided methods use Gemini-2.5-Pro as the judge. This controls judge identity in the rubric comparison. The reported gains support the rubric's value.

S3. The expert-rubric comparison tests both criteria and response rankings.

Table 4 compares generated and expert rubrics on the same candidate responses. Against RubricEM, semantic F1 increases from 0.43 to 0.51 on HealthBench-Consensus and from 0.47 to 0.53 on ResearchRubrics. All four response-ranking measures also increase on both datasets. These results support agreement with expert rubrics in both criteria and response order.

Critical weaknesses

1. The training curves and tables do not agree, and one benchmark name is not confirmed.

At step 800, Figures 4 and 8 give lower S2R scores than Table 2 on all five benchmarks. HealthBench and SQAv2 end near 45 and 68, versus 65.2 and 90.4 for S2R (Gemini3). Initial scores also differ from the Qwen3-8B-SFT row. Figure 8 starts DR Bench2 at 43, but Table 2 gives 29.6. Furthermore, Figure 8 starts three baselines at S2R's score, although Table 2 gives them different backbones. The paper gives no procedure for these curves. Section 4.1 uses SQAv2 for ScholarQA-CS2, but H.1.2 says this identity “requires confirmation.” H.2.2 also leaves the release and aggregation unconfirmed.

2. The construction data and evaluation references are not clearly distinguished.

Appendix F gives data roles but no training query count, split identifiers, or reference-report sources. Neither does it document overlap between construction inputs and evaluation references. The reason why this matters is that Stage I uses reference reports to recover requirements, while Table 4 tests agreement with expert criteria.

3. The paper omits details necessary to implement and train the method.

H.5 gives stage descriptions but no prompts or operational LowDisc and Δ rules. With that being said, Equation 4 does not specify how to select items for division in practice. Table 6 gives hardware and a minimum ablation budget, but no learning rate, rollout batch size, or full training budget. G.3 says optimizer settings “are required,” and H.2.8 refers to run records, but neither gives the necessary values.

4. The reported gains lack evaluation counts and uncertainty estimates.

Table 1 gives paired Cohen's d, but Tables 1, 3, and 4 give no evaluation counts or uncertainty estimates. An effect size does not show the uncertainty of an estimate. The counts and uncertainty are necessary to assess how firmly the comparisons support the preference and expert-rubric gains.

Questions for authors
  1. Which run, checkpoint, subset, and scorer produced each S2R row in Tables 2–3? Which conditions produced each curve in Figures 4 and 8? Why are the step-0 scores different from the Qwen3-8B-SFT row? How did you produce the Quest-35B, DR Tulu, and Tournament-GRPO curves in Figure 8? Which ScholarQA release and aggregation rule does SQAv2 use?

  2. What are the training and evaluation settings? (e.g. query counts and split identifiers, prompts and operational rules for LowDisc and Δ, optimizer settings, rollout settings, and training budgets for the full method and ablations)

4 comments
AuthorsOpenReview ↗

Response to Reviewer xVvx (1/4)

We thank the reviewer for checking the curves against the tables number by number. The reviewer notes that the method "gives clear controls on rubric changes"; this response is about making the results behind that claim checkable. Changes are marked in blue in the revised PDF.


W1 & Q1. About the agreement between the training curves (Figs. 4, 8) and Table 2, where each number comes from, and the SQAv2 name.

A: The submitted curves differed from Table 2 in two ways. They stop at step 800, whereas Table 2 reports the final checkpoint at step 2,080, and they were scored during training under a different evaluation setting. We have re-evaluated every curve under the Table 2 setting; the remaining gap reflects only the training after step 800.

The submitted curves were scored with the training-time tools (local Crawl4AI browsing with truncated pages), while Table 2 used Jina browsing. The DRB2 curve reported the equal-weight mean of three dimension-level pass rates, while Table 2 reports the official overall score. The HealthBench curve also used a different sample of 1,000 examples. The revised curves and Table 2 now share one setting: Serper search and Jina browsing; DRB (100 tasks, Gemini-2.5-Pro), DRB2 (132 tasks, Gemini-2.5-Pro), HealthBench (1,000 examples, GPT-4.1), ResearchQA (776 questions, GPT-4.1-mini) and SQAv2 (100 questions, Gemini-2.5-Flash). Each point averages three training seeds (42, 123, 456), each evaluated three times. Table 2 was rerun under this setting, and its S2R (Gemini 3) row changes by at most 0.7 points.

S2R (Gemini 3)DRBDRB2HealthBenchResearchQASQAv2
Step 0 (= Qwen3-8B-SFT row, Table 2)38.6730.0437.9268.8472.82
Step 800 (last point in Fig. 4)45.3137.7648.2973.1684.30
Step 1,20048.9641.6053.9776.5086.68
Step 1,60052.4445.7459.6278.8088.76
Step 2,080 (= Table 2)54.6949.0664.9280.5890.21

(1) Why is step 800 below Table 2? Step 800 is 38% of the 2,080-step schedule. At that point S2R has reached 37–41% of its total gain on DRB, DRB2, HealthBench and ResearchQA, and 66% on SQAv2. By step 1,600 it has reached 80–92% on the five benchmarks.

(2) Why does step 0 differ from the Qwen3-8B-SFT row? Both are the same checkpoint; the difference came from the evaluation setting above. On DRB2, 43 was the equal-weight mean and 29.6 the official overall score, i.e., the fraction of all criteria passed per task, averaged over tasks. In the revision, step 0 equals the Qwen3-8B-SFT row on all five benchmarks (30.04 on DRB2).

(3) How were the Figure 8 curves produced? All four policies start from the same Qwen3-8B-SFT checkpoint and share the 8,881 RL queries, the training and tool budgets, the GPT-5 rubric generator and the Gemini-2.5-Pro reward judge. Only the reward construction differs, which is why they share a starting point. The S2R and Tournament-GRPO curves come from the same training runs as their rows in the controlled block of Table 2. The QUEST reward is adapted to Qwen3-8B and is now labeled QUEST-8B; DR Tulu is retrained under the same setting. Results on each method's original backbone are listed separately in Table 2.

(4) Which run, checkpoint, subset and scorer produced the S2R rows of Tables 2–3? Both S2R rows of Table 2 and the downstream columns of Table 3 use the step-2,080 checkpoints of three training seeds, each evaluated three times, with the subsets and judges listed above. The preference columns of Table 3 are measured before RL on 298 DRB report pairs and 1,176 SQAv2 annotations, scored by Gemini-2.5-Pro. Figure 4 follows the same training runs as Table 3 (Gemini 3 generator); Figure 8 uses the GPT-5 generator.

(5) Which ScholarQA release and aggregation rule does SQAv2 use?

SQAv2 denotes ScholarQA-CS2 from AstaBench (Bragg et al., 2026). We use the sqa_test task of astabench 0.5.4 (suite configuration v1.0.0), i.e., the 100 questions of the official test set; it is judged by the official gemini-2.5-flash, and the score is the mean of the four official metrics: answer coverage, answer relevance, citation precision, and citation recall. The "requires confirmation" sentences in H.1.2 and H.2.2 are deleted, and we have stated this release and aggregation rule in the dataset description of Appendix H.1 and in Table H2.

Revision: Figures 4 and 8 now show curves evaluated under the Table 2 setting; Section 4.3 states that Tables 2 and 3 report the step-2,080 checkpoint; Appendix H.3 gives the evaluation setting of both figures and the scores at steps 1,200, 1,600 and 2,080, and Appendix G gives the training configuration.

AuthorsOpenReview ↗

Response to Reviewer xVvx (2/4)

W2. About the separation between construction data and evaluation references.

A: The construction inputs and the evaluation references are separate; the expert rubrics of Table 4 never enter rubric construction. The four items the reviewer lists are now given in Appendix H.1, summarized below.

(1) Training query counts. The RL data are identical to DR Tulu's; the SFT data follow DR Tulu's construction pipeline and add the multipaper_qa subset of OpenScholar (Table H1).

StageCountComposition
SFT27,869 trajectories16,052 built with DR Tulu's pipeline (long-form 10,251: OpenScholar 5,704, Search Arena 3,547, ScholarQA 1,000; short-form 5,801: HotpotQA 1,176, MegaScience 814, TaskCraft 583, WebWalkerSilver 1,438, BrowseComp 916, PopQA and TyDi QA 874; the ScholarQA, HotpotQA and MegaScience parts, which DR Tulu did not release, are regenerated following its Appendix F.1), plus 11,817 from OpenScholar multipaper_qa
RL8,881 long-form promptsSearchArena and OpenScholar 4,881, RaR-Science 3,000, RaR-Medicine 1,000

(2) Split identifiers. We define no split of our own; all training data are used only for training. For Table 4, we hold out 20 HealthBench-Consensus conversations and 5 ResearchRubrics queries from the official benchmarks as few-shot examples and evaluate on the remaining 3,651 conversations and 96 queries. Training and evaluation queries come from different sources, as in DR Tulu, CaRR and ASearcher.

(3) Reference-report sources. Each query has 10 reference reports, generated independently by four deep research systems (Gemini Deep Research, OpenAI Deep Research, Perplexity Deep Research and Grok Deeper Search) and six models equipped with the same search tools (GPT-5, Gemini-2.5-Pro, Claude-Sonnet-4, DeepSeek-R1, Kimi-K2-Instruct and GLM-4.5). Each system sees only the query (Appendix H.1.2).

(4) Overlap. None, in either direction. The expert rubrics do not come from reference reports: the HealthBench-Consensus criteria were pre-written by physicians and assigned to conversations by majority agreement, and the ResearchRubrics criteria were written by experts directly from the prompt without assuming a reference answer. The reference reports never see the expert rubrics: generation receives only the query, and the expert rubrics are used only to evaluate Table 4. The evaluation references of Table 4 are therefore independent of the construction inputs, and all methods are compared with the expert rubrics on the same candidate responses.

Revision: Appendix H.1.1 gives the training data and Table H1; H.1.2 states the reference-report sources and why they do not overlap with the expert rubrics; H.1.3 and Tables H2–H4 list the evaluation data.

AuthorsOpenReview ↗

Response to Reviewer xVvx (3/4)

W3 & Q2. About the details needed to implement and train the method, and the training and evaluation settings.

A: We have added every detail the reviewer lists; the values are given below.

(1) Prompts and operational rules for LowDisc, Δ, and Eq. 4. At each step and for each query, we rescale the scores of every rubric in R_q^0 on the K = 8 rollouts to [0, 1] and take the five rubrics with the lowest standard deviation as LowDisc candidates; ties are broken by the higher mean score, then by the larger global weight w_i α_r. In one call, the rubric generator lists Δ_r^t for each candidate, i.e., the response differences that fall within its evaluation content and direction but are not separated by its judgment conditions; candidates with Δ_r^t ≠ ∅ form A_t(q) in Eq. 4. Each such rubric is decomposed into at most five finer criteria, including one shared criterion, whose integer weights are rescaled to α_r. A decomposition replaces r only if it passes review (preservation, scope, redundancy, overlap, conflict) and its weighted score has a larger standard deviation than r on the same responses; otherwise r is retained. Each step starts again from R_q^0. Appendix H.5 gives the full prompt of every step of the pipeline and Algorithm 1, and Section 3.3 now states the selection rule after Eq. 4.

(2) Values missing from Table 6. Table 6 is replaced by Tables G1 (SFT) and G2 (RL). RL uses AdamW (β = (0.9, 0.95), weight decay 0), a constant learning rate of 5 × 10⁻⁷, 32 prompts per batch, 8 rollouts per prompt, and a KL coefficient of 0.001. The full method and every ablation train for 2,080 steps (7.5 epochs), using about 2,770 GPU hours per run on eight H20 GPUs, with three seeds (42, 123, 456). These budgets replace "at least 75 steps", and Section 4.1 summarizes the core training settings.

(3) G.3 and H.2.8. G.3 now points to Tables G1 and G2. H.2.8 now defines the target score θ of each benchmark as the highest baseline score at step 2,080, and states that GPU-hours cover policy training only, excluding API calls to the rubric generator and the judge. The revision no longer reports average rubric length, so the tokenizer item is removed.

(4) Query counts, splits, and evaluation settings. See our answer to W2: RL uses 8,881 queries, and we define no custom split (Appendix H.1). The evaluation settings are given in W1 & Q1 and Appendix Table H2.

Revision: Section 3.3 (one sentence after Eq. 4); Section 4.1 (training details); Appendix G (Tables G1, G2); Appendix H.2.8; Appendix H.5 (operational rules, the prompts of every step, and Algorithm 1).

AuthorsOpenReview ↗

Response to Reviewer xVvx (4/4)

W4. About evaluation counts and uncertainty.

A: The revision now states the evaluation counts of Tables 1, 3 and 4 in the main text, and Appendix I.8 adds run-to-run standard deviations, query-level 95% intervals and paired permutation tests.

Counts. The preference columns of Tables 1 and 3 use 298 non-tied DRB preference pairs and 1,176 human annotations (198 ties) on 588 SQAv2 report pairs. The downstream columns of Table 3 use the 132 DeepResearch Bench II tasks and the 776 ResearchQA questions. Table 4 uses 3,651 HealthBench-Consensus conversations and 96 ResearchRubrics queries, after holding out 20 conversations and 5 queries as few-shot examples. These counts are stated in Secs. 4.1 and 4.4, the captions of Tables 3 and 4, and Appendix Table H4.

Uncertainty. We report two sources separately. (1) Run-to-run variation: preference evaluation and Table 4 regenerate the rubrics three times, and each downstream variant in Table 3 is trained with three seeds and evaluated three times per seed. Means ± sample SDs are in Appendix Tables 7 (for Table 1), I2 (Table 3) and I3 (Table 4), and Table I4b lists the Table 3 scores of each training seed. (2) Query sampling: preference comparisons in Tables 1 and 3 are paired within each rubric generation, and downstream comparisons in Table 3 are paired within each training seed after averaging the three evaluations of each query. 10,000 query-cluster bootstrap samples and 10,000 paired permutations, keeping all report pairs and annotations of a query together, give 95% intervals for the differences and two-sided p-values. For Table 4, we report paired differences and permutation p-values, with Holm correction over the 13 metrics of each comparison.

Results. (1) Table 1: the differences between S2R (GPT-5) and the strongest baseline in each column are positive in all three rubric generations and under both scoring judges. The accuracy differences are +0.34 to +1.34 percentage points on DRB and +1.36 to +1.70 on SQAv2, and the paired Cohen's d differences are +0.023 to +0.043 on DRB and +0.014 to +0.034 on SQAv2. All of their 95% intervals include zero (Appendix Table I5b). (2) Table 3: for the preference columns, the intervals of all seven ablations include zero in all three rubric generations. On downstream RQA, w/o Stage I (Rubric Refinement), w/o Structured Scope, w/o Hierarchical Review (Flat Review), w/o Fixed Scope and w/o Scope Constraint give p < 0.05 for every training seed, and w/o Stage II and w/o Response-Aware Decomp. (Static Rubric Decomp.) for two of the three seeds. (3) Table 4: for full S2R versus RubricEM, 11 of the 13 metrics have Holm-adjusted p < 0.05; the ResearchRubrics τb and Top-1 have unadjusted p = 0.16 and 0.18, and Holm-adjusted p = 0.32 for both.

Revision: Secs. 4.1 and 4.4 and the captions of Tables 3 and 4 state the counts; Sec. 4.2 reports the Table 1 results as point estimates and Sec. 4.3 the downstream tests for Table 3, and the Table 1 caption points to Appendix I.8; Appendix Table H4 gives the evaluation splits, and I.8 reports run-to-run variation (Tables 7, I2, I3 and I4b), query-level intervals and paired tests for Tables 1 and 3 (Tables I5b and I5c), and paired tests for Table 4 (Table I3c).


Summary. The reviewer's concerns were the mismatch between curves and tables, possible overlap between construction and evaluation data, missing implementation details, and missing uncertainty. The curves and Tables 2–3 now share one evaluation setting, and each number states its run, checkpoint, subset and scorer (W1 & Q1); construction and evaluation data do not overlap (W2); every requested implementation and training value is given (W3 & Q2); and the comparisons now carry evaluation counts, run-to-run standard deviations and query-level intervals (W4). If these resolve the concerns, we would appreciate the reviewer's reassessment; if not, we would be glad to know what remains.

Reviewer_CTpQOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes Scope2Rubric(S2R), a two-stage query-specific rubric for long-form deep research. Stage I extracts full task requirements from query and reference analyses, groups them into weighted evaluation directions and generates and reviews rubrics within each direction at three levels. Stage II runs during GRPO training: it finds rubrics that do not separate the current rollouts and splits them inside their fixed direction, keeping the direction and rubric weights unchanged. Experiments on human-preference alignment shows S2R achieves 76.39% and 53.15% accuracy on DeepResearchBench and ScholarQA-CS2.

Critical strengths
  1. The problem is well motivated. The core idea, fixing what to evaluate before deciding how finely to evaluate it, is clear and reasonable. The theoretical analysis in the appendix formalizes this separation.

  2. The evaluation goes beyond preference accuracy and downstream performance. It also covers direction quality, alignment with expert rubrics, individual rubric quality, and redundancy within and across directions. S2R performs well on these analyses.

Critical weaknesses
  1. The training curves appear inconsistent with the reported results. In Figure 4, the S2R curves end around 45 on HealthBench and 68 on SQAv2, while Table 2 reports 65.2 and 90.4, respectively. Similar discrepancies appear in Figure 8.
  2. The paper does not clearly specify the RL training data, reference report sources, GRPO hyperparameters, or the criterion for identifying low-discriminability rubrics.
  3. There is no Stage-I-only baseline to isolate the contribution of dynamic rubric decomposition. The additional LLM inference cost of scope recovery and rubric refinement is also not quantified.
  4. The downstream comparisons are not well controlled. The methods should be compared under the same backbone and training budget.
  5. There are citation format issues. For example, “ResearchQA Yifei et al. (2026)” should be “ResearchQA (Yifei et al., 2026)”. The paper also contains what appears to be unedited AI-generated text, e.g., “...represented by the inherited SQAv2 column requires confirmation before these names can be treated as interchangeable”.
Questions for authors
  1. Why do the y-axes and endpoints of Figures 4 and 8 differ from the final scores in Tables 2 and 3?
  2. What are the results of the same-checkpoint runs of Quest, DR Tulu, and Tournament-GRPO shown in Figure 8?
  3. Which queries are used for RL training, and where do the reference reports used in Stage I come from?
  4. How many preference pairs does each benchmark contain, and how are ties and chance-level accuracy on SQAv2 handled?
  5. Which judge scores each downstream benchmark, and how does it relate to the Gemini-2.5-Pro reward judge?
  6. What are the generation and judging token costs of Stage I and of Stage II per training step?
  7. How did you obtain the unstarred baseline scores in Table 8?
4 comments
AuthorsOpenReview ↗

Response to Reviewer CTpQ (1/4)

We thank the reviewer for the clear review. The reviewer finds the core idea, "fixing what to evaluate before deciding how finely to evaluate it", "clear and reasonable"; the concerns are about whether the results behind it can be trusted. Changes are marked in blue in the revised PDF.


W1 & Q1. About the training curves (Figs. 4, 8) vs. the reported results (Tables 2–3), and why their y-axes and endpoints differ.

A: The curves end below Table 2 because they stop at step 800, whereas Tables 2 and 3 report the model at the end of training (step 2,080). The submitted curves were also scored during training under a different evaluation setting. We have re-evaluated every curve under the Table 2 setting.

Under the unified setting, step 0 of each curve equals the Qwen3-8B-SFT row of Table 2, and step 2,080 equals the S2R row. The scores of S2R (Gemini 3) at each checkpoint are below.

S2R (Gemini 3)DRBDRB2HealthBenchResearchQASQAv2
Submitted Fig. 4 endpoint (as cited)––~45–~68
Step 0 (= Qwen3-8B-SFT row, Table 2)38.6730.0437.9268.8472.82
Step 800 (last point in Fig. 4)45.3137.7648.2973.1684.30
Step 1,60052.4445.7459.6278.8088.76
Step 2,080 (= Table 2)54.6949.0664.9280.5890.21

Step 800 is 38% of the 2,080-step schedule, and the scores keep rising afterwards: HealthBench goes from 48.29 to 64.92 and SQAv2 from 84.30 to 90.21. The submitted figures showed only the first 800 steps, and each panel's y-axis covered only the score range of that interval, so its upper limit was below the final scores in Tables 2 and 3. The three curves in Figure 4 come from the same training runs as Table 3. Their ResearchQA scores at step 2,080 are 80.58, 77.78 and 77.12, identical to the corresponding rows of Table 3. The curves in Figure 8 are re-evaluated under the same setting.

Revision: Figures 4 and 8 now show curves evaluated under the Table 2 setting; Section 4.3 states that Tables 2 and 3 report the step-2,080 checkpoint; Appendix H.3 gives the evaluation setting of both figures and the scores at steps 1,200, 1,600 and 2,080.


W2 & Q3. About the RL training data, reference-report sources, GRPO hyperparameters, and the low-discriminability criterion.

A: The revision now gives these four items in Appendix H.1, Table G2, and H.5; the values are below.

(1) RL training data (first part of Q3). RL uses all 8,881 long-form prompts released by DR Tulu: 4,881 from SearchArena and OpenScholar, 3,000 from RaR-Science, and 1,000 from RaR-Medicine (Table H1). These queries are used only for training and come from different sources than the evaluation benchmarks.

(2) Reference-report sources (second part of Q3). Each query has 10 reference reports, generated independently by four deep research systems (Gemini Deep Research, OpenAI Deep Research, Perplexity Deep Research, and Grok Deeper Search) and six models equipped with the same search tools (GPT-5, Gemini-2.5-Pro, Claude-Sonnet-4, DeepSeek-R1, Kimi-K2-Instruct, and GLM-4.5). Each system sees only the query (Appendix H.1.2).

(3) GRPO hyperparameters. They follow DR Tulu: AdamW (β = (0.9, 0.95), weight decay 0), a constant learning rate of 5 × 10⁻⁷, and a KL coefficient of 0.001; 32 prompts per step with 8 rollouts each, temperature and top-p of 1.0, responses of up to 16,384 tokens, and at most 10 tool calls. The full method and every ablation train for 2,080 steps (7.5 epochs) with three seeds (42, 123, 456) (Table G2).

(4) Low-discriminability criterion. A rubric that gives nearly the same score to the 8 responses of a query cannot separate them. Concretely, at each step and for each query, we rescale the scores of every initial rubric on the 8 responses to [0, 1] and take the five rubrics with the lowest standard deviation as candidates, breaking ties by higher mean score. A candidate is replaced only if its decomposition passes review and the finer criteria spread the scores of the same 8 responses more widely than the original rubric; otherwise it is kept. Appendix H.5 gives the full rules, Algorithm 1, and the prompts.

Revision: Table G2 in Appendix G; H.1.1 and Table H1; H.1.2; and the operational rules, Algorithm 1, and prompts in H.5.

AuthorsOpenReview ↗

Response to Reviewer CTpQ (2/4)

W3 & Q6. About the Stage-I-only baseline, and the token costs of Stage I and Stage II.

A: We have added both. With Stage I only, all four metrics are about one point below the full method; Stage I runs once per query before training, and Stage II adds about half to the judging tokens of each step.

Stage-I-only baseline. Table 3 of the revision adds w/o Stage II, which trains with the initial Stage I rubrics throughout and otherwise matches the full method; both use Gemini 3 as the rubric generator. Preference accuracy averages three rubric generations, and downstream scores average three training seeds:

VariantDRBSQAv2DRB2RQA
Stage I only (w/o Stage II)75.3952.1348.0579.51
Full S2R76.4053.1549.0680.58
Gain from Stage II+1.01+1.02+1.01+1.07

With query-paired tests, the RQA gain has p < 0.05 in two of the three training seeds, while the 95% intervals of the DRB2 gain and both preference-accuracy gains include zero (Table I5c). Stage II therefore gives a consistent but small improvement. This improvement comes from response information: a static control that splits the rubrics to the same item count without reading responses is only 0.12–0.23 above Stage I only (Table 3).

Cost of Stage I. Stage I builds each query's rubrics once before training and is not called during training. Measured on five queries each from DRB and SQAv2 (Gemini 3, averaged over three generations per query), it uses 23.7 calls and about 253K tokens per query (230K input and 24K output). Of these, 15.1 generation calls use about 179K tokens and 8.6 review calls use about 75K tokens. A control that generates rubrics directly without review uses 9.3 calls and about 95K tokens per query.

Cost of Stage II per training step. Each step has 32 queries with 8 rollouts each. On the generation side, Stage II makes one decomposition call per query and one review call when finer rubrics are produced, using about 0.95M tokens per step. On the judging side, scoring the finer rubrics adds about 0.68M tokens per step; scoring the initial rubrics, which is needed without Stage II as well, takes about 1.41M tokens per step, so Stage II increases judging tokens by about 48%. These figures assume that about half of the queries yield finer rubrics, which matches our measurement: on average, 5.28 of the 10.94 initial rubrics per query are decomposed during training (48.3%). With GPT-5 as the generator and 2,080 training steps, one S2R training run costs about USD 18,710 in API fees, about USD 5,540 more than RubricEM.

Revision: Table 3 and Section 4.3 now include the Stage-I-only baseline; Tables I2 and I5c in Appendix I.8 give its standard deviations and paired tests; Table I4c reports the calls and tokens of Stage I, and Table I4d the generation and judging usage and cost of each training run.


W4 & Q2. About comparing under the same backbone and training budget, and the same-checkpoint runs in Figure 8. A: We have compared the methods under the same backbone and training budget, and S2R leads on all five benchmarks. All methods start from the same Qwen3-8B-SFT checkpoint and share the training queries, tools, rollout budget, steps (2,080), rubric generator and reward judge with S2R. Mean results over three training seeds:

Controlled blockDRBDRB2HBRQASQAv2Avg.
Qwen3-8B-SFT38.6730.0437.9268.8472.8249.66
Tournament-GRPO51.4741.1254.6878.5486.7662.51
OpenRS (PAMR)47.8337.7253.8477.2183.9260.10
RubricEM51.4742.3154.0878.7887.6362.85
S2R (GPT-5)53.1645.3659.4682.4689.6366.01

For the three same-checkpoint baselines in Figure 8, the five-benchmark averages at step 2,080 are 61.42 for QUEST-8B, 59.66 for DR Tulu and 62.51 for Tournament-GRPO, compared with 66.01 for S2R (GPT-5). All four methods use GPT-5 as the rubric generator; per-benchmark scores are in the Figure 8 supplemental-checkpoint table in Appendix H.3. The remaining groups in Table 2 now serve only as system-level context.

Revision: the controlled block of Table 2, Figure 8 and its caption, and the Figure 8 supplemental-checkpoint table in Appendix H.3.

AuthorsOpenReview ↗

Response to Reviewer CTpQ (3/4)

W5. About the citation format and the unedited text.

A: Thank you for catching these; both are corrected. Parenthetical citations now use \citep throughout (e.g., "ResearchQA (Yifei et al., 2026)"); following the ICLR template, \citet is kept only where the authors are part of the sentence.

We have deleted the quoted sentence, deleted one similar statement in H.2.2 and revised another in H.2.3, and stated the verified information in the appendix: SQAv2 denotes ScholarQA-CS2 from AstaBench, evaluated on the sqa_test task of astabench 0.5.4 (the 100-question official test set). H.1.3 gives the release, and Table H2 the scoring. The SQAv2 column of the revised Table 2 follows this protocol.


Q4. Preference pairs per benchmark, and how ties and near-chance SQAv2 accuracy are handled.

A: DRB has 298 preference pairs, and SQAv2 has 588 report pairs with 1,176 human annotations. Ties are kept as human labels, and a prediction counts as correct only if it matches the human label exactly. Under this scoring, chance on SQAv2 is 41.58%, not 50%. The gap between S2R and the strongest baseline lies within sampling error, and the revised paper no longer presents it as a significant advantage.

Pairs and ties. DRB yields 298 preference pairs from its official human scores; report pairs with equal human scores are excluded. SQAv2 uses 588 of the 600 official report pairs; for the remaining 12 development pairs, the official release contains two different reports, so the annotated report cannot be identified (Appendix Table H4). Two annotators label each report pair separately, giving 1,176 annotations, and all 198 tie annotations are kept. A prediction (A wins, B wins, or tie) counts as correct only if it matches the human label exactly: score-based methods predict a tie only when the two scores are exactly equal, and failed predictions count as errors with no half credit.

Why near 50% is not chance. Guessing A or B at random gets half of the 978 directional annotations right and none of the 198 ties, so its accuracy is only 41.58%. The two annotators agree on only 330 report pairs; on the other 258, any single prediction can match at most one annotator, so the highest attainable accuracy is 78.06%.

Results and uncertainty. With GPT-5 as the shared rubric generator, S2R reaches 53.09% on SQAv2 and the strongest baseline, RubricEM, 51.45%; on DRB the two reach 76.06% and 74.94%. With query-paired bootstrapping within each rubric generation, S2R's SQAv2 gain over RubricEM is +1.62 to +1.70 points, with 95% CIs of about (−1.9, 5.1) and paired permutation p = 0.36–0.39; its DRB gain is +0.67 to +1.34 points, with p = 0.73–0.90. All of these gaps lie within query-sampling error.

Revision: Section 4.1 now states the pair counts, tie rule, chance level, and ceiling; Section 4.2 reports the Table 1 results as point estimates only; Appendix H.4 gives the definitions, Appendix I.8 the per-run intervals and tests, and Table 7b the 95% intervals over the three runs.


Q5. Which judge scores each downstream benchmark, and how it relates to the Gemini-2.5-Pro reward judge.

A: DRB and DRB2 are scored by Gemini-2.5-Pro, the same model as the reward judge; HealthBench, ResearchQA, and SQAv2 are scored by GPT-4.1, GPT-4.1-mini, and Gemini-2.5-Flash, respectively, and Gemini-2.5-Flash belongs to the same model family as the reward judge.

Appendix Table H2 lists the evaluated items, judge, and score computation of each benchmark. For DRB and DRB2, whose judge is the reward judge, we rescore the same reports of the controlled block in Table 2 with DeepSeek-V3.2, keeping the RACE aggregation for DRB and the fraction of passed criteria for DRB2. Under DeepSeek-V3.2, S2R (GPT-5) exceeds the strongest controlled baseline by 4.65 points on DRB and 4.63 points on DRB2 (46.27 vs. 41.62 and 38.42 vs. 33.79); the gaps under Gemini-2.5-Pro are 1.69 and 3.05 points. Its advantage over the controlled baselines therefore does not arise only under the reward judge. Its gains over Qwen3-8B-SFT are 1.90 and 1.69 points smaller under DeepSeek-V3.2. DeepSeek-V3.2 gives every method lower absolute scores, so we do not attribute this difference solely to fitting Gemini-2.5-Pro, but we cannot rule that out either.

In the revised paper, Section 4.1 now states the judge of each benchmark and its relation to the reward judge, and Section 4.2 reports the rescoring results; the last two columns of Appendix Table 8 give the full scores, and Appendix H.4 gives the setup.

AuthorsOpenReview ↗

Response to Reviewer CTpQ (4/4)

Q7. How the unstarred baseline scores in Table 8 were obtained. A: We obtained them ourselves under the unified protocol of Sec. 4.1; they are not copied from the original papers. They come from four sources. We trained and evaluated the controlled block from the same Qwen3-8B-SFT checkpoint. We ran proprietary systems through their official APIs, and open systems with the code or weights released by their authors. Rubric RL methods without released weights, such as Tournament-GRPO and ARISE-RL, were retrained with their released code and then evaluated. Every row uses the same items and judges: Gemini-2.5-Pro scores DRB and DRB2, and GPT-4.1, GPT-4.1-mini, and Gemini-2.5-Flash score HealthBench, ResearchQA, and SQAv2 (Table H2). Table 8 is the complete version of Table 2; its revised caption states this, and the new Appendix Table H6 lists the source of every row's scores and the configuration of each original system. Our claims about the reward construction rest on the controlled comparison (see W4 & Q2).


Summary. The reviewer's main concerns were the mismatch between curves and tables, the uncontrolled downstream comparison, and the missing Stage-I-only baseline and cost. The curves and tables now share one evaluation setting (W1 & Q1); under the same backbone, training budget, generator and judge, S2R leads the strongest controlled baseline by 3.16 points on average (W4 & Q2); Stage I alone is about one point below the full method, and the costs of both stages are reported (W3 & Q6). If these resolve the concerns, we would appreciate the reviewer's reassessment; if not, we would be glad to know what remains.

Reviewer_S2hxOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The authors propose a training free method of generating task-specific rubrics for long-form deep research tasks. The pipeline at a high level has two stages:

(I) given a task, the pipeline generates all query specific requirements along with an evaluation direction which also has the spice fic criteria, weight, content to be included and boundary. All directions together form the scope of evaluation. This step ensures coverage of requirements. This step also does hierarchical refinement of the rubric.

(II) in the second stage the pipeline decomposes rubrics that fail to separate responses into finer criteria

The authors also train downstream LLMs with GRPO using rubric as reward and show the effectiveness of the rubrics they generate.

The authors compare their system on two axes: (a) preference accuracy (alignment) and (b) downstream task performance with the post-trained models. They evaluate (a) on two datasets/tasks and (b) on 5 datasets.

For human alignment, they show their system achieves the best preference accuracy and cohen’s d with the two datasets. Same for the downstream task performance. The paper also shows that the rubrics generated by their proposed method are less clustered together and represent different dimensions. They also show that training with their rubrics leads to more efficient training along with downstream generalization.

Critical strengths

(a) the paper tackles an important aspect of rubric creation: coverage and discriminability

(b) the paper has a nice inference-only method of generating rubrics

(c) the experiments are well thought out: evaluating the rubrics and showing the downstream effectiveness of the rubrics is good

Critical weaknesses

(A) The main paper is lacking an example that can help a reader go through the proposed pipeline in Section 3 (there is an overview figure but that does not have an example).

(B) It is unclear how the implicit requirements in Stage 1 are derived (line 172). The paper gives a reference but it does not have any more information regarding this.

(C) It’s unclear which responses Stage II scores. Are they policy rollouts in training? Or, benchmark candidate reports in evaluation?

(D) Missing prompts etc for the pipeline, reproducibility is an issue

(E) In the results, experiments compare against baselines and take numbers from prior work instead of running on their own. The paper is missing details on lets say the different models used by the baselines. For example, Dr Tulu also uses LLM as a judge when doing evolving rubrics, but details around the LLM used there are missing. The higher result number we see in Table 1/2 could also be a factor of the proposed method using stronger LLMs (like Gemini and GPT-5), but there is no control around it.

(F) No significance reporting

(G) A major chunk of the details etc are in the appendix eg: how Figure 5 is calculated.

Writing/missing details:

(i) Fix the citations; use ~\citep{}

(ii) Missing computation costs

(iii) Unclear what the dataset sizes etc are

(iv) Figure 4 and Table 3 are missing what datasets they report the metrics on

(v) Near line #400, Figure 3 is mentioned, but it doesn’t really fit the context or support the claim in the paragraph.

(vi) It is unclear what dataset Figure 6 represents

(vii) Sentence 1705 might contain a left over coding-agent text, which might be pointing to incomplete information about the implementation.

Questions for authors

(A) How many rubrics are decomposed in Stage II compared to the rubrics generated in Stage I? How many refinement rounds does Stage I do? Do all baselines in Figure 7 do the same number of steps of refinement?

(B) Can you give examples of scope and directions? I see in Figure 9 there is an example, but all the directions seem to be content:semantic? Do you also discover other aspects that are non semantic, like factuality or tone or style?

(C) How do you choose responses for stage 2?

(D) In the results: do you have performance (preference accuracy) after stage 1 (for Table 3)? To isolate the effect of stage 2? Also for Table 4 what is the precision/recall after Stage 1?

(E) Did you observe any reward hacking behavior like length increase etc with the rubrics?

(F) Do the different datasets cover different capabilities/domains etc? Did you notice any differences in the performance based on the domains?

7 comments
AuthorsOpenReview ↗

Response to Reviewer S2hx (1/5)

We thank the reviewer for the thorough review, down to individual lines of the appendix. The reviewer notes that the paper "tackles an important aspect of rubric creation: coverage and discriminability"; the concerns are that the pipeline is hard to follow and reproduce, and that the gains may come from stronger LLMs rather than from the method. This response addresses both, with the derivation steps and all prompts, and with comparisons in which the LLMs are held fixed. Changes are marked in blue in the revised PDF.


W-B. How the implicit requirements in Stage 1 are derived (line 172). A: They are derived in four steps, now described in Sec. 3.2, with the full prompt in Appendix H.5. (1) S2R analyzes the query's task context along five constraints: task intent, user intent, domain norms, audience, and output genre. (2) These constraints determine which of seven research stages, from defining the task object to reviewing boundaries and uncertainty, an ideal report requires. (3) Candidate requirements are proposed only within the required stages. (4) A candidate is retained only if it meets five conditions: the task context supports it, it lies within its stage's scope, omitting it causes a failure rather than only an optional loss, it is not a generic quality requirement, and no existing direction covers it. Retained requirements that serve one evaluation function form an implicit direction.

Example. For the ResearchQA query "What makes the Microsoft COCO Dataset important for research in vision and language, and what are its key features?", the explicit requirements are COCO's research value and its key features. "What makes … important" implies a third requirement: explaining how specific features support specific research tasks. In a human-labeled pair, response A links COCO's 80 categories and varied object interactions to models that understand and describe complex scenes, whereas B lists more specific features (e.g., multiple reference captions per image) without explaining how they provide training or evaluation signal. With the same per-item scores, explicit directions alone prefer B (0.794 vs. 0.847), removing only this implicit direction still prefers B (0.838 vs. 0.880), and the full rubric set prefers A (0.866 vs. 0.797), matching the human preference.


W-C & Q-C. Which responses Stage II scores, and how they are chosen. A: Both, depending on the setting; each setting uses all of its responses, without selection. In RL training, Stage II uses all eight rollouts sampled from the current policy for each query, the same group from which GRPO computes advantages. Evaluations have no policy rollouts, so Stage II uses a fixed set of candidate responses per query: the preference evaluation of Table 1 uses the two judged reports of each pair, the same input given to the response-grounded and hybrid baselines, and no method sees the human preference labels during rubric construction; Tables 4 and 9 use the Table 4 candidates on ResearchRubrics (one from each of eight models) and additional responses from the same eight models on HealthBench-Consensus and DeepResearch Bench II; Figure 7 uses eight Qwen3-8B-SFT responses per query, shared by all four methods. We have stated both settings in Sec. 3.3, and Appendix H.5 lists the responses used in each evaluation.


W-D. About the missing prompts and reproducibility. A: Appendix H.5 now gives the prompt of every step of the pipeline, and the models and run settings are now stated.

The prompt boxes in H.5 cover, in Stage I, explicit direction extraction, implicit direction discovery, supplementary discovery from one reference report, cross-report aggregation and boundary refinement, direction review and revision, direction weighting, rubric generation, and three-level rubric review; in Stage II, decomposition and the review of decompositions; and reward judging. Each box gives its inputs and JSON output schema. H.5 also gives the Stage I review limits and stopping rule (at most 10 rounds per level, 3.3 on average), the Stage II rules for selecting candidates and accepting decompositions, and the full training procedure (Algorithm 1). The rubric generator is GPT-5 (with Gemini 3 and Qwen3-8B variants of S2R) and the reward judge is Gemini-2.5-Pro; Appendix Tables G1 and G2 give the SFT and RL configurations, and Table H5 the shared construction settings.

Revision: Appendix H.5 (all prompt boxes, the Stage I and Stage II rules, and Algorithm 1); Appendix Tables G1, G2, and H5.

AuthorsOpenReview ↗

Response to Reviewer S2hx (2/5)

W-E. About the baseline numbers and the control for stronger LLMs. A: We have re-run the baselines, listed the models each baseline uses, and fixed the LLMs in Tables 1 and 2. With the LLMs fixed, S2R still leads.

(1) All unstarred cells in Table 2 were run by us under the protocol of Sec. 4.1; starred cells are published scores obtained under the same protocol. (2) The new Appendix Table H6 lists the backbone, rubric generator and reward judge of every baseline; for example, DR Tulu uses GPT-4.1 to generate rubrics and GPT-4.1-mini as the judge.

(3) In Table 1, every rubric method uses GPT-5 as the generator and Gemini-2.5-Pro for scoring; task-grounded and hybrid methods receive the same 10 reference reports per query, response-grounded and hybrid methods refine rubrics with the two reports of each preference pair, and every refining method runs four refinement rounds under the same per-call limits (Appendix Table H5). S2R reaches 76.06% and 53.09% accuracy on DRB and SQAv2, compared with 74.94% and 51.45% for the strongest baseline (RubricEM in both columns). With the smaller open Qwen3-8B as the shared generator, S2R exceeds RubricEM by 2.35 and 2.84 points, and its average accuracy (62.99%) still exceeds that of Tournament-GRPO (62.38%) and QUEST (62.09%) with GPT-5. Under DeepSeek-V3.2 rescoring, S2R's average-accuracy gain over RubricEM remains positive with all three generators. The controlled block of Table 2 starts from the same Qwen3-8B-SFT checkpoint and retrains the strongest Table 1 baseline of each rubric type (Tournament-GRPO, OpenRS (PAMR), and RubricEM) with the same 8,881 RL queries, training and tool budgets, GPT-5 generator, and Gemini-2.5-Pro reward judge, reporting means over three training seeds; S2R averages 66.01 over five benchmarks, compared with 62.85 for the strongest baseline.

The advantage therefore holds with the LLMs fixed. See Table 1, the controlled block of Table 2, Sec. 4.2, and Appendix Table H6 (H.3).


W-F. About significance reporting. A: We have added it. Appendix I.8 now reports run-to-run means ± standard deviations, 95% query-clustered bootstrap intervals for paired differences, and paired permutation tests for Tables 1–4 (Tables I5b, I5c and I3c). Preference results are paired by query within each rubric generation, and downstream results by query within each training seed; every comparison reports the difference interval and a two-sided p-value from 10,000 permutations, and Table 4 applies a Holm correction to the 13 tests of each comparison.

(1) Table 1: relative to RubricEM, the strongest baseline, S2R (GPT-5) gains 0.67–1.34 percentage points on DRB and 1.62–1.70 on SQAv2; the 95% intervals include zero in all three rubric generations (p = 0.36–0.90), so these gaps lie within query-sampling error, and Sec. 4.2 therefore reports point estimates only. (2) Controlled block of Table 2: S2R (GPT-5)'s five-benchmark average exceeds RubricEM with intervals of (2.01, 4.14), (2.13, 4.25) and (2.14, 4.30) over the three training seeds, each with p = 0.0001; on HealthBench and ResearchQA, p = 0.0001 against the strongest baseline in all three seeds, while on DRB the intervals include zero in all three seeds. (3) Table 3: on ResearchQA, five of the seven ablations give p < 0.05 in all three seeds; the intervals for both preference-accuracy columns include zero. (4) Table 4: against both RubricEM and Stage I alone, the full method's precision, recall and F1 have Holm-adjusted p ≤ 0.0013 on both datasets.

Revision: Secs. 4.2 and 4.4 now state the test results for the controlled block of Table 2 and for Table 4, and Sec. 4.3 states the downstream tests for Table 3; Appendix I.8 gives the full results.


W-G. Details only in the appendix. We have moved these definitions into the main text, keeping judges, formulas, and prompts in the appendix. (1) Figure 5 (Sec. 4.4) now states the evaluation sets (all 101 ResearchRubrics prompts and all 132 DeepResearch Bench II tasks), the comparators, what each of the five metrics computes, and the preference pairs used for direction-weight utility (2,828 and 2,708 report pairs labeled by expert-rubric scores). Its configurations and formulas, previously spread over four places in the appendix, are merged into Appendix H.2.4. (2) Figure 7 (Sec. 4.4) now defines within-direction coverage, semantic redundancy, score dependence, and within-direction weight utility. (3) Table 4 (Sec. 4.4) now defines semantic precision, recall, and F1. (4) Figure 8 (Sec. 4.4) now states that the target score for steps to threshold is the strongest baseline's score at the end of training. (5) Short-form transfer (Sec. 4.5) now states the judge, the metric, and the number of sampled questions.

AuthorsOpenReview ↗

Response to Reviewer S2hx (3/5)

W-(i). Citations. Fixed: parenthetical citations now use \citep throughout; \citet remains only where the authors are part of the sentence.


W-(ii). Computation cost. We have added it; Appendix I.6 summarizes all costs. (1) Stage I builds the rubrics of each query once before training: 23.7 calls and about 253K tokens per query (Table I4c). (2) Stage II runs at every training step: decomposition and review take about 0.95M tokens per step, and judging the finer rubrics adds about 0.68M tokens per step, about 48% more than the roughly 1.41M judging tokens needed with the initial rubrics alone. (3) Each RL run trains for 2,080 steps and uses about 2,770 H20 GPU-hours (Table G2); in the controlled block, the API cost of rubric generation and reward judging is about USD 18,710 per training seed for S2R (GPT-5) and USD 13,170 for RubricEM (Table I4d).


W-(iii). Dataset sizes. Appendix H.1 now tabulates all data sizes: Table H1 gives the training data (27,869 SFT examples and 8,881 RL queries), Tables H2 and H3 give the downstream evaluation sizes (e.g., 100 DeepResearch Bench tasks, 776 ResearchQA test-mini questions, and 1,000 sampled HealthBench conversations), and Table H4 gives the rubric-quality evaluation data (e.g., 298 DeepResearch Bench preference pairs and 588 SQAv2 report pairs with 1,176 annotations). Section 4.1 now points to these tables.


W-(iv). Datasets of Figure 4 and Table 3. The five panels of Figure 4 report downstream scores on DR Bench, DR Bench2, HealthBench, ResearchQA, and SQAv2. In Table 3, the DRB and SQAv2 columns report preference accuracy on the DeepResearch Bench and SQAv2 preference pairs, and the DRB2 and RQA columns report downstream scores on DeepResearch Bench II and ResearchQA. The caption of each now states these datasets.


W-(v). Figure 3 near line 400. We agree. That paragraph argues for alignment with expert-written rubrics, and Figure 3 contains no expert rubrics, so it cannot support this claim. We have removed the reference there, leaving Table 4 as the evidence, and now discuss Figure 3 in the paragraph "Intra- and Inter-Direction Rubric Set Quality" of Sec. 4.4, where it supports a narrower claim: S2R's rubric clusters align with its recovered directions (ARI = 0.62) and are better separated than those of three baselines (silhouette 0.31 vs. 0.12–0.17). The caption now also specifies the dataset, embedding model, clustering method, and t-SNE settings.


W-(vi). Dataset of Figure 6. Figure 6 reports results on ResearchRubrics; the caption now states this.


W-(vii). Sentence 1705. This sentence is deleted and replaced by the settings we used. The scoring-dependence probes for Figure 7 follow RADAR, with each query as a task slice: GPT-5.5 (reasoning) generates 20 high-condition and 20 low-condition probes for each criterion, and Claude Sonnet 4.6 scores every probe on each criterion of the query on a 0–4 scale at temperature 0; scores are divided by 4, criterion pairs with a self-effect below 0.40 in either direction are excluded, and leakage is computed within each query before averaging. Two similar statements in the same part of the appendix now also state what we did: within-direction recall is averaged uniformly over directions within each query, and each correctness judgment is binary, with criteria that the evidence cannot support scored 0. See Appendices H.2.6 and H.2.7.


Q-A. Stage II decompositions, Stage I rounds, and refinement steps in Figure 7. (1) At each training step, Stage II decomposes 48.3% of the Stage I rubrics on average, i.e., 5.28 of the 10.94 rubrics per query. Each decomposed rubric is split only once, into at most five finer criteria (4.20 on average), which replace it in scoring. A query is therefore scored with 27.82 rubrics per step on average, compared with 10.94 after Stage I.

(2) During RL training, Stage I performs 3.3 refinement rounds per query on average, with a maximum of ten; it stops early when review finds no issue in a round or when a revised candidate fails the check. In comparisons that require a matched budget, namely rubric construction for Table 1 and Figure 7, every refining method runs a fixed four rounds.

(3) Yes. All four methods in Figure 7 undergo a fixed four rounds of the same response-aware decomposition: they use the Stage II decomposition and review prompts of Scope2Rubric and the same eight Qwen3-8B-SFT candidate responses per query, and each decomposes within its own directions. We have stated this setting in the caption of Figure 7 and in Appendix H.2.7, and Appendix H.5 now gives the Stage I round limit and stopping rule together with these prompts.

Reviewer_NnGjOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

Scope2Rubric proposes a two-stage framework for constructing query-specific evaluation rubrics for long-form research reports. Stage I extracts requirements from the query and reference reports, groups them into weighted evaluation directions, and generates rubrics with hierarchical review. Stage II decomposes insufficiently discriminative rubrics using candidate-response differences while preserving the direction boundaries and weight budgets. The resulting scores are also used as GRPO rewards.

Critical strengths
  1. Relevant problem and useful conceptual separation. Rubric generators can omit requirements shared failures do not expose, or reward salient but irrelevant differences between reports. Fixing a task-grounded scope before response-aware refinement directly addresses these failure modes.

  2. Sensible weight accounting. Preserving each direction’s total weight under decomposition prevents a direction from gaining importance merely because it acquires more rubric items. Equations (3), (5), and (7) make this aspect clear.

  3. Broad intended evaluation. The paper examines preference alignment, expert-rubric agreement, downstream RL, and transfer. Holding Gemini-2.5-Pro fixed as the scoring model for rubric-guided preference comparisons is a useful control.

Critical weaknesses

1. Essential implementation and training details are absent, including in the appendices

This is the most serious issue. The framework is described largely through instructions such as “review,” “verify,” “decompose,” and “revise until complete,” without specifying the procedures sufficiently for reproduction.

Missing details include:

  • The actual prompts and output schemas, despite Appendix H.5 being titled “Implementation and Prompts.”
  • The operational definition and threshold for LowDisc, and how unresolved task-relevant differences Δrt\Delta_r^t are detected.
  • Review stopping rules, maximum iterations, failed-generation handling, and the stated deterministic conversion from priority judgments to weights.

Table 6 supplies hardware and model-family names but no substantive optimizer or rollout hyperparameters; “at least 75 steps” is not an exact training budget. Appendix H.1.2 explicitly states that the ScholarQA release “requires confirmation,” and H.2.9 says that the 2Wiki evaluator still needs identification. These are unresolved details of the reported experiments, not optional refinements to reporting.

The long appendix therefore does not remedy the reproducibility gap.

2. The downstream evaluation does not isolate the effect of the proposed reward construction

Tables 2 and 8 combine published scores, newly obtained scores, different backbones, different system configurations, and some 100-sample evaluations. Such a table can provide context, but it cannot establish that S2R’s reward construction causes its advantage over other rubric methods.

The decisive experiment would retrain strong baselines from the same Qwen3-8B-SFT checkpoint, using the same training queries, references, tools, rollout budget, judge, optimization settings, and evaluation protocol. This is not established for the main baselines. Even the common judge in preference evaluation does not establish that rubric generators receive matched model capacity, references, or construction budgets.

The comparison also lacks construction-cost controls. S2R adds multiple extraction and review stages; an equally resourced rubric-first baseline is needed to distinguish the benefit of the intermediate representation from the benefit of additional inference.

3. The ablations do not adequately separate the core mechanisms

Table 3 is useful, but the variants are underspecified and do not isolate several central choices:

  • Stage I alone versus the complete two-stage method.
  • Response-aware decomposition versus an equally fine-grained static rubric set.
  • Hierarchical review versus flat review with matched inference budget.
  • Fixed direction weights versus alternative normalization.
  • Query-only scope recovery versus reference-assisted recovery.

Appendix I.5 discusses removing hierarchy, but does not report corresponding results. It also acknowledges that removing directions changes the review structure simultaneously. Consequently, the results do not cleanly attribute gains to the scope representation itself.

The explanation that unconstrained variants introduce irrelevant criteria is plausible, but should be supported by directly measured scope drift and task-irrelevant additions for those variants—not inferred solely from lower final scores.

4. Parts of the theoretical analysis are overstated or incomplete

Some appendix results are valid bookkeeping or error-decomposition observations, but stronger claims have gaps:

  • Proposition D.1: Normalized within-direction weights and boundary preservation do not imply equality with the target utility. The proof introduces the additional condition
∑rαrβrk=αk/wi,\sum_r \alpha_r\beta_{rk}=\alpha_k/w_i,

which requires exact preservation of atomic importance. This condition does not follow from the stated normalization requirement unless built explicitly into the definition of decomposition.

  • Proposition D.4: Preserving direction weights does not generally preserve the atomic importance vector used to define drift in Equation (22). With one direction containing two requirements, changing their effective weights from (0.5,0.5)(0.5,0.5) to (0.9,0.1)(0.9,0.1) preserves the direction budget but creates atomic drift of 0.4. The paper must distinguish direction-level invariance from atomic-level invariance.

  • Proposition D.3: The conditional-expectation argument does not establish that the method’s low-discriminability trigger is necessary. Refinement can reveal additional relevant information even when the original criterion is already discriminative. Moreover, the actual weighted LLM scores are not shown to implement the Bayes predictors analyzed in the proposition.

Proposition D.2 assumes corrections improve coverage and weighting without introducing new errors; its conclusion is therefore conditional accounting, not an explanation of why the implemented LLM pipeline satisfies those assumptions.

Questions for authors
  1. What exact datasets, splits, training budgets, and tool configurations produced each reported RL result? Can all main baselines be compared under one controlled configuration?
  2. What are the complete prompts, review stopping rules, decomposition thresholds, reference sources, and model identifiers?
  3. How do the training curves reconcile with Table 2, and what are the query-level confidence intervals and seed-level variation?
  4. Does Stage II restart from Rq0R_q^0 at every update, as Equation (6) indicates, or accumulate decompositions across training?
  5. What is the measured performance–cost trade-off against a budget-matched rubric-first baseline?
5 comments
AuthorsOpenReview ↗

Response to Reviewer NnGj (1/5)

We thank the reviewer for the precise review; the counterexample to Proposition D.4 and the list of conditions for a controlled comparison directly shaped this revision. Changes are marked in blue in the revised PDF.


W1 & Q2. About the implementation and training details. A: Every item the reviewer lists is now specified in the appendix; we give each below.

Prompts and output schemas. Appendix H.5 now gives a prompt box, with its inputs and JSON output schema, for each of the 11 pipeline steps (eight in Stage I, two in Stage II, and the reward judge); Algorithm 1 gives the full training procedure of S2R.

LowDisc and Δ. At each step and for each query, we rescale the scores of every rubric in R_q^0 on the 8 rollouts to [0, 1] and take the five rubrics with the lowest standard deviation as candidates; ties are broken by higher mean score, then larger global weight w_i α_r, then rubric index. Instead of a threshold, we take a fixed number of candidates; the cap equals the K_max = 5 that DR Tulu uses to limit rubric scoring cost. Δ is detected by the rubric generator within the decomposition call: it lists response differences that fall within the evaluation content of r, are consistent with its direction, and are not distinguished by the judgment conditions of r; if it finds none, it returns an empty list and r is retained. Only candidates with such differences (Δ_r^t ≠ ∅) are decomposed, and r is replaced with the finer criteria only if the decomposition passes review and the weighted score of the finer criteria has a larger standard deviation than r on the same 8 responses.

Stopping rules, failure handling and weights. For each query, Stage I repeats the three-level review and revision of Section 3.2: single-rubric, intra-direction and inter-direction. A revised candidate replaces the current version only if it resolves the targeted issues without losing any confirmed requirement. Review stops, keeping the last accepted version, once a round of review finds no remaining issue at any of the three levels or the revised candidate fails this check; each query runs at most 10 rounds and 3.3 rounds on average, so no training query is removed. Stage II makes one decomposition call and one review call per query at each step; if the decomposition, the review or the score check fails, r is retained at this step without retry. Each rubric is decomposed into at most five finer criteria, one level only, and every step starts again from R_q^0. Weights are not looked up from priority labels. In Stage I, the generator proposes a raw weight with a reason for each direction and each rubric according to its task importance and omission impact, and the reviewed weights are normalized so that Σ_i w_i = 1 and Σ_r α_r = 1 within each direction; in Stage II, the integer weights of the finer criteria are rescaled to sum to α_r (Eq. 5). The deterministic part is these two normalizations.

Training configuration. The former Table 6 is now split into an SFT table (G1) and an RL table (G2), completed after the hyperparameter tables of DR Tulu: AdamW (β = (0.9, 0.95), weight decay 0), a constant learning rate of 5 × 10⁻⁷ and a KL coefficient of 0.001; 32 prompts per step with 8 rollouts each, temperature and top-p of 1.0, responses of up to 16,384 tokens and at most 10 tool calls. "At least 75 steps" is removed: the full method and every ablation are trained for 2,080 steps (7.5 epochs), and the final checkpoint is evaluated; each RL run uses about 2,770 H20 GPU hours.

ScholarQA release. The "requires confirmation" sentence in Appendix H.1.2 is deleted. SQAv2 denotes ScholarQA-CS2 from AstaBench. We use the sqa_test task of astabench 0.5.4 (suite configuration v1.0.0; the 100-question official test set), judged by gemini-2.5-flash and scored as the mean of the four official metrics. Appendix H.1.3 gives the release, and Table H2 the evaluator and scoring.

2Wiki evaluator. The corresponding sentence in H.2.9 is also deleted. 2WikiMultiHopQA is evaluated on 1,000 randomly sampled questions, judged by GPT-4.1 following the released code of DR Tulu (Table H3).

Reference sources and model identifiers (Q2). Each query has 10 reference reports, generated independently by four deep research systems and six models equipped with the same search tools; Appendix H.1.2 lists them. The rubric generator is GPT-5 (with Gemini 3 and Qwen3-8B variants of S2R), and the reward judge is Gemini-2.5-Pro (Table G2); Table H5 gives the construction-stage scoring model.

Revision: Tables G1 and G2 in Appendix G; H.1.2 and H.1.3; Tables H2 and H3; and in H.5, the Stage I and Stage II subsections, Algorithm 1 and the prompt boxes.

AuthorsOpenReview ↗

Response to Reviewer NnGj (2/5)

W2 & Q1 & Q5. About isolating the effect of the reward construction. A: We have run the decisive experiment the reviewer describes. All methods are retrained from the same Qwen3-8B-SFT checkpoint with identical training queries (the 8,881 DR Tulu RL queries), references, tools, rollout budget (8 rollouts per query for 2,080 steps), reward judge (Gemini-2.5-Pro), optimization settings (Appendix Table G2) and evaluation protocol (Sec. 4.1). The controlled baselines are the strongest baseline of each rubric type in Table 1, all with GPT-5 as the rubric generator. Values are means over three training seeds:

Controlled blockDRBDRB2HBRQASQAv2Avg.
Qwen3-8B-SFT38.6730.0437.9268.8472.8249.66
Tournament-GRPO51.4741.1254.6878.5486.7662.51
OpenRS (PAMR)47.8337.7253.8477.2183.9260.10
RubricEM51.4742.3154.0878.7887.6362.85
S2R (GPT-5)53.1645.3659.4682.4689.6366.01

S2R exceeds the strongest controlled baseline on all five benchmarks, by 1.69 to 4.78 points, and by 3.16 points on average. The remaining groups in Table 2 now serve only as system-level context. The captions of Tables 2 and 8 state that every score without ∗ is obtained by us under the protocol of Sec. 4.1, and Appendix Table H6 lists, row by row, whether each score comes from our training, an official API, released code, or released weights.

On model capacity and references in the preference evaluation: every rubric method in Table 1 uses GPT-5 as the generator and Gemini-2.5-Pro for scoring. Every baseline whose pipeline accepts evidence receives the same 10 reference reports as S2R, and Appendix Table H5 lists the shared construction settings. S2R reaches 64.58% average accuracy, compared with 63.19% for the strongest baseline, RubricEM; without reference reports, the two reach 62.75% and 60.62%.

For the data, budget and tools behind each RL result (Q1): Appendix Tables G1 and G2 give the settings of the controlled block, and the new Appendix Table H6 gives the RL data, training steps and tools of each baseline in its original configuration. All evaluation sets are official full sets or fixed subsets: 100 DRB tasks, 132 DRB2 tasks, the official 100-question SQAv2 test set, the 776-question ResearchQA test-mini set, and 1,000 HealthBench examples sampled with seed 42.

Q5: a budget-matched rubric-first control. Table 3 of the revision adds w/o Stage I (Rubric Refinement). It generates rubrics directly from the same query and reference reports with Gemini 3, revises them for several rounds with the improve-rubrics prompt of Chasing the Tail, without responses, until each query uses as many LLM calls as full S2R, and then applies Stage II as usual. Appendix Table I4c reports the construction cost on five queries each from DRB and SQAv2: this control uses 23.7 calls and about 241K tokens per query, and S2R uses 23.7 calls and about 253K tokens. Under this budget, S2R is still 1.68 and 2.16 points higher in DRB and SQAv2 preference accuracy and 4.34 and 3.57 points higher on downstream DRB2 and RQA. Without the extra calls, rubric-first uses only 9.3 calls and about 95K tokens per query, and the gaps are 2.91 and 2.81 points and 4.80 and 4.04 points. Matching the number of calls closes only 10–42% of the gap, so most of the benefit of S2R comes from the intermediate representation rather than from additional inference.

Revision: Table 3 and Section 4.3 now include this control, and the new Table I4c in Appendix I.8 reports the calls and tokens of each variant.

AuthorsOpenReview ↗

Response to Reviewer NnGj (3/5)

W3. About ablations that separate the core mechanisms. A: We have added all five comparisons the reviewer lists, specified every variant, and directly measured scope drift and task-irrelevant additions for each variant. All Table 3 variants use Gemini 3 as the rubric generator; the new Appendix Table H7 marks, row by row, the mechanisms each variant retains or removes, and Appendix H.2.3 describes how each replacement variant is constructed. Each comparison below changes only the choice the reviewer names. Preference accuracy averages three rubric generations, and downstream scores average three training seeds:

Requested comparisonVariantDRBSQAv2DRB2RQA
—Full S2R76.4053.1549.0680.58
Stage I alonew/o Stage II75.39 (−1.01)52.13 (−1.02)48.05 (−1.01)79.51 (−1.07)
Equally fine-grained static rubricsw/o Response-Aware Decomp. (Static Rubric Decomp.)75.62 (−0.78)52.30 (−0.85)48.17 (−0.89)79.63 (−0.95)
Flat review, matched budgetw/o Hierarchical Review (Flat Review)75.17 (−1.23)52.01 (−1.14)46.84 (−2.22)78.93 (−1.65)
Alternative normalizationGlobal Positive Normalization74.94 (−1.46)51.79 (−1.36)46.39 (−2.67)78.48 (−2.10)

Static Rubric Decomp. splits Rq0\mathcal R_q^0 to S2R's item count without responses. It improves over Stage I alone by only 0.12–0.23 and trails the full method by 0.78–0.95, so the Stage II gain comes mainly from response information rather than finer granularity. Flat Review merges the three review levels into one call per round and uses the same number of calls per query as hierarchical review (23.7), with about 4% more tokens (265K vs. 253K; Table I4c). Global Positive Normalization removes the direction level and normalizes by the sum of all positive weights, as in DR Tulu, OnlineRubrics, and HealthBench. The fifth comparison, query-only scope recovery, is the S2R w/o reference reports row of Table 1 (GPT-5 generator): its DRB and SQAv2 accuracies are 74.16% and 51.33%, 1.90 and 1.76 points below the reference-assisted method. Downstream, the RQA differences have p < 0.05 in all three seeds for Flat Review and Global Positive Normalization, and in two seeds for w/o Stage II and Static Rubric Decomp. (Table I5c).

On Appendix I.5. The original w/o Structured Scope both removes directions and switches to flat review. The new Flat Review keeps directions and only switches to flat review. The two differ only in whether directions exist, so their gap measures the scope representation itself: 0.56 and 0.62 points on DRB and SQAv2, and 0.27 and 0.30 points on DRB2 and RQA. Scope-first construction as a whole matters more: the full method exceeds the call-matched w/o Stage I (Rubric Refinement) by 1.68–4.34 points across the four metrics (Table 3). We have rewritten I.5 to report this contrast and removed the leftover drafting note.

Direct measurements. On a panel of 16 DRB2 and 16 RQA queries across three training seeds, we audit the items added by Stage II. At step 2,080, 96.6% of S2R's added items stay within both their direction and the task scope, versus 69.9% for w/o Scope Constraint. Task-irrelevant additions (Intent Necessity score 1) make up 1.68% under S2R and 14.35% under w/o Scope Constraint. The direction-weight distribution of w/o Fixed Scope reaches a total variation of 0.140 from step 0 by step 2,080, versus 0 for S2R. These measurements now directly support the explanation that unconstrained variants introduce irrelevant criteria.

Revision: Table 3 and Section 4.3 now include these variants; Appendix Table H7 and H.2.3 define every variant, and Appendix I.5 reports the scope-representation contrast; Appendix I.4.1 gives the alternative normalization and the three direct measurements; Tables I2, I4b, and I5c in Appendix I.8 give the standard deviations, per-seed scores, and paired tests.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.