acceptodds
Under review as a conference paper at ICLR 2027

GRADS: Graph-Regulated Agentic Data Synthesis with Controllable Difficulty

Abstract

Deep research agents are increasingly deployed to retrieve precise information from large corpora, yet multi-hop, long-horizon tasks remain challenging. Synthetic data generation has emerged as a scalable solution, with existing pipelines typically expanding seed tasks to increase their apparent depth and breadth. However, this expansion does not reliably translate into necessary reasoning: newly introduced evidence may be only weakly coupled to the answer, creating a mismatch between intended and empirical difficulty. To address this problem, we introduce GRADS, a graph-guided agentic data synthesis framework with explicit difficulty control. GRADS runs a closed loop of seed selection, evidence expansion, question–answer construction, and verification while maintaining a graph of the evidence and inference dependencies required to derive each answer. At each step, it conditions generation on a relevant subgraph, preserving indispensable inference without accumulating irrelevant context. Experiments across security, software-engineering, and scientific-document domains show that GRADS consistently generates verified tasks with empirically separated difficulty distributions, as measured by independent solvers. Moreover, under matched training budgets, post-training on this data delivers consistent gains at every level: SFT raises the solution-termination rate to over 95% while cutting tool calls and turns by up to 75%, RL trims tool use by up to a further 19.5%, and on our held-out, domain-specific benchmarks, the resulting 4B model matches Sonnet 4 on software engineering and surpasses it on the scientific and security domains.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Agentic Data Synthesis, Deep Search Agent, Large Language Model, Agent Evaluation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_GptAOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

GRADS synthesizes research questions from domain corpora by maintaining a graph of source-grounded claims and purportedly necessary inference dependencies. A selector conditions question generation on a subgraph with a requested depth, while several verifiers check grounding and reasoning. Experiments compare difficulty control across synthesis methods and solver models, then evaluate SFT and RL on the generated data in software engineering, physics, and security domains.

Critical strengths
  1. The separation of exploration from context selection is a sensible design. An immutable evidence archive, source pointers, and mechanical quotation checks make the generated answers more inspectable. The graph also gives the controller a specific object to revise when an expansion adds context without adding a useful dependency.

  2. The evaluation goes beyond reporting the generator's own depth labels. Scores decrease across the requested bands for three solver models in three domains, and SFT on GRADS trajectories improves over the synthesis baselines on the shared benchmarks. These are meaningful signs of practical utility. The paper also appropriately acknowledges that the additional RL accuracy gains are not statistically distinguishable from SFT.

Critical weaknesses
  1. [Major] The evidence does not yet establish that graph depth measures unavoidable reasoning. Computing depth in code is exact only after the LLM has decided which dependencies are necessary. The graph verifiers inspect these dependencies, but the paper does not report an independent audit or solver interventions showing that the intended hops cannot be bypassed. Appendix C makes the enforcement particularly unclear: the final Verifier reports source-necessity failures as not automatically fatal, and necessity is omitted from its listed pass conditions. How are such findings propagated back into graph selection and acceptance? The lower scores on harder bands support empirical difficulty, but do not identify its cause or validate the claimed chain. A targeted audit of shortcut solutions and source-removal tests would address this central uncertainty. The paper should also distinguish its contribution from FORT-Searcher (Deng et al., 2026), which directly studies the gap between graph complexity and realized search difficulty using shortcut diagnostics and adversarial refinement.

  2. [Major] The post-training evaluation is not specified well enough to establish generalizable gains from the synthesis method. The benchmarks are described as frozen and held out, but their construction and separation from training evidence are unclear. Disjoint QA pairs can still share source records or answer-bearing claims, so the unit of separation matters here. The tables also report the best benchmark checkpoint without explaining a separate validation-based selection rule. Even the claimed matched optimization budget is inconsistent: Table 12 gives every arm 24 steps, whereas Table 3 lists a GRADS GitHub checkpoint at step 26. Equal trajectory counts also leave materially different amounts of supervised text across arms. These issues do not erase the observed gains, but weaken their attribution to data quality. Please specify the evidence-level overlap policy, select checkpoints independently of the test set, and provide a consistent budget comparison that accounts for supervised tokens.

Questions for authors

See weaknesses.

3 comments

We sincerely thank the reviewer for the careful reading of our manuscript and the thoughtful, constructive feedback. We appreciate the reviewer’s recognition of the potential value of GRADS and the opportunity to clarify several important aspects of our work.

Response to W1:

1. Necessity verification and iterative refinement.

In GRADS, the Graph Verifier is responsible for assessing the necessity of dependencies in the evidence graph. Rather than relying on a single verification pass, GRADS repeatedly re-evaluates and updates the dependency graph throughout the agentic synthesis process as new evidence becomes available. This iterative verification allows dependencies to be reconsidered and refined in light of newly discovered information.

The Graph Verifier communicates its feedback to the Brain, which orchestrates the overall synthesis process. Importantly, a necessity-related warning from the final Verifier does not automatically trigger rejection. Instead, the Brain determines whether to continue exploration, deepen or broaden the evidence graph, revise the question, request corrections from other agents, or withdraw the task, taking into account the remaining iteration budget and the verification feedback.

Therefore, GRADS employs a dynamic verification-and-refinement mechanism, rather than relying solely on a terminal binary acceptance decision. The nonfatal treatment of necessity warnings is intended to support iterative correction, not to imply that such warnings are irrelevant.

We acknowledge, however, that this process does not formally guarantee the strict necessity of every reasoning step across all generated tasks. Our empirical results nevertheless show a consistent negative correlation between graph-defined depth and solver performance across all three domains, supporting depth as a meaningful operational proxy for task difficulty. We will clarify this distinction and the verification workflow in the revised manuscript.

2. Distinction from FORT-Searcher.

We thank the reviewer for pointing out FORT-Searcher (Deng et al., 2026). Both approaches recognize that graph complexity does not necessarily imply realized search difficulty. FORT addresses this gap through shortcut-aware synthesis, including controls for evidence co-coverage, single-clue selectivity, exposed constants, and prior-knowledge shortcuts, together with adversarial refinement and solver-trajectory diagnostics.

In contrast, GRADS focuses on explicitly controlling the depth of source-grounded claim dependencies during iterative synthesis. Its provenance-grounded dependency graph serves as a target-conditioned control structure for selecting relevant evidence subgraphs at specified difficulty levels, enabling systematic difficulty calibration across heterogeneous domain-specific corpora and supporting downstream SFT and RL.

Thus, while FORT primarily emphasizes shortcut resistance and realized search costs, GRADS emphasizes explicit difficulty control through graph-defined dependency depth. We view these approaches as complementary and will clarify their relationship in the revised manuscript.

Response to W2:

We thank the reviewer for highlighting the need to clarify the evaluation protocol, evidence-level overlap, checkpoint selection, and training-budget comparability.

1. Training-Benchmark Separation and Evidence Overlap

The training, validation, and test QA pairs are disjoint. To further quantify potential evidence-level overlap, we leverage the provenance information recorded during synthesis, including source records (e.g., document pages or GitHub issues) and finer-grained evidence spans (e.g., individual comments or paragraphs).

We calculate the intersection-over-union (IoU) of distinct cited source records and spans across the training, and test sets:

$ \mathrm{IoU}(Test,Train) = \frac{|Test \cap Train |}{|Test \cup Train |} $

Evidence overlapGitHubPhysicsSecurity
Cited records (IoU)11%7%3%
Cited spans (IoU)6%2%2%

Here, cited records refer to distinct source records referenced during synthesis, whereas cited spans refer to finer-grained evidence units within those records. These results indicate limited overlap in cited evidence, particularly at the span level.

We acknowledge that low cited-span overlap does not, by itself, rule out shared answer-bearing information or alternative uncited evidence. We will clarify the provenance-based overlap analysis and its limitations in the revised manuscript.

2. Validation and checkpoint selection.

We apologize for not describing the SFT validation protocol sufficiently clearly.

During SFT, we constructed a separate validation set of 10 disjoint QA pairs, with an Easy:Medium:Hard ratio of 3:3:4. We kept this validation set small to limit the computational overhead of repeated agent-based evaluation during training.

We first saved a series of checkpoints and selected the top three checkpoints according to their performance on this validation set and subsequently evaluated these checkpoints on the full benchmark. The reported best-performing checkpoint was selected from these three based on benchmark performance and was then used for the subsequent RL stage.

We acknowledge that this final selection may introduce benchmark-based model-selection bias, even though the initial checkpoint screening was performed using a separate validation set.

3. Supervised-token differences across synthesis methods.

We agree that equal trajectory counts and optimizer steps do not guarantee identical numbers of loss-bearing supervised tokens.

Our SFT trajectories are reconstructed from the synthesis trajectories and include tool interactions and evidence-reading behavior. Consequently, trajectory lengths vary across synthesis methods, even under the same trajectory-count and optimization-step budgets.

The results discussed in lines 1203–1204 provide evidence that longer trajectories do not consistently yield better downstream performance. However, we acknowledge that this observation does not isolate the causal effect of supervised-token count.

We therefore characterize our comparison as using a shared training recipe with matched trajectory counts and optimizer steps, rather than strictly matched supervised-token budgets.

4. Correction of the training-step inconsistency.

We thank the reviewer for identifying the inconsistency between Tables 3 and 12. The value "26" in Table 3 is a typographical error; the correct SFT training-step count is 24, as reported in Table 12.

Thanks for your reply. Consider the idea that make me not feel that interesting, and the comparison of results are from an outdated model sonnet or the results are trained on a 4B model. I will still maintain my negative score.

Reviewer_FX57OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

GRADS synthesizes multi-hop question-answer tasks from domain corpora by maintaining a provenance-linked evidence DAG. An agent team explores records, generates questions and claim-level rubrics, checks support and dependency necessity, and selects a subgraph whose longest necessary-dependency chain targets an Easy, Medium, or Hard band. Numeric graph properties are computed in code, while the graph's semantic dependencies are proposed and checked by LLM agents. Experiments cover GitHub records, CMS physics documents, and security-agent traces. The reported scores decrease across requested difficulty bands for three solver models, and SFT of Qwen3.5-4B on GRADS trajectories outperforms three synthesis baselines under a shared training recipe. Subsequent short-budget GRPO yields small, statistically unresolved score gains and further reduces tool use.

Critical strengths

A concrete synthesis mechanism with unusually explicit provenance and verification. The immutable evidence history, source pointers, deterministic quotation/reference checks, and separate semantic verifiers address the practical problem of generating traceable training tasks rather than merely producing longer questions. The full role prompts make the intended system substantially easier to inspect. Consistent utility across three different record domains. GRADS has decreasing Easy-to-Hard mean scores for all three solver tiers (Table 4), and its SFT checkpoints outperform all baseline arms in each domain (Tables 3 and 12). The large Security improvement, from 0.1414 to 0.4453, is particularly encouraging. The paper also appropriately distinguishes the modest, statistically unresolved RL score changes from the more substantial SFT gains.

Critical weaknesses

The central claim of indispensable reasoning depth is not adequately validated. Code computes the longest path of a graph whose nodes and necessary edges are nevertheless determined by LLMs. An accurate count of that representation does not by itself establish the shortest reasoning route available from the actual corpus; node granularity, directly stated conclusions, and alternative evidence can change the apparent depth. The three-model score ordering and moderate depth-score correlation support depth as an empirical proxy, but do not establish hop necessity. Section 5.6 acknowledges the absence of a human audit, while the conclusion invokes an independent judge and graph builder without clearly specifying what independence means or what validation it supports beyond the described role-based checks. More concretely, Section 4.3 says delivery requires the exact requested band, but the Brain prompt explicitly permits an item at depth 4 against a target of 10 (p. 17), and the final Verifier prompt makes necessity findings nonfatal and omits necessity from its pass condition (p. 19). The graph verifier does perform a necessity check, so these discrepancies are not proof that deficient items were accepted; they leave the actual end-to-end acceptance guarantee unclear. The post-training comparison is insufficiently specified to sustain the strong matched-budget interpretation. The shared recipe and favorable scores are useful evidence, but Table 12 reports different trajectory counts and substantially different amounts of loss-bearing supervision: Physics uses 169.0k supervised tokens for GRADS versus 84.5k for TaskCraft, and GitHub uses 145.4k versus 79.6k. This limits attribution of the gain specifically to graph-regulated structure rather than the amount and content of supervision; it does not imply that all gains are explained by length, and GRADS uses fewer supervised tokens than ClaudeCode in Security. More seriously, the GRADS GitHub checkpoint is listed at step 26 in Table 3, all arms have 24 steps in Table 12, and GRPO initialization is listed at GitHub SFT step 56 in Table 13. The manuscript does not adequately specify source-level train/test separation, benchmark construction, or whether the reported best checkpoints were selected using an independent validation set. It mentions bootstrap/permutation tests but provides no comparison-specific uncertainty intervals, leaving the reliability of the best-checkpoint margins difficult to assess.

Questions for authors

Clarification and validation of the central claim: In the deletion test, what is removed: a graph claim, its source span and duplicates, or all access to that information in the solver's corpus? How are alternate derivations considered? What does “independent judge and independent graph builder” in the conclusion mean: separate role calls, blind reannotation, different models, or another form of independence? Please describe the validation protocol and results, including whether dependency judgments were made without seeing the generating graph or requested band. A concrete existing task with its sources, graph, and an attempted shortcut would substantially help assess this issue. Clarification of acceptance and selection: How does the actual delivery gate reconcile Section 4.3 with the Brain's permission to deliver under-target items and the Verifier's nonfatal necessity findings? Please report the fraction of delivered items satisfying the requested band on the selected graph, separately from the full-graph depths in Table 1. In Equation 6, is the intended closure over ancestors in the original graph, and can the selector change the anchor as Appendix C suggests? Clarification of the training comparison: Please reconcile GitHub SFT steps 24, 26, and 56, explain how the best checkpoints were selected, and describe how related source records were separated between SFT, RL, validation, and benchmark tasks. Does the existing evidence support a graph-specific training advantage after accounting for supervised-token differences, or should the claim be restricted to utility under the shared sequence-based recipe? A token-matched follow-up would be useful, but is a suggested additional experiment rather than an assumed existing result. Clarification of the statistical evidence: What are the exact sample counts used for each reported score/depth average and post-training comparison? Table 1 states 20 per cell, Appendix A.1 uses benchmarks of 100/100/135, Table 7 uses 30/30/40 band weights, and Table 8 uses counts out of 100. Please provide the actual bootstrap/permutation uncertainty for the principal SFT and band comparisons and explain how the stated ±0.043 RL noise floor was obtained.

7 comments

We sincerely thank the reviewer for the careful reading of our paper and the detailed, constructive feedback. We appreciate the opportunity to clarify several aspects of GRADS, particularly the validation of dependency necessity, the difficulty-control mechanism, and the experimental evaluation protocol.

  • Clarification and validation of the central claim:
  1. The deletion test is a claim-level counterfactual assessment, not a physical deletion experiment on the solver-accessible corpus. For a candidate dependency (u \rightarrow v), the Graph Builder examines the recorded evidence and assesses whether (v) remains derivable without relying on claim (u) (Appendix C). If the dependency is not necessary, the corresponding semantic relation may be retained with necessary: false, excluding it from depth computation.

    Importantly, this procedure does not remove the underlying source span, its duplicates, or other occurrences of the same information from the solver's corpus. Near-duplicate graph nodes are addressed separately to prevent artificial depth inflation, while detail nodes are excluded from the necessary reasoning spine.

    Thus, our measured depth represents the longest chain of dependencies judged necessary within the constructed evidence graph, rather than a formally established lower bound on all possible reasoning paths.

  2. GRADS addresses potential shortcuts at three levels:

  • Dependency assessment: The Graph Builder applies the counterfactual deletion criterion to each proposed necessary edge. If the dependent claim remains derivable without its proposed prerequisite, the edge should not contribute to depth.

  • Graph verification: The Graph Verifier examines redundant dependencies, near-duplicate nodes, and transitive shortcuts that could artificially inflate the measured depth.

  • Final answer verification: The final Verifier evaluates source necessity from the perspective of the shortest plausible derivation, checking whether the answer remains derivable without particular cited evidence.

    These mechanisms reduce redundant dependencies within the recorded evidence. However, they do not exhaustively search the entire solver-accessible corpus for alternative evidence paths. For example, an uncited document might directly establish an intermediate or final conclusion, allowing a solver to bypass the intended chain.

We therefore acknowledge that the existing verification procedure provides evidence-relative necessity judgments, rather than a guarantee that every hop is unavoidable under unrestricted corpus access.

  1. Both the judge and graph builder use Claude Opus 4.8, but are executed as separate role-specific calls with independent, isolated contexts.

    For judging, each judge operates in a fresh context and receives the solver's response and the corresponding evaluation rubrics. Judgments are performed independently, without sharing conversational histories or intermediate judgments across evaluation instances.

    For graph construction, each graph-building turn also operates with a separate context containing only the information relevant to constructing or updating that graph. Previous unrelated context is not carried over.

    Thus, “independent” refers to context isolation and independent role execution, rather than the use of different underlying models or independent human reannotation. We will clarify this distinction in the revised manuscript to avoid overstating the degree of independence.

  2. In GRADS, both the final Verifier and the Graph Verifier use Opus 4.8, but they operate in separate contexts with distinct responsibilities and inputs.

    The final Verifier evaluates the generated task using question- and answer-related information, including the question, reference answer, and rubrics. It does not receive the original evidence graph.In contrast, the Graph Verifier explicitly receives the graph constructed by the Graph Builder and evaluates its structure and the necessity of its dependency edges. Therefore, the Graph Verifier's dependency judgments are not blind to the generating graph. This is an internal graph-validation procedure, rather than blind reannotation or independent graph reconstruction.

    The purpose of this design is to provide complementary checks on the generated QA item and its underlying evidence structure. However, we acknowledge that these checks do not independently establish that all graph-defined dependencies are unavoidable under unrestricted access to the corpus.

Overall, GRADS provides a structured and source-grounded mechanism for controlling graph-defined reasoning depth, and our experiments examine its relationship with empirical solver difficulty. We acknowledge that the current verification mechanisms do not constitute a formal proof of globally unavoidable reasoning depth. We will clarify this distinction and the precise scope of independent validation in the revised manuscript.

  • Clarification of acceptance and selection:
  1. In GRADS, the Graph Verifier is responsible for assessing the consistency and necessity of dependencies in the evidence graph. As new evidence becomes available, the graph can be re-examined and refined throughout the agentic synthesis process. The Graph Verifier's feedback is communicated to the Brain, which orchestrates subsequent actions, including further exploration, graph refinement, question revision, and task termination.

    Importantly, a necessity-related warning from the final Verifier does not automatically terminate the synthesis process. Instead, the Brain can use this feedback to request further corrections, subject to the remaining iteration budget. This design allows potentially recoverable issues to be addressed through iterative refinement rather than immediate rejection.

    However, intermediate verification feedback and final delivery eligibility are distinct. The Brain's permission to stop exploration below a target depth does not itself establish that the resulting task satisfies the requested difficulty band. The selected-subgraph depth is checked separately through the difficulty-control mechanism.

  2. We clarify that GRADS uses a code-level depth-selection mechanism to constrain the selected evidence subgraph to the requested difficulty band. This selection operates on the graph structure, rather than relying on the LLM's subjective estimate of depth.

    The selected subgraph is then dereferenced into source-grounded context and passed back to the Brain for subsequent question construction and refinement. Additional evidence discovered during later iterations may extend or modify the graph, but this should be distinguished from the depth of the subgraph selected at a particular iteration.

    Thus, full-graph depth and selected-subgraph depth measure different aspects of the synthesis process. Table 1 reports the former, whereas the requested-band constraint applies to the latter.

  3. The intended ancestor-closure constraint in Equation (6) refers to the necessary ancestors of the anchor in the original evidence graph. The purpose of this constraint is to preserve the prerequisite claims supporting the selected answer, rather than allowing arbitrary removal of dependencies to obtain a shorter chain. Regarding anchor selection, the anchor (a) is treated as fixed in the optimization formulated in Equation (6). The Selector is intended to select an appropriate evidence subgraph around this anchor, rather than redefine the answer target during that optimization.

  • Clarification of the training comparison:
  1. We apologize for the inconsistent checkpoint-step labels. The values 26 in Table 3 and 56 in Table 13 are typographical errors; both should be 24, consistent with the SFT optimization budget reported in Table 12. All compared SFT methods were trained for 24 optimizer steps under the same training configuration.

    For checkpoint selection, we saved a checkpoint every two optimization steps. During SFT, we constructed a separate validation set of 10 QA pairs, with an Easy:Medium:Hard distribution of 3:3:4. This set was kept small to limit the computational cost of repeated agent-based evaluation.

    We first ranked the saved checkpoints using this validation set and shortlisted the top three candidates. These three checkpoints were subsequently evaluated on the full benchmark, and the highest-scoring checkpoint was selected for the subsequent RL stage.

  2. The training, validation, and benchmark QA pairs are disjoint. To further quantify evidence reuse, we analyzed the source provenance recorded during synthesis, including source records (such as GitHub issues and document pages) and finer-grained evidence spans (such as comments and paragraphs).

    We calculated the intersection-over-union of distinct cited source records and spans across these splits: $ \mathrm{IoU}(T, E)=\frac{|T\cap E|}{|T\cup E|}. $

Evidence overlapGitHubPhysicsSecurity
Cited records (IoU)11%7%3%
Cited spans (IoU)6%2%2%

Here, (T), (E) denote the sets of cited evidence units in training and testing, respectively.

The analysis shows that relatively small fractions of the union of cited evidence are shared, particularly at the span level.

  1. We agree that matching trajectory counts and optimizer steps does not guarantee equal supervised-token budgets. Our SFT training sequences are reconstructed from the synthesis trajectories and contain tool interactions, retrieved evidence, and intermediate reasoning behavior. Consequently, the number of supervised tokens varies across synthesis methods, even when trajectory counts, optimization steps, and other training settings are held constant.

    As discussed in lines 1203–1204, the observed results do not exhibit a consistent relationship in which longer training trajectories yield better downstream performance. This suggests that trajectory length alone does not straightforwardly explain the observed ranking of methods. Nevertheless, we acknowledge that these comparisons do not isolate the causal effect of supervised-token volume.

    We therefore interpret the existing results as evidence of the practical utility of GRADS-generated trajectories under a shared sequence-based training recipe, rather than definitive evidence of a graph-specific causal advantage under strictly token-matched supervision.

  • Clarification of the statistical evidence:
  1. We thank the reviewer for pointing out the ambiguity in the sample-count reporting. We revisited our evaluation pipeline and confirmed that each domain-specific benchmark contains 100 QA items.

    For computational efficiency, we partitioned each benchmark into five batches of 20 items and evaluated these batches concurrently. The statement “n = 20 per cell” in Table 1 mistakenly refers to the evaluation batch size rather than the total number of evaluated items.

    The reported aggregate scores and depths were computed from the combined evaluation results, not from a single 20-item batch. We will correct the table caption to distinguish the evaluation batch size from the statistical sample size and report the exact per-band counts. For our main benchmark evaluations, we use a fixed benchmark of 100 QA items per domain (GitHub, Physics, and Security). The same benchmark is used consistently for score and depth calculations and downstream post-training comparisons, ensuring that performance differences are evaluated on the same set of tasks.

  2. For the Security domain, our synthesis pipeline originally generated 135 QA items. The analysis in Appendix A.1 was conducted using all 135 synthesized items, whereas the final evaluation benchmark was constructed by randomly selecting 100 items without preference for any particular difficulty level or performance outcome. The remaining 35 items were excluded from both training and benchmark evaluation. The 30/30/40 distribution in Table 7 represents the Easy/Medium/Hard composition used for benchmark evaluation.

  3. The reported ±0.043 corresponds to a 95% confidence interval estimated from repeated evaluations of the same model on the same benchmark.

We acknowledge that this repeated-evaluation estimate does not directly provide bootstrap or permutation uncertainty for the principal SFT and difficulty-band comparisons.

Reviewer_df9EOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes GRADS, a multi-agent pipeline that synthesizes multi-hop QA items at a requested difficulty from closed domain corpora. While agents explore the corpus, an LLM graph builder maintains an evidence graph of source-grounded claims and "necessary" dependency edges. Difficulty is defined as the longest necessary chain, and a selector picks a subgraph in the requested Easy/Medium/Hard band; the question and a weighted claim rubric are then written from that subgraph and verified. On GitHub/JIRA issues, CMS physics notes and security traces, the authors report that only GRADS yields depth and solver scores that move monotonically with the requested band, compared with TaskCraft, WebShaper and a prompt-only variant. Fine-tuning Qwen3.5-4B on about 60 GRADS trajectories per domain gives the best score on the authors' in-domain benchmarks, and a short GRPO stage mainly reduces tool calls.

Critical strengths
  1. Tying the difficulty label to a measured chain of necessary hops, with provenance checks, is a sensible way to close the gap between nominal and realized difficulty.
  2. Applying synthesis to issue trackers, physics notes and security traces rather than Wikipedia or the open web is a useful setting.
  3. The Easy > Medium > Hard ordering holds for three solver tiers in all three domains (Table 4). The paper also reports synthesis cost per item (Table 9) and releases the pipeline code.
Critical weaknesses
  1. The evidence that difficulty is "realized" comes almost entirely from inside the pipeline. Opus 4.8 builds the graph, verifies the items, solves them and judges the answers, and there is no human audit of hop necessity and no shortcut test.
  2. The difficulty metric is confounded with rubric size. The score is weighted recall over rubric claims, and in the released code the rubric grows from 4–8 claims (Easy) to 20–40 (Hard), so the monotone trend may partly reflect a longer answer key rather than deeper reasoning.
  3. The training evidence is narrow. SFT uses about 60 trajectories per domain, the ~100-item test set is produced by GRADS itself with the same rubric format and judge, the best checkpoint is picked on that test set, and there is no standard benchmark such as BrowseComp or GAIA.
  4. The baseline comparison is not clearly fair. Supervised tokens are not matched (e.g., 169k vs. 84.5k for TaskCraft on Physics, Table 12), and TaskCraft's near-ceiling scores on GitHub (0.95–1.0, Table 1) raise questions about how it was ported. The Physics margin over the Claude-Code arm (0.339 vs. 0.301) is also within the paper's own ±0.043 noise level.
Questions for authors
  1. Does the band ordering hold under a metric that does not scale with rubric size, e.g., correctness of the final answer only?
  2. How were TaskCraft and WebShaper adapted to these corpora, and how was their depth measured?
  3. How does the SFT model perform on an evaluation set not generated by GRADS, with checkpoints selected on a separate validation split?
3 comments

Response to Q1:

We thank the reviewer for highlighting the potential confounding effect of rubric size.

For multi-hop research tasks, more complex questions often involve additional factual requirements and intermediate reasoning steps, which naturally motivate more fine-grained evaluation criteria. This practice is consistent with prior rubric-based benchmarks, such as ResearchRubrics (Sharma et al., 2025), which employs 20–43 task-specific criteria for deep-research questions, and PaperBench (Starace et al., 2025), which hierarchically decomposes complex research-replication tasks into independently gradable components.

In GRADS, rubric size increases with difficulty because deeper dependency chains generally involve more intermediate claims and supporting evidence that must be evaluated. However, we acknowledge that this introduces a potential confounding factor when comparing aggregate rubric scores across difficulty bands. We therefore supplement the aggregate results with category-wise analyses of Core, Inference, and Detail claims to investigate whether the observed difficulty ordering reflects more than differences in rubric size.

To further investigate this concern, we analyze solver performance separately for the three rubric categories used in our evaluation: Core, Inference, and Detail.

DomainRubric CategoryEasyMediumHard
GitHubCore0.7660.6750.414
Inference0.8110.7460.473
Detail0.7570.7450.679
PhysicsCore0.6820.3530.220
Inference0.7440.4210.364
Detail0.7190.5770.538
SecurityCore0.7340.6370.485
Inference0.6960.6540.538
Detail0.7450.7220.654

Across all three domains, the scores for Core and Inference claims decrease more sharply from Easy to Hard than those for Detail claims. For example, in GitHub, Core and Inference scores decrease by 0.352 and 0.338, respectively, compared with only 0.078 for Detail claims. Similar patterns appear in Physics and Security.

These results suggest that the observed difficulty increase is not uniform across rubric categories: the degradation is concentrated in claims associated with the central answer and inferential reasoning, rather than supporting details.

We acknowledge that this category-wise analysis remains rubric-based and therefore does not fully eliminate rubric-size confounding or replace a final-answer-only correctness evaluation. Nevertheless, it provides additional evidence that the difficulty ordering reflects more than a uniform decline in claim coverage.

Response to Q2:

We thank the reviewer for requesting additional implementation details.

TaskCraft. We used the released TaskCraft generation code without modifying its internal generation logic. Adaptation to our domains was implemented through external corpus and tool interfaces.

  • Corpus preparation: GitHub issues were represented as documents containing titles, metadata, bodies, and comments; CMS documents were converted from PDF notes to text; and Security challenges were represented using challenge specifications and relevant attempt records.
  • Retrieval: TaskCraft's web-search interface was replaced with retrieval over the corresponding local domain corpus.
  • Answer format: We retained TaskCraft's native short-answer format, representing its atomic answer as a single rubric claim.
  • Security-specific adaptation: We applied two deterministic post-processing adjustments to improve answerability: replacing file-name/ID/URL answers with vulnerability descriptions and providing vulnerability findings as context for Easy questions referring to synthetic file names.

WebShaper. Because the original synthesis implementation was not publicly available, we reimplemented its Knowledge Projection (KP) expansion mechanism based on the published methodology and checked the representation against its released dataset.

For GitHub, we constructed typed relation graphs over issue clusters, including issue-to-issue links and attribute relations. Starting from a candidate answer issue, the Expander iteratively added relation hops while evaluating the remaining candidate answer set. Dependency-type relations were preferred, and attribute constraints were introduced when needed to uniquely identify the answer. Attribute intersections contributed to breadth rather than relation-chain depth.

For Physics and Security, the available corpus structures did not support the same directional relation-chain expansion: Physics clusters primarily contained undirected document relationships, while Security clusters were organized around individual challenges. We therefore used WebShaper's intersection-based difficulty operation, requiring increasing distinguishing properties for Easy, Medium, and Hard, respectively.

Depth measurement. We distinguish two quantities:

  1. Constructed depth: the complexity level produced by each generator according to its native expansion mechanism. For TaskCraft, this corresponds to its depth-expansion operations; for WebShaper, it corresponds to its relation-expansion operations where applicable. Intersection-based constraints are reported separately.
  2. Measured dependency depth: a common post-generation metric applied to all synthesized items. An LLM graph builder examines the question, answer, and rubric claims, identifies necessary claim dependencies, and computes the longest dependency path using the same graph-depth routine applied to GRADS.

This common measurement protocol enables comparison under a shared operational definition of dependency depth, although the underlying semantic dependency judgments remain LLM-assessed.

We will make the distinction between native construction complexity and measured dependency depth explicit in the revised manuscript.

Response to Q3:

We appreciate the reviewer's suggestion to evaluate the fine-tuned models on independently generated benchmarks.

GRADS targets the synthesis of both post-training trajectories and verifiable benchmarks for specialized domain corpora. General-purpose benchmarks such as BrowseComp and GAIA do not directly reproduce our domain-specific evidence sources, retrieval interfaces, and task requirements. This motivates our evaluation using held-out tasks constructed from the target corpora.

Regarding checkpoint selection, we clarify that SFT used a separate validation set of 10 disjoint QA pairs, distributed across Easy, Medium, and Hard with a ratio of 3:3:4. We first selected the top three checkpoints based on validation performance and then evaluated these candidates on the full benchmark.

Regarding additional evaluation, the original fine-tuned checkpoints were produced in an industrial research environment and are no longer accessible to the authors preparing this response. Consequently, we cannot perform additional cross-generator or external-benchmark evaluations of these checkpoints during the rebuttal period.

We acknowledge that evaluation on GRADS-generated benchmarks, even with disjoint QA pairs, does not fully establish generalization to independently generated task distributions. We will therefore clarify the scope of our post-training conclusions and distinguish the observed in-domain improvements from broader generalization claims.

Thanks for the rebuttal, which have addressed most of my concerns. I am satisfied with the response and would raise the score. However, as indicated by my confidence score, I am less familiar with the scope this paper falls into. Some of the other reviewers' concerns seem valid and please address them as well. I would not fight for acceptance for this paper.

Comments on the paper

1 comment

We sincerely thank the Area Chair and all reviewers for their thoughtful feedback and constructive discussion. We summarize the key strengths recognized by the reviewers, our rebuttal clarifications, and the remaining limitations to facilitate the final assessment.

1. Strengths Recognized by Reviewers

Methodology and Novelty

  • Structured Difficulty Control: GRADS introduces a provenance-grounded evidence graph that explicitly controls task complexity through dependency-depth-based subgraph selection. Reviewer df9E recognized this as “a sensible way to close the gap between nominal and realized difficulty.”
  • Verifiable Agentic Synthesis: The immutable evidence archive, source pointers, mechanical quotation checks, and complementary verification agents provide an inspectable synthesis framework (FX57, GptA).

Experimental Effectiveness

  • Consistent Difficulty Differentiation: All three solver tiers exhibit decreasing performance from Easy to Hard across GitHub, Physics, and Security. Reviewers df9E, FX57, and GptA all acknowledged this empirical pattern.
  • Post-Training Utility: SFT on GRADS-generated trajectories outperforms the evaluated synthesis baselines across all three domains under a shared training recipe, demonstrating practical utility for training domain-specific research agents.
  • Broad Application Settings: Experiments span software engineering, scientific documents, and security traces, demonstrating applicability beyond conventional open-web information-seeking tasks.

2. Key Rebuttal Clarifications

C1. Necessity Verification and Difficulty Control (FX57, GptA)

We clarified the iterative graph verification and refinement procedure, the distinction between Graph Verifier and final Verifier, and the code-level subgraph-selection mechanism. We also provided a concrete multi-document GitHub example illustrating the intended reasoning dependencies.

Importantly, we explicitly distinguished graph-defined dependency depth from a formal guarantee of globally unavoidable reasoning. Our contribution is operational difficulty control supported by empirical solver behavior, rather than a proof that all alternative reasoning paths are impossible.

C2. Rubric-Size Confounding (df9E)

We provided additional category-wise analyses separating Core, Inference, and Detail claims. Across all three domains, Core and Inference scores decline more sharply with increasing difficulty than Detail scores.

These findings provide additional evidence that the difficulty trend is concentrated in answer-bearing and reasoning-related requirements, rather than reflecting a uniform decline in rubric coverage.

C3. Experimental Protocol and Reproducibility (df9E, FX57, GptA)

We clarified the baseline adaptation procedures, the distinction between native construction complexity and measured dependency depth, and the checkpoint-selection protocol.

We also corrected inconsistent training-step labels and provided evidence-overlap statistics. Cited-span overlap between training and testing is 6%, 2%, and 2% for GitHub, Physics, and Security, respectively.

We further clarified that the comparison uses matched trajectory counts and optimizer steps, rather than strictly matched supervised-token budgets.

C4. Scope of Empirical Conclusions (FX57, GptA)

We acknowledge that independent shortcut audits, fully external benchmarks, and benchmark-independent final checkpoint selection would strengthen the conclusions.

Nevertheless, the current results establish several meaningful empirical findings: controllable graph-defined difficulty, consistent solver-performance separation, and improved downstream post-training performance under the evaluated protocol.

We have accordingly narrowed our claims to reflect the evidence without overstating theoretical guarantees or broader generalization.

3. Final Considerations

We recognize that important limitations remain, particularly regarding globally unavoidable reasoning chains.

Nevertheless, we respectfully emphasize that these limitations constrain the scope of our conclusions but do not negate the empirical contributions already demonstrated: a source-grounded and inspectable graph-regulated synthesis framework, consistent difficulty differentiation across nine domain–solver combinations, and practical post-training improvements across three specialized domains.

We believe GRADS offers a meaningful methodological contribution to controllable agentic data synthesis.

We respectfully hope the Area Chair will consider the paper's methodological novelty, demonstrated empirical utility, and our substantive rebuttal clarifications in the final assessment.

Thank you again for your time.

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.