- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
GRADS synthesizes research questions from domain corpora by maintaining a graph of source-grounded claims and purportedly necessary inference dependencies. A selector conditions question generation on a subgraph with a requested depth, while several verifiers check grounding and reasoning. Experiments compare difficulty control across synthesis methods and solver models, then evaluate SFT and RL on the generated data in software engineering, physics, and security domains.
Critical strengths
-
The separation of exploration from context selection is a sensible design. An immutable evidence archive, source pointers, and mechanical quotation checks make the generated answers more inspectable. The graph also gives the controller a specific object to revise when an expansion adds context without adding a useful dependency.
-
The evaluation goes beyond reporting the generator's own depth labels. Scores decrease across the requested bands for three solver models in three domains, and SFT on GRADS trajectories improves over the synthesis baselines on the shared benchmarks. These are meaningful signs of practical utility. The paper also appropriately acknowledges that the additional RL accuracy gains are not statistically distinguishable from SFT.
Critical weaknesses
-
[Major] The evidence does not yet establish that graph depth measures unavoidable reasoning. Computing depth in code is exact only after the LLM has decided which dependencies are necessary. The graph verifiers inspect these dependencies, but the paper does not report an independent audit or solver interventions showing that the intended hops cannot be bypassed. Appendix C makes the enforcement particularly unclear: the final Verifier reports source-necessity failures as not automatically fatal, and necessity is omitted from its listed pass conditions. How are such findings propagated back into graph selection and acceptance? The lower scores on harder bands support empirical difficulty, but do not identify its cause or validate the claimed chain. A targeted audit of shortcut solutions and source-removal tests would address this central uncertainty. The paper should also distinguish its contribution from FORT-Searcher (Deng et al., 2026), which directly studies the gap between graph complexity and realized search difficulty using shortcut diagnostics and adversarial refinement.
-
[Major] The post-training evaluation is not specified well enough to establish generalizable gains from the synthesis method. The benchmarks are described as frozen and held out, but their construction and separation from training evidence are unclear. Disjoint QA pairs can still share source records or answer-bearing claims, so the unit of separation matters here. The tables also report the best benchmark checkpoint without explaining a separate validation-based selection rule. Even the claimed matched optimization budget is inconsistent: Table 12 gives every arm 24 steps, whereas Table 3 lists a GRADS GitHub checkpoint at step 26. Equal trajectory counts also leave materially different amounts of supervised text across arms. These issues do not erase the observed gains, but weaken their attribution to data quality. Please specify the evidence-level overlap policy, select checkpoints independently of the test set, and provide a consistent budget comparison that accounts for supervised tokens.
Questions for authors
See weaknesses.
3 comments
Response to W2:
We thank the reviewer for highlighting the need to clarify the evaluation protocol, evidence-level overlap, checkpoint selection, and training-budget comparability.
1. Training-Benchmark Separation and Evidence Overlap
The training, validation, and test QA pairs are disjoint. To further quantify potential evidence-level overlap, we leverage the provenance information recorded during synthesis, including source records (e.g., document pages or GitHub issues) and finer-grained evidence spans (e.g., individual comments or paragraphs).
We calculate the intersection-over-union (IoU) of distinct cited source records and spans across the training, and test sets:
$ \mathrm{IoU}(Test,Train) = \frac{|Test \cap Train |}{|Test \cup Train |} $
| Evidence overlap | GitHub | Physics | Security |
|---|---|---|---|
| Cited records (IoU) | 11% | 7% | 3% |
| Cited spans (IoU) | 6% | 2% | 2% |
Here, cited records refer to distinct source records referenced during synthesis, whereas cited spans refer to finer-grained evidence units within those records. These results indicate limited overlap in cited evidence, particularly at the span level.
We acknowledge that low cited-span overlap does not, by itself, rule out shared answer-bearing information or alternative uncited evidence. We will clarify the provenance-based overlap analysis and its limitations in the revised manuscript.
2. Validation and checkpoint selection.
We apologize for not describing the SFT validation protocol sufficiently clearly.
During SFT, we constructed a separate validation set of 10 disjoint QA pairs, with an Easy:Medium:Hard ratio of 3:3:4. We kept this validation set small to limit the computational overhead of repeated agent-based evaluation during training.
We first saved a series of checkpoints and selected the top three checkpoints according to their performance on this validation set and subsequently evaluated these checkpoints on the full benchmark. The reported best-performing checkpoint was selected from these three based on benchmark performance and was then used for the subsequent RL stage.
We acknowledge that this final selection may introduce benchmark-based model-selection bias, even though the initial checkpoint screening was performed using a separate validation set.
3. Supervised-token differences across synthesis methods.
We agree that equal trajectory counts and optimizer steps do not guarantee identical numbers of loss-bearing supervised tokens.
Our SFT trajectories are reconstructed from the synthesis trajectories and include tool interactions and evidence-reading behavior. Consequently, trajectory lengths vary across synthesis methods, even under the same trajectory-count and optimization-step budgets.
The results discussed in lines 1203–1204 provide evidence that longer trajectories do not consistently yield better downstream performance. However, we acknowledge that this observation does not isolate the causal effect of supervised-token count.
We therefore characterize our comparison as using a shared training recipe with matched trajectory counts and optimizer steps, rather than strictly matched supervised-token budgets.
4. Correction of the training-step inconsistency.
We thank the reviewer for identifying the inconsistency between Tables 3 and 12. The value "26" in Table 3 is a typographical error; the correct SFT training-step count is 24, as reported in Table 12.
Thanks for your reply. Consider the idea that make me not feel that interesting, and the comparison of results are from an outdated model sonnet or the results are trained on a 4B model. I will still maintain my negative score.
We sincerely thank the reviewer for the careful reading of our manuscript and the thoughtful, constructive feedback. We appreciate the reviewer’s recognition of the potential value of GRADS and the opportunity to clarify several important aspects of our work.
Response to W1:
1. Necessity verification and iterative refinement.
In GRADS, the Graph Verifier is responsible for assessing the necessity of dependencies in the evidence graph. Rather than relying on a single verification pass, GRADS repeatedly re-evaluates and updates the dependency graph throughout the agentic synthesis process as new evidence becomes available. This iterative verification allows dependencies to be reconsidered and refined in light of newly discovered information.
The Graph Verifier communicates its feedback to the Brain, which orchestrates the overall synthesis process. Importantly, a necessity-related warning from the final Verifier does not automatically trigger rejection. Instead, the Brain determines whether to continue exploration, deepen or broaden the evidence graph, revise the question, request corrections from other agents, or withdraw the task, taking into account the remaining iteration budget and the verification feedback.
Therefore, GRADS employs a dynamic verification-and-refinement mechanism, rather than relying solely on a terminal binary acceptance decision. The nonfatal treatment of necessity warnings is intended to support iterative correction, not to imply that such warnings are irrelevant.
We acknowledge, however, that this process does not formally guarantee the strict necessity of every reasoning step across all generated tasks. Our empirical results nevertheless show a consistent negative correlation between graph-defined depth and solver performance across all three domains, supporting depth as a meaningful operational proxy for task difficulty. We will clarify this distinction and the verification workflow in the revised manuscript.
2. Distinction from FORT-Searcher.
We thank the reviewer for pointing out FORT-Searcher (Deng et al., 2026). Both approaches recognize that graph complexity does not necessarily imply realized search difficulty. FORT addresses this gap through shortcut-aware synthesis, including controls for evidence co-coverage, single-clue selectivity, exposed constants, and prior-knowledge shortcuts, together with adversarial refinement and solver-trajectory diagnostics.
In contrast, GRADS focuses on explicitly controlling the depth of source-grounded claim dependencies during iterative synthesis. Its provenance-grounded dependency graph serves as a target-conditioned control structure for selecting relevant evidence subgraphs at specified difficulty levels, enabling systematic difficulty calibration across heterogeneous domain-specific corpora and supporting downstream SFT and RL.
Thus, while FORT primarily emphasizes shortcut resistance and realized search costs, GRADS emphasizes explicit difficulty control through graph-defined dependency depth. We view these approaches as complementary and will clarify their relationship in the revised manuscript.