- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The paper proposes U2, a framework for learning to generate challenging, source-grounded questions from academic literature. A GPT-OSS-20B generator is optimized using GRPO with format-dependent rewards and a factual-consistency gate supplied by an SFT-trained verifier. For short-form questions, difficulty is measured through sampled-answer entropy from a tool-enabled solver that is not given the source paper. Multi-agent procedures filter questions and construct reference answers.
The framework produces U2-Bench, which has three 500-question tracks: research reports, fact-heavy non-STEM QA, and calculation-heavy STEM QA. Validation includes model and tool-access comparisons, generator checkpoint evaluations, human assessment of 82 question–answer pairs, and a Fact500 ambiguity audit with a controlled repair intervention. A same-base U2-Train experiment reports improvements of 4.8, 3.3, and 3.6 percentage points on GAIA, FRAMES, and HLE, respectively.
Critical strengths
-
The paper directly investigates whether ambiguity can inflate the difficulty reward. Section 6 audits all 500 Fact500 questions without revealing the intended source or reference answer. For the 20 identified defects, the authors compare three versions: the original question, a minimal repair that restores the missing constraint, and a placebo paraphrase. Repairs reduce normalized entropy by 0.139 relative to placebo, and the reported 95% confidence interval excludes zero. This supports the interpretation that omitted constraints increase entropy within the identified defective items. It directly addresses a central failure mode of the proposed reward.
-
The paper demonstrates the feasibility and practical utility of building literature-grounded evaluations automatically. The framework produces three distinct 500-question evaluation tracks and demonstrates their use for both model assessment and downstream training. The matched source-access analysis on Fact500 is particularly useful. Accuracy increases from 55.6% without tools to 75.8% with web search and code, and to 86.2% when the intended source is supplied (Appendix C.2). These results support the utility of the generated questions for evaluating evidence acquisition and source-grounded problem solving. U2-Train also provides preliminary evidence of training utility through improvements on three external benchmarks.
Critical weaknesses
- Major — Existing validation does not sufficiently establish the robustness of benchmark scores or the capability interpretations drawn from them.
The paper includes meaningful validation: human evaluation, error-taxonomy analysis, a complete Fact500 ambiguity audit, and independent Python calculations. Nevertheless, important gaps remain in connecting these checks to the reliability of the reported scores.
Section 3.2 reports that 84.1% of the 82 human-reviewed question–reference pairs were fully correct. Appendix A.5 distinguishes uniqueness defects, scoring issues, granularity disagreements, and reference-answer errors, so the remaining 15.9% should not be read as a reference-error or ambiguity rate. However, several things remain unclear: the sample’s track composition and representativeness, the inter-annotator agreement, and whether the identified corrections were incorporated into the evaluated snapshot.
Scoring sensitivity is substantial. GPT-5.5 receives 75.8% under the LLM judge but 42.2% under rule-based scoring (Section 5.2). This discrepancy alone does not show that the judge is wrong, since accepting semantically equivalent answers could explain part of it. However, independent expert adjudication of actual solver outputs is needed to determine whether the difference reflects valid flexibility or erroneous acceptance.
For Calc500, Appendix D.1 reports independent Python solutions, but not their coverage, agreement rate, or how disagreements were handled. Appendix E explicitly acknowledges the absence of an independent validity audit. Consequently, attributing Calc500 score differences specifically to assumption selection and multi-step derivation remains insufficiently supported.
These gaps materially limit conclusions about benchmark difficulty and model capability. Independent experts should separately validate question uniqueness, reference correctness, and judge decisions on stratified samples from both QA tracks. The authors should also report whether correcting the identified Fact500 defects and reference errors changes the principal results.
- Major — The experiments demonstrate overall training benefits but do not isolate the additional contribution of entropy-based difficulty selection.
Appendix C.1 compares the generator before and after training under a fixed source pool and screening policy. With 500 candidates at each checkpoint, usable yield increases from 3.6% at step 0 to 28.4% at step 200. This supports the effectiveness of the combined training procedure within the stated protocol. Figure 5 also provides a meaningful comparison between difficulty oracles.
However, as Appendix C.1 acknowledges, these experiments do not separate the difficulty reward from the formatting, grounding, and other rewards. They establish improved generation under the combined objective, but not the specific added value of entropy-based optimization.
Similarly, U2-Train’s improvements over the base model show positive transfer after additional training. They do not establish that entropy-selected questions outperform equally sized, otherwise comparable training data. The gains could come from additional optimization, source content, answer verification, or dataset composition. Appendix E acknowledges the missing matched training-data controls and the lack of training-seed variability, but these remain consequential evidence gaps.
The decisive controls are:
RL without the difficulty reward. Solver training on randomly selected versus entropy-selected questions, with dataset size, verification, and training budget matched.
Comparing these variants on held-out sources and held-out solvers would test whether the differences generalize beyond the selection protocol. Multiple training seeds would establish whether the downstream gains are reproducible. A compute-matched generate-and-filter baseline would also clarify whether RL offers an advantage over simply generating more candidates.
- Minor issues and reporting corrections (not decisive for my recommendation):
Section 5.1 contains an arithmetic inconsistency. On 500 questions, an increase from 55.6% to 75.8% corresponds to 278 versus 379 correct answers, a net increase of 101. The reported 135 recoveries and 16 regressions instead imply a net increase of 119. These cannot describe the same paired evaluation and require clarification. The representative examples do not substantiate the claimed complexity. Table 7 effectively supplies the requested answer category within the question. Table 8 supplies the formula and all required quantities, leaving only substitution and unit conversion. These examples do not show that the entire benchmark is easy, but they are weak illustrations of extensive searching or source-grounded derivation. Some reproducibility details are insufficiently specified. The paper provides the optimization objectives, oracle sample count, entropy threshold, and annotation committee configuration. However, the following need further specification:
- the actual reward coefficients;
- the entropy-sampling temperature and canonicalization details;
- the verifier’s training size and held-out performance;
- the U2-Train dataset size, solver reward, and training configuration.
Questions for authors
Clarification requests:
- Human validation and corrections: How were the 82 reviewed pairs sampled, and how are they distributed across tracks? What was the inter-annotator agreement? How were the errors and disagreements identified in Appendix A.5 reflected in the final references and the reported scores?
- Fact500 audit and evaluated snapshot: Were the 20 defects identified in Section 6 removed or repaired before the reported evaluations? If not, how do the principal results change after correction? Was an independently reviewed sample used to estimate defects that the audit may have missed?
- Scoring and numerical verification: What are the evaluation judge’s exact model ID and complete prompt? Are expert judgments available for solver outputs on which the scoring methods disagree? For the independent Python solutions in Appendix D.1, what proportion of Calc500 was covered, what was the agreement rate, and how were disagreements resolved?
Clarification and requested controls (could materially affect my assessment):
U2-Train protocol and controls: What are the training-set size and track proportions, the answer-generation and verification models, and the GSPO reward? Which external benchmark splits, sample sizes, and evaluation budgets were used, and how was overlap between training and evaluation data checked? Are results available for removing the difficulty reward, or for training on matched randomly selected data?
6 comments
Weakness 3: Inconsistent recovery and regression counts in Section 5.1.
Response. Thank you for identifying this reporting inconsistency. The reported accuracies correspond to 278 correct answers without tools and 379 with tools, giving a net increase of 101 questions, or 20.2 percentage points. The submitted recovery/regression pair of 135/16 does not reproduce these totals.
We will replace the inconsistent breakdown with the supported aggregate statement:
Tool access increases accuracy from 55.6% to 75.8%, corresponding to 101 additional correct answers on the 500-question set.
The aggregate accuracy result remains unchanged. Any paired recovery/regression breakdown must use the same item-level labels and reproduce these margins.
Weakness 4 + Question 4: Implementation details and verifier training.
Response. The three learned components have distinct training procedures: the generator uses GRPO, the verifier uses SFT, and the solver uses GSPO.
Generator. Section 2.2 defines the task-specific reward components and normalized answer entropy. Appendix A.2 gives the GRPO objective, including the clipped policy-ratio surrogate, group-relative advantage, KL penalty, and numerical stabilization. All formats share a factual gate: a failed paragraph-level source-consistency check sets the trajectory reward to zero.
Open-ended generation combines format, lexical self-containment, paper selection, and coverage. Short-form generation combines format, self-containment, section selection, and difficulty; calculation-heavy generation additionally uses a quantitative-structure reward.
Verifier. Section 2.3 and Appendix A.2 specify supervised fine-tuning on MAS-adjudicated paragraph labels using next-token cross-entropy under a fixed binary output format. The verifier is frozen during generator optimization. The label-construction procedure includes independent assessment, source-span extraction, cross-examination, and rotated adjudication.
Solver. Section 7 initializes a fresh GPT-OSS-20B solver from the same base model as the generator and difficulty model, and trains it with GSPO on the generated, validated questions. The reported transfer results concern this complete same-base pipeline.
We will consolidate the remaining numerical hyperparameters, reward coefficients, data partitions, sampling and canonicalization settings, and external evaluation configurations in the implementation description, while keeping the three components separate.
Additional concern: The appendix examples do not adequately illustrate the technical complexity.
Response. We will supplement the appendix with two technical examples from our broader generated calculation-question pool, with explicit provenance, identifiers, and solution sketches.
Example 1: Cutoff-dependent integration, item 597151fddde298da.
For , determine the limit of the following integral over an ellipsoidal domain:
The solver must account for the ellipsoidal cutoff and its finite boundary contribution. Applying spherical angular averaging before treating the cutoff would incorrectly give zero. The result is
Neither this expression nor its derivation is supplied in the question. The task requires reasoning about cutoff geometry, boundary contributions, and the limiting procedure.
Example 2: Convergence of a matrix-integral observable, item a570a95f6559e793.
Determine whether the second moment exists for the six-dimensional bosonic Yang–Mills matrix integral.
The partition function is finite, but the second moment diverges logarithmically along commuting-matrix directions. The key reasoning step is to distinguish normalization of the measure from convergence of the observable.
Together, these examples illustrate reasoning about regulator-dependent finite contributions and observable convergence, complementing the aggregate evaluations with concrete technical problems and solution explanations.
Thank you for the thoughtful response and for correcting the Section 5.1 numbers. I appreciate the clarifications. However, some of my main concerns would benefit from more concrete evidence. Any information on the following would be very helpful:
Corrected results. Since the reported Fact500 scores use the original set, it would be helpful to see whether the main results change after removing or repairing the 20 identified defects.
Judge validity. Section 5.2 states that the judge and prompt are reported explicitly, but I could not find the short-form judge model or prompt in the paper. Could you share them? Given the gap between LLM-judge and rule-based scores (75.8 vs. 42.2), a small manual check of cases where the two methods disagree would also help readers interpret the scores.
Calc500 checks. Could you share the coverage and agreement rate of the independent Python solutions (Appendix D.1)?
Entropy selection. If results on random vs. entropy selection, or without the difficulty reward, are available, I would be glad to see them. Otherwise, slightly scoping the related claims may be sufficient.
Details. If possible, please include the U2-Train data size, how overlap with the evaluation benchmarks was checked, and key values not given in the paper (reward coefficients, oracle sampling temperature, and verifier training size and accuracy).
I understand time is limited. Partial answers are welcome.
Thank you for recognizing the source-blind audit, controlled repair experiment, source-access analysis, and training application. We clarify the evidence supporting benchmark quality and distinguish the contributions established by the existing experiments.
Response. Our quality-control procedure combines source-grounded generation, reference-answer adjudication, professional human review, and source-blind validity analysis. These stages address different aspects of benchmark quality.
During generation, an SFT-trained factual verifier checks each reasoning paragraph against its source context. Reference construction uses independent answer proposals, cross-critique, and rotated adjudication, with tolerance-aware numerical agreement for calculation-heavy questions.
The professional review of 82 question–reference pairs reports 84.1% fully correct references, 98.8% question relevance, and 96.3% full source support. Answer uniqueness is assessed separately, with 89.0% judged to admit a single clear answer. As explained in Appendix A.5, the remaining non-fully-correct cases include distinct categories such as reference incompleteness, granularity disagreements, and uniqueness defects. They should not be combined into a single ambiguity rate. We will clarify the sampling frame and annotation protocol; the original study does not report inter-annotator agreement.
The complete Fact500 source-blind audit independently tests question specification by searching for materially different supported answers satisfying every explicit constraint. The controlled intervention then compares original questions, minimal repairs restoring a missing qualifier, and placebo paraphrases.
Rebuttal Table 4: Controlled changes in normalized entropy, reported in Section 6.
The repair effect persists against the placebo control, supporting the explicit-constraint check used by the reject-or-rewrite procedure. The reported Fact500 scores describe the fixed evaluation set; this intervention measures entropy changes rather than post-repair leaderboard accuracy.
For response scoring, semantic evaluation follows the approach used in SimpleQA [1], BrowseComp [2], HLE [3], and FRAMES [4]. Open-ended evaluation adopts DeepEval [5], whose original paper provides human-alignment experiments. We will distinguish this published methodological evidence from U2-specific evaluator calibration.
For calculation-heavy questions, Appendix D.1 describes the reference-construction procedure and sandboxed Python checks. We will keep these checks distinct from the complete Fact500 validity audit and clarify their coverage and disagreement-handling protocol separately from model accuracy.
Response. The paper provides evidence at both the generation and downstream-training stages.
Under a fixed source pool and screening policy, Appendix C.1 evaluates 500 candidates at each generator checkpoint:
Rebuttal Table 5: Generator improvement under the combined training objective.
The simultaneous improvements in usable yield, well-posedness, and entropy support the effectiveness of the combined generation objective within the stated screening protocol. The difficulty-model ablation additionally shows that oracle choice changes the challenge level of the selected questions.
U2-Train demonstrates external training utility: the same-base GPT-OSS-20B solver improves from 59.3 to 64.1 on GAIA, 76.2 to 79.5 on FRAMES, and 16.6 to 20.2 on HLE. We attribute these gains to the complete pipeline and distinguish them from the marginal contribution of entropy selection, which requires matched component-level controls.
Regarding oracle dependence, U2-Bench evaluates the selected questions across GPT, Gemini, and Claude families. U2-Train further studies a separate setting in which the generator, difficulty model, and target solver are distinct instances initialized from the same GPT-OSS-20B base model. Thus, its difficulty signal targets the base model’s own capability gaps rather than relying on GPT-5.5 as the difficulty oracle.
[1] Measuring Short-Form Factuality in Large Language Models.
[2] BrowseComp: A Simple Yet Challenging Benchmark for Browsing Agents.
[3] Humanity’s Last Exam.
[4] Fact, Fetch, and Reason: A Unified Evaluation of Retrieval-Augmented Generation.
[5] LiveResearchBench: A Live Benchmark for User-Centric Deep Research in the Wild.