- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
VADE proposes a data engine that automatically creates verifiable professional analysis tasks from public sources. It builds reference analyses from evidence, verifies them through independent reproduction, and generates scoring contracts to enable scalable evaluation and training. The authors validate VADE across finance, law, medicine, and energy, showing high expert-reviewed quality, strong difficulty in exposing model weaknesses, and improved agent performance after training on verified trajectories.
Critical strengths
- The authors focus on the problem of data synthesis for professional domains.
- The synthesized data is manually reviewed, with 93.5% of tasks meeting a high bar for source fidelity, answer correctness, and evaluation fairness.
- The evaluation results show clear performance gaps among different models, indicating that the benchmark can effectively distinguish model capabilities.
Critical weaknesses
- Although the paper claims that VADE can be applied to professional domains in general (without specifying a particular domain), the experiments are mainly conducted in the finance domain. VADE synthesizes 1,000 tasks for Finance, while only 85 tasks are synthesized for both Energy and Medicine. Moreover, the training experiments are also only conducted in the finance domain. This raises concerns about whether VADE can be effectively applied to other professional domains such as Energy and Medicine, or whether it is mainly effective for Finance.
- The experiments do not compare VADE with previous approaches or even simple baselines, making it difficult for readers to understand the relative performance and positioning of VADE.
- The paper does not include ablation studies, making it unclear whether each component of VADE is necessary, such as the impact of removing Independent Reproduction or how the Question-first approach performs.
Questions for authors
- What specific public sources were used for the Energy and Medicine domains? The paper does not provide this information.
- Why is there such a large difference in the scale of experiments across different domains? What evidence demonstrates that VADE can generalize well beyond the finance domain?
- The paper does not include ablation studies to evaluate the contribution of each component in VADE. How much does each stage of the pipeline contribute to the final task quality?
3 comments
Response to Reviewer 8vDn (2/3)
W3 + Q3. Ablation: what each step contributes
For the ablation and to show how much each step contributes to task quality, we go through the pipeline step by step in the table below. The “Ablation” column describes how removing or replacing a step affects the final task quality, and the “Result” column gives the numbers we measured. The steps are listed in the order in which they run, and each is defined in §2 and Appendix A.1. Below the table, we discuss in separate paragraphs the steps the reviewer names and the other steps that need further explanation. The table is in Appendix F.5 of the revised paper (Table 26).
| Step | What it does | Ablation | Result |
|---|---|---|---|
| Discovery routes | Propose candidate analyses from the sources | Not a check; nothing to remove | The ten routes are listed in Appendix A.2, Table 7 |
| Novelty check | Removes candidates that duplicate an earlier one | Without it, the candidates it removes stay | 10.5% of the candidates in finance, Figure 3 |
| Integration | Ties every relationship of a candidate to a verbatim quotation of the sources; a candidate the sources do not support exits | Without it, the candidates it removes stay | 31.3% of the candidates in finance, Figure 3 |
| Difficulty screen | Ranks candidates without solving them; the easiest exit | Without it, the candidates it removes stay. It affects difficulty, not correctness | 13.9% of the candidates in finance, counted together with those whose reference analysis could not be completed, Figure 3 |
| Building the reference analysis | Derives every output from the quotations | Without it there is no answer. The last row replaces it with direct writing | See the last row |
| Code checks | Code matches every quotation in a worker’s output against the sources and recomputes every step | Without them, an output with a quotation that is not in the sources goes on to the next step | About 6% of the worker outputs in finance and 9% in law are rejected for such a quotation and redone |
| Independent reproduction | A second worker derives the answer without seeing the reference analysis, and the two are compared | Without it, the defects it exposes stay in the reference analysis. We also replaced it with a simple inspector to measure the effect on task quality | It exposed a defect in the reference analysis of 5.9% of the finance candidates and 16.2% of the law candidates (§3.2). The inspector did not correctly identify 43% of them (below) |
| Question written last | The question is written from the verified reference analysis | We wrote the question first | 5 of 20 questions needed their requirements corrected, against 1 of 20 under the current pipeline (below) |
| Validity review | Judges the question from what the solver sees | Without it, a returned task is delivered as it was before the return | 21.2% of the delivered finance tasks and 5.6% of the delivered law tasks were returned by it (below) |
| Consistency review | Checks the question, the reference analysis, and the scoring contract against one another | Without it, a returned task is delivered as it was before the return | 44.5% of the delivered finance tasks and 24.0% of the delivered law tasks were returned by it (below) |
| Wording polish | Edits only the wording of the question | Without it, the question is delivered with its original wording | 32% of the polish reviews in finance edited the wording (below) |
| Whole pipeline | All of the above | The model writes a task directly from the same sources | Defective reference answer in 16 of 50 tasks, against 4 of 50 for VADE (Appendix F.3) |
Independent reproduction. This step guards against the most serious kind of defect, a reference answer that is itself wrong. To measure its effect, we changed this one step and kept everything else. In independent reproduction, a second worker, the reproducer, does not see the reference analysis and derives the answer from the quotations alone. We replaced it with an inspector, which receives the reference analysis together with the quotations and is asked to find what is wrong, with the same model and budget. We ran this on 133 cases in which independent reproduction had found a defect in production, using the version before the repair (Appendix F.4, Table 25).
| Defects | Found | Partly found | Not found | |
|---|---|---|---|---|
| Finance | 56 | 35 | 3 | 18 |
| Law | 77 | 41 | 12 | 24 |
| Total | 133 | 76 (57%) | 15 (11%) | 42 (32%) |
(Continued in the next comment.)
Response to Reviewer 8vDn (3/3)
“Partly found” means the inspector raised a concern about the right item, but the problem it described was not the actual defect.
The inspector missed 32% of the defects entirely, and 29 of these 42 change the core conclusion of the task. It reads the reference analysis and checks whether each step holds, so an analysis that is internally consistent passes, even if the same quotations also support another reasonable answer. The reproducer never sees the reference analysis; in such a case it arrives at the other answer, and the comparison shows the difference. The direct-generation baseline gives the same result: of its 16 defective tasks, 4 were found only by reproduction and comparison (Appendix F.3).
Question written last.
We have two sets of results. The first is a comparison during development (§7.2, Appendix F.2): we wrote the question first and then searched for its answer, 20 times, with the same verification as the current pipeline. In 5 of them the question itself had to be corrected before it passed: the basis of a comparison, a threshold, or the range over which the conclusion holds. Under the current pipeline this happened in 1 of 20. The second is the direct-generation baseline, which also writes the question before any answer is verified: in 7 of its 50 tasks the question and the reference answer do not match, against 1 of the 50 VADE tasks (Appendix F.3). Both have the same cause: a question written first fixes requirements that the sources may not support.
Validity review and consistency review.
The table covers only the delivered tasks. It gives the share that each review returned at least once and the average number of returns per task (§3.2; Appendix F.5, Table 27).
| Finance: returned at least once | Finance: returns per task | Law: returned at least once | Law: returns per task | |
|---|---|---|---|---|
| Validity review | 21.2% | 0.24 | 5.6% | 0.06 |
| Consistency review | 44.5% | 0.60 | 24.0% | 0.32 |
| Either review | 56.1% | 0.84 | 27.2% | 0.37 |
A returned task is not discarded. It goes back with the problems the review found, each confirmed against the task by a separate worker; these are repaired and the task is reviewed again, at most five times (§2.3, Appendix A.1). The consistency review returns the most because it is the first check after the question and the scoring contract are written, and a mismatch between the two is the most common cause of repair (§3.2). After all of these steps, the expert review found a substantive problem in 6.5% of the finance tasks and in 1 of 50 law tasks (§3.4).
Wording polish.
The polish makes sure the question tells the solver everything the scoring expects and nothing more. In finance, 32% of the polish reviews edited the wording. The table gives the share of these edits that made each kind of change; one edit can make several, so the rows add up to more than 100% (Appendix F.5, Table 28).
| What the edit does | Share of the edits |
|---|---|
| States the form the answer should take | 34% |
| Makes explicit a requirement that the question only implied | 34% |
| Removes a hint to the answer | 29% |
| Replaces internal vocabulary | 22% |
| Names a source less specifically | 8% |
| Removes a request that is not scored | 7% |
§7.1 also reports the result of replacing the model at each step with a weaker one.
Q1. Sources of the energy and medicine domains
The revised paper now lists the sources of the energy and medicine runs in Appendix A.5, with a pointer from §3.1.
In energy, the sources are the primary records of the bodies that regulate and operate the U.S. energy sector: event reports from the nuclear regulator (NRC); orders and filings from the federal energy regulator (FERC) and the state public utility commissions; statistics from the Energy Information Administration (EIA); publications of the grid reliability organization (NERC); and market data and reports from the regional grid operators (PJM, SPP, CAISO, ERCOT, ISO-NE) and their market monitors, together with federal regulations and the Federal Register.
In medicine, the sources are FDA review documents, labels and their revision histories, approval letters, and safety communications; ClinicalTrials.gov records and their version histories; and PubMed. Like the SEC filings used in finance, they are primary records published by regulators or official bodies.
We thank the reviewer again for the comments, and we are glad to clarify any of these points further.
Response to Reviewer 8vDn (1/3)
We thank the reviewer for the careful review and the specific questions. We respond to each point below. Section, table, and figure numbers refer to the revised paper, in which changes are marked in blue.
W1 + Q2. Scale of the experiments and evidence beyond finance
Since submission we have added tasks in law, energy, and medicine, evaluated models on the new tasks, and repeated the training experiment of finance in each of them. We summarize the results here; the details are in Sections 1–3 of our response to all reviewers and in the revised paper.
We extended the task sets from 372 to 642 in law, from 85 to 211 in energy, and from 85 to 205 in medicine, with the same pipeline (§3.3). The performance of current models on these tasks is reported in §4.2 (Table 2; by task category in Appendix B, Table 13).
We reviewed the quality of the tasks outside finance. An expert with a legal background reviewed 50 law tasks and found a substantive problem in 1. In energy and medicine, two models reviewed 50 tasks each under the same criteria and flagged 2 to 4 of them; a re-review by the authors confirmed 2 in energy and none in medicine (§3.4, Appendix H).
We repeated the training experiment of finance (§5) in law, energy, and medicine. After fine-tuning on the tasks of a domain, the item pass rate of Qwen3.5-9B rises on the held-out VADE tasks, from 27.6% to 44.7% in law, from 43.6% to 67.9% in energy, and from 46.9% to 53.8% in medicine. It also rises on two external benchmarks, from 47.5% to 63.3% on LAB in law and from 26.4% to 42.8% on LifeSci Diamond in medicine (§5.2, Table 4).
Building a task costs a median of 24.4M tokens (§3.2), and we gave priority to finance and law for two reasons.
First, these are the domains where the evaluation of professional agents is most established. The Finance Agent Benchmark (Vals AI) and the Legal Agent Benchmark (LAB, Harvey) have public leaderboards and are routinely run on newly released models. Both are written by experts, which limits their scale and makes them hard to use for training; this is the problem VADE addresses (§1).
Second, the expert-written benchmarks in these two domains give us an external reference. Our finance and law tasks follow their task categories, solving environments, and judging procedures (§3.3, §4.1), and the models trained on VADE data are tested on them: four finance benchmarks in the paper, and LAB in law since submission (§5).
W2. Comparison with previous approaches and a simple baseline
We have added both: a comparison with previous approaches and a simple baseline.
For the comparison with previous approaches, we compare VADE with existing task-synthesis methods and with expert-written benchmarks on the form of the tasks they produce: their sources, how they are solved, the form of the answer, and how it is scored (Section 4 of our response to all reviewers; Appendix G, Table 29). VADE and the expert-written benchmarks, which the paper uses for external evaluation (§5), share one task form: multi-turn tool use over professional primary documents, answered with a long analysis and scored with a rubric. None of the other synthesis methods produces tasks of this form.
As a simple baseline, we built direct generation, following the prompt-based generation that Autodata uses as its control: the same model, given the same source files, writes the question, the reference answer, and the scored items directly. The scored content of the reference answer is defective in 16 of the 50 tasks generated this way, against 4 of the 50 VADE tasks built from the same sources (Fisher’s exact test, two-sided, p = 0.005). The design and the defects by type are in Section 4 of our response to all reviewers and in Appendix F.3 of the revised paper (Table 24).