acceptodds
Under review as a conference paper at ICLR 2027

VADE: A VERIFIABLE AGENT DATA ENGINE FROM PROFESSIONAL SOURCES

Abstract

Language agents have demonstrated their practical value, particularly in software engineering, where abundant public code and executable tests support training and evaluation at scale. Extending this success to other professional domains, such as law and finance, is not straightforward: many agent tasks involve open-ended analysis, making high-quality outcomes difficult to verify and, in turn, difficult to use as scalable supervision. As a result, high-quality tasks in these domains are still largely expert-authored, making them expensive to scale and limiting their use for training. To address this bottleneck, we introduce VADE, a data engine that automatically constructs verifiable analysis tasks from publicly available sources used in professional practice. VADE exploits a simple asymmetry: professional analyses are difficult to solve because the relevant evidence must first be found and interpreted, but much easier to verify once that evidence is fixed. During task synthesis, the engine first constructs a reference analysis from source evidence and asks an independent agent to reproduce its conclusions from the same evidence without seeing the reference analysis. Only after the analysis is verified is it compiled into an open-ended task with an item-level scoring contract, allowing the same task to support both evaluation and training. We apply VADE across four professional domains, with independent post-hoc expert review finding 93.5% of our constructed tasks to meet a high bar for source fidelity, answer correctness, and evaluation fairness. In finance, these tasks expose substantial remaining model errors, particularly unsupported facts and inferences, while training on verified solver trajectories improves held-out performance and yields gains that transfer to public finance benchmarks such as Finance Agent Benchmark v2. Together, these results suggest a practical path toward scalable, verifiable supervision for open-ended professional tasks.

Primary area:
datasets and benchmarks
Keywords:
Language Agents, Data Synthesis, Agent Benchmark, Task Generation, Agent Evaluation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_8vDnOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

VADE proposes a data engine that automatically creates verifiable professional analysis tasks from public sources. It builds reference analyses from evidence, verifies them through independent reproduction, and generates scoring contracts to enable scalable evaluation and training. The authors validate VADE across finance, law, medicine, and energy, showing high expert-reviewed quality, strong difficulty in exposing model weaknesses, and improved agent performance after training on verified trajectories.

Critical strengths
  1. The authors focus on the problem of data synthesis for professional domains.
  2. The synthesized data is manually reviewed, with 93.5% of tasks meeting a high bar for source fidelity, answer correctness, and evaluation fairness.
  3. The evaluation results show clear performance gaps among different models, indicating that the benchmark can effectively distinguish model capabilities.
Critical weaknesses
  1. Although the paper claims that VADE can be applied to professional domains in general (without specifying a particular domain), the experiments are mainly conducted in the finance domain. VADE synthesizes 1,000 tasks for Finance, while only 85 tasks are synthesized for both Energy and Medicine. Moreover, the training experiments are also only conducted in the finance domain. This raises concerns about whether VADE can be effectively applied to other professional domains such as Energy and Medicine, or whether it is mainly effective for Finance.
  2. The experiments do not compare VADE with previous approaches or even simple baselines, making it difficult for readers to understand the relative performance and positioning of VADE.
  3. The paper does not include ablation studies, making it unclear whether each component of VADE is necessary, such as the impact of removing Independent Reproduction or how the Question-first approach performs.
Questions for authors
  1. What specific public sources were used for the Energy and Medicine domains? The paper does not provide this information.
  2. Why is there such a large difference in the scale of experiments across different domains? What evidence demonstrates that VADE can generalize well beyond the finance domain?
  3. The paper does not include ablation studies to evaluate the contribution of each component in VADE. How much does each stage of the pipeline contribute to the final task quality?
3 comments
AuthorsOpenReview ↗

Response to Reviewer 8vDn (1/3)

We thank the reviewer for the careful review and the specific questions. We respond to each point below. Section, table, and figure numbers refer to the revised paper, in which changes are marked in blue.

W1 + Q2. Scale of the experiments and evidence beyond finance

Since submission we have added tasks in law, energy, and medicine, evaluated models on the new tasks, and repeated the training experiment of finance in each of them. We summarize the results here; the details are in Sections 1–3 of our response to all reviewers and in the revised paper.

We extended the task sets from 372 to 642 in law, from 85 to 211 in energy, and from 85 to 205 in medicine, with the same pipeline (§3.3). The performance of current models on these tasks is reported in §4.2 (Table 2; by task category in Appendix B, Table 13).

We reviewed the quality of the tasks outside finance. An expert with a legal background reviewed 50 law tasks and found a substantive problem in 1. In energy and medicine, two models reviewed 50 tasks each under the same criteria and flagged 2 to 4 of them; a re-review by the authors confirmed 2 in energy and none in medicine (§3.4, Appendix H).

We repeated the training experiment of finance (§5) in law, energy, and medicine. After fine-tuning on the tasks of a domain, the item pass rate of Qwen3.5-9B rises on the held-out VADE tasks, from 27.6% to 44.7% in law, from 43.6% to 67.9% in energy, and from 46.9% to 53.8% in medicine. It also rises on two external benchmarks, from 47.5% to 63.3% on LAB in law and from 26.4% to 42.8% on LifeSci Diamond in medicine (§5.2, Table 4).

Building a task costs a median of 24.4M tokens (§3.2), and we gave priority to finance and law for two reasons.

First, these are the domains where the evaluation of professional agents is most established. The Finance Agent Benchmark (Vals AI) and the Legal Agent Benchmark (LAB, Harvey) have public leaderboards and are routinely run on newly released models. Both are written by experts, which limits their scale and makes them hard to use for training; this is the problem VADE addresses (§1).

Second, the expert-written benchmarks in these two domains give us an external reference. Our finance and law tasks follow their task categories, solving environments, and judging procedures (§3.3, §4.1), and the models trained on VADE data are tested on them: four finance benchmarks in the paper, and LAB in law since submission (§5).

W2. Comparison with previous approaches and a simple baseline

We have added both: a comparison with previous approaches and a simple baseline.

For the comparison with previous approaches, we compare VADE with existing task-synthesis methods and with expert-written benchmarks on the form of the tasks they produce: their sources, how they are solved, the form of the answer, and how it is scored (Section 4 of our response to all reviewers; Appendix G, Table 29). VADE and the expert-written benchmarks, which the paper uses for external evaluation (§5), share one task form: multi-turn tool use over professional primary documents, answered with a long analysis and scored with a rubric. None of the other synthesis methods produces tasks of this form.

As a simple baseline, we built direct generation, following the prompt-based generation that Autodata uses as its control: the same model, given the same source files, writes the question, the reference answer, and the scored items directly. The scored content of the reference answer is defective in 16 of the 50 tasks generated this way, against 4 of the 50 VADE tasks built from the same sources (Fisher’s exact test, two-sided, p = 0.005). The design and the defects by type are in Section 4 of our response to all reviewers and in Appendix F.3 of the revised paper (Table 24).

AuthorsOpenReview ↗

Response to Reviewer 8vDn (2/3)

W3 + Q3. Ablation: what each step contributes

For the ablation and to show how much each step contributes to task quality, we go through the pipeline step by step in the table below. The “Ablation” column describes how removing or replacing a step affects the final task quality, and the “Result” column gives the numbers we measured. The steps are listed in the order in which they run, and each is defined in §2 and Appendix A.1. Below the table, we discuss in separate paragraphs the steps the reviewer names and the other steps that need further explanation. The table is in Appendix F.5 of the revised paper (Table 26).

StepWhat it doesAblationResult
Discovery routesPropose candidate analyses from the sourcesNot a check; nothing to removeThe ten routes are listed in Appendix A.2, Table 7
Novelty checkRemoves candidates that duplicate an earlier oneWithout it, the candidates it removes stay10.5% of the candidates in finance, Figure 3
IntegrationTies every relationship of a candidate to a verbatim quotation of the sources; a candidate the sources do not support exitsWithout it, the candidates it removes stay31.3% of the candidates in finance, Figure 3
Difficulty screenRanks candidates without solving them; the easiest exitWithout it, the candidates it removes stay. It affects difficulty, not correctness13.9% of the candidates in finance, counted together with those whose reference analysis could not be completed, Figure 3
Building the reference analysisDerives every output from the quotationsWithout it there is no answer. The last row replaces it with direct writingSee the last row
Code checksCode matches every quotation in a worker’s output against the sources and recomputes every stepWithout them, an output with a quotation that is not in the sources goes on to the next stepAbout 6% of the worker outputs in finance and 9% in law are rejected for such a quotation and redone
Independent reproductionA second worker derives the answer without seeing the reference analysis, and the two are comparedWithout it, the defects it exposes stay in the reference analysis. We also replaced it with a simple inspector to measure the effect on task qualityIt exposed a defect in the reference analysis of 5.9% of the finance candidates and 16.2% of the law candidates (§3.2). The inspector did not correctly identify 43% of them (below)
Question written lastThe question is written from the verified reference analysisWe wrote the question first5 of 20 questions needed their requirements corrected, against 1 of 20 under the current pipeline (below)
Validity reviewJudges the question from what the solver seesWithout it, a returned task is delivered as it was before the return21.2% of the delivered finance tasks and 5.6% of the delivered law tasks were returned by it (below)
Consistency reviewChecks the question, the reference analysis, and the scoring contract against one anotherWithout it, a returned task is delivered as it was before the return44.5% of the delivered finance tasks and 24.0% of the delivered law tasks were returned by it (below)
Wording polishEdits only the wording of the questionWithout it, the question is delivered with its original wording32% of the polish reviews in finance edited the wording (below)
Whole pipelineAll of the aboveThe model writes a task directly from the same sourcesDefective reference answer in 16 of 50 tasks, against 4 of 50 for VADE (Appendix F.3)

Independent reproduction. This step guards against the most serious kind of defect, a reference answer that is itself wrong. To measure its effect, we changed this one step and kept everything else. In independent reproduction, a second worker, the reproducer, does not see the reference analysis and derives the answer from the quotations alone. We replaced it with an inspector, which receives the reference analysis together with the quotations and is asked to find what is wrong, with the same model and budget. We ran this on 133 cases in which independent reproduction had found a defect in production, using the version before the repair (Appendix F.4, Table 25).

DefectsFoundPartly foundNot found
Finance5635318
Law77411224
Total13376 (57%)15 (11%)42 (32%)

(Continued in the next comment.)

AuthorsOpenReview ↗

Response to Reviewer 8vDn (3/3)

“Partly found” means the inspector raised a concern about the right item, but the problem it described was not the actual defect.

The inspector missed 32% of the defects entirely, and 29 of these 42 change the core conclusion of the task. It reads the reference analysis and checks whether each step holds, so an analysis that is internally consistent passes, even if the same quotations also support another reasonable answer. The reproducer never sees the reference analysis; in such a case it arrives at the other answer, and the comparison shows the difference. The direct-generation baseline gives the same result: of its 16 defective tasks, 4 were found only by reproduction and comparison (Appendix F.3).

Question written last.

We have two sets of results. The first is a comparison during development (§7.2, Appendix F.2): we wrote the question first and then searched for its answer, 20 times, with the same verification as the current pipeline. In 5 of them the question itself had to be corrected before it passed: the basis of a comparison, a threshold, or the range over which the conclusion holds. Under the current pipeline this happened in 1 of 20. The second is the direct-generation baseline, which also writes the question before any answer is verified: in 7 of its 50 tasks the question and the reference answer do not match, against 1 of the 50 VADE tasks (Appendix F.3). Both have the same cause: a question written first fixes requirements that the sources may not support.

Validity review and consistency review.

The table covers only the delivered tasks. It gives the share that each review returned at least once and the average number of returns per task (§3.2; Appendix F.5, Table 27).

Finance: returned at least onceFinance: returns per taskLaw: returned at least onceLaw: returns per task
Validity review21.2%0.245.6%0.06
Consistency review44.5%0.6024.0%0.32
Either review56.1%0.8427.2%0.37

A returned task is not discarded. It goes back with the problems the review found, each confirmed against the task by a separate worker; these are repaired and the task is reviewed again, at most five times (§2.3, Appendix A.1). The consistency review returns the most because it is the first check after the question and the scoring contract are written, and a mismatch between the two is the most common cause of repair (§3.2). After all of these steps, the expert review found a substantive problem in 6.5% of the finance tasks and in 1 of 50 law tasks (§3.4).

Wording polish.

The polish makes sure the question tells the solver everything the scoring expects and nothing more. In finance, 32% of the polish reviews edited the wording. The table gives the share of these edits that made each kind of change; one edit can make several, so the rows add up to more than 100% (Appendix F.5, Table 28).

What the edit doesShare of the edits
States the form the answer should take34%
Makes explicit a requirement that the question only implied34%
Removes a hint to the answer29%
Replaces internal vocabulary22%
Names a source less specifically8%
Removes a request that is not scored7%

§7.1 also reports the result of replacing the model at each step with a weaker one.

Q1. Sources of the energy and medicine domains

The revised paper now lists the sources of the energy and medicine runs in Appendix A.5, with a pointer from §3.1.

In energy, the sources are the primary records of the bodies that regulate and operate the U.S. energy sector: event reports from the nuclear regulator (NRC); orders and filings from the federal energy regulator (FERC) and the state public utility commissions; statistics from the Energy Information Administration (EIA); publications of the grid reliability organization (NERC); and market data and reports from the regional grid operators (PJM, SPP, CAISO, ERCOT, ISO-NE) and their market monitors, together with federal regulations and the Federal Register.

In medicine, the sources are FDA review documents, labels and their revision histories, approval letters, and safety communications; ClinicalTrials.gov records and their version histories; and PubMed. Like the SEC filings used in finance, they are primary records published by regulators or official bodies.

We thank the reviewer again for the comments, and we are glad to clarify any of these points further.

Reviewer_qX25OpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

VADE constructs professional agent tasks by grounding a reference analysis in source evidence, independently reproducing it, and then generating the question and scoring contract. The paper evaluates the resulting tasks across several domains and demonstrates their usefulness for fine-tuning finance agents.

Critical strengths

1/ A well-motivated and original architecture. Verifying the reference analysis before writing the question makes sense for professional tasks without an execution oracle. Evidence grounding, independent reproduction, and targeted repair form a coherent and useful system design.

2/ Solid evidence of quality and training utility. Two finance reviewers audited 186 tasks, with 93.5% requiring no substantive revision. Fine-tuning raises Qwen3.5-27B's held-out task pass rate from 6.7% to 21.3%, with further gains on public subsets of external finance benchmarks. Together, these results provide meaningful support for the generated data's usefulness.

Critical weaknesses

Data quality needs broader validation: The human audit covers only finance, and the paper does not report agreement or disagreement resolution between the two reviewers. This limits confidence in the reported quality assessment, particularly when extending the conclusions to other domains.

Suggestion: Since expert reviews are already available, it would be useful to compare LLM task-quality judgments against them. Reliable agreement could support larger quality audits, with similar validation needed before extending to other domains.

Questions for authors

Please see Critical Weaknesses

3 comments
AuthorsOpenReview ↗

Response to Reviewer qX25

We thank the reviewer for the constructive review. We respond to the weakness and to the suggestion below. Section, table, and figure numbers refer to the revised paper, in which changes are marked in blue.

W1. Broader validation of data quality

On the two points raised here, we report below the agreement between the two finance reviewers, a human review in law, and a review by models in all four domains; all three are in Appendix H of the revised paper.

Agreement between the two finance reviewers and how disagreements were resolved.

The table gives the number of tasks in which each reviewer found a substantive problem (Appendix H, Table 30).

Reviewer 1Reviewer 2BothAgreementCohen’s κ
Task realism000100%—
Source fidelity000100%—
Correctness of the conclusion52298.4%0.56
Scoring fairness93295.7%0.32
Whole task145494.1%0.40

Of the 186 tasks, 15 were flagged by at least one reviewer. We asked one of the reviewers to re-examine these 15 tasks one by one against the sources: 12 were confirmed to have a substantive problem and the other 3 were not. This is where the 93.5% in the paper comes from (174/186). Of the 12, 9 concern scoring fairness (the scoring contract asks for more than the question requires), 2 an information cutoff set too early, and 1 a misstatement in the reference analysis.

Human review in law.

One expert with a legal background reviewed 50 law tasks under the same criteria.

Law (50)
Task realism0
Source fidelity0
Correctness of the conclusion1
Scoring fairness1
Whole task with a substantive problem1

The one problem found is that the scoring contract requires the answer to state a specific deadline that has nothing to do with the regulatory judgment the question asks for.

Review by models in all four domains.

We used two models, Claude Fable 5.2 and GPT-6 Astra, to review 50 tasks from each domain under the same four criteria. Like the human reviewers, each model receives the question, the reference analysis, the scoring contract, and the source documents, and can reopen the official sources and recompute. The table gives the number of tasks in which a substantive problem was flagged (Fable 5.2 / Astra).

Finance (50)Law (50)Energy (50)Medicine (50)
Task realism0 / 00 / 00 / 10 / 0
Source fidelity1 / 10 / 10 / 00 / 0
Correctness of the conclusion0 / 00 / 10 / 11 / 1
Scoring fairness14 / 121 / 42 / 22 / 2
Flagged by the model14 / 121 / 42 / 43 / 3
Confirmed by human review5120

The last row gives how many of the flagged tasks were confirmed by human review. In finance and law, the tasks are among those reviewed by the experts. In energy and medicine, authors with a computer-science background re-reviewed the flagged tasks against the sources, assisted by an LLM, and confirmed 2 in energy and none in medicine. On the finance tasks, both models found every problem the experts found and flagged others that were not confirmed (see our reply to the suggestion below).

Suggestion. Comparing LLM judgments of task quality with expert review

We thank the reviewer for this suggestion. The review reported under W1 is a direct attempt at it: the models judge task quality independently under the four criteria of the human review, and their judgments are compared with the human judgments on the same tasks. The results are as follows.

ModelConfirmed / flaggedFalse alarmsMissesAgreement with the expertsCohen’s κ
Fable 5.25/14 = 35.7%9082.0%0.44
Astra5/12 = 41.7%7086.0%0.52

The models missed none of the five problems confirmed by the experts but raised 9 and 7 false alarms, mostly by treating equivalent wording or ordinary precision requirements as unfair. We therefore see them as a first screen whose flags an expert then checks, not as a replacement for expert review. The full comparison is in Appendix H of the revised paper.

I thank the authors for their efforts in addressing the reviewers’ comments. After carefully considering the rebuttal, I remain cautiously positive about the paper overall and will maintain my original rating 3.

Thank you for reviewing our responses and for your positive assessment!

Your comments have been valuable in guiding these improvements, and we appreciate your time and consideration.

Reviewer_8TztOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The authors propose VADE a Data Engine that can generate questions, reference answers and grading criteria from different data sources. The goal is create an on-demand benchmark for Question Answering for professional tasks like Law and Finances by using various file sources. The idea is to finetune an existing agent on these generated questions and answers, which they did via LoRA and Qwen. They showed that their finetuned agent performs better in the VADE generated held out set.

Critical strengths
  • Being able to generate a Q&A benchmark for new problem domains is both interesting and relevant
  • The authors created 4 benchmarks in interesting domains VADE-Finance, VADE-Law, VADE-Energy-Preview, and VADE-MedicinePreview
  • I like the idea of using a verifier to make sure the answer can be reached from multiple independent models as that supports that the question is solvable.
  • many different models have been tested from deepseek, Qwen to GLM and MiniMax and they showed that the models finetuned on VADE generted set can perform better on VADE generated test set which shows that there is some potential that the generated data has signal.
Critical weaknesses
  • The paper is poorly written unfortunately and its very hard to follow, for example figure 1 is hard to understand. It is not clear what is the exact input to the model and how the questions are exactly created and how their reference answers are generated. There are also phrases that do not make much sense such as "Discover a relationship worth analysing" and "Question written last then gated from the solver's view" which are not explained. Also terms like "reference analysis" and "scoring contract" are not explained properly. Also some of the sentences are very strange for example, consider this one "The reference analysis is then verified by independent reproduction. A second worker, the reproducer, receives only the analytical objective, the evidence fragments, and a neutral description of each required output, and must derive every output from the evidence alone. ", much of it seems to contain terms that are not explained.

  • It is not clear whether the generated tasks are useful or not. The way the authors validated the usefulness is by having the model finetune on the generated training set and test it on a held out set. However, the held out set has also been generated by VADE which must be similar in distribution as the training set so it seems like we are just overfitting here. The authors need to evaluate the model on an actual real held out set that hasn't been generated by VADE. Also it is not clear whether there is data contamination between the test and the training set where the model being trained could be memorizing answers that it can use for the test set.

  • There is a lot of missing details on how VADE actually works in terms of creating QA Tasks. Answers to these questions are missing: what is the exact input to VADE? what LLMs and prompts were being used to generate the questions and the reference answers? how were the example output generated and how accurate is the LLM as a judge being used to validate the agents for question answering? and how do we make sure that grading criteria is exhaustive enough and accurate?

  • Generating QA tasks for different domains and topics has lots of existing works. For example, FM2DS is a published work (https://arxiv.org/abs/2412.07030) that shows how to create good QA tasks for multi-hop long horizon queries given a set of topics. The authors need to compare between VADE and FM2DS (or other works where a system is being used to generate QA tasks) to see if VADE actually improves the quality of such task generation compared to existing methods.

Questions for authors

Please address the weaknesses above.

3 comments
AuthorsOpenReview ↗

Response to Reviewer 8Tzt (1/3)

W2. Usefulness of the tasks: evaluation outside VADE, and contamination between training and test

We would first like to clarify a key point. To verify that our data are useful and to test for overfitting, we evaluate the models fine-tuned on our data not only on VADE’s own test set, but also on four external benchmarks. Our training and test sets are also fully separated, not only by task but also by source: tasks that share a source, a family, or near-duplicate content are kept on the same side of the split (§5.1).

We use four expert-written benchmarks for finance agents: FAB v1, FAB v2, BigFinanceBench, and FrontierFinance. The results are reported in §5.2 and Table 3 of the main text: the item pass rate of both models rises on all four benchmarks, and the pass rate of Qwen3.5-27B, for example, rises from 29.6% to 51.9% on FAB v2. Appendix E and Appendix B.1 describe how the models were trained and how the benchmarks were run.

In addition, during the rebuttal period we ran the same training experiment in law, energy, and medicine, and evaluated the fine-tuned models on VADE's held-out tasks and on external benchmarks. The results are in Section 3 of our response to all reviewers and in §5, Table 4, and Appendix E of the revised paper.

The table below gives the pass rate (P) and the item pass rate (R) of Qwen3.5-9B before fine-tuning and after four epochs, on every benchmark we evaluated in each domain.

Trained onEvaluated onBase PBase REpoch 4 PEpoch 4 R
FinanceVADE-Finance-Test0.040.80.054.8
FAB v130.046.542.071.5
FAB v211.135.425.963.6
BigFinanceBench6.030.010.056.1
FrontierFinance1.017.11.038.6
LawVADE-Law-Test1.627.612.744.7
LAB0.847.54.163.3
EnergyVADE-Energy-Test0.043.68.067.9
MedicineVADE-Medicine-Test0.046.914.353.8
LifeSci Diamond0.026.40.042.8

The gains on the external benchmarks are comparable to those on VADE’s own test tasks. In finance, the item pass rate rises by 21.5 to 28.2 points on the four external benchmarks and by 14.0 points on VADE-Finance-Test. A model that had only overfitted to VADE’s distribution would not improve on the external benchmarks in this way.

W4. Comparison with FM2DS and other task-generation methods

We thank the reviewer for pointing us to FM2DS.

FM2DS and VADE produce different kinds of tasks. The table below compares the two.

FM2DSVADE
TaskMultimodal QAOpen-ended professional analysis
Answer1.95 words on averageA long analysis (median 7,570–9,927 characters, Appendix B.1)
SolvingA single responseMulti-turn tool use: median 16–30 turns in finance (Appendix B.1)
Sources2–3 Wikipedia pages per questionProfessional primary documents (Appendix A.5, Table 12)
Access to the sourcesGiven in the promptThe solver must locate and read them with tools (§2.4)
ScoringExact match and F1 against the reference answerItem by item against a scoring contract (11.9 answer atoms and 8.0 fatal conditions on average, §3.3)

FM2DS’s “multi-hop” refers to reasoning across the documents it is given, not to a long horizon of agent actions.

VADE tasks are much more challenging: an 8B model reaches 55.8 exact match on FM2DS’s benchmark, while a 9B model passes none of the VADE finance test tasks (§5.2, Table 3).

The two kinds of tasks share neither an answer form nor a scoring basis, so the quality of the tasks themselves cannot be compared on one scale. To answer the reviewer’s question about quality, we did two things.

First, we compared VADE with existing task-synthesis methods, including FM2DS, and with expert-written benchmarks on the form of the tasks they produce. None of the synthesis methods produces tasks that require multi-turn tool use over professional primary documents and are answered with a long, rubric-scored analysis. The full comparison is in Appendix G of the revised paper (Table 29).

Second, we built a direct-generation baseline, following the prompt-based generation that Autodata uses as its control: the same model, given the same source files, writes the question, the reference answer, and the scored items directly. The scored content of the reference answer is defective in 16 of the 50 tasks generated this way, against 4 of the 50 VADE tasks (p = 0.005). The details are in Section 4 of our response to all reviewers and in Appendix F.3 of the revised paper (Table 24).

In addition, we replaced independent reproduction with a review of the reference analysis by a model, which is how several existing methods check their tasks (Table 5 of the paper). The review failed to find 32% of the defects that independent reproduction had found (Appendix F.4, Table 25).

AuthorsOpenReview ↗

Response to Reviewer 8Tzt (2/3)

W3. Details of how VADE works

"what is the exact input to VADE?"

The input to VADE is a collection of source files. §2.1 and §3.1 define the input and give its categories and its size. The first sentence of §2.1 reads: “Sources are registered in the form in which they are obtained: web pages, structured filing data, PDF documents, tables and workbooks, audio and video, and images.” Appendix A further gives the complete construction procedure, a step-by-step description, and a concrete example that follows one task from its source data to the reference analysis, the independent reproduction, and the final question.

§3.1 also reports the number of source files: 96,524 in finance and 53,498 in law. In the revised paper, Appendix A.5 (Table 12) breaks them down by channel. In finance, 78.6% of the files come from the SEC and 11.3% from banking regulators (FDIC, Federal Reserve, OCC). In law, 51.5% come from the SEC, 24.8% from federal codes, regulations, and the Federal Register, and the rest from courts and federal and state agencies. Appendix A.5 also lists the sources of the energy and medicine runs.

"what LLMs and prompts were being used?"

§3.1: "The orchestrator and the workers at every stage ran mainly on GPT-5.6 Sol; steps that require several agents also drew on other model families, including DeepSeek V4 Flash, Grok 4.6, and Claude Opus 5."

As for the prompts, Appendix A.6 of the revised paper gives the full prompts of two key steps, independent reproduction and the validity review, and what the other prompts require is described step by step in Appendix A.1. The supplementary material submitted with the paper contains an implementation of the pipeline, including the prompts of all 17 roles: the orchestrator and 16 mutually isolated workers, reviewers, and judges.

"how were the example output generated?"

"Example output" may mean three things; we answer each.

  • If it refers to the answers of the evaluated models: the solving environments are described in §4.1 and Appendix B, which specify the tools, the information-cutoff filter, and the budgets (at most 120 tool calls and one hour per task in finance; at most 200 turns and one hour in law). The models' actual outputs on specific tasks are shown in Appendix C.4 and Appendix D.

  • If it refers to the reference answer of each task: its construction and independent reproduction are described in §2.2 and Appendix A.

  • If it refers to the examples shown in the paper: they are all tasks actually produced by the pipeline. Appendix A.3 walks one task through discovery, reference construction, and independent reproduction; Figure 2, Appendix A.4, and Appendix D show delivered finance tasks.

"how do we make sure that grading criteria is exhaustive enough and accurate?"

On "accurate". The scored items are not written from the question; they are generated from the answer atoms of the reference analysis. §2.2 requires: "A reference analysis counts only once a second worker has reproduced it independently and every atom agrees." In other words, the correct answer of every scored item has been derived a second time, from the same evidence, by an agent that cannot see it, and is kept only if the two agree. In production, this comparison exposed a defect on the task side, which was then repaired, in 5.9% of the finance candidates and 16.2% of the law candidates (§3.2). The direct-generation baseline in W4 shows the same from the other side: without these checks, the reference answer is defective in 16 of 50 tasks, against 4 of 50 VADE tasks.

On "exhaustive". There are three layers of assurance.

  1. The question and the scored items correspond one to one. §2.3: "code checks that every requirement of the question maps to atoms and every atom to a requirement." Appendix A.4 gives the complete scoring contract of one task, in which the four requirements of the question are mapped, one by one, to nine answer atoms.

  2. Reasonable alternative answers are accepted. The scoring contract explicitly lists the accepted alternatives and the tolerances. The question states the basis, the period, and the information cutoff that make its answer unique, which the validity review checks, and the consistency review checks that "reasonable alternatives are accepted". In finance, the validity review returned 22% of the tasks at first review and the consistency review 47% (§3.2); a returned task is repaired and reviewed again, and is delivered only after it passes.

  3. Human verification. The fourth criterion of the human review in §3.4 is "whether a reasonable alternative exists that the scoring contract does not accept". Of the 186 tasks reviewed, 15 were flagged by at least one of the two reviewers; on re-examination against the sources, 12 were confirmed to have a substantive problem, 9 of which concern scoring fairness: the scoring contract asks for more than the question requires (§3.4, Appendix H).

AuthorsOpenReview ↗

Response to Reviewer 8Tzt (3/3)

W3. Details of how VADE works

"how accurate is the LLM as a judge?"

We compared the judge with human grading directly. One reviewer with a finance background and one reviewer with a legal background independently graded 30 finance answers and 30 law answers respectively, sampled from model answers to our finance and law tasks, half of which the judge had passed and half failed. The reviewers received exactly what the judge receives: the question, the scoring contract, the reference analysis, the evidence, and the answer. They did not see any ruling of the judge. The table compares the judge's rulings with the reviewer's.

LevelItemsJudge: pass → reviewer: passJudge: fail → reviewer: failOverall agreementCohen’s κ
Scored items632573 / 58638 / 4696.7%0.77
Whole answer6024 / 3025 / 3081.7%0.63

The third and fourth columns give, of the items the judge passed, how many the reviewer also passed, and of the items the judge failed, how many the reviewer also failed. The judge's ruling is the final ruling, after the review of the items on which the two judges differ.

On scored items, the judge and the reviewers agree on 96.7% of the rulings. Agreement on whole answers is lower because an answer passes only if every item passes, so a single item on which the two differ changes the verdict. The disagreements go in both directions. This comparison is in Appendix B of the revised paper (Table 14).

W1. Clarity of the writing

We have revised §2 and Figure 1 to make the method easier to follow. The changes are as follows.

"Figure 1 is hard to understand".

Figure 1 is redrawn in the revised paper: its wording now follows exactly the text of §2, and the two phrases the reviewer mentions have been rewritten.

The pipeline is complex, so Figure 1 gives an overview of its core steps in four stages, and §2.1–2.4 explain the four stages in turn. The complete procedure is in Appendix A.1, with the full flowchart (Figure 4) and a worked example (Appendix A.3).

"reference analysis" and "scoring contract".

Both terms are defined in the second paragraph of §2. In the revised paper, we state the definition of the reference analysis more directly:

"The reference analysis is the complete answer to the question, built from the sources and hidden from the solver. It reaches its conclusion through answer atoms: the values and categories that a correct answer must contain."

"The scoring contract turns the reference analysis into scored items: every answer atom, and the fatal conditions that fail an answer outright."

The sentence on independent reproduction.

In the revised paper, we have replaced "a neutral description of each required output" with "a description of each required output that does not reveal its value", and §2 now points to where the terms of this sentence are explained in full: the "Independent reproduction" paragraph of Appendix A.1 and Figure 5 list what the reproducer can and cannot see, and Appendix A.3 gives a worked example.

What the input is, how the questions are created, and how the reference answers are generated. These are the same questions as in W3, and we have answered them in our reply to W3.

We thank the reviewer again for the comments, and we are glad to clarify any of these points further.

Comments on the paper

3 comments
AuthorsOpenReview ↗

Response to all reviewers (1/2)

We thank all reviewers for their time and their careful comments. This response collects the new results that several of our individual replies refer to. Changes in the revised paper are marked in blue and listed at the end.

1. New tasks since submission (8vDn W1, Q2; qX25 W1)

DomainTasks in the paperNow
Law372642
Energy85211
Medicine85205

Law still covers 12 practice areas and 4 task types. In energy and medicine, the task categories have grown from the 7 and 5 listed in Appendix B to 10 and 6. All new tasks went through the same full pipeline as the finance tasks, and the VADE pipeline itself was not changed. These numbers are in §3.3 of the revised paper.

2. Results on newer models (8vDn W1, Q2)

The table gives the overall pass rate of each model in each domain, i.e. the share of tasks on which every item of the scoring contract passes. This is Table 2 of the revised paper (§4.2); Table 13 in Appendix B gives the rates by task category.

ModelFinanceLawEnergyMedicine
GPT-6.1 Sol54.561.771.178.0
Grok 4.766.246.357.357.1
Claude Opus 5.564.939.446.034.1
DeepSeek V4.1 Flash27.311.829.925.9
GLM-5.328.610.629.417.1
MiMo V2.6 Flash28.69.323.214.6

3. Training outside finance (8Tzt W2; 8vDn W1, Q2)

We repeated the training experiment of §5 in law, energy, and medicine. We fine-tuned Qwen3.5-9B on solving trajectories of the tasks in each domain and evaluated it on held-out VADE tasks. We also evaluated it on two external benchmarks, LAB in law and LifeSci Diamond in medicine.

BaseEpoch4
PRPR
LawVADE-Law-Test1.627.612.744.7
LAB0.847.54.163.3
EnergyVADE-Energy-Test0.043.68.067.9
MedicineVADE-Medicine-Test0.046.914.353.8
LifeSci Diamond0.026.40.042.8

P is the pass rate, the share of tasks on which every scored item passes. R is the item pass rate, the share of an answer’s scored items that pass. In each domain, the training and test tasks share no task and no source. The results are in §5 of the revised paper (Table 4), and the setup is in Appendix E.

AuthorsOpenReview ↗

Response to all reviewers (2/2)

4. Quality of the generated tasks (8Tzt W4; 8vDn W2)

To assess what VADE’s construction process adds to task quality, we implemented a direct-generation baseline, following the prompt-based generation that Autodata uses as its control.

We took 50 delivered finance tasks, three or four from each of 13 industry groups, and gave the source files of each task, unchanged, to the same model that built it. In a single session, the model writes a question, a reference answer, and the scored items; there are no evidence fragments, no independent reproduction, and no review.

We then put the 50 directly generated tasks and the 50 VADE tasks through the same two checks: a review of the reference answer against the sources, and an independent reproduction followed by a comparison.

The scored content of the reference answer is defective in 16 of the 50 directly generated tasks, against 4 of the 50 VADE tasks (two-sided Fisher’s exact test, p = 0.005). Of the 16, the review found 12, and 4 were found only by the reproduction and comparison. Three LLM annotators who did not know which group a task came from assigned the type of each defect by majority vote (pairwise κ = 0.90–0.95).

Defect in the scored content of the reference answerDirect generationVADE
Source published after the information cutoff40
Required output missing31
Material item from the sources omitted41
Wrong basis, period, or population31
Numeric error21
Misdescription of the sources10
Tasks with at least one defect16/504/50

One directly generated task has two types of defect. The full setup, including the generation prompt, and the results are in Appendix F.3 of the revised paper (Table 24).

Relation to prior work. The table compares VADE with the most closely related task-synthesis methods on the form of the tasks they produce.

WorkSourcesSolvingAnswerScoring
FM2DSWikipediaSingle-turn QAShort answerReference-answer matching
Autodata (legal setting)Professional primary documentsSingle-turn QALong answerRubric
AGORAProfessional primary documentsMulti-turn tool useShort answerReference-answer matching
QUESTOpen webMulti-turn tool useShort or long answerRubric
VADEProfessional primary documentsMulti-turn tool useLong answerRubric

FM2DS generates multimodal multi-hop QA with concise answers, whereas VADE targets open-ended professional analysis. QUEST’s open-ended setting uses a reference report and rubrics to evaluate reports, and Autodata uses solver and verifier feedback to refine generated training data. In VADE, the reference analysis is reproduced independently from the sources before the question and the item-level scoring contract are written. Our baseline follows the control used in Autodata and is not a reproduction of these methods. The full comparison, which also covers further synthesis methods and the expert-written benchmarks (LAB, FAB, FrontierFinance, BigFinanceBench), is in Appendix G of the revised paper (Table 29).

5. Changes in the revised paper

  • Abstract: the sentence on the expert review is worded more precisely.
  • Figure 1: redrawn with larger text, a legend for the symbols, and wording that matches §2.
  • §2: a direct definition of the reference analysis, a clearer statement of what the reproducer receives, and pointers to the complete procedure in Appendix A.
  • §3.1 and Appendix A.5 (Table 12): source files by channel, and the sources of the energy and medicine runs.
  • §3.2: how often independent reproduction and the two reviews found a problem in production, and how many of the delivered tasks each review returned.
  • §3.3: the extended task sets in law, energy, and medicine.
  • §3.4 and Appendix H (Tables 30–32): agreement between the reviewers, the expert review in law, and the review by models.
  • §4.1 and Appendix B (Table 14): agreement of the judge with human grading.
  • §4.2 (Table 2) and Appendix B (Table 13): results of more recent models on the extended sets.
  • §5 (Table 4) and Appendix E: training in law, energy, and medicine, and the evaluation on LAB and LifeSci Diamond.
  • Appendix A.6: prompts of key steps.
  • Appendix F.3 (Table 24) and F.4 (Table 25): the direct-generation baseline, and a simple inspector in place of independent reproduction.
  • Appendix F.5 (Tables 26–28): what each step of the pipeline contributes.
  • Appendix G (Table 29): the form of VADE’s tasks compared with existing synthesis methods and expert-written benchmarks.
  • Minor corrections of wording and references.
AuthorsOpenReview ↗

Follow-up on our responses and revised manuscript

Dear Reviewers and Area Chair,

As the discussion period draws to a close, we would appreciate your feedback on our responses and revised manuscript.

In response to the reviews, we have:

  1. Responded to all questions and concerns raised by each reviewer.
  2. Added 516 tasks across three additional domains.
  3. Conducted three additional SFT experiments using different training datasets and added evaluation results on five additional benchmarks.
  4. Incorporated feedback from three domain experts into our responses.
  5. Incorporated our responses to all reviewer concerns into the revised manuscript.

We have not yet received any feedback on our rebuttal. Please let us know whether any concerns remain so that we can clarify them before the discussion ends.

Thank you for your time and consideration.

Best regards,
The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.