- Rating
- 2(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes AssetForge, which is a pipeline used to synthesize training tasks for business automation agents. Derived from the AutomationBench, an LLM Author builds tasks from a task-specific rubric with reusable construction assets. Generated candidates pass through self-checks, a code Gate, an LLM review, and up to two repair rounds. The authors finetuned Qwen3.6-35B-A3B on 6000 accepted tasks with DeepSeek V4.1 Flash or GLM-5.3 supervision. Several experimental results demonstrate the effectiveness of the proposed pipeline.
Critical strengths
-
To build an executable business task, the author must set up the app data, the policy evidence, and the success checks. In existing pipelines, this work is repeated for every task. AssetForge makes these parts reusable, so the author can focus on the business logic. The higher first-submission acceptance rate suggests this saves real effort.
-
With the backbone, supervisor, example count, SFT recipe, and evaluator fixed, AssetForge-GLM beats the rubric-only pipeline by 10.84 points, and first-trial gain in all six domains. This is about twice what the baseline gains from quadrupling its own data. Such an achievement suggests that the construction method is effective in improving the performance of the backbone.
Critical weaknesses
-
The experiments do not support the main claims. Section 4.3 is called an ablation study, but it does not remove or vary any part of AssetForge. The two pipelines also use different rubric text, so we cannot tell whether the gain comes from the assets or from the new rubric instructions. Ablation studies of the pipeline's components are needed. The only baseline is the authors' own rubric-only pipeline, with no comparison to other synthesis methods, and the AppWorld results compare only against the base model, not the rubric-only model. In addition, the assets are built from AutomationBench's own engine and checks, so the training tasks may simply look more like the test tasks; the contamination audit does not rule this out, and it is unclear whether any design choices or checkpoints were tuned on public-600. Finally, the paper reports call counts (Table 7) but not the tokens or time spent building tasks, nor the human effort to build the 61 assets. Since each AssetForge call includes the asset catalog, fewer calls may not mean lower cost, so the claim that AssetForge reduces construction work is not supported.
-
Limited novelty and unclear presentation. The main ideas already exist: spec-first synthesis (NexForge), executable validation with repair (FACET), and reusable environment components (SWE-Next). The paper does not explain what is new beyond applying them to business tasks. The writing also makes the contribution hard to follow. In the introduction, business automation is first mentioned only in paragraph 3, so the first two paragraphs read like a general data-synthesis paper, and the method introduced in the final paragraph was already described in the previous one. In the method section, Section 3.1 discusses Figure 1 but never cites it, the figure and the text use different terms, and some figure elements are never explained (e.g., the AssetForge controller). The repair budget and the source of the rubrics are also not described. The introduction and method sections need a thorough revision.
Questions for authors
-
Which part of the gain comes from the assets, and which from the changed rubric instructions? Can you report results for (a) the rubric-only pipeline with the AssetForge rubric text but no assets, and (b) AssetForge without the Gate, the model-based review, or the repair step?
-
Were any design choices (asset selection, rubric wording, number of epochs, etc.) based on public-600 results? Can you report results on a held-out or private AutomationBench split?
-
How many Author and reviewer tokens, and how much time, did each pipeline need to produce 6,000 accepted tasks? How much human effort went into building the 61 assets?
-
What is technically new in AssetForge compared with NexForge, FACET, and SWE-Next?
-
What is the "AssetForge controller" in Figure 1?
-
What are the comparisons against other existing synthesis pipelines (e.g., FACET or NexForge) adapted to AutomationBench?
7 comments
Thank you for the response and the new experiments. The token-matched run, the scaling results, and the Rubrics row on AppWorld make the improvement over the rubric-only pipeline more convincing. However, several main concerns remain:
- Source of the gain. The AssetForge rubric also adds other instructions (Figure 5), so it is still unclear how much of the gain comes from the assets. The requested ablations of the Gate, review, and repair steps are also missing.
- Baselines. There is still no comparison with another synthesis method. If FACET or NexForge are hard to adapt, a cited method, an existing public synthetic dataset, or a simple baseline without assets would also help.
- Test-set similarity. The response confirms that engine APIs and public task characteristics shaped the assets and rubrics, which supports my concern. It also does not say whether checkpoints were selected on public-600.
- Construction cost. Token usage, time, and the human effort to build the 61 assets are still not reported.
I will keep my current score for now.
Response 2 to Reviewer o1Hs
Thank you for the follow-up. We take your concerns seriously and have promptly worked to address them.
1. Source of the gain
Gate, review and repair already exist in Rubrics and are held fixed across methods (Appendix F.2); they are not our claimed contributions. We compare construction with and without assets, using a Rubric adapted to each setting.
If your concern is how these shared steps affect synthesis costs in Rubrics and AssetForge, we take it seriously and provide the following statistics and conditional estimates:
| Method | No Gate | No review | No repair |
|---|---|---|---|
| Rubrics | Reviewer calls: 0% to +127.45% (bound) | Total construction tokens: -1.3% to -2.9% | Author tokens: -11.2% to -11.5%; new drafts: +2.04% |
| AssetForge | Reviewer calls: 0% to +36.15% (bound) | Total construction tokens: -0.37% to -0.50% | Author tokens: -3.6% to -3.9%; new drafts: +1.49% |
Estimates hold other work fixed and assume unchanged first-pass yield, with each repair costing 25% of a new draft.
For both AssetForge and the Rubrics baseline, removing any of these stages can reduce qualified yield; producing 6,000 qualified tasks would then require more candidates, offsetting direct savings.
2. Baselines
The Rubrics baseline in our paper serves as the simple baseline without assets you suggested. Directly reusing FACET or NexForge prompts without adapting their domain and infrastructure assumptions can reduce synthesis quality through incompatibility.
The same applies to AssetForge's Rubric: many extra instructions teach the Author how to use assets, such as reading descriptions and parameter contracts from catalog/, selecting assets and building initial state from them. Removing the assets but keeping these instructions leaves necessary inputs missing, which can confuse the Author and hurt task quality. Therefore, the appropriate "simple baseline without assets" is precisely the paper's Rubrics baseline: it is adapted for working without assets and tells the Author everything it must construct, including the constraints and checks (Figure 5).
Our contribution is thus reusable task-construction assets with a task-specific Rubric interface, not reusable assets alone: the assets and their adapted instructions form the design we evaluate.
3. Test-set similarity
To address possible public-600 test-instance leakage, Appendix B.2 reports zero flagged overlaps for both 6,000-example training sets across five checks covering prompt hashes, state/assertion fingerprints, record IDs, lexical overlap and semantic similarity. These checks found no evidence of public-600 test-instance contamination. During seed-based asset construction, we also blocked official benchmark and dataset-hosting URLs and benchmark-related keywords, including "AutomationBench", in the agent's search tool, using O*NET and vendor documentation as sources. This restriction was designed to keep benchmark examples out of asset construction.
By the way, We trained the paper's models for five epochs using the stated datasets and hyperparameters and evaluated only the final checkpoint, without selecting among checkpoints based on public-600 scores. The token-matched run uses its planned training endpoint.
4. Construction cost
Totals cover Author, review, repair and asset preparation for 6,000 accepted tasks; per-task values divide by 6,000. One step is one model generation call. Generation time varies with concurrency and API conditions, so we report model-generation steps.
| Method | Total input (B) | Total output (B) | Total model steps (k) | Input/accepted task (M) | Output/accepted task (k) | Steps/accepted task |
|---|---|---|---|---|---|---|
| Rubrics | 80.600 ± 0.779 | 1.200 ± 0.018 | 525.9 ± 8.4 | 13.433 ± 0.130 | 200.0 ± 3.1 | 87.65 ± 1.41 |
| AssetForge | 45.458 ± 0.087 | 0.529 ± 0.014 | 400.3 ± 1.7 | 7.576 ± 0.014 | 88.1 ± 2.3 | 66.71 ± 0.28 |
Rubrics reconstructs the task-specific application setup, world state, evidence and graders for each task; AssetForge reuses these components. The higher Rubrics step and output budgets model this repeated setup and validation. Both use the same Reviewer configuration.
61 assets recorded 7.94M input/11.64M output tokens and 383 steps: below 0.05% of combined tokens and 0.10% of steps in either 6k budget. Agents generated and checked the assets; humans reviewed plausibility.
We hope these clarifications, cost of assets construction and task synthesis, together with the token-matched, scaling and AppWorld comparisons, address your concerns. Thank you for the continued discussion; we welcome further questions and can release the corresponding models and selected training data if useful.
Response to Reviewer o1Hs
Thank you for examining the method and controls closely. We immediately prioritized the token-budget check and completed it with three scaling runs. We address the experimental design and implementation below.
1. Contribution and comparison
Both 6k arms share self-checks, Gate, review, repair, GLM supervision and training/evaluation settings. Their Rubrics describe the available resources: compose assets or build components from scratch (Appendix E).
Our insight is to reuse environment construction so Authors can focus on task quality. Assets provide recurring application records, links, rules and checks, freeing effort for business logic and diversity. Authors choose task objectives and constraints.
2. Token budget and scaling
Broader workflows and longer trajectories are outputs of AssetForge. We retained all 6,000 GLM examples and shortened SFT to 640.3M input tokens (340 updates), matching the reported Rubrics budget of approximately 640M (128M × 5 epochs). Only the horizon and its warmup/decay schedule change.
All values are percentages; domains/pass@1 use T1, avg3 uses T1–T3. The budget-controlled gain is 10.00 points.
The five-epoch subset runs are compared with historical Rubrics checkpoints:
The three pass@1 gains are 12.83/10.00/9.33 points; AssetForge-1.5k beats Rubrics-6k by 7.00 points using 75% fewer examples.
3. AppWorld and benchmark design
The AppWorld comparison now includes the Rubrics checkpoint:
On Normal, Rubrics is 5.3/8.9 TGC/SGC points below the base; on Challenge it gains 1.0/2.2 points. Limited workflow diversity may explain this uneven transfer. Both AssetForge models exceed Rubrics on both splits.
Public task characteristics and application distributions informed Rubrics. Asset seeds came from O*NET/vendor documentation, excluding benchmark sites/keywords from search; engine APIs supplied executable interfaces. Appendix B.2 reports zero flags across five overlap detectors. New subset sizes/token budgets were fixed before scoring; AppWorld tests a separate environment.
4. Prior methods
NexForge builds capability-driven resources; FACET grounds terminal artifacts in executable state; SWE-Next reuses repository environments. AssetForge makes application state, business evidence, operations and outcome checks reusable across workflows (Appendix D.4).
5. Construction work and validity
After one-time asset preparation, 6,000 accepted tasks require 6,665 candidates versus Rubrics' 8,469; repairs fall from 5,058–5,178 to 1,417–1,504 (Table 7). Appendix D.3 gives a 22-assertion example: correct execution passes, no-action fails, and checks protect out-of-scope records. Review checks request–evidence–assertion consistency.
6. Controller
The Figure 1 controller is code that controls candidate counts, starts/stops generation, runs checks/review and tracks versions, with at most two repairs. The supplementary code exposes these stages and assets.
We hope these clarifications of the method and appendices, together with the new experiments, address the main concerns. If needed, we can also release the corresponding models and part of the training data. Thank you for the close reading; we remain available for further discussion.