acceptodds
Under review as a conference paper at ICLR 2027

AssetForge: Asset-Augmented Task Synthesis for Business Automation Agents

Abstract

Training business automation agents requires executable tasks that connect a natural-language request to an application state and a verifiable end state. Traditional Rubrics requires Authors to recreate application interfaces, initial records, policy evidence, and outcome checks for each task, entangling business logic with repeated infrastructure work. We introduce \method, an asset-augmented task-construction pipeline that pairs a reusable construction-asset library with task-specific Rubrics. The assets provide application interfaces, world-state components, policy/evidence carriers, and native assertion definitions; the Rubric directs their instantiation for a particular business task. Executable checks and an additional model-based review validate each candidate, return targeted feedback, and support bounded repair before acceptance. On AutomationBench v1.0.6 public-600, fine-tuning Qwen3.6-35B-A3B on 6,000 examples yields 41.17% on the official strict \pOne metric, 4.4 the 9.33% base score (+31.83 points). With the same backbone and GLM-5.3 supervision, reaches 41.17% versus 30.33% for traditional Rubrics, a 10.84-point gain (35.7% relative) at the same accepted-example scale (6,000 versus 6,000 examples). Both DeepSeek- and GLM-supervised AssetForge models improve AppWorld completion over the base model on Test-Normal and Test-Challenge. These results establish the advantage of AssetForge's complete construction pipeline on AutomationBench public: the coupled asset–Rubric interface directs Authors toward business logic, while executable checks and model-based review validate their tasks.

Primary area:
foundation or frontier models, including LLMs
Keywords:
asset-augmented task synthesis, business automation agents, reusable construction assets, synthetic task generation, LLM agent training

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_o1HsOpenReview ↗
Rating
2(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes AssetForge, which is a pipeline used to synthesize training tasks for business automation agents. Derived from the AutomationBench, an LLM Author builds tasks from a task-specific rubric with reusable construction assets. Generated candidates pass through self-checks, a code Gate, an LLM review, and up to two repair rounds. The authors finetuned Qwen3.6-35B-A3B on 6000 accepted tasks with DeepSeek V4.1 Flash or GLM-5.3 supervision. Several experimental results demonstrate the effectiveness of the proposed pipeline.

Critical strengths
  • To build an executable business task, the author must set up the app data, the policy evidence, and the success checks. In existing pipelines, this work is repeated for every task. AssetForge makes these parts reusable, so the author can focus on the business logic. The higher first-submission acceptance rate suggests this saves real effort.

  • With the backbone, supervisor, example count, SFT recipe, and evaluator fixed, AssetForge-GLM beats the rubric-only pipeline by 10.84 points, and first-trial gain in all six domains. This is about twice what the baseline gains from quadrupling its own data. Such an achievement suggests that the construction method is effective in improving the performance of the backbone.

Critical weaknesses
  • The experiments do not support the main claims. Section 4.3 is called an ablation study, but it does not remove or vary any part of AssetForge. The two pipelines also use different rubric text, so we cannot tell whether the gain comes from the assets or from the new rubric instructions. Ablation studies of the pipeline's components are needed. The only baseline is the authors' own rubric-only pipeline, with no comparison to other synthesis methods, and the AppWorld results compare only against the base model, not the rubric-only model. In addition, the assets are built from AutomationBench's own engine and checks, so the training tasks may simply look more like the test tasks; the contamination audit does not rule this out, and it is unclear whether any design choices or checkpoints were tuned on public-600. Finally, the paper reports call counts (Table 7) but not the tokens or time spent building tasks, nor the human effort to build the 61 assets. Since each AssetForge call includes the asset catalog, fewer calls may not mean lower cost, so the claim that AssetForge reduces construction work is not supported.

  • Limited novelty and unclear presentation. The main ideas already exist: spec-first synthesis (NexForge), executable validation with repair (FACET), and reusable environment components (SWE-Next). The paper does not explain what is new beyond applying them to business tasks. The writing also makes the contribution hard to follow. In the introduction, business automation is first mentioned only in paragraph 3, so the first two paragraphs read like a general data-synthesis paper, and the method introduced in the final paragraph was already described in the previous one. In the method section, Section 3.1 discusses Figure 1 but never cites it, the figure and the text use different terms, and some figure elements are never explained (e.g., the AssetForge controller). The repair budget and the source of the rubrics are also not described. The introduction and method sections need a thorough revision.

Questions for authors
  • Which part of the gain comes from the assets, and which from the changed rubric instructions? Can you report results for (a) the rubric-only pipeline with the AssetForge rubric text but no assets, and (b) AssetForge without the Gate, the model-based review, or the repair step?

  • Were any design choices (asset selection, rubric wording, number of epochs, etc.) based on public-600 results? Can you report results on a held-out or private AutomationBench split?

  • How many Author and reviewer tokens, and how much time, did each pipeline need to produce 6,000 accepted tasks? How much human effort went into building the 61 assets?

  • What is technically new in AssetForge compared with NexForge, FACET, and SWE-Next?

  • What is the "AssetForge controller" in Figure 1?

  • What are the comparisons against other existing synthesis pipelines (e.g., FACET or NexForge) adapted to AutomationBench?

7 comments

Response to Reviewer o1Hs

Thank you for examining the method and controls closely. We immediately prioritized the token-budget check and completed it with three scaling runs. We address the experimental design and implementation below.

1. Contribution and comparison

Both 6k arms share self-checks, Gate, review, repair, GLM supervision and training/evaluation settings. Their Rubrics describe the available resources: compose assets or build components from scratch (Appendix E).

Our insight is to reuse environment construction so Authors can focus on task quality. Assets provide recurring application records, links, rules and checks, freeing effort for business logic and diversity. Authors choose task objectives and constraints.

2. Token budget and scaling

Broader workflows and longer trajectories are outputs of AssetForge. We retained all 6,000 GLM examples and shortened SFT to 640.3M input tokens (340 updates), matching the reported Rubrics budget of approximately 640M (128M × 5 epochs). Only the horizon and its warmup/decay schedule change.

ModelFin.HRMkt.Ops.SalesSupp.pass@1avg3
Qwen3.6-35B-A3B (base)127618679.339.39
Rubrics (6k)33272943341630.3330.39
AssetForge-GLM (6k, original)39513845443041.1740.22
AssetForge-GLM (6k, matched tokens)37494348392640.3339.89

All values are percentages; domains/pass@1 use T1, avg3 uses T1–T3. The budget-controlled gain is 10.00 points.

The five-epoch subset runs are compared with historical Rubrics checkpoints:

ModelFin.HRMkt.Ops.SalesSupp.pass@1avg3
Qwen3.6-35B-A3B (base)127618679.339.39
Rubrics (1.5k)27241933291524.5026.78
AssetForge-GLM (1.5k)38463740382537.3335.78
Rubrics (≈2.5k)28192841301727.1729.06
AssetForge-GLM (2.5k)35463841372637.1738.50
Rubrics (≈4k)34302640291328.6727.06
AssetForge-GLM (4k)36503938412438.0037.11
Rubrics (6k)33272943341630.3330.39
AssetForge-GLM (6k, original)39513845443041.1740.22

The three pass@1 gains are 12.83/10.00/9.33 points; AssetForge-1.5k beats Rubrics-6k by 7.00 points using 75% fewer examples.

3. AppWorld and benchmark design

The AppWorld comparison now includes the Rubrics checkpoint:

MethodLLMTest-Normal TGCTest-Normal SGCTest-Challenge TGCTest-Challenge SGC
ReActQwen3.6-35B-A3B (base)59.535.730.212.2
ReActRubrics54.226.831.214.4
ReActAssetForge-GLM64.346.434.817.3
ReActAssetForge-DS71.455.439.123.0

On Normal, Rubrics is 5.3/8.9 TGC/SGC points below the base; on Challenge it gains 1.0/2.2 points. Limited workflow diversity may explain this uneven transfer. Both AssetForge models exceed Rubrics on both splits.

Public task characteristics and application distributions informed Rubrics. Asset seeds came from O*NET/vendor documentation, excluding benchmark sites/keywords from search; engine APIs supplied executable interfaces. Appendix B.2 reports zero flags across five overlap detectors. New subset sizes/token budgets were fixed before scoring; AppWorld tests a separate environment.

4. Prior methods

NexForge builds capability-driven resources; FACET grounds terminal artifacts in executable state; SWE-Next reuses repository environments. AssetForge makes application state, business evidence, operations and outcome checks reusable across workflows (Appendix D.4).

5. Construction work and validity

After one-time asset preparation, 6,000 accepted tasks require 6,665 candidates versus Rubrics' 8,469; repairs fall from 5,058–5,178 to 1,417–1,504 (Table 7). Appendix D.3 gives a 22-assertion example: correct execution passes, no-action fails, and checks protect out-of-scope records. Review checks request–evidence–assertion consistency.

6. Controller

The Figure 1 controller is code that controls candidate counts, starts/stops generation, runs checks/review and tracks versions, with at most two repairs. The supplementary code exposes these stages and assets.

We hope these clarifications of the method and appendices, together with the new experiments, address the main concerns. If needed, we can also release the corresponding models and part of the training data. Thank you for the close reading; we remain available for further discussion.

Thank you for the response and the new experiments. The token-matched run, the scaling results, and the Rubrics row on AppWorld make the improvement over the rubric-only pipeline more convincing. However, several main concerns remain:

  • Source of the gain. The AssetForge rubric also adds other instructions (Figure 5), so it is still unclear how much of the gain comes from the assets. The requested ablations of the Gate, review, and repair steps are also missing.
  • Baselines. There is still no comparison with another synthesis method. If FACET or NexForge are hard to adapt, a cited method, an existing public synthetic dataset, or a simple baseline without assets would also help.
  • Test-set similarity. The response confirms that engine APIs and public task characteristics shaped the assets and rubrics, which supports my concern. It also does not say whether checkpoints were selected on public-600.
  • Construction cost. Token usage, time, and the human effort to build the 61 assets are still not reported.

I will keep my current score for now.

Response 2 to Reviewer o1Hs

Thank you for the follow-up. We take your concerns seriously and have promptly worked to address them.

1. Source of the gain

Gate, review and repair already exist in Rubrics and are held fixed across methods (Appendix F.2); they are not our claimed contributions. We compare construction with and without assets, using a Rubric adapted to each setting.

If your concern is how these shared steps affect synthesis costs in Rubrics and AssetForge, we take it seriously and provide the following statistics and conditional estimates:

MethodNo GateNo reviewNo repair
RubricsReviewer calls: 0% to +127.45% (bound)Total construction tokens: -1.3% to -2.9%Author tokens: -11.2% to -11.5%; new drafts: +2.04%
AssetForgeReviewer calls: 0% to +36.15% (bound)Total construction tokens: -0.37% to -0.50%Author tokens: -3.6% to -3.9%; new drafts: +1.49%

Estimates hold other work fixed and assume unchanged first-pass yield, with each repair costing 25% of a new draft.

For both AssetForge and the Rubrics baseline, removing any of these stages can reduce qualified yield; producing 6,000 qualified tasks would then require more candidates, offsetting direct savings.

2. Baselines

The Rubrics baseline in our paper serves as the simple baseline without assets you suggested. Directly reusing FACET or NexForge prompts without adapting their domain and infrastructure assumptions can reduce synthesis quality through incompatibility.

The same applies to AssetForge's Rubric: many extra instructions teach the Author how to use assets, such as reading descriptions and parameter contracts from catalog/, selecting assets and building initial state from them. Removing the assets but keeping these instructions leaves necessary inputs missing, which can confuse the Author and hurt task quality. Therefore, the appropriate "simple baseline without assets" is precisely the paper's Rubrics baseline: it is adapted for working without assets and tells the Author everything it must construct, including the constraints and checks (Figure 5).

Our contribution is thus reusable task-construction assets with a task-specific Rubric interface, not reusable assets alone: the assets and their adapted instructions form the design we evaluate.

3. Test-set similarity

To address possible public-600 test-instance leakage, Appendix B.2 reports zero flagged overlaps for both 6,000-example training sets across five checks covering prompt hashes, state/assertion fingerprints, record IDs, lexical overlap and semantic similarity. These checks found no evidence of public-600 test-instance contamination. During seed-based asset construction, we also blocked official benchmark and dataset-hosting URLs and benchmark-related keywords, including "AutomationBench", in the agent's search tool, using O*NET and vendor documentation as sources. This restriction was designed to keep benchmark examples out of asset construction.

By the way, We trained the paper's models for five epochs using the stated datasets and hyperparameters and evaluated only the final checkpoint, without selecting among checkpoints based on public-600 scores. The token-matched run uses its planned training endpoint.

4. Construction cost

Totals cover Author, review, repair and asset preparation for 6,000 accepted tasks; per-task values divide by 6,000. One step is one model generation call. Generation time varies with concurrency and API conditions, so we report model-generation steps.

MethodTotal input (B)Total output (B)Total model steps (k)Input/accepted task (M)Output/accepted task (k)Steps/accepted task
Rubrics80.600 ± 0.7791.200 ± 0.018525.9 ± 8.413.433 ± 0.130200.0 ± 3.187.65 ± 1.41
AssetForge45.458 ± 0.0870.529 ± 0.014400.3 ± 1.77.576 ± 0.01488.1 ± 2.366.71 ± 0.28

Rubrics reconstructs the task-specific application setup, world state, evidence and graders for each task; AssetForge reuses these components. The higher Rubrics step and output budgets model this repeated setup and validation. Both use the same Reviewer configuration.

61 assets recorded 7.94M input/11.64M output tokens and 383 steps: below 0.05% of combined tokens and 0.10% of steps in either 6k budget. Agents generated and checked the assets; humans reviewed plausibility.

We hope these clarifications, cost of assets construction and task synthesis, together with the token-matched, scaling and AppWorld comparisons, address your concerns. Thank you for the continued discussion; we welcome further questions and can release the corresponding models and selected training data if useful.

Reviewer_VjepOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces AssetForge, an asset-augmented pipeline for synthesizing executable business-automation tasks. It integrates reusable construction assets with rubric-guided task generation, executable validation, model-based review, and repair. On AutomationBench public-600, AssetForge achieves 41.17% pass@1, compared with 30.33% for the Rubrics baseline under the same GLM-5.3 supervision and 6,000-example training scale. It also improves task construction acceptance rates.

Critical strengths
  1. The paper presents a practical pipeline for synthesizing executable business-automation tasks with reusable resources and explicit verification.
  2. The reported improvements in task acceptance rates and downstream agent performance demonstrate the practical effectiveness of the overall pipeline.
Critical weaknesses
  1. Limited methodological novelty. The main contribution of AssetForge is to provide reusable construction assets within an existing rubric-guided task-synthesis pipeline. While this design improves task construction efficiency and downstream performance, the underlying idea of providing structured resources to facilitate task generation is relatively intuitive. The paper does not sufficiently establish a fundamentally new synthesis methodology or a substantial conceptual advance beyond this engineering integration. Consequently, the novelty and broader research significance of the work remain limited.
  2. Confounding factors in the experimental comparison. Although both methods use 6,000 training examples, the AssetForge corpus contains substantially more tokens (176M vs. 128M) and covers more applications per task (3.86 vs. 2.42). These differences make it difficult to isolate the contribution of the proposed approach from differences in training data volume and task complexity.
Questions for authors
  1. What is the fundamental methodological insight of AssetForge beyond integrating reusable construction resources into the task-synthesis pipeline?
  2. Can the authors provide an equal-token-budget comparison to better isolate the contribution of the proposed approach?
3 comments

Response to Reviewer Vjep

We appreciate your questions about novelty and training volume. They prompted an immediate equal-token comparison and three additional scaling runs, all now complete. We first clarify the construction principle, then present these results.

1. Methodological insight

We treat recurring environment construction as reusable infrastructure for task synthesis. Business-automation tasks need consistent application records, cross-application links, policies, operations and success checks. Rubrics describe the desired task but leave the Author to rebuild this infrastructure for every candidate, diverting effort from task quality.

AssetForge packages this support as reusable assets so the Author can focus on realistic business logic, diverse workflows and coherent outcomes. Appendix D.4 demonstrates reuse across Finance and HR; the supplementary code exposes the 61-asset catalog and construction stages.

The matched 6k arms share self-checks, Gate, review, repair, GLM supervision and training/evaluation settings. The asset–Rubric interface raises first-submission acceptance from 69.43% to 88.72% and pass@1 from 30.33% to 41.17%.

2. Equal training-token budget

The original comparison fixes example count; broader workflows and longer trajectories are outputs of AssetForge. We retained all 6,000 examples and shortened SFT to 640.3M input tokens (340 updates), matching the reported Rubrics budget of approximately 640M (128M × 5 epochs). Warmup and decay follow this horizon; other training/evaluation settings remain unchanged.

ModelFin.HRMkt.Ops.SalesSupp.pass@1avg3
Qwen3.6-35B-A3B (base)127618679.339.39
Rubrics (6k)33272943341630.3330.39
AssetForge-GLM (6k, original)39513845443041.1740.22
AssetForge-GLM (6k, matched tokens)37494348392640.3339.89

With training exposure controlled, pass@1 remains 10.00 points higher. Entries are percentages; domain scores and pass@1 use trial 1, and avg3 averages all three trials.

3. Additional scaling experiments

The new nested, five-epoch runs below show sample efficiency alongside historical Rubrics checkpoints:

ModelFin.HRMkt.Ops.SalesSupp.pass@1avg3
Qwen3.6-35B-A3B (base)127618679.339.39
Rubrics (1.5k)27241933291524.5026.78
AssetForge-GLM (1.5k)38463740382537.3335.78
Rubrics (≈2.5k)28192841301727.1729.06
AssetForge-GLM (2.5k)35463841372637.1738.50
Rubrics (≈4k)34302640291328.6727.06
AssetForge-GLM (4k)36503938412438.0037.11
Rubrics (6k)33272943341630.3330.39
AssetForge-GLM (6k, original)39513845443041.1740.22

At 1.5k examples, AssetForge surpasses Rubrics-6k by 7.00 points with 75% fewer examples. Across the three smaller scales, its pass@1 gains over the Rubrics points are 12.83/10.00/9.33 points.

Thank you for encouraging a sharper account of the contribution. The discussion above expands the method's reuse principle, while the new experiments test it under tighter budgets. If there is interest, we can make the relevant models and selected training data publicly available. We would be glad to discuss whether these results resolve your concerns.

The rebuttal resolved my concerns, I raised my score to 3.

Thank you for your support of our paper. Your suggestions indeed help us improve our work.

Reviewer_oyVWOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

AssetForge is a pipeline for synthesizing executable business-automation tasks. It combines a reusable library of construction assets with task-specific rubrics, then applies execution checks, model review, and bounded repair. The assets cover application interfaces, initial state, policy/evidence, and native outcome assertions. Fine-tuning on the resulting tasks improves AutomationBench performance relative to a Rubrics pipeline at the same accepted-example count, and improves AppWorld performance over the base model. The central open question is how much of the gain is attributable to the asset design, rather than the coupled changes in training supervision and task complexity.

Critical strengths
  1. The pipeline addresses a concrete and consequential bottleneck: generating tasks whose requests, initial application states, and verifiable outcomes agree. The matched-supervisor Rubrics comparison is stronger than a base-model-only comparison. At 6,000 accepted examples with the same GLM supervisor, the reported AutomationBench pass@1 rises from 30.33% to 41.17%, approximately 10.83 percentage points using the underlying counts (Table 1). This is a practically meaningful gain.

  2. The paper provides useful evidence about execution and data construction, including failure/repair cases, overlap audits, and repeated inference evaluations. It also discloses the token-budget difference and distinguishes the complete pipeline effect from a single-component effect. These details support the credibility and practical utility of the system, even though the causal contribution of the asset interface remains unresolved.

Critical weaknesses
  1. The comparison does not isolate the contribution of reusable assets. The two 6,000-example corpora contain approximately 176M versus 128M tokens, a 37.5% difference, and also differ in trajectory complexity and application/assertion composition (Appendix Table 4). The result supports an advantage of the complete pipeline at equal accepted-example count, but not an advantage at equal training-token budget or a specific mechanism attributable to assets. A stronger construction baseline with the same native engine/API documentation and validation workflow is also needed to distinguish asset organization from simply supplying better environment information.

  2. The transfer experiment lacks the relevant trained control. AppWorld compares the base model with AssetForge-trained models but omits the matched Rubrics checkpoint (Table 2). Thus it shows that AssetForge training transfers relative to the base model, but cannot establish that the asset pipeline transfers better than ordinary synthetic-task training. Together with the budget issue, this leaves the distinctive methodological contribution insufficiently identified and motivates my current weak-reject recommendation.

Questions for authors
  1. Decision-critical comparison: Does the AssetForge advantage persist when the training-token budget is matched? Please distinguish this comparison from the existing equal-accepted-example comparison.

  2. Decision-critical transfer check: How does the already trained, matched GLM Rubrics checkpoint perform on both AppWorld splits under the same evaluation settings?

  3. Clarification: How are the reusable assets constructed and selected, and what additional capability do they provide beyond giving the task author equally complete native engine/API/schema documentation with the same checks and repair budget? Please also clarify whether reported construction-cost savings include asset preparation.

4 comments

Response to Reviewer oyVW

Thank you for distinguishing the equal-example result from the token-budget and transfer questions. We prioritized the token control immediately and completed it alongside three AssetForge scaling runs. We also report the Rubrics AppWorld results below.

1. Equal training-token budget

The original comparison fixes the supervisor and example count; broader workflows and longer trajectories are outputs of our pipeline. We retained all 6,000 AssetForge-GLM examples and shortened SFT to 640.3M input tokens (340 updates), matching the reported Rubrics budget of approximately 640M (128M × 5 epochs). Warmup and decay follow the shorter horizon; the backbone, optimizer and evaluation settings remain unchanged.

ModelFin.HRMkt.Ops.SalesSupp.pass@1avg3
Qwen3.6-35B-A3B (base)127618679.339.39
Rubrics (6k)33272943341630.3330.39
AssetForge-GLM (6k, original)39513845443041.1740.22
AssetForge-GLM (6k, matched tokens)37494348392640.3339.89

At the matched budget, AssetForge retains a 10.00-point advantage. Domains and pass@1 report the first trial; avg3 is the three-trial mean (all %).

Five-epoch nested subsets are compared below with the historical Rubrics scaling points:

ModelFin.HRMkt.Ops.SalesSupp.pass@1avg3
Qwen3.6-35B-A3B (base)127618679.339.39
Rubrics (1.5k)27241933291524.5026.78
AssetForge-GLM (1.5k)38463740382537.3335.78
Rubrics (≈2.5k)28192841301727.1729.06
AssetForge-GLM (2.5k)35463841372637.1738.50
Rubrics (≈4k)34302640291328.6727.06
AssetForge-GLM (4k)36503938412438.0037.11
Rubrics (6k)33272943341630.3330.39
AssetForge-GLM (6k, original)39513845443041.1740.22

The pass@1 margins are 12.83, 10.00 and 9.33 points at 1.5k/2.5k/4k. Even 1.5k AssetForge examples outperform Rubrics-6k by 7.00 points, using 75% fewer examples.

2. Rubrics on AppWorld

For your transfer question, we add the Rubrics checkpoint to Table 2's 35B comparison:

MethodLLMTest-Normal TGCTest-Normal SGCTest-Challenge TGCTest-Challenge SGC
ReActQwen3.6-35B-A3B (base)59.535.730.212.2
ReActRubrics54.226.831.214.4
ReActAssetForge-GLM64.346.434.817.3
ReActAssetForge-DS71.455.439.123.0

Rubrics falls below the base on Normal by 5.3 TGC/8.9 SGC points, while gaining 1.0/2.2 points on Challenge. Narrower workflows may favor specialization over transfer to related environments. AssetForge outperforms Rubrics on both splits.

3. Asset construction, selection and value

We use O*NET occupational/software information and corresponding vendor documentation, such as Google Workspace, as seeds. An agent with search and code tools creates standardized application specifications, usage descriptions and categories; its search excludes the AutomationBench website and related keywords. Assets passing format validation enter the library. The Author selects and combines them according to the operations and evidence needed for each task.

Our central insight is to separate reusable environment construction from task synthesis. API documentation describes application operations; assets also provide initial records, cross-application links, readable rules and executable checks. A rubric alone leaves the Author to rebuild this support per candidate. Reusing it lets the Author focus on realistic business logic, diverse workflows and coherent objectives while choosing each task's rules and outcomes. Appendices D.3–D.4 illustrate composition and reuse; the supplementary code exposes the catalog and construction stages.

The 61 assets are prepared once and reused. Table 7 counts task generation after library construction.

We hope the token control and AppWorld comparison address your two decision-critical questions, and that the expanded account of the assets in the method makes their role clearer. If useful, we can also release the corresponding model checkpoints and a subset of the training data. Thank you for prompting these checks; we welcome your assessment and follow-up questions.

Additional Response to Reviewer oyVW

Thank you again for your thoughtful questions. We would like to further clarify your third question: what our method provides beyond complete API and schema documentation, and how asset preparation enters the construction cost.

Assets and their adapted Rubric form our contribution. With documentation alone, the Author still has to repeatedly construct the task-specific records, relationships, policy evidence and outcome checks, determine how to connect them, and validate their consistency. It has guidance on what to construct, but no ready-made assets or instructions for using those assets. AssetForge supplies both reusable construction components and an adapted Rubric explaining how to select, configure and compose them. This shifts recurring infrastructure work into reusable preparation, allowing the Author to focus on business logic, workflow diversity and task quality.

For example, API documentation explains how to read messages and update spreadsheet rows. The assets and their adapted Rubric help the Author assemble the records, discoverable policies and executable checks needed for a coherent workflow. The Author still determines the particular scenario, business rules, exceptions and intended outcomes.

The assets and their adapted Rubric are inseparable parts of the interface we evaluate: the Rubric explains how to use the supplied components, and the components make those instructions actionable. Removing the assets while retaining instructions that depend on them creates an incomplete configuration. The corresponding no-assets baseline is our Rubrics pipeline, whose adapted instructions explain how to construct the required elements and satisfy their constraints. Gate, review and repair remain shared across methods, as stated in Appendix F.2. Our comparison therefore evaluates construction with and without assets, using the corresponding Rubric interface in each setting.

Construction cost. The 61 assets are prepared once and reused. The statistics below include asset preparation, Author, review and repair for 6,000 accepted tasks per method. One-time preparation accounts for approximately 1.1% of AssetForge's total tokens. Per-task values divide totals by 6,000; one step is one model generation call.

MethodTotal input (B)Total output (B)Total model steps (k)Input/task (M)Output/task (k)Steps/task
Rubrics80.600 ± 0.7791.200 ± 0.018525.9 ± 8.413.433 ± 0.130200.0 ± 3.187.65 ± 1.41
AssetForge45.458 ± 0.0870.529 ± 0.014400.3 ± 1.77.576 ± 0.01488.1 ± 2.366.71 ± 0.28

± denotes half the estimated range, not a standard deviation. API interruptions, retries and incomplete repair/review records may introduce additional accounting uncertainty not captured by this range. Output includes reasoning tokens.

Rubrics requires the Author to reconstruct and validate the supporting infrastructure for each task, whereas AssetForge provides reusable components and their usage guidance. This repeated work is the mechanism represented by the higher Rubrics token and step budgets in our statistics.

We hope this explanation, together with the token-matched, scaling and AppWorld comparisons in our first response, clarifies the role of the integrated Assets–Rubric interface. Thank you for the continued discussion.

Dear Reviewer oyVW,

We sincerely thank you for your constructive review and for raising these concerns, which have greatly helped us improve our paper. Based on the reviewers' concerns, we have uploaded a revised version of the paper, incorporating the experiments—including the equal training-token budget, additional scaling experiments, construction cost, and AppWorld comparison—into the appropriate sections, and updating the relevant descriptions and figures for greater clarity.

Could you please read our response and briefly share whether your main concerns have been addressed? With the author response period ending on November 18 (AoE), we would greatly appreciate your follow-up before then. Thank you for your time and effort.

Dear Reviewer oyVW,

Thank you again for your review and constructive suggestions. With the discussion period drawing to a close, we wanted to follow up briefly in case you have not yet had a chance to read our earlier replies. The revised manuscript now includes both comparisons you requested. With matched training-token budgets, AssetForge-GLM achieves 40.33% strict pass@1 versus 30.33% for Rubrics (Table 3). On AppWorld, it outperforms the trained Rubrics baseline on both task and scenario goal completion across both test splits (Table 2). We would be grateful if you could revisit your assessment in light of our rebuttal and the revised manuscript.

Comments on the paper

1 comment

Global response to the Area Chair and all reviewers

Dear Area Chair and Reviewers,

Thank you for the constructive reviews. Your questions helped us identify missing comparisons and clarify the paper's contribution. As the rebuttal period draws to a close, we summarize how we addressed these concerns and how your feedback shaped the revised manuscript.

Main concerns raised

  • The original comparison matched example counts, but AssetForge produced longer trajectories. Would its advantage persist at the same training-token budget, at smaller training-set sizes, and on AppWorld against a trained Rubrics control?
  • Given the shared validation workflow, what do the reusable assets and their adapted Rubric contribute beyond API documentation? How does AssetForge compare with an external synthesis method or dataset?
  • How closely do the training tasks resemble AutomationBench public-600, and did its scores influence checkpoint selection? Do the construction-cost estimates include asset preparation?

Our responses and results

  • Training budget and scaling. To match training-token exposure, we retained all 6,000 AssetForge-GLM examples and shortened training. At approximately 640M input tokens per model, AssetForge-GLM achieved 40.33% strict pass@1 versus 30.33% for Rubrics. We then trained on subsets of 1,500, 2,500, and 4,000 examples to see whether the advantage persisted with less data. AssetForge-GLM outperformed the corresponding Rubrics checkpoints at all four scales. Even with 1,500 examples, it reached 37.33%, seven points above Rubrics trained on 6,000.
  • Transfer to AppWorld. To test transfer to a different environment, we evaluated the trained Rubrics checkpoint under the same ReAct settings. AssetForge-GLM improved task and scenario goal completion over Rubrics by 10.1 and 19.6 points on Test-Normal and by 3.6 and 2.9 points on Test-Challenge.
  • Synthesis baseline and attribution. Both pipelines share self-checks, the code Gate, model-based review, and bounded repair. With Rubrics, the Author builds each task's state, evidence, and checks from scratch, whereas AssetForge supplies reusable components and a Rubric adapted to them. The comparison therefore evaluates the assets and adapted Rubric together. For a direct comparison at the same 6,000-example scale, we have begun adapting an external synthesis method to business tasks.
  • External 35B models. While that comparison is underway, we broadened the evaluation of independently trained models. Occamy-1.0 is relevant because it also trains on synthesized business workflows. Nex-N2.5-mini offers a possible indirect link to NexForge. Its model card traces its multimodal foundations to Nex-N2, and the NexForge paper documents the use of its synthetic data in training that earlier family. Both AssetForge variants achieved 41.17% strict pass@1, compared with published scores of 32.30% for Nex-N2.5-mini and 27.60% for Occamy. We also ran Apodex-1.1-mini and Sage-A1.5 on all 600 tasks for three trials each. Their strict pass@1 scores were 16.17% and 8.17%, respectively. Both AssetForge variants outperformed them in all six domains on the first trial.
  • Benchmark similarity and construction cost. Public task characteristics informed rubric design, and we evaluated the final checkpoints at prespecified endpoints without selecting them by public-600 scores. To examine distributional similarity, we compared both AssetForge corpora with public-600 on application combinations, assertion counts, and assertion types. On all three measures, training-to-public distances exceeded the median distance between two halves of public-600. Our cost estimates include one-time preparation of all 61 assets. Producing 6,000 accepted tasks required an estimated 45.46B input tokens and 400,300 model calls for AssetForge, versus 80.60B input tokens and 525,900 calls for Rubrics.

Changes to the revised manuscript

  • Figures and tables. Figure 1 and Table 1 add Occamy, Apodex, and Sage alongside the existing Nex-N2.5-mini reference. Table 19 gives the per-domain, three-trial results for Apodex and Sage. Table 2 adds the trained Rubrics control on AppWorld, Table 3 reports the matched-token experiment, and Figure 4a shows the four training-set sizes.
  • Method and evaluation details. We revised the contribution statement to identify the shared validation stages, explained the controller's role and two-round repair limit, and documented checkpoint selection. Section 4.3 is now titled Additional analyses to reflect its scope. Appendix C.1 describes asset preparation and selection, and Appendix B.5 reports construction costs and their uncertainty.

If useful and there is an opportunity, we could release our model weights and selected training data at publication.

We thank all reviewers for their time and constructive discussion. We hope the Area Chair and reviewers will consider these results and clarifications when reassessing the paper.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.