acceptodds
Under review as a conference paper at ICLR 2027

TrajFoundry: Experience-Driven Trajectory Synthesis for Multimodal Agentic Reasoning

Abstract

Training multimodal agents to reason over visual observations, executable actions, and environment feedback benefits from grounded trajectories that expose intermediate decisions and link them to verifiable outcomes. Producing such supervision is costly, but each attempt also yields reusable production experience—task constructions, solver outcomes, and artifact-validation evidence—that can support subsequent synthesis. We introduce **TrajFoundry**, an experience-driven framework that organizes these results as production experience for subsequent synthesis. These retained outcomes support different parts of subsequent production. *Compositional Task Reuse* preserves executable task families for re-instantiation and structural derivation, including compatible composition; *Validated Skill Reuse* turns valid-task solver outcomes into candidate Skills adopted only after paired Skill replay; and *Targeted Artifact Recovery* uses artifact-validation evidence to repair defective production artifacts at the affected stage while preserving unaffected work. The producer model remains fixed; improvements come from carrying forward reusable task families, validated Skills, and recovery records across production rounds. Under full production-cost accounting, **TrajFoundry** produces 591 grounded unique trajectories per 1K cost, compared with 480 for the pre-specified Skill-guided static control and 497 for the adapted open-source EnvScaler baseline. Carry-versus-reset results show gains on newly instantiated tasks alongside recovered prior work. At matched accepted tokens, the **TrajFoundry** corpus improves mean structural and environment OOD success by 6.4 points over the Skill-guided static control, and the same corpus advantage transfers to a second student backbone. Our code is available at https://anonymous.4open.science/r/TrajFoundry-B61F/.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Multimodal Agentic Reasoning

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_yCJoOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents TRAJFOUNDRY, a framework for generating multimodal agent trajectories by retaining production experience across rounds while keeping the producer model fixed. The framework consists of three mechanisms: compositional reuse of executable task families, validation reuse skills, and target artifact recovery. In experiments, TRAJFOUNDRY produces 591 grounded unique trajectories per 1k normalized cost, compared with 480 for static production and 497 for EnvScaler-adapted. The main contribution is kind of a meta policy for trajectory/experience synthesis, to get high-quality generated data in a more efficient way.

Critical strengths

The framework addresses trajectory synthesis in a more efficient way. The paper does not rely solely on generating more accepted trajectories. It evaluates downstream learning with matched training tokens and a fixed training recipe, and removes production-time Skill cards from student examples.

Critical weaknesses
  1. I highly doubt the figure is purely AI-drawn without human adjustment, especially Figure 2. The drawing lines and arrows are messy, and there are very small images inside the figure. The writing and experiment analysis part also require human revision.

  2. The improvement over other baselines is not that much, but the task itself is very costly. And the novelty of this work is limited. This is just an extension of previous trajectory synthesis methods from the trajectory level to the meta level.

Questions for authors
  1. In the appendix, the authors do not provide an example to show how the framework can efficiently improve the synthesis. Only from the text, I cannot tell it from previous works. I think an example might also help to clarify the novelty. Can the authors provide one or two examples?

  2. What are the adoption rate and negative-transfer rate among families that actually receive a new Skill?

Reviewer_TCEyOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper presents a trajectory synthesis pipeline for multimodal agents. Since generating agentic data is costly and time-consuming, the authors propose a way to reuse what each attempt leaves behind, including failed attempts: executable task constructions, solver outcomes on valid tasks, and artifact-validation evidence. They introduce TRAJFOUNDRY, which uses three mechanisms. Compositional Task Reuse keeps executable task families so they can be re-instantiated or composed into new tasks. Validated Skill Reuse turns solver outcomes into Skills that are adopted only after paired replay shows they help. Targeted Artifact Recovery repairs defective artifacts at the stage that failed. Under equal production cost, TRAJFOUNDRY yields 591 grounded unique trajectories per 1K cost, compared with 480 for the Skill-guided static control and 497 for an adapted EnvScaler. Fine-tuning a Qwen3.5-9B student (LoRA-SFT) on the generated data improves internal OOD accuracy by 6.4 points and the external average over three benchmarks (TIR-Bench, Agentic-MME, VTC-Bench) by 3.8 points over the static control.

Critical strengths
  1. The paper’s major strength is leveraging three residues and using them in subsequent data production. It distinguishes between two groups of outputs: 1) the good trajectories (accepted), which are of training standard, and 2) the leftovers (task generators, solver outcomes, validation evidence), which have production value. Agentic data is costly and time-consuming, so this is a good direction.

  2. The routing rule is interesting. Solver mistakes on valid tasks become Skill evidence, and broken artifacts go to repair. This blocks two problems: a) Fixing a correct task to match a wrong answer. b) Polluting Skills with evidence from broken tasks.

Critical weaknesses
  1. I could not find much novelty in this paper; each component is already proposed, as the authors acknowledge in Appendix G. The stated question is only whether combining the components helps. That is an empirical question, not a new method.

  2. The cost accounting in the paper is contradictory: every method has the same total of 4.27K cost units (Table 9), but raw resource use differs widely (Table 10), and the proposed method uses more of every reported resource than the skill-guided static control.

  3. The method is proposed along with in-house environments and test suites, so evaluation is very narrow, along with contradictory ablations. The ablation numbers contradict the main table; the full system shows 507 in F.4 but 591 in Table 1. Also, there are no multimodal trajectory synthesis baselines reported. The only external method, EnvScaler, is heavily modified; it was built for text-based tool use, not multimodal agents. MM-Traj / T3-Agent is left out as a baseline, and TaskCraft and DataEnvGym are not discussed, which are relevant parts of the literature.

Questions for authors
  1. Can the authors reframe the contribution around what is actually new?
  2. Add the evaluation dataset beyond the authors' own world. Can the authors add at least one external setting and report the results?
  3. Compare with existing agentic synthesis methods.
  4. Why does TrajFoundry have the lowest model-inference cost (2.38) despite consuming the most tokens (52.7M)?
Reviewer_hH76OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes TrajFoundry, a framework for synthesizing multimodal agent trajectories by reusing experience from previous production rounds. It combines compositional task reuse, validated skill reuse, and targeted artifact recovery to improve data generation efficiency and downstream agent performance.

Critical strengths
  1. Comprehensive experiments. The paper provides extensive evaluations of production efficiency, data quality, and downstream agent performance, including component ablations and multiple model backbones.
  2. Practical value. Reusing task constructions, solving experience, and validation feedback improves trajectory production efficiency and downstream generalization.
Critical weaknesses

1. Limited methodological novelty.

The three core components closely resemble existing approaches, including RST (Li et al., 2026) for task reuse, Trace2Skill (Ni et al., 2026) for skill extraction, and FACET (Shi et al., 2026) and VISA (Zeng et al., 2026) for validation-guided recovery. The main contribution appears to be integrating these existing ideas into a unified pipeline, rather than introducing a fundamentally new method.

2. Limited generalizability to real-world agent environments.

The entire trajectory production pipeline is evaluated in only two controlled synthetic environments with deterministic state transitions and programmatic verification. While downstream evaluation includes external benchmarks, it remains unclear whether the proposed framework can effectively operate in more realistic, dynamic agent environments.

3. Potential confounding factors in downstream training comparisons.

The paper matches total accepted training tokens across datasets, but the SFT loss is computed only on assistant tokens (Appendix D.7). Different datasets may therefore contain different amounts of effective supervision, making it difficult to attribute the observed performance gains entirely to data quality.

4. Insufficient evidence for the broader effectiveness of Skill reuse.

Appendix F.4 reports that only 7 out of 64 task families produced Skills that passed validation, and only 5 of these improved performance on an independent test. This raises questions about how broadly the Skill reuse mechanism contributes to the overall improvements.

5. Limited comparison with existing end-to-end systems.

The main external baseline, EnvScaler, is adapted to the authors' custom environments and evaluation pipeline. Although this enables controlled comparisons, it remains unclear whether the reported advantages would hold against existing systems in their original settings.

Questions for authors

see weaknesses

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.