- Rating
- 4
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The authors present Web2LongTraj, a pipeline to synthesize long horizon agentic trajectories from web corpora. The automatic data synthesis pipeline created 1.15 million agentic trajectories, and further use AnchorStretch and ArtifactStitch to expand the length and volume of the dataset. The resulting dataset showed gains in a range of agentic and memory benchmarks, compared to the original mid-training data of Olmo-3-7B and Olmo-3-32B.
Critical strengths
- Fair comparison and solid ablation: The dataset is compared against the original Olmo mid-training data under equal or additive token budget and aligned training parameters. The ablation experiments covered the design choices extensively.
- Dual focus of agentic and long context: The dataset improves both agentic (tool calling and multi step interaction) and long term memory benchmarks through strategic concatenation of short agentic episodes.
Critical weaknesses
- The control is non-agentic: the control mixture (Dolma 3 Dolmino and Longmino) contains no tool-use or trajectory data, so Tables 1 measures agentic data versus none in mid-training, instead of an already agentic focused mixture, such as Nemotron-SFT-Agentic-v2 or MidTool. The web-based corpora is therefore claimed to be a complementary source of agentic and long horizon data.
- Expansion through recombination: the 10.6B long context tokens in the dataset were constructed from 1.90B-token pool through concatenation. AnchorStretch and ArtifactStitch introduced in later training stages contain seen token substrings in earlier stages and training steps.
Questions for authors
- Could you provide evaluation results of your models on Terminal Bench? It's a long term agentic benchmark that may benefit from diverse knowledge in web corpora.
- Artifact Stitch is introduced in the last training stage and is probably meant to be of the highest quality and length among the dataset. Why there are no llm-powered filters or refinements on quality or coherence of the multi task trajectory?
4 comments
Response to Reviewer gQ1p (2)
As the table shows, TOUCAN + Open-SWE-Traces is a strong public agentic baseline (improving over Olmo 3 long data by +3.4 and +8.7 points under Instruct and Think SFT, respectively, and outperforming single-component ablations in some settings). However, the data synthesized by our full Web2LongTraj pipeline achieves the highest overall average score under both post-training recipes (35.6 vs. 34.2 under Instruct SFT and 32.8 vs. 31.6 under Think SFT, outperforming this baseline on 15 out of 16 benchmark comparisons across the two settings), without requiring manually engineered task environments, MCP servers, or Docker sandboxes. We have added this controlled comparison and analysis to Table 1 and Section 3.2 of the revised paper.
For Weakness 2
Weakness 2: Expansion through recombination: the 10.6B long context tokens in the dataset were constructed from 1.90B-token pool through concatenation. AnchorStretch and ArtifactStitch introduced in later training stages contain seen token substrings in earlier stages and training steps.
Response:
- What is expanded here is the supervision structure conditioned on long contexts. A short action-observation episode and the exact same episode resumed after many related intervening tasks constitute two different prediction problems: the latter requires the model to resume an earlier task after completing various distracting intermediate tasks without forgetting the information and details of the earlier task. When the same continuation appears after a large number of distracting trajectories rather than inside a short trajectory, the conditional prediction problem that the model must solve is fundamentally different.
The controlled design in Table 1 of the paper directly tests and rules out the hypothesis that "gains come from repeated exposure to the same substrings" while strictly aligning the number of substring repetitions:
Controlled Recipe in Table 1 (Fixed 60B budget, identical 1.90B trajectory pool & Stage 2 overlap) | Instruct: 8-Benchmark Mean | Think: 8-Benchmark Mean |
|---|---|---|
SeedTraj (Stage 3 uses only official Olmo long documents) | 32.7 | 28.1 |
↳ w/o dependency split (recombined in equal volume & order as AnchorStretch, without splitting dependencies) | 32.4 | 27.7 |
SeedTraj + AnchorStretch | 34.8 | 30.2 |
↳ unrelated sessions (recombined in equal volume from the same pool as ArtifactStitch, randomly concatenated sessions) | 32.7 | 27.1 |
SeedTraj + ArtifactStitch | 34.2 | 29.2 |
- Dependency-splitting control (
w/o dependency splitvs.AnchorStretch): The results show that simply packing already-seen trajectories into a long context without stretching dependencies (w/o dependency split) yields no gain in the 8-benchmark average; only when the exact same set of trajectories is placed between the anchor and continuation to form genuine long-range dependencies (AnchorStretch) does the 8-benchmark average improve overSeedTrajby+2.1 / +2.0points (and outperformw/o dependency splitby+2.4 / +2.4points). unrelated sessionsvs.ArtifactStitch: As shown in the last two rows of the table above,unrelated sessionssamples an equal volume of tokens from the sameSeedTrajpool and randomly concatenates them into multi-task sessions, yielding no gain overSeedTrajin the 8-benchmark average either, whereasArtifactStitchwith shared-artifact graph walks outperformsunrelated sessionsby+1.4(Instruct) and+2.1(Think) points. These controls consistently confirm that downstream gains are driven by cross-trajectory dependency and context organization structures rather than repeated substring exposure.
Response to Reviewer gQ1p (3)
For Question 1
Question 1: Could you provide evaluation results of your models on Terminal Bench? It's a long term agentic benchmark that may benefit from diverse knowledge in web corpora.
Response:
1. Supplementary Evaluation Results on Terminal-Bench 2.0: Fine-Grained Verification Confirms Significant Cross-Domain Long-Horizon Gains from Web2LongTraj
The latest versions of Terminal-Bench (v4.0/v4.1) specifically removed problems solvable by small and medium-sized models, calibrating single-task difficulty to 2 to 12 hours for a human expert (even 120B-scale models achieve a full-task pass rate of only 0.3% on the official leaderboard). Therefore, to provide a more discriminative comparison for open-weight models at the Olmo-3 scale, we evaluated all five main recipes of Olmo-3-32B (Instruct SFT) from Table 2 of our paper on Terminal-Bench 2.0:
| Data Recipe (Olmo-3-32B, 20B Stage 2 + 20B Stage 3) | Full-Task Pass@1 (N = 89) | Per-Task Partial Score Mean ([0,1] x 100) | Official Unit-Test Pass Rate (N = 305) | Fraction of Tasks with Effective Progress (>= 1 test passed) |
|---|---|---|---|---|
| Olmo 3 long data (control baseline) | 0.0% (0/89) | 15.9 | 19.7% (60/305) | 39.3% (35/89) |
| SeedTraj | 0.0% (0/89) | 18.7 | 21.6% (66/305) | 47.2% (42/89) |
| SeedTraj + AnchorStretch | 1.1% (1/89) | 21.5 | 24.3% (74/305) | 47.2% (42/89) |
| SeedTraj + ArtifactStitch | 0.0% (0/89) | 21.9 | 26.2% (80/305) | 44.9% (40/89) |
| Web2LongTraj (Full Method) | 1.1% (1/89) | 22.0 | 26.6% (81/305) | 49.4% (44/89) |
The experimental results show that Web2LongTraj consistently performs best across all fine-grained execution metrics: Web2LongTraj increases the overall pass rate across all 305 official unit tests from 19.7% to 26.6% and improves the mean per-task partial score from 15.9 to 22.0; on the 55 medium-difficulty tasks that make up the bulk of the benchmark, the mean partial score improves by +9.1 points (14.7 -> 23.8). The number of tasks achieving substantive sub-goal breakthroughs expands from 35/89 to 44/89, and the number of tasks completing more than half of their unit tests increases from 13/89 (14.6%) to 23/89 (25.8%).
2. Why Binary Full-Task Pass@1 Suffers from a Floor Effect, and the Complementary Design of Our Benchmark Suite
The default full-task Pass@1 metric of Terminal-Bench uses a strict "all-or-nothing" criterion (a task receives 1 point only if 100% of its pytest assertions pass, whereas passing 4/5 still receives 0 points), and its official Terminus-2 agent disables native tool-calling interfaces, requiring the model to output strictly escaped JSON keystroke sequences in plain text to drive an interactive tmux terminal. Because our study is designed to strictly isolate the contribution of different mid-training data recipes (fixing a 60B-token mid-training budget at 7B and a 40B-token budget at 32B, followed by general SFT without targeted fine-tuning on any downstream benchmark or specific terminal scaffold), the binary full-task Pass@1 metric is compressed into a 0.0% - 1.1% floor range and cannot sensitively reflect differences between mid-training recipes on its own.
This is precisely why our main experiments systematically combine the following execution-feedback and coding benchmarks:
- Real command execution and feedback in a Docker Shell (
InterCode-Bash): Placing the model in a real Docker container shell to iteratively manipulate the filesystem based on execution feedback under a fair matched budget,Web2LongTrajimproves Olmo-3-7B from75.8to80.6(+4.8) under Instruct SFT and from60.9to78.1(+17.2) under Think SFT; on Olmo-3-32B, it similarly improves from81.1to84.8(+3.7) under Instruct SFT and from70.8to80.0(+9.2) under Think SFT. - Multi-step interaction over shared environment state and code transfer (-bench, MBPP$+$, and BigCodeBench): On the dynamic environment -bench, the 7B model improves by
+3.6/+9.5and the 32B model improves by+1.9/+11.4; on the coding benchmarks MBPP$+$ and BigCodeBench, the 7B model improves by+6.9/+11.9and+3.0/+4.4, while the 32B model improves by+3.2/+6.9and+0.5/+1.0(Instruct / Think SFT), validating the effectiveness of the web-mined trajectories and the two long-horizon expansion operators inWeb2LongTrajon real command execution and software engineering tasks.
Response to Reviewer gQ1p (4)
For Question 2
Question 2: Artifact Stitch is introduced in the last training stage and is probably meant to be of the highest quality and length among the dataset. Why there are no llm-powered filters or refinements on quality or coherence of the multi task trajectory?
Response:
(1) We would like to clarify that the primary source of ultra-long sequences in our dataset is AnchorStretch (10.00B tokens across 64K-256K windows, with mean dependency distances spanning 52K-227K tokens), which undergoes multi-round independent LLM auditing. In contrast, ArtifactStitch totals 0.11B tokens (0.18% of the 60B total budget), with an average session length of 8.2K tokens, and replaces the official short-document quota in the 256K sub-stage (as noted in Appendix D of our paper, short-context data is mixed in even during the 256K long-context training stage, consistent with the Olmo 3 long-context recipe, to preserve the model's short-context capabilities as the context window grows).
(2) All member trajectories in ArtifactStitch have already passed the three strict LLM gates (Gates 1, 2, and 3, which cumulatively filter out 99.64% of raw web pages). We deliberately avoid generative LLM refinement in order to keep the source records 100.0% byte-for-byte authentic (with 100.0% valid tool calls under the merged tool declaration) and avoid fabricating unverified cross-post intermediate states.
(3) Our independent blind coherence audit of 2,000 randomly sampled ArtifactStitch sessions using gpt-6.1-sol shows that sessions generated by the deterministic artifact-enrichment graph already achieve an 89.8% edge-level shared working-context correctness rate (environment compatibility), with 88.7% having zero cross-task hard conflicts, without any LLM rewriting. Therefore, no additional generative LLM filtering or refinement is needed to achieve both multi-task contextual coherence and byte-for-byte source fidelity.
We sincerely thank you once again for your dedicated review and strong endorsement of our work!
Response to Reviewer gQ1p (1)
We sincerely thank you for your dedicated review and strong endorsement of our paper (
4: Clear acceptance), for appreciating our fair comparisons and solid ablations under equal or additive token budgets and aligned training parameters, and for recognizing the dual contribution of our method in simultaneously improving both agentic (tool calling and multi-step interaction) and long-term memory benchmarks.For Weakness 1
Response:
Nemotron-SFT-Agentic-v2andMidTool: the former consists of short-horizon conversational data designed for the SFT stage (data from the same family is already used as a post-training component in our Think SFT mixture), whileMidTool, which we discuss and compare in Section 4 (Related Work), has a maximum mid-training context window of only8K. Neither can directly provide long-horizon mid-training trajectories spanning a256Kcontext window at a10B+token scale, and there is currently no off-the-shelf public long-horizon agent trajectory corpus at a comparable scale and256Kcontext length for direct comparison.AnchorStretchandArtifactStitchare source-agnostic expansion operators that can also be directly applied to stretch and stitch trajectories generated from executable environments, which is another important dimension of how our approach complements environment-first synthesis.MidToolpaper also uses an equal-budget Olmo Dolmino mixture (20Btokens) as its mid-training control baseline, which matches our design of using official Olmo 3 data as the control (and our paper validates this at a larger60Bbudget).OpenHands,SWE-agent,OpenCode, andmini-swe-agent, averaging31.5Ktokens per trajectory). We replacedSeedTrajwith an equal 1.90B-token volume of TOUCAN trajectories in Stage 2 (8Kwindow), and in Stage 3 we used ourAnchorStretchandArtifactStitchmethods to pack a1:1token mixture of TOUCAN and Open-SWE-Traces into the64K / 128K / 256Klong-context windows. We then ran experiments using the exact same training (mid-training and post-training) and evaluation setups as Table 1 of the paper. The results are shown below: