- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
TrajLong is a framework that converts agent trajectories into long-context supervision for mid-training, targeting three capabilities: precise evidence retrieval, cross-evidence aggregation, and temporal state tracking. Its main contribution is a capability-oriented data construction pipeline that combines shared contexts with multiple question–solution pairs. The authors evaluate two Qwen3 models across six long-context and twelve agent benchmarks, reporting broad improvements after supervised fine-tuning. Matched-budget ablations show gains over both raw and loss-masked trajectory baselines, while additional analyses explore task-dependent relationships between long-context and agent capabilities.
Critical strengths
Practical motivation: The paper addresses how to better use agent trajectories for mid-training, connecting long-context learning with downstream agent capabilities.
Concrete data construction: The three complementary task types have explicit validation procedures. State tracking is particularly compelling, with programmatic reconstruction and safeguards against answer leakage.
Broad empirical validation: Experiments cover two model sizes and diverse long-context and agent benchmarks. Matched-budget ablations help isolate the benefits of capability-targeted compilation over raw and loss-masked trajectories.
Critical weaknesses
-
Limited model-family coverage: Both evaluated models belong to the Qwen3 family. Although they differ in size and architecture, the results do not establish whether the method generalizes to other model families.
-
Uneven gains on search and tool use: For the 14B model, improvements are less consistent than on coding tasks.
Questions for authors
-
Why are the 14B gains on search and tool-use benchmarks less consistent than on coding tasks? Which task characteristics determine when TrajLong is beneficial?
-
How was the 3:4:3 mixture ratio selected? Could alternative ratios improve search and tool-use performance while preserving coding gains?
6 comments
Response to Reviewer tVqr (Part2)
Question 2: Why choose 3:4:3, and can other ratios improve search/tool use while preserving coding gains?
The 3:4:3 ratio provides an approximately balanced allocation based on data availability. Our additional 1:1:1 experiment yields similar overall performance.
- The 3:4:3 ratio reflects data availability and balanced capability coverage
We allocate input tokens to Retrieval, MultidocQA, and State Tracking in a 3:4:3 ratio, corresponding to 3B, 4B, and 3B tokens within the 10B trajectory-data budget. This choice reflects data availability while giving the three capabilities approximately balanced training exposure.
- Overall performance remains similar with a 1:1:1 mixture
We retain the Qwen3-14B backbone, the budget of 30B background plus 10B trajectory tokens, and two epochs of downstream SFT, and assign approximately 3.33B tokens to each data type. The table reports post-SFT Avg@3 scores. See Table 22, Appendix E.8.
| Category | Benchmark | 3:4:3 (paper Avg@3) | 1:1:1 (Avg@3) | Difference |
|---|---|---|---|---|
| Search | BrowseComp | 11.77 | 11.23 | −0.54 |
| Search | WideSearch | 30.42 | 31.21 | +0.79 |
| Search | FSC-V2 | 43.53 | 43.22 | −0.31 |
| Tool use | MCP-Atlas | 44.00 | 45.00 | +1.00 |
| Tool use | SkillsBench | 11.11 | 11.11 | 0.00 |
| Coding | SWE-Verified | 54.40 | 55.29 | +0.89 |
| Coding | SWE-Pro | 36.43 | 37.06 | +0.63 |
| Long context | MRCR | 20.18 | 20.93 | +0.75 |
| Long context | GraphWalks | 32.30 | 32.10 | −0.20 |
| Long context | GSM-Infinite | 6.67 | 7.33 | +0.66 |
Across ten benchmarks, six improve, one is unchanged, and three decrease slightly, with differences ranging from −0.54 to +1.00 percentage points. The uniform mixture preserves coding benefits while improving some search and tool-use tasks.
- The mixed model covers three capabilities within the same budget
Each specialist uses all 10B trajectory tokens for one capability, whereas All divides the same budget across three capabilities, with approximately 3–4B tokens per capability. All thus achieves broader capability coverage within the same total budget, while the reduced supervision for each individual capability may also affect its score relative to a specialist.
The 3:4:3 All model also outperforms the best specialist on FSC-V2 and both SWE benchmarks. Together with the 1:1:1 results, this supports complementary benefits under a fixed budget.
Best regards,
Paper 39899 Authors
Kind Reminder for Discussion
Dear Reviewer tVqr,
As the discussion period draws to a close, we are very interested in hearing any further feedback you might have and would be pleased to clarify or address any remaining concerns. Please do not hesitate to let us know if you have additional comments or questions.
Thank you again for your time and thoughtful review.
Best regards, Paper 39899 Authors
Response to Reviewer tVqr (Part1)
Dear Reviewer tVqr,
We sincerely thank you for your careful review of our work and your valuable feedback.
We have added experiments on the NVIDIA Nemotron family and a uniform data mixture, and use task requirements and further-SFT results to explain the gains across tasks.
To assess cross-family generalization, We add validation on the NVIDIA Nemotron family: all three compilation strategies improve all four long-context and all seven search/coding benchmarks.
We initialize from
NVIDIA-Nemotron-3-Nano-30B-A3B-Baseand train Retrieval, MultidocQA, and State Tracking variants. The Baseline also undergoes mid-training and downstream SFT, using the same mid-training settings, training-token budget, and SFT recipe. Selected High-thinking / Acc results (%) are shown below; full results are in the revised PDF (Appendix E.9, Tables 23–24).All three strategies also outperform the Baseline on all seven agent benchmarks. Together with the paper's Qwen3 experiments, these results support TrajLong's effectiveness across model families.
Your question highlights the need to connect gains on agent tasks to their underlying capability requirements. Matching supervision to task requirements and allowing sufficient downstream adaptation help realize TrajLong's gains; model capacity may also affect their magnitude. The results show targeted benefits when compiled supervision matches a task's demands, while further SFT more fully brings out the potential gains from mid-training.
The updated Figure 6 in Section 5.3.3 compares specialist supervision with unmasked Raw Trajectories under matched budgets and the same SFT recipe. MultidocQA improves BrowseComp and MCP-Atlas, matching their demands for integrating dispersed evidence and handling dependencies across steps. State Tracking improves WideSearch and SkillsBench, aligning with their demands for continuously integrating information, maintaining intermediate results, and advancing the task.
Search and tool-use tasks require coordinated evidence integration, planning, and execution. At 14B scale, these complex behaviors may depend more on sufficient downstream adaptation, and model capacity may affect how much of the mid-training improvement translates into final task performance.
In Table 3, TrajLong-All improves over the same Raw baseline across search, tool-use, and coding tasks.
Increasing SFT for the same mid-trained model from two to three epochs further improves BrowseComp and SWE-Pro (Section 5.3.4, Table 5). These results suggest that mid-training raises the attainable level of downstream capability, with further SFT more fully realizing that potential as improved task performance.