acceptodds
Under review as a conference paper at ICLR 2027

TrajLong: Co-Designing Agentic and Long-Context Supervision for Mid-Training

17 views

Abstract

LLM agents for coding, search, and workplace tasks increasingly rely on long-context capabilities to effectively aggregate and reason over extended interaction histories. Recent work has incorporated agent trajectories into mid-training stage, drawing on their naturally long and interaction-rich structure. Yet how to organize the information within these trajectories into effective mid-training supervision remains underexplored. In this work, we investigate the relationship between long-context and agent atomic capabilities and introduce TrajLong, a novel framework that compiles trajectories into long-context training tasks with dense supervision, targeting three representative atomic capabilities: evidence grounding, cross-evidence aggregation, and temporal state maintenance. We mid-train Qwen3-14B-Base and Qwen3-30B-A3B-Base with data compiled by TrajLong, followed by supervised fine-tuning. Experiments on 6 long-context and 12 agent benchmarks demonstrate broad performance gains, with controlled ablations showing improvements over raw and masked trajectory baselines. Capability-level analyses further reveal task-dependent associations between long-context and agent atomic capabilities. These findings suggest that shared capability demands provide a principled basis for designing mid-training data to strengthen downstream agent capabilities.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Long Context, Mid-Training, LLM Agents, Data Synthesis

Reviews

Among:
Shape:
1234this paper, Top 32%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.50 over 4 reviews, above 68% of the others.
Reviewer_tVqrOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

TrajLong is a framework that converts agent trajectories into long-context supervision for mid-training, targeting three capabilities: precise evidence retrieval, cross-evidence aggregation, and temporal state tracking. Its main contribution is a capability-oriented data construction pipeline that combines shared contexts with multiple question–solution pairs. The authors evaluate two Qwen3 models across six long-context and twelve agent benchmarks, reporting broad improvements after supervised fine-tuning. Matched-budget ablations show gains over both raw and loss-masked trajectory baselines, while additional analyses explore task-dependent relationships between long-context and agent capabilities.

Critical strengths

Practical motivation: The paper addresses how to better use agent trajectories for mid-training, connecting long-context learning with downstream agent capabilities.

Concrete data construction: The three complementary task types have explicit validation procedures. State tracking is particularly compelling, with programmatic reconstruction and safeguards against answer leakage.

Broad empirical validation: Experiments cover two model sizes and diverse long-context and agent benchmarks. Matched-budget ablations help isolate the benefits of capability-targeted compilation over raw and loss-masked trajectories.

Critical weaknesses
  1. Limited model-family coverage: Both evaluated models belong to the Qwen3 family. Although they differ in size and architecture, the results do not establish whether the method generalizes to other model families.

  2. Uneven gains on search and tool use: For the 14B model, improvements are less consistent than on coding tasks.

Questions for authors
  1. Why are the 14B gains on search and tool-use benchmarks less consistent than on coding tasks? Which task characteristics determine when TrajLong is beneficial?

  2. How was the 3:4:3 mixture ratio selected? Could alternative ratios improve search and tool-use performance while preserving coding gains?

6 comments
AuthorsOpenReview ↗

Response to Reviewer tVqr (Part1)

Dear Reviewer tVqr,

We sincerely thank you for your careful review of our work and your valuable feedback.

We have added experiments on the NVIDIA Nemotron family and a uniform data mixture, and use task requirements and further-SFT results to explain the gains across tasks.

Weakness 1: Does TrajLong generalize to model families beyond Qwen3?

To assess cross-family generalization, We add validation on the NVIDIA Nemotron family: all three compilation strategies improve all four long-context and all seven search/coding benchmarks.

We initialize from NVIDIA-Nemotron-3-Nano-30B-A3B-Base and train Retrieval, MultidocQA, and State Tracking variants. The Baseline also undergoes mid-training and downstream SFT, using the same mid-training settings, training-token budget, and SFT recipe. Selected High-thinking / Acc results (%) are shown below; full results are in the revised PDF (Appendix E.9, Tables 23–24).

ModelsLongBench V2MRCRGraphWalksWideSearchSWE-VerifiedSWE-Pro
Nemotron-3-Nano-30B-A3B Baseline22.7820.6529.9423.8128.7310.14
TrajLong (Retrieval)23.57 (+0.79)26.14 (+5.49)31.22 (+1.28)29.78 (+5.97)49.87 (+21.14)32.10 (+21.96)
TrajLong (MultidocQA)24.40 (+1.62)21.07 (+0.42)33.11 (+3.17)32.32 (+8.51)47.12 (+18.39)32.84 (+22.70)
TrajLong (State Tracking)25.13 (+2.35)21.82 (+1.17)31.79 (+1.85)29.30 (+5.49)46.87 (+18.14)34.20 (+24.06)

All three strategies also outperform the Baseline on all seven agent benchmarks. Together with the paper's Qwen3 experiments, these results support TrajLong's effectiveness across model families.

Weakness 2 / Question 1: Why are 14B search/tool-use gains less consistent than coding gains, and which task characteristics determine effectiveness?

Your question highlights the need to connect gains on agent tasks to their underlying capability requirements. Matching supervision to task requirements and allowing sufficient downstream adaptation help realize TrajLong's gains; model capacity may also affect their magnitude. The results show targeted benefits when compiled supervision matches a task's demands, while further SFT more fully brings out the potential gains from mid-training.

  1. Gains align with task demands for evidence integration and state maintenance

The updated Figure 6 in Section 5.3.3 compares specialist supervision with unmasked Raw Trajectories under matched budgets and the same SFT recipe. MultidocQA improves BrowseComp and MCP-Atlas, matching their demands for integrating dispersed evidence and handling dependencies across steps. State Tracking improves WideSearch and SkillsBench, aligning with their demands for continuously integrating information, maintaining intermediate results, and advancing the task.

  1. The 14B model may depend more on sufficient downstream adaptation

Search and tool-use tasks require coordinated evidence integration, planning, and execution. At 14B scale, these complex behaviors may depend more on sufficient downstream adaptation, and model capacity may affect how much of the mid-training improvement translates into final task performance.

  1. The gains from mid-training emerge more fully with further SFT

In Table 3, TrajLong-All improves over the same Raw baseline across search, tool-use, and coding tasks.

Increasing SFT for the same mid-trained model from two to three epochs further improves BrowseComp and SWE-Pro (Section 5.3.4, Table 5). These results suggest that mid-training raises the attainable level of downstream capability, with further SFT more fully realizing that potential as improved task performance.

AuthorsOpenReview ↗

Response to Reviewer tVqr (Part2)

Question 2: Why choose 3:4:3, and can other ratios improve search/tool use while preserving coding gains?

The 3:4:3 ratio provides an approximately balanced allocation based on data availability. Our additional 1:1:1 experiment yields similar overall performance.

  1. The 3:4:3 ratio reflects data availability and balanced capability coverage

We allocate input tokens to Retrieval, MultidocQA, and State Tracking in a 3:4:3 ratio, corresponding to 3B, 4B, and 3B tokens within the 10B trajectory-data budget. This choice reflects data availability while giving the three capabilities approximately balanced training exposure.

  1. Overall performance remains similar with a 1:1:1 mixture

We retain the Qwen3-14B backbone, the budget of 30B background plus 10B trajectory tokens, and two epochs of downstream SFT, and assign approximately 3.33B tokens to each data type. The table reports post-SFT Avg@3 scores. See Table 22, Appendix E.8.

CategoryBenchmark3:4:3 (paper Avg@3)1:1:1 (Avg@3)Difference
SearchBrowseComp11.7711.23−0.54
SearchWideSearch30.4231.21+0.79
SearchFSC-V243.5343.22−0.31
Tool useMCP-Atlas44.0045.00+1.00
Tool useSkillsBench11.1111.110.00
CodingSWE-Verified54.4055.29+0.89
CodingSWE-Pro36.4337.06+0.63
Long contextMRCR20.1820.93+0.75
Long contextGraphWalks32.3032.10−0.20
Long contextGSM-Infinite6.677.33+0.66

Across ten benchmarks, six improve, one is unchanged, and three decrease slightly, with differences ranging from −0.54 to +1.00 percentage points. The uniform mixture preserves coding benefits while improving some search and tool-use tasks.

  1. The mixed model covers three capabilities within the same budget

Each specialist uses all 10B trajectory tokens for one capability, whereas All divides the same budget across three capabilities, with approximately 3–4B tokens per capability. All thus achieves broader capability coverage within the same total budget, while the reduced supervision for each individual capability may also affect its score relative to a specialist.

The 3:4:3 All model also outperforms the best specialist on FSC-V2 and both SWE benchmarks. Together with the 1:1:1 results, this supports complementary benefits under a fixed budget.

Best regards,

Paper 39899 Authors

AuthorsOpenReview ↗

Kind Reminder for Discussion

Dear Reviewer tVqr,

As the discussion period draws to a close, we are very interested in hearing any further feedback you might have and would be pleased to clarify or address any remaining concerns. Please do not hesitate to let us know if you have additional comments or questions.

Thank you again for your time and thoughtful review.

Best regards, Paper 39899 Authors

Reviewer_Szq6OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces TRAJLONG, a framework that transforms agent interaction trajectories into structured long-context supervision for LLM mid-training. The approach targets three atomic capabilities: evidence grounding, cross-evidence aggregation, and temporal state maintenance. The authors construct training examples from coding and search agent trajectories and perform mid-training on Qwen3, followed by downstream supervised fine-tuning. Experiments across long-context and agent benchmarks demonstrate improvements over direct SFT and raw-trajectory mid-training. The paper further investigates the relationships between long-context and agentic capabilities through learning dynamics and correlation analyses.

Critical strengths
  1. The paper addresses an important question of how to better utilize agent trajectories during mid-training, beyond conventional next-token prediction or loss masking. The connection between long-context reasoning and agentic capabilities provides a useful perspective for training-data design.
  2. The three compilation strategies are intuitive and complementary. In particular, reconstructing file histories and generating state-tracking questions offers a meaningful way to provide supervision for reasoning about evolving environments and interaction histories.
  3. The authors evaluate two model architectures across a broad collection of long-context, search, tool-use, and coding benchmarks. The matched-token comparisons against raw and masked trajectories are valuable. The additional analyses of different capabilities and training stages provide useful insights.
Critical weaknesses
  1. Limited methodological novelty relative to existing trajectory-compilation approaches. Prior work such as ACC already explores converting agent trajectories into long-context QA supervision. TRAJLONG extends this direction through capability-specific task construction, but the individual tasks resemble existing retrieval, multi-document QA, and state-tracking objectives.
  2. Some improvements over the strongest baselines are relatively small. While the gains over SFT-only are substantial, the improvements over unmasked raw-trajectory mid-training are pretty small. For example, Table 3 reports only +0.8 percentage points on SWE-bench Verified and +0.5 points on SWE-bench Pro. Additionally, the combined model does not consistently outperform the specialized variants.
  3. The relationship between long-context and agentic capabilities is not conclusively established. The correlation analyses rely on very few model variants. Consequently, the reported correlations may be sensitive to individual observations and do not establish that improving a particular long-context capability causally improves the corresponding agentic capability. Other factors, such as overall training quality or domain-specific exposure, may also contribute.
  4. The observed gains may arise partly from introducing synthetic QA examples, teacher-generated reasoning traces, or increased effective supervision density, rather than the particular decomposition into three atomic capabilities. Further ablations against generic trajectory-based QA construction or alternative compilation strategies would strengthen the central claim. The Retrieval construction also closely resembles the MRCR evaluation format, including ordinal selection, verbatim reproduction, and random prefixes, raising questions about task-format transfer versus broader generalization.
  5. Although the appendix provides detailed filtering and validation procedures, quantitative statistics about generated QA quality, acceptance rates, evidence-dependency complexity, and potential teacher errors are limited. The reliance on private agent trajectories and multiple teacher models also makes independent reproduction challenging.
Questions for authors
  1. What is the benefit of capability-targeted compilation compared with generic synthetic QA supervision? Could the authors provide a matched-budget comparison with an ACC-style mid-training baseline or a generic trajectory-to-QA approach using the same source trajectories and teacher models?
  2. How sensitive is performance to the mixture of the three capabilities? The 3:4:3 ratio appears empirically chosen, and the combined model sometimes underperforms individual variants. Have alternative mixture ratios been evaluated, and is there evidence of interference between capabilities?
  3. How is the quality and difficulty of the generated supervision assessed? Could the authors provide statistics on QA acceptance rates, the number and separation of supporting evidence items, the depth of state dependencies, and independent correctness evaluations?
  4. How should the reported capability correlations be interpreted given the small number of observations? Can the authors provide statistical uncertainty, additional independently trained variants, or more direct experiments that distinguish shared underlying capabilities from correlations driven by general model improvements?
6 comments
AuthorsOpenReview ↗

Response to Reviewer Szq6 (Part1)

Dear Reviewer Szq6,

We sincerely thank you for your careful review and valuable feedback.

Weakness 1: What is TrajLong's methodological advance over ACC and existing retrieval, QA, and state-tracking objectives?

Your question highlights the importance of explaining what capability-targeted compilation adds beyond existing trajectory-to-QA methods. TrajLong expands supervision by mining new capability-targeted tasks from existing interaction logs. The main additions relative to ACC are as follows.

  1. ACC (Su et al., 2026) retains the original task question, uses the final answer as the target, and aggregates tool responses and environmental observations across turns into a long context. Its task coverage therefore depends on the available original task QA pairs. TrajLong mines tool observations, cross-source evidence, and file histories to construct multiple new questions per trajectory (Sections 3.3–3.4).

  2. Retrieval identifies unique targets among natural distractors; MultidocQA combines sources with independent solution checks; State Tracking reconstructs file states and provenance through operation replay (Appendices D.2–D.4).

The matched-budget ACC and same-source Generic QA comparisons also support these designs; see Weakness 4 / Question 1 below, Table 3 (Section 5.2), and Table 18 (Appendix E.1).

Weakness 4 / Question 1: Do gains reflect capability targeting or generic QA, teacher reasoning, and supervision density? Please compare with ACC or Generic QA using the same sources, teachers, and budget.

We appreciate the suggestion to separate the effects of reasoning, supervision density, and task construction.

  1. QA retains most long-context gains without explicit teacher reasoning

From the same MultidocQA source pool, we remove reasoning while retaining context, question, and validated answer. Both versions use 10B task-data tokens and the same SFT recipe. wCoT/woCoT denote with/without reasoning; macro-averages cover six long-context and twelve agent tasks. Scores are Avg@3 (%), and differences are in pp (Section 5.2, Table 4).

BenchmarkRaw (unmasked)MultidocQA wCoTMultidocQA woCoTWithout CoT − RawWithout CoT − With CoT
LongBench V231.0732.6032.34+1.27−0.26
GraphWalks31.3833.4633.10+1.72−0.36
CorpusQA61.8063.4263.12+1.32−0.30
Long-context macro-average34.5135.6635.52+1.01−0.14
BrowseComp11.8213.9313.47+1.65−0.46
MCP-Atlas43.8046.7346.20+2.40−0.53
SWE-Verified53.6052.1352.53−1.07+0.40
Agent macro-average27.7627.1627.29−0.47+0.13

QA-only beats Raw on all six long-context tasks and retains approximately 88% of the gain. The GraphWalks gain has a paired 95% interval of [0.90, 2.50].

  1. TrajLong outperforms raw trajectories when both use full-token supervision

Unmasked Combined Trajectories and TrajLong both compute loss at all valid prediction positions, using Qwen3-14B, the same 40B-token budget, and downstream SFT. TrajLong-All improves all seven tasks (Section 5.2, Table 3).

  1. TrajLong outperforms ACC across task categories

We upsample ACC's approximately 1B-token corpus to 10B tokens. Both methods use Qwen3-14B, 30B background tokens, two SFT epochs, and DeepSeek-generated reasoning. Post-SFT Avg@3 results are below (Appendix E.1, Table 18).

Model / training setupOolong-SynthGraphWalksCorpusQABrowseCompNL2RepoSWE-Pro
ACC-14B-All54.1230.1657.657.336.3921.47
TRAJLONG-14B-All58.5032.3063.8311.7715.0336.43

TrajLong improves all six tasks. Its advantages on NL2Repo and SWE-Pro are also larger in mid-training than in direct SFT (Section 5.3.4, Table 5).

  1. Capability targeting improves on Generic QA from the same trajectories

Generic QA prompts the teacher with each trajectory and its associated task to generate ordinary QA, without explicitly targeting Retrieval, MultidocQA, or State Tracking. Both methods share the source trajectories, teacher, reasoning format, 40B-token budget, downstream SFT, and evaluation settings.

TrajLong-All improves all eight benchmarks. The table reports post-SFT Avg@3 and pp differences (Section 5.2, Table 3).

BenchmarkA. Generic QAB. Capability-targeted QA (TrajLong-All)B − A (pp)
MRCR18.3220.18+1.86
GraphWalks30.1132.30+2.19
Oolong-Synth55.6758.50+2.83
CorpusQA62.1063.83+1.73
FSC-V240.5243.53+3.01
SkillsBench9.5411.11+1.57
SWE-Verified52.8954.40+1.51
SWE-Pro35.4136.43+1.02
AuthorsOpenReview ↗

Response to Reviewer Szq6 (Part2)

Weakness 2: How meaningful are the minor gains on SWE-Verified/Pro over unmasked raw trajectories?

We agree that gains should be assessed against the strong unmasked baseline.

  1. SWE gains over the strong Raw baseline are statistically significant

We compare TrajLong-All with unmasked Combined Trajectories (Raw). Avg@3 is the mean of three complete evaluation-run means; SD measures variation across those three means.

BenchmarkRaw: mean ± SDTrajLong: mean ± SD
SWE-Verified53.60 ± 1.7354.40 ± 1.27
SWE-Pro35.93 ± 1.5736.43 ± 0.58
SWE-Bench-Multilingual38.22 ± 2.3639.22 ± 0.51

Model means and 95% confidence intervals:

ModelSWE-VerifiedSWE-ProSWE-Bench-Multilingual
TrajLong-All54.40 [50.67, 58.13]36.43 [33.47, 39.44]39.22 [34.56, 44.00]
Combined Trajectories53.60 [49.40, 57.80]35.93 [32.59, 39.19]38.22 [33.56, 42.78]

TrajLong has higher means and lower SDs on all three benchmarks. Additional paired item-level analyses confirm significant gains on SWE-Verified and SWE-Pro.

  1. Nemotron experiments extend the gains to another model family

The 14B model's capacity may limit gains on complex agent tasks. We also train from NVIDIA-Nemotron-3-Nano-30B-A3B-Base, with the same mid-training settings, token budget, and downstream SFT recipe for the Baseline and TrajLong variants.

Selected High-thinking / Acc results (%) follow; full results are in the revised PDF (Appendix E.9, Tables 23–24).

ModelsLongBench V2MRCRGraphWalksWideSearchSWE-VerifiedSWE-Pro
Nemotron-3-Nano-30B-A3B Baseline22.7820.6529.9423.8128.7310.14
TrajLong (Retrieval)23.57 (+0.79)26.14 (+5.49)31.22 (+1.28)29.78 (+5.97)49.87 (+21.14)32.10 (+21.96)
TrajLong (MultidocQA)24.40 (+1.62)21.07 (+0.42)33.11 (+3.17)32.32 (+8.51)47.12 (+18.39)32.84 (+22.70)
TrajLong (State Tracking)25.13 (+2.35)21.82 (+1.17)31.79 (+1.85)29.30 (+5.49)46.87 (+18.14)34.20 (+24.06)

All three strategies improve all four long-context and seven agent benchmarks. These results further demonstrate TrajLong's effectiveness in another model family.

Weakness 5 / Question 3: Please report QA acceptance and independent correctness, teacher errors, supporting-evidence counts and distances, and state-dependency depths.

We report quality and difficulty statistics in Appendix A.5, Tables 10–11.

  1. Correctness is independently assessed on all retained QA examples

We use DeepSeek-v4-pro to generate answers and GPT-OSS-120B as an independent judge to evaluate the final answers for all retained QA examples after filtering. Correctness is the proportion judged correct; retention statistics are measured at the trajectory-record level.

StatisticMultidocQARetrievalState Tracking
Source trajectory screening76.8%68.2%64.6%
Compilation pipeline74.5%67.8%61.2%
End-to-end (records)57.2%46.2%39.5%
QA produced per retained record5.152.523.04
Teacher answer correctness (all post-filter QA samples)78.6%74.2%75.9%
  1. Questions combine multiple supporting items across long distances

MultidocQA uses 3.35 supporting items on average; 26.3% involve four distinct sources and 4.3% at least five. Trajectories average 22.7 candidate items and 43,019 tokens. Non-supporting evidence remains as distractors, and source groups are shuffled.

First-to-last supporting-evidence spans are approximately 20,000–30,000 tokens; adjacent-block gaps average several thousand to 10,000 tokens.

For Retrieval, target-containing files undergo seven operations on average. Search responses average 6.7 candidate passages, with 7–11 search groups per trajectory.

  1. Typical examples require multiple state updates

We count effective target-file updates between the state anchor and dependency boundary, and interleaved changes to other files.

Snapshot (median / P95)Provenance (median / P95)
Effective state updates on target file3 / 174 / 22
Mutations to other files4 / 325 / 35

The medians characterize typical multi-step examples; P95 captures longer reconstruction processes.

Weakness 5: How can others reproduce the pipeline and results given private trajectories and multiple teacher models?

We agree that accessible materials are essential for reproduction.

  1. We will open-source the construction code, complete prompts, training/evaluation configurations, releasable examples, and usage instructions.

  2. The appendix documents preprocessing (A.1, Table 6), construction and teacher roles (D.2–D.4), training/evaluation settings (D.1, Tables 14–15; B–C), and examples (F, Tables 27–29). DeepSeek generates solutions; MultidocQA uses GPT-OSS-120B for grouping and Gemini for question generation.

AuthorsOpenReview ↗

Response to Reviewer Szq6 (Part3)

Weakness 3 / Question 4: How robust are the correlations, and do they imply causality? Please add uncertainty estimates, independently trained variants, and controls for general training or domain effects.

  1. Additional configurations and resampling broaden the analysis

We expanded from five to ten training configurations and increased repeated training runs. ICL uses eight final mid-training checkpoints (36 task pairs); SFT covers ten configurations, six long-context and nine agent tasks.

Two-sided exact permutation tests provide p-values. Item-level bootstrap measures sensitivity to benchmark sampling; leave-one-variant-out analysis checks dependence on individual variants.

StageBenchmark pairSpearman ρTwo-sided pItem-level bootstrap percentile range (2.5th–97.5th)Leave-one-variant-out ρ range
ICLRULER-QA1–Code Summary0.9940.0000992[0.731, 0.976][0.991, 1.000]
SFTGSM-Infinite–WideSearch0.8060.007215[0.603, 0.879][0.733, 0.883]
ICLBrowseCompLong–Planning Preference0.9380.001786[0.586, 0.892][0.925, 0.943]
ICLRULER-Multivalue–Tool Selection0.7560.046429[0.493, 0.869][0.632, 0.934]
SFTGSM-Infinite–SkillsBench0.7030.027421[0.556, 0.845][0.591, 0.790]
SFTLongBench V2–FSC-V20.6690.039769[0.519, 0.816][0.577, 0.778]

Both ranges are positive for all six pairs (Appendix E.10, Table 25).

  1. Matched-budget training improves corresponding long-context and agent tasks

Comparisons fix Qwen3-14B, 30B background plus 10B trajectory tokens, optimization, and downstream SFT. Below is one task per strategy; full results are in Appendix E.10, Table 26. Scores and gains use the paper's BoN@3 results; 95% intervals use the supplementary item-level evaluations.

Compilation variantBenchmarkBoN@3: Raw → Compiled (%)Gain (pp)Item-level reanalysis: 95% interval (pp)
RetrievalMRCR30.93 → 33.29+2.36[1.13, 3.91]
MultidocQAGraphWalks37.00 → 39.61+2.61[1.53, 3.69]
State TrackingWideSearch45.43 → 50.56+5.13[2.10, 8.27]

These training comparisons complement the associations with gains under controlled training settings.

Weakness 4: Given Retrieval's similarity to MRCR, do gains transfer beyond that task format?

Your question highlights transfer across task formats. Retrieval benefits extend beyond MRCR both before and after SFT.

  1. Before SFT, Retrieval improves over unmasked Raw on RULER-CWE and GraphWalks under matched settings (Appendix E.5, Figure 10). These require word-frequency extraction and graph queries, unlike MRCR's prefix-conditioned reproduction of an earlier response.

  2. With the same downstream SFT recipe, gains extend to aggregation, search, tool-use, and browser tasks (Section 5.3.3, Figure 6), demonstrating transfer across input and output formats.

Weakness 2 / Question 2: How sensitive are results to the mixture ratio, and could capability interference explain the combined model trailing specialist variants?

  1. The 3:4:3 ratio reflects data availability and balanced capability coverage

We allocate input tokens to Retrieval, MultidocQA, and State Tracking in a 3:4:3 ratio, corresponding to 3B, 4B, and 3B tokens within the 10B trajectory-data budget. This choice reflects data availability while giving the three capabilities approximately balanced training exposure.

  1. Uniform mixing gives similar overall results

At 1:1:1, each type receives approximately 3.33B tokens, fixing Qwen3-14B, 30B background plus 10B trajectory tokens, and two SFT epochs. Results are post-SFT Avg@3 (Appendix E.8, Table 22).

CategoryBenchmark3:4:3 (paper Avg@3)1:1:1 (Avg@3)Difference
SearchBrowseComp11.7711.23−0.54
SearchWideSearch30.4231.21+0.79
SearchFSC-V243.5343.22−0.31
Tool useMCP-Atlas44.0045.00+1.00
Tool useSkillsBench11.1111.110.00
CodingSWE-Verified54.4055.29+0.89
CodingSWE-Pro36.4337.06+0.63
Long contextMRCR20.1820.93+0.75
Long contextGraphWalks32.3032.10−0.20
Long contextGSM-Infinite6.677.33+0.66

Six of ten tasks improve, one is unchanged, and three decrease slightly; differences range from −0.54 to +1.00 pp. Gains persist under both approximately balanced mixtures.

  1. Specialists receive more training data per capability

Each specialist uses all 10B tokens for one capability; All allocates approximately 3–4B to each. This reduced exposure can contribute to gaps relative to specialists. All still outperforms the best specialist on FSC-V2 and both SWE benchmarks, supporting complementary benefits within a fixed budget.

Best regards,

Paper 39899 Authors

Reviewer_hgsmOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper introduces a new technique for constructing long context training data from agent trajectories for midtraining. The approach pulls evidence from agent trajectories to construct context, questions are generated based on the context, and answers based on the question and context are generated by an LLM and checked against reference answers. These triplets are serialized by shared context into long context data that is used for midtraining. There are four different variants of TrajLong, 3 that target capabilities and one that combines the 3 capabilities. Midtraining is unmasked on two backbones: Qwen3-14B-Base and Qwen3-30B-A3B-Base. Midtraining on the 4 variants of TrajLong produces 4 variants of the 14B model, the 30B model is only tested with the combined variant.

They perform experiments with a number of long-context benchmarks, agent benchmarks, and ICL benchmarks. They conclude that midtraining on TrajLong data improves performance on ICL benchmarks and that it improves performance on agent and long context benchmarks after SFT. They also analyze correlations between long context and agent performance.

Critical strengths

Their explanation of the approach and its motivation is good. They evaluate on a broad range of long context and agent benchmarks.

The construction of the data itself is careful, for the three target capabilities they ensure the included questions are answerable given the context and ensure the answers are consistent/agree with the reference answers.

Critical weaknesses

It is unclear that the improvements seen in the benchmarks are not just a result of distillation from the teacher model. It appears as though the TrajLong data contains some reasoning from the teacher model (Table 18 and D.3 Solution Generation). Critically, the raw trajectories do not contain the teacher-generated reasoning that are in TrajLong so the existing ablation cannot isolate the organization of TrajLong from the effect of including teacher reasoning.

Based on the ablation study, the benefit of TrajLong over just midtraining on raw trajectories may not be significant. The gains reported in Table 3 are small and confidence intervals are not reported.

Figure 1 right is mislabeled, the “Agent Trajectories” appear to match the results for “Combined Trajectories (with Mask)” presented in Table 3. A fairer comparison would be with “Combined Trajectories” since the other midtraining variants in that figure are unmasked. At the very least it should be labeled in the figure. This is important because the masked version is worse than the unmasked version in a majority of the benchmarks in Table 3.

The paper presents its improvement over “SFT only” in Figure 1 and Tables 1-3 as a primary result. This supports the usefulness of midtraining in general but is not convincing evidence for TrajLong. The secondary result, the ablation study, is more directly related to TrajLong and their argument.

The analysis in section 5.3.2 relies on spearman correlations that are not significant. With 36 correlations and only five training variants none of the correlations can reach statistical significance when accounting for multiple testing. This also affects the correlations reported in figures 2, 5, 11, and 12.

Update 11/17 I’d like to sincerely thank the authors for the revision. The corrected Figure 1 and the teacher-reasoning ablation are good changes that address two of my concerns.

Several critical concerns are not sufficiently addressed. The evidence they provide does not show an improvement from TrajLong over just midtraining with unmasked raw trajectories. Table 4 reports a 12-task agent Avg@3 of 27.76 for unmasked raw trajectories. Averaging the 12 agent scores reported in Table 2 for TrajLong-14B-All gives 27.84, a negligible increase over midtraining on raw trajectories.

Similarly, table 4 reports a six-task long-context avg@3 of 34.51 for unmasked raw trajectories. Averaging the 6 long-context scores in tables 1 and 20 gives 35.56, a ~1pp improvement.

In a comment, the authors provide item level intervals for the Table 3 comparisons. TrajLong-All’s intervals overlap with those of combined trajectories on every benchmark. They state that paired analyses show significant gains on SWE-verified and SWE-pro, but these are not reported. The difference intervals they do report in Table 26 only cover a subset of the variants and benchmarks, and not TrajLong-All.

The benchmarks added were placed in the appendix rather than Table 3. The main text still states that TrajLong-All outperforms raw trajectories on “all seven benchmarks”. On the added benchmarks it is below raw trajectories on both BrowseComp and SealQA, and is below/level with SFT-only on BrowseComp, SealQA and GSM-inf.

The expanded correlation analysis does not account for multiple testing.

I find the idea interesting, and I think a mixed or negative result could be a meaningful contribution. I do not think the paper’s framing is accurate. It presents TrajLong as an improvement over raw trajectories but the evidence, particularly on agent benchmarks, is not conclusive. I will keep my scores.

Questions for authors
  1. Is TrajLong still beneficial if teacher generated reasoning is removed?
  2. How does TrajLong compare to ACC in midtraining? It appears they were only compared in SFT.
  3. What are the confidence intervals for the benchmark results reported in the ablation study?
  4. Are there any additional variants that can be included in section 5.3.2? A bootstrap CI from sampling benchmark items would also help address the significance of these correlations.
  5. Why are GSM-inf, LongBenchQA, BC, SealQA reported in Figure 6 but not Table 3? They should be included.
4 comments
AuthorsOpenReview ↗

Response to Reviewer hgsm (Part1)

Dear Reviewer hgsm,

We sincerely thank you for your careful review and suggestions.

Weakness 1 / Question 1: Do gains persist without teacher-generated reasoning, separating trajectory organization from teacher distillation?

We agree that this ablation helps isolate the role of teacher reasoning.

  1. The no-reasoning ablation matches the task-data budget and SFT recipe

From the same MultidocQA source pool, we remove reasoning but retain contexts, questions, and validated final answers, matching the 10B-token task-data budget and downstream SFT (Section 5.2, Table 4).

BenchmarkSFT-onlyRawMultidocQA wCoTMultidocQA woCoTWithout CoT − RawWithout CoT − With CoT
LongBench V229.3431.0732.6032.34+1.27−0.26
MRCR16.6119.8119.9320.05+0.24+0.12
GraphWalks24.1231.3833.4633.10+1.72−0.36
Oolong-Synth53.1756.6757.3357.17+0.50−0.16
GSM-Inf7.446.337.227.33+1.00+0.11
CorpusQA55.9361.8063.4263.12+1.32−0.30
Long-context mean31.1034.5135.6635.52+1.01−0.14
BC12.5311.8213.9313.47+1.65−0.46
BC-ZH18.2720.0718.3418.69−1.38+0.35
WideSearch24.9629.0430.4830.15+1.11−0.33
FSC-V235.0940.8742.6642.25+1.38−0.41
SealQA24.3225.9324.6825.10−0.83+0.42
MCP-Atlas42.4043.8046.7346.20+2.40−0.53
SkillsBench6.489.266.947.87−1.39+0.93
WebArena27.3228.6229.8829.60+0.98−0.28
TB 2.17.1216.489.3610.49−5.99+1.13
NL2Repo6.2317.6615.7115.90−1.76+0.19
SWE-Verified42.1353.6052.1352.53−1.07+0.40
SWE-Pro24.2135.9335.1135.25−0.68+0.14
Agent mean22.5927.7627.1627.29−0.47+0.13

The QA-only long-context macro-average is 35.52, above 34.51 for unmasked Raw and close to 35.66 for the reasoning version, retaining approximately 88% of the original gain. The agent average exceeds SFT-only and wCoT.

  1. Capability-targeted construction adds gains with the same teacher and trajectories

Generic QA directly prompts the teacher to generate questions based on each trajectory and corresponding task. Both methods share source trajectories, teacher, reasoning format, 40B mid-training tokens, SFT, and evaluation settings. Below are post-SFT Avg@3 (%) results (Table 3 in Section 5.2).

BenchmarkA. Generic QAB. Capability-targeted QA (TrajLong-All)B − A (pp)
MRCR18.3220.18+1.86
GraphWalks30.1132.30+2.19
Oolong-Synth55.6758.50+2.83
CorpusQA62.1063.83+1.73
FSC-V240.5243.53+3.01
SkillsBench9.5411.11+1.57
SWE-Verified52.8954.40+1.51
SWE-Pro35.4136.43+1.02

TrajLong-All gains 1.02–3.01 pp on all eight tasks, supporting capability-targeted construction.

Weakness 2 / Question 3: Are the small Table 3 gains over raw trajectories significant? Please provide confidence intervals.

From supplementary evaluation records, 10,000 paired bootstrap resamples use identical sampled items across models, sampling with replacement at the original sample size. We recompute means and paired differences; tables show Avg@3 means and 95% CIs.

ModelOolong-SynthGraphWalksCorpusQA
TrajLong-All58.50 [53.17, 63.83]32.35 [29.86, 34.95]63.93 [59.68, 68.09]
SFT-only53.00 [47.50, 58.50]24.12 [21.63, 26.72]55.93 [51.57, 60.18]
Search Trajectories57.33 [51.67, 63.00]31.62 [29.09, 34.24]61.80 [57.45, 66.06]
Code Trajectories55.50 [50.00, 60.83]32.77 [30.27, 35.35]61.80 [57.45, 66.16]
Combined Trajectories56.67 [51.00, 62.17]31.38 [28.82, 34.02]61.80 [57.65, 66.06]
Combined Trajectories (with Mask)54.83 [49.33, 60.50]31.24 [28.72, 33.90]60.99 [56.64, 65.35]
ModelFSC-V2SkillsBenchSWE-VerifiedSWE-Pro
TrajLong-All43.53[39.44, 47.57]11.11 [6.02, 18.06]54.40 [50.67, 58.13]36.43 [33.47, 39.44]
SFT-only35.09[31.20, 39.03]6.48 [2.31, 11.57]42.13 [38.47, 45.80]24.21 [21.71, 26.81]
Search Trajectories40.05[36.06, 44.04]9.72 [5.56, 14.81]50.27 [46.53, 54.07]31.37 [28.55, 34.20]
Code Trajectories39.49[35.60, 43.48]8.80 [4.63, 13.89]52.60 [48.73, 56.47]34.38 [31.51, 37.35]
Combined Trajectories40.87 [36.93, 44.81]9.26 [4.63, 14.35]53.60 [49.40, 57.80]35.93 [32.59, 39.19]
Combined Trajectories (with Mask)42.20[38.11, 46.24]6.48 [3.24, 10.65]53.20 [49.40, 57.00]33.88 [30.96, 36.84]

Under the same mid-training budget and SFT recipe, TrajLong-All improves all seven means over unmasked Raw. Additional paired item-level analyses further confirm that TrajLong-All's gains over unmasked Raw on SWE-Verified and SWE-Pro are statistically significant.

Weakness 3: Figure 1 appears to use a masked baseline. Please include the unmasked baseline or clarify the label.

Thank you for identifying this issue. We have redrawn Figure 1 to include both masked and unmasked Combined Trajectories; its caption specifies matched mid-training token budgets and downstream SFT.

AuthorsOpenReview ↗

Response to Reviewer hgsm (Part2)

Weakness 4 / Question 2: Please compare TrajLong with other mid-training methods, especially ACC, to establish its specific contribution.

We agree that comparisons with other mid-training approaches are central to assessing TrajLong’s contribution. Matched-budget comparisons with raw trajectories and ACC, together with direct-SFT experiments, support the value of TrajLong's data construction itself.

  1. TrajLong outperforms raw trajectories under the same mid-training budget

Table 3 fixes Qwen3-14B, 40B mid-training tokens (30B background plus 10B trajectory-related tokens), and the downstream SFT recipe. TrajLong-All outperforms both masked and unmasked combined-trajectory baselines on all seven evaluated benchmarks, demonstrating the additional value of capability-targeted compilation over direct trajectory training.

The same-source Generic QA comparison under ‘Weakness 1 / Question 1’ also keeps the teacher model, mid-training budget, and downstream SFT fixed. TrajLong-All improves all eight benchmarks by 1.02–3.01 percentage points, further supporting capability-targeted construction under the same training procedure. The seven shared tasks appear in Table 3, Section 5.2.

  1. TrajLong data also improves direct SFT

Table 5 in Section 5.3.4 holds the SFT-token budget fixed and incorporates TrajLong data directly into SFT, improving all six evaluated benchmarks over the original SFT baseline. With the same mixing ratio and training budget, SWE-Pro also exceeds the result using ACC data by 3.89 percentage points. This further supports the supervision value of the compiled data itself.

  1. Matched-budget ACC comparisons show a larger advantage from mid-training

To compare the two data-construction methods at the same scale, we repeatedly sample the approximately 1B-token ACC corpus to supply 10B task-data tokens. ACC and TrajLong both use Qwen3-14B, the same 30B-token background corpus, and a total mid-training budget of 40B input tokens. Both then receive two epochs of the same downstream SFT recipe, and reasoning in both datasets is generated by DeepSeek. This matches the backbone, background data, training-token budget, and downstream adaptation.

The table reports Avg@3 (%), combining the mid-training-plus-SFT comparison (Appendix E.1, Table 18) with direct-SFT results (Section 5.3.4, Table 5).

Model / training setupOolong-SynthGraphWalksCorpusQABrowseCompNL2RepoSWE-Pro
Qwen3-14B (original)52.1727.0256.431.875.594.29
ACC-14B-All54.1230.1657.657.336.3921.47
TRAJLONG-14B-All58.5032.3063.8311.7715.0336.43
ACC-SFT-Only56.0030.6257.8910.167.0027.22
TRAJLONG-SFT-only56.8328.0459.2713.569.4031.11

Under this matched setup, TrajLong-All outperforms ACC-14B-All on all six benchmarks, spanning long-context, search, and coding tasks. The gains are achieved with the same mid-training token budget and downstream SFT.

The coding results also highlight the value of the mid-training stage. TrajLong's advantage over ACC on NL2Repo increases from 2.40 pp under direct SFT to 8.64 pp under mid-training plus SFT; on SWE-Pro, it increases from 3.89 to 14.96 pp. ACC and TrajLong use matched budgets and recipes within each setting. These larger coding gains support using capability-targeted supervision during mid-training to build transferable capabilities that subsequent SFT can more fully express.

AuthorsOpenReview ↗

Response to Reviewer hgsm (Part3)

Weakness 5 / Question 4: With few variants, are the correlations in Figures 2, 5, 11, and 12 significant after multiple-testing correction? Please add variants and item-level bootstrap intervals.

We sincerely thank the reviewer for the concrete suggestions on additional variants, resampling, and multiple-testing analysis. The expanded analysis and matched-budget results provide complementary evidence linking long-context capabilities and agent performance.

  1. Additional variants and item-level resampling support the positive associations

We expanded the analysis from five training configurations to ten and also increased the number of repeated training runs. Relative to Figure 5, the added configurations are Search Trajectories, Code Trajectories, Raw Masked, Code Masked, and Search Masked. ICL uses the eight final mid-training checkpoints with paired evaluations, covering all 36 task pairs. SFT covers ten configurations across six long-context and nine agent tasks.

We obtain p-values from two-sided exact permutation tests. Item-level bootstrap assesses sensitivity to evaluation samples, and leave-one-variant-out analysis checks whether an association depends on a single variant.

StageBenchmark pairSpearman ρTwo-sided pItem-level bootstrap percentile range (2.5th–97.5th)Leave-one-variant-out ρ range
ICLRULER-QA1–Code Summary0.9940.0000992[0.731, 0.976][0.991, 1.000]
SFTGSM-Infinite–WideSearch0.8060.007215[0.603, 0.879][0.733, 0.883]
ICLBrowseCompLong–Planning Preference0.9380.001786[0.586, 0.892][0.925, 0.943]
ICLRULER-Multivalue–Tool Selection0.7560.046429[0.493, 0.869][0.632, 0.934]
SFTGSM-Infinite–SkillsBench0.7030.027421[0.556, 0.845][0.591, 0.790]
SFTLongBench V2–FSC-V20.6690.039769[0.519, 0.816][0.577, 0.778]

For all six associations, both bootstrap percentile and leave-one-variant-out ranges are positive (Table 25, Appendix E.10).

  1. Matched-budget training improves corresponding long-context and agent tasks

The training comparisons fix Qwen3-14B, 30B background tokens, 10B trajectory-data tokens, optimization settings, and the downstream SFT recipe. In Table 3, TrajLong-All outperforms both masked and unmasked combined-trajectory baselines on all seven Avg@3 benchmarks.

The table below reports the gains of each compiled variant over unmasked Raw. Scores and gains use the paper's BoN@3 results, while the 95% intervals for the gains come from the additional item-level evaluations. See Table 26, Appendix E.10.

Compilation variantBenchmarkBoN@3: Raw → Compiled (%)Gain (pp)Item-level reanalysis: 95% interval (pp)
RetrievalMRCR30.93 → 33.29+2.36[1.13, 3.91]
RetrievalWebArena40.71 → 42.84+2.13[1.31, 3.98]
MultidocQAGraphWalks37.00 → 39.61+2.61[1.53, 3.69]
MultidocQAMCP-Atlas75.41 → 77.46+2.05[1.03, 4.81]
State TrackingGSM-Infinite17.67 → 26.00+8.33[3.00, 13.67]
State TrackingWideSearch45.43 → 50.56+5.13[2.10, 8.27]
State TrackingSkillsBench19.44 → 29.17+9.73[3.12, 18.75]

For example, State Tracking improves GSM-Infinite, WideSearch, and SkillsBench together, consistent with the positive associations above. The correlation analysis and training results jointly support organizing trajectory supervision around retrieval, cross-evidence aggregation, and state tracking.

Question 5: Please add GSM-inf, LongBenchQA, BC, and SealQA to the Table 3 ablation.

We agree that including these benchmarks will make the ablation results more complete. We have added GSM-Inf, LongBench V2, BrowseComp, and SealQA to Appendix E.1, Table 19, supplementing Table 3. The original table selected a subset of benchmarks for space. Below, we provide the supplementary results for its six original training settings and the three specialist compilation variants, covering nine settings in total.

Training variantGSM-Inf (Avg@3)LongBench V2 (Avg@3)BC (Avg@3)SealQA (Avg@3)
SFT-only7.4429.3412.5324.32
Search Trajectories7.0029.5412.4826.51
Code Trajectories6.6728.9412.2224.95
Combined Trajectories6.3331.0711.8225.93
Combined Trajectories (with Mask)6.4431.5312.9226.37
TrajLong-14B-Retrieval6.8932.0713.1426.13
TrajLong-14B-MultidocQA7.2232.6013.9324.68
TrajLong-14B-State Tracking12.2231.8712.7231.35
TrajLong-14B-All6.6731.8711.7724.32

These results complete the picture of how the compilation strategies perform across tasks.

Best regards,

Paper 39899 Authors

AuthorsOpenReview ↗

Response to Reviewer hgsm‘s Remaining Concern (11.18)

Dear Reviewer hgsm,

Thank you for acknowledging the Figure 1 correction and the teacher-reasoning ablation. Here we want to clarify our research objective, the long-context improvements, and the distinct patterns of transfer to agent tasks, with additional results.

  1. The central research objective is to understande which long-context capabilities transfer to which agent tasks, and how supervision design shapes that transfer. Our twelve agent benchmarks span search, tool use, and coding, with different demands on evidence integration, state maintenance, tool interaction, and code execution. These tasks have different optimization priorities and capability bottlenecks, and it is nearly impossible to design a single general-purpose dataset that benefits all three agentic task categories a lot simultaneously, so the same supervision design can produce different transfer benefits.

    On Qwen3-14B, State Tracking and MultidocQA improve search averages by 1.97/0.47 pp and tool-use averages by 1.26/0.62 pp, respectively. On Nemotron, their gains are 2.28/1.93 pp for search and 2.46/1.90 pp averaged over the two SWE benchmarks. Alongside aggregate performance, these gain patterns help characterize the connections between long-context and agent capabilities.

  2. Matched-budget training improves long-context performance. TrajLong-All and MultidocQA outperform Raw on all six long-context benchmarks, improving the average by 1.05 and 1.15 percentage points, respectively. State Tracking improves the long-context average by 1.57 points. On agent tasks, All remains essentially comparable to Raw, MultidocQA is slightly lower, and State Tracking improves the average by 0.75 points. The additional Nemotron comparison covers four long-context and seven search/coding benchmarks. MultidocQA and State Tracking improve the long-context average by 1.23 and 0.92 points, and the agent average by 1.92 and 2.33 points, respectively.

  3. The transfer patterns correspond to the targeted capabilities.

  • MultidocQA reorganizes trajectories into multi-evidence question answering. Its gains on LongBench V2, GraphWalks... align with its focus on evidence integration and dependency reasoning.
  • State Tracking improves GSM-Infinite, WideSearch, and SkillsBench, suggesting that intermediate-state supervision can support long-context reasoning, information aggregation, and multi-step execution.
  • TrajLong-All exceeds Raw and both specialist variants on FSC-V2 and the two SWE benchmarks, suggesting complementary benefits from combining capabilities.

We will refine and clarify our conclusions with the observed long-context improvements and task-dependent agent benefits.


Complete comparison

Avg@3 % for Qwen3-14B under matched mid-training budgets and downstream.

CategoryBenchmarkRawTrajLong-AllMultidocQAState Tracking
Long-contextAvg34.5135.56 +1.05+1.0535.66 +1.15+1.1536.08 +1.57+1.57
SearchBrowseComp11.8211.7713.9312.72
SearchBrowseComp-ZH20.0719.3818.3419.58
SearchWideSearch29.0430.4230.4832.98
SearchFSC-V240.8743.5342.6640.97
SearchSealQA25.9324.3224.6831.35
SearchAvg25.5525.88 +0.34+0.3426.02 +0.47+0.4727.52 +1.97+1.97
Tool useMCP-Atlas43.8044.0046.7345.00
Tool useSkillsBench9.2611.116.9412.04
Tool useWebArena28.6229.4829.8828.42
Tool useAvg27.2328.20 +0.97+0.9727.85 +0.62+0.6228.49 +1.26+1.26
CodingTB 2.116.4814.239.3614.61
CodingNL2Repo17.6615.0315.7115.40
CodingSWE-Verified53.6054.4052.1353.07
CodingSWE-Pro35.9336.4335.1135.89
All agent tasksAvg27.7627.84 +0.09+0.0927.16 −0.59−0.5928.50 +0.75+0.75

Additional Results for NVIDIA-Nemotron-3-Nano-30B-A3B under matched mid-training budgets and downstream.

CategoryBenchmarkRawMultidocQAState Tracking
Long-contextLongBench V223.1324.4025.13
Long-contextMRCR21.2221.0721.82
Long-contextGraphWalks30.9133.1131.79
Long-contextCorpusQA58.6260.2258.82
SearchBrowseComp14.2416.2216.67
SearchBrowseComp-ZH23.4626.7627.64
SearchWideSearch30.5932.3229.30
SearchFSC-V244.2146.6347.66
SearchSealQA21.0121.2323.63
CodingSWE-Verified44.2947.1246.87
CodingSWE-Pro31.8732.8434.20
agent tasksAvg29.9531.87 +1.92+1.9232.28 +2.33+2.33

Paired significance results:

Overlapping individual-model intervals do not imply nonsignificant paired differences. The TrajLong-All–Raw comparisons show positive gain intervals and two-sided p-values below 0.05 on both SWE benchmarks.

BenchmarkPaired itemsAvg@3 gain over Raw ppppPaired 95% CI ppppTwo-sided p
SWE-Verified500+0.800[0.615, 1.133]0.012
SWE-Pro730+0.502[0.412, 1.329]0.037
Reviewer_D2FYOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces TRAJLONG, a framework that converts agent interaction trajectories into capability-targeted supervision for long-context mid-training. It constructs retrieval, multi-document question answering, and state-tracking tasks to support evidence grounding, cross-evidence aggregation, and temporal state maintenance. Experiments on Qwen3-14B and Qwen3-30B-A3B suggest improvements across several long-context and agent benchmarks. Comparisons with raw and masked trajectory training under matched token budgets provide supporting evidence for the proposed approach.

Critical strengths
  1. The framework offers a relatively clear and well-motivated connection between supervision design and the information-processing demands of agent tasks.

  2. The evaluation covers a reasonably broad range of downstream tasks and includes controlled comparisons with raw and masked trajectory baselines.

Critical weaknesses
  1. The respective contributions of capability-targeted task construction, teacher-generated reasoning, and quality filtering could be clarified further. Although the current comparisons are informative, some uncertainty remains about how much each component contributes to the observed gains. Addressing this point would help readers better understand the contribution of the proposed framework.

  2. The robustness and practical value of the additional improvements would benefit from further supporting evidence. Some gains over the matched-budget raw trajectory baseline are relatively modest, so information on result variability would help assess their consistency. A more complete accounting of data-generation costs would also make the overall cost–performance trade-off easier to evaluate.

Questions for authors
  1. The combined variant performs well overall, although individual task variants achieve better results on some benchmarks. How was the 3:4:3 mixing ratio selected? Is there any evidence indicating how sensitive the results are to this choice?

  2. Could the authors share any available information on result variability, particularly for tasks showing relatively modest improvements over raw trajectory training?

  3. To better understand the practical cost–performance trade-off, it would be helpful to know the approximate costs of data compilation and training. How much additional computation does teacher-model inference require?

4 comments
AuthorsOpenReview ↗

Response to Reviewer D2FY (Part1)

Dear Reviewer D2FY,

We sincerely thank you for your careful review of our work and your valuable feedback.

Weakness 1: How much do task construction, teacher reasoning, and quality filtering each contribute?

Your question highlights the contribution of our work. Task construction supplies effective QA supervision, reasoning adds long-context gains, and filtering checks target reliability.

  1. Capability-targeted compilation improves on raw trajectories, ACC, and Generic QA

Using Qwen3-14B with a matched 40B-token budget and SFT recipe, TrajLong-All outperforms unmasked Combined Trajectories (Raw) on all seven benchmarks; both use full-token supervision (Section 5.2, Table 3).

With matched budgets, the same 30B background corpus and SFT recipe, TrajLong also beats ACC on all six tasks (Appendix E.1, Table 18).

Generic QA prompts the teacher to generate questions based on each trajectory and corresponding task, without explicitly targeting specific Retrieval, MultidocQA, or State Tracking capabilities. We match source trajectories, teacher, reasoning format, 40B-token budget, SFT and evaluation. Scores are Avg@3 (%). Table 3 (Section 5.2) in paper also reports the results.

BenchmarkA. Generic QAB. Capability-targeted QA (TrajLong-All)B − A (pp)
MRCR18.3220.18+1.86
GraphWalks30.1132.30+2.19
Oolong-Synth55.6758.50+2.83
CorpusQA62.1063.83+1.73
FSC-V240.5243.53+3.01
SkillsBench9.5411.11+1.57
SWE-Verified52.8954.40+1.51
SWE-Pro35.4136.43+1.02

TrajLong-All improves all eight tasks by 1.02–3.01 pp under the same source and teacher.

  1. QA retains approximately 88% of the long-context gain without reasoning

We remove reasoning from the same MultidocQA source pool, retaining context, question and validated answer. Both versions use 10B task-data tokens and identical SFT. wCoT/woCoT mean with/without reasoning; macro-averages cover all six long-context and twelve agent tasks. Scores are reported in Avg@3 (%) (Section 5.2, Table 4).

BenchmarkRaw (unmasked)MultidocQA wCoTMultidocQA woCoTWithout CoT − RawWithout CoT − With CoT
LongBench V231.0732.6032.34+1.27−0.26
GraphWalks31.3833.4633.10+1.72−0.36
CorpusQA61.8063.4263.12+1.32−0.30
Long-context macro-average34.5135.6635.52+1.01−0.14
BrowseComp11.8213.9313.47+1.65−0.46
MCP-Atlas43.8046.7346.20+2.40−0.53
SWE-Verified53.6052.1352.53−1.07+0.40
Agent macro-average27.7627.1627.29−0.47+0.13

QA-only beats Raw on all six long-context tasks and retains approximately 88% of the gain, only 0.14 pp below wCoT. Its GraphWalks gain has a paired 95% interval of [0.90, 2.50]; the agent macro-average is preserved after reasoning removal.

  1. Filtering reduces ambiguity and checks answers against the evidence

We remove malformed records, incomplete workflows and unproductive repetition, retaining error recovery and progressive exploration. Checks cover Retrieval target uniqueness and answer fidelity, MultidocQA answerability, evidence support and solution consistency, and State Tracking replay, state reconstruction and teacher-reference agreement. These checks reduce unsupported or ambiguous targets. See Appendix A.1, Table 6, and D.2–D.4 for rules; Appendix A.5, Table 10 for quality statistics.

Weakness 2 / Question 2: How consistent are the modest gains over matched-budget raw trajectories, and what is their practical value?

We agree that small gains should be assessed for consistency and significance. Supplementary evaluations and paired analyses support the gains.

  1. Three evaluation runs show higher means and lower SDs

Supplementary evaluations use matched mid-training, SFT and inference settings and three complete runs on the same items. Avg@3 and SD are the mean and standard deviation of the three run-level means. The matched-budget setup is in Section 5.2, Table 3.

BenchmarkRaw: mean ± SDTrajLong: mean ± SD
SWE-Verified53.60 ± 1.7354.40 ± 1.27
SWE-Pro35.93 ± 1.5736.43 ± 0.58
SkillsBench9.26 ± 2.8911.11 ± 0.80

TrajLong achieves higher means and lower SDs across all three benchmarks.

  1. Paired analysis supports the significance of the gains

We use 10,000 paired bootstrap resamples, drawing the same items for both models and recomputing their mean-score difference. The paired 95% confidence intervals for the GraphWalks and FSC-V2 gains are [0.16, 1.77] and [0.51, 4.81], respectively; both are above zero.

Additional paired analyses confirm significant SWE-Verified and SWE-Pro gains over unmasked Raw.

AuthorsOpenReview ↗

Response to Reviewer D2FY (Part2)

Question 1: Why use a 3:4:3 mixture, and how sensitive are results to this ratio?

The 3:4:3 ratio provides an approximately balanced allocation based on data availability. Our additional 1:1:1 experiment yields similar overall performance.

  1. The 3:4:3 ratio reflects data availability and balanced capability coverage

We allocate input tokens to Retrieval, MultidocQA, and State Tracking in a 3:4:3 ratio, corresponding to 3B, 4B, and 3B tokens within the 10B trajectory-data budget. This choice reflects data availability while giving the three capabilities approximately balanced training exposure.

  1. Overall performance remains similar with a 1:1:1 mixture

We retain the Qwen3-14B backbone, the budget of 30B background plus 10B trajectory tokens, and two epochs of downstream SFT, and assign approximately 3.33B tokens to each data type. The table reports post-SFT Avg@3 scores. See Table 22, Appendix E.8.

CategoryBenchmark3:4:3 (paper Avg@3)1:1:1 (Avg@3)Difference
SearchBrowseComp11.7711.23−0.54
SearchWideSearch30.4231.21+0.79
SearchFSC-V243.5343.22−0.31
Tool useMCP-Atlas44.0045.00+1.00
Tool useSkillsBench11.1111.110.00
CodingSWE-Verified54.4055.29+0.89
CodingSWE-Pro36.4337.06+0.63
Long contextMRCR20.1820.93+0.75
Long contextGraphWalks32.3032.10−0.20
Long contextGSM-Infinite6.677.33+0.66

Across ten tasks, six improve, one is unchanged, and three decrease slightly, with differences of −0.54 to +1.00 pp. These results show limited sensitivity to this change in ratio.

  1. Mixing provides broader coverage within the same budget

Specialists use all 10B tokens for one capability, whereas All assigns approximately 3–4B to each; this reduced exposure can affect individual scores. TrajLong-All still outperforms the best specialist on FSC-V2 and both SWE benchmarks, supporting complementary benefits under a fixed budget.

Weakness 2 / Question 3: What are the data-compilation, teacher-inference, and training costs relative to the performance gains?

TrajLong converts a one-time offline construction effort into reusable supervision at the same student-training token budget as Raw. We separate the measured training reference from estimated data-construction costs.

  1. Both methods use the same 40B-token mid-training budget

Combined Trajectories and TrajLong use 30B background plus 10B raw-trajectory or compiled-data tokens, followed by the same SFT recipe (Section 4; Appendix D.1, Tables 14–15). As a measured reference, the Qwen3-14B Combined Trajectories baseline uses FlashAttention-3 on 256 H800 GPUs for 55.4 hours, totaling approximately 14,200 H800 GPU-hours. This is the baseline's training cost, excluding data construction.

  1. Data construction mainly requires teacher inference and lightweight CPU processing

Trajectory parsing, evidence organization, state replay, and data formatting require only a small amount of CPU computation; teacher inference is performed through an API. For a 10B-token compiled corpus, assuming five QA pairs per shared context and contexts accounting for approximately 90% of training tokens, solution generation requires approximately 45B input tokens and 1B output tokens. Using DeepSeek API prices at data-generation time, an 80% input-cache hit rate, and a 30% allowance for question construction, verification, and unsuccessful attempts gives an estimated cost of RMB 17,800–35,700. This expense and the CPU processing overhead form the one-time data-construction investment.

  1. Reuse amortizes construction costs across runs and models

The compiled supervision can be reused without repeating teacher inference or trajectory compilation. Across N uses, each run's share of the one-time construction cost falls to 1/N; another run mainly requires mid-training, SFT, and data loading. Table 3 shows improvements on all seven benchmarks under the same training budget. Improved supervision and repeated reuse together support TrajLong's cost-effectiveness.

Best regards,

Paper 39899 Authors

AuthorsOpenReview ↗

Kind Reminder for Discussion

Dear Reviewer D2FY,

As the discussion period draws to a close, we are very interested in hearing any further feedback you might have and would be pleased to clarify or address any remaining concerns. Please do not hesitate to let us know if you have additional comments or questions.

Thank you again for your time and thoughtful review.

Best regards, Paper 39899 Authors

AuthorsOpenReview ↗

Sincerely invite reviewer to further discuss

Dear Reviewer,

The discussion period is coming to an end in one day. We have posted responses to the proposed concerns and included additional discussion.

We totally understand that this is quite a busy period, so we deeply appreciate it if you could take some time to return further feedback on whether our responses solve your concerns. If there are any other comments, we will try our best to address them.

Best regards,

Paper 39899 Authors

Comments on the paper

4 comments
AuthorsOpenReview ↗

Revision Summary

Dear Reviewers,

Thank you for your thoughtful and constructive feedback. We have revised the manuscript to address your comments, with the following major updates:

  • We revised Figure 1 and its caption to include both masked and unmasked Combined Trajectories and specify the matched mid-training token budgets and downstream SFT recipe. We also updated Figure 6 using the corrected unmasked Raw baseline scores.

  • We expanded Section 5.2, Tables 3–4, with a same-source, same-teacher Generic QA comparison and a teacher-reasoning ablation. TrajLong-All outperforms Generic QA on all seven benchmarks in Table 3, while QA-only MultidocQA retains approximately 88% of the long-context gain.

  • We added Appendix E.1, Tables 18–19, with a matched-budget ACC mid-training comparison and additional ablation results on GSM-Infinite, LongBench V2, BrowseComp, and SealQA. TrajLong-All outperforms ACC on all six evaluated tasks under the same mid-training token budget and downstream SFT recipe.

  • We added Appendix A.5, Tables 10–11, reporting record retention, independently judged correctness for all retained QA examples, supporting-evidence statistics, and state-dependency depths. We specify DeepSeek-v4-pro as the answer generator and GPT-OSS-120B as the independent correctness judge.

  • We added Appendix E.8, Table 22, comparing the default 3:4:3 mixture with 1:1:1 under matched training settings. Across ten benchmarks, score differences range from −0.54 to +1.00 percentage points, indicating limited sensitivity to this change in ratio.

  • We added Appendix E.9, Tables 23–24, evaluating NVIDIA Nemotron-3-Nano-30B-A3B. All three compilation strategies improve over the matched Baseline on all four long-context and seven search/coding benchmarks.

  • We added Appendix E.10, Tables 25–26, with expanded training configurations and repeated runs, exact permutation tests, item-level bootstrap, and leave-one-variant-out analyses. We report additional capability associations and present the paper's training gains alongside intervals from supplementary item-level evaluations.

We appreciate the reviewers' feedback, which has helped improve the clarity and empirical support of our work.

As we are nearing the conclusion of the discussion period, we would like to kindly inquire about the extent to which we have successfully addressed the concerns outlined in your review. We'd appreciate it if you could find time to check the response and further consider your evaluations of our paper. We look forward to further discussions on these new clarifications.

Best regards,

Paper 39899 Authors

Reviewer_hgsmOpenReview ↗

Review updated, scores unchanged

Thank you for the revision and analysis. I updated my review. The correction to figure 1 and the teacher-reasoning ablation address two of my concerns, but a main concern remains: averaged over the twelve agent benchmarks TrajLong-All performs about the same as midtraining on raw trajectories. The text and framing of the paper does not acknowledge this. I am keeping my scores.

AuthorsOpenReview ↗

Response to Reviewer hgsm‘s Remaining Concern (11.18)

Dear Reviewer hgsm,

Thank you for your follow-up and for acknowledging the concerns we have addressed. Regarding the remaining concern, we would like to clarify that a central objective of our work is to understand which long-context capabilities transfer to which agent tasks, and how supervision design shapes this transfer.

We have posted a detailed response under your review part, and we would be pleased to address any remaining concerns. Please let us know if you have additional comments or questions.

Paper 39899 Authors

AuthorsOpenReview ↗

General Response: Key Contributions and Additional Evidence

Dear Area Chair and Reviewers,

We sincerely thank all reviewers for their feedback. We would like to highlight the central contribution of our work and its broader significance:

Mid-training is a key stage in frontier model development: it extends context windows, strengthens the ability to handle long-range dependencies, and lays the foundation for downstream agentic behavior. This makes two questions central to advancing frontier models: how to turn existing agent trajectories into high-quality long-context mid-training data, and how to co-design long-context supervision with downstream agent capabilities.

TrajLong's main value lies in addressing both questions through scalable data construction and empirical analysis. Retrieval, MultidocQA, and State Tracking mine tool observations, dispersed evidence, and evolving file states to construct targeted supervision. Our experiments examine how this supervision develops long-context capabilities and transfers to downstream agent tasks. We believe that combining these two perspectives can play an important role in advancing the long-context and agentic capabilities of frontier LLMs.

We have added controlled comparisons, expanded the evaluation, and revised the manuscript:

  1. New comparisons support the value of capability-targeted construction. ACC retains the original task QA, whereas TrajLong constructs additional questions from evidence within the trajectory. Under matched mid-training budgets and downstream SFT, TrajLong-All outperforms ACC on all six evaluated tasks (Appendix E.1, Table 18). We also added Generic QA, which prompts the teacher with each trajectory and its associated task to generate ordinary QA. Matching source trajectories, teacher, reasoning format, 40B mid-training tokens, SFT, and evaluation, TrajLong-All improves eight tasks by 1.02-3.01 percentage points. It also outperforms both masked and unmasked raw-trajectory baselines on all seven Table 3 tasks, supporting the value of task construction beyond loss masking.

  2. Gains remain without explicit teacher reasoning. The new MultidocQA ablation removes reasoning while retaining contexts, questions, and validated answers, with the same 10B task-data tokens and downstream SFT. QA-only improves all six long-context tasks over unmasked Raw and retains approximately 88% of the long-context gain. The long-context Avg is 35.52 without reasoning, compared with 34.51 for Raw and 35.66 with reasoning (Section 5.2, Table 4). This supports compiled QA supervision.

  3. The benefits transfer across task formats and model families. Retrieval improves non-MRCR tasks before SFT and selected aggregation, search, tool-use, and browser tasks after SFT (Appendix E.5, Figure 10; Section 5.3.3, Figure 6). New NVIDIA Nemotron-3-Nano-30B-A3B experiments show that all three compilation strategies improve all four long-context and seven search/coding benchmarks over a Baseline with matched training settings (Appendix E.9, Tables 23-24). Further SFT also improves BrowseComp and SWE-Pro for the same mid-trained Qwen3-14B model (Section 5.3.4, Table 5), suggesting that downstream adaptation can more fully realize the capability gains established during mid-training.

  4. Additional statistical analyses strengthen the empirical evidence. We expanded from five to ten training configurations and increased repeated training runs. Exact permutation tests, item-level bootstrap, and leave-one-variant-out analyses examine capability associations, sensitivity to benchmark sampling, and dependence on individual configurations (Appendix E.10, Tables 25-26).

  5. The gains persist with a uniform mixture. The original 3:4:3 ratio reflects data availability and approximately balanced capability coverage. With the same total budget and SFT recipe, switching to 1:1:1 changes scores by -0.54 to +1.00 percentage points across ten benchmarks (Appendix E.8, Table 22). Specialists use all 10B task tokens for one capability, while the mixed model allocates approximately 3-4B to each; this difference in training exposure can contribute to gaps on individual tasks.

  6. Data quality and practical reuse. We added retention rates, independent correctness assessments of all retained QA examples, supporting-evidence statistics, and state-dependency depths (Appendix A.5, Tables 10-11). Our cost response distinguishes one-time offline compilation and teacher inference from student training. Raw and TrajLong use the same 40B-token mid-training budget, and compiled supervision can be reused across runs and model families without repeating its construction.

We have also redrawn Figure 1 to include both masked and unmasked baselines, and added four ablation benchmarks in Appendix E.1, Table 19. We appreciate the reviewers' suggestions, which strengthen the evidence for capability-targeted trajectory compilation as an effective approach to agentic mid-training.

Best regards,

Paper 39899 Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.