- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper studies how to select source environments for agent trajectory SFT. It proposes TIP to distinguish task semantics (TS) from interaction realization (IR), then compares two source groups on BFCL-V4 Memory using three Qwen3 models. IR-SFT consistently outperforms TS-SFT, and improves over the original model on two of the three students. Case studies illustrate differences in memory writing, retrieval, and answer completion.
Critical strengths
-
The research question is interesting and practically relevant. Separating task requirements from interaction conditions offers a useful perspective on why seemingly similar tasks may transfer poorly.
-
The performance gap is substantial and consistent across the three models and memory interfaces. The trajectory examples help explain where their behaviors differ.
Critical weaknesses
-
The study compares only two pooled source groups on one target benchmark. The authors acknowledge this limited coverage, which is good. However, it directly limits the main claim: the observed ordering may reflect these particular dataset choices rather than a general relationship between interaction proximity and transfer.
-
IR-SFT uses 4,217 examples versus 3,145 for TS-SFT, without matching supervised-token budgets. Dataset composition also differs, and no source ablations identify what drives the gain. Although authors acknowledge these confounds, they leave a central question unresolved: is the advantage related to IR proximity, data quantity, or particular source datasets?
-
The experiments do not sufficiently validate the usefulness of the proposed TS/IR distances. Defining these distances and observing a performance gap between two groups does not establish that they reliably identify better training sources. The case studies illustrate individual successes and failures, but do not explain how these relate to the distance measures. More direct evidence that the measured proximity corresponds to transfer outcomes across different sources is needed.
Questions for authors
Please see the concerns raised in Critical Weaknesses.
4 comments
3. Connecting the Case Studies to the IR Distance Attributes (W3)
-
Cases 1–2 illustrate the interaction properties represented by history policy, state persistence, and action families.
- These attributes characterize how information is exposed, preserved across turns, and accessed through available actions.
- In Figure 3, the 30B IR-SFT model saves the required facts before the context reset and uses them to answer afterward, whereas TS-SFT fails to construct usable memory.
- The 8B RecSum diagnostic further demonstrates the behavioral difference: IR-SFT invokes query tools in 461/465 executions, compared with 0/465 for TS-SFT (Table 7).
-
Case 3 illustrates the execution-control and completion behaviors represented by controller and termination.
- The 4B TS-SFT model retrieves relevant evidence but continues retrieving without answering, while IR-SFT fails earlier by omitting the required memory write.
- The 8B diagnostic also shows that query-tool use can occur without a final answer, as observed in 105 IR-SFT executions.
-
These cases connect the IR attributes to concrete differences in memory construction, retrieval, and answer completion, complementing the direct distance–transfer evidence in Table R1.
We recognize the importance of the issues you raised, and we will incorporate all these additional results and clarifications into the revised manuscript. With this new evidence, we hope our contributions can be viewed in a fuller light. Thank you again for your valuable feedback.
Follow-Up: Additional Evidence on Target Coverage, Source Composition, and Transfer Behavior
Thank you again for your constructive feedback. Continuing our earlier response and the November 16 manuscript revision, we provide additional experiments and behavioral analyses below to further strengthen our findings.
4. Extending the Analysis to an Additional Target (W1)
To examine whether the observed source–target relationship extends beyond Memory, we additionally evaluated SciCode on Qwen3-8B using two source groups constructed by TS/IR proximity under matched supervised-token budgets and update counts (averaged over three repeats). As shown in Table R4, TS-SFT drops below the baseline by 10.76 pp, whereas IR-SFT improves performance by 5.44 pp, confirming that the TS/IR transfer pattern generalizes beyond the Memory setting.
Table R4: Step accuracy on the additional SciCode target under matched training budgets.
| Training Condition | Source Proximity to SciCode | Source Benchmark Families | SciCode Step Accuracy (%) | Δ vs. Baseline (pp) |
|---|---|---|---|---|
| Baseline | — | — | 25.12 | — |
| TS-SFT | TS-closer / IR-farther | AlgoTune, DS-1000, QCircuitBench | 14.35 | −10.76 |
| IR-SFT | TS-farther / IR-closer | HumanEval, MBPP, BigCodeBench | 30.56 | +5.44 |
5. Interpreting Source Composition Through Trajectory-Level Interaction Statistics (W2)
To examine what different source datasets provide as supervision beyond coarse dataset labels, we analyzed retained SFT examples for tool use and explicit storage writes (Table R5):
- Tool-call presence alone does not explain transfer differences: StructMemEval invokes tools in 99.61% of examples yet causes −10.32 pp negative transfer.
- Positive transfer occurs without explicit storage-write demonstrations: VehicleMemBench, ToolMaze, and TravelBench contain no explicit storage-write calls in their retained SFT trajectories—for instance, VehicleMemBench only covers querying and execution via context summaries or read-only retrieval—yet all achieve positive transfer, alongside ToolMisuseBench which includes explicit writes.
Table R5: Selected source-side interaction statistics and individual-source transfer on Qwen3-8B (Memory baseline: 18.06%).
| Group | Source | ≥1 Tool Call (%) | ≥1 Explicit Write Call (%) | Memory Accuracy (%) | Δ vs. Baseline (pp) |
|---|---|---|---|---|---|
| TS | StructMemEval | 99.61 | 0.39 | 7.74 | −10.32 |
| IR | VehicleMemBench | 100.00 | 0.00 | 23.08 | +5.02 |
| IR | ToolMaze | 100.00 | 0.00 | 18.78 | +0.72 |
| IR | ToolMisuseBench | 100.00 | 100.00 | 21.72 | +3.66 |
| IR | TravelBench | 84.81 | 0.00 | 18.64 | +0.58 |
6. Linking TS/IR Distances to Transfer Outcomes and Target Behaviors (W3)
Complementing the individual source results in Table R1 and our case-level analyses, we evaluated aggregate target-side behavior on the full-pool TS-SFT and IR-SFT checkpoints across all three students (Table R6):
- Across all three models, queried-fact presence at query time reaches 24.64%–28.64% after IR-SFT, versus 4.90%–11.73% after TS-SFT.
- On Qwen3-4B, comparable write-trigger rates (37.24% vs. 35.74%) still yield a 15.70 pp accuracy gap.
- On Qwen3-30B-A3B and Qwen3-8B, IR-SFT surpasses the baseline despite lower queried-fact presence, indicating that fact availability alone does not determine task accuracy. Together with our case-level analyses, these results highlight the importance of coordinating information retrieval with execution control and answer termination.
Write-trigger rate is measured over prerequisite entries, and queried-fact presence is measured at query time. These expanded diagnostics and failure cases are incorporated into Sections 6.2–6.3 and Appendices A.3.
Table R6: Target-side behavioral statistics across the three students (TS-SFT / IR-SFT, averaged over three repeats).
| Student | Write-Trigger Rate (TS / IR, %) | Queried Facts Present (TS / IR, %) | Memory Accuracy (TS / IR, %) |
|---|---|---|---|
| Qwen3-30B-A3B | 18.62 / 45.35 | 9.73 / 28.64 | 9.39 / 25.52 |
| Qwen3-8B | 38.74 / 59.46 | 4.90 / 28.16 | 8.39 / 24.95 |
| Qwen3-4B | 37.24 / 35.74 | 11.73 / 24.64 | 2.29 / 17.99 |
Overall Summary of Our Responses to W1–W3
-
W1 (Target and Experimental Coverage): Our original TIP profiling and individual-source experiments establish the relationship between source–target proximity and transfer across eight source families, with all four IR-closer sources outperforming all four TS-closer sources by at least 9.39 pp. The additional SciCode evaluation extends this investigation to a second target, providing further evidence beyond the original Memory setting.
-
W2 (Training Exposure and Source Composition): We conducted supervised-token-matched experiments across three independent training seeds, preserving an 11.73 pp IR–TS advantage. Source-removal ablations further show that removing VehicleMemBench still yields +4.52 pp, while removing GateMem improves TS-SFT but leaves it 1.36 pp below baseline. The additional source-side trajectory statistics further characterize the interaction supervision underlying these transfer differences.
-
W3 (Distance–Transfer Relationship and Behavioral Interpretation): We provided quantitative TS/IR distances and complete unit-level profiles, directly connected individual-source proximity to transfer outcomes, and reported sensitivity analyses under alternative memory-boundary encodings, 20 field-weight configurations, and attribute removal. The original case studies, together with the additional aggregate behavioral diagnostics, further connect these relationships to memory construction, information preservation, retrieval, and answer completion.
Overall, our investigation examines cross-environment transfer from complementary perspectives: source–target relationships, experimental controls, and observable agent behaviors. Our goal is to understand how task requirements and interaction realization jointly characterize transfer outcomes, rather than to establish a universal advantage of one dimension over the other.
We would appreciate your reassessment of these analyses together with the revised manuscript. Please let us know whether they address your main concerns, or which issue remains unresolved.
Response to Reviewer HtwH
We sincerely thank the reviewer for the constructive feedback and valuable suggestions. In response to the concerns raised, we present additional experimental results and analyses below.
Following our main evaluation protocol, every checkpoint is evaluated over three complete repeats.
1. Direct Evidence Linking TS/IR Distances to Transfer Across Individual Sources (W1 & W3)
All four IR-closer sources achieve positive transfer, while all four TS-closer sources produce negative transfer (Table R1).
The separation holds across all eight individual sources, not merely the original pooled groups.
Table R1: Source-to-target distances and single-source SFT results on Qwen3-8B.
Family-level distances average constituent benchmark-unit distances from Eqs. (5)–(6), weighted by retained example counts.
2. Disentangling IR Proximity from Data Quantity and Individual Source Contributions (W2 & W1)
Table R2: Supervised-token budget-matched SFT on Qwen3-8B.
Each token-matched result averages three independently trained checkpoints, one per training seed.
Table R3: Leave-one-family-out SFT ablations on Qwen3-8B.