acceptodds
Under review as a conference paper at ICLR 2027

Understanding Cross-Environment Agent Transfer through Task Semantics and Interaction Realization

Abstract

Teacher trajectories collected from executable benchmark variants provide a scalable source of supervision for agent post-training. However, given a target benchmark, it remains unclear how source benchmark variants should be selected. Source benchmarks may resemble the target in Task Semantics (TS)—including the task goal, domain, core capabilities, and success conditions—or in Interaction Realization (IR)—including how information is accessed, actions are executed, feedback is provided, and state is managed. We investigate which type of proximity is more predictive of effective transfer. To this end, we propose TIP, an analysis framework that characterizes benchmark relationships along the TS and IR dimensions separately. We generate and filter teacher trajectories from two groups of source benchmarks, fine-tune Qwen3 models via trajectory SFT, and evaluate transfer to a target benchmark. We find that sources more closely aligned with the target in interaction realization yield larger performance gains, whereas sources with greater task-semantic similarity consistently induce negative transfer across the evaluated models. Our study develops a benchmark-relation framework beyond task-semantic similarity, clarifies when trajectories generated by one benchmark can provide useful training signals for another, and offers testable guidance for source-data selection and the analysis of agent transfer across benchmarks.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
LLM Agents, Training Data Selection, Transfer Learning, Supervised Fine-Tuning

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_HtwHOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies how to select source environments for agent trajectory SFT. It proposes TIP to distinguish task semantics (TS) from interaction realization (IR), then compares two source groups on BFCL-V4 Memory using three Qwen3 models. IR-SFT consistently outperforms TS-SFT, and improves over the original model on two of the three students. Case studies illustrate differences in memory writing, retrieval, and answer completion.

Critical strengths
  1. The research question is interesting and practically relevant. Separating task requirements from interaction conditions offers a useful perspective on why seemingly similar tasks may transfer poorly.

  2. The performance gap is substantial and consistent across the three models and memory interfaces. The trajectory examples help explain where their behaviors differ.

Critical weaknesses
  1. The study compares only two pooled source groups on one target benchmark. The authors acknowledge this limited coverage, which is good. However, it directly limits the main claim: the observed ordering may reflect these particular dataset choices rather than a general relationship between interaction proximity and transfer.

  2. IR-SFT uses 4,217 examples versus 3,145 for TS-SFT, without matching supervised-token budgets. Dataset composition also differs, and no source ablations identify what drives the gain. Although authors acknowledge these confounds, they leave a central question unresolved: is the advantage related to IR proximity, data quantity, or particular source datasets?

  3. The experiments do not sufficiently validate the usefulness of the proposed TS/IR distances. Defining these distances and observing a performance gap between two groups does not establish that they reliably identify better training sources. The case studies illustrate individual successes and failures, but do not explain how these relate to the distance measures. More direct evidence that the measured proximity corresponds to transfer outcomes across different sources is needed.

Questions for authors

Please see the concerns raised in Critical Weaknesses.

4 comments
AuthorsOpenReview ↗

Response to Reviewer HtwH

We sincerely thank the reviewer for the constructive feedback and valuable suggestions. In response to the concerns raised, we present additional experimental results and analyses below.

Following our main evaluation protocol, every checkpoint is evaluated over three complete repeats.

1. Direct Evidence Linking TS/IR Distances to Transfer Across Individual Sources (W1 & W3)

  • All four IR-closer sources achieve positive transfer, while all four TS-closer sources produce negative transfer (Table R1).

    • To examine source-level transfer at a finer granularity than the pooled groups, we evaluate Qwen3-8B fine-tuned independently on each of the eight source benchmark families (characterized under TIP by their Task Semantics distance DTSD_{\mathrm{TS}} and Interaction Realization distance DIRD_{\mathrm{IR}}, where lower values indicate closer proximity).
    • IR-closer sources (DIR∈[0.1875,0.3882]D_{\mathrm{IR}} \in [0.1875, 0.3882]) achieve 18.64%–23.08% accuracy, improving over the 18.06% baseline by +0.58 to +5.02 pp.
    • TS-closer sources (DTS∈[0.2083,0.4500]D_{\mathrm{TS}} \in [0.2083, 0.4500], DIR∈[0.6458,0.8750]D_{\mathrm{IR}} \in [0.6458, 0.8750]) achieve 6.95%–9.25% accuracy, yielding −8.81 to −11.11 pp relative to baseline.
  • The separation holds across all eight individual sources, not merely the original pooled groups.

    • Every IR source outperforms every TS source. Even the lowest-performing IR source (18.64%) exceeds the highest-performing TS source (9.25%) by 9.39 pp.
    • These results directly connect the measured proximity distinction to source-level transfer outcomes: task-semantic similarity alone does not guarantee beneficial transfer, whereas closer interaction realization consistently identifies the beneficial sources in our comparisons.

Table R1: Source-to-target distances and single-source SFT results on Qwen3-8B.

GroupBenchmark FamilyDTS↓D_{\mathrm{TS}} \downarrowDIR↓D_{\mathrm{IR}} \downarrowMean Accuracy (%)Δ\Delta Baseline (pp)
—Baseline——18.06—
TSGateMem0.20830.75008.32−9.74
TSStructMemEval0.28330.64587.74−10.32
TSMemGround0.40830.87509.25−8.81
TSAMemGym0.45000.70316.95−11.11
IRToolMaze0.76090.187518.78+0.72
IRVehicleMemBench0.83330.210923.08+5.02
IRToolMisuseBench0.91400.317121.72+3.66
IRTravelBench0.82700.388218.64+0.58

Family-level distances average constituent benchmark-unit distances from Eqs. (5)–(6), weighted by retained example counts.

2. Disentangling IR Proximity from Data Quantity and Individual Source Contributions (W2 & W1)

  • IR-SFT retains an 11.73 pp advantage after matching supervised-token budgets (Table R2).
    • We matched the supervised-token budgets of the two pooled groups (742,238 for TS vs. 742,598 for IR, differing by less than 0.05%).
    • Despite using fewer retained examples than TS-SFT, token-matched IR-SFT achieves 22.46% vs. 10.73%, preserving an 11.73 pp advantage and the ordering IR-SFT > Baseline > TS-SFT.
    • Thus, the substantial IR-SFT advantage persists under nearly identical supervised-token budgets, directly addressing the data-quantity confound.

Table R2: Supervised-token budget-matched SFT on Qwen3-8B.

Training ConditionSupervised TokensMean Accuracy (%)Δ\Delta Baseline (pp)
Baseline—18.06—
TS-SFT (Token-Matched, 3-seed mean)742,23810.73−7.33
IR-SFT (Token-Matched, 3-seed mean)742,59822.46+4.40

Each token-matched result averages three independently trained checkpoints, one per training seed.

  • Source-removal ablations show that the transfer pattern does not depend on the inclusion of a particular memory-related source (Table R3).
    • IR without VehicleMemBench: Removing the memory-focused source reduces accuracy from 24.95% to 22.58%, but the remaining 3,235 retained examples still achieve +4.52 pp over baseline. Thus, positive IR transfer does not require VehicleMemBench.
    • TS without GateMem: Removing this large long-context source improves accuracy from 8.39% to 16.70% (+8.31 pp), but performance remains −1.36 pp below baseline. Source composition affects the magnitude of degradation, but negative transfer persists.
    • Together with the single-source results in Table R1, these ablations show that the observed transfer pattern is not confined to either dataset or the original pooled mixtures.

Table R3: Leave-one-family-out SFT ablations on Qwen3-8B.

Training ConditionRetained ExamplesMean Accuracy (%)Δ\Delta Baseline (pp)
Baseline—18.06—
TS-SFT3,1458.39−9.68
IR-SFT4,21724.95+6.88
TS w/o GateMem1,98116.70−1.36
IR w/o VehicleMemBench3,23522.58+4.52

3. Connecting the Case Studies to the IR Distance Attributes (W3)

  • Cases 1–2 illustrate the interaction properties represented by history policy, state persistence, and action families.

    • These attributes characterize how information is exposed, preserved across turns, and accessed through available actions.
    • In Figure 3, the 30B IR-SFT model saves the required facts before the context reset and uses them to answer afterward, whereas TS-SFT fails to construct usable memory.
    • The 8B RecSum diagnostic further demonstrates the behavioral difference: IR-SFT invokes query tools in 461/465 executions, compared with 0/465 for TS-SFT (Table 7).
  • Case 3 illustrates the execution-control and completion behaviors represented by controller and termination.

    • The 4B TS-SFT model retrieves relevant evidence but continues retrieving without answering, while IR-SFT fails earlier by omitting the required memory write.
    • The 8B diagnostic also shows that query-tool use can occur without a final answer, as observed in 105 IR-SFT executions.
  • These cases connect the IR attributes to concrete differences in memory construction, retrieval, and answer completion, complementing the direct distance–transfer evidence in Table R1.

We recognize the importance of the issues you raised, and we will incorporate all these additional results and clarifications into the revised manuscript. With this new evidence, we hope our contributions can be viewed in a fuller light. Thank you again for your valuable feedback.

AuthorsOpenReview ↗

Follow-Up: Additional Evidence on Target Coverage, Source Composition, and Transfer Behavior

Thank you again for your constructive feedback. Continuing our earlier response and the November 16 manuscript revision, we provide additional experiments and behavioral analyses below to further strengthen our findings.

4. Extending the Analysis to an Additional Target (W1)

To examine whether the observed source–target relationship extends beyond Memory, we additionally evaluated SciCode on Qwen3-8B using two source groups constructed by TS/IR proximity under matched supervised-token budgets and update counts (averaged over three repeats). As shown in Table R4, TS-SFT drops below the baseline by 10.76 pp, whereas IR-SFT improves performance by 5.44 pp, confirming that the TS/IR transfer pattern generalizes beyond the Memory setting.

Table R4: Step accuracy on the additional SciCode target under matched training budgets.

Training ConditionSource Proximity to SciCodeSource Benchmark FamiliesSciCode Step Accuracy (%)Δ vs. Baseline (pp)
Baseline——25.12—
TS-SFTTS-closer / IR-fartherAlgoTune, DS-1000, QCircuitBench14.35−10.76
IR-SFTTS-farther / IR-closerHumanEval, MBPP, BigCodeBench30.56+5.44

5. Interpreting Source Composition Through Trajectory-Level Interaction Statistics (W2)

To examine what different source datasets provide as supervision beyond coarse dataset labels, we analyzed retained SFT examples for tool use and explicit storage writes (Table R5):

  • Tool-call presence alone does not explain transfer differences: StructMemEval invokes tools in 99.61% of examples yet causes −10.32 pp negative transfer.
  • Positive transfer occurs without explicit storage-write demonstrations: VehicleMemBench, ToolMaze, and TravelBench contain no explicit storage-write calls in their retained SFT trajectories—for instance, VehicleMemBench only covers querying and execution via context summaries or read-only retrieval—yet all achieve positive transfer, alongside ToolMisuseBench which includes explicit writes.

Table R5: Selected source-side interaction statistics and individual-source transfer on Qwen3-8B (Memory baseline: 18.06%).

GroupSource≥1 Tool Call (%)≥1 Explicit Write Call (%)Memory Accuracy (%)Δ vs. Baseline (pp)
TSStructMemEval99.610.397.74−10.32
IRVehicleMemBench100.000.0023.08+5.02
IRToolMaze100.000.0018.78+0.72
IRToolMisuseBench100.00100.0021.72+3.66
IRTravelBench84.810.0018.64+0.58

6. Linking TS/IR Distances to Transfer Outcomes and Target Behaviors (W3)

Complementing the individual source results in Table R1 and our case-level analyses, we evaluated aggregate target-side behavior on the full-pool TS-SFT and IR-SFT checkpoints across all three students (Table R6):

  • Across all three models, queried-fact presence at query time reaches 24.64%–28.64% after IR-SFT, versus 4.90%–11.73% after TS-SFT.
  • On Qwen3-4B, comparable write-trigger rates (37.24% vs. 35.74%) still yield a 15.70 pp accuracy gap.
  • On Qwen3-30B-A3B and Qwen3-8B, IR-SFT surpasses the baseline despite lower queried-fact presence, indicating that fact availability alone does not determine task accuracy. Together with our case-level analyses, these results highlight the importance of coordinating information retrieval with execution control and answer termination.

Write-trigger rate is measured over prerequisite entries, and queried-fact presence is measured at query time. These expanded diagnostics and failure cases are incorporated into Sections 6.2–6.3 and Appendices A.3.

Table R6: Target-side behavioral statistics across the three students (TS-SFT / IR-SFT, averaged over three repeats).

StudentWrite-Trigger Rate (TS / IR, %)Queried Facts Present (TS / IR, %)Memory Accuracy (TS / IR, %)
Qwen3-30B-A3B18.62 / 45.359.73 / 28.649.39 / 25.52
Qwen3-8B38.74 / 59.464.90 / 28.168.39 / 24.95
Qwen3-4B37.24 / 35.7411.73 / 24.642.29 / 17.99
AuthorsOpenReview ↗

Overall Summary of Our Responses to W1–W3

  • W1 (Target and Experimental Coverage): Our original TIP profiling and individual-source experiments establish the relationship between source–target proximity and transfer across eight source families, with all four IR-closer sources outperforming all four TS-closer sources by at least 9.39 pp. The additional SciCode evaluation extends this investigation to a second target, providing further evidence beyond the original Memory setting.

  • W2 (Training Exposure and Source Composition): We conducted supervised-token-matched experiments across three independent training seeds, preserving an 11.73 pp IR–TS advantage. Source-removal ablations further show that removing VehicleMemBench still yields +4.52 pp, while removing GateMem improves TS-SFT but leaves it 1.36 pp below baseline. The additional source-side trajectory statistics further characterize the interaction supervision underlying these transfer differences.

  • W3 (Distance–Transfer Relationship and Behavioral Interpretation): We provided quantitative TS/IR distances and complete unit-level profiles, directly connected individual-source proximity to transfer outcomes, and reported sensitivity analyses under alternative memory-boundary encodings, 20 field-weight configurations, and attribute removal. The original case studies, together with the additional aggregate behavioral diagnostics, further connect these relationships to memory construction, information preservation, retrieval, and answer completion.

Overall, our investigation examines cross-environment transfer from complementary perspectives: source–target relationships, experimental controls, and observable agent behaviors. Our goal is to understand how task requirements and interaction realization jointly characterize transfer outcomes, rather than to establish a universal advantage of one dimension over the other.

We would appreciate your reassessment of these analyses together with the revised manuscript. Please let us know whether they address your main concerns, or which issue remains unresolved.

Reviewer_6h7sOpenReview ↗
Rating
4(+1)
Confidence
4
Informativeness
3(+1)
Presentation
2
Soundness
3(+1)
Read review
Paper summary

The paper asks which kind of source–target resemblance predicts transfer when an agent is fine-tuned on teacher trajectories from other benchmarks. It separates Task Semantics (TS: task goal, domain, core capabilities, success conditions) from Interaction Realization (IR: interaction structure and feedback, action space, context and state management, system orchestration). It proposes TIP, which labels benchmark units with a controlled vocabulary and computes one distance per dimension from indicator and Jaccard terms averaged over equally weighted fields (Eqs. 4–6). The target is BFCL-V4 Memory: 465 questions, 155 each for key–value (KV), vector, and recursive-summarization (RecSum) memory, asked after a context reset. Five variants of MemGround, GateMem, StructMemEval, and AMemGym form a "TS group" described as closer in TS and farther in IR. Seven variants of VehicleMemBench, ToolMisuseBench, ToolMaze, and TravelBench form an "IR group" with the reverse relation. DeepSeek-V4-Flash generates the trajectories. After filtering, 3,145 TS and 4,217 IR examples fine-tune Qwen3-30B-A3B-Instruct-2507, Qwen3-8B, and Qwen3-4B-Instruct-2507 (one epoch, one seed), and each checkpoint is evaluated three times. IR-SFT exceeds TS-SFT for every student and every interface, by 15.70–16.56 pp in mean accuracy. Relative to each student's own baseline, IR-SFT changes mean accuracy by +8.67, +6.88, and −3.23 pp, and TS-SFT by −7.46, −9.68, and −18.92 pp (30B, 8B, 4B; Table 3). Three case studies (Figure 3) and query-tool counts for one model (Table 7) relate the gap to memory writing, retrieval, and answer completion.

Critical strengths
  • IR-SFT beats TS-SFT in all nine student × interface cells. Within each student, the lowest IR-SFT repeat exceeds the highest TS-SFT repeat (24.95 vs. 10.11, 23.23 vs. 9.25, 15.27 vs. 2.58; Table 6). By my computation from Table 6, the gap is 7–18 times the larger evaluation SD of the two conditions. The TS-SFT collapse is a clear negative result in its own right: training on related memory-QA trajectories can remove a student's ability to use memory tools.
  • Eqs. 4–6, Tables 1 and 9, and the full prompt in Appendix A.4, including three memory-specific boundary checks, make TIP concrete enough to audit and reuse. Section 7 states the main limitations itself: no causal isolation of either dimension, one training seed, pooled groups, and untested aggregation weights.
Critical weaknesses
  • The TS/IR proximity at the center of the claim is asserted, not shown. Eqs. 5–6 define D_TS and D_IR, but no value, group aggregate, or field assignment is reported for any unit. Table 2 gives qualitative labels only and notes that "These labels describe the group comparison, not identical properties of every constituent variant." Appendix A.4 says "The specification and prompt do not replace unit-level assignments or numerical distances," but neither appears in the paper. The labels also hinge on a boundary decision. Section 5.1 says the target has "requirements for memory construction, state persistence, and retrieval," and the A.4 prompt says to "assign an action to TS when it is itself a task requirement," yet Figure 1 and Section 1 treat memory writing as IR. If persistence across a context reset were coded as a TS requirement, the TS group might no longer be closer in TS, and the two dimensions would no longer favor different groups. Without distances and a sensitivity analysis, the paper's question, "which type of proximity is more predictive of effective transfer", cannot be answered from the evidence.
  • A single confounded contrast cannot attribute the gap to IR proximity. The study has one group per relation type, one target, and one training seed. The groups differ in size (4,217 vs. 3,145 examples, +34%), in supervised-token exposure (the pooled count ∑z∈Dg∑ℓmz,ℓ\sum_{z\in\mathcal D_g}\sum_{\ell} m_{z,\ell}, the group analogue of ZuZ_u in Eq. 2; never reported), and in composition. VehicleMemBench supplies 23% of the IR examples (982), and its variant names (none, gold, key_value, summary) suggest memory-mechanism conditions that may resemble the target's KV and RecSum interfaces.
Questions for authors
  • Please report D_TS and D_IR against BFCL-V4 Memory for each of the 12 variants and each unit within them, with the field assignments, the comparable sets AfA_f and FXF_X of Eq. 6 for each pair (u,t)(u,t), and the rule that turns unit-level distances into the Table 2 relations. Does every TS-group unit have lower D_TS and higher D_IR than every IR-group unit? If not, which units invert the relation?
  • How were memory construction and state persistence coded for the target, given the A.4 instruction to "assign an action to TS when it is itself a task requirement"? Do the Table 2 relations hold if persisting information across a context reset is coded as a TS core capability, under alternative field weights, and with each attribute left out in turn?
  • What do the VehicleMemBench variants involve? Do key_value and summary expose memory-write and retrieval operations analogous to the target's KV and RecSum interfaces? Please report leave-one-family-out runs for at least one student under the same recipe: at minimum IR without VehicleMemBench (3,235 examples) and TS without GateMem (1,981 examples).
  • Figure 2 claims "Matched training tokens," but Section 1, the Figure 4 caption, and Section 7 say exposure differs. Which is correct? Please report total and supervised tokens for each pooled dataset, whether sequences were packed into the 131,072-token window, and the number of optimizer updates per run. Please also report a run with the IR pool subsampled to the TS pool's supervised-token budget.
  • For each source variant, what fraction of retained examples contain at least one tool call and at least one state-writing call, and how many tool calls does an example contain on average? For each checkpoint on the target, what fraction of the 111 prerequisite entries trigger a memory write, and what fraction of queried facts are in memory at query time? These numbers would test whether the gap reduces to the propensity to call tools.
  • How many candidates were screened in Section 4.1? How were the 12 variants and the two groups chosen, and was the grouping fixed before any training run? Were any other groupings trained or evaluated but not reported?
5 comments
AuthorsOpenReview ↗

Response to Reviewer 6h7s

We sincerely thank the reviewer for the detailed assessment and highly constructive questions. Below, we provide the requested experimental results and analyses.

Following our main evaluation protocol, every checkpoint is evaluated over three complete repeats.

1. Variant-/Unit-Level TIP Distances, Aggregation, and Inversion Checks (W1, Q1)

  • Distance computation and aggregation.

    • The 8 source benchmark families and 12 variants comprise 23 benchmark units. Following Eqs. (4)–(6) and the controlled-vocabulary attributes in Table 9, categorical attributes use a 0/1 indicator distance, while set-valued attributes use Jaccard distance. We first average comparable attributes within each field, then equally average the four comparable fields within each dimension. Unit-level distances are aggregated to the variant, family, and pooled-group levels, weighted by retained example counts.
    • We will provide the complete unit-level field assignments and the pair-specific comparable attribute and field sets AfA_f and FXF_X used in Eq. (6) in the appendix of the revised manuscript.
  • Numerical distances and cross-group inversion checks (Table R1).

    • Variant level: All 12 source variants maintain strict cross-group separation in both DTSD_{\mathrm{TS}} and DIRD_{\mathrm{IR}}. Every TS-group variant has a lower DTSD_{\mathrm{TS}} than every IR-group variant ([0.2083, 0.4500] vs. [0.7609, 0.9140]) and a higher DIRD_{\mathrm{IR}} ([0.6354, 0.8750] vs. [0.1875, 0.3882]). There are no cross-group ranking inversions.
    • Unit level: The 23 constituent units also maintain strict cross-group separation in DTSD_{\mathrm{TS}}. For DIRD_{\mathrm{IR}}, only one IR-group unit, TravelBench / unsolved (DIR=0.7917D_{\mathrm{IR}}=0.7917), exhibits cross-group ranking inversions.

Table R1: Constituent units, retained example counts, and TIP distances to BFCL-V4 Memory for all 12 source variants.

GroupBenchmark Family / Variant / UnitUnitsRetained ExamplesDTS↓D_{\mathrm{TS}}\downarrowDIR↓D_{\mathrm{IR}}\downarrowDTSD_{\mathrm{TS}} vs. DIRD_{\mathrm{IR}}
TSGateMem / long_context31,1640.20830.7500<
TS↳ access-control3230.20830.7500<
TS↳ active-forgetting3710.20830.7500<
TS↳ utility4700.20830.7500<
TSStructMemEval / mem_agent12540.28330.6458<
TS↳ mem-agent2540.28330.6458<
TSMemGround / trpg_success11440.40830.8750<
TS↳ trpg_qa1440.40830.8750<
TSAMemGym / native17260.45000.6354<
TS↳ overall-on-policy7260.45000.6354<
TSAMemGym / awi18570.45000.7604<
TS↳ overall-on-policy8570.45000.7604<
TSPooled TS Group73,1450.34520.7237<
IRVehicleMemBench / none1840.83330.2188>
IR↳ direct_context_baselines840.83330.2188>
IRVehicleMemBench / gold13880.83330.2188>
IR↳ direct_context_baselines3880.83330.2188>
IRVehicleMemBench / key_value12460.83330.1875>
IR↳ key_value_memory2460.83330.1875>
IRVehicleMemBench / summary12640.83330.2188>
IR↳ recursive_summary_memory2640.83330.2188>
IRToolMaze / legacy_full79260.76090.1875>
IR↳ baseline_p02770.70830.1875>
IR↳ explicit_permanent_c1_p2180.78330.1875>
IR↳ explicit_permanent_multipath_p21520.78330.1875>
IR↳ explicit_transient_p12680.78330.1875>
IR↳ implicit_permanent_c1_p450.78330.1875>
IR↳ implicit_permanent_multipath_p4970.78330.1875>
IR↳ implicit_transient_p31090.78330.1875>
IRToolMisuseBench / api_model_train21,5190.91400.3171>
IR↳ crud_state_change7300.87500.3333>
IR↳ file_artifact_creation7890.95000.3021>
IRTravelBench / legacy_full37900.82700.3882>
IR↳ multi_turn2800.70830.4792>
IR↳ single_turn3700.87500.1667>
IR↳ unsolved1400.93750.7917>
IRPooled IR Group164,2170.84530.2772>

2. Encoding Target Memory Requirements and Boundary Sensitivity Analysis (W1, Q2)

  • How memory construction and state persistence are encoded for the target.

    • The TS/IR boundary: In TIP, Task Semantics (TS) describes what the task specification requires: task goal, task domain, core capability requirements, and success conditions. Interaction Realization (IR) describes how these requirements are implemented through environmental interfaces and executable protocols: interaction structure and feedback, action space, context and state management, and system orchestration.
    • Mapping memory construction and persistence: We follow the rule that an action belongs to TS when it is itself a task requirement, while its implementation mechanism belongs to IR. The target's requirements to actively modify state and retrieve information are captured in the TS field core_capabilities as {interpret_requirements, information_seeking, act_on_environment}, with act_on_environment capturing the required state construction. In contrast, the concrete interaction mechanisms—read/write tool interfaces (action_families), within-session context policies (history_policy), and state persistence across resets (state_persistence)—are encoded in the corresponding IR fields.
  • Sensitivity to boundary encoding, field weights, and attribute removal.

    • Following the reviewer's suggestions, we conducted three sensitivity analyses. Both the pooled-group relations in Table 2—TS is closer in DTSD_{\mathrm{TS}}, while IR is closer in DIRD_{\mathrm{IR}}—and strict cross-group separation at the variant level remain unchanged.
    • Memory-boundary recoding: Under an expanded capability vocabulary, we add retain_and_use_history_across_context_resets to TS. The confirmed-only setting assigns this capability only to environments that explicitly require state persistence across context resets. The inclusive setting assigns it to all source benchmarks involving history retention across stages.
    • Alternative field weights: We test one equal-weight baseline and 20 non-uniform configurations, reweighting individual fields or pairs of fields and then renormalizing.
    • Leave-one-attribute-out analysis: We remove each controlled-vocabulary attribute in turn and recompute the distances.

Table R2: Sensitivity to memory-boundary recoding, field weights, and attribute removal.

Sensitivity SettingPooled-Group Relations PreservedCross-Group Variant Separation PreservedMinimum Cross-Group Variant Gap (DTSD_{\mathrm{TS}})Minimum Cross-Group Variant Gap (DIRD_{\mathrm{IR}})
Original default settingYesYes0.31090.2472
Memory-boundary recodingYesYes0.31590.2472
Alternative field weightsYesYes0.08120.1428
Leave-one-attribute-outYesYes0.08120.1548

The TS gap is min⁡IRDTS−max⁡TSDTS\min_{\mathrm{IR}}D_{\mathrm{TS}}-\max_{\mathrm{TS}}D_{\mathrm{TS}}, and the IR gap is min⁡TSDIR−max⁡IRDIR\min_{\mathrm{TS}}D_{\mathrm{IR}}-\max_{\mathrm{IR}}D_{\mathrm{IR}}. Positive values indicate strict cross-group separation among variants.

3. VehicleMemBench Mechanisms and Leave-One-Family-Out Ablations (W2, Q3)

  • Mechanisms of the VehicleMemBench variants.

    • VehicleMemBench defines four memory-initialization conditions in a vehicle-control setting. none provides no historical state. gold and summary directly provide preprocessed preference facts or textual summaries in context, respectively. key_value organizes historical state in a structured store accessed through a read-only retrieval tool.
    • The trajectories used for SFT come from the querying and execution stage. summary examples directly receive the summary and contain no explicit memory-write or memory-retrieval calls. key_value examples contain read-only memory queries and vehicle operations, but no memory writes.
  • Leave-one-family-out ablations on Qwen3-8B (Table R3).

    • Using the same training recipe, we fine-tuned Qwen3-8B on the IR pool without VehicleMemBench and the TS pool without GateMem. After removing VehicleMemBench, the IR group still achieves a positive gain of +4.52 pp. Removing GateMem improves TS accuracy from 8.39% to 16.70%, which remains 1.36 pp below baseline.

Table R3: Leave-one-family-out SFT ablations on Qwen3-8B.

Training ConditionRetained ExamplesMean Accuracy (%)Δ\Delta Baseline (pp)
Baseline—18.06—
TS-SFT (Full Pool)3,1458.39−9.68
IR-SFT (Full Pool)4,21724.95+6.88
TS w/o GateMem1,98116.70−1.36
IR w/o VehicleMemBench3,23522.58+4.52

4. Token Exposure Accounting and Multi-Seed Experiments with Token-Matched Subsampling (W2, Q4)

  • Main-experiment token accounting and training settings (Table R4).
    • Thank you for catching the discrepancy between Figure 2 and the text. In the main experiments, both groups share the same training recipe—one epoch, global batch size GBS=8\mathrm{GBS}=8, and the same sequence-packing capacity—rather than matched total or supervised token counts. We apologize for the imprecise “Matched training tokens” shorthand in Figure 2 and will replace it with “Shared training recipe” in the revised manuscript.
    • Training uses dynamic packing: multiple complete trajectories can be packed into a single microbatch with independent attention boundaries (up to the 131,072-token limit, without requiring full occupancy), and one optimizer update can accumulate multiple microbatches.

Table R4: Token counts and optimizer updates for the source pools used in the main experiments.

Applicable ModelsSource GroupTotal TokensSupervised TokensOptimizer Updates
30B, 8B, 4BTS59,867,798371,119393
30B, 8B, 4BIR8,226,4392,133,345527
  • Supervised-token-matched experiments across three training seeds (Table R5).
    • We subsample the IR pool to 371,299 supervised tokens per epoch, matching the TS pool (371,119) to within 0.05%. Training both conditions on Qwen3-8B for two epochs yields cumulative supervised-token counts of 742,598 for IR and 742,238 for TS across three independent seeds.
    • IR-SFT > Baseline > TS-SFT holds for every seed, confirming that the IR-SFT advantage persists under matched supervision budgets.

Table R5: Supervised-token-matched transfer across three training seeds on Qwen3-8B.

Training ConditionSupervised TokensSeed 123 (%)Seed 1234 (%)Seed 2234 (%)Mean Accuracy (%)Δ\Delta Baseline (pp)
Baseline————18.06—
TS-SFT (Token-Matched)742,238 (371,119 / epoch)10.399.1812.6210.73−7.33
IR-SFT (Token-Matched)742,598 (371,299 / epoch)23.8723.0820.4322.46+4.40

5. Source-Side Tool-Use Statistics and Target-Side Memory Behavior Diagnostics (Q5)

  • Source-side trajectory statistics and single-source transfer (Table R6).
    • Frequent tool use alone is insufficient to avoid negative transfer: In the TS group, StructMemEval invokes tools in 99.61% of examples (averaging 4.799 calls per example). However, its interactions primarily involve read-only queries over existing state under a distant interaction realization (DIR=0.6458D_{\mathrm{IR}}=0.6458), and single-source SFT accuracy still falls to 7.74% (−10.32 pp).
    • Explicit storage-writing calls are not necessary for positive transfer: Six of the seven IR variants (63.98% of IR examples) have a 0.00% explicit storage-write rate, yet single-source training on each of the three zero-write IR families—VehicleMemBench, ToolMaze, and TravelBench—achieves positive transfer (+0.58 to +5.02 pp). This suggests that interaction structures such as multi-step action control and feedback handling can also effectively support the target task.

Table R6: Source-side tool use, explicit storage writing, and single-source transfer on Qwen3-8B.

GroupBenchmark Family / VariantExamples with ≥1 Tool Call (%)Examples with ≥1 Explicit Storage-Write Call (%)Mean Tool Calls per ExampleFamily-Level Single-Source SFT Accuracy (%)Δ\Delta Baseline (pp)
TSGateMem / long_context0.000.000.0008.32−9.74
TSStructMemEval / mem_agent99.610.394.7997.74−10.32
TSMemGround / trpg_success0.000.000.0009.25−8.81
TSAMemGym (Family Aggregate)0.000.000.0006.95−11.11
TS↳ AMemGym / native0.000.000.000——
TS↳ AMemGym / awi0.000.000.000——
TSPooled TS Group8.040.030.3888.39−9.68
IRVehicleMemBench (Family Aggregate)100.000.003.02723.08+5.02
IR↳ VehicleMemBench / none100.000.002.679——
IR↳ VehicleMemBench / gold100.000.002.255——
IR↳ VehicleMemBench / key_value100.000.005.163——
IR↳ VehicleMemBench / summary100.000.002.284——
IRToolMaze / legacy_full100.000.008.02318.78+0.72
IRToolMisuseBench / api_model_train100.00100.001.00021.72+3.66
IRTravelBench / legacy_full84.810.003.47018.64+0.58
IRPooled IR Group97.1536.023.47724.95+6.88
Reviewer_zMjvOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies which types of similarity between source and target environments are associated with better transfer when performing supervised fine-tuning on teacher-generated interaction trajectories. The authors distinguish Task Semantics (TS) from Interaction Realization (IR) and propose TIP, a framework that describes environments using attributes such as task goals, capability requirements, action spaces, feedback, and state management, and computes distances along the two dimensions separately.

The experiments compare two pooled source groups: one closer to the target in task semantics and the other closer in interaction realization. The authors generate trajectories using the same teacher, fine-tune three Qwen3 student models separately on each group, and evaluate them on three memory interfaces in BFCL-V4 Memory. IR-SFT outperforms TS-SFT across all models and interfaces and surpasses the original model on two of the three models, whereas TS-SFT produces negative transfer on all three. Through execution examples and a subset of behavioral statistics, the authors further connect the results to behaviors such as memory writing, retrieval, and answer completion.

Critical strengths
  1. The paper addresses a practically relevant question. When selecting training data for agents, similarity in task content does not necessarily mean that the training trajectories contain the operations required by the target environment. Examining task requirements and interaction mechanisms separately provides a useful perspective on why data from related tasks may fail to transfer effectively.
  2. The advantage of IR-SFT over TS-SFT holds across all three student models and all three memory interfaces, with substantial performance gaps. The paper also reports performance degradation after fine-tuning and cases where frequent tool calls fail to lead to a completed answer. These results allow the analysis to cover both transfer gains and behavioral regressions, rather than focusing solely on performance improvements.
Critical weaknesses
  1. The connection between TIP's specific design and the experimental results is insufficiently validated. In the current pipeline, TIP is primarily used to describe source environments and organize the two groups of training data; its attributes and distances are not directly used in subsequent training or evaluation. The paper does not systematically examine the relationship between each source's TS/IR distances and its individual transfer gains, nor does it explain what additional value the framework provides over simpler grouping methods. Meanwhile, the two groups differ in benchmark composition, trajectory content, sample count, and training exposure. The experiments therefore support a difference in transfer performance between these two datasets, but are insufficient to attribute that difference primarily to interaction similarity or to establish whether TIP can effectively guide the selection of new training sources.

  2. The scope of the conclusions remains unclear. All three interfaces belong to the same Memory target, and all three students are from the Qwen3 family. In this target, writing information to persistent memory in advance is an important prerequisite for answering subsequent questions, so matching interaction behaviors may play a particularly prominent role. Whether other types of agent tasks exhibit the same pattern remains to be tested. Although the authors acknowledge these limitations, the abstract's claim of clarifying when trajectories from one benchmark are useful for another still extends beyond what the current experiments directly support. In addition, each condition uses only one training seed; repeated evaluations do not establish robustness to training randomness.

Questions for authors
  1. In the main results, the original 4B model has a higher mean accuracy than the 8B and 30B-A3B models. Have the authors investigated this ordering? Is it primarily due to differences in post-training across checkpoints, memory construction behavior, tool-use strategies, or other factors? How does this observation affect the interpretation of transfer gains across students?
  2. What motivated the choice of these three Qwen3 checkpoints? Were newer-generation models considered when the experiments were conducted? If so, why was the current model set selected? For models with stronger tool-use or memory capabilities before fine-tuning, do the authors expect the same ordering between the source groups?
  3. What specific role did TIP play in determining the final source groups? Could the authors provide the actual TS/IR distances for the source units and explain how these distances determined group membership? Since the experiments mainly compare two pooled datasets, what further evidence could support TIP's attribute and distance design beyond the differences between these particular datasets? What additional explanatory or predictive value does TIP provide over directly grouping environments by features such as tool use and state management?
4 comments
AuthorsOpenReview ↗

Response to Reviewer zMjv

Thank you for the thoughtful review and constructive feedback. We provide additional analyses and experimental evidence to address the key concerns.

Following our main evaluation protocol, every checkpoint is evaluated over three complete repeats.

1. TIP-Guided Source Selection and Its Value Beyond Simple Heuristics (W1, Q3)

  • TIP explicitly guides source selection through task and interaction proximity.
    • TIP profiles environments using eight evidence-grounded fields covering Task Semantics (TS) and Interaction Realization (IR). Their distances, DTSD_{\mathrm{TS}} and DIRD_{\mathrm{IR}}, range from 0 to 1, where smaller values indicate greater source–target similarity.
    • Computed from task specifications and executable protocols before training, these distances guide the construction of contrasting source groups: TS-closer/IR-farther versus TS-farther/IR-closer. Table R1 confirms this separation across all 12 source variants, demonstrating TIP's direct role in selecting training sources rather than merely describing datasets afterward.

Table R1: TIP source-to-target distances for all 12 benchmark variants.

GroupBenchmark Family / VariantDTS↓D_{\mathrm{TS}}\downarrowDIR↓D_{\mathrm{IR}}\downarrow
TSGateMem / long_context0.20830.7500
TSStructMemEval / mem_agent0.28330.6458
TSMemGround / trpg_success0.40830.8750
TSAMemGym / native0.45000.6354
TSAMemGym / awi0.45000.7604
IRVehicleMemBench / key_value0.83330.1875
IRToolMaze / legacy_full0.76090.1875
IRVehicleMemBench / none, gold, summary0.83330.2188
IRToolMisuseBench / api_model_train0.91400.3171
IRTravelBench / legacy_full0.82700.3882

Distances follow Eqs. (5)–(6). VehicleMemBench's none, gold, and summary variants share the reported distances and are combined for brevity.

  • TIP captures interaction compatibility beyond simple tool-use or state-management heuristics:
    • Tool availability does not imply interaction compatibility. TIP distinguishes task-required capabilities (TS) from available actions, feedback mechanisms, and execution protocols (IR). Environments offering similar tools may therefore demonstrate substantially different action–feedback behaviors.
    • State existence does not imply state accessibility. TIP separates the history policy and state persistence attributes within context and state management (Appendix A.4, Table 9), distinguishing how information enters the agent's context from how it is maintained and accessed across stages. This matters when the target requires information to be preserved in memory and accessed after a context reset.

2. Direct Validation of TIP Distances Through Individual-Source Transfer (W1, Q3)

  • Single-source experiments directly connect TIP distances to transfer outcomes.
    • We independently fine-tuned Qwen3-8B on all eight source benchmark families. All four IR-closer sources achieve positive mean transfer, whereas all four TS-closer but IR-distant sources produce negative transfer.
    • Every individual IR source outperforms every individual TS source. Even the weakest IR source (18.64%) exceeds the strongest TS source (9.25%) by 9.39 pp (Table R2).

Table R2: TIP distances and individual-source transfer on Qwen3-8B.

GroupBenchmark FamilyDTS↓D_{\mathrm{TS}} \downarrowDIR↓D_{\mathrm{IR}} \downarrowMean Accuracy (%)Δ\Delta Baseline (pp)
—Baseline——18.06—
TSGateMem0.20830.75008.32−9.74
TSStructMemEval0.28330.64587.74−10.32
TSMemGround0.40830.87509.25−8.81
TSAMemGym0.45000.70316.95−11.11
IRToolMaze0.76090.187518.78+0.72
IRVehicleMemBench0.83330.210923.08+5.02
IRToolMisuseBench0.91400.317121.72+3.66
IRTravelBench0.82700.388218.64+0.58

Family-level distances average constituent benchmark-unit distances from Eqs. (5)–(6), weighted by retained example counts.

  • The results illustrate TIP's additional value for trajectory-SFT source selection.
    • StructMemEval and AMemGym involve interactive memory mechanisms but exhibit larger IR distances and substantial negative transfer (−10.32 and −11.11 pp).
    • Conversely, the semantically distant ToolMisuseBench (DTS=0.9140D_{\mathrm{TS}}=0.9140) is closer in IR (DIR=0.3171D_{\mathrm{IR}}=0.3171), yielding +3.66 pp.
    • These results demonstrate how TIP distinguishes sources with different transfer outcomes beyond task-domain resemblance or coarse tool-use labels.

3. Matched-Token and Multi-Seed Robustness (W1, W2)

  • IR-SFT retains its advantage under matched supervised-token budgets across three independent training seeds.
    • We match Qwen3-8B supervised-token budgets to within 0.05% and repeat training across three independent seeds.
    • IR-SFT > Baseline > TS-SFT holds for every seed, with an average IR–TS advantage of 11.73 pp (Table R3).

Table R3: Supervised-token-matched transfer across three training seeds on Qwen3-8B.

Training ConditionSupervised TokensSeed 123 (%)Seed 1234 (%)Seed 2234 (%)Mean Accuracy (%)
Baseline————18.06
TS-SFT (Token-Matched)742,23810.399.1812.6210.73
IR-SFT (Token-Matched)742,59823.8723.0820.4322.46

Each seed represents an independently trained checkpoint evaluated over three complete repeats.

4. Model Selection Rationale and Behavioral Analysis of the 4B Baseline (Q1)

  • The original Qwen3 checkpoints were selected to cover complementary model configurations within a common family.

    • Parameter scale and architecture. Qwen3-4B and Qwen3-8B are dense models, whereas Qwen3-30B-A3B is a sparse MoE with approximately 30B total parameters but only 3B activated per token, covering different scales and computational architectures.
    • Post-training. Qwen3-4B-Instruct-2507 and Qwen3-30B-A3B-Instruct-2507 use updated 2507 instruction tuning, whereas Qwen3-8B uses the standard Qwen3 checkpoint.
    • Context mechanisms and length. Both 2507 checkpoints natively support 262,144 tokens, while Qwen3-8B natively supports 32,768 tokens, extended to 131,072 through YaRN (factor 4). All three use the same 131,072-token inference window.
  • Checkpoint differences and memory-behavior evidence help explain the stronger 4B baseline.

    • The 4B baseline achieves 21.22%, compared with 18.06% for 8B and 16.85% for 30B-A3B. Its advantage is particularly pronounced on KV (16.13% vs. 7.74% and 8.17%) and RecSum (33.55% vs. 28.39% and 28.82%).
    • The KV advantage is strongly concentrated in historical fact preservation. Across three evaluation repeats, 4B answers 34/75 notetaker questions correctly, compared with 3/75 for 8B and 2/75 for 30B-A3B. These differences account for approximately 79.5% and 86.5% of 4B's KV accuracy advantage over the two models.
    • Memory traces provide a corresponding behavioral explanation. 4B preserves work-related and HR facts absent from the other checkpoints' final memory snapshots. Despite similar KV query-tool call counts for 4B and 8B (36 vs. 38), 4B answers substantially more questions correctly, highlighting the importance of preserving answer-relevant information rather than simply invoking tools.
    • These behavioral differences, together with checkpoint-specific post-training and architecture, help explain why baseline memory performance does not follow nominal parameter size alone.

5. Generalization to Stronger and Cross-Family Baselines (Q2)

  • Newer-generation and cross-family models preserve the IR advantage from stronger baselines.
    • We additionally evaluate Qwen3.6-35B-A3B and GLM-4.7-Flash, whose baselines reach 34.12% and 36.42%.
    • Under matched supervised-token budgets, both retain IR-SFT > Baseline > TS-SFT, with substantial IR–TS advantages of 23.44 pp and 11.90 pp, respectively (Table R4).

Table R4: Supervised-token-matched transfer on newer-generation and cross-family students.

ModelTraining ConditionSupervised TokensMean Accuracy (%)Δ\Delta Baseline (pp)IR−TS (pp)
Qwen3.6-35B-A3BBaseline—34.12——
TS-SFT (Token-Matched)386,62218.35−15.77—
IR-SFT (Token-Matched)387,86941.79+7.67+23.44
GLM-4.7-FlashBaseline—36.42——
TS-SFT (Token-Matched)342,25534.05−2.37—
IR-SFT (Token-Matched)341,91745.95+9.53+11.90

These additional findings will be reflected in the revised manuscript, together with clearer explanations of the key points raised. We hope they provide a stronger basis for reconsidering our work. Thank you again for your time and thoughtful feedback.

Thank you again for your constructive feedback. Building on our earlier response and the November 16 manuscript revision, we provide additional experiments and behavioral analyses below.

6. Source-Composition Ablations (W1)

  • We conduct two targeted source-removal ablations on Qwen3-8B under the same training recipe to isolate individual source contributions.

    • Specifically, we remove VehicleMemBench from the IR group and GateMem from the TS group (Table R5).

Table R5: Targeted source-removal ablations on Qwen3-8B.

Training ConditionRetained ExamplesMemory Accuracy (%)Δ vs. Baseline (pp)
Baseline—18.06—
TS-SFT, full pool3,1458.39−9.68
IR-SFT, full pool4,21724.95+6.88
TS without GateMem1,98116.70−1.36
IR without VehicleMemBench3,23522.58+4.52
  • Source composition alters transfer magnitude without reversing transfer directions.

    • Removing VehicleMemBench (23.3% of IR examples) still yields a +4.52 pp gain over baseline, while removing GateMem raises TS-SFT accuracy by 8.31 pp yet leaves it 1.36 pp below baseline.
    • Combined with our single-source (Table R2) and token-matched (Table R3) results, these ablations confirm that pooled outcomes are not driven by a single dominant source.

7. Extending the Analysis to an Additional Target (W2)

To examine whether the observed source–target relationship extends beyond Memory, we additionally evaluated SciCode on Qwen3-8B using two source groups constructed by TS/IR proximity under matched supervised-token budgets and update counts. As shown in Table R6, TS-SFT drops below the baseline by 10.76 pp, whereas IR-SFT improves performance by 5.44 pp, providing further evidence that the observed TS/IR transfer pattern is not limited to the Memory setting.

Table R6: Step accuracy on the additional SciCode target under matched training budgets.

Training ConditionSource Proximity to SciCodeSource Benchmark FamiliesSciCode Step Accuracy (%)Δ vs. Baseline (pp)
Baseline——25.12—
TS-SFTTS-closer / IR-fartherAlgoTune, DS-1000, QCircuitBench14.35−10.76
IR-SFTTS-farther / IR-closerHumanEval, MBPP, BigCodeBench30.56+5.44

8. Interpreting Source Composition Through Trajectory-Level Interaction Statistics (W1, Q3)

To examine what different source datasets provide as supervision beyond coarse dataset labels, we analyzed retained SFT examples for tool use and explicit storage writes (Table R7):

  • Tool-call presence alone does not explain transfer differences: StructMemEval invokes tools in 99.61% of examples yet yields −10.32 pp negative transfer.
  • Positive transfer occurs without explicit storage-write demonstrations: VehicleMemBench, ToolMaze, and TravelBench contain no explicit storage-write calls in their retained SFT trajectories—for instance, VehicleMemBench uses preprocessed summaries or read-only retrieval in its querying and execution trajectories—yet all achieve positive transfer. In contrast, ToolMisuseBench also achieves positive transfer while including explicit storage writes.

Table R7: Selected source-side interaction statistics and individual-source transfer on Qwen3-8B (Memory baseline: 18.06%).

GroupSource≥1 Tool Call (%)≥1 Explicit Write Call (%)Memory Accuracy (%)Δ vs. Baseline (pp)
TSStructMemEval99.610.397.74−10.32
IRVehicleMemBench100.000.0023.08+5.02
IRToolMaze100.000.0018.78+0.72
IRToolMisuseBench100.00100.0021.72+3.66
IRTravelBench84.810.0018.64+0.58
AuthorsOpenReview ↗

Overall Summary of Our Responses to W1, W2, and Q1–Q3

W1 (TIP Validation and Experimental Controls): Our earlier response provided actual TS/IR distances for all source variants, individual-source transfer results across eight benchmark families, and supervised-token-matched experiments across three independent training seeds. The additional source-removal ablations further examine the contributions of prominent source families to the pooled results.

W2 (Scope of Conclusions): Beyond the original BFCL-V4 Memory target, we have added SciCode as an additional target. Together with the Qwen3.6 and GLM evaluations, these experiments extend our investigation across different target settings and student models.

Q1 (Checkpoint Differences): Our earlier response examined the stronger 4B baseline through checkpoint-specific architecture, post-training, notetaker performance, and memory traces. In particular, the 4B model correctly answers 34/75 notetaker questions, compared with 3/75 and 2/75 for the 8B and 30B-A3B models, respectively, providing behavioral evidence for differences in historical-fact preservation.

Q2 (Student Model Selection): We clarified the complementary model configurations covered by the original Qwen3 checkpoints and extended the evaluation to Qwen3.6-35B-A3B and GLM-4.7-Flash, providing additional evidence across model scales and families.

Q3 (TIP's Explanatory Value): We provided actual TS/IR distances and unit-level profiles, connected source-level proximity to individual-source transfer outcomes, and further examined source-side interaction statistics. These comparisons illustrate why coarse features such as tool-call presence and explicit storage-writing frequency alone are insufficient to explain the observed transfer differences.

We sincerely appreciate your time and thoughtful feedback. We hope these additional analyses and the revised manuscript provide further clarification, and we would be grateful for your further consideration and reassessment.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response: Manuscript Revision and Additional Evidence

Dear Area Chair and Reviewers,

We sincerely thank all reviewers for taking the time to carefully read our paper and provide thoughtful and constructive feedback. We truly appreciate the effort you have invested in understanding our work and identifying areas that could be improved.

We have spent considerable time revisiting the manuscript in light of the reviews, both to strengthen the supporting evidence and to make the overall motivation and presentation clearer. We have now uploaded a revised manuscript with key changes highlighted in blue. The main updates are summarized below:

  1. TIP Profiling and Robustness (Section 4, Appendix A.4). We provide quantitative TS/IR distances, complete unit-level field assignments, and sensitivity analyses across alternative encodings, weights, and attributes, making the source–target comparisons more transparent and verifiable.

  2. Transfer Analysis and Experimental Controls (Section 6.1, Appendices A.5–A.6). We add individual-source experiments across eight benchmark families, source-removal ablations, and supervised-token-matched experiments across three training seeds. The observed 9.39 pp source-level separation and 11.73 pp matched-budget difference strengthen our findings beyond the original pooled comparison.

  3. Model Coverage and Behavioral Insights (Sections 6.2–6.3, Appendices A.3 and A.7). We extend evaluation to stronger and cross-family student models and add quantitative behavioral diagnostics connecting source interaction properties with target-side memory construction, retrieval, and task completion.

Together, these revisions strengthen our investigation of how task semantics and interaction realization jointly inform source-data selection for agent post-training. Our goal is to better understand the complementary roles of these two dimensions in characterizing source–target relationships, rather than to establish that one dimension universally dominates the other.

We genuinely care about this work and have tried to address the reviewers' concerns as carefully as possible while preserving the original research direction. We hope the revised manuscript makes our motivation, methodology, and findings clearer.

As the discussion period approaches its end, we would be very grateful if the reviewers could take another look at the revised manuscript and share their thoughts. In particular, we would appreciate knowing whether the additional evidence and clarifications address the main concerns, or whether there are any remaining issues that we should further clarify.

Thank you again for your time, patience, and thoughtful consideration. We would be very happy to continue the discussion.

Best regards,

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.