acceptodds
Under review as a conference paper at ICLR 2027

TeleTune: Evolving Agent Skills From Offline Telemetry

Abstract

Computer-use agents need to capture procedural knowledge of how people use software. User telemetry, i.e., logs of interface states and user actions, offers a scalable source of this knowledge. However, learning reusable skills from these logs requires addressing three challenges: (1) Goal Underspecification, since logs do not record the goal behind each action; (2) Non-Replayability, since past activity cannot be replayed to evaluate skill updates; and (3) Interleaved Trajectories, since logs may mix several tasks without marking their boundaries. To address these challenges, we introduce TeleTune, a framework for learning a textual skill library from offline logs without recorded goals, cannot be replayed during optimization, and may interleave tasks. The library contains workflows describing cross-instance procedures and skills guiding individual task decisions, both provided as context to the agent at test time. TeleTune uses action-prediction errors on logged trajectories to propose library edits and keep only those that improve held-out action-prediction accuracy, which we call skillguided progress. The learned workflows also enable retrieval of demonstrations that cover the subgoals of a new task. At test time, the agent is provided with the learned library and the workflow-based retrieved demonstrations. Experiments on WorkArena and Online-Mind2Web show that TeleTune outperforms random retrieval, Agent Workflow Memory (AWM), and their combination. We find that the best baseline varies by setting, whereas TeleTune achieves average success rates of 77.1% and 80.6%, respectively, improving over the strongest baseline on each benchmark by 6.7% and 7.7%. Under the heaviest perturbation of the WorkArena training data, TeleTune keeps the highest average success rate at 68.5%, 6.3% above the strongest baseline. Finally, our analyses show that skill optimization and workflow-based retrieval are complementary, that optimizing on fixed logs costs 5 to 75 times fewer tokens than validating the same edits with live episodes, and that skill-guided progress tracks the live success rate.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Agent Skill, LLM Agent, Computer Use Agent, Web Agent

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_JQo6OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

TeleTune learns a workflow library from action logs without explicit goal labels. It infers goals, proposes library changes from action prediction errors, and selects changes using offline valiidation. Experiments on WorkArena and Online-Mind2Web show improved agent performace without replaying the environment during optimization.

Critical strengths
  1. Offline optimization addresses a practical problem: improving agents when repeated environment interaction is costly or unavailable.
  2. Live evaluations and component ablations support the method. For example, WorkArena L1 success rises from 80.3% for the strongest reported baseline to 92.3%.
Critical weaknesses
  1. Evidence linking action prediction quality to task success is narrow: the main analysis uses four checkpoints from one WorkArena L1 run. Matching a logged action may also penalize other valid actions.
  2. Training mainly uses successful demonstrations, while interruptions are synthetic. Online-Mind2Web also uses a filtered subset and exclude blocked episodes, limiting conclusions about noisy real-world logs.
Questions for authors
  1. Does the offline score reliably rank library changes across other tasks, and how are equivalent valid actions handled?
  2. How many episodes are blocked for each method, and do results hold on a shared evaluation subset?
  3. How sensitive is TeleTune to failed demonstrations or incorrectly inferred goals?
4 comments
AuthorsOpenReview ↗

Response to Reviewer JQo6 (Part 1)

We thank the reviewer for noting that "offline optimization addresses a practical problem," and that "live evaluations and component ablations support the method." We address the reviewer's comments below.


 

W1 & Q1: The evidence linking skill-guided progress to task success comes from four checkpoints of one WorkArena L1 run, and matching a logged action may penalize other valid actions.

While we only evaluate four checkpoints in Figure 3, we note that skill-guided progress is an offline approximation and is used only to compare two libraries on the same dev steps. The final libraries selected this way improve live success in every GPT-5.4 setting (Table 1) and with Qwen-3.6 (Table 2).

Regarding valid alternatives, we agree that a predicted action can be a valid next step even when it differs from the logged action. However, two properties of our setup limit the effect. First, we use teacher forcing, so the predictor always continues from the logged history. A valid alternative therefore costs at most one step and does not affect the later steps. Second, we use skill-guided progress only to compare two libraries on the same dev steps. If a step has several valid actions, both libraries lose that step, and the comparison changes only when one library leads the predictor to the logged action. In that case, we reward the library for following the convention of the logged users, which is often the organization's preferred procedure.

To measure how often a mismatch is a valid alternative, we take the 618 mismatched dev predictions on WorkArena L1 and judge each with an LLM judge and manual verification. We then re-score Q with equivalence-aware matching, which also counts a valid alternative as a match.

Benchmark# mismatched dev predictionsMismatches judged to be a valid next step (%)Final-library dev Q (%): original matchingFinal-library dev Q (%): equivalence-aware matching# accept/reject decisions that change (of total steps)Checkpoint ranking changes (yes/no)
WorkArena L161843.573.477.61/22no

On WorkArena L1, 43.5% of the 618 mismatches are valid next steps, so exact matching underestimates Q (73.4% vs. 77.6% with equivalence-aware matching). Re-scoring changes only 1 of 22 accept/reject decisions and leaves the checkpoint ranking unchanged. Therefore, we conclude that valid alternatives only lower the absolute value of Q, but rarely change which library we select.

AuthorsOpenReview ↗

Response to Reviewer JQo6 (Part 2)

W2 & Q2: Training mainly uses successful demos with synthetic interruptions, and Online-Mind2Web uses a filtered subset and excludes blocked episodes; how many episodes are blocked per method, and do the results hold on a shared subset?

Real organizational telemetry with natural interleaving is privacy-sensitive (see our Ethics Statement), and to our knowledge, no public dataset releases such sessions together with a live environment for evaluation. Experimenting on production-grade telemetry would also mean working with data that is not available for others to build on. We therefore simulate user telemetry on publicly available benchmarks, and we try to be as close as possible to actual user telemetry while making sure our work is easy to build upon. Concretely, we remove the goals from all training trajectories, never replay them during optimization, and interleave tasks with a controlled perturbation. Below, we clarify what each benchmark covers.

Online-Mind2Web is the closest to the target setting. The training trajectories were recorded by human annotators on real websites when Mind2Web was collected. They do not have goals during optimization, and they cannot be replayed because the sites have changed since. Thus, this is very close to our motivating setting.** The annotators still performed each task after it was specified to them, so these trajectories do not cover failed or abandoned attempts (which we address below), and TeleTune still improves over the strongest baseline there by +7.7% (80.6 vs. 72.9), with the largest gain on hard tasks (64.7 → 75.3).

WorkArena complements this with a controlled setting in which we can vary the interruption rate. Under the heaviest perturbation (q=1.0), TeleTune keeps the highest WorkArena average (68.5 vs. 62.2 for the strongest baseline), while AWM falls to 49.3 on L1.

Regarding failed or abandoned attempts, TeleTune currently assumes that most logged trajectories reach their goal. Still, we agree that sensitivity to unsuccessful logs should be measured. We therefore mix unsuccessful trajectories into the training pool using Qwen3.6-35B-A3B. We replace a fraction ρ of the training trajectories with (a) failed agent trajectories on the same training task instances or (b) abandoned trajectories truncated at a random step, and rebuild every method from each pool. Results are on WorkArena L1.

Methodclean (ρ=0)failed, ρ=0.5abandoned, ρ=0.5
Random-k55.945.451.7
AWM60.458.156.2
AWM + Random-k63.758.561.1
TeleTune75.672.174.3

TeleTune remains the best method when half of the training trajectories are unsuccessful, staying at least 13.2% above the strongest baseline in each pool.

[Paper Update] We added a limitations section (Appendix G) stating that our training logs are successful demonstrations and that interleaving is controlled. All revisions are marked in blue in the updated PDF.

Regarding the filtered subset, the filter is applied once, before any method is run, and depends only on website category. Since all methods share the filter, it does not favor TeleTune, but we agree that it limits conclusions to in-domain categories. We additionally evaluate on the 178 excluded tasks (out of the 300 Online-Mind2Web tasks in total):

MethodExcluded tasks (178): success rate (%)In-domain tasks (122, Table 1): success rate (%)
Naive57.363.9
Random-k60.972.9
AWM + Random-k61.270.2
TeleTune66.580.6

TeleTune remains the best method on the excluded tasks (66.5% vs. 61.2% for the strongest baseline).

For blocked episodes, we report the number of blocked episodes for each method and two stricter success rates on Online-Mind2Web. Each seed has 122 episodes per method.

(a) Blocked = failure counts every blocked episode as a failure.
(b) The shared unblocked subset holds the tasks that are not blocked for any method in any seed (57 of 122 tasks).

MethodBlocked episodes per seed (of 122)(a) Blocked = failure(b) Shared unblocked subset (57 tasks)
Naive38 / 39 / 3942.1 ± 3.167.3 ± 5.1
Random-k44 / 40 / 4844.8 ± 2.976.6 ± 4.1
AWM46 / 43 / 4743.4 ± 2.573.7 ± 1.8
AWM + Random-k44 / 44 / 4042.6 ± 3.571.9 ± 2.5
TeleTune44 / 44 / 4250.0 ± 2.883.0 ± 3.7

All methods are blocked at a similar rate. TeleTune is blocked on 42 to 44 episodes per seed, which is within the range of the baselines (38 to 48). TeleTune also remains the best method under both stricter success rates. Blocking therefore does not explain TeleTune's gain.

AuthorsOpenReview ↗

Response to Reviewer JQo6 (Part 3)

Q3: How sensitive is TeleTune to failed demonstrations or incorrectly inferred goals?

For failed demonstrations, please see the table in W2 above, where we replace half of the training trajectories with failed or abandoned ones.

Inferred goals are used only by the predictor during optimization, and at test time the agent receives the real task goal. An edit induced by a wrong goal is kept only if it also raises skill-guided progress on held-out trajectories. The goal-inference prompt also forbids describing the action sequence and retries outputs that break its rules (Appendix A.1). To measure sensitivity directly:

(a) Goal quality: agreement between inferred and ground-truth goals on the training trajectories, judged by Qwen3.6-35B-A3B.

Training trajectoriesInferred goal agrees with ground-truth goal (%)
WorkArena L196.7
WorkArena L291.2
Mind2Web88.8

(b) Goal ablation on WorkArena L1 and L2. Each row changes only the goals given to the predictor during optimization.

Goals used during optimizationWorkArena L1WorkArena L2
Ground-truth goals91.862.7
Inferred goals (ours)92.361.8
No goal88.560.3

The inferred goals agree with the ground-truth goals on 88.8% to 96.7% of the training trajectories. Optimizing with inferred goals is within one point of optimizing with ground-truth goals on both levels (92.3 vs. 91.8 on L1 and 61.8 vs. 62.7 on L2). Removing the goal lowers success by 3.8 points on L1 and 1.5 points on L2. TeleTune therefore benefits from having a goal, and the errors in the inferred goals have little effect on the final library.

AuthorsOpenReview ↗

A gentle reminder

Dear Reviewer JQo6,

With only two days remaining in the discussion period, we wanted to kindly follow up on our rebuttal. We have addressed your questions regarding the relationship between skill-guided progress and task success, sensitivity to successful demonstrations, the use of filtered data in Online-Mind2Web, and sensitivity to failed demonstrations or incorrectly inferred goals. We have also revised the manuscript to incorporate your feedback.

If any concerns remain unresolved, please feel free to let us know, and we would be happy to provide further clarification. We would greatly appreciate it if you could revisit your score in light of our responses and additional experimental results.

Thank you again for your time and thoughtful feedback!

Reviewer_NDrqOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The authors propose a new skill induction method for computer-use agents that learns from offline trajectories (figure 2 illustrates it well). The evaluate on two web agent benchmarks, WorkArena and Online-Mind2Web. They test on 1 proprietary LLM (GPT-5.4) and 2 open-weight LLMs (Qwen3.6-35B and Mistral Medium 3.5) and show improvement over Agentic Worflow Memory, which is an older but well cited skill induction method.

Critical strengths
  • Use both proprietary frontier models like GPT-5.4 and self-hostable models like Qwen3.6 and Mistral Medium 3.5.
  • Include both specialized benchmarks like WorkArena++ and open-web benchmarks like Online-Mind2Web
Critical weaknesses
  • Although the focus of the work is on computer use (and the method appears to be general purpose enough), it does not include any non-web benchmarks, such as OSWorld. This work would be stronger if it shows that the method can generalize to CUAs. Otherwise, narrowing down the claim of the paper to focus on web agents would be better.
  • It doesn’t focus on more recent methods like ASI (https://arxiv.org/pdf/2504.06821) and SkillWeaver (https://arxiv.org/abs/2504.07079). AWM is a good baseline but I expect stronger baselines to be considered.
Questions for authors
  • Is there any reason why you excluded more recent skill induction approaches? If you have a good reason for selecting the current benchmarks, I may update my score based.
  • Stronger models like GLM and Kimi families would’ve made the results stronger. What’s the decision making for selecting Mistral Medium 3.5 rather than Kimi or GLM? If you have updated results showing good performance of this method for Kimi or GLM-family models, this method would be more convincing (primarily to reinforce the results for Qwen3.8, which is already promising).
  • How do you handle changes in websites for Online-Mind2Web? What about errors caused by CAPTCHA vs model capabilities; what if one model got unfairly blocked more often than another?
4 comments
AuthorsOpenReview ↗

Response to Reviewer NDrq (Part 1)

We thank the reviewer for noting that we "use both proprietary frontier models like GPT-5.4 and self-hostable models like Qwen3.6 and Mistral Medium 3.5" and "include both specialized benchmarks like WorkArena++ and open-web benchmarks like Online-Mind2Web." We address the reviewer's comments below.


 

W1: No non-web benchmark such as OSWorld is included; the paper should either show generalization to computer-use agents or narrow its claim to web agents.

We thank the reviewer for raising this, and we would like to clarify that TeleTune makes no web-specific assumptions. However, we agree that testing it with baselines on OSWorld will be important for the scope, so we run the naive agent, Random-k, AWM, and TeleTune on 170 OSWorld tasks each. We report the results in the table below.

MethodOSWorld Task Success (%)
Naive41.8
Random-k36.5
AWM34.7
TeleTune34.1

All three methods that learn from logs fall behind the naive agent. Since no method helps, we suspect that computer-use agents in desktop environments differ from web agents in ways that none of these methods handle, and we will therefore narrow our claim from computer-use agents to web agents.

[Paper Update] We narrowed the claims from "computer-use agents" to "web agents" in the first sentence of the abstract, in the sentence of the introduction (Section 1) that introduces TeleTune, in the problem formulation (Section 2.1), and in the conclusion (Section 6). We also discussed desktop environments as future work in the new limitations section (Appendix G), which mentions the OSWorld result above. All revisions are marked in blue in the updated PDF.


 

W2 & Q1: Recent skill-induction methods such as ASI and SkillWeaver are not compared; why were they excluded?

We excluded them because they require conditions that our setting rules out:

  • SkillWeaver (Zheng et al., 2025) learns skills by exploring a website, proposing its own practice tasks, and testing the synthesized APIs by execution.
  • ASI (Wang et al., 2025) induces programmatic skills from online episodes and verifies each one by re-executing it and judging the result with an LLM evaluator conditioned on the task goal.

Both need live environment access during learning and, for verification, a known goal or outcome. TeleTune instead focuses on learning from logs that have no stated goals and cannot be replayed. AWM is the closest applicable baseline because its workflow induction runs offline on recorded trajectories, and we run it on the same goal-free logs.

Despite these differences, following the reviewer's suggestion, we add two comparisons using Qwen3.6-35B-A3B. The first is an offline adaptation of ASI that induces programmatic skills from the same training trajectories with inferred goals but skips execution-based verification. The second is SkillWeaver with live exploration. Note this latter method has privileged access to the environment, which our method and the other baselines we test do not, and which is impossible in our motivating settings.

MethodEnvironment access during learningWorkArena L1WorkArena L2
TeleTuneNone75.670.6
ASI (offline)None37.851.6
SkillWeaver (live exploration)Live environment40.455.9

Due to time constraints, we evaluate ASI (offline) and SkillWeaver on 50 samples for each level. The gap comes mainly from the sorting tasks. TeleTune succeeds on 29 of 30 episodes for each sorting task, while the two baselines mostly succeed on 0 or 1 episodes of these tasks. Their induced functions are rarely called, and they do not help when they are called. On L1, ASI calls a function at least once in only 22% of episodes, and SkillWeaver in only 12% (6 episodes, none of which succeed). The functions that ASI induces are action macros parameterized by element ids (e.g., fill_input_field(bid, value)). They save steps, but they do not store knowledge such as how to perform a multi-column sort.

[Paper Update] We cited ASI and SkillWeaver and added this discussion to the end of the first paragraph of Section 5 (Related Work), marked in blue in the updated PDF.

AuthorsOpenReview ↗

Response to Reviewer NDrq (Part 2)

Q2: Why was Mistral Medium 3.5 selected rather than GLM or Kimi models?

We selected the two additional backbones to differ from GPT-5.4 in family and scale. Qwen-3.6 is a 35B mixture-of-experts model with 3B active parameters, and Mistral-3.5 is a 128B dense model. We agree that more open-weight families would strengthen the claim, and we add GLM5.3-Flash. Following our protocol, we learn skills using GLM itself and test it on WorkArena L1/L2.

WorkArena L1WorkArena L2
Naive90.0%88.0%
AWM + Random-k95.0%90.0%
TeleTune99.0%94.0%

TeleTune is again the best method on both levels. Due to the high cost, we only run 100 samples for L1 and 50 samples for L2, and we spend around $113 on this experiment. We chose GLM over Kimi here because it’s cheaper.


 

Q3: How are website changes handled on Online-Mind2Web, and could CAPTCHA blocking affect methods unequally?

Website changes are part of the setting we target. The training trajectories were recorded when Mind2Web was collected, the live sites have changed since, and TeleTune does not replay them. The learned library is written over visible labels and placeholders rather than selectors (Appendix F), which is why it can transfer to the current versions of the sites. The test tasks come from Online-Mind2Web, whose live tasks are curated for the current sites (Xue et al., 2025). All methods run on the same 122 tasks with the same seeds and within the same time window, so drift on the test side affects all methods alike.

Regarding CAPTCHA, we use the discounted success rate, which excludes blocked episodes. To check that blocking does not favor any method, we report the number of blocked episodes for each method and two stricter success rates on Online-Mind2Web. Each seed has 122 episodes per method.

(a) Blocked = every blocked episode will count as a failure.
(b) The shared unblocked subset, which the tasks are not blocked for any method in any seed (57 of 122 tasks).

MethodBlocked episodes per seed (of 122)(a) Blocked = failure(b) Shared unblocked subset (57 tasks)
Naive38 / 39 / 3942.1 ± 3.167.3 ± 5.1
Random-k44 / 40 / 4844.8 ± 2.976.6 ± 4.1
AWM46 / 43 / 4743.4 ± 2.573.7 ± 1.8
AWM + Random-k44 / 44 / 4042.6 ± 3.571.9 ± 2.5
TeleTune44 / 44 / 4250.0 ± 2.883.0 ± 3.7

All methods are blocked at a similar rate. TeleTune is blocked on 42 to 44 episodes per seed, which is within the range of the baselines (38 to 48). TeleTune also remains the best method under both stricter success rates. When every blocked episode counts as a failure, TeleTune reaches 50.0% vs. 44.8% for the strongest baseline (Random-k). On the 57 tasks that are never blocked, TeleTune reaches 83.0% vs. 76.6%. Blocking therefore does not explain TeleTune's gain.

AuthorsOpenReview ↗

A gentle reminder

Dear Reviewer NDrq,

With only two days remaining in the discussion period, we wanted to kindly follow up on our rebuttal. In our response, we have addressed your questions by including a non-web benchmark, compare with recent skill-induction methods, and adding the GLM model. We have also updated the PDF accordingly to incorporate your comments. If there is anything still unresolved, please let us know, and we are happy to answer. We would be grateful if you would consider revisiting your score in light of our responses. Thank you!

AuthorsOpenReview ↗

A gentle reminder

Dear Reviewer NDrq,

Since this is the last day of the discussion period, we wanted to kindly follow up on our rebuttal. We hope our responses have addressed your concerns, but if anything remains unresolved, please let us know, and we would be happy to provide further clarification.

We would also be grateful if you could take our responses into consideration when reassessing your evaluation.

Thank you!

Reviewer_CiRAOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes TeleTune, a framework for learning workflows and skills from offline trajectories without recorded task goals or environment access during optimization. It infers goals from logged action and observation sequences, proposes skill edits based on action-prediction errors, and accepts edits that improve held-out action prediction accuracy. At inference, the learned library supports retrieval of complementary demonstrations. Experiments on WorkArena and Online-Mind2Web show improvements over demonstration-retrieval and workflow-induction baselines.

Critical strengths
  1. The paper addresses a practical problem: improving agents from existing user activity logs when the underlying task is unknown and the environment is unavailable.
  2. The proposed method leverages action prediction errors under the teacher-forcing setting to guide skill optimization, which makes sense and addresses the challenges of the considered setting.
  3. The evaluation includes live task execution, multiple backbones, and ablations of skill optimization and demonstration retrieval. Reporting offline optimization costs also helps assess practical utility.
Critical weaknesses
  1. The experiments only partially capture the proposed telemetry setting. Training uses successful oracle trajectories or task-specific human demonstrations. Interleaving is simulated by inserting chunks from other trajectories, rather than evaluated on naturally occurring human activity. The use of successful trajectories also leaves unclear how the method would handle failed attempts, abandoned tasks, or ambiguous outcomes. These limitations weaken the claim that ordinary user telemetry can directly support the proposed learning process.
  2. The baselines do not adequately isolate the contribution of the optimization procedure. The main comparisons are against random demonstrations and AWM, neither of which receives comparable iterative optimization using action-prediction feedback.
  3. Handling of interleaved actions is unclear. The paper does not clearly explain whether interleaved action segments are separated, assigned different goals, or excluded from action-prediction feedback. If randomly inserted task switches are included in the feedback, prediction errors may reflect unpredictable interruptions rather than deficiencies in the skill library. The paper does not sufficiently discuss how to address this challenge.
Questions for authors

Please see the weakness.

4 comments
AuthorsOpenReview ↗

Response to Reviewer CiRA (Part 1)

We thank the reviewer for noting that the paper "addresses a practical problem”, that using action-prediction errors under teacher forcing "makes sense and addresses the challenges of the considered setting," and that "reporting offline optimization costs also helps assess practical utility." We address the reviewer's comments below.


 

W1: Experiments partially capture the telemetry setting: training uses successful oracle or human demonstrations, interleaving is simulated, and failed or abandoned attempts are not studied.

We agree that our experiments approximate actual user telemetry. Real organizational telemetry with natural interleaving is privacy-sensitive (see our Ethics Statement), and to our knowledge, no public dataset releases such sessions together with a live environment for evaluation. Experimenting on production-grade telemetry would also mean working with data that is not available for others to build on. We therefore simulate user telemetry on publicly available benchmarks, and we try to be as close as possible to actual user telemetry while making sure our work is easy to build upon. Concretely, we remove the goals from all training trajectories, never replay them during optimization, and interleave tasks with a controlled perturbation. Below, we clarify what each benchmark covers.

Online-Mind2Web is the closest to the target setting. Online-Mind2Web provides no demonstrations, so we draw the training trajectories from the original Mind2Web training set and align them with the test tasks by website category (Appendix D). The training trajectories were recorded by human annotators on real websites when Mind2Web was collected (Deng et al., 2023). They do not have goals during optimization, and they cannot be replayed because the sites have changed since. Thus, this is very close to our motivating setting. The annotators still performed each task after it was specified to them, so these trajectories do not cover the failed or abandoned attempts the reviewer highlights (which we address below), and TeleTune still improves over the strongest baseline there by +7.7% (80.6 vs. 72.9), with the largest gain on hard tasks (64.7 → 75.3).

WorkArena complements this with a controlled setting in which we can vary the interruption rate. Under the heaviest perturbation (q=1.0), TeleTune keeps the highest WorkArena average (68.5 vs. 62.2 for the strongest baseline), while AWM falls to 49.3 on L1.

Regarding failed or abandoned attempts, TeleTune currently assumes that most logged trajectories reach their goal. Still, we agree that sensitivity to unsuccessful logs should be measured. We therefore mix unsuccessful trajectories into the training pool using the same Qwen3.6-35B-A3B model we used in the paper. We replace a fraction ρ of the training trajectories with (a) failed agent trajectories on the same training task instances or (b) abandoned trajectories truncated at a random step, and rebuild every method from each pool. Results are on WorkArena L1.

Methodclean (ρ=0)failed, ρ=0.5abandoned, ρ=0.5
Random-k55.945.451.7
AWM60.458.156.2
AWM + Random-k63.758.561.1
TeleTune75.672.174.3

TeleTune remains the best method when half of the training trajectories are unsuccessful. Its success drops by 3.5 points with failed trajectories and by 1.3 points with abandoned ones, and it stays at least 13.2 points above the strongest baseline in each pool.

[Paper Update] We added a limitations section (Appendix G) stating that our training logs are successful demonstrations and that interleaving is controlled rather than naturally occurring. We also scoped the closing claim of the conclusion (Section 6) to goal-free logs of web tasks and pointed it to Appendix G. All revisions are marked in blue in the updated PDF.

AuthorsOpenReview ↗

Response to Reviewer CiRA (Part 2)

W2: The baselines do not isolate the contribution of the optimization procedure, since neither Random-k nor AWM receives iterative optimization with action-prediction feedback.

We thank the reviewer for this point. The submission contains two controls that hold everything except the optimization fixed.

  1. In Table 3, retrieval is held fixed at random. Adding the optimized library raises success by +18.6 on L1 (65.1 → 83.7), +6.6 on L2 (54.2 → 60.8), and +2.2 on Online-Mind2Web (72.9 → 75.1).
  2. In Figure 3, the first evaluated checkpoint reaches 74.1% on L1. Its library holds only the induced workflows and no skills. This is comparable to AWM (75.7%) and below AWM + Random-k (80.3%). The offline optimization loop then raises the same pipeline to 92.3%.

These show that the gap to the baselines comes from the optimization, not from the initial workflows or the retrieval. We agree that a baseline with comparable iterative optimization makes the comparison stronger. Following the suggestion, we add optimization baselines on the same training logs and backbone:

MethodWorkArena L1WorkArena L2Online-Mind2WebAvg.
(a) TeleTune without the validation gate (accept every proposed edit)90.0%61.6%69.3%73.6%
(b) Trace2Skill w/ one-shot skill distillation from all train-split88.9%62.6%81.2%77.6%
(c) GEPA w/ skill-guided progress as its fitness88.9%60.6%83.3%77.6%
TeleTune92.3%61.8%80.6%78.2%

TeleTune has the highest average (78.2%). Removing the validation gate lowers Online-Mind2Web success from 80.6% to 69.3%, so validating each set of edits on held-out trajectories matters most on live websites. All three optimization baselines learn from the same training logs, and all of them are well above AWM + Random-k on L1 (80.3%). This supports that the gain over the original baselines comes from optimizing the library offline against the logs.


 

W3: It is unclear how interleaved segments are handled, and prediction errors at task switches may reflect unpredictable interruptions rather than gaps in the library.

We thank the reviewer for raising this. We do not separate the interleaved segments or assign them different goals. For each perturbed trajectory, we infer only one task goal, which is the goal of the main task. The chunks of the secondary tasks are used purely as a perturbation. The randomly inserted task switches are not included in the action-prediction feedback, so the prediction errors that the optimizer reads do not come from unpredictable interruptions. Empirically, this is consistent with TeleTune keeping the highest WorkArena average at every interruption rate (Figure 4, Table 7).

The validation step also protects the library against overfitting to unpredictable interruptions. We keep a set of edits only if it strictly raises the mean skill-guided progress on the full dev split (Section 2.2, Table 5). An edit that only fits one arbitrary task switch does not transfer to the held-out trajectories, so we reject it.

[Paper Update] We described how perturbed trajectories are handled in Section 4.2 (two new sentences after the definition of the interruption rate) and in a new paragraph, "Perturbed WorkArena Training Pools", in Appendix D. We also discussed segmentation as a direction for naturally interleaved logs in Appendix D and in the new limitations section (Appendix G). All revisions are marked in blue in the updated PDF.

AuthorsOpenReview ↗

A gentle reminder

Dear Reviewer CiRA,

With only two days remaining in the discussion period, we wanted to kindly follow up on our rebuttal. We have addressed your questions regarding failed or abandoned attempts, added comparisons with two external baselines (Trace2Skill and GEPA), and clarified how our method handles interleaved segments. We have also revised the manuscript to incorporate your feedback.

If any concerns remain unresolved, please feel free to let us know, and we would be happy to provide further clarification. We would greatly appreciate it if you could revisit your score in light of our responses and additional experimental results.

Thank you again for your time and thoughtful feedback!

I appreciate the authors for the detailed response. The added experiments addressed my concern about the telemetry setting. However, comparison with baselines that have comparable optimization cost shows that the improvements of the proposed method are reduced. So I'll maintain my original score.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Additional Rebuttal Experiments

  1. Robustness to failed and abandoned trajectories (CiRA W1; JQo6 W2/Q3)

    We evaluated TeleTune's robustness to unsuccessful training logs by replacing 50% of WorkArena L1 training trajectories with either failed agent trajectories or randomly truncated (abandoned) trajectories. TeleTune achieved 72.1% with failed trajectories and 74.3% with abandoned trajectories, compared with 75.6% on clean demonstrations. It consistently outperformed the strongest baseline by at least 13.2% across these settings, demonstrating robustness to imperfect training data.

  2. Additional skill optimization baselines (CiRA W2)

    We introduced three additional baselines: Trace2Skill (one-shot skill distillation), GEPA (iterative optimization using skill-guided progress), and TeleTune without the validation gate. Across WorkArena L1/L2 and Online-Mind2Web, TeleTune achieved the highest average success rate (78.2%), compared with 77.6% for both Trace2Skill and GEPA, and 73.6% without validation. Removing the validation gate particularly reduced Online-Mind2Web performance from 80.6% to 69.3%, supporting the importance of validating skill edits on held-out trajectories.

  3. Generalization to non-web environments (NDrq W1)

    We evaluated TeleTune and three baselines on 170 OSWorld desktop tasks per method. Since none of the log-based methods improved over the naive agent, these results motivated us to narrow the paper's claims to web agents and identify desktop environments as an important direction for future work.

  4. Comparison with recent skill-induction methods (NDrq W2/Q1)

    We added comparisons with ASI (adapted for offline learning) and SkillWeaver (using live environment exploration), evaluated on 50 WorkArena tasks per difficulty level. TeleTune achieved 75.6%/70.6% on L1/L2, outperforming ASI (37.8%/51.6%) and SkillWeaver (40.4%/55.9%).

  5. Generalization to an additional model family (NDrq Q2)

    We extended our evaluation to GLM5.3-Flash, learning and evaluating skills using the same backbone. On WorkArena L1/L2, TeleTune achieved 99.0%/94.0%, compared with 95.0%/90.0% for AWM + Random-k and 90.0%/88.0% for the naive agent. These results provide additional evidence that TeleTune's improvements generalize across model families.

  6. CAPTCHA blocking and evaluation fairness (NDrq Q3; JQo6 W2/Q2)

    We measured blocked-episode counts across three Online-Mind2Web evaluation seeds and introduced two stricter evaluation protocols. Blocking rates were comparable across methods, indicating that TeleTune's improvements were not attributable to differential CAPTCHA blocking.

  7. Evaluation on excluded Online-Mind2Web tasks (JQo6 W2/Q2)

    To investigate whether the original website-category filtering influenced our conclusions, we additionally evaluated on the 178 Online-Mind2Web tasks excluded from the original 122-task in-domain evaluation. TeleTune achieved 66.5%, outperforming the strongest baseline, AWM + Random-k (61.2%). This demonstrates that TeleTune's benefits extend beyond the originally selected website categories.

  8. Goal inference quality and ablation (JQo6 Q3)

    We measured agreement between inferred and ground-truth goals and compared optimization using inferred, ground-truth, or no goal information. Using inferred goals yielded 92.3%/61.8% success on WorkArena L1/L2, comparable to 91.8%/62.7% with ground-truth goals, while removing goals reduced performance to 88.5%/60.3%. These results demonstrate that goal inference is effective and that its remaining errors have limited impact on downstream performance.


 

Overall

These additional experiments strengthen the empirical evaluation of TeleTune across several dimensions: robustness to unsuccessful logs, comparisons with stronger optimization and skill-induction baselines, generalization across environments and model families, evaluation fairness under CAPTCHA blocking and dataset filtering, and the reliability of inferred goals.

We also revised the manuscript to clarify how interleaved segments are handled, narrow the scope of our claims to web agents, acknowledge the limitations of simulated telemetry and successful demonstrations, and incorporate additional discussion of recent related work. All updates are highlighted in blue.

We thank the reviewers and ACs again for their efforts to help strengthen our work!

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.