- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
TeleTune learns a workflow library from action logs without explicit goal labels. It infers goals, proposes library changes from action prediction errors, and selects changes using offline valiidation. Experiments on WorkArena and Online-Mind2Web show improved agent performace without replaying the environment during optimization.
Critical strengths
- Offline optimization addresses a practical problem: improving agents when repeated environment interaction is costly or unavailable.
- Live evaluations and component ablations support the method. For example, WorkArena L1 success rises from 80.3% for the strongest reported baseline to 92.3%.
Critical weaknesses
- Evidence linking action prediction quality to task success is narrow: the main analysis uses four checkpoints from one WorkArena L1 run. Matching a logged action may also penalize other valid actions.
- Training mainly uses successful demonstrations, while interruptions are synthetic. Online-Mind2Web also uses a filtered subset and exclude blocked episodes, limiting conclusions about noisy real-world logs.
Questions for authors
- Does the offline score reliably rank library changes across other tasks, and how are equivalent valid actions handled?
- How many episodes are blocked for each method, and do results hold on a shared evaluation subset?
- How sensitive is TeleTune to failed demonstrations or incorrectly inferred goals?
4 comments
Response to Reviewer JQo6 (Part 2)
W2 & Q2: Training mainly uses successful demos with synthetic interruptions, and Online-Mind2Web uses a filtered subset and excludes blocked episodes; how many episodes are blocked per method, and do the results hold on a shared subset?
Real organizational telemetry with natural interleaving is privacy-sensitive (see our Ethics Statement), and to our knowledge, no public dataset releases such sessions together with a live environment for evaluation. Experimenting on production-grade telemetry would also mean working with data that is not available for others to build on. We therefore simulate user telemetry on publicly available benchmarks, and we try to be as close as possible to actual user telemetry while making sure our work is easy to build upon. Concretely, we remove the goals from all training trajectories, never replay them during optimization, and interleave tasks with a controlled perturbation. Below, we clarify what each benchmark covers.
Online-Mind2Web is the closest to the target setting. The training trajectories were recorded by human annotators on real websites when Mind2Web was collected. They do not have goals during optimization, and they cannot be replayed because the sites have changed since. Thus, this is very close to our motivating setting.** The annotators still performed each task after it was specified to them, so these trajectories do not cover failed or abandoned attempts (which we address below), and TeleTune still improves over the strongest baseline there by +7.7% (80.6 vs. 72.9), with the largest gain on hard tasks (64.7 → 75.3).
WorkArena complements this with a controlled setting in which we can vary the interruption rate. Under the heaviest perturbation (q=1.0), TeleTune keeps the highest WorkArena average (68.5 vs. 62.2 for the strongest baseline), while AWM falls to 49.3 on L1.
Regarding failed or abandoned attempts, TeleTune currently assumes that most logged trajectories reach their goal. Still, we agree that sensitivity to unsuccessful logs should be measured. We therefore mix unsuccessful trajectories into the training pool using Qwen3.6-35B-A3B. We replace a fraction ρ of the training trajectories with (a) failed agent trajectories on the same training task instances or (b) abandoned trajectories truncated at a random step, and rebuild every method from each pool. Results are on WorkArena L1.
| Method | clean (ρ=0) | failed, ρ=0.5 | abandoned, ρ=0.5 |
|---|---|---|---|
| Random-k | 55.9 | 45.4 | 51.7 |
| AWM | 60.4 | 58.1 | 56.2 |
| AWM + Random-k | 63.7 | 58.5 | 61.1 |
| TeleTune | 75.6 | 72.1 | 74.3 |
TeleTune remains the best method when half of the training trajectories are unsuccessful, staying at least 13.2% above the strongest baseline in each pool.
[Paper Update] We added a limitations section (Appendix G) stating that our training logs are successful demonstrations and that interleaving is controlled. All revisions are marked in blue in the updated PDF.
Regarding the filtered subset, the filter is applied once, before any method is run, and depends only on website category. Since all methods share the filter, it does not favor TeleTune, but we agree that it limits conclusions to in-domain categories. We additionally evaluate on the 178 excluded tasks (out of the 300 Online-Mind2Web tasks in total):
| Method | Excluded tasks (178): success rate (%) | In-domain tasks (122, Table 1): success rate (%) |
|---|---|---|
| Naive | 57.3 | 63.9 |
| Random-k | 60.9 | 72.9 |
| AWM + Random-k | 61.2 | 70.2 |
| TeleTune | 66.5 | 80.6 |
TeleTune remains the best method on the excluded tasks (66.5% vs. 61.2% for the strongest baseline).
For blocked episodes, we report the number of blocked episodes for each method and two stricter success rates on Online-Mind2Web. Each seed has 122 episodes per method.
(a) Blocked = failure counts every blocked episode as a failure.
(b) The shared unblocked subset holds the tasks that are not blocked for any method in any seed (57 of 122 tasks).
| Method | Blocked episodes per seed (of 122) | (a) Blocked = failure | (b) Shared unblocked subset (57 tasks) |
|---|---|---|---|
| Naive | 38 / 39 / 39 | 42.1 ± 3.1 | 67.3 ± 5.1 |
| Random-k | 44 / 40 / 48 | 44.8 ± 2.9 | 76.6 ± 4.1 |
| AWM | 46 / 43 / 47 | 43.4 ± 2.5 | 73.7 ± 1.8 |
| AWM + Random-k | 44 / 44 / 40 | 42.6 ± 3.5 | 71.9 ± 2.5 |
| TeleTune | 44 / 44 / 42 | 50.0 ± 2.8 | 83.0 ± 3.7 |
All methods are blocked at a similar rate. TeleTune is blocked on 42 to 44 episodes per seed, which is within the range of the baselines (38 to 48). TeleTune also remains the best method under both stricter success rates. Blocking therefore does not explain TeleTune's gain.
Response to Reviewer JQo6 (Part 3)
Q3: How sensitive is TeleTune to failed demonstrations or incorrectly inferred goals?
For failed demonstrations, please see the table in W2 above, where we replace half of the training trajectories with failed or abandoned ones.
Inferred goals are used only by the predictor during optimization, and at test time the agent receives the real task goal. An edit induced by a wrong goal is kept only if it also raises skill-guided progress on held-out trajectories. The goal-inference prompt also forbids describing the action sequence and retries outputs that break its rules (Appendix A.1). To measure sensitivity directly:
(a) Goal quality: agreement between inferred and ground-truth goals on the training trajectories, judged by Qwen3.6-35B-A3B.
| Training trajectories | Inferred goal agrees with ground-truth goal (%) |
|---|---|
| WorkArena L1 | 96.7 |
| WorkArena L2 | 91.2 |
| Mind2Web | 88.8 |
(b) Goal ablation on WorkArena L1 and L2. Each row changes only the goals given to the predictor during optimization.
| Goals used during optimization | WorkArena L1 | WorkArena L2 |
|---|---|---|
| Ground-truth goals | 91.8 | 62.7 |
| Inferred goals (ours) | 92.3 | 61.8 |
| No goal | 88.5 | 60.3 |
The inferred goals agree with the ground-truth goals on 88.8% to 96.7% of the training trajectories. Optimizing with inferred goals is within one point of optimizing with ground-truth goals on both levels (92.3 vs. 91.8 on L1 and 61.8 vs. 62.7 on L2). Removing the goal lowers success by 3.8 points on L1 and 1.5 points on L2. TeleTune therefore benefits from having a goal, and the errors in the inferred goals have little effect on the final library.
A gentle reminder
Dear Reviewer JQo6,
With only two days remaining in the discussion period, we wanted to kindly follow up on our rebuttal. We have addressed your questions regarding the relationship between skill-guided progress and task success, sensitivity to successful demonstrations, the use of filtered data in Online-Mind2Web, and sensitivity to failed demonstrations or incorrectly inferred goals. We have also revised the manuscript to incorporate your feedback.
If any concerns remain unresolved, please feel free to let us know, and we would be happy to provide further clarification. We would greatly appreciate it if you could revisit your score in light of our responses and additional experimental results.
Thank you again for your time and thoughtful feedback!
Response to Reviewer JQo6 (Part 1)
We thank the reviewer for noting that "offline optimization addresses a practical problem," and that "live evaluations and component ablations support the method." We address the reviewer's comments below.
W1 & Q1: The evidence linking skill-guided progress to task success comes from four checkpoints of one WorkArena L1 run, and matching a logged action may penalize other valid actions.
While we only evaluate four checkpoints in Figure 3, we note that skill-guided progress is an offline approximation and is used only to compare two libraries on the same dev steps. The final libraries selected this way improve live success in every GPT-5.4 setting (Table 1) and with Qwen-3.6 (Table 2).
Regarding valid alternatives, we agree that a predicted action can be a valid next step even when it differs from the logged action. However, two properties of our setup limit the effect. First, we use teacher forcing, so the predictor always continues from the logged history. A valid alternative therefore costs at most one step and does not affect the later steps. Second, we use skill-guided progress only to compare two libraries on the same dev steps. If a step has several valid actions, both libraries lose that step, and the comparison changes only when one library leads the predictor to the logged action. In that case, we reward the library for following the convention of the logged users, which is often the organization's preferred procedure.
To measure how often a mismatch is a valid alternative, we take the 618 mismatched dev predictions on WorkArena L1 and judge each with an LLM judge and manual verification. We then re-score Q with equivalence-aware matching, which also counts a valid alternative as a match.
On WorkArena L1, 43.5% of the 618 mismatches are valid next steps, so exact matching underestimates Q (73.4% vs. 77.6% with equivalence-aware matching). Re-scoring changes only 1 of 22 accept/reject decisions and leaves the checkpoint ranking unchanged. Therefore, we conclude that valid alternatives only lower the absolute value of Q, but rarely change which library we select.