acceptodds
Under review as a conference paper at ICLR 2027

HyWe: Unifying Search and Operation for Real-World Hybrid Web Tasks

12 views

Abstract

Real-world hybrid web tasks need both information gathering and page interaction, yet search-based and operation-based agents are developed and evaluated separately. To bridge this gap, we introduce HyWe, a framework that unifies both capabilities within one model and trajectory. Its two modes use dedicated prompts and tool sets to sustain sub-goal execution; explicit switching selects the needed capability, and shared history lets each mode build on the other's findings. To provide demonstrations of these behaviors, we develop HyWeCollector, a fully automatic pipeline that uses task prototypes, four switching patterns, and multi-agent synthesis to produce verified hybrid trajectories. We train HyWe-35B on these demonstrations to unify efficient information gathering and browser interaction in a single 35B web agent. To evaluate hybrid task completion, we introduce HyWeBench: human-written tasks on websites distinct from training, with human-written sub-goal checklists and an outcome-based evaluation pipeline. Hybrid training enables HyWe-35B to achieve success on this benchmark, improving on the untrained base model by percentage points and outperforming existing search-based and operation-based agents. Even before training, the same base model solves of tasks with HyWe, versus with all tools under one prompt. Beyond hybrid tasks, HyWe-35B achieves and success on Online-Mind2Web and WebVoyager, outperforming existing open-source baselines and cutting action cost per successful task by and compared with Fara-7B.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM agents, web agents, search agents, hybrid web tasks, mode switching, agentic trajectory synthesis

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_fMaLOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes a training dataset, a training model, and an evaluation benchmark for solving real-world hybrid web tasks that combine search and operation. Specifically, this paper developed HYWECOLLECTOR, an automated method for generating training data; HYWE-35B, a switchable trained model based on Qwen3.6-35B-A3B; and HYWEBENCH: 100 human-written tasks on websites, a benchmark. The experimental results show that the proposed system achieved better performance compared to an untrained Qwen3.6-35B-A3B model and other existing search-based and operation-based agents.

Critical strengths
  • S1. The most significant strength is that the study highlighted the need to distinguish between search and web navigation. Since this aspect had indeed been overlooked in existing research, defining it as a task represents an important contribution.

  • S2. The proposed HYWE framework is indeed a reasonable approach for achieving high performance on this task.

  • S3. The creation of HYWECOLLECTOR and HYWEBENCH represents a significant contribution from the perspective of the dataset.

Critical weaknesses
  • W1. The main concern is how realistic this task is. The task shown in Fig. 1, “Find the book and chapter of the verse ‘Blessed are the poor in spirit,’ then play its …”, is like an artificial combination of two tasks, searching and navigation, that should naturally be separate. It is unclear how practical this is.

  • W2. While there is some contribution in the construction of the training dataset itself, the training method itself is a very naive approach.

  • W3. The comparison methods may not be enough. For example, it is natural that the model outperforms the untrained Qwen3.6-35B-A3B, which is listed as a direct baseline in Table 1. If “Switching” is a key methodological novelty, the results of a method that removes the training signal specifically for “Switching” should be included. It is not strictly clear whether the performance improvement stems from training through “Switching” or from improvements in the capabilities of “Search” or “Operation” themselves.

  • W4. A benchmark of only 100 samples is likely insufficient for accurately assessing the performance of this task in the future. However, this is not a major limitation.

Questions for authors

For more details, please refer to “Critical Weaknesses.” In particular, I would like the authors to clearly distinguish whether the performance improvement is due to switching or simply to training (as in W3).

12 comments
AuthorsOpenReview ↗

Response to Reviewer fMaL

W1. Realism of Hybrid Tasks and the Dependency Between Search and Operation

We agree that describing the task in Figure 1 as "first look up the book and chapter, then play the video" emphasizes the solution steps rather than fully conveying a unified user goal. What we intend to express is that the user wants to obtain or interact with specific web content, and achieving this goal requires first locating the right target and then operating on it.

Taking Figure 1 as an example, this request is best understood around the unified goal of "playing the American Sign Language (ASL) video corresponding to this scripture quote on Deaf Bible." Given only a scripture excerpt from the user, the agent must first identify that it corresponds to Matthew 5:3, and then locate and play the corresponding video based on that reference. The identified book, chapter, and verse directly determine which page to open and interact with next; merely reporting the scripture reference or a video link does not fulfill the goal of playing the video. Thus, the two stages exhibit a clear information dependency centered on locating and using the same piece of content.

This structure also appears across other cases in the paper:

Case & SourceUnified User GoalDependency Between Information and Operation
Deaf Bible, Figure 1Play the ASL video corresponding to a given scripture quoteIdentifying the scripture reference determines the target video page, followed by page interaction to start playback
Kiddy Charts, Appendix A.1Create a reward chart for a child with a specified theme, name, and behavior itemsIdentifying the right generator along with its theme and behavior options informs the subsequent configuration and chart generation
Online-Mind2Web, Appendix A.2Display a comparison chart of female unemployment trends between Illinois and MichiganLocating the corresponding comparison page enables operating the state selectors so the webpage displays the requested two-state comparison

Notably, the third case comes from the independent Online-Mind2Web benchmark, illustrating that a similar task structure—"finding relevant content and then operating the webpage to obtain the desired result"—naturally exists in established web benchmarks as well. For such requests, the value of information acquisition lies in enabling subsequent operations, while the target of those operations is constrained by the entities and conditions identified earlier.

Furthermore, the human-curated evaluation checklists in HyWeBench score trajectories based on acquired facts and final page outcomes, without requiring specific tools, fixed execution paths, or mandatory calls to SWITCH. Completing the same goal using only the browser is equally judged as a success. Whether to use search and when to transition to page interaction are decided autonomously by the agent based on task progress. Our core research question is how an agent can leverage the complementarity of both capabilities to accomplish these interdependent tasks more effectively.

W2. Role of SFT and the Model's Ability to Complete Subtasks Within Each Mode

We adopt a standard SFT paradigm focused on addressing web task behaviors that the base model has not yet fully mastered: consistently completing the current subtask within the appropriate mode and utilizing intermediate results to drive subsequent execution. Simply providing a model with search and operation tools does not automatically mean it can use them effectively to complete tasks; this capability gap is precisely what our framework and training data are designed to bridge.

HyWe organizes execution through mode-specific prompts, toolsets, and a shared history, while our training data provides demonstrations aligned with this inference process. Each training segment supervises all assistant steps within the active mode, covering information retrieval, evidence verification, page operation, and the decision of whether to continue execution, switch modes, or terminate the task. Consequently, SFT trains the model both on how to complete subtasks within each respective mode and on how to leverage previously acquired information to advance subsequent steps.

Our existing results demonstrate the effectiveness of this training. Using the same base model with access to both search and operation tools, the HyWe framework alone raises the task success rate from 11.0% (under naive merging) to 28.0% prior to training; within the same HyWe framework, SFT yields the following further improvements:

HyWeBench MetricBase ModelHyWe-35B (Post-SFT)Gain (Percentage Points)
Task Success Rate28.0%47.0%+19.0
Information Subgoal Completion Rate74.8%82.2%+7.4
Operation Subgoal Completion Rate43.7%68.8%+25.1

These results come from Tables 2–3 of the main paper, where both models are evaluated under the HyWe framework. The gains are reflected not only in overall task success rate, but also in the substantially higher completion rate of operation subgoals. The reward chart case study in Appendix A.1 concretely illustrates this difference: the base model successfully locates the relevant information but stalls at the information acquisition stage, whereas the post-SFT model transitions into webpage operation, configures the name, theme, and behavior items, and ultimately fulfills all seven subgoals. This indicates that even after acquiring the required information, the model still needs to learn how to translate that information into subsequent actions.

Training gains also generalize to external web benchmarks: task success improves from 51.7% to 59.0% on Online-Mind2Web and from 76.8% to 88.3% on WebVoyager (Table 1 in the paper). Thus, these improvements are not confined to our proposed HyWeBench.

We believe that the substantial gains achieved via standard SFT demonstrate the practical value of providing supervision tailored to mode-specific execution and subtask hand-offs. Together, the pre-training framework comparison, the within-framework SFT gains, and the improvements on external benchmarks validate the overall effectiveness of our framework and training data design. We will clarify the contributions of our training objective and data organization in the revision; the isolated contribution of switching supervision is further examined via the ablation studies in W3.

W3. Attributing Gains Between Within-Mode Capability Improvements and Mode-Switching Training

We appreciate this insightful question, which seeks to disentangle whether the training gains stem merely from improved standalone search and operation capabilities, or whether dedicated mode organization and switching supervision provide additional benefits. In the original paper, the comparison between the base model and HyWe-35B encompasses both factors; therefore, the improvement from 28.0% to 47.0% should be interpreted as the combined gain of the entire training recipe rather than being attributed solely to switching supervision.

To further disentangle these factors, we conduct two sets of controlled comparisons.

First, we add a naive merging SFT baseline trained with the exact same dataset and training strategy. Specifically, we convert the same training trajectories used for HyWe-35B into a unified prompt format where both search and operation tools are simultaneously available, retaining all task content and actual tool interactions while removing explicit mode isolation and the SWITCH protocol, and fine-tune from the same base model using the identical training setup. Each model is evaluated under its corresponding execution harness. This controlled comparison tests whether HyWe's mode organization and corresponding training format still provide benefits when both models undergo identical web task training.

Post-SFT ModelData & Execution FormatHyWeBench Success RateInformation Completion RateOperation Completion Rate
Naive merging SFT baselineSame data & training strategy; unified tool format and execution harness38.078.554.9
HyWe-35BSame data & training strategy; HyWe format and execution harness47.082.268.8

We hypothesize that the performance gap primarily stems from the timing of transitioning from search to webpage operation. Given the same demonstrations, HyWe explicitly supervises when to conclude search via SWITCH, and upon switching, continues execution from the accumulated evidence using operation-specific prompts and tools. By contrast, naive merging must implicitly infer this hand-off timing from a continuous stream of mixed tool calls; as a result, even after locating the right page, it often continues searching or answers prematurely, skipping necessary clicks, configurations, or state verifications. Such failures directly lower the operation completion rate and consequently reduce overall task success.

Second, fixing the same HyWe-35B checkpoint, we compare S-only, O-only, and S+O inference settings. S-only and O-only restrict the agent to the tools and prompts of the respective single mode, whereas S+O allows the model to switch autonomously. This comparison evaluates how many tasks the fine-tuned model can solve within a single mode alone versus how much combining both modes further improves completion rates.

Inference Mode (Same HyWe-35B Checkpoint)HyWeBench Success RateInformation Completion RateOperation Completion Rate
S-only9.077.06.88
O-only35.065.454.9
S+O47.082.268.8

This ablation across S-only and O-only modes on HyWe-35B further confirms that even after fine-tuning—while single-mode performance improves over the base model—operating in a single mode alone cannot match the full S+O performance under the HyWe framework. This demonstrates the substantial benefit brought by the hybrid switching framework itself, and explains why the base model sees more limited gains prior to SFT: it lacks the learned ability to switch to the appropriate mode for subtasks at different stages of execution.

Reviewer_X4EZOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper introduces HyWe, a web-agent framework in which one model works in either a search mode or a browser-operation mode, with an explicit SWITCH between them and a shared history. It adds HyWeCollector, an automatic pipeline for generating hybrid training trajectories, and HyWeBench, 100 human-written hybrid tasks. The problem is well motivated, since search and operation agents have complementary strengths, and the results are strong: the same base model rises from 11.0% to 28.0% with the HyWe harness and to 47.0% after fine-tuning, matching frontier models about 70–80 times larger. The pipeline is systematic and the paper is clearly written and detailed.

Due to the novelty of their benchmark, data collection process, agent harness design (splitting into operations and search) and the training mechanism they introduced to adapt models to a new paradigm, I give this paper a weak accept and will consider changing the rating to a strong accept if the authors can address above concerns.

Critical strengths
  • S1. Well-motivated problem. The authors show empirically that search-based and operation-based agents are complementary. In their pilot, a search agent recovers only about 40% of the facts an operation agent obtains through browser interaction, while it solves 29% of sampled tasks within four steps versus 6% for the operation agent, with median operation-to-search cost ratios of 1.9× in steps, 9.0× in tokens, and 6.3× in time.
  • S2. Meaningful empirical results. HyWe-35B more than doubles the best single-mode agent on HyWeBench (47.0% vs. 22.0% for XYZ-Aquila-mini; Table 2) and matches frontier models with roughly 1/80 and 1/69 of their parameters. It also leads the open-source agents on Online-Mind2Web and WebVoyager and improves the base model on deep search (e.g. HLE 30.5 → 47.8; Table 4).
  • S3. Systematic implementation. HyWeCollector is carefully staged: capability probing on 1,110 real websites decides which facts each mode can obtain; 36 search and 48 operation task prototypes and four switching patterns are composed into tasks; a planner–executor–controller setup runs them; and a reviewer and refiner verify completion and remove guidance leaked from the plan before acceptance. Of 25,662 tasks run, 3,735 trajectories are kept.
  • S4. Clear presentation and reproducibility. The paper is well organized, with extensive implementation details (Appendices C, F, G, I, K: algorithms, training settings, prompts, judge prompts).
Critical weaknesses
  • W1. Missing matched unified-tool SFT baseline.

    • The only unified-tool (naive merging) result is 11% for the untrained base model. The trained model, HyWe-35B, is evaluated only inside the HyWe harness. So the 28.0 → 47.0 gain from training is never compared with the gain that the same training would give a unified-tool agent.
    • Suggested experiment: Train both designs on equivalent trajectory content — the same HyWeCollector trajectories, re-rendered into a single-prompt, all-tools format for the unified agent — with the same backbone and training budget, and evaluate each under its native harness. This ablation is useful because it isolates how much of the gain comes from the modes and how much from the training data.
  • W2. Insufficient validation of HyWeCollector.

    • The teacher gets help the student does not. During collection, the executor receives the current phase goal and verified earlier results, and a controller approves each switch and answer against the private plan. The deployed student gets none of this (Algorithm 1). The authors remove leaked guidance with a reviewer and refiner. The actions in each kept trajectory were therefore still chosen with help from a plan the student never sees.
    • The pipeline's benefit is untested. Collection uses frontier models (Qwen3.8-Max and Kimi-K3; Appendix J), and only 3,735 of the 25,662 tasks run are kept after review, refinement, and deduplication (about 15%). The paper never shows that this machinery beats a simpler way of producing the same data.
    • The natural missing baseline - Autonomous teacher rejection sampling: let the same teacher models attempt the same tasks without sub-goals or a controller, keep only the runs that pass the same verification procedure, and match the generation budget. Without this comparison, the benefit of controller-assisted synthesis over simpler trajectory generation remains unclear, both in data yield and in downstream performance.
  • W3. Training-data source contributions are not isolated.

    • Two data sources, one result. The 8,055 hybrid SFT samples are mixed with 7,000 public search-based trajectories from XYZ Agentic Team (2026), adapted to HyWe's format and stored in one shuffled dataset. No ablation shows whether the external data is needed, or how much gain each one of them explains.
    • Why it matters. Without this ablation, it is unclear how much of HyWe-35B's lead over it comes from HyWe's own contributions.
    • Suggested experiments. Hybrid-only, search-only, and combined-data runs, evaluated on HyWeBench, the operation benchmarks, and the deep-search benchmarks, would show which data source drives which gain.
  • W4. Limited analysis of the inference framework and its costs.

    • Four changes at once. The framework comparison (11.0% → 28.0%, Table 3a) changes mode-specific instructions, tool isolation, explicit switching, and shared history together. The authors acknowledge this: "Our ablations assess the framework as a whole" and isolating individual design contributions is left open. Although these mechanisms are partly coupled, targeted ablations (e.g. mode-specific prompts with all tools visible, or isolated tools without specialized prompts) would show which of them drive the improvement.
    • Switching may add hidden cost. Each switch changes the system prompt and tool schema, which may reduce prompt-cache (KV-prefix) reuse at inference. The trained model switches far more often (2.10 switches per task on HyWeBench vs. 0.47 before training; Table 2). Actions per successful task do not capture this; token, latency, or cost measurements would.
  • W5. Each result comes from a single run.

    • Every system runs each HyWeBench task only once, and no repeated runs, standard deviations, or confidence intervals are reported in Tables 2–3. Agent runs on live websites vary from one run to the next. Several headline comparisons are close: HyWe-35B (47.0) vs. Kimi-K3 (46.0) and Qwen3.8-Max (44.0), and the untrained model with both modes (28.0) vs. operation mode alone (26.0). With 100 tasks, one task is one percentage point, so these gaps may be within run-to-run noise.
    • Suggested experiment: Run each system at least three times per task and report the mean and standard deviation (or pass@k), at least for the close comparisons.
Questions for authors

These questions are also meant to be considered as candidates for ablations.

  • Q1. Is the judge reliable? HyWeBench is scored by Kimi-K3 alone and no agreement with human judgment is reported. Kimi-K3 also helps generate the training data and is a competing system (46.0 vs. 47.0 for HyWe-35B). How well does the judge agree with human labels, and do the results change with a different judge?
  • Q2. Does the training work for other models? Only one base model (Qwen3.6-35B-A3B) is trained. Do the gains hold for other model families or sizes?
  • Q3. Why doesn't more switching save actions? On Online-Mind2Web, training raises the share of tasks with a switch from 37.3% to 93.7%, yet actions per success stay flat (30.9 vs. 31.8, Table 1). If switching is meant to save effort, why does it not do so here?
4 comments

W1. Matched Naive-Merging SFT Baseline

We agree that comparing untrained base models does not substitute for a matched post-SFT comparison. We therefore trained a matched naive-merging SFT model starting from the exact same Qwen3.6-35B-A3B backbone, using the exact same training instances and SFT hyperparameters as HyWe-35B.

Specifically, we converted the HyWe-35B training trajectories into the naive-merging format: all search (S) and operation (O) tools are exposed simultaneously under a single unified prompt, removing mode-specific tool isolation and explicit SWITCH actions while concatenating cross-mode segments into a continuous history. All task instructions, tool calls, observations, and final answers are preserved identically. At evaluation, each model uses its corresponding inference framework on the 100 HyWeBench tasks under the same 100-step budget, tool backend, and grading rubric:

Training StatusTraining Data & FormatInference FrameworkTask SR (%)Info CR (%)Op CR (%)
Base (no SFT)—Naive merging11.036.915.8
Base (no SFT)—HyWe28.074.843.7
SFT (Matched Baseline)Same data, naive-merging formatNaive merging38.078.554.9
SFT (HyWe-35B)Same data, HyWe formatHyWe47.082.268.8

While SFT substantially improves the naive-merging model over its base checkpoint (+27.0 points, from 11.0% to 38.0%), HyWe-35B still outperforms the matched naive-merging SFT baseline by +9.0 points in Task SR (47.0% vs. 38.0%) and +13.9 points in Operation CR (68.8% vs. 54.9%) on identical training data. This confirms that mode-decoupled tool spaces and explicit SWITCH coordination yield gains beyond what unified-prompt SFT learns from the same demonstrations.

W2. Autonomous Teacher Rollouts and Role of HyWeCollector

To isolate the role of guided collection, we ran an autonomous teacher rollout baseline on 100 collection tasks by removing the controller and subgoal guidance while keeping the exact same post-hoc verification gate (Sec. 3.2, Eq. 7). Autonomous teacher rollouts achieved only a 4% verification pass rate, compared to 15% for the full HyWeCollector pipeline.

Even strong teacher models struggle to autonomously coordinate multi-stage search-and-operation dependencies, verify intermediate completion criteria, and time mode switches over long horizons; pure post-hoc rejection sampling without in-process constraints suffers from severe sample inefficiency. In HyWeCollector, subgoals specify the immediate stage objective and required handoff states, while the controller checks stage completion against actual observations before permitting mode switches or termination.

Regarding teacher-student information asymmetry: external guidance is used strictly as sampling scaffolding during demonstration collection. Before training, the reviewer/refiner strips all controller prompts and feedback, retaining only the user instruction, executed tool calls, environment observations, and reasoning. The student policy is thus trained exclusively on context available at deployment time.

W3. Individual Contributions of External Search vs. Hybrid Trajectories

We ablated the training mixture on Qwen3.6-35B-A3B under the HyWe inference framework by training exclusively on external search trajectories, exclusively on HyWeCollector hybrid trajectories, and on their combination:

Training ConfigurationExternal Search Traj.Hybrid Traj.Task SR (%)Info CR (%)Op CR (%)
Base model (no SFT)0028.074.843.7
Search-only SFT7,000024.078.515.8
Hybrid-only SFT08,05540.077.258.4
Combined (HyWe-35B)7,0008,05547.082.268.8

Training on search trajectories alone slightly improves Info CR (78.5%) but biases the policy against switching into operation mode, causing Op CR to drop from 43.7% to 15.8% and lowering overall Task SR below the untrained base model (24.0% vs. 28.0%). Training on HyWeCollector hybrid data alone raises Op CR to 58.4% and Task SR to 40.0%, establishing hybrid trajectories as the primary driver of cross-mode coordination. Combining both sources further strengthens deep information retrieval without degrading switching behavior, reaching 47.0% Task SR.

W4. Component Interdependence and Mode-Switching Prefill Cost

(1) Component interdependence. Mode-specific prompts, tool isolation, explicit SWITCH, and shared history form a coupled transition mechanism: prompts and tool isolation constrain the active action space to the current subgoal, SWITCH swaps both simultaneously when task demands change, and shared history passes verified entities and constraints across modes. As shown in W1, removing tool isolation and SWITCH (naive merging) reduces post-SFT Task SR by 9.0 points under matched training data.

(2) Prefill overhead of mode switching. Mode switches rebuild the prompt prefix and incur additional prefill tokens. Assuming intra-mode prefix KV-cache reuse and full history re-prefill upon each switch, we compare theoretical prefill tokens per task between the untrained base model and HyWe-35B:

BenchmarkSwitches / Task (Base → HyWe-35B)Theoretical Prefill (10k tokens/task)Task SR (Base → HyWe-35B)
Online-Mind2Web0.37 → 1.31 (+0.94)3.09 → 3.57 (+15.4%)51.7% → 59.0% (+7.3%)
WebVoyager0.17 → 1.04 (+0.87)2.39 → 2.70 (+13.1%)76.9% → 88.3% (+11.4%)

Although HyWe-35B switches more actively, theoretical prefill increases by only 13.1%–15.4% while improving Task SR by 7.3–11.4 points, because most switches occur early in the trajectory (Type A1 initial O→S routing or Type A2 early S→O fallback) when accumulated history is short. Moreover, compared with operation-heavy GUI baselines, offloading retrieval to compact search calls substantially reduces total prefill volume:

BenchmarkBaseline MethodTheoretical Prefill: Baseline → HyWe (10k tokens/task)Relative Change
Online-Mind2WebGUI-Owl-1.5-8B8.31 → 4.32−48.1%
Online-Mind2WebFara-7B5.17 → 3.64−29.6%
WebVoyagerGUI-Owl-1.5-8B6.02 → 2.16−64.0%
WebVoyagerFara-7B3.78 → 2.34−38.0%

W5. Multi-Run Mean and Variance on HyWeBench

We repeated the evaluation across 3 independent full runs (100 tasks per run) for the primary systems on HyWeBench:

SystemReported Single-Run SR (%)3-Run Mean ± Std (%)
XYZ-Aquila-mini (Search baseline)22.020.7 ± 2.1
Fara-7B (Operation baseline)13.014.3 ± 1.2
Qwen3.6-35B-A3B (HyWe S+O)28.028.3 ± 2.5
Kimi-K3 (HyWe S+O)46.044.3 ± 4.7
Qwen3.8-Max (HyWe S+O)44.045.7 ± 3.8
HyWe-35B (HyWe S+O)47.046.7 ± 4.7

All single-run numbers reported in the paper fall within one standard deviation of the 3-run mean, and relative model rankings remain unchanged.

Q1. Judge Reliability, Human Agreement, and Cross-Judge Stability

Because Kimi-K3 was used in both data synthesis and HyWeBench grading, we re-evaluated the exact same HyWe-35B trajectories using three external LLM judges (Claude-Sonnet-5-5, GPT-5.6-Luna, Qwen3.8-Max) and two independent human experts (h1, h2):

JudgeClaude-Sonnet-5-5GPT-5.6-LunaKimi-K3Qwen3.8-MaxHuman h1Human h2
Task SR (%)44.045.047.050.046.045.0

Pairwise Judge Agreement (%):

JudgeQwen3.8-MaxKimi-K3GPT-5.6-LunaClaude-Sonnet-5-5h1h2
Qwen3.8-Max100%83%83%84%88%87%
Kimi-K383%100%88%89%93%92%
GPT-5.6-Luna83%88%100%89%93%92%
Claude-Sonnet-5-584%89%89%100%94%93%
h188%93%93%94%100%97%
h287%92%92%93%97%100%

The two human experts reach 97% agreement (45.0%–46.0% SR). Aside from Qwen3.8-Max being slightly more lenient (50.0% SR, 87%–88% human agreement), Kimi-K3, GPT-5.6-Luna, and Claude-Sonnet-5-5 all achieve 92%–94% agreement with human experts and yield closely aligned success rates (44.0%–47.0%), showing no evidence of self-preference inflation from Kimi-K3.

Q2. Effectiveness on a Smaller Backbone (Qwen3.5-9B)

To test transferability to smaller models, we trained HyWe-9B from Qwen3.5-9B using the same data and recipe and evaluated both under the HyWe S+O framework on HyWeBench:

ModelTask SR (%)Info CR (%)Op CR (%)
Qwen3.5-9B (Base)8.064.813.5
HyWe-9B (SFT)30.0 (+22.0)72.8 (+8.0)34.3 (+20.8)

Despite the weaker zero-shot capacity of the 9B backbone (8.0% SR), HyWe SFT improves Task SR by +22.0 points and Op CR by +20.8 points, demonstrating that our data and training recipe generalize across model scales.

Q3. Why Overall Steps/SR Remains Similar on Online-Mind2Web Despite More Switching

The Steps/SR metric in the main paper divides total steps across all episodes (including failed ones) by the number of solved tasks, and HyWe-35B solves additional harder, longer-horizon tasks where the base model fails. To compare efficiency on equal footing without survivorship bias, we examine the 116 tasks solved by both Base and HyWe-35B:

Per-Task Mean on 116 Commonly Solved TasksBaseHyWe-35BChange
Total Steps18.0316.44−1.59 (−8.8%)
Mode Switches0.371.31+0.94
O-Side (Operation) Calls14.488.46−6.02
S-Side (Search) Calls2.175.67+3.50

On commonly solved tasks, HyWe-35B reduces average total steps by 8.8% (18.03 → 16.44): taking +0.94 more switches and +3.50 S-side search calls eliminates 6.02 costly O-side GUI actions per task on average.

Reviewer_X4EZOpenReview ↗

Final response to rebuttal

I thank the authors for the extensive new experiments, especially the SFT baseline, which convincingly shows that the mode-based design outperforms unified-prompt training on identical data (47.0 vs. 38.0). The SFT baseline, data-source ablation and repeated runs resolves W1, W3, and W5. However, since the core idea combines existing search and operation tool sets, W2 and W4 remain only partly addressed, and the repeated runs show HyWe-35B on par with rather than ahead of Kimi-K3 and Qwen3.8-Max, I maintain my score of weak accept.

Reviewer_4PegOpenReview ↗
Rating
3
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

The paper argues that many real web tasks are hybrid, involving the use of both search-tools for efficient information gathering, as well as live page interaction, when a browser interaction is needed. The paper's main contributions are: A) a framework for web information gathering agent to work in both search-based and operation-based mode, B) an automatic data generation pipeline to synthesize trajectories for this hybrid approach of search + page interaction. SFT with the generated data considerably improves the Qwen3.6-35B-A3B on multiple web information gathering benchmarks.

Critical strengths

The pilot study is nicely done to motivate the need for switching between search-based and operation-based modes. The data generation process is pretty well engineered to output high quality training data to use for SFT.

Critical weaknesses

Overall there is limited novelty in the work, and there is prior work [1] that has tried to address this similar work. The primary contribution is mainly the well engineered data generation pipeline to get high quality SFT trajectories.

It should be investigated in more detail why the framework (which is claimed as one of the main contributions) by itself adds little value. S+O beats O-only by only 2 points (28.0 vs. 26.0), so majority of the gain is coming from SFT on the trajectories.

The authors should demonstrate that the SFT trajectories is not overly targeted to the eval benchmarks. For this, it would be good to show that the model's general capabilities aren't affected from this training (this can be done by showing that performance on other benchmarks like coding)

[1] Beyond Browsing: API-Based Web Agents

Questions for authors

Given that HyWeBench has only 100 examples, it is recommended to do multiple runs per system on this approach and report the mean / variance across runs.

The same modes are used to generate the data while also used as the judge for the HyWeBench benchmark. It should be clarified where there is any issue of self-preference bias for this benchmark given it is LLM graded.

Would be good to show ablations that show the benefit of using the 7,000 XYZ search trajectories vs the hybrid trajectory data.

4 comments

W1. Novelty and Distinction from Beyond Browsing

We thank the reviewer for pointing out the connection to Beyond Browsing [1]. Here, it is important to distinguish the capabilities carried by different types of APIs: calling site-specific backend APIs and invoking a real search engine for open-web retrieval address fundamentally different problems.

Beyond Browsing invokes site-specific backend APIs via Python to access site data and perform in-site operations alongside browser GUI interactions. Under our capability taxonomy, this primarily falls under programmatic site access and manipulation—i.e., the Operation (O) side. Moreover, code-driven web interaction is a standard tool modality already supported by mainstream frameworks such as browser-use and Playwright. In contrast, HyWe's Search (S) mode queries the live Google search engine via Serper combined with webpage reading to perform cross-website information retrieval, whereas its Operation (O) mode handles subgoals that require live page interactions.

Comparison DimensionBeyond Browsing [1]HyWe
API Type & RoleSite-specific backend APIs accessed via Python for site functions and dataLive Google search engine queried via Serper, combined with webpage reading
Coordinated CapabilitiesProgrammatic site backend access/manipulation and browser GUI interactionOpen-web search (S) and webpage operation (O)
Web Task EvaluationSelf-hosted WebArena built on controlled website replicasHyWeBench, Online-Mind2Web, and WebVoyager on live public websites
Open-Web Search EvaluationNo standalone open-web search or deep-research benchmark evaluation reportedHLE, xbench-DeepSearch-2510, and BrowseComp-EN

Therefore, although both works involve APIs and browsers, their research focuses differ. Beyond Browsing studies how to leverage site-provided programmatic interfaces to complete web tasks, whereas HyWe focuses on the complementarity between open-web search and webpage operation—specifically how to select capabilities, switch modes, and pass accumulated evidence across modes in real-world tasks. While WebArena tasks may include in-site search and information lookup, this remains distinct from cross-website open-web search evaluation.

Our core contributions lie in: (1) designing mode-specific prompts and toolsets, an explicit SWITCH action, and a shared cross-mode interaction history; (2) introducing HyWeCollector, which organizes training trajectories across four switching patterns to train the model on completing subgoals and deciding when to switch; and (3) building HyWeBench to separately evaluate the completion of information-acquisition and operation subgoals.

W2. Contribution of the Framework Itself and Post-SFT Single-Mode Ablations

The reviewer notes that the base model achieves a 28.0% success rate under S+O, which is only 2 percentage points higher than the 26.0% achieved under O-only. While this gap is modest, it is important to distinguish between two comparisons: the gain from adding the search mode versus the gain from the framework's organizational structure itself.

Using the same base model with access to both search and operation tools, naively merging both toolsets into a single prompt yields a success rate of 11.0%, whereas adopting the HyWe framework raises it to 28.0% (Table 3a in the paper). This improvement occurs prior to our SFT, demonstrating that the mode-organized framework itself already brings substantial benefits. By contrast, the comparison between 26.0% and 28.0% varies the available modes within HyWe, examining whether an un-finetuned base model can further exploit search capabilities on its own.

From the perspective of subgoal metrics, adding search increases the base model's information completion rate from 57.6% to 74.8%, but its operation completion rate drops slightly from 47.8% to 43.7%. This indicates that acquiring more information does not automatically translate into more complete page operations. We view this as evidence that the base model remains limited in retrieval, operation, and the hand-off between the two, which underscores the value of training on high-quality trajectories to adapt the model to this workflow. After SFT, the information completion rate reaches 82.2%, the operation completion rate reaches 68.8%, and the overall task success rate reaches 47.0%.

To further disentangle single-mode capability improvements from the gains of combining both modes after training, we supplement an ablation evaluating the exact same HyWe-35B checkpoint under S-only and O-only settings (where S-only and O-only retain only the prompts and tools of the respective mode while removing SWITCH and instructions for the other mode):

Model CheckpointFramework / ModeTask Success RateInformation Subgoal Completion RateOperation Subgoal Completion Rate
Base ModelNaive merging of S+O tools11.036.915.8
Base ModelHyWe, S-only7.073.815.8
Base ModelHyWe, O-only26.057.647.8
Base ModelHyWe, S+O28.074.843.7
HyWe-35BHyWe, S-only9.077.018.6
HyWe-35BHyWe, O-only35.065.454.9
HyWe-35BHyWe, S+O47.082.268.8

This ablation across S-only and O-only modes on HyWe-35B further confirms that even after fine-tuning—while single-mode performance improves over the base model—operating in a single mode alone cannot match the results under the HyWe S+O dual-mode setting (47.0% vs. 35.0%). This demonstrates that the dual-mode switching mechanism itself raises the performance ceiling substantially, whereas the base model sees only a modest gain under S+O compared to the single-mode (O-only) setting (28.0% vs. 26.0%) because it has not yet acquired the ability to switch to the appropriate mode across different subgoal stages.

W3. Evaluation Specificity and the Trade-off Between Domain Adaptation and General Capabilities

We acknowledge that HyWe-35B shows some performance degradation on certain general-purpose tasks compared to the Qwen3.6-35B-A3B base model. We selected three benchmarks across distinct domains from those evaluated in the official Qwen3.6-35B-A3B model card, with our measured results reported below:

BenchmarkBase Model AccuracyHyWe-35B AccuracyHyWe − Base (Percentage Points)
GPQA86.077.7−8.3
LiveCodeBench v680.479.4−1.0
RealWorldQA85.375.2−10.1

Our current SFT uses only 15K pure multi-step tool-use trajectories without mixing in any general instruction or subject-reasoning replay data; consequently, experiencing distribution shift on single-turn disciplinary QA (GPQA) and static visual QA (RealWorldQA) is a common phenomenon in pure domain-specific SFT, whereas performance on LiveCodeBench v6—which similarly tests multi-step logic—remains virtually intact (−1.0). We believe that, weighed against the gains in the target domain, this trade-off is acceptable given our positioning of HyWe-35B as a specialized model for web tasks: on HyWeBench, task success rises from 28.0% to 47.0% and operation subgoal completion improves from 43.7% to 68.8%, accompanied by improvements on Online-Mind2Web, WebVoyager, and multiple deep-search benchmarks. Notably, these gains are not confined to our in-house benchmark. While training tasks in HyWeCollector are automatically synthesized and verified, the tasks and subgoal checklists in HyWeBench are manually authored and evaluated on websites disjoint from those used for trajectory collection; together with the results on external benchmarks, these settings provide evidence of generalization within the web domain.

Accordingly, we conclude that domain-specific SFT substantially enhances the model's web search, operation, and coordination capabilities at the cost of a moderate reduction in general-purpose benchmark retention; for our target application of specialized web agents, we consider this trade-off reasonable and acceptable. Incorporating general instruction data mixing and related training recipes to better preserve general capabilities and output stability is a promising direction for future work.

Q1. Multiple Evaluation Runs and Mean / Variance on the 100-Task Benchmark

We repeated the evaluation of the main systems on HyWeBench across 3 independent runs, aggregating scores over the full 100-task set per run and reporting the mean and standard deviation of the task success rate:

SystemSingle-Run Success Rate (Paper)Success Rate (Mean ± Std)
XYZ-Aquila-mini, Search Baseline22.0%20.7 ± 2.1
Fara-7B, Operation Baseline13.014.3 ± 1.2
Qwen3.6-35B-A3B, HyWe S+O28.028.3 ± 2.5
Kimi-K3, HyWe S+O46.044.3 ± 4.7
Qwen3.8-Max, HyWe S+O44.045.7 ± 3.8
HyWe-35B, HyWe S+O47.046.7 ± 4.7

As shown above, the multi-run averages closely align with the single-run results reported in the main paper, confirming that our conclusions hold reliably across multiple runs.

Q2. Judge Model Self-Preference and Result Stability Across Alternative Judges

To examine whether our evaluation is affected by judge-specific bias, we re-evaluated the exact same set of HyWe-35B trajectories on HyWeBench using three additional LLM judges (Claude-Sonnet-5-5, GPT-5.6-Luna, and Qwen3.8-Max) alongside two human experts (denoted as h1 and h2), and compared their judgments against our original Kimi-K3 judge.

JudgeTask Success Rate
Claude-Sonnet-5-544.0
GPT-5.6-Luna45.0
Kimi-K347.0
Qwen3.8-Max50.0
h146.0
h245.0

Pairwise Judge Agreement Rate.

JudgeQwen3.8-MaxKimi-K3GPT-5.6-LunaClaude-Sonnet-5-5h1h2
Qwen3.8-Max100%83%83%84%88%87%
Kimi-K383%100%88%89%93%92%
GPT-5.6-Luna83%88%100%89%93%92%
Claude-Sonnet-5-584%89%89%100%94%93%
h188%93%93%94%100%97%
h287%92%92%93%97%100%

First, different models indeed exhibit slight variations when evaluating trajectories against subgoal checklists, which is related to their intrinsic biases and model capabilities. Among the models above, except for Qwen3.8-Max—which shows lower agreement with the human annotators—all other models achieve over 90% agreement with the human experts, so we believe that Kimi-K3, GPT-5.6-Luna, and Claude-Sonnet-5-5 all provide reasonable and reliable judgments.

Q3. Respective Contributions of External Search Trajectories and Hybrid Trajectories

We added an ablation comparing models trained exclusively on external search trajectories, exclusively on HyWeCollector hybrid trajectories, and on the combined mixture of both, alongside the un-finetuned base model for reference:

Training ConfigurationExternal Search TrajectoriesHybrid Training SamplesHyWeBench Success RateInformation Completion RateOperation Completion Rate
Base Model (No SFT)0028.074.843.7
External Search Data Only7,000024.078.515.8
Hybrid Data Only08,05540.077.258.4
Combined Data (HyWe-35B)7,0008,05547.082.268.8

As shown in the table, training solely on external search data actually degrades overall task success compared to the base model (24.0% vs. 28.0%): because it only reinforces search behavior, the model rarely switches to the operation mode (causing operation completion to drop to 15.8%). Conversely, training solely on hybrid data yields a strong overall improvement (40.0%) and exhibits mode-switching behavior similar to HyWe-35B, but lacks the additional search proficiency provided by the external search trajectories, resulting in a lower overall success rate than the full HyWe-35B model (47.0%).

Comments on the paper

1 comment
AuthorsOpenReview ↗

Note to Area Chair: Revisions Summary & Clarification on Reviewer fMaL

Dear Area Chair,

First, we would like to express our sincere gratitude to all the reviewers for their thoughtful questions and constructive suggestions. We have completed the requested additional experiments and incorporated the results into our revised manuscript, which has significantly strengthened the completeness and rigor of our work.

Second, we would kindly like to draw your attention to our final exchange with Reviewer fMaL. While we genuinely appreciate their feedback, we feel there may remain a slight misunderstanding regarding the core contribution of our work, and we would like to briefly clarify two key points:

  1. Switching frequency in HyWeBench: Reviewer fMaL noted that 81% of the reference trajectories in HyWeBench involve only a single switch. We would like to clarify that this proportion was intentionally designed to keep the benchmark's difficulty well-calibrated; moreover, actual evaluation is based on the completion of human-annotated subgoals rather than strictly matching the reference switching pattern.

  2. Lightweight router vs. unified modeling: Reviewer fMaL suggested simply connecting separate search and web-operation agents via a lightweight router. Respectfully, this perspective touches directly on the central challenge our work aims to address, as supported by two key observations:

    • Directly combining these three components is insufficient to cover the cross-modal capabilities required by the task. As shown in the capability comparison table in our response, a router would need to integrate interaction histories from both modes to decompose tasks, assess progress, and determine handoff timing. Likewise, each single-mode agent must comprehend observations from the other mode to continue executing the correct subtask after handoff. Consequently, all three components would still need to acquire the corresponding cross-modal capabilities.
    • Direct supervision of SWITCH is critical to HyWe's success rate. Holding the training data, remaining training settings, and inference framework constant, simply masking the training loss on SWITCH outputs causes the task success rate to drop from 47.0% to 30.0%, approaching the base model's 28.0%, even though the model can still invoke SWITCH autonomously at inference time. Further details can be found in our Capability Comparison Table and Ablation Results.

HyWe's core contribution is to jointly learn search, web operation, and cross-modal coordination within a single unified model—enabling it to decompose tasks, switch at appropriate junctures, and leverage observations from the other mode to continue execution. Thank you very much for your time and for taking these analyses and experimental findings into account in your holistic evaluation of our paper.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.