- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper proposes a training dataset, a training model, and an evaluation benchmark for solving real-world hybrid web tasks that combine search and operation. Specifically, this paper developed HYWECOLLECTOR, an automated method for generating training data; HYWE-35B, a switchable trained model based on Qwen3.6-35B-A3B; and HYWEBENCH: 100 human-written tasks on websites, a benchmark. The experimental results show that the proposed system achieved better performance compared to an untrained Qwen3.6-35B-A3B model and other existing search-based and operation-based agents.
Critical strengths
-
S1. The most significant strength is that the study highlighted the need to distinguish between search and web navigation. Since this aspect had indeed been overlooked in existing research, defining it as a task represents an important contribution.
-
S2. The proposed HYWE framework is indeed a reasonable approach for achieving high performance on this task.
-
S3. The creation of HYWECOLLECTOR and HYWEBENCH represents a significant contribution from the perspective of the dataset.
Critical weaknesses
-
W1. The main concern is how realistic this task is. The task shown in Fig. 1, “Find the book and chapter of the verse ‘Blessed are the poor in spirit,’ then play its …”, is like an artificial combination of two tasks, searching and navigation, that should naturally be separate. It is unclear how practical this is.
-
W2. While there is some contribution in the construction of the training dataset itself, the training method itself is a very naive approach.
-
W3. The comparison methods may not be enough. For example, it is natural that the model outperforms the untrained Qwen3.6-35B-A3B, which is listed as a direct baseline in Table 1. If “Switching” is a key methodological novelty, the results of a method that removes the training signal specifically for “Switching” should be included. It is not strictly clear whether the performance improvement stems from training through “Switching” or from improvements in the capabilities of “Search” or “Operation” themselves.
-
W4. A benchmark of only 100 samples is likely insufficient for accurately assessing the performance of this task in the future. However, this is not a major limitation.
Questions for authors
For more details, please refer to “Critical Weaknesses.” In particular, I would like the authors to clearly distinguish whether the performance improvement is due to switching or simply to training (as in W3).
12 comments
W2. Role of SFT and the Model's Ability to Complete Subtasks Within Each Mode
We adopt a standard SFT paradigm focused on addressing web task behaviors that the base model has not yet fully mastered: consistently completing the current subtask within the appropriate mode and utilizing intermediate results to drive subsequent execution. Simply providing a model with search and operation tools does not automatically mean it can use them effectively to complete tasks; this capability gap is precisely what our framework and training data are designed to bridge.
HyWe organizes execution through mode-specific prompts, toolsets, and a shared history, while our training data provides demonstrations aligned with this inference process. Each training segment supervises all assistant steps within the active mode, covering information retrieval, evidence verification, page operation, and the decision of whether to continue execution, switch modes, or terminate the task. Consequently, SFT trains the model both on how to complete subtasks within each respective mode and on how to leverage previously acquired information to advance subsequent steps.
Our existing results demonstrate the effectiveness of this training. Using the same base model with access to both search and operation tools, the HyWe framework alone raises the task success rate from 11.0% (under naive merging) to 28.0% prior to training; within the same HyWe framework, SFT yields the following further improvements:
| HyWeBench Metric | Base Model | HyWe-35B (Post-SFT) | Gain (Percentage Points) |
|---|---|---|---|
| Task Success Rate | 28.0% | 47.0% | +19.0 |
| Information Subgoal Completion Rate | 74.8% | 82.2% | +7.4 |
| Operation Subgoal Completion Rate | 43.7% | 68.8% | +25.1 |
These results come from Tables 2–3 of the main paper, where both models are evaluated under the HyWe framework. The gains are reflected not only in overall task success rate, but also in the substantially higher completion rate of operation subgoals. The reward chart case study in Appendix A.1 concretely illustrates this difference: the base model successfully locates the relevant information but stalls at the information acquisition stage, whereas the post-SFT model transitions into webpage operation, configures the name, theme, and behavior items, and ultimately fulfills all seven subgoals. This indicates that even after acquiring the required information, the model still needs to learn how to translate that information into subsequent actions.
Training gains also generalize to external web benchmarks: task success improves from 51.7% to 59.0% on Online-Mind2Web and from 76.8% to 88.3% on WebVoyager (Table 1 in the paper). Thus, these improvements are not confined to our proposed HyWeBench.
We believe that the substantial gains achieved via standard SFT demonstrate the practical value of providing supervision tailored to mode-specific execution and subtask hand-offs. Together, the pre-training framework comparison, the within-framework SFT gains, and the improvements on external benchmarks validate the overall effectiveness of our framework and training data design. We will clarify the contributions of our training objective and data organization in the revision; the isolated contribution of switching supervision is further examined via the ablation studies in W3.
W3. Attributing Gains Between Within-Mode Capability Improvements and Mode-Switching Training
We appreciate this insightful question, which seeks to disentangle whether the training gains stem merely from improved standalone search and operation capabilities, or whether dedicated mode organization and switching supervision provide additional benefits. In the original paper, the comparison between the base model and HyWe-35B encompasses both factors; therefore, the improvement from 28.0% to 47.0% should be interpreted as the combined gain of the entire training recipe rather than being attributed solely to switching supervision.
To further disentangle these factors, we conduct two sets of controlled comparisons.
First, we add a naive merging SFT baseline trained with the exact same dataset and training strategy. Specifically, we convert the same training trajectories used for HyWe-35B into a unified prompt format where both search and operation tools are simultaneously available, retaining all task content and actual tool interactions while removing explicit mode isolation and the SWITCH protocol, and fine-tune from the same base model using the identical training setup. Each model is evaluated under its corresponding execution harness. This controlled comparison tests whether HyWe's mode organization and corresponding training format still provide benefits when both models undergo identical web task training.
| Post-SFT Model | Data & Execution Format | HyWeBench Success Rate | Information Completion Rate | Operation Completion Rate |
|---|---|---|---|---|
| Naive merging SFT baseline | Same data & training strategy; unified tool format and execution harness | 38.0 | 78.5 | 54.9 |
| HyWe-35B | Same data & training strategy; HyWe format and execution harness | 47.0 | 82.2 | 68.8 |
We hypothesize that the performance gap primarily stems from the timing of transitioning from search to webpage operation. Given the same demonstrations, HyWe explicitly supervises when to conclude search via SWITCH, and upon switching, continues execution from the accumulated evidence using operation-specific prompts and tools. By contrast, naive merging must implicitly infer this hand-off timing from a continuous stream of mixed tool calls; as a result, even after locating the right page, it often continues searching or answers prematurely, skipping necessary clicks, configurations, or state verifications. Such failures directly lower the operation completion rate and consequently reduce overall task success.
Second, fixing the same HyWe-35B checkpoint, we compare S-only, O-only, and S+O inference settings. S-only and O-only restrict the agent to the tools and prompts of the respective single mode, whereas S+O allows the model to switch autonomously. This comparison evaluates how many tasks the fine-tuned model can solve within a single mode alone versus how much combining both modes further improves completion rates.
| Inference Mode (Same HyWe-35B Checkpoint) | HyWeBench Success Rate | Information Completion Rate | Operation Completion Rate |
|---|---|---|---|
| S-only | 9.0 | 77.0 | 6.88 |
| O-only | 35.0 | 65.4 | 54.9 |
| S+O | 47.0 | 82.2 | 68.8 |
This ablation across S-only and O-only modes on HyWe-35B further confirms that even after fine-tuning—while single-mode performance improves over the base model—operating in a single mode alone cannot match the full S+O performance under the HyWe framework. This demonstrates the substantial benefit brought by the hybrid switching framework itself, and explains why the base model sees more limited gains prior to SFT: it lacks the learned ability to switch to the appropriate mode for subtasks at different stages of execution.
Response to Reviewer fMaL
W1. Realism of Hybrid Tasks and the Dependency Between Search and Operation
We agree that describing the task in Figure 1 as "first look up the book and chapter, then play the video" emphasizes the solution steps rather than fully conveying a unified user goal. What we intend to express is that the user wants to obtain or interact with specific web content, and achieving this goal requires first locating the right target and then operating on it.
Taking Figure 1 as an example, this request is best understood around the unified goal of "playing the American Sign Language (ASL) video corresponding to this scripture quote on Deaf Bible." Given only a scripture excerpt from the user, the agent must first identify that it corresponds to Matthew 5:3, and then locate and play the corresponding video based on that reference. The identified book, chapter, and verse directly determine which page to open and interact with next; merely reporting the scripture reference or a video link does not fulfill the goal of playing the video. Thus, the two stages exhibit a clear information dependency centered on locating and using the same piece of content.
This structure also appears across other cases in the paper:
Notably, the third case comes from the independent Online-Mind2Web benchmark, illustrating that a similar task structure—"finding relevant content and then operating the webpage to obtain the desired result"—naturally exists in established web benchmarks as well. For such requests, the value of information acquisition lies in enabling subsequent operations, while the target of those operations is constrained by the entities and conditions identified earlier.
Furthermore, the human-curated evaluation checklists in HyWeBench score trajectories based on acquired facts and final page outcomes, without requiring specific tools, fixed execution paths, or mandatory calls to SWITCH. Completing the same goal using only the browser is equally judged as a success. Whether to use search and when to transition to page interaction are decided autonomously by the agent based on task progress. Our core research question is how an agent can leverage the complementarity of both capabilities to accomplish these interdependent tasks more effectively.