- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
ACTS generates search-agent training tasks by controlling evidence acquisition before writing questions. It combines entity and document-navigation hypergraphs, branch selection, and separate decisions for retaining evidence and continuing exploration. Solver trajectories then train student models. Experiments evaluate downstream gains, construction ablations, and comparisons with QUEST.
Critical strengths
- Consistent downstream gains. All three students improve across four benchmarks. For example, Qwen3.8-27B improves from 49.05% to 67.61% on BrowseComp (Table 1, p.8), supporting the practical value of the data recipe.
- Concrete construction controls. The dual hypergraphs, max–min selection over semantic and navigation evidence, and separation of retention from exploration address distinct construction choices, with six ablations (§§3–4). Beyond existing browse-then-write methods such as QUEST, these controls are the main methodological contribution.
Critical weaknesses
- Construction paths may not be necessary for solving the tasks. Sources counts graph URLs supplied to the writer, rather than sources visited by the solver. Node recall of 59.95% and sequence LCS recall of 41.52% show path overlap but do not rule out shorter retrieval routes (Appendix B.5, p.17). This limits the claim that construction controls determine the acquisition difficulty faced by solvers.
- Budget differences and small samples limit attribution. ACTS and QUEST use similar trajectory counts, but total tool calls differ by about 1.79×, and matched token budgets for supervision are not established. On BC-130, the selector answers only four and three more questions correctly than random and entity-only selection (§4.4; Table 2). Data-efficiency and selector gains therefore remain uncertain.
Questions for authors
- Acquisition difficulty: How do sources visited in existing solver trajectories align with construction paths? If needed, a small test with a short retrieval budget and the same solver could assess path necessity; alternatively, please narrow the difficulty-control claim.
- Budgets and uncertainty: Please report supervised tokens, update counts, and construction budgets for ACTS and QUEST. If they differ materially, a small comparison with equal supervision would help. For the selector, please report paired uncertainty using existing BC-130 outcomes.
5 comments
Response to Q2
Budgets and uncertainty: Please report supervised tokens, update counts, and construction budgets for ACTS and QUEST. If they differ materially, a small comparison with equal supervision would help. For the selector, please report paired uncertainty using existing BC-130 outcomes.
We appreciate your suggestions on reporting training and construction budgets and quantifying uncertainty in the selector comparison.
The original comparison already uses comparable supervised-token budgets. Although ACTS has more tool calls, it averages fewer supervised tokens per tool call: approximately 148, compared with 238 for QUEST. On BC-130, ACTS scores 89/130 (68.46%), versus QUEST's 78/130 (60.00%), a gain of 8.46 percentage points.
We additionally compare the two methods under comparable total training-token budgets: 0.990B for ACTS and 1.003B for QUEST. We retain all 4,670 complete ACTS trajectories and shorten training from 198 to 120 optimizer updates, matching QUEST's 120 updates. ACTS scores 84/130 (64.62%), compared with 78/130 (60.00%) for QUEST, retaining a 4.62-percentage-point advantage.
| Quantity | ACTS (Table 3) | ACTS (120 updates) | QUEST |
|---|---|---|---|
| Training trajectories | 4,670 | 4,670 | 4,516 |
| Training tokens | 1.654B | 0.990B | 1.003B |
| Supervised tokens | 220.88M | 132.34M | 200.27M |
| Optimizer updates | 198 | 120 | 120 |
| SFT GPU hours (H200) | 478.00 | 262.46 | 331.80 |
| BC-130 | 89/130 (68.46%) | 84/130 (64.62%) | 78/130 (60.00%) |
These results show that ACTS outperforms QUEST on BC-130 with both a slightly larger but comparable supervised-token budget and a smaller one. Its advantage also persists under comparable total training-token budgets and the same number of optimizer updates.
All three models start from the same Qwen3.8-27B base and use the same training templates. B denotes billion and M million.
ACTS constructs questions from a fixed, offline KILT database. Its document retrieval and reading are performed locally, with no live-web search or external search API calls. ACTS allows at most 27 expansion steps: three policies, three steps per policy per round, and three rounds. During expansion, ACTS uses Kimi-K3 to extract facts for the evidence graphs. Its construction costs also include model inference for question writing and editing (Kimi-K3). QUEST uses Claude Sonnet 4.5 to collect web evidence and write questions. Its construction costs include model inference for evidence processing and question generation, Serper searches, Jina page retrieval, GPT-5-mini page summarization, and GPT-5 evaluation-script generation. Both trajectory sets use the same Kimi-K3 teacher (reasoning effort: max), tools, and maximum of 500 tool calls.
For the selector, we computed paired 95% confidence intervals from the existing Table 2 outcomes on the same 130 questions. Full ACTS answers 59 correctly, random selection 55, and entity-only selection 56.
| Comparison | Accuracy difference | Paired 95% confidence interval |
|---|---|---|
| Full ACTS vs. random | +3.08 points | [-6.15, 12.31] points |
| Full ACTS vs. entity-only | +2.31 points | [-6.92, 11.54] points |
We resample questions while keeping each question's results from the two models together. Both intervals include zero. We will report them with Table 2 so that the four- and three-question differences are presented with their uncertainty.
We hope these responses and the new experiments address the reviewer's concerns, and we would be grateful if the reviewer would consider further strengthening the support for our paper. We are happy to answer any further questions.
Thank the authors for the additional analyses. I maintain my rating of 3 (weak acceptance), with stronger support for the complete data recipe. The remaining uncertainty concerns component-level attribution and the strength of the difficulty-control claim.
Response to Q1
Thank you. We would like to clarify how the sources visited by the teacher align with the construction paths.
For each question, we compare the URLs opened in the teacher's trajectory with the source URLs recorded during that question's construction, excluding the answer's sources. Among the 6,741 trajectories with recorded non-answer sources, 66.37% open at least one of these URLs. Pairing the same trajectories with other questions' construction records reduces this rate to 0.24% and 0.31%. These results show that independent teacher search revisits some of the sources used to construct the same question, and that this overlap is specific to that question.
As suggested, we randomly sampled 100 retained ACTS questions that had passed GLM-5.1's no-tool screening with thinking disabled. We then asked the original teacher, Kimi-K3 (reasoning effort: max), to solve them again under different retrieval budgets. We generated one new trajectory per question per budget and graded the final answer to assess whether the teacher could answer with fewer retrieval calls.
Retrieval calls count searches and page openings (search + open). Find and local Python remain available in every condition and are also included in total tool calls. The normal budget allows up to 500 total tool calls.
With reasoning enabled, Kimi-K3 answers 58/100 questions with zero retrieval and 83/100 under the normal budget. The normal budget yields 13 more correct answers than the setting limited to five retrieval calls.
These results show that a stronger reasoning-enabled model can solve many retained questions without retrieval or with only a few retrieval calls. It may already know information that another model needs to retrieve, or find a shorter route to the answer. The amount of retrieval needed and the solution path therefore depend on the solver. We will make this dependence explicit, limit our acquisition-difficulty claims to the tested models and settings, and interpret construction-path recovery as overlap with the tested teacher's search, rather than a shortest or required solution route.