- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
Answer2Path uses known answers or desired states to guide web-agent trajectory acquisition. It audits webpage evidence, induces reusable workflows, and evaluates them on held-out tasks without outcomes. Experiments on WebArena Shopping also examine guidance timing, modification tasks, and a small loop for generating new task–outcome pairs.
Critical strengths
- Complementing reverse task synthesis such as OS-Genesis, the paper tests whether known outcomes yield experience that remains useful after those outcomes are removed. Fixed-K query transfer improves from 88.3% to 97.2% with identical workflow induction, and improves across all three source seeds.
- The symmetric evidence check also improves supported yield from 29.0% to 45.4%, beyond official-success gains alone. Leakage audits, token accounting, modification tasks, and independent scaffold/backbone replications give the acquisition finding substantial practical support.
Critical weaknesses
- The main fixed-K study does not include Full GT-only acquisition. Smaller placement controls are informative, but the added transfer value of explicit path-seeking guidance over simply revealing outcomes remains less certain than the overall outcome-conditioning benefit.
- The semantic evidence audit is incomplete: 124/155 combines literal passes with semantic review of failures. The symmetric mechanical audit supports improvement, but this provisional figure cannot establish the full rate of genuinely evidence-supported trajectories.
- Modification transfer includes A2P-only compact compilation. As disclosed, it evaluates the combined pipeline; it cannot independently confirm that better acquired trajectories caused the transfer gain. The clean query result carries the stronger evidence here.
- Pooled tests reuse tasks across seeds and nested K values, with shared workflows. The positive per-seed results are reassuring, but the clustering unit and effective sample size need clarification before interpreting the reported significance.
- Minor corrections: Table 11's A2P + same-family gain is 14.2 percentage points, rather than 14.1. Figure 3 should distinguish flywheel admission from the main experiments' eligibility rule. Keep the seven-pair expansion result explicitly framed as a feasibility pilot.
Questions for authors
- Can the existing Full GT-only trajectories be evaluated with the same fixed-K protocol to clarify the incremental value of path-seeking guidance?
- What does a condition-blind semantic audit of a matched sample from both methods show, including literal-check passes and failures?
- How do the confidence intervals and paired tests account for repeated tasks, nested K values, and shared workflows?
- How much of modification transfer remains when both conditions use identical workflow compilation?