- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 3
Read review
Paper summary
The paper studies when and how previously discovered workflows should be applied on current RAG-systems. The paper argues that the proper workflow is determined by the state of the retrieved corpus. And they supported this with 3 observation:
- Workflow Reversal: the same workflow helps in one evidence state and hurts in another.
- Noise Amplification Loop: further retrieval under noisy evidence makes the evidence worse.
- Success Mismatch: trajectory success does not reflect a workflow’s incremental gain.
Critical strengths
- Conditioning workflow after evidence retrieval feels very sensible, and to my knowledge, is a novel idea.
- Very solid experiment: (a) All data splits (discovery, tree-growing, certification, RL, development, test) are question-disjoint. (b) The ablations cover the natural alternatives. (c) The paper compares its method with many other systems thoroughly.
- The empirical gain is clear: without any executor training, the method outperforms all other baselines in macro F1, and with RL training, it gets better performance with lower compute.
Critical weaknesses
- Writing and organization. Although the findings are interesting, the paper is quite hard to follow for me. (a) The introduction relies on terms that are only defined in section 4 or the appendix (e.g., paired runs, full-registry control, simultaneous LCB, region–workflow assignment). (b) “Workflow” is only defined in section 4.1, and its meaning differs from common usage. (c) A concrete running example early in the paper of the 3-stage system is highly recommended.
- The workflow is chosen only once after the first retrieval. At deployment, the map is queried once on the initial evidence state, and the chosen workflow stays fixed until execution ends. However, since each retrieval/workflow actions might change the state, it is reasonable to analyse what if we re-choose the workflow after every few retrievals.
Questions for authors
- Did you try recomputing the evidence state and re-querying the map after each retrieval? If not, what prevents it?
- The bare protocol and the prompt-based baselines. The bare protocol reaches ~0.399 macro F1 (Tables 40/43), but it is not shown in Table 1. Most prompt-based baselines score below or around it (IRCoT 0.389, Adaptive-RAG 0.382, CRAG 0.395, SeaKR 0.402). Could you add the bare protocol to Table 1, and explain on whether the ported implementations of these baselines may be under-performing compared with their native versions?
- Improving readability. Could the authors revise the paper to make the method easier to follow? Specifically: (a) add a concrete running example early in the main text that walks one question through the full pipeline (b) define key terms (workflow, paired runs, full-registry control, simultaneous LCB) at or before their first use in the introduction
4 comments
Author Response to Reviewer 9FZi (2/3)
W2. Why select the workflow only once when evidence changes during execution?
Selecting a workflow once keeps action order, queries, and stopping decisions adaptive throughout execution. Your suggested experiment tests whether switching workflows as evidence changes improves performance further.
New evidence: original three-benchmark setting. All variants retain the same library, map, executor, and cumulative budget. Rerouting recomputes the current state, updates guidance and the action mask, and preserves evidence and history.
| Routing frequency | Macro F1 | Utility gain vs. bare (pts, 95% CI) | Switches/query | Extra state evals/query | Latency (s) |
|---|---|---|---|---|---|
| Once after initial retrieval | 0.528 | 9.88 [8.62, 11.13] | 0 | 0.0 | 5.68 |
| After every retrieval | 0.529 | 9.93 [8.71, 11.18] | 0.8 | 2.0 | 5.91 |
| After every two retrievals | 0.527 | 9.79 [8.52, 11.06] | 0.4 | 0.7 | 5.77 |
Routing after every retrieval increases the F1 point estimate by 0.1 points and latency by 0.23 seconds. Once-only routing achieves nearly identical F1 with lower overhead at the original horizon. This is consistent with our discovery process, which distills reusable guidance from complete historical trajectories to support multi-step execution. Certification evaluates complete continuations under the once-selected policy. The switching variants are assessed through their complete test trajectories.
Q1. Have you tried recomputing the state and re-querying the map after each retrieval?
Yes. After each retrieval, the agent preserves its trajectory, recomputes the state, and updates its guidance and allowed actions within the remaining budget. Returning to bare restores all eight actions.
| Original-budget routing | Macro F1 | Utility gain vs. bare (pts, 95% CI) | Query regression (%) | Latency (s) |
|---|---|---|---|---|
| Once | 0.528 | 9.88 [8.62, 11.13] | 0.7 | 5.68 |
| Every retrieval | 0.529 | 9.93 [8.71, 11.18] | 0.8 | 5.91 |
| Every two retrievals | 0.527 | 9.79 [8.52, 11.06] | 0.7 | 5.77 |
On long-horizon BrowseComp-Plus [1], repeated routing gives a larger accuracy point improvement and uses more retrievals:
| Long-horizon routing | Accuracy (%) | Gain over bare (pts, paired 95% CI) | Evidence recall (%) | Retrievals/query | USD/query |
|---|---|---|---|---|---|
| Bare | 38.8 | 0 | 31.4 | 11.2 | 0.264 |
| Once | 40.4 | +1.6 [−0.8, 4.0] | 35.2 | 11.8 | 0.312 |
| Every retrieval | 41.1 | +2.3 [−0.1, 4.6] | 37.4 | 13.6 | 0.355 |
The point improvement from repeated routing is larger at the long horizon: accuracy rises from 40.4% to 41.1%, and evidence recall from 35.2% to 37.4%. Retrievals increase from 11.8 to 13.6 per question. These measurements quantify the benefit and cost of your proposed extension. The original certificate covers once-selected continuations, and the dynamic variants have the direct test results shown above.
[1] BrowseComp-Plus: A More Fair and Transparent Evaluation Benchmark of Deep-Research Agent.
Author Response to Reviewer 9FZi (3/3)
Q2. How does bare compare with the prompt baselines, and could interface adaptation depress their results?
Bare provides the direct reference you request. Its Table 43 scores use the same evaluation setting as Table 1:
| Bare protocol | NQ | HotpotQA | 2Wiki | TriviaQA | MuSiQue | Five-benchmark macro |
|---|---|---|---|---|---|---|
| F1 | 0.450 | 0.439 | 0.455 | 0.526 | 0.123 | ≈0.399 |
| EM | 0.412 | 0.393 | 0.409 | 0.497 | 0.102 | 0.3626 |
| Method | Five-benchmark macro F1 |
|---|---|
| Bare | ≈0.399 |
| IRCoT | 0.389 |
| Adaptive-RAG | 0.382 |
| CRAG | 0.395 |
| SeaKR | 0.402 |
| Frozen REMEDY-RAG | 0.508 |
Bare is an agent with all eight operations. Each baseline uses its published controller with the same executor, corpus, retriever, and budget. The interface is selected before testing by development utility (Section B.1). Here, native means the original prompt and tool-call syntax within this shared setting.
New interface audit:
| Method | Native dev F1 | JSON dev F1 | Selected interface | Invalid calls (%) | Budget exhaustion (%) |
|---|---|---|---|---|---|
| IRCoT | 0.420 | 0.418 | Native | 0.7 | 1.5 |
| Adaptive-RAG | 0.433 | 0.435 | JSON | 1.4 | 3.6 |
| CRAG | 0.419 | 0.417 | Native | 2.4 | 4.1 |
| SeaKR | 0.442 | 0.440 | Native | 1.0 | 2.5 |
| Paired interface recheck | Native minus selected JSON F1 (0–1 scale, 95% CI) |
|---|---|
| Adaptive-RAG | −0.001 [−0.009, 0.008] |
Three baselines use their native interfaces. Native and JSON development F1 differ by only 0.2 points for all four methods. For Adaptive-RAG, the paired recheck gives a native-minus-JSON difference of −0.001, with a 95% CI spanning zero. The selected interfaces therefore preserve closely matched performance in this audit. Table 1 compares the controllers under a common Qwen3-8B executor, retriever, corpus, and budget, including the shared corpus for CRAG's retrieval calls.
Q3. Can you explain the pipeline through one example and define the key terms directly?
Your requested example follows one question through deployment using a workflow learned and certified offline:
- Discover: separate historical traces yield guidance to resolve a missing relation with an entity-specific query. This guidance and its allowed actions form a workflow. The executor decides the sequence and stopping time.
- Verify: paired runs compare that workflow against the same question's full-registry control, which retains all eight actions. A learned evidence region and its candidate form a region–workflow assignment. Independent certification computes simultaneous LCBs for regional mean gains across the frozen assignments. For R5, mean gain is 20.5 utility points and its lower bound is 16.3 (Table 11), supporting admission.
- Assemble: for “Who is the father of the director of Red 2?”, initial evidence supplies the director, Dean Parisot. The state enters R5, activating the missing-hop workflow. The executor retrieves Dean Parisot's father relation, verifies the chain, and answers Aldo Parisot (Figure 11).
Historical questions establish the deployment rule offline. A new question uses one evidence-state lookup and executes the selected guidance.
Hi authors,
Thanks for your response. I have read the response and I remain the positive rating.
Author Response to Reviewer 9FZi (1/3)
Thank you for your time and for your constructive suggestions on clarity and evaluation. We address your questions below with a concrete example, dynamic-routing results, and a direct audit of the baseline interfaces.
F1 uses the 0–1 scale. One utility point equals 0.01U.
Response. The terms you identify have the following operational meanings:
Figure 11 illustrates “Who is the father of the director of Red 2?” Initial retrieval identifies Dean Parisot. The next step is to find his father. Discover produces a missing-hop workflow from separate historical questions. Verify assigns it to R5 using separate growing and certification sets. Its certification gain is 20.5 utility points with LCB 16.3 (Table 11). Assemble maps this example's state (0.60,0.27,0.09,0.33) to R5. The executor decomposes the relation, retrieves “Who is Dean Parisot's father?”, verifies the evidence, and answers Aldo Parisot. The regional certificate comes from independent questions, and this example shows how the learned rule is used at deployment.