acceptodds
Under review as a conference paper at ICLR 2027

Not Every Workflow Helps: Discover, Verify, and Assemble Workflows for Agentic RAG

Abstract

Agentic retrieval-augmented generation (RAG) uses workflows to retrieve and process evidence. However, training RAG policies is computationally expensive, while handcrafting workflows requires repeated case review and validation. Reusing past workflows offers an alternative, but previously effective workflows may become counterproductive under new conditions. We investigate failure modes in RAG workflow reuse and identify three phenomena: Workflow Reversal, Success Mismatch, and Noise Amplification Loop. These findings motivate verifying whether a workflow provides a benefit under the evidence conditions in which it is applied. To avoid ineffective reuse, we propose REMEDY-RAG with three stages: Discover, Verify, and Assemble. It extracts workflows from past executions, learns evidence-conditioned assignments from paired gains over no intervention, and independently verifies these assignments. An external workflow map selects a verified workflow or no intervention based on the initial evidence state. With the same verified workflow, evidence-conditioned admission raises macro F1 from 47.0 to 50.3 across three QA benchmarks. Across five benchmarks, REMEDY-RAG with a frozen Qwen3-8B executor reaches 50.8 macro F1, surpassing the strongest baseline while using 80.1% less offline adaptation compute. The map complements policy training and improves the performance of Qwen3-32B and GPT-5.4 without target-side rediscovery, reverification, or tuning. These results highlight the value of learning and verifying when past experience improves agentic RAG.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Agentic Retrieval-Augmented Generation, Evidence-State Routing, Workflow Reuse, Large Language Models, Conditional Utility Verification

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_9FZiOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
1
Soundness
3
Read review
Paper summary

The paper studies when and how previously discovered workflows should be applied on current RAG-systems. The paper argues that the proper workflow is determined by the state of the retrieved corpus. And they supported this with 3 observation:

  1. Workflow Reversal: the same workflow helps in one evidence state and hurts in another.
  2. Noise Amplification Loop: further retrieval under noisy evidence makes the evidence worse.
  3. Success Mismatch: trajectory success does not reflect a workflow’s incremental gain.
Critical strengths
  1. Conditioning workflow after evidence retrieval feels very sensible, and to my knowledge, is a novel idea.
  2. Very solid experiment: (a) All data splits (discovery, tree-growing, certification, RL, development, test) are question-disjoint. (b) The ablations cover the natural alternatives. (c) The paper compares its method with many other systems thoroughly.
  3. The empirical gain is clear: without any executor training, the method outperforms all other baselines in macro F1, and with RL training, it gets better performance with lower compute.
Critical weaknesses
  1. Writing and organization. Although the findings are interesting, the paper is quite hard to follow for me. (a) The introduction relies on terms that are only defined in section 4 or the appendix (e.g., paired runs, full-registry control, simultaneous LCB, region–workflow assignment). (b) “Workflow” is only defined in section 4.1, and its meaning differs from common usage. (c) A concrete running example early in the paper of the 3-stage system is highly recommended.
  2. The workflow is chosen only once after the first retrieval. At deployment, the map is queried once on the initial evidence state, and the chosen workflow stays fixed until execution ends. However, since each retrieval/workflow actions might change the state, it is reasonable to analyse what if we re-choose the workflow after every few retrievals.
Questions for authors
  1. Did you try recomputing the evidence state and re-querying the map after each retrieval? If not, what prevents it?
  2. The bare protocol and the prompt-based baselines. The bare protocol reaches ~0.399 macro F1 (Tables 40/43), but it is not shown in Table 1. Most prompt-based baselines score below or around it (IRCoT 0.389, Adaptive-RAG 0.382, CRAG 0.395, SeaKR 0.402). Could you add the bare protocol to Table 1, and explain on whether the ported implementations of these baselines may be under-performing compared with their native versions?
  3. Improving readability. Could the authors revise the paper to make the method easier to follow? Specifically: (a) add a concrete running example early in the main text that walks one question through the full pipeline (b) define key terms (workflow, paired runs, full-registry control, simultaneous LCB) at or before their first use in the introduction
4 comments
AuthorsOpenReview ↗

Author Response to Reviewer 9FZi (1/3)

Thank you for your time and for your constructive suggestions on clarity and evaluation. We address your questions below with a concrete example, dynamic-routing results, and a direct audit of the baseline interfaces.

F1 uses the 0–1 scale. One utility point equals 0.01U0.01U.

W1. Key terms appear late, and the three-stage system needs a concrete example.

Response.{\color{green}\mathbf{Response.}} The terms you identify have the following operational meanings:

  • Workflow: reusable textual guidance and an allowed subset of the eight actions. The executor chooses action order and when to stop.
  • Bare/full-registry control: the same agent with default guidance and all eight actions available, including retrieval and verification.
  • Paired runs: workflow and bare continuations of the same question and initial evidence, with matched budgets and decoding seeds. Their utility difference measures the workflow's contribution.
  • Region–workflow assignment: a frozen candidate for a region defined by the evidence-state tree.
  • Simultaneous LCB: lower confidence bounds adjusted across the frozen regional assignments, used to admit positive regional mean gains.

Figure 11 illustrates “Who is the father of the director of Red 2?” Initial retrieval identifies Dean Parisot. The next step is to find his father. Discover produces a missing-hop workflow from separate historical questions. Verify assigns it to R5 using separate growing and certification sets. Its certification gain is 20.5 utility points with LCB 16.3 (Table 11). Assemble maps this example's state (0.60,0.27,0.09,0.33)(0.60,0.27,0.09,0.33) to R5. The executor decomposes the relation, retrieves “Who is Dean Parisot's father?”, verifies the evidence, and answers Aldo Parisot. The regional certificate comes from independent questions, and this example shows how the learned rule is used at deployment.


AuthorsOpenReview ↗

Author Response to Reviewer 9FZi (2/3)

W2. Why select the workflow only once when evidence changes during execution?

Response.{\color{green}\mathbf{Response.}} Selecting a workflow once keeps action order, queries, and stopping decisions adaptive throughout execution. Your suggested experiment tests whether switching workflows as evidence changes improves performance further.

New evidence: original three-benchmark setting. All variants retain the same library, map, executor, and cumulative budget. Rerouting recomputes the current state, updates guidance and the action mask, and preserves evidence and history.

Routing frequencyMacro F1Utility gain vs. bare (pts, 95% CI)Switches/queryExtra state evals/queryLatency (s)
Once after initial retrieval0.5289.88 [8.62, 11.13]00.05.68
After every retrieval0.5299.93 [8.71, 11.18]0.82.05.91
After every two retrievals0.5279.79 [8.52, 11.06]0.40.75.77

Routing after every retrieval increases the F1 point estimate by 0.1 points and latency by 0.23 seconds. Once-only routing achieves nearly identical F1 with lower overhead at the original horizon. This is consistent with our discovery process, which distills reusable guidance from complete historical trajectories to support multi-step execution. Certification evaluates complete continuations under the once-selected policy. The switching variants are assessed through their complete test trajectories.


Q1. Have you tried recomputing the state and re-querying the map after each retrieval?

Response.{\color{green}\mathbf{Response.}} Yes. After each retrieval, the agent preserves its trajectory, recomputes the state, and updates its guidance and allowed actions within the remaining budget. Returning to bare restores all eight actions.

Original-budget routingMacro F1Utility gain vs. bare (pts, 95% CI)Query regression (%)Latency (s)
Once0.5289.88 [8.62, 11.13]0.75.68
Every retrieval0.5299.93 [8.71, 11.18]0.85.91
Every two retrievals0.5279.79 [8.52, 11.06]0.75.77

On long-horizon BrowseComp-Plus [1], repeated routing gives a larger accuracy point improvement and uses more retrievals:

Long-horizon routingAccuracy (%)Gain over bare (pts, paired 95% CI)Evidence recall (%)Retrievals/queryUSD/query
Bare38.8031.411.20.264
Once40.4+1.6 [−0.8, 4.0]35.211.80.312
Every retrieval41.1+2.3 [−0.1, 4.6]37.413.60.355

The point improvement from repeated routing is larger at the long horizon: accuracy rises from 40.4% to 41.1%, and evidence recall from 35.2% to 37.4%. Retrievals increase from 11.8 to 13.6 per question. These measurements quantify the benefit and cost of your proposed extension. The original certificate covers once-selected continuations, and the dynamic variants have the direct test results shown above.

[1] BrowseComp-Plus: A More Fair and Transparent Evaluation Benchmark of Deep-Research Agent.


AuthorsOpenReview ↗

Author Response to Reviewer 9FZi (3/3)

Q2. How does bare compare with the prompt baselines, and could interface adaptation depress their results?

Response.{\color{green}\mathbf{Response.}} Bare provides the direct reference you request. Its Table 43 scores use the same evaluation setting as Table 1:

Bare protocolNQHotpotQA2WikiTriviaQAMuSiQueFive-benchmark macro
F10.4500.4390.4550.5260.123≈0.399
EM0.4120.3930.4090.4970.1020.3626
MethodFive-benchmark macro F1
Bare≈0.399
IRCoT0.389
Adaptive-RAG0.382
CRAG0.395
SeaKR0.402
Frozen REMEDY-RAG0.508

Bare is an agent with all eight operations. Each baseline uses its published controller with the same executor, corpus, retriever, and budget. The interface is selected before testing by development utility F1−0.1C\mathrm{F1}-0.1C (Section B.1). Here, native means the original prompt and tool-call syntax within this shared setting.

New interface audit:

MethodNative dev F1JSON dev F1Selected interfaceInvalid calls (%)Budget exhaustion (%)
IRCoT0.4200.418Native0.71.5
Adaptive-RAG0.4330.435JSON1.43.6
CRAG0.4190.417Native2.44.1
SeaKR0.4420.440Native1.02.5
Paired interface recheckNative minus selected JSON F1 (0–1 scale, 95% CI)
Adaptive-RAG−0.001 [−0.009, 0.008]

Three baselines use their native interfaces. Native and JSON development F1 differ by only 0.2 points for all four methods. For Adaptive-RAG, the paired recheck gives a native-minus-JSON difference of −0.001, with a 95% CI spanning zero. The selected interfaces therefore preserve closely matched performance in this audit. Table 1 compares the controllers under a common Qwen3-8B executor, retriever, corpus, and budget, including the shared corpus for CRAG's retrieval calls.


Q3. Can you explain the pipeline through one example and define the key terms directly?

Response.{\color{green}\mathbf{Response.}} Your requested example follows one question through deployment using a workflow learned and certified offline:

  1. Discover: separate historical traces yield guidance to resolve a missing relation with an entity-specific query. This guidance and its allowed actions form a workflow. The executor decides the sequence and stopping time.
  2. Verify: paired runs compare that workflow against the same question's full-registry control, which retains all eight actions. A learned evidence region and its candidate form a region–workflow assignment. Independent certification computes simultaneous LCBs for regional mean gains across the frozen assignments. For R5, mean gain is 20.5 utility points and its lower bound is 16.3 (Table 11), supporting admission.
  3. Assemble: for “Who is the father of the director of Red 2?”, initial evidence supplies the director, Dean Parisot. The state (0.60,0.27,0.09,0.33)(0.60,0.27,0.09,0.33) enters R5, activating the missing-hop workflow. The executor retrieves Dean Parisot's father relation, verifies the chain, and answers Aldo Parisot (Figure 11).

Historical questions establish the deployment rule offline. A new question uses one evidence-state lookup and executes the selected guidance.

Hi authors,

Thanks for your response. I have read the response and I remain the positive rating.

Reviewer_dBWCOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
3
Read review
Paper summary

The paper studies when workflows distilled from past agentic RAG executions should be reused. It then proposes REMEDY-RAG: Discover clusters de-instantiated recommendations from historical traces into 5 candidate workflows; verify estimated paired utility gains over a no-intervention control and fit a depth-3 policy tree over a 4-dimensional evidence state (SNR, Cov, Con, Conf); an independent certification stage retains only region–workflow assignments with positive simultaneous 95% lower confidence bounds, defaulting to no intervention otherwise. With a frozen Qwen3-8B executor, REMEDY-RAG reaches 50.8 macro F1 across five Wikipedia QA benchmarks.

Critical strengths

S1: The deployment question — when a discovered workflow should be used — is well motivated.

S2: The certify-before-deploy methodology is sound: frozen components, question-disjoint splits, paired gains against a no-intervention control, and simultaneous confidence bounds for admission.

S3: Strong empirical results under matched budgets: a frozen 8B executor beats the strongest trained baseline with ~80% less adaptation compute, and the map transfers to larger executors and unseen domains without target-side tuning.

S4: The appendices are unusually thorough, covering pool-size/depth sensitivity, context-augmented routing controls, cross-map split sensitivity, held-out reliability audits, and detailed cost accounting.

S5: Deployment overhead is minimal — one probe retrieval plus a table lookup, with no executor retraining — making the approach practically attractive.

Critical weaknesses

W1: Limited conceptual novelty: conditioning the RAG strategy on post-retrieval evidence has been established by Adaptive-RAG, CRAG, and post-retrieval routing work; the core contribution reduces to proposing a simple selection mechanism with a 4-dimensional feature vector and a depth-3 decision tree.

  • W2: The choice of evidence features is not analyzed — no leave-one-feature-out ablation of the deployed map, no exploration of richer post-retrieval features, and the routing-model comparison (Context-Tree/MLP) reuses the same frozen encoder representations rather than testing genuinely richer state estimation.

  • W3: The workflow library is small, and discovery is under-examined: pool size only varies in {3, 5, 8} with a fixed clustering-based discovery procedure, so it is unclear whether larger or more diverse libraries would further improve coverage or gains.

  • W4: Evaluation is confined to Wikipedia QA with top-3 retrieval from a fixed corpus; there are no experiments on challenging agentic search benchmarks (e.g., BrowseComp, BrowseComp-Plus, GAIA) where evidence is noisier, horizons are longer, and the diagnosed failure modes should matter most.

  • W5: The three named phenomena largely confirm known behaviors in the adaptive-RAG and skill-reuse literature, and the paper offers limited insight into how to improve RAG performance beyond selecting a workflow from a small library.

  • W6: The presentation is dense relative to the simplicity of the proposed method, with key design choices scattered across the appendices.

Questions for authors

Q1: Why these four state features? Could you provide a leave-one-feature-out ablation of the deployed map, and comment on whether richer post-retrieval evidence features (e.g., claim-level support, answer consistency, LLM-graded sufficiency) improve routing?

Q2: How does REMEDY-RAG perform on web-scale, long-horizon agentic search benchmarks such as BrowseComp, BrowseComp-Plus, or GAIA?

9 comments
AuthorsOpenReview ↗

Author Response to Reviewer dBWC (1/4)

Thank you for taking the time to examine our work and for your valuable suggestions on the features, workflow library, and search benchmarks. Below, we clarify the contribution and present the experiments addressing your concerns.

F1 uses the 0–1 scale. One utility point equals 0.01U0.01U.

W1. Is the contribution simply post-retrieval routing with four features and a shallow tree?

Response.{\color{green}\mathbf{Response.}} The central insight is that learning a reusable procedure also requires learning where it adds value. Success on an easy question can give a workflow credit for an answer the agent would already obtain. Positive average gain can also hide harm in particular evidence states. REMEDY-RAG learns where a workflow improves the same agent's continuation and where bare execution is preferable.

Adaptive-RAG [1] selects retrieval strategies from query complexity. CRAG [2] evaluates retrieved documents to trigger predefined corrective branches. SkillGen [3] verifies a skill's aggregate net benefit through paired executions. REMEDY-RAG focuses on the evidence conditions under which a learned workflow provides positive incremental value. This makes the conditions for reuse part of what is learned from experience.

Table 32 tests this distinction while retaining the same candidate:

Deployment of the same workflow  Intervention (%)  Utility gain over bare (points)
Global net-gain admission  100.00  3.35
Evidence-region net-gain admission  76.17  7.15

Learning where to use the same candidate more than doubles its deployment gain. With regional candidates and matched 75.93% coverage, uncertainty-aware admission further raises gain from 9.14 to 9.92 points and reduces query regression from 1.61% to 0.68%. These controls show why reusable experience needs evidence-dependent deployment conditions. A shallow tree makes those conditions inexpensive to apply.

[1] Adaptive-RAG: Learning to Adapt Retrieval-Augmented Large Language Models through Question Complexity.

[2] Corrective Retrieval Augmented Generation.

[3] SkillGen: Verified Inference-Time Agent Skill Synthesis.


W2. Why these four features, and does richer state estimation help?

Response.{\color{green}\mathbf{Response.}} The four features describe complementary aspects of the decision. SNR measures retrieval relevance, Cov estimates evidence sufficiency, and Con measures contradiction among relevant evidence. Conf measures the executor's closed-book answer confidence. Together they capture missing evidence, distraction, conflict, and the executor's prior knowledge.

New evidence. Each feature removal refits and independently certifies the map, fixing the library, executor, 4,000 growing questions, 2,000 certification questions, and depth three. The richer state appends relationship-level support and overall sufficiency judgments from the frozen 8B model using only the question and initial evidence.

StateMacro F1Utility change vs. four features (points, 95% CI)Query regression (%)Extra diagnosis (output tokens / s)
Four features0.528400.7—
Without SNR0.5158−1.32 [−1.92, −0.72]1.1—
Without Cov0.5116−1.78 [−2.39, −1.15]1.3—
Without Con0.5223−0.63 [−1.15, −0.20]0.8—
Without Conf0.5197−0.93 [−1.46, −0.36]0.9—
Four features + support + sufficiency0.5297−1.41 [−2.04, −0.79]0.8161 / 1.32

Every feature removal reduces utility after relearning, with the largest losses from Cov and SNR. The richer judgments improve F1 by 0.13 points and require 161 extra output tokens and 1.32 seconds per question. Their additional diagnosis cost lowers utility by 1.41 points under the shared budget. Under the evaluated budget, adding relation-level evidence support and overall evidence sufficiency to the four-feature state yields only a marginal F1 improvement and lowers deployment utility once diagnostic costs are accounted for. These results support the four features as a compact state that preserves useful routing information at low cost.


AuthorsOpenReview ↗

Author Response to Reviewer dBWC (2/4)

W3. Would larger or more diverse workflow libraries improve coverage and gains?

Response.{\color{green}\mathbf{Response.}} The expanded comparison tests up to 16 candidates and uses k-center representative selection to increase diversity. Discovery traces, embeddings, prompts, and call budgets are matched. Tree depth is three, and certification uses 2,000 questions.

DiscoveryCandidatesGrowing questionsExpected questions/​candidateDeployed candidatesIntervention (%)F1Utility gain over bare (points, 95% CI)GPU⁠-⁠h
Clustering54,000800575.70.5289.88 [8.62, 11.13]27.8
Clustering84,000500678.10.53110.25 [8.94, 11.56]27.3
Clustering164,000250568.90.5188.62 [7.40, 9.95]28.4
k-center164,000250672.30.5249.27 [8.13, 10.44]27.7
Clustering1612,800800778.50.53010.00 [8.78, 11.25]55.9

Eight candidates give the highest F1 and utility point estimates at the original growing budget. With 16 candidates, k-center selection improves coverage from 68.9% to 72.3% and utility from 8.62 to 9.27 points. Increasing the growing budget to 800 questions per candidate recovers clustering-based F1 to 0.530 at 55.9 GPU-hours. Library diversity helps at a fixed budget, and larger libraries benefit from more observations per candidate. With fewer observations per candidate, a larger library may increase the risk of overfitting in the router by favoring workflows whose estimated gains are driven by a small subset of training questions.


W4. Does the evaluation cover challenging search beyond Wikipedia QA?

Response.{\color{green}\mathbf{Response.}} Table 1 includes OTT-QA and biomedical MedHop. Frozen transfer reaches F1 0.416 and 0.351, respectively, compared with bare scores of 0.268 and 0.233 (Table 43). To address your long-horizon search question, the new BrowseComp-Plus v1 [1] evaluation covers 830 questions over 100,195 web documents.

All methods share GPT-5.4, retrieval, and the eight-operation interface. The source libraries and maps are frozen. Short/long budgets allow 4/20 retrievals, 6/30 interactions, and 1,024/8,192 generated tokens. Auxiliary generation counts toward these totals. Accuracy differences have paired 95% CIs.

BudgetDeploymentAccuracy (%)Change vs. bare (points, 95% CI)Evidence recall (%)Retrievals/queryUSD/query
ShortBare31.8024.83.30.054
ShortRegional mean gate32.9+1.1 [−1.2, 3.4]26.33.50.057
ShortSource LCB map, once34.0+2.2 [−0.2, 4.7]28.13.40.059
ShortSource LCB map, after each retrieval34.3+2.5 [−0.1, 4.9]29.03.80.065
LongBare38.8031.411.20.264
LongRegional mean gate39.6+0.8 [−1.6, 3.2]33.112.00.294
LongSource LCB map, once40.4+1.6 [−0.8, 4.0]35.211.80.312
LongSource LCB map, after each retrieval41.1+2.3 [−0.1, 4.6]37.413.60.355

Source-map reuse increases evidence recall at both horizons. At the long horizon, once-only routing raises recall from 31.4% to 35.2%, and dynamic routing reaches 37.4%. Dynamic routing also achieves the highest accuracy point estimate, 41.1%, at higher search cost. The paired accuracy intervals include zero. The results show how frozen workflow reuse performs on a much larger web corpus with a budget of up to 20 retrievals.

[1] BrowseComp-Plus: A More Fair and Transparent Evaluation Benchmark of Deep-Research Agent.


AuthorsOpenReview ↗

Author Response to Reviewer dBWC (3/4)

W5. What actionable insight do the three phenomena provide beyond familiar RAG behavior?

Response.{\color{green}\mathbf{Response.}} The three diagnostics guide how experience is learned and reused:

  1. Use evidence to identify repair needs. In Figure 2, evidence features predict deficiency with AUC 0.861, compared with 0.601 for executor signals. Executor signals are stronger at predicting answer correctness (0.746 versus 0.665). Retrieval helps missing-hop evidence (+0.186) and harms noisy evidence (−0.147). Evidence features therefore provide the relevant signal for choosing a repair.
  2. Choose the repair before spending the remaining budget. Section 3.2 reports same-region re-entry of 68.4% in noisy states and 31.2% in low-coverage states. Evidence quality also worsens in noisy states. The diagnosis helps determine whether further retrieval is useful.
  3. Assign credit through incremental gain. Section 3.3 reports a success–incremental-gain correlation of 0.02 (95% CI [−0.34, 0.59]). Paired comparisons with the same agent's bare continuation measure the workflow's own contribution.

These insights also improve execution within a selected workflow. In Table 23, regional assignments and coverage are fixed:

Execution specificationThree-benchmark macro F1
Bare instructions, full action set0.448
Learned guidance, full action set0.503
Default guidance, workflow action mask0.481
Learned guidance and action mask0.528

Learned guidance alone contributes 5.5 F1 points under this control. Combining guidance with the action mask reaches 0.528 F1. Workflow content improves execution, and evidence-dependent deployment identifies where that content adds value.


W6. The presentation obscures a simple method and scatters key design choices.

Response.{\color{green}\mathbf{Response.}} A workflow consists of reusable guidance and an allowed action set. The executor chooses its steps and stopping time. Discover extracts workflows from historical trajectories. Verify learns their regional gains against the same unrestricted agent and tests frozen assignments on separate questions. Assemble performs one lookup after the initial retrieval and activates an admitted workflow or bare execution.

The method uses five candidates and a depth-three tree over four state scores. Question-disjoint discovery, growing, and certification sets contain 2,000, 4,000, and 2,000 questions. Admission requires at least 120 certification questions per region and a positive simultaneous lower bound. Paired comparisons occur offline. Online routing requires an initial evidence diagnosis and a table lookup.

In Figure 11, the question asks for the father of Red 2's director. Initial evidence identifies the director, Dean Parisot. The map selects the missing-hop workflow to resolve the father relation. The executor retrieves that relation, verifies the chain, and answers Aldo Parisot.


Reviewer_52vjOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

REMEDY-RAG learns when to apply reusable workflows in agentic RAG. It extracts a small workflow library from agent traces, compares each workflow with the bare agent through paired continuations, and learns workflow assignments from features of the evidence available after retrieval. It then checks these assignments on a separate certification split. A workflow is used only where the lower confidence bound on its regional mean utility gain is positive; elsewhere, the system falls back to the bare agent. The workflows combine textual guidance with restrictions on available actions. The paper evaluates the frozen system, an optional RL variant, and several controls for routing, certification, transfer, and cost.

Critical strengths
  1. The paper provides direct evidence for the value of conditional deployment. In Table 32, replacing a global admission decision with regional admission for the same candidate increases mean utility gain from 3.35 to 7.15 points. At the same intervention rate, the confidence-bound rule achieves 9.92 points versus 9.14 for mean-based ranking, with fewer harmful regional assignments. These controls help separate better selection from simply using workflows less often. Guidance-only, action-mask-only, and stronger-router comparisons further examine which parts of the system matter.

  2. The evaluation goes beyond a single successful workflow library. Three independent pipeline builds achieve five-benchmark macro-F1 of 0.508, 0.515, and 0.510, supporting repeatability despite differences in the discovered workflows. The primary frozen system reaches 0.5083 versus 0.4970 for CW-GRPO, with reported adaptation costs of 27.8 versus 139.8 GPU-hours, including construction costs. Experiments with other executors and additional domains provide evidence that the approach is useful beyond its construction setting, although they do not establish that the same deployment rules remain optimal.

Critical weaknesses
  1. The comparison with existing baselines establishes a useful system more clearly than it establishes when the proposed learning and verification procedure is necessary. The paper includes meaningful controls, but their implications deserve a more precise discussion. For example, comparative outcome regression retains 91.2% of the paired router's utility gain at the largest audited training budget (Table 30). Meanwhile, Table 32 supports regional selection and shows an additional benefit from confidence-bound admission at matched intervention rates. Taken together, these results suggest that much of the benefit may come from conditioning workflow use on the evidence, while pairing and certification provide further advantages whose size depends on the setting. The paper should distinguish these contributions more clearly when comparing with prior methods. My concern is not primarily whether it improves on one particular skill framework, but whether the comparisons establish what existing selection methods are missing and when the additional machinery is worth using.

  2. The evidence for transferable usage conditions is weaker than the evidence for a useful system. Rebuilding the method for GPT-5.4 preserves four of five workflow roles, but only four of eight regional assignments match the transferred source map (§C.8.3). The transferred map still improves performance, so this is not a failure of transfer; it suggests that reusable procedures and the conditions for applying them have different degrees of portability. The reliability evidence also has a specific boundary: the primary rule uses approximate normal/Sidak bounds, whereas the auxiliary calibration analysis uses bootstrap max-t bounds on a centered empirical bank. That analysis does not directly establish calibration of the primary rule, and neither certificate automatically applies after changing executors or domains. The main text should make these limits clearer when explaining what has been learned about when workflows help.

Questions for authors
  1. Given that comparative outcome regression recovers 91.2% of the paired router's gain at the largest training budget, what do the existing comparisons establish as the main advantage over simpler selection methods: better performance with limited training data, better routing, or fewer harmful interventions? Please relate the results in Tables 30–32 to these distinct claims and clarify which comparisons hold the workflow library, training budget, and intervention rate fixed.
  2. Section C.8.3 reports changes in admission for R2/R4 and workflow choice for R7/R8 after rebuilding for GPT-5.4. Do the existing regional estimates distinguish executor-dependent changes in utility from uncertainty in the estimated assignments? What evidence supports reusing the source deployment map without recertification?
  3. Which reported calibration results evaluate the primary normal/Sidak admission rule directly, and which evaluate only the auxiliary bootstrap rule? Please clarify the empirical support for the reliability claim attached to the deployed primary system.
4 comments
AuthorsOpenReview ↗

Author Response to Reviewer 52vj (1/3)

Thank you for the time you devoted to our paper and for your thoughtful comments on selection, transfer, and reliability. We address your questions below with controlled comparisons and new calibration results.

F1 uses the 0–1 scale. One utility point equals 0.01U0.01U, where U=F1−0.1CU=\mathrm{F1}-0.1C.

W1. What do pairing and certification contribute beyond simpler conditional selection?

Response.{\color{green}\mathbf{Response.}} You correctly identify conditional deployment as a major source of the gain. In Table 32, regional admission raises utility gain from 3.35 to 7.15 points for the same candidate. Selecting candidates by region with a development-selected mean threshold reaches 9.58 points.

Deployment rule, Table 32Intervention (%)Utility gain over bare (points)
Global mean gate100.003.35
Same candidate, regional mean gate76.177.15
Regional candidates, development-selected mean threshold77.219.58
Regional candidates, coverage-matched mean ranking75.939.14
Regional candidates, bootstrap LCB75.939.92

Pairing improves sample efficiency. In Table 30, paired learning with 1,500 growing questions achieves 8.92 points, compared with 8.77 for outcome regression using 4,000. Both use 2,000 certification questions. At 4,000 growing questions, outcome regression recovers 91.2% of the paired router's gain, leaving a 0.85-point advantage.

Certification reduces harmful interventions at matched coverage. At 75.93% coverage, the bootstrap LCB rule gains 0.78 utility points over mean ranking (95% CI [0.51, 1.05]) and reduces query-level regression from 1.61% to 0.68% (Table 32). Together, these controls identify three benefits: conditional use provides the main gain, pairing improves data efficiency, and uncertainty-aware admission reduces harm. Certification is particularly useful with noisy regional estimates or costly regressions. The primary certification stage costs 8.6 GPU-hours (Table 9).


W2. How portable are the learned usage conditions, and what does the reliability evidence cover?

Response.{\color{green}\mathbf{Response.}} Your distinction between procedure reuse and deployment conditions is supported by the transfer results. Section C.8.3 preserves four of five workflow roles and four of eight regional decisions. The target-side comparison shows how much direct reuse achieves and how much target-specific admission adds.

New evidence: GPT-5.4 deployment. Direct transfer uses the frozen source map. Recertification retains the source library, tree, and eight provisional candidates, and evaluates admission on 2,000 independent target questions. All strategies route using GPT-5.4's own confidence.

StrategyTarget construction questionsMacro F1Utility gain over bare (points, 95% CI)Target adaptation cost (USD)
Bare00.54800
Direct source-map transfer00.6018.22 [7.40, 9.01]0
Source library/tree/candidates, target recertification2,0000.6129.30 [8.45, 10.15]386.40
Full target reconstruction8,0000.61810.94 [8.90, 12.94]1,274.80

Direct transfer improves target F1 by 5.3 points with zero target adaptation cost. Recertification adds 1.1 F1 points and approaches full reconstruction using one quarter of its target construction questions. Certification is specific to the executor and distribution being evaluated. The target evaluation establishes the benefit of direct reuse, and target recertification supplies executor-specific admission evidence.

The original 4.7% null false-admission rate in Section C.5.9 evaluates bootstrap max-tt. The new matched resampling audit directly evaluates the deployed normal/Šidák rule on all eight fixed provisional assignments:

Primary ruleAll-zero FWER (%, 95% CI)Mixed-null FWER (%, 95% CI)
Original regional sample counts5.3 [4.1, 6.9]4.6 [3.5, 6.1]
120 questions per region5.0 [3.8, 6.5]4.2 [3.1, 5.6]

The primary rule's false-admission estimates are consistent with the nominal 5% level. This directly supports its empirical calibration on the centered bank. The certificate provides approximate simultaneous control of regional mean gains under the frozen system and stated sampling assumptions.

AuthorsOpenReview ↗

Author Response to Reviewer 52vj (2/3)

Q1. Which comparisons isolate sample efficiency, routing quality, and harmful interventions?

Response.{\color{green}\mathbf{Response.}} Tables 30–32 separate the effects of the learning objective, deployment conditions, and admission rule.

Learning objective, Table 30(a). Both routers use the same five workflows, four state features, 4–128–6 MLP, growing observations, and 2,000 certification questions. Both use paired bootstrap admission, with intervention rates determined by each learned policy. Entries are mean ± SD over ten matched seeds.

Growing questionsOutcome regression: utility gainPaired-gain regression: utility gain
1,5007.10 ± 0.628.92 ± 0.42
4,0008.77 ± 0.469.62 ± 0.34

Both objectives target the same population contrast:

E[Uw−U0∣e]=E[Uw∣e]−E[U0∣e].\mathbb{E}[U_w-U_0\mid e]=\mathbb{E}[U_w\mid e]-\mathbb{E}[U_0\mid e].

Pairing improves learning efficiency at finite sample sizes. Table 30(b) also shows that retaining paired covariance tightens confidence bounds. Outcome-based inference yields identical bounds when it retains the same covariance on the same observations.

Deployment conditions, Table 32. The library, state features, eight regions, 4,000 growing questions, and 2,000 certification questions are fixed. For the same candidate, global gating yields 3.35 utility points at 100% coverage. Regional gating yields 7.15 points at 76.17% coverage. This comparison measures the benefit of conditional use, including its change in coverage.

Admission, Tables 31–32. Frozen candidates and intervention counts are matched within each comparison:

Audit  Rule  Intervention (%)  Utility  Query regression (%)
Table 31, fixed-candidate bank  Mean ranking  75.31  Reference  1.58
Table 31, fixed-candidate bank  Bootstrap LCB  75.31  +0.64 vs. mean  0.74
Table 32, regional candidates  Mean ranking  75.93  9.14 over bare  1.61
Table 32, regional candidates  Bootstrap LCB  75.93  9.92 over bare  0.68

The paired utility improvements have 95% CIs [0.39, 0.89] and [0.51, 1.05], respectively. These bootstrap comparisons establish the benefit of uncertainty-aware selection at fixed coverage. The normal/Šidák calibration below directly evaluates the deployed primary rule.


Q2. Do changed regional assignments reflect executor effects or estimation uncertainty? What supports transfer without recertification?

Response.{\color{green}\mathbf{Response.}} To distinguish executor effects from estimation uncertainty, the new diagnostic compares gains on identical regional members. It fixes initial evidence and groups questions using the source confidence score. Each workflow is compared with the corresponding executor's bare continuation.

RegionCandidateQuestions8B gain (points, 95% CI)GPT-5.4 gain (points, 95% CI)GPT-5.4 minus 8B (points, paired 95% CI)
R2Source33810.9 [6.7, 15.1]9.4 [4.8, 14.0]−1.5 [−7.1, 4.2]
R4Source405−7.9 [−12.6, −3.2]9.6 [5.0, 14.2]+17.5 [12.3, 22.8]
R5Source85917.6 [14.2, 21.0]3.2 [−0.4, 6.8]−14.4 [−20.0, −8.8]
R7Source75117.4 [14.1, 20.7]13.2 [9.8, 16.6]−4.2 [−8.3, 0.1]
R7Target-rebuilt751—15.4 [11.6, 19.2]—
R8Source50012.1 [8.0, 16.2]10.4 [6.1, 14.7]−1.7 [−7.7, 4.3]
R8Target-rebuilt500—17.1 [13.2, 21.0]—

R4 shows a change from negative to positive gain, and R5 shows a substantial reduction, with regional membership fixed. Their paired intervals identify clear executor effects. For R2, R7, and R8, the cross-executor intervals include zero. Both source candidates retain positive target gains in R7/R8, and target-rebuilt candidates achieve higher point estimates.

The deployment evaluation uses the target executor's confidence to route questions. It measures the resulting policy on its target-defined regions:

GPT-5.4 strategyTarget construction questionsMacro F1Utility gain over bare (points, 95% CI)
Bare00.5480
Direct transfer00.6018.22 [7.40, 9.01]
Target recertification2,0000.6129.30 [8.45, 10.15]
Full target reconstruction8,0000.61810.94 [8.90, 12.94]

Direct reuse improves target F1 by 5.3 points with zero target construction questions. Recertification adds 1.1 F1 points and supplies target-specific admission evidence. These results support reusing the source map as an effective starting point and refining admission with a modest target budget.

AuthorsOpenReview ↗

Author Response to Reviewer 52vj (3/3)

Q3. Which calibration results directly evaluate the deployed normal/Šidák rule?

Response.{\color{green}\mathbf{Response.}} Table 11 directly audits the primary system: all six admitted regions have positive independent-audit mean gains. Section C.5.9's 4.7% null false-admission rate evaluates the auxiliary bootstrap rule on its fixed-candidate bank. The following new experiment directly tests both rules on matched samples from the primary certification bank.

The 2,000-question certification bank retains all eight provisional assignments, including R1/R4. Gains are centered within dataset × region. At least 1,000 question-level resamples preserve paired continuations. Each replicate evaluates both the deployed rule Gˉ−2.49 SE>0\bar G-2.49\,\mathrm{SE}>0 and bootstrap max-tt, with minimum regional support 120. The mixed setting assigns regional means of 00, −0.01-0.01, or +0.05+0.05. FWER measures the probability of admitting at least one region with a nonpositive assigned mean. Intervals are Wilson 95% CIs.

Rule  Questions per region  All-zero FWER (%)  Mixed-null FWER (%)  Power at gain +0.05 (%)
Normal/Šidák  Original counts  5.3 [4.1, 6.9]  4.6 [3.5, 6.1]  46.9
Bootstrap max-tt  Original counts  4.2 [3.1, 5.6]  3.8 [2.8, 5.2]  44.3
Normal/Šidák  120  5.0 [3.8, 6.5]  4.2 [3.1, 5.6]  23.1
Bootstrap max-tt  120  4.1 [3.0, 5.5]  3.5 [2.5, 4.8]  20.4

The primary rule's all-zero false-admission estimates, 5.3% and 5.0%, are consistent with the nominal 5% level within Monte Carlo uncertainty. Reducing regional support to 120 mainly lowers power. This audit directly supports primary-rule calibration on the recorded bank and complements the positive held-out gains in Table 11. Both assess regional mean gains for the frozen executor and distribution.

Dear Reviewer,

With a few hours remaining in the discussion period, we sincerely thank you for your time and thoughtful review of our paper. We hope our responses have addressed your concerns.

Best regards,
The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.