acceptodds
Under review as a conference paper at ICLR 2027

From Multi-Agent to Single-Agent: When Is Skill Distillation Beneficial?

Abstract

Multi-agent systems (MAS) for structured data-science tasks externalize analytical control through workflows, but repeatedly executing orchestration incurs deployment overhead. Distilling workflows into single-agent skills raises a selective-transfer question: which components should survive? We distinguish capability resources, which expand available actions and knowledge, from pipeline guidance, which constrains explored solutions. On the same causal-estimation instances, capability-matched pipeline retention changes normalized utility by points under method-selection accuracy but under numerical error. We introduce Behavior-Outcome Freedom (), a pre-synthesis diagnostic of signed behavior–outcome rank mismatch, and give a candidate-conditional account through Signed Anchor-Rank Transfer with an explicit remainder for coarse representations. AdaSkill preserves validated capabilities, removes runtime orchestration, and calibrates an -threshold on module-level development interventions, freezing it before target construction. Across 16 matched interventions, structural effect decreases with (, ); 15 atomic treatments localize the reversal to pipeline guidance. Across 11 datasets and four task families, directly executed skills combine strong performance with lower latency than source workflows. Workflow-excluded construction tests the calibrated policy separately from the mechanism analysis.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM agents, multi-agent systems, skill distillation, workflow distillation, agent evaluation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_bM8qOpenReview ↗
Rating
3(−1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies when pipeline guidance should be retained when distilling multi-agent workflows into directly executable single-agent skills. It distinguishes capability resources from pipeline guidance and introduces Behavior-Outcome Freedom, a diagnostic intended to guide this retention decision. The proposed AdaSkill framework preserves validated capabilities, removes runtime orchestration, and uses a threshold calibrated on development interventions to decide whether to retain pipeline guidance. The authors present a candidate-conditional theoretical account, analyze 16 matched interventions and 15 atomic treatments, and evaluate performance and latency across 11 datasets spanning four task families. They also report workflow-excluded construction experiments to evaluate the calibrated policy separately from the mechanism analysis.

Critical strengths
  1. A useful distinction between capabilities and workflow constraints.

The paper asks which parts of a multi-agent workflow should survive conversion into a single-agent skill. It separates tools and declarative knowledge from pipeline rules and runtime orchestration (Sections 3.1 and 3.4). This is a useful way to frame the problem. Keeping useful resources does not require keeping every ordering rule or decision gate. The distinction gives a concrete design principle for building lower-overhead agents.

  1. An experimental design aimed at isolating the role of pipeline guidance.

The paper includes component ablations and Full-versus-Discard comparisons that hold tools, knowledge, the evaluator, and the execution budget fixed (Sections 4.3–4.4). It also tests threshold selection with the target workflow regime excluded, then executes the resulting skills (Section 4.5). These controls are relevant to the central question and go beyond an end-to-end performance comparison. The evaluation covers four task families and reports both performance and deployment overhead.

Critical weaknesses

My only concern is the internal consistency of the reported data and diagnostic statistics. I have two specific concerns below. If the authors cannot provide a convincing explanation supported by the underlying results, I may recommend rejection with a rating of 1 (Clear rejection).

  1. The binary score variances in Table 4 do not match the stated six-run protocol.

Section B.3 (p. 20, lines 1059–1061 and 1073) states that each diagnostic item is run six times and that BIRD has no execution failures. Table 4 (p. 23, lines 1195–1215) reports two identifiable BIRD SRR items with a mean score variance of 0.250. Section B.2 defines SRR as a binary score.

With k successes in six runs, the usual sample variance is k(6-k)/30. For an item with varying scores, the possible values are 0.1667, 0.2667, and 0.3000. The mean over two such items can only be 0.1667, 0.2167, 0.2333, 0.2667, 0.2833, or 0.3000. None rounds to 0.250. Using population variance does not explain the whole table either: binary population variance cannot exceed 0.25, but BIRD EX is 0.275 and CE Real MSA is 0.300.

Please clarify the effective run counts, variance denominator, and aggregation rule for each row. Please provide the six raw binary scores for each of the two BIRD SRR items and the code used to produce Table 4.

  1. The CE Real MSA variance in Table 4 conflicts with the Freedom range in Table 11 under the stated definitions.

Table 4 reports five valid CE Real MSA diagnostic items, six runs per item, and a mean score variance of 0.300. Table 11 (p. 30) gives the corresponding output-space Freedom range as [0.000, 0.592]. The relevant definitions appear in Equation (2), Section B.2, and Table 3.

For six binary scores, the maximum sample variance is 0.300. It occurs only with three correct and three incorrect runs. The reported mean therefore requires this split for all five items, under the stated protocol and rounding precision.

MSA checks agreement with one reference method label. The canonical output distance checks whether two method labels differ. Across the 15 distinct run pairs, nine correct–incorrect pairs have different scores. At most 12 pairs can have different method labels, since the three correct runs share the reference label.

Let U indicate different method labels and V indicate different scores. Then P(V=1)=0.6, P(U=1)<=0.8, and V=1 implies U=1. For these binary variables, Spearman correlation equals Pearson correlation. The minimum correlation is sqrt[(0.60.2)/(0.80.4)] = 0.6124. Freedom is therefore at most 1-0.6124 = 0.3876, or 0.388 after rounding. This is below the reported maximum of 0.592.

This calculation assumes six runs per item, binary MSA, one reference method label, binary label-distance, and the 15 distinct non-diagonal pairs. If the implementation differs, please explain the difference. Please provide the per-run method labels, binary scores, effective run counts, and per-item Freedom values for these five items. If corrections are needed, please also check whether they affect the reported correlations, threshold calibration, and routing conclusions.

These are requests to resolve statistical inconsistencies, not allegations of data fabrication. Table-entry errors, mismatched result versions, or undocumented implementation choices could explain them. The run-level records are needed to establish which explanation applies.

Questions for authors
  1. How was the BIRD SRR mean score variance of 0.250 in Table 4 calculated? Under the stated six-run protocol, the mean sample variance over two score-varying binary items cannot take this value. Please provide the six raw scores for each item, the variance denominator, and the aggregation code. Were any runs excluded, or were different sampling budgets used?

  2. Were the CE Real MSA entries in Tables 4 and 11 computed from the same runs? Under the stated definitions, a mean sample variance of 0.300 requires three correct and three incorrect runs for each item. This limits output-space Freedom to approximately 0.388, below the reported maximum of 0.592. Please provide the per-run method labels and scores, and explain any differences in run counts, distance definitions, or data versions.

  3. How does AdaSkill account for differences between candidate pipelines on the same task? Freedom is measured from the Base Agent before skill construction. Section 3.3 also notes that candidates with the same Freedom can have opposite effects. If two qualified source pipelines share the same task, metric, and diagnostic setup, would AdaSkill give both the same retention decision? What information allows the method to distinguish a helpful pipeline from a harmful one in this setting?

  4. How are conflicting module decisions handled when constructing one skill? For example, CE retains method-selection guidance but discards numerical-estimation guidance. However, method selection also affects numerical error. Could you show a concrete example of how a source rule is assigned, retained, or removed when it affects both metrics? What determines which instruction takes priority if the retained guidance constrains a decision assigned to the discard route?

6 comments
AuthorsOpenReview ↗

Official Response to Reviewer bM8q (1/2)

We sincerely thank Reviewer bM8q for the remarkably statistical audit and for recognizing the value of our capabilities-vs-constraints distinction and experimental controls.


[W1, W2, Q1, Q2] Resolution of Statistical Discrepancies in Tables 4 and 11

Both discrepancies come from a text typo in Sections 4.6, B.3, and C.1, which mistakenly reported N=6N = 6 (from the Section C.5 sweep) instead of the actual n=5n = 5 runs executed. Because all tables and routing decisions were computed directly from the n=5n = 5 raw logs, every reported number reproduces identically and zero results change.

Variance and Aggregation Rule. The main diagnostic batch runs n=5n = 5 trials per item (M=10M = 10 for T2SQL, M=6M = 6 for CE). For each identifiable item (where behavior and score distances vary across the 5 runs), we compute the unbiased sample variance with Bessel's correction (ddof=1\text{ddof} = 1, denominator n−1=4n - 1 = 4) and average across the KoutK_{\text{out}} identifiable items. With n=5n = 5 and kk successes, binary sample variance is k(5−k)/20k(5-k)/20, which equals 0.2000.200 for k∈{1,4}k \in \{1, 4\} and 0.3000.300 for k∈{2,3}k \in \{2, 3\}.

Under n=5n = 5, all discrepancies resolve directly:

1. Table 4 Binary Score Variances (BIRD SRR, BIRD EX, Spider EX)

Table 4 RowKoutK_{\text{out}}Mean Var.Identifiable Item IDsRaw Binary Scores (n=5n = 5)Item Var. (n−1=4n-1=4)
BIRD SRR20.250q758 / q1368[1,1,1,1,0] / [1,1,1,0,0]0.200 / 0.300
BIRD EX40.275q1454 / q538 / q1195 / q337[0,0,0,0,1] / [1,1,0,0,1] / [1,1,0,0,1] / [1,0,0,0,1]0.200 / 0.300 / 0.300 / 0.300
Spider EX20.250q91/ q51[1,0,0,0,0] / [0,0,1,1,0]0.200 / 0.300

Run exclusions. Zero runs were excluded for CE or BIRD. In Spider, 3 of 5 runs on q25 timed out and the remaining 2 valid runs both scored 1 (zero variance), so q25 was non-identifiable.

2. CE Real MSA Variance (Table 4) vs. Freedom Range (Table 11)

Tables 4 and 11 were computed from the exact same n=5n = 5 runs. Your bound Fout≤0.388F_{\text{out}} \le 0.388 is exact for n=6n = 6. Under n=5n = 5 ((52)=10\binom{5}{2} = 10 pairs), a variance of 0.3000.300 occurs when k=2k = 2 or k=3k = 3. When k=2k = 2 (2 correct runs sharing the reference label, and 3 incorrect runs with three distinct wrong labels), there are 66 score-differing pairs (P(V=1)=0.6P(V=1) = 0.6) and 99 label-differing pairs (P(U=1)=0.9P(U=1) = 0.9). Your formula yields ρ=(0.6×0.1)/(0.9×0.4)=0.4082\rho = \sqrt{(0.6 \times 0.1)/(0.9 \times 0.4)} = 0.4082 and Fout=1−ρ=0.592F_{\text{out}} = 1 - \rho = \mathbf{0.592} (item q25 below).

Item [Ref Label]Selected Methods across n=5n = 5 RunsRaw MSA ScoresSample Var.FoutF_{\text{out}}
q26 [iv]rdd, iv, iv, rdd, ols0,1,1,0,00.3000.388
q5 [ols]ols, ols, matching, ols, matching1,1,0,1,00.3000.000
q15 [matching]ols, matching, ols, matching, matching0,1,0,1,10.3000.000
q25 [iv]rdd, iv, iv, ols, matching0,1,1,0,00.3000.592
q11 [did]did, did, ols, did, matching1,1,0,1,00.3000.198
q10 [rdd]rdd, rdd, rdd, rdd, rdd1,1,1,1,10.000Unidentifiable

Over the 5 identifiable items, mean variance is 0.300 (Table 4), mean FoutF_{\text{out}} is 0.236 (Tables 7 and 11), and the range is [0.000, 0.592] (Table 11). The family-level CE MSA routing input (0.167, Section 4.5) averages Textbook (0.098) and Real (0.236).

3. Impact on Results and Sensitivity Sweep (N∈[4,20]N \in [4, 20])

Zero downstream numbers change because all reported results already used this n=5n = 5 batch. In our separate 20×2020 \times 20 sweep on CE Real (Section C.5 and Figure 9), we executed 20 runs per item and subsampled NN without replacement (audit/mn_sweep.py). Because FF stabilizes once N≥5N \ge 5 (shifting by only 0.009 on MSA and 0.000 on MRE between N=5N=5 and N=6N=6), the n=5n = 5 main batch required no re-execution. We will update the run-count text in Sections 4.6, B.3, and C.1 from 66 to 55.

Subsampled Runs (NN)45 (Main Batch)6 (Sweep Operating Pt.)71020
MSA Mean FF (±\pm SD)0.128 (.099)0.143 (.099)0.152 (.089)0.151 (.083)0.164 (.080)0.167 (.067)
MRE Mean FF (±\pm SD)0.875 (.242)0.871 (.175)0.871 (.143)0.868 (.139)0.874 (.108)0.864 (.078)

4. Raw Records and Code for Verification (supplementary.zip)

  • raw/A_batch1_n5/ and raw/C_budget_sweep_ce_real_n20/ contain the time-stamped run directories (2026-04-01) with results.json and full trajectory logs.
  • audit/per_run_records.csv and audit/per_item_audit.csv list all per-run method labels, SQL queries, raw scores, variances, and FF values.
  • audit/recompute_from_raw.py and audit/mn_sweep.py recompute Tables 4 and 11 and the NN-sweep table directly from the raw JSON logs with 100% agreement.
AuthorsOpenReview ↗

Official Response to Reviewer bM8q (2/2)

[Q3] Distinguishing Candidate Pipelines on the Same Task

Yes. Two qualified pipelines on the same task and metric receive the same routing decision from FF. AdaSkill does not estimate candidate-specific direction aπa_\pi, providing best-in-class average value rather than individual-candidate safety.

  • What FF decides (whether to constrain). Computed on Base Agent runs before synthesis, FF measures whether contracting behavior toward successful anchors preserves (F<1F<1) or reverses (F>1F>1) outcome-rank order. As an effect modifier rather than a candidate-lift estimator, FF assigns all candidates on the same task-metric module the same Retain/Discard decision.
  • What qualification decides (admissibility, Section 3.1). AdaSkill checks only whether a source workflow is published, high-performing on the same objective, clause-aligned to scored decisions, and interface-compatible (all fixed prior to target evaluation). This yields a grounded structural proposal, not a positive-transfer guarantee.
  • What lies outside scope (aπa_\pi). Realized direction and strength (aπ,Δπ,χπ,B2,ha_\pi, \Delta_\pi, \chi^2_{\pi,B}, h) are unestimated and excluded from converter inputs (as Section 3.3 notes, candidates sharing FF can have opposite alignment). With a pre-frozen threshold, routing attains best-in-class average value over calibrated structural units rather than individual safety (Prop. 3.1; Appendix A.8), while surviving errors are diagnosed via execution evidence (Figure 10). We will state this boundary explicitly in Section 3.3.

[Q4] Handling Conflicting Module Decisions in a Single Skill

Conflicting module decisions are resolved when generating SKILL.md by assigning each source rule to the metric module that scores the decision step it directly controls, not by runtime priority arbitration. A Retain module fixes the choice for its own step, while a Discard module omits source rules for downstream steps instead of adding opposing directives (Section C.2), so the two never conflict.

  • Concrete assignment and removal in CAIS. rule_based_select_method and estimator-validity checks (such as the IV relevance test; Appendix F.1) govern estimator selection and are retained as metric-scoped prose in SKILL.md under MSA (Retain, Figure 16, e.g., forbidding matching/PSM on RCTs and selecting OLS or IV). Conversely, clauses governing post-selection tuning, support-gap heuristics, and F04 backtracking control numerical estimation and are removed under MRE (Discard, Figure 10), leaving covariate choice unconstrained once the estimator is chosen while keeping tool advisory outputs available on both routes.
  • Scope of claims on cross-metric coupling. Assigning rules by scored decision step does not eliminate cross-metric coupling, as choosing the right method does not determine numerical accuracy alone (captured by Rπ,BR_{\pi,B} in Eq. 3; e.g., in Figure 10, AdaSkill and Full both select IV yet produce estimates of 4049.36 versus 1469.96 under different covariates). We claim no theoretical optimality for hybrid routing (Appendix A.8), and Table 8's gains over Full (MSA 89.7/96.8 versus 84.6/90.3; MRE 23.1/15.9/4.3 versus 26.3/17.3/7.6) reflect the empirical effect of the composed skill as a whole.
Reviewer_bM8qOpenReview ↗

Thank you for the clarification

Thank you for the detailed response and for providing the raw records and recomputation code. The five-run records explain the BIRD SRR variance of 0.250 and the CE Real MSA Freedom range of [0, 0.592]. These resolve my two original concerns about the internal consistency of Tables 4 and 11. Please update the run counts and related pair counts consistently throughout the paper.

I have one follow-up question about MSA scoring. The supplementary README notes that some Synthetic reference labels (e.g., iv_encouragement and did_twfe) differ from the labels requested in the diagnostic prompt (iv and did), and the diagnostic code compares the strings directly. I understand that Synthetic MSA is excluded from the CE MSA routing coordinate. Could you clarify whether the final Synthetic MSA evaluation uses a label mapping, and whether correcting this mismatch would change any reported results or conclusions?

Reviewer_M6sgOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper studies how to turn a multi-agent workflow for data-science tasks into a skill for a single agent, and asks which parts of the workflow are worth keeping. The authors argue that tools and domain knowledge should always be kept, but step-by-step pipeline guidance only helps when there is a clear "right way" to solve the task. To tell these cases apart, they propose a measure called Behavior-Outcome Freedom, which checks whether runs that behave differently also score differently, and their method, AdaSkill, keeps the guidance only when this measure is low. The main evidence is an experiment where the same guidance on the same task helps under one metric and hurts under another. Experiments on 11 datasets across four task types show that the benefit of guidance falls as Freedom rises, and the resulting skills perform comparably to the original multi-agent systems while running much faster.

Critical strengths

1. The framing, and the experiment that supports it. I think the main idea here is a good one. When people distill a multi-agent system into a skill, they usually try to carry everything over. The paper argues that tools and domain knowledge are a different kind of thing from step-by-step pipeline rules: the first adds abilities, the second restricts how the agent searches. The scorer-reversal experiment shows this nicely. With the same workflow, tools and test instances, the pipeline rules help method-selection accuracy (+20.0 on Textbook, +25.0 on Real) but hurt numerical error (−10.3, −2.1). Only the metric changes, so the effect has to come from the rules themselves. This is the main reason I found the paper worth reading. "Keep everything" is clearly not a safe default, and that lesson applies well beyond these four tasks.

2. A cheap check before building the skill. Freedom is computed from a small number of base-agent runs (around $6 for causal estimation) before any skill is synthesized. The Full vs Discard comparisons are properly matched, and across 16 of them in four quite different task families the benefit of the pipeline goes down as Freedom goes up. The trend also holds with trace-based distances and with GPT-5.1, and the final skills are much faster than the source systems. If this holds up, it would be genuinely useful in practice. That is also why I weigh the held-out results below so heavily: the practical value of the paper depends on this signal working for task families it wasn't tuned on.

Critical weaknesses

1. Weak held-out evidence for the frozen threshold. The practical claim is that a threshold fitted on other task families tells you what to do for a new one. The current results don't clearly show this.

  • Headline table is partly in-sample. For causal discovery and Text-to-SQL, Table 1's AdaSkill numbers don't match the LOFO skills in Table 8. The causal-discovery numbers equal the Full arm, while LOFO routes causal discovery to Discard.
  • Held-out routing errors. Two of the six qualifying modules (CD F1, T2SQL EX) are routed against the sign of their own effect (App. A.10). The fold thresholds are just midpoints between the nearest modules, and the AUC of 1.0 is in-sample.
  • Routing mixed with synthesis noise. The LOFO skill sometimes beats both arms it is choosing between (CE Textbook MSA: 89.7 vs 84.6 / 76.9), so part of the RQ3 gain likely comes from regenerating the skill.

2. Small effects and a missing baseline. It's hard to tell whether Freedom adds anything beyond direct validation.

  • Effects within noise. BIRD EX differs by 2 of 147 questions, and the CD effects (1.5–3.8 F1) are smaller than their 3-run standard deviations (5–7).
  • Few data points. Three contrasts rest on fewer than three identifiable questions, and the trend comes from only four workflow regimes. The low MSA Freedom also holds partly by construction (App. C.1).
  • No direct-validation baseline. Freedom already needs gold-scored runs, and calibration already required building both skill versions. The paper never compares against simply building both and testing them on a small dev split.
Questions for authors

Clarifications:

  1. Do Tables 1, 15 and 17 report the full-data policy? Please add LOFO results to the main table.
  2. How do you reconcile the two held-out routing errors with the claim that the frozen policy generalizes to excluded workflows?
  3. Which backbone do the source multi-agent systems use? If it isn't Sonnet 4.6, the latency and accuracy comparisons are confounded.

Requested analyses. These would strengthen the paper and could raise my score:

  1. Results over several skill-synthesis seeds per arm, to separate routing from synthesis noise.
  2. A comparison with directly evaluating Full vs Discard on a small dev split at the same cost as computing Freedom.
2 comments
AuthorsOpenReview ↗

Official Response to Reviewer M6sg (1/2)

We thank Reviewer M6sg for recognizing the capabilities-versus-pipeline framing, the scorer-reversal experiment, and the cheap check before building the skill.


[W1] Generalization of Thresholds Fitted on Prior Task Families

There does exist a boundary that separates whether pipeline guidance pays off: below it the pipeline is beneficial, and above it, harmful. As long as a new family's FF falls outside the ambiguous band [0.426,0.692][0.426, 0.692] (the cut points produced across the four LOFO folds), we route directly—in the LOFO experiment, all 4 modules outside this band routed correctly in 100% of folds. Only when FF falls inside the band is direct measurement required, which follows from the held-out test itself (below).


[W1-1, Q1] In-Sample Main Table and Adding LOFO Results

Yes, Tables 1, 15, and 17 originally report the full-data policy, which is why the CD result in Table 1 equals the Full arm (whereas LOFO routes CD to Discard). We add the AdaSkill (LOFO) column below:

  • Under LOFO, CE and FE route identically to full-data; CD goes to Discard and T2SQL EX to Retain.
  • Across all 7 modules, LOFO retains 92.2% of the full-data normalized lift (31.09% vs. 33.71%).
Dataset / MetricAdaSkill (Full-Data)AdaSkill (LOFO)
CE (3-dataset mean MSA ↑ / MRE ↓)95.5 / 14.495.5 / 14.4
FE (2-dataset mean AUC ↑)74.174.1
CD AutoMPG / DWD / Sachs / Child (F1 ↑)65.2 / 70.1 / 92.4 / 95.266.7 / 66.3 / 89.3 / 91.8
CD Mean F1 ↑ (Construct Cost / Latency)80.7 ($8.5 / 3.1m)78.5 ($8.5 / 3.1m)
T2SQL BIRD-147 (EX / NSF / SRR ↑)67.4 / 94.1 / 80.866.7 / 93.8 / 80.2
T2SQL Spider-120 (EX / NSF / SRR ↑)66.7 / 60.5 / 7.565.8 / 60.2 / 7.5

[W1-2, Q2] Explaining the Two Held-Out Routing Errors (CD F1 and T2SQL EX)

These two errors are not due to insufficient sample size; they follow from the mathematical property of the "hold out the boundary point" test itself. Order the NN available FF values as f1<⋯<fk<t<fk+1<⋯<fNf_1<\dots<f_k<t<f_{k+1}<\dots<f_N, and let gLg_L, gBg_B, gRg_R be the gaps below, across, and above the boundary pair (fk−fk−1f_k-f_{k-1}, fk+1−fkf_{k+1}-f_k, fk+2−fk+1f_{k+2}-f_{k+1}). Holding out fkf_k moves the midpoint to (fk−1+fk+1)/2(f_{k-1}+f_{k+1})/2, so fkf_k is misrouted iff gB<gLg_B<g_L, and symmetrically fk+1f_{k+1} iff gB<gRg_B<g_R. Thus both are correct iff gBg_B is the largest of the three.

In our data gL=0.435g_L=0.435, gB=0.084g_B=0.084, gR=0.096g_R=0.096, so gBg_B is the smallest—both boundary modules are misrouted. The criterion depends only on the relative lengths of the three local gaps, so denser sampling does not systematically eliminate it; survival depends on where new modules land. It is a mathematical property of LOFO, not a design flaw or insufficient sampling.


[W1-3, Requested Analysis 1] Separating Routing Gains from Skill Re-Synthesis Randomness

We fully agree that this experiment is necessary and have carried it out. Multi-seed experiments establish that the direction of advantage between Full and Discard is completely stable across all seeds, so routing is not driven by random noise. The gain decomposition follows below.

  • Multi-seed stability (CE Textbook, 3 seeds). Across all 3 seeds, Full beats Discard on MSA (86.3% vs. 77.8%) and Discard beats Full on MRE (25.5% vs. 27.6%), with zero direction flips. AdaSkill is best or tied-best on MRE in every seed (24.5%) and beats Discard on MSA in all three (89.7% vs. 77.8%).
Arm (CE Textbook, 3 seeds)MSA Seed 1 / 2 / 3MSA Mean ±\pm SD ↑MRE Seed 1 / 2 / 3MRE Mean ±\pm SD ↓
Base Agent61.5 / 59.0 / 61.560.7 ±\pm 1.431.3 / 32.0 / 31.431.6 ±\pm 0.4
Full (Retain All)84.6 / 84.6 / 89.786.3 ±\pm 2.926.3 / 27.2 / 29.327.6 ±\pm 1.5
Discard (Remove All)76.9 / 82.1 / 74.477.8 ±\pm 3.923.1 / 27.0 / 26.525.5 ±\pm 2.1
AdaSkill (LOFO Composed)89.7 / 92.3 / 87.289.7 ±\pm 2.623.1 / 26.4 / 24.024.5 ±\pm 1.7
  • Pure routing, zero re-synthesis (7 modules). LOFO switching strictly between the pre-synthesized Full and Discard candidates (LOFO Routing Only) reaches a mean lift of 24.64%, +2.53 points above Always-Full (22.11%)—55% of the 26.72% static ceiling. Adding multi-metric composition (CE, T2SQL) raises it to 31.09% (+6.45 points). Both tables will be added to §4.5.
Policy (7 Modules, Normalized Lift %)Skill Candidates UsedMean Normalized Lift
Always Discard / Always FullFixed Discard / Full candidates21.90% / 22.11%
LOFO Routing Only (Zero re-synthesis)Selects pre-synthesized Full or Discard per module24.64%
Best Pre-Synthesized Candidate per ModuleBest of the two fixed candidates per module26.72%
Executed LOFO AdaSkill (Routing + Composition)Selected (CD, FE) + Hybrid Composed (CE, T2SQL)31.09%
AuthorsOpenReview ↗

Official Response to Reviewer M6sg (2/2)

[W2] Small Effect Sizes, Limited Sample Counts, and Direct-Validation Baseline

[W2-1] Small Individual Contrast Effects Within Run-to-Run Noise

These two boundary modules have very small point effects, and they are exactly the modules for which we already stated direct measurement is required (CD F1 F=0.602F=0.602, T2SQL EX F=0.686F=0.686). Our conclusion rests on the overall slope across all 16 contrasts, not any single data point:

  • Per unit increase in FF, the gain drops by −9.18-9.18 points (95% CI [−13.41,−4.94][-13.41, -4.94], p<0.001p<0.001, r=−0.798r=-0.798); on the 13 core contrasts the slope is −9.72-9.72.
  • Removing the weakest 1 to 6 modules in ascending order of effect size leaves the overall correlation ordering unchanged (Table 6). We will mark contrasts below 1 SD in Table 7.

[W2-2] Few Identifiable Questions and Metric-Representation Overlap

  1. On the small-sample concern. Removing the 3 contrasts with fewer than 3 valid samples leaves the 13 route-grade contrasts, where the slope remains strong (−9.72-9.72, CI [−14.00,−5.45][-14.00, -5.45]); removing any single task family also leaves a negative slope (−5.58-5.58 to −9.87-9.87).
  2. On definitional overlap. Partial constructed correlation does exist (Appendix C.1: when method labels agree, MSA necessarily agrees, forcing ρS≥0\rho_S \ge 0). Exactly one audit genuinely removes the method label, the response-free trace with both the final answer and selected-method fields stripped (§4.3): MSA and MRE stay clearly separated (0.550.55 vs. 1.121.12), and though the gap compresses from 0.9140.914 to 0.570.57, both the reversal and the binary routing are unchanged (0.55<0.644<1.120.55<0.644<1.12). The execution signature (Table 5) refines rather than removes the label, since estimator identity is retained at the coarse level; it tests representational stability, and separation persists (Textbook: 0.1810.181 vs. 1.0851.085; Real: 0.2550.255 vs. 0.7620.762), with exec coordinates preserving the matched ordering (r=−0.793r=-0.793).

[W2-3, Requested Analysis 2] Direct Validation at Equal Cost

At equal budget, direct validation ties on 82.9% of splits (equivalent to a coin flip), whereas reaching a stable decision costs 12.9x more than Freedom.

Freedom runs the cheap unguided Base Agent (USD 0.029/run on BIRD), whereas direct validation must first synthesize two complete Skills and then run skill-augmented trajectories on both (USD 0.759 per question on T2SQL).

Comparison RegimeCost Ratio (Direct Validation / Probe)MetricAgreement with FFDev Split TiesTest Gain (Direct Validation minus FF)95% Interval
Equal Dollar (m=1m = 1, $0.76 of $1.47 budget)1.0xBIRD EX49.3%92.4%+0.81 pts[−2.74,+5.48][-2.74, +5.48]
Equal Dollar (m=1m = 1, $0.76 of $1.47 budget)1.0xBIRD NSF52.5%73.3%-0.55 pts[−2.25,+0.02][-2.25, +0.02]
Equal Dollar Mean (EX + NSF)1.0xCombined50.9%82.9%+0.13 pts[−1.95,+2.41][-1.95, +2.41]
Equal Run Count (m=25m = 25, $18.98)12.9xBIRD EX41.2%30.2%+0.43 pts[−2.74,+5.48][-2.74, +5.48]
Equal Run Count (m=25m = 25, $18.98)12.9xBIRD NSF70.2%0.2%-0.54 pts[−3.01,0.00][-3.01, 0.00]
Equal Run Count Mean (EX + NSF)12.9xCombined55.7%15.2%-0.06 pts[−2.10,+1.85][-2.10, +1.85]
  • Equal dollar (m=1m=1). After Skill construction the budget covers only 1 development question per arm. Full and Discard tie in 82.9% of cases (EX 92.4%, NSF 73.3%), forcing random tie-breaking (50.9% agreement with FF).
  • Equal run count (m=25m=25). Direct validation now separates the two arms, yet its test performance is indistinguishable from Freedom (−0.06-0.06 points, 95% CI [−2.10,+1.85][-2.10, +1.85]) at $18.98 (12.9x Freedom; 5.2x on CE).
  • Combinatorial advantage. For kk independent metrics, enumerating all retain/discard combinations needs 2k2^k Skills, and coordinate-wise ablation needs 2k2k variants and kk times the evaluation budget; Freedom needs one batch of Base Agent trajectories for all kk modules.

[Q3] Backbone Model Used by the Four Source Multi-Agent Systems

All four source systems use Sonnet 4.6 (claude-sonnet-4-6). We re-ran all four (APEX-SQL, CAIS, MATMCD, FELA) on the same claude-sonnet-4-6 as Base Agent, EvoSkill, Compiler, Full, Discard and AdaSkill, so there is no model confounding. "Native" in §4.1 and Appendix B.1 means each source system's own orchestration and operating configuration, not a different backbone, and we will state this explicitly. FELA's per-run cost is an operating-range estimate (Appendix A.10, Figure 15).


[Presentation] Improving Readability and Clarity

Thank you for stating this candidly. We accept the criticism. We will (i) move the operational policy (Eq. 6, Prop. 3.1) into its own subsection, (ii) add a box fixing the three unit systems, and (iii) add a notation table for FF, FoutF_{\text{out}}, FtraceF_{\text{trace}}, FexecF_{\text{exec}} with an end-to-end routing example.

Reviewer_p95sOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes AdaSkill, which converts multi-agent workflows into single-agent skills. It introduces Behavior-Outcome Freedom to decide whether to retain pipeline guidance while preserving tools and domain knowledge. Experiments on 11 datasets across four data-science tasks show competitive performance with reduced execution overhead.

Critical strengths
  1. The paper addresses a practical problem: how to convert expensive multi-agent workflows into reusable single-agent skills.
  2. The Full-versus-Discard ablations are useful and show that retaining pipeline guidance can sometimes hurt performance.
Critical weaknesses
  1. Unclear multi-agent setting. I am not convinced that all four source systems are really multi-agent systems. For example, CAIS seems more like a staged tool pipeline with deterministic components. It is unclear whether the gains come from multi-agent distillation or simply simplifying existing workflows.

  2. Limited evidence for Freedom. The proposed metric is interesting, but why is it better than simpler measures like output diversity, score variance, or baseline accuracy? Some behavioral representations also overlap with the evaluation metrics, which may partly explain the correlation. The threshold is calibrated on only a few task families, so its generalizability is unclear.

  3. Efficiency gains are not well isolated. Much of the speedup seems to come from removing multi-agent orchestration, which the Full and Discard baselines also do. I would like to see how much efficiency improvement actually comes from Freedom-based selection, especially after accounting for the diagnostic and construction costs.

Questions for authors
  1. Which parts of the source workflows actually involve multiple independent LLM agents, rather than deterministic tools or fixed pipelines?
  2. How does Freedom compare with simpler routing heuristics under the same budget?
  3. How well does the calibrated threshold generalize to unseen workflows? In Appendix Table 8, the family-level decision does not select the better candidate for several causal discovery datasets.
  4. What is the additional efficiency benefit of Freedom-based selection over a direct workflow-to-skill conversion? What is the break-even point when including the diagnostic cost?
4 comments
AuthorsOpenReview ↗

Official Response to Reviewer p95s (1/2)

We thank Reviewer p95s for recognizing the practical value of distilling expensive workflows into reusable single-agent skills and our Full vs. Discard ablations showing when pipeline guidance helps or hurts.


[W1, Q1] Multi-Agent Source Workflows vs. Staged Pipelines

All four source systems coordinate multiple role-specialized components, falling along the multi-agent spectrum from cooperative agent teams. Specifically, CAIS coordinates eight role modules (input_parser, query_interpreter, dataset_analyzer, method_selector, method_validator, etc.), combining LLM stages with deterministic decision trees—matching our operational definition in §1 ("workflows to organize analytical control across roles, stages, shared state, tool calls, verification, and repair") and Verma et al. (2025).

Deterministic routing or validation modules do not disqualify a system from being multi-agent; standard frameworks such as AutoGen (round-robin selection) and MetaGPT (SOP pub/sub) rely on deterministic orchestration.

More importantly, we deliberately selected four multi-agent systems with different degrees of agentic structure to test source systems of different orchestration styles. The breakdown below records, for each source, which components are independent LLM-agent roles and which are deterministic.

Source SystemIndependent LLM-Agent Roles and OrchestrationDeterministic Tools and Components
APEX-SQL (T2SQL)8 LLM role modules (planner, dual-path schema pruner, semantic linker, table explorers, synthesizer, macro/micro SQL executors, 8-run vote arbitrator)Schema formatter, rule-based tip matcher, BM25 retriever, SQLite engine, vote clustering
CAIS (CE)5 LLM stages over shared state (input parser, dataset analyzer, query interpreter, argument configurator, explanation generator)Rule-based method_selector tree, method_validator checks, statistical estimators
MATMCD (CD)2 LLM agents (retrieval domain-context agent and causal-constraint agent interacting via multi-round debate and refinement)Statistical graph search (GES/PC) and constrained edge-orientation solver
FELA (FE)6 LLM agents over shared memory (proposer, synthesizer, creator, code generator, idea critic, code critic)UCB bandit selector, island evolutionary manager, Python executor, LightGBM evaluator

Why our findings hold across all four architectures. Every source system externalizes control into mandatory ordering rules and validation gates. In our Full vs. Discard ablations (§4.4), both arms remove runtime orchestration and share identical tools and domain knowledge, differing solely in whether source pipeline rules are retained. Thus, our gains isolate selective rule retention rather than generic workflow simplification.


[W2, Q2] Comparison with Simpler Heuristics and Metric-Representation Overlap

Under the exact same Base Agent diagnostic runs and budget, Freedom (FF) separates the CE scorer reversal by 10.33x on Textbook and 2.95x on Real, achieving the strongest correlation with structural lift across both the 10 per-question contrasts (ρ=−0.58\rho = -0.58) and all 16 contrasts (r=−0.80,p=0.0002r = -0.80, p = 0.0002).

1. Comparison Against Simpler Heuristics (audit/simple_diagnostics.py)

Diagnostic HeuristicSpearman ρ\rho with Lift (τpipe\tau_{\text{pipe}})Scorer Reversal Separation (Textbook / Real)Scale-Invariant & Directional (F>1F > 1)?
Behavior-Outcome Freedom FF (Ours)-0.58 (r=−0.80r = -0.80 across all 16)10.33x / 2.95x (0.098→1.0120.098 \to 1.012 / 0.236→0.6970.236 \to 0.697)Yes (Proposition A.7)
Unsigned Rank Association (1−∣ρS∣1 - \vert{}\rho_S\vert{})-0.537.23x / 2.67x (0.098→0.7090.098 \to 0.709 / 0.236→0.6300.236 \to 0.630)Scale-invariant, but collapses sign (ρS≶0\rho_S \lessgtr 0)
Score Variance / SD across Runs+0.401.69x / 1.43xNo (confounded by binary vs. continuous scale)
Baseline Accuracy / Utility+0.18 (r=+0.03,p=0.90r = +0.03, p = 0.90 across all 16)1.12x / 1.07x (fails to reverse)No
Output Diversity (Mean Behavioral Distance)+0.041.00x / 1.00x (identical across metrics)No
  • Output Diversity and Baseline Accuracy fail. MSA and MRE share the exact same Base Agent runs (1.00x diversity ratio), yet rule retention helps MSA (+20.0 to +25.0 pts) and harms MRE (-10.3 to -2.1 pts); baseline utility is likewise uncorrelated with lift (r=0.033,p=0.904r = 0.033, p = 0.904).
  • Score SD is scale-dependent, and Unsigned Association hides anti-concordance. Raw Score SD is mechanically larger on binary {0,1}\{0, 1\} metrics (MSA, EX) than on continuous [0,1][0, 1] metrics (MRE, NSF), limiting separation to 1.43x–1.69x. Meanwhile, 1−∣ρS∣1 - \vert{}\rho_S\vert{} folds anti-concordant MRE questions (ρS,q<0,Fq>1\rho_{S,q} < 0, F_q > 1, up to 1.3551.355 in Table 11) back below 11, dropping separation from 10.33x to 7.23x.
AuthorsOpenReview ↗

Official Response to Reviewer p95s (2/2)

2. Auditing Metric-Representation Overlap

Because CE's default behavioral distance uses the selected estimator class (overlapping with MSA; Appendix C.1), we verified that the correlation holds even when the method label is removed or refined (§4.3, Table 5):

  • Response-free process traces (FtraceF_{\text{trace}}), stripping both final answers and selected-method fields, preserve the full reversal (0.550.55 for MSA vs. 1.121.12 for MRE; gap 0.570.57) and maintain strong negative correlation across all 16 contrasts (r=−0.79,p<0.001r = -0.79, p < 0.001, Figure 3b).
  • Hierarchical execution signatures (FexecF_{\text{exec}}) over packages, dataframe columns, and hyperparameters yield 0.1810.181 (MSA) vs. 1.0851.085 (MRE) on Textbook (gap 0.900.90) and 0.2550.255 vs. 0.7620.762 on Real (gap 0.510.51), preserving cross-task correlation when replacing canonical CE coordinates (r=−0.793,p=0.0002r = -0.793, p = 0.0002, Appendix C.2).

[W2, Q3] Generalization of the Calibrated Threshold to Unseen Workflows

Across Leave-One-Family-Out (LOFO) folds, the frozen threshold retains 92.2% of full-data lift (31.09% vs. 33.71%) and beats Always-Full (22.11%) and Always-Discard (21.90%) in 97.8% of bootstrap samples.

  • Why CD shifts in Table 8, and why regret is minimal. Across all 6 modules, −F-F orders treatment signs with zero inversions (t=0.644t = 0.644, midpoint of CD F1 at 0.6020.602 and T2SQL EX at 0.6860.686). Holding out CD shifts the fold midpoint to 0.4260.426, routing CD to Discard. However, on Child, DWDClimate, and Sachs, Full exceeds Discard by only 3.1–3.8 F1 pts—smaller than the 3-run SD of 5–7 F1 pts (Table 15)—while on AutoMPG Discard beats Full (66.7 vs. 65.2). Routing CD to Discard still outperforms Source (65.3), Base (72.5), and Compiler (71.8) by +6.0 to +13.2 F1 pts, while all non-boundary modules are routed with 100% accuracy.

[W3, Q4] Isolating Efficiency Gains and Break-Even Analysis

Beyond the 2.4x–17.7x macro speedup from orchestration removal, FF-based pruning cuts per-query execution cost by an additional 7%–52% over Always-Full by removing high-FF validation and backtracking loops. Including upfront probe and conversion costs, AdaSkill breaks even within 10–40 queries against Always-Full (CD, FE, T2SQL) and 20–75 queries against Direct Conversion (CE, T2SQL).

1. Inference Savings Beyond Orchestration Removal

Retaining high-FF pipeline rules inside a single-agent skill still forces mandatory multi-step checks, retries, and backtracking at inference time:

  • Adding pipeline rules alone (Pipeline Only) inflates per-query cost by +88.6% on FE and +125.6% on T2SQL over the Base Agent, whereas Discard (Tools + Knowledge) cuts cost by -5.2% on FE and -36.7% on T2SQL (Figures 5 and 10).
  • Compared directly against matched Always-Full (Appendix A.10), FF-based pruning lowers per-query cost across every task family: -7.2% in CE ($0.223 →\to $0.207), -17.1% in CD ($3.108 →\to $2.577), -21.5% in FE ($6.177 →\to $4.846), and -51.8% on T2SQL Discard modules ($0.512 →\to $0.247).

2. Break-Even Point Including Diagnostic and Construction Costs

Because Direct Conversion (Workflow Compiler / Always-Full) and AdaSkill share identical one-time conversion costs (CconvC_{conv} = $7.91, $2.36, $5.50, $3.97 for CE, CD, FE, T2SQL), CconvC_{\text{conv}} cancels out in head-to-head comparisons, leaving only the one-time diagnostic probe cost (CprobeC_{\text{probe}}). Using per-family probe costs (which we will also update in Figure 8 and add to Appendix E):

Task FamilyOne-Time Probe Cost (CprobeC_{probe})Per-Query Saving vs. Always-FullBreak-Even vs. Always-FullComparison vs. Direct Conversion (Workflow Compiler)
CD$5.47 (4 datasets ×\times 5 runs)$0.531 / query~10 queries+8.9 F1 pts higher accuracy (80.7 vs. 71.8)
FE$15.30 (Taobao, 5 runs)$1.331 / query~12 queries+1.3 AUC pts higher accuracy (74.1 vs. 72.8) & 17% faster (35.6m vs. 42.9m)
T2SQL$10.51 (BIRD $1.47 + Spider $9.04)$0.265 / query~40 queries (~6 queries on BIRD routing)Saves $0.140/query (breaks even at ~75 queries) + +4.6 EX pts on Spider
CE$8.20 (3 datasets ×\times 30 runs)$0.016 / query~510 queriesSaves $0.401/query ($0.207 vs. $0.608, breaks even at ~20 queries) + +4.7 MSA / -4.4 MRE pts

Even against native Source Workflows (amortizing both Cconv+CprobeC_{\text{conv}} + C_{\text{probe}}), AdaSkill breaks even in 15 queries on CD, 2 on FE, and 54 on T2SQL (while cutting CE latency by 2.3x and MRE by 19.7 pts).

Thank you for the detailed response and the additional experiments. I find Behavior–Outcome Freedom useful for deciding how much workflow guidance to retain. The causal estimation example illustrates this well. On the same instances, retaining the pipeline helped the agent select the right method but hurt numerical estimation.

I’m keeping my original rating of 2 (Weak Reject). I see AdaSkill primarily as a practical heuristic for configuring an existing LLM agent. The additional experiments make a stronger case that the heuristic is useful, but I still find the methodological novelty limited.

AuthorsOpenReview ↗

Response on Methodological Novelty and Analytical Framework

Thank you for recognizing our response and finding our method useful for configuring LLM agents.

Regarding novelty, AdaSkill is designed to replace previous heuristic and brute-force MAS-to-skill methods. Previous methods either copy the entire workflow indiscriminately (e.g., Workflow Compiler) or rely on trial-and-error search on a validation set (e.g., EvoSkill or direct validation) that scales as 2k2^k across kk modules. In contrast, AdaSkill introduces a formal framework to diagnose and distill MAS workflows.

While splitting and routing components is intentionally simple as a final operation, the core contribution of our paper is the whole analytical framework behind it. Specifically, we introduce Behavior-Outcome Freedom (F) and innovatively link pre-synthesis behavior-outcome geometry to component-level transfer gains, supported by theoretical bounds and controlled experiments. This analytical framework and takeaway (which Reviewers M6sg and bM8q also highlighted as a core strength) is the central value of our work.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response to the Area Chair and All Reviewers

We thank the Area Chair and all reviewers for their constructive feedback and for recognizing our paper's core strengths.

  • Selective Transfer Framing. We convert high-overhead workflows into low-latency single-agent skills by separating capability resources (always retained) from pipeline guidance (conditionally retained) (bM8q, M6sg, p95s).
  • Controlled Mechanism Evidence. Our capability-matched Full vs. Discard ablations and scorer reversal experiment show that the same pipeline helps method selection (+20.0 to +25.0 pts) but harms numerical estimation (-10.3 to -2.1 pts) on the same instances (bM8q, M6sg, p95s).
  • Low-Cost Pre-Synthesis Diagnostic. Behavior-Outcome Freedom (FF) diagnoses this structural sensitivity from a few base-agent runs before skill synthesis (bM8q, M6sg).

Below is a summary of our new experiments, raw execution logs, and clarifications.

  1. Complete Raw Logs and 100% Statistical Reproduction (Reviewer bM8q). Every output-space entry in Tables 4 and 11 reproduces identically from our time-stamped logs (2026-04-01) and verification script in supplementary.zip. In our sensitivity sweep (Section C.5, which ran up to 20 executions per item and subsampled NN without replacement), N=6N = 6 was identified as a recommended operating point, and the main-batch descriptions in Sections 4.6, B.3, and C.1 inadvertently wrote N=6N = 6 instead of the executed n=5n = 5 main-batch run count. Updating those text statements to n=5n = 5 changes zero performance, Freedom, threshold, or routing numbers; the USD 6.12 diagnostic allowance in Figure 8 is a modeled cost for one 6×66 \times 6 execution unit (Appendix A.10); our response to p95s reports the measured per-family probe costs.
  2. Multi-Seed Synthesis, Routing Isolation, and Direct Validation Baseline (Reviewer M6sg). Across 3 independent skill-synthesis seeds per arm on CE Textbook, the scorer reversal holds on every seed (Full > Discard by +8.5 MSA pts, Discard < Full by -2.0 MRE pts), and AdaSkill achieves the best 3-seed mean on both metrics (89.7% MSA and 24.5% MRE), though its MSA margin over Full is within seed noise at n=3n = 3 and we do not rest on it. Isolating pure LOFO routing on fixed pre-synthesized candidates (zero re-synthesis) still captures 55% of the pre-synthesized selection headroom (+2.53 lift over Always-Full). Comparing FF against Direct Validation ("Racing") shows that at equal dollar cost (m=1m = 1 dev question/arm), Direct Validation ties on 73% to 92% of dev splits and therefore selects essentially at random, while matching our 50-run sample size (m=25m = 25/arm) costs 5.2x to 12.9x more for the same test accuracy (-0.06 pts difference).
  3. Simpler Heuristics, Workflow Breakdown, and Break-Even Volume (Reviewer p95s). Under the same runs and budget, FF separates the CE scorer reversal by 2.95x to 10.33x and correlates more strongly with structural lift (r=−0.80,p=0.0002r = -0.80, p = 0.0002 across all 16 contrasts and ρ=−0.58\rho = -0.58 on per-question contrasts) than Unsigned Association (ρ=−0.53\rho = -0.53), Score SD (ρ=+0.40\rho = +0.40), Baseline Utility (ρ=+0.18\rho = +0.18), and Output Diversity (ρ=+0.04\rho = +0.04). Pruning high-FF pipeline loops reduces per-query cost by 7% to 52% over Always-Full, breaking even on diagnostic and construction costs in 10 to 40 queries on CD, FE, and T2SQL against Always-Full and 20 to 75 queries on CE and T2SQL against Direct Conversion.
  4. Unified Backbone and Held-Out Table 1 Results (Reviewers M6sg, p95s). All four native source workflows were re-run on the exact same claude-sonnet-4-6 backbone as AdaSkill and all baselines. We also add the held-out LOFO column to Table 1, which retains 92.2% of full-data lift (31.09% vs. 33.71%) and beats both fixed policies in 97.8% of bootstrap samples. We narrow the generalization claim accordingly: across the four folds there are 7 held-out module evaluations and 5 route correctly, the 2 errors being exactly the two modules whose midpoint is the pooled cut, each failing only in the fold that removes it. Every module outside the observed fold-threshold range [0.426,0.692][0.426, 0.692] routes correctly in every fold, so for a new family whose FF falls inside that range we now recommend measuring the matched Full-versus-Discard effect or defaulting to Discard.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.