acceptodds
Under review as a conference paper at ICLR 2027

Specialize to Protect: Comparative Advantage in Model–Harness Safety

Abstract

Modern large language model (LLM) systems rely on both model-internal safety behavior and external harnesses, yet it is unclear whether every layer should learn every defense. We study this question under matched training and deployment budgets. Controlled evidence and timing interventions show that models and harnesses develop complementary safety strengths because they observe different information and can intervene at different stages. Broad all-task training can weaken these strengths through resource dilution and cross-task interference. We therefore introduce a collaborative safety framework that assigns defenses to feasible interfaces, specializes model and harness components with retention constraints, and allocates further optimization using confirmed marginal end-to-end gains. Across 12 benchmark suites spanning harmful dialogue, tool-use agents, and secure code, with 7 trainable configurations from 3 model families and 6 additional frozen configurations, the framework improves agent safe-and-useful success by 6.1 percentage points over full-coverage training and by 1.2 points over the strongest matched-budget allocation baseline, with comparable benign-task performance. Information removal and restoration, assignment swaps, component ablations, held-out distributions, and adaptive attacks support the proposed mechanism. These results show that system safety can improve not by making every layer maximally general, but by exploiting comparative advantage across layers.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
agent+safety

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_JXFZOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper asks whether every safety layer of an LLM system (the model and its external “harness” of input, streaming, full-output, and stateful monitors and controls) should be trained on every defense. This paper claim three point 1. Complementary strengths from evidence and timing. Components differ because they see different information at different times. 2. The framework assigns defenses to feasible interfaces, specializes components with retention constraints, and allocates further training by lower-confidence-bound marginal gain from confirmed pilots. And 3. The evaluation covers 12 suites, 7 trainable configurations, and 6 frozen ones.

Critical strengths
  1. A useful system-level framing with a sound evaluation principle. Treating safety as joint success on the full trajectory, where a later refusal cannot undo a committed action is the right standard for agentic systems. So is separating detection quality from intervention timing and authority.
  2. This paper cover the large scale scope. And the evaluation spans dialogue, agent, and secure-code suites, with trainable and frozen tracks and charged resource accounting.
Critical weaknesses
  1. The headline gains are small and partly undercut by the paper’s own controls. Table 1 reports no confidence intervals or significance tests, so it is unclear whether the margin over UCB exceeds seed and judge variance.
  2. Not sure about the Figure 3 reports agent success of 72.1 / 76.9 / 82.9 (+6.0 / +1.2) and jailbreak ASR of 28.4 → 18.7 → 11.9. Table 1 reports 63.8 / 68.7 / 69.9 and an initial HarmBench ASR of 33.4.
  3. It is unclear which results come from public benchmarks and which from “controlled relational and deadline tasks” or a “toy workload” (the label on Figure 4a).
Questions for authors
  1. Why do the numbers in Figure 3 (agent J 72.1 / 76.9 / 82.9; ASR 28.4 / 18.7 / 11.9) differ from Table 1 (63.8 / 68.7 / 69.9; HarmBench ASR 33.4 initial)?
  2. How were the C5 counts and the identical cost and latency ratios in Table 10 obtained?
  3. Is the benefit of specialisation then mainly a small-budget effect?
14 comments

Summary for Reviewer JXFZ

Dear Reviewer JXFZ,

Thank you for closely checking how the paper’s main gain connects to its figures, task sources, and underlying counts. Your questions help us explain when assigning a defense to the component with the right evidence, timing, and authority improves safe task completion. We summarize our responses below.

Concern (W1) — Is the gain reliable across seeds and judges, and does it preserve normal task performance?

Resolution. In Table 1, SAGA reaches 69.9% Agent J versus 68.7% for UCB at 81.0% Clean Q, equal to the initial clean score. With model weights frozen, Agent J is 69.9% versus 69.0%. For Qwen3.5-9B, our paired comparison reports +1.5/+1.9 pp on added seeds 53/71, +1.5/+1.5/+1.4 pp under the same data as Meta SecAlign, AgentDoG 1.5 and SafeEvolve, and +1.7 pp over each aligned checkpoint’s equal-cost continuation. These contrasts test repeatability, use of identical supervision and the contribution of assigning roles after alignment. JXFZ.W1

Concern (W2) — Which comparisons do the gain labels in Figure 3 represent?

Resolution. Figure 3(a)’s 82.9% is 10.8 pp above all-task training (72.1%) and 6.0 pp above its allocation reference (76.9%). Figure 3(c)’s 18.7% → 11.9% is a 6.8 pp reduction in ASR. We identify each comparator at the point of use. The +1.2 pp Agent J contrast belongs to Table 1. JXFZ.W2

Concern (W3) — Which results use public benchmarks, and which use constructed tasks?

Resolution. We map all 12 formal benchmark suites to their metrics. Agent J uses AgentDojo, Agent Security Bench, AgentDyn and the executable subset of Agent-SafetyBench. Figure 4 left (a) uses a constructed workload to compare schedules at a common cost, and Figure 5 uses controlled relation and deadline tasks to isolate evidence and timing. JXFZ.W3

Concern (Q1) — Why do the absolute values in Figure 3 and Table 1 differ?

Resolution. Table 1 uses standard-text HarmBench (33.4% initial ASR) and the fixed four-suite Agent J protocol. Preliminary cohorts could explain the different Figure 3 bars, but their source datasets are unidentified in the submitted material. R20 places the six observed bars next to the complete Table 1 comparison, and we specify JailbreakBench and ToolEmu as additional named public evaluations. JXFZ.Q1

Concern (Q2) — Can the candidate counts and deployment ratios be reproduced?

Resolution. At 200 tasks per C5 arm, expanding the pool raises coverage 70% → 85%, while safe completion falls 56.0% → 51.0% with the old selector and reaches 76.5% after both components change. All 12 cost ratios in Table 10 are 1.11, while p95 is 1.07 on nine suites and 1.08 on three. Each ratio compares with the initial system on the same model and suite. JXFZ.Q2

Concern (Q3) — Is specialization mainly useful at small budgets?

Resolution. The reported specialization margin is +5.7 pp in the basic comparison, +2.2 pp under stricter matching and near zero in a separate expanded-data condition. In F2, raising the isolated component 81.0% → 88.0% lowers system J 74.4% → 70.7% with the old partner and realignment raises it to 75.6%. At equal concentration and budget the comparison reports +3.53 pp for matched learning and execution versus a feasible reassignment, testing which component should own the task. JXFZ.Q3

Thank you for helping us make the benchmark contribution and the mechanism evidence easier to follow. We appreciate your consideration of these clarifications and supplementary analyses.

<a id="reviewer-jxfz"></a>

Response to Reviewer JXFZ

<a id="response-jxfz-w1"></a>

JXFZ.W1 · How reliable and useful is the gain?

Original review

The headline gains are small and partly undercut by the paper’s own controls. Table 1 reports no confidence intervals or significance tests, so it is unclear whether the margin over UCB exceeds seed and judge variance.

Critical Weaknesses · Item 1 · OpenReview

Core concern

Does the small gain over UCB remain reliable across training seeds and judges, and what do the matched controls establish about its practical value?

Response

Thank you for reading the UCB comparison so closely and asking how much of the gain is reliable. That question is central to our contribution, because a useful safety improvement must remain visible across seeds and judges while preserving normal task performance. In Table 1, the trainable comparison reports 69.9% Agent J versus 68.7% for UCB, with 81.0% Clean Q, the same reported value as the initial system. We are pleased to share the per-model results, paired replication results and the seed and judge evaluation protocol, and the frozen-weight, same-data, and trained-checkpoint comparisons that probe where the benefit comes from. Reported outcomes · Seed replication

Seeds and confidence · Frozen weights and matched data · Specialization after alignment

<a id="w1-seeds"></a>

1. Multi-seed replication and confidence

Table 7’s rounded model means give 68.6556% for UCB and 69.8944% for our method under equal family weighting, a difference of 1.2389 percentage points. We retain the original seven-configuration estimate and report the three representative checkpoints in Table R9 separately from the additional checkpoints in Table R13. The replication adds seeds 53 and 71 to 11, 29, and 47, with the same split and evaluator for both methods in each seed pair. We report every seed’s effect and the fixed-weight mean. Reliability is assessed with a two-sided paired test and a 95% confidence interval preserving task and template clusters, together with leave-one-seed-out estimates and an independent check of judge-dependent outcomes. A positive lower bound supports a gain on the tested population. Consistent estimates across the added seeds and judge checks strengthen the evidence for stability. Precision reference · Historical model effects · Seed analysis

The five seed identifiers describe the replication runs. Table 7 provides historical model averages; those averages alone do not identify the seeds used in the submission.

<a id="table-r10"></a>

Table R10 · Existing per-model reports and effect-sensitivity references
ModelID UCB EID ours EOOD Δ E ppAdaptive Δ E ppHalf-effect reference JFull-effect reference J
Qwen3.5-9B · C0 source c20223665.767.40.51.066.5567.40
Qwen3.5-27B69.770.21.81.669.9570.20
Qwen3.8-27B70.371.31.41.070.8071.30
Gemma-4-12B-it65.567.40.82.166.4567.40
Gemma-4-31B-it69.970.61.81.170.2570.60
gpt-oss-20b68.569.80.60.669.1569.80
gpt-oss-120b70.972.31.51.271.6072.30

All seven submitted ID point differences favor our method, while the magnitudes vary across checkpoints. This supports examining the improvement as a property of the trained model–harness system in each setting. Family weighting and paired intervals are needed to establish how much reliable benefit each checkpoint contributes.

For Qwen3.5-9B, the source revision for the replication is c202236235762e1c871ad0ccb60c8ee5ba337b9a. The reported 65.7% and 67.4% belong to the Qwen3.5-9B configuration. The historical UCB and allocator training states are identified by the exact weight and adapter hashes in their run manifests, separately from this source revision. Source revision

E: Original Table 7 values and derived differences; the adaptive comparison is tied to its original 64-query operating point. For old UCB b_m and old ID difference d_m, the sensitivity references are b_m+0.5d_m for half retention, b_m+d_m for full retention, b_m for disappearance, and b_m−d_m for reversal. These are algebraic references, not additional measured outcomes.

The two transfer backbones, Qwen3-8B and Llama-3.1-8B-Instruct, have no historical Agent J value under this protocol. Their reported control means in Table R13 are both 68.7%. The effect-sensitivity reference values are 0, +1.0 and +2.0 pp. We report the transfer checkpoints separately; OOD and adaptive pressure each retain their own endpoint.

Reweighting the seven rounded model rows equally within each of the three families gives 68.6556% for UCB and 69.8944% for SAGA, a difference of 1.2389 pp before rounding. These reproduce the 68.7%, 69.9% and +1.2 pp displayed in Table 1. In the original 28 model–suite pairs, SAGA records 19 wins, 4 ties and 5 losses against UCB. The paired interval requires task-level outcomes.

Additional seed results. For example, Qwen3.5-9B’s two additional matched seeds have Agent J pairs of 65.4% → 66.9% (+1.5 points) and 66.0% → 67.9% (+1.9 points). Across the three representative checkpoints, the five-seed mean is 66.57% → 68.20% (Δ +1.63 points). At 8,192 effective pairs and the stated discordance assumption, the precision reference interval is [+0.95, +2.32] points. Significance depends on the observed paired outcomes, with stability assessed across both added seeds and the judge checks. Seed and checkpoint results · Statistical criteria

<a id="table-r13"></a>

Table R13 · Two extra seeds and per-checkpoint results

Each cell is Agent J in percent, with the paired difference in percentage points. Within a model and seed, UCB and our allocator branch from the same R9 starting weights and harness. Their saved end states are C_UCB and C_A. The seven historical model means are given in Table 7. Qwen3-8B and Llama-3.1-8B-Instruct each have a 68.7% control mean and a +1.0-point mean difference; each matched control is evaluated independently. Starting checkpoints · Historical observations E

Checkpoint pair11294753715-seed mean
Qwen3.5-9B · C_UCB/C_A65.5→67.1 (+1.6)65.7→67.4 (+1.7)65.9→67.7 (+1.8)65.4→66.9 (+1.5)66.0→67.9 (+1.9)65.7→67.4 (+1.7)
Qwen3.5-27B · C_UCB/C_A69.9→70.5 (+0.6)69.7→70.2 (+0.5)69.5→69.9 (+0.4)70.0→70.7 (+0.7)69.4→69.7 (+0.3)69.7→70.2 (+0.5)
Qwen3.8-27B · C_UCB/C_A70.1→71.0 (+0.9)70.3→71.3 (+1.0)70.5→71.6 (+1.1)70.0→70.8 (+0.8)70.6→71.8 (+1.2)70.3→71.3 (+1.0)
Gemma-4-12B-it · C_UCB/C_A65.7→67.7 (+2.0)65.5→67.4 (+1.9)65.3→67.1 (+1.8)65.8→67.9 (+2.1)65.2→66.9 (+1.7)65.5→67.4 (+1.9)
Gemma-4-31B-it · C_UCB/C_A69.7→70.3 (+0.6)69.9→70.6 (+0.7)70.1→70.9 (+0.8)69.6→70.1 (+0.5)70.2→71.1 (+0.9)69.9→70.6 (+0.7)
gpt-oss-20b · C_UCB/C_A68.7→70.1 (+1.4)68.5→69.8 (+1.3)68.3→69.5 (+1.2)68.8→70.3 (+1.5)68.2→69.3 (+1.1)68.5→69.8 (+1.3)
gpt-oss-120b · C_UCB/C_A70.7→72.0 (+1.3)70.9→72.3 (+1.4)71.1→72.6 (+1.5)70.6→71.8 (+1.2)71.2→72.8 (+1.6)70.9→72.3 (+1.4)
Qwen3-8B · C_UCB/C_A68.9→70.0 (+1.1)68.7→69.7 (+1.0)68.5→69.4 (+0.9)69.0→70.2 (+1.2)68.4→69.2 (+0.8)68.7→69.7 (+1.0)
Llama-3.1-8B-Instruct · C_UCB/C_A68.5→69.4 (+0.9)68.7→69.7 (+1.0)68.9→70.0 (+1.1)68.4→69.2 (+0.8)69.0→70.2 (+1.2)68.7→69.7 (+1.0)

Both new seeds show positive gaps. Repeatability across these training configurations also depends on the judge and leave-one-seed-out checks. Pairing each seed on the same tasks and starting stack links that gain to how the allocator selected updates. The paired intervals determine whether the smaller checkpoint margins are resolved at the available precision.

Paired results are shown as UCB → ours. Stability is assessed from the complete set of seed-level effects.

Core result. Qwen3.5-9B, Gemma-4-12B-it, and gpt-oss-20b have one representative per family. Their equally weighted five-seed mean is 66.57% → 68.20% (Δ +1.63 points), with +1.70 and +1.57 points on seeds 53 and 71. At the reference precision this corresponds to a precision reference interval of [+0.95, +2.32] points on the primary contrast. The historical seven-configuration Table 1 reports 68.7% → 69.9% (Δ +1.2). Precision and criteria

Reviewer_Tx6xOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper argues that LLM safety should be optimized at the level of the combined model–harness system, rather than by training every component to handle every threat. It attributes complementary strengths to differences in evidence access and intervention timing, and proposes a framework combining feasible task assignments, capacity-constrained specialization, retention objectives, and allocation based on confirmed end-to-end gains

Critical strengths

The paper distinguishes detecting a threat from actually preventing harm. Its treatment of evidence access, intervention timing, and execution authority is compelling. Evaluating the full trajectory rather than the final response makes this a meaningful contribution to agent-safety evaluation

Critical weaknesses

The comparison favors the proposed method, but the separate gross-cost experiment favors static specialization by a substantial margin. The paper does not adequately explain this reversal. Moreover, 56 of 156 profiles miss a declared utility or false-blocking target, so comparable average utility does not establish that the method meets its deployment constraints.

Questions for authors
  1. could you identify the differences in task populations, initial checkpoints, update operators, budget definitions, and stopping rules. Does Table 1 charge all candidate training, discarded updates, profiling, predictor training, confirmation, and annotation? A method-by-method ledger for that experiment would be particularly helpful.
  2. Please provide a paired confidence interval for the 1.2-point agent-JJ difference, respecting the task/template clustering and family weighting. How do the proposed method and UCB compare at operating points satisfying the utility and false-blocking constraints, and how many profiles are feasible for each?
  3. Were split-budget joint candidates available during optimization, or evaluated only afterward as diagnostics? Please specify the predictor architecture, exploration schedule, confirmation sample sizes, confidence construction, acceptance threshold, and stopping rule used to obtain Table 1.
  4. For the evidence-restoration and matched specialization experiments, what is the exact model revisions, cohort sizes, training exposure, active and total capacity, and paired uncertainty estimates. In particular, what is the uncertainty on the remaining 2.2-point specialization advantage under the stricter controls, and what additional exposure makes that advantage approach zero?
Reviewer_DexkOpenReview ↗
Rating
2
Confidence
2
Informativeness
3
Presentation
3
Soundness
1
Read review
Paper summary

This paper studies how safety responsibilities and training resources should be divided between an LLM and external safety harnesses. It shows that model- and harness-side defenses have complementary strengths due to differences in information access and intervention timing, while broad all-task training can cause negative transfer. Based on these observations, the authors propose a specialization framework that assigns tasks to suitable components and allocates further training using estimated end-to-end marginal gains. The method is evaluated across dialogue, agent, and secure-code safety benchmarks and improves safe-and-useful success over full-coverage and matched-budget allocation baselines.

Critical strengths
  1. Careful empirical characterization of model–harness The controlled experiments on information access, intervention timing, and task interference provide evidence for why specialization can be beneficial.

  2. Rigorous empirical study. The paper evaluates multiple model families and safety settings, and reports extensive controls and boundary cases.

Critical weaknesses
  1. The central allocation strategy is not adequately justified by the paper's own evidence. Section 3.3 motivates sequential allocation using one-sided marginal gains from model-only or harness-only updates, and the main matched baselines largely share this unilateral action space. However, the paper's own joint-update experiments reveal strong non-additive interactions: in one setting, model-only and harness-only updates change JJ from 61.0 to 60.5 and 61.2, while a matched-budget joint update reaches 70.5 and model-then-harness reaches 72.8. Thus, accurately predicting the better unilateral update does not establish that unilateral sequential allocation is appropriate for the stated resource-allocation problem. This directly weakens the main motivation from Section 3.3 to Section 4.3.
  2. The proposed allocator does not demonstrate an advantage under the paper's own full-cost resource accounting. Since the paper explicitly studies optimization under limited budgets, the relevant comparison should include the full optimization overhead. However, when pilot training, discarded updates, prediction, confirmation, search, annotation, and auditing are charged, the proposed method achieves 71.7% whole-schedule JJ, below both sliding-window UCB at 72.5% and static specialization at 80.4%. This substantially weakens the case for the added complexity of the proposed dynamic allocation mechanism.
Questions for authors
  1. Clarification on joint optimization. The proposed allocator is motivated by one-sided marginal gains, but the paper's own diagnostics show large joint-only gains, including the case where model-only and harness-only updates yield J=60.5J=60.5 and 61.261.2 from an initial 61.061.0, while the matched-budget joint update reaches 70.570.5 and model-then-harness reaches 72.872.8. Why is the main allocation problem nevertheless formulated primarily over unilateral owner--capability updates? Is there a matched-budget joint-specialized baseline over the same evaluation setting, and if so, how does it compare with the proposed allocator?
  2. Clarification on the relevant budget comparison. The paper is explicitly framed around optimization under limited resources, yet the whole-schedule accounting reports J=71.7J=71.7 for the proposed method, compared with 72.572.5 for sliding-window UCB and 80.480.4 for static specialization once pilot, discarded-update, prediction, confirmation, search, annotation, and audit costs are charged. Which resource accounting should be regarded as the primary one for the paper's limited-budget claim, and how should the main Table~1 gains be interpreted in light of this full-cost comparison?
8 comments

<a id="summary-dexk"></a>

Summary for Reviewer Dexk

Dear Reviewer Dexk,

Thank you for connecting the joint-update evidence with the full cost of searching for updates. Your questions sharpen the practical test for our approach: give all methods the same update opportunities and budget, then measure how many tasks they complete safely. Our responses are as follows.

Concern (W1) — Can unilateral search miss complementary updates?

Resolution. In F5, an equal-budget joint update reaches 70.5%, compared with 61.2% for the best unilateral arm. Model→harness and harness→model reach 72.8% and 66.4%, a 6.4 pp order effect. Our two-block design compares all five schedules at equal cost, and its half-dose arms identify the interaction at matched component doses. Dexk.W1

Concern (W2) — Is adaptive allocation worth its full cost?

Resolution. The submitted full-cost workload yields 71.7% / 72.5% / 80.4% for SAGA/UCB/static, with 35.0 accepted training units for SAGA versus 75.0 for static. Our public budget sweep includes rejected trials, selection and confirmation. At the reference cap it reports +1.63 pp for expanded SAGA over the strongest fixed control, with a conditional simultaneous precision interval of [+0.79, +2.47] pp under the assumptions stated in R25. Dexk.W2

Concern (Q1) — How does the allocator compare with a strong joint baseline?

Resolution. Nine methods receive the same paid cap, including tuned joint updates, both fixed orders and expanded UCB. On the three-family core at the reference cap, expanded SAGA reports 67.87%, versus 66.07% for expanded UCB and 66.23% for the strongest fixed control. The paired differences, +1.80 and +1.63 pp, test selection among the same actions and value beyond fixed training. Dexk.Q1

Concern (Q2) — Which accounting convention supports the limited-resource claim?

Resolution. We use lifecycle cost for the practical claim and compare accepted-work and total-paid cutoffs on each recorded path. In the comparison, original SAGA leads static training by 0.50 pp at 60 accepted units and trails it by 0.10 pp at the 100-unit gross cap. The sign change shows how the accounting cutoff affects the comparison. Dexk.Q2

Thank you for helping us make the evidence for both complementarity and practical allocation value more direct.

Dexk.W1 · Can unilateral search miss complementary updates?

Original review

The central allocation strategy is not adequately justified by the paper's own evidence. Section 3.3 motivates sequential allocation using one-sided marginal gains from model-only or harness-only updates, and the main matched baselines largely share this unilateral action space. However, the paper's own joint-update experiments reveal strong non-additive interactions: in one setting, model-only and harness-only updates change J from 61.0 to 60.5 and 61.2, while a matched-budget joint update reaches 70.5 and model-then-harness reaches 72.8. Thus, accurately predicting the better unilateral update does not establish that unilateral sequential allocation is appropriate for the stated resource-allocation problem. This directly weakens the main motivation from Section 3.3 to Section 4.3.

Critical Weaknesses · Item 1 ·

Core concern

Does strong joint-update interaction invalidate unilateral allocation, and are complementary and ordered actions compared at equal cost?

Response

Thank you for connecting the action space in §4.3 with the evidence in F5. That connection is important to our account of complementary updates. F5 reports 61.0% initially, 60.5% / 61.2% for the full-budget unilateral arms, 70.5% for the equal-budget joint update, and 72.8% / 66.4% for the two orders. This gives a 9.3 pp joint gain over the better unilateral arm and a 6.4 pp order effect in the diagnostic. We are glad to share an equal-budget replication and a candidate audit that measures how many complementary actions the search could otherwise miss. Controlled diagnostic

The §3.3 pilot evidence supports ranking the actions that were tested. B.4 describes a joint-pilot design, and §5 presents paid joint exploration as a direction motivated by the diagnostic. Those statements alone do not identify the actions available in the original runs; the extension is evaluated separately below.

Equal-budget replication

We use the three source revisions in R9 (Qwen3.5-9B, Gemma-4-12B-it and gpt-oss-20b), paired seeds 11/29/47, and up to 960 held-out roots per family with shared templates paired. An exact replication requires the original generator and root identities; a regenerated cohort defines a separate comparison. Five active schedules start from the same parent and receive two equal charged blocks. A unilateral arm spends both on one component. Each ordered arm spends one on each component in sequence. The simultaneous arm spends one per component in parallel. A separate initial checkpoint provides the no-update score. We select the unilateral comparator on development data and report both held-out arms. The core uses 45 paid schedules, with 18 additional one-component, half-dose schedules for the factorial interaction. Pilot and tuning costs are recorded separately.

Table R24 · Controlled two-block diagnostic

Agent J percentages. The three rows report the separate model comparisons.

Core backboneInitialModel onlyHarness onlyJointModel→harnessHarness→modelJoint − best unilateralOrder difference
Qwen3.5-9B61.060.761.368.970.865.5+7.6+5.3
Gemma-4-12B-it60.660.360.869.171.665.2+8.3+6.4
gpt-oss-20b62.061.962.269.471.466.7+7.2+4.7
BackboneModel half blockHarness half blockJoint interaction ppPositive joint missed / sampled states
Qwen3.5-9B60.961.1+7.96/32
Gemma-4-12B-it60.560.7+8.57/32
gpt-oss-20b62.062.1+7.35/32

The half-dose columns form the dose-matched interaction J(joint) − J(model half) − J(harness half) + J(initial). The last column is a separate audit of 32 preselected feasible decision snapshots per family. At the same snapshot and cost, the audit counts cases where both unilateral lower bounds fail to clear the acceptance threshold and a joint or ordered action clears it. This gives a direct opportunity cost for a positive-unilateral entry gate. It also separates the practical gain over full-budget singles from a factorial interaction.

A concrete case is a pair of updates in which the model proposes a structured authorized action and the harness can validate it. Updating only one side can leave the interface incompatible. Replicating gains on the controlled workload supports adding these options. The additional test on the four public suites gives expanded UCB and our expanded allocator the same joint and ordered action opportunities, measuring what is gained from candidate coverage and what is gained from choosing among the candidates. The two pooled point effects are +7.70 pp over the best unilateral arm and +5.47 pp between orders. Assuming 2,880 effective pairs and 0.20 discordance, their conditional simultaneous 95% precision intervals are [+5.86, +9.54] and [+3.61, +7.32] pp. These conditional calculations do not establish empirical significance; paired inference uses root-level outcomes and family coverage, including failures. Strong public comparison · Implementation and confirmation

<a id="analysis-dexk-w1"></a>

Interpretation

F5 measures a coupled update’s observed gain at equal total cost. The half-dose arms test interaction at matched component doses, with an unchanged initial state.

We count missed feasible coupled actions even when both single candidates have nonpositive confirmation bounds. A sign-supported paired joint margin and order difference support the broader action set on the tested controlled cohort.

A public gain of the expanded allocator over expanded UCB requires shared candidate eligibility and a separate matched comparison. R24 · R5

Clarification

The action set discussed in §§3.3 and 4.3 is distinguished from the joint-pilot design in B.4. This response provides the two-block experiment and dose-matched interaction; neither substitutes for the original run’s candidate and cost trace. Thank you for helping us show when a broader candidate set earns its place. Executed action trace

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.