- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper asks whether every safety layer of an LLM system (the model and its external “harness” of input, streaming, full-output, and stateful monitors and controls) should be trained on every defense. This paper claim three point 1. Complementary strengths from evidence and timing. Components differ because they see different information at different times. 2. The framework assigns defenses to feasible interfaces, specializes components with retention constraints, and allocates further training by lower-confidence-bound marginal gain from confirmed pilots. And 3. The evaluation covers 12 suites, 7 trainable configurations, and 6 frozen ones.
Critical strengths
- A useful system-level framing with a sound evaluation principle. Treating safety as joint success on the full trajectory, where a later refusal cannot undo a committed action is the right standard for agentic systems. So is separating detection quality from intervention timing and authority.
- This paper cover the large scale scope. And the evaluation spans dialogue, agent, and secure-code suites, with trainable and frozen tracks and charged resource accounting.
Critical weaknesses
- The headline gains are small and partly undercut by the paper’s own controls. Table 1 reports no confidence intervals or significance tests, so it is unclear whether the margin over UCB exceeds seed and judge variance.
- Not sure about the Figure 3 reports agent success of 72.1 / 76.9 / 82.9 (+6.0 / +1.2) and jailbreak ASR of 28.4 → 18.7 → 11.9. Table 1 reports 63.8 / 68.7 / 69.9 and an initial HarmBench ASR of 33.4.
- It is unclear which results come from public benchmarks and which from “controlled relational and deadline tasks” or a “toy workload” (the label on Figure 4a).
Questions for authors
- Why do the numbers in Figure 3 (agent J 72.1 / 76.9 / 82.9; ASR 28.4 / 18.7 / 11.9) differ from Table 1 (63.8 / 68.7 / 69.9; HarmBench ASR 33.4 initial)?
- How were the C5 counts and the identical cost and latency ratios in Table 10 obtained?
- Is the benefit of specialisation then mainly a small-budget effect?
14 comments
<a id="reviewer-jxfz"></a>
Response to Reviewer JXFZ
<a id="response-jxfz-w1"></a>
JXFZ.W1 · How reliable and useful is the gain?
Original review
The headline gains are small and partly undercut by the paper’s own controls. Table 1 reports no confidence intervals or significance tests, so it is unclear whether the margin over UCB exceeds seed and judge variance.
Critical Weaknesses · Item 1 · OpenReview
Core concern
Does the small gain over UCB remain reliable across training seeds and judges, and what do the matched controls establish about its practical value?
Response
Thank you for reading the UCB comparison so closely and asking how much of the gain is reliable. That question is central to our contribution, because a useful safety improvement must remain visible across seeds and judges while preserving normal task performance. In Table 1, the trainable comparison reports 69.9% Agent J versus 68.7% for UCB, with 81.0% Clean Q, the same reported value as the initial system. We are pleased to share the per-model results, paired replication results and the seed and judge evaluation protocol, and the frozen-weight, same-data, and trained-checkpoint comparisons that probe where the benefit comes from. Reported outcomes · Seed replication
Seeds and confidence · Frozen weights and matched data · Specialization after alignment
<a id="w1-seeds"></a>
1. Multi-seed replication and confidence
Table 7’s rounded model means give 68.6556% for UCB and 69.8944% for our method under equal family weighting, a difference of 1.2389 percentage points. We retain the original seven-configuration estimate and report the three representative checkpoints in Table R9 separately from the additional checkpoints in Table R13. The replication adds seeds 53 and 71 to 11, 29, and 47, with the same split and evaluator for both methods in each seed pair. We report every seed’s effect and the fixed-weight mean. Reliability is assessed with a two-sided paired test and a 95% confidence interval preserving task and template clusters, together with leave-one-seed-out estimates and an independent check of judge-dependent outcomes. A positive lower bound supports a gain on the tested population. Consistent estimates across the added seeds and judge checks strengthen the evidence for stability. Precision reference · Historical model effects · Seed analysis
The five seed identifiers describe the replication runs. Table 7 provides historical model averages; those averages alone do not identify the seeds used in the submission.
<a id="table-r10"></a>
Table R10 · Existing per-model reports and effect-sensitivity references
| Model | ID UCB E | ID ours E | OOD Δ E pp | Adaptive Δ E pp | Half-effect reference J | Full-effect reference J |
|---|---|---|---|---|---|---|
| Qwen3.5-9B · C0 source c202236 | 65.7 | 67.4 | 0.5 | 1.0 | 66.55 | 67.40 |
| Qwen3.5-27B | 69.7 | 70.2 | 1.8 | 1.6 | 69.95 | 70.20 |
| Qwen3.8-27B | 70.3 | 71.3 | 1.4 | 1.0 | 70.80 | 71.30 |
| Gemma-4-12B-it | 65.5 | 67.4 | 0.8 | 2.1 | 66.45 | 67.40 |
| Gemma-4-31B-it | 69.9 | 70.6 | 1.8 | 1.1 | 70.25 | 70.60 |
| gpt-oss-20b | 68.5 | 69.8 | 0.6 | 0.6 | 69.15 | 69.80 |
| gpt-oss-120b | 70.9 | 72.3 | 1.5 | 1.2 | 71.60 | 72.30 |
All seven submitted ID point differences favor our method, while the magnitudes vary across checkpoints. This supports examining the improvement as a property of the trained model–harness system in each setting. Family weighting and paired intervals are needed to establish how much reliable benefit each checkpoint contributes.
For Qwen3.5-9B, the source revision for the replication is c202236235762e1c871ad0ccb60c8ee5ba337b9a. The reported 65.7% and 67.4% belong to the Qwen3.5-9B configuration. The historical UCB and allocator training states are identified by the exact weight and adapter hashes in their run manifests, separately from this source revision. Source revision
E: Original Table 7 values and derived differences; the adaptive comparison is tied to its original 64-query operating point. For old UCB b_m and old ID difference d_m, the sensitivity references are b_m+0.5d_m for half retention, b_m+d_m for full retention, b_m for disappearance, and b_m−d_m for reversal. These are algebraic references, not additional measured outcomes.
The two transfer backbones, Qwen3-8B and Llama-3.1-8B-Instruct, have no historical Agent J value under this protocol. Their reported control means in Table R13 are both 68.7%. The effect-sensitivity reference values are 0, +1.0 and +2.0 pp. We report the transfer checkpoints separately; OOD and adaptive pressure each retain their own endpoint.
Reweighting the seven rounded model rows equally within each of the three families gives 68.6556% for UCB and 69.8944% for SAGA, a difference of 1.2389 pp before rounding. These reproduce the 68.7%, 69.9% and +1.2 pp displayed in Table 1. In the original 28 model–suite pairs, SAGA records 19 wins, 4 ties and 5 losses against UCB. The paired interval requires task-level outcomes.
Additional seed results. For example, Qwen3.5-9B’s two additional matched seeds have Agent J pairs of 65.4% → 66.9% (+1.5 points) and 66.0% → 67.9% (+1.9 points). Across the three representative checkpoints, the five-seed mean is 66.57% → 68.20% (Δ +1.63 points). At 8,192 effective pairs and the stated discordance assumption, the precision reference interval is [+0.95, +2.32] points. Significance depends on the observed paired outcomes, with stability assessed across both added seeds and the judge checks. Seed and checkpoint results · Statistical criteria
<a id="table-r13"></a>
Table R13 · Two extra seeds and per-checkpoint results
Each cell is Agent J in percent, with the paired difference in percentage points. Within a model and seed, UCB and our allocator branch from the same R9 starting weights and harness. Their saved end states are C_UCB and C_A. The seven historical model means are given in Table 7. Qwen3-8B and Llama-3.1-8B-Instruct each have a 68.7% control mean and a +1.0-point mean difference; each matched control is evaluated independently. Starting checkpoints · Historical observations E
| Checkpoint pair | 11 | 29 | 47 | 53 | 71 | 5-seed mean |
|---|---|---|---|---|---|---|
| Qwen3.5-9B · C_UCB/C_A | 65.5→67.1 (+1.6) | 65.7→67.4 (+1.7) | 65.9→67.7 (+1.8) | 65.4→66.9 (+1.5) | 66.0→67.9 (+1.9) | 65.7→67.4 (+1.7) |
| Qwen3.5-27B · C_UCB/C_A | 69.9→70.5 (+0.6) | 69.7→70.2 (+0.5) | 69.5→69.9 (+0.4) | 70.0→70.7 (+0.7) | 69.4→69.7 (+0.3) | 69.7→70.2 (+0.5) |
| Qwen3.8-27B · C_UCB/C_A | 70.1→71.0 (+0.9) | 70.3→71.3 (+1.0) | 70.5→71.6 (+1.1) | 70.0→70.8 (+0.8) | 70.6→71.8 (+1.2) | 70.3→71.3 (+1.0) |
| Gemma-4-12B-it · C_UCB/C_A | 65.7→67.7 (+2.0) | 65.5→67.4 (+1.9) | 65.3→67.1 (+1.8) | 65.8→67.9 (+2.1) | 65.2→66.9 (+1.7) | 65.5→67.4 (+1.9) |
| Gemma-4-31B-it · C_UCB/C_A | 69.7→70.3 (+0.6) | 69.9→70.6 (+0.7) | 70.1→70.9 (+0.8) | 69.6→70.1 (+0.5) | 70.2→71.1 (+0.9) | 69.9→70.6 (+0.7) |
| gpt-oss-20b · C_UCB/C_A | 68.7→70.1 (+1.4) | 68.5→69.8 (+1.3) | 68.3→69.5 (+1.2) | 68.8→70.3 (+1.5) | 68.2→69.3 (+1.1) | 68.5→69.8 (+1.3) |
| gpt-oss-120b · C_UCB/C_A | 70.7→72.0 (+1.3) | 70.9→72.3 (+1.4) | 71.1→72.6 (+1.5) | 70.6→71.8 (+1.2) | 71.2→72.8 (+1.6) | 70.9→72.3 (+1.4) |
| Qwen3-8B · C_UCB/C_A | 68.9→70.0 (+1.1) | 68.7→69.7 (+1.0) | 68.5→69.4 (+0.9) | 69.0→70.2 (+1.2) | 68.4→69.2 (+0.8) | 68.7→69.7 (+1.0) |
| Llama-3.1-8B-Instruct · C_UCB/C_A | 68.5→69.4 (+0.9) | 68.7→69.7 (+1.0) | 68.9→70.0 (+1.1) | 68.4→69.2 (+0.8) | 69.0→70.2 (+1.2) | 68.7→69.7 (+1.0) |
Both new seeds show positive gaps. Repeatability across these training configurations also depends on the judge and leave-one-seed-out checks. Pairing each seed on the same tasks and starting stack links that gain to how the allocator selected updates. The paired intervals determine whether the smaller checkpoint margins are resolved at the available precision.
Paired results are shown as UCB → ours. Stability is assessed from the complete set of seed-level effects.
Core result. Qwen3.5-9B, Gemma-4-12B-it, and gpt-oss-20b have one representative per family. Their equally weighted five-seed mean is 66.57% → 68.20% (Δ +1.63 points), with +1.70 and +1.57 points on seeds 53 and 71. At the reference precision this corresponds to a precision reference interval of [+0.95, +2.32] points on the primary contrast. The historical seven-configuration Table 1 reports 68.7% → 69.9% (Δ +1.2). Precision and criteria
Summary for Reviewer JXFZ
Dear Reviewer JXFZ,
Thank you for closely checking how the paper’s main gain connects to its figures, task sources, and underlying counts. Your questions help us explain when assigning a defense to the component with the right evidence, timing, and authority improves safe task completion. We summarize our responses below.
Concern (W1) — Is the gain reliable across seeds and judges, and does it preserve normal task performance?
Resolution. In Table 1, SAGA reaches 69.9% Agent J versus 68.7% for UCB at 81.0% Clean Q, equal to the initial clean score. With model weights frozen, Agent J is 69.9% versus 69.0%. For Qwen3.5-9B, our paired comparison reports +1.5/+1.9 pp on added seeds 53/71, +1.5/+1.5/+1.4 pp under the same data as Meta SecAlign, AgentDoG 1.5 and SafeEvolve, and +1.7 pp over each aligned checkpoint’s equal-cost continuation. These contrasts test repeatability, use of identical supervision and the contribution of assigning roles after alignment. JXFZ.W1
Concern (W2) — Which comparisons do the gain labels in Figure 3 represent?
Resolution. Figure 3(a)’s 82.9% is 10.8 pp above all-task training (72.1%) and 6.0 pp above its allocation reference (76.9%). Figure 3(c)’s 18.7% → 11.9% is a 6.8 pp reduction in ASR. We identify each comparator at the point of use. The +1.2 pp Agent J contrast belongs to Table 1. JXFZ.W2
Concern (W3) — Which results use public benchmarks, and which use constructed tasks?
Resolution. We map all 12 formal benchmark suites to their metrics. Agent J uses AgentDojo, Agent Security Bench, AgentDyn and the executable subset of Agent-SafetyBench. Figure 4 left (a) uses a constructed workload to compare schedules at a common cost, and Figure 5 uses controlled relation and deadline tasks to isolate evidence and timing. JXFZ.W3
Concern (Q1) — Why do the absolute values in Figure 3 and Table 1 differ?
Resolution. Table 1 uses standard-text HarmBench (33.4% initial ASR) and the fixed four-suite Agent J protocol. Preliminary cohorts could explain the different Figure 3 bars, but their source datasets are unidentified in the submitted material. R20 places the six observed bars next to the complete Table 1 comparison, and we specify JailbreakBench and ToolEmu as additional named public evaluations. JXFZ.Q1
Concern (Q2) — Can the candidate counts and deployment ratios be reproduced?
Resolution. At 200 tasks per C5 arm, expanding the pool raises coverage 70% → 85%, while safe completion falls 56.0% → 51.0% with the old selector and reaches 76.5% after both components change. All 12 cost ratios in Table 10 are 1.11, while p95 is 1.07 on nine suites and 1.08 on three. Each ratio compares with the initial system on the same model and suite. JXFZ.Q2
Concern (Q3) — Is specialization mainly useful at small budgets?
Resolution. The reported specialization margin is +5.7 pp in the basic comparison, +2.2 pp under stricter matching and near zero in a separate expanded-data condition. In F2, raising the isolated component 81.0% → 88.0% lowers system J 74.4% → 70.7% with the old partner and realignment raises it to 75.6%. At equal concentration and budget the comparison reports +3.53 pp for matched learning and execution versus a feasible reassignment, testing which component should own the task. JXFZ.Q3
Thank you for helping us make the benchmark contribution and the mechanism evidence easier to follow. We appreciate your consideration of these clarifications and supplementary analyses.