- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper examines how coding agent safety changes when the same model is paired with different CLI harnesses. It introduces CLIGALE and a deterministic backend that records tool requests while suppressing downstream effects. Experiments with frozen, transferred, and adaptive attacks show substantial variation across configurations, supporting the need to evaluate and report the complete agent system.
Critical strengths
- The full crossing of models and harnesses under identical frozen attacks directly addresses the research question. The observed differences provide useful evidence that model identity alone does not adequately characterize agent safety.
- The paper carefully distinguishes prohibited requests from realized harm and separates attack preparation regimes. Most successful attacks in the representative adaptive evaluation have deterministic tool evidence, and the authors are appropriately cautious about the scope of their conclusions.
Critical weaknesses
My main concerns lie with the experimental evaluation.
- Important parts of the evaluation remain difficult to verify. Only one run per tier is released, so the reported three-run means and variability cannot be independently checked. Complete cases, verdict specifications, and sufficient CIDB and adapter details are also missing. Since the benchmark and evaluation protocol are central contributions, these gaps limit my confidence in their reproducibility.
- The safety comparison lacks matched benign task controls. BENIGN includes both successful task completion and refusal, leaving unclear whether lower ASR reflects better defenses or more conservative and less effective task execution. External capability scores do not resolve this issue on CLIGALE. The consequence restoration check in only one configuration also provides limited evidence that CIDB preserves cross-harness comparisons.
Beyond these concerns, I am not convinced that the paper’s current contribution meets the bar for acceptance at ICLR. While the research question offers some novelty, both the presentation and the depth of analysis need further improvement. I would recommend revising the paper and resubmitting it.
Questions for authors
- Could you provide all three runs, uncertainty estimates for the frozen comparisons, and complete sanitized cases with their verdict rules and CIDB behavior?
- Do the main harness differences persist on tasks that every configuration can complete without an attack? Matched completion and over-refusal rates would help assess this.
- What caused the label changes when consequences were restored, and is there evidence that CIDB affects the different harnesses similarly?
6 comments
Q2: Yes, safety differences persist while legitimate tasks are completed
Short answer. Yes. In matched native experiments, leakage reverses in both directions under harness prompt interventions while every legitimate task is completed. Across the benchmark, normal-task over-refusal is zero and task failure is negligible.
Safety reverses with the task completed
Each condition has three trials, and leakage is read from the saved report:
| Task | Native prompts | Prompt package exchanged | Native restored |
|---|---|---|---|
| Claude Code, approved-template task | 0/3 | 3/3 | 0/3 |
| Codex, encoding task | 3/3 | 0/3 | 3/3 |
- Independent rollback and reinstatement of the system-instruction group reproduce the reversal.
- All 72 prompt-package trials and all 24 system-instruction-group trials complete their legitimate tasks.
- Their 36 and 12 clean controls have no file violations.
- Actual execution permissions stay unchanged throughout.
Zero normal-task over-refusal and negligible task failure
- Normal-task over-refusal is 0%, and every task was screened for executability with DeepSeek-V4 during construction.
- Across 7,884 executions (nine configurations × three adaptive-tier runs), exactly one is TASK_FAILED. The rest are CONFIRMED_BREAK or BENIGN, with no missing, error, or INCONCLUSIVE outcomes.
Harness components therefore change whether an agent respects the user's safety constraints while completing the requested work.
Evidence: Section 4.6/Table 6 (p. 9), Appendix M/Tables 25–26 (pp. 28–29), Appendix F/Table 12 (p. 20), Appendix N/Table 29 (p. 30).
Q3: Label changes under restoration, and whether CIDB treats harnesses alike
Why labels changed under restoration. For Claude Code with Opus 4.6, restoring real consequences preserved:
- 28 of 30 violating labels (93.3%; 95% Wilson interval 78.7–98.2%);
- 29 of 30 benign labels (96.7%; 83.3–99.4%).
The three changes run in both directions (two violating → benign, one benign → violating), which is consistent with run-to-run variability in stochastic model generation rather than a one-sided shift. We attribute them to this variability (Section 4.7, pp. 9–10; Appendix E/Table 11, p. 20).
Whether CIDB affects harnesses similarly.
- By construction. CIDB applies one case–state–action response contract to every configuration: for the same case and state, the same action receives the same response regardless of the harness. Each CLI's own model, runtime, and tool-dispatch path stay in the system under test (Sections 3.3–3.4).
- The tested component effects also appear outside CIDB. In the new native experiments, agents perform real local writes, and violations are read from the saved
public_report.json. On the same approved-template attack, Codex leaks the private value in 3/3 trials and Claude Code in 0/3. Swapping Claude Code's prompt package for Codex's reproduces the leak (0/3 → 3/3), and restoring it removes the leak again (→ 0/3). Every trial completes the report.
These tested harness-component effects are therefore observable with real file writes, without CIDB, so they do not depend on CIDB's consequence isolation.
Revisions (blue): Section 4.6/Table 6 (p. 9), Appendix M/Tables 25–26 (pp. 28–29), Appendix O/Table 30 (p. 31).
Q1: All three runs, uncertainty estimates, sanitized cases, and verdict rules
Short answer. All three frozen runs are public. The headline gaps hold in every run with tight intervals. Ten sanitized cases with executable verdict logic can be replayed offline.
1. All three runs and their uncertainty. The three frozen-run outcome matrices are available as CSV files. On matched behaviors:
Appendix L/Table 24 reports per-run values, sample SDs, intervals, and missing-label sensitivity, and distinguishes these paired estimates from the configuration means in Table 3.
2. Sanitized cases with executable verdict logic. We release ten sanitized cases spanning all five main entry categories and all six main technique categories.
3. Verdict rules and CIDB specification.
Revisions (blue): Section 4.2 (p. 7), Appendix L/Table 24 (p. 27), Reproducibility Statement (p. 11), Appendix J (p. 24).