- Rating
- 2
- Confidence
- 1(−1)
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper presents an empirical study of how LLM agents handle conflicting information and authorization boundaries in impossible software-repair tasks. The authors propose a process-level analysis framework (Information → Interpretation → Action) and evaluate three LLMs under different peer-information, instruction, and tool settings. The results reveal model-specific decision patterns and complementary behaviors in multi-agent collaboration.
Critical strengths
S1. Meaningful research question.
The paper highlights the limitations of outcome-only safety evaluation and provides a useful process-level perspective for understanding agent behavior under conflicting constraints.
S2. Multi-dimensional experimental design.
The study systematically examines the effects of peer information, claimed authority, instructions, and tools, and extends the analysis to multi-agent collaboration, providing interesting behavioral insights.
Critical weaknesses
W1. Limited validation of process-level analysis.
The paper reveals behavioral differences beyond final outcomes but does not demonstrate whether process-level indicators provide additional value for assessing agent safety or predicting violations.
W2. Limited experimental robustness.
The study uses only seven tasks, with some conditions having 2–5 valid episodes. The lack of repeated trials and uncertainty analysis weakens the reliability of the conclusions.
W3. Unreliable process-level annotations.
Interpretation and deliberation rely on manual annotations without reported inter-annotator agreement. Differences in reasoning visibility across models further raise concerns about comparability.
W4. Limited realism of experimental settings.
Agents are explicitly informed of task infeasibility and potential rule-violating shortcuts, rather than encountering these naturally during execution. This raises concerns about whether the observed behaviors reflect realistic agent deployments.
Questions for authors
See the critical weaknesses above
2 comments
Thank you for the detailed and candid response. I appreciate the authors' clarifications and acknowledge the potential value of the process-level analysis. However, I still have concerns about the limited sample sizes, lack of repeated trials, and reliability of the manual annotations.
Overall, the rebuttal has alleviated some of my concerns. If a score of 2.5 were available, I would choose it. For now, I will maintain my rating of 2 and lower my confidence to 1.
Response to Reviewer onA1 (W1–W4)
W1 — Limited validation of process-level analysis
Accepted. The paper argues that outcome-only evaluation is insufficient (§6.1–6.2) but never demonstrates that process-level indicators change a safety judgement. The demonstration is available in Table 2 and we did not frame it as one.
An outcome-only reading of the three protected edits gives Sol 2, Gemini 1, Fable 0 — three small numbers with overlapping intervals, inviting the conclusion that one mitigation serves all three. The per-condition detail says otherwise. Within the forged-authorization condition, strict instruction held fixed, Sol's deliberation column shows the authority conflict already live under typed tools (1/7 deliberations, 0/7 edits) and rising under open tools (5/7, 2/7): the gate is action cost, after interpretation has turned permissive. Gemini's single edit appears only under the benchmark prompt and is self-reverted on the next turn (§4.1, Table 2 note c); under the strict instruction it refuses the same bait and the edit never becomes a live option: the gate sits earlier, at interpretation.
Same outcome column, different position in the evidence → interpretation → action chain, therefore different mitigations: a harder write path for one, contract text for the other. An outcome-only evaluation would have prescribed the wrong one for one of them.
Two limits we state ourselves. This rests on three protected edits, Gemini's half on one; it shows the framework separates these cases, not that it predicts them. And the deliberation column it uses is the single-analyst annotation discussed under W3 — the agreement statistic we commit to there covers this argument too.
W2 — Limited experimental robustness
Accepted, and we would rather be direct than argue around it. We ran one episode per task per cell, so there are no repeated trials and no within-cell variance estimate. That was a budget decision, not a methodological claim: with fixed compute we chose breadth over depth at any single cell.
That permits the structural result, and the prompt-wording effect survives testing at this N. It does not permit a powered per-condition estimate on crossing, where the corpus holds three events: detecting a difference of the observed size at 80% power needs about 130 episodes per arm, roughly twenty times our per-cell N. We therefore report that outcome as a bounded null — the three non-forged conditions at [0.0%, 2.5%]. See the General Response for the scope we claim.
W3 — Unreliable process-level annotations
Accepted on both halves.
On agreement: the labels came from a single-analyst hand-adjudication pass (§3.4). We did not compute an inter-annotator agreement statistic, and we will not generate one retrospectively and present it as though it had governed the original coding. We will state this as a limitation and commit to a double-coded subsample with Cohen's κ for the camera-ready. The adjudication worksheet is released so the judgements can be inspected directly.
On comparability: not comparing raw reasoning rates across models (§3.4) addresses only part of the reviewer's concern. The larger threat is differential exclusion. Table 2's N column carries it: strict-parser failures remove a very different share of each model's episodes, and Gemini is worst affected, its Strict+Open cells falling to N = 2–4. The failure is an envelope problem rather than task incompetence — predominantly literal newlines inside JSON reasoning payloads. We did not repair the parser mid-study, deliberately: doing so would have made repaired episodes incomparable with every episode already scored under the strict contract. We would make the same choice again, but the revision will state the cost rather than leave it in the N column: Gemini is coverage-limited, its cells are the smallest in the table, and its rates rest on a non-randomly selected subsample. We will flag every affected cell.
W4 — Limited realism
Accepted as a real gap; it will be in the Limitations section. Two points bear on it, neither closing it:
The residual gap — the agent is told a shortcut exists rather than finding it — stands.