- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper focus on whether a weaker LLM can detect when a stronger agent composes tools to boost capabilities from masked, incomplete traces. The authors introduce NegoBench, a multi-issue negotiation benchmark. Execution logs deterministically label each round as L1 (communication only), L2 (isolated tool use), or L3 (composed tool use). Building on this, they propose SCI: it first classifies the evidence format of each round (bundled, itemized, or narrative), then applies corresponding rules to identify tool composition.
Critical strengths
This experimental setting reflects a real-world need.
Critical weaknesses
-
The paper has notable formatting flaws. In Figure 3, subfigures (a) and (b) overlap with each other, making it impossible to read the performance of the respective methods. In the appendix, only RQs 6, 8, 10, 12, and 13 are present, with the remaining RQs missing entirely. It feels like the authors did not carefully read the paper before submission.
-
Figure 3 shows that SCI is basically tied with the baselines in most cases, so it is hard to see any clear advantage.
Questions for authors
The prompt for SCI feels overly hand-crafted, with so many rules that look tuned specifically to fit NegoBench's trace format. This undermines the "domain-agnostic" claim and makes one wonder whether the gains actually come from stratification or just a finer rubric. There is an ablation missing: giving the baseline the exact same detailed rules without any stratification.
2 comments
W1: Formatting
We thank the reviewer for raising the formatting issues. The revised paper will separate the subfigures of Figure 3 so that each method's performance is clearly readable, and will renumber the research questions consecutively and update all cross-references.
W2: SCI performance relative to baselines
Averaged across all visibility conditions and monitors, SCI obtains a G-mean of 0.53 compared with 0.46 for baseline and 0.45 for sequential (RQ1, Section 4.2), with non-overlapping bootstrap 95% confidence intervals (SCI [0.492, 0.563], baseline [0.425, 0.490], sequential [0.416, 0.482]). SCI outperforms both reference strategies in all six visibility conditions, with the largest margins under Expanded visibility where intermediate tool-call records are available (SCI 0.50 to 0.56 vs. baseline and sequential 0.40 to 0.44). SCI maintains the highest G-mean in most awareness conditions (Figure 3(b)), and its detection does not require prior tool knowledge, showing near-zero change between K0 and K1 while baseline and sequential degrade (RQ3, Section 4.4). The advantage holds across all three monitor models (Appendix D.2). The appendix further compares SCI against alternative detection strategies including Profile-then-Probe and a rule-based dependency detector (Appendix D.5, RQ12). In combination with the ablation above, these results demonstrate that SCI provides a consistent and decomposable advantage in the conditions most relevant to deployment. The revised paper will foreground this conditional performance profile so that readers can identify where the method operates best.
Summary
The requested ablation, the corrected figures, and the per-condition results resolve all of the reviewer's concerns. The exact ablation the reviewer requested shows that stratification drives a 36% relative improvement over an identical rubric at the same inference budget, SCI reaches a G-mean of 0.722 against 0.197 for the baseline on the full 180-episode run of the most demanding visibility condition with Unaware agents, SCI outperforms both reference strategies in every one of the six visibility conditions with non-overlapping confidence intervals, and SCI maintains a strong lead when coordination spans multiple turns. The revised paper will also correct Figure 3 and the appendix numbering.
We would be deeply grateful if the reviewer would reconsider the soundness assessment and the overall score in light of these results, especially as the reviewer recognized that our setting reflects a real-world need. We would be glad to discuss any remaining points.
We thank Reviewer GUgf for the constructive feedback.
We thank reviewer GUgf for recognizing that "this experimental setting reflects a real-world need" and for the direct question about whether SCI's gains come from the stratification mechanism or the detailed rubric. The ablation below isolates each component and answers this question directly.
Q1: Stratification ablation
The review asks for "giving the baseline the exact same detailed rules without any stratification." We conducted exactly this four-method ablation on a CoT-Only subset (o4-mini monitor, K0 where the monitor does not receive the agent's tool inventory, Unaware agents, on randomly selected episodes). Four methods isolate each component.
The ablation confirms that stratification itself drives SCI's detection gains beyond the detailed rubric. Adding verbatim SCI compositional indicators to the baseline prompt (unstratified rubric) raises G-mean from 0.00 to 0.39 [0.28, 0.48]. The original baseline scores 0.00 on this subset because CoT-Only is the most demanding visibility condition. Action inputs and outputs are hidden, so composition must be inferred from the agent's reasoning alone, a setting where the baseline struggles to identify compositional signals. The baseline recognizes the individual operations of an executed composition but does not aggregate them into one coordinated operation, and it labels 168 of the 173 turns as L2, 5 as L1, and none as L3. The full 180-episode run of the same configuration with Unaware agents confirms that this result reflects the condition and extends well beyond the subset. On all 1,563 turns, SCI reaches a G-mean of 0.722 with an L3 recall of 53.2%, against 0.197 and 4.0% for the baseline, at a comparable specificity of 98.1% and 98.3%. SCI's compositional indicators are designed for exactly this condition, since they teach the monitor to read repeated rationales, named steps, and narrative coordination cues as evidence of a single coordinated operation. Even under the same single-call inference budget, adding evidence-type classification (stratified one-call) raises G-mean to 0.53 [0.43, 0.61], a 36% relative improvement over the unstratified rubric at an identical inference budget, demonstrating that stratification itself improves detection at no additional computational cost. The full two-step architecture reaches 0.74 [0.66, 0.80]. This decomposition demonstrates that SCI's gains extend well beyond those of a more detailed prompt. The evidence-type classification step provides a structurally distinct source of detection signal by routing different trace patterns to type-appropriate compositional indicators, enabling the monitor to recognize coordination whether it surfaces as bundled outputs, itemized operations, or goal-directed reasoning.
On the concern that the rules are tuned to NegoBench, the SCI indicators describe how coordination appears in a trace, as bundled outputs, itemized operations, or goal-directed reasoning, and contain no negotiation-specific tool names or scenario rules. Two results confirm this generality beyond the within-turn trace format. SCI's detection does not depend on the tool inventory (+0.01 between K0 and K1, against −0.03 and −0.04 for baseline and sequential), and SCI maintains a strong lead when we broaden L3 to also cover dependencies that agents carry across turns through dialogue history without composition operators, reaching a G-mean of 0.60 against 0.00 for the baseline.