- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies the leakage of inference-time harness information in autonomous multi-agent systems and proposes Agent Harness Distillation (AHD). Through black-box probes, AHD induces a target system to describe its execution structure, aggregates the results of multiple interactions, and uses a coding agent to construct and iteratively optimize an executable harness. The paper evaluates the reconstructed system’s task performance, structural similarity, and transfer across backbone models, and proposes a defense that returns fabricated workflow descriptions.
Critical strengths
- The paper studies the leakage of orchestration logic, tool use, and execution workflows, extending the analysis of intellectual property leakage from models and static topologies to dynamic harnesses.
- It measures the agreement of extracted information with both attacked and clean executions, rather than with attacked executions alone.
- The evaluation goes beyond measuring information leakage: it constructs executable systems and assesses capability transfer across multiple target systems and backbone models.
Critical weaknesses
W1. Novelty and attribution of gains. Appendix E states that harness-level distillation has not been explored in prior work. This statement is too broad. AgentDistill transfers capabilities from teacher agents to smaller models through tool modules without updating model parameters; Black-Box Skill Stealing studies the extraction of proprietary agents’ skills and workflows through public interfaces. Section 7 states that existing methods rely on privileged access to subagent outputs, but MASLeak and CIA also start from a black-box setting in which only final outputs are observable. Please clarify the specific contributions of AHD relative to these works and provide comparisons on subtasks where their settings are compatible.
In addition, AHD uses both a strong coding agent and iterative optimization based on validation performance. Two controls are missing: (1) automatic optimization without target information, with all other conditions held constant; and (2) directly asking the target to summarize its execution workflow, followed by the same reconstruction pipeline. These comparisons would quantify the additional gains from extracting target information over autonomous harness design and direct querying. In Figure 1, “Train” could be labeled more specifically as “harness optimization,” with an explanation of how the extracted target structures guide code modifications and how validation performance is used to select them.
W2. Extracted information may include probe-induced structures. Section 2.2 defines the attack objective as recovering the harness under normal execution, but the extracted information agrees more closely with attacked executions (IF, approximately 83%) than with clean executions (CT, approximately 48%). The average PP estimated from Table 5 is approximately 44%. Please explain which execution steps account for these differences and clarify what is meant in Section 6.2 by the statement that the attack “does not fundamentally alter the underlying inference-time harness.” Agreement across repeated clean executions of the same task would serve as a useful reference for distinguishing the system’s inherent variability from changes introduced by the probes.
W3. Threat model and evaluation protocol. The paper allows the attacker to deploy the agent locally and freely configure its backbone model, while restricting the attacker’s observations to final answers. Please clarify whether model configuration includes custom API endpoints. If the attacker can record requests sent to an endpoint under their control, their observations also include the prompts and any tool or subagent messages sent to that endpoint. For the evaluation, the paper should specify the rules for defining and matching atomic components, who or what decides whether two components match and how the reliability of those judgments is checked, sample sizes at each stage, the number of repetitions, and the statistical meaning of “±,” as well as how failures and timeouts are handled.
W4. Distillation outcomes under the defense. The defense experiments report extraction quality and the target system’s task accuracy, but do not report the performance of the final reconstructed system when the defense is enabled. I suggest running the full AHD pipeline against a defended target and reporting the reconstructed system’s task performance and structural similarity. It would also be useful to examine whether fabricated information is removed during aggregation and denoising, leaving information gaps, or remains and affects subsequent reconstruction. The defense should also be compared, under the same conditions, with standard instructions to keep internal information confidential or refuse to disclose it, and the exact defense instructions used in Table 1 should be specified.
Questions for authors
- With the same coding agent and budget but no target information, how does the optimized harness perform? How does a baseline that directly asks the target to summarize its workflow perform?
- How consistent are execution structures across two clean runs of the same task? Relative to this, how much does the prefix alone or the suffix alone change them?
- Can the attacker specify arbitrary model API endpoints? Could the authors provide a worked example from an execution log to the final structural score?
- After the full AHD pipeline, how does the reconstructed system perform when the target is defended versus undefended?
References
- Qiu et al. AgentDistill: Training-Free Agent Distillation with Generalizable MCP Boxes. 2025. https://arxiv.org/abs/2506.14728
- Wang et al. Black-Box Skill Stealing Attack from Proprietary LLM Agents: An Empirical Study. 2026. https://arxiv.org/abs/2604.21829
- Wang et al. MASLeak: Investigating and Exposing Intellectual Property Leakage Vulnerabilities in Multi-Agent Systems. USENIX Security 2026. https://www.usenix.org/conference/usenixsecurity26/presentation/wang-liwen
- Wu et al. CIA: Inferring the Communication Topology from LLM-based Multi-Agent Systems. ACL 2026. https://aclanthology.org/2026.acl-long.815/
Please engage with the authors responses
Dear Reviewers,
This is a polite reminder to engage with the authors responses to your comments.
Thank you again for your time and effort in reviewing,
Kind regards,
Your ICLR Area Chair