- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper studies cross-principal information sharing among LLM agent delegates. It introduces SharedOS, an execution layer that enforces owner-issued grants, and SharedEval, which labels what each requester is entitled to receive. The framework is used to evaluate responder delegates in two scenarios: a bilateral setting (PAIR) and a multi-party network (NET).
Critical strengths
-
The topic is important. As agents increasingly act on behalf of different owners, how they handle information sharing across principals is a practical and growing concern.
-
In particular, the multi-party setting of NET is relatively well defined. Its design could support future multi-agent evaluation in information-sharing contexts.
Critical weaknesses
-
The motivation is hard to follow. To evaluate multi-agent systems, one could build a single shared sandbox environment and drive each agent through its own harness SDK. Existing sandboxes already support this. Thus, it is initially unclear why SharedOS is needed. The need is clearer if each agent brings its own sandbox and permission system, in which case a common governing layer such as SharedOS is necessary. The paper would benefit from clarifying this.
-
There are several internal inconsistencies that the authors should clarify (see Questions).
-
Many design decisions are not justified, which makes the work hard to understand and evaluate. For example, both PAIR and NET are constructed by the authors themselves, but the paper does not explain the basis for their design or why the tasks were designed this way. It is also unclear why requester reframing is treated as an experimental variable in PAIR. Moreover, the requester-side variation in PAIR is limited, and it does not necessarily support the general conclusion that "the boundary is set on the responder's side."
Questions for authors
-
PAIR and NET seems to be designed by the authors themselves. How were the settings of these two scenarios designed, and how was their quality ensured? For example, how can the authors ensure that these scenarios are representative for testing cross-principal delegation, specifically information sharing?
-
In PAIR, some gold labels in relationship-v2 conflict with the agent-facing D2 policy in appendix d.9.3. For example, 52 of 90 sampled D2 leaks involve items that the category contract treats as public, and 33 of the executive delegate's authorized work actions are withheld by the labels but granted by the D2 text. Why do they conflict? Since the policy and the judgment criteria disagree, do these conflicts undermine the validity of the results? How much do they affect the reported numbers and the findings?
-
How was NET executed? Sec 4.3 reports runs with 60 agents, while appendix c mentions that NET is not a production execution target. and that the 60-principal v2 world has "no integrated runner." While the abstract presents NET findings as a main result.
-
The NET ladder result rests on 15 of 136 forbidden edges (the intersection that every policy rung delivered), and the D1–D2 difference in Lw corresponds to a single edge. Why did 4 of the 19 primary edges fail to deliver under some rung, and is this attrition independent of the policy? Can the authors report results on all 19 (or all 49 scorable) edges, treating failures explicitly?