- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2(+1)
Read review
Paper summary
The paper studies how agents solve tasks that require accessing information held by other agents, and how the allocation of such knowledge affects the final outcome. The authors adapt existing tasks (AgentDojo and τ -bench) to two different settings, namely centralized (C) and socially distributed (SD) environments. In the former, a single agent holds the service-side knowledge, while in the latter, it is distributed among multiple agents. The authors also introduce Incognita, a framework that splits a task into sub-components that fit the C and SD environments.
Critical strengths
- The paper addresses an important and interesting problem, i.e., that of distributed problem solving (actually, a slightly simplified version of it where agents only miss pieces of information they have to retrieve from other agents)
- The abstract and intro (up to the very end) are quite well written and easy to follow
- Some results are genuinely interesting (Section 5 on safety) and tell us that malicious pieces of information, once passed to a model, can trigger even if their frequency is low
Critical weaknesses
- The last part of the introduction (lines 64-77) sounds off from the previous part, as it introduces the results in quite a messy and Claude-ish way. I would reserve the numerical results for the appropriate section and leave this part for a summary and take-home points of the paper.
- The mathematical formulation looks more like an ornament rather than being useful for developing the framework. Cosider Equation 2: it simply enforces that the agent solving the task does not possess the information required to solve it. In this sense, it reads more like a necessary condition for the C/SD environments to exist, but it does not tell anything about the hardness of the task. Also, Eq. 3 is kind of tautological (just rewrite it in terms of the numerator and denominator, and each cancels out and produces the final result)
- Results in Section 4 are quite unsurprising: on the one hand, frontier models can beat the C environment while they struggle in the SD environment. While on its own this is not bad, the paper lacks a convincing explanation of why that is the case.
- Almost all of the successful attacks in Section 5 come from a single model, with no clear trend that tells us whether the C/SD environments are the cause or at least correlate with the numbers.
Edit: After rebuttal, I have decided to raise the Soundness score.
Questions for authors
- Can the authors provide the source code?
- Do you have any explanation for why GPT 5.4 performs much better than other models?