acceptodds
Under review as a conference paper at ICLR 2027

Acting Through Others: Evaluating Agent Competence and Safety in Socially Distributed Environments

Abstract

What can an agent accomplish when the knowledge it needs and the authority to act belong to others? In such settings, progress requires discovering whom to ask, what others can provide, and how to get actions carried out. We introduce Incognita, a framework that transforms existing benchmark tasks into matched environments with centralized or socially distributed knowledge and action authority, while retaining task objectives and success criteria. In distributed conditions, agents interact with role-isolated participants whose capabilities are initially undisclosed. Using tasks adapted from AgentDojo and τ-bench, we investigate the dual demands of acting through others: acquiring information needed for progress while resisting information that would redirect that progress. In exploratory security tests, the effects of distributed access varied across models and attack families. Lower exposure to malicious information did not necessarily translate into fewer successful attacks. Task completion was also lower under distributed access in our competence study. Most failed distributed runs in that study stalled before obtaining the information needed to act. These complementary findings motivate a relational view of agent evaluation. Incognita makes the dependencies underlying action experimentally tractable, allowing us to examine how competence and vulnerability take shape through an agent’s relations with others.

Primary area:
datasets and benchmarks
Keywords:
LLM agents, agent evaluation, socially distributed environments, prompt injection, AI safety

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_YQ2DOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2(+1)
Read review
Paper summary

The paper studies how agents solve tasks that require accessing information held by other agents, and how the allocation of such knowledge affects the final outcome. The authors adapt existing tasks (AgentDojo and τ -bench) to two different settings, namely centralized (C) and socially distributed (SD) environments. In the former, a single agent holds the service-side knowledge, while in the latter, it is distributed among multiple agents. The authors also introduce Incognita, a framework that splits a task into sub-components that fit the C and SD environments.

Critical strengths
  1. The paper addresses an important and interesting problem, i.e., that of distributed problem solving (actually, a slightly simplified version of it where agents only miss pieces of information they have to retrieve from other agents)
  2. The abstract and intro (up to the very end) are quite well written and easy to follow
  3. Some results are genuinely interesting (Section 5 on safety) and tell us that malicious pieces of information, once passed to a model, can trigger even if their frequency is low
Critical weaknesses
  1. The last part of the introduction (lines 64-77) sounds off from the previous part, as it introduces the results in quite a messy and Claude-ish way. I would reserve the numerical results for the appropriate section and leave this part for a summary and take-home points of the paper.
  2. The mathematical formulation looks more like an ornament rather than being useful for developing the framework. Cosider Equation 2: it simply enforces that the agent solving the task does not possess the information required to solve it. In this sense, it reads more like a necessary condition for the C/SD environments to exist, but it does not tell anything about the hardness of the task. Also, Eq. 3 is kind of tautological (just rewrite it in terms of the numerator and denominator, and each cancels out and produces the final result)
  3. Results in Section 4 are quite unsurprising: on the one hand, frontier models can beat the C environment while they struggle in the SD environment. While on its own this is not bad, the paper lacks a convincing explanation of why that is the case.
  4. Almost all of the successful attacks in Section 5 come from a single model, with no clear trend that tells us whether the C/SD environments are the cause or at least correlate with the numbers.

Edit: After rebuttal, I have decided to raise the Soundness score.

Questions for authors
  1. Can the authors provide the source code?
  2. Do you have any explanation for why GPT 5.4 performs much better than other models?
Reviewer_Kq7rOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

Incognita reallocates information and action authority in existing benchmarks while preserving task goals and scoring. Centralized and distributed agents both act through role-holders. Retail completion falls from 160/216 to 98/216 trials; security experiments separately track exposure, transmission, and committed attacks.

Critical strengths

S1: Preserving source scorers and using a matched messaging interface makes access distribution a tractable evaluation variable. S2: The Retail gap is supported by task-clustered uncertainty and persists after excluding corrected task cards. Failure-location diagnostics add useful detail beyond final success rates.

Critical weaknesses

W1: Security coverage rests on nine Banking user tasks and three Slack workflows. Repeated model/payload trials increase episode counts without providing equally broad independent task coverage (Table 1; Appendix H). W2: The Banking Type 2 commitment reversal is fragile across coordinators: GPT-5.4 accounts for 12 of 14 SD events, and the task-clustered interval touches zero (Section 5; Appendix H). It supports a specific observed vulnerability, with limited evidence for a general safety ranking. W3: The directory intervention provides limited evidence about mechanisms: GPT controls predate treatment by about a month, while the concurrent Sonnet completion-gain interval includes zero. These acknowledged limits constrain interpretation beyond the observed reduction in early failures (Section 4.2; Appendix C).

Questions for authors

Q1: How are Banking Type 2 commitments distributed across the nine source tasks, and is the contrast dominated by particular tasks? Q2: How does the task-clustered Type 2 commitment interval change when GPT-5.4 is excluded? Q3: In matched task/repetition cells, how often does the directory convert an early failure into completion versus a later failure?

Reviewer_o327OpenReview ↗
Rating
4
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper proposes Incognita, a framework for adapting τ\tau-bench and AgentDojo tasks to distributed environments where the coordinating agent must explore to determine who has what information or permissions. Using a sub-set of adapted tasks from the aforementioned benchmark datasets, the authors contrast the performance of various LLM agents between the centralized and distributed domains. They find that although performance appears to degrade in the distributed domain, this is not consistent across models and tasks. Furthermore, the authors also evaluate the difference in vulnerability to attacks. This is accomplished by evaluating different attack mechanisms on translated tasks from AgentDojo and comparing their effectiveness across centralized and distributed environments.

Critical strengths

1. Contributes a useful and interesting dataset for evaluating LLMs in distributed environments. The authors adapt a variety of tasks to the socially distributed environment, allowing them to evaluate disparate domains (retail, airline, banking, slack), goals, and security paradigms (under attack vs. not under attack). Therefore the developed dataset is rich and could prove valuable for the development of agents that can operate safely and effectively in distributed environments.

2. Experimental results are easy to appreciate and interesting. The authors conduct a series of experiments providing interesting results on how task completion success can deteriorate when the environment is distributed rather than cetntralized. I believe these experiments are interesting and useful for practioners looking to study LLMs in distributed environments, and that their conclusions might inspire future research in this area. The results are presented cleanly, and are easy to understand. I also find the addition of results on the travel of malicious information to be interesting.

Critical weaknesses

1. Limited task translation. The authors translate 18 τ\tau-retail tasks and 5 τ\tau-airline tasks while the source dataset has far more tasks. The same is true for AgentDojo. Because one of the primary contributions of Incognita is the transformation of centralized to distributed tasks, it would be helpful if the outcome dataset was larger.

Questions for authors
  1. You qualify in L191 and elsewhere that "D requires at least one such dependency to cross holders in SD." How much did the gap between C/SD on a particular task depend on how many dependencies crossed holders? Is there some way to measure the gap between C/SD as a function of how distributed the information becomes?

  2. Is there a known optimal route for task completion? If so, how does the length of this route differ between the distributed and centralized environments?

  3. Would it be possible to include a discussion of future research directions? One that I think could be particularly interesting is to compare agent completion rates when the exploration strategy is set by an established exploration algorithm.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.