acceptodds
Under review as a conference paper at ICLR 2027

RightSource: Joint Planning of Evidence and Authorization for Tool-Using Agents

Abstract

Tool-using agents must determine not only whether an action satisfies its task requirements, but also whether the appropriate party has authorized it. These decisions become difficult when relevant information is incomplete, evidence comes from different sources, and confirmations apply only to particular actions. A successful factual check does not establish permission, while requesting every possible confirmation can introduce unnecessary interaction. We introduce **RightSource**, a framework for synthesizing verifiable decision policies that distinguish *factual evidence* from *action-scoped authorization*. RightSource represents uncertainty as a set of possible worlds and records confirmations separately according to their source and scope. It plans which observations and confirmations to acquire, allowing subsequent queries to depend on earlier outcomes. Before permitting an action, the framework checks that its safety conditions hold in every remaining possible world and that all required authorizations have been obtained. Within the specified finite model and registered sources, the resulting policies provide checkable guarantees for permitted actions. We evaluate RightSource on retail workflows adapted from τ-bench and prompt-injection scenarios from AgentDojo. The results show fewer retail queries with similar observed task-completion rates, and fewer argument-scope violations in the tested injection scenarios, supporting joint planning of evidence acquisition and authorization.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
AI agents, authorization, verified synthesis, evidence acquisition, agent safety

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_KsHxOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

RightSource formalizes tool-agent decisions over a finite registry of possible worlds. The state is (B,K,U)(B,K,U): the remaining world cell BB, the set KK of action prerequisites confirmed by sources acting within declared authority (Kq(o)⊆Auth(q)K_q(o)\subseteq \mathrm{Auth}(q)), and the unused queries UU. An action may be exposed only if it is safe in every world in BB and all of its prerequisites across BB are in KK (Eq. 2). The paper shows this rule is strictly stronger than ordinary decision-region stopping (Props. 4–5). It solves the finite problem by exact memoized dynamic programming, which is optimal over grounded non-repeating certified policies (Thm. 15), with a loss-aware variant (Cor. 16). The planner is wrapped in a fail-closed execution path with signed map activation and receipt checks, and an optional Pair-SAT refinement with an LLM proposer handles implicit support.

The evaluation has three parts:

Constructed finite studies. Four studies (AuthorityWitness-1, InformationRouting-1, SourceGround-8, and a constructed-loss grid) compare exact planning (EX) against machine-first, best-fixed+skip, and two-step lookahead (L2). Retail ablation. In a shared-gate τ-bench Retail ablation, EX uses 1.93% fewer acquisition interactions than a fixed order, with similar success and 13.6% higher latency. AgentDojo comparison. RightSource reaches 8.2% ASR versus 16.5% for a regex tool filter, and 3.8% argument-scope violations versus 18.2%.

Critical strengths

S1. A clean, correct formal object that separates “what is true” from “who has allowed it.”

The certification predicate CAct(B,K)\mathrm{CAct}(B,K) requires common safety over the remaining worlds plus action-specific, authority-scoped confirmation. It is the paper’s one atomic idea, and it is stated precisely. A minimal counterexample shows that ordinary decision-region stopping can act with unauthorized mass 1 (Prop. 4). There is also a fail-closed soundness result relative to the declared registry (Thm. 11, Prop. 12). The running example, where a passed compatibility test is not owner approval, gives agent-safety work useful vocabulary. The explicit trusted-boundary table (Table 7) is the right way to state a guarantee of this kind. This is why I do not consider the paper technically wrong.

S2. Unusually disciplined, self-falsifying evaluation.

The authors compare against the exhaustive best fixed order rather than a convenient baseline. They decompose the gap (Eq. 9) and report every strict-gap condition (Table 16). In Retail they use a prespecified non-inferiority margin with a task-cluster bootstrap, and they stratify interaction savings by paired outcome (Table 25). They also state exactly what the artifact can and cannot reproduce (Table 31). This matters because it makes the numbers trustworthy. My negative assessment rests on what these numbers show, not on doubts about them.

Critical weaknesses

W1. The distinguishing contribution, exact joint planning of evidence and authorization, is not shown to matter by the paper’s own evidence, and the formal novelty is thin.

Empirically, simple controls match or nearly match EX everywhere:

L2 matches EX in all 72 SourceGround-8 conditions, and EX improves on best-fixed+skip by only 0.46% in the primary setting. Best-fixed+skip equals EX in all 1,512 constructed-loss conditions. A descriptor-first index rule matches EX in all five InformationRouting settings. In the shared-gate Retail ablation, EX saves 0.081 acquisition interactions per trajectory over a fixed order, the comparison with L2 is unresolved, and EX pays +13.6% mean latency. In addition, −6 of the net −28 interactions come from pairs where EX fails and the fixed order succeeds (Table 25). By the paper’s own statement in Sec. 3.3, the AgentDojo comparison does not isolate planning from gating.

Theoretically, the formal additions are small:

Sec. 6 concedes that general belief-state planning subsumes the (B,K,U)(B,K,U) formulation, and that DRD/HEC with confirmation-augmented regions can represent it. Sec. 2 notes that KK is recoverable from (B,U)(B,U). The information state that standard adaptive-information-gathering methods carry, the partial realization of observed query–outcome pairs (Golovin & Krause, 2011), already determines KK. The separation in Props. 4–5 is therefore about which stopping predicate one writes on that state, not about a missing state component. Thm. 15 is exact DP over a finite acyclic problem with an exponential state bound (Thm. 18). No scalable variant carries an approximation guarantee; AA-HEC-F explicitly inherits none.

What remains is a confirmation conjunct in the stopping predicate, plus a negative finding that simple orderings suffice. Yet the abstract concludes that the results support “joint planning of evidence acquisition and authorization.” The body’s evidence supports the certification gate, not the planning. This mismatch sits at the level of the central thesis and is the main reason for my recommendation.

W2. The guarantee and the agent-level evidence concern different objects, and the agent-level setup is under-specified in ways that could account for the paper’s only large effect.

The guarantee is relative to a complete, correctly labeled registry: worlds, safe-action labels, R(w,a)R(w,a), Auth(q)\mathrm{Auth}(q), outcome maps, and K0K_0. The hardest part of real authorization therefore sits inside the trusted base (Sec. 3.1, Table 7). Two key facts are missing:

How these registries were produced for the 115 Retail and 97 AgentDojo tasks: by whom, from what information, and at what cost. Who answers confirmation queries in AgentDojo, which has no interactive user.

If each AgentDojo registry encodes which action categories its user task needs, RightSource is effectively a per-task allowlist. It is then compared against a generic regex deny-list and Prompt Guard. It should instead be compared against defenses that also use task-derived capability or provenance policies on AgentDojo (CaMeL, Debenedetti et al., 2025; Progent, Shi et al., 2025; FIDES, Costa et al., 2025), or against AgentDojo’s own tool-filter defense.

The threat setting is also narrowed:

one experiment-defined injection per user task and a single fixed-template attack; 97 of the 629 benchmark cases; no adaptive attack that stays inside authorized action categories, which is exactly where the 11 residual argument-scope violations occur.

Finally, the agent-level results cannot be rerun from the supplied artifact (Table 31). Because the AgentDojo row is the paper’s only practically large effect, these gaps prevent me from crediting it to the proposed framework.

Other, non-decisive issues.

(a) Presentation.

The introduction never states the contributions in plain language. The main text is largely compressed caveats spread across eight named sub-studies. The agent integration (how agent, registry, compiler, and gate interact at each step) is not specified at a level that permits reimplementation. The Pair-SAT/LLM-proposer scaling study (App. F, six proposer models) is effectively a separate paper and dilutes focus.

(b) Positioning. Joint tool–argument clarification has already been formalized as a POMDP with an expected-value-of-perfect-information objective for question selection (SAGE-Agent, Suri et al., Findings of ACL 2026). Exact cost-optimal decision trees are classically NP-hard (Hyafil & Rivest, 1976), which frames Thm. 18. arxiv

(c) Modeling. Human responses are deterministic per world, cannot be withdrawn, and are noise-free. Real authorization is time-varying.

Questions for authors

[Clarification — most likely to change my score] Registry provenance in the agent evaluations. For Retail and AgentDojo: Who authored the worlds, safe-action labels, R(w,a)R(w,a), Auth(q)\mathrm{Auth}(q), outcome maps, and K0K_0? From what information? In particular, were injection tasks, ground-truth solutions, or the scoring scope visible to the author or generator? How long did authoring take per task, and how many registries failed activation or needed correction? If an LLM generated the registries, what is the failure rate? [Clarification] Confirmations in AgentDojo. Which source answers confirmation queries, and on what basis? Is the user authorization scope used to score MunauthargM^{\mathrm{arg}}_{\mathrm{unauth}} derived from the same specification that populates the registry or K0K_0? If so, part of the 3.8% vs. 18.2% gap measures agreement between the gate and its own specification. [Experiment] Two controls that would let me attribute the AgentDojo effect. (i) A gate-only ablation: identical registry and gate, but a trivial query policy (fixed order or always-ask). (ii) At least one per-task capability/provenance defense (CaMeL or Progent) and AgentDojo’s standard tool filter, run on the full user×injection matrix, plus one adaptive attack that keeps the injected call inside an authorized category. If (i) matches RightSource, the security result belongs to gating, not to joint planning. [Clarification / Experiment] Practical value of exact planning. Is there any non-constructed task family where EX beats both L2 and best-fixed+skip by a practically meaningful margin, say ≥10% of expected acquisition cost? If not, would you restate the thesis as “authority-scoped certification is necessary; simple acquisition orderings are near-optimal”? [Clarification] Scale. What are ∣Ω∣|\Omega|, ∣Q∣|Q|, and ∣R∣|R| for the Retail registries? How does EX planning time scale with them? What happens, and with what guarantee, when exact compilation becomes infeasible? [Clarification] What the framework adds formally. Under the deterministic per-world human model, is there any reachable case where KK is not a function of the observed query–outcome history? If not, please state precisely what the framework adds beyond conditioning the stopping predicate on the partial realization.

Future work, not affecting my assessment: stochastic or withdrawable confirmations; certification of open-ended arguments.

Reviewer_Wxp9OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This is a significantly AI generated paper that has a complex writing style to an apparently very simple problem with an even simpler standard solution of dynamic programming. The AI tools have played their part in making the problem and solution look best paper worthy but careful read (5 th time) enlightened me to a simpler and modest world.

The paper is about deciding whether to apply a patch or seek permission from user in a repository repair task. This has to be performed in a sequence because there may be many patches some may be rejected by the user and hence asking for permission is unnecessarily costly while some others may be consequential edits and require permission and should be asked.

This boils down to a very standard priority scheduling problem with cost optimization. And the likey solution is dynamic programming which the authors have produced and presented.

Critical strengths

The only strength I see is a simple but concrete example that helped me understand the writeup.

Critical weaknesses

The most important weakness is the writing. And it is not the authors fault. It is the fault of the AI tools that the authors used. This makes me wonder where these AI tools are going with their poor and inconsistent text generation style as well as incoherent blabbering type text.

I did not understand why should we write a paper about the problem of priority scheduling when there have existed nearly 100 years of work in operations research that has explored several solutions to this specific problem including dynamic programming based solutions.

I also do not understand a word of the Introduction which uses complex incoherent words for very simple concepts such as state. I mean why are you using the word "world" for a scheduling state? So confusing.

What is new dynamic programming? What were the major challenges you faced that cannot be solved using existing techniques?

Questions for authors

Write the problem definition in your own words.

Write the primary challenges for which solutions do not exist.

Reviewer_aBmTOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper presents RightSource, a framework for the synthesis of verifiable decision policies that distinguish factual evidence from action-related authorization. The background of the framework is that an agent has the necessary rights for a task before it is executed. The framework is evaluated using retail workflows adapted from τ-bench, as well as prompt injection scenarios from AgentDojo. Unfortunately, the paper is insufficiently developed for submission / full review.

Critical strengths

Due to the state of the paper and the weaknesses listed below, I was unable to extract critical strenghts

Critical weaknesses
  • The introduction provides no motivation, contribution, or research gap.

  • Terms are used without being introduced, or are only introduced in entirely different places, such as the appendix, e.g., possible worlds, source contract, receipt, source scopes, present commitments, Dev-fixed…

  • The claims stated in the abstract and elsewhere do not reflect the actual results, e.g., Retail: −1.93% interactions, +0.58 pp success (CI includes 0), +13.6% latency, cost saving ≈ −1.41%.

  • The paper is written in a way that makes the authors' reasoning, conclusions, and decisions incomprehensible/not traceable.

Questions for authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.