- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
RightSource formalizes tool-agent decisions over a finite registry of possible worlds. The state is : the remaining world cell , the set of action prerequisites confirmed by sources acting within declared authority (), and the unused queries . An action may be exposed only if it is safe in every world in and all of its prerequisites across are in (Eq. 2). The paper shows this rule is strictly stronger than ordinary decision-region stopping (Props. 4–5). It solves the finite problem by exact memoized dynamic programming, which is optimal over grounded non-repeating certified policies (Thm. 15), with a loss-aware variant (Cor. 16). The planner is wrapped in a fail-closed execution path with signed map activation and receipt checks, and an optional Pair-SAT refinement with an LLM proposer handles implicit support.
The evaluation has three parts:
Constructed finite studies. Four studies (AuthorityWitness-1, InformationRouting-1, SourceGround-8, and a constructed-loss grid) compare exact planning (EX) against machine-first, best-fixed+skip, and two-step lookahead (L2). Retail ablation. In a shared-gate τ-bench Retail ablation, EX uses 1.93% fewer acquisition interactions than a fixed order, with similar success and 13.6% higher latency. AgentDojo comparison. RightSource reaches 8.2% ASR versus 16.5% for a regex tool filter, and 3.8% argument-scope violations versus 18.2%.
Critical strengths
S1. A clean, correct formal object that separates “what is true” from “who has allowed it.”
The certification predicate requires common safety over the remaining worlds plus action-specific, authority-scoped confirmation. It is the paper’s one atomic idea, and it is stated precisely. A minimal counterexample shows that ordinary decision-region stopping can act with unauthorized mass 1 (Prop. 4). There is also a fail-closed soundness result relative to the declared registry (Thm. 11, Prop. 12). The running example, where a passed compatibility test is not owner approval, gives agent-safety work useful vocabulary. The explicit trusted-boundary table (Table 7) is the right way to state a guarantee of this kind. This is why I do not consider the paper technically wrong.
S2. Unusually disciplined, self-falsifying evaluation.
The authors compare against the exhaustive best fixed order rather than a convenient baseline. They decompose the gap (Eq. 9) and report every strict-gap condition (Table 16). In Retail they use a prespecified non-inferiority margin with a task-cluster bootstrap, and they stratify interaction savings by paired outcome (Table 25). They also state exactly what the artifact can and cannot reproduce (Table 31). This matters because it makes the numbers trustworthy. My negative assessment rests on what these numbers show, not on doubts about them.
Critical weaknesses
W1. The distinguishing contribution, exact joint planning of evidence and authorization, is not shown to matter by the paper’s own evidence, and the formal novelty is thin.
Empirically, simple controls match or nearly match EX everywhere:
L2 matches EX in all 72 SourceGround-8 conditions, and EX improves on best-fixed+skip by only 0.46% in the primary setting. Best-fixed+skip equals EX in all 1,512 constructed-loss conditions. A descriptor-first index rule matches EX in all five InformationRouting settings. In the shared-gate Retail ablation, EX saves 0.081 acquisition interactions per trajectory over a fixed order, the comparison with L2 is unresolved, and EX pays +13.6% mean latency. In addition, −6 of the net −28 interactions come from pairs where EX fails and the fixed order succeeds (Table 25). By the paper’s own statement in Sec. 3.3, the AgentDojo comparison does not isolate planning from gating.
Theoretically, the formal additions are small:
Sec. 6 concedes that general belief-state planning subsumes the formulation, and that DRD/HEC with confirmation-augmented regions can represent it. Sec. 2 notes that is recoverable from . The information state that standard adaptive-information-gathering methods carry, the partial realization of observed query–outcome pairs (Golovin & Krause, 2011), already determines . The separation in Props. 4–5 is therefore about which stopping predicate one writes on that state, not about a missing state component. Thm. 15 is exact DP over a finite acyclic problem with an exponential state bound (Thm. 18). No scalable variant carries an approximation guarantee; AA-HEC-F explicitly inherits none.
What remains is a confirmation conjunct in the stopping predicate, plus a negative finding that simple orderings suffice. Yet the abstract concludes that the results support “joint planning of evidence acquisition and authorization.” The body’s evidence supports the certification gate, not the planning. This mismatch sits at the level of the central thesis and is the main reason for my recommendation.
W2. The guarantee and the agent-level evidence concern different objects, and the agent-level setup is under-specified in ways that could account for the paper’s only large effect.
The guarantee is relative to a complete, correctly labeled registry: worlds, safe-action labels, , , outcome maps, and . The hardest part of real authorization therefore sits inside the trusted base (Sec. 3.1, Table 7). Two key facts are missing:
How these registries were produced for the 115 Retail and 97 AgentDojo tasks: by whom, from what information, and at what cost. Who answers confirmation queries in AgentDojo, which has no interactive user.
If each AgentDojo registry encodes which action categories its user task needs, RightSource is effectively a per-task allowlist. It is then compared against a generic regex deny-list and Prompt Guard. It should instead be compared against defenses that also use task-derived capability or provenance policies on AgentDojo (CaMeL, Debenedetti et al., 2025; Progent, Shi et al., 2025; FIDES, Costa et al., 2025), or against AgentDojo’s own tool-filter defense.
The threat setting is also narrowed:
one experiment-defined injection per user task and a single fixed-template attack; 97 of the 629 benchmark cases; no adaptive attack that stays inside authorized action categories, which is exactly where the 11 residual argument-scope violations occur.
Finally, the agent-level results cannot be rerun from the supplied artifact (Table 31). Because the AgentDojo row is the paper’s only practically large effect, these gaps prevent me from crediting it to the proposed framework.
Other, non-decisive issues.
(a) Presentation.
The introduction never states the contributions in plain language. The main text is largely compressed caveats spread across eight named sub-studies. The agent integration (how agent, registry, compiler, and gate interact at each step) is not specified at a level that permits reimplementation. The Pair-SAT/LLM-proposer scaling study (App. F, six proposer models) is effectively a separate paper and dilutes focus.
(b) Positioning. Joint tool–argument clarification has already been formalized as a POMDP with an expected-value-of-perfect-information objective for question selection (SAGE-Agent, Suri et al., Findings of ACL 2026). Exact cost-optimal decision trees are classically NP-hard (Hyafil & Rivest, 1976), which frames Thm. 18. arxiv
(c) Modeling. Human responses are deterministic per world, cannot be withdrawn, and are noise-free. Real authorization is time-varying.
Questions for authors
[Clarification — most likely to change my score] Registry provenance in the agent evaluations. For Retail and AgentDojo: Who authored the worlds, safe-action labels, , , outcome maps, and ? From what information? In particular, were injection tasks, ground-truth solutions, or the scoring scope visible to the author or generator? How long did authoring take per task, and how many registries failed activation or needed correction? If an LLM generated the registries, what is the failure rate? [Clarification] Confirmations in AgentDojo. Which source answers confirmation queries, and on what basis? Is the user authorization scope used to score derived from the same specification that populates the registry or ? If so, part of the 3.8% vs. 18.2% gap measures agreement between the gate and its own specification. [Experiment] Two controls that would let me attribute the AgentDojo effect. (i) A gate-only ablation: identical registry and gate, but a trivial query policy (fixed order or always-ask). (ii) At least one per-task capability/provenance defense (CaMeL or Progent) and AgentDojo’s standard tool filter, run on the full user×injection matrix, plus one adaptive attack that keeps the injected call inside an authorized category. If (i) matches RightSource, the security result belongs to gating, not to joint planning. [Clarification / Experiment] Practical value of exact planning. Is there any non-constructed task family where EX beats both L2 and best-fixed+skip by a practically meaningful margin, say ≥10% of expected acquisition cost? If not, would you restate the thesis as “authority-scoped certification is necessary; simple acquisition orderings are near-optimal”? [Clarification] Scale. What are , , and for the Retail registries? How does EX planning time scale with them? What happens, and with what guarantee, when exact compilation becomes infeasible? [Clarification] What the framework adds formally. Under the deterministic per-world human model, is there any reachable case where is not a function of the observed query–outcome history? If not, please state precisely what the framework adds beyond conditioning the stopping predicate on the partial realization.
Future work, not affecting my assessment: stochastic or withdrawable confirmations; certification of open-ended arguments.