- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper studies applicability auditing for procedural-memory reuse: a retrieved procedure can be topically relevant yet inapplicable under current conditions, and the agent must distinguish evidence of inapplicability from insufficient evidence. The proposed system, RGA (Rule-Grounded Applicability Auditor), separates four responsibilities: (1) a learned router selects candidate observations per rule; (2) a learned verifier predicts a three-valued semantic verdict (TRUE/FALSE/UNKNOWN); (3) a deterministic qualification module checks source authority, binding, temporal validity, and completeness, overriding any unqualified verdict to UNKNOWN; and (4) a deterministic controller aggregates verdicts via Kleene three-valued logic over target compatibility (T), direct applicability (D), and recovery feasibility (R), mapping (T,D,R) to REUSE / ADAPT / REJECT / DEFER. Training uses structured counterfactual supervision on the audit (evidence IDs + verdicts), never the final action.
The paper also introduces RGABENCH, a four-track offline benchmark (State from ToolSandbox, Retail from τ-bench, and controlled Authority/Renderer tracks) with counterfactual groups as the statistical unit. On Qwen3-8B, RGA reaches 84.90% macro Action accuracy and 5.17% Harm vs. 44.97% / 45.68% for the strongest mechanism-level control (a Skill-Pro-style binary gate). Supporting studies: a Direct-4Action SFT comparison (three seeds), frozen-output qualification replay, verdict-space collapse ablations, compute-matched binary-vs-three-valued training, 40-family AppWorld offline transfer, and a 196-episode end-to-end AppWorld execution study (success 19.9% → 29.6%).
Critical strengths
1. The frozen-output qualification replay is a genuinely strong causal-attribution design, and it is reported without overclaiming. Because the learned model outputs are frozen and only the deterministic gate is toggled (Table 4), the paper can show that evidence admissibility alone changes 111/768 Renderer actions (Action 81.77 → 96.22, Harm 14.58 → 0.13) with no change on Authority. This isolates the qualification contribution from the learned audit in a way most agent papers cannot, and the authors explicitly note Authority's gain arises upstream of the gate rather than claiming the gate does everything. The verdict-space studies reinforce this: both the frozen collapse (Table 11) and the compute-matched native binary training (Table 12, Full-3V 100.0 vs. 75.0 on Retail; 94.01 vs. 64.32/74.87 on Renderer) show binary training cannot recover the information the four-action boundary needs. Together these make the paper's core thesis — preserve UNKNOWN; separate admissibility from semantics — well-supported, not merely asserted.
2. Statistical and leakage discipline well above the norm for this area. The paper uses independent groups (families / source-tasks / counterfactual groups) as units, not model calls (Appendix J); applies Holm correction across the 24 Direct-4A comparisons (Table 19); reports a 156-file AppWorld contamination audit with zero canary/instruction/action-label leakage (Table 9); and — importantly — does not hide its negatives. It states plainly that Direct-4A beats RGA on State and Retail (Table 3), that the AppWorld offline transfer shows no Harm reduction and no significant paired result after correction (Tables 5, 10; §K.3), and that the end-to-end repair branch succeeds only 48.9% of the time (Table 6). This candour makes the genuine wins (Authority, Renderer) credible.
Critical weaknesses
1. The headline 84.90 vs. 44.97 comparison is against deliberately handicapped controls, and the method's advantage is concentrated on the two tracks the authors fully control. The "strongest evaluated mechanism-level control" is a Skill-Pro-style binary gate stripped of UNKNOWN and provenance, and the AWM/ExpeL-style baseline just predicts one of four actions directly — none is a faithful reimplementation of the cited systems, which were not built for this interface. More tellingly, when compared against Direct-4A, a baseline given the identical full context (Table 3), RGA loses on State (−4.13) and Retail (−4.17) and wins decisively only on Authority (+50.52) and Renderer (+61.11) — the two tracks that are "controlled held-out," authored by the authors, and whose action distributions are hand-designed to stress exactly the admissibility/representation axes RGA is built for. State (ToolSandbox) and Retail (τ-bench), the two externally-derived tracks, do not favour RGA. So the fair reading is "explicit auditing helps specifically when the benchmark isolates evidence admissibility and representation shifts" — which the authors do say (§5.3) — but the macro-average headline and abstract frame it far more broadly than the Direct-4A comparison supports. The equal-weight macro average gives the two author-controlled tracks half the total weight.
2. The end-to-end result — the only test of whether any of this helps a real agent — is thin and partly structural. The 196-episode AppWorld study (Table 6) is the paper's single piece of evidence that applicability auditing improves task completion rather than audit-label accuracy. The improvement (19.9% → 29.6%, +9.7 points, 19 episodes over 49 independent families) is reported with family-cluster bootstrap machinery (§O.7) but no p-value or CI is actually given for the headline success difference — only branch-conditional diagnostics. With 49 independent units and a 9.7-point difference, significance is plausible but unestablished as written. Worse, part of the construction builds in the result: the "different-family" condition (49/196 episodes) is designed so the correct action is REJECT, and REJECT in the RGA pipeline falls back to the same planning path as the unaudited agent (§O.6) — so on a quarter of episodes the two pipelines are, by design, doing nearly the same thing, and the "initially unresolved" condition is hand-split 25/24 into ADAPT/REJECT outcomes. The gain therefore rests largely on the same-instance (positive-control) and sibling conditions. This is suggestive but not yet a convincing demonstration of downstream benefit, and the 48.9% repair-branch success rate (which the authors flag) shows the bottleneck has merely moved downstream.
Secondary:
- RGABENCH's realism and the ledger/policy assumption. RGA assumes a supplied rule ledger and evidence policy at inference (§3.1), and the ledgers are template-instantiated. The hard problem in practice — where do correct per-procedure applicability rules and a correct evidence policy come from? — is bracketed out. The paper is explicit about this, but it narrows the contribution: RGA is an auditor given a good contract, not a system that discovers the contract. Any comparison to end-to-end reuse systems that lack this scaffolding is correspondingly uneven.
- Single backbone. All results use Qwen3-8B. Given that the method's value is a structured decision layer over a frozen LLM's verdicts, whether the gains survive on a weaker or stronger verifier (where the learned-verification bottleneck in Appendix G.2 — 61.46 Action at 97.92 evidence exactness — would shift) is untested and matters for the generality claim.
Questions for authors
-
For the headline 19.9% → 29.6% (Table 6), please report the family-level paired sign test and the family-cluster bootstrap CI you define in §O.7. Also report the per-condition success breakdown (same / sibling / different / unresolved) for both pipelines, so readers can see how much of the +9.7 comes from conditions where REJECT collapses the two pipelines together.
-
Against Direct-4A (the matched-context baseline), RGA wins only on the two author-controlled tracks. Can you add an externally-sourced track that stresses admissibility/representation (not authored by the team), or argue why State/Retail are a fair test of the admissibility axis at all given RGA does not beat Direct-4A there?
-
How robust is RGA to an imperfect ledger or evidence policy (missing a rule, an over-permissive source list)? A degradation curve as ledger/policy quality drops would show whether the method's value survives realistic contract noise, which is the main deployment concern.
-
Were the AWM/ExpeL/Skill-Pro controls given any opportunity to express abstention or provenance within this interface, or are they structurally incapable by construction? If the latter, the macro-average comparison in the abstract should be reframed, since it compares against interfaces that cannot represent the paper's key quantity.
-
The factorized diagnostic (Appendix G.2) localises the remaining errors to semantic verification (FALSE→TRUE, UNKNOWN→TRUE). How do Authority/Renderer results change with a second backbone, and is the qualification gate's benefit larger or smaller when the verifier is weaker?
3 comments
Thank you for the clarifications and the additional Mistral-7B results. Seeing consistent gains on the Renderer track across two model families strengthens my confidence that the qualification mechanism improves action accuracy. After considering the rebuttal, I am maintaining my original rating of 3.
Thank You for Your Feedback and Consideration
We sincerely thank the reviewer for taking the time to consider our rebuttal and additional experiments. We appreciate your positive feedback on the cross-backbone results and your recognition of the qualification mechanism's effectiveness. Thank you again for your thoughtful evaluation and constructive comments, which have helped us strengthen our work.
Follow-up on Our Rebuttal and Invitation for Further Discussion
Dear Reviewer,
We sincerely appreciate the time and effort you have devoted to reviewing our work.
Following your valuable feedback, we have provided responses and clarifications to the concerns raised in your review. We would be grateful to hear whether these explanations have helped resolve your concerns.
We remain very open to further discussion and would be happy to elaborate on any points that may still require clarification.
Thank you again for your thoughtful consideration.
Best regards,
The Authors