acceptodds
Under review as a conference paper at ICLR 2027

RGA: Evidence-Grounded Applicability Audits for Reliable Procedural Memory Reuse

Abstract

A procedure retrieved from memory can be relevant to a language agent’s task yet inapplicable under current conditions. Reliable reuse therefore requires distinguishing evidence of inapplicability from insufficient evidence. We introduce the Rule-Grounded Applicability Auditor (RGA) to assess retrieved procedures before execution, given explicit applicability rules and an evidence policy. RGA combines learned evidence routing and three-valued semantic verification with deterministic evidence qualification and action aggregation. Unresolved conditions remain UNKNOWN, allowing the controller to distinguish REUSE, ADAPT, REJECT, and DEFER. Targeted counterfactual supervision trains the auditor to produce rule-level audits near applicability boundaries without directly supervising the final action. We also introduce RGABENCH, a controlled offline benchmark covering state conflicts, public-task transfer, source qualification, and evidencerepresentation shifts. With equal weighting across its four tracks, RGA achieves 84.90% action accuracy and 5.17% Harm, defined as the rate of incorrect executable decisions, compared with 44.97% and 45.68%, respectively, for the strongest evaluated mechanism-level control adapted to the same interface. Under this contract, ablations and compute-matched binary training show that collapsing unresolved evidence into true or false reduces action accuracy or increases incorrect executable decisions. These results support evidence-grounded applicability auditing as an explicit decision layer between procedural-memory retrieval and execution.

Primary area:
neurosymbolic & hybrid AI systems (physics-informed, logic & formal reasoning, etc.)
Keywords:
LLM agents procedural memory applicability auditing evidence grounding agent reliability

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_PFEEOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies applicability auditing for procedural-memory reuse: a retrieved procedure can be topically relevant yet inapplicable under current conditions, and the agent must distinguish evidence of inapplicability from insufficient evidence. The proposed system, RGA (Rule-Grounded Applicability Auditor), separates four responsibilities: (1) a learned router selects candidate observations per rule; (2) a learned verifier predicts a three-valued semantic verdict (TRUE/FALSE/UNKNOWN); (3) a deterministic qualification module checks source authority, binding, temporal validity, and completeness, overriding any unqualified verdict to UNKNOWN; and (4) a deterministic controller aggregates verdicts via Kleene three-valued logic over target compatibility (T), direct applicability (D), and recovery feasibility (R), mapping (T,D,R) to REUSE / ADAPT / REJECT / DEFER. Training uses structured counterfactual supervision on the audit (evidence IDs + verdicts), never the final action.

The paper also introduces RGABENCH, a four-track offline benchmark (State from ToolSandbox, Retail from τ-bench, and controlled Authority/Renderer tracks) with counterfactual groups as the statistical unit. On Qwen3-8B, RGA reaches 84.90% macro Action accuracy and 5.17% Harm vs. 44.97% / 45.68% for the strongest mechanism-level control (a Skill-Pro-style binary gate). Supporting studies: a Direct-4Action SFT comparison (three seeds), frozen-output qualification replay, verdict-space collapse ablations, compute-matched binary-vs-three-valued training, 40-family AppWorld offline transfer, and a 196-episode end-to-end AppWorld execution study (success 19.9% → 29.6%).

Critical strengths

1. The frozen-output qualification replay is a genuinely strong causal-attribution design, and it is reported without overclaiming. Because the learned model outputs are frozen and only the deterministic gate is toggled (Table 4), the paper can show that evidence admissibility alone changes 111/768 Renderer actions (Action 81.77 → 96.22, Harm 14.58 → 0.13) with no change on Authority. This isolates the qualification contribution from the learned audit in a way most agent papers cannot, and the authors explicitly note Authority's gain arises upstream of the gate rather than claiming the gate does everything. The verdict-space studies reinforce this: both the frozen collapse (Table 11) and the compute-matched native binary training (Table 12, Full-3V 100.0 vs. 75.0 on Retail; 94.01 vs. 64.32/74.87 on Renderer) show binary training cannot recover the information the four-action boundary needs. Together these make the paper's core thesis — preserve UNKNOWN; separate admissibility from semantics — well-supported, not merely asserted.

2. Statistical and leakage discipline well above the norm for this area. The paper uses independent groups (families / source-tasks / counterfactual groups) as units, not model calls (Appendix J); applies Holm correction across the 24 Direct-4A comparisons (Table 19); reports a 156-file AppWorld contamination audit with zero canary/instruction/action-label leakage (Table 9); and — importantly — does not hide its negatives. It states plainly that Direct-4A beats RGA on State and Retail (Table 3), that the AppWorld offline transfer shows no Harm reduction and no significant paired result after correction (Tables 5, 10; §K.3), and that the end-to-end repair branch succeeds only 48.9% of the time (Table 6). This candour makes the genuine wins (Authority, Renderer) credible.

Critical weaknesses

1. The headline 84.90 vs. 44.97 comparison is against deliberately handicapped controls, and the method's advantage is concentrated on the two tracks the authors fully control. The "strongest evaluated mechanism-level control" is a Skill-Pro-style binary gate stripped of UNKNOWN and provenance, and the AWM/ExpeL-style baseline just predicts one of four actions directly — none is a faithful reimplementation of the cited systems, which were not built for this interface. More tellingly, when compared against Direct-4A, a baseline given the identical full context (Table 3), RGA loses on State (−4.13) and Retail (−4.17) and wins decisively only on Authority (+50.52) and Renderer (+61.11) — the two tracks that are "controlled held-out," authored by the authors, and whose action distributions are hand-designed to stress exactly the admissibility/representation axes RGA is built for. State (ToolSandbox) and Retail (τ-bench), the two externally-derived tracks, do not favour RGA. So the fair reading is "explicit auditing helps specifically when the benchmark isolates evidence admissibility and representation shifts" — which the authors do say (§5.3) — but the macro-average headline and abstract frame it far more broadly than the Direct-4A comparison supports. The equal-weight macro average gives the two author-controlled tracks half the total weight.

2. The end-to-end result — the only test of whether any of this helps a real agent — is thin and partly structural. The 196-episode AppWorld study (Table 6) is the paper's single piece of evidence that applicability auditing improves task completion rather than audit-label accuracy. The improvement (19.9% → 29.6%, +9.7 points, 19 episodes over 49 independent families) is reported with family-cluster bootstrap machinery (§O.7) but no p-value or CI is actually given for the headline success difference — only branch-conditional diagnostics. With 49 independent units and a 9.7-point difference, significance is plausible but unestablished as written. Worse, part of the construction builds in the result: the "different-family" condition (49/196 episodes) is designed so the correct action is REJECT, and REJECT in the RGA pipeline falls back to the same planning path as the unaudited agent (§O.6) — so on a quarter of episodes the two pipelines are, by design, doing nearly the same thing, and the "initially unresolved" condition is hand-split 25/24 into ADAPT/REJECT outcomes. The gain therefore rests largely on the same-instance (positive-control) and sibling conditions. This is suggestive but not yet a convincing demonstration of downstream benefit, and the 48.9% repair-branch success rate (which the authors flag) shows the bottleneck has merely moved downstream.

Secondary:

  • RGABENCH's realism and the ledger/policy assumption. RGA assumes a supplied rule ledger and evidence policy at inference (§3.1), and the ledgers are template-instantiated. The hard problem in practice — where do correct per-procedure applicability rules and a correct evidence policy come from? — is bracketed out. The paper is explicit about this, but it narrows the contribution: RGA is an auditor given a good contract, not a system that discovers the contract. Any comparison to end-to-end reuse systems that lack this scaffolding is correspondingly uneven.
  • Single backbone. All results use Qwen3-8B. Given that the method's value is a structured decision layer over a frozen LLM's verdicts, whether the gains survive on a weaker or stronger verifier (where the learned-verification bottleneck in Appendix G.2 — 61.46 Action at 97.92 evidence exactness — would shift) is untested and matters for the generality claim.
Questions for authors
  1. For the headline 19.9% → 29.6% (Table 6), please report the family-level paired sign test and the family-cluster bootstrap CI you define in §O.7. Also report the per-condition success breakdown (same / sibling / different / unresolved) for both pipelines, so readers can see how much of the +9.7 comes from conditions where REJECT collapses the two pipelines together.

  2. Against Direct-4A (the matched-context baseline), RGA wins only on the two author-controlled tracks. Can you add an externally-sourced track that stresses admissibility/representation (not authored by the team), or argue why State/Retail are a fair test of the admissibility axis at all given RGA does not beat Direct-4A there?

  3. How robust is RGA to an imperfect ledger or evidence policy (missing a rule, an over-permissive source list)? A degradation curve as ledger/policy quality drops would show whether the method's value survives realistic contract noise, which is the main deployment concern.

  4. Were the AWM/ExpeL/Skill-Pro controls given any opportunity to express abstention or provenance within this interface, or are they structurally incapable by construction? If the latter, the macro-average comparison in the abstract should be reframed, since it compares against interfaces that cannot represent the paper's key quantity.

  5. The factorized diagnostic (Appendix G.2) localises the remaining errors to semantic verification (FALSE→TRUE, UNKNOWN→TRUE). How do Authority/Renderer results change with a second backbone, and is the qualification gate's benefit larger or smaller when the verifier is weaker?

3 comments
AuthorsOpenReview ↗

Follow-up on Our Rebuttal and Invitation for Further Discussion

Dear Reviewer,

We sincerely appreciate the time and effort you have devoted to reviewing our work.

Following your valuable feedback, we have provided responses and clarifications to the concerns raised in your review. We would be grateful to hear whether these explanations have helped resolve your concerns.

We remain very open to further discussion and would be happy to elaborate on any points that may still require clarification.

Thank you again for your thoughtful consideration.

Best regards,

The Authors

Thank you for the clarifications and the additional Mistral-7B results. Seeing consistent gains on the Renderer track across two model families strengthens my confidence that the qualification mechanism improves action accuracy. After considering the rebuttal, I am maintaining my original rating of 3.

AuthorsOpenReview ↗

Thank You for Your Feedback and Consideration

We sincerely thank the reviewer for taking the time to consider our rebuttal and additional experiments. We appreciate your positive feedback on the cross-backbone results and your recognition of the qualification mechanism's effectiveness. Thank you again for your thoughtful evaluation and constructive comments, which have helped us strengthen our work.

Reviewer_NnT5OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

RGA checks whether a stored procedure can be reused in a new situation. It judges individual rules as TRUE, FALSE, or UNKNOWN, checks whether the supporting evidence is admissible, and selects REUSE, ADAPT, REJECT, or DEFER. The paper introduces RGABench and evaluates the approach on controlled tasks and AppWorld.

Critical strengths
  1. The method separates interpreting evidence from deciding whether that evidence is acceptable. This makes the reuse decision easier to inspect and avoids treating missing evidence as permission to act.
  2. The ablations are useful. Holding model predictions fixed, qualification improves Renderer accuracy from 81.77% to 96.22%. The matched binary/three-valued comparison also helps explain the value of retaining UNKNOWN.
  3. RGABench tests several distinct failure cases, including stale states, unreliable sources, and changes in evidence format. The analysis accounts for related examples by testing at the group level, which makes the comparisons more convincing.
Critical weaknesses
  1. Direct-4A and RGA use different training budgets and curricula. The comparison therefore does not establish how much of the gain comes from structured auditing rather than additional training.
  2. The practical evaluation is limited. Offline AppWorld does not show a reduction in Harm or significant corrected comparisons. The execution study uses official solutions as memories, including some target-task solutions shared by both pipelines. A breakdown by memory condition is needed to understand where the gains come from.
Questions for authors
  1. How does Direct-4A perform with comparable training data, curriculum, and budget?
  2. Does the execution improvement remain after excluding target-task solution memories? Please report results by memory condition and confidence intervals.
  3. Did both execution pipelines have the same repair opportunities and tool/model-call budgets?
1 comment
AuthorsOpenReview ↗

Follow-up on Our Rebuttal and Invitation for Further Discussion

Dear Reviewer,

Thank you again for your valuable feedback and constructive suggestions.

We have provided detailed responses to the questions and concerns raised in your review. We would appreciate any feedback on whether our clarifications have satisfactorily addressed your concerns.

We welcome further discussion and would be glad to address any remaining questions or provide additional explanations if needed.

Thank you for your time and consideration.

Best regards,

The Authors

Reviewer_2rcsOpenReview ↗
Rating
2(+1)
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This work introduces RGA framework that decides if a procedure retrieved from agent's memory is relevant and safe to use or not. the paper attempts to motivate that procedure might be relevant but unapplicaple on current task. This makes agent "unsafe". RGA attempts to evaluate if it should be reused, adopted or rejected. This is achieved by combining learned evidence routing and three-valued semantic verification with deterministic evidence qualification and action aggregation. Paper also introduces a benchmarks to measure this more closely called as RGABench.

Critical strengths
  1. The proposed pipeline decomposes the problem carefully, the design choices have been well motivated, which makes pipeline intutive and relevant. The mechanism also provides a clear way to enforce evidence policies independently of the language model's predictions.
  2. The evaluations are strong and the technique outperforms most of the baselines used in the paper. Experiments on Appworld also improve understanding the usecase bit better.
Critical weaknesses
  1. The importance of the problem is not well motivated, the paper argues that existing procedural-memory systems may retrieve relevant procedures that are not applicable to the current state. While this is plausible, the paper does not convincingly establish how prevalent or consequential this failure mode is in existing agents. I would be curious to understand how frequently this happens in real world agentic setups in more popular agentic benchmarks? curently the evaluations and the motivation is bottlenecked with RGABench, which makes the claims weaker.
  2. RGABench explicitly evaluates source qualification, entity binding, temporal validity, and representation shifts, all of which align closely with the proposed architecture. Such controlled tests are useful for isolating failure modes, but the resulting gains do not necessarily establish that the approach generalizes to realistic procedural-memory reuse. In particular, the paper should more clearly distinguish performance on a benchmark constructed to test applicability auditing from evidence that existing agents suffer substantially from these failures in the wild.
Questions for authors
  1. Can the authors provide evidence that applicability failures are a significant source of errors in existing procedural-memory agents, ideally through an analysis of naturally occurring failures rather than only controlled counterfactual examples?
  2. How does RGA compare with a strong prompted LLM baseline that receives the same procedure, rules, evidence, and policy and is instructed to check applicability before deciding whether to execute?
  3. To what extent do the gains on RGABench transfer to tasks and failure modes that were not explicitly represented in the benchmark construction?
  4. What are the additional inference costs of RGA, and how do these costs compare with the benefits in end-to-end task completion?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.