acceptodds
Under review as a conference paper at ICLR 2027

OrgMem: Benchmarking Self-Evolving Agents for Organization-Specific Evolving Audit Environments

Abstract

Large language model (LLM) agents increasingly rely on external memory to reuse experience across long-horizon interactions, yet existing evaluations provide limited insight into whether historical experience remains useful when the underlying decision environment evolves. This challenge is particularly important in organizational auditing, where procurement policies, market conditions, vendor behavior, and fraud-risk patterns may change over time, causing superficially similar cases to require different judgments. We introduce AuditEvo, a synthetic benchmark for evaluating self-evolving agents under organization-specific and implicitly evolving procurement environments. AuditEvo contains 40,000 evaluated cases organized into 100 chronological organizational chains with evolving policy, market, vendor, and risk conditions. At each step, an agent observes the current case and chronologically available history, predicts an audit outcome of approval, rejection, or investigation, and immediately receives verified feedback that may be reused in subsequent decisions. A deterministic audit oracle provides reproducible labels while keeping latent environment states and oracle rationales hidden from evaluated agents. Experiments across diverse memory and reflection frameworks and multiple LLM backbones show that current systems struggle particularly with risk-regime cases and that retrieving historically related cases alone does not ensure reliable decisions when their applicability changes. Motivated by this limitation, we introduce Environment-Aware JitRL, which maintains an evolving playbook that uses feedback to adapt state similarity and thereby condition historical experience retrieval for action-value estimation. Across evaluated backbones, the method improves average audit accuracy over JitRL by approximately 2–7 percentage points. AuditEvo provides a controlled testbed for studying memory reuse, adaptation, and negative transfer under persistent multi-environment evolution.

Primary area:
datasets and benchmarks
Keywords:
Agent Systems, Datasets, Evaluation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_QQf6OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces AuditEvo, a procurement audit benchmark designed for dynamic organizational environments, aimed at investigating memory reuse and continuous adaptation. The benchmark comprises 100 chronologically evolving organizational business chains, totaling 40,000 scored cases, with each chain supplemented by 31 calibration cases. As procurement policies, market conditions, supplier behaviors, and risk patterns shift over time, the applicability of historical experience fluctuates accordingly. Given completed transactions, active policies, and past case records, the system under evaluation must generate an audit judgment: approve, reject, or flag for investigation. Following its prediction, the system receives the ground-truth label, which it then uses to update its internal memory. The authors employ a deterministic audit oracle to generate labels, utilizing temporal constraints and the separation of public information from hidden states to prevent information leakage. Experiments are conducted across six large language models, comparing various approaches to memory retrieval, reflection, and online learning. Furthermore, the authors propose "Environment-Aware JitRL," which leverages feedback to update business manuals, adjust case representations, and refine the similarity metrics used for retrieving historical experience—ultimately using this retrieved experience to estimate action values. Experimental results demonstrate that this method improves audit accuracy by 2 to 7 percentage points compared to the original JitRL. Comparisons with a static-environment control group confirm that environmental evolution increases task difficulty, highlighting the distinct advantages of the proposed method in dynamic, ever-changing scenarios.

Critical strengths
  1. This work establishes a distinctive evaluation scenario for dynamic organizational settings. AuditEvo organizes procurement policies, market conditions, supplier behaviors, and evolving risks into a continuous stream of cases, raising the critical question of whether historical experience remains directly applicable. The strict temporal constraints imposed by the "deterministic oracle"—along with the immediate provision of labels following each prediction—create an excellent controlled experimental environment for studying continuous adaptation in dynamic settings. See Section 3 and Appendix A.3 for details.
  2. The paper explores continuous adaptation through the lens of memory reuse—a highly significant research topic. The study focuses on whether past experiences remain valid in new contexts, addressing key aspects such as memory retrieval, the updating of experiences, and the handling of obsolete information. By conducting experiments across multiple models and frameworks—complemented by controlled experiments in static environments—the authors derive several valuable insights. Furthermore, the performance gains achieved via "Environment-Aware JitRL" provide preliminary evidence that dynamically adjusting how past experiences are utilized is an effective strategy. See Tables 2 and 5 for details.
Critical weaknesses
  1. Insufficient diagnosis of adaptability. Metrics such as accuracy and Macro-F1 are influenced by a combination of factors: policy comprehension, historical retrieval, cross-case reasoning, and the assessment of experience applicability. Evaluations in static environments merely demonstrate that environmental shifts increase task difficulty; they fail to pinpoint whether errors stem from a failure to retrieve relevant evidence, flawed reasoning, or the inappropriate application of outdated experiences. The "Case-overlap F1" metric, which tracks only the overlap of case IDs, cannot disentangle these distinct sources of error. (See Sections 4.2–4.3, and Appendices A.2 and A.6.)
  2. Lack of clarity regarding model roles and total computational cost. While the paper specifies the backbone models under evaluation, it does not detail the models used for decision-making, the LLM-based evaluator, or the organization of the playbook. Furthermore, it is unclear whether the evaluator is updated in tandem when the backbone model is changed. Although Table 2 provides latency figures for retrieval and memory writes, it is uncertain whether these include the costs of updating the "reflection playbook" or reconstructing historical representations. Consequently, it is difficult to assess the method's benefits relative to its overall computational efficiency. (See Section 5, and Appendices A.7 and A.8.)
  3. Absence of comparisons with related methods and mechanism-specific ablation studies. While the set of baselines is reasonably broad, it omits works focused on the evolution of experiences—such as ReMem and EvoMem—without providing a justification for their exclusion. Since the "playbook" influences both retrieval representations and the decision-making context, the study lacks a control condition that keeps the retrieval similarity constant (e.g., by fixing the playbook prompts). Without this, it is impossible to isolate the specific contribution of the dynamic retrieval mechanism. (See Table 2, and Equations 4 and 6.)
Questions for authors
  1. Is it possible to break down errors based on the type of environmental change or the specific decision node? Specifically, can you distinguish between failures in retrieving historical evidence, errors in the reasoning/audit process, and misjudgments regarding the applicability of past experiences to the current situation?

  2. What specific models are used for the decision model evaluator and the content organization module? If the backbone model is replaced, will these modules also be updated? Furthermore, do the latency figures provided in the paper account for the full reflection and memory-update process?

  3. Could you include a comparison with ReMem and EvoMem, or explain why such a comparison is not feasible? Additionally, how would performance be affected if the decision prompts within the playbooks were kept constant while only the retrieval similarity threshold remained fixed?

  4. Regarding the Macro-F1 metric: is it calculated by aggregating all predictions, or by calculating the score for each organizational chain individually and then taking the average? Based on the methodology described in the paper, the theoretical lower bound for GPT-5 combined with your approach should be approximately 56.62%, yet Table 2 reports a value of 55.97%. Could you please clarify this discrepancy? Providing the calculation script or a confusion matrix would also be helpful.

Reviewer_xph2OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper introduces AuditEvo, a synthetic benchmark for evaluating whether memory-based agents can reuse historical experience when organizational decision environments evolve. The benchmark contains 40,000 scored procurement-audit cases across 100 chronological organizational chains, with changing policies, market conditions, vendor behavior, and risk patterns. Agents classify each completed transaction as approval, rejection, or investigation using current evidence and previously resolved cases. Verified feedback becomes available immediately after each prediction, while hidden environment states and future information remain inaccessible. A deterministic audit oracle generates reproducible labels.

Experiments across multiple language models and memory frameworks show that existing systems struggle particularly with risk detection and that retrieving related historical cases does not consistently produce accurate decisions. To address this limitation, the authors propose Environment-Aware JitRL. The method maintains an evolving playbook that uses feedback to revise case representations and similarity criteria, retrieve applicable experiences, and adjust action preferences through historical reward estimates without updating model parameters.

The proposed method improves overall accuracy over JitRL by approximately 2–7 percentage points across evaluated backbones. The study highlights the importance of adapting how experience is selected and reused, although its findings are limited to a synthetic audit environment.

Critical strengths
  1. The research question is clear and practically meaningful.
    The paper examines not only whether agents remember past experience, but whether that experience remains applicable after the environment changes. This distinction matters: retrieving similar cases can still lead to incorrect decisions when policies or market conditions differ.

  2. The tasks require cross-case reasoning.
    Risks such as split purchasing, vendor concentration, and bid rotation are difficult to identify from individual transactions. Distributing evidence across chronological cases makes historical memory an important requirement for task completion rather than an optional source of context.

  3. The temporal and feedback protocols are clearly defined.
    Agents can access only historical cases that closed before the current audit time, and the current outcome is revealed only after prediction. Applicable policies are determined by the transaction’s request date. These choices help prevent future-information leakage and support evaluation of continual experience accumulation.

  4. Ground-truth labels are reproducible, and evidence sufficiency is checked.
    Deterministic rules generate labels without relying on subjective LLM judgments. The authors also reconstruct labels using an independent program restricted to publicly available evidence and verify that natural-language descriptions preserve necessary fields. These checks strengthen the benchmark’s internal consistency and verifiability.

  5. The proposed method aligns well with the research problem.
    Environment-Aware JitRL updates case representations and similarity criteria using feedback, directly addressing the possibility that previously similar cases may no longer be applicable. It implements adaptation through changes in how historical experience is selected.

  6. The evaluation includes a useful control for environmental change.
    The appendix fixes environmental conditions while preserving historical dependencies and feedback. Under this setting, the proposed method’s advantage over JitRL decreases. This supports its particular usefulness in evolving environments and provides more explanatory evidence than overall accuracy gains alone.

Critical weaknesses
  1. The synthetic environment is highly structured, limiting its representativeness of real audits.
    Cases are generated from predefined rules and risk patterns, with clean, policy-violation-only, and risk-only cases deliberately separated. No scored cases contain both policy violations and risk findings. Real audits frequently involve missing information, overlapping issues, and conflicting evidence. The benchmark is therefore useful for testing historical reasoning under controlled conditions, but transfer to real-world settings remains unverified.

  2. Immediate, accurate feedback is a strong assumption.
    After every prediction, the model immediately receives the correct label and uses it to update its memory and playbook. In practice, audit outcomes may require lengthy investigations and may remain disputed or contain errors. The paper does not evaluate delayed, missing, or noisy feedback, leaving the method’s dependence on ideal feedback conditions unclear.

  3. Missing component ablations make the source of improvement unclear.
    The proposed method combines an evolving playbook, dynamic features, similarity adjustments, and direct playbook guidance for model predictions. Comparison with the original JitRL alone cannot establish whether gains come from better retrieval, richer prompting, or additional reflection. The fixed-environment control is useful but does not replace these component ablations.

  4. The analysis does not directly characterize adaptation to environmental change.
    The paper primarily reports accuracy averaged over entire sequences and compares fixed versus evolving environments. It does not sufficiently examine performance drops after changes, the number of feedback examples needed for recovery, or when historical experience causes negative transfer. Without these temporal analyses, it is difficult to determine whether the method adapts faster or simply performs classification better overall.

  5. Computational and information budgets are insufficiently transparent.
    The methods differ in summarization, reflection, memory updates, and historical representation reconstruction, but retrieval and ingestion latency alone do not capture their full costs. Model calls, token consumption, and end-to-end latency are not comprehensively reported under a common accounting protocol. Furthermore, top-k is generally set to 10 but can be adjusted for certain components without sufficiently clear budget comparisons. These gaps limit fair assessment of performance and cost.

  6. The scope of “self-evolving agents” exceeds what the task directly evaluates.
    Each case ultimately requires a single three-way classification, without active investigation, tool selection, requests for additional evidence, or planning subsequent actions. This setting evaluates memory and classification adaptation with feedback, but provides limited evidence about broader long-horizon autonomous agent capabilities.

Questions for authors
  1. “Implicit environmental change” covers different information conditions that are easy to confuse.
    Policy changes are communicated through complete replacement documents without highlighting revisions, whereas changes in markets, vendors, and risk states must be inferred from historical transactions. These settings impose different demands. Section 3 should more clearly distinguish directly observable information, changes requiring comparison, and states requiring inference.

  2. “Approve/reject” can be mistaken for pre-purchase authorization.
    The main text uses approve, reject, and investigation, but Appendix A.3 clarifies that the task involves retrospective auditing of completed transactions, including payment information. Operational approval status is also distinct from the model’s audit opinion. This essential task definition should be explicit when the task is first introduced.

  3. How the playbook changes retrieval remains abstract.
    Section 5 and Appendix A.7 describe generating lessons, adding features, revising weights, and reconstructing historical representations. However, they do not sufficiently explain how a natural-language lesson becomes a computable feature or how feature-specific similarities are calculated. The equations describe the overall structure but do not make the execution procedure clear. A worked example tracing feedback through playbook updates to changed retrieval results would help.

  4. The revealed ground-truth label and the action stored in memory are easy to conflate.
    Although the benchmark provides the correct outcome after each prediction, JitRL’s action-value records store the model’s original prediction and its correctness reward, rather than replacing the action with the ground-truth label. This distinction is explained in detail only in Appendix A.8, despite being essential to understanding the equations in Section 5.

  5. Case-overlap F1 is easy to misinterpret.
    This metric measures overlap between retrieved cases and annotated reference sets, rather than evidence sufficiency or retrieval correctness, because the reference sets do not exhaust all valid evidence. Moreover, some methods’ source-case identifiers are reconstructed from summaries or links rather than obtained from direct retrieval logs. The extent to which these scores support fair comparisons across methods needs clearer explanation.

Reviewer_68iJOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces AuditEvo, a new synthetic benchmark for evaluating LLM agents' ability to reuse historical experience in evolving procurement-audit environments. The benchmark contains 40000 cases across 100 chronological organizational chains, where policies, market conditions, vendor behaviors, and risk patterns change over time. Agents must make audit decisions based on current information and historical cases. Experiments across multiple LLM backbones and memory frameworks reveal current limitations in adapting historical experience to changing environments. To address this, this paper further proposes Environment-Aware JitRL, which maintains an evolving playbook to adapt experience retrieval based on feedback. The proposed method achieves approximately 2–7% improvement in audit accuracy over JitRL.

Critical strengths
  • Novel benchmark setting. AuditEvo combines organization-specific procurement workflows with evolving policies, market conditions, vendor behavior, and fraud risks in chronological case chains. This provides a new and distinctive setting for evaluating agents’ ability to infer implicit environmental changes, and selectively reuse historical experience that remains applicable.

  • New method contribution. Environment-Aware JitRL uses a feedback-driven playbook to adapt case representations and retrieval criteria, prioritizing historical experiences that are more compatible with the current decision context.

  • Controlled evaluation design. The deterministic audit oracle and chronological feedback protocol is well designed . The fixed-environment comparison provides a useful control for examining the additional challenges introduced by environmental evolution.

Critical weaknesses
  • Insufficient detail on data construction and expert validation. The paper does not adequately document the procurement data sources, scale, temporal coverage, or their mapping to simulator parameters, leaving empirical rules and author-defined assumptions difficult to distinguish. Appendix A.3 also omits the expert-review protocol, disagreement-resolution procedures, and inter-annotator agreement (IAA).
  • Insufficiently documented identifiability validation. The reported identifiability depends on unspecified rule families, parameter ranges, and calibration-case selection procedures. The public-only oracle’s implementation and independence from the data generator require clarification to assess the strength of this validation.
  • Limited structural diversity in the synthetic dataset. Table 4 reports no cases containing both policy violations and risk findings, while Section 3.3 restricts each risk-target case to a single primary risk family. These choices simplify evaluation but leave more complex cases involving multiple concurrent issues untested.
  • Insufficient ablation study. The playbook affects both retrieval and the base policy, making it difficult to attribute improvements to environment-aware retrieval. Ablations should separate contextual guidance, critic computation, representation adaptation, and action-probability correction.
Questions for authors
  1. Identifiability analysis. Could the authors clarify the detector rule families and parameter ranges used in the identifiability analysis? How were the 31 calibration cases per chain constructed or selected?
  2. Playbook implementation. How are lessons from feedback translated into features and similarity functions? What criteria are used to revise existing features?
  3. Playbook ablations. Could the authors separate the effect of including the playbook in the prompt from its effect on retrieval, under comparable inference budgets?
  4. Outdated memory and negative transfer. Table 5 compares fixed and evolving environments, but does not directly show whether outdated memories hurt performance. Could the authors include recent-history and structured-retrieval baselines, along with an ablation of historical feedback labels, to isolate this effect?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.