acceptodds
Withdrawn from ICLR 2027

ReplayLens: Auditing Agents' Use of Outcomes

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

When an agent reuses logged experience, a changed decision may reflect the recorded score, the action's name, or the record's position in storage. Standard memory evaluations do not reveal which relationship drives that change. We introduce ReplayLens, a black-box audit that changes one relationship in the stored history at a time, holds the remaining interface fixed, and measures the resulting decision. Four interventions target four relationships. Outcome reassignment swaps which scores belong to which actions. Pair transport moves intact action–score pairs to new record slots. Consistent renaming relabels actions in both history and menu. Key-slot reassignment changes both score attachment and position. A constructive separation shows why the audit is needed: two memory writers with identical endpoint accuracy respond differently to the same replay, so conventional evaluation cannot resolve the underlying dependence. On black-box LLM interfaces, swapping scores changes decisions while moving intact pairs does not, separating score attachment from record order. A bounded-memory study exposes ingestion-order sensitivity that endpoint comparison misses. In sequential experiment planning, altered historical scores redirect exploration and reduce final utility despite fresh measurements. A code-debugging agent with sealed hidden tests shows the same pattern outside model selection. ReplayLens provides a relationship-level audit for deciding whether logged experience can be merged, reordered, or reindexed safely.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Agent Memory, Black-Box Auditing

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_tHTrOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper propose ReplayLens to find that how current LLM agents use historical outcomes based on a block-box method. The authors perturbs some conditions to see what will change for the LLM's choice. For example, when the records are reordered, the results can change significantly. These methods help to check the organization problems.

Critical strengths
  1. The authors split the research aim into several specific problems.
  2. The experiments are extensive and relatively solid.
Critical weaknesses
  1. The overall novelty of this paper is limited. It merely contains several perturbation of historical conditions.
  2. The experiment results are reasonable, but have already met common scene in this field, which brings limited innovation.
  3. Actually, I cannot get the exact usage of the proposed method in real world. The influence of historical text is highly compound with the specific model, harness, hyper-param, etc. A single condition changes will significantly change the conclusion, which may happen frequently in the real world deployment.
  4. The appendix is too extensive and difficult to navigate. It mixes complex archival results, making it really hard to identify the evidence supporting the main claims. Besides, although in the main body, the authors refer to the appendix somewhere, it is hard to clearly connect them.
Questions for authors

Please see the weakness part, especially for the third one.

Reviewer_eF9rOpenReview ↗
Rating
3
Confidence
1
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

This paper asks a critical question in interpretability. When an agent indeed benefits from past experiences, how do we know which parts of those experiences actually are impactful and helpful for the decisions? To tackle this problem, this paper introduced four intervention methods for this audition.

Critical strengths
  1. The motivation is important, and the solution of this problem would significantly improve reliability and real deployment of agents.

  2. It proposes a novel methodology based on controlled transformations of stored experience. Each intervention method is informative and the design seems suitable for the task.

  3. The paper explicitly argues which relationships each intervention preserves and changes.

  4. Empirical experiments are extensive and convincing.

Critical weaknesses

I am not very knowledgeable about this field. But my observation is that, the method only applies to a rather rigid, well-defined action-outcome logs. This homogeneity may restrict the applicability.

Questions for authors

See weaknesses.

Reviewer_BqxwOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

this paper presents an intervention (or perturbation) study for agent memory. the authors study which part of an agent's stored experience or memory contribute to its later actions. they build ReplyLens, a black box framework for "auditing" agent memory and experience. this framework would change on part of the agent history at each time and rebuild memory with the same writer. it then queries the llm or the agent with the same question to measure what changes in agent actions and task reward.

Critical strengths

+ agent memory analysis is an important and timely topic, especially understanding why or which part of agent memory is contributing to task rewards.

+ the authors present a large number of evaluation results to test the proposed method.

Critical weaknesses

- even as someone who does research in the field of agent memory, i find this paper very hard to follow and understand. the writing is very abstract and uses many terms that the research community does not conventionally use. after quite some time i finally understand that by "audit" the authors refer to an input-level intervention or perturbation study for agent memory; on the other hand, in conventional ML, "audit" is often used in algorithm fairness studies. the authors never directly discuss what research problem they are trying to tackle, what challenges are there, and what could be the significance if the problem is resolved. overall i am quite lost after reading this paper and do not understand what the authors are trying to convey.

- the evaluation section is very arbitrary and the presentation is confusing, with no standard benchmarks/datasets that the community is familiar with and no legit baseline methods that can be used for comparison. every task is built by the authors themselves. the UCI datasets (sec 4.6) are standard, but they are used only to produce scores. on baselines: for a diagnostic method, the baseline should be another way to diagnose memory use; the authors even cite many of these in section 2. the only comparison in this paper is "endpoint evaluation", which was never meant to answer which relation drives a choice. the reference rules and random search describe the agent’s behavior, but they are not baselines for the method.

Questions for authors
  • in the final version, please consider polishing figures 1 and 2. currently the font is very small, and the choice of color (especially figure 1) makes it difficult to read.

My major concerns and questions are in "weaknesses".

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.