- Rating
- 1
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper propose ReplayLens to find that how current LLM agents use historical outcomes based on a block-box method. The authors perturbs some conditions to see what will change for the LLM's choice. For example, when the records are reordered, the results can change significantly. These methods help to check the organization problems.
Critical strengths
- The authors split the research aim into several specific problems.
- The experiments are extensive and relatively solid.
Critical weaknesses
- The overall novelty of this paper is limited. It merely contains several perturbation of historical conditions.
- The experiment results are reasonable, but have already met common scene in this field, which brings limited innovation.
- Actually, I cannot get the exact usage of the proposed method in real world. The influence of historical text is highly compound with the specific model, harness, hyper-param, etc. A single condition changes will significantly change the conclusion, which may happen frequently in the real world deployment.
- The appendix is too extensive and difficult to navigate. It mixes complex archival results, making it really hard to identify the evidence supporting the main claims. Besides, although in the main body, the authors refer to the appendix somewhere, it is hard to clearly connect them.
Questions for authors
Please see the weakness part, especially for the third one.