- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper introduces AgentPedia, an open knowledge commons in which store agent findings for reuse. Three types of objects are stored in the accumulated knowledge: content as the knowledge units, indices that organize the units, and rules that govern the changing process. The knowledge can be produced in a flexible way, therefore, things from previous computation could be reused for later tasks. Across multiple various evaluation experiments, the proposed method is shown to reduce page reads and API costs, while also raising answer completeness.
Critical strengths
- The proposed method is shown to preserve intermediate findings and reduce repeated research, which could provide some insights for future work in this direction.
- The proposed approach is shown to provide benefits on reducing page access and API cost.
Critical weaknesses
- The experiments and evaluation is limited, it only tests a domain-specific task, and it is unclear whether things can be generalized to different tasks and domains. There is also a lack of comparisons to related caching and retrieval methods.
- There is a lack of more detailed analysis and ablation study on how different components and design choices of the proposed approach will influence the performance.
Questions for authors
- How does the proposed approach compare with semantic caching and other retrieval methods over previous answers under the same total token budget?
- How will the task relatedness influence the results? For example, if two tasks are more related, maybe the proposed approach might be more useful. Moreover, how about the case where there might contradicting information, will there be mechanisms to handle such scenarios?
- Do benefits persist on naturally occuring tasks across domains and heterogeneous agent models? I think there should be more evaluation and analysis on more different settings.