- Rating
- 3
- Confidence
- 4
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper proposes RADAR, an auditable runtime that makes an agent's analytical choices inspectable and supports their revision through execution feedback, for data analysis tasks.
Critical strengths
-
The authors open-sourced the codebase of RADRA, and I highly appreciate it.
-
The authors done comprehensive evaluations through multiple benchmarks, i.e., DABStep benchmark, KramaBench, and DA-Code. Specifically, it outperforms the state-of-the-art baseline (DS-STAR), and also shows high generalizability across various evaluation setups.
-
I think the idea is novel, and also important, since auditing the execution traces for any kinds of agents are an important topic to discuss. In addition, the authors provided comprehensive ablation studies to support their idea.
Critical weaknesses
- The key weakness I think is the absolute performance of the RADAR. Most importantly, DS-STAR is not the state-of-the-art data science agent now. As one can see in DABStep Leaderboard, now there are four more agents, i.e., NVIDIA KGMON Data Explorer, DataPilot, gg-agent-gpt-1104-1, CambioML energent.ai DS Agent. Specifically, NIVIDA KGMON Data Explorer achieved almost 90% of accuracy, which is much more higher than RADAR. I would like to recommend an acceptance if the authors can resolve this concern.
Questions for authors
-
What is the key difference between DS-STAR's data analyzer which generates descriptions for each data, and RADAR's source map construction? Can authors provide comparative ablation study to verify that RADAR's source map construction is better design choice for data science tasks?
-
DS-STAR also proposes simple query-aware exploration, which compares the cosine similarity between the embeddings of the data description and the user query. On the other hand, RADAR uses BM25 to retrieve the useful data files when the number of files are too large (e.g., in KramaBench). What is a better choice? Can authors provide ablation studies on this? For example, maybe we can check some precisions, recalls in KramaBench subsets
-
In Wildfire domain in KramaBench, why RADAR fails to achieve a better score when using oracle inputs? This is weird, and I think there should be a sufficient explanation in dedicated section.
-
Can you provide qualitative examples for visualization tasks in DA-Code benchmark?
-
Is the proposed idea can be applied to any kind of data science agents? If so, it would be good to add on SOTA methods, and see even we can get better scores.
2 comments
Thank you for your thoughtful feedback and for reconsidering your assessment. We are glad that the additional experiments addressed your concerns. We will incorporate these results and discussions into the final revision.
Thank you for the detailed response and for conducting the additional experiments during the rebuttal period. The new empirical results and clarifications effectively address my primary concerns. Accordingly, I am happy to raise my score from 2 to 3. Please make sure to incorporate these results and discussions into the final revision.