- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper introduces a benchmark Kolab consisting of 105 hours of screen-recorded human-agent interaction across 45 professional domains. The human-agent collaboration is evaluated using the collaboration burstiness, workflow topology, and AI rentention rate metrics. Frontier multimodal LLMs were assessed on their understanding of human-agent collaborative workflow.
Critical strengths
- The benchmark is much more comprehensive than existing benchmarks as it covers a wide variety of professional workflows from 10 different sectors with 36 different occupations. Prior benchmarks provide limited grounding in real collaborative workflow or only cover the software development domain.
- The authors show some interesting findings through the use of their proposed metrics such as i) burstier workflows reflects more planning before interaction with agent and ii) vastly different AI retention pattern across different professional domains.
Critical weaknesses
- Since one of the paper’s contribution is also the introduction of three different metrics to analyse human-agent collaboration dynamics, one important analysis appears to be missing which is the relationship between collaboration burstiness and AI retention rate (see question section below for more details).
- The paper does not have code and dataset which prevented me from assessing the quality of the benchmark which is a core contribution of this paper.
Questions for authors
- Can the authors clarify what are the definitions for , , in equation 1.
- Has any analysis been conducted on the relationship between collaboration burstiness and the number of human rejections? Does a high collaboration burstiness leads to less number of human rejections on the final agent output?
- Similarly, has any analysis been conducted on the relationship between collaboration burstiness and the number of adoptions?
4 comments
Dear XVsL,
Thank you for acknowledging that our benchmark is much more comprehensive than existing benchmarks and our work has interesting findings through the use of our proposed metrics.
With the rebuttal period closing today, we wanted to gently check in to see if our responses and additional results have addressed your concerns. Please let us know if you have any remaining questions or need further clarification.
Thank you very much,
Authors
Official comment by Reviewer XVsL
Thank you for the rebuttal. Since most of my concerns for the paper is addressed. I will increase my rating.
Thank you so much!
Thanks for your feedback!
The relationship between collaboration burstiness and AI retention rate
Thanks for your suggestion on this analysis about collaboration burstiness and AI retention rate. We added these results to the paper section 4.2 Finding #3.
Collaboration burstiness and the number of human rejections?
We have analyzed this relationship between collaboration burstiness and the number of human rejections, and it does not support the hypothesis that burstiness reduces rejections. The relationship goes the other way.
In bursty sessions, human–AI interactions cluster in short episodes separated by longer periods of independent work. One possible explanation is that humans delegate substantial work to agents and subsequently scrutinize their outputs, leading to more rejections. However, this hypothesis remains to be tested for future work.
Relationship between collaboration burstiness and the number of adoptions?
Collaboration burstiness is unrelated to the raw count of adoptions (not statistically significant), but it is negatively related to the adoption rate with p<0.005.
Code and dataset.
We added the code and some of the data points as a supplementary material.
Definitions for a, w, t in equation 1.
It was defined in Appendix A.3. Let G=(V,E) be a workflow graph, where V is the set of nodes, each corresponding to a discrete human or AI action, and E⊆V×V is the set of directed edges, with (u,v)∈E indicating that action v directly follows from action u within the workflow. Each node v∈V is annotated by three functions:
an actor label a: V \to \{H, A} indicating whether the action was performed by the human (H) or the agent (A);
a workflow-step label w:V→{1,…,K}.
indicating which of the K steps of the task the action belongs to; and a time interval τ(v)=[ts(v),te(v)]⊂R≥0. recording when the action began and ended.
The paper has been updated and no longer includes graph notation in the main text, following reviewer kKh8's suggestion.