acceptodds
Under review as a conference paper at ICLR 2027

Kolab: Benchmarking Human-Agent Collaboration in Long-Horizon Workflows

Abstract

When humans collaborate with AI agents on long-horizon tasks, what happens to the agents' contributions? For solving long-horizon tasks, humans place greater importance on ongoing interaction with AI agents (e.g., revising plans) than on an agent's ability to complete the task in a single attempt. However, most agentic benchmarks have focused more on automation. Strong performance on these benchmarks does not mean that an agent can collaborate effectively when deployed in the real world. Professional workflows offer a useful setting for examining this question, as they often involve interdependent steps. To address this gap, we introduce Kolab, a new dataset of 100+ hours of screen recordings of 45 professionals from 36 occupations collaborating with AI agents on realistic, long-horizon problems. The dataset also contains collaboration and output quality judgments from the participating experts. We evaluate (1) how well agents collaborate with professionals and (2) whether they can serve as verifiers of human–agent collaboration. With Kolab, we find that frontier models, both agentic and non-agentic, tend to overestimate collaboration with experts and struggle to distinguish between high- and low-quality collaboration. As frontier agents cannot yet understand long-horizon collaboration with experts, we position Kolab as a valuable dataset for systematically studying human–agent collaboration grounded in real professional workflows.

Primary area:
datasets and benchmarks
Keywords:
human-agent collaboration, dataset, benchmark, long-horizon workflow

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 4 reviews, above 85% of the others.
Reviewer_XVsLOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces a benchmark Kolab consisting of 105 hours of screen-recorded human-agent interaction across 45 professional domains. The human-agent collaboration is evaluated using the collaboration burstiness, workflow topology, and AI rentention rate metrics. Frontier multimodal LLMs were assessed on their understanding of human-agent collaborative workflow.

Critical strengths
  1. The benchmark is much more comprehensive than existing benchmarks as it covers a wide variety of professional workflows from 10 different sectors with 36 different occupations. Prior benchmarks provide limited grounding in real collaborative workflow or only cover the software development domain.
  2. The authors show some interesting findings through the use of their proposed metrics such as i) burstier workflows reflects more planning before interaction with agent and ii) vastly different AI retention pattern across different professional domains.
Critical weaknesses
  1. Since one of the paper’s contribution is also the introduction of three different metrics to analyse human-agent collaboration dynamics, one important analysis appears to be missing which is the relationship between collaboration burstiness and AI retention rate (see question section below for more details).
  2. The paper does not have code and dataset which prevented me from assessing the quality of the benchmark which is a core contribution of this paper.
Questions for authors
  1. Can the authors clarify what are the definitions for aa, ww, τ\tau in equation 1.
  2. Has any analysis been conducted on the relationship between collaboration burstiness and the number of human rejections? Does a high collaboration burstiness leads to less number of human rejections on the final agent output?
  3. Similarly, has any analysis been conducted on the relationship between collaboration burstiness and the number of adoptions?
4 comments
AuthorsOpenReview ↗

Thanks for your feedback!

The relationship between collaboration burstiness and AI retention rate

Thanks for your suggestion on this analysis about collaboration burstiness and AI retention rate. We added these results to the paper section 4.2 Finding #3.

Collaboration burstiness and the number of human rejections?

Does a high collaboration burstiness leads to less number of human rejections on the final agent output?

We have analyzed this relationship between collaboration burstiness and the number of human rejections, and it does not support the hypothesis that burstiness reduces rejections. The relationship goes the other way.

Collaboration burstiness vs.Spearman's ρp
Number of rejections+0.300.002
Rejection rate (rejects/all responses)+0.290.003

In bursty sessions, human–AI interactions cluster in short episodes separated by longer periods of independent work. One possible explanation is that humans delegate substantial work to agents and subsequently scrutinize their outputs, leading to more rejections. However, this hypothesis remains to be tested for future work.

Relationship between collaboration burstiness and the number of adoptions?

Collaboration burstiness is unrelated to the raw count of adoptions (not statistically significant), but it is negatively related to the adoption rate with p<0.005.

Collaboration burstiness vs.Spearman's ρp
Number of adoptions+0.040.72
Adoption rate R_A (adopts/all responses)−0.300.002

Code and dataset.

We added the code and some of the data points as a supplementary material.

Definitions for a, w, t in equation 1.

It was defined in Appendix A.3. Let G=(V,E)G = (V, E) be a workflow graph, where VV is the set of nodes, each corresponding to a discrete human or AI action, and E⊆V×VE \subseteq V \times V is the set of directed edges, with (u,v)∈E(u,v) \in E indicating that action vv directly follows from action uu within the workflow. Each node v∈Vv \in V is annotated by three functions:

  • an actor label a: V \to \{H, A} indicating whether the action was performed by the human (H) or the agent (A);

  • a workflow-step label w:V→{1,…,K}.w: V \to \{1, \dots, K\}.

indicating which of the KK steps of the task the action belongs to; and a time interval τ(v)=[ts(v),te(v)]⊂R≥0.\tau(v) = [t_s(v), t_e(v)] \subset \mathbb{R}_{\geq 0}. recording when the action began and ended.

The paper has been updated and no longer includes graph notation in the main text, following reviewer kKh8's suggestion.

Dear XVsL,

Thank you for acknowledging that our benchmark is much more comprehensive than existing benchmarks and our work has interesting findings through the use of our proposed metrics.

With the rebuttal period closing today, we wanted to gently check in to see if our responses and additional results have addressed your concerns. Please let us know if you have any remaining questions or need further clarification.

Thank you very much,

Authors

Reviewer_XVsLOpenReview ↗

Official comment by Reviewer XVsL

Thank you for the rebuttal. Since most of my concerns for the paper is addressed. I will increase my rating.

Thank you so much!

Reviewer_SmzvOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces KOLAN, a dataset spanning 105 hours of screen recordings, chat logs and a partcipants’ collaboration rating in which 45 professionals from 36 occupations work with an AI assistant on long-horizon tasks. For a subset of these sessions, the authors build temporal workflow graphs in which human and AI actions are nodes. The nodes carry a label about how the human uses each AI output (adopt, modify or reject).

The paper describes collaboration with three metrics: burstiness, workflow branching and AI retention rate. It then benchmarks six MLLMs on two kinds of task. The first task is recognizing or forecasting the adopt, modify or reject. The second is predicting the participants’ rating from video, chat log, graph or combination of these. The models perform poorly on both tasks. Most over-predict adaptation and under-predict modification and rejection. None beats a constant “always 5” baseline on rating error, and their predictions correlate only weakly with the human ratings.

Critical strengths
  1. The dataset contribution is strong, covering a significant scale of 105h real-world expert Human-AI collaboration data across many different applications, which increases the dataset’s domain diversity coverage. The data quality appears high, with many valuable artifacts, such as screen recordings, chat logs and expert human-AI collaboration ratings. This provides a valuable contribution, even beyond the benchmark.
  2. The measurements of human-AI collaboration are meaningful. Collecting how human users interact with AI generated content in the simple ‘adopt’, ‘modify’ and ‘reject’ taxonomy, and the subjective expert rating of AI output satisfaction is a direct way of measuring human-AI interaction quality
Critical weaknesses
  1. The reported evaluation results are inconsistent in multiple places. For example on page 7, claude is described as best on both adoption tasks, and quotes an F1 of 0.87. However, table 5 shows claude below gemini 3.1 Pro and GPT-6 on task 1a and an F1 score of 0.672 (instead of 0.87). The MAE and Spearman values on page 8 also differ from table 6. The number of adoption examples is 242 according to Table 3, 298 according to page 5 or 290 in table 11.
  2. The presentation of evaluation results is confusing and unfocused. On page 5, the analysis is described as focused on the observed collaborative patterns, rather than on comparing professional domains. However, finding #3 does exactly the opposite. As the workflow differences within and across professions aren't described in detail, I agree with the statement on page 5 that this analysis result #3 shouldn't be claimed here. Section 5 has the same problem. It reports many numbers (precision, recall, F1, per-label F1, within-1, and seven input combinations for four tasks and six models) with little interpretation, and much of it is repeated in Tables 12–15 in the appendix. Some metrics add little: within-1 tells the same story as MAE, since the constant mode-baseline beats every model on both (Table 6). Table 5's caption doesn't say that the F1 scores are macro-averaged or why Claude is excluded from the best results, and the task naming is inconsistent (Task 1a is called both "recognition" and "prediction", and Experiments 1c and 1d in Table 11 are never defined). As a result, the more interesting findings are easy to miss, such as chat-only input missing rejections that happen outside the chat (Figure 6b). Finally, the conclusion is only three sentences long, there is no limitations section, and the future work describes the authors' own plans rather than what the field should take from these results. I suggest focusing Section 5 on one or two core findings and moving the detailed breakdowns to the appendix.
Questions for authors

Clarifications

  1. Which of the conflicting numbers in Weakness 1 are correct?
  2. According to Appendix A.8, Claude was queried manually through its web interface, on fixed subsets for Tasks 1a, 2a and 2b (e.g., a class-balanced subset of 75 examples for Task 1a). Can you evaluate Claude through the API on the same full sets as the other models? Otherwise, its scores can't be compared with the rest.
  3. What is the agreement between the two annotators on the adopt, modify and reject labels? And how well do humans perform on the forecasting and rating tasks, given the same input as the models? Without this, it's unclear whether the low scores reflect a model limitation or a task that can't be predicted from the input.
  4. The annotation guidelines (Appendix A.4.1) use REJECT only when the user explicitly states that they reject an output, while the model prompts (Appendix A.6) also count ignoring an output as REJECT. Which definition do the labels follow?
  5. The recordings show participants' real work. How were they anonymized, and did participants (and any clients whose data appears) consent to the public release?

Suggestion

  1. Workflow branching counts nodes with more than one outgoing edge, regardless of who acts or for how long. Since every node has a start and end time, did you consider measuring how long human and agent actions actually overlap? This might capture concurrent work more directly.
5 comments
AuthorsOpenReview ↗

Thanks for your feedback!

Which of the conflicting numbers in Weakness 1 are correct?

As mentioned in the general comment, now we have 100 annotated sessions (100 videos) and all the numbers in the paper and the appendix are consistent. We have updated our paper to reflect the updated numbers.

Now, the upper-bound F1-score (Experiment 1a) of the best model (Claude) run on the full instances (692 instances) is 0.634. In Table 5 of the updated paper, GPT-6’s score is 0.713 but it is evaluated on fewer instances (279 instances instead of the full 692 instances) due to cost limitation.

For MAE and Spearman values, the scores in the writing for paragraph “Performance on predicting collaboration satisfaction and output quality” are now consistent with Table 6.

The number of ADOPT instances is 456 and it is consistent across the paper.

On page 5, this analysis result #3 shouldn't be claimed here.

We assume you are referring to Finding #3, “AI retention patterns vary across professional domains,” in Section 4.2. This finding was intended as a descriptive analysis to illustrate variations in human–agent collaboration across professional domains, rather than a systematic comparison of professions.

We understand your concern. In the updated paper, we have moved this descriptive analysis to the Appendix and replaced it with another analysis of AI retention x collaboration burstiness as we addressed Reviewer XVsL's suggestion.

Table 5's caption doesn't say that the F1 scores are macro-averaged or why Claude is excluded from the best results

Claude was not highlighted in Table 5 because the majority of participants used Claude, which could introduce a potential bias in the evaluation. However, we still included its results for completeness and discussed them in the text (Lines 365–366 in the initial submission).

In the updated paper, we have revised Table 5 to explicitly say “Macro F1” and highlighted Claude's results as well.

The task naming consistencies, longer conclusion, restructuring

We have updated the writing to reflect this. Now all task names are consistent and Experiment 1c (step-level forecasting) and 1d (workflow-level forecasting) are elaborated in the Appendix due to page limitation. The conclusion section now includes limitations and what the field should take from these results.

Evaluating Claude through its API

Unfortunately, our university policy does not allow the usage of Claude API. But we ensured that Claude did not have access to the ground-truth labels during evaluation.

We acknowledge that evaluating Claude through its interface is not directly comparable to API-based evaluations of Gemini and GPT-6. However, we believe that including Claude results in our paper provides a useful complementary reference, particularly because our dataset captures real-world workflows in which professionals interact with AI agents through user interfaces.

The interannotator agreement

Since we have updated the paper to incorporate additional annotated data from our existing collection, now we have annotated all the 100 sessions with 692 labeled instances. Four human annotators labeled the data with the final score of a percent agreement of 86.7% and a Krippendorff’s alpha of 0.80 (substantial agreement). We also added this information in Appendix A6.

Please see our response about interannotator agreement details in the general comment.

AuthorsOpenReview ↗

Human Performance

Human performance on the forecasting and rating tasks?

Results for comparing human performances vs. models for the forecasting task (1b) with the same 115 instances. GPT-6 only has 29 overlapping instances so we exclude GPT-6 results. N=115 instances which is 16.61% of the total 692 instances. The Always-ADOPT baseline predicts ADOPT for every instance, the same with the MODE baseline in our paper. For completeness, we also show results with Always-MODIFY and Always-REJECT baselines.

ScoringMacro-F1
Humans0.414
Gemini Pro0.353
Claude0.342
Qwen3-30B0.385
Gemini Flash0.275
Qwen3-32B0.273
Always-MODIFY baseline0.178
Always-REJECT baseline0.178
Always-ADOPT baseline0.149

These results suggest that forecasting human adoption behavior remains challenging for both humans and frontier models. We highlight two considerations. First, our human evaluators were given randomly sampled instances from professional domains they are not familiar with (e.g., aerospace engineering, 3D animation, GIS, and music composition). In contrast, AI agents are trained on large-scale data which may include that information of those professions so we expect they have better knowledge on what is good and bad. Second, AI agents are expected to adapt to individual users over time, making this capability particularly important for effective long-horizon collaboration. Even with longer context (experiment 1c and 1d in the Appendix) AI agents do not consistently perform better.

Results for comparing human performances vs. models for prediction collaboration rating (task 2a) with N= 85 instances, which is 24.4% of instances used in experiment 2a. GPT-6 is not compared because it has fewer instances. Humans always have lower MAE and higher Spearman rho. The Always-5 baseline predicts a rating of 5 for every instance, the same with the MODE baseline in our paper.

MAE (lower is better)Spearman's ρ (higher is better)
Humans0.890.50
Claude0.950.25
Qwen32B0.990.22
Qwen30B1.110.14
Gemini Pro1.490.20
Always-5 baseline1.52no variance

We didn’t ask humans to label all instances due to resource and time constraints. We also prioritized Task 2a over Task 2b because collaboration satisfaction and output quality are positively correlated in our dataset (Figure 15 in the Appendix).

AuthorsOpenReview ↗

REJECT Definition, anonymization, suggestion on workflow concurrency

REJECT definition

The definition of reject is defined in the main paper of the first version of submission (not the rebuttal version) line 209-210. Ignoring is included as a REJECT. We have revised the Appendix to reflect this update.

The recordings show participants' real work. How were they anonymized, and did participants consent to the public release?

Any parts of the videos containing personally identifiable information (PII), including client information (e.g., email addresses, names, physical addresses, and company names), are covered with gray boxes or blurred. All participants provided consent for the public release of their data and were informed that their data would be made publicly available.

Suggestion: Workflow branching counts nodes with more than one outgoing edge, regardless of who acts or for how long. Since every node has a start and end time, did you consider measuring how long human and agent actions actually overlap? This might capture concurrent work more directly.

Thanks for your suggestion! We agree that timing can offer a more direct view of concurrency than branching, so we tried to run the experiment with this time-based concurrency measure. Because every node has start and end times, we computed human–agent overlap as the total time during which at least one human node and at least one agent node were active simultaneously, reported as a share of session length (and of agent-active time). We computed it for all 100 sessions.

Overlap is strongly related to branching (Spearman ρ = +0.78, p < 0.001). It also supports our finding that branching workflows are associated with lower perceived collaboration experience. Sessions with more human–agent overlap are rated lower (ρ = −0.30, p < 0.01), a pattern comparable to branching W (ρ = −0.28, p < 0.01).

Reviewer_kKh8OpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper releases a benchmark based on screen recordings between human and AI collaboration on real-world computer-use tasks like art design, architecture and engineering. The data is newly collected with paid experts from 45 professionals from 36 occupations, totaling 100+ hours of recording. In the process, experts are free to work on the task with claude code agents. The collected trace is used for both analyzing the human ai collaboration behavior as well as evaluating the model’s capability to predict collaboration outcomes of an agent action. The authors conduct thorough analysis on the traces, and draw conclusions on the Human AI process itself as well as models’s ability to assess the collaboration process.

Critical strengths

I think the problem is well motivated and the overall data annotation process and experiments design is reasonable. The dataset can be generally useful for the community, and the analysis and results are relatively interesting and informative. For example, the analysis of the AI retention rate across domains is interesting and is complementary to some other related work like “How Well Does Agent Development Reflect Real-World Work?”

Critical weaknesses

I think the presentation can be improved –

  • A few claims are not rigorous – e.g., this sentence in the abstract “For solving long-horizon tasks, humans place greater importance on ongoing interaction with AI agents (e.g., revising plans) than on an agent’s ability to complete the task in a single attempt” is a strong claim that needs to be validated.
  • I am not sure if the graph-based notation is necessary in section 4. I also feel like the task description and formalization may come earlier than the data collection part?

I am happy to adjust my rating after the authors have a careful pass in improving the writing and presentation of this paper.

Questions for authors
  • Please fix the overlapping text in figure 1.
4 comments
AuthorsOpenReview ↗

Thanks for your feedback!

We have updated the abstract, improved the presentation and writing of the paper based on your comments, restructured the section positions, and introduced the task and formalization earlier in the paper.

We'd be grateful if you could reconsider your rating paper in light of these revisions. If you have any further suggestions, please let us know, and we'll be happy to revise it again if needed. Thanks!

Dear reviewer kKh8,

Thank you again for acknowledging the importance of our benchmark and for your constructive feedback about the writing. We have updated our paper following your feedback!

As the discussion period is coming to a close, please let us know if these resolve your concerns or if any further updates are needed. If you feel your concerns have been addressed, we’d truly appreciate it if you will consider adjusting your score.

Thank you very much,

Authors

Thank you for the update! I think my concerns are addressed and I've adjusted my score accordingly.

Thank you very much for updating your score!

Reviewer_oQYNOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces a dataset and benchmark for studying human–agent collaboration in long-horizon professional workflows. It includes 90 sessions involving professionals from 36 occupations, each bringing a real work problem requiring at least an hour of effort, five interdependent steps, and a concrete deliverable. The dataset captures chat logs, recordings, files, deliverables, and professional ratings of collaboration and output quality on a five-point scale.

Agent outputs are labeled according to whether the human adopts, modifies, or rejects them. A valuable aspect of this setting is that these responses can occur outside the chat interface—for example, when an architect transfers an agent-generated layout into Revit. Six models are evaluated on recognizing and forecasting these labels and predicting human ratings. Forecasting F1 ranges from 0.4245 to 0.562, compared with a majority-class baseline of 0.249, while the rating-prediction results do not outperform the baseline.

Critical strengths

The primary contribution is the dataset. Recruiting professionals and collecting realistic, extended workflows across 36 occupations is difficult and valuable. Using participants’ own work problems and requiring concrete deliverables strengthens the practical relevance of the benchmark.

Capturing human responses beyond the chat window is particularly useful: adoption, modification, and rejection may only become apparent through actions in external applications. The public release of session recordings, chat logs, files, and relevant artifacts could support further research on these interactions.

Critical weaknesses

The reliability of the benchmark and the sources of evaluation variability need clearer characterization. Although the paper reports standard deviations across three runs, it is unclear what changes between runs and how this variability is defined at the instance level. This makes it difficult to distinguish model instability from other sources of evaluation noise.

The reliability of the adoption labels also requires clarification. In particular, it is unclear how consistently annotators distinguish adoption, modification, and rejection, especially when the relevant behavior occurs outside the chat interface. Without inter-annotator agreement statistics and a description of disagreement resolution, the amount of ambiguity in the benchmark is difficult to assess.

Questions for authors
  1. What varies across the three evaluation runs? Are the same instances and context inputs used each time?
  2. How many annotators independently labeled each instance, and what was their agreement on adoption, modification, and rejection?
  3. How were ambiguous cases and annotator disagreements resolved, particularly for actions performed outside the chat interface?
2 comments
AuthorsOpenReview ↗

Thanks for your feedback!

Thank you for recognizing the value of our dataset, the challenges of collecting realistic workflow videos from working professionals, and the usefulness of our benchmark task on capturing human responses beyond the chat window.

What varies across the three evaluation runs?

Nothing changes for all the three runs. The prompts and LLMs and agent settings remain the same for all the three runs. We have updated the paper to make this setting clear.

Interannotator agreement for adoption, modification, and rejection, especially when the relevant behavior occurs outside the chat interface

As mentioned in the general comment, we have updated the paper to incorporate additional annotated data from our existing collection, so now we have annotated all the 100 sessions with 692 labeled instances. Four human annotators labeled the data with the final score of a percent agreement of 86.7% and a Krippendorff’s alpha of 0.80 (substantial agreement).

In the first round, all four annotators labeled 112 instances and achieved 84.1% agreement and a Krippendorff’s alpha of 0.76. In the second round, we randomly sampled 54 new instances, which were labeled by all four annotators, achieving 86.7% agreement and a Krippendorff’s alpha of 0.80.

Out of these 166 instances used for computing inter-annotator agreement scores,126 (75.9%) are not happening on the AI chat interface. The inter-annotator agreement for this subset (combined round 1 and round 2) is 86.8% for percent agreement and 0.8 for Krippendorff’s alpha.

  • [Round 1] For instances occurring outside the chat interface (91 instances), the inter-annotator agreement scores are:

    • Percent agreement: 85.2%
    • Krippendorff’s α: 0.78
  • [Round 2] For instances occurring outside the chat interface (35 instances), the inter-annotator agreement scores are:

    • Percent agreement: 91.0%
    • Krippendorff’s α: 0.87

A description of disagreement resolution and solving ambiguity

Disagreements are solved through multiple rounds of discussions and we resolve the disagreement as a team, including with discussions with the professionals who recorded their screens. Then, we update the annotation guideline accordingly. We updated the Appendix A6. Data Processing and Annotation Details to contain this information.

The annotation guideline is added in the supplementary material: Annotation Guidelines.pdf.

How many annotators are independently labeled each instance?

In the initial submission, we already described this in line 201 (Section 4.1): two human annotators independently labeled the instances.

After updating the paper during this rebuttal period, now we have four human annotators who independently labeled each instance, besides the instances used for the interannotator agreement session. We added this information to the revised paper in line 191.

Dear reviewer oQYN,

Thank you for acknowledging that our dataset of human-AI collaboration in 100 realistic professional workflows is valuable and the benchmark task of capturing human responses beyond the chat window is particularly useful.

With the rebuttal period closing today, we wanted to gently check in to see if our responses and additional results have addressed your concerns. Please let us know if you have any remaining questions or need further clarification.

Thank you very much,

Authors

Comments on the paper

2 comments
AuthorsOpenReview ↗

General Comment

We thank the reviewers for their constructive feedback. We have updated the paper and added supplementary materials accordingly. Here are our main updates:

  • Expanded dataset and annotations. Our dataset now contains 100 videos, all human-annotated with adopt/modify/reject labels. Four human annotators achieved a percent agreement of 86.7% and a Krippendorff’s α of 0.80, indicating strong annotation reliability.
  • Updated experiments and analyses. We revised the paper and updated all experimental results to incorporate the additional data points and analyses.
  • Additional supplementary materials. We uploaded our code, annotation guidelines (PDF), and sample data points.

Regarding Inter-Annotator Agreement

We have updated the paper to incorporate additional annotated data from our existing collection. All 100 sessions are now annotated, comprising 692 labeled instances.

We conducted two rounds of inter-annotator agreement evaluation with four human annotators:

  • Round 1: All four annotators independently labeled 112 instances, achieving *percent agreement = 84.1% and Krippendorff’s α = 0.76.
  • Round 2: We randomly sampled 54 new instances, independently labeled by all four annotators, achieving percent agreement = 86.7% and Krippendorff’s α = 0.80 (substantial agreement).
AuthorsOpenReview ↗

Summary of Reviews and Revisions

Dear Area Chair and Reviewers,

Thank you for your time and effort in reviewing our paper and providing constructive feedback.

We appreciate the reviewers' recognition of the following strengths of our work:

  1. Comprehensive and valuable dataset. All four reviewers (oQYN, kKh8, Smzv, XVsL) agree that our dataset is comprehensive and valuable for the community. Collecting videos of 50 professionals (40 different domains) collaborating with AI agents in realistic long-horizon workflow with various artifacts (video, annotated graph, chat log, deliverables) is useful even beyond the benchmark task (Smzv).
  2. Well-motivated problem and rigorous data collection. The problem is well-motivated, and data quality is high with rigorous data and annotation process (kKh8, Smzv).
  3. Meaningful workflow metrics. Reviewers recognized the value of the proposed metrics for analyzing human–agent collaboration (Smzv, XVsL).
  4. Useful benchmark tasks and informative findings. The benchmark task itself of capturing human responses beyond the chat window is particularly useful and experiment results and analysis are interesting and informative (oQYN, kKh8, XVsL)

We also have addressed the concerns raised by the reviewers and updated the paper accordingly:

  1. For the inter-annotator agreement of ADOPT/MODIFY/REJECT labels by four human annotators (oQYN, Smzv), the Krippendorf's alpha score is 0.80, indicating high reliability.
  2. Improvement of the presentation: we removed the graph notation from Section 4 (kKh8) and updated the paper to have consistent wording and numbers (Smzv).
  3. For the relationship between collaboration burstiness and AI retention rate (XVsL), we added this analysis to Section 4.2 Finding #3.

We hope that AC and reviewers could take these into consideration in the final assessment of our paper.

Thank you very much,

Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.