- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This work studies the using of time as an implicit coordination signal for social interaction. The authors use a game-"The Mind", where two players must play cards in ascending order with the passage of time as the only available signal. The experiments design two rule-based strategies and genereate five types of rule-based partner. Then, the authors let humans, computer-use agents, and text-based LLMs play with these partners, respectively. The results indicate that humans and stronger models often behave in ways consistent with Bayesian reasoning. Moreover, humans show much stronger online adaptation to their partners.
Critical strengths
-
The problem being studied is very Interesting. The ability of using time as the coordination signal is overlooked by most of the agent research work.
-
The authors have conduct comprehensive experiments. They compare humans, CUAs, and text-based LLMs under a shared framework. The results indicate that agents lack the online adapation ability and also raise a promising research direction.
Critical weaknesses
-
The interaction setting is still quite controlled. Most experiments pair humans or models with fixed rule-based partners. This makes the analysis cleaner, but it is unclear whether the same conclusions hold when both players are adaptive, especially in Human-LLM or Human-CUA interaction.
-
The timing setup is not fully comparable across humans and LLMs. Humans experience continuous wall-clock time, while text-based LLMs are explicitly given simulated elapsed time or WAIT counts. This makes some of the human-LLM comparison difficult to interpret.
Questions for authors
-
Have the authors considered directly pairing humans with LLMs or CUAs? It would be interesting to see whether the temporal coordination strategies observed with rule-based partners also emerge in human-AI interaction.
-
In the +Calibration conditions, both the human and the rule-based partner may adapt over time. Can the authors better disentangle how much of the performance improvement comes from human adaptation versus partner adaptation?
-
In the human experiment, was the use of external timing tools such as a system clock or phone explicitly prohibited? Since the interface in Fig. 6 itself does not show elapsed time, this seems relevant to the claim that humans rely on implicit temporal perception.
3 comments
Thanks for the detailed responses. My concerns have been addressed.
Dear Reviewer UNcY,
Thanks again for your helpful suggestions and questions! We are glad our responses addressed your concerns. We really appreciate your thoughtful feedback!
Best,
Authors
We thank the reviewer for their helpful comments and questions!
Weakness #1: but it is unclear whether the same conclusions hold when both players are adaptive, especially in Human-LLM or Human-CUA interaction.
Thank you for raising this point! We note that we do already consider settings where both players are adaptive: the rule-based players with calibration are adaptive, and we also include LLMs vs LLMs in our experiments.
We did not conduct Human-LLM interaction because it would be extremely unnatural (if not impossible), as LLMs and humans process time differently. LLMs can only respond to prompts and have no intrinsic “sense” of time, while humans directly experience the continuous passage of time. In this case, it is infeasible to make humans and LLMs use their “native” ability about time in a shared game.
Another reason why we did not include Human-CUA, Human-LLM, or Human-Human experiments is because identifying a player’s strategy and measuring adaptation requires the partner’s strategy to be known. When both players’ strategies are unknown and co-evolving, it becomes difficult to identify which player is adapting to whom, and the strategy-fitting analysis becomes much more noisy. Additionally, a shared set of partners makes humans, LLMs and CUAs comparable with each other in their adaption and fitting strategies. We will elaborate on these points in the camera-ready revision.
Weakness #2: The timing setup is not fully comparable across humans and LLMs.
Yes, we agree that absolute timing is not directly comparable between humans and text-based LLMs. Please note that we have already flagged this in Section 3.3.3 and the final limitations paragraph in the Discussion.
This incongruity is rooted in how LLMs operate: they can only respond to prompts and have no intrinsic “sense” of time. The distinction between human and LLM treatment of time is motivated by our broader question: i.e., how a non-time-embodied system can perform the time-based social coordination task. This is also one of the reasons why we additionally included CUAs, which can be evaluated under the same experimental conditions as humans.
Importantly, only RQ1 (but not RQs2-4) relies on the direct comparison of absolute timing between humans and LLMs. We will make the limitation more explicit when we discuss RQ1 in the camera-ready revision.
Question #1: Have the authors considered directly pairing humans with LLMs or CUAs?
Thank you for this question! In our current experiments, we limit the human/LLM/CUA players with rule-based players for a more clean analysis of the behavior patterns, as discussed in our response above.
Question #2: In the +Calibration conditions, both the human and the rule-based partner may adapt over time. Can the authors better disentangle how much of the performance improvement comes from human adaptation versus partner adaptation?
This is an interesting question! To answer this question, we collected the errors made by human players and their +Calibration partners separately and divided the play history into three bins. We find human error rates dropped significantly from the first to the last third with both Linear (18.8%->12.6%, paired t(29) = 2.73, p = 0.011) and Bayesian partners (14.6%->10.7%, paired t(29) = 2.60, p = 0.014). The partners’ error rates changed by less than 1.5 percentage points with no significant results (p>0.49). This suggests that the performance improvement in the +Calibration conditions comes mainly from human adaptation rather than partner calibration. We did a similar analysis with GPT6 CUAs and we did not find such adaptation. This is consistent with our analysis in Sec 4.3. We will add a brief analysis of the results to Sec 4.3.
Question #3: In the human experiment, was the use of external timing tools such as a system clock or phone explicitly prohibited?
Thanks for raising this question. We did not explicitly prohibit participants from using external timing tools. However, as the experiment is restricted to desktop/laptop users and our experiment setting is made on a timescale of seconds, we believe that the system clock would provide little useful information about time passage given that a system clock typically displays at minute-level resolution.