- Rating
- 1
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The paper proposes a benchmark called ChameleonBench that contains 100 tasks (respectively 20 for GUI and 80 for coding). This benchmark aims at evaluating LLMs across Linux, MacOS and Windows. The paper finds that models yield mixed results across OSes (for the same given task), with some models more consistent than others.
Critical strengths
None
Critical weaknesses
The paper fails at identifying correctly the literature gaps beyond spanning three operating systems. The writing seems heavily performed by AI tools, without much polishing. A result of this is that the text is often difficult to follow. Experiments are run once per model. The analysis is lacking, the abstract states that Claude-Opus-5 has the highest (robust) success rate, which can be simply counfounded by the fact that it is the most intelligent model used in the study. No research question is explicitly stated nor answered.
Questions for authors
-
What criteria made you select the six models in the paper? They vary greatly in intelligence.
-
Why did you run the experiments once per OS and per task? The results as they are now seem to represent a noisy estimate.
-
What are the explicit research questions you answer? What do you want us, the readers to get out of the paper?
-
Why did you go for 30 mechanism families when you have so little amount of tests? From your paper we understand that we have 100/30= 3.3 tests per mechanism family. Why not having fewer number of families with more tests per family, if it is a variable you want to study?
-
How did you weight the tasks for RSR? The paper mentions "Robust Success Rate, which requires completing the same task on all three", before mentioning "RSR is the weighted proportion of tasks completed on all three deployments".
-
What difference in interpretation should we get from Success Rate and Robust Success Rate?
-
What is the difference between Chameleon and the previous related work beyond adding MacOS? Why not extending OSWorld with MacOS for instance?
-
Compared to previous studies related to the field, what new insights do we get?
-
How was the synthetic data built? Are you going to open source your code and/or the traces?