acceptodds
Under review as a conference paper at ICLR 2027

ChameleonBench: Evaluating the Cross-Environment Robustness Across Heterogeneous Systems

Abstract

Language models increasingly automate programming and desktop workflows, yet success in one environment leaves open whether the same tasks can be completed under different execution conditions. We use environment broadly to denote the external conditions governing observations, available actions, and execution outcomes, including operating systems, software versions, configurations, permissions, hardware resources, network conditions, and runtime state. We present ChameleonBench, a benchmark of 100 tasks—80 Coding and 20 GUI—constructed around native execution behaviors that vary across environments. We focus on native Linux, macOS, and Windows deployments, which provide differences in filesystem, runtime, and application behavior while supporting equivalent task objectives and success criteria. The tasks form 30 mechanism families, with Coding covering individual mechanisms and their combinations. Using a fixed interaction harness and independent verification, we evaluate six models once per task and deployment, yielding 1,800 runs. With equal weighting across families, the strongest model achieves 75.2% mean success across deployments but 57.9% Robust Success Rate, which requires completing the same task on all three. The other models achieve 7.3–23.0% robust success. Grouping by mechanism family also yields more accurate predictions of performance differences across operating systems, on average, than grouping by interaction type and target OS alone. Our results show why task completion and its preservation across environments should be evaluated together.

Primary area:
datasets and benchmarks
Keywords:
agent benchmarking, agent reliability, ai agents

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_2H5kOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper proposes a benchmark called ChameleonBench that contains 100 tasks (respectively 20 for GUI and 80 for coding). This benchmark aims at evaluating LLMs across Linux, MacOS and Windows. The paper finds that models yield mixed results across OSes (for the same given task), with some models more consistent than others.

Critical strengths

None

Critical weaknesses

The paper fails at identifying correctly the literature gaps beyond spanning three operating systems. The writing seems heavily performed by AI tools, without much polishing. A result of this is that the text is often difficult to follow. Experiments are run once per model. The analysis is lacking, the abstract states that Claude-Opus-5 has the highest (robust) success rate, which can be simply counfounded by the fact that it is the most intelligent model used in the study. No research question is explicitly stated nor answered.

Questions for authors
  1. What criteria made you select the six models in the paper? They vary greatly in intelligence.

  2. Why did you run the experiments once per OS and per task? The results as they are now seem to represent a noisy estimate.

  3. What are the explicit research questions you answer? What do you want us, the readers to get out of the paper?

  4. Why did you go for 30 mechanism families when you have so little amount of tests? From your paper we understand that we have 100/30= 3.3 tests per mechanism family. Why not having fewer number of families with more tests per family, if it is a variable you want to study?

  5. How did you weight the tasks for RSR? The paper mentions "Robust Success Rate, which requires completing the same task on all three", before mentioning "RSR is the weighted proportion of tasks completed on all three deployments".

  6. What difference in interpretation should we get from Success Rate and Robust Success Rate?

  7. What is the difference between Chameleon and the previous related work beyond adding MacOS? Why not extending OSWorld with MacOS for instance?

  8. Compared to previous studies related to the field, what new insights do we get?

  9. How was the synthetic data built? Are you going to open source your code and/or the traces?

Reviewer_Lbw9OpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces a benchmark for assessing whether models can reliably handle user requests across heterogeneous systems — specifically Windows, macOS, and Linux. It evaluates 100 tasks (80 Coding + 20 GUI) across the three systems.

Critical strengths
  • Well-formatted figures and tables.

  • Extensive experimental results.

  • A relatively fair benchmark for cross-platform evaluation.

Critical weaknesses
  • The paper's value to the broader community is not clearly established.
Questions for authors
  1. Mechanism organization is hard to follow without a concrete example. Could you explain it in more detail? Specifically, how exactly are tasks classified into mechanism families?

  2. What is the key challenge for future model updates? In other words, what does this benchmark reveal that should guide the next generation of models?

  3. What do you mean when you say the agent is "the same" across platforms? I wonder whether there are subtle differences in tool registration across deployments that could affect comparability. For commercial agents, will there be any adaptors?

Other Comments:

The cross-environment performance plots in Figure 1 are visually attractive but difficult for a human reader to interpret.

Reviewer_xHiwOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper introduces ChameleonBench, which tests whether language-model agents keep completing tasks across native Linux, macOS and Windows deployments. Its 100 tasks, 80 Coding and 20 GUI, are paired across the three OSs with equivalent inputs and verifier-based success criteria, and grouped into 30 families of environmental mechanisms. Six models are run once per task and OS. The strongest, Claude-Opus-5, reaches 75.2% mean SR but only 57.9% robust success, while the others stay between 7.3% and 23.0%. Family-level analyses also reveal reversals of the OS rankings

Critical strengths
  1. All 100 tasks are paired across native Linux, macOS and Windows under one shared contract with independent verifiers. Most prior benchmarks instead rely on migrated subsets, platform-specific tasks or synthetic perturbations.
  2. Defining families by the full set of mechanisms separates single requirements from combinations. Combined mechanisms lower mean SR by 18.2 to 31.3, and family grouping best predicts OS differences.
  3. The reporting is unusually complete, with full family-level results, weighting and leave-one-family-out sensitivity, reference and naive controls, and annotated traces. The aggregate tables can be recomputed exactly from the family-level results.
Critical weaknesses
  1. Single runs with binary outcomes and no confidence intervals, plus 12 of 28 Coding families with one task, mean RSR, EG and family reversals cannot be separated from noise.
  2. Environment is defined broadly, but only three OS deployments are evaluated. Most mechanisms concern filename, path and encoding semantics and GUI covers two families, so it mainly measures filesystem quirks, not general robustness.
  3. Some evaluation choices may confound the attribution to environments. The evaluator strips Python encoding variables the agent session apparently kept, Deepseek's GUI runs use high reasoning, and unimplemented submissions count as environment failures.
Questions for authors
  1. A footnote mentions reruns despite the one-run protocol. How many of the 1,800 outcomes are reruns, and how do results change without the model used for task selection?
  2. Could you repeat a subset with 3 to 5 runs per task and OS to give confidence intervals for RSR and EG, and check whether family-level reversals exceed run-to-run variance?
  3. Why are the Python encoding variables removed only at evaluation and not in the agent sessions? How many Windows failures in the decoding-related families remain once the two environments are matched?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.