acceptodds
Under review as a conference paper at ICLR 2027

Evaluating Agent Collaboration at Scale via Simulated Software Worlds

Abstract

Scaling agent collaboration requires understanding how agents choose work, coordinate changes, and manage shared resources. We introduce Software World, a framework for studying these decisions through the maintenance of real, interdependent Python packages. Software World has three components: an environment where agents maintain software together, a method for constructing worlds and evaluating changes on unseen packages, and infrastructure for running and recording simulations at scale. Agents choose their own tasks to improve software efficiency while preserving correctness. We use human-authored tests and benchmarks to measure correctness and speedups in dependent packages that agents never see during the simulation. Across 31 simulations in six-package worlds with four models, we collect nearly 14K agent work-session traces. Across all four model families, agents produce larger held-out speedups when they can collaborate than when they work in isolation. Claude Opus 5 agents frequently initiate exchanges, while GPT-5.6 Sol agents make no contact when communication is optional. Assigning each agent a package can improve performance compared with letting agents divide up the work themselves. When agents share a budget through a voting system, some fail to coordinate access, leaving shared resources unused after their individual budgets run out. We will release our framework and dataset upon acceptance.

Primary area:
datasets and benchmarks
Keywords:
software ecosystem, software agents, agent evaluation, agent collaboration, agent simulation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_HgrBOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates how LLM agents should select tasks, coordinate code changes, exchange information, and allocate shared resources when maintaining interdependent software packages over long-running, open-ended tasks.

The authors introduce SOFTWARE WORLD, a framework consisting of three components:

  1. SW-ENV: A collaborative environment built around real Python packages, where agents autonomously select optimization tasks and interact asynchronously through issues, pull requests, code reviews, and releases.
  2. SW-BENCH: A world-construction and evaluation methodology that uses real software dependencies and human-authored tests and performance benchmarks to measure improvements on downstream packages that agents cannot access or modify.
  3. SW-HARBOR: An execution infrastructure supporting parallel agent sessions, persistent memory, configurable budgets, and comprehensive experiment logging. The authors conduct 31 simulations in six-package worlds, evaluating four model families: GPT-5.6 Sol, Claude Opus 5, Gemini 3.8 Flash, and Kimi K2.7 Code. The experiments yield nearly 14,000 completed work-session traces. The framework is also evaluated in a larger world containing 26 packages. The main studies investigate communication, ownership and task division, and shared-budget allocation.

The results show that enabling collaboration improves held-out downstream performance across all four model families, although more communication or additional agents do not necessarily improve outcomes. Explicit package ownership generally helps organize work, while shared-budget voting can leave resources unused when agents fail to coordinate effectively. The paper's main contributions are a configurable framework for studying agent collaboration in realistic software ecosystems, an evaluation methodology based on downstream performance, and empirical findings about how organizational choices influence collaborative outcomes.

Critical strengths
  1. The evaluation captures cross-package collaboration benefits rather than merely measuring local task completion.

The paper combines real software dependencies, asynchronous collaboration, and hidden downstream tests to evaluate whether optimizations to one package actually benefit projects that depend on it. Compared with evaluating isolated repository modifications or individual-agent task success, this design better captures the value of information sharing, compatibility maintenance, and coordinated optimization across software boundaries. The case studies in the appendix further illustrate how cross-package communication can reveal issues that local tests miss.

  1. The framework systematically investigates organizational mechanisms and provides inspectable interaction traces.

Beyond comparing collaborative and isolated work, the paper examines proactive communication, explicit ownership, multi-agent task division, and shared-budget voting. The experiments reveal meaningful patterns: communication frequency does not directly determine performance, explicit ownership can reduce duplicated work, and shared-budget mechanisms can fail to use available resources effectively. The associated issues, pull requests, session traces, and evaluation records also improve interpretability and provide reusable infrastructure for future research.

Critical weaknesses
  1. Comparisons of cross-model collaboration behavior are potentially confounded by differences in agent harnesses.

In the main experiments, Claude Opus 5 runs on Claude Code, whereas the other models run on Pi-mono. Different harnesses may affect context management, tool interactions, workflows, and session behavior. Consequently, observed differences in proactive communication, collaboration, and resource management cannot necessarily be attributed entirely to the underlying models. Although Section 6 examines execution-setting sensitivity for GPT-5.6 Sol, this does not establish whether harness differences affect comparisons involving the other models. This issue weakens the interpretation of the paper's cross-model findings about collaborative behavior.

  1. The empirical coverage of the main conclusions is limited, leaving their generalizability across software ecosystems and collaboration structures uncertain.

The principal studies of communication, ownership, and budget allocation center on a single six-package world built around requests, urllib3, and their downstream dependencies. The 26-package experiment primarily demonstrates framework extensibility rather than independently validating all the central findings across broader software ecosystems. Different ecosystems may have substantially different dependency topologies, task distributions, and maintenance requirements, potentially changing the relative benefits of collaboration and task division. The current evidence therefore supports the findings most directly within the studied simulation settings, without fully establishing their applicability to a broader range of open-source software maintenance scenarios.

Questions for authors
  1. Model–harness confounding: Can the authors provide a controlled comparison across models using the same agent harness, tool permissions, context management, and session configuration? In particular, does Claude Opus 5's proactive communication behavior remain different from that of the other models under a unified execution framework? This would affect how confidently the observed differences can be attributed to the models themselves.

  2. Generalizability across software ecosystems: Can the main findings about communication, ownership, and shared-budget allocation be replicated in software worlds with different dependency topologies and maintenance characteristics? In particular, does explicit ownership continue to outperform unassigned task division when dependencies are more complex or optimization opportunities are more unevenly distributed?

3 comments

Thank you for your review! We've answered your question on "generalizability across software ecosystems" in the general responses [1] above. Here we address your other questions point by point.

Model–harness confounding: Can the authors provide a controlled comparison across models using the same agent harness, tool permissions, context management, and session configuration? In particular, does Claude Opus 5's proactive communication behavior remain different from that of the other models under a unified execution framework? This would affect how confidently the observed differences can be attributed to the models themselves.

  1. First, we use the Pi harness whenever possible. Our Opus simulation runs on Claude Code as Anthropic limits Claude Code subscription usage to its official harness. In our submission, we already compare harnesses between codex and pi for GPT-5.6-Sol: It reaches the same final score on Codex and on Pi (1.049 after 600 sessions), while a Pi session costs less than half as much ($1.02 versus \$2.14). This matches one recent work that shows harness choice changes task success little but cost substantially.

  2. We ran an additional experiment to compare the difference between the claude code and pi mono harness using Anthropic APIs. Overall the same conclusion holds: the pi mono harness costs less to achieve the same performance, and the agent behavior is very similar to what we observed using the Claude Code harness: We use the same collaborative simulation setup on the 6-repo network world. We ran the simulation for 162 sessions at $370 of API spend.

  • On Pi, Opus reaches 1.081 speedup after 162 completed sessions. That is close to the Claude Code run at the same number of sessions (1.090–1.108), at less than half the cost per session (about $2.5 against about \$5). At the same spend, the Claude Code run had reached only 1.039–1.048.
  • The communication pattern is also similar: 29% of the Pi run's sessions contain an interaction with another maintainer, against 31% for the Claude Code run at the same number of sessions. In both runs, the requests-gssapi and responses maintainer agents have the most communication records.

Generalizability across software ecosystems: Can the main findings about communication, ownership, and shared-budget allocation be replicated in software worlds with different dependency topologies and maintenance characteristics? In particular, does explicit ownership continue to outperform unassigned task division when dependencies are more complex or optimization opportunities are more unevenly distributed?

First we ran additional experiments on the 26-package world and a new data-science world of six packages: it includes numpy and pandas with different coding setups (e.g., the core libraries include compiled code). These worlds have “different dependency topologies and maintenance characteristics”.

We have not finished communication and ownership ablation experiments in the two other worlds, but we find several examples where agents indeed collaborate in similar ways like what we find in world 1:

  • Agents maintaining downstream libraries report bugs to upstream: In the Gemini simulation, the dask and pandas maintainers reported five bugs in new NumPy and pandas releases.
  • Upstream agents solicit feedback and use cases from the downstream libraries: In the Kimi simulation, the NumPy maintainer asked the dask maintainer which NumPy operations matter most for dask, and improved the code based on the feedback. Afterwards a held-out scikit-image task that calls np.tensordot ran 1.3x faster.

Additionally, the networking world already has “unevenly distributed opportunities”. In our experiments, we create a world by first selecting core libraries (here requests and urllib3) and then including four downstream consumers. So naturally there is more work to do on these two core repos than the others. And we find that in the experiment without code ownership, Opus agents put more of their changes into those two core packages (though they produced fewer changes collectively compared to assigned code ownership).

AuthorsOpenReview ↗

Thank you for your review!

Thank you again for reviewing and the constructive and encouraging feedback. As the discussion period ends soon, please let us know if any questions remain before the discussion closes. If you feel your concerns have been addressed, we would appreciate it if you would consider updating your score. Thank you very much!

I thank the authors for their efforts and additional experiments addressing my concerns. The cross-harness comparison and experiments on additional software worlds provide useful evidence and further strengthen the paper. Overall, I maintain my original positive assessment.

Reviewer_CQPPOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper contributes a benchmark that can test how agents collaborate. IMO, the main contribution of the paper is in the analysis of how agents collaborate to improve Python packages. I can not validate the results 100%, but they are quite as expected.

Critical strengths

It is clear that the authors worked very hard to build this infrastructure. They need to build the sandbox, define the tasks, the tools, the environment, the way the agent communicates, etc. There are many things to define and to check that they are working and explainable, so it is impressive if it works. I like the paper's main idea, especially where they simulated a world when agents are collaborating. It reminds me of the HF incident :) , I am wondering how this will play out if they put agents without their guardrails.

I think that the most interesting part is agent communication patterns: what they are saying, how they are dividing the work, etc. This should be the main point of the paper,

Critical weaknesses

The conclusion of the experiment and the lesson learned are quite disappointing Overall, the novelty of the paper is borderline

Questions for authors
  1. What is the one thing that is most novel, in your opinion, in your paper?
  2. Do you assume that all agents are benign? How do you test that the agents did no harm?
  3. How will it work for smaller open-source models?
  4. What is the most important lesson learned for me if I want to build several agents that will improve a feature in my product?
  5. What was the overall cost of this experiment? How does it scale?
  6. You wrote at the end that researchers can take this and further analyse it, run it with additional models etc., . I am wondering if you can give a deeper analysis and recommendation on best practices
4 comments

Thank you for your review! We’ve answered your questions regarding the novelty of our work, the generalizable insights and lessons, and “overall cost of experiments” in the general responses [2] [3] above. Here we address your other questions point by point.

The conclusion of the experiment and the lesson learned are quite disappointing Overall, the novelty of the paper is borderline What is the most important lesson learned for me if I want to build several agents that will improve a feature in my product? What is the one thing that is most novel, in your opinion, in your paper? You wrote at the end that researchers can take this and further analyse it, run it with additional models etc., . I am wondering if you can give a deeper analysis and recommendation on best practices

We address these points in the general response above (novelty and insights), and summarize them here. The most novel part of our work is the simulation itself - there is a lack of difficult, realistic environments for evaluating frontier multi-agent systems and we fill that gap. Agents pursue a broad goal in a world built from real software libraries, choose their own tasks and whether to collaborate, and are scored on held-out packages they never see. For building several agents that improve one product, our main lesson is to give each agent clear ownership of one part and let the agents communicate. Communication is a main way agents find work, and without clear ownership, adding agents mostly duplicates work. We also find that some frontier models stop looking for work early, so the choice of model matters for such a team. The revised paper adds more case studies (appendix D) providing deeper analysis from the agents' traces.

Do you assume that all agents are benign? How do you test that the agents did no harm?

We do not assume agents are benign. But here are two explanations given our auditing:

  1. In our qualitative auditing we do not find malicious or ill-intended agent behavior. This reflects a latent incentive design in our simulation. Since it's an interconnected world and each party has their own identity and own work allocation. (1) If one agent "lies" about their work (e.g., over claiming their contribution or code impacts), other agents can catch the wrong claims easily by running code on their end. (2) Further, everyone will benefit from others' work in the society (as one improves its own software the dependencies can also get benefited). As a result, sabotaging other's code is not in their interest.
  2. In a narrow sense, "harm" can be defined as making the software worse and reducing efficiency. This is directly captured by our metrics.

That being said, besides our agent auditing, deriving quantitative metrics to measure malicious behavior is an interesting direction for future work.

How will it work for smaller open-source models?

We’ve already included two open-source models, namely, Kimi-2.7-code and the newly added GLM-5.3, in the new experiments. We are running one experiment with the qwen3.8-27B models in world 1.

AuthorsOpenReview ↗

Additional results on open source models

How will it work for smaller open-source models?

Besides the Kimi-2.7-code and GLM-5.3 results, we added results from two more open-weight models, Qwen3.8-27B and Inkling-2-Small. Both improve the libraries less than GLM-5.3 and Kimi-2.7-code so far. Inkling levels off early at about 1.02, and Qwen improves slowly but keeps rising. We serve Qwen3.8-27B ourselves using vLLM on H100 GPUs, and we use the inkling model provided from a public API.

SessionsQwen3.8-27BInkling-2-SmallGLM-5.3Kimi-2.7-code
841.0131.0121.0351.015
1831.0161.0201.0381.047
2611.0221.0201.0381.076
3751.0291.0181.0391.092

We hope the new experiments and clarifications address your concerns. If anything is still unclear, please let us know before the discussion period closes and we will reply promptly. If your concerns are resolved, we would be grateful if you would consider raising your score. Thank you very much!

Thank you for the reply. I do not think that deeper analysis is required. I think that better motivation and results should be the main effort. I appreciate that you added another experiment, but I could not understand if they changed the final takeaway of the paper. IMO, the simulation is not enough in terms of novelty.

AuthorsOpenReview ↗

Additional note on "better motivation and results"

Thank you for engaging with our response!

Regarding "better motivation and results", I think we've improved it in this rebuttal, as summarized in General Response 3.

The Software World simulation is a means of understanding agents’ collaborative behavior when we give them full agency, and the configurable setup has enabled us to measure new capabilities of the agents and discover insights for improving the model training. We appreciate the reviewer acknowledging the effort involved in building the simulation infra, and we also agree that the simulation itself should not be the sole point of novelty (but still, to the best of our knowledge, our simulation setup is still distinct from prior work). If helpful, we want to bring to the reviewer’s attention again the interesting insights that we’ve discovered via our work:

The simulation setup is scalable and measures new capabilities of the agents. As the capabilities of frontier agents grow quickly, we are in a constant need of new benchmarks to agents capabilities. Our framework can systematically assess agents’ capabilities to collaborate for a broad goal instead of prescribed tasks—i.e., all the agents are only told to collaborate and find improvements to the software and they decide what to work on and whether to collaborate. y7vw and HgrB agree that this is different from past work. Further, our framework is scalable: we build the pipeline to systematically select and build new worlds from real-world program libraries. We showcased that it can create worlds of different sizes and structures to test agents differently.

Our insights and takeaways and include:

  • Communication helps agents discover tasks. We find that agent collaboration helps them identify optimizations opportunities and bugs they miss alone. This is reflected in our case studies, but also we can measure it quantitatively. For Gemini and Opus, in collaborative setups they almost never have idle work sessions (only 1%), but in isolated runs, they report about half of their sessions as idle (53% and 48%).

  • Clear ownership helps scale agent teams. In our experiments, we find that without clear code ownership, adding agents duplicates work, and each merged change takes more agent work. This is aligned with Anthropic's recent prompting guide for Claude Fable 5.2: telling the model that it owns the code base raises how often it consolidates duplicated code from under 10% to about 75% of runs. And we believe by releasing this work we can enable finding other similar insights outside the frontier labs.

  • Our simulation helps identify issues in frontier models. For example, gpt-6-astra, a recent flagship model, barely speeds up the held-out packages (1.005, against 1.059 for gpt-5.6-sol in the same world). Its agents decide early that no work remains and make no change at all in 91% of their sessions (28% for gpt-5.6-sol). We also tested different prompts that ask them to explore and aggressively find work: that did not change the outcome, achieving similar results (1.003).

We appreciate all your feedback, and we hope this can make our contribution and novelty clearer. Please let us know if there are any remaining concerns.

Reviewer_y7vwOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper introduces SOFTWAREWORLD, a framework for studying agents that maintain real, dependent Python packages. Agents choose optimization tasks and work through issues, pull requests, reviews, and releases. The framework evaluates changes on hidden downstream packages using correctness tests and instruction-count speedups. Across 31 six-package simulations with four models, the paper studies communication, ownership, and budget sharing. It also reports four runs in a larger 26-package world.

Critical strengths
  1. The environment supports useful studies of agent collaboration. Real package dependencies create a clear need to share findings and coordinate changes, while agents choose their own tasks.
  2. Hidden downstream evaluation is a strong design choice. It tests whether local changes help other packages, and the recorded code changes and discussions help explain agent behavior.
Critical weaknesses
  1. The score can hide correctness failures. When a downstream test gate fails, the evaluator keeps its last passing performance value, so the aggregate score may combine results from different code states and may not represent a working final system.
  2. The communication comparison changes more than communication. Collaborative agents can inspect other repositories, while isolated agents cannot, so the gains cannot be clearly attributed to communication.
  3. The evaluation covers few independent software settings. Most findings come from one six-package world centered on requests and urllib3, with only 17 held-out tasks across four packages.
  4. Instruction counts are a limited measure of software efficiency. They improve measurement stability, but do not directly show wall-clock gains, especially for HTTP packages where I/O and waiting can matter.
Questions for authors
  1. Can you report final correctness pass rates and scores using only the final code state? Do the main conclusions hold without carrying earlier passing values?
  2. Can you separate communication access from repository visibility, using a condition where agents can read other repositories but cannot exchange messages?
  3. Which held-out tasks directly use upstream human-written benchmarks, which are adapted, and which are newly written by agents? Please provide the task-level fidelity audit results.
  4. How many independent simulations support each main comparison? Can you report variation across repeated runs for the key collaboration and ownership results?
  5. How much of the total gain comes from requests and urllib3? Do the collaboration benefits remain across other dependency groups and under package-balanced scoring?
2 comments

Thank you for your review! We’ve answered your question on “variation across repeated runs”, “The evaluation covers few independent software settings”, and “Instruction counts are a limited measure of software efficiency” in the general responses [1] [2] above. Here we address your other questions point by point.

Can you report final correctness pass rates and scores using only the final code state? Do the main conclusions hold without carrying earlier passing values?

Yes, and the main conclusions hold. Among the three worlds tested, we only find the following cases where an agent broke the tests.

RunCarryStrict
Gemini, World 21.1441.101
Gemini, World 31.3051.294
Kimi, World 31.1241.049
(Or see table 1 in the updated paper)

However Gemini remains the highest-scoring model in all three worlds, and the only change in ranking is that Kimi falls just below Sol (1.060) in World 3. We don't find a difference between carry and strict rule for World-1 score.

Can you separate communication access from repository visibility, using a condition where agents can read other repositories but cannot exchange messages?

We are running additional experiments on this and will post the results when they conclude.

Which held-out tasks directly use upstream human-written benchmarks, which are adapted, and which are newly written by agents? Please provide the task-level fidelity audit results.

All the evaluation tasks are based on the raw human written tests, and we only adapt them to a unified format to execute the experiments and track the metrics (instruction counts and wall clock time). Taking cachecontrol as an example, it contains benchmarking code (https://github.com/psf/cachecontrol/blob/master/examples/benchmark.py) , which doesn’t track the instruction count or the results. We convert it to run in a docker container with metrics tracking, so the results can be reproduced. We audit the tasks and report them in table 6 in the appendix.

How much of the total gain comes from requests and urllib3? Do the collaboration benefits remain across other dependency groups and under package-balanced scoring?

In our experiments, we select the held-out packages that only evaluate a world’s core libraries (here requests and urllib3). So the gains come from changes in requests and urlib3. In this case, consumer maintainers act as collaborators that facilitate the work in the core libraries, but their own changes don't contribute directly to the score. Additionally, in this setup no agent knows which packages will be evaluated, and it avoids agents from optimizing for the evaluation.

AuthorsOpenReview ↗

Additional results on separating communication access

Can you separate communication access from repository visibility, using a condition where agents can read other repositories but cannot exchange messages?

We’ve finalized the experiments for the read-only setup in World-1. We configured the system so that each maintainer can read the code and commit history from every other package, but cannot open issues or pull requests on them or message other maintainers. We ran it for about 600 sessions with Gemini and Sol:

ModelCollaborativeRead-onlyIsolated
Gemini1.2511.1961.194
Sol1.0591.0591.053

For Gemini there’s clear gain from communication, where there is none for Sol. For Gemini, we record the performance at different stages during the run: we find that being able to inspect other repo’s code (read-only) made it faster for Gemini to improve the performance compared to the isolated condition, but the two runs plateaued to roughly the same performance in the end. On the other hand, the collaborative setup outperformed both conditions at all steps.

Session CountCollaborativeRead-onlyIsolated
1281.1631.1481.102
2591.2021.1771.163
3711.2121.1871.177
5001.2221.1951.192
6001.2511.1961.194

This experiment suggests that, for models capable of collaboration, they can benefit from different levels of information and communication access to other agents.

As the discussion period is coming to a close, please let us know if these clarifications resolve your concerns or if any further details are needed. If you feel your concerns have been addressed, we hope you will consider raising your score. Thank you very much!

Comments on the paper

5 comments
AuthorsOpenReview ↗

General Response

We thank the reviewers for their thoughtful and constructive feedback. We are encouraged that y7vw finds our evaluation setting "a strong design choice", HgrB also supports our evaluation setup, and CQPP “likes the paper's main idea” and appreciates our effort in implementing the simulation system.

We categorize main critiques from the reviewers into three groups—(1) experiment design and improvements (2) evaluation details and clarification and (3) sharing generalizable insights and findings. Since the submission, we have extended this work in several directions the reviewers ask about. We address the shared points from the reviewers in this general response, and we answer each review in its own section. We have also uploaded a revised PDF with additional results and changes requested by reviewers.

AuthorsOpenReview ↗

[General Response 1: Improving experiment design]

More worlds and models (y7vw, HgrB)

  • For the six-repo world, we added experiments on four other models, including GPT-6-astra, GLM-5.3, Opus-5.5 and Fable-5.1.
  • We added a new data-science world of six packages: it includes numpy and pandas with different coding setups (e.g., the core libraries include compiled code). We also expanded the 26-package world with additional model runs. The figure 1 and 2 in the updated paper shows the topology of the worlds and additional metadata.
  • We carefully analyzed the runs and found new results. For example:
    • We find similar agent collaboration behavior in the new worlds as in World-1: downstream maintainers report bugs to upstream libraries, and upstream maintainers ask downstream ones which operations matter to them.
    • Across all three worlds, held-out speedup keeps growing with compute with gemini-3.8-flash scores highest in every world (Section 3, Fig. 3).
    • We also find that several recent frontier models (gpt-6-astra, claude-opus-5-5, claude-fable-5-1) stop looking for work early, idling in over 90% of their sessions.

Experiment variability (y7vw)

  • Because each simulation is costly (on average, it takes about $1k in API cost and three days of wall-clock time), we measure run-to-run variation in one setting, and we run one simulation per condition in the other experiments
  • We measure run-to-run variation in two scenarios:
    • Within the same harness, we ran Sol three times using the Pi harness in the six-package networking world under the collaborative setup. They end up achieving similar speed-ups at 1.049, 1.059 and 1.063 (standard deviation 0.007), with similar trends in the evaluation results during the simulation (see figure 8 in the updated paper).
    • We compare Sol runs using the Codex and Pi harness: the Codex harness ends at 1.049, close to the Pi results.

Given the variability, we find that collaboration clearly helps Gemini (+0.057) and Opus (+0.024), while Sol (+0.006) and Kimi (−0.003) fall within noise. We have updated the paper including this result.

AuthorsOpenReview ↗

[General Response 2: Evaluation details]

The overall cost of this experiment (CQPP)

In the latest paper, we ran 89 simulations totaling 50k+ agent sessions. On average, a session costs about $1.88, with Claude Opus 5 being the most expensive at $5.86 per session. We find that the Pi harness lowers the cost per session without lowering performance (in the Sol ablation, it costs $1.02 per session on Pi against $2.14 on Codex), so we use it whenever we can. For GPT and Claude models, we use provider's subscription plans to save cost.

Reporting Metrics other than instruction counts (y7vw)

We use instruction count as the primary metric as both recent work (arxiv:2607.01211) and our experiments find that wall clock time varies under different settings and can change experiment conclusions. We validated our instruction count metrics and found it robust and stable across different machines and setups. (As we mentioned in the paper).
We additionally report the wallclock time of results: we run each three times on the same machine and we do not parallelize to make sure results are stable and comparable.

expinstructionwall-clockwall-clock variation
Gemini1.2511.2401.240–1.244
Opus1.1591.1471.143–1.153
Kimi1.1051.1071.096–1.110
Sol1.0591.0551.047–1.061
GLM1.0381.0261.023–1.043
Astra1.0051.0091.005–1.010

Comparing another Astra evaluation run on a different server:

expinstructionwall-clockwall-clock variation
Astra (server 1)1.0051.0091.005–1.010
Astra (server 2)1.0051.0131.010–1.014

The instruction count remains unchanged but the wall clock time metric shows more variability.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.