- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper investigates how LLM agents should select tasks, coordinate code changes, exchange information, and allocate shared resources when maintaining interdependent software packages over long-running, open-ended tasks.
The authors introduce SOFTWARE WORLD, a framework consisting of three components:
- SW-ENV: A collaborative environment built around real Python packages, where agents autonomously select optimization tasks and interact asynchronously through issues, pull requests, code reviews, and releases.
- SW-BENCH: A world-construction and evaluation methodology that uses real software dependencies and human-authored tests and performance benchmarks to measure improvements on downstream packages that agents cannot access or modify.
- SW-HARBOR: An execution infrastructure supporting parallel agent sessions, persistent memory, configurable budgets, and comprehensive experiment logging. The authors conduct 31 simulations in six-package worlds, evaluating four model families: GPT-5.6 Sol, Claude Opus 5, Gemini 3.8 Flash, and Kimi K2.7 Code. The experiments yield nearly 14,000 completed work-session traces. The framework is also evaluated in a larger world containing 26 packages. The main studies investigate communication, ownership and task division, and shared-budget allocation.
The results show that enabling collaboration improves held-out downstream performance across all four model families, although more communication or additional agents do not necessarily improve outcomes. Explicit package ownership generally helps organize work, while shared-budget voting can leave resources unused when agents fail to coordinate effectively. The paper's main contributions are a configurable framework for studying agent collaboration in realistic software ecosystems, an evaluation methodology based on downstream performance, and empirical findings about how organizational choices influence collaborative outcomes.
Critical strengths
- The evaluation captures cross-package collaboration benefits rather than merely measuring local task completion.
The paper combines real software dependencies, asynchronous collaboration, and hidden downstream tests to evaluate whether optimizations to one package actually benefit projects that depend on it. Compared with evaluating isolated repository modifications or individual-agent task success, this design better captures the value of information sharing, compatibility maintenance, and coordinated optimization across software boundaries. The case studies in the appendix further illustrate how cross-package communication can reveal issues that local tests miss.
- The framework systematically investigates organizational mechanisms and provides inspectable interaction traces.
Beyond comparing collaborative and isolated work, the paper examines proactive communication, explicit ownership, multi-agent task division, and shared-budget voting. The experiments reveal meaningful patterns: communication frequency does not directly determine performance, explicit ownership can reduce duplicated work, and shared-budget mechanisms can fail to use available resources effectively. The associated issues, pull requests, session traces, and evaluation records also improve interpretability and provide reusable infrastructure for future research.
Critical weaknesses
- Comparisons of cross-model collaboration behavior are potentially confounded by differences in agent harnesses.
In the main experiments, Claude Opus 5 runs on Claude Code, whereas the other models run on Pi-mono. Different harnesses may affect context management, tool interactions, workflows, and session behavior. Consequently, observed differences in proactive communication, collaboration, and resource management cannot necessarily be attributed entirely to the underlying models. Although Section 6 examines execution-setting sensitivity for GPT-5.6 Sol, this does not establish whether harness differences affect comparisons involving the other models. This issue weakens the interpretation of the paper's cross-model findings about collaborative behavior.
- The empirical coverage of the main conclusions is limited, leaving their generalizability across software ecosystems and collaboration structures uncertain.
The principal studies of communication, ownership, and budget allocation center on a single six-package world built around requests, urllib3, and their downstream dependencies. The 26-package experiment primarily demonstrates framework extensibility rather than independently validating all the central findings across broader software ecosystems. Different ecosystems may have substantially different dependency topologies, task distributions, and maintenance requirements, potentially changing the relative benefits of collaboration and task division. The current evidence therefore supports the findings most directly within the studied simulation settings, without fully establishing their applicability to a broader range of open-source software maintenance scenarios.
Questions for authors
-
Model–harness confounding: Can the authors provide a controlled comparison across models using the same agent harness, tool permissions, context management, and session configuration? In particular, does Claude Opus 5's proactive communication behavior remain different from that of the other models under a unified execution framework? This would affect how confidently the observed differences can be attributed to the models themselves.
-
Generalizability across software ecosystems: Can the main findings about communication, ownership, and shared-budget allocation be replicated in software worlds with different dependency topologies and maintenance characteristics? In particular, does explicit ownership continue to outperform unassigned task division when dependencies are more complex or optimization opportunities are more unevenly distributed?
3 comments
Thank you for your review!
Thank you again for reviewing and the constructive and encouraging feedback. As the discussion period ends soon, please let us know if any questions remain before the discussion closes. If you feel your concerns have been addressed, we would appreciate it if you would consider updating your score. Thank you very much!
I thank the authors for their efforts and additional experiments addressing my concerns. The cross-harness comparison and experiments on additional software worlds provide useful evidence and further strengthen the paper. Overall, I maintain my original positive assessment.
Thank you for your review! We've answered your question on "generalizability across software ecosystems" in the general responses [1] above. Here we address your other questions point by point.
First, we use the Pi harness whenever possible. Our Opus simulation runs on Claude Code as Anthropic limits Claude Code subscription usage to its official harness. In our submission, we already compare harnesses between codex and pi for GPT-5.6-Sol: It reaches the same final score on Codex and on Pi (1.049 after 600 sessions), while a Pi session costs less than half as much ($1.02 versus \$2.14). This matches one recent work that shows harness choice changes task success little but cost substantially.
We ran an additional experiment to compare the difference between the claude code and pi mono harness using Anthropic APIs. Overall the same conclusion holds: the pi mono harness costs less to achieve the same performance, and the agent behavior is very similar to what we observed using the Claude Code harness: We use the same collaborative simulation setup on the 6-repo network world. We ran the simulation for 162 sessions at $370 of API spend.
First we ran additional experiments on the 26-package world and a new data-science world of six packages: it includes numpy and pandas with different coding setups (e.g., the core libraries include compiled code). These worlds have “different dependency topologies and maintenance characteristics”.
We have not finished communication and ownership ablation experiments in the two other worlds, but we find several examples where agents indeed collaborate in similar ways like what we find in world 1:
np.tensordotran 1.3x faster.Additionally, the networking world already has “unevenly distributed opportunities”. In our experiments, we create a world by first selecting core libraries (here requests and urllib3) and then including four downstream consumers. So naturally there is more work to do on these two core repos than the others. And we find that in the experiment without code ownership, Opus agents put more of their changes into those two core packages (though they produced fewer changes collectively compared to assigned code ownership).