acceptodds
Under review as a conference paper at ICLR 2027

GitSwarm: Cumulative Inference Over Structured Persistent Memory

Abstract

Long-horizon problem solving and scientific research require computation to accumulate across successive attempts. Partial solutions, experimental findings, and even unsuccessful approaches can inform later work, yet organizing inference-time computation into a reusable body of knowledge remains a challenge. We introduce cumulative inference and instantiate it in GitSwarm, an asynchronous, decentralized system in which homogeneous agents collaborate through structured persistent memory. Agents independently explore, experiment, verify, refine, and synthesize previous work in a shared, branchable Git repository. Atomic commits preserve intermediate artifacts, while explicit provenance records dependencies across branches. We evaluate GitSwarm on long-horizon problem solving and GPU-backed experimental research. On IMOProofBench-Advanced, it solves all 30 problems in one run using GPT-5.5. On ProgramBench, it achieves an 79.4% mean score as compared to 65.1% achieved by the reported baseline. Across three neural architecture research tasks — Residual Matrix Transformer, Looped Transformer, and NanoChat — GitSwarm discovers non-trivial improvements over the respective starting architectures through successive experimentation. Analysis of ProgramBench runs shows that 94.7% of contributions are subsequently built upon, while the selected solution’s ancestry covers 82–93% of the contribution graph. Together, these results demonstrate strong problem-solving performance and extensive accumulation of intermediate work, and extend repository-mediated inference to sustained experimental research.

Primary area:
foundation or frontier models, including LLMs
Keywords:
multi-agent, llms, swarms, long-horizon reasoning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_fNUJOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The authors of this paper propose the concept of cumulative inference and implements it in GitSwarm, an asynchronous system in which homogeneous language-model workers collaborate through a shared, branchable Git repository. Atomic commits preserve intermediate artifacts. Workers can independently explore, verify, and refine. Final readout uses plurality voting. The evaluation covers 30 mathematical proofs on IMOProofBench-Advanced, a 50-task subset of ProgramBench, three GPU-backed architecture-research tasks, controlled synthetic tests, and trace-level analyses of reuse.

Critical strengths

The paper presents a clear and technically concrete systems abstraction for long-horizon agent collaboration. In such a system, persistence, durable branching, and explicit cross-branch provenance are implemented as inspectable repository operations rather than only as prompt-level conventions. The evaluation is broad, combining fixed-answer benchmarks, open-ended GPU research, controlled substrate tests, and detailed trace analyses. The research histories provide useful evidence that workers can recover failed ideas, integrate independently developed changes, and reuse artifacts over time.

Critical weaknesses

There is no matched intervention on the same tasks, model, seeds, token budget, and wall-clock budget comparing GitSwarm with and without the proposed substrate. The synthetic tests are simplified and do not establish causal effects on benchmark performance. Several research comparisons also differ in model, training-token budget, hardware, starting checkpoint, or completion status. In addition, the 94.7% reuse and ancestry statistics largely rely on worker-declared 'uses=' links or selected-lineage structure, which demonstrate reuse but not causal influence.

Questions for authors

See weaknesses above and some additional question as below:

  1. How are 'uses=' declarations validated against actual artifact access and downstream influence, and can the authors estimate the false-positive or purely nominal-reference rate?
  2. Can the authors report per-task ProgramBench results and uncertainty estimates across independent seeds, and explain how the fixed 50-task subset affects the generality of the headline result?
2 comments

We thank the reviewer for the constructive assessment and for recognizing GitSwarm's contributions to persistent, structured collaboration. The review identifies an important limitation: our original analyses demonstrate observable reuse, but do not isolate its causal contribution to final performance. We have now run a new controlled experiment addressing this concern.

1. Matched intervention on real benchmark tasks

We evaluate five difficult ProgramBench tasks (FFmpeg, GROMACS, pandoc, ctags, and PHP-src) with three configurations:

  • GitSwarm (GS5): Five concurrent workers sharing branchable Git history and provenance.
  • Isolated lines (L5): Five concurrent workers with separate persistent Git histories.
  • Latest-state lines (L5-last): Five independent workers accessing only their own latest state.

All configurations use GPT-5.5 at high reasoning effort, the same tools, five concurrent workers, fresh contexts per episode, identical wall-clock budgets, and the same three-selector final vote.

Task-balanced mean hidden-test score (%):

Configuration4 h6 h8 h10 h
GitSwarm14.616.521.423.1
Isolated lines11.714.917.012.4
Latest-state lines10.812.214.415.3

GitSwarm outperforms isolated lines at every tested budget, including a 10.7-point advantage at 10 hours.

Unlike the original single-agent comparison, this intervention controls for parallelism, fresh contexts, and final-answer selection. Consequently, the improvement cannot be attributed to these factors alone.

We emphasize the remaining limitations. These are wall-clock-matched rather than token-matched experiments: GitSwarm uses approximately 10–20% more tokens than isolated lines. Each task-budget cell is also a single run, except for one repeated comparison. We therefore interpret the results as evidence of a shared-memory benefit under matched concurrency and wall-clock budgets, not a complete estimate of token efficiency or an ablation of every component of the substrate.

2. Synthetic controls and research-task comparability

We agree with the reviewer's distinction.

The synthetic experiments in Appendix C establish operational properties of persistence, branching, and attribution under controlled conditions. They do not establish that these properties individually cause benchmark improvements. The new ProgramBench intervention supplies direct evidence concerning real-task performance.

Similarly, the three GPU research studies are not uniformly matched performance comparisons. The RMT, Looped Transformer, and NanoChat studies differ in their experimental controls and resource conditions. We present them as case studies of sustained research, showing how independent developments are integrated, earlier conclusions are revisited, and negative experimental evidence informs later work.

We will make this separation more prominent: the controlled ProgramBench comparison supports the performance claim, while the research histories demonstrate mechanisms and behaviors observed during sustained experimentation.

3. Validating uses= dependencies

We agree that worker-declared dependencies cannot be assumed valid without independent checks.

We therefore audited all 13,830 references to non-parent commits across 40 new GitSwarm runs, comparing them against worker traces, code, and notes.

  • 84.8% were associated with identifiable code transfer, defined as at least three substantive lines present in the referenced commit but not the citing commit's parent. The corresponding rate for randomly selected available uncited commits was 2.5%.
  • 14.6% informed a documented decision without detectable code transfer.
  • Approximately 0.6% were inspected but had no documented code or decision reuse.
  • Only 0.04% (5 of 13,830) were nominal references with no trace-visible inspection or use.

We also examined omissions. Among commits opened but not cited, only 1.2% supplied additional code unavailable through the declared sources. Our analysis estimates that declared sources account for 96.6% of identifiable code reuse.

These results substantially strengthen the interpretation of the original reuse measurements: the dependency records overwhelmingly correspond to trace-visible code or evidence use.

They do not establish that every cited contribution causally improves performance. That distinct, system-level question is addressed by the new shared-memory intervention.

4. Per-task scores, uncertainty, and the 50-task subset

We agree that the original results would benefit from clearer per-task reporting and uncertainty estimates.

The original 50-task benchmark evaluation consists of single-seed runs. We therefore cannot provide a reliable across-seed confidence interval for the entire headline result.

Two existing sources help characterize variability. On nine development tasks, three seeds per configuration yield 62.06 ± 2.44% for GitSwarm (Appendix G.6). In the new controlled experiment, three repetitions of the ctags 10-hour comparison produce 33.7 ± 0.9 for GitSwarm and 7.2 ± 2.6 for isolated lines. We will make the limited scope of these uncertainty estimates explicit.

The original 50-task subset excludes the nine development tasks and was stratified by prior-performance quintiles, test-suite-size quintiles, language, and historical runtime, with deterministic tie-breaking.

Its prior macro score was 62.31%, compared with 62.98% for the full held-out population; its median hidden-test count was 947, compared with 965. These similarities support the representativeness of the sample on the measured characteristics, although they do not establish generalization to every ProgramBench task.

We will correct Appendix B.1's simplified description of the sampling process, provide a clearer per-task breakdown, and explicitly limit the headline inference to the evaluated subset and budgets.

Summary

We appreciate the reviewer's emphasis on separating observed reuse, causal performance effects, and conclusions drawn from heterogeneous research studies.

The new intervention directly strengthens the first performance-related gap by comparing shared and isolated multi-worker computation under matched wall-clock budgets. The provenance audit independently supports the fidelity of our reuse measurements. We will revise the manuscript to add the new experimental results.

We believe these additions substantially strengthen the paper's empirical foundation while clarifying the remaining limits of causal attribution and generality.

Reviewer_tHoYOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The authors of GitSwarm present an agentic framework based on the versioning system git. In their approach, each agent operates on their own git work tree. Each agent is identically initialized and tasked to work autonomously on one of the existing commits. The agent can then perform one of 3 actions: contributing agents can add to the git history, nominating agents can publish a nomination for an existing commit and agents that abstain can terminate a session without publication. Put simply, the idea is to require agents to follow the principle of using a history graph in a revision-controlled project as it has been best practice in all of software engineering. There is no central planner. The authors extend the standard history graph by additionally introducing "semantic-provenance edges" which describe dependencies from any possible other past commit that may not be on the same branch and not the neighboring commit. Further they introduce a harness to handle scheduling and validation. Final output is decided by plurality vote. They apply this framework to two settings. Quantitative research based on choosing and running experiments to improve performance up on a specified setting. The second setting is problem solving where agents are tasked to write proofs or solve coding tasks with verifiable outcomes. Their main contributionis a paradigm shift from agent orchestration to independent agents using a structured history.

Critical strengths

-The idea of using a history graph in software development is well established and has been proven successful over the past decades, applying it to an agent framework may have some reasonable merit (though almost all commercial and academic agent frameworks for software development have been routinley trained/tasked to use git.)

  • The evaluation on ProgramBench with roughly matched compute shows that GitSwarm's process organization can turn extra inference compute into progress compared to a single long running agent. (Unfortunately this is the only such evidence. With two additional caveats, the performance of the single agents may dramatically depend on the way it is prompted and a fair comparison would be to other agentic frameworks or at least multiple unstructured agents)
Critical weaknesses

The two main research questions remain unanswered:

  1. Question: Does the history based versioning memory make the inference "accumulate" and cause the described gains in performance. -> The authors claim the goal is to make inference compute "accumulate" across many independent agents and doing this without using a central planner. This inference compute accumulation has never been made measurable. As the authors admit multiple times for example line 429 following "They establish observed reuse, not the causal contribution of individual dependencies to performance." (Also on line 453)
  2. Question: The advantage of multiple autonomous agents over orchestration has not been established. It remains unclear why the presented paradigm should be superior to established agentic frameworks using different roles. If we regard agentic frameworks as comprising mainly two tasks: deciding the next global step and performing the next local step, it is unclear why an agent with the full history context delegating the second task to agents with full local task context shouldn’t use less input context and a better understanding of local and global context. The authors don't convincingly show why an agent doing both should be superior and do not compare their paradigm to orchestration but only to single running agents.

The paper further has critical scientific and methodological weaknesses:

  • novelty is at most incremental. The main paper leaves out related work entirely. There is very close work for example https://arxiv.org/abs/2506.02049
  • The main paper method section is extremely short and omits core details, for example how the branching works, how results are validated and how the research task is prompted.
  • The appendix is extremely large (almost 50 pages) and clearly (entirely) AI generated as it for example mentions worker IDs used as story characters (w0273, w2985) and absurd precision such as the number 0.5677622963183501. There are broken references in the appendix and cross-references to the main that do not exist.
  • The two main changes to git are not well justified, the plurality readout seems to produce less accurate results than a central selector in more time (ll. 450–452) and the uses= list benefits are not shown on any real task.
Questions for authors
  • the "no roles" claim conflicts with the appendix as you say that for the nanochat runs you use "verifier" and "critic" roles
  • missing controls: this paper misses to credibly show that this paradigm shift in orchestration gives significant performance benefits over matched controls. Every benchmark comparison you do is against one agent that is repeatedly told to continue. Comparing to one single agent cannot separate the effect of shared memory from parallelism, fresh contexts and vote-based selection. Autoscientist (https://arxiv.org/abs/2605.28655), Karpathys autoresearch etc.
  • as I understand every agent has to search the full history to find a starting point to increment on, what evidence do you have that this uses less resources or has better performance than a simple search over a log book. Especially if the agents can use any results of past commits without restrictions.
8 comments

We thank the reviewer for the detailed assessment. We agree that the original submission did not sufficiently isolate the performance contribution of shared memory from parallelism, fresh contexts, and final selection. We have now conducted a new controlled experiment addressing this central concern. We also clarify the scope of our claims concerning decentralized orchestration, novelty, and provenance.

1. Does shared history actually cause inference-time computation to accumulate?

The reviewer correctly identifies a limitation acknowledged in our original submission (Section 4): observed dependencies and artifact reuse do not establish that access to prior work improves downstream performance.

We have now run the corresponding intervention on five hard ProgramBench tasks: FFmpeg, GROMACS, pandoc, ctags, and PHP-src.

All configurations use the same GPT-5.5 model (high reasoning effort), tools, five concurrent workers, fresh contexts per worker episode, wall-clock budgets, and (except for GitSwarm's own plurality readout) an independent three-selector final vote.

We compare GitSwarm's shared repository against five isolated persistent Git histories (L5) and five independent latest-state-only histories (L5-last).

Configuration4 h6 h8 h10 h
GitSwarm16.217.221.722.4
GitSwarm (Selector)14.616.521.423.1
Isolated lines11.714.917.012.4
Latest-state lines10.812.214.415.3

In GitSwarm (selector), we replace the plurality readout in GitSwarm with a plurality vote among three independent central selection agents. Isolated lines and Latest-state lines use the same selection mechanism.

GitSwarm outperforms isolated lines at all four budgets. At 10 hours, the difference is 22.4% and 23.1% versus 12.4%, or 10.7 percentage points.

To estimate stochastic variation, we repeated the ctags 10-hour comparison three times. GitSwarm obtained 33.7 ± 0.9, compared with 7.2 ± 2.6 for isolated lines.

This addresses the main gap in the original evaluation. The original 94.7% reuse statistic demonstrated that contributions were subsequently referenced; the new experiment establishes that access to shared work can also improve final benchmark performance.

2. Why decentralized workers rather than centralized orchestration?

We agree that the paper does not establish that decentralized workers outperform centralized orchestration in general. We do not intend to make that claim.

Our contribution is a concrete instantiation and evaluation of cumulative inference: independently scheduled episodes preserve and reuse a persistent body of work without a central intellectual planner. The absence of such a planner describes GitSwarm's design choices, not an experimentally demonstrated superiority over planners.

This design has several useful properties, as also pointed out in the work the reviewer cites, EvoGit (Huang et al., 2025), whose decentralized architecture "enables fully asynchronous agent operation without centralized scheduling or inter-agent communication."

First, no single agent must maintain the complete research history in its active context. The repository stores the history, and workers selectively retrieve relevant contributions. ProgramBench workers typically inspect 5–7 candidates even as the available graph grows substantially (Appendix G.5).

Second, independent workers can challenge earlier decisions rather than inherit a single coordinator's judgment. The Looped Transformer experiment illustrates this: a candidate rejected after a small evaluation was recovered through independent re-evaluation.

Third, the system can continue publishing and integrating independent work without requiring an intellectual decision from a designated planner whenever a worker becomes available. The execution harness still centrally manages scheduling and validates publication requirements.

There are no roles, plans or hand-offs to engineer. Every worker has the same actions (contribute, nominate, or abstain on the benchmarks; contribute or abstain in the research tasks), which also gives a uniform interface for future training.

These are architectural properties and motivations, not evidence that decentralized orchestration is universally better or cheaper. Direct comparison with a strong centralized planner remains valuable future work.

The new controlled baselines instead answer a more focused question: when worker concurrency, fresh contexts, and selection are held fixed, does shared access to previous work improve outcomes? Our results support that claim.

3. Novelty and relationship to EvoGit

We thank the reviewer for identifying EvoGit (Huang et al., 2025). We agree that Git-based decentralized agent collaboration predates our work and should not be presented as our invention. We have referenced EvoGit in the main-paper and added a discussion in the related works section in the Appendix. Our existing related-work analysis in Appendix A also discusses other repository-based systems, including Agora.

GitSwarm differs from EvoGit in its contribution semantics and experimental objectives.

Contribution protocol. EvoGit primarily evolves code through small mutations and crossover, accepting modifications that improve upon their parents. GitSwarm workers instead contribute bounded pieces of work that may represent implementations, partial solutions, experiments, evaluations, refutations, or integrations. Negative results are preserved rather than necessarily discarded.

Semantic provenance. GitSwarm separates the physical Git parent of a contribution from its additional uses= dependencies. These record code and evidence drawn from other branches, including findings that inform a decision without appearing as inherited code. This distinction matters for analyses of research synthesis and negative-result reuse.

Autonomous readout and evaluation. EvoGit includes periodic human guidance for branch promotion and pruning. GitSwarm's benchmark protocol instead uses autonomous nomination and a deterministic plurality readout.

Broader experimental setup. EvoGit demonstrates feasibility through two case studies (a web application and a generator of bin-packing solvers). We evaluate GitSwarm quantitatively against matched baselines on ProgramBench (50 tasks) and proof benchmarks, report multi-day GPU research runs on three research problems, and in this rebuttal add a controlled comparison of the memory structure at matched compute.

We do not claim that every component is individually novel. Our contribution is the combined compounding-inference protocol, its explicit treatment of cross-branch evidence dependencies, and evidence of how shared work affects final-task performance.

4. Missing implementation details

We agree that several details currently require consulting the appendix. We will bring the essential protocol into the main paper.

Each worker starts in a private Git worktree, inspects the shared history and in-flight activity, and independently selects a starting commit. It can create an evaluated contribution, nominate an existing candidate, or abstain. A contribution records both its Git parent and semantic sources through uses=. Workers document their assessments in EVALUATION.md; the harness checks publication constraints and reference integrity but does not certify semantic correctness. Research tasks use their specified validation metrics for candidate selection.

We will summarize these mechanics more explicitly in Section 2 and direct readers to the complete task prompts and validation requirements in Appendix I.

5. Appendix quality and the AI-usage statement

We acknowledge the presentation issues, including overly precise numerical values and broken cross-references. We have addressed this in the updated draft

The worker identifiers cited in the review are identifiers from actual execution traces, included to make research trajectories auditable. They are not fictional characters. Nevertheless, the technical argument should not depend on lengthy worker-by-worker narratives, and we will simplify the presentation while retaining reproducibility information.

The appendix is extensive because it includes experimental protocols, research histories, measurements, and analyses omitted from the main paper for space. We will improve its organization, repair broken references, and amend the AI-usage disclosure to ensure that it accurately describes the tools used in preparing the manuscript. We take responsibility for the correctness of all submitted content.

6. Readout and provenance: what benefit is established?

We distinguish the design rationale from the empirical claims.

Readout. We do not claim that plurality readout is more accurate than centralized selection. In the new controlled experiments, evaluating plurality readout and an independent three-selector vote on the same GitSwarm graphs produces final scores within 0.6 points, with a mean difference of under 0.1 points across the 20 paired comparisons (shown below). The original development-set comparison slightly favored the central selector, but the difference was not statistically significant (Appendix G.6).

Configuration4 h6 h8 h10 h
GitSwarm16.217.221.722.4
GitSwarm (Selector)14.616.521.423.1

We use plurality readout because it permits final selection using the same homogeneous worker interface without introducing a privileged intellectual decision-maker. Its accuracy advantage is not established.

Provenance. We distinguish the benefit of cross-worker access from the additional benefit of explicitly recording individual dependency edges.

The new experiment establishes a benefit from access to shared work, but does not separately isolate the effect of the uses= field. We nevertheless have evidence that the field records substantive dependencies.

We audited 13,830 non-parent uses= provenance references across 40 new GitSwarm runs. Of these,

  • 84.8% were associated with identifiable code transfer, compared with 2.5% for randomly selected available uncited commits;
  • 14.6% informed documented decisions without detectable code transfer;
  • approximately 0.6% were inspected without documented reuse;
  • and only 0.04% were nominal references without trace-visible access or use.

Further, 10.4% of workers in these runs followed a commit's declared provenance to inspect a previously unopened source. Thus, provenance is not merely descriptive: workers also use it to navigate earlier work.

These findings support provenance-fidelity and practical use.

7. Full-history inspection and comparison with a logbook

Workers are not required to inspect the full history. They begin from a compact commit overview and selectively open candidate artifacts. Appendix G.5 shows approximately 5–7 candidate inspections per ProgramBench worker, even late in runs with many more available candidates.

A conventional logbook is a reasonable alternative. GitSwarm includes a log-like overview, but its entries are linked to immutable code, evaluations, diffs, and semantic sources that can be inspected independently.

We do not claim that this representation has been shown to outperform every possible logbook implementation at equal cost. The new comparison instead establishes that access to shared, structured prior work is useful relative to isolated workers. Identifying which representation of shared work is most effective requires further controlled ablations.

Summary

The original submission established the feasibility of compounding inference through extensive trace analyses and multi-day research histories, but did not fully isolate its performance contribution. The new controlled experiment closes an important part of that gap: across five hard ProgramBench tasks and multiple wall-clock budgets, shared-work GitSwarm outperforms parallel workers with isolated histories under the same concurrency and selection protocol.

We will revise the manuscript to add the new experimental results and make the above distinction central and narrow claims not established by the experiments, and provide details on how the research tasks are prompted

As of now, we have updated the manuscript to clarify prior work, add details to the methods section, and improve the presentation issues identified by the reviewer. The changes are highlighted using a cyan colored text.

Reviewer_pWWCOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces the concept of "cumulative reasoning" and implements a framework within GitSwarm, where homogeneous agents collaborate asynchronously and in a decentralized manner using shared, branchable Git repositories. Intermediate artifacts are preserved through atomic commits, while cross-branch dependencies are tracked via explicit provenance. The system is evaluated on long-horizon problem-solving tasks and GPU-accelerated experimental research; it demonstrates strong performance in mathematical proof generation and program reconstruction, and has yielded substantial improvements to initial architectures across three distinct neural architecture research tasks.

Critical strengths

By framing "cumulative reasoning" as a distinct problem, the work emphasizes that partial solutions, experimental findings, and even failed paths should be treated as reusable knowledge, directly addressing a critical bottleneck in long-horizon collaborative agent systems.

The use of persistent, branchable Git repositories as a shared memory, leveraging atomic commits for intermediate artifacts and explicit provenance for cross-branch dependencies, enables an asynchronous, decentralized workflow that eliminates the need for a central planner, resulting in a clear and highly extensible architecture.

Critical weaknesses

The repository is piled high with intermediate results, yet the system fails to truly process them, identifying which are correct, which are erroneous, which contradict one another, or which should be consolidated. If subsequent users cite flawed entries, the errors may propagate and compound.

Contributors might make omissions, introduce mistakes, or simply cite existing work out of convenience. Using such a collection as evidence of "cumulative progress" lacks a solid foundation. In mathematical proofs, for instance, many earlier efforts are examined or compared, yet they do not necessarily become part of the final, definitive proof. This highlights that "preservation and reuse" are distinct from "achieving a superior final outcome."

While extensive prior computation can be valuable, the subsequent burden, parsing the history, verifying the validity of results, and preventing redundant work, demands significant effort. Academic papers often oversimplify the notion that "more computation is always better," but in reality, this is not consistently cost-effective.

Questions for authors

Please refer to the weaknesses.

3 comments

We thank the reviewer for recognizing the importance of preserving intermediate work in long-horizon reasoning and for raising three important questions: how incorrect results are handled, whether reuse translates into better final solutions, and whether the benefits justify the overhead.

1. How does GitSwarm prevent incorrect or contradictory results from propagating?

GitSwarm does not assume that committed results are correct. The harness validates publication requirements and provenance references, but not semantic correctness. Instead, each contributing worker records its evaluation, checks, and known limitations in EVALUATION.md. Subsequent workers may independently verify, repair, extend, challenge, or disregard previous contributions.

This distinction is intentional: for open-ended scientific research, the correctness or value of a contribution may not be established when it is produced. Rather than deleting uncertain or negative results, GitSwarm preserves them so later workers can reconsider the evidence.

Our research traces demonstrate several concrete instances of this behavior:

  • Residual Matrix Transformer (Figure 3(b), Appendix E.2): An unsuccessful single-token memory mechanism was revisited approximately 49.5 hours later. Another worker generalized the idea to multiple previous tokens, leading to the temporal memory mixer incorporated into the final architecture.
  • Looped Transformer (Figure 4(b), Appendix E.3): The eventual winning adapter was initially rejected following an evaluation on only 128 development examples. Another worker re-evaluated the same checkpoint on 512 examples, and a third confirmed the improvement on the full development set. Workers subsequently identified that 128-example evaluations misranked 5 of 12 comparable model pairs and adopted a more reliable multi-stage evaluation procedure.
  • Reuse of negative evidence: Negative architectural results were later reused as controls.

These are observational statistics based on worker-assigned labels, not a guarantee that erroneous results cannot propagate. They nevertheless show that preserving a result does not imply accepting it as correct or continuing from it. GitSwarm enables subsequent workers to revisit and challenge earlier conclusions.

2. Does preservation and reuse actually improve the final outcome?

We agree that preservation, reuse, and improved final performance are distinct claims. Our original trace analyses established extensive reuse but did not causally isolate its effect on performance. We have now conducted a controlled experiment addressing this concern.

We compare three configurations on five hard ProgramBench tasks (FFmpeg, GROMACS, pandoc, ctags, and PHP-src):

  • GitSwarm: Five workers with access to shared, branchable Git history and provenance.
  • GitSwarm (Selector) : Five workers with access to shared, branchable Git history and provenance. The plurality voting based selection is replaced by a central selection mechanism.
  • Isolated lines : Five workers, each maintaining a private persistent Git history without access to other workers' work.
  • Latest-state lines (L5-last): Five independent workers, each seeing only the latest state of its own line.

In GitSwarm (selector), we replace the plurality readout in GitSwarm with a plurality vote among three independent central selection agents. Isolated lines and Latest-state lines use the same selection mechanism.

All configurations use GPT-5.5 with high reasoning effort, the same tools, five concurrent workers, fresh contexts for each episode, identical wall-clock budgets, and, except for GitSwarm's own plurality readout, the same three-selector final vote.

Mean hidden-test score (%):

Configuration4 h6 h8 h10 h
GitSwarm16.217.221.722.4
GitSwarm (selector)14.616.521.423.1
Isolated lines11.714.917.012.4
Latest-state lines10.812.214.415.3

GitSwarm outperforms isolated lines at every tested budget. These comparisons hold concurrency, context resets, and final selection fixed, showing that providing workers access to one another's prior work improves final-task performance beyond independent parallel trajectories.

3. Do contributors cite existing work nominally?

On the concern that contributors may cite existing work out of convenience, we audited all uses= references across all runs included in the experiments presented in the previous section. We compare the citations against the citing worker's tool trace, code and notes:

  • 84.8% transfer code from the cited commit (at least three substantive lines present in the source but not in the parent; 2.5% for a random uncited commit available at the same time);
  • 14.6% inform a documented decision (no code transferred, but the source is named in the citing worker's notes, e.g. as a control or a competing result);
  • 0.6% were inspected only;
  • 0.04% are nominal: neither inspected, reused nor mentioned;

Conversely, citations rarely omit real dependencies. Of the commits a worker opened but did not cite, only 1.2% supplied code that is not also available from its cited sources, so the declared sources account for 96.6% of identifiable code reuse. These observations show that contributors do not cite uses= references nominally or out of convenience.

4. Does navigating the growing history create excessive overhead?

The reviewer correctly identifies retrieval and coordination costs as important limitations. GitSwarm does not require workers to read the entire repository history.

As documented in Appendix G.5, ProgramBench workers typically open only 5–7 candidate commits, and proof workers approximately three. Late in a 100-worker ProgramBench run, workers inspect approximately six candidates despite having around 72 available.

GitSwarm also has a mechanism in place to address potential redundant work. Workers inspect in-flight activity and may abstain when equivalent work is already underway. In our trace analysis, 94.9% of 1,421 ProgramBench abstentions were supported by subsequently materialized work. This reduces some redundant work, although abstentions and repository inspection still incur inference costs.

Thus, our claim is not that additional computation is always useful or free. We present a way of organising additional computation which provides performance benefits over existing ways of organising additional computation. Structured persistent memory can make previously completed work available for subsequent improvement. The new controlled experiment demonstrates a benefit in final performance, while our overhead analyses characterize some of the corresponding costs.

We appreciate the reviewer's distinction between preserving work and obtaining better solutions. The revised evidence directly addresses that distinction and supports GitSwarm's central claim: sharing and reusing work across independent inference episodes can improve long-horizon problem solving. We will revise the manuscript to incorporate these new experimental results

AuthorsOpenReview ↗

Further feedback.

Thank you again for your helpful feedback. Your comments have helped us improve the paper.

Since the discussion period is coming to an end, we wanted to follow up and see if our responses have addressed your concerns.

If there is anything we may have missed, we would really appreciate the opportunity to address it before the discussion closes.

Thanks for your help and time.

Comments on the paper

2 comments

We thank all three reviewers for their thoughtful feedback and constructive suggestions. We appreciate their recognition of the importance of cumulative inference and the questions raised regarding its empirical validation, novelty, and practical effectiveness.

A central concern shared across the reviews is whether GitSwarm's performance gains arise from the accumulation and reuse of prior work, rather than simply from parallelism, fresh contexts, or final-answer selection. We agree that our original evaluation did not fully disentangle these factors. To address this, we have conducted substantial new experiments and analyses:

1. New controlled experiment isolating shared-memory benefits. We compare GitSwarm against five independent workers with persistent private histories (L5) and five independent workers retaining only their latest states (L5-last). Across five hard ProgramBench tasks, all configurations use the same model, tools, concurrency, fresh-context protocol, wall-clock budgets, and final selection mechanism. GitSwarm outperforms isolated workers at every evaluated budget (4–10 hours), with gains of 1.7–10.7 percentage points. These experiments provide direct evidence that access to other workers' prior work improves final-task performance beyond independent parallel execution. We also report the remaining limitations, including differences in token consumption and limited repeated-seed evaluations.

2. Large-scale validation of actual knowledge reuse. We audited 13,830 cross-commit provenance references against execution traces, code, and notes. We find that 84.8% correspond to identifiable code transfer, while most remaining references inform documented decisions. Only 0.04% are nominal references without trace-visible use. This substantially strengthens the evidence that provenance reflects meaningful reuse rather than merely recorded citations.

3. Clarifying the scope of our contributions. We distinguish the observed performance benefits of shared memory from the separate questions of whether explicit provenance edges independently improve performance and whether decentralized coordination outperforms centralized orchestration. We do not claim to have established either of the latter. We will also strengthen the comparison with related work (including EvoGit), clarify implementation details and experimental configurations, and correct presentation inconsistencies.

We believe these additions substantially strengthen the paper's central contribution: independent inference episodes can build upon shared, persistent work to achieve better final outcomes than isolated trajectories under comparable wall-clock budgets.

We thank the reviewers again for motivating these additional experiments and clarifications. We provide detailed responses and supporting evidence in the individual reviewer threads.

As the discussion period comes to an end, we would like to thank all the reviewers for useful suggestions which helped us improve the work. We have updated the submission with the additional experiments run during the rebuttal period and the other suggested modifications. We summarize the primary changes made to the paper below (highlighted in blue in the revised PDF):

Main paper

  • Positioning (Section 2). We relate GitSwarm to the close prior systems, CORAL and EvoGit, and point readers to the related-work discussion in Appendix A.
  • Worker protocol (Section 2.3). We describe how each worker starts in a fresh worktree with read-only access to all earlier commits, branches from a single chosen commit, records other influential commits in its uses= list, and combines work from several branches without merging.
  • Publication and validation (Section 2.3). A new paragraph specifies the mandatory artifacts in every contribution (MY_THOUGHTS.md and EVALUATION.md) and the checks the harness applies before publishing a contribution or a nomination.
  • Controlled comparison (Sections 3.1 and 3.3, Figure 3). A new section compares GitSwarm with GitSwarm-central (three-selector vote), Isolated lines, Latest-state lines and Flat memory, at four wall-clock budgets (4–10 h) on the five difficult ProgramBench tasks, selected following Agensh. All arms use the same model, concurrency and budgets.
    • Both GitSwarm variants outperform the independent-worker baselines at every budget, by 1.7 to 10.7 points.
    • Plurality and central readouts perform similarly.
    • A simpler shared store (Flat memory) captures much of the performance benefit of sharing at these budgets; at these budgets, the main benefit of version control is to keep shared work organized and auditable.
  • Provenance audit (Section 3.5). We summarize a new audit of uses= declarations: 84.8% of references to commits other than the Git parent carry code from the cited commit, against 2.5% for randomly chosen uncited commits.
  • Pointer to coordination analyses (Section 3.5). We point readers to Appendix G for worker activity types, coordination mechanisms and design tradeoffs.

Appendix

  • Related work (Appendix A). New discussions of EvoGit, CORAL and Agensh, and how GitSwarm differs from each.
  • Controlled-comparison details (Appendix D). Per-task scores for every method and budget, run-to-run variation from three repeated runs of one task and budget (D.2), and the full uses= audit (D.3).
  • Research-task prompts (Appendix I). The system-prompt addenda for all three research tasks (RMT, Looped Transformer and NanoChat), and a description of their user prompts.
  • Correction (Appendix E.4). We removed an incorrect sentence describing worker roles in the NanoChat run; it referred to an earlier version of the system.

We hope these changes address the reviewers' concerns. Please let us know if you have any additional questions or concerns. If your main concerns have been addressed, we would be grateful if you would consider updating your evaluation.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.