- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The authors of this paper propose the concept of cumulative inference and implements it in GitSwarm, an asynchronous system in which homogeneous language-model workers collaborate through a shared, branchable Git repository. Atomic commits preserve intermediate artifacts. Workers can independently explore, verify, and refine. Final readout uses plurality voting. The evaluation covers 30 mathematical proofs on IMOProofBench-Advanced, a 50-task subset of ProgramBench, three GPU-backed architecture-research tasks, controlled synthetic tests, and trace-level analyses of reuse.
Critical strengths
The paper presents a clear and technically concrete systems abstraction for long-horizon agent collaboration. In such a system, persistence, durable branching, and explicit cross-branch provenance are implemented as inspectable repository operations rather than only as prompt-level conventions. The evaluation is broad, combining fixed-answer benchmarks, open-ended GPU research, controlled substrate tests, and detailed trace analyses. The research histories provide useful evidence that workers can recover failed ideas, integrate independently developed changes, and reuse artifacts over time.
Critical weaknesses
There is no matched intervention on the same tasks, model, seeds, token budget, and wall-clock budget comparing GitSwarm with and without the proposed substrate. The synthetic tests are simplified and do not establish causal effects on benchmark performance. Several research comparisons also differ in model, training-token budget, hardware, starting checkpoint, or completion status. In addition, the 94.7% reuse and ancestry statistics largely rely on worker-declared 'uses=' links or selected-lineage structure, which demonstrate reuse but not causal influence.
Questions for authors
See weaknesses above and some additional question as below:
- How are 'uses=' declarations validated against actual artifact access and downstream influence, and can the authors estimate the false-positive or purely nominal-reference rate?
- Can the authors report per-task ProgramBench results and uncertainty estimates across independent seeds, and explain how the fixed 50-task subset affects the generality of the headline result?
2 comments
4. Per-task scores, uncertainty, and the 50-task subset
We agree that the original results would benefit from clearer per-task reporting and uncertainty estimates.
The original 50-task benchmark evaluation consists of single-seed runs. We therefore cannot provide a reliable across-seed confidence interval for the entire headline result.
Two existing sources help characterize variability. On nine development tasks, three seeds per configuration yield 62.06 ± 2.44% for GitSwarm (Appendix G.6). In the new controlled experiment, three repetitions of the ctags 10-hour comparison produce 33.7 ± 0.9 for GitSwarm and 7.2 ± 2.6 for isolated lines. We will make the limited scope of these uncertainty estimates explicit.
The original 50-task subset excludes the nine development tasks and was stratified by prior-performance quintiles, test-suite-size quintiles, language, and historical runtime, with deterministic tie-breaking.
Its prior macro score was 62.31%, compared with 62.98% for the full held-out population; its median hidden-test count was 947, compared with 965. These similarities support the representativeness of the sample on the measured characteristics, although they do not establish generalization to every ProgramBench task.
We will correct Appendix B.1's simplified description of the sampling process, provide a clearer per-task breakdown, and explicitly limit the headline inference to the evaluated subset and budgets.
Summary
We appreciate the reviewer's emphasis on separating observed reuse, causal performance effects, and conclusions drawn from heterogeneous research studies.
The new intervention directly strengthens the first performance-related gap by comparing shared and isolated multi-worker computation under matched wall-clock budgets. The provenance audit independently supports the fidelity of our reuse measurements. We will revise the manuscript to add the new experimental results.
We believe these additions substantially strengthen the paper's empirical foundation while clarifying the remaining limits of causal attribution and generality.
We thank the reviewer for the constructive assessment and for recognizing GitSwarm's contributions to persistent, structured collaboration. The review identifies an important limitation: our original analyses demonstrate observable reuse, but do not isolate its causal contribution to final performance. We have now run a new controlled experiment addressing this concern.
1. Matched intervention on real benchmark tasks
We evaluate five difficult ProgramBench tasks (FFmpeg, GROMACS, pandoc, ctags, and PHP-src) with three configurations:
All configurations use GPT-5.5 at high reasoning effort, the same tools, five concurrent workers, fresh contexts per episode, identical wall-clock budgets, and the same three-selector final vote.
Task-balanced mean hidden-test score (%):
GitSwarm outperforms isolated lines at every tested budget, including a 10.7-point advantage at 10 hours.
Unlike the original single-agent comparison, this intervention controls for parallelism, fresh contexts, and final-answer selection. Consequently, the improvement cannot be attributed to these factors alone.
We emphasize the remaining limitations. These are wall-clock-matched rather than token-matched experiments: GitSwarm uses approximately 10–20% more tokens than isolated lines. Each task-budget cell is also a single run, except for one repeated comparison. We therefore interpret the results as evidence of a shared-memory benefit under matched concurrency and wall-clock budgets, not a complete estimate of token efficiency or an ablation of every component of the substrate.
2. Synthetic controls and research-task comparability
We agree with the reviewer's distinction.
The synthetic experiments in Appendix C establish operational properties of persistence, branching, and attribution under controlled conditions. They do not establish that these properties individually cause benchmark improvements. The new ProgramBench intervention supplies direct evidence concerning real-task performance.
Similarly, the three GPU research studies are not uniformly matched performance comparisons. The RMT, Looped Transformer, and NanoChat studies differ in their experimental controls and resource conditions. We present them as case studies of sustained research, showing how independent developments are integrated, earlier conclusions are revisited, and negative experimental evidence informs later work.
We will make this separation more prominent: the controlled ProgramBench comparison supports the performance claim, while the research histories demonstrate mechanisms and behaviors observed during sustained experimentation.
3. Validating uses= dependencies
We agree that worker-declared dependencies cannot be assumed valid without independent checks.
We therefore audited all 13,830 references to non-parent commits across 40 new GitSwarm runs, comparing them against worker traces, code, and notes.
We also examined omissions. Among commits opened but not cited, only 1.2% supplied additional code unavailable through the declared sources. Our analysis estimates that declared sources account for 96.6% of identifiable code reuse.
These results substantially strengthen the interpretation of the original reuse measurements: the dependency records overwhelmingly correspond to trace-visible code or evidence use.
They do not establish that every cited contribution causally improves performance. That distinct, system-level question is addressed by the new shared-memory intervention.