- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper proposes Hackathon-24h, a protocol for evaluating coding agents over 24 hours. An agent builds one of four systems from scratch in Python (a MySQL-compatible database, an AMQP message queue, an HTTP/1.1 server, or a Redis-compatible store). A separate test loop takes a snapshot of the workspace every 30 minutes, scores it against 13 external industrial test suites (257 tests in total), and gives the agent anonymized feedback. Each run is summarized by its peak, final and AUC (time-averaged) score. The main study runs 10 models on the 4 tasks with the Claude Code harness, one 24-hour run each (40 runs, 960 agent-hours). Small extra studies vary the harness (6 runs), remove the feedback (1 run) and replicate 4 cells (3 runs each). The paper reports that trajectories show behaviors a final-state score hides, such as losing earlier correctness ("hidden regression") and progress that appears only late ("delayed success"), and that changing the harness can move scores as much as changing the model.
Critical strengths
- Real systems with public specifications (database, message queue, HTTP server, Redis store).
- Scored by established external test suites, not by tests the authors wrote.
- Each suite is checked against the real production system (MySQL, RabbitMQ, nginx, Redis).
- The agent cannot see the test code or the scoring machinery, so the scores are hard to game/manipulate.
- Tracking the whole run shows what a final score misses: GLM-5 reaches 70% on the message queue in 3 hours, then ends at 47%.
- GLM-5 and Claude Opus 4.6 end with the same database tests, at hour 22.5 and hour 3 respectively; only AUC tells them apart.
Critical weaknesses
- Each model–task result in the main table comes from a single 24-hour run, so it is unclear how stable it is.
- The paper's own repeated runs differ a lot. In three identical runs, GLM-5's peak on the message queue is 70%, 47% and 13% (Table 10).
- This run-to-run variation is as large as the differences the paper draws conclusions from: between models, between harnesses (about 15-17 points), and with or without feedback.
- The harness study uses only 6 runs and the feedback study only 2, which is too few to support general conclusions.
- The paper never checks whether tests give the same result on unchanged code. Throughput and concurrency tests can be noisy, so some reported "regressions" may be measurement noise rather than real breakage.
- All models run inside Claude Code, a harness designed for Claude models, and Claude Opus 4.6 ranks first on every task. Part of the ranking may reflect how well each model fits this harness rather than model ability.
- The budget is wall-clock time, so faster APIs or fewer rate limits let some models do more work in the same 24 hours. This mixes model ability with provider speed.
- All four systems must be built in Python. What if such systems are written in languages like C, Go or Rust? Results may not carry over to other languages, and throughput-based tests may partly measure Python's speed.
- Per-checkpoint scores are available only on request, and the generated code, snapshots and agent logs are not released. Others cannot check the reported trajectory patterns.
- Each run takes 24 hours and up to billions of tokens, so reproducing or extending the results is very expensive.
- The contamination check compares the generated code with only a few Python projects and skips the message queue entirely, so copying from other public implementations is not ruled out.
Questions for authors
Q1. If the same snapshot is scored several times, how often does a test flip between pass and fail? How many of the 16 "hidden regression" cells remain after removing flaky tests? Q2. With 3 runs per cell for more cells, do the harness effect (Table 4) and the feedback effect (Table 5) stay larger than the run-to-run spread shown in Table 11? Q3. How were non-Anthropic models connected to Claude Code (API proxy, settings, context handling), and were they given any model-specific tuning? Would the model ranking change under a model-neutral harness such as OpenCode? Q4. How much does API speed or rate limiting affect the number of iterations a model completes in 24 hours? Did any runs hit rate limits or provider outages? Q5. Will the per-checkpoint scores, agent logs and generated code be released publicly, so that others can verify the trajectory phenomena without re-running 960 agent-hours?
2 comments
Thank you for the response.
-
The repeated scoring of unchanged snapshots (1.44% flaky, all 16 regression cells confirmed) answers my question on test noise.
-
The response now treats the harness and feedback comparisons as descriptive case studies. The main table still rests on single runs, and the paper's own replicated cells vary by up to 57 points (GLM-5: 70%, 47%, 13%). The model rankings and the claim that changing the harness moves scores as much as changing the model should be removed or clearly qualified in the paper.
-
26 of 40 main runs hit rate-limit, overload or server errors, which confirms that the 24-hour score partly measures provider reliability, not only the model.
-
All models still run in Claude Code, and ranking stability across harnesses is left to future work.
-
Agent logs and generated code remain unreleased beyond the 16 selected checkpoint pairs.
With the comparative claims withdrawn, the contribution is mainly a descriptive study of four tasks with single runs. I am keeping my rating at 2.
Rebuttal for Reviewer QB4w
We thank the reviewer for the detailed review and for recognizing the external conformance tests, evaluation isolation, and value of trajectory analysis beyond final-state scoring.
1. Run variability and comparative claims — W1–W4 / Q2
Our core empirical contribution is the characterization of regression, delayed success, and temporal dynamics that final-state scoring misses; we do not claim configuration effects over run-to-run variation or stable rankings. Appendix D.2 reports three runs per configuration across four configurations to illustrate long-horizon run variability. The harness and feedback comparisons therefore serve as descriptive sensitivity case studies, rather than evidence of general performance benefits.
2. Evaluation noise and regression validation — W5 / Q1
We now provide one regression checkpoint pair for each of the 16 reported cells, with before/after source snapshots and evaluation records (https://anonymous.4open.science/r/Hackathon-24h-anonymous/regression_cases.zip). Each of the 32 unchanged snapshots was evaluated four additional times, yielding 128 re-evaluations.
We identify a test as flaky on a snapshot if its pass/fail outcome varies across the original evaluation and four re-evaluations. Under this definition, 31 of the 2,154 test–snapshot combinations (1.44%) exhibit flakiness. After removing tests identified as flaky on either snapshot, all 16 hidden-regression cells remain, with 88 originally observed test regressions confirmed: each test passes in all five evaluations before the change and fails in all five afterward.
3. Harness dependence and model integration — W6 / Q3
Non-Anthropic models were connected to Claude Code through provider-supplied Anthropic-compatible APIs, with no model-specific tuning. All models used the same task prompts and controller logic. Context handling was delegated to Claude Code, with session resumption across iterations and a common fresh-session fallback on context-overflow errors. These integration and context-handling mechanisms are documented in our released evaluation framework.
Using Claude Code across models provides a common setting for studying long-horizon trajectories. Our evaluation focuses on the deployed agent system, including model–harness compatibility. The OpenCode and Cline sensitivity case studies complement this analysis by illustrating how observed trajectories differ across harnesses; assessing ranking stability across harnesses is a direction for future work.
4. Wall-clock budget and provider effects — W7 / Q4
Our 24-hour budget measures what a deployed agent can build within a fixed delivery deadline, including the API speed and service reliability encountered during execution. The framework handles rate-limit and transient API errors through retries and recovery. Across the 40 main-table trajectories, 26 contained explicit rate-limit, overload, or server-side errors. Three trajectories had error-to-resumption intervals exceeding 10 minutes (16.2, 24.5, and 38.2 minutes), including retries, backoff, and subsequent inference. All such intervals counted toward the fixed budget. We will clarify this treatment in the experimental setup.
5. Python scope and contamination — W8 / W11
Python provides a common implementation setting for this study. Our main contribution is the trajectory-based evaluation protocol, which uses external client interfaces and can accommodate implementations in C, Go, or Rust. The reported outcomes, particularly throughput-sensitive results, remain specific to the Python setting; evaluating other languages is a natural extension.
The contamination analysis examines overlap with selected Python reference implementations. MQ was not included because we did not identify a suitable Python reference for this comparison. Together with the reference-implementation evaluations, Dolos provides evidence concerning direct reuse of the examined references, rather than an exhaustive check against all public implementations.
6. Artifact access and reproduction cost — W9–W10 / Q5
We have publicly released the per-checkpoint scores for all 40 trajectories, enabling independent inspection and recomputation of peak, final, and AUC (https://anonymous.4open.science/r/Hackathon-24h-anonymous/supplementary.zip). Beyond the selected regression cases, the complete logs and generated implementations will remain unreleased because sensitive-information screening and redistribution review are not yet complete.
Our evaluation framework is already open-source and supports reproduction and extension to additional models, harnesses, and providers. The 24-hour horizon captures late progress and regression that shorter evaluations can miss, while the scores and selected cases support trajectory inspection and regression validation without repeating the full 960-agent-hour study.
We hope the additional validation and scope clarifications address the reviewer’s concerns.