acceptodds
Under review as a conference paper at ICLR 2027

LunarZero: A Single-Binary Coding Agent that Runs on Pooled Free-Tier Language Models with Quota-Aware Failover

Abstract

Agentic software engineering systems are commonly evaluated under the assumption that their language model is continuously available. This assumption excludes an important deployment setting: users and institutions that cannot afford sustained access to premium APIs or the infrastructure required to self-host capable models. Such users often depend on free or otherwise rate-limited access, where individual models have limited request and token budgets, availability changes over time, and failures may prevent an agent from completing a task even when alternative models remain accessible. We study this setting empirically. We build LunarZero, a terminal coding agent that treats models from multiple providers as a shared pool and maintains a persistent per-model ledger of request and token usage, failure-class cooldowns, and pre-streaming failover, and we evaluate it on all 300 tasks of SWE-bench Lite using the official instance images and evaluator. Across 886 evaluator-scored runs the pooled configuration resolves of tasks at zero model cost, using 98 distinct models from ten providers. We compare it against a matched control that uses the same agent and protocol pinned to the model ranked highest by the router's quality metric and used most by the pooled system. On the 186 tasks attempted by both, the pooled system resolves against for the control, a paired difference of percentage points ( CI ; exact McNemar, ): under the evaluated configuration we find no statistically detectable difference in task resolution. The configurations nevertheless exhibit substantially different failure modes. The pooled system abandons tasks without producing a patch far more frequently ( pp, ), whereas the control more frequently exhausts its time budget once its provider becomes unavailable ( pp, ); the pooled system also completes tasks faster and sustains far more total model usage than any individual free tier. Ablations show the hand-designed cooldown policy to be conservative: uniform exponential backoff increases resolution by pp on its paired subset () at roughly the request volume. We characterize 47,406 failover events, of which over a quarter arise from catalogue drift rather than rate limiting. These results characterize model availability as a distinct systems constraint: pooling rate-limited models can substantially increase aggregate availability and throughput without necessarily improving task resolution. We specify the routing policy and evaluation protocol in full, and report the accounting behind every claim, so that the study can be replicated independently.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
Coding agents, SWE-bench, empirical evaluation, model routing, rate limiting, free tiers, software engineering tools

Reviews

Among:
Shape:
1234this paper, Bottom 25%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.75 over 4 reviews, above 25% of the others.
Reviewer_epUrOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents LunarZero, a coding agent designed to operate with free-tier, rate-limited LLMs. It combines models from multiple providers into a unified pool through quota tracking, model routing, failure-specific cooldowns, and automatic failover. The authors evaluate LunarZero on SWE-bench Lite, using 98 models from 10 providers to complete all 300 tasks at zero model API cost, achieving a resolution rate of 42.3%. In a paired comparison on 186 tasks against a single-model control, pooling does not significantly improve the resolution rate but leads to substantially different failure modes. The ablation study further shows that a simple uniform exponential backoff strategy outperforms the proposed failure-specific cooldown policy.

Critical strengths

Practically relevant problem. Existing coding-agent research typically assumes continuous model availability. This paper instead studies a realistic setting where free-tier models have limited, dynamic, and exhaustible quotas.

Valuable empirical findings. Beyond resolution rates, the paper analyzes task abandonment, timeouts, model failovers, and API failures. The substantially different failure modes of pooled and single-model configurations, together with the improvement from uniform exponential backoff, provide useful insights into availability-aware routing.

Transparent discussion of limitations. The authors appropriately acknowledge that a non-significant difference does not establish equivalence and explicitly exclude four ablations confounded by quota exhaustion rather than drawing unsupported conclusions.

Critical weaknesses
  1. Potential confounding in the main experimental comparison.

Section 4 states that all configurations start with a fresh ledger at 00:00 UTC, while Appendix K notes that the control started 3.5 hours before a quota reset, following heavy prior usage. Since resetting the local ledger does not reset the providers' actual quotas, it is unclear whether the pooled and control configurations started with comparable resource availability.

Furthermore, the two configurations use different concurrency levels (6 vs. 3), which may affect rate limiting and execution efficiency. Given that model availability is the central variable studied in this paper, the authors should clarify the actual initial quota states and justify the concurrency settings. If necessary, a more tightly controlled paired comparison would help establish the reliability of the main findings.

  1. Limited methodological novelty and insufficient comparison with existing multi-model routing approaches.

The main technical components, including quota tracking, cooldowns, cross-model failover, and history reconstruction, have similar counterparts in existing LLM gateways and agent systems, such as LiteLLM and OpenRouter. The paper's main contribution therefore appears to lie in its empirical analysis of free-tier model pooling rather than in a novel routing method.

Although the authors include an ablation using uniform exponential backoff, the evaluation lacks a comparison with an independent, simple multi-model routing baseline. Notably, Table 3 shows that uniform exponential backoff improves the resolution rate from 27% to 45% on the same tasks, further raising questions about the benefits of the proposed hand-designed policy. Including a simple existing routing or fallback strategy as a baseline would help clarify LunarZero's contribution beyond established approaches.

  1. Insufficient evidence for the proposed explanation of the different failure modes.

The paper observes a substantially higher task abandonment rate under pooling (30.6% vs. 2.7%) and uses execution traces to suggest that weaker models often fail to continue tool execution after failover. However, the analysis does not systematically quantify how many abandoned tasks actually experienced a model switch, or disentangle the effects of model capability differences from those of cross-model execution continuity.

A more detailed analysis of the existing traces, particularly the relationship between abandonment, model switches, the final executing model, and tool-calling behavior, would strengthen the explanation of this important finding.

Questions for authors

Section 4 states that the selected control model accounts for 71.4% of the pooled configuration's steps, whereas Table 4 in Appendix D reports 71.4% as the share of the entire Kilo provider, which includes 17 models used during evaluation. Could the authors clarify whether this percentage refers to the individual model or the provider?

Table 2 reports an average of 32.9 failovers per task for the single-model control, while failover is defined as a mid-step switch between models. Given that the control is described as being pinned to a single model, what exactly do these failover events represent?

2 comments

(Part 1 of 2)

We sincerely thank the reviewer for a careful and insightful review. Your question about the control's 32.9 failovers was the starting point of an audit that uncovered the most serious error in our original paper. We will correct it in the paper, and we have run the tightly controlled comparison you recommended. We respond to each point below.

Critical Weaknesses:

W1. Potential confounding in the main experimental comparison.

Section 4 states that all configurations start with a fresh ledger at 00:00 UTC, while Appendix K notes that the control started 3.5 hours before a quota reset, following heavy prior usage. Since resetting the local ledger does not reset the providers' actual quotas, it is unclear whether the pooled and control configurations started with comparable resource availability.

Furthermore, the two configurations use different concurrency levels (6 vs. 3), which may affect rate limiting and execution efficiency. Given that model availability is the central variable studied in this paper, the authors should clarify the actual initial quota states and justify the concurrency settings. If necessary, a more tightly controlled paired comparison would help establish the reliability of the main findings.

Response: Thank you for this precise diagnosis; it matches what our audit found. The quota states were not comparable, the concurrency difference was not justified, and, as you note, resetting the local ledger does not reset provider quotas. The revised paper will document the actual conditions.

We ran the tightly controlled paired comparison you recommend. It was pre-registered, with:

  • a strict no-failover control;
  • one fresh UTC day per arm;
  • concurrency 3 everywhere;
  • two disjoint 60-task blocks.

Its results are clear:

  • resolution showed no detectable difference (+10.0+10.0 pp, p=0.42p = 0.42; −1.7-1.7 pp, p=1.00p = 1.00);
  • timeouts fell sharply under pooling (−27.5 and −55.0-55.0 pp; Holm p=0.022p = 0.022 and p<0.001p < 0.001).

Full results are in the tables of our general comment.

W2. Limited methodological novelty and insufficient comparison with existing multi-model routing approaches.

The main technical components, including quota tracking, cooldowns, cross-model failover, and history reconstruction, have similar counterparts in existing LLM gateways and agent systems, such as LiteLLM and OpenRouter. The paper's main contribution therefore appears to lie in its empirical analysis of free-tier model pooling rather than in a novel routing method.

Although the authors include an ablation using uniform exponential backoff, the evaluation lacks a comparison with an independent, simple multi-model routing baseline. Notably, Table 3 shows that uniform exponential backoff improves the resolution rate from 2727\\% to 4545\\% on the same tasks, further raising questions about the benefits of the proposed hand-designed policy. Including a simple existing routing or fallback strategy as a baseline would help clarify LunarZero's contribution beyond established approaches.

Response: Thank you for this suggestion; we have added the independent baseline you describe.

Novelty. We agree that each component has counterparts in LiteLLM, OpenRouter and agent systems, and that the contribution is empirical. The revised paper will say so directly and state the contributions as:

  • a pre-registered comparison of our router, a strict single model, uniform backoff and an independent gateway under matched quota and concurrency;
  • the finding that availability and resolution separate under exhaustible quotas;
  • an evaluation protocol for exhaustible model access, motivated by the failures we found in our own original evaluation;
  • a taxonomy of 47,406 failover events and a complete routing specification.

Independent baseline. We added a LiteLLM gateway with a quality-ordered fallback chain over the same 65 models, with the same agent and protocol.

  • It resolved at least as many tasks as our router (26 vs 16; 19 vs 15).
  • It timed out on 31 and 38 of 60 tasks, because 99.999.9\\% of its successful calls went to one rate-limited model, and it resolved about a third as many tasks per task-hour.

The hand-designed policy. We re-tested uniform backoff under controlled conditions. It matched or exceeded our cooldown policy (23 vs 16; 18 vs 15; not significant) and reached a similar throughput, while making 43–51 failovers per task against about 4. The benefit therefore comes from switching inside the agent loop, not from our cooldown design. We withdraw the earlier 2727\\% \to 45\\% result as evidence for any mechanism and will remove the original ablation table (Table 3).

(Part 2 of 2)

W3. Insufficient evidence for the proposed explanation of the different failure modes.

The paper observes a substantially higher task abandonment rate under pooling (30.630.6\\% vs. 2.72.7\\%) and uses execution traces to suggest that weaker models often fail to continue tool execution after failover. However, the analysis does not systematically quantify how many abandoned tasks actually experienced a model switch, or disentangle the effects of model capability differences from those of cross-model execution continuity.

A more detailed analysis of the existing traces, particularly the relationship between abandonment, model switches, the final executing model, and tool-calling behavior, would strengthen the explanation of this important finding.

Response: Thank you for proposing this analysis; we have carried it out in full. It does not support our original explanation, which we will remove.

Model switches. Of the 7474 abandoned pooled tasks:

\textbf{Abandoned tasks} & \textbf{Count} \\\\ \hline \text{Answered by a single model throughout} & 31 \\\\ \text{Answered by two or more models} & 41 \\\\ \text{No answer} & 2 \\\\ \hline \text{Ended after one step (of the above)} & 27 \end{array}$$ **The first model.** Abandonment depends mainly on which model answers first. `command-a-reasoning` answered first in 29 tasks, and 25 of them were abandoned. **Tool-calling behaviour.** The transcript classification attributes 24 of those 25 tasks to model behaviour: the model wrote out a fix and made no tool call. Among the other 49 abandoned tasks, it found two harness defects: unparsed markup tool calls (13 tasks) and output cut-offs accepted as the end of the turn (20 tasks). **Switching alone.** The replication shows that switching does not explain abandonment: uniform backoff switched between more models per task (7–8 against 4) yet abandoned fewer tasks. The revised paper will report this analysis and mention switching to weaker models only as a plausible cause that we have not tested. **Questions For Authors:** Section 4 states that the selected control model accounts for $71.4\\%$ of the pooled configuration's steps, whereas Table 4 in Appendix D reports $71.4\\%$ as the share of the entire Kilo provider, which includes 17 models used during evaluation. Could the authors clarify whether this percentage refers to the individual model or the provider? **Response:** Thank you for catching this error. It refers to the provider: $71.4\\%$ is Kilo's share across its 17 models, and the control model's own share is $8.5\\%$. We will correct Section 4 and remove the stated reason for choosing the control. Table 2 reports an average of 32.9 failovers per task for the single-model control, while failover is defined as a mid-step switch between models. Given that the control is described as being pinned to a single model, what exactly do these failover events represent? **Response:** Thank you for this question. It was the starting point of our audit and uncovered the most serious error in the original paper. The failovers were real switches to other models: pinning the control did not disable failover, so any failed request could move a step to another model. - The pinned model answered 86–$99\\%$ of steps in tasks 1–120, and none after its provider stopped answering. - Only 34 of 186 tasks used it exclusively. We then built a true no-failover mode, verified that every answered step came from the pinned model, and used it as the strict arm of the replication, where it recorded 0 failovers. In the revised paper, we will relabel the control row "preferred model + fallback". We thank the reviewer again for a review that went to the heart of the paper. In summary, we have run the tightly controlled comparison with an independent gateway baseline and carried out the trace analysis you suggested. The revised paper will correct the control's description, report these results, replace the abandonment explanation with the trace analysis, and correct the $71.4\\%$ figure. We hope these changes address your concerns, and we would be grateful if you would take them into account in your final assessment. We remain fully available during the discussion period and would be glad to answer any further questions, run additional analyses on our records, or make any further revisions you suggest.
Reviewer_8sBNOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies how one can use the free quota for different models across multiple providers to build a free tier terminal agent. To this end, the authors build LunarZero, a hand-engineered terminal agent and routing policy to pool the free tier models from multiple providers. The proposed agent is then evaluated on SWE-bench lite and compared against the same agent only using a single model. No statistically significant result between the two configurations is observed.

Critical strengths

The problem of routing across a large pool of free tier models from multiple providers is very interesting and relevant. The evaluation on swe-bench lite is reasonable, though maybe a harder bench mark such as swe-bench pro would be worth considering as well.

Critical weaknesses

My main concerns are:

  • the code is not released
  • As the routing policy forms a major component of the agent, it would be important to understand in more detail how it was designed.
  • I think it is worth clarifying why the agent is free. It is a property of the free model apis rather than the agent per se. Any agent pointed at free model endpoints would be free. Also, these free endpoints often come with caveats such a prompts being used for training which is worth mentioning under the paper's current framing.
  • Quota aware routing across and within tasks seems like a super interesting question here and I would have liked to see this tackled and discussed by the paper. It is mentioned as a problem but not addressed.
  • It would make sense to also estimate output tokens for a given request to ensure a model has quota to complete the task. Another question is to consider whether a model has quota to actually finish the remaining turns required to complete the task instead of just checking the current request can be completed.
  • Switching models would lose provider side prefix caching and adds latency, can this be quantified?
  • Figure 2 is a bit overwhelming for an overview figure, I think this could be improved.
  • Why was the control not run on the full set? Would it be possible to properly rerun the control also on the 186 tasks with fresh quota? This seems to strongly impact the comparison.
  • The throughput speed up seems to be confounded with abandoned tasks ending earlier?
  • Related to the point above is the higher rate of abandoning tasks an issue with the harness?
Questions for authors

see above

3 comments

(Part 1 of 2)

We sincerely thank the reviewer for a detailed and thoughtful list of concerns. Each one led to a concrete change: a matched comparison with a true single-model control, a measurement of switching latency, a classification of abandoned tasks, documentation of provider data terms, and a simpler overview figure for the revised paper. We respond to each point below.

Critical Weaknesses:

My main concerns are:

W1. the code is not released

Response: Thank you for raising this; we understand that verifiability matters. We are not able to release the agent's implementation. To make every result verifiable without it, we provide reviewers and ACs, in a restricted comment:

  • anonymized per-task records for every configuration (1,871 tasks);
  • the scripts that regenerate every number and figure;
  • the pre-registration;
  • the gateway configuration and its routing log;
  • the dated catalogue snapshot.

The revised Section 3 will also give every routing parameter, so the policy can be reimplemented.

W2. As the routing policy forms a major component of the agent, it would be important to understand in more detail how it was designed.

Response: Thank you for this question. The policy was hand-designed during interactive development and frozen before the benchmark harness existed; no parameter was tuned on the benchmark. The revised Section 3 will give the full specification, including every weight and adjustment term. The score of a model is

S=wq⋅q100+ws⋅s+wh⋅h−0.05,n+ν+λ+π,S = w_q \cdot \frac{q}{100} + w_s \cdot s + w_h \cdot h - 0.05\\,n + \nu + \lambda + \pi,

combining quality qq, speed ss and availability hh, with (wq,ws,wh)=(0.25,0.55,0.20)(w_q, w_s, w_h) = (0.25, 0.55, 0.20) for conversational requests and (0.75,0.05,0.20)(0.75, 0.05, 0.20) otherwise; a task-type adjustment ν∈[−0.05,+0.10]\nu \in [-0.05, +0.10]; and λ=π=0\lambda = \pi = 0 in every run. Sticky routing keeps a session's model for 30 minutes unless another scores at least 12 quality points higher.

We present it as one reasonable hand-designed policy, not an optimal one. The replication supports this framing: uniform backoff performed as well as our cooldown classes, so the specific heuristics were not what mattered.

W3. I think it is worth clarifying why the agent is free. It is a property of the free model apis rather than the agent per se. Any agent pointed at free model endpoints would be free. Also, these free endpoints often come with caveats such a prompts being used for training which is worth mentioning under the paper's current framing.

Response: Thank you for this insightful point; we agree completely. Being free is a property of the endpoints, and the revised paper will describe the system as an agent evaluated on free-tier endpoints. The claim will be about availability, not cost.

We documented each provider's data terms, which the revised paper will include:

  • May train on submitted data by default: Google AI Studio's unpaid tier and Mistral's free mode.
  • Unless the user opts out: Cohere.
  • Through upstream providers: some OpenRouter :free endpoints and Kilo's free models.
  • State that they do not train on API data: Groq, Ollama Cloud and Z.AI.

The revised paper will also state that free-tier endpoints should not receive proprietary code.

W4. Quota aware routing across and within tasks seems like a super interesting question here and I would have liked to see this tackled and discussed by the paper. It is mentioned as a problem but not addressed.

Response: Thank you for highlighting what we now see as the central open question. Our results sharpen it. The replication shows that the real choice is between waiting for a better model and switching to whichever model is available now:

  • waiting leads to timeouts on 33–6363\\% of tasks;
  • switching avoids them but leads to more abandoned tasks;
  • both reach similar resolution.

We do not solve this problem, and the revised paper will say so. It will present the wait-versus-switch decision, across and within tasks, as the main direction for future work.

W5. It would make sense to also estimate output tokens for a given request to ensure a model has quota to complete the task. Another question is to consider whether a model has quota to actually finish the remaining turns required to complete the task instead of just checking the current request can be completed.

Response: Thank you for this constructive suggestion; we agree. The router estimates prompt tokens only and checks only whether the current request fits. It does not estimate output tokens or the remaining turns of a task. The revised paper will list both as limitations and as concrete directions for future work.

(Part 2 of 2)

W6. Switching models would lose provider side prefix caching and adds latency, can this be quantified?

Response: Thank you for this suggestion; we have measured it. We compared the time to first token on switched steps with continuing steps of the same model and prompt size, using only steps without failed attempts:

\textbf{Measure} & \textbf{Estimate} \\\\ \hline \text{Added time to first token per switch} & \approx 1.0\text{ s } (95\\%\text{ CI } 0.4\text{–}1.5\text{ s}) \\\\ \text{Switch to a model new to the task} & \approx 1.6\text{ s; cached prompt share } 66\\% \to 1\\% \\\\ \text{Share of task wall time: switches} & \approx 2\\% \\\\ \text{Share of task wall time: failed attempts} & 14\text{–}16\\% \end{array}$$ These estimates are observational, because switches are not randomized. The revised paper will report them. W7. Figure 2 is a bit overwhelming for an overview figure, I think this could be improved. **Response:** Thank you for this feedback. We will replace it with a simple diagram of the routing loop: agent step, router, provider pool, and the ledger that records failures and sets cooldowns. W8. Why was the control not run on the full set? Would it be possible to properly rerun the control also on the 186 tasks with fresh quota? This seems to strongly impact the comparison. **Response:** Thank you for this important question; we agree that it strongly affected the comparison. The control was stopped by hand at 186 tasks, after its main provider stopped answering on September 21. Our audit also found that it could still fail over, so it was not a true single-model control. Re-running 186 or 300 tasks for every arm, with each arm on its own fresh quota day, was not feasible within the discussion period. We therefore ran a pre-registered replication on two disjoint 60-task blocks with: - a true no-failover control; - fresh quota for every arm; - matched concurrency. The results are in our general comment. W9. The throughput speed up seems to be confounded with abandoned tasks ending earlier? **Response:** Thank you for spotting this confound. You are right, and we withdraw the $1.7\times$ claim. Under matched conditions, we now measure resolved tasks per task-hour: pooled 3.49 and 2.54, strict 0.98 and 0.82. Because both arms resolved a similar number of tasks, this gain comes from avoiding rate-limit waits, not from abandoned tasks ending early. W10. Related to the point above is the higher rate of abandoning tasks an issue with the harness? **Response:** Thank you for this question, which led us to find two real defects in our harness. In part, yes. We classified the transcripts of all 74 abandoned tasks in the original pooled run: - **`command-a-reasoning`, 25 tasks.** In 24, the model wrote out a fix and made no tool call. This is model behaviour: the same adapter carried 80 well-formed tool calls from this model elsewhere. - **Other models, 49 tasks.** We found two harness defects: - in 13 tasks, a tool call arrived as model-native markup in the text, which our parser does not recover; - in 20 tasks, a reply cut off at the output limit was accepted as the end of the turn. These defects mostly affect free gateway models and inflate the pooled router's abandonment. They are present in every configuration, including the replication, and the revised paper will report them. **Questions For Authors:** see above **Response:** Thank you. Each question is answered under the corresponding point above. We thank the reviewer again. Your points led to new measurements (switching latency and the causes of abandonment), new experiments under matched conditions, and changes to the paper that clarify what "free" means and simplify Figure 2. We hope these changes address your concerns, and we would be grateful if you would take them into account in your final assessment. We remain fully available during the discussion period and would be glad to answer any further questions, run additional analyses on our records, or make any further revisions you suggest.
Reviewer_8sBNOpenReview ↗

Response to authors

Thanks to the authors for the rebuttal. Unfortunately, several of my concerns regarding the control ending early and having a failover, the harness issues, and the throughput being affected by abandoned tasks seem to be confirmed by the rebuttal. I appreciate some of the promised changes such as the overview figure and additional clarifications, but I do think with all these changes it substantially impacts the paper and would require another round of review. That said, I appreciate the authors' efforts and will make my decision during the AC reviewer discussion.

Reviewer_ExNrOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes LunarZero, a coding agent that combines multiple free LLM APIs into a single pool and uses quota-aware routing and automatic failover so that it can keep running when API usage limits are reached. On SWE-bench Lite, it resolves 42.3% of all 300 tasks using only free APIs. However, when compared with a single-model baseline on the same set of tasks, it does not show a statistically significant improvement in resolution rate, though the analysis shows that the two approaches differ in their failure modes. I view the paper’s main contribution as an empirical analysis of how API availability and quota limits affect agent performance, rather than a new routing algorithm.

Critical strengths
  1. Realistic problem setting and meaningful failure analysis

This paper empirically analyzes how the limited availability of free LLM APIs affects agent execution. In particular, it shows that while pooling multiple models allows more work to be carried out, switching between models introduces a problem where relatively weaker models fail to complete the task. I also find it valuable that the paper systematically analyzes failure types that arise in real systems, such as rate limits, model access restrictions, and API compatibility issues.

Critical weaknesses
  1. Limited novelty and insufficient comparison with existing routing systems

The core components of LunarZero, namely quota-aware routing, availability tracking, the cooldown policy, and automatic failover, are techniques already used in existing LLM routing systems. Systems such as LiteLLM offer similar functionality, and the proposed routing policy is heuristic rather than learned. I therefore consider the methodological novelty to be limited.

In addition, the paper does not compare directly against existing multi-provider routing systems and uses only a single model as the baseline. As a result, it is unclear what LunarZero contributes beyond existing approaches.

  1. Limitations in experimental design and reproducibility

The experiments rely on a single task ordering and a shared API quota. Section 4 states that experiments start at the quota reset, but Appendix K notes that the control run started 3.5 hours before the reset, when prior usage was already high. The pooled and control runs also differed in the number of tasks executed concurrently.

Four of the six ablations were invalid due to quota exhaustion, and the remaining two were evaluated on a limited number of tasks under cold-start conditions. I therefore believe additional repeated experiments under consistent quota conditions are needed. Another limitation is that the code and execution logs are not released, which makes it difficult to verify the results independently.

Questions for authors

Q1. Differentiation from existing routing systems

Compared with existing quota-aware routing and failover systems such as LiteLLM, what is the distinctive technical contribution of LunarZero? A direct comparison against existing routing policies using the same agent and model pool would make the contribution of the proposed system much clearer.

Q2. Fairness of experimental conditions

Section 4 states that experiments start at the quota reset, but Appendix K notes that the control run started 3.5 hours before the reset. Could the authors clarify whether the actual API quota state was the same for both runs, and why different concurrency levels were used for the pooled and control runs?

Q3. Clarification on model usage

Section 4 states that the Nemotron model chosen as the control handled 71.4% of all steps in the pooled system. However, Table 4 reports 71.4% as the usage share of the entire Kilo provider, which includes 17 models. Could the authors clarify whether this figure refers to the usage share of a single model or of the provider as a whole?

Q4. Reproducibility of the ablation results

The paper reports that changing the cooldown policy to uniform backoff improved the success rate by 18 percentage points. Does this result hold under different task orderings and quota states? I would also like to know whether the four invalid ablations have been rerun under the same initial quota conditions.

3 comments

(Part 1 of 3)

We sincerely thank the reviewer for a clear and fair review. We agree with your reading that the contribution is empirical rather than a new routing algorithm, and the revised paper will say so directly. Following your suggestions, we added a direct comparison with a LiteLLM gateway using the same agent and model pool, and re-ran all comparisons under consistent quota conditions. We respond to each point below.

Critical Weaknesses:

W1. Limited novelty and insufficient comparison with existing routing systems

The core components of LunarZero, namely quota-aware routing, availability tracking, the cooldown policy, and automatic failover, are techniques already used in existing LLM routing systems. Systems such as LiteLLM offer similar functionality, and the proposed routing policy is heuristic rather than learned. I therefore consider the methodological novelty to be limited.

In addition, the paper does not compare directly against existing multi-provider routing systems and uses only a single model as the baseline. As a result, it is unclear what LunarZero contributes beyond existing approaches.

Response: Thank you for this fair assessment, and for the concrete suggestion of a LiteLLM baseline, which we have now run.

Novelty. We agree that quota tracking, cooldowns and failover each exist in systems such as LiteLLM and OpenRouter, and that our policy is heuristic. The revised paper will no longer present them as methodological contributions. The paper's contributions are empirical and methodological:

  1. a pre-registered, controlled comparison of four ways to run a coding agent on free-tier models (our router, a strict single model, uniform backoff and an independent LiteLLM gateway) under matched quota and concurrency;
  2. the finding that resolution and availability separate under exhaustible quotas: switching models leaves resolution unchanged but removes most timeouts and roughly triples resolved tasks per hour, at the cost of more abandoned tasks;
  3. an evaluation protocol for exhaustible model access, motivated by the failures we found in our own original evaluation, each of which silently changed what was being compared;
  4. a taxonomy of 47,406 failover events, and a complete routing specification.

Comparison with an existing system. The gateway used the same agent and the same 65 models, with a quality-ordered fallback chain and LiteLLM's documented reliability settings, on the same tasks under the same protocol.

& \textbf{B1 pooled} & \textbf{B1 gateway} & \textbf{B2 pooled} & \textbf{B2 gateway} \\\\ \hline \text{Resolved (of 60)} & 16 & 26 & 15 & 19 \\\\ \text{Timeouts} & 2 & 31 & 2 & 38 \\\\ \text{Resolved per task-hour} & 3.49 & 1.12 & 2.54 & 0.81 \end{array}$$ - The gateway resolved at least as many tasks (uncorrected $p = 0.021$ in block 1 and $p = 0.34$ in block 2). - It timed out far more often. Its log shows why: $99.9\\%$ of its successful calls went to one model, because its first choice returned 429 on nearly every request. This comparison also shows where the availability gain comes from. Both arms that switch inside the agent loop at every step, our router and uniform backoff, resolved 2.5–3.5 tasks per task-hour. The gateway's fixed fallback chain settled on one rate-limited model and resolved 0.8–1.1. The gain therefore comes from per-step switching across the pool, not from our specific heuristics. A differently configured gateway might perform better; we used its documented settings. Because the hand-designed details made no difference, the open problem is the wait-versus-switch decision itself, which a learned policy could address. The revised paper will name this as future work. Full results are in the tables of our general comment.

(Part 2 of 3)

W2. Limitations in experimental design and reproducibility

The experiments rely on a single task ordering and a shared API quota. Section 4 states that experiments start at the quota reset, but Appendix K notes that the control run started 3.5 hours before the reset, when prior usage was already high. The pooled and control runs also differed in the number of tasks executed concurrently.

Four of the six ablations were invalid due to quota exhaustion, and the remaining two were evaluated on a limited number of tasks under cold-start conditions. I therefore believe additional repeated experiments under consistent quota conditions are needed. Another limitation is that the code and execution logs are not released, which makes it difficult to verify the results independently.

Response: Thank you for raising these issues. We agree that they required new experiments, and we ran them.

Repeated experiments under consistent conditions. The pre-registered replication ran each of four arms on its own fresh UTC day, at concurrency 3, on two disjoint 60-task blocks, with a strict single-model control that cannot switch (results in our general comment).

Ablations. Uniform backoff, the ablation you highlight, was re-tested as an arm of the replication (see Q4). Our audit also showed that all other original ablations ran after the main provider stopped answering. The revised paper will therefore report them only with that caveat and draw no mechanism conclusions from them.

Verification. We provide per-task records for every configuration, the analysis scripts that regenerate every number, the pre-registration and the dated catalogue snapshot to reviewers and ACs in a restricted comment. The agent's implementation is not included.

Questions For Authors:

Q1. Differentiation from existing routing systems

Compared with existing quota-aware routing and failover systems such as LiteLLM, what is the distinctive technical contribution of LunarZero? A direct comparison against existing routing policies using the same agent and model pool would make the contribution of the proposed system much clearer.

Response: Thank you for this question; it helped us state the contribution more precisely. The distinctive contribution is empirical (see W1). On the technical side, the one difference that matters in our data is where routing happens. LunarZero routes inside the agent loop: at every step it re-scores the whole pool, filtering by the step's context need and recent failures, and keeps a session on one model unless a clearly better one is available. A gateway sees individual requests and follows a fixed fallback chain.

We ran the direct comparison you suggest, with the same agent, the same 65 models and the same protocol (full results in our general comment):

  • Resolution: the gateway resolved at least as many tasks (26 vs 16 and 19 vs 15).
  • Availability: it timed out on 31 and 38 of 60 tasks, against 2, and resolved about a third as many tasks per task-hour (1.12 vs 3.49 and 0.81 vs 2.54).
  • Why: it fell back along its fixed chain and settled on one model, which served 99.999.9\\% of its successful calls, and then had to wait out that model's rate limits. Our router spread each task across about four models.

Per-step switching across the pool therefore changes availability, not resolution. This is what the paper reports, without claiming a better routing algorithm.

Q2. Fairness of experimental conditions

Section 4 states that experiments start at the quota reset, but Appendix K notes that the control run started 3.5 hours before the reset. Could the authors clarify whether the actual API quota state was the same for both runs, and why different concurrency levels were used for the pooled and control runs?

Response: Thank you for noticing this inconsistency. The quota states were not the same, and Section 4 is wrong. The control started at 20:27 UTC, right after another configuration, and ran across three UTC days. The concurrency difference (6 vs 3) was not justified. The replication fixes both, with one fresh UTC day per arm and concurrency 3 everywhere, and the revised paper will document the original conditions.

(Part 3 of 3)

Q3. Clarification on model usage

Section 4 states that the Nemotron model chosen as the control handled 71.471.4\\% of all steps in the pooled system. However, Table 4 reports 71.471.4\\% as the usage share of the entire Kilo provider, which includes 17 models. Could the authors clarify whether this figure refers to the usage share of a single model or of the provider as a whole?

Response: Thank you for catching this error. The figure refers to the whole provider: 71.471.4\\% is Kilo's share across its 17 models, and the control model's own share is 8.58.5\\%. We will correct Section 4 and remove the stated reason for choosing the control.

Q4. Reproducibility of the ablation results

The paper reports that changing the cooldown policy to uniform backoff improved the success rate by 18 percentage points. Does this result hold under different task orderings and quota states? I would also like to know whether the four invalid ablations have been rerun under the same initial quota conditions.

Response: Thank you for asking us to test this result again. The answer changed our conclusions.

The 18-point gain does not hold. Under controlled conditions, uniform backoff resolved 23 vs 16 tasks (block 1) and 18 vs 15 (block 2) against our cooldown policy. Neither difference is significant (p=0.14p = 0.14 and 0.51). We withdraw the 18-point claim, and the revised paper will state that our cooldown classes did not improve resolution.

The four other ablations were re-run, one per fresh day. All re-runs took place after the main provider stopped answering, so none can be compared with the reference run. Their paired differences were −3.4, +5.1, −3.3 and +8.3+8.3 pp, all with p≥0.42.p \geq 0.42. The revised paper will report them only with this caveat.

We thank the reviewer again. In summary, we have added the direct LiteLLM comparison you suggested and re-run the comparisons under consistent quota and concurrency. The revised paper will present the contribution as empirical, report these results, correct the 71.471.4\\% figure, and withdraw the cooldown claim. We hope these changes address your concerns, and we would be grateful if you would take them into account in your final assessment. We remain fully available during the discussion period and would be glad to answer any further questions, run additional analyses on our records, or make any further revisions you suggest.

Reviewer_YshdOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

LunarZero is a single-binary coding agent that pools free-tier language models using persistent quota tracking, failure-specific cooldowns, and model failover. The paper evaluates the system on all 300 SWE-bench Lite tasks, reporting a resolution rate of 42.3% without model-access charges. A matched comparison with a single-model configuration finds no statistically detectable resolution difference, but substantially different failure profiles: pooling produces more patchless terminations, whereas the single-model configuration experiences more timeouts. The contribution is primarily an empirical systems study of how exhaustible model access affects coding-agent reliability and efficiency.

Critical strengths

S1. The paper addresses a practically important deployment constraint. Model availability is often treated as fixed in coding-agent evaluations, although free-tier users face shared quotas, throttling, and changing access conditions. Studying these constraints separately from model capability is valuable for resource-limited users and institutions. The paper highlights that increasing access to inference does not necessarily improve task completion.

S2. The evaluation provides informative failure analysis and useful negative results. The study covers the full SWE-bench Lite test set using the official evaluator and distinguishes incorrect patches, patchless termination, and timeouts. This decomposition reveals operational differences that aggregate resolution rates obscure. The authors also explicitly exclude four quota-confounded ablations from their mechanism claims and report that uniform exponential backoff outperforms their hand-designed cooldown policy. These disclosures strengthen the study’s transparency and provide useful engineering observations.

Critical weaknesses

W1. Contradictory descriptions of the control’s initial quota state undermine the central comparison. Section 4 states that every configuration starts at midnight UTC with fresh quota conditions. However, Appendix K states that the single-model control began 3.5 hours before a quota reset following heavy prior use. If the latter describes the reported experiment, the control’s timeouts and exhaustion cannot be attributed solely to the absence of pooling. The configurations also use different concurrency levels—six tasks for pooling and three for the control—which can affect throttling and execution time. Matching benchmark instances does not resolve these confounds. The actual execution conditions must be clarified before the comparison can support conclusions about pooling.

W2. A single shared-quota trajectory provides insufficient support for the statistical conclusions. Tasks within each configuration consume the same quota pool, so earlier tasks affect the resources available to subsequent tasks. Section 4 and Appendix F acknowledge that task-level bootstrap intervals understate uncertainty, but the paper nevertheless relies on these intervals and task-level significance tests to characterize differences. One task ordering and a short evaluation window cannot establish whether the observed effects persist across independent quota cycles. Moreover, the resolution difference interval of −13.4 to +2.7 percentage points permits a substantial disadvantage for pooling; failure to detect a difference should not be interpreted as comparable performance.

W3. Limited reproducibility and incomplete resource accounting weaken the feasibility claims. The reproducibility statement and Appendix J explicitly state that neither the implementation nor per-task records are released. Meanwhile, the routing specification leaves important scoring weights and adjustment terms insufficiently defined for faithful reimplementation. The resource argument also overreaches: Table 4 lists several providers without a published daily token cap, so the largest known cap cannot establish that no individual free tier could support the evaluation. The relationship between kilo’s reported 200-request daily cap and its 6,334 model steps likewise requires clarification of quota scope and evaluation dates. These gaps prevent independent verification of central system-level claims.

W4. The efficiency measurements do not establish an improvement in useful throughput. Section 5.1 derives the reported 1.7-fold advantage from average task duration, but the pooled configuration also terminates substantially more tasks without producing a patch. Faster termination can therefore reflect faster failure rather than more efficient problem solving. Different concurrency settings further complicate the interpretation. Successful tasks per unit of total elapsed time, measured under matched concurrency and quota conditions, would more directly support the claimed throughput benefit.

Questions for authors
  1. Initial quota conditions: Which description corresponds to the reported control experiment: the fresh-quota protocol in Section 4 or the heavily used quota state described in Appendix K? Are results available with matched initial server-side quota and concurrency?

  2. Statistical reliability: Have the configurations been repeated across independent quota cycles or task orderings? If so, do the failure-profile differences and cooldown-ablation gains remain consistent when uncertainty is assessed across runs?

  3. Reproducibility and accounting: Can you provide anonymized task records, complete routing parameters, and a dated model-catalogue snapshot? How do the reported provider step counts reconcile with the applicable account-, model-, or provider-level limits?

  4. Effective throughput: Under matched concurrency, does pooling improve the number of successfully resolved tasks per hour of total elapsed time? How much of the reported duration reduction is attributable to early patchless termination?

3 comments
AuthorsOpenReview ↗

Response to the review (Part 1 of 3)

(Part 1 of 3)

We sincerely thank the reviewer for a thorough and constructive review. Your questions about the control's quota state and concurrency led us to audit every run record and to run a new pre-registered replication under matched conditions. The paper is substantially stronger as a result. We respond to each point below.

Critical Weaknesses:

W1. Contradictory descriptions of the control's initial quota state undermine the central comparison. Section 4 states that every configuration starts at midnight UTC with fresh quota conditions. However, Appendix K states that the single-model control began 3.5 hours before a quota reset following heavy prior use. If the latter describes the reported experiment, the control's timeouts and exhaustion cannot be attributed solely to the absence of pooling. The configurations also use different concurrency levels—six tasks for pooling and three for the control—which can affect throttling and execution time. Matching benchmark instances does not resolve these confounds. The actual execution conditions must be clarified before the comparison can support conclusions about pooling.

Response: Thank you for this careful reading. You identified a real inconsistency, and investigating it led to the most important corrections we will make.

Original conditions. Appendix K was correct and Section 4 was not. The control started at 20:27 UTC, ten minutes after another configuration finished, and ran across three UTC days at concurrency 3, against 6 for the pooled run. Our audit found two further problems:

  • the control was not strictly single-model, because a failed request could still move a step to another model;
  • its main provider stopped answering on September 21, after which the pinned model answered no steps.

The original comparison therefore does not isolate pooling. We will relabel it "preferred model + fallback" and report it only in a new audit section.

New matched comparison. To remove these confounds, we ran a pre-registered replication:

  • a strict control with all failover disabled, in which every answered step came from the pinned model;
  • one configuration per fresh UTC day;
  • concurrency 3 in every arm;
  • two disjoint 60-task blocks.

Under these conditions, pooling did not change resolution (block 1: +10.0+10.0 pp, p=0.42p = 0.42; block 2: −1.7-1.7 pp, p=1.00p = 1.00). It sharply reduced timeouts (−27.5 and −55.0-55.0 pp; Holm p=0.022p = 0.022 and p<0.001p < 0.001). In block 1, the comparison uses the 40 tasks that both arms attempted after an infrastructure failure, as the pre-registration specifies. Full results are in the tables of our general comment.

W2. A single shared-quota trajectory provides insufficient support for the statistical conclusions. Tasks within each configuration consume the same quota pool, so earlier tasks affect the resources available to subsequent tasks. Section 4 and Appendix F acknowledge that task-level bootstrap intervals understate uncertainty, but the paper nevertheless relies on these intervals and task-level significance tests to characterize differences. One task ordering and a short evaluation window cannot establish whether the observed effects persist across independent quota cycles. Moreover, the resolution difference interval of −13.4 to +2.7 percentage points permits a substantial disadvantage for pooling; failure to detect a difference should not be interpreted as comparable performance.

Response: Thank you for this point, which directly shaped the design of the new experiments. We agree with both parts.

Independent quota cycles. Each arm of the replication ran on two separate fresh quota days, on two disjoint task blocks. We report the blocks separately rather than pooling them:

  • the timeout reduction replicated in both blocks;
  • resolution showed no detectable difference in either block;
  • the abandonment difference reversed sign between the blocks, so we draw no conclusion from it.

With two blocks, variation across quota cycles can be shown but not formally estimated. The revised paper will state this, and the primary outcomes are Holm-corrected.

Non-detection is not equivalence. We fully agree. The revised paper will state this explicitly, and the abstract and conclusion will say "no detectable difference", never "comparable performance". The new intervals (for example, −13.3 to +8.3+8.3 pp in block 2) still allow a meaningful difference in either direction.

(Part 2 of 3)

W3. Limited reproducibility and incomplete resource accounting weaken the feasibility claims. The reproducibility statement and Appendix J explicitly state that neither the implementation nor per-task records are released. Meanwhile, the routing specification leaves important scoring weights and adjustment terms insufficiently defined for faithful reimplementation. The resource argument also overreaches: Table 4 lists several providers without a published daily token cap, so the largest known cap cannot establish that no individual free tier could support the evaluation. The relationship between kilo's reported 200-request daily cap and its 6,334 model steps likewise requires clarification of quota scope and evaluation dates. These gaps prevent independent verification of central system-level claims.

Response: Thank you for pushing us on verifiability. We have addressed each of the four gaps.

Per-task records. In a restricted comment to reviewers and ACs, we provide anonymized records for every configuration (1,871 tasks): outcome, steps, tokens, the model that answered each step, and timings. We also provide:

  • the scripts that regenerate every reported number;
  • the pre-registration;
  • the gateway configuration and its routing log;
  • the dated catalogue snapshot.

The agent's implementation is not included.

Routing parameters. The revised Section 3 will give the complete specification:

S=wq⋅q100+ws⋅s+wh⋅h−0.05,n+ν+λ+πS = w_q \cdot \frac{q}{100} + w_s \cdot s + w_h \cdot h - 0.05\\,n + \nu + \lambda + \pi

  • The weights are (wq,ws,wh)=(0.25,0.55,0.20)(w_q, w_s, w_h) = (0.25, 0.55, 0.20) for conversational requests and (0.75,0.05,0.20)(0.75, 0.05, 0.20) for all other requests.
  • ν=+0.10\nu = +0.10 for a reasoning model on a reasoning request, −0.05-0.05 for a reasoning model on a conversational request, and up to +0.10+0.10 for long-context requests.
  • λ=π=0\lambda = \pi = 0 in every run.
  • Sticky routing keeps a session's model for 30 minutes, unless another available model scores at least 12 quality points higher.

Resource claim. You are right that it overreached. We withdraw "no single free tier could have served the evaluation", and Table 4 will label its caps as catalogue values, not documented provider limits.

Kilo's cap and step count. Kilo's 6,334 steps were spread over 17 models. The busiest model answered up to 788 steps on September 20 (UTC), the single day of the pooled run, well above the catalogued 200 requests. Two facts explain this:

  • our ledger did not enforce the cap, because each agent process loaded the shared file once and rewrote it without locking, so concurrent tasks overwrote each other's counts;
  • Kilo served these requests without refusing, so the catalogued figure was not its operative limit either.

The revised paper will correct both points, including the description of the ledger in Appendix I.

W4. The efficiency measurements do not establish an improvement in useful throughput. Section 5.1 derives the reported 1.7-fold advantage from average task duration, but the pooled configuration also terminates substantially more tasks without producing a patch. Faster termination can therefore reflect faster failure rather than more efficient problem solving. Different concurrency settings further complicate the interpretation. Successful tasks per unit of total elapsed time, measured under matched concurrency and quota conditions, would more directly support the claimed throughput benefit.

Response: Thank you for suggesting a better measure; we adopted it directly and withdraw the 1.7×1.7\times claim. The replication reports resolved tasks per task-hour under matched concurrency and fresh quota:

\textbf{Arm} & \textbf{Block 1} & \textbf{Block 2} \\\\ \hline \text{pooled} & 3.49 & 2.54 \\\\ \text{backoff} & 2.97 & 2.74 \\\\ \text{strict} & 0.98 & 0.82 \\\\ \text{gateway} & 1.12 & 0.81 \end{array}$$ That is 3.1–3.6 times as many resolved tasks per hour. The gain does not come from early termination: pooled and strict resolved a similar number of tasks, but the strict model spent much of its time budget waiting out rate limits and timed out on 13 of 40 and 35 of 60 tasks, against 2 of 60 for pooled in each block. Uniform backoff reached a similar throughput (2.97 and 2.74). The gain therefore comes from switching itself, not from our particular cooldown policy, and the revised paper will say so.

(Part 3 of 3)

Questions For Authors:

Initial quota conditions: Which description corresponds to the reported control experiment: the fresh-quota protocol in Section 4 or the heavily used quota state described in Appendix K? Are results available with matched initial server-side quota and concurrency?

Response: Thank you for asking us to settle this clearly. Appendix K describes the reported control; Section 4 is wrong, and we will correct it and document the actual conditions. Matched results are now available from the pre-registered replication (see W1 and our general comment). Server-side quota cannot be observed directly, so we approximated matched conditions by giving every arm its own fresh UTC day at the same concurrency.

Statistical reliability: Have the configurations been repeated across independent quota cycles or task orderings? If so, do the failure-profile differences and cooldown-ablation gains remain consistent when uncertainty is assessed across runs?

Response: Thank you for this question. Answering it required new experiments, which we have run. Each arm ran on two independent quota days, on two disjoint task blocks.

  • Timeouts: the difference was consistent in both blocks (Holm p=0.022p = 0.022 and p<0.001p < 0.001).
  • Abandonment: not consistent. Pooled abandoned fewer tasks than strict in block 1 and more in block 2.
  • Cooldown gain: it did not hold. Uniform backoff matched or exceeded our cooldown policy in both blocks (23 vs 16 and 18 vs 15 resolved; p=0.14p = 0.14 and 0.51). The submitted 18-point gain is withdrawn.

The earlier second seed was affected by a provider failure, and the revised paper will report it as such, not as a replication.

Reproducibility and accounting: Can you provide anonymized task records, complete routing parameters, and a dated model-catalogue snapshot? How do the reported provider step counts reconcile with the applicable account-, model-, or provider-level limits?

Response: Thank you for these concrete requests. Yes to all three: the records and the catalogue snapshot (2026-09-11, the one compiled into the evaluated binary) are in the restricted comment, and the routing parameters are given in W3 above and will be added to Section 3. On reconciliation: the catalogued Kilo cap applied per model, and observed use exceeded it because our ledger under-counted concurrent use and the provider did not refuse requests. We cannot confirm the provider's actual account-level limit, and the revised paper will say so.

Effective throughput: Under matched concurrency, does pooling improve the number of successfully resolved tasks per hour of total elapsed time? How much of the reported duration reduction is attributable to early patchless termination?

Response: Thank you for this question. Yes: under matched concurrency, pooling resolved 3.1–3.6 times as many tasks per task-hour as the strict model (see W4). Early termination explains only a small part of the original duration reduction. In the original run, abandoned tasks took 4.0 minutes on average; excluding them, mean task time was still 9.2 minutes for pooled against 13.4 for the control, and the median time per resolved task was 9.1 against 13.5 minutes. In the replication, the duration gap is driven by the strict arm's timeouts.

We thank the reviewer again for a review that materially improved the paper. In summary, we have run a pre-registered replication under matched quota and concurrency and measured throughput as resolved tasks per hour, and we provide per-task records, analysis scripts and the full routing specification. The revised paper will clarify the original conditions, report these results, and withdraw every claim the evidence does not support. We hope these changes address your concerns, and we would be grateful if you would take them into account in your final assessment. We remain fully available during the discussion period and would be glad to answer any further questions, run additional analyses on our records, or make any further revisions you suggest.

Comments on the paper

2 comments

General response: audit, pre-registered replication, contributions, and planned revisions

We sincerely thank all four reviewers for careful and constructive reviews. Your questions about the control's quota state and concurrency, the 71.471.4\\% figure, the control's 32.932.9 failovers, novelty relative to LiteLLM, and throughput led us to audit every run record and to run a new pre-registered replication. This comment reports the results and the changes we will make to the paper; each reviewer also receives an individual reply.

1. What the audit of the original run found

  • The control could still fail over: its 32.932.9 failovers per task were real switches. We will relabel it preferred model + fallback.
  • Its main provider stopped answering on September 21, partway through the control.
  • The control did not start on fresh quota (Appendix K is correct; Section 4 is not), and concurrency was 66 vs 33.
  • 71.471.4\\% is Kilo's share across 1717 models; the control model's own share is 8.58.5\\%.
  • Our ledger did not enforce catalogued caps under concurrency (unlocked file rewrites).

2. Pre-registered replication

Two disjoint blocks of 6060 SWE-bench Lite tasks; each arm on its own fresh UTC day at concurrency 33. Arms: our pooled router; a strict single model with all failover disabled; uniform backoff; and an independent LiteLLM 1.106.2 gateway over the same 6565 models. Block 1 strict has 4040 tasks after an infrastructure failure.

\textbf{Arm} & \textbf{Resolved (B1 / B2)} & \textbf{Timeouts (B1 / B2)} & \textbf{Resolved per task-hour} \\\\ \hline \text{pooled} & 16 \\;/\\; 15 & 2 \\;/\\; 2 & 3.49 \\;/\\; 2.54 \\\\ \text{backoff} & 23 \\;/\\; 18 & 4 \\;/\\; 4 & 2.97 \\;/\\; 2.74 \\\\ \text{strict} & 9 \\,(\text{of } 40) \\;/\\; 16 & 13 \\;/\\; 35 & 0.98 \\;/\\; 0.82 \\\\ \text{gateway} & 26 \\;/\\; 19 & 31 \\;/\\; 38 & 1.12 \\;/\\; 0.81 \end{array}$$ Pooled minus strict (pp, $95\\%$ paired-bootstrap CI, Holm-adjusted exact McNemar $p$): $$\begin{array}{lcc} \textbf{Outcome} & \textbf{Block 1 } (n=40) & \textbf{Block 2 } (n=60) \\\\ \hline \text{Resolved} & +10.0\\ [-7.5,\\, +27.5],\\ p=0.42 & -1.7\\ [-13.3,\\, +8.3],\\ p=1.00 \\\\ \text{Abandoned} & -22.5\\ [-42.5,\\, -2.5],\\ p=0.10 & +21.7\\ [+8.3,\\, +35.0],\\ p=0.009 \\\\ \text{Timeout} & -27.5\\ [-45.0,\\, -10.0],\\ p=0.022 & -55.0\\ [-68.3,\\, -41.7],\\ p<0.001 \end{array}$$ **3. What the evidence supports** - **Resolution:** pooling does not improve it; backoff and the gateway resolved at least as many tasks. - **Availability:** switching inside the agent loop at every step (pooled, backoff) removes most timeouts and resolves $2.5$–$3.5$ tasks per task-hour, against $0.8$–$1.1$ for the strict model and the gateway, whose fallback chain settled on one rate-limited model ($99.9\\%$ of its successful calls). - **Cost:** pooling abandons more tasks, partly because of two harness defects we found. **4. Our contributions** Empirical and methodological, not a new routing algorithm: 1. **A pre-registered, controlled comparison** of a pooled router, a strict single model, uniform backoff and an independent LiteLLM gateway under matched quota and concurrency. 2. **The finding that availability and resolution separate** under exhaustible quotas: per-step switching leaves resolution unchanged but removes most timeouts and roughly triples resolved tasks per hour. The central decision is whether to wait for a strong model or switch to an available one. 3. **An evaluation protocol for exhaustible model access** (fresh quota per arm, matched concurrency, a control that cannot switch, a per-step record of the answering model), motivated by the failures in our own original evaluation. 4. **A failure taxonomy** of $47{,}406$ failover events by cause, and **a complete routing specification**. **5. Changes we will make to the paper** - Rewrite the abstract, introduction and conclusion around the replication. - Add the replication and an audit section covering point 1. - Add the complete routing parameters, the latency per switch (about $1.0$ s, roughly $2\\%$ of task time), a classification of abandoned tasks, and provider data terms; simplify Figure 2. - Withdraw: that pooling improves accuracy; the $1.7\times$ figure; the $18$-point cooldown gain; "no single free tier could have served the evaluation"; the "weaker model after failover" explanation; and the novelty of the routing mechanisms. Per-task records and analysis scripts are provided to reviewers and ACs in a restricted comment. We are grateful for reviews that made this a more honest and more useful paper, and we hope the reviewers will take these results into account in their final assessment.
AuthorsOpenReview ↗

Per-task records and analysis scripts (reviewers and ACs only)

Per-task records and analysis scripts (reviewers and ACs only)

As stated in our general comment, this anonymous link gives reviewers and ACs the data behind every number in our replies:

https://anonymous.4open.science/r/s46018-records-430D/

The repository contains:

  • Per-task records of every configuration (1,8711{,}871 tasks): outcome, evaluator verdict, steps, tokens, the model that answered each step, and timings;
  • Analysis scripts that regenerate every reported number and the outcomes-per-arm figure from those records;
  • The pre-registration, with every deviation dated, and the run log;
  • The gateway configuration and its per-attempt routing log;
  • The dated model catalogue compiled into the evaluated binary;
  • Supporting analyses: classification of abandoned tasks, latency per model switch, and provider data terms.

Model transcripts, credentials and the agent's implementation are not included. README.md explains which folder is which. Running python3 replication_analysis.py and python3 audit.py (Python ≥3.11\geq 3.11, standard library only) reproduces the shipped results byte for byte.

Please do not redistribute. Thank you for taking the time to examine our data.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.