- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper presents LunarZero, a coding agent designed to operate with free-tier, rate-limited LLMs. It combines models from multiple providers into a unified pool through quota tracking, model routing, failure-specific cooldowns, and automatic failover. The authors evaluate LunarZero on SWE-bench Lite, using 98 models from 10 providers to complete all 300 tasks at zero model API cost, achieving a resolution rate of 42.3%. In a paired comparison on 186 tasks against a single-model control, pooling does not significantly improve the resolution rate but leads to substantially different failure modes. The ablation study further shows that a simple uniform exponential backoff strategy outperforms the proposed failure-specific cooldown policy.
Critical strengths
Practically relevant problem. Existing coding-agent research typically assumes continuous model availability. This paper instead studies a realistic setting where free-tier models have limited, dynamic, and exhaustible quotas.
Valuable empirical findings. Beyond resolution rates, the paper analyzes task abandonment, timeouts, model failovers, and API failures. The substantially different failure modes of pooled and single-model configurations, together with the improvement from uniform exponential backoff, provide useful insights into availability-aware routing.
Transparent discussion of limitations. The authors appropriately acknowledge that a non-significant difference does not establish equivalence and explicitly exclude four ablations confounded by quota exhaustion rather than drawing unsupported conclusions.
Critical weaknesses
- Potential confounding in the main experimental comparison.
Section 4 states that all configurations start with a fresh ledger at 00:00 UTC, while Appendix K notes that the control started 3.5 hours before a quota reset, following heavy prior usage. Since resetting the local ledger does not reset the providers' actual quotas, it is unclear whether the pooled and control configurations started with comparable resource availability.
Furthermore, the two configurations use different concurrency levels (6 vs. 3), which may affect rate limiting and execution efficiency. Given that model availability is the central variable studied in this paper, the authors should clarify the actual initial quota states and justify the concurrency settings. If necessary, a more tightly controlled paired comparison would help establish the reliability of the main findings.
- Limited methodological novelty and insufficient comparison with existing multi-model routing approaches.
The main technical components, including quota tracking, cooldowns, cross-model failover, and history reconstruction, have similar counterparts in existing LLM gateways and agent systems, such as LiteLLM and OpenRouter. The paper's main contribution therefore appears to lie in its empirical analysis of free-tier model pooling rather than in a novel routing method.
Although the authors include an ablation using uniform exponential backoff, the evaluation lacks a comparison with an independent, simple multi-model routing baseline. Notably, Table 3 shows that uniform exponential backoff improves the resolution rate from 27% to 45% on the same tasks, further raising questions about the benefits of the proposed hand-designed policy. Including a simple existing routing or fallback strategy as a baseline would help clarify LunarZero's contribution beyond established approaches.
- Insufficient evidence for the proposed explanation of the different failure modes.
The paper observes a substantially higher task abandonment rate under pooling (30.6% vs. 2.7%) and uses execution traces to suggest that weaker models often fail to continue tool execution after failover. However, the analysis does not systematically quantify how many abandoned tasks actually experienced a model switch, or disentangle the effects of model capability differences from those of cross-model execution continuity.
A more detailed analysis of the existing traces, particularly the relationship between abandonment, model switches, the final executing model, and tool-calling behavior, would strengthen the explanation of this important finding.
Questions for authors
Section 4 states that the selected control model accounts for 71.4% of the pooled configuration's steps, whereas Table 4 in Appendix D reports 71.4% as the share of the entire Kilo provider, which includes 17 models used during evaluation. Could the authors clarify whether this percentage refers to the individual model or the provider?
Table 2 reports an average of 32.9 failovers per task for the single-model control, while failover is defined as a mid-step switch between models. Given that the control is described as being pinned to a single model, what exactly do these failover events represent?
2 comments
(Part 2 of 2)
W3. Insufficient evidence for the proposed explanation of the different failure modes.
The paper observes a substantially higher task abandonment rate under pooling ( vs. ) and uses execution traces to suggest that weaker models often fail to continue tool execution after failover. However, the analysis does not systematically quantify how many abandoned tasks actually experienced a model switch, or disentangle the effects of model capability differences from those of cross-model execution continuity.
A more detailed analysis of the existing traces, particularly the relationship between abandonment, model switches, the final executing model, and tool-calling behavior, would strengthen the explanation of this important finding.
Response: Thank you for proposing this analysis; we have carried it out in full. It does not support our original explanation, which we will remove.
Model switches. Of the abandoned pooled tasks:
\textbf{Abandoned tasks} & \textbf{Count} \\\\ \hline \text{Answered by a single model throughout} & 31 \\\\ \text{Answered by two or more models} & 41 \\\\ \text{No answer} & 2 \\\\ \hline \text{Ended after one step (of the above)} & 27 \end{array}$$ **The first model.** Abandonment depends mainly on which model answers first. `command-a-reasoning` answered first in 29 tasks, and 25 of them were abandoned. **Tool-calling behaviour.** The transcript classification attributes 24 of those 25 tasks to model behaviour: the model wrote out a fix and made no tool call. Among the other 49 abandoned tasks, it found two harness defects: unparsed markup tool calls (13 tasks) and output cut-offs accepted as the end of the turn (20 tasks). **Switching alone.** The replication shows that switching does not explain abandonment: uniform backoff switched between more models per task (7–8 against 4) yet abandoned fewer tasks. The revised paper will report this analysis and mention switching to weaker models only as a plausible cause that we have not tested. **Questions For Authors:** Section 4 states that the selected control model accounts for $71.4\\%$ of the pooled configuration's steps, whereas Table 4 in Appendix D reports $71.4\\%$ as the share of the entire Kilo provider, which includes 17 models used during evaluation. Could the authors clarify whether this percentage refers to the individual model or the provider? **Response:** Thank you for catching this error. It refers to the provider: $71.4\\%$ is Kilo's share across its 17 models, and the control model's own share is $8.5\\%$. We will correct Section 4 and remove the stated reason for choosing the control. Table 2 reports an average of 32.9 failovers per task for the single-model control, while failover is defined as a mid-step switch between models. Given that the control is described as being pinned to a single model, what exactly do these failover events represent? **Response:** Thank you for this question. It was the starting point of our audit and uncovered the most serious error in the original paper. The failovers were real switches to other models: pinning the control did not disable failover, so any failed request could move a step to another model. - The pinned model answered 86–$99\\%$ of steps in tasks 1–120, and none after its provider stopped answering. - Only 34 of 186 tasks used it exclusively. We then built a true no-failover mode, verified that every answered step came from the pinned model, and used it as the strict arm of the replication, where it recorded 0 failovers. In the revised paper, we will relabel the control row "preferred model + fallback". We thank the reviewer again for a review that went to the heart of the paper. In summary, we have run the tightly controlled comparison with an independent gateway baseline and carried out the trace analysis you suggested. The revised paper will correct the control's description, report these results, replace the abandonment explanation with the trace analysis, and correct the $71.4\\%$ figure. We hope these changes address your concerns, and we would be grateful if you would take them into account in your final assessment. We remain fully available during the discussion period and would be glad to answer any further questions, run additional analyses on our records, or make any further revisions you suggest.
(Part 1 of 2)
We sincerely thank the reviewer for a careful and insightful review. Your question about the control's 32.9 failovers was the starting point of an audit that uncovered the most serious error in our original paper. We will correct it in the paper, and we have run the tightly controlled comparison you recommended. We respond to each point below.
Critical Weaknesses:
W1. Potential confounding in the main experimental comparison.
Section 4 states that all configurations start with a fresh ledger at 00:00 UTC, while Appendix K notes that the control started 3.5 hours before a quota reset, following heavy prior usage. Since resetting the local ledger does not reset the providers' actual quotas, it is unclear whether the pooled and control configurations started with comparable resource availability.
Furthermore, the two configurations use different concurrency levels (6 vs. 3), which may affect rate limiting and execution efficiency. Given that model availability is the central variable studied in this paper, the authors should clarify the actual initial quota states and justify the concurrency settings. If necessary, a more tightly controlled paired comparison would help establish the reliability of the main findings.
Response: Thank you for this precise diagnosis; it matches what our audit found. The quota states were not comparable, the concurrency difference was not justified, and, as you note, resetting the local ledger does not reset provider quotas. The revised paper will document the actual conditions.
We ran the tightly controlled paired comparison you recommend. It was pre-registered, with:
Its results are clear:
Full results are in the tables of our general comment.
W2. Limited methodological novelty and insufficient comparison with existing multi-model routing approaches.
The main technical components, including quota tracking, cooldowns, cross-model failover, and history reconstruction, have similar counterparts in existing LLM gateways and agent systems, such as LiteLLM and OpenRouter. The paper's main contribution therefore appears to lie in its empirical analysis of free-tier model pooling rather than in a novel routing method.
Although the authors include an ablation using uniform exponential backoff, the evaluation lacks a comparison with an independent, simple multi-model routing baseline. Notably, Table 3 shows that uniform exponential backoff improves the resolution rate from 27 to 45 on the same tasks, further raising questions about the benefits of the proposed hand-designed policy. Including a simple existing routing or fallback strategy as a baseline would help clarify LunarZero's contribution beyond established approaches.
Response: Thank you for this suggestion; we have added the independent baseline you describe.
Novelty. We agree that each component has counterparts in LiteLLM, OpenRouter and agent systems, and that the contribution is empirical. The revised paper will say so directly and state the contributions as:
Independent baseline. We added a LiteLLM gateway with a quality-ordered fallback chain over the same 65 models, with the same agent and protocol.
The hand-designed policy. We re-tested uniform backoff under controlled conditions. It matched or exceeded our cooldown policy (23 vs 16; 18 vs 15; not significant) and reached a similar throughput, while making 43–51 failovers per task against about 4. The benefit therefore comes from switching inside the agent loop, not from our cooldown design. We withdraw the earlier 27 result as evidence for any mechanism and will remove the original ablation table (Table 3).