- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper introduces BOTTLED, a benchmark assessing whether LLM agents can create inexpensive task-specific solutions for million-instance workloads under fixed budgets. Experiments evaluate ten models across three tasks against zero-shot and distillation baselines, identifying substantial quality gaps, resource-management failures, and occasional large reductions in reported API costs.
Critical strengths
-
The paper formulates a practically important evaluation setting: converting general model capabilities into inexpensive solutions for large repetitive workloads. Its resource-constrained, artifact-agnostic protocol accommodates programs, distilled models, and hybrid strategies.
-
The evaluation covers ten models on workloads containing millions of instances. Zero-shot and distillation baselines contextualize agent performance, while budget-relaxation experiments help distinguish resource-management failures from other causes of underperformance.
Critical weaknesses
-
Reported dollar costs exclude local GPU and CPU expenses. Distillation baselines match token budgets without documented time-budget matching, weakening claims about end-to-end savings and fair resource-constrained comparisons between competing approaches.
-
Only three tasks and two agent runs per model-task pair support broad capability comparisons. Missing agent-level uncertainty, paired analyses, and harness ablations make model rankings and general conclusions insufficiently robust.
Questions for authors
-
Report total costs, GPU-hours, CPU-hours, and wall-clock times for agents and students. Do the savings and distillation comparisons persist under matched compute budgets and model-matched teachers across all three tasks?
-
Can you provide additional independent runs, confidence intervals for model rankings, and paired zero-shot versus bottled comparisons on identical instances? How sensitive are conclusions to harness configuration and resource budgets?
11 comments
(2/6) Response to Reviewer aGgp
W1.2 Distillation baselines match token budgets without documented time-budget matching; do resource-constrained comparisons between competing approaches still preserve the conclusions?
Yes, our conclusion, that bottling is often outperformed by a naive distillation baseline, is preserved under matched time budgets. Confined to the agents' 600-minute budget, the better distilled student still beats 46 of 90 bottling runs.
Setup. The distillation uses the same hardware as the agents (a single A100 40GB GPU, 8 CPU cores and 64GB RAM) and the same API budget (5 Mtok, spent on GLM 5.3 Flash teacher labels). For each task and student, we combine the wall clock times for teacher labelling, student training, and student inference over the full workload, averaged over three runs. For a given task, we report the proportion of the 30 bottling runs of that task that were outperformed by the student. On RAID, the full distillation pipeline slightly exceeds 600 minutes; for fair comparison, we additionally include the score of the distillation pipeline if it had been stopped at 600 minutes. Instances assumed missing in this case are scored as trivial answers (on RAID, i.e., AI detection, this amounts to a 50% predicted probability that the document was AI-written).
| Task | Method | API budget | Teacher labelling (minutes) | Student training (minutes) | Student inference (minutes) | Total (minutes) | Bottled runs beaten (of 30) |
|---|---|---|---|---|---|---|---|
| MAVE | Student: Qwen3-0.6B | 5 Mtok | 46 | 38 | 293 | 377 | 18 |
| MAVE | Student: SmolLM2-360M | 5 Mtok | 46 | 18 | 285 | 349 | 21 |
| ESCI | Student: Qwen3-0.6B | 5 Mtok | 18 | 46 | 168 | 232 | 15 |
| ESCI | Student: SmolLM2-360M | 5 Mtok | 18 | 27 | 168 | 213 | 8 |
| RAID | Student: Qwen3-0.6B | 5 Mtok | 17 | 37 | 597 | 651 | 10 |
| RAID | Student: Qwen3-0.6B, stopped at 600 min | 5 Mtok | 17 | 37 | 546 | 600 | 10 |
| RAID | Student: SmolLM2-360M | 5 Mtok | 17 | 23 | 595 | 635 | 8 |
| RAID | Student: SmolLM2-360M, stopped at 600 min | 5 Mtok | 17 | 23 | 560 | 600 | 8 |
| All | Agent (bottling) | 5 Mtok | – | – | – | 600 (budget) | – |
Results. The distillation baseline still outperforms 46 of 90 bottling runs under matched time budgets. On MAVE and ESCI, the full pipeline finishes in 213 to 377 minutes, well within the agents' 600-minute budget. On RAID, the distillation pipeline slightly exceeds the budget; however, by the 600th minute, the students had already labeled over 90% of the workload; even assuming that the remaining answers are missing, the students beat the same number of bottling runs as when run to completion (10 and 8 of 30, respectively). Time budgets therefore do not confound our conclusion.
(3/6) Response to Reviewer aGgp
W2 Robustness of conclusions
W2.1: Task variety
We strive to cover the most interesting use cases of LLMs on large repetitive workloads, and our capability comparisons extend to a fourth task beyond classification/extraction, which we add below.
We consider TL;DR generation from 3.8 million Reddit posts [1] and evaluate all 10 models under both zeroshot and bottled setups.
Setup. Similar to our original experiments, we evaluate each LLM under both zero-shot and bottling modes. Under zero-shot mode, we ask the LLM to summarize 1000 posts, one at a time. Under the bottling mode, we ask the LLM agent to summarize all 3.8 million posts under a budget of 10 hours and 5 million tokens. We evaluate the answers using “win rate”, i.e., how often the “system one” model Jev [2] prefers the answer's summary to the author’s summary [5]. Because evaluating all 3.8M samples using Jev is infeasible, we evaluate the same 1000 summaries for both zero-shot and bottled solutions, and compare their 95% CIs over 2000 resamplings. We report the results below:
| Model | Zero-shot win rate [95% CI] | Bottled win rate [95% CI] |
|---|---|---|
| Qwen 3.8 Max | 0.996 [0.992, 0.999] | 0.731 [0.703, 0.757] |
| Sonnet 5 | 0.997 [0.993, 1.000] | 0.730 [0.702, 0.755] |
| GPT 5.6 Terra | 0.997 [0.993, 1.000] | 0.705 [0.677, 0.734] |
| Qwen 3.8 Flash | 0.995 [0.990, 0.999] | 0.660 [0.630, 0.690] |
| GLM 5.3 Flash | 0.997 [0.993, 1.000] | 0.640 [0.611, 0.670] |
| Opus 5 | 0.998 [0.995, 1.000] | 0.625 [0.596, 0.654] |
| GLM 5.3 | 0.995 [0.990, 0.999] | 0.501 [0.470, 0.530] |
| Gemini 3.1 Pro | 0.990 [0.983, 0.995] | 0.460 [0.430, 0.491] |
| GPT 5.6 Sol | 0.999 [0.997, 1.000] | 0.039 [0.028, 0.051] |
| Gemini 3.1 Flash-Lite | 0.988 [0.980, 0.994] | 0.037 [0.026, 0.049] |
On TL;DR, bottling reveals capability differences that zero-shot evaluation generally does not. All ten LLMs are almost indistinguishable zero-shot, with win rates between 0.988 and 0.999. On the other hand, no bottled CI overlaps with its corresponding zero-shot CI; bottled win rates range from 0.037 to 0.731 (a range of 0.694). The gap between zero-shot and bottling capability, observed in our main experiments on MAVE, ESCI and RAID, persists on open-ended generation.
We also stress that our original 3 tasks correspond directly to industrial use cases. TypeSafe AI introduces its recently released Jev [2] as "a new class of frontier models built to make fast, structured decisions" for use cases such as "map-reducing over big data", i.e., making the same decision for every item of a large dataset. [2] lists these decisions as "classify, route, score, extract, or branch". Our tasks cover three of the five: MAVE extracts an attribute's value from a product listing (or abstains), ESCI classifies a query–product pair into one of four labels, and RAID scores a text with a continuous output. ESCI and RAID also correspond directly to two of Jev's three question types for text [3]: ESCI is a choice question over 4 labels, and RAID is a yes/no (noul) question. Our tasks represent a class of use cases important enough to warrant a frontier model.
W2.2 Only two agent runs per model-task pair
Following the reviewer’s request, we added a third independent run per task-model pair; this run preserves our central claim that bottling consistently falls short of zero-shot model competence; please see W2.3.
W2.3 agent-level uncertainty
Even after accounting for agent-level uncertainty (run-to-run variation), our main claim (there exists a consistent zeroshot-bottling gap) remains preserved. We conducted three independent bottling runs for every model-task pair (90 runs in total). Below we report each LLM’s bottled relative gain (aggregated over tasks) as the mean ± standard deviation across runs.
| Model | Zero-shot gain [95% CI] | Bottled gain (mean ± std over 3 runs) |
|---|---|---|
| Opus 5 | 0.606 [0.544, 0.657] | 0.394 ± 0.140 |
| Gemini 3.1 Pro | 0.586 [0.534, 0.630] | 0.281 ± 0.184 |
| GLM 5.3 | 0.584 [0.542, 0.622] | 0.271 ± 0.028 |
| GPT 5.6 Sol | 0.573 [0.515, 0.620] | 0.149 ± 0.164 |
| GPT 5.6 Terra | 0.572 [0.508, 0.622] | 0.288 ± 0.104 |
| Qwen 3.8 Max | 0.568 [0.505, 0.620] | 0.361 ± 0.032 |
| GLM 5.3 Flash | 0.543 [0.487, 0.589] | 0.398 ± 0.021 |
| Sonnet 5 | 0.541 [0.473, 0.597] | 0.212 ± 0.096 |
| Qwen 3.8 Flash | 0.539 [0.477, 0.591] | 0.266 ± 0.036 |
| Gemini 3.1 Flash-Lite | 0.473 [0.397, 0.538] | 0.061 ± 0.017 |
For 10/10 LLMs, the mean bottled gain lies more than one standard deviation below the lower end of the corresponding zero-shot 95% CI. Only Opus 5 comes close (0.394 + 0.140 = 0.534 vs. 0.544) but still falls slightly short. This indicates that while individual runs can vary a lot, bottling consistently falls short of zeroshot competence.
(1/6) Response to Reviewer aGgp
Thank you for your engagement with our work and for asking interesting questions. Please note that we added a third independent bottling run per model-task pair (90 runs in total, in response W2.2) and our main claims remain unchanged; all our answers below reflect this updated suite of runs (90 main runs vs. the original 60).
W1 Cost accounting
W1.1: Reported dollar costs exclude local GPU and CPU expenses; do end-to-end savings persist under matched resources?
Yes, end-to-end savings from bottling persist even after accounting for local compute (GPU and CPU): across our 10 LLMs, bottling remains between 32 and 5,197 times cheaper than zero-shot labeling. To confirm, we recomputed the mean cost of bottling per LLM after considering local compute expenses.
Setup. Bᴏᴛᴛʟᴇᴅ provides, to each agent, a single A100 40GB GPU, along with 8 CPU cores and 64GB RAM on our local infrastructure. We charge this hardware at a representative on-demand VM price: $1.99/h for an A100 40GB instance [6]. This is a realistic upper bound on our compute cost: the Lambda instance offers close to 4× our CPU cores and 3× our RAM; hence the savings below are conservative.
For each LLM, we computed the total bottling cost as the sum of its (i) mean API cost and (ii) the hourly VM rate scaled by the mean wall time of a bottling run on that LLM. The total zero-shot cost is the projected API cost of labelling the full workload with the same LLM. As in our aggregate results (Figure 1), all values are averaged over the three tasks; for bottling, they are further averaged over three runs. Below, we report the results for the LLMs with the lowest and highest savings, respectively. We have added the results for all LLMs in the paper.
GLM 5.3 Flash, the LLM with the lowest zeroshot-to-bottled cost ratio, still costs 32x less when bottled than when run zero-shot (\13.89vs451) and achieves a bottled gain that is 73% of its zero-shot gain (0.398 bottled vs 0.543 zero-shot). For LLMs with higher API prices, savings can be much larger: GPT 5.6 Terra costs 5,197x less when bottled (\8.37vs43,507). Across all 10 LLMs, the median saving is 901x. Even after including local compute, bottling remains orders of magnitude cheaper than zero-shot labelling.