acceptodds
Under review as a conference paper at ICLR 2027

Agent in a Bottle: Can LLM Agents Turn Their Capabilities Into Cheap, Scalable Artifacts?

12 views

Abstract

Large language models (LLMs) can solve many narrow tasks, but querying them separately for millions of related instances can be prohibitively expensive. Can LLM agents autonomously create cheaper solutions for such workloads? We call this ability bottling: the ability to turn general capabilities into task-specific solutions that balance answer quality and amortised cost. We introduce BOTTLED, a benchmark in which agents receive an entire unlabelled workload and must complete it under fixed time, compute and LLM API budgets. Agents choose their own approach, such as training a small model or writing a reusable program. Across ten models and three tasks, we find that strong zero-shot task performance does not reliably translate into strong bottling capabilities. Models with similar zero-shot scores can differ substantially after bottling, and 48 of 60 runs score below the lower bound of the 95% confidence interval of their underlying model’s zero-shot-performance. Moreover, 31 of 60 runs underperform the stronger of two naive small-model distillation baselines with the same token budget. Nevertheless, bottling can yield substantial savings: on query-product relevance classification, Opus 5 retains approximately 82% of its zero-shot macro-F1 at roughly 657 times lower reported cost. BOTTLED provides a basis for evaluating and improving agents’ ability to invest limited resources in reusable solutions for large, repetitive workloads.

Primary area:
foundation or frontier models, including LLMs
Keywords:
large language models, agents, benchmarking, amortization, generalists and specialists

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_aGgpOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces BOTTLED, a benchmark assessing whether LLM agents can create inexpensive task-specific solutions for million-instance workloads under fixed budgets. Experiments evaluate ten models across three tasks against zero-shot and distillation baselines, identifying substantial quality gaps, resource-management failures, and occasional large reductions in reported API costs.

Critical strengths
  1. The paper formulates a practically important evaluation setting: converting general model capabilities into inexpensive solutions for large repetitive workloads. Its resource-constrained, artifact-agnostic protocol accommodates programs, distilled models, and hybrid strategies.

  2. The evaluation covers ten models on workloads containing millions of instances. Zero-shot and distillation baselines contextualize agent performance, while budget-relaxation experiments help distinguish resource-management failures from other causes of underperformance.

Critical weaknesses
  1. Reported dollar costs exclude local GPU and CPU expenses. Distillation baselines match token budgets without documented time-budget matching, weakening claims about end-to-end savings and fair resource-constrained comparisons between competing approaches.

  2. Only three tasks and two agent runs per model-task pair support broad capability comparisons. Missing agent-level uncertainty, paired analyses, and harness ablations make model rankings and general conclusions insufficiently robust.

Questions for authors
  1. Report total costs, GPU-hours, CPU-hours, and wall-clock times for agents and students. Do the savings and distillation comparisons persist under matched compute budgets and model-matched teachers across all three tasks?

  2. Can you provide additional independent runs, confidence intervals for model rankings, and paired zero-shot versus bottled comparisons on identical instances? How sensitive are conclusions to harness configuration and resource budgets?

11 comments
AuthorsOpenReview ↗

(1/6) Response to Reviewer aGgp

Thank you for your engagement with our work and for asking interesting questions. Please note that we added a third independent bottling run per model-task pair (90 runs in total, in response W2.2) and our main claims remain unchanged; all our answers below reflect this updated suite of runs (90 main runs vs. the original 60).

W1 Cost accounting

W1.1: Reported dollar costs exclude local GPU and CPU expenses; do end-to-end savings persist under matched resources?

Yes, end-to-end savings from bottling persist even after accounting for local compute (GPU and CPU): across our 10 LLMs, bottling remains between 32 and 5,197 times cheaper than zero-shot labeling. To confirm, we recomputed the mean cost of bottling per LLM after considering local compute expenses.

Setup. Bᴏᴛᴛʟᴇᴅ provides, to each agent, a single A100 40GB GPU, along with 8 CPU cores and 64GB RAM on our local infrastructure. We charge this hardware at a representative on-demand VM price: $1.99/h for an A100 40GB instance [6]. This is a realistic upper bound on our compute cost: the Lambda instance offers close to 4× our CPU cores and 3× our RAM; hence the savings below are conservative.

For each LLM, we computed the total bottling cost as the sum of its (i) mean API cost and (ii) the hourly VM rate scaled by the mean wall time of a bottling run on that LLM. The total zero-shot cost is the projected API cost of labelling the full workload with the same LLM. As in our aggregate results (Figure 1), all values are averaged over the three tasks; for bottling, they are further averaged over three runs. Below, we report the results for the LLMs with the lowest and highest savings, respectively. We have added the results for all LLMs in the paper.

ModelZero-shot costBottled API costBottled wall-clockBottled local computeBottled totalZero-shot / bottled cost ratioZero-shot gainBottled gain
Lowest ratio: GLM 5.3 Flash$451$0.666.6 h$13.23$13.8932x0.5430.398
Highest ratio: GPT 5.6 Terra$43,507$5.281.6 h$3.09$8.375,197x0.5720.288

GLM 5.3 Flash, the LLM with the lowest zeroshot-to-bottled cost ratio, still costs 32x less when bottled than when run zero-shot (\13.89vs13.89 vs \\451) and achieves a bottled gain that is 73% of its zero-shot gain (0.398 bottled vs 0.543 zero-shot). For LLMs with higher API prices, savings can be much larger: GPT 5.6 Terra costs 5,197x less when bottled (\8.37vs8.37 vs \\43,507). Across all 10 LLMs, the median saving is 901x. Even after including local compute, bottling remains orders of magnitude cheaper than zero-shot labelling.

AuthorsOpenReview ↗

(2/6) Response to Reviewer aGgp

W1.2 Distillation baselines match token budgets without documented time-budget matching; do resource-constrained comparisons between competing approaches still preserve the conclusions?

Yes, our conclusion, that bottling is often outperformed by a naive distillation baseline, is preserved under matched time budgets. Confined to the agents' 600-minute budget, the better distilled student still beats 46 of 90 bottling runs.

Setup. The distillation uses the same hardware as the agents (a single A100 40GB GPU, 8 CPU cores and 64GB RAM) and the same API budget (5 Mtok, spent on GLM 5.3 Flash teacher labels). For each task and student, we combine the wall clock times for teacher labelling, student training, and student inference over the full workload, averaged over three runs. For a given task, we report the proportion of the 30 bottling runs of that task that were outperformed by the student. On RAID, the full distillation pipeline slightly exceeds 600 minutes; for fair comparison, we additionally include the score of the distillation pipeline if it had been stopped at 600 minutes. Instances assumed missing in this case are scored as trivial answers (on RAID, i.e., AI detection, this amounts to a 50% predicted probability that the document was AI-written).

TaskMethodAPI budgetTeacher labelling (minutes)Student training (minutes)Student inference (minutes)Total (minutes)Bottled runs beaten (of 30)
MAVEStudent: Qwen3-0.6B5 Mtok463829337718
MAVEStudent: SmolLM2-360M5 Mtok461828534921
ESCIStudent: Qwen3-0.6B5 Mtok184616823215
ESCIStudent: SmolLM2-360M5 Mtok18271682138
RAIDStudent: Qwen3-0.6B5 Mtok173759765110
RAIDStudent: Qwen3-0.6B, stopped at 600 min5 Mtok173754660010
RAIDStudent: SmolLM2-360M5 Mtok17235956358
RAIDStudent: SmolLM2-360M, stopped at 600 min5 Mtok17235606008
AllAgent (bottling)5 Mtok–––600 (budget)–

Results. The distillation baseline still outperforms 46 of 90 bottling runs under matched time budgets. On MAVE and ESCI, the full pipeline finishes in 213 to 377 minutes, well within the agents' 600-minute budget. On RAID, the distillation pipeline slightly exceeds the budget; however, by the 600th minute, the students had already labeled over 90% of the workload; even assuming that the remaining answers are missing, the students beat the same number of bottling runs as when run to completion (10 and 8 of 30, respectively). Time budgets therefore do not confound our conclusion.

AuthorsOpenReview ↗

(3/6) Response to Reviewer aGgp

W2 Robustness of conclusions

W2.1: Task variety

We strive to cover the most interesting use cases of LLMs on large repetitive workloads, and our capability comparisons extend to a fourth task beyond classification/extraction, which we add below.

We consider TL;DR generation from 3.8 million Reddit posts [1] and evaluate all 10 models under both zeroshot and bottled setups.

Setup. Similar to our original experiments, we evaluate each LLM under both zero-shot and bottling modes. Under zero-shot mode, we ask the LLM to summarize 1000 posts, one at a time. Under the bottling mode, we ask the LLM agent to summarize all 3.8 million posts under a budget of 10 hours and 5 million tokens. We evaluate the answers using “win rate”, i.e., how often the “system one” model Jev [2] prefers the answer's summary to the author’s summary [5]. Because evaluating all 3.8M samples using Jev is infeasible, we evaluate the same 1000 summaries for both zero-shot and bottled solutions, and compare their 95% CIs over 2000 resamplings. We report the results below:

ModelZero-shot win rate [95% CI]Bottled win rate [95% CI]
Qwen 3.8 Max0.996 [0.992, 0.999]0.731 [0.703, 0.757]
Sonnet 50.997 [0.993, 1.000]0.730 [0.702, 0.755]
GPT 5.6 Terra0.997 [0.993, 1.000]0.705 [0.677, 0.734]
Qwen 3.8 Flash0.995 [0.990, 0.999]0.660 [0.630, 0.690]
GLM 5.3 Flash0.997 [0.993, 1.000]0.640 [0.611, 0.670]
Opus 50.998 [0.995, 1.000]0.625 [0.596, 0.654]
GLM 5.30.995 [0.990, 0.999]0.501 [0.470, 0.530]
Gemini 3.1 Pro0.990 [0.983, 0.995]0.460 [0.430, 0.491]
GPT 5.6 Sol0.999 [0.997, 1.000]0.039 [0.028, 0.051]
Gemini 3.1 Flash-Lite0.988 [0.980, 0.994]0.037 [0.026, 0.049]

On TL;DR, bottling reveals capability differences that zero-shot evaluation generally does not. All ten LLMs are almost indistinguishable zero-shot, with win rates between 0.988 and 0.999. On the other hand, no bottled CI overlaps with its corresponding zero-shot CI; bottled win rates range from 0.037 to 0.731 (a range of 0.694). The gap between zero-shot and bottling capability, observed in our main experiments on MAVE, ESCI and RAID, persists on open-ended generation.

We also stress that our original 3 tasks correspond directly to industrial use cases. TypeSafe AI introduces its recently released Jev [2] as "a new class of frontier models built to make fast, structured decisions" for use cases such as "map-reducing over big data", i.e., making the same decision for every item of a large dataset. [2] lists these decisions as "classify, route, score, extract, or branch". Our tasks cover three of the five: MAVE extracts an attribute's value from a product listing (or abstains), ESCI classifies a query–product pair into one of four labels, and RAID scores a text with a continuous output. ESCI and RAID also correspond directly to two of Jev's three question types for text [3]: ESCI is a choice question over 4 labels, and RAID is a yes/no (noul) question. Our tasks represent a class of use cases important enough to warrant a frontier model.

W2.2 Only two agent runs per model-task pair

Following the reviewer’s request, we added a third independent run per task-model pair; this run preserves our central claim that bottling consistently falls short of zero-shot model competence; please see W2.3.

W2.3 agent-level uncertainty

Even after accounting for agent-level uncertainty (run-to-run variation), our main claim (there exists a consistent zeroshot-bottling gap) remains preserved. We conducted three independent bottling runs for every model-task pair (90 runs in total). Below we report each LLM’s bottled relative gain (aggregated over tasks) as the mean ± standard deviation across runs.

ModelZero-shot gain [95% CI]Bottled gain (mean ± std over 3 runs)
Opus 50.606 [0.544, 0.657]0.394 ± 0.140
Gemini 3.1 Pro0.586 [0.534, 0.630]0.281 ± 0.184
GLM 5.30.584 [0.542, 0.622]0.271 ± 0.028
GPT 5.6 Sol0.573 [0.515, 0.620]0.149 ± 0.164
GPT 5.6 Terra0.572 [0.508, 0.622]0.288 ± 0.104
Qwen 3.8 Max0.568 [0.505, 0.620]0.361 ± 0.032
GLM 5.3 Flash0.543 [0.487, 0.589]0.398 ± 0.021
Sonnet 50.541 [0.473, 0.597]0.212 ± 0.096
Qwen 3.8 Flash0.539 [0.477, 0.591]0.266 ± 0.036
Gemini 3.1 Flash-Lite0.473 [0.397, 0.538]0.061 ± 0.017

For 10/10 LLMs, the mean bottled gain lies more than one standard deviation below the lower end of the corresponding zero-shot 95% CI. Only Opus 5 comes close (0.394 + 0.140 = 0.534 vs. 0.544) but still falls slightly short. This indicates that while individual runs can vary a lot, bottling consistently falls short of zeroshot competence.

Reviewer_3HrwOpenReview ↗
Rating
3(+2)
Confidence
3(+1)
Informativeness
2
Presentation
3
Soundness
2(+1)
Read review
Paper summary

The authors investigate the problem of how well an LLM agent can provide a solution script to solve some specific problem (on MAVE, ESCI and RAID datasets, all are ). They compare it with directly as LLMs themselves (zero-shot) and compare these two results, calling this "bottling".

Authors find that, good zero-shot performance not necessarily lead to good bottling performance. Moreover, some bottles perform worse than small LLMs finetuned with glm-5.3-flash answers.

Interestingly, though these are two different tasks, authors find bottled performance are always worse than zero-shot performance, though sometimes they can be quite similar.

This work can provide an interesting setting on capability bottling and LLM-driven researches in these aspects.

Critical strengths
  1. Clear Presentation and Experiment Settings. The paper is generally clear and easy to follow. The experiment settings are tidy, sound and clear.

  2. The idea to let LLM agents wrap their ability into bottles are interesting, testing whether LLMs are capable of producing small artifacts for a certain problem.

Critical weaknesses

I am mainly concerned about (1) analysis and (2) coverage on "bottles" are limited. This might be caused by authors use opencode default as agent harness to test all LLMs, and use one same prompt and token budget to let LLM agents create "bottles" for each problem.

I am interested (and I believe the community would be interested) to see like: (1) performance of more kinds of "bottles" (e.g. fixed program scripts, trained small LLMs, scripts to call their own), (2) how and why certain LLMs choose to use that specific kind of "bottles".

However, in the paper, (1) there is no specific presentation on what bottle each LLMs choose, (2) the one same prompt and token budget may lead to different LLMs use only limited kinds of "bottle" (as presented in footnote 2 in page 4, authors claim that " cexec always turns out to be zero, as models always build a self-contained artifact", though authors explicitly allow LLMs to call their own apis to execute tasks.

To me, I would like to see authors provide more detailed reports and analysis on how each bottle types perform and why agents choose to encapsulate themselves into those kind of bottle types. For example, if we have different inference-time budget limits, would different bottles be optimal in different inference-time budgets?

This leads to some other minor concerns. For example, I think research-time budget and inference-time budget are not the same right? Research-time budget requires LLM agents to research and come up with a good solution, while inference-time budget is limitation to that solution. They are different restrictions. (Of-course you may add them up and use one total budget, but that sounds a bit wierd anyway). For different inference budgets, are there different best bottles? For different research budgets, how would different LLMs come up with different bottles? There should be a 2D plane of ResearchBudget X InferenceBudget X Score For Each LLM and Each Dataset. Or some 1D curve of Budget X Score for Each LLM and Each Dataset could provide some information. However, for now, only one point of Budget X Score for Each LLM and Each Dataset is presented, and there are no analysis on bottles chosen w.r.t. budgets/prompts/LLMs/harness/etc.

Now, I am unclear what kind of bottles each LLM choose and why.

Update: authors have solved many of the above concerns during rebuttal.

Questions for authors

Please see weakness. Could you provide further analysis on what kind of bottles each LLMs choose and why? Reading the intro and method section, I feel authors' ambition to encourage LLMs to research and investigate like routing, fixed scripts, small LLM distillation, call themselves, etc., a wide range of bottles, but later parts in the paper there are very little analysis on them.

Overall this is a well-presented interesting paper, and I'll consider to raise my score.

9 comments
AuthorsOpenReview ↗

(1/4) Response to Reviewer 3Hrw

Thank you for your engagement with our work. Please note that we added a third independent bottling run per model-task pair (90 runs in total, cf. our response to Reviewer zVZR's W4.2) and our main claims remain unchanged; all our answers below reflect this updated suite of runs (90 main runs vs. the original 60).

W1 Analysis and coverage on “bottles”

W1.1 What bottle types does each LLM build?

Across our main experiments (3 tasks, 10 LLMs, 3 runs each), agents choose to build a variety of bottle types. For our analysis, we divide bottle types into:

  • Rules: scripts without learned components.
  • Non-generative ML: hand-built features or a non-generative model trained by the agent, e.g. a classifier on DeBERTa embeddings.
  • Pretrained generative model: a generative language model deployed without finetuning.
  • Finetuned generative model: a generative language model finetuned by the agent.
  • Task model: a model built for the task (but not trained on benchmark dataset itself), fetched from the web.
  • API calls: the agent's LLM answering instances using its API.
  • Other: none of the above.

We labelled each run using an agentic judge (GLM 5.3 Flash) that analyzes the full trace, identifying the bottle type that produced most of the answers; for runs without output, it identifies the bottle that the agent was last building. This judge agreed with 15/15 of our manual labels. We show the breakdown of its labels across LLMs below:

LLMRulesNon-generative MLPretrained generative modelFinetuned generative modelTask modelAPI calls*Other
GLM 5.32421———
GLM 5.3 Flash—621———
GPT 5.6 Sol151——11
GPT 5.6 Terra33——12—
Gemini 3.1 Flash-Lite33——12—
Gemini 3.1 Pro—43—2——
Opus 5—8—1———
Qwen 3.8 Flash36—————
Qwen 3.8 Max2511———
Sonnet 5115—2——
All LLMs1545144651

(*Runs labeled "API calls" wrote only <0.01% of answers with local tools (write or apply_patch); judge considered "API calls" the closest fitting label.)

Agents chose a diverse set of bottles. Half our runs (45/90) chose a non-generative ML solution; quite many chose rules-based scripts and pretrained generative models instead (15, 14 respectively); remaining types (finetuned generative model and task model) also appeared in some runs (10/90). Interestingly, even two LLMs from the same model family can differ substantially in bottle choice: Opus 5 chose non-generative ML in 8/9 runs, while Sonnet 5 deployed a pretrained generative model in 5/9. This provides initial evidence that Bᴏᴛᴛʟᴇᴅ elicits a broad range of bottle types even under a fixed harness and prompt; we substantiate this further below.

W1.2 Does the fixed harness/prompt/LLM/budget limit the coverage over different kinds of bottles?

We find that it does not. Different bottle types appear across our 90 runs (W1.1); further, even the same LLM on the same task can choose different bottles across independent runs. To confirm, we list bottles chosen by each of three runs of every LLM–task pair (R = rules, N = non-generative ML, P = pretrained generative model, F = finetuned generative model, T = task model, A = API calls, O = other):

LLMMAVEESCIRAID
GLM 5.3R, R, PP, F, NN, N, N
GLM 5.3 FlashF, P, NN, N, PN, N, N
GPT 5.6 SolN, P, RA, N, NO, N, N
GPT 5.6 TerraR, R, AR, A, TN, N, N
Gemini 3.1 Flash-LiteR, R, TN, A, AN, N, R
Gemini 3.1 ProN, P, PN, P, NN, T, T
Opus 5F, N, NN, N, NN, N, N
Qwen 3.8 FlashR, R, RN, N, NN, N, N
Qwen 3.8 MaxR, R, FP, N, NN, N, N
Sonnet 5P, P, RP, P, PN, T, T

Even with the task, prompt, harness, budgets and LLM all fixed, the bottle often changes from run to run: in 20/30 LLM–task pairs, atleast 2 unique bottles were chosen over 3 independent runs. Concrete examples with quotes from three runs of GLM 5.3 on ESCI (P, F, N in the table):

Pretrained generative model (macro-F1: 0.442): "Plan: local vLLM inference on the A100 with a compact instruct model."

Finetuned generative model (macro F1: 0.551). "[...] fine-tune on GLM labels; the fine-tuned model may approach GLM accuracy [...] So fine-tune path is good."

Non-generative ML (macro-F1: 0.550). "[...] semantically-similar retrieval approach: embed queries and products and compute similarity + train classifier. Cross-encoder is simpler and better. Go with cross-encoder."

Notably, all 3 bottle types improve substantally (macro F1 0.442-0.551) over the trivial macro-F1 of 0.198 on ESCI. Our setup does not restrict agents to limited kinds of bottles.

AuthorsOpenReview ↗

(2/4) Response to Reviewer 3Hrw

W1.3: cexecc_{\mathrm{exec}} always turns out to be zero, as models always build a self-contained artifact

We note that cexec=0c_{\mathrm{exec}} = 0 (i.e., the API cost of running the bottle/artifact is zero, line 215) only indicates that the bottles built by the LLMs tested by us did not call the LLM APIs at inference time; this does not limit the diversity/coverage over bottle types (W1.1, W1.2). Agents are well aware of their API and use it heavily for building the bottles themselves. As an example, one GLM 5.3 run on ESCI used its API to label thousands of samples to finetune a local generative model (W1.2).

One possible bottle design could indeed route a few hard instances to the LLM at inference time; current agents do not choose to do so. Bᴏᴛᴛʟᴇᴅ allows for, and is well-positioned to evaluate such hybrid bottles if future agents choose them. We have rephrased footnote 2 accordingly, "in our runs, no agent chose to call the LLM API at inference time".

W1.4: How do different bottle types perform?

We find that different bottle types can reach similar top scores, while runs of the same bottle type can still vary widely. Below, we report, for each task, the average, minimum and maximum score of each bottle type across all runs on that task:

BottleMAVE (F1)ESCI (macro F1)RAID (AUROC)
Rules0.278 [0.000, 0.453] (n=13)0.332 (n=1)0.604 (n=1)
Non-generative ML0.227 [0.000, 0.468] (n=5)0.389 [0.198, 0.550] (n=16)0.681 [0.500, 0.925] (n=24)
Pretrained generative model0.213 [0.001, 0.368] (n=7)0.379 [0.294, 0.442] (n=7)—
Finetuned generative model0.360 [0.208, 0.455] (n=3)0.551 (n=1)—
Task model0.034 (n=1)0.394 (n=1)0.703 [0.606, 0.902] (n=4)
API calls0.000 (n=1)0.198 [0.198, 0.198] (n=4)—
Other——0.500 (n=1)

First, there exist multiple best candidate bottle types for a given task. On MAVE, the best rule-based script (F1 = 0.453) is close to the best non-generative ML solution (F1 = 0.468); on ESCI, the finetuned generative model (macro F1 = 0.551) matches the best non-generative ML solution (macro F1 = 0.550). The most optimal bottle type is not known in advance; Bᴏᴛᴛʟᴇᴅ leaves this choice open to allow for more innovative solutions by future agents.

Second, while different bottle types can compete with each other, performance can also vary widely within any single bottle type. For example, runs choosing rule-based scripts deliver performances ranging from 0.000 to 0.453 F1 on MAVE, and those choosing non-generative ML solutions range from 0.500 (trivial) to 0.925 AUROC on RAID.

Bᴏᴛᴛʟᴇᴅ evaluates autonomous agents on both capabilities: choosing a suitable bottle, and building it well.

AuthorsOpenReview ↗

(3/4) Response to Reviewer 3Hrw

W1.5: Why/how do agents build certain bottles?

We find that (i) agents consider many bottle types but empirically test only a few, and (ii) the same stated reason, on the same task, leads to different bottling types across runs.

While it is difficult to determine the actual reasons driving an LLM's choices [1], we offer initial insights into the LLM agents' exploration. Each Bᴏᴛᴛʟᴇᴅ run's trace records the agent’s activity before arriving at a bottle type. We therefore use an agentic judge (GLM 5.3 Flash) to determine the following:

Extent of consideration: the number of bottle types (out of those in W1.1) the agent considered, i.e. mentioned as an option;

Extent of experimentation: the number of bottle types it tested, i.e. ran on at least a sample of the workload;

Basis of bottle choice: the primary reason given by the agent for its final choice of one bottle type over another:

  • time budget: e.g., the runtime of an option.
  • API budget: e.g., the token cost of an option.
  • Environment: resources offered/blocked by the agent's environment.
  • Data: e.g. regularities in the inputs, the difficulty of the task.
  • Other: a reason that fits none of the above.
  • Arbitrary: no particular reason given.
  • None: no bottle type chosen (we did not observe this case).

(i) How much do agents explore before choosing a bottle type?

To answer this, we quantify the agents' exploration process by extents of consideration vs experimentation below:

#(bottle types)#Runs that considered this many#Runs that tested this many
0-5
1541
21134
3+7410

Almost all agents consider many different bottle types before arriving at a solution. 74/90 runs (82%) considered, i.e., named in their reasoning traces, 3+ different bottle types (last row in the table above).

On the other hand, far fewer agents test a lot of different bottle types. Only 10/90 runs (11%) tested 3+ bottle types. In fact, 59 of the 74 runs (80%) that considered 3 or more bottle types, tested only 1 or 2. Most agents choose bottles largely through reasoning, with relatively less measurement.

(ii) Why, or on what basis do agents choose bottle types?

We answer this by aggregating the agents' stated basis (per the judge) for the bottle choice across our 90 runs.

Most agents justify their bottle with feasiblity under the API/time budget. In 71/90 runs (79%), the agent's stated basis for its bottle type is feasibility under the API budget (41/90) or the time budget (30/90). In contrast, only 8/90 runs (9%) base their choice mainly on observations from the data.

Next, we compare the agents' stated reasons with their actual choice of bottle. Below we report numbers of different bottle types chosen by agents that stated the API budget as the main reason for their bottle choice:

TaskRunsRulesNon-generative MLPretrained generative modelFinetuned generative modelTask model#(Bottle types) built
MAVE128—31—3
ESCI171115——3
RAID1219——23

While most agents cite budget as the main reason, the budget does not actually determine the bottle type. On each of the three tasks, different runs citing the API budget choose three different bottle types. Consider MAVE (data extraction) runs that stated API budget as their basis: 8/12 runs wrote rules-based scripts, 3 used a pretrained generative model, and 1 finetuned a generative model. We quote from two different Sonnet 5 MAVE runs that arrived at different bottle types through similar reasoning:

(1) Rules-based script: "The budget math rules out calling an LLM on each of the 4.7M rows, since even minimal prompts overshoot the roughly 5M token allowance by a wide margin. So the extraction has to lean on algorithmic methods, most likely matching value spans that already appear in the title or description text."

(2) Pretrained generative model: "With only a 5 million token budget across 4.7M lines, individual API calls per line aren't feasible, so I'm leaning toward using the GPU for local inference with a small open-source LLM downloaded from Hugging Face to handle extraction at scale."

Both agents' reasonings start off ruling out zero-shot API calls, but one reasons itself into an algorithmic approach ("matching value spans") while the other decides on a small language model ("small open-source LLM"). Agents may indeed reach very different bottle types on the same basis.

These findings are an initial view into agents' exploration processes, and we believe they add empirical strength to our study. Upon acceptance, we will release the full traces of all 90 bottling runs rendered as HTML pages, so that readers can audit them independently and gain further insights into agent behaviour.

Reviewer_zVZROpenReview ↗
Rating
3
Confidence
1
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces BOTTLED, a benchmark that asks whether an agent given a whole unlabeled workload (up to millions of instances) and a fixed token/compute/API budget can amortize its general ability into a reusable artifact—a small fine-tuned model, a rule program, or a hybrid—rather than calling the LLM once per instance. They run 10 models across three tasks (MAVE attribute extraction, ESCI query-product classification, RAID AI detection) and report both retained zero-shot performance and the dollar cost.

Critical strengths
  • The framing is genuinely useful: "bottling" as amortization under a budget is a real industrial question (RPA, document parsing, support triage) that most agent benchmarks ignore. The Pareto curve in Fig. 1 is a nice way to make the cost/quality tradeoff visible.
  • The accounting model (caching = 0.1× uncached output tokens, outbound MCP calls cost twice) is more honest than most agent harnesses, and the OpenCode-based common harness lets them isolate model effect from harness effect.
  • The headline finding—zero-shot F1 ≈ 0.44 for both GPT-5.6 Terra and Sonnet 5, but bottled F1 of 0.405 vs 0.154—is a clean, non-obvious result. It suggests benchmarking agents only on zero-shot single-instance tasks misses a whole axis of capability.
  • The ablation that disables budget termination (Fig. 3) is the right control: it shows the gap isn't just "agents burn budget and get cut off," though that does explain 21/60 runs.
Critical weaknesses
  • Three tasks is thin for a benchmark that wants to define a new axis. MAVE/ESCI/RAID are all classification/extraction; there's no open-ended generation, no multi-step tool-use task, no task where the artifact has to compose itself from libraries. The claim "bottling separates models" might just be "these three classification tasks happen to separate models."
  • The agent has 10 hours wall-clock + 5M tokens—that's a very specific budget. Sensitivity to budget scaling (1M? 50M?) is almost absent; the reader can't tell whether the winners win because they're better bottlers or because they plan better under this particular budget.
  • The "naive distillation baseline" (Qwen3-0.6B, SmolLM2-360M fine-tuned on 5M tokens of GLM-5.3-Flash outputs) is a strawman in places—no task-specific prompting, no curriculum, no filtering. Beating it by 29/60 runs is not that impressive; I'd want to see a reasonable distillation recipe.
  • The anonymized repo exists, but the paper doesn't say how reproducible the agent's exploration is (temperature, planning seeds). With only 2 runs per cell, 95% CIs are huge and several rankings inside Table 1 will not survive a third run.
Questions for authors

Q1. You only evaluate three tasks, and all three are classification / structured extraction. Can you motivate why this task family is representative of "repetitive workloads" generally—would a code-generation or open-ended writing task show the same zero-shot-vs-bottled disconnect? Q2. Table 1 shows GPT-5.6 Sol failed 4/6 runs (never produced out.txt), and 21/60 total runs were budget-terminated. How many of these failures are harness/interface bugs versus genuine model inability? A short failure-mode breakdown would help. Q3. With only 2 runs per cell and bootstrap CIs as wide as ±0.1 F1, several adjacent rankings in Table 1 (e.g., GLM-5.3 Flash vs Gemini 3.1 Pro on ESCI) will not survive a third run. Will you commit to releasing the raw trajectories so readers can re-run their own aggregation?

6 comments
AuthorsOpenReview ↗

(1/5) Response to Reviewer zVZR

Thank you for your recognition of our work. Below, we address the remaining questions. Please note that we added a third independent bottling run per model-task pair (90 runs in total, in response W4.2) and our main claims remain unchanged; our answers below reflect this updated suite of runs (90 main runs vs. the original 60).

W1 Task variety

W1.1: MAVE/ESCI/RAID are all classification/extraction; there’s no open-ended generation, no multi-step tool-use task, no task where the artifact has to compose itself from libraries.

While we instantiate Bᴏᴛᴛʟᴇᴅ with three tasks, we acknowledge that adding more tasks would be beneficial. As a step toward tasks beyond classification/extraction, we add TL;DR generation from Reddit posts [1]; this is close to open-ended generation. We evaluate all 10 models under both zeroshot and bottled setups.

Setup. Similar to our original experiments, we evaluate each LLM under both zero-shot and bottling modes. Under zero-shot mode, we ask the LLM to summarize 1000 posts, one at a time. Under the bottling mode, we ask the LLM agent to summarize all 3.8 million posts under a budget of 10 hours and 5 million tokens. We evaluate the summaries using “win rate”, i.e., how often the “system one” model Jev [2] prefers the model’s summary to the author’s original summary [5]. Because evaluating all 3.8 million samples using Jev is infeasible, we evaluate the same 1000 summaries for both zero-shot and bottled solutions, and compare their 95% CIs over 2000 samplings with replacement. We report the results below:

ModelZero-shot win rate [95% CI]Bottled win rate [95% CI]
Qwen 3.8 Max0.996 [0.992, 0.999]0.731 [0.703, 0.757]
Sonnet 50.997 [0.993, 1.000]0.730 [0.702, 0.755]
GPT 5.6 Terra0.997 [0.993, 1.000]0.705 [0.677, 0.734]
Qwen 3.8 Flash0.995 [0.990, 0.999]0.660 [0.630, 0.690]
GLM 5.3 Flash0.997 [0.993, 1.000]0.640 [0.611, 0.670]
Opus 50.998 [0.995, 1.000]0.625 [0.596, 0.654]
GLM 5.30.995 [0.990, 0.999]0.501 [0.470, 0.530]
Gemini 3.1 Pro0.990 [0.983, 0.995]0.460 [0.430, 0.491]
GPT 5.6 Sol0.999 [0.997, 1.000]0.039 [0.028, 0.051]
Gemini 3.1 Flash-Lite0.988 [0.980, 0.994]0.037 [0.026, 0.049]

Observations and conclusions.

(i) All LLMs achieve win rates of 98%+ against the authors’ references, i.e., they are competent at the zero-shot task.

(ii) No bottled solution’s win-rate CI overlaps with that of the corresponding zeroshot solution. Strikingly, Gemini 3.1 Flash-Lite builds a near-trivial solution (~3.7% win rate) despite being competent at the zeroshot task (98.8% win-rate). We conclude that the zeroshot-bottling gap persists for this task.

Tasks involving multi-step tool-use and the composition of libraries into artifacts are natural extensions for future work and we appreciate the reviewer’s suggestion.

W1.2: The claim “bottling separates models” might just be “these three classification tasks happen to separate models.”

We appreciate this concern. We emphasize that:

(i) Zeroshot competence at the tasks themselves does not separate models nearly as much as bottling does. In the table below, we report, for each task (including the newly added Reddit TL;DR), the range of the task's native metric across all models. Bottled performance spans much wider ranges than zero-shot performance: for instance, on ESCI, bottled macro F1 spans a range of 0.317 across models, more than four times the zero-shot range (0.077). Differences in zero-shot classification capability alone therefore cannot account for the differences in bottling performance across models.

TaskZero-shot range across modelsBottled range across models
MAVE (F1)0.0840.386
ESCI (macro F1)0.0770.317
RAID (AUROC)0.1450.283
TL;DR (win rate)0.0110.694

(ii) The original suite of 3 tasks is itself substantially diverse: MAVE requires a model to copy an attribute's value verbatim from a product listing or abstain, ESCI is four-way classification, and RAID requires a continuous output. ESCI and RAID alone match two of the three text-based question types [3] of the recently released Jev [2], which TypeSafe describes as "a new class of frontier models built to make fast, structured decisions": ESCI is akin to a choice question over four labels, and RAID is a yes/no (noul) question answered with a probability. ESCI and RAID therefore represent a class of tasks important enough to warrant a frontier model.

(iii) The newly added task of Reddit TL;DR generation validates our conclusion (bottling capability falls well below zeroshot capability and differs across models, cf. W1.1) beyond the family of classification/extraction tasks.

AuthorsOpenReview ↗

(2/5) Response to Reviewer zVZR

W2 Sensitivity to budget scaling

We fix a single budget of 5 million tokens to allow for fair comparison across models; to check whether this choice influences our conclusions, we repeated the bottled runs of a strong bottler (GLM 5.3 Flash) and a weak one (Gemini 3.1 Flash-Lite) on all three tasks at six further API budgets, from 0.5 to 50 Mtok (a 100x range).

GLM 5.3 Flash (zero-shot gain: 0.54)

API budget (Mtok)0.512.55102550
Relative gain0.000.120.440.400.310.460.43
Mtok used0.510.921.453.337.828.8726.40

Gemini 3.1 Flash-Lite (zero-shot gain: 0.47)

API budget (Mtok)0.512.55102550
Relative gain0.040.010.040.060.190.060.00
Mtok used0.200.120.220.610.520.320.53

(i) The strong bottler (GLM 5.3 Flash) does not perform substantially differently at larger budgets. Going from 2.5 to 50 Mtok, it spends 18× more tokens (from 1.45 to 26.40 Mtok), yet its gain stays comparable (0.44 at 2.5 Mtok, 0.43 at 50 Mtok). Its performance drops substantially only at budgets of 1 Mtok or lower, which are evidently not sufficient for our tasks.

(ii) The weak bottler (Gemini 3.1 Flash-Lite) does not lose because of the budget: it spends only 0.36 Mtok on average, even when given up to 50 Mtok. Its best gain at any budget (0.19 at 10 Mtok) is still lower than the strong bottler's gain at the same budget (0.31 at 10 Mtok).

(iii) 5 Mtok is a sensible budget for our workloads. It is small enough that an agent cannot simply label the workload zero-shot; at the same time, it allows a capable agent to bottle well: the strong bottler's gain already plateaus at 2.5 Mtok, half our budget.

This ablation adds further empirical validation to our choice of budget, and we thank the reviewer for suggesting it.

W3 Distillation baseline

W3.1: Distillation baseline is naive; beating it by 29/60 runs is not that impressive.

We clarify that the purpose of the distillation baseline is to measure how far an agent gets beyond a simple, untuned pipeline with the same API budget. We do not claim that bottling beats it; rather, we claim that bottling often fails to beat it (§4.1: "Bottling is often outperformed by a naive distillation baseline", line 363). Out of our original 60 runs, only 29 beat the better of the two distilled students (as the reviewer pointed out); even after adding a third run per model-task pair, only 44 out of 90 runs beat the better of the two distilled students.

(i) This baseline is deliberately weak, and therefore it is surprising that it still outperforms bottling in around half of the runs (46 of 90). A stronger recipe (curriculum, filtering, and so on) would score higher and beat even more bottling runs, providing even further evidence for our claim (also see W3.2).

(ii) The distillation baseline does indeed use task-specific prompting: its teacher labels are produced with exactly the same task prompt as our zero-shot baseline (Appendix B). Only the students' training is (deliberately) left untuned.

We revised §4.1 to state the purpose of this baseline explicitly, and we appreciate the opportunity to provide clarification.

AuthorsOpenReview ↗

(3/5) Response to Reviewer zVZR

W3.2: Reasonable distillation recipe

As argued in W3.1, a stronger distillation baseline can only strengthen our conclusion, since more bottling runs would fall below it. We confirm this on ESCI with a stronger recipe: using GLM 5.3 Flash labels and the same 5 Mtok budget as the original baseline, we fine-tune mDeBERTa-v3-base [4] as a classifier; we call this the "strong distillation baseline". We perform a learning-rate sweep and select the best checkpoint on held-out teacher labels. We present the results below.

Macro F1Bottled runs beaten (of 30)
GLM 5.3 Flash, zero-shot0.57330
GLM 5.3 Flash, bottled (mean of 3 runs)0.46223
Original distillation baseline (better student)0.36615
Strong distillation baseline0.50924

The strong distillation baseline outperforms 24 of the 30 bottling runs on ESCI, compared with the naive baseline that beats 15 out of 30 runs. This confirms our expectation that bottling would further underperform a stronger distillation baseline.

We note that the strong distillation baseline involves human-picked design decisions conditioned on the task, including the decision to perform distillation at all. We chose a multilingual encoder because ESCI is multilingual. We also chose to employ a classification head, and perform a learning-rate sweep for model selection. Bᴏᴛᴛʟᴇᴅ asks agents to make precisely these decisions autonomously; the strong baseline therefore serves as a human-engineered reference point. Notably, the strongest bottlers do reach/surpass it (e.g., one GLM 5.3 run at 0.551 vs the strong baseline at 0.509) but not often (only 6 out of 30 runs for ESCI), leaving considerable room for current agents to improve in general.

W4 Reproducibility of agents’ exploration

W4.1: Reproducibility of the agent’s exploration (temperature, planning seeds)

Agent exploration cannot be made deterministic for the models we evaluate: some providers reject non-default temperatures and expose no seed (e.g., Anthropic for Sonnet 5 and Opus 5 [6]), and others do not guarantee determinism even with a seed (e.g., OpenAI [7]). We therefore use provider defaults and support the robustness of our claims with multiple runs per agent (see W4.2). In addition, we strive for reproducibility via:

  • Pinned LLM API endpoints. Appendix A lists the exact models, pinning their release dates and a snapshot of the token prices considered in our results. Model providers are pinned to the original model developers.
  • Data sampling and shuffling. In the anonymized repository, we provide scripts to download and organize the datasets exactly as used in our evaluations, down to the exact seeds used for zero-shot sampling (under scripts/prepare_data/).
  • Harness configuration. We provide the exact prompts (under prompts/ in the repo) and Python environments (under requirements_agents_env.txt) given to agents.
  • Raw traces. In the camera-ready version, we will add the full agent trajectories and the artifacts generated by the agents as rendered webpages, so that readers can audit our experiments independently.

W4.2 With only 2 runs per cell, 95% CIs are huge and several rankings inside Table 1 will not survive a third run.

We have added a third independent run for every model–task pair; our main experiments now comprise 90 runs in total.

Agent trajectories are indeed highly stochastic, and our central claims do not depend on model rankings; we state this explicitly in the revision. The pair mentioned by the reviewer (GLM 5.3 Flash vs Gemini 3.1 Pro on ESCI) nevertheless retains its order after the third run, and the gap increases (0.472 vs 0.375 macro F1 with two runs, 0.462 vs 0.316 with three); on aggregate gain, 4 out of 45 model pair orderings flip after adding the third run.

Our central claim (§4.1: "Bottling usually falls well short of zero-shot performance") holds even under a deliberately strict test. The 95% bootstrap CIs belong to the zero-shot evaluation, which is performed on a 1,000-instance sample; bottled solutions are scored on the full workload. We count a bottled run as "falling short" only if it scores below the lower bound of its model's zero-shot CI. The wider the CI, the lower this bound, and the harder it is for a bottled run to count as falling short. We note that even under this strict test, 25 of the 30 new runs fall short (and 48 of 60 in the original two runs). Also, 20 of the 30 new runs (and 40 of the 60 original runs) fall below the CI of even the worst zero-shot model.

The third independent run therefore confirms our conclusions: while individual runs vary, bottling consistently falls short of zero-shot performance.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Thank you and closing remarks

We would like to thank the reviewers for their time and for constructive suggestions that strengthened our paper substantially.

We are pleased that the paper was unanimously well-received; all three reviewers support acceptance after the initial concerns from Reviewers 3Hrw and aGgp were resolved.

  • Reviewer zVZR recognized bottling as "a real industrial question". The reviewer's feedback motivated a fourth task (Reddit TL;DR, Appendix C), a budget ablation from 0.5 to 50 Mtok (Appendix D), and a third run per model–task pair (90 runs in total vs. 60 originally). We also clarified that the naive distillation baseline is deliberately weak: it tests whether agents beat an untuned pipeline with the same API budget. Our conclusion that agents often (46/90 runs) do not even beat this baseline, is robust precisely because it is a naive baseline (§4.1).

  • Reviewer 3Hrw found the experiment settings "tidy, sound and clear", and recognized our analyses of the artifacts that agents build and their exploration processes as "important, good and inspiring results to show" (§4.3, Figure 4; Appendices H, I). We especially appreciate the reviewer's curiosity and extended engagement with our work.

  • Reviewer aGgp commended the evaluation setting as "practically important". Following the reviewer's questions, we showed that our conclusions hold under local-compute-inclusive costs (Appendix G), time- and model-matched distillation (Appendices B.2, B.3 respectively), a second harness (Appendix E) and uncertainty analyses (Appendix F).

Bᴏᴛᴛʟᴇᴅ provides a protocol to track how future frontier models (generalists) can autonomously build cheap, specialized solutions tailored to given workloads. Upon acceptance, we will link a webpage containing the full traces of our agent runs, so that interested readers can independently audit the behavior of current agents on our tasks.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.