acceptodds
Under review as a conference paper at ICLR 2027

Markets, Not Planners: Decentralized Orchestration of LLM Agents with Private Information

Abstract

As LLM agents proliferate, built by different parties and with different capabilities and costs, orchestrating them is more like assembling labor across the economy than a computer calling a subroutine. Existing orchestration is typically centralized, with a single planner assigning every task, but this creates a bottleneck as agent pools grow, requires private information (e.g., agents' execution costs), and can easily be manipulated, such that a single inserted preference nearly doubles a favored agent's task share under a centralized LLM allocator. We introduce AgentLance, a repeated labor market with decentralized participation and bidding. Agents bid using private cost estimates and strategy notes, while the platform selects winners using bids and success probabilities predicted by an estimator. A VCG-style payment rule encourages cost-aware bidding. Complex tasks are handled by hierarchical delegation: winning agents can decompose work and subcontract it through the same mechanism. Across mathematical reasoning, code generation, knowledge-intensive QA, and agentic tasks, AgentLance matches agents to their specializations, shifts work toward cheaper agents as cost sensitivity rises, and consistently outperforms single-model, centralized-orchestration, and market baselines. Diagnosing market failures, including inaccurate cost self-estimation and suboptimal bidding, then correcting them in controlled experiments yields further gains, charting a path toward more efficient agent economies.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Multi-agent systems, agent orchestration, decentralized market, cost-aware task allocation

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_sYbvOpenReview ↗
Rating
3(+1)
Confidence
1
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

AgentLance assigns tasks to LLM agents through a repeated market with private execution costs. Workers decide whether and how much to bid. The platform estimates their success probabilities from reputation and uses a quality-adjusted allocation and VCG-style payment rule. Winners can subcontract complex tasks. Experiments compare the system with fixed models, centralized routers, and other market designs.

Critical strengths
  1. The system combines participation, bidding, reputation, and subcontracting in a concrete setting. The routing baselines and component ablations help explain how these choices affect performance.
  2. The cost-estimation experiments show that enforcing cost-based bidding alone can reduce performance, while correcting both estimates and bids improves it. The result shows that the benefit of cost-based bidding depends on the accuracy of the estimates.
Critical weaknesses
  1. For agent orchestration, the distinctive contribution of bidding is not sufficiently isolated. The system also changes worker reasoning, reflective notes, reputation-based success prediction, and subcontracting. The centralized baselines already receive cost information. Without subcontracting, AgentLance averages 52.2, compared with 51.6 for CARROT; the larger gain also changes how tasks are executed. The experiments therefore leave unclear what additional allocation capability comes from workers' cost reports, beyond the other changes to the orchestration system.
  2. The economic interpretation then depends on the cost-accounting boundary. Section 4.1 counts task execution cost but excludes the computation used for bidding, estimation, reflection, and delegation. This is a stated limitation of the prototype. However, the claim that the overhead becomes less important for complex tasks is not quantified. Given the modest allocation gain without subcontracting, the current comparison does not establish whether the market retains a net advantage over cost-aware routing when its operation is included.
Questions for authors
  1. How much of the reported gain comes from cost information supplied by workers, rather than reputation-based prediction and subcontracting?
  2. Does the advantage over cost-aware routing persist when the computation needed to operate the market is included?
2 comments

Thank you for the helpful comments!

Weakness 1 & Question 1: Isolating the contribution of bidding

How much of the reported gain comes from cost information supplied by workers, rather than reputation-based prediction and subcontracting?

We added a new control to separate the contributions of bids and reputation. In the reputation-only condition, we retain the allocator and reputation information but set all bids to zero, so allocation depends solely on estimated success probabilities. We compare this with random assignment, AgentLance’s worker-generated bids, and bids set to realized execution costs. All results are averaged over 3 runs.

Allocationα = 1α = 2α = 5α = 10Average
Random assignment52.350.444.735.145.6
Reputation only56.355.048.936.649.2
Reputation + bids (AgentLance w/o subcontract)56.355.351.645.652.2
Reputation + true-cost bids57.156.855.553.555.7

Reputation and bids each contribute, and they are complementary. Reputation improves accuracy and raises the score by about 4 points over random assignment for α≤5. Bids reduce execution cost, and their contribution grows as cost matters more, adding a further 2.7 and 9.0 points on top of reputation at α=5 and α=10. With both, AgentLance is 10.5 points above random assignment at α=10, compared with 1.5 points for reputation alone. With more accurate bids, the score improves further.

The contributions of the other components are ablated in the paper. Table 2 in Section 4.4 shows that removing reflective notes lowers the average score from 52.2 to 48.6, and replacing the VCG-style payment with pay-as-bid lowers it by 2.5 points. Table 1 in Section 4.3 shows that adding subcontracting raises the average score from 52.2 to 54.2, and on the tasks where subcontracting is invoked it improves accuracy by 27.3% and reduces cost by 64.1%.

We will include the new control and clarify the respective contributions of these components in the revision.

Weakness 2 & Question 2: Market communication overhead

Does the advantage over cost-aware routing persist when the computation needed to operate the market is included?

Our work aims to provide a proof of concept for market-based task allocation, so we focus on relatively simple tasks within our experimental budget. We excluded communication overhead from the original evaluation because our focus was on the allocation mechanism. Since communication cost does not scale proportionally with task execution cost, we expect its relative contribution to decrease for more complex and expensive tasks.

In the table below we report results including communication overhead in the cost calculation. The best result among all methods is in bold, and the second-best is italicized. AgentLance with subcontracting retains the highest average score across all methods. We will add these results to our paper.

Methodα = 1α = 2α = 5α = 10Average
GPT-5 nano44.644.142.840.643.0
Grok 4.356.354.549.340.550.1
Gemini 3 Flash56.552.942.324.644.1
DeepSeek-V3.252.150.144.334.745.3
Best Single Model (per α)56.554.549.340.650.1
Centralized LLM Planner54.953.947.139.848.9
IRT-Router52.650.144.334.745.4
CARROT55.954.450.046.151.6
MarketBench (direct)55.151.239.720.841.7
MarketBench (self-knowledge)49.548.245.139.745.6
AgentLance w/o subcontract55.954.549.541.450.3
AgentLance w/ subcontract57.155.252.343.652.1
Reviewer_byTUOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces AgentLance, which coordinates heterogeneous agents with private execution costs through bidding and reputation records, and supports subcontracting for complex tasks. The experiments cover a broad range of settings, but the reliability of the numerical predictions underlying allocation and the contribution of the market mechanism itself remain insufficiently established.

Critical strengths
  1. The paper addresses a meaningful problem. Modeling agent coordination under private costs as a market offers a useful perspective on task allocation among heterogeneous agents.

  2. The experimental coverage is broad. Evaluations across tasks and mechanism ablations provide useful observations, particularly on how cost estimation and bidding strategies affect each other.

Critical weaknesses
  1. The numerical estimates underlying allocation lack sufficient validation. AgentLance relies on prompted LLM estimates of execution costs and success probabilities, yet cost predictions exhibit substantial errors that historical information does not consistently reduce, while success probabilities lack calibration evidence. Prior work reports similar limitations in cost prediction and verbalized confidence (Bai et al., 2026; Xiong et al., 2024). Although Section 5.2 investigates cost estimation and bidding strategies through controlled interventions, the implementation details, particularly how accurate execution costs are obtained before task execution, remain insufficiently specified. These limitations raise concerns about the reliability of market allocation and the reproducibility of the reported improvements.

  2. The existing comparisons do not sufficiently isolate the contribution of the market mechanism itself. The centralized planner selects a single execution model, whereas AgentLance can decompose GAIA tasks through subcontracting. The MarketBench comparisons and mechanism ablations still leave an important question unanswered: how much does the market add over non-market orchestration that can also decompose tasks and use historical feedback? Moreover, the reported score excludes market communication overhead, so an end-to-end cost advantage has not been established.

[1] Xiong, M., Hu, Z., Lu, X., Li, Y., Fu, J., He, J., & Hooi, B. (2024). Can LLMs Express Their Uncertainty? An Empirical Evaluation of Confidence Elicitation in LLMs. The Twelfth International Conference on Learning Representations.

[2] Bai, L., Huang, Z., Wang, X., Sun, J., Mihalcea, R., Brynjolfsson, E., Pentland, A., & Pei, J. (2026). How Do AI Agents Spend Your Money? Analyzing and Predicting Token Consumption in Agentic Coding Tasks. arXiv preprint arXiv:2604.22750.

Questions for authors
  1. Given the substantial cost prediction errors, how reliably can the market identify lower-cost executors? Do the reported gains persist when errors increase or become systematically biased toward overestimation or underestimation? For long-horizon tasks, please also explain how cost and success-probability estimates account for tool failures and changes in the environment during execution.

  2. How much of the market's advantage remains when a non-market orchestrator uses the same model pool and tools, can also decompose tasks, receives the same historical feedback and budget, and is evaluated with the costs of all orchestration calls included?

8 comments

Thank you for the constructive comments!

Weakness 1: Reliability of market allocation

We clarify below how execution costs are obtained in the controlled experiments, how our findings relate to prior work, and why the concerns are solvable.

Implementation details. In the controlled experiments in Section 5.2, accurate costs are measured from actual task executions rather than predicted. We execute each model on each task in advance and cache the realized execution cost, which is then used as oracle information in the accurate-cost intervention. We will clarify this procedure in the revision.

Relationship to prior work. Our findings are consistent with the limitations identified by the reviewer. Xiong et al. (2024) [1] find that verbalized confidence is often overconfident, although prompting, sampling, and aggregation can mitigate this tendency. Bai et al. (2026) [2], already cited in Section 5.1, report weak-to-moderate correlations between models’ predicted and realized token consumption in agentic coding tasks. We agree that historical information does not consistently improve cost estimates in our experiments, and that allocation performance alone does not establish probability calibration. We will make these limitations explicit and expand the discussion of both studies.

Paths to improvement. Recent work shows that both kinds of estimates are improving quickly. On cost, Xie et al. (2026) [3] predict output length from internal model representations and reduce error by about 29% over the best baseline. On confidence, Tian et al. (2023) [4] show that suitable prompting substantially improves the calibration of verbalized confidence, and Li et al. (2026) [5] and Damani et al. (2026) [6] train models to output calibrated confidence. Notably, the new experiment under response to Question 1 shows that AgentLance remains effective even when estimation errors are large.

[1] Xiong M, Hu Z, Lu X, et al. Can llms express their uncertainty? an empirical evaluation of confidence elicitation in llms[C]//International Conference on Learning Representations. 2024, 2024: 23650-23678.

[2] Bai L, Huang Z, Wang X, et al. How do ai agents spend your money? analyzing and predicting token consumption in agentic coding tasks[J]. arXiv preprint arXiv:2604.22750, 2026.

[3] Xie H, Chen Y, Wang L, et al. Predicting LLM output length via entropy-guided representations[J]. arXiv preprint arXiv:2602.11812, 2026.

[4] Tian K, Mitchell E, Zhou A, et al. Just ask for calibration: Strategies for eliciting calibrated confidence scores from language models fine-tuned with human feedback[C]//Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing. 2023: 5433-5442.

[5] Li Y, Xiong M, Wu J, et al. Conftuner: Training large language models to express their confidence verbally[J]. Advances in Neural Information Processing Systems, 2026, 38: 53484-53513.

[6] Damani M, Puri I, Slocum S, et al. Beyond binary rewards: Training lms to reason about their uncertainty[C]//International conference on learning representations. 2026, 2026: 59456-59493.

Question 1

Given the substantial cost prediction errors, how reliably can the market identify lower-cost executors? Do the reported gains persist when errors increase or become systematically biased toward overestimation or underestimation? For long-horizon tasks, please also explain how cost and success-probability estimates account for tool failures and changes in the environment during execution.

We conducted an additional experiment to directly evaluate robustness to cost estimation errors. In the controlled setting of Section 5.2, we set each agent’s bid to its realized execution cost multiplied by log-normal noise, with σ controlling the noise level. At σ = 2, bids can differ substantially from actual costs. We also evaluated three additional conditions: uniform overestimation by a factor of 3, uniform underestimation to one-third of actual cost, and allocation without cost information. All results are averaged over three runs.

Cost reportα = 1α = 2α = 5α = 10
True cost57.1 ± 1.956.8 ± 0.955.5 ± 1.053.5 ± 1.7
Noise σ = 0.2557.9 ± 0.956.5 ± 0.854.6 ± 1.853.0 ± 2.3
Noise σ = 0.557.2 ± 1.056.9 ± 1.455.7 ± 1.253.1 ± 0.9
Noise σ = 157.5 ± 0.856.4 ± 1.955.0 ± 2.051.0 ± 2.8
Noise σ = 257.7 ± 0.557.2 ± 1.153.4 ± 1.348.5 ± 0.6
Overestimation ×357.9 ± 1.257.3 ± 0.856.4 ± 0.354.4 ± 0.6
Underestimation ×1/358.0 ± 0.656.3 ± 1.254.6 ± 1.450.6 ± 0.9
No cost information56.3 ± 1.155.0 ± 1.148.9 ± 1.436.6 ± 4.5

Robustness to estimation noise. The market is highly tolerant of cost estimation errors. For σ≤0.5, scores match those obtained with true costs. Even at σ=2, the score drops by only 2.0 and 5.0 points at α=5 and α=10, and remains 4.5 and 11.9 points above using no cost information.

Robustness to systematic bias. The market is also robust to systematic bias. Scores do not drop when all agents overestimate by 3×, and drop by only 2.9 points at α=10 when all agents underestimate to 1/3.

We also measured the actual bias of LLM bids. The median ratio of bid to true cost lies between 1.0 and 1.2, indicating no pronounced systematic bias overall.

Long-horizon tasks. Regarding long-horizon tasks, cost and success-probability estimates are made before execution, and payment is fixed at allocation and does not change with realized cost. The agent therefore bears the risk of cost overruns during execution, which encourages it to leave a margin in its bid. In addition, agents’ histories reflect realized costs and outcomes from tool-using runs, including recorded failures and retries, which can inform subsequent estimates.

Weakness 2

Isolating the contribution of the market mechanism

We evaluated the existing baselines using their original designs, therefore we did not add task decomposition into them. To distinguish the contribution of market allocation from that of subcontracting, we also report AgentLance without subcontracting in Table 1. This variant outperforms all the baselines, indicating that AgentLance’s advantage over baselines is not solely attributable to decomposition.

Following your suggestion, we additionally equipped the Centralized Planner with decomposition. We report this comparison in our response to Question 2.

Communication overhead

Our work aims to provide a proof of concept for market-based task allocation, so we focus on relatively simple tasks within our experimental budget. We excluded communication overhead from the original evaluation, anticipating that it would account for a smaller share of total cost as agents tackle increasingly complex and expensive tasks.

In the table below we report results including communication overhead in the cost calculation. The best result among all methods is in bold, and the second-best is italicized. AgentLance with subcontracting retains the highest average score across all methods. We will add these results to our paper.

Methodα = 1α = 2α = 5α = 10Average
GPT-5 nano44.644.142.840.643.0
Grok 4.356.354.549.340.550.1
Gemini 3 Flash56.552.942.324.644.1
DeepSeek-V3.252.150.144.334.745.3
Best Single Model (per α)56.554.549.340.650.1
Centralized LLM Planner54.953.947.139.848.9
IRT-Router52.650.144.334.745.4
CARROT55.954.450.046.151.6
MarketBench (direct)55.151.239.720.841.7
MarketBench (self-knowledge)49.548.245.139.745.6
AgentLance w/o subcontract55.954.549.541.450.3
AgentLance w/ subcontract57.155.252.343.652.1

Question 2

How much of the market's advantage remains when a non-market orchestrator uses the same model pool and tools, can also decompose tasks, receives the same historical feedback and budget, and is evaluated with the costs of all orchestration calls included?

Following your suggestion, we added a Centralized Planner with decomposition. The planner selects a manager for each task; the manager decides whether to decompose a GAIA task, and the planner assigns any resulting subtasks without bidding. This provides a non-market counterpart to AgentLance’s subcontracting workflow. Results are averaged over three runs.

Methodα = 1α = 2α = 5α = 10Avg
Centralized Planner55.154.348.242.049.9
Centralized Planner + decomposition58.656.254.852.155.4
AgentLance w/o subcontract56.355.351.645.652.2
AgentLance w/ subcontract57.656.254.848.454.2

Without access to any agent’s true cost, AgentLance performs on par with the Centralized Planner with decomposition for α≤5. The Centralized Planner achieves this performance with more information than AgentLance. It observes each agent’s historical realized execution costs, whereas in AgentLance costs are private to the agents and must be estimated by the agents themselves. At α=10 the score depends heavily on cost, and this information advantage puts the planner ahead. Table 3 in Section 5.2 supports this further, and we extract two of its rows below.

AgentLance w/o subcontract (from Table 3)α = 1α = 2α = 5α = 10Average
+ true cost55.354.453.550.453.4
+ true cost + cost-based bidding57.857.857.053.356.5

When agents in AgentLance know their true costs, AgentLance improves substantially even without decomposition, from 51.6 to 53.5 at α=5 and from 45.6 to 50.4 at α=10. With cost-based bidding added, it outperforms the Centralized Planner with decomposition for α≥2, by 1.2 to 2.2 points. This indicates that the gap at large α comes from cost information, and suggests that the market mechanism itself is advantageous when information is sufficient.

We’d also like to clarify that all multi-model methods use the same execution-model pool, and all methods have access to the same GAIA tools: bash, Python, and search. The single-model baselines use only their designated model, with the same tool access.

Reviewer_fgjKOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

Single AI systems are increasingly being composed into "multi-agent" workflows. But what are good ways to compose such workflows? The authors cleverly draw on market mechanisms to design a new orchestration scheme for AI agents: AgentLance. AgentLance lets agents engage in market behavior (like bidding for tasks, strategizing, subcontracting). They evaluate potential performance and cost gains in four benchmark evals.

Critical strengths

I think the authors do a good job of motivating their algorithm. I found the idea to draw on market mechanisms quite clever and generally well-explained.

I also appreciate the authors' deep dive in the results in Section 5 to better understand their approach at a granular level. I appreciate the authors being upfront on their limitations in this regard.

I also think looking at not just performance but cost was a good decision (though see below on need for more detail in the results).

More generally, I think the work could have a positive cross-disciplinary impact (bridging AI/ML, especially the multi-agent community, and econ / mechanism design).

Critical weaknesses

My biggest holdups, however, are on the results and their presentation. It was hard to grasp how significant the apparent boosts in accuracy actually are, but is that "meaningful"? There are no error bars or statistical analyses (to my reading?) reported, so it's hard to know if it is statistically significant. This also applies to the ablations: are there significant differences from removing each part (I know there are differences, but hard to tell if they "matter"!)

Moreover, the authors note that they looked at four benchmarks; however, it seems that all main results are reported aggregating over all benchmarks? Does performance and cost differ across benchmarks? (and what is the variance, e.g., reported with error bars, around such results?)

Understanding the above is crucial, in my opinion, for assessing the scientific contribution of the work.

Questions for authors
  • I'm curious how the authors see their work connected to some other orchestration work like LATTE https://arxiv.org/pdf/2605.06320? Which to my knowledge, has a DSL that lets it do things somewhat like subcontracting (agents can claim work)?

  • In Fig D.1, do the authors know why GPT-5-nano seems to go less than 0 for alpha 1 but not alpha levels? But then DeepSeek does for alpha 10? (again, error bars here would help to get a sense of variance!)

5 comments

Thank you for the insightful comments!

Weakness 1. Significance test

We estimated 95% confidence intervals for the mean score difference between AgentLance and each baseline, using 100,000 paired bootstrap resamples stratified by benchmark:

BaselineMean score difference (AgentLance − baseline) (%)95% CI
GPT-5 nano+11.2[7.38, 15.20]
Grok 4.3+4.1[0.63, 7.92]
Gemini 3 Flash+10.2[2.17, 20.78]
DeepSeek-V3.2+8.9[4.34, 13.66]
Centralized Planner+4.3[1.50, 7.40]
IRT+8.8[4.29, 13.40]
CARROT+2.6[−0.69, 6.22]
MarketBench direct+11.4[3.45, 21.92]
MarketBench self-knowledge+6.7[3.61, 9.80]

AgentLance achieves positive mean score differences against all nine baselines. Benchmark-stratified paired bootstrap 95% confidence intervals exclude zero for eight comparisons, including Centralized Planner and both MarketBench variants. Against CARROT, AgentLance achieves a positive mean improvement of 2.62 score points, although the 95% confidence interval (−0.69 to 6.22) includes zero.

We also estimated the confidence intervals for ablation variants:

MethodMean score difference (AgentLance w/o subcontract − ablation) (%)95% CI
w/o Reflective Notes+3.6[1.88, 5.40]
w/o VCG-style Payment+2.5[0.78, 4.30]

Both ablations show statistically significant score reductions, supporting the contribution of Reflective Notes and VCG-style Payment to AgentLance’s performance. We will update the paper to include confidence intervals.

Weakness 2. Benchmark-wise results

The tables below report benchmark-wise scores across four values of α. The best result among all methods is in bold, and the second-best is italicized.

α = 1

MethodOlympiadBenchBigCodeBenchSuperGPQAGAIAAverage95% CI (AgentLance − method)
GPT-5 nano63.8461.2038.5414.6544.56[8.63, 17.58]
Grok 4.384.7567.7158.2814.2656.25[−1.82, 4.48]
Gemini 3 Flash86.4167.8163.837.8156.46[−2.81, 5.36]
DeepSeek-V3.271.9362.6455.9617.7452.07[0.82, 10.35]
Centralized LLM Planner80.0766.0759.4014.9655.12[−1.31, 6.18]
IRT-Router73.1863.9557.2916.1752.65[0.34, 9.58]
CARROT84.8567.8563.837.1855.93[−1.79, 5.11]
MarketBench (direct)82.8867.3863.857.2155.33[−1.72, 6.54]
MarketBench (self-knowledge)74.8466.0744.2914.4649.91[3.78, 11.72]
AgentLance w/o subcontract84.7968.1957.9414.4456.34[−1.70, 4.17]
AgentLance w. subcontract83.5069.5460.6416.7057.60Reference

α = 2

MethodOlympiadBenchBigCodeBenchSuperGPQAGAIAAverage95% CI (AgentLance − method)
GPT-5 nano63.6861.0738.4113.3044.11[7.21, 17.06]
Grok 4.384.1667.4357.908.5354.50[−2.60, 6.10]
Gemini 3 Flash86.1567.6263.66−5.7252.93[−1.84, 9.17]
DeepSeek-V3.271.8562.6255.9310.1450.13[0.96, 11.10]
Centralized LLM Planner83.7065.8958.838.9654.34[−2.53, 6.24]
IRT-Router71.8562.6255.9310.1450.13[0.96, 11.10]
CARROT84.3767.7063.662.0254.44[−2.50, 6.06]
MarketBench (direct)82.6567.2163.70−6.4651.78[−0.72, 10.39]
MarketBench (self-knowledge)73.7065.9244.1412.5049.06[2.75, 11.70]
AgentLance w/o subcontract83.7768.4559.0210.0655.32[−3.01, 5.05]
AgentLance w. subcontract83.8063.5859.1118.2256.18Reference

α = 5

MethodOlympiadBenchBigCodeBenchSuperGPQAGAIAAverage95% CI (AgentLance − method)
GPT-5 nano63.2060.6838.029.2442.79[7.55, 16.70]
Grok 4.382.4066.5656.75−8.6849.26[1.39, 10.13]
Gemini 3 Flash85.3767.0663.15−46.3042.32[3.18, 24.63]
DeepSeek-V3.271.6362.5555.82−12.6544.34[5.10, 15.92]
Centralized LLM Planner81.7665.8758.30−13.0048.23[1.89, 11.61]
IRT-Router71.6362.5555.82−12.6544.34[5.10, 15.92]
CARROT82.9467.2663.15−13.2050.04[0.24, 9.55]
MarketBench (direct)81.9766.6863.24−47.9041.00[4.48, 25.95]
MarketBench (self-knowledge)71.2265.0645.417.4447.28[3.42, 11.79]
AgentLance w/o subcontract81.9167.7055.920.9551.62[−0.17, 6.77]
AgentLance w. subcontract82.2669.0257.9510.0854.83Reference

α = 10

MethodOlympiadBenchBigCodeBenchSuperGPQAGAIAAverage95% CI (AgentLance − method)
GPT-5 nano62.4060.0237.372.4940.57[2.18, 13.11]
Grok 4.379.4665.1354.83−37.3640.51[0.79, 15.66]
Gemini 3 Flash84.0866.1162.30−113.9424.64[6.46, 47.28]
DeepSeek-V3.271.2762.4355.64−50.6434.67[6.69, 20.82]
Centralized LLM Planner76.0865.2061.17−34.3142.03[0.25, 13.07]
IRT-Router71.2762.4355.64−50.6434.67[6.69, 20.82]
CARROT80.6066.5462.30−24.9946.11[−4.33, 9.51]
MarketBench (direct)81.2665.8262.48−115.7323.46[7.79, 48.43]
MarketBench (self-knowledge)70.0464.9346.01−5.0143.99[−1.20, 9.63]
AgentLance w/o subcontract79.7066.3357.20−20.8745.59[−3.65, 10.21]
AgentLance w. subcontract80.5766.6457.18−10.8748.38Reference

AgentLance with subcontracting achieves the highest average score across all four α settings, demonstrating strong performance across cost preferences. While some baselines excel on individual benchmarks at specific α values, AgentLance adapts task allocation to changing cost–performance trade-offs.

Question 1

I'm curious how the authors see their work connected to some other orchestration work like LATTE https://arxiv.org/pdf/2605.06320? Which to my knowledge, has a DSL that lets it do things somewhat like subcontracting (agents can claim work)?

Thank you for pointing us to this work. We briefly summarize the connection and difference here.

LATTE lets a team of agents maintain a shared task graph that evolves during execution, and idle agents can directly claim open nodes. This design is effective at improving team efficiency. Like AgentLance, it is decentralized in that agents themselves decide which part of the work to take on, rather than being assigned by a central controller.

The two differ in their assumptions. LATTE assumes cooperative agents that claim work on a first-come, first-served basis, without prices. AgentLance targets settings where agents have heterogeneous execution costs that are private information, which is common in practice, and therefore uses bids and a payment rule to decide who performs each task. The two approaches are complementary. One direction for future work is to use LATTE’s task graph to organize decomposed subtasks and to allocate its open nodes through bidding. We will include this in the discussion.

Question 2

In Fig D.1, do the authors know why GPT-5-nano seems to go less than 0 for alpha 1 but not alpha levels? But then DeepSeek does for alpha 10? (again, error bars here would help to get a sense of variance!)

Across three runs, GPT-5-nano’s final cumulative profit at α=1 was −$9.007 ± $5.466, while DeepSeek’s at α=10 was −$0.274 ± $0.234 (mean ± SD). Both were negative in all three runs.

These losses arise from different mechanisms. At α = 1, GPT-5-nano’s losses primarily stem from subcontracting payments exceeding its primary-task revenue: subtask budgets constrain submitted bids but do not cap the final critical payments. At α = 10, DeepSeek’s losses primarily result from execution costs exceeding payments received, reflecting the inaccurate cost estimates. We will clarify these mechanisms and add pointwise shading indicating ±1 SD across runs to Fig. D.1.

Dear Reviewer fgjK,

Thank you again for your constructive feedback and thoughtful comments. As the discussion phase is moving towards its end, we would be grateful to know whether our response has addressed your concerns. We would be glad to clarify any remaining questions!

Best, Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.