- Rating
- 3(+1)
- Confidence
- 1
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
AgentLance assigns tasks to LLM agents through a repeated market with private execution costs. Workers decide whether and how much to bid. The platform estimates their success probabilities from reputation and uses a quality-adjusted allocation and VCG-style payment rule. Winners can subcontract complex tasks. Experiments compare the system with fixed models, centralized routers, and other market designs.
Critical strengths
- The system combines participation, bidding, reputation, and subcontracting in a concrete setting. The routing baselines and component ablations help explain how these choices affect performance.
- The cost-estimation experiments show that enforcing cost-based bidding alone can reduce performance, while correcting both estimates and bids improves it. The result shows that the benefit of cost-based bidding depends on the accuracy of the estimates.
Critical weaknesses
- For agent orchestration, the distinctive contribution of bidding is not sufficiently isolated. The system also changes worker reasoning, reflective notes, reputation-based success prediction, and subcontracting. The centralized baselines already receive cost information. Without subcontracting, AgentLance averages 52.2, compared with 51.6 for CARROT; the larger gain also changes how tasks are executed. The experiments therefore leave unclear what additional allocation capability comes from workers' cost reports, beyond the other changes to the orchestration system.
- The economic interpretation then depends on the cost-accounting boundary. Section 4.1 counts task execution cost but excludes the computation used for bidding, estimation, reflection, and delegation. This is a stated limitation of the prototype. However, the claim that the overhead becomes less important for complex tasks is not quantified. Given the modest allocation gain without subcontracting, the current comparison does not establish whether the market retains a net advantage over cost-aware routing when its operation is included.
Questions for authors
- How much of the reported gain comes from cost information supplied by workers, rather than reputation-based prediction and subcontracting?
- Does the advantage over cost-aware routing persist when the computation needed to operate the market is included?
2 comments
Weakness 2 & Question 2: Market communication overhead
Does the advantage over cost-aware routing persist when the computation needed to operate the market is included?
Our work aims to provide a proof of concept for market-based task allocation, so we focus on relatively simple tasks within our experimental budget. We excluded communication overhead from the original evaluation because our focus was on the allocation mechanism. Since communication cost does not scale proportionally with task execution cost, we expect its relative contribution to decrease for more complex and expensive tasks.
In the table below we report results including communication overhead in the cost calculation. The best result among all methods is in bold, and the second-best is italicized. AgentLance with subcontracting retains the highest average score across all methods. We will add these results to our paper.
| Method | α = 1 | α = 2 | α = 5 | α = 10 | Average |
|---|---|---|---|---|---|
| GPT-5 nano | 44.6 | 44.1 | 42.8 | 40.6 | 43.0 |
| Grok 4.3 | 56.3 | 54.5 | 49.3 | 40.5 | 50.1 |
| Gemini 3 Flash | 56.5 | 52.9 | 42.3 | 24.6 | 44.1 |
| DeepSeek-V3.2 | 52.1 | 50.1 | 44.3 | 34.7 | 45.3 |
| Best Single Model (per α) | 56.5 | 54.5 | 49.3 | 40.6 | 50.1 |
| Centralized LLM Planner | 54.9 | 53.9 | 47.1 | 39.8 | 48.9 |
| IRT-Router | 52.6 | 50.1 | 44.3 | 34.7 | 45.4 |
| CARROT | 55.9 | 54.4 | 50.0 | 46.1 | 51.6 |
| MarketBench (direct) | 55.1 | 51.2 | 39.7 | 20.8 | 41.7 |
| MarketBench (self-knowledge) | 49.5 | 48.2 | 45.1 | 39.7 | 45.6 |
| AgentLance w/o subcontract | 55.9 | 54.5 | 49.5 | 41.4 | 50.3 |
| AgentLance w/ subcontract | 57.1 | 55.2 | 52.3 | 43.6 | 52.1 |
Thank you for the helpful comments!
Weakness 1 & Question 1: Isolating the contribution of bidding
We added a new control to separate the contributions of bids and reputation. In the reputation-only condition, we retain the allocator and reputation information but set all bids to zero, so allocation depends solely on estimated success probabilities. We compare this with random assignment, AgentLance’s worker-generated bids, and bids set to realized execution costs. All results are averaged over 3 runs.
Reputation and bids each contribute, and they are complementary. Reputation improves accuracy and raises the score by about 4 points over random assignment for α≤5. Bids reduce execution cost, and their contribution grows as cost matters more, adding a further 2.7 and 9.0 points on top of reputation at α=5 and α=10. With both, AgentLance is 10.5 points above random assignment at α=10, compared with 1.5 points for reputation alone. With more accurate bids, the score improves further.
The contributions of the other components are ablated in the paper. Table 2 in Section 4.4 shows that removing reflective notes lowers the average score from 52.2 to 48.6, and replacing the VCG-style payment with pay-as-bid lowers it by 2.5 points. Table 1 in Section 4.3 shows that adding subcontracting raises the average score from 52.2 to 54.2, and on the tasks where subcontracting is invoked it improves accuracy by 27.3% and reduces cost by 64.1%.
We will include the new control and clarify the respective contributions of these components in the revision.