- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The authors consider the setting of a company of employees using AI agents to complete work tasks. They consider two different policies the employees could take in their AI usage: (1) token-max, where employees try to maximize the number of tokens they use in each task and (2) outcome-max, where employees try to maximize the output they produce per token used. They design a workplace simulation based on existing benchmarks like SWE-bench and MATH-500. In the simulation, a collection of employees (simulated by LLMs) interact with another LLM to solve benchmark tasks. The human-simulating LLM is instructed to either follow the token-max or outcome-max policies through prompting. The simulation also incorporates employee persona features, adaptive memories across rounds, and firing of low-performing employees plus mutation and replication of high-performing employees. The experiments show that the outcome-max prompt provides comparable (and sometimes higher) accuracy with fewer tokens than token-max.
Critical strengths
- The paper is generally well structured and mostly well-written, although there is room for improving clarity (see comments below).
- The appendix appears to contain everything needed to understand the details of the experiments (prompts, parameters, etc).
Critical weaknesses
The main issue with this submission is that it sets up a complex system to show something that is intuitively clear: if we maximize for token usage, then this is wasteful if it doesn't come with a corresponding increase in quality. I struggle to see what this paper provides beyond this observation. In particular:
- There is a mismatch between the complexity of the model and simulation considered and the simplicity / inevitability of the results: under any reasonable model, optimizing for successful outcomes per token should perform better than optimizing for maximum tokens. It's not clear what we learn by running a complex multi-agent simulation with adaptive behavior and evolution and then observing that yes, optimizing for value per cost provides better value than optimizing for maximum cost. Some of the results are then about details of the simulation (e.g., Fig 3b and Fig 6b about LLM employee personas and lineages), which do not provide generalizable insights. The monetary values chosen in the simulation appear arbitrary.
- The two theorems state facts that follow immediately from their assumptions and do not provide broader insights.
Questions for authors
- The most crucial question: what do we learn from the paper that (a) isn't clear already and (b) isn't about the details of the multi-agent simulation?
- Why does "reported token metric" include only input and output? are input tokens that are cache writes charged as both input and cache write? And are output tokens that are cache reads charged as both output and cache read?
- What does a "tactic library" contain?
- Where do the "employee lineages" in Figure 6 come from? And what are they? They don't seem to be explained in the text.
- How are fines implemented in the simulation?
Comments
- The design of the figures and tables is quite nice in general!
- The related work summaries are overly succinct, making the relevance to the present work unclear. E.g., "Reflexion adapts verbal memory"; not clear what this means or why it's relevant. Longer sentences elaborating more would help. A couple of sentences summarizing the literature in each area overall would also help, in addition to listing off individual papers.
- What is "employee acceptance"? and what does "a shipped defect is " mean?
- clarifying what kind of policy is would be helpful. Is it the policy of the employee choosing how to use AI?
- "token billing motivates the archetype without establishing that billing alone produces it" can be made clearer.
- is the idea behind making a function of that employees with higher fee-sharing will be more productive due to higher incentive? this is worth spelling out.
- Where does the baseline come from? the full utility for each party should be spelled out, as (4) seems to just define some components.
- Theorem 1 is pretty clear from the definitions: rewarding tokens only helps if more tokens means more value. Do we learn anything from it?
- Theorem 2 doesn't seem to be saying anything non-trivial: if the loss to employees equals the loss to companies, then of course they have identical minimizers; if , then the written functions are identical.
- Eq (6) defined , but what are and ? (ah, under the two policies -- this should be explicit)
- SWE-chat user personas should be explained around line 373
- The labels in Fig 5b are poorly placed -- it's hard to see which label is associated with which point and the text also overlaps data (and the error bars are truncated)
4 comments
Rebuttal (Part 2)
Comment fwJw-Q4: What is an employee lineage, and where does it come from?
Response: Lineage records simulated ancestry, not employee ability. L3 maintains 12 simulated employees. Each period, descendants of the top three replace the bottom three, with rule-governed changes to behavioral settings. Shared colors mark common ancestors and show which supervision styles survive the specified selection rule.
Comment fwJw-Q5: How are fines implemented?
Response: Fines are implemented via prompts and accounting, with no real money transfers. The prompt states penalties for defective delivery, while accounting calculates expected losses. Combined instructions do not isolate fine effects, and actual monetary responses require human evidence.
Minor Comments
Comment fwJw-C1: The related work is too brief, and the connection to Reflexion and similar work is unclear
Response: We link Reflexion to our text-based memory updates (L2), noting that both retain fixed model weights. We will include clearer related work.
Comment fwJw-C2: What are employee acceptance and a shipped defect?
Response: Acceptance ends the interaction; a shipped defect is an accepted incorrect result. With acceptance a and external-test correctness y, correct delivery is ay and defective delivery is a(1−y). Correct work can also be rejected. We will explain it more.
Comment fwJw-C3: Does π denote the employee's policy for using AI?
Response: π denotes the employee's AI interaction policy under common resource limits. It specifies feedback, checking, and acceptance. L1 fixes these rules; L2 updates supervisor or solver text rules separately and identifies the role being changed.
Comment fwJw-C4: What does “token billing motivates the archetype…” mean?
Response: Token billing makes computation measurable, motivating the study of usage as an input proxy. We do not claim billing alone causes the token-max archetype; actual human adoption requires separate behavioral evidence.
Comment fwJw-C5: Why does Q depend on φ? Does a higher share imply greater employee effort?
Response: φ is the provider's share, not the employee bonus. The general model allows Q to depend on φ, but the fixed-output accounts do not estimate that behavioral response. Increasing φ redistributes payoff from the firm to the provider; employee bonuses and output remain fixed in these accounts.
Comment fwJw-C6: What is the baseline, and how should each party's complete utility be written?
Response: With production artifacts and task value held fixed, the baseline uses token billing and fixed wages; OutcomeShare adds acceptance, sharing, and liability. Transfers like fees and fines cancel in the sum, but serving, effort, and defect costs remain in the full utility.
| Party | Net-return calculation |
|---|---|
| Provider | Production and verification revenue − cost of providing those model services |
| Employee | Wage + delivery bonus − effort cost − expected fine |
| Firm | Value of correct deliveries − production payments to the provider − wages and bonuses − verification fees − defect losses + fines received |
Comment fwJw-C7: Does Theorem 1 merely restate the definition?
Response: Theorem 1 states the threshold for improving efficiency. For expected correct output Y, cost C > 0, and added cost ΔC > 0, efficiency increases exactly when ΔY/ΔC > Y/C. The algebra is direct; the empirical difficulty is estimating ΔY before continuing.
Comment fwJw-C8: If the two losses in Theorem 2 are equal, does the theorem have substantive content?
Response: Theorem 2 specifies how to calibrate defect liability. With fixed audit probability p > 0, collectible fine F, and defect damage matches expected liability to damage. Under risk neutrality, other payoffs fixed in effort, and attained minima, the defect-related losses have identical minimizers.
Comment fwJw-C9: Which policy do the quantities in Eq. (6) describe?
Response: Output and tokens in the equation refer to the same policy. For task j, is external-test success and is execution-agent token use; . Subscripts tok and out distinguish the token-use and outcome policies. We report success and mean tokens alongside efficiency.
Comment fwJw-C10: SWE-chat user personas should be explained at first mention
Response: SWE-chat persona definitions describe observed human behavior, distinct from prescribed simulated styles. We explain these categories near first use and separate conversation scores from external correctness.
Comment fwJw-C11: Figure 5b has overlapping labels, unclear correspondence, and clipped error bars
Response: Will revise that.
Thanks for the replies!
I agree that the problem of deciding good token / AI budget allocations and policies is very important for companies. But I'm skeptical that the LLM simulations described here provide informative and generalizable takeaways beyond the (intuitively clear) statement that optimizing for impact / output is better than optimizing for tokens (and the also intuitively clear principles listed above). I think companies are doing all of these things already
Thanks for the thoughtful reply. However, I would respectfully disagree that companies are already implementing all of these practices (at least before we started doing this work). In reality, many companies are still primarily focused on cutting token budgets to avoid excessive consumption, and some even continue to use token usage as a key performance metric or leaderboar see many report:
- https://newsletter.semianalysis.com/p/tokenbudgeting-our-conversations?embedding_publication_id=8645961
- https://www.tcs.com/content/dam/global-tcs/en/pdfs/insights/global-studies/report/enterprise-ai-study-key-findings.pdf?
- https://www.theinformation.com/articles/tokenmaxxing-tide-may-turning
What is largely missing is a systematic framework for evaluating how effectively AI is being used and for designing policies that encourage better AI utilization. This is precisely what we aim to contribute in this paper. Beyond showing that optimizing for impact rather than token consumption is preferable, we propose a practical framework for guiding AI usage through appropriate evaluation rules and incentive mechanisms. For example, companies could establish
- Good task-specific evaluation criteria and define appropriate stopping rules
- Introduce mechanisms we proposed, such as sharing outcome-based revenue with employees or applying penalties for inefficient AI usage.
These mechanisms are intended to encourage employees to use AI more effectively rather than simply consume fewer tokens. Our goal is not merely to demonstrate an intuitive principle through LLM simulations, but to provide actionable policy guidance for companies on how to organize, evaluate, and incentivize AI usage. We believe the simulation results offer useful insights into how different policies shape behavior, and we are also conducting related experiments in real-world industry settings to further investigate their practical implications.
Rebuttal (Part 1)
We thank you for your valuable opinions and questions. All the responses have been updated in the revised PDF.
Major Comments
Comment fwJw-W1: Do the complex simulations yield only intuitive results? What is the general significance of lineages and monetary amounts?
Response: The contribution is to turn intuition into an empirical account of conditions and mechanisms. “More use need not be better” does not tell an enterprise which work wastes computation, whether reducing usage harms hard tasks, or when continuation is worthwhile. Our traces distinguish checking after a correct result without added benefit, edits that introduce regressions, and a compatibility question that repairs a defect.
These findings motivate designing continuation and stopping together. Enterprises can evaluate the marginal value of additional computation rather than use volume as evidence of effectiveness. Lineage colors track simulated ancestry; monetary scenarios compare payoffs under explicit value, cost, and sharing assumptions. They explain processes and application conditions, not actual employee ability or company profits.
The token-max problem is of critical importance to enterprises, as maximizing token utilization does not necessarily lead to better outcomes and may introduce inefficiencies or redundant work (We already got many invitations for this topic). We hope our work will encourage enterprises to pay closer attention to the underlying mechanism design challenges and reconsider how computational resources should be allocated to achieve more effective and reliable outcomes. We sincerely hope you will give careful consideration to the significance and contributions of our work.
Comment fwJw-W2: Do the two theorems follow directly from their assumptions?
Response: Theory states the decision conditions; experiments establish the observed consequences. The algebra is straightforward. The first proposition compares marginal computational return with current efficiency, but the additional correct output ΔY is unknown before continuing and requires empirical investigation. It does not supply an optimal stopping algorithm. The second matches expected liability to defect damage with other payoffs fixed; it does not prove full contract alignment. The principal contribution is the empirical characterization and its explanation. These conditions connect behavior, cost, and enterprise decisions.
Comment fwJw-Q1: What does the paper establish beyond common sense and simulation details?
Response: Four practical principles follow:
These principles follow from observed trade-offs; their combined effectiveness still needs validation with real tasks and personnel.
Comment fwJw-Q2: Why count only input and output? Are cache reads and writes charged twice?
Response: Token counts and billed cost are distinct. Cache reads/writes are input categories, separated before pricing to avoid double counting; output counts newly generated content. The original main table counts execution-agent input and output, whereas matched mathematics includes both roles. Some historical cache fields cannot be recovered consistently, so L3 prices recorded input/output. Token savings therefore do not imply equal billing savings.
Comment fwJw-Q3: What is the tactic library?
Response: Tactic libraries are text rules updated by task feedback. Supervisor and solver roles are updated separately. Joint rule learning is not automatically beneficial; we retain training and reflection costs, amortizing them over 27 tasks to evaluate net quality and cost.