acceptodds
Under review as a conference paper at ICLR 2027

From Token-Max to Outcome-Max: How You Use AI Determines Its Productivity

Abstract

Generative artificial intelligence (AI) models can perform increasingly complex tasks, yet greater AI usage does not necessarily translate into proportional productivity gains. We identify token-max as one source of this inefficiency: when token consumption is treated as productive effort, agents are encouraged to over-exert and expend computation beyond what is necessary. We instead propose outcome-max, which rewards independently verified task completion per unit cost and induces a principled stopping rule. Then, to study these objectives, we develop a three-level simulation framework spanning immediate interaction, long-run behavioral adaptation, and organizational collaboration. Across all three levels, outcome-max improves the efficiency of AI-assisted production while largely preserving verified task performance. To further align these incentives with outcome-max, we introduce OutcomeShare, an incentive mechanism. Theory and simulation show that OutcomeShare can induce participation while generating shared gains for employees, firms, and LLM providers. Together, our results suggest that AI productivity not just depends on model capability, but also on the objectives governing AI use.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
AI productivity, Mechanism design, Multi-agent simulation, Token Economics

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_fwJwOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
3
Soundness
3
Read review
Paper summary

The authors consider the setting of a company of employees using AI agents to complete work tasks. They consider two different policies the employees could take in their AI usage: (1) token-max, where employees try to maximize the number of tokens they use in each task and (2) outcome-max, where employees try to maximize the output they produce per token used. They design a workplace simulation based on existing benchmarks like SWE-bench and MATH-500. In the simulation, a collection of employees (simulated by LLMs) interact with another LLM to solve benchmark tasks. The human-simulating LLM is instructed to either follow the token-max or outcome-max policies through prompting. The simulation also incorporates employee persona features, adaptive memories across rounds, and firing of low-performing employees plus mutation and replication of high-performing employees. The experiments show that the outcome-max prompt provides comparable (and sometimes higher) accuracy with fewer tokens than token-max.

Critical strengths
  1. The paper is generally well structured and mostly well-written, although there is room for improving clarity (see comments below).
  2. The appendix appears to contain everything needed to understand the details of the experiments (prompts, parameters, etc).
Critical weaknesses

The main issue with this submission is that it sets up a complex system to show something that is intuitively clear: if we maximize for token usage, then this is wasteful if it doesn't come with a corresponding increase in quality. I struggle to see what this paper provides beyond this observation. In particular:

  1. There is a mismatch between the complexity of the model and simulation considered and the simplicity / inevitability of the results: under any reasonable model, optimizing for successful outcomes per token should perform better than optimizing for maximum tokens. It's not clear what we learn by running a complex multi-agent simulation with adaptive behavior and evolution and then observing that yes, optimizing for value per cost provides better value than optimizing for maximum cost. Some of the results are then about details of the simulation (e.g., Fig 3b and Fig 6b about LLM employee personas and lineages), which do not provide generalizable insights. The monetary values chosen in the simulation appear arbitrary.
  2. The two theorems state facts that follow immediately from their assumptions and do not provide broader insights.
Questions for authors
  1. The most crucial question: what do we learn from the paper that (a) isn't clear already and (b) isn't about the details of the multi-agent simulation?
  2. Why does "reported token metric" include only input and output? are input tokens that are cache writes charged as both input and cache write? And are output tokens that are cache reads charged as both output and cache read?
  3. What does a "tactic library" contain?
  4. Where do the "employee lineages" in Figure 6 come from? And what are they? They don't seem to be explained in the text.
  5. How are fines implemented in the simulation?

Comments

  • The design of the figures and tables is quite nice in general!
  • The related work summaries are overly succinct, making the relevance to the present work unclear. E.g., "Reflexion adapts verbal memory"; not clear what this means or why it's relevant. Longer sentences elaborating more would help. A couple of sentences summarizing the literature in each area overall would also help, in addition to listing off individual papers.
  • What is "employee acceptance"? and what does "a shipped defect is d=a(1−y)d=a (1-y)" mean?
  • clarifying what kind of policy π\pi is would be helpful. Is it the policy of the employee choosing how to use AI?
  • "token billing motivates the archetype without establishing that billing alone produces it" can be made clearer.
  • is the idea behind making QQ a function of ϕ\phi that employees with higher fee-sharing will be more productive due to higher incentive? this is worth spelling out.
  • Where does the baseline Uk0U^0_k come from? the full utility UkU_k for each party kk should be spelled out, as (4) seems to just define some components.
  • Theorem 1 is pretty clear from the definitions: rewarding tokens only helps if more tokens means more value. Do we learn anything from it?
  • Theorem 2 doesn't seem to be saying anything non-trivial: if the loss to employees equals the loss to companies, then of course they have identical minimizers; if pF=κVpF = \kappa V, then the written functions are identical.
  • Eq (6) defined η\eta, but what are ηout\eta_\text{out} and ηtok\eta_\text{tok}? (ah, under the two policies -- this should be explicit)
  • SWE-chat user personas should be explained around line 373
  • The labels in Fig 5b are poorly placed -- it's hard to see which label is associated with which point and the text also overlaps data (and the error bars are truncated)
4 comments
AuthorsOpenReview ↗

Rebuttal (Part 1)

We thank you for your valuable opinions and questions. All the responses have been updated in the revised PDF.

Major Comments

Comment fwJw-W1: Do the complex simulations yield only intuitive results? What is the general significance of lineages and monetary amounts?

Response: The contribution is to turn intuition into an empirical account of conditions and mechanisms. “More use need not be better” does not tell an enterprise which work wastes computation, whether reducing usage harms hard tasks, or when continuation is worthwhile. Our traces distinguish checking after a correct result without added benefit, edits that introduce regressions, and a compatibility question that repairs a defect.

These findings motivate designing continuation and stopping together. Enterprises can evaluate the marginal value of additional computation rather than use volume as evidence of effectiveness. Lineage colors track simulated ancestry; monetary scenarios compare payoffs under explicit value, cost, and sharing assumptions. They explain processes and application conditions, not actual employee ability or company profits.

The token-max problem is of critical importance to enterprises, as maximizing token utilization does not necessarily lead to better outcomes and may introduce inefficiencies or redundant work (We already got many invitations for this topic). We hope our work will encourage enterprises to pay closer attention to the underlying mechanism design challenges and reconsider how computational resources should be allocated to achieve more effective and reliable outcomes. We sincerely hope you will give careful consideration to the significance and contributions of our work.

Comment fwJw-W2: Do the two theorems follow directly from their assumptions?

Response: Theory states the decision conditions; experiments establish the observed consequences. The algebra is straightforward. The first proposition compares marginal computational return with current efficiency, but the additional correct output ΔY is unknown before continuing and requires empirical investigation. It does not supply an optimal stopping algorithm. The second matches expected liability to defect damage with other payoffs fixed; it does not prove full contract alignment. The principal contribution is the empirical characterization and its explanation. These conditions connect behavior, cost, and enterprise decisions.

Comment fwJw-Q1: What does the paper establish beyond common sense and simulation details?

Response: Four practical principles follow:

  • Measure correct output against total cost, checking hard-task performance separately.
  • Tie additional work to unmet requirements, testing that edits preserve correct behavior.
  • Separate acceptance from correctness, accounting for defects, false rejections, and checking fees.
  • Establish net value before allocating it, distinguishing break-even from improvement over baseline.

These principles follow from observed trade-offs; their combined effectiveness still needs validation with real tasks and personnel.

Comment fwJw-Q2: Why count only input and output? Are cache reads and writes charged twice?

Response: Token counts and billed cost are distinct. Cache reads/writes are input categories, separated before pricing to avoid double counting; output counts newly generated content. The original main table counts execution-agent input and output, whereas matched mathematics includes both roles. Some historical cache fields cannot be recovered consistently, so L3 prices recorded input/output. Token savings therefore do not imply equal billing savings.

Comment fwJw-Q3: What is the tactic library?

Response: Tactic libraries are text rules updated by task feedback. Supervisor and solver roles are updated separately. Joint rule learning is not automatically beneficial; we retain training and reflection costs, amortizing them over 27 tasks to evaluate net quality and cost.

AuthorsOpenReview ↗

Rebuttal (Part 2)

Comment fwJw-Q4: What is an employee lineage, and where does it come from?

Response: Lineage records simulated ancestry, not employee ability. L3 maintains 12 simulated employees. Each period, descendants of the top three replace the bottom three, with rule-governed changes to behavioral settings. Shared colors mark common ancestors and show which supervision styles survive the specified selection rule.

Comment fwJw-Q5: How are fines implemented?

Response: Fines are implemented via prompts and accounting, with no real money transfers. The prompt states penalties for defective delivery, while accounting calculates expected losses. Combined instructions do not isolate fine effects, and actual monetary responses require human evidence.

Minor Comments

Comment fwJw-C1: The related work is too brief, and the connection to Reflexion and similar work is unclear

Response: We link Reflexion to our text-based memory updates (L2), noting that both retain fixed model weights. We will include clearer related work.

Comment fwJw-C2: What are employee acceptance and a shipped defect?

Response: Acceptance ends the interaction; a shipped defect is an accepted incorrect result. With acceptance a and external-test correctness y, correct delivery is ay and defective delivery is a(1−y). Correct work can also be rejected. We will explain it more.

Comment fwJw-C3: Does π denote the employee's policy for using AI?

Response: π denotes the employee's AI interaction policy under common resource limits. It specifies feedback, checking, and acceptance. L1 fixes these rules; L2 updates supervisor or solver text rules separately and identifies the role being changed.

Comment fwJw-C4: What does “token billing motivates the archetype…” mean?

Response: Token billing makes computation measurable, motivating the study of usage as an input proxy. We do not claim billing alone causes the token-max archetype; actual human adoption requires separate behavioral evidence.

Comment fwJw-C5: Why does Q depend on φ? Does a higher share imply greater employee effort?

Response: φ is the provider's share, not the employee bonus. The general model allows Q to depend on φ, but the fixed-output accounts do not estimate that behavioral response. Increasing φ redistributes payoff from the firm to the provider; employee bonuses and output remain fixed in these accounts.

Comment fwJw-C6: What is the baseline, and how should each party's complete utility be written?

Response: With production artifacts and task value held fixed, the baseline uses token billing and fixed wages; OutcomeShare adds acceptance, sharing, and liability. Transfers like fees and fines cancel in the sum, but serving, effort, and defect costs remain in the full utility.

PartyNet-return calculation
ProviderProduction and verification revenue − cost of providing those model services
EmployeeWage + delivery bonus − effort cost − expected fine
FirmValue of correct deliveries − production payments to the provider − wages and bonuses − verification fees − defect losses + fines received

Comment fwJw-C7: Does Theorem 1 merely restate the definition?

Response: Theorem 1 states the threshold for improving efficiency. For expected correct output Y, cost C > 0, and added cost ΔC > 0, efficiency increases exactly when ΔY/ΔC > Y/C. The algebra is direct; the empirical difficulty is estimating ΔY before continuing.

Comment fwJw-C8: If the two losses in Theorem 2 are equal, does the theorem have substantive content?

Response: Theorem 2 specifies how to calibrate defect liability. With fixed audit probability p > 0, collectible fine F, and defect damage κV,F=κV/pκV, F=κV/p matches expected liability to damage. Under risk neutrality, other payoffs fixed in effort, and attained minima, the defect-related losses have identical minimizers.

Comment fwJw-C9: Which policy do the quantities in Eq. (6) describe?

Response: Output and tokens in the equation refer to the same policy. For task j, yjy_j is external-test success and TjT_j is execution-agent token use; η=106Σyj/ΣTjη = 10⁶Σy_j/ΣT_j. Subscripts tok and out distinguish the token-use and outcome policies. We report success and mean tokens alongside efficiency.

Comment fwJw-C10: SWE-chat user personas should be explained at first mention

Response: SWE-chat persona definitions describe observed human behavior, distinct from prescribed simulated styles. We explain these categories near first use and separate conversation scores from external correctness.

Comment fwJw-C11: Figure 5b has overlapping labels, unclear correspondence, and clipped error bars

Response: Will revise that.

Thanks for the replies!

I agree that the problem of deciding good token / AI budget allocations and policies is very important for companies. But I'm skeptical that the LLM simulations described here provide informative and generalizable takeaways beyond the (intuitively clear) statement that optimizing for impact / output is better than optimizing for tokens (and the also intuitively clear principles listed above). I think companies are doing all of these things already

Thanks for the thoughtful reply. However, I would respectfully disagree that companies are already implementing all of these practices (at least before we started doing this work). In reality, many companies are still primarily focused on cutting token budgets to avoid excessive consumption, and some even continue to use token usage as a key performance metric or leaderboar see many report:

What is largely missing is a systematic framework for evaluating how effectively AI is being used and for designing policies that encourage better AI utilization. This is precisely what we aim to contribute in this paper. Beyond showing that optimizing for impact rather than token consumption is preferable, we propose a practical framework for guiding AI usage through appropriate evaluation rules and incentive mechanisms. For example, companies could establish

  • Good task-specific evaluation criteria and define appropriate stopping rules
  • Introduce mechanisms we proposed, such as sharing outcome-based revenue with employees or applying penalties for inefficient AI usage.

These mechanisms are intended to encourage employees to use AI more effectively rather than simply consume fewer tokens. Our goal is not merely to demonstrate an intuitive principle through LLM simulations, but to provide actionable policy guidance for companies on how to organize, evaluate, and incentivize AI usage. We believe the simulation results offer useful insights into how different policies shape behavior, and we are also conducting related experiments in real-world industry settings to further investigate their practical implications.

Reviewer_JXDvOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies how supervisory objectives and delivery incentives can improve the resource efficiency of AI-assisted work. The authors compare token-max, which maximizes interaction token consumption, with outcome-max, which maximizes externally verified successes per unit of compute cost. They develop a three-level simulation framework comprising single-task supervised execution, textual policy adaptation by employees and production agents, and employee population selection under different evaluation metrics. Employees are simulated using persona-conditioned LLMs, and task correctness is measured separately from employee acceptance.

Experiments cover SWE-bench Verified, HumanEval+, GSM8K, and MATH-500. The prescribed outcome-max behaviors generally reduce production-agent token consumption substantially while maintaining similar or somewhat higher aggregate success rates, although performance declines on some difficult tasks. The paper further proposes OutcomeShare, which allocates value through verified-outcome sharing and liability for defective delivery. Its theoretical analysis characterizes conditions for efficiency improvement and incentive alignment in a simplified verification model.

The mechanism evaluation includes experiments with liability and verification prompts, alongside delivery and payoff calculations based on synthetic inspection signals, fixed production outcomes, and predefined economic parameters. The positive results primarily support the efficiency advantages of prescribed behavioral policies and the feasibility of value sharing under certain parameter settings, rather than establishing how real employees respond to monetary incentives or how the complete contract performs in deployment.

Critical strengths
  1. The paper studies supervision and stopping decisions as determinants of AI efficiency and demonstrates a practically meaningful cost–quality trade-off. Rather than focusing solely on model capabilities, the paper examines how users guide AI systems, request additional checks, and terminate interactions. Results across multiple task types show that supervisory policies can substantially change resource consumption while yielding similar aggregate success rates. For example, on SWE-bench, average production-agent token consumption decreases from 21.1k to 9.4k, while aggregate success changes from 48.8% to 48.2%. The adaptation and population-selection experiments further illustrate how these policies can be updated through experience and reinforced by evaluation rules.

  2. The clear distinction between external correctness, employee acceptance, and defective delivery avoids treating perceived completion as actual quality. The paper uses independent task evaluators to assess artifacts rather than treating employee acceptance as the success label. This design identifies both continued checking of correct artifacts and premature acceptance of incorrect ones. Paired examples further illustrate that additional checks can be redundant but can also uncover genuine compatibility defects. The reported declines in hard-task success and reductions in correct deliveries under liability mechanisms extend the analysis beyond token savings alone.

Critical weaknesses
  1. The main experiments prescribe continuation and stopping behaviors, making it difficult to establish the contribution of the objectives or incentives themselves.

The token-max instructions explicitly request additional edge cases, refactoring, tests, documentation, and explanations, whereas the outcome-max instructions explicitly require task completion at minimum cost and prohibit additional elaboration. The experiments therefore change the objective, task scope, verification strategy, and stopping tendency simultaneously. The substantial resource-use differences cannot be attributed specifically to objective optimization, much less interpreted as evidence that monetary incentives induce the corresponding behaviors.

The objective-only ablation does not resolve this concern: the paired efficiency gain is 4.5%, with a 95% confidence interval of [−31.9%,+62.4%][-31.9\%, +62.4\%], and success rates remain unchanged. The neutral reference is also not strictly matched to the main adaptation conditions. Consequently, the incremental benefit over standard, task-oriented, or budget-aware supervision remains unclear.

  1. The validation of OutcomeShare is insufficient to establish incentive alignment and improved payoffs for all three parties under the complete mechanism.

Under assumptions of risk neutrality, a fixed effective audit probability, collectible fines, and other payoff terms that do not vary with verification effort, Theorem 2 makes the two loss functions identical by setting pF=κVpF = \kappa V. This result is sound, but verification under the actual OutcomeShare mechanism may change correct-delivery volume, bonuses, throughput, and supervision costs. It therefore does not establish incentive alignment for the complete contract.

There is also a gap in the empirical evidence. The LLM liability prompts simultaneously require verification, and no monetary deductions are actually executed. The payoff model instead relies on synthetic inspection signals generated from gold labels and assumes ideal calibration; it therefore does not validate the responses of real participants or the behavior of an actual verifier. The claimed three-way gains are also highly parameter-dependent. The central scenario in Table S21 assumes a valuation of $240 per reference hour and reports a firm-profit point estimate of $275, with a 95% confidence interval of $[-$1{,}176, +$2{,}161]$. At a valuation of $80 per reference hour, firm profit is −$1,508. Moreover, the provider’s participation condition compares revenue rather than profit net of serving costs.

Questions for authors
  1. Fair comparisons and the role of objective framing Can the existing records support a standard task-oriented or budget-aware supervision baseline matched on tasks, models, personas, and budgets, rather than a baseline explicitly instructed to maximize token consumption? What specific cohort or model differences exist between the objective-only condition and the main policy-package condition? Clarifying these details would help determine how much of the reported advantage comes from objective framing rather than prescribed continuation rules.

  2. Correspondence between the theory and the complete contract In the actual OutcomeShare model, which private and social payoff terms satisfy Theorem 2’s assumption that they remain constant with respect to verification effort? When verification changes bonuses, correct-delivery volume, and throughput, does any existing analysis establish alignment of the complete objectives, or is the theorem used only to calibrate liability for defective delivery? This distinction affects my assessment of the mechanism’s contribution.

  3. Three-party participation and the robustness of payoff gains Based on the existing sensitivity analyses, over what ranges of inspection quality, inspection cost, task valuation, and sharing rates can employee participation, firm break-even, and provider participation net of serving costs be satisfied simultaneously? Please distinguish improvement over a loss-making baseline from achieving nonnegative net payoffs, and clarify whether the conclusions rely solely on point estimates under fixed parameters.

  4. The impact of valid-run filtering on the main results The main software experiment moves from 15 scheduled tasks with repeated runs to a primary paired analysis covering 13 shared tasks. Please report the number and reasons for excluded runs in each condition, along with the consistently applied exclusion rules. Do the cost advantage and hard-task quality gap persist when all scheduled tasks are included in the denominator? This information is important for assessing the credibility of the efficiency gains and performance-preservation claims.

2 comments
AuthorsOpenReview ↗

Rebuttal (Part-1)

Thanks for the insightful questions. We have already revised our PDF to include an updated version that covers all our answers.

Major Comments

Comment JXDv-W1: Objectives, scope, verification, and stopping rules change together; a fair ordinary baseline is missing

Response: Matched baselines improve the comparison; the unit of evaluation remains the complete policy. The mathematics experiment fixes tasks, drafts, models, and iteration limits, with gold answers hidden. Ordinary supervision provides a common reference for objective-only and rule-based policies.

Mathematics supervision policyContinuation rateMean total tokensExternally verified success
Ordinary supervision1.3%1,01893.8%
Outcome: goal only2.5%1,00991.3%
Outcome: with rules0.0%92993.8%
Token: goal only8.8%1,05095.0%
Token: with rules68.8%2,06195.0%

An objective label cannot substitute for behavioral evidence. Outcome-only usage is close to ordinary supervision, whereas token rules substantially increase continuation. In software, ordinary, budget-aware, outcome-only, and outcome-rule policies all accept identical drafts (86.11% correct), without additional revisions. Because rule packages also change wording, these comparisons isolate neither a single rule nor the effect of real monetary incentives.

Comment JXDv-W2: Do the theory and experiments establish complete incentive alignment and gains for all three parties?

Response: Efficiency creates value; verification and allocation determine who benefits. OutcomeShare connects productivity to reliable delivery, checking costs, and participation. We ran task/code-only checkers on 432 software artifacts, with correctness determined by hidden tests. These measurements are separate from earlier synthetic accounting.

Acceptance processCorrect deliveryDefective deliveryAdditional checking fee per thousand checks
Original supervisor acceptance, no additional check85.42%14.58%$0
Additional Flash-Lite check81.71%14.58%$0.22
Additional Pro check81.48%6.25%$6.40

Pro reduced defective delivery by 8.33 percentage points, but also reduced correct delivery by 3.94 points and added $6.40 per 1,000 checks. Stricter verification has both benefits and costs. The theorem calibrates defect liability with other payoffs fixed; fixed-production accounts do not identify human incentive responses. The results show it indeed build an incentive alignment for all three parties.

Comment JXDv-Q1: Can you provide matched standard or budget-aware baselines? How does the original objective-only study differ?

Response: The old objective-only result cannot be attributed to objective wording alone. Its 4.5% efficiency gain and 95% interval [-31.9%, 62.4%] also reflect differences in models and styles.

StudyExecution modelSupervision modelSupervision style
Original main supervision policiesClaude Haiku 4.5Claude Sonnet 4.614 settings
Original objective-only studyDeepSeek v4 FlashClaude Haiku 4.5Midpoint on each behavioral dimension
New five-policy mathematics comparisonGemini 3.5 Flash-LiteGemini 3.5 Flash-LiteCommon neutral style

The new matched comparisons fix these conditions and include ordinary and budget-aware baselines. In matched mathematics tasks, the outcome objective alone saves 0.9% of tokens, with a paired interval containing zero. This near-zero result motivates evaluating concrete supervision rules rather than assuming a new objective label improves behavior.

AuthorsOpenReview ↗

Rebuttal (Part 2)

Comment JXDv-Q2: Which terms are fixed with respect to verification effort in the theorem? Does it cover the full contract?

Response: The theorem aligns defect liability with other payoffs fixed in effort. At effort e, private loss is c(e)+pFδ(e) and social loss is c(e)+κVδ(e), where c is checking cost, δ is defect probability, and κV is defect damage. With risk neutrality, fixed audit probability p > 0, collectible fine F, and attained minima, pF=κV yields identical minimizers. Wages, bonuses, and other payoffs must be fixed in e. Changes in bonuses, correct delivery, or serving costs require a fuller analysis.

Comment JXDv-Q3: Under which parameters do all parties participate and the firm break even? Are these only point estimates?

Response: Breaking even is weaker than improving on every party's baseline. The example uses ordinary-supervision outputs with full Pro screening; amounts are dollars per 1,000 attempts. Provider/employee shares are 10%/20%; serving cost is 50% of API price. Defect loss and the fine equal task value; wage equals mean production fee, effort cost is half the wage, and audit probability is 25%.

Task value as a multiple of mean production feeProvider net returnEmployee net returnFirm net returnOriginal firm net return at the same task value
10×+3.41+1.78−2.62+4.36
20×+4.15+3.15+1.76+10.38

At a task value of 20× production fee, all payoffs are nonnegative, but the firm earns less than its baseline. Under these assumptions, preserving provider/employee baseline payoffs while all parties break even requires value of 15.98× production fee (95% interval [11.85, 25.35]).

Feasibility depends on valuation, shares, and costs. Each policy is evaluated over 18,225 combinations: task value spans 2–50× production fee, provider shares 5%–20%, and employee shares 10%–40%, with screening, serving costs, defect losses, and liability also varied.

RequirementFraction of prespecified combinations satisfying it
Each party weakly improves on its own baseline0.91%
All three net returns are nonnegative28.50%
Provider and employee retain baseline returns, and all three returns are nonnegative26.29%

These fractions describe the chosen grid, not deployment-success probabilities.

Comment JXDv-Q4: Why do 15 tasks become 13 shared tasks? Do the conclusions survive inclusion of every run?

Response: Each policy has valid records for 15 tasks, but only 13 tasks are shared. Identical exclusion rules apply in sequence. Astropy counts differ because some failures were removed at earlier steps. The table accounts for the 310 paired records from 420 scheduled attempts per policy.

Filtering stepToken-use policyOutcome policy
Scheduled attempts420420
Empty result or execution error−29−33
Standard-test launch failure−27−27
Remaining patch-capture failures on the same Astropy task−28−26
Valid records336334
No valid counterpart under the other policy−26−24
Final matched records310310

The all-scheduled analysis retains failures and the hard-task quality loss. Failed attempts remain in the success denominator and count as unsuccessful.

Task subsetToken-use successOutcome-policy successToken-use mean tokensOutcome-policy mean tokens
All tasks39.05%38.33%20,80810,791
Hard tasks26.43%16.43%15,39813,132

Overall recorded tokens fall by 48.14%, but outcome-max produces 27.11% fewer correct tasks per token on hard tasks. Some failure-related usage is unrecoverable, so recorded usage is not a complete bill, and missing usage is not zero.

Conclusion:

Thanks again for supporting our paper. We want to highlight that our key contribution is to identify and explain the conditions under which these divergent outcomes arise through controlled experiments and behavioral trajectory analysis, providing empirical evidence for when further computation is beneficial and when it should stop. It will support communities in thinking about the good use of AI.

Reviewer_gevnOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

Tasks are: SWE-bench Verified: fixing real GitHub issues in Python repositories HumanEval+: writing small Python functions GSM8K: solving grade-school math word problems MATH-500: solving competition math problems

Methods are: Token-max: the LLM employee is prompted to keep the agent working and maximize tokens used Outcome-max: the LLM employee is prompted to get a correct solution at minimum cost and stop Objective-only: the employee is told only the goal, with no instructions on how to behave Personas: employees vary in style, such as giving examples, fact-checking, or asking for revisions Adaptation (L2): the employee, the agent, or both update a text playbook of tactics over rounds Selection (L3): a simulated firm keeps and copies employees ranked by tokens used or by solves per cost Verification rules: employees are told about a gate check or a fine for approving wrong work OutcomeShare: an accounting simulation that splits verified value among provider, firm, and employee and fines defects

Findings are:

  • outcome-max used far fewer tokens than token-max
  • outcome-max lost success on hard SWE-bench tasks
  • when both employee and agent learned, outcome-max had higher success with less tokens
  • the firm ranking employees by tokens lost money, the firm ranking by tokens was profitable
Critical strengths

S1. The paper identifies a timely problem since firms are actually starting to reward or limit employee token use.

S2. The paper is clear about its limitations and gives useful examples (Table 2) showing when extra interaction is wasteful and when it actually fixes a bug.

Critical weaknesses

W1. If you optimize for token maximization, the maximum output you achieve will be lower than what you could achieve by optimizing directly for output maximization. In other words, aligning your optimization objective with your evaluation objective leads to better results. I see this as common sense. Since the token-max agents are explicitly prompted to maximize tokens, the efficiency gain follows by design.

W2. The key empirical question is whether humans, when asked to use more tokens, still ask questions that are useful for their work rather than random ones. Experiments with LLMs cannot tell us how humans would behave here. In fact, when LLMs are given only the objective without behavior instructions, the two objectives perform almost the same (Table S4).

W3. Humans also differ from LLMs in how much information they can process. Even if human and LLM supervisors asked similar questions under token-max, a human may not be able to read and verify such long outputs. Since the results depend on the supervisor judging when the work is correct, this is another weak link.

W4. The findings depend heavily on the prompts and setup, as shown by the gap almost disappearing with objective-only prompts (Table S4), and because these are LLM simulations on small samples, they may not hold with other models, tasks, or random seeds.

Questions for authors

What are your thoughts on the extent to which LLM simulations of humans can represent actual human behavior? It seems to me that this is a fundamental aspect of this paper and an invalid assumption.

1 comment
AuthorsOpenReview ↗

Rebuttal (Part-1)

Thanks for the insightful questions; we have already made the change in the PDF for all the responses.

Major Comments

Comment gevn-W1: If the prompt already requests more tokens, is the efficiency gap predetermined?

Response: We want to highlight that Asking for more tokens does not necessarily predetermine lower productivity: useful checks could repair enough defects to justify their cost. We find that, given the same model capability, supervision can shape whether additional computation corrects errors or instead leads to redundant work and even introduces new errors. Our key contribution is to identify and explain the conditions under which these divergent outcomes arise through controlled experiments and behavioral trajectory analysis, providing empirical evidence for when further computation is beneficial and when it should stop. For example,

  • Across MATH-500 problems and two seeds, with shared drafts, models, and limits, the token objective alone and token rules used 1,050 and 2,061 tokens, respectively; both achieved 76/80 successes. Equal aggregate counts do not establish equivalence. Gold answers were hidden from supervisors.

  • In software, ordinary, budget-aware, outcome-only, and outcome-rule policies accepted identical drafts (62/72 correct) without revision. Token rules repaired one draft and broke four; the net success difference remains uncertain. The empirical contribution is to measure what additional work achieves. These are complete-policy comparisons, not isolated effects of token count or a single rule.

So we really appreciate you considering the insight in our study.

Comment gevn-W2: People asked to use more tokens may ask useful questions; objective-only differences are small

Response: The marginal value of computation depends on what the extra work resolves. A follow-up about backward compatibility repaired an initially failing patch. Across all scheduled hard-task attempts, success was 26.43% under token-max and 16.43% under outcome-max; the latter also produced fewer correct tasks per token. Lower usage alone therefore does not establish higher productivity.

Changing the stated objective does not ensure an effective intervention. In matched mathematics tasks, the outcome objective alone saved 0.9% of tokens, with a paired interval containing zero. Concrete checking and stopping rules need evaluation. The simulations establish consequences of prescribed policies, not how people would respond to usage or monetary incentives.

Comment gevn-W3: Does simulated acceptance overlook people's limited reading capacity?

Response: Short reports reduce displayed text by 34.7% but increase total tokens by 49.1%, with the same 13.89% false acceptance rate. All initial drafts were accepted. This shows that compressing visible text does not automatically improve acceptance quality or reduce total computation. This is not a human reading or comprehension experiment, but an evaluation of how interface constraints alter the workflow and background costs.

MeasureFull-code viewShort-report interface
Newly displayed text tokens per task460.5300.7
Hidden-test correctness86.11%86.11%
False acceptance13.89%13.89%
Total system tokens, including report generation1,074.11,601.1

Comment gevn-W4: Do the results depend on prompts, models, small samples, and random seeds?

Response: Within-model comparisons establish efficiency differences with a defined scope. New matched experiments fix drafts, models, and limits. The table reports effects within each model group, not a ranking across models or a population prediction.

Execution modelToken-use successOutcome-policy successEfficiency gain and 95% interval
Claude Haiku 4.551.9%61.6%+120.0% [79.6%, 170.1%]
DeepSeek v4 Pro58.1%61.0%+276.5% [234.8%, 322.7%]

More adaptation does not guarantee better outcomes. Across three independent training runs, fixed rules achieved 93.83% correctness versus 91.36% when both roles updated. Training and reflection costs are amortized over 27 test tasks. The selected prompts, styles, tasks, and seeds still limit generalization.

Comment gevn-Q: To what extent can LLM simulations represent people?

Response: Simulations provide controlled evaluation of supervisory policies. For example, [1] LLMs can exhibit human-like context-dependent choices, although their underlying decision-making mechanisms may differ. However, in simulated environments, they demonstrate controllable behavioral trends that are qualitatively similar to those observed in real humans. We also compare SWE-chat uses LLM annotation, not external test correctness or causal incentive evidence (See Figure 1).

[1] Do LLMs Choose Like Humans? Using Cognitive Theory to Evaluate LLM Decision-Making EMNLP 2026 findings.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.