acceptodds
Under review as a conference paper at ICLR 2027

Can More LLM Users Reduce the Welfare Cost of Selfish Advice?

11 views

Abstract

Large language models (LLMs) increasingly guide decisions whose consequences extend beyond their users. We study a shared LLM *advisor* that learns from user feedback while other players follow no regret learning in the *volunteer's dilemma*, a multiplayer public good game. We compare a **selfish** LLM benchmark that optimizes its users' expected payoffs with a **selfless** benchmark that optimizes total expected welfare. Our central quantity is the social welfare gap, the selfless benchmark's total welfare minus the selfish benchmark's at the same adoption level. To analyze this gap, we first establish convergence of the other players' response and the advisor's mean learning dynamics. When the other players continue volunteering, both objectives select the same policy, so the gap is zero. When LLM users alone supply the public good, this gap has a unique maximum as adoption varies. When the maximum is interior, the gap first widens and then narrows. Broader adoption brings more of those who benefit from the public good into the selfish objective. Higher contribution costs relative to the benefit each player receives increase the gap and move an interior maximum toward lower adoption. Our experiments on Qwen3 8B reveal action probabilities close to theoretical predictions and generally smaller average welfare gaps at higher adoption levels. These findings suggest that wider LLM adoption can reduce the welfare cost of serving users alone, so losses at limited adoption need not persist as more users join.

Primary area:
optimization
Keywords:
Algorithmic game theory, large language models, multi-agent learning, no regret learning, social welfare, LLM adoption

Reviews

Among:
Shape:
1234this paper, Top 32%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.50 over 4 reviews, above 68% of the others.
Reviewer_JLQFOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies the volunteer’s dilemma: someone in a group has to perform a costly task, like checking a shared proposal, but everyone benefits once it is done, so each person hopes someone else will do it. In the model, some players follow the same AI advisor, which learns from its users’ feedback and aims to maximize their rewards (the “selfish” advisor). The other players decide on their own and gradually learn which action pays off better.

The authors compare the total welfare achieved by this selfish advisor with that of a hypothetical “selfless” advisor that aims to maximize everyone’s welfare. They call the difference the social welfare gap. When both groups continue volunteering (type 2), the two advisors have the same optimal policy within this regime, and the gap is zero. When only AI users volunteer (type 1), the gap first increases and then decreases as more people use the AI, because more of those benefiting from volunteering are included in the advisor’s objective. Higher volunteering costs increase the gap and shift its maximum toward lower AI adoption levels.

Critical strengths
  • I think the paper provides a solid theoretical analysis. Rather than simply assuming that players reach an equilibrium, the authors first study how the independent players respond to the AI advisor and then analyze the advisor's learning dynamics and convergence. Based on these results, they derive the welfare outcomes and show how the gap changes with AI adoption. I also found it interesting that the gap does not simply decrease as more players use the AI, but can first increase and then decrease, with the contribution costs affecting where the maximum occurs.

  • I appreciate that the authors also test their theoretical predictions using a real LLM rather than relying only on simulations. They fine-tune Qwen3-8B using PPO and compare the learned volunteering probabilities with the theoretical predictions, finding relatively small differences (2.23 percentage points on average). The experiments are also presented transparently, and the authors acknowledge that the welfare results are more exploratory and do not fully validate the theoretical conclusions.

Critical weaknesses
  • My main concern is that the welfare results are derived separately for type 1 and type 2. Theorem 4.1 shows that the welfare gap eventually decreases as more players use the AI, but only within type 1. Figure 4(a) shows that AI users sometimes prefer type 2, so a selfish advisor could choose a different regime from the selfless one. In that case, the welfare gap could be quite different from what the paper reports. I think this is an important limitation, since it is unclear whether the main conclusion still holds when both advisors can freely choose their policies.
  • I also have some concerns about the experimental validation of the welfare results. In Section 5.2, both LLMs are trained using individual user rewards, but with different pretrained reference policies. As I understand it, the selfless advisor is therefore not directly trained to maximize total welfare. This makes it somewhat unclear how strongly the experiments support the comparison between selfish and selfless advice. I think training the two advisors with their respective objectives would provide a better evaluation.
Questions for authors

Questions

  • What happens if both advisors can choose their policies freely, rather than being restricted to type 1 or type 2? Figure 4(a) suggests that the selfish advisor might prefer type 2, even when the selfless advisor would prefer type 1. Would the main conclusions about the welfare gap still hold in that case?
  • I was wondering how much the learning outcome depends on the initial pretrained policy. Theorem 3.1 suggests that the advisor can converge to different regimes depending on this initialization. Does this mean that the main welfare results depend on choosing a suitable reference policy, rather than describing the outcome a selfish advisor would naturally learn?
  • Remark 3 mentions that total welfare can decrease with higher AI adoption, even while the welfare gap becomes smaller. How should we interpret the main finding in this case? It seems that a smaller gap does not necessarily mean that society benefits from having more AI users.
  • In the welfare experiments, both LLMs are trained using individual user rewards, but with different reference policies. Would it be possible to train the selfless advisor directly using total welfare as its reward? I think this would make the comparison between selfish and selfless advice more convincing.

Suggestions for Future Work

Compared to other ICLR submissions I have reviewed, I appreciated the quality of the theoretical work. However, I sometimes found it difficult to follow the mathematical intuition behind certain steps. I think cleaner explanations of the main theoretical results compared to the extant literature would make the paper easier to follow. Similarly, while you acknowledged the current limitations of the current experiments, strengthening the empirical validation with additional experiments would be appreciated.

Reviewer_VgXLOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies volunteer’s dilemma, where there are two groups of users. One group is advised by a LLM and one group is learning by themselves. The goal is to understand how the social welfare gap varies across different parameter regimes, especially the LLM adoption rate.

Critical strengths

I think the setting of how LLMs affect the dynamics in volunteer dilemma is interesting.

Critical weaknesses

See my questions above. I dont think the theoretical modeling captures the motivation.

Questions for authors

See above.

Reviewer_jyJ3OpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies how the adoption of a shared LLM advisor affects social welfare in the volunteer’s dilemma. The population consists of advisor-guided players and an aggregate population of exponential-weights learners. The authors characterize the latter population’s limiting response and analyze the advisor’s reduced mean learning dynamics under KL regularization. They then compare policies that maximize guided-user welfare and total social welfare within contribution regimes. The main result establishes that the welfare gap in the regime where only guided users contribute is single-peaked in adoption, and characterizes how its peak changes with contribution costs. Experiments with Qwen3-8B provide preliminary comparisons between learned behavior and the analytical predictions.

Critical strengths
  1. An insightful scenario. The mixed-population setting captures a meaningful issue: an advisor’s recommendations affect people whose welfare is excluded from its objective, while adoption changes the population covered by that objective. This provides a useful setting for studying the externalities of shared AI advice.
  2. A reasonable temporal structure. The distinction between period-level interaction and batch-level advisor updates is a sensible abstraction of frequent decisions combined with periodic model adaptation. It gives the interaction between the two populations a clear operational motivation.
  3. Solid technical analysis. The paper contains substantial analytical work, particularly the single-peakedness result and the comparative statics of the peak. Although some benchmark calculations are straightforward, the main shape result requires nontrivial analysis.
  4. Preliminary validation with an actual LLM. Training and evaluating Qwen3-8B in the simulated game provides an initial check of whether the theoretical abstraction captures aspects of language-model behavior. The agreement in final action probabilities is encouraging.
Critical weaknesses
  1. The presentation could be more economical and readable. The paper introduces many related symbols, coordinates, and indices, which makes it difficult to retain a clear picture of the model and follow the argument. Some notation and repeated explanations appear avoidable. A more compact presentation, together with a concise notation table, would help readers focus on the central mechanism and results.
  2. The purpose and interpretation of the experiments are not sufficiently clear. Section 5.1 compares final action probabilities under a common reference with predicted fixed points, whereas Section 5.2 compares welfare under separately calibrated references targeting two benchmark policies. This distinction is important but requires careful reading across the main text and appendix. In particular, both arms of the welfare experiment use individual-payoff feedback, so the comparison’s implications for learning under different welfare objectives are limited. The organization and figures could more clearly identify what is varied, what is measured, and which theoretical claim each experiment supports. I found it necessary to revisit the figures and experimental descriptions several times to reconstruct the intended comparison.
  3. The significance of the main qualitative conclusion is somewhat limited. A narrowing welfare gap at high adoption is reasonably intuitive because the advisor’s user objective encompasses more of the beneficiaries. The rise-and-fall pattern is also qualitatively plausible in this setting, although proving its precise shape is nontrivial. Much of the additional contribution therefore lies in the properties and location of the peak. The paper could better explain how these characterizations inform concrete decisions about advisor design or deployment. At present, the practical implications of the peak analysis seem less developed than its mathematical characterization.
Questions for authors
  1. Have the authors examined whether Qwen3-8B converges to stable advice when it reasons over stored interaction history without parameter updates?
  2. Could the authors clarify the role and necessity of LoRA/PPO adaptation for Qwen3-8B in this narrowly specified binary-action task?
  3. How robust are the welfare results if ordinary players maintain individual learning scores instead of sharing a representative score?
  4. How do switches between contribution regimes affect the welfare gap when both benchmarks are optimized globally?
Reviewer_QzZcOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper studies a shared LLM advisor in the volunteer’s dilemma, showing that wider adoption can eventually narrow the social welfare gap between advice optimized for users alone and advice optimized for all players.

Critical strengths
  • The question is relevant and cleanly formulated. A shared LLM advisor learns from pooled user feedback alongside no-regret players in the volunteer's dilemma, and the social welfare gap is defined at matched adoption level and feasible policy set, separating the objective's cost from incomplete optimization (Sections 1, 2.5).

  • The theoretical results are solid. They establish convergence of the other players' response and of the advisor's mean dynamics (Lemma 1; Theorem 3.1), a zero gap when other players keep volunteering, and, within type 1, a single-peaked gap in adoption with interpretable comparative statics in c/G (Theorem 4.1; Corollary 4.1).

  • The Qwen3 8B experiments show dynamics close to the theoretical predictions. The authors are transparent about the exploratory nature of the welfare comparison, reporting post-hoc grid refinement and retaining calibration misses rather than excluding them (Section 5; Appendix E.3).

Critical weaknesses
  • Both arms train on guided users' individual payoffs with identical PPO and KL settings. The "user" and "social" labels refer only to different theory-calibrated SFT references, which also initialize each arm (Section 5.2; Appendix E.3). The update optimizes neither aggregate user welfare nor total welfare (Appendix C.1.5). The design is disclosed and labeled exploratory, but Figure 2(b) mainly reflects the choice of anchor and does not isolate the effect of optimizing user versus social welfare.

  • The welfare-gap analysis is restricted to type 1 policies (Definitions 1–2; Section 4.2), yet the conclusion states the adoption claim without this qualifier. Theorem B.1 and Figure 4(a) show that users' preferred regime can reverse as adoption grows. If each objective optimizes over the full interval, the two optima can therefore fall in different regimes, and the gap need not follow the type 1 curve. The paper neither characterizes this case nor states for which (N, c/G) the restriction is non-binding (Appendix B.6).

Questions for authors
  1. Starting from a shared SFT initialization and reference, how do arms trained on a shared aggregate-user-welfare reward and a shared total-welfare reward compare? Please also report the welfare difference between the two untrained calibrated references, to show how much of the measured gap comes from PPO rather than from the anchor.

  2. Can you characterize the gap over the full policy interval (Definition 2 with I = [0, 1]), where either objective may switch regimes as ρ varies? Does single-peakedness in ρ (Theorem 4.1(ii)) survive, and for which (N, c/G)?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.