- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper studies the volunteer’s dilemma: someone in a group has to perform a costly task, like checking a shared proposal, but everyone benefits once it is done, so each person hopes someone else will do it. In the model, some players follow the same AI advisor, which learns from its users’ feedback and aims to maximize their rewards (the “selfish” advisor). The other players decide on their own and gradually learn which action pays off better.
The authors compare the total welfare achieved by this selfish advisor with that of a hypothetical “selfless” advisor that aims to maximize everyone’s welfare. They call the difference the social welfare gap. When both groups continue volunteering (type 2), the two advisors have the same optimal policy within this regime, and the gap is zero. When only AI users volunteer (type 1), the gap first increases and then decreases as more people use the AI, because more of those benefiting from volunteering are included in the advisor’s objective. Higher volunteering costs increase the gap and shift its maximum toward lower AI adoption levels.
Critical strengths
-
I think the paper provides a solid theoretical analysis. Rather than simply assuming that players reach an equilibrium, the authors first study how the independent players respond to the AI advisor and then analyze the advisor's learning dynamics and convergence. Based on these results, they derive the welfare outcomes and show how the gap changes with AI adoption. I also found it interesting that the gap does not simply decrease as more players use the AI, but can first increase and then decrease, with the contribution costs affecting where the maximum occurs.
-
I appreciate that the authors also test their theoretical predictions using a real LLM rather than relying only on simulations. They fine-tune Qwen3-8B using PPO and compare the learned volunteering probabilities with the theoretical predictions, finding relatively small differences (2.23 percentage points on average). The experiments are also presented transparently, and the authors acknowledge that the welfare results are more exploratory and do not fully validate the theoretical conclusions.
Critical weaknesses
- My main concern is that the welfare results are derived separately for type 1 and type 2. Theorem 4.1 shows that the welfare gap eventually decreases as more players use the AI, but only within type 1. Figure 4(a) shows that AI users sometimes prefer type 2, so a selfish advisor could choose a different regime from the selfless one. In that case, the welfare gap could be quite different from what the paper reports. I think this is an important limitation, since it is unclear whether the main conclusion still holds when both advisors can freely choose their policies.
- I also have some concerns about the experimental validation of the welfare results. In Section 5.2, both LLMs are trained using individual user rewards, but with different pretrained reference policies. As I understand it, the selfless advisor is therefore not directly trained to maximize total welfare. This makes it somewhat unclear how strongly the experiments support the comparison between selfish and selfless advice. I think training the two advisors with their respective objectives would provide a better evaluation.
Questions for authors
Questions
- What happens if both advisors can choose their policies freely, rather than being restricted to type 1 or type 2? Figure 4(a) suggests that the selfish advisor might prefer type 2, even when the selfless advisor would prefer type 1. Would the main conclusions about the welfare gap still hold in that case?
- I was wondering how much the learning outcome depends on the initial pretrained policy. Theorem 3.1 suggests that the advisor can converge to different regimes depending on this initialization. Does this mean that the main welfare results depend on choosing a suitable reference policy, rather than describing the outcome a selfish advisor would naturally learn?
- Remark 3 mentions that total welfare can decrease with higher AI adoption, even while the welfare gap becomes smaller. How should we interpret the main finding in this case? It seems that a smaller gap does not necessarily mean that society benefits from having more AI users.
- In the welfare experiments, both LLMs are trained using individual user rewards, but with different reference policies. Would it be possible to train the selfless advisor directly using total welfare as its reward? I think this would make the comparison between selfish and selfless advice more convincing.
Suggestions for Future Work
Compared to other ICLR submissions I have reviewed, I appreciated the quality of the theoretical work. However, I sometimes found it difficult to follow the mathematical intuition behind certain steps. I think cleaner explanations of the main theoretical results compared to the extant literature would make the paper easier to follow. Similarly, while you acknowledged the current limitations of the current experiments, strengthening the empirical validation with additional experiments would be appreciated.