acceptodds
Under review as a conference paper at ICLR 2027

DSPG: Learning Distribution-Conditioned Policies for Heterogeneous-Agent Economies

Abstract

With richer micro data and cheaper computation, macroeconomic policy evaluation has shifted from aggregates to the distribution behind them. Heterogeneous-agent models are the standard tool for that question, yet solving them in general equilibrium means confronting an infinite-dimensional fixed point: household policies govern how the cross-sectional distribution of assets and income evolves, and that distribution feeds back into policies through equilibrium prices. Existing solvers face a trade-off between distributional information and computational cost: low-dimensional methods (Krusell-Smith, DeepHAM, StructuralRL) reduce the distribution to a few moments, while high-dimensional methods such as DEQN take the distribution itself as input but solve too slowly to be practical. We propose DSPG (Distribution-based Structural Policy Gradient), which introduces the complete cross-sectional distribution directly into structural reinforcement learning. In a Huggett economy with aggregate risk and an explicit market-clearing condition, DSPG attains a low Euler residual of 2.5×10⁻³ in 956 s, against 6.5×10⁻³ for the strongest convergent baseline solver. A U-Net is the best DSPG network, which needs only 783 parameters to beat a 480k-parameter MLP (614× smaller) at only 3.5% higher runtime cost, and whose parameter count does not grow with the asset grid, the dominant dimension of the distribution. DSPG is the state of the art on this economy once accuracy, runtime, and model size are considered together, and it is robust across all 75 economies of a sweep that varies aggregate risk, idiosyncratic risk, and risk aversion, without retuning.

Primary area:
reinforcement learning
Keywords:
heterogeneous-agent macroeconomics, reinforcement learning, differentiable simulation, computational economics, mean field games

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_5KvyOpenReview ↗
Rating
2(+1)
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper proposes DSPG, a way to solving heterogenerous agent economy with aggregate risk. The policy conditions on the joint distribution of assets and income. Authors also introduce a compact U-Net representation that shares the parameters along the asset grid.

Critical strengths
  • U-net is an interesting way to exploit the ordered asset grid.
  • The paper gives an interesting economically motivated computational structure.
  • The appendices give important limitations with good transparency.
Critical weaknesses
  1. Appendix B states that continuation consumption is evaluated at the current distribution and that the diagnostic uses the current return. It evaluates future-state consumption as c^(a′,e′,z′;g)\widehat c(a',e',z';g), not at the next population distribution. The appendix calls these quantities numerical diagnostics under a particular convention, but the main results interpret them as evidence of equilibrium solution accuracy.

  2. The theoretical gradient is not clearly connected to the training objective. Appendix A defines the conditional household objective as JT(θ;Eθˉ)=∑t=0T−1βt(dtθ)⊤utθ.J_T(\theta;\mathcal E^{\bar\theta}) = \sum_{t=0}^{T-1}\beta^t(d_t^\theta)^\top u_t^\theta. This weights current utility by the household’s current state distribution. Appendix B describes the original training score as J~T=∑tβtgt+1⊤u(ct),\widetilde J_T = \sum_t\beta^t g_{t+1}^{\top}u(c_t), and Figure 1 uses the updated-histogram as well. The manuscript says that this is retained for reporting, separately from the conditional household formulation, but does not establish unambiguously which loss produced the reported policy updates. Even in a one-period problem, weighting u(c0)u(c_0) by d1θd_1^\theta instead of the fixed d0d_0 changes the objective and its derivative.

  3. The comparisons do not yet establish the claimed accuracy frontier. DSPG benefits from steady state information for initialization and policy bounds, but StructuralRL uses a constant-share initialization. The DEQN implementation is a budgeted adaptation with an additional auctioneer term, and its losses are still declining at the final episode. The classical outer iteration is stopped after a documented two-cycle. These experiments characterize particular implementations and budgets, but they do not establi

Questions for authors

see weaknesses.

1 comment

Dear Reviewer 5Kvy,

The discussion phase ends in one day. We understand you are busy, but we would greatly appreciate it if you could take a look at our updated results and let us know whether your concerns have been addressed. Reviewer ucMW is also waiting for your judgment on whether the experimental concerns have been resolved in order to decide whether to raise the score. We would be very grateful for any response.

Thank you very much for your time.

Reviewer_ucMWOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper deals with heterogeneous agent-based economic models. The authors state that macroeconomic analysis requires tracking more and more expressive representations of the economy's entire distribution rather than simple aggregates. For this, agent-based models would be suitable candidates to simulate and represent the entire economy at a more detailed level, but finding an equilibrium within these models is more difficult. This difficulty has grown with richer models, and the authors then flag that existing literature either simplifies too much, or is too slow to compute. To this end, the authors propose DSPG, a method that strikes a better balance between accuracy and speed.

As I infer, this can primarily be done now that AI has the ability to process high-dimensional inputs. As the agents can effectively be given the entire "state of the economy" as an observation, they can one-shot their consumption levels without needing to iterate on the interest rate of the economy.

The authors additionally compare a U-Net approach versus a standard MLP and find that the U-Net achieves better results at a significantly reduced parameter count.

Critical strengths

It may be difficult to assess since I am not working with these types of problems on an experimental level, but the authors have managed to well exploit recent AI advances in order to effectively optimize agent based economies while keeping the resolution high.

Critical weaknesses
  • Primarily, I am a bit worried about the presentation. I feel one has to be actively working on this subject in order to properly understand the paper. As it stands, the paper does not provide a background section to pull in readers that are perhaps familiar with concepts of the paper, but are not actively working on similar work.

  • The claim that agent based models are what everyone is shifting to is, in my view, quite strong and not supported by a large amount of related work. Are the authors able to defend this statement more firmly?

Questions for authors
  • Some items are left unclear to me; what exactly is the simulated setup? A 2000 period burn-in and 4000-period panel does not tell me much. Can the authors explain more indepth how a full rollout looks like from the perspective of the agents, and how it would compare with previous methods.

  • Are the authors able to provide a more background section to help readers that are just outside of this space to familiarize themselves with the concepts that are being tackled?

  • Are the authors able to provide an overview of the metrics and how they could be interpreted? For example, in table 3 values sit extremely close together and I am left wondering if there is even any statistical significance.

Reviewer_DfcUOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes DSPG, a structural policy-learning method that conditions household policies on the full discretized wealth distribution rather than a small set of moments. It uses an analytic conditional gradient and computes prices through the market-clearing identity. In a Huggett-type economy, a compact U-Net achieves a reported Euler diagnostic of 2.52*10^(-3) in 956 seconds, outperforming the reported StructuralRL-NN and DEQN results. The paper also presents information ablations, a 75-calibration risk sweep, and a partial-equilibrium VFI comparison. However, the gradient derivation is conditional, and convergence or uniqueness of the detached-price outer fixed point is not established.

Critical strengths
  1. With the U-Net, objective, clearing procedure, budget, and seeds held fixed, the full histogram reaches 2.5210^(-3) Euler error, compared with 4.76–4.8010^(-3) for the tested compressed inputs. This directly supports the claim that the distribution contains useful policy information beyond the tested moment summaries.

2)The method incorporates economic structure rather than learning the entire solution pipeline. Household transitions, budget accounting, and market clearing are handled structurally, while learning focuses on the policy. This makes the approach interpretable as a model-based alternative to learning all dynamics from data.

  1. The U-Net is extremely parameter-efficient in reported comparison, using 783 parameters versus 480,440 for MLP-128 and achieves a lower reported Euler diagnostic. This indicates that an ordered-grid architecture can represent the policy compactly in this setting.

  2. Numerical evaluation is broad within the chosen model. Multiple seeds, a 600-run architecture/batch-size/learning-rate ablation, a 75-calibration risk sweep, Euler-error profiles, clearing residuals, and a partial-equilibrium VFI benchmark provide complementary checks. In the VFI comparison, DSPG reports 5.6612 versus VFI's 5.6534, with lower across-seed variation than the listed PPO, SAC, and DDPG results.

Critical weaknesses
  1. The gradient derivation assumes a fixed environment, excludes interpolation/clipping breakpoints, and requires derivative bounds. The paper does not establish convergence or uniqueness of the detached-price outer fixed point.

  2. Potentially unfair DEQN comparison. The DEQN is trained with a limited budget and its losses are still falling at the final episode. StructuralRL also uses a different learning-rate schedule. The headline gaps may therefore reflect training protocols as well as algorithmic differences.

  3. The experiments focus on one Huggett-type model and a 200-point asset grid. The calibration sweep shows robustness within this setup, but not in richer HANK models or under finer grids and additional aggregate states. This suggeats limited external validity.

  4. The U-Net's parameter count may not grow with grid length, but end-to-end runtime and accuracy under grid refinement are not demonstrated. The paper should also clarify whether the learned-moments baseline received comparable tuning to the full-histogram model.

Questions for authors
  1. There is the partial-equilibrium VFI comparison (Appendix C, Table 4) and wealth-profile Euler-error analyses. Can you extend this validation to the general-equilibrium setting and explain how the residual errors relate to the learned policy's equilibrium accuracy?

  2. How do runtime and numerical accuracy change with asset-grid refinement and additional idiosyncratic or aggregate states? Please distinguish the U-Net's parameter-count scaling from end-to-end computational scaling.

  3. Please rerun DEQN until convergence, or report the comparison explicitly as DSPG versus a budgeted DEQN adaptation. Please also clarify the differences in training schedules and compute budgets for StructuralRL.

  4. Can you state precisely what is guaranteed by the conditional-gradient derivation, including the fixed-environment, breakpoint, and derivative-bound assumptions? What evidence or theory supports convergence and uniqueness of the detached-price outer fixed point?

  5. Is the four-learned-moments baseline tuned with a comparable optimization budget to the full-histogram model? Please report its tuning protocol and uncertainty so the information ablation can be interpreted fairly.

4 comments

Dear Reviewer DfcU,

The discussion phase ends in one day. We understand you are busy, but we would greatly appreciate it if you could take a look at our updated results and let us know whether your concerns have been addressed. Reviewer ucMW is also waiting for your judgment on whether the experimental concerns have been resolved in order to decide whether to raise their score. We would be very grateful for any response.

Thank you very much for your time.

Dear Reviewer DfcU,

Apologies for reaching out again. Reviewer ucMW has noted that they are inclined to raise their score but would first like to see whether the other reviewers' concerns, particularly regarding the experiments, are resolved. As the discussion phase is ending very soon, we would be truly grateful for your view on whether our revised experiments address your main concern. Your response would mean a great deal to us.

Thank you very much.

I must say that the rebuttal demonstrates that the authors are responsive to technical criticism by addig meaningful experiments, clarifying the theoretical scope, and correcting methodological and reporting issues. However, there is still concern that the theoretical analysis does not establish convergence or uniqueness of the detached-price outer fixed point. In addition, although the DEQN comparison has been improved, the baseline is still described as a budgeted adaptation rather than a fully converged implementation.

Overall, the revised manuscript is substantially stronger, and I appreciate the authors' efforts. Nevertheless, I will retain my original rating as I believe the remaining limitations prevent me from recommending a full acceptance.

We thank the reviewer for the constructive assessment, and especially for acknowledging that the added experiments address the technical concerns raised previously.

We would like to clarify that this paper is not a theoretical contribution; its main contribution is improved computational efficiency. A convergence (let alone uniqueness) proof for the detached-price outer fixed point would require conditions we cannot guarantee at this level of generality. Moreover, treating households as price takers is a standard assumption, effectively part of the model specification, which is precisely why the price gradient with respect to household policies must be detached.

For the same reason, we assess convergence through numerical equilibrium-condition checks. We acknowledge that this is less rigorous theoretically and have stated it explicitly as a limitation of this study.

We thank the reviewer again for the timely response.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.