acceptodds
Under review as a conference paper at ICLR 2027

Vela: Nonlinear Solutions to DSGE Models via Multi-Agent Reinforcement Learning in JAX

Abstract

Dynamic stochastic general equilibrium (DSGE) models are widely used by macroeconomic institutions for policy-making and forecasting. However, traditional approaches often rely on log-linearization around a pre-specified steady state, yielding linear approximations whose validity is limited to its neighborhood. We present Vela, a JAX-based framework for defining and solving DSGE models through parallel multi-agent reinforcement learning. We implement the Kydland–Prescott RBC and Smets–Wouters DSGE models in Vela. Using OECD data for ten G20 economies from 1960 to 2026, we estimate empirical impulse responses through vector autoregression (VAR) and compare their directions with model predictions. Solutions learned via multi-agent self-play achieve higher accuracy than a conventionally solved Smets–Wouters baseline in eight of the ten G20 economies studied, with statistically significant improvements. The learned policies capture nonlinear dynamics, including interactions between simultaneous shocks, and can be meta-trained across a continuous range of economic calibration parameters. We also examine the design choices that affect learned policy quality and compare the resulting solutions with conventional DSGE solution methods and empirical data.

Primary area:
reinforcement learning
Keywords:
Multi-Agent Reinforcement Learning, Computational Economics, Dynamic Stochastic General Equilibrium (DSGE), Macroeconomic Modeling

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_UNUWOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper proposes Vela, a JAX-based MARL framework for solving DSGE models. Built on the IPPO algorithm together with several engineering design choices, it is used to solve two representative DSGE models, the Kydland–Prescott RBC and the Smets–Wouters model. The authors estimate VARs on empirical data for ten economies, impose a one-time, unanticipated 5% shock on the model side, and read the response direction off the cumulative relative IRF 15 quarters after the shock, comparing the direction agreement of Vela's responses with those implied by the VARs. Vela achieves significantly higher direction agreement than the conventional numerical solution in eight of the ten economies, and the authors further report that Vela's responses are nonlinear under large simultaneous shocks. The paper's core contributions are the innovative use of MARL to solve DSGE models, the JAX-based engineering framework Vela, and higher IRF direction agreement together with more nonlinear IRFs on representative DSGE models.

Critical strengths

(1) The paper innovatively explores representing DSGE models with multi-agent systems (MAS) and using MARL algorithms to solve for global nonlinear solutions of DSGE models, and proposes a set of validation protocols.

(2) From an engineering perspective, it provides a complete solution framework; it adopts several engineering implementation details that are worth learning from and that make the algorithm work in practice, and it is implemented in high-performance computing languages such as JAX.

Critical weaknesses

(1) The authors claim that Vela is a nonlinear "solution" to DSGE models, but the validation is carried out through the direction-agreement rate of IRFs, and the authors take the conventional numerical method's lower score on this metric as evidence that Vela is better. This is a wrong conclusion obtained from a wrong validation procedure. Both DSGE models in the paper have numerical true solutions that can be computed within a minute; the reason the conventional solution does not score well on the authors' metric is not that it solves the model incorrectly, but that the model's structure is too simplified to explain real data well. This is a problem of the model's structure rather than of the method, and the authors conflate the two by using the model's explanatory power for real data to validate the performance of a solving algorithm. This means the results themselves are problematic: we cannot regard the output as a "solution" to the DSGE model, and hence the paper's central claim that "Vela is a nonlinear solution to DSGE models" does not stand.

(2) The paper lacks sufficient ablation experiments and comparisons against baselines: baseline algorithms are only mentioned in the main text but never compared experimentally, and robustness is not verified with respect to different economic parameters or different algorithm hyperparameters. As a result, we do not know which parameters Vela is sensitive to, or whether it is robust — questions that are central to its practical value.

(3) Other, smaller issues:

(3.1) In the code, several engineering optimizations of the RL setup (some mappings, clipping, and changes to the reward function) have, in my reading, already changed the structure of the SW model itself. Are the reported nonlinear responses then produced by these changes in model structure rather than by the solution? This makes Vela's results and the conventional numerical results not directly comparable.

(3.2) There is already a large body of research using MARL to solve DSGE models (e.g., TaxAI at AAMAS); the incremental value and novelty of this work are therefore low.

(3.3) Some of the solution settings are economically rather confusing (perhaps). For example, the projection method maps proposed actions back into the feasible set, whereas in my understanding one only needs to model the current effective cash-on-hand and let the household choose expenditure shares for the next period that sum to one — in which case the policy could never leave the feasible set, and this would also be closer to how people actually decide; the projection itself is counterintuitive. Furthermore, in the code the prices observed by the household are those set by the firms in the previous period, whereas under the standard market-clearing setup the clearing price should be formed within the current period, so the household should decide based on the current price set by the firms. Having firms set prices rather than letting prices be determined by market clearing is itself a very uncommon setup. Does market clearing still hold? Does Walras' law still hold? These questions are of first-order importance from an economic perspective.

Questions for authors

(1) Can the authors provide an accuracy report for Vela's solution — for example, Euler equation errors, the gap relative to the baselines, or other self-consistency metrics?

(2) Can the authors explain how this work differs from existing research such as TaxAI?

(3) Is Vela sensitive to economic parameters and to algorithm hyperparameters? Can the authors provide robustness checks or ablation results?

4 comments

We thanks reviewer for their time and constructive feedback.

Addressing weaknesses and questions

(W1/Q1) Solution Self-Consistensy Metrics and First-order optimality

Thank you for highlighting the need for a deeper analysis of solution self-consistensy metrics. Do do so, we performed additional experiments on the first-order optimality. The first-order condition (FOC) requires that, at an interior optimum, a small feasible change in an action cannot improve expected lifetime utility: ∂Qπ(s,a)∂aj=0\frac{\partial Q^\pi(s,a)}{\partial a_j}=0. We assess local optimality by perturbing each action coordinate by h=±0.01h=\pm0.01 and simulating H=80H=80 periods, with all agents following their frozen policies thereafter. We average return improvements over two stochastic continuation branches, using matched random draws for the original and perturbed trajectories. For each parallel environment, we compute Rj=max⁡!(0,ΔJ+h+,ΔJ−h−)R_j=\max\\!\left(0,\frac{\Delta J_{+}}{h_{+}},\frac{\Delta J_{-}}{h_{-}}\right). Figure 1 tracks these residuals during training for five household and three representative-firm action coordinates across six random seeds. Residuals are averaged over evaluated states within each run; the curves show the across-run median, with interquartile bands. Residuals decline by aproximately four magnitudes from 10110^1 to 10−310^{-3}, indicating substantial improvements in local policy optimality through multi-agent self-play. Stochastic estimation and numerical constraint enforcement can contribute to the remaining residuals.

(W2/Q3) Algorithm Hyperparameters Ablation Study

We compare training configurations using agents’ base economic rewards, excluding constraint-violation and action-magnitude penalties from the evaluation metric.

We examine the constraint-violation penalty, which discourages infeasible action proposals (Figure 2), and the action-magnitude penalty, which discourages abrupt adjustments (Figure 3). Both ablations show that removing the respective penalty substantially reduces the economic rewards attained by households and firms.

Appendix E additionally examines the historical-policy population buffer size (Figure 4) and the number of parallel training environments (Figure 5). Within the tested configurations, larger policy populations and more parallel environments improve training performance.

(W2/Q3) Economic Hyperparameters Ablation Study

To assess sensitivity to economic calibration, we repeated the empirical directional-accuracy evaluation across parameter values within the training domain-randomization ranges. We varied one parameter at a time while holding the others at their range midpoints. Figure 6 reports the country-averaged accuracy difference: Δc=100(AccuracyVela,c−AccuracySW2002,c)\Delta_c=100\left(\mathrm{Accuracy}_{\mathrm{Vela},c}-\mathrm{Accuracy}_{\mathrm{SW2002},c}\right) Vela’s empirical directional-accuracy advantage remains positive across all tested calibration values, although its magnitude varies by parameter. The advantage is relatively insensitive to the shock persistence parameters ρ\rho, but varies substantially with the production elasticities αK\alpha_K and αL\alpha_L, with the largest gains occurring at intermediate values. Lower capital depreciation rates δ\delta yield smaller gains, while the advantage peaks at intermediate depreciation rates. Overall, these results suggest that Vela’s empirical advantage persists across the tested calibration ranges, with greater sensitivity to production and depreciation parameters than to shock persistence.

(W3.1) Limited Comparability with Conventional Smets Wouters

We modified the original Smets–Wouters specification to improve training convergence and reduce susceptibility to reward hacking in MARL setup. Nevertheless, the reviewer’s concern about limited comparability is well-founded. To address it, we constructed a revised conventional DSGE baseline incorporating the following features of Vela:

  • No consumption habit formation.
  • No variable capital utilization or fixed production costs, with separate capital and labor elasticities.
  • Delayed dividend payments and profit taxation.
  • A constant nominal money supply in place of the Taylor rule.
  • Vela-style calibration and five shock processes.

We solved this revised baseline using first-order perturbation in gEconpy and repeated the empirical directional-accuracy evaluation described in Section 4.1. The results are reported below:

\hline \text{Country} & \text{Productivity} & \substack{\text{Labor}\\\\\text{preference}} & \text{Investment} & \substack{\text{Money}\\\\\text{preference}} & \text{Average} \\\\ \hline \text{Türkiye} & \substack{\mathbf{+33.33}\\\\{[+29.0,+36.9]}} & \substack{-19.64\\\\{[-24.3,-14.5]}} & \substack{\mathbf{+15.77}\\\\{[+10.7,+21.1]}} & \substack{\mathbf{+14.29}\\\\{[+9.0,+19.6]}} & \substack{\mathbf{+10.94}\\\\{[+8.3,+13.6]}} \\\\ \text{Italy} & \substack{\mathbf{+33.33}\\\\{[+29.0,+36.9]}} & \substack{\mathbf{+5.65}\\\\{[+1.0,+10.8]}} & \substack{\mathbf{+5.95}\\\\{[+0.8,+11.3]}} & \substack{\mathbf{+60.71}\\\\{[+55.7,+65.2]}} & \substack{\mathbf{+26.41}\\\\{[+23.8,+29.0]}} \\\\ \text{United States} & \substack{\mathbf{+41.07}\\\\{[+36.4,+45.1]}} & \substack{-29.46\\\\{[-34.3,-24.3]}} & \substack{\mathbf{+30.65}\\\\{[+25.3,+35.9]}} & \substack{\mathbf{+50.89}\\\\{[+45.6,+55.9]}} & \substack{\mathbf{+23.29}\\\\{[+20.6,+25.9]}} \\\\ \text{Japan} & \substack{\mathbf{+33.33}\\\\{[+29.0,+36.9]}} & \substack{\mathbf{+14.29}\\\\{[+9.0,+19.6]}} & \substack{-14.58\\\\{[-18.8,-9.8]}} & \substack{\mathbf{+2.68}\\\\{[-1.8,+7.7]}} & \substack{\mathbf{+8.93}\\\\{[+6.3,+11.6]}} \\\\ \text{South Korea} & \substack{\mathbf{+37.20}\\\\{[+32.3,+41.6]}} & \substack{-8.04\\\\{[-12.7,-3.0]}} & \substack{\mathbf{+32.14}\\\\{[+27.4,+37.3]}} & \substack{\mathbf{+44.35}\\\\{[+39.1,+49.7]}} & \substack{\mathbf{+26.41}\\\\{[+23.8,+29.1]}} \\\\ \text{Australia} & \substack{\mathbf{+41.07}\\\\{[+36.4,+45.1]}} & \substack{\mathbf{+16.07}\\\\{[+10.7,+21.3]}} & \substack{\mathbf{+3.87}\\\\{[-0.7,+8.9]}} & \substack{\mathbf{+22.62}\\\\{[+17.7,+27.9]}} & \substack{\mathbf{+20.91}\\\\{[+18.2,+23.6]}} \\\\ \text{Canada} & \substack{\mathbf{+41.07}\\\\{[+36.4,+45.1]}} & \substack{\mathbf{+9.82}\\\\{[+4.9,+15.1]}} & \substack{\mathbf{+14.88}\\\\{[+9.8,+20.2]}} & \substack{\mathbf{+34.52}\\\\{[+29.3,+39.9]}} & \substack{\mathbf{+25.07}\\\\{[+22.4,+27.7]}} \\\\ \text{France} & \substack{\mathbf{+33.33}\\\\{[+29.0,+36.9]}} & \substack{\mathbf{+36.61}\\\\{[+31.3,+41.9]}} & \substack{\mathbf{+18.45}\\\\{[+13.3,+23.8]}} & \substack{\mathbf{+60.71}\\\\{[+55.7,+65.2]}} & \substack{\mathbf{+37.28}\\\\{[+34.7,+39.8]}} \\\\ \text{Germany} & \substack{\mathbf{+33.33}\\\\{[+29.0,+36.9]}} & \substack{-44.94\\\\{[-49.6,-39.8]}} & \substack{\mathbf{+4.46}\\\\{[-0.2,+9.6]}} & \substack{\mathbf{+23.21}\\\\{[+17.9,+28.6]}} & \substack{\mathbf{+4.02}\\\\{[+1.4,+6.7]}} \\\\ \text{United Kingdom} & \substack{\mathbf{+33.33}\\\\{[+29.0,+36.9]}} & \substack{-81.85\\\\{[-85.6,-77.4]}} & \substack{\mathbf{+30.95}\\\\{[+25.8,+36.3]}} & \substack{\mathbf{+9.82}\\\\{[+4.5,+15.2]}} & \substack{-1.93\\\\{[-4.6,+0.7]}} \\\\ \hline \text{Average} & \substack{\mathbf{+36.04}\\\\{[+33.4,+38.7]}} & \substack{-10.15\\\\{[-35.1,+14.8]}} & \substack{\mathbf{+14.26}\\\\{[+3.6,+24.9]}} & \substack{\mathbf{+32.38}\\\\{[+17.3,+47.5]}} & \substack{\mathbf{+18.13}\\\\{[+9.5,+26.8]}} \\\\ \hline \end{array}$$ The table shows that Vela’s average empirical directional-accuracy advantage increases from 9.69 percentage points against the original Smets–Wouters baseline to 18.13 percentage points against the revised baseline. Thus, under this metric, the original comparison was more favorable to the conventional baseline. **(W3.2/Q2) How does Vela differ from TaxAI?** Compared with TaxAI, which adopts an agent-based modeling (ABM) approach grounded in the Bewley–Aiyagari model with up to $10^5$ households agents, Vela focuses on DSGE environments with representative agents and multiple coupled economic decisions and constraints. We aim to develop a reusable framework across economic models, while TaxAI focuses exclusively on optimal taxation. We view these contributions as complementary and will expand our related-work section to cover a broader range of MARL applications to economics.

(W3.3) Does Walras law hold?

Budget accounts (namely for the household) are balanced by design via @constraint primitive:

Ybal,t=∑j=12(wj,tLj,t+rj,tKKj,t)+(1−τπ)Π~t−1Pt=Ct+It+Sreal,tY_{\mathrm{bal},t}=\frac{\sum_{j=1}^{2}\left(w_{j,t}L_{j,t}+r^K_{j,t}K_{j,t}\right)+(1-\tau_\pi)\widetilde{\Pi}_{t-1}}{P_t}=C_t+I_t+S_{\mathrm{real},t}

Here, wLwL denotes labor income, rKKr^K K capital rental income, Π~t−1\widetilde{\Pi}_{t-1} lagged nominal profits payed as dividends, and PtP_t the current price level. CtC_t and ItI_t denote real consumption and investment expenditures, while Sreal,tS_{\mathrm{real},t} is unspent income retained as money. Figure 7 shows household accounting discrepancies of order 10−610^{-6}\\% across the displayed shock trajectory over 256 parallel environments, establishing accounting consistency.

Next, we compare real production value with household income, the counterpart of income-based GDP in the Smets–Wouters model:

Ybal,t≈Qt=∑j=12pj,tYj,tPtY_{\mathrm{bal},t}\approx Q_t=\frac{\sum_{j=1}^{2}p_{j,t}Y_{j,t}}{P_t}

Although this relationship is not imposed by construction, it holds approximately in our simulations, with a one-period overproduction spike followed by rapid adjustment (Figure 8). Together, these results indicate agreement between production-, income-, and expenditure-based GDP for the learned policies.

Futhermore as reward function implies penalizing agents for proposing actions that violate market clearing contraints we investigated how this penalty term progresses over the course of PPO training. Figure 9 shows a sustained decline, indicating that proposed actions require progressively smaller feasibility corrections.

Thank you for the additional experiments and explanations; I appreciate the effort. The new results address most of my concerns, and I agree they show that Vela is robust within its current framework.

I still lean toward keeping my current score, mainly because I am not yet convinced that the accuracy evidence establishes Vela recovers the correct global solution. The perturbation-based residual is informative about local optimality, but Euler-equation errors and a comparison of policy functions against existing numerical solutions would speak more directly to this question. SW is a heavily studied model with a rich literature on higher-order perturbation and global solutions, so such a comparison should be relatively inexpensive and would considerably strengthen the paper.

On TaxAI: my mention was about model complexity. SW has no infinite-dimensional state, whereas TaxAI solves an Aiyagari model with taxation and the MARL-BC paper at last year's AAMAS solves Krusell–Smith—both substantially more complex objects for a solution algorithm. I agree that studying IRFs from MARL solutions is novel; that is precisely why I would like to see stronger evidence for their accuracy.

Reviewer_nzesOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposed a JAX-based Multi-Agent Reinforcement Learning, MARL, framework to solve the nonlinear DSGE model and eliminating the linear assumptions required in the conventional numerical method. Vela models different objects (e.g., households and firms) as different independent agents and certain constraint projections are introduced to guarantee the feasibility of the solutions. IPPO and multi-agent self-play mechanism are also deployed to enhance training stability. In addition, domain randomization is also introduced to improve generalization. The authors implemented Vela on two different models, including RBC and Smets-Wouter, illustrating its advantages over the conventional methods.

Critical strengths
  1. Valuable framework design and promising engineering implementation: This works integrated: 1. heterogeneous economic agents, 2. dynamic economic constraints, and 3. MARL training into a unified JAX framework for large-scale parallel simulation. Even though most of the key components in this framework is not new, such integration shows promising practicability.

  2. Comprehensive validation: The authors conduct tests on some complicated models and compared the simulation results to the real-world data, providing certain preliminary evidence to implementing MARL for macroeconomic modeling。

Critical weaknesses
  1. Potentially unfair comparison with baseline: The most critical issue about this manuscript is that Vela modifies several components in the original Smets-Wouters model. Such modification might make the improvement from the proposed framework become suspicious and unclear as the better accuracy against the real-world data might resulted by the revised models instead of the nonlinearity preserved. in the new framework.

  2. Limited novelty in methodology: The major components (IPPO, self-play, projection techniques, and domain randomization) in Vela are from the existing study and lack of clear contribution to methodology. Meanwhile, the authors should include more discussion and comparison about the computational efficiency and scalability of Vela against the existing solutions for DSGE models.

Questions for authors

Please see the comments in the weakness section.

2 comments

We appreciate reviewer for careful reading and constructive feedback.

Addressing weaknesses and questions

(W1) Limited comparability with vanilla Smets Wouters

We modified the original Smets–Wouters specification to improve training convergence and reduce susceptibility to reward hacking in MARL setup. Nevertheless, the reviewer’s concern about limited comparability is well-founded. To address it, we constructed a revised conventional DSGE baseline incorporating the following features of Vela:

  • No consumption habit formation.
  • No variable capital utilization or fixed production costs, with separate capital and labor elasticities.
  • Delayed dividend payments and profit taxation.
  • A constant nominal money supply in place of the Taylor rule.
  • Vela-style calibration and five shock processes.

We solved this revised baseline using first-order perturbation in gEconpy and repeated the empirical directional-accuracy evaluation described in Section 4.1. The results are reported below:

\hline \text{Country} & \text{Productivity} & \substack{\text{Labor}\\\\\text{preference}} & \text{Investment} & \substack{\text{Money}\\\\\text{preference}} & \text{Average} \\\\ \hline \text{Türkiye} & \substack{\mathbf{+33.33}\\\\{[+29.0,+36.9]}} & \substack{-19.64\\\\{[-24.3,-14.5]}} & \substack{\mathbf{+15.77}\\\\{[+10.7,+21.1]}} & \substack{\mathbf{+14.29}\\\\{[+9.0,+19.6]}} & \substack{\mathbf{+10.94}\\\\{[+8.3,+13.6]}} \\\\ \text{Italy} & \substack{\mathbf{+33.33}\\\\{[+29.0,+36.9]}} & \substack{\mathbf{+5.65}\\\\{[+1.0,+10.8]}} & \substack{\mathbf{+5.95}\\\\{[+0.8,+11.3]}} & \substack{\mathbf{+60.71}\\\\{[+55.7,+65.2]}} & \substack{\mathbf{+26.41}\\\\{[+23.8,+29.0]}} \\\\ \text{United States} & \substack{\mathbf{+41.07}\\\\{[+36.4,+45.1]}} & \substack{-29.46\\\\{[-34.3,-24.3]}} & \substack{\mathbf{+30.65}\\\\{[+25.3,+35.9]}} & \substack{\mathbf{+50.89}\\\\{[+45.6,+55.9]}} & \substack{\mathbf{+23.29}\\\\{[+20.6,+25.9]}} \\\\ \text{Japan} & \substack{\mathbf{+33.33}\\\\{[+29.0,+36.9]}} & \substack{\mathbf{+14.29}\\\\{[+9.0,+19.6]}} & \substack{-14.58\\\\{[-18.8,-9.8]}} & \substack{\mathbf{+2.68}\\\\{[-1.8,+7.7]}} & \substack{\mathbf{+8.93}\\\\{[+6.3,+11.6]}} \\\\ \text{South Korea} & \substack{\mathbf{+37.20}\\\\{[+32.3,+41.6]}} & \substack{-8.04\\\\{[-12.7,-3.0]}} & \substack{\mathbf{+32.14}\\\\{[+27.4,+37.3]}} & \substack{\mathbf{+44.35}\\\\{[+39.1,+49.7]}} & \substack{\mathbf{+26.41}\\\\{[+23.8,+29.1]}} \\\\ \text{Australia} & \substack{\mathbf{+41.07}\\\\{[+36.4,+45.1]}} & \substack{\mathbf{+16.07}\\\\{[+10.7,+21.3]}} & \substack{\mathbf{+3.87}\\\\{[-0.7,+8.9]}} & \substack{\mathbf{+22.62}\\\\{[+17.7,+27.9]}} & \substack{\mathbf{+20.91}\\\\{[+18.2,+23.6]}} \\\\ \text{Canada} & \substack{\mathbf{+41.07}\\\\{[+36.4,+45.1]}} & \substack{\mathbf{+9.82}\\\\{[+4.9,+15.1]}} & \substack{\mathbf{+14.88}\\\\{[+9.8,+20.2]}} & \substack{\mathbf{+34.52}\\\\{[+29.3,+39.9]}} & \substack{\mathbf{+25.07}\\\\{[+22.4,+27.7]}} \\\\ \text{France} & \substack{\mathbf{+33.33}\\\\{[+29.0,+36.9]}} & \substack{\mathbf{+36.61}\\\\{[+31.3,+41.9]}} & \substack{\mathbf{+18.45}\\\\{[+13.3,+23.8]}} & \substack{\mathbf{+60.71}\\\\{[+55.7,+65.2]}} & \substack{\mathbf{+37.28}\\\\{[+34.7,+39.8]}} \\\\ \text{Germany} & \substack{\mathbf{+33.33}\\\\{[+29.0,+36.9]}} & \substack{-44.94\\\\{[-49.6,-39.8]}} & \substack{\mathbf{+4.46}\\\\{[-0.2,+9.6]}} & \substack{\mathbf{+23.21}\\\\{[+17.9,+28.6]}} & \substack{\mathbf{+4.02}\\\\{[+1.4,+6.7]}} \\\\ \text{United Kingdom} & \substack{\mathbf{+33.33}\\\\{[+29.0,+36.9]}} & \substack{-81.85\\\\{[-85.6,-77.4]}} & \substack{\mathbf{+30.95}\\\\{[+25.8,+36.3]}} & \substack{\mathbf{+9.82}\\\\{[+4.5,+15.2]}} & \substack{-1.93\\\\{[-4.6,+0.7]}} \\\\ \hline \text{Average} & \substack{\mathbf{+36.04}\\\\{[+33.4,+38.7]}} & \substack{-10.15\\\\{[-35.1,+14.8]}} & \substack{\mathbf{+14.26}\\\\{[+3.6,+24.9]}} & \substack{\mathbf{+32.38}\\\\{[+17.3,+47.5]}} & \substack{\mathbf{+18.13}\\\\{[+9.5,+26.8]}} \\\\ \hline \end{array}$$ The table shows that Vela’s average empirical directional-accuracy advantage increases from 9.69 percentage points against the original Smets–Wouters baseline to 18.13 percentage points against the revised baseline. Thus, under this metric, the original comparison was more favorable to the conventional baseline.

(W2) Limited methodological novelty

Our contribution is primarily in economic modeling and computational infrastructure, rather than a new MARL algorithm. We formulate DSGE models as general-sum partially observable stochastic games and use established methods namely IPPO and population-based self-play as the optimization backbone. We apply this framework to a medium-scale DSGE specification and demonstrate nonlinear responses and higher empirical directional accuracy relative to the evaluated conventional baselines.

Reviewer_HNsKOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

DSGE models are a class of dynamic models that are widely used for macroeconomic modeling. The key thing about DSGE models is that they are "micro-founded": that is, they assume that various agents in the system (usually consumers, firms, and perhaps government) have utility functions and make decisions over time to optimize these utility functions. Solving the DSGE therefore consists in finding an equilibrium consisting of mutually optimal policies for these agents, and parameters of the system can then be calibrated so that the resulting equilibrium reflects features of empirical reality. The hope is that because DSGEs are micro-founded, after calibration, one can use the model to look at counterfactual future scenarios and have some confidence that these counterfactuals will make sense because they will still reflect equilibrium behavior. Usually DSGE models are solved by classical algorithmic techniques. Recently, there have been various attempts to use deep learning instead. This paper takes the particular approach that the DSGE model should essentially be a MARL environment; by using deep MARL techniques, one can hopefully find optimal policies. They show that using their training environment, it is possible to empirically recover solutions that agree to some extent with standard DSGE models, and also have learned models respond to shocks in an economically meaningful way.

Critical strengths

The engineering work seems quite good and useful. It will be a good building block to solving DSGE and DSGE-like models via MARL, which is a promising idea.

Critical weaknesses

The critical weakness is that the "E" in DSGE stands for "equilibrium", and the authors appear to make no attempt to test in any way whether or not they are at equilibrium. Showing agreement with existing models, and behavior that is consistent with known empirical findings, does not establish this.

For example, I don't see any explicit discussion of how they handle market-clearing constraints. There are also no tests that the agent policies are optimal given the rest of the system, for example checking if agents are best-responding or if first-order optimality conditions are satisfied. There is some discussion of projecting onto constraint sets, which enforces some feasibility constraints, but still doesn't establish strategic or market equilibrium.

These are key aspects of DSGEs. Without them, it is hard to claim that this really is a DSGE solution.

Questions for authors

Please address the issue of testing for market and strategic equilibrium.

Consider citing Hill et al., "Solving Heterogeneous General Equilibrium Economic Models with Deep Reinforcement Learning". There may be some other related work in this direction too.

Update at the end of the discussion period: My feeling about this paper remains the same. The reported results in the paper show the technique does "work" in the sense of training the agents to show economically plausible behavior, which is not a trivial achievement. The engineering work to make this happen is a useful building block. However, the lack of testing for equilibrium is a serious limitation. I would encourage the authors to try to overcome this, at least by measuring and reporting residuals or ideally by modifying the environment to enforce at least some of the equilibrium conditions. Some of the needed pieces may already be there, for example, there is a differentiable QP solver in the loop already, perhaps this could be used to exactly compute equilibrium clearing prices?

3 comments

We thank the reviewer for the careful consideration of our work. Below we address each weakness and question one by one.

Addressing weaknesses and questions

(W1) Market Clearing Constraints Handling

Constraints defined through the @constraint primitive are enforced at each step using OSQP/LBFGS projection. This ensures feasibility of the specified household constraints; prices are set by firms. We conducted additional simulations of a positive TFP shock to verify household budget accounting:

Ybal,t=∑j=12(wj,tLj,t+rj,tKKj,t)+(1−τπ)Π~t−1Pt=Ct+It+Sreal,tY_{\mathrm{bal},t}=\frac{\sum_{j=1}^{2}\left(w_{j,t}L_{j,t}+r^K_{j,t}K_{j,t}\right)+(1-\tau_\pi)\widetilde{\Pi}_{t-1}}{P_t}=C_t+I_t+S_{\mathrm{real},t}

Here, wLwL denotes labor income, rKKr^K K capital rental income, Π~t−1\widetilde{\Pi}_{t-1} lagged nominal profits payed as dividends, and PtP_t the current price level. CtC_t and ItI_t denote real consumption and investment expenditures, while Sreal,tS_{\mathrm{real},t} is unspent income retained as money. Figure 1 shows household accounting discrepancies of order 10−610^{-6}\\% across the displayed shock trajectory over 256 parallel environments, establishing accounting consistency.

Next, we compare real production value with household income, the counterpart of income-based GDP in the Smets–Wouters model:

Ybal,t≈Qt=∑j=12pj,tYj,tPtY_{\mathrm{bal},t}\approx Q_t=\frac{\sum_{j=1}^{2}p_{j,t}Y_{j,t}}{P_t}

Although this relationship is not imposed by construction, it holds approximately in our simulations, with a one-period overproduction spike followed by rapid adjustment (Figure 2). Together, these results indicate agreement between production-, income-, and expenditure-based GDP for the learned policies.

Futhermore as reward function implies penalizing agents for proposing actions that violate market clearing contraints we investigated how this penalty term progresses over the course of PPO training. Figure 3 shows a sustained decline, indicating that proposed actions require progressively smaller feasibility corrections.

(W2/Q1) First-order optimality (FOC) and strategic equilibria

Thank you for highlighting the need for a deeper analysis of solution first-order optimality. The first-order condition (FOC) requires that, at an interior optimum, a small feasible change in an action cannot improve expected lifetime utility: ∂Qπ(s,a)∂aj=0\frac{\partial Q^\pi(s,a)}{\partial a_j}=0. We assess local optimality by perturbing each action coordinate by h=±0.01h=\pm0.01 and simulating H=80H=80 periods, with all agents following their frozen policies thereafter. We average return improvements over two stochastic continuation branches, using matched random draws for the original and perturbed trajectories. For each parallel environment, we compute Rj=max⁡!(0,ΔJ+h+,ΔJ−h−)R_j=\max\\!\left(0,\frac{\Delta J_{+}}{h_{+}},\frac{\Delta J_{-}}{h_{-}}\right). Figure 4 tracks these residuals during training for five household and three representative-firm action coordinates across six random seeds. Residuals are averaged over evaluated states within each run; the curves show the across-run median, with interquartile bands. Residuals decline by aproximately four magnitudes from 10110^1 to 10−310^{-3}, indicating substantial improvements in local policy optimality through multi-agent self-play. Stochastic estimation and numerical constraint enforcement can contribute to the remaining residuals.

(Q2) Additional citation and related work

Thank you for highlighting Hill et al., Solving Heterogeneous General Equilibrium Economic Models with Deep Reinforcement Learning. We will cite this paper and discuss it in the related-work section. Their approach learns value functions and uses explicit optimization and numerical market clearing, whereas Vela directly learns agents’ policies through strategic interaction.

Thank you for your responses to my review as well as to other reviewers, which attempt to address the concerns about equilibrium. There is not enough time left in the author-reviewer discussion period for me to understand to what extent these new results do resolve my concerns. However, rest assured that I will consider them carefully during the remaining reviewer discussion period and update my review if warranted.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.