acceptodds
Under review as a conference paper at ICLR 2027

Economics-Informed and Coordination-Aware Multi-Agent RL for Multi-Government Economic Policy Design

Abstract

Reinforcement learning (RL) for economic policy design faces two persistent failures beyond toy settings: learned policies can drift away from basic economic sense, and independently trained fiscal, monetary, and pension authorities can conflict after deployment. We address both with two plug-in mechanisms for economic Markov games. Economics-Informed MARL (EI-MARL) regularises each policy toward classical economic rules through an annealed KL prior and Euler consistency. Multi-Government Coordination via Latent Communication (MG-Coord) lets heterogeneous government agents exchange latent intent messages and aligns their gradients with a conflict-aware regulariser while preserving separate institutional rewards. Across representative tasks, EI-MARL improves sample efficiency 3.4 and long-run welfare 18.1%; in the three-government benchmark, MG-Coord raises GDP by 27.1% and lowers the Gini coefficient by 8 points over communication-only coordination. Together, the mechanisms outperform the strongest previously reported hybrid baseline while adding <2% per-step overhead at agents.

Primary area:
learning on time series and dynamical systems
Keywords:
LLM

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_VTFoOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes two mechanisms for reinforcement learning in economic policy design. EI-MARL regularizes government policies toward classical economic rules with a decaying penalty and adds an Euler-consistency loss for households. MG-Coord combines communication between government agents with a gradient-conflict penalty while retaining separate institutional rewards. Experiments in simulated economic environments report improvements in sample efficiency, welfare, and joint policy outcomes.

Critical strengths

The paper considers a relevant policy-design problem involving fiscal, monetary, and pension authorities with distinct objectives. Explicitly retaining these separate objectives is a potentially useful starting point for studying interactions among policy authorities.

Critical weaknesses
  1. The presentation is incomplete, and the underlying economic environment is insufficiently specified. Multiple references to sections, tables, and appendices remain unresolved, and Section 5.3 contains only a heading. More substantially, Sections 3 and 4.1 describe economic roles and interactions without specifying the budget equations, market-clearing mechanism, feasible action sets, transition dynamics, or precise reward functions. These details are repeatedly deferred to an appendix that is absent from the provided PDF; no supplementary-material link is visible on the submission page. Consequently, it is difficult to assess the economic consistency of the environment or interpret and reproduce the reported results.

  2. The methodological novelty and justification for the design choices are insufficiently established. Section 4.2 uses policy regularization toward reference rules, an approach related to established policy-guidance methods such as Distral and Kickstarting. Section 4.3 combines attention-based communication with gradient alignment. The paper does not sufficiently establish what is technically new in these components or their combination. It also provides limited justification for why these particular techniques are well matched to the stated difficulties. Although the experiments include component ablations and a parameter-sharing baseline, they do not adequately distinguish the proposed contribution from relevant existing approaches to policy guidance and coordination.

  3. The application does not establish a sufficiently clear economic insight. Economic rules provide reference policies, but their applicability to the simulated economy and the interpretation of deviations from them receive limited discussion. Likewise, the paper does not adequately connect conflicts between gradients in shared parameters to concrete policy externalities or institutional trade-offs. The experiments primarily report aggregate performance improvements without explaining the economic mechanisms behind them. Applying established techniques to economic policy design could be valuable, but the paper does not sufficiently demonstrate what new understanding of economic behavior or policy coordination this application provides.

Questions for authors
  1. Where can the complete economic model and implementation details be found? In particular, how are budget feasibility, action bounds, and market clearing implemented, and what are the exact institutional reward functions?

  2. What distinguishes the proposed mechanisms from existing policy-guidance and coordination methods, and what evidence supports their suitability for this setting beyond the reported component ablations?

  3. What economic mechanism explains the reported coordination gains, and how does alignment of shared-parameter gradients correspond to the policy trade-offs among the three authorities?

Reviewer_hsXsOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper studies economic-policy learning with multiple government institutions that retain distinct rewards. EI-MARL adds an annealed penalty toward classical economic policies and a household Euler-consistency term. MG-Coord exchanges eight-dimensional intent messages through attention and penalizes negative cosine similarity between institutional return gradients. The reported experiments cover five economic task families and a fiscal/central-bank/pension configuration. The motivation is relevant, but the supplied manuscript does not contain the environment appendix, implementation details, or the benchmark-validation results it repeatedly invokes. Several central mathematical and numerical descriptions also do not agree with one another.

The proposal is instantiated with DDPG for continuous-action institutions and households. It combines action-prior regularization, an exponentially decaying prior coefficient, eight-dimensional messages, attention aggregation, and a cosine-based gradient-conflict penalty. The empirical claims concern both learning efficiency and economic outcomes; assessing them requires the transition equations, institutional rewards, welfare aggregation, and exact update sequence, because a simulator can favor a reference policy through its construction. The present recommendation therefore concerns the completeness and internal consistency of this particular study, rather than the legitimacy of economic MARL as a research direction (pp. 3–7, Sections 3–5).

Critical strengths
  1. Relevant problem formulation. Keeping fiscal, monetary, and pension authorities' rewards distinct addresses a meaningful limitation of treating all institutions as a single benevolent controller. Sections 1 and 4.1 explain concrete sources of interference, and Section 4.3 proposes a mechanism that could be implemented without imposing an identical reward on every institution (pp. 1, 4–5).

    A formulation with distinct institutional objectives can expose conflicts that a common-reward benchmark hides. This is a useful modeling choice if the rewards and interaction rules are fully specified; the current incompleteness does not eliminate that motivation.

  2. Interpretable inductive bias. The proposed Saez/Taylor/pension reference policies and household Euler condition provide a concrete route for injecting domain knowledge. Annealing the action-prior penalty could allow useful early guidance while retaining later flexibility. This is a sensible design motivation even though the stated KL interpretation and household objective need correction (p. 4, Section 4.2, Eqs. 1–3).

    The soft-prior idea also permits a practical comparison between learning with and without a domain anchor. The appropriate contribution would be an empirically validated inductive bias rather than a claim that classical rules are universally optimal. The annealing and household ablations show that the authors recognize the need to separate these design choices (p. 7).

  3. The paper attempts to isolate communication from its conflict penalty. Table 3 includes independent, parameter-shared, communication-only, and full variants. The reported full-versus-communication-only changes are substantial: GDP 2.58e10 to 3.28e10, welfare 1.03e6 to 1.24e6, Gini 0.36 to 0.28, and survival 58 to 86 years. The GDP change is approximately 27.13%, and the Gini change is 0.08 in absolute units, consistent with an eight-point change on a 0–100 scale (p. 7, Table 3). These are promising reported results, subject to the deficiencies below.

    The arithmetic of these reported differences checks out. In particular, the Gini change is an absolute 0.08, not an 8% relative reduction; the corresponding relative reduction is approximately 22.2%. Preserving the units makes the result interpretable without overstating it. Statistical and protocol qualifications remain necessary.

Critical weaknesses
  1. Critical parts of the submitted study are absent. Section 3 defers the transition models, task catalogue, economic equations, and calibration to “Appendix ??”; Section 4.3 likewise defers pseudocode; the conclusion refers to missing reproducibility/ethics details. The supplied PDF instead ends after two reference pages. More seriously, the heading “5.3 Q3: Validation on Established Benchmarks” is followed immediately by Section 6 with no experiment or results. The abstract's strongest-hybrid comparison and 10,000-agent, under-2% overhead claims therefore lack their promised supporting section. This is not a request to expand into new tasks: it concerns evidence expressly claimed in the submission (pp. 3, 5, 8–10; abstract).

    The consequence is that neither task difficulty nor baseline suitability can be independently assessed. Listing 11 role types and a 25-task catalogue does not specify their dynamics, and a platform-agnostic method still requires an exact evaluated instance. A response should distinguish existing submitted evidence from results that would need to be newly produced.

  2. The coordination objective is not specified sufficiently to reconstruct its gradients. Equation 6 uses cosine similarity between gradients with respect to theta, whereas the text describes conflict in the shared message-encoder space and then updates psi and attention parameters. Equation 4 uses institutional encoders f_psi^g, and Section 4.1 uses institution-indexed policies, without defining precisely which parameters are shared. If gradients belong to disjoint policy parameter vectors, their cosine is not a meaningful common-space conflict measure; if the intended derivatives are each institution's return gradient through shared communication parameters, those common parameters and the higher-order differentiation must be defined. “Pooled advantage” is also unspecified despite the DDPG instantiation and the claimed preservation of separate objectives. The current description leaves the main proposed update ambiguous (pp. 4–5, Eqs. 4–6 and training loop).

    The relevant question is whether every return is differentiated with respect to the same explicitly defined vector. For example, ∇ψJg\nabla_{\psi}J_g and ∇ψJg′\nabla_{\psi}J_{g'} can be compared when ψ\psi is genuinely shared; institution-specific parameter blocks require an embedding into a common space and a defined coupling. Differentiating a cosine of such gradients also involves higher-order terms unless the gradients are detached. Detachment changes what the conflict loss can optimize. The submission does not resolve those choices, so the concern is specification, not proof that its cosine is necessarily invalid.

  3. The economic regularizer is not the KL that is written, and the complete household objective is unspecified. Equation 1 gives DKL(πθ∥πecon)D_{\mathrm{KL}}(\pi_{\theta}\Vert\pi_{\mathrm{econ}}), but the economic reference policies are deterministic and the implementation is said to use squared error on the action mean. If a policy assigns positive mass away from the reference point, it is not absolutely continuous with respect to that point mass, so the stated KL is generally infinite. Distinct deterministic DDPG actions do not solve this problem: their point-mass KL is also not a finite squared error. The squared penalty can be legitimate if it is defined directly or justified by a specified softened reference distribution. The reported convergence divergence of 0.42 nats is otherwise not operationally defined. Separately, Eq. 3 is the government loss and need not include the household Euler term; the actual missing item is a complete household loss, its Euler weight, conditional-expectation estimator, and optimization path despite the household ablation (pp. 4, 6–7, Eqs. 1–3; Section 5.1).

    For illustration, if both the learned policy and reference are specified as nondegenerate Gaussians, with the reference given by N(aecon(o),s2I)\mathcal N(a_{\mathrm{econ}}(o),s^2 I) then a quadratic mean-deviation term can be proportional to part of their KL under stated variance assumptions. The scale ss then determines both regularization strength and any reported value in nats. Without that convention, “0.42 nats” and squared action units are not interchangeable. This distinction matters to the meaning of the proposed economic prior, not just notation.

  4. The central sample-efficiency claim cannot be reconciled with the reporting protocol and figure. Table 2 says the numbers are reported at 2M training steps, but the same table and text give 2.8M, 9.4M, and 12.7M steps to reach the stated threshold. The caption may refer only to welfare/survival reporting with longer separate sample-efficiency runs, but the submission does not explain these different reporting clocks. Figure 3 runs to 12M and its caption claims a 9.4M DDPG crossing; visually, the DDPG mean remains below the horizontal threshold through the plotted range, while EI-MARL crosses near 2M rather than the stated 2.8M. The arithmetic 9.4/2.8 = 3.36 supports the rounded 3.4x ratio only if those crossing times are correct. Please specify the training/evaluation clocks, threshold, interpolation rule, and per-task values and reconcile them with the plotted curves. This is a concrete inconsistency, not a statistical-significance objection based only on a small seed count (pp. 6–7, Table 2/Figure 3, lines 290–323 and 336–338).

    The plotted crossing observation is approximate because it is read from the figure. The authors can resolve it directly with the existing numerical records: identify the exact horizontal threshold, the task or task average represented, the first-crossing rule, and the evaluation checkpoint spacing. If Figure 3 is illustrative rather than the source of the 3.4x number, that must be stated and the actual measurement shown.

  5. Aggregate results do not establish the breadth or reliability of the claimed effects. Table 2 averages five different task families without defining each welfare scale, normalization, task weight, or which economic reference rule applies to monopoly regulation and “996” work culture. Table 3 is described as a five-seed average but omits dispersion, while the ablations provide only brief percentage statements. The reader cannot determine whether improvements hold across the five tasks or are dominated by one task/scale. The claim that joint deployment beats every single-institution baseline is not accompanied by those baseline results. In addition, the 1.6 ms overhead at 1,000 agents is not a percentage overhead or evidence for the claimed percentage at 10,000 agents; the measurement denominator, hardware, training-versus-inference scope, and cost of differentiating the conflict penalty are missing (pp. 6–8, Tables 2–3, Sections 5.1–5.2 and conclusion).

These details are particularly important when comparing welfare across tax, pension, inflation, monopoly, and labor-culture settings. Mean welfare is meaningful only after the aggregation convention is specified. Likewise, a claim about eliminating institutional conflict requires the individual objectives or trade-offs, not GDP alone, because a higher GDP can coexist with deterioration in another authority’s objective. The reported multiple metrics are helpful but do not replace those definitions.

Questions for authors
  1. Which exact environment, transition/reward equations, task list, calibration procedure, and pseudocode were intended to be supplied in the referenced appendices? Can the authors identify existing submitted material supporting Section 5.3 and the 10,000-agent/strongest-hybrid claims? If those results were not submitted, the claims should be withdrawn and the remaining study assessed at its demonstrated scope (Weakness 1).

    A useful response would identify exact pages or an existing artifact and state which headline claims remain supported if the missing Section 5.3/scaling material is unavailable.

  2. In Eq. 6, what is the common parameter vector for every institutional gradient? Please give the actual shared-parameter loss, how gradients pass through other agents' messages/policies, whether second derivatives are retained, the conflict weight, and the definition of pooled advantage (Weakness 2).

    A short executable update sequence or pseudocode with parameter sharing and stop-gradient operations marked would resolve this more effectively than another qualitative account of coordination.

  3. What probability distributions make the claimed KL finite, and how exactly is the 0.42-nat statistic calculated? If training uses action-mean squared error instead, please state that objective explicitly and give the full household loss including the Euler weight and expectation estimator (Weakness 3).

    Please distinguish a distributional KL from a quadratic action penalty in both training and reporting. This clarification could repair the objective description without requiring a new method.

  4. Can the authors reconcile the 2M budget, the reported threshold times, and Figure 3 using the existing per-seed learning records? Please provide task-wise normalized welfare, seed dispersion for Table 3, and the actual single-institution and scaling comparisons underlying the corresponding claims (Weaknesses 4–5).

    Please identify whether each answer corrects a figure/caption, a reporting convention, or the underlying recorded results. The distinction determines whether clarification alone is sufficient or whether central experiments need reevaluation.

Reviewer_AQkhOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

EI-MARL guides economic policy learning with classical rules. MG-Coord penalizes gradient conflicts in shared communication parameters. The authors report improved welfare in simulation.

Critical strengths
  1. The paper addresses an important problem of central interest to the community.
  2. The proposed method is reasonable.
Critical weaknesses

The paper appears to be unfinished.

  1. The central empirical evidence is incomplete and internally inconsistent. Section 5.3 on page 8 consists only of a heading. Q4 refers to “Sec. ??”.
  2. In Figure 3, EI-MARL crosses the plotted threshold at around 2M steps, whereas DDPG remains below it through 12M steps. This contradicts the crossing times of 2.8M and 9.4M steps reported in the caption.
  3. The paper misrepresents several cited studies. For example, line 116 describes R-MABM as using 1,000 RL agents. The cited study instead uses only 1–20 RL-controlled firms.
Questions for authors

N/A

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.