acceptodds
Under review as a conference paper at ICLR 2027

Progressive Residual Symbolic Interpretation of Multi-Agent Reinforcement Learning Policies

Abstract

Symbolic policy surrogates expose learned decision rules, but a single selected formula can obscure explanatory multiplicity and residual decision structure. We introduce a progressive residual symbolic interpretation framework for multi-agent reinforcement learning policies that characterizes a population of low-complexity symbolic explanations and then allocates residual symbolic corrections path by path using minimum description length (MDL). The resulting population is structurally diverse but functionally organized. After compression to 24 representative paths, MDL selects nonzero residual refinements for 20 and retains the Stage-I form for four. On 200 independently generated nominal episodes, the frozen refinements reduce agent-balanced NMSE by (95% CI: –), showing that the gains persist beyond the discovery trajectories. The discovered symbolic structures transfer without structural rediscovery: on held-out matched states from four independently trained target MATD3 policies, fixed-structure coefficient recalibration achieves an agent-balanced NRMSE of and . Finally, six selected symbolic controllers jointly replace all neural agents and preserve the principal closed-loop voltage-control profile across nominal and unseen % and % operating shifts. Across million raw symbolic-action evaluations, the final controller produces no native action clamps, non-finite evaluations, or rollout failures. Together, this establishes population-level characterization, progressive residual refinement, cross-policy structural reuse, and closed-loop functional validation.

Primary area:
interpretability and explainable AI
Keywords:
Multi-Agent Reinforcement Learning, Symbolic Regression, Policy Interpretability, Interpretable Control

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_nNRkOpenReview ↗
Rating
2(−1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies symbolic interpretation of multi-agent reinforcement learning (MARL) policies in the context of active voltage control on the MAPDN case33 distribution network with six photovoltaic (PV) inverter agents trained with MATD3.

Critical strengths

(1) Well-motivated and coherent conceptual framing. (2) Cross-policy structural reuse is a genuinely interesting result. (3) Closed-loop joint replacement is a strong evaluation.

Critical weaknesses

(1) Evidence is limited to a single environment, a single algorithm, and a single teacher. All conclusions rest on MAPDN case33 with six MATD3 agents. The paper's framing ("multi-agent reinforcement learning policies" generally) is considerably broader than the evidence. Nothing is known about whether structural multiplicity, MDL depth selection, or cross-policy reuse behaves similarly on larger networks (e.g., case123), other MARL algorithms (IPPO, MADDPG, QMIX), or non-power-systems tasks. (2) No comparison against alternative interpretable-policy baselines. The paper never compares against the natural competitors.

Questions for authors

(1) What is the end-to-end computational cost (CPU-hours, hardware) of Stage I + Stage II + selection + closed-loop evaluation, and how does it scale with the number of agents?

1 comment
AuthorsOpenReview ↗

Response to Reviewer nNRk: Expanded Evaluation and Computational Cost

Thank you for recognizing the paper's conceptual framing, cross-policy structural reuse, and joint closed-loop evaluation. The revised manuscript evaluates these contributions across two MARL algorithms, two network scales, independently trained teachers, and six interpretable-policy baselines, alongside a computational-cost analysis. All changes and additions are highlighted in blue in the revised manuscript. Unless otherwise indicated, line numbers refer to the revision.

1. Experimental breadth and generalizability (Critical Weakness 1). Building on the original four-target structural reuse experiment (original Sec. 4.3 and App. B.4), PRSI is now evaluated on 33-bus MATD3, 33-bus HASAC, and 141-bus MATD3, extending joint symbolic control from 6 to 22 inverters. Across all three settings, residual refinement lowers within-inverter mean held-out action NRMSE relative to Stage I for every inverter (Fig. 3, lines 421–428). MDL adapts explanation depth path by path, selecting residual corrections on 20/24, 21/25, and 78/105 paths, respectively, while keeping the compact Stage-I formulas on the remaining 35 paths, where forced refinement adds a median of 3 symbolic search-complexity units (Table 23, lines 430–437, 2190–2199). Fixed symbolic structures also transfer across four independently trained target teachers in each setting, with coefficient refitting improving held-out fidelity over direct source-coefficient reuse at the largest calibration budget. Under joint nominal MATD3 deployment, PRSI's voltage-violation-rate difference from the neural teachers stays within 0.087 percentage points on 33 buses and 0.837 on 141 buses (lines 414–418). These results demonstrate consistent residual refinement, adaptive explanation depth, and cross-teacher structural reuse across different MARL algorithms and network sizes.

2. Comparison with interpretable policies (Critical Weakness 2). We compare PRSI with six baselines—direct PySR, OLS, Lasso, CART, DAgger+CART, and VIPER—on held-out fidelity, model size, and joint closed-loop control across nine matched operating conditions (Table 1, Tables 19–20). PRSI achieves the lowest nine-condition mean voltage-violation rate among all seven methods on both MATD3 networks. On 141 buses, it achieves 4.383%, a 24.2% reduction relative to direct PySR (5.781%; paired reduction 1.399 percentage points, 95% CI [1.198, 1.600]), and below the next-best baseline, CART (5.188%). Under the two conditions combining reduced PV generation with elevated demand, the mean reduction relative to direct PySR reaches 4.546 percentage points on 141 buses and 0.273 on 33 buses (lines 408–412). On 33-bus MATD3, PRSI achieves 1.596% (paired reduction against PySR 0.090 percentage points, 95% CI [0.057, 0.122]); on 33-bus HASAC, it reduces direct PySR's violation rate by 43.6% (2.909% to 1.641%). PRSI also delivers more compact symbolic rules: within-inverter mean symbolic search complexity is lower than DSR across all three settings (Table 1, Panel B), with a 34.1–63.0% reduction at comparable cumulative search time on 33-bus MATD3 (lines 439–444). Together, these results demonstrate compact symbolic interpretation with effective joint regulation in physically coupled multi-agent systems.

3. Computational cost and hardware (Question 1). Appendix D.4, Table 24 quantifies Stage-I plus Stage-II search costs (lines 2200–2225). At 1,600 iterations per search and one search seed, estimated per-inverter costs are 104.007 min (33-bus MATD3), 107.738 min (33-bus HASAC), and 118.934 min (141-bus MATD3). Across the two MATD3 workloads, the inverter count increases 3.7-fold (6 to 22), while per-inverter search time increases by 14.4%, with aggregate search workloads of 10.40 and 43.61 hours, respectively. The recorded 141-bus Stage-I execution completed 198 searches across three seeds in 4.57 wall-hours using six workers (27.32 summed task-hours). Representative-path selection was performed on a single MacBook Pro with 32 GiB unified memory, using four worker processes and one BLAS thread per process. The 141-bus closed-loop evaluation covers nine conditions × 100 paired episode seeds × 239 control steps (lines 2106–2115). Complete process CPU times for the full pipeline were not separately recorded; we report the search-time estimates and measured 141-bus Stage-I wall time.

Reviewer_PwZcOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This work proposes a symbolic policy distillation framework for interpreting multi-agent reinforcement learning policies. It characterizes a family of symbolic explanations through Pareto-based selection and introduces an MDL-based mechanism to determine whether each expression requires additional residual refinement. The framework is evaluated in a multi-agent power voltage control environment.

Critical strengths

This work studies an important challenge which is to convert neural multi-agent policies into executable symbolic representations.

Critical weaknesses
  1. The developed framework adopts the progressive residual symbolic interpretation and mainly combines established techniques, such as symbolic regression, Pareto selection, and MDL-based model selection, without contributing to new methodology. The technical contribution of this work is limited. Also, the literature review is outdated without reviewing and comparing to SOTA symbolic interpretation studies.

  2. The evaluation is too limited to support reliable claims. It's restricted to a 6-inverter (agent) power distribution grid without meaningful comparative methods. Even in power system standard, a 33-bus power distribution grid hosting 6 inverters is a toy system, which doesn't support complex interactions.

Questions for authors
  1. How were the 6 expressions selected for closed-loop evaluation? Were the selection criteria determined independently of the final evaluation episodes?

  2. How should the reported results be interpreted in terms of behavioral equivalence rather than empirical similarity?

  3. How does path-wise residual refinement compare with alternative approaches, such as budget-matched symbolic regression and uniformly applied two-stage refinement, in terms of model fidelity and expression complexity?

  4. How to reliably evaluate the symbolic explanations in practice in front of domain experts, say for the six symbolic inverter controllers?

5 comments

The author(s) didn't provide a response, so I maintain my evaluation.

AuthorsOpenReview ↗

Response to Reviewer PwZc: Methodological Contribution and Expanded Evaluation

Thank you for recognizing the importance of extracting executable symbolic policies from multi-agent reinforcement learning. We would like to clarify the methodological contribution and present the expanded evidence for its effectiveness. All changes and additions are highlighted in blue in the revised manuscript. Line numbers refer to the revision unless indicated otherwise.

1. Methodological contribution (Critical Weakness 1). PRSI's contribution is not a new symbolic regression search algorithm, but a path-specific mechanism for determining how much symbolic structure is needed to interpret a learned policy. Existing symbolic regression, Pareto selection, and residual fitting provide useful components, but they do not themselves determine whether different explanations of the same policy warrant different refinement depths. PRSI preserves representative formulas along the complexity–fidelity frontier and compares each proposed residual correction against the zero correction using an integrated MDL criterion. The criterion jointly encodes symbolic complexity and the remaining validation error, accounting for temporal dependence within episodes (lines 222–231). Consequently, different formulas for the same inverter can receive different levels of refinement. The revision also discusses recent symbolic policy distillation with dataset aggregation (Acero & Li, 2024) and successive symbolic residual fitting (Leite & Schoenauer, 2025), clarifying the distinction between these approaches and pathwise explanation-depth selection (Section 2, lines 87–111). The value of this mechanism is directly tested through budget-matched single-stage search and uniform-refinement ablations (Section 4.2).

2. Experimental scope and meaningful comparisons (Critical Weakness 2). The revised evaluation covers 33-bus MATD3, 33-bus HASAC, and 141-bus MATD3, including two MARL algorithms and extending joint symbolic control from 6 to 22 inverters. In these networks, local inverter actions interact through shared voltage dynamics, making joint deployment a substantive evaluation beyond independent action fitting. We additionally compare PRSI against six interpretable-policy baselines: direct PySR, OLS, Lasso, CART, DAgger+CART, and VIPER, measuring held-out action fidelity, model size, and joint closed-loop voltage regulation (Table 1, Tables 19–20). PRSI achieves the lowest nine-condition mean voltage-violation rate among all seven methods on both MATD3 networks: 1.596% on 33 buses and 4.383% on 141 buses. On the 141-bus network, this is a 24.2% reduction relative to direct PySR (5.781%; paired difference 1.399 percentage points, 95% CI [1.198, 1.600]), while the next-best baseline, CART, reaches 5.188% (lines 332–347, 373–376). Across the two MATD3 networks, the advantage is particularly pronounced under reduced PV generation and elevated demand, with mean reductions of 0.273 and 4.546 percentage points relative to direct PySR (lines 408–412). These comparisons demonstrate the practical value of compact symbolic policies under joint feedback in larger, physically coupled multi-agent systems.

AuthorsOpenReview ↗

Responses to Q1–Q4: Evaluation Protocol, Ablations, and Expert Assessment

We provide the following clarifications and experimental evidence in response to the four questions.

Q1. Selection of the six deployed expressions. The selection and final evaluation were separated in the original submission (original Appendix A.5). Each candidate was screened through single-inverter substitution under eight development conditions with PV/demand shifts of up to 20%, using 50 paired episodes per condition. Eligible candidates had to complete all rollouts with finite actions and zero native clamps. Selection followed a predefined lexicographic criterion: minimizing the worst-case discrepancy from the neural teacher in voltage-violation rate, followed by voltage deviation, controllability, and episode return. The six selected formulas were frozen before joint deployment and final evaluation, which covered nominal operation and eight previously unseen ±10% and ±30% shifts, with 100 paired episodes per condition. Thus, the final shifted operating conditions were not involved in controller selection. Table 14 identifies all six expressions (lines 1065–1079, 1715–1743).

Q2. Interpretation of behavioral agreement. The paper establishes empirical behavioral preservation under joint feedback, rather than formal policy equivalence. We evaluate two complementary aspects: held-out action fidelity on identical observations and joint closed-loop performance, where the six symbolic controllers determine subsequent network states. All paired comparisons use matched external PV and demand trajectories. Under nominal operation, the symbolic-minus-neural episode-return difference is −0.000824 (paired 95% CI [−0.005566, 0.003917]), with a voltage out-of-control-rate difference of −0.000015. Across eight unseen operating shifts, the largest absolute mean return difference is 0.0132 (Table 15, lines 1740–1758). Furthermore, across 1,290,600 raw symbolic-action evaluations, the joint controller produces zero native action clamps, non-finite evaluations, or rollout failures (Table 16, lines 1759–1778). These results establish empirical behavioral agreement and numerical reliability under the evaluated joint-feedback conditions.

Q3. Benefits of selective refinement over direct search and uniform refinement. At comparable cumulative search budgets (58.84 versus 59.13 hours), PRSI reduces within-inverter mean symbolic search complexity by 34.1–63.0% relative to single-stage PySR on 33-bus MATD3, while achieving a lower joint voltage-violation rate (1.596% versus 1.623%, paired reduction 0.027 percentage points, 95% CI [0.007, 0.047]). Table 22 reports per-inverter held-out action fidelity and complexity (lines 439–445, 2176–2189). Against uniform two-stage refinement, both methods use the same residual candidate sets, but PRSI determines explanation depth independently for each path. MDL selects nonzero corrections on 119/154 paths, retaining compact Stage-I formulas on the remaining 35. Forcing corrections on these paths adds a median of 3 symbolic search-complexity units and 66.57 validation bits, for a median absolute test NRMSE change of only 0.00196 (Table 23, lines 2152–2159, 2190–2199). Across all three teacher–network settings, residual refinement reduces within-inverter mean held-out action NRMSE relative to Stage I for every inverter (Fig. 3, lines 421–428). The central benefit is adaptive allocation of symbolic complexity, producing compact representations that remain effective under joint feedback.

Q4. Expert assessment of interpretability. We conducted a blinded interpretability evaluation with five domain experts, who independently assessed 238 complete controllers across seven methods and 34 inverter policies. Their expertise spanned interpretable machine learning, power systems and control, applied mathematics, safety and reliability, and human-centered explainable AI. They examined complete deployed rules without access to method identities or performance results, using four criteria: structural readability, input-to-action traceability, physical interpretability, and behavioral auditability. The composite ratings achieved ICC(2,1) = 0.808 (95% agent-cluster bootstrap CI [0.780, 0.832]). Among the six original MATD3 inverter agents, PRSI received the highest overall interpretability scores for inverters A4 and A5 (4.10 and 4.40 out of 5, respectively), compared with the six alternative methods evaluated for each inverter. Tables 25–26 report the complete expert ratings and criterion-specific agreement (lines 2206–2213, 2226–2242, 2274–2281).

Reviewer_zBPeOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
3
Read review
Paper summary

This paper uses a 2 stage approach for symbolic discovery of equations that govern a policy. The authors start with a frozen multi-agent policy for voltage control and use PySR to fit symbolic equations that recover the original policy, using samples labeled from the policy. They propose fitting the error residual as a new symbolic equation, and chain multiple equations to minimize error. Since PySR provides multiple candidate generations, the authors propose using Minimum descrption length (MDL) as a measure to select which residual formula is useful so they do not sacrifice interpretability, if a formula proposed is very complex. The method has 2 discovery stages to fit the equation, where the first stage finds the main candidate equations and uses a pareto front of complexity vs NMRSE to retain candidates, and the 2nd stage fits the residual. Additionally, after a policy is fit for 1 stage, they reuse the same formula but change the coefficients for cross-policy transfer by minimizing MSE.

Critical strengths
  1. The MDL metric is quite useful for selecting symbolic equations to maintain interpretability. It would have been easier to just get the best fit residual but using MDL makes it more interpretable.
  2. Using the pareto front is a clever trick to select first stage equations, ensuring that the best operating point which provides the middle ground tradeoff between complexity and error is helpful in maintaining interpretability
Critical weaknesses

The paper is quite underdeveloped and does not read like a complete paper, which makes it difficult to recommend for ICLR.

  1. The paper experiments with a single domain, MAPDN case33. The authors have not cited MAPDN case33, nor have described it in detail. Considering that this is a non-standard RL environment, a detailed description should have been provided. The original paper that introduced MAPDN is present in the introduction but only mentioned. It should have been described better.

The single domain limitation severely limits evaluation of strength of proposed technique. It would have been useful to see at least a few more environments. If finding environments was challenging, it would be useful to just test with more cases from MAPDN.

  1. There is no true element of multi-agent systems as part of the contribution. To break it down, the proposed method uses a single policy's interactions to fit an equation and then learns the weights for every term for cross-policy transfer. While this shows behaviour multiplicity, it is quite obvious this is going to be the case because the controllers are exactly the same, and all do the same task. The same could also be shown for single-agent environments, which opens up a lot of comparisons and scope. I would have expected the paper to start with single-agent environments and then move to multi-agent systems.

  2. Symbolic discovery of policies has been explored for quite some time, which opens up scope of comparisons. As a paper proposing a new technique, baseline comparisons are needed to assess strengths. I can immediately think of a few baselines, like linear regression, decision trees using VIPER, PIRL (Verma et al). All these methods learn symbolic equations to describe a policy, so they serve as direct baselines. Additionally, the first learned symbolic policy could have been tested for other agents to see if relearning coefficients is necessary or not.

  3. While it is smart to fit residuals as a new formula, I am surprised that a single pass with more complexity does not just recover the same equation using 2 passes. The paper provides no comparisons with a single-pass more complex PySR run, which also weakens my evaluation of this paper.

This paper needs to go through a major revision before it can be considered for ICLR. At minimum, I expect missing baselines and more environments to show that the method transfers.

Questions for authors

Please address weaknesses.

2 comments
AuthorsOpenReview ↗

Response to Reviewer zBPe: Clarification of Multi-Agent Contributions, Structural Multiplicity, and Cross-Policy Reuse

Thank you for recognizing the value of MDL-based symbolic selection and Pareto-based representation. We clarify the multi-agent contribution, structural multiplicity, and cross-policy reuse, and have added the requested baselines and a matched-budget single-stage search comparison. All changes and additions are highlighted in blue in the revised manuscript. Unless otherwise indicated, line numbers refer to the revision.

1. MAPDN and the multi-agent contribution. The original Introduction already cited Wang et al. (2021) when introducing active voltage control of distribution networks (original p. 1, lines 43–48). The revision provides a consolidated description of the environment, local observations, reactive-power actions, and network configurations in Section 3.1 (lines 145–159). The six inverters are not identical controllers acting independently: they share a voltage-regulation objective but occupy different buses, receive location-dependent observations, and interact through shared voltage dynamics (Table 12, lines 1646–1657). Each inverter's action changes voltages subsequently observed by other agents. PRSI builds interpretable local controllers and verifies them under simultaneous multi-agent feedback, where local imitation accuracy does not necessarily predict system-level performance. The original submission already included both single-inverter substitution and simultaneous replacement of all six neural controllers (original Section 4.4 and Appendix A.5). The new baselines demonstrate the importance of this distinction. At A2, CART's jointly deployed controllers produce higher violation rates in every one of the nine operating conditions, despite lower held-out action MAE (0.00157 versus 0.00357; lines 324–331). The revision further extends the evaluation to 141-bus MATD3 with 22 jointly controlled inverters and 33-bus HASAC, covering two network scales and two MARL algorithms.

2. Structural multiplicity and cross-policy reuse. The manuscript examines two distinct properties. Structural multiplicity concerns alternative symbolic representations of the same frozen neural policy: different mathematical structures can produce closely aligned actions on identical observations. The original Section 4.1 explicitly analyzes this relationship between structural similarity and functional distance, rather than assuming that formulas fitted to the same teacher must have identical structures. Cross-policy structural reuse addresses a separate question: whether a discovered mathematical structure remains useful when the neural teacher changes through independent training. This does not follow automatically from controllers sharing a voltage-regulation objective.

Cross-inverter transfer would additionally change the physical control task, since different inverters operate at different buses with location-specific measurements and voltage sensitivities (Table 12, lines 1646–1657). Our cross-policy experiment therefore holds the physical inverter fixed while varying the independently trained neural teacher. In the original 33-bus MATD3 experiment, structures discovered from the seed-1 teacher were transferred to four independently trained target teachers (seeds 2–5), preserving each inverter's symbolic basis and STOP/CONTINUE decision (original Section 3.4 and Appendix B.4; revised Table 2, lines 378–381). Fixed-structure coefficient refitting reduces agent-balanced held-out NRMSE from 4.4488 with unchanged coefficients to 0.3213, compared with 0.4742 for affine recalibration (Table 8, lines 1535–1563). The revision further tests transfer between MATD3 and HASAC at the same physical inverters: at A1, the MATD3-derived structure performs better on both target algorithms; at A2, the HASAC-derived structure performs better on both, consistently across four target seeds (Table 2, lines 350–354, 378–390). These experiments demonstrate that symbolic structures can remain reusable across independently trained policies, with structural preferences that depend on the inverter and learning algorithm.

We further address the requested comparative evaluation through six interpretable-policy baselines and a controlled single-stage symbolic regression experiment. Line numbers refer to the revised manuscript unless marked original.

3. Interpretable-policy baselines. We benchmark six baselines: direct PySR (DSR60), OLS, Lasso, CART, DAgger+CART, and critic-weighted VIPER. PIRL requires a task-specific program grammar for these controllers, so we discuss it in Related Work (original p. 3, lines 135–139) rather than benchmarking it. Our VIPER adaptation represents the continuous inverter command using 21 discrete action levels and maps the predicted classification-tree output back to a control action (lines 1091–1105). Tables 19–20 report per-inverter action fidelity and model size, while Table 1 evaluates all seven methods under joint control across nine matched operating conditions. PRSI achieves the lowest mean voltage-violation rate among all seven methods on both MATD3 networks: 1.596% on 33 buses and 4.383% on 141 buses. On the larger network, PRSI reduces violations by 24.2% relative to direct PySR (5.781%; paired reduction 1.399 percentage points, 95% CI [1.198, 1.600]) and outperforms the next-best baseline, CART (5.188%; Table 1, lines 332–347, 373–376). These results demonstrate the effectiveness of compact symbolic policies under joint feedback across different network scales.

4. Single-stage search versus progressive residual refinement. To test whether increasing the complexity of a single PySR search could recover the benefits of progressive refinement, we conducted a budget-matched experiment on 33-bus MATD3. The direct-search control uses the union of both stages' input variables, a maximum expression size of 60, and 90 searches, with 59.13 cumulative search-hours versus 58.84 for PRSI. At this comparable budget, PRSI reduces within-inverter mean symbolic search complexity by 34.1–63.0%. Under simultaneous six-inverter deployment, PRSI also achieves a lower mean voltage-violation rate (1.596% versus 1.623%; paired reduction 0.027 percentage points, 95% CI [0.007, 0.047]). Table 22 reports per-inverter held-out action fidelity and complexity (lines 439–445, 2176–2189). This separate ablation uses a different frozen direct-search model set from the seven-method DSR60 baseline (lines 2147–2150). These results demonstrate that progressive refinement produces more compact symbolic decision rules while maintaining effective joint closed-loop regulation.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.