acceptodds
Under review as a conference paper at ICLR 2027

Decentralized Preference-Based Reinforcement Learning Under Byzantine Attacks: Stability Analysis and Resilient Aggregation Design

Abstract

Preferences can specify cooperative goals that are hard to encode as numerical rewards. We study fully decentralized preference-based control, where each agent learns a private reward model from local comparisons and exchanges critic predictions to estimate the team-average reward signal. Byzantine messages can bias policy updates and hence the trajectories compared next; on unbounded state spaces, learning errors and state growth can reinforce each other. We propose PARC, an actor–critic method combining stability-constrained policy mixing with topology-preserving sender-screened clipping (TPSC). At each receiver, TPSC first uses all received predictions to bound every coordinate, then replaces suspicious senders' predictions by its own or shrinks them toward it. Because every sender keeps its slot and weight, consensus holds under the original graph conditions for any adapted screen, accurate or not, at any fixed positive retention level. For linear reward models and critics, under stability and excitation conditions, we prove almost-sure utility identification and bounded critic consensus and tracking, with tracking error controlled by aggregation bias. An average stationarity bound for the team objective separates aggregation, approximation, occupancy and reward-misspecification errors. On ten linear–quadratic plants, screening improves attacked gains when corrupted predictions fall outside the honest range, and reduced retention when they fall inside it. Under distance-constrained corruption, PARC outperforms Remove-then-Clip (RTC) under attack in LQ and has smaller clean-to-attack losses than Mean and RTC in VMAS flocking.

Primary area:
reinforcement learning
Keywords:
multi-agent reinforcement learning, preference-based reinforcement learning, decentralized reinforcement learning, robust aggregation, consensus

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_oAhnOpenReview ↗
Rating
3
Confidence
1
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies decentralized MAMDP with Byzantine agents and rewards learned from private preference comparisons. However, I am not very familiar with this topic.

Critical strengths

This paper studies a complex task, combining preference learning, multi-agent games, and Byzantine attacks, and seems to address it well.

Critical weaknesses
  1. For readers not familiar with the topic, it is helpful to provide a detailed MAMDP formulation to make the problem definition clearer.

  2. The experimental setup is wholly set by the authors. It is not clear why these scenarios are useful and meaningful in practice.

Questions for authors

What is the practical meaning (e.g., the application scenario) of this paper?

2 comments
AuthorsOpenReview ↗

Response to Reviewer oAhn

Thank you for the review.

W1. Formalization

The revised Sec. 2.2 and Figure 1 caption state the information flow among the four models (see our reply to Reviewer AVaL). The setting in brief:

  • System. A cooperative Markov decision process ⟨G,S,A,K,r⋆,γ⟩\langle G,S,A,K,r^\star,\gamma\rangle on a possibly unbounded state space, with nn cooperative agents and at most HH Byzantine agents whose identities are unknown.
  • Observations. All agents observe the shared states, actions and trajectories; labels, comparison records and reward models are private.
  • Preferences. On each shared pair of trajectory segments, agent i's supervisor gives a label drawn from a Bradley–Terry model with latent utility χ(Z)⊤ηi⋆\chi(Z)^\top\eta_i^\star (Eqs. (1)–(2)).
  • Byzantine agents. They may send arbitrary finite critic messages, differing across receivers and coordinates; their physical policy varies slowly (Assumption 1).
  • Objective. Maximize the discounted honest-team utility (Eq. (3)) without a trusted coordinator, while keeping the state process stable.

W2. Why these scenarios are meaningful

Both tasks abstract control problems in which operators' preferences are hard to write as rewards and communication may be tampered with. Networked LQ models coupled subsystems with an open-loop unstable plant, as in grid frequency control or traffic networks, and lets us detect instability of a learned controller exactly. VMAS flocking is a public benchmark (Bettini et al., 2022) for formation control, where each vehicle's operator judges the formation from that vehicle's own position. Plants are generated at random with seeds fixed in advance, all runs of each design are reported, and each attack targets one defense: clipping, retention, or distance-based removal.

Q. Practical applications

Three settings combine stakeholders with different priorities, communication that may be tampered with, and an existing conservative controller that is a candidate safe policy:

ApplicationCompeting prioritiesCandidate safe controller
Multi-UAV formationformation accuracy vs energy usestation-keeping controller
Connected vehicle platoonspacing vs ride comfortadaptive cruise control
Frequency regulation by distributed generatorsfrequency deviation vs operating costdroop control

Thank you for your reply. Since I'm not familiar with the topic of this paper, I maintain my score and confidence.

Reviewer_AVaLOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper studies decentralized preference-based reinforcement learning in cooperative multi-agent MDPs with heterogeneous private utilities and Byzantine communication. Agents learn from private comparisons of shared trajectories to optimize the honest agents’ average utility. The proposed PARC framework learns individual reward models and communicates reward- and value-critic predictions to construct the team signal used for policy optimization. It combines safe-policy mixing to control state growth with TPSC, a reward aggregation rule based on coordinate clipping and sender screening. For linear reward models and critics, the analysis establishes utility identification, critic boundedness and consensus with aggregation-dependent tracking error, and an approximate policy-stationarity bound accounting for aggregation, approximation, and occupancy errors. Experiments on networked LQ control and VMAS flocking demonstrate the usefulness of preference feedback and show that screening and reduced retention mitigate out-of-range and in-range attacks, respectively.

Critical strengths
  1. The formulation is interesting. Combining heterogeneous private preferences with decentralized multi-agent learning and Byzantine communication raises a meaningful information-sharing problem where the agents must construct a cooperative learning signal without observing one another’s private feedback or knowing which neighbors are trustworthy.

  2. The analysis accounts for the limitations of resilient aggregation. With heterogeneous utilities, agreement among honest critics does not necessarily imply accurate estimation of their average reward. The paper explicitly retains this aggregation bias and traces its effect on policy optimization, making clear what consensus alone can guarantee.

  3. The consensus argument does not require accurate sender screening, where honest agents may be flagged and Byzantine agents may escape detection. The Accommodation of these errors makes the guarantee more meaningful than assuming successful attacker identification.

Critical weaknesses

Writing and Presentation

The organization and presentation of the paper make the formulation and experimental evidences unnecessarily difficult to understand. For examples:

  • Section 2 introduces the concept “critic messages” before explaining their contents.

  • The distinction between the private reward model and the communicating team-reward critic, and the resulting four-model architecture (policy, private reward model, team-reward critic and the team-value critic), should be explained earlier through a clear information flow.

  • The experiments similarly need a compact setup paragraph or table covering the MDP, preference generation, networks, communication graph, attackers, and evaluation metric of each experiment. These details are currently distributed across the main text and Appendix G.

  • Table 1 emphasizes pairwise effects, while absolute clean/attacked results, reward-error plots, and some baseline comparisons require consulting separate appendix tables. Switching among cost ratios, normalized gains, and attack-induced losses further largely increases the reading burden.

Experiment Limitations

The empirical scope and its connection to the main guarantees remain limited.

  • All attacked experiments use one Byzantine sender, although the theory permits multiple simultaneous attackers under suitable graph conditions. Robustness at larger attacker budgets remains untested despite its theoretical coverage.

  • VMAS preference heterogeneity is insufficiently specified. The paper does not explicitly define each agent’s teacher utility and preference-label generation rule, making it difficult to determine whether and how their underlying preferences differ.

  • The connection to the theoretical error bounds is mostly indirect. Control performance is measured, but critic disagreement, error against the team targets, and the practical magnitude of the theoretical residuals are not directly characterized. A controlled heterogeneity study would make these guarantees more informative.

  • Task and scale coverage is limited. LQ provides useful controlled tests, and VMAS includes neural policies and critics, but the evaluation remains confined to small teams and one nonlinear task. How this work translates to larger teams, more complex tasks, or more realistic communication networks is unclear. At least one additional standard MARL environment such as SMAC or multi-agent MuJoCo is needed to establish the practical relevance of the proposed method.

Questions for authors
  1. What precisely generates each agent’s VMAS preference labels? Do agents have different underlying utility functions, and if so, how are these defined and how much do they differ?

  2. Why communicate predictions from a separate team-reward critic instead of private reward estimates followed by robust aggregation or consensus? What benefit does the additional learning of a team-reward critic for each agent provide?

  3. How practical is obtaining a safe policy satisfying the theoretical assumptions in applications beyond the evaluated environments?

5 comments
AuthorsOpenReview ↗

Response to Reviewer AVaL (1/3): presentation (P1–P4) and Q2

Thank you for the detailed review. We adopt all four presentation suggestions, and N1–N4 were designed around your experimental points. We number your points in the order of your review: P for presentation, E for experiments and Q for questions.

Your pointOur responsePart
P1–P2, Q2Critic messages defined; information flow among the four models; why the team-reward critic is separate1
P3, P4Setup table before the results (new Table 1); compared rules' means in the results table (now Table 2)1
E1, E4N1 (two attackers, H = 2) and N2 (20 agents, sparse ring)2
Q3Role of the safe policy2
E2 / Q1, E3Teacher definition and N3; critic-level measurements in N43

P1–P2. Critic messages and the four models

A critic message is the vector of predictions of agent i's post-innovation reward-critic and value-critic proposals at common probe inputs (Sec. 3.2). The revised Sec. 2.2 defines both where they first appear, the Figure 1 caption states what is exchanged, and the top row of Figure 1 traces the loop:

labels→ηi→r^i→(ωi,vi)⇄neighbors→θi→next segments\text{labels}\to\eta_i\to\hat r_i\to(\omega_i,v_i)\rightleftarrows\text{neighbors}\to\theta_i\to\text{next segments}

  • Private reward model ηi\eta_i is fit to agent i's own comparison labels (Eq. (5)) and yields the local reward estimate r^i\hat r_i. It is never communicated.
  • Team-reward critic ωi\omega_i takes r^i\hat r_i as its local innovation and is mixed with neighbors by TPSC, so it tracks the honest-team average reward. It is communicated as predictions at probes.
  • Team-value critic viv_i is updated by TD(λ\lambda) from r^i\hat r_i and mixed in the same way.
  • Policy θi\theta_i is updated from the two critics (Eq. (7)) and generates the next compared segments. It is not communicated.

Labels thus enter only through ηi\eta_i; only critic predictions are exchanged.

Q2. Why a separate team-reward critic?

Because it is the robust consensus on private reward estimates that you describe. Its recursion (Eq. (6) followed by TPSC) takes agent i's private estimate r^i\hat r_i as the innovation, mixes by robust aggregation, and targets the projection of the honest-average reward (Eq. (9)). We keep it as a separate persistent state for two reasons. First, ηi\eta_i must stay unmixed: it is fit to agent i's own labels and identifies agent i's own utility (Theorem 1). Second, a team average needs repeated mixing: on a sparse graph, one round of neighbor aggregation gives only a neighborhood average, so a persistent state has to be mixed over many rounds. That state is ωi\omega_i, and it supplies the actor's reward term directly (Eq. (7)). The revised Sec. 3.1 states this design choice.

P3. Experimental setup

The revised PDF places the LQ and VMAS settings at the start of Sec. 5 as the new Table 1; the N1, N2 and N4 settings are in Appendix H.

StudyTask, agentsPreferencesCritics / actorGraph, HAttacksMetric
LQ (Sec. 5.1–5.2)networked LQ, 10 agents, 20-dim stateBT labels, 10-step segments, agent-specific quadratic costslinear or 2×64 ReLU critics; linear Gaussian actor6-neighbor ring, 1boundary, in-interval, distance-constrainedgain (Sec. 5.2); cost ratio (Sec. 5.1)
VMAS (Sec. 5.3)flocking, 6 agentsBT labels, 32/64/192-step segments, collected in phases 0–31, then frozen reward models2×128 tanh critics; PPO actor; 448 phasescomplete, 1sparse distance-constrainednative team return
N1LQ, 10 agentsas LQ, 800 pairs per agent per phase2×64 ReLUcomplete, 2boundary, distance-constrainedgain
N2LQ, 20 agents, 40-dim stateas LQ, 800 pairs per agent per phase2×64 ReLU6-neighbor ring, 1boundary, distance-constrainedgain
N4LQ, 10 agents, frozen policiesnumeric utilities with heterogeneity τ\taulinear critics6-neighbor ring, 1boundary, in-intervalcritic diagnostics

P4. Absolute results and metrics

For each contrast, the results table (former Table 1, now Table 2) lists the two rules' mean attacked gains beside the paired improvement; its caption gives the clean-gain range and points to Table 8 for all per-rule results. Sec. 5.1 now points to the reward-error curves (Fig. 4). For the main comparison: under the boundary attack PARC's gain goes from 10.56 to 9.88 pp (94% retained) and IntervalOnly's from 10.46 to 2.49 pp (24%); under the in-interval attack PARC$_{0.1}$ goes from 10.61 to 8.68 pp (82%).

In this response, gain is 100 (J0−J)/J0100\,(J_0-J)/J_0 in percentage points (pp), where JJ is the stationary honest-team cost of the learned controller and J0J_0 the cost of the plant's common half-gain reference policy (Appendix G.1). A negative gain means the learned controller is worse than this reference. The paper reports the same quantity as a fraction (9.88 pp = 0.0988).

AuthorsOpenReview ↗

Response to Reviewer AVaL (2/3): more attackers, larger networks, safe policy (E1, E4, Q3)

N1 and N2 show the same pattern: each baseline loses most or all of its clean gain under at least one of the two attacks, whereas PARC keeps at least 84% of its clean gain under both.

E1. Multiple Byzantine senders

The theory requires a (2H+1)(2H+1)-robust graph. The LQ ring and the six-node VMAS graph are 3-robust, which supports H = 1; H = 2 requires 5-robustness, which the complete graph on the 10-agent system provides. In N1 two coordinated attackers send identical, receiver-specific messages, and PARC's distance screen flags the two farthest senders.

Gain in pp, mean over 10 plants. The last column takes each plant's lower gain before averaging.

RuleCleanBoundaryDistance-constrainedLower of the two attacks
PARC10.599.288.978.96
IntervalOnly10.57−13.988.72−13.98
RTC10.3410.30−17.52−17.52
Mean10.40−56.27−17.78−56.27
  • Prespecified comparison. PARC − IntervalOnly under the boundary attack is +23.25 pp [21.49, 25.03], positive on 10/10 plants.
  • Every baseline has a blind spot. IntervalOnly and Mean end worse than the half-gain reference (negative gain) under the boundary attack. RTC has the highest gain there, but drops to −17.52 under the distance-constrained attack. PARC loses 1.3 and 1.6 pp. In each plant's lower gain it exceeds IntervalOnly by 22.93 pp [21.24, 24.68] and RTC by 26.47 pp [24.37, 28.81], on 10/10 plants, while the four rules' clean gains are within 0.25 pp.

RTC is our message-space adaptation (Appendix G.2). The in-interval geometry, handled by reduced retention (Sec. 5.2), was not rerun at H = 2.

E4. Scale and benchmark coverage

N2 doubles the team to 20 agents on a sparse six-neighbor ring, certified 3-robust by exact enumeration of all node subsets, with five new plants. Gain in pp, mean over 5 plants:

RuleCleanBoundaryDistance-constrainedLower of the two attacks
PARC5.184.674.684.65
IntervalOnly4.831.854.671.85
RTC4.504.39−0.50−0.50
Mean4.48−6.920.67−6.92

PARC is best or tied in every column. The prespecified comparison, PARC − IntervalOnly in each plant's lower gain, is +2.81 pp [2.50, 3.11], on 5/5 plants; the differences from RTC and Mean are +5.15 [4.60, 5.57] and +11.57 [9.95, 13.35]. One attacker reaches 6 of the 19 honest receivers, so exposure is lower than in N1.

On benchmarks: the paper's claims concern a specific problem class, in which private heterogeneous preferences, corrupted critic messages and possible state instability interact. LQ exercises all three with exact stability checks. VMAS, a standard MARL benchmark (Bettini et al., 2022) included in the BenchMARL library (Bettini et al., 2024), tests the preference and aggregation components with neural actors and critics on a nonlinear task. We have not added a further benchmark, and the revised conclusion states this scope.

Q3. Is a safe policy practical?

By design, the learned policy is executed with probability at most c/(1+∣s∣)c/(1+|s|), so the safe profile dominates as the state grows (Sec. 3.1). It has to be stabilizing, not performant: LQR in our LQ tasks, and candidates such as adaptive cruise control, station-keeping or droop control elsewhere, each to be verified for its application. The guarantee covers execution with this fallback. Reported LQ performance goes further and evaluates the learned controller alone: all runs of the main comparison (240), N1 (120) and N2 (60) have finite stationary cost, which requires a stable learned controller.

AuthorsOpenReview ↗

Response to Reviewer AVaL (3/3): preferences and the link to theory (E2 / Q1, E3)

E2 / Q1. VMAS preference generation and heterogeneity

Agents share the utility form and coefficients; their preferences differ because each evaluates the utility from its own position. Let AA be the six agents plus the scripted target, pip_i agent ii's position, and dijd_{ij} the simulator's surface distance:

Φi(s)=16∑j∈A∖{i}(∥pi−pj∥2−0.1)2,ri,t=Φi(st)−Φi(st+1)−0.1∑j∈A∖{i}1{dij≤0.005}\Phi_i(s)=\tfrac{1}{6}\sum_{j\in A\setminus\{i\}}\big(\lVert p_i-p_j\rVert_2-0.1\big)^2,\qquad r_{i,t}=\Phi_i(s_t)-\Phi_i(s_{t+1})-0.1\sum_{j\in A\setminus\{i\}}\mathbf{1}\{d_{ij}\le 0.005\}

The segment utility is Ui=∑tri,tU_i=\sum_t r_{i,t}, and labels follow P(segment 1 preferred)=σ(Ui(segment 1)−Ui(segment 2))P(\text{segment 1 preferred})=\sigma\big(U_i(\text{segment 1})-U_i(\text{segment 2})\big). Each labeling phase gives every agent 256 pairs (64/64/128 of lengths 32/64/192; 192 training, 64 validation). Labels are collected in phases 0–31, after which the reward models are frozen. The revised Sec. 5.3 summarizes this, and Appendix G.7 states the definition.

N3 measures the resulting differences on 1,000 common segment pairs per checkpoint and length, over the six VMAS seeds (honest agents, last labeling checkpoint):

DiagnosticL=32L=32L=64L=64L=192L=192
Correlation of two agents' utility differences0.120.120.39
Agreement of an agent's preference with the honest-team direction67.5%68.1%75.5%

On short segments two agents' utility differences are nearly uncorrelated: they prefer opposite segments on 46–47% of pairs (37% at L=192L=192), with label probabilities differing by 8–12 pp. One agent's comparisons are therefore only a noisy proxy for the team objective. LQ is more heterogeneous, since agents also have distinct cost matrices QiQ_i and RiR_i: correlation 0.08, 47% opposite pairs, 22 pp probability difference, 64% agreement.

E3. Connection to the theoretical bounds

N4 is a controlled diagnostic of the mechanism behind Theorem 2. Policies are frozen and trajectories shared, so only aggregation differs between runs; critics are linear; and riτ=rˉ+τ(ri−rˉ)r_i^{\tau}=\bar r+\tau(r_i-\bar r) varies heterogeneity while keeping the team objective fixed. The tenth node follows the protocol with its own, non-team utility in the clean condition and sends boundary messages under attack. We report the three quantities you listed.

Critic disagreement and aggregation residual. With identical honest utilities (τ=0\tau=0, clean), PARC and IntervalOnly have disagreement and net aggregation shift below 1e−13, despite the tenth node's different utility; Mean's reward-critic shift is 3.3e−3. With heterogeneous utilities (τ=1\tau=1, boundary attack):

RuleReward-critic disagreementReward-critic net shiftValue-critic net shift
PARC0.00470.00320.0014
IntervalOnly0.00580.00590.0012
Mean0.03510.03290.0128

Net shift is the realized version of Eq. (10), ∥∑tαtΔt∥/∑tαt\lVert\sum_t\alpha_t\Delta_t\rVert/\sum_t\alpha_t over 1,000-step windows; the robust rules' net shift is 6–11 times smaller than Mean's. PARC's reward-critic shift is 0.0008, 0.0016 and 0.0032 at τ=0.25\tau=0.25, 0.50.5 and 11. The attack is defined relative to the honest spread, so this sweep varies attack amplitude together with heterogeneity.

Error against team targets. Function-space distance of the reward critic to an honest-average run (Mean without the tenth node's messages, which has zero aggregation shift and shares every other detail), at τ=1\tau=1:

RuleCleanBoundary attack
PARC0.0230.008
IntervalOnly0.0210.010
Mean0.0140.060

Under attack the robust rules stay 6–7.5 times closer to the reference than Mean (value critic: 0.055 and 0.100 versus 0.478). In the clean condition Mean is closest: the clean runs still contain the tenth node's non-team messages, and trimming costs some accuracy under heterogeneous utilities, a trade-off the paper keeps explicit. The cost does not show in control performance (clean gains within 0.25 pp in N1). We use this run rather than the analytic target of Eq. (9) because, within 20,000 critic steps, the robust rules' error to that target is still decreasing even at τ=0\tau=0 (parameter-space error 3.10 at phase 1, 1.62 at phase 20), whereas Mean under attack at τ=1\tau=1 stops improving, at 1.94.

Magnitude of the theoretical residuals. The generic constants are worst-case certificates over all admissible messages and screens. They grow exponentially with the number of cooperative agents (Lemma 14) and are not numerical predictions. The theorem contributes the structure above; the measured net shifts, 1e−3 to 6e−3 per unit of critic time for the robust rules, give the practical size of the aggregation term.

Reviewer_aLDGOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

Thispaper studies decentralized preference-based reinforcement learning in the presence of Byzantine agents. Each honest agent learns a reward model from private preference comparisons and exchanges critic predictions to estimate the honest team’s average reward. The authors propose PARC, which combines topology-preserving robust aggregation with stability-constrained policy mixing. A notable part of the work is that its consensus guarantees do not require accurate identification of malicious senders. For linear reward models and critics they establish reward identification, critic consensus and bounded tracking error, and an approximate policy-stationarity bound. Numerical experiment on linear–quadratic systems and nonlinear flocking tasks evaluate robustness under different attacks.

Critical strengths

The paper addresses an interesting coupled problem: Byzantine corruption of critic messages affects both policy updates and the trajectories used for subsequent preference learning, potentially amplifying instability on unbounded state spaces. To my mind, the main strengths are:

  1. TPSC provides a distinctive consensus guarentee that doesnt require accurate screening of malicious senders, under the stated graph and positive-retention conditions (Section3.2, Theorem 2).

  2. The theoretical analysis connects reward identification, critic consensus and tracking, and approximate policy stationarity, explicitly showing how aggregation bias propagates into the final error bound (Theorems 1–3).

  3. The experiments include informative ablations distinguishing the benefits of screening against out-of-range attacks from reduced retention against in-range attacks, alongside evaluations in both linear–quadratic and nonlinear flocking environments (section 5).

Critical weaknesses
  1. The significance of the advance over closely related work needs clearer justification. The authors acknowledge PARC adopts the stabilization mechanism of Jiang et al. (2026), which already studies resilient actor–critic learning on unbounded state spaces with critic-tracking and actor-convergence guarantees. The main additions appear to be preference identification and TPSC’s screening-independent consensus guarantee. A clearer account of the new technical challenges, and of what TPSC guarantees beyond simpler aggregation methods such as IntervalOnly, would help establish the significance of these contributions.

  2. The practical scope of the guarantees is limited by strong stability and exploration assumptions. The analysis requires a known safe policy that stabilizes execution uniformly over admissible Byzantine physical policies, slowly varying and convergent Byzantine physical behaviour, and uniform preference and critic excitation (Assumptions 1, 3–4). Explaining how these conditions can be verified/enforced would strengthen the practical relevance of the results. It would also help to clarify that the policy guarantee concerns execution with the safe fallback, rather than the standalone learned controller (Eq. (3)).

Questions for authors
  1. The paper addresses identification under evolving trajectory distributions and controls the resulting critic perturbation through Eq. (22). Which aspects of these arguments require a substantial extension of Jiang et al. (2026), beyond standard stochastic-approximation and time-scale-separation techniques?

  2. The ablations clarify when screening helps, and Appendix E.6 provides complementary fixed-input error comparisons. Can these comparisons establish conditions under which TPSC achieves tighter critic-tracking or stationarity bounds than IntervalOnly across their respective learning runs, or is the performance advantage primarily supported by the empirical results?

3 comments
AuthorsOpenReview ↗

Response to Reviewer aLDG (1/2): W1, Q1, W1b

Thank you for the careful review and the precise questions.

W1 / Q1. Increment over Jiang et al. (2026)

We adopt their synchronized safe-policy mixing and use standard stochastic-approximation tools. The new proof obligations come from closing the preference–critic–policy loop, and each can be checked against both papers.

  1. Reward identification inside the loop. Jiang et al. use numeric rewards. Here utilities are identified from comparisons whose distribution changes with the policies, and hence with corrupted messages. This adds a third time scale and block-level Markov score noise with within-block actor and physical-policy updates, controlled by the block transport conditions of Lemma 8 (Eqs. (49)–(51)). Curvature must survive the truncation that unbounded comparison features require (Lemma 9).
  2. Identification error as critic forcing. The reward-parameter error multiplies unbounded feature and trace fields and comes with no convergence rate. Eq. (22) is a standard product bound; what makes it applicable is Theorem 1 together with the critic-time window intensities of Lemma 12, which stand in for a rate.
  3. Deterministic critic radius. Their Theorem 3.10 bounds the critic error by the limit superior of path-dependent quantities of the realized consensus matrices. Our Theorem 2 gives a deterministic radius Dx/δxD_x/\delta_x with finite DxD_x (Lemma 11); under Lemma 13 it becomes a certificate in the honest-utility spread that covers all admissible messages and screens and vanishes when honest utilities coincide.
  4. Team-objective stationarity. Their Theorem 3.11 gives convergence to the limit set of a projected differential inclusion. Our Theorem 3 gives an average stationarity bound for the discounted team objective under attack, with aggregation, critic-approximation and occupancy terms separated (Lemma 19).
  5. Aggregation. They apply coordinatewise W-MSR to critic parameters. TPSC adds sender screening with replacement or shrinkage while keeping every slot and weight, and its honest-support certificate holds for every adapted screen and fixed positive retention (Appendix E.1). Messages are predictions at common probes, so neural critics need no aligned parameter coordinates.

To Q1 directly: the Robbins–Siegmund step in Theorem 1 and the product bound (22) are standard. The extension lies in establishing their hypotheses in this setting, namely items 1–2: block-level Markov noise with within-block policy updates, curvature under truncated excitation, and critic forcing by a reward error without a rate. The revised Sec. 1.2 summarizes these differences.

W1b. What TPSC guarantees beyond IntervalOnly

IntervalOnly is the TPSC member with an empty screen. What TPSC adds is that any adapted screen, accurate or not, keeps the consensus certificate, with unchanged constants at full retention and weaker ones under reduced retention (Lemma 14). Screening heuristics can therefore be added without re-deriving graph conditions. Whether a given screen helps is characterized on fixed inputs by Eqs. (114), (119) and (120), and the measured gains follow that analysis:

  • Out-of-range attacks: large gain. PARC − IntervalOnly under the boundary attack is +23.25 pp [21.49, 25.03] in N1 and +2.83 pp [2.52, 3.13] in N2. In N4 the boundary attack moves PARC's critics 1.6–2.0 times less than IntervalOnly's (function-space distance between attacked and clean runs at τ=1\tau=1: 0.017 vs 0.028 on the reward critic, 0.079 vs 0.156 on the value critic; the realized net shifts are in our reply to Reviewer AVaL).
  • Distance-constrained attacks: no gain, no loss. The difference is +0.25 pp in N1 and +0.01 pp in N2. Such a message is by construction closer than the farthest honest peer, so a farthest-sender screen seldom selects it, and the protection comes from interval clipping, as in Appendix G.5.
  • In-interval attacks: retention, not screening. Screening at full retention leaves in-range values unchanged, so it cannot remove an in-range Byzantine contribution; reduced retention is the component that acts here (C3–C4).
AuthorsOpenReview ↗

Response to Reviewer aLDG (2/2): W2, W2b, Q2

W2. Can the assumptions be verified or enforced?

  • Safe policy (Assumption 1). For a linear plant with a specified family of admissible Byzantine rows, a common quadratic Lyapunov function is an LMI feasibility problem. Eq. (39) gives one sufficient mixing constant. The choice ε(s)=1/(1+∥s∥)\varepsilon(s)=1/(1+\lVert s\rVert) used in our LQ experiments follows the argument of Jiang et al. (Proposition 3.9), where the drift is required outside a compact set; Appendix G.1 of the revised PDF states the conditions for this choice.
  • Byzantine physical behaviour. The proofs of Theorems 1–2 use the rate bound dΠ(ϖt+1,ϖt)≤CΠ aθ,td_\Pi(\varpi_{t+1},\varpi_t)\le C_\Pi\,a_{\theta,t} from Assumption 1; the limit ϖ⋆\varpi^\star enters in Theorem 3, where it fixes the objective. Some restriction on adversarial physical actions is necessary for any stability guarantee.
  • Uniform excitation (Assumptions 3–4). Under the regularity conditions of Appendices D.1 and D.3 the feature and comparison moment matrices are continuous in the execution descriptor, so on the compact executed family pointwise positivity gives uniform constants. N5 checks the pointwise condition at 30 policy points: the value and reward feature moments are full rank (231 and 496 dimensions) with smallest eigenvalues 6.0e−4 and 1.3e−4, and the bounded comparison moments are full rank on the 9-dimensional identifiable subspace at all 270 agent–policy points, with smallest eigenvalue 2.6e−3.

W2b. The guarantee concerns execution with the fallback

Agreed; the revised PDF states this after Theorem 3. The experiments go one step further than the guarantee. LQ performance is evaluated for the learned controller alone, without the fallback: every run of the main comparison (240), N1 (120) and N2 (60) has finite stationary cost, which requires a stable learned controller, and at the 30 policies sampled in N5 its spectral radius is 0.815–0.871.

Q2. Can TPSC be shown to have tighter bounds than IntervalOnly?

On fixed inputs, yes, and exactly. Eq. (119) is an identity for the shift on a given record of raw messages. When every coordinate of each Byzantine message lies outside its raw interval, Eq. (120) gives the envelope difference: screening tightens the envelope if and only if the removed Byzantine exposure exceeds the removed honest exposure. Eq. (114) is the analogue for retention. The recorded-message replays in Appendix G.5 show this margin to be positive for the distance screen on 10/10 plants and negative in conditional mean for a random screen.

Across separate learning runs the two rules generate different messages and trajectories, so the paper does not order their tracking or stationarity bounds. Theorem 2 holds with any valid DxD_x, so a smaller certified DxD_x for a restricted attack class would tighten it directly; against an unrestricted adversary, who may send in-range messages, the two worst-case certificates coincide. The cross-run advantage is therefore empirical: N1 and N2 measure it in control performance, and N4 at the critic level under shared trajectories. Appendix G.5 of the revised PDF makes this separation explicit.

I thank the authors for their clarifications and I acknowledge that I have read them. The responses address my concerns, I was already scoring the paper a 3, I believe that this is still a fair score.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General response to all reviewers

We thank the reviewers for their careful reading and concrete suggestions. We completed five additional studies in response and answer every point below. We have uploaded a revised PDF; changes are highlighted in red, and Appendix H is new.

The paper in three points.

  1. Preference data are generated by the policies, which are shaped by possibly corrupted critic messages. On unbounded state spaces, we analyze reward identification, critic consensus and execution stability within this single loop.
  2. TPSC keeps every sender's slot and weight, so its consensus certificate holds for any adapted screen, accurate or not, at any fixed positive retention.
  3. Errors are traced from preference identification through critic tracking to an average stationarity bound for the team objective, with aggregation, approximation and occupancy terms kept separate.

Additional studies. All runs of each reported design are included. Intervals are pointwise 95% paired bootstrap intervals over independent plants or policy seeds.

QuestionSettingResult
N1More attackers10-agent LQ, complete graph, H = 2, two coordinated attackers; 10 plants, 120 runsPARC − IntervalOnly under the boundary attack: +23.25 pp [21.49, 25.03], 10/10 plants. PARC is the only rule whose gain stays positive under both attacks.
N2Larger, sparser network20-agent LQ, six-neighbor ring certified 3-robust, H = 1; 5 new plants, 60 runsPARC − IntervalOnly: +2.83 pp [2.52, 3.13] under the boundary attack; +2.81 pp [2.50, 3.11] in each plant's lower gain across the two attacks, 5/5 plants.
N3How preferences differCommon segment pairs at frozen VMAS and LQ checkpointsUtility differences of two honest agents are nearly uncorrelated on short segments (r = 0.12 in VMAS, 0.08 in LQ); an agent's preference matches the honest-team direction on 64–75.5% of pairs.
N4Critic-level link to Theorem 2Frozen policies, shared trajectories, linear critics, four heterogeneity levels; 440 critic-only runsWith identical honest utilities the robust rules have zero aggregation shift. Under the boundary attack their shift is 6–11 times smaller than Mean's, and their critics stay about 5–9 times closer than Mean's to an honest-average reference run.
N5Assumption diagnostics30 sampled LQ policiesFeature and comparison moments are full rank at every point; learned-controller spectral radii are 0.815–0.871.

Revision. In the main text, the revised PDF defines critic messages and the four models where they first appear (Sec. 2.2 and the Figure 1 caption), adds a setup table ahead of the results (new Table 1), reports both compared rules' means beside each paired improvement in the results table (former Table 1, now Table 2), relates PARC to Jiang et al. (2026) in Sec. 1.2, and states the scope of the guarantee after Theorem 3 and of the experiments in the conclusion. Sec. 5 now summarizes N1–N4, Appendix G.7 defines the VMAS teachers, and Appendix H reports N1–N5 in full.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.