acceptodds
Desk rejected at ICLR 2027

Learning What to Share: Latent Representations for FedRL with Observation Distractors

Desk rejected at ICLR 2027, so left out of the paper lists and the map.

Abstract

Federated reinforcement learning lets agents in different environments learn a shared policy by averaging updates without exchanging trajectories. This setting becomes harder when each observation contains task-irrelevant signals: the policy must learn from returns which coordinates matter, and each client learns that lesson from a different noise realisation. We study whether the federation should share a learned representation rather than raw-observation policies. We formalise the setting as a federated exogenous block MDP, then train federated PPO with an encoder whose gradient is routed through the value loss, with or without a self￾supervised objective. The objectives include a JEPA-style multi-step predictor, a behavioural metric, and a multi-step inverse model. A linear–Gaussian analysis shows why a noise-free representation is common across clients and why policy￾gradient learning on raw observations pays a fixed-step-size penalty that grows with the number of distractors. On six MuJoCo tasks with clients that differ in physical parameters, the critic-trained encoder outperforms FedAvg, FedProx, and FedKL baselines under 64 appended Gaussian distractors. The gain is largely due to gradient routing rather than model width. Self-supervised objectives help on some tasks and hurt on others. For the predictive agent, the gain grows with the number of i.i.d. distractors, but its size changes with temporal correlation: it is reduced on several tasks at ρ = 0.99 and remains positive on most. Federation improves the critic-trained encoder on all four tasks with a single-agent comparison, although it is not the source of the basic advantage.

Primary area:
reinforcement learning
Keywords:
Federated reinforcement learning, Latent world models, observation distractors, Proximal policy optimization (PPO)

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_8NEPOpenReview ↗
Rating
1
Confidence
1
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

I do not have sufficient expertise to assess this submission reliably and have alerted the AC.

Critical strengths

None

Critical weaknesses

None

Questions for authors

None

1 comment
AuthorsOpenReview ↗

Response to Reviewer 8NEP

Thank you for your time, and for flagging your confidence level. The summary comment above lists the changes in the revision, and our replies to Reviewers Dw1n and KzaE address their points. We would be glad to answer any questions.

Reviewer_KzaEOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies federated RL in settings where the client observations contain task-irrelevant signals (so-called distractors). It addresses the question whether the federation should share a learned latent representation rather than raw-observation policies. To this end, the paper formalizes the setting as a federated exogenous block MDP and introduces a latent federated PPO with a shared encoder. The encoder is trained through the value loss only, with a stop-gradient on the actor.

A linear-Gaussian analysis shows that a noise-free representation can be safely shared across heterogeneous clients, while raw policy-gradient learning struggles as the number of distractors increases. In experiments on six MuJoCo tasks, with 64 appended i.i.d. Gaussian distractors, the critic-trained encoder outperforms the best raw baseline. The paper argues that the gain comes from critic routing rather than model width. The advantage is specific to the distractor setting and disappears when observations are clean. Notably, federation helps but is not the source of the basic advantage, since the encoder already wins in a single-agent setting.

Critical strengths

The paper studies an interesting problem and addresses a concrete gap. Client-specific distractors are a real challenge in federated RL, and the question of what to share is practically relevant. Framing the problem in terms of training signals rather than aggregation strategies is an interesting angle. While prior work on federated RL has focused on how to aggregate updates under heterogeneity, this paper investigates client-specific exogenous noise, and asks whether the shared representation can be trained to discard what is irrelevant to control.

The paper attempts to isolate routing, model capacity, module-sharing, and optimizer choice as separate factors. This makes it easier to attribute the observed improvements to specific design choices.

The authors are generally transparent and clear in how they present their findings and limitations. The paper reports cases where auxiliary objectives hurt performance, where distractors lead to better results than clean observations (a finding the authors acknowledge they cannot yet explain), and where their theoretical analysis differs from the implemented algorithm.

Critical weaknesses

The technical contribution appears somewhat incremental. Training an encoder through the critic loss while stopping gradients from the actor is already established in prior work [1,2], as is the use of self-predictive objective in the style of JEPA/SPR [3,4], or the exogenous Block MDP formalization [5]. The main novelty therefore seems to be the combination of these existing components and its extension to the federated setting.

However, the federated framing appears somewhat stronger than the empirical evidence warrants. The results suggest that federation contributes only modestly to the overall gains, as the advantage over the raw baseline also holds in the single-agent setting across all ten seeds. This leaves a gap in the paper's story: while the paper is framed around federated RL, a substantial part of the performance gains comes from the encoder and also carries over to the single-agent setting. It therefore remains unclear what the federated setting itself contributes to the broader insight.

Synthetic distractor setting. All distractors are appended i.i.d. Gaussian (or AR(1)) coordinates known a priori to be reward-irrelevant. It is unclear how well this setup reflects nuisance signals in real-world applications, which limits the generalizability of the findings. Including, for example, distractors that are partially correlated with the reward could strengthen the paper.

Auxiliary-objective comparisons are confounded by optimizer choice. Using different optimizers for different objectives makes it difficult to determine whether the observed differences are due to the objectives themselves or to the choice of optimizer.

Minor points that did not affect my overall recommendation: The paper contains several line breaks within inline equations, for example in Lines 215, 735, and 744.

Overall, these concerns currently outweigh the strengths of the paper. However, I'm open to revising my assessment based on how the authors address the concerns and questions raised in my review during the rebuttal phase.

To be fully transparent, the appendix is over 56 pages long, so I did not check every detail line by line. Instead, I spot-checked the key claims about the mechanism and the theoretical analysis.


Questions for authors
  • What do the results tell us about why critic routing works in the federated setting, beyond what is already known about critic-trained representations and JEPA/SPR-style objectives? In particular, is there evidence that the federated setting introduces a new challenge or insight that would not arise in the single-agent or standard multi-agent setting?

  • Given that the advantage holds in the single-agent setting across all ten seeds, and that federation has a significant effect on only two of the four tasks, can the authors point to a result that specifically demonstrates the value of federation? For example, is there an outcome that ≥2\geq 2 clients can achieve together but no single agent can achieve on its own?

  • How does the method perform with more complex nuisance signals, such as distractors that are partially correlated with the reward? Is there any evidence that the advantage generalizes beyond appended independent Gaussian or AR(1) noise?

  • How common and realistic are the assumptions of L2-optimal heads and small steps? How much do the actual learning trajectories deviate from the theoretical analysis when these assumptions do not hold?

1 comment
AuthorsOpenReview ↗

Response to Reviewer KzaE

Thank you for the detailed and constructive review. We answer point by point.

W1. "The technical contribution appears somewhat incremental" [1–5]. We agree that the components are known, and the revision says so.

  • Critic routing is credited to SAC-AE [1] and CURL [2] (§1, §2).
  • The predictive objective is credited to SPR and JEPA, and the formalisation to the exogenous block MDP.

We use the critic-trained encoder; we do not propose it. Our contribution is what federation does with this design:

  • (i) how averaging treats client-specific distractors, module by module, with a theory and a test in the number of clients (§5, §8);
  • (ii) what sharing costs each client, and which choices change that cost (§8);
  • (iii) controls that separate routing, width and objective, and probe the optimiser (§7).

W2, Q1, Q2. What does federation contribute? "Is there an outcome that ≥ 2 clients can achieve together but no single agent can achieve on its own?" We agree that federation is not the source of the basic advantage: trained alone, the encoder still beats PPO, and federation raises its return on all four tasks, significantly on two (Table 16). The new Section 8 (fresh seeds 30–39, App. H) measures what federation adds.

  • Such an outcome, on Ant (exploratory, post hoc; App. H.4): the server model's mean return is shown above that of every client trained alone, for all three agents (one-sided p ≤ 0.011, uncorrected). Elsewhere it is not shown above the best-placed client alone, and is sometimes below it, so we claim no such outcome there. Over all four tasks, sharing raises the global return over the clients' average alone in 7 of 8 latent-agent–task pairs.
  • A federation-specific insight (Q1): adding clients removes distractor weight from FedAvg-PPO's critic (Holm p = 0.0004), but not from its actor. The theory predicts this pooling for layers trained by regression; critic routing keeps out the policy gradient, which can pay for noise (§5).
  • A federation-specific challenge (Q1): on three of four tasks, only 5–35% of latent clients earn their solo return, and the loss falls on unusual physics (exploratory). Pairs of clients raise the lowest own return there, whether or not their physics are adjacent (Holm p ≤ 0.024). An exit rule built on this cost (seeds 40–49, App. H.5) beat a random exit for the worst-off client (Holm p ≤ 0.009), but none of its three target tests held.

W3, Q3. Synthetic distractors, and distractors "partially correlated with the reward". We added two designs (§9, App. E.9; seeds 0–9; descriptive):

  • an exogenous AR(1) term is added to the reward and shown in one distractor dimension;
  • one dimension shows a noisy copy of the previous reward.

Results:

  • In both designs, the critic-trained encoder beat FedAvg-PPO on every seed of Walker2d, Hopper and MassHopper.
  • The exogenous term roughly halved its Walker2d margin.
  • Under the exogenous term, its actor gave the correlated dimension 17–35 times the gradient share of an i.i.d. distractor, but on average under 1% of the total.

Beyond appended noise: a rotation shared by all clients, mixing state and noise, is not shown to cost the latent agents return. Per-client rotations drop both below FedAvg-PPO under full averaging; a private reading layer raises both, but not to their level alone (§8, App. H.2).

W4. The objectives are confounded by the optimiser. You were right. We ran each objective under PPO's optimiser and under its own (§7.2, App. E.8; six tasks, ten seeds).

  • Under PPO's optimiser, SimSR is above the critic-trained encoder, and the other two objectives are below it (Holm p ≤ 0.039).
  • Under their own optimisers, none is shown to differ (Holm p ≥ 0.199). By mean, the ranking reverses, though AC-State and SimSR then differ by less than 0.001.

The introduction and §7.2 now state this dependence.

Q4. How realistic are L2-optimal heads and small steps?

  • Status: App. A.1 labels each assumption's status in our runs.
  • Measured (App. A.8, seeds 30–39, descriptive): the critic's final layer is not at the least-squares optimum. Calibrated for the refit's sampling error, its loss is 16–56% above a pooled refit on Walker2d, Hopper and MassHopper, and 4–10% on Ant. The difference is only 0.5–4.5% of the returns' variance. The representation's net drift per round falls from 0.7–1.0% of its norm to 0.03–0.10%. This drift bounds the step size only from below.
  • Simulated (App. A.7): the predicted slopes in m and K survive most relaxations. Full-batch Adam weakens the encoder's slope in K, and random starts often end on a saddle plateau.
  • Exact check (App. A.6): the implementation has the sign symmetry that Theorem 4(a2) needs, in all 145 configurations tested.

Minor. The breaks in inline equations (lines 215, 735 and 744) are fixed. The appendix now opens with a guide (Tables 3–4).

We are glad to clarify anything further.

Reviewer_Dw1nOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The authors investigate representation learning in heterogeneous federated reinforcement learning (FedRL). Specifically they consider the setting when agents encounter task-irrelevant distractors on observation space. The authors propose sharing a latent encoder trained through the critic’s value loss, while preventing policy gradients from updating the representation, and compare this approach with several self-supervised objectives. Experiments across six MuJoCo tasks demonstrate substantial improvements over conventional federated PPO baselines, with gradient routing accounting for most of the gains and additional representation objectives providing mixed benefits. A idealized analysis explains how distractors increase policy-gradient variance and why sharing representations across clients can help suppress nuisance information.

Critical strengths

To me this is a very relevant topic. The strength of the paper is in the empirical results, which, if I understand correctly, demonstrate that critic-only representation learning can dramatically improve PPO under observation distractors in federated environments.

Specific Strengths:

  • Table 1 reports that the critic-trained encoder achieves up tp~8 times the return of the strongest raw-observation federated baseline across six MuJoCo tasks.

  • The ablation study in Section 7.2 shows that letting policy gradients update the encoder largely eliminates the advantage, while a much smaller critic-trained encoder retains it. Thi

  • Table 2 (page 9) measures actor sensitivity to distractors and finds that critic-routed architectures are substantially less sensitive. This supports the proposed explanation.

Critical weaknesses

Despite the strength of some empirical results, there are several things that I am concerned about:

  • The PPO baseline is really bad. It's an order of magnitude worse than the critic-trained encoder. Is it really that bad, or could a more refined parameter-tuning improve this?

  • In Section 7.2, results shsow that adding distractors actually improves the critic-trained encoder's absolute performance on three tasks. The authors acknowledge they cannot explain this (I credit them for the honesty!). This finding is very surprising. If the benefit comes primarily from removing irrelevant information, why should adding that information improve performance relative to clean observations? It suggests another optimization mechanism is involved.

  • It's not clear to me what to take from this paper with respect to federated learning. It seems that a lot of improvement comes from critic-driven representation learning and not necessarily the federation. Only the hopper example is there a significant benefit from FL. I think more work is needed to be done in theory and numerically to make the case for federated learning.

Questions for authors
  • Methodology: Given that the encoder is trained exclusively through the critic’s value loss, what ensures that the learned representation preserves all information necessary for optimal control, rather than merely for value prediction? In particular, how would the method behave when distinct states have similar values but require different optimal actions?

  • Do you now have any explanation as to what is going on in Section 7.2?!

1 comment
AuthorsOpenReview ↗

Response to Reviewer Dw1n

Thank you for the careful review, and for crediting our honesty about Section 7.2. We answer point by point.

W1. "The PPO baseline is really bad… could a more refined parameter-tuning improve this?" In part, yes, and the revision says so (§7.1, App. E.7). We swept ten FedAvg-PPO configurations on four tasks (seeds 0–4; descriptive).

  • The best configuration closes 6% of the gap on Walker2d and 33% on Hopper, where the critic-trained encoder still wins every seed.
  • It closes 72% on MassHopper.
  • On Ant, a smaller GAE λ lifts FedAvg-PPO's mean above the encoder's.

The introduction now states this, and the abstract now scopes our claim to the baselines' main settings. Without distractors, our FedAvg-PPO beats the encoder on four of six tasks (Table 15), so the large gap at the main settings arises with the distractors. Clean single-agent PPO reaches about 2,600 on Walker2d, 2,500 on Hopper, 2,000 on HalfCheetah, 7,700 on InvertedDoublePendulum and 390 on Ant after 0.75M steps (Table E.7.3). An L1 input penalty on PPO recovers 59% of the narrow critic-trained encoder's gain; the narrow encoder and AC-State still beat it in 9 of 12 agent–task pairs (App. E.4).

W2 and Q2. Why do distractors improve the critic-trained encoder? "Do you now have any explanation as to what is going on in Section 7.2?!" In part. A controlled study (§7.2, App. E.5; seeds 0–9; tests and readings fixed before the runs) found the following.

  • On Hopper and MassHopper, 64 zero-valued inputs give a significantly lower return than 64 noise inputs. The noise itself helps there, consistent with (but not proof of) noise acting as a regulariser.
  • On Walker2d, the zeros were not shown to lower the return. Appended zeros alone raised it over clean inputs (descriptive), so extra input width may explain the gain there.
  • On Walker2d and Hopper, noise held fixed within an episode does far worse than fresh noise, and even worse than clean inputs (descriptive), so held noise may harm.
  • On fresh seeds 30–39, the rise holds on average (+221, p = 0.007) but not on MassHopper (App. H.2).

The readings thus differ by task and conflict on Walker2d, so the mechanism is not established (§7.2).

W3. "It's not clear to me what to take from this paper with respect to federated learning… Only the hopper example is there a significant benefit from FL." We agree that critic routing, not federation, creates the basic advantage. In the single-agent comparison (Table 16), federation raises the encoder's return on all four tasks, significantly on Hopper and Ant. The new Section 8 (fresh seeds 30–39, App. H) measures what federation adds for each client:

  • Sharing raises the global return over the clients' average trained alone in 7 of 8 latent-agent–task pairs.
  • Picked after the fact (exploratory; App. H.4): on Ant, the server model's mean return is shown above that of every client trained alone. On the other tasks, it is not shown above the best-placed client trained alone, and is sometimes below it.
  • Adding clients removes distractor weight from FedAvg-PPO's critic (Holm p = 0.0004), but not from its actor. The theory predicts this pooling for layers trained by regression (§5).
  • There is a cost: on three of four tasks, most latent clients earn less than alone. An exit rule derived from this (seeds 40–49, App. H.5) protects the worst-off client better than a random exit (Holm p ≤ 0.009), but it was not shown to reach its three targets; the worst-off client still fell well short of its return trained alone.

Q1. Does a value-trained encoder keep what control needs? Not in general: states with equal values but different optimal actions can share a latent. Linear probes (§7.3, App. E.6; descriptive) measure what it keeps in practice.

  • The encoder's latent keeps most of the kinematic state: held-out R² of 0.89–0.96 on Walker2d, Hopper and MassHopper, but 0.55 on Ant.
  • AC-State, which targets the controllable state, keeps more on all four tasks: 0.97–0.99, and 0.90 on Ant.

So a value-trained latent keeps most of the state on these tasks, but less than an objective built for control, especially on Ant.

We hope this addresses your concerns and are happy to clarify anything further.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of the revision

Summary of the revision

We thank all reviewers. Dw1n found the topic very relevant and the empirical results the paper's strength. KzaE found the problem interesting, and noted the effort to separate the factors behind the gains and the transparent reporting. Their main questions fall into three groups. The revision answers each with new, completed studies:

  1. What does federation itself contribute? (Dw1n W3; KzaE W2, Q1, Q2) → new Section 8 and Figure 2, with Appendix H (fresh seeds 30–39). A participation rule derived from it is tested on seeds 40–49 (App. H.5).
  2. Are the comparisons fair? PPO tuning (Dw1n W1) → §7.1, App. E.7. Objectives under a matched optimiser (KzaE W4) → §7.2, App. E.8.
  3. Why do distractors help the critic-trained encoder? (Dw1n W2, Q2) → §7.2, App. E.5.

Other changes:

  • What a value-trained latent keeps (Dw1n Q1): linear probes, §7.3, App. E.6.
  • Reward-correlated distractors (KzaE W3, Q3): §9, App. E.9.
  • The theory's assumptions (KzaE Q4): status of each in our runs (App. A.1), an exact symmetry check (A.6), simulations under realistic optimisation (A.7), and measurements on trained agents (A.8).
  • Positioning (KzaE W1): SAC-AE and CURL are credited (§1, §2), and the contributions are restated (§1).
  • Two results narrow our claims, and the introduction now states both. A wider tuning sweep (seeds 0–4, descriptive) closes most of FedAvg-PPO's gap on MassHopper and lifts its mean above the critic-trained encoder's on Ant. The ranking of the self-supervised objectives depends on the optimiser.
  • Presentation (KzaE): the inline equations are fixed, and the appendix opens with a guide (Tables 3–4).

In the revised PDF, new and reworded text is shown in blue. We reply to each reviewer below.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.