acceptodds
Under review as a conference paper at ICLR 2027

BRIDGE: Robustness as Dynamic Generalization without Parameter Updates

Abstract

Robust reinforcement learning (RL) uses max-min optimization to withstand worst-case disturbances, often sacrificing nominal performance even when the encountered environment admits a less conservative policy. We introduce *BRIDGE* (Bridging Robustness and In-context Decision GEneralization), which provides an alternative perspective on robustness as dynamic generalization to infer the disturbance from context and *activate a tailored policy without parameter updates*. In-context RL (ICRL) provides a natural implementation, but requires pretraining experience that teaches how disturbances affect the environment and how to counteract them. Our key insight is that the max-min training of canonical robust RL already generates this experience through a curriculum of adversaries and specialized intermediate policies. Motivated by this, we propose the *CORE* pretraining pipeline to distill these experts into an ICRL model and use the final robust controller to collect its initial deployment context. These initial interactions preserve the robust baseline's behavior with minimal risk while providing context for tailored control. Having used ICRL to reduce conservatism around a single nominal task, we take BRIDGE one step further: using adversarial experience to strengthen ICRL's own generalization to unseen tasks under deployment disturbances. This introduces a critical challenge that the context must now reveal both what a new task requires and how its execution is disturbed. To support this joint inference, we propose the *ADAPT* architecture for policy transformers to learn to disentangle task and disturbance information from CORE curricula across training tasks. Our analysis quantifies when contextual specialization improves worst-case return over a conventional robust policy, including on unseen nominal tasks. Experiments on Dark Room, Meta-World, and 11 continuous-control tasks demonstrate improved robust control and unseen-task generalization, with and without disturbances.

Primary area:
reinforcement learning
Keywords:
In-context Learning, Generalization, Robustness, Max-min Optimization, Reinforcement Learning, In-context RL

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_oSR9OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes BRIDGE, an in-context reinforcement learning approach that addresses the tradeoff between robustness and performance via online adaptation without parameter updating using context information fed in to a transformer network. Using a two stage-training, the BRIDGE trains a transformer network that can adaptively change its behavior based on observations, instead of having a single policy across all situations. The algorithm was empirically demonstrated in simulated environments against several different baselines.

Critical strengths
  • The problem is well formulated and addresses the important question of trading off robustness vs performance
  • The core insight is straightfoward and compelling.
  • Strong theoretical analysis with guarantees
  • Well designed ablation studies and experimentation step across various different tasks and benchmarks.
  • Clear writing and easy to understand sentences / figures.
Critical weaknesses
  • One of the building blocks for this paper is using In-Context Reinforcement via Transformer Network trained by adversarial curriculum. The former was proposed in DPT(Lee), the later in RARL(Pinto et al 2017). While the authors did an extensive ablation study and theoretical proof on why this would work, I think the paper could be made stronger why this would be the best combination of ideas to use for robust in-context RL and share their insights in this regard.
  • Furthermore, while the adversarial learning setup by RARL is an max-min formulation, Genetic Curriculum (Song et al 2022, 2025) showed that from a curricular point of view, this may not be the best due to adversarial examples often exceeding trainee's capacity to learn new behaviors at a given time. The work instead proposed a curricular approach to gradually introduce different types of adversarial approaches. Given how the proposed work is also dependent on generating a good training curriculum, I am curious on how other adversarial curricular learning approaches could be incorporated to this paper, and whether the approach proposed by the paper is indeed the best.
  • Other than adversarial sampling, there's also diversity based sampling (Eysenbach et al, 2018) and curiosity based sampling (Pathak et al 2017). Given how robustness is often associated with how well the algorithm performs against a variety of different conditions, it might have been better if the paper explored with various different approaches towards how to make a training curriculum.
  • As RARL is already an concept that existed for several years, I think the paper could be made much more impact and be more novel if the authors could take one more step and show that adversarial training is whether the best match for training DPT.
Questions for authors

My request to the authors would be to address the points made in the critical weaknesses.

10 comments

Dear Reviewer oSR9,

We sincerely thank the reviewer for the encouraging assessment of our problem formulation, core insight, theory, and experiments.

The four comments raise two closely related themes, which we address together:

  • THEME I. the optimality of our design and implementation as well as our technical (algorithmic and architectural) contributions in addition to the idea/paradigm and theoretical contributions (W1 and W4).
  • THEME II. the comparison with alternative curricula to the adversarial one (W2 and W3).

THEME I. Optimality of the design and our contributions (W1 and W4).

W1: "One of the building blocks for this paper is using In-Context Reinforcement via Transformer Network trained by adversarial curriculum. The former was proposed in DPT(Lee), the later in RARL(Pinto et al 2017). While the authors did an extensive ablation study and theoretical proof on why this would work, I think the paper could be made stronger why this would be the best combination of ideas to use for robust in-context RL and share their insights in this regard."

W4: "As RARL is already an concept that existed for several years, I think the paper could be made much more impact and be more novel if the authors could take one more step and show that adversarial training is whether the best match for training DPT."

These two comments raise two closely related questions, which we answer in turn.

Q1. Our instantiations are deliberately straightforward, and they fit together naturally

We appreciate the reviewer's interest in whether this is the best combination, and we would like to gently clarify that we do not claim it is, nor is proving its optimality within our intended contribution.

As the first work to implement BRIDGE, we deliberately chose straightforward, canonical instantiations, so that the gains can be attributed to the proposed frameworks (BRIDGE and CORE) themselves, rather than to a strong individual component. Exploring more advanced instantiations is important follow-up work.

That said, the reviewer's question is indeed important, and our theoretical analysis gives a principled answer: a desired curriculum for BRIDGE must provide three things

  • coverage of all disturbances (R1),
  • targeting of the disturbances that decide the worst case (R2),
  • high-quality expert labels (R3).

These requirements are explained and derived in detail in our response to W2 and W3 below, where they also serve as the criterion for comparing alternatives. Our choices are in fact well suited to them:

  • Why supervised pretraining. It is one of the cleanest ICRL frameworks. Its main lever is the pretraining data distribution, which is exactly what CORE designs. Because the transformer imitates each disturbance's optimal action, the gap to the disturbance-specific optimal policy is easy to control, and Theorem 5.1 separates expert, context, and pretraining error. This is what makes R1 to R3 the right design targets.
  • Why a RARL-style max-min process. It is the canonical, well-understood way to generate an adversarial curriculum [1], as the reviewer notes, and we adapt it for ICRL data generation (Q2). Its adversary finds the disturbances where the worst case is decided (R2), and its warm-started experts trained to convergence give accurate labels (R3). Phase I adds coverage (R1).
  • An additional benefit: adversarial variation also improves ICRL's generalization to unseen tasks even without disturbances (Fig. 3b, Fig. 4a and 4c).

We thank the reviewer for prompting us to make this connection explicit, and we will add it to the paper.

Q2. Our contributions go beyond combining DPT and RARL, at both the idea level and the technical level

  • Idea and paradigm. BRIDGE reframes robustness as in-context generalization without parameter updates, and CORE turns the byproducts of max-min training, which robust RL normally discards, into pretraining data. These contributions hold for any viable choice of the components.
  • Technical A: ADAPT, a new architecture. We do not directly use the DPT model [2]. Instead, we propose ADAPT, which infers the task and the disturbance jointly from context, using separate task and adversary latents with three auxiliary objectives. Trained on the same CORE data, it raises return over a single-stream model from 6.1 to 13.9 on disturbed Meta-World and from 157.2 to 186.0 on Quadruped-Run (Table 2).
  • Technical B: CORE has Phases I and II supported by both theoretical analysis and empirical evidence, and CORE modifies RARL in Phase II for ICRL data generation. RARL takes a few alternating gradient steps per round and keeps only one final policy. CORE instead warm-starts each expert from the previous one and trains it to convergence against a frozen adversary, so each expert is a high-quality label. It also keeps every (adversary, expert) pair, which is exactly the data supervised pretraining needs, and seeds its runs from Phase I experts. Notably, training to convergence is also why GC's [3] capacity concern does not arise here (Q3).

THEME II. Comparison with alternative curricula (W2 and W3).

W2: "Given how the proposed work is also dependent on generating a good training curriculum, I am curious on how other adversarial curricular learning approaches could be incorporated to this paper, and whether the approach proposed by the paper is indeed the best."

W3: "Given how robustness is often associated with how well the algorithm performs against a variety of different conditions, it might have been better if the paper explored with various different approaches towards how to make a training curriculum."

This is a great question. In essence, both comments ask which other curricula could be used in CORE, and whether they would be better. Our theoretical analysis gives a precise yardstick for comparing curricula. BRIDGE's worst-case return is min⁡ψ(J⋆(ψ)−regret(ψ))\underset{\psi}{\min} \left( J^\star(\psi) - \text{regret}(\psi) \right), where J⋆(ψ)J^\star(\psi) is the best return attainable under disturbance ψ\psi, and the regret at ψ\psi shrinks with the number of nearby experts (proof of Lemma G.6).

Notably, our theoretical analysis reveals that a desired curriculum should provide:

  • (R1) Coverage: experts near every disturbance.
  • (R2) Targeting: more experts where J⋆(ψ)J^\star(\psi) is low, since those disturbances set the worst case.
  • (R3) Label quality: experts close to their matching disturbance's optimum, which is the expert error in Theorem 5.1.

CORE meets these by design:

  • Phase I samples uniformly (R1).
  • Phase II's adversary sends each new expert to where the current expert is weakest (R2).
  • Experts are trained to convergence (R3).

In particular, the choice of curriculum decides whether both R1 and R2 can be satisfied. To understand why both of them are necessary, we provide an illustrative toy example below.

Toy example. Three disturbances A, B, and C, with best attainable returns J⋆J^\star of 40, 60, and 90, and a budget of 12 experts. The regret at a disturbance is 60 divided by the number of experts near it. How to read it: each bar's full length is J⋆J^\star (one block is 5 return units), the solid part █ is BRIDGE's return, the light part ░ is the regret, and ? means no guarantee. The worst-case guarantee of a curriculum is its shortest solid bar.

Curricula Defined by #Experts at A / B / CA (J⋆J^\star = 40)B (J⋆J^\star = 60)C (J⋆J^\star = 90)Worst-case Guarantee
R1 only (4 / 4 / 4)█████░░░█████████░░░███████████████░░░25
R2 only (12 / 0 / 0)███████░??????????????????????????????none
R1 + R2 (6 / 4 / 2)██████░░█████████░░░████████████░░░░░░30
  • R1 only: experts are spread evenly, so the hard disturbance A loses as much as the easy C, and A's short bar sets the worst case at 25.
  • R2 only: all experts sit at A, so B and C have no nearby experts and no guarantee at all.
  • R1 + R2: moving experts from C to A raises A's bar to 30, while C's extra regret is harmless because C stays far above the worst case.

Real data agrees. On disturbed held-out Meta-World at equal budget (Episode 40), the results are 7.4 ± 0.6 for Phase I only (R1), 9.5 ± 0.6 for Phase II only (R2), and 13.1 ± 0.5 for both.

Reviewer_3TdsOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

This paper proposes BRIDGE for robust reinforcement learning. It uses deployment context to adapt to different disturbances without updating model parameters. CORE reuses intermediate policies from adversarial training as experts for different disturbances. ADAPT extends this approach to unseen tasks by learning task and disturbance representations. The paper provides theoretical analysis and evaluates the method on Dark Room, Meta-World, and several continuous-control tasks.

Critical strengths

1.A simple but insightful core idea. The strongest aspect of the paper is not a new adversarial robust-training algorithm per se, but the observation that intermediate policies that would normally be discarded during max-min training can be interpreted as disturbance-specialized experts. Reusing this trajectory as supervision for a context-conditioned policy is a clever way to extract additional value from robust training. 2.The theory explains when the method can help. The gain from using specialized experts must be larger than the errors from expert quality, context, and pretraining. I see this result mainly as an explanation of the method, not a way to predict exact performance. 3.The experiments support the method. With the same budget, Phase I alone gives a return of (7.4\pm0.6) at Episode 40. Phase I+II reaches (13.1\pm0.5). Adding robust initialization raises it to (13.9\pm0.2). These results support the value of adversarial search. The architecture ablations and linear probes also support the design of ADAPT. 4.The authors are careful about their claims. The appendix explains that orthogonality does not mean statistical independence. It also treats the extra representation objectives as helpful training goals, not guarantees of true disentanglement.

Critical weaknesses

1.The full training cost is unclear. CORE needs repeated expert training and adversary search. Phase-I tasks and separate Phase-II runs can run in parallel. However, steps within each Phase-II run are sequential. The paper does not clearly report the total cost of building the curriculum. It is unclear how well the method scales when more experts are needed.

2.The scaling of expert distillation is not tested. One model must learn from many experts. Similar contexts may require different actions, which can make the training targets conflict. The theory accounts for this through residual expert-action uncertainty and prediction error. It also gives an example where context cannot identify the relevant disturbance. However, the experiments do not test whether performance drops as the number of experts grows.

3.The need for Phase I is not fully clear. The same-budget results show that Phase II works better than spending more budget on Phase I alone. But they do not show whether Phase I is necessary. A useful comparison would skip Phase I and start several independent adversarial training runs.

Questions for authors

1.How does performance change as the number and diversity of experts in CORE increase? 2.Can the authors report the total cost of CORE, including expert training and adversary search? 3.Have the authors tested a version that skips Phase I and starts several independent adversarial training runs?

2 comments

Dear Reviewer 3Tds,

We sincerely thank the reviewer for the thoughtful review, and for recognizing the core idea, the role of the theory, the experimental support, and the care in our claims. Each weakness is paired with one of the reviewer's questions, so we answer them together: the necessity of Phase I (W3, Q3), the scaling of expert distillation (W2, Q1), and the total cost of CORE (W1, Q2). We ran the experiment the reviewer requested for each.


W3 and Q3. Phase I is necessary: skipping it lowers return from 13.1 to 9.5

W3: "The need for Phase I is not fully clear. The same-budget results show that Phase II works better than spending more budget on Phase I alone. But they do not show whether Phase I is necessary. A useful comparison would skip Phase I and start several independent adversarial training runs."

Q3: "Have the authors tested a version that skips Phase I and starts several independent adversarial training runs?"

We ran exactly this comparison, and Phase I is necessary. We skipped Phase I and started the independent max-min runs from randomly initialized policies, with the same total budget of 20 experts per task. On disturbed held-out Meta-World:

Curriculum (equal budget)Episode 5Episode 40
Phase I only1.9 ± 0.57.4 ± 0.6
Phase II only (new)2.7 ± 0.39.5 ± 0.6
Phase I + II (CORE)3.8 ± 0.613.1 ± 0.5
Phase I + II with robust context collection (full BRIDGE)6.5 ± 0.413.9 ± 0.2

Each phase alone falls short, and the two together do best. Phase I plays two roles that Phase II cannot replace:

  • Coverage. BRIDGE's worst case is taken over all disturbances, and the regret at a disturbance is small only when experts lie near it (proof of Lemma G.6). Phase II concentrates experts on the hard disturbances. Without Phase I, the remaining disturbances have no nearby experts, so even a mild disturbance can become BRIDGE's worst case.
  • Competent, diverse starting points. Each Phase II run starts from a Phase I expert trained under a different sampled disturbance, so the adversary probes real weaknesses of a good policy and different runs explore different hard regions. Started from random policies, the first adversaries find only trivial failures of an untrained policy, and early rounds are spent learning the task rather than handling disturbances.

To understand why both of them are necessary, we provide an illustrative toy example below.

Toy example of the coverage argument. Three disturbances A, B, and C, with best attainable returns J⋆J^\star of 40, 60, and 90, and 12 experts. The regret at a disturbance is 60 divided by the number of experts near it. Each bar's full length is J⋆J^\star (one block is 5 return units), the solid part █ is BRIDGE's return, the light part ░ is the regret, and ? means no guarantee. The worst case is the shortest solid bar.

Experts at A / B / CA (J⋆J^\star = 40)B (J⋆J^\star = 60)C (J⋆J^\star = 90)Worst case
Coverage only (4 / 4 / 4)█████░░░█████████░░░███████████████░░░25
Targeting only (12 / 0 / 0)███████░??????????????????????????????none
Coverage + targeting (6 / 4 / 2)██████░░█████████░░░████████████░░░░░░30

Coverage (Phase I) alone leaves the hard disturbance A at 25, targeting (Phase II) alone leaves B and C with no guarantee, and combining both raises the worst case to 30.

We will add the Phase II only row to Table 1 and this rationale to Section 3.

W2 and Q1. Scaling of expert distillation

W2: "The scaling of expert distillation is not tested. One model must learn from many experts. Similar contexts may require different actions, which can make the training targets conflict. [...] However, the experiments do not test whether performance drops as the number of experts grows."

Q1: "How does performance change as the number and diversity of experts in CORE increase?"

More experts help: return rises steadily as the curriculum grows, with no sign of conflicting targets. We trained ADAPT on curricula of 10, 20, and 40 experts per task, with the architecture and training recipe fixed. Larger curricula also use more independent Phase II runs, so the number and the diversity of experts grow together. Disturbed held-out Meta-World:

Experts per task (Phase I + Phase II runs × steps)Episode 5Episode 40
10 (5 + 5 × 1)5.4 ± 0.511.8 ± 0.4
20 (10 + 5 × 2, ours)6.5 ± 0.413.9 ± 0.2
40 (20 + 10 × 2)6.8 ± 0.415.2 ± 0.3

Why conflicts stay limited. Supervised pretraining learns the posterior-weighted mixture of the experts' actions given the context (Lemma G.2). Two experts give conflicting targets only when their contexts look alike but their actions differ.

  • Different disturbances leave different traces in the context. A disturbance changes the transitions the context records, so experts for different disturbances rarely share a context.
  • ADAPT is built to read these traces. The adversary latent is trained through conditioned dynamics prediction, which pushes it to encode how the current disturbance changes the transitions.
  • The residual conflict is the context term of Theorem 5.1, as the reviewer notes, and it shrinks as context accumulates. Adding experts therefore improves coverage while conflicts stay limited, as the rising returns show.

W1 and Q2. The total cost of CORE

W1: "The full training cost is unclear. [...] The paper does not clearly report the total cost of building the curriculum. It is unclear how well the method scales when more experts are needed."

Q2: "Can the authors report the total cost of CORE, including expert training and adversary search?"

CORE costs more than one robust RL run, and the extra cost buys a converged specialist for every disturbance in the curriculum. We measure cost in units of one nominal SAC [1] training run (about 29 minutes on Quadruped-Run):

SACRARL [2] / QARL [3]CORE + ADAPT
Total RL compute per training task1.0×about 2.0×22.0× (Meta-World, Quadruped-Run); 7.4× (Dark Room)
Wall-clock with parallel workers1.0× (29 min)about 1.7× (49 to 51 min)about 3.4× (about 1.6 h on Quadruped-Run)
Transformer pretraining (one A5000 GPU, no environment steps)nonenoneunder 2 h
Cost on a new taska full training runa full training runa few context episodes, no parameter updates

How the total is computed.

  • Phase I: each expert costs 1.0×, and all of them train in parallel.
  • Phase II: each run costs about 2.4× in total (about 1.2× for its experts and 1.2× for its adversaries), split across its steps. Warm-starting lets each expert converge within its share. Meta-World, for example, uses 10 Phase I experts and 5 Phase II runs: 10 × 1.0 + 5 × 2.4 = 22.0.
  • Wall-clock is about 1.0× for Phase I plus 2.4× for one Phase II run, since runs train in parallel.

How the cost scales when more experts are needed:

  • Compute grows linearly: 1.0× per Phase I expert and about 2.4× per Phase II run.
  • Wall-clock is set by the sequential steps of one Phase II run, so it stays at about 3.4× SAC as long as new runs train in parallel. Our 40-expert curriculum (W2 and Q1) doubles the number of runs but keeps two steps per run, so its wall-clock is unchanged.
  • The cost is offline and paid once. Deployment needs no parameter updates, and the pretrained model is reused across tasks and disturbances.

We will add this breakdown to Appendix C.3.


On presentation. We will make the design rationale of the two phases explicit in Section 3, state the warm-starting of Phase II experts in Algorithm 1, and add the cost breakdown, so that the paper requires less effort to follow.

We thank the reviewer again for these precise suggestions. We hope the new results address the concerns, and we look forward to the discussion.

References

  • [1] T. Haarnoja, A. Zhou, P. Abbeel, and S. Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. ICML, 2018.
  • [2] L. Pinto, J. Davidson, R. Sukthankar, and A. Gupta. Robust adversarial reinforcement learning. ICML, 2017.
  • [3] A. Reddi, M. Tölle, J. Peters, G. Chalvatzaki, and C. D'Eramo. Robust adversarial reinforcement learning via bounded rationality curricula. ICLR, 2024.
Reviewer_q2pMOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper addresses the conservatism problem in robust reinforcement learning. Traditional max-min optimization protects against worst-case disturbances but often sacrifices nominal performance. The authors propose BRIDGE (Bridging Robustness and In-context Decision GEneralization), which reframes robustness as a dynamic generalization problem—using in-context RL to infer the encountered disturbance and activate a specialized policy without parameter updates. The main contributions are: (1) CORE pretraining pipeline that distills intermediate expert policies from adversarial training curricula into an ICRL transformer; (2) using the final robust controller to collect initial deployment context safely; (3) ADAPT architecture with dual projection heads that disentangle task and disturbance representations. Theorem 5.1 provides performance bounds. Experiments on Dark Room, Meta-World, and 11 MuJoCo tasks show improvements in both single-task robust control and unseen-task generalization.

Critical strengths
  1. Fresh perspective with good motivation. The core insight is clever: adversarial training already generates a curriculum of disturbances and specialized policies, so why not leverage this for in-context learning? This addresses a real problem—robust policies are often overly conservative in typical conditions. The connection between max-min training and ICRL pretraining data is well-motivated. Using the final robust controller to collect initial context is a practical solution to the cold-start problem.
  2. Thorough experimental validation. The experiments cover diverse settings: discrete and continuous actions, sparse and dense rewards, different disturbance types. The ablations are well-designed. Table 1 shows Phase II improves return from 7.4 (Phase I alone with same budget) to 13.1, clearly demonstrating the value of adversarial boundary expansion. Table 1 also shows robust initialization helps (6.5 to 13.9 at Episode 40). Table 2 demonstrates the cumulative benefit of ADAPT components—full ADAPT gets 13.9 on Meta-World versus 6.1 for single-stream baseline. The linear probing results in Table 7 are interesting: task latent gets 91% task recognition vs 35% for adversary latent, and vice versa for adversary recognition (85% vs 26%), suggesting reasonable separation even though the cross-prediction accuracies of 26-35% indicate some residual entanglement.
Critical weaknesses
  1. Theory-practice connection could be stronger. Theorem 5.1 provides a bound but contains several unestimated constants, making it hard to assess practical tightness. The exponential 1/(d+1) dependence on dimension is mentioned but not explored—how does this affect high-dimensional disturbance spaces? The coverage assumptions (joint distribution coverage Rθ ≤ CQe,X and local probability mass conditions) are stated formally but it's not clear how to verify them or how they relate to the experimental train/test splits. For instance, in the MuJoCo physical parameter experiments, does the 8×8 test grid satisfy the coverage conditions relative to the 4×4 training grid? The theorem gives conditions under which BRIDGE helps, but doesn't translate this into practical guidance for when practitioners should use it.
  2. Some design choices need more justification. The paper uses K0=5 episodes for initial context collection but doesn't ablate this choice. What happens with 1, 3, or 10 episodes? Is there a point of diminishing returns? The auxiliary loss weights (λ1=0.1, λ2=1.0, λ3=0.05) appear hand-tuned but no sensitivity analysis is provided. While Appendix C.3 reports some training times, the full computational cost of CORE (Phase I expert training + Phase II sequential adversarial search + ICRL distillation) relative to standard RARL isn't quantified in terms of total environment steps or wall-clock time. This matters for assessing the practical tradeoff.
Questions for authors
  1. Can you provide a full computational cost breakdown? How many total environment steps does CORE require (Phase I + Phase II + distillation) compared to training a standard RARL/QARL policy? This would help assess the practical tradeoff.
  2. Have you ablated the context length K0=5? What's the performance with K0=1, 3, or 10 episodes? Understanding this sensitivity would be useful.
  3. Table 7 shows cross-prediction accuracies of 26-35%, which is above random chance. Does this residual information leakage affect control performance on unseen tasks? Can you quantify the relationship between separation quality and task performance?
  4. For the physical parameter generalization experiments, how do the test parameters (8×8 grid) relate to the training parameters (4×4 grid) in terms of the coverage assumptions in Theorem 5.1? Can you empirically verify whether the test conditions satisfy the theoretical requirements?
  5. Under what conditions should practitioners choose BRIDGE over standard robust RL? Can you provide some empirical rules of thumb based on properties like disturbance dimensionality, available training budget, or deployment scenario characteristics?
4 comments

Dear Reviewer q2pM,

We sincerely thank the reviewer for the careful and constructive review, and for recognizing the fresh perspective, the motivation, and the thorough experiments.

The comments fall into three themes, which we address in turn:

  • connecting the theory to practice (W1, Q4, Q5)
  • design choices and computational cost (W2, Q1, Q2)
  • information shared between the two latents (Q3)

Each theme comes with new experiments as evidence.


W1, Q4 and Q5. Connecting the theory to practice

W1: "Theory-practice connection could be stronger. [...] The theorem gives conditions under which BRIDGE helps, but doesn't translate this into practical guidance for when practitioners should use it."

We agree that this connection deserves to be explicit, and we thank the reviewer for asking for it. Theorem 5.1 is meant to explain what the gain depends on and how each factor scales. Like most finite-sample guarantees, its constants are worst-instance quantities, so it does not predict exact returns. Each of its terms, however, corresponds to something a practitioner can measure. The four answers below make that correspondence concrete.

1. Every term of Theorem 5.1 maps to a diagnostic that can be measured

"Theorem 5.1 provides a bound but contains several unestimated constants, making it hard to assess practical tightness."

Term in Theorem 5.1The practical questionHow to measure itWhat we observe
Tailored-control advantageDo disturbance-specific experts beat one robust policy?Compare the experts' returns with the robust baseline's worst caseLarge: on Quadruped-Run, specialists reach about 195 versus 105 for RARL [1], and ADAPT (186) recovers about 90% of the gap
Expert errorAre the expert labels close to optimal?Train the experts 3× longer and measure the improvementOnly about 4.5% improvement
Context uncertaintyCan a few episodes reveal the disturbance?Vary the amount of contextGains saturate by about five episodes (answer 5)
Pretraining errorAre there enough experts?Vary the number of expertsReturn rises from 11.8 to 13.9 to 15.2 with 10, 20, and 40 experts per task
CoverageAre there experts near every disturbance, and more near the hard ones?Phase I only, Phase II only, or both, at equal budget7.4, 9.5 (new), 13.1

We will add this table to Section 5, so that the theorem reads as a checklist rather than only as a bound.

2. High-dimensional disturbances: only the hardest disturbances need fine coverage

"The exponential 1/(d+1) dependence on dimension is mentioned but not explored—how does this affect high-dimensional disturbance spaces?"

  • What d is. It is the dimension of the disturbance family (2 for a mass and friction family), not the number of weights in an adversary network (Appendix G.5).

  • The effect of a large d. Guaranteeing small regret at every disturbance requires experts near every disturbance, and the number needed grows exponentially with d. Appendix G.5 also shows that this exponent cannot be improved without additional structure.

  • Why this is less severe for the worst case: only the hardest disturbances need small regret. BRIDGE's worst-case return is min⁡ψ(J⋆(ψ)−regret(ψ))\underset{\psi}{\min} \left( J^\star(\psi) - \text{regret}(\psi) \right), where J⋆(ψ)J^\star(\psi) is the best return attainable under the disturbance ψ\psi, so easier disturbances only need regret below their slack. Applying the key step of our proof (Lemma G.6) separately to the hard set (disturbances whose best return is within a margin γ\gamma of the lowest) and to the rest gives

    worst-case gain≥Δ−max⁡(Bhard,Beasy−γ)\text{worst-case gain} \ge \Delta - \max\left( B^{\text{hard}}, B^{\text{easy}} - \gamma \right),

    where Δ\Delta is the tailored-control advantage, and BhardB^{\text{hard}} and BeasyB^{\text{easy}} are the error term of Theorem 5.1 evaluated with the curriculum mass near the hard set and elsewhere. When thin coverage keeps the easy disturbances within their slack, the rate is governed by the size of the hard set, not by the full dimension d.

  • This is exactly CORE's two-phase design. Phase I provides thin coverage everywhere, and Phase II concentrates experts on the hard disturbances. At equal budget, using both phases reaches 13.1, compared with 7.4 for Phase I only and 9.5 for Phase II only.

We will add this refinement as a remark after Theorem 5.1.

3. The 8×8 grid: coverage holds in the central 4×4 cells, and the outer cells test extrapolation

"For instance, in the MuJoCo physical parameter experiments, does the 8×8 test grid satisfy the coverage conditions relative to the 4×4 training grid?" (also Q4)

The central 4×4 cells satisfy the coverage conditions. The outer cells do not, and we include them deliberately as an extrapolation test.

  • Inside the 4×4 region, test configurations come from the same population as pretraining, so Theorem 5.1 and Corollary G.10 apply.
  • Outside it, the guarantees do not apply, and the gains there (for example Walker-Run and Cheetah-Run, Fig. 12b and 13a) are empirical evidence of extrapolation. We will state this explicitly in the paper.
  • An empirical check of effective coverage, without labels (Q4). ADAPT's dynamics-prediction error measures how familiar the observed transitions are. It stays low across the 4×4 core and the surrounding ring, where returns are high, and rises only at the extreme corners.

Why BRIDGE works beyond its guarantee. Our analysis is deliberately worst-case. It assumes only mild regularity of the task family, so it needs coverage to rule out an uncovered disturbance that demands an entirely new behavior. Physical control tasks carry much stronger structure, which acts as a strong prior for extrapolation:

  1. Shared mechanics. Every cell of the grid is the same robot under the same physical laws, and only two parameters change. The skills learned inside the grid (balance, gait) remain the right skills outside it.
  2. Smooth dependence on the parameters. A slightly heavier torso calls for slightly more of the same correction, so the trend learned inside the grid continues outside it.
  3. Parameter changes act like forces. To the controller, a heavier torso or a lower friction is an additional force on the body (the premise of RARL [1], inherited from H-infinity control). Phase II experts were trained under adversarial forces, so the responses that the outer cells require are already in the curriculum.

BRIDGE exploits this structure, and the evidence matches: DPT [2], pretrained on the same 4×4 grid but without the adversarial experts, drops sharply at the outer cells, while ADAPT largely retains its central-region level and degrades gradually (Fig. 12b, 13a, and 14).

In practice, BRIDGE therefore reaches further than its worst-case analysis guarantees. Bringing such helpful task structure into the analysis is a natural direction for future work.

How to check the conditions in general. The local-mass condition holds by construction for any disturbance inside the range that Phase I samples uniformly. The joint coverage condition concerns the mismatch between pretraining and deployment contexts. Our experiments with random and expert context (Fig. 3a, Fig. 4a and 4b) vary the context law substantially and show that performance is stable under this mismatch.

4. When to choose BRIDGE over standard robust RL

Q5: "Under what conditions should practitioners choose BRIDGE over standard robust RL? Can you provide some empirical rules of thumb based on properties like disturbance dimensionality, available training budget, or deployment scenario characteristics?"

PropertyBRIDGE is the better choice whenRelated term
Conflict among disturbancesDifferent disturbances call for different strategies, so specialists clearly beat one robust policy (Quadruped-Run: ADAPT 192 nominal and 186 disturbed, versus 110 and 105 for RARL)Tailored-control advantage
Deployment scenarioThe disturbance stays fixed during a deployment and shows up in the transitions within a few episodesContext uncertainty
Disturbance dimensionalityThe family is low-dimensional, or its hard cases are concentrated (answer 2)Coverage
Training budgetOffline compute is available and the pretrained model is reused across deploymentsCost (answer 7)

When does standard robust RL may remain preferable? From the analysis above, possible scenarios include when the disturbance changes within an episode, when the context cannot reveal it (the example in Appendix G.6), or for a single deployment under a tight training budget.

A quick test before committing: train a few specialists and compare them with the robust policy. If the gap is small, BRIDGE has little to gain; otherwise, BRIDGE may provide large gain.

W2, Q1 and Q2. Design choices and computational cost

5. Context length K0

Q2: "Have you ablated the context length K0=5? What's the performance with K0=1, 3, or 10 episodes?"

Following this advice, we conducted an ablation on K0. Results show that ~5 episodes is the sweet spot: a few episodes reveal the disturbance, while more conservative context slows adaptation. Episode numbers count online adaptation episodes after the K0 collection episodes (10 seeds, mean ± SEM):

K0Meta-World, Ep. 5Meta-World, Ep. 40Quadruped-Run, Ep. 40
0 (no robust collection)3.8 ± 0.613.1 ± 0.5174.8 ± 4.5
14.6 ± 0.513.3 ± 0.4178.5 ± 4.1
35.8 ± 0.413.7 ± 0.3183.4 ± 3.2
5 (ours)6.5 ± 0.413.9 ± 0.2186.0 ± 2.6
105.1 ± 0.412.4 ± 0.4172.4 ± 3.6
  • Up to five episodes, more context helps, as the context-uncertainty term of Theorem 5.1 predicts: each episode under the disturbance reveals more about it (Proposition G.7), and the gains saturate quickly.
  • Beyond five, return drops. The extra episodes come from the conservative robust controller, so a longer context biases the model toward conservative behavior, which later adaptation must overcome.

6. Auxiliary loss weights

"The auxiliary loss weights (λ1=0.1, λ2=1.0, λ3=0.05) appear hand-tuned but no sensitivity analysis is provided."

Performance is stable: halving or doubling any weight keeps return between 13.3 and 13.8 (default 13.9). Disturbed held-out Meta-World, Episode 40:

Weight varied0.5×1× (ours)2×
λ1 (task anchoring, 0.1)13.4 ± 0.313.9 ± 0.213.7 ± 0.3
λ2 (dynamics prediction, 1.0)13.3 ± 0.413.9 ± 0.213.8 ± 0.3
λ3 (orthogonality, 0.05)13.5 ± 0.313.9 ± 0.213.7 ± 0.2
  • No per-task tuning. The weights only balance the natural scales of the three losses (a contrastive loss, a dynamics regression loss, and a cosine similarity in [0, 1]), and one setting is used unchanged on Dark Room, Meta-World, and all 11 MuJoCo tasks.
  • Return drops only at the extremes. Removing objectives entirely lowers it (Table 2: 10.6 without orthogonality, 8.7 without dynamics prediction as well, and 6.1 for the single-stream baseline), and so does a 20× larger orthogonality weight (12.8, answer 8).

7. Computational cost

Q1: "Can you provide a full computational cost breakdown? How many total environment steps does CORE require (Phase I + Phase II + distillation) compared to training a standard RARL/QARL policy?"

Thank you for this suggestion. We measure cost in units of one nominal SAC [3] training run (about 29 minutes on Quadruped-Run):

SACRARL [1] / QARL [4]CORE + ADAPT
Total RL compute per training task1.0×about 2.0×22.0× (Meta-World, Quadruped-Run); 7.4× (Dark Room)
Wall-clock with parallel workers1.0× (29 min)about 1.7× (49 to 51 min)about 3.4× (about 1.6 h on Quadruped-Run)
Transformer pretraining (one A5000 GPU, no environment steps)nonenoneunder 2 h
Cost on a new taska full training runa full training runa few context episodes, no parameter updates

How the total is computed.

  • Phase I: each expert costs 1.0×, and all of them train in parallel.
  • Phase II: each run costs about 2.4× in total (about 1.2× for its experts and 1.2× for its adversaries), split across its steps. Warm-starting lets each expert converge within its share. Meta-World, for example, uses 10 Phase I experts and 5 Phase II runs: 10 × 1.0 + 5 × 2.4 = 22.0.
  • Wall-clock is about 1.0× for Phase I plus 2.4× for one Phase II run, since runs train in parallel.

Two points put this cost in context:

  • It is offline and paid once. Deployment needs no parameter updates, and the same pretrained model is reused across tasks and disturbances.
  • The budget is well spent. At equal budget, spending everything on Phase I reaches 7.4, compared with 13.1 for the two-phase curriculum (Table 1).

Q3. Information shared between the two latents

8. Shared information does not by itself hurt control, and separation and return improve together

Q3: "Table 7 shows cross-prediction accuracies of 26-35%, which is above random chance. Does this residual information leakage affect control performance on unseen tasks? Can you quantify the relationship between separation quality and task performance?"

The reviewer is right that the separation is not perfect (chance is about 7% for the 15 tasks and 12.5% for the 8 perturbation targets), and we do not claim full disentanglement (Appendix G.10).

  • Why shared information does not by itself hurt control. The action decoder receives both latents. What matters for control is that the two latents together keep the decision-relevant information (Appendix G.10 decomposes the prediction error into information lost by the representation plus decoding error). Information that appears in both latents is not lost.
  • Why some sharing is expected. Both latents come from one shared encoder, and the effect of a disturbance depends on the task being performed.
  • The relationship, quantified. We ran the linear probes of Table 7 on every variant of the Table 2 ablation, so that separation can be read next to return:
VariantTask ID from task latent (↑)Task ID from adversary latent (↓)Adversary ID from adversary latent (↑)Adversary ID from task latent (↓)Meta-World returnQuadruped-Run return
Single-stream (one shared feature)58%n/a51%n/a6.1 ± 0.5157.2 ± 8.2
Dual-stream + task anchoring86%49%56%44%8.7 ± 0.6166.2 ± 4.8
+ dynamics prediction89%42%82%34%10.6 ± 0.4181.2 ± 3.4
+ orthogonality (full ADAPT)91%35%85%26%13.9 ± 0.2186.0 ± 2.6

As the objectives are added, each latent predicts its own factor better, cross-prediction falls, and return rises on both benchmarks. This is a correlation across four variants rather than a causal claim, and we present it as such. Separation does not help without limit: a 20× larger orthogonality weight lowers cross-prediction to about 21% but reduces return from 13.9 to 12.8, consistent with the decoder needing some information shared by both latents.


We thank the reviewer again for these questions, which helped us tie the analysis to practice. We hope the clarifications and new results address the concerns, and we look forward to discussions.

References

  • [1] L. Pinto, J. Davidson, R. Sukthankar, and A. Gupta. Robust adversarial reinforcement learning. ICML, 2017.
  • [2] J. Lee, A. Xie, A. Pacchiano, Y. Chandak, C. Finn, O. Nachum, and E. Brunskill. Supervised pretraining can learn in-context reinforcement learning. NeurIPS, 2023.
  • [3] T. Haarnoja, A. Zhou, P. Abbeel, and S. Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. ICML, 2018.
  • [4] A. Reddi, M. Tölle, J. Peters, G. Chalvatzaki, and C. D'Eramo. Robust adversarial reinforcement learning via bounded rationality curricula. ICLR, 2024.

Comments on the paper

2 comments

Dear Reviewers and Area Chairs,

We thank the review team for their valuable efforts and deeply appreciate the reviewers' recognition of our work:

  • Insightful core idea (oSR9, q2pM, 3Tds)
  • Important, well-formulated problem (oSR9, q2pM)
  • Strong theoretical analysis (oSR9, 3Tds)
  • Thorough experiments and well-designed ablations (oSR9, q2pM, 3Tds)
  • Practical solution to the cold-start problem (q2pM)
  • Clear writing and presentation (oSR9, q2pM)

New Empirical Experiments

Following the reviewers' feedback, we conducted these additional studies (full results in each response):

  1. Ablation on the necessity of CORE Phase I (Reviewer 3Tds, W3 & Q3). Removing Phase I and giving its budget to Phase II alone (independent max-min runs) drops final return by a notable margin, confirming the necessity of Phase I. This is consistent with our analysis: the worst-case guarantee needs both coverage (CORE Phase I) and challenging adversary targeting (CORE Phase II). Phase I also gives Phase II diverse, competent starting points for better empirical optimization results.
  2. Scaling the number of experts (Reviewer 3Tds, W2 & Q1). We trained ADAPT on half, the default, and twice the experts per task. Return keeps rising, so distilling many specialists into one model does not suffer from conflicting targets, as the context tells them apart.
  3. Ablation on context length (Reviewer q2pM, Q2). Varying initial robust episodes from 0 to 10 shows performance peaks at five episodes: a few suffice to reveal the disturbance, while a longer conservative context slows later adaptation.
  4. Sensitivity to auxiliary loss weights (Reviewer q2pM, W2). We halved and doubled each weight. Performance is reasonably stable, and we use one setting for all benchmarks without per-task tuning.
  5. Latent separation versus performance (Reviewer q2pM, Q3). Running linear probes on every architecture variant shows representation separation and return improve together, while forcing too much separation hurts.
  6. Full computational cost (Reviewer q2pM, Q1; Reviewer 3Tds, W1 & Q2). We report compute and wall-clock time for each phase against SAC, RARL, and QARL. Notably, the cost of BRIDGE and CORE is one-time and largely parallel, with zero parameter updates at deployment.
  7. Empirical diagnostics for Theorem 5.1 (Reviewer q2pM, W1). Measuring the terms of our bound shows specialists clearly outperform a single robust policy, and experts trained 3× longer barely improve, showing high label quality.

Additional Improvements

We will also incorporate into our updated manuscript the following new analysis and discussion:

  • Clarified contributions (Reviewer oSR9, W1 & W4):
    • Idea and Paradigm: We propose BRIDGE to reframe robustness as in-context generalization without parameter updates, and CORE to turn the discarded byproducts of max-min training into pretraining data. We deliberately instantiate both with simple, canonical components to showcase the value of the proposed frameworks themselves, rather than relying on state-of-the-art components. We do not claim the instantiated combination is optimal, and investigating other implementations is an exciting future direction.
    • Technical Algorithm and Architecture: While Phase II of CORE resembles RARL (a classic deep robust RL algorithm), CORE modifies RARL to make it suitable for pretraining data collection (e.g., warm-started experts trained to convergence, every adversary and expert pair kept), and ADAPT is a new dual-stream ICRL architecture for robust deployment.
  • Curriculum design principles from theory (Reviewer oSR9, W2 & W3; Reviewer 3Tds, W3): coverage, targeting, and label quality, which explain the two-phase design and give a yardstick for comparing alternative curricula (genetic, diversity, curiosity).
  • Refined analysis for high-dimensional disturbances (Reviewer q2pM, W1): the worst case needs fine coverage only near the hardest disturbances, which is what Phase II provides.
  • Theory connected to practice (Reviewer q2pM, W1 & Q5): each term of Theorem 5.1 maps to a measurable diagnostic, with rules of thumb for when to choose BRIDGE.
  • Extrapolation explained (Reviewer q2pM, Q4): physical task structure (shared mechanics, smooth dependence on parameters, parameter changes acting like forces) lets BRIDGE work beyond the region its guarantee covers, and ADAPT's dynamics-prediction error confirms effective coverage beyond the training grid.

Thanks again for all your precious time and consideration! We believe the above changes help us notably improve the quality of our work. We are more than happy to provide further analysis and additional materials, and we look forward to the discussion.

Best regards,

Authors

Dear Reviewers,

Thank you for your contributions to the review process. As the author–reviewer discussion period ends on November 18, I would like to kindly remind everyone to review the authors' rebuttals and engage in the discussion.

In particular, please indicate whether your main concerns have been adequately addressed and clarify any remaining issues that may affect your final assessment.

Your feedback will be very helpful for the subsequent reviewer–AC discussion and meta-review process.

Thank you for your time and effort!

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.