- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes BRIDGE, an in-context reinforcement learning approach that addresses the tradeoff between robustness and performance via online adaptation without parameter updating using context information fed in to a transformer network. Using a two stage-training, the BRIDGE trains a transformer network that can adaptively change its behavior based on observations, instead of having a single policy across all situations. The algorithm was empirically demonstrated in simulated environments against several different baselines.
Critical strengths
- The problem is well formulated and addresses the important question of trading off robustness vs performance
- The core insight is straightfoward and compelling.
- Strong theoretical analysis with guarantees
- Well designed ablation studies and experimentation step across various different tasks and benchmarks.
- Clear writing and easy to understand sentences / figures.
Critical weaknesses
- One of the building blocks for this paper is using In-Context Reinforcement via Transformer Network trained by adversarial curriculum. The former was proposed in DPT(Lee), the later in RARL(Pinto et al 2017). While the authors did an extensive ablation study and theoretical proof on why this would work, I think the paper could be made stronger why this would be the best combination of ideas to use for robust in-context RL and share their insights in this regard.
- Furthermore, while the adversarial learning setup by RARL is an max-min formulation, Genetic Curriculum (Song et al 2022, 2025) showed that from a curricular point of view, this may not be the best due to adversarial examples often exceeding trainee's capacity to learn new behaviors at a given time. The work instead proposed a curricular approach to gradually introduce different types of adversarial approaches. Given how the proposed work is also dependent on generating a good training curriculum, I am curious on how other adversarial curricular learning approaches could be incorporated to this paper, and whether the approach proposed by the paper is indeed the best.
- Other than adversarial sampling, there's also diversity based sampling (Eysenbach et al, 2018) and curiosity based sampling (Pathak et al 2017). Given how robustness is often associated with how well the algorithm performs against a variety of different conditions, it might have been better if the paper explored with various different approaches towards how to make a training curriculum.
- As RARL is already an concept that existed for several years, I think the paper could be made much more impact and be more novel if the authors could take one more step and show that adversarial training is whether the best match for training DPT.
Questions for authors
My request to the authors would be to address the points made in the critical weaknesses.
10 comments
Q2. Our contributions go beyond combining DPT and RARL, at both the idea level and the technical level
- Idea and paradigm. BRIDGE reframes robustness as in-context generalization without parameter updates, and CORE turns the byproducts of max-min training, which robust RL normally discards, into pretraining data. These contributions hold for any viable choice of the components.
- Technical A: ADAPT, a new architecture. We do not directly use the DPT model [2]. Instead, we propose ADAPT, which infers the task and the disturbance jointly from context, using separate task and adversary latents with three auxiliary objectives. Trained on the same CORE data, it raises return over a single-stream model from 6.1 to 13.9 on disturbed Meta-World and from 157.2 to 186.0 on Quadruped-Run (Table 2).
- Technical B: CORE has Phases I and II supported by both theoretical analysis and empirical evidence, and CORE modifies RARL in Phase II for ICRL data generation. RARL takes a few alternating gradient steps per round and keeps only one final policy. CORE instead warm-starts each expert from the previous one and trains it to convergence against a frozen adversary, so each expert is a high-quality label. It also keeps every (adversary, expert) pair, which is exactly the data supervised pretraining needs, and seeds its runs from Phase I experts. Notably, training to convergence is also why GC's [3] capacity concern does not arise here (Q3).
THEME II. Comparison with alternative curricula (W2 and W3).
W2: "Given how the proposed work is also dependent on generating a good training curriculum, I am curious on how other adversarial curricular learning approaches could be incorporated to this paper, and whether the approach proposed by the paper is indeed the best."
W3: "Given how robustness is often associated with how well the algorithm performs against a variety of different conditions, it might have been better if the paper explored with various different approaches towards how to make a training curriculum."
This is a great question. In essence, both comments ask which other curricula could be used in CORE, and whether they would be better. Our theoretical analysis gives a precise yardstick for comparing curricula. BRIDGE's worst-case return is , where is the best return attainable under disturbance , and the regret at shrinks with the number of nearby experts (proof of Lemma G.6).
Notably, our theoretical analysis reveals that a desired curriculum should provide:
- (R1) Coverage: experts near every disturbance.
- (R2) Targeting: more experts where is low, since those disturbances set the worst case.
- (R3) Label quality: experts close to their matching disturbance's optimum, which is the expert error in Theorem 5.1.
CORE meets these by design:
- Phase I samples uniformly (R1).
- Phase II's adversary sends each new expert to where the current expert is weakest (R2).
- Experts are trained to convergence (R3).
In particular, the choice of curriculum decides whether both R1 and R2 can be satisfied. To understand why both of them are necessary, we provide an illustrative toy example below.
Toy example. Three disturbances A, B, and C, with best attainable returns of 40, 60, and 90, and a budget of 12 experts. The regret at a disturbance is 60 divided by the number of experts near it. How to read it: each bar's full length is (one block is 5 return units), the solid part █ is BRIDGE's return, the light part ░ is the regret, and ? means no guarantee. The worst-case guarantee of a curriculum is its shortest solid bar.
| Curricula Defined by #Experts at A / B / C | A ( = 40) | B ( = 60) | C ( = 90) | Worst-case Guarantee |
|---|---|---|---|---|
| R1 only (4 / 4 / 4) | █████░░░ | █████████░░░ | ███████████████░░░ | 25 |
| R2 only (12 / 0 / 0) | ███████░ | ???????????? | ?????????????????? | none |
| R1 + R2 (6 / 4 / 2) | ██████░░ | █████████░░░ | ████████████░░░░░░ | 30 |
- R1 only: experts are spread evenly, so the hard disturbance A loses as much as the easy C, and A's short bar sets the worst case at 25.
- R2 only: all experts sit at A, so B and C have no nearby experts and no guarantee at all.
- R1 + R2: moving experts from C to A raises A's bar to 30, while C's extra regret is harmless because C stays far above the worst case.
Real data agrees. On disturbed held-out Meta-World at equal budget (Episode 40), the results are 7.4 ± 0.6 for Phase I only (R1), 9.5 ± 0.6 for Phase II only (R2), and 13.1 ± 0.5 for both.
Dear Reviewer oSR9,
We sincerely thank the reviewer for the encouraging assessment of our problem formulation, core insight, theory, and experiments.
The four comments raise two closely related themes, which we address together:
THEME I. Optimality of the design and our contributions (W1 and W4).
These two comments raise two closely related questions, which we answer in turn.
Q1. Our instantiations are deliberately straightforward, and they fit together naturally
We appreciate the reviewer's interest in whether this is the best combination, and we would like to gently clarify that we do not claim it is, nor is proving its optimality within our intended contribution.
As the first work to implement BRIDGE, we deliberately chose straightforward, canonical instantiations, so that the gains can be attributed to the proposed frameworks (BRIDGE and CORE) themselves, rather than to a strong individual component. Exploring more advanced instantiations is important follow-up work.
That said, the reviewer's question is indeed important, and our theoretical analysis gives a principled answer: a desired curriculum for BRIDGE must provide three things
These requirements are explained and derived in detail in our response to W2 and W3 below, where they also serve as the criterion for comparing alternatives. Our choices are in fact well suited to them:
We thank the reviewer for prompting us to make this connection explicit, and we will add it to the paper.