- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper shows pipeline systems are optimized in an unintended way (i.e., role changes of a module) when they are trained end-to-end using RL. The authors refer to this phenomenon as role drift, a failure mode. They observe this phenomenon across four different system including RAG, multi-hop QA, code generation, and LLM routing.
Critical strengths
- The paper is quite straightforward in writing and easy to follow.
- Empirical demonstrations of the described phenomenon across four different pipeline systems.
Critical weaknesses
- Fundamentally, the paper treats role drift as a failure mode, but I'm not fully convinced about the justification on why the roles should be preserved necessarily. The original modular role in the pipeline systems is merely an architectural inductive bias, not necessarily an optimal design in the first place. As such, end-to-end RL may simply discover more effective and optimized system as a whole (i.e., beneficial adaptation).
- The identified notion, role drift, appears to reframe already known optimization behaviors in RL, and so it might not be surprising to observe. While the paper mentions existing concepts such as reward hacking and shortcut learning to distinguish, it is fully unclear what additional meaningful values this new framing provides.
- The authors argue that the role drift matters because of deployment settings. However, some evaluations seem to be rather artificial, as performance drop is seen under deliberately modified conditions or assumptions about specific behavior (e.g., answer masking, additional latency constraints, strong dependence on retrieved documents).
Questions for authors
Please see weaknesses above.