acceptodds
Withdrawn from ICLR 2027

Do Modules Stay in Their Lane? Role Drift in Compound AI Systems

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

When a compound AI system is trained end-to-end with reinforcement learning, the terminal reward only shows how good the final output is, but cannot tell whether each module performed its assigned job. A module can deviate from its assigned role while still improving the terminal reward. We call this failure role drift, and demonstrate it in four compound systems. For example, in a multihop QA pipeline, a decomposer is supposed to pose subquestions and a solver is supposed to answer them concisely. After training, the decomposer begins writing the gold answer into its subquestions, with leakage rising from 14% to 73.5%. Similar drift appears in a retrieval-augmented QA pipeline, a code generation pipeline, and an LLM routing system. Each module is capable of performing its assigned role, but the terminal reward pays more for leaving the role than for keeping it. The drift is invisible in the reward curve and becomes harmful when deployment conditions change: when the leaked answers in the decomposer's subquestions are masked, the system's performance drops by 8.9 points. Monitoring the terminal reward alone is not sufficient to ensure that a compound system's modules behave as intended.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Large Language Models, Compound AI Systems, Multi-Agent Systems, Alignment, AI Safety

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_79kQOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper shows pipeline systems are optimized in an unintended way (i.e., role changes of a module) when they are trained end-to-end using RL. The authors refer to this phenomenon as role drift, a failure mode. They observe this phenomenon across four different system including RAG, multi-hop QA, code generation, and LLM routing.

Critical strengths
  1. The paper is quite straightforward in writing and easy to follow.
  2. Empirical demonstrations of the described phenomenon across four different pipeline systems.
Critical weaknesses
  1. Fundamentally, the paper treats role drift as a failure mode, but I'm not fully convinced about the justification on why the roles should be preserved necessarily. The original modular role in the pipeline systems is merely an architectural inductive bias, not necessarily an optimal design in the first place. As such, end-to-end RL may simply discover more effective and optimized system as a whole (i.e., beneficial adaptation).
  2. The identified notion, role drift, appears to reframe already known optimization behaviors in RL, and so it might not be surprising to observe. While the paper mentions existing concepts such as reward hacking and shortcut learning to distinguish, it is fully unclear what additional meaningful values this new framing provides.
  3. The authors argue that the role drift matters because of deployment settings. However, some evaluations seem to be rather artificial, as performance drop is seen under deliberately modified conditions or assumptions about specific behavior (e.g., answer masking, additional latency constraints, strong dependence on retrieved documents).
Questions for authors

Please see weaknesses above.

Reviewer_kYt2OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper investigates whether end-to-end reinforcement learning preserves assigned module roles in four compound systems: retrieval-augmented QA, question decomposition, code generation, and LLM routing. It tracks role-specific probes alongside task rewards and uses stricter evaluation and deployment perturbations to argue that improving terminal reward can conceal functional drift and downstream failures.

Critical strengths

Monitoring module behavior alongside terminal reward offers a useful diagnostic perspective across several applications. The original-test rescoring in the coding experiment provides particularly direct evidence that increasing proxy reward can conceal deteriorating task performance.

Critical weaknesses
  1. The causal and deployment interpretations are stronger than the controls establish.

The RAG context-accuracy gap falls to 1.1 percentage points after six epochs, but an aggregate gap does not establish per-example answer invariance (Section 4.1). The updated-facts results provide additional supporting evidence, although the current answer accuracy and changes in paired predictions remain unreported. For decomposition, Table 2 pools outputs across questions and training stages, thereby confounding the proposed reward mechanism. Independent-solver transfer improves by 6.5 points before masking; reversing that advantage through gold-answer masking does not by itself establish harmful co-adaptation.

  1. The incremental contribution and generality need stronger support.

The paper acknowledges relevant predecessors, including "When Does Multi-Agent RL Improve LLM Workflows?" Workflow, Scale, and Policy-Sharing Tradeoffs (Zeng et al., 2026) documents RL-induced role capture and explicitly warns that aggregate accuracy can conceal drift; Steering RL Training: Benchmarking Interventions Against Reward Hacking (Wong, Engels, and Nanda, 2025) demonstrates the test-overwrite behavior studied here. The cross-system synthesis is useful, but evidence for additional mechanisms beyond these acknowledged antecedents remains limited. One reported seed/configuration per system, exclusively Qwen models, and the deliberate 7B/0.5B capacity imbalance further limit the broader structural claims.

Questions for authors
  1. Can you report paired prediction changes under RAG context swaps and current-answer accuracy on the updated-facts evaluation, alongside the existing outdated-answer counts?

  2. Does the leakage probe count the final answer, intermediate hop answers, or both, and does this match Appendix B's masking procedure? Please provide all four pre-/post-RL × masked/unmasked accuracies to reconcile the abstract's 8.9-point performance-drop claim with Table 3's change in relative advantage.

  3. For routing, how does the supervised baseline perform after tuning its decision threshold on validation data for the same utility? Since the objective rewards correct selective routing above always-escalate, what evidence distinguishes weak cost incentives from policy-learning or calibration limitations?

Reviewer_QMFQOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

This paper considers compound AI systems trained with reinforcement learning. It studies how the roles of different modules in the system drift with RL training. The paper analyzes four compound systems (RAG, Decomposer-Solver, Solver-Evaluator, LLM Router) made up of Qwen models and sees how module roles change when the system is trained with RL. From these settings, the paper tries to answer why RL produces role drift and what its impact are. They conclude there is role drift because the reward does not incentivize role-faithful behavior.

Critical strengths
  • The writing in most places is quite clear.
  • The figures are easy to read.
  • The authors test several models (though in the same family) across a few different settings.
  • The authors do point out what seems to me to be the correct reason for role drift.
Critical weaknesses
  • The main finding does not seem to me particularly interesting or novel. I believe most researchers in the area would expect drift because RL does not incentivize role-faithful behavior. This has been widely documented already in past work, eg. via reward hacking in self-play as mentioned and in other settings. I am unsure what the purpose of the analysis in this paper is or what the reader is meant to take away.
  • There is no clearly scoped scientific study here. The paper reads like a case study of some settings the authors tried and then defined metrics post-hoc seeing what happened. What is the key question? What are the metrics? I did not feel like any non-obvious scientific questions were asked or that robust answer was given anywhere.
  • The paper argues against a view (which seems reasonable to me) that suggests role drift is unimportant because RL improves the terminal performance whereas the roles are hand-designed inductive biases and not necessarily optimal. They argue against this by saying the improvement from RL can be illusory, citing and example where a model does not learn a strict format not rewarded during training. This does not prove the point and is simply an issue with environment design. If the authors believe the strict match is what is important, that the reward should be designed to maximize that. The other examples also cite some train-inference mismatch, but again the main purpose of careful environment design is to minimize this transfer. There is no universal reason a system that follows its "roles" should perform any better in this scenario and this is not shown in any results.
  • Though I agree with the authors there might be some limits of careful environment design, it is worth acknowledging that it has thus far been extremely successful in training widely capable models without the kinds of issues cited here.
  • Despite pointing out what seems to me to be the correct reason for role drift and arguing that it is undesirable, the authors do not offer any potential solution or mitigation.
Questions for authors

See above.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.