MA-RL-VLA: Counterfactual Credit-Aware PPO for Multi-Agent Reinforcement Fine-Tuning of Vision-Language-Action Models

Reinforcement learning (RL) has proven effective for fine-tuning single-agent vision-language-action (VLA) models, yet extending this paradigm to multi-agent vision-language navigation (MA-VLN) remains largely unexplored. We present MA-RL-VLA, a centralized-training / decentralized-execution (CTDE) framework whose core design is a mathematically consistent integration of counterfactual credit into PPO for VLA-scale actors. The counterfactual baseline is identical in form to COMA's policy-marginalized baseline, and because one actor pass already yields the full distribution over the four discrete navigation actions, it is computed by exact enumeration at the cost of four small-critic evaluations per agent step. The detached, clipped credit weight is folded inside both PPO branches as a combined advantage ${\text{Act}}_i=\text{sg}[{\tilde{w}}_i]{\text{bA}}_i$, preserving PPO's pessimistic clipping under negative credit and thereby suppressing free-riding. The reward separates a policy-invariant potential-based progress term (fixed, never tuned) from auxiliary coordination terms with fixed weights selected on training-split tuning scenes. On CoNavBench (Habitat, $N=3$) with 8 matched seeds, MA-RL-VLA improves Team Success Rate by $+19.1$ points over behavior cloning (95% CI $[16.3,21.9]$) and $+6.4$ points over standard COMA-style credit ($[3.9,8.9]$; exact paired permutation $p=0.008$), and cuts exploration redundancy by $40.7\%$. Controlled ablations attribute the gain to the credit signal ($+8.0$), the objective form ($+2.3$ over the legacy outer-weight surrogate), and the exact estimator. Our evidence is confined to one simulated benchmark and one VLA backbone; we state precisely which guarantees do and do not transfer to the deployed update.