- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies federated adaptation for tool-using vision-language agents under a fixed upload budget. The proposed DREL method divides LoRA parameters into a shared reasoning partition and a private execution partition, with role-based gradient routing during training. Despite having three times the adapter parameters, DREL does not outperform a byte-matched FedAvg-32 baseline. Removing the private partition at inference improves accuracy in the three evaluated cells. The paper presents these findings as a negative result and emphasizes the importance of matching communication budgets.
Critical strengths
- The byte-matched comparison is useful. Reporting deployed parameters separately from uploaded parameters makes the trade-off clear and avoids attributing gains to an unmatched communication budget.
- Removing the private partition while keeping the shared parameters unchanged provides a useful diagnostic. The resulting improvement gives a concrete observation to investigate beyond the overall accuracy comparison.
Critical weaknesses
- The methodological contribution is limited. Global and local adapters already appear in personalized federated learning methods such as FedDPA. The more specific contribution here is the reasoning–execution split and role-based routing, but their benefits remain unclear. Matching upload budgets strengthens the evaluation, but is not by itself a strong methodological contribution.
- The negative finding is fairly narrow. The main experiment uses one backbone, four simulated clients, and two seeds, while the private-partition intervention covers only three cells. These results describe the evaluated configuration, but provide limited evidence about when or why the broader design fails. This leaves the paper with a relatively modest empirical contribution.
- Important explanations remain unresolved. The completed routing ablation also changes the auxiliary loss coefficients, and the controlled ablation was not completed. Repairing the training data also changes the direction of the observed comparison. It is therefore difficult to separate limitations of the proposed design from issues in the training data or configuration.
Questions for authors
See weakness
2 comments
Thank you for the response and additional experiments. I appreciate the correction of the routing claim, the transparent reporting of the data-repair results, and the repeated evaluations to assess uncertainty. These additions help clarify the scope and limitations of the findings.
Answer to DmYc: confounded ablation corrected
We agree with nearly every point in this review, and we have used the discussion period to close the two gaps you identified rather than to defend the submitted version.
The confounded ablation: corrected, and it reverses our own claim. You are right that the available routing comparison changed the auxiliary coefficients along with the routing flag, so it identified nothing. We ran the matched-coefficient version: coefficients held at orth 0.001 and role 0.03, only the routing flag off, both seeds, all eight cells. Routing is worth −0.25 points (5 of 8 cells worse, sign p = 0.73) — a null. In particular the "+0.94 for no-routing" number in our submission was produced by the coefficient change, not by routing, and we withdraw the interpretation we attached to it. The submitted sentence saying the unconfounded ablation "did not complete" is now obsolete and will be replaced.
The data repair: we did not hide the sign flip, but we also could not explain it, and still cannot. The pre-registered repair shifts the estimate by +1.89 points and flips its sign on one seed, which is exactly the instability you describe. Rather than adjudicate it, we now report both the original and repaired conditions side by side and state that neither resolves the comparison. What the repair does show is a concrete training defect — clients had been trained on tools they do not own in 2.8–17.8% of calls — which is a data problem, not an aggregation one, and we no longer present it as evidence about the design.
"Narrow" — we now bound the measurement instead of implying precision. The GTA evaluator samples at temperature 0.8. Repeated evaluation of one identical adapter varies by up to 8 points on a single cell. We repeated the two headline conditions three times per cell: the difference is −0.60 points, CI [−2.03,+0.83], 2 of 8 cells favouring DREL, against the −1.74 we submitted from one sample. We therefore claim only that no advantage is detectable at this scale, not that the design is inferior.
What we think survives. One result has a consistent sign across clients: deleting the shared partition is worse in 7 of 8 cells tested (−1.27 points, CI [−2.63,+0.10]), while deleting the private partition is not measurably different (+1.62, 5 of 7). We would reframe the contribution around that boundary rather than around the aggregate comparison, and we agree the global/private split itself is not novel relative to FedDPA — our claim is about what the boundary does under a byte-matched budget, not about inventing the split.