acceptodds
Under review as a conference paper at ICLR 2027

Byte-Matched Evaluation of Rank-Split Federated Adaptation for Tool-Using Vision-Language Agents

Abstract

Tool-using language agents interleave task reasoning with tool-specific execution. In federated deployments, clients own heterogeneous tool environments and cannot share raw trajectories, which makes it natural to give every client **private adaptation capacity** alongside a shared, uploaded partition. We test that premise under a **fixed upload budget** rather than a fixed parameter count. Rank-split federated adaptation (DREL) spends a client's per-event upload on a shared rank-32 reasoning partition while keeping a rank-64 execution partition private, so at the identical **87,293,952-parameter (333 MiB)** upload it deploys exactly **three times** the adapter parameters of a plain rank-32 FedAvg control. On a four-client GTA protocol with two seeds, the threefold capacity does not convert into accuracy: DREL does not show an accuracy advantage over the byte-matched rank-32 FedAvg control (**58.4% versus 60.1%** on the pre-specified eight-cell macro endpoint, **-1.74 points**, 95% CI [-3.56, +0.22]). We then ask where that capacity goes by intervention rather than argument: zeroing the private partition at inference, with the uploaded partition byte-identical, **raised mean accuracy from 58.53% to 60.85%** across the 3 cells evaluated (3 of 3 improving), where the byte-matched control scores 59.30%. We also expose a confounded comparison in our own earlier work that had motivated dropping the method's routing component, and report lower observed GAIA accuracy than local adaptation on the one client evaluated. The reusable lesson is a protocol one: **federated rank comparisons are confounded unless upload is matched exactly.**

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
Personalized Federated Learning, Language Agents, Tool-Augmented LLMs, Evaluation Methodology, Communication-Efficient Adaptation

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_DmYcOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies federated adaptation for tool-using vision-language agents under a fixed upload budget. The proposed DREL method divides LoRA parameters into a shared reasoning partition and a private execution partition, with role-based gradient routing during training. Despite having three times the adapter parameters, DREL does not outperform a byte-matched FedAvg-32 baseline. Removing the private partition at inference improves accuracy in the three evaluated cells. The paper presents these findings as a negative result and emphasizes the importance of matching communication budgets.

Critical strengths
  1. The byte-matched comparison is useful. Reporting deployed parameters separately from uploaded parameters makes the trade-off clear and avoids attributing gains to an unmatched communication budget.
  2. Removing the private partition while keeping the shared parameters unchanged provides a useful diagnostic. The resulting improvement gives a concrete observation to investigate beyond the overall accuracy comparison.
Critical weaknesses
  1. The methodological contribution is limited. Global and local adapters already appear in personalized federated learning methods such as FedDPA. The more specific contribution here is the reasoning–execution split and role-based routing, but their benefits remain unclear. Matching upload budgets strengthens the evaluation, but is not by itself a strong methodological contribution.
  2. The negative finding is fairly narrow. The main experiment uses one backbone, four simulated clients, and two seeds, while the private-partition intervention covers only three cells. These results describe the evaluated configuration, but provide limited evidence about when or why the broader design fails. This leaves the paper with a relatively modest empirical contribution.
  3. Important explanations remain unresolved. The completed routing ablation also changes the auxiliary loss coefficients, and the controlled ablation was not completed. Repairing the training data also changes the direction of the observed comparison. It is therefore difficult to separate limitations of the proposed design from issues in the training data or configuration.
Questions for authors

See weakness

2 comments
AuthorsOpenReview ↗

Answer to DmYc: confounded ablation corrected

We agree with nearly every point in this review, and we have used the discussion period to close the two gaps you identified rather than to defend the submitted version.

The confounded ablation: corrected, and it reverses our own claim. You are right that the available routing comparison changed the auxiliary coefficients along with the routing flag, so it identified nothing. We ran the matched-coefficient version: coefficients held at orth 0.001 and role 0.03, only the routing flag off, both seeds, all eight cells. Routing is worth −0.25 points (5 of 8 cells worse, sign p = 0.73) — a null. In particular the "+0.94 for no-routing" number in our submission was produced by the coefficient change, not by routing, and we withdraw the interpretation we attached to it. The submitted sentence saying the unconfounded ablation "did not complete" is now obsolete and will be replaced.

The data repair: we did not hide the sign flip, but we also could not explain it, and still cannot. The pre-registered repair shifts the estimate by +1.89 points and flips its sign on one seed, which is exactly the instability you describe. Rather than adjudicate it, we now report both the original and repaired conditions side by side and state that neither resolves the comparison. What the repair does show is a concrete training defect — clients had been trained on tools they do not own in 2.8–17.8% of calls — which is a data problem, not an aggregation one, and we no longer present it as evidence about the design.

"Narrow" — we now bound the measurement instead of implying precision. The GTA evaluator samples at temperature 0.8. Repeated evaluation of one identical adapter varies by up to 8 points on a single cell. We repeated the two headline conditions three times per cell: the difference is −0.60 points, CI [−2.03,+0.83], 2 of 8 cells favouring DREL, against the −1.74 we submitted from one sample. We therefore claim only that no advantage is detectable at this scale, not that the design is inferior.

What we think survives. One result has a consistent sign across clients: deleting the shared partition is worse in 7 of 8 cells tested (−1.27 points, CI [−2.63,+0.10]), while deleting the private partition is not measurably different (+1.62, 5 of 7). We would reframe the contribution around that boundary rather than around the aggregate comparison, and we agree the global/private split itself is not novel relative to FedDPA — our claim is about what the boundary does under a byte-matched budget, not about inventing the split.

Thank you for the response and additional experiments. I appreciate the correction of the routing claim, the transparent reporting of the data-repair results, and the repeated evaluations to assess uncertainty. These additions help clarify the scope and limitations of the findings.

Reviewer_zPBjOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper presents rank-split federated adaptation (DREL), a federated learning technique that trains tool-using agents in an environment where different clients have their own set of tools. DREL trains two LoRA adapters: one shared and one private. The evaluation shows negative performance of DREL, such as a 1.74-point degradation for the four-client GTA protocol compared to the upload-matched FedAvg-32 baseline on average.

Critical strengths
  1. The proposed separation of reasoning and execution is a simple technique and intuitively makes sense, so it would be able to be adopted in many scenarios if that works.
Critical weaknesses
  1. The paper is mainly about reporting negative results, and the evaluation covers a limited design space: the paper does not systematically examine alternative shared/private rank allocations or sensitivity to DREL's hyperparameters. Although the private-adapter removal and training-data repair experiments provide useful diagnostic evidence, they do not provide analysis to understand why DREL underperforms.
Questions for authors
  1. Can you provide more analysis on whether that task can really be decomposed into reasoning vs. execution, and whether the DREL's LoRA adapters were really able to learn such decomposition?
2 comments
AuthorsOpenReview ↗

Answer to zPBj: design space and the noise floor

We take the rating of 1 seriously and will not argue with the substance of it: as submitted, this is a negative result with an unexplored design space. Two of your three concerns we can now answer with data; on the third we disagree, and we want to be explicit about why rather than appear evasive.

"Why does DREL underperform?" — we can now separate this from noise. The submitted number (−1.74, CI [−3.56,+0.22]) came from a single evaluation per cell. The GTA evaluator samples at temperature 0.8, and repeated evaluation of an identical adapter varies by up to 8 points on a single cell. We re-ran the two headline conditions three times per cell. On repeat means the difference is −0.60 points, CI [−2.03,+0.83], with 2 of 8 cells favouring DREL. So the correct conclusion is not "DREL underperforms by 1.7 points" but "we cannot resolve a difference at this scale, in either direction". Our submitted point estimate overstated the effect, and we will correct it.

"Does the decomposition work at all?" — we now have a two-sided test. Deleting the shared partition costs −1.27 points, 7 of 8 cells worse; deleting the private partition costs nothing measurable (+1.62, 5 of 7 cells); and the matched-coefficient routing ablation that was missing at submission gives −0.25 points, 5 of 8 cells, i.e. a null. The partition boundary is doing something; the token-role routing on top of it is not measurably doing anything.

"No rank-allocation or hyperparameter sensitivity" — we decline the rank sweep deliberately, and offer a stronger substitute. Changing r_R changes the number of uploaded bytes. Our entire claim is defined at an exactly matched payload (both arms transmit 87,293,952 float32 = 333 MiB), so a rank sweep would silently convert a byte-matched comparison into an unmatched one and the resulting numbers would not be comparable to anything else in the paper. What we did instead is address the variability you are implicitly pointing at: we measured the evaluation noise floor directly, by repeatedly evaluating identical weights, and we report every headline effect as a mean over three repeats with a standard error instead of a single sample. We also report seed and client dispersion explicitly. If you would still prefer a rank sweep under an adjusted-budget framing, we will add it as a follow-up rather than fold it into a comparison it would invalidate.

We have not fixed the underlying weakness you identify: one backbone, four simulated clients. We say so in the revision rather than implying generality.

Thanks for the response. While the new results are helpful, the weaknesses I pointed out in the original review are not resolved. If I understand your new results correctly, the proposed method still does not bring performance improvement, and the evaluation scope is still very narrow. Therefore, I want to keep my original score.

Reviewer_C5PrOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies federated learning for tool-using agents. It is motivated by the fact that different clients have different tools. The authors propose DREL, a method that separates reasoning and execution. They use a shared LoRA partition for reasoning and a private LoRA partition for execution. The authors evaluate DREL against several baselines. The results show that DREL does not outperform the FedAvg baseline. They also find that disabling the private execution partition improves accuracy in all three of the tested settings.

Critical strengths

The paper addresses an important problem of federated learning for agents. The main comparison is done under the same communication budget, which makes the comparison more fair. I also like that the authors report the negative results.

Critical weaknesses
  1. DREL does not outperform the simpler FedAvg baseline under the same communication budget, even though it uses more parameters. Also, disabling the private execution partition improves accuracy in all three tested settings. This raises questions about whether the private partition is actually useful.

  2. The main idea behind DREL is to separate reasoning and tool execution. But the paper does not clearly show whether the two LoRA partitions actually learn these different roles. This makes it hard to know whether the proposed separation works. Reasoning tokens may contain tool-specific information, while execution tokens may also contain reasoning information.

Questions for authors
  1. Why does the private execution partition seem to hurt accuracy in all tested settings?

  2. Is there any evidence that the reasoning and execution partitions actually learn different types of information?

2 comments
AuthorsOpenReview ↗

Answer to C5Pr: partition role, with the missing ablations

We thank Reviewer C5Pr for the two most useful questions in this round, and we answer them with new experiments rather than argument. Both concern whether the reasoning and execution partitions earn their names.

Why does the private partition hurt? It does not clearly hurt, and we should not have left that impression. Our submitted claim rested on a single evaluation of three cells. The GTA evaluator samples at temperature 0.8, and on duplicate evaluations of the same adapter we observe differences of up to 8 points on one cell (e.g. control client C/seed42: 64.0, 55.8, 55.8). A one-point effect cannot be read from one sample. We therefore re-ran the two headline conditions three times each across all eight client-seed cells. On repeat means, zeroing the private partition gives +1.62 points, improving in 5 of 7 evaluated cells — directionally as we reported, but not significant, and two cells get worse. The honest statement is that the private partition contributes nothing measurable, not that it is harmful.

Do the partitions learn different things? We now have a two-sided test instead of an argument. If the boundary is real, deleting the shared partition should hurt and deleting the private one should not. It behaves that way: zeroing the shared partition costs −1.27 points, with seven of eight cells worse, while zeroing the private partition costs nothing measurable. We also ran the ablation that was missing at submission: holding the auxiliary coefficients fixed and switching only the routing flag, routing is worth −0.25 points (5 of 8 cells worse without it) — indistinguishable from zero. The earlier "+0.94 for no-routing" was an artefact of changing the coefficients at the same time, and we withdraw it.

On your conceptual objection, we agree. You are right that Thought tokens name the tool and Code tokens instantiate that decision, so the two spans are entangled and a token-role label is not a semantic guarantee. We now state this as a limitation of the design premise rather than assuming the split is clean, and the null routing effect is consistent with your reading: if the labels do not isolate the intended content, routing on them should do little, which is what we measure.

What we do not claim. With eight cells and one backbone we cannot establish that partitioned sharing fails in general. What we can support is narrower: at an exactly matched 333 MiB upload, this partition does not beat its byte-matched control (−0.60 points, CI [−2.03,+0.83], 2 of 8 cells favouring DREL), and the only sub-component with a consistent sign across clients is the shared partition.

Thanks for the additional experiments. Based on the new results, the private partition still does not show a measurable benefit. Also, DREL still does not outperform the baseline, so I will keep my original score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.