acceptodds
Under review as a conference paper at ICLR 2027

Concordia: Self-Improving Synthetic Tables for Federated LLMs

Abstract

Adapting large language models (LLMs) to tabular prediction in regulated cross-silo settings is difficult when institutions cannot share raw records, validation sets, synthetic tables, generators, or downstream model updates. This challenge is amplified by non-IID client distributions, extreme class imbalance, and limited communication budgets, where only one or two rounds may be feasible. Synthetic tables offer a natural on-premise training substrate, but existing pipelines often treat generation as static or optimize proxy objectives disconnected from the evolving downstream learner. We propose Concordia, a tri-level limited-shot framework that turns private validation utility into an online signal for refining client-local synthetic generation. Each client performs local LoRA adaptation on synthetic tables, learns a capacity-limited utility scorer from private validation feedback to reweight synthetic samples, and uploads only this scorer. The server pools and redistributes scorers within each task, and each client uses the pooled scorer feedback to refine its own generator with group-relative policy optimization (GRPO). Across finance and healthcare tabular benchmarks, Concordia improves early-round MCC under severe heterogeneity and imbalance, with the clearest gains on the long-tail Travel task where positives are near 0.01; the finance tasks are single-client streams and measure local self-improvement, while healthcare evaluates multi-client scorer pooling. Generator audits further suggest that scorer-guided refinement changes the support of synthetic training data, preserving coherent feature dependencies in finance and emphasizing multi-feature risk patterns in healthcare.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
federated learning, synthetic tabular data, large language models, LoRA adaptation, GRPO, utility scorer, cross-silo privacy, class imbalance, limited-shot communication

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_hVuoOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2(+1)
Read review
Paper summary

Concordia studies LLM based tabular prediction across institutions that cannot share data, synthetic tables, generators or LoRA updates, and that can communicate only once or twice. Each client trains a LoRA predictor on synthetic rows from a local LLM generator, learns a small LoRA scorer from changes in validation MCC to weight these rows, and uploads only the scorer. The server pools the scorers within a task and sends them back, and each client then refines its generator with GRPO. On three finance datasets, each treated as a single client, and on the Heart Disease sites, the paper reports MCC gains over the Qwen backbones after one and two rounds.

Critical strengths

S1: The paper looks at a realistic and under-studied setting, regulated cross-silo deployment where even model updates cannot leave the institution and only one or two rounds are possible. It also states clearly what can and cannot be shared.

S2: Using private validation utility to guide the synthetic data generator, instead of only making the data look realistic, is a reasonable direction, and it links sample reweighting with GRPO based refinement of the generator.

S3: The authors are open about several limitations. The finance results are single client self-improvement, there is no formal privacy guarantee, and transfer to the VA site is uneven. Appendices C, E and F give fairly complete hyperparameters, prompts and payload sizes.

Critical weaknesses

W1: The baselines do not support the main claims. (a) The local adaptation procedure for the single-node baselines is not clearly specified, despite Section 4.1 and Appendix C.7 stating that methods use the same LoRA configuration and local optimization budget. (b) Section 4.1 excludes local supervised learners, imbalance aware training and static synthesizers because they "provide no signal that can cross the export boundary". These methods do not need to cross the boundary, since every client already holds its own data. In the finance experiments there is no communication between clients at all, so local XGBoost or CatBoost with class weights, LoRA on the real local data and LoRA on static synthetic data are all allowed and are the most relevant comparisons. The next sentence then says "They are included only to test whether relaxing the communication boundary to standard update aggregation is sufficient", so it is unclear whether these methods were run at all. (c) FedLoRA is said to collapse to near zero MCC, but no numbers are reported.

W2: The evidence for the method is weak. First, no ablation supports the claim that the three levels are jointly necessary. The paper does not report synthetic LoRA without the scorer, scorer weighting without GRPO, GRPO with a local scorer compared with the pooled one, or a random or shuffled scorer. The scorer signal is also very weak, since a single change of validation MCC over 15 steps is credited to the average score of the last batch of 4 samples, and the healthcare validation sets have only 17 to 41 rows. Second, the results are statistically fragile. The healthcare test sets have 13 to 31 rows. Switzerland has a positive rate of 0.94 in its training split, so its 13 test rows probably contain only one or two negatives, and standard deviations reach about 40 MCC points with 3 seeds. The same small validation set is used to train the scorer, to define the GRPO reward and to choose the MCC threshold. Several results get worse in round 2, for example the Hungarian model on VA drops from 72.76 to 34.3 and the Switzerland model on VA from 42.01 to -9.64. An MCC of 90.22 on LendingClub is unusually high for charged off prediction and may point to leakage from features recorded after loan origination, and the feature list is not given. Finally, the privacy argument is weak. Section 2 forbids sending "parameter-efficient task-model updates such as LoRA or adapter weights", yet the uploaded scorer is itself a LoRA adapter (rank 8 on q and v, about 2.95M parameters, Appendix C.5 and F) trained on private validation feedback, and no membership or property inference test is reported. The federated part is only tested in healthcare with three small clients, and the strongest result, on Travel, does not involve federation.

W3: The writing is hard to follow and contains several inconsistencies. (a) The text uses many coined terms, such as tri-level limited-shot, scorer pooling, compliance-constrained and reduced-exposure channel, and repeats the list of artifacts that stay on premise many times. Section 3 would be clearer with a concrete example. (b) Section 4.1 says the generators start from Qwen3-4B-Instruct, but Appendix C.1 says Qwen3-4B is the backbone for both the predictor and the generator. (c) Algorithm 1 refines the generator after receiving the pooled scorers, while Appendix C.2 refines it before uploading the scorer. (d) The generator audit uses two hand picked pairs (Tables 3 and 4), and Figure 2 has no axes, no round labels and no description of the embedding method, so it cannot support the claim that refinement changes the support of the synthetic data.

Questions for authors

Q1 (see W1): Are the single node baselines zero-shot? Please add local GBDT with class weights on the real data, LoRA on the real data with the same threshold tuning, LoRA on static synthetic data, and the FedLoRA numbers. If Concordia does not beat the first two in finance, the case for training only on synthetic data needs to be reconsidered.

Q2 (see W2 ): Please report the component ablations listed in W2, including a shuffled or random scorer. Could the authors also use cross validation or larger held out splits, more seeds, and separate data for threshold selection and scorer training? Please list the LendingClub features and explain the 90.22 MCC.

Q3 (see W2 ): Can the authors report membership and property inference attacks on the uploaded scorers, compared with a task LoRA of similar size, to support treating the scorer differently from the LoRA updates that the threat model forbids?

Q4 (see W3 ): Please clarify which checkpoint the generator starts from and in which order generator refinement and scorer upload happen. I also suggest adding a small worked example in Section 3, and redrawing Figure 2 with axes, round labels and a description of the embedding method, together with a quantitative comparison of the synthetic data before and after refinement instead of two hand picked pairs.

3 comments
AuthorsOpenReview ↗

Response to Reviewer hVuo

Thank you for recognizing the realism of this under-studied regulated setting, the direction of guiding generation by private validation utility rather than realism alone, and our openness about limitations. We keep that openness below.

W1, baselines. You are right that these methods need not cross the boundary and that the sentence did not say why we excluded them. Single-node rows are zero-shot (see FeeU Q1). The setting adapts an LLM without gradient updates on real records, given documented memorization under LoRA (Li et al., 2026; Bossy et al., 2026). The relevant comparison is the same LLM predictor trained on synthetic versus real records, and Table 3 reports it on all six tasks together with the tabular learners and FedLoRA. Section 2 now states the reason.

W2, ablations and signal strength. You are right that the evidence on individual components is thinner than the claim. The fixed-generator row in Table 3 isolates scorer-weighted synthetic training. Section 1 now describes the levels by role, and the claims stay with what the tables show.

W2, statistical fragility. You are right that healthcare test sets have 13–31 rows, and Switzerland has very few negatives. These sites form the public multi-site clinical benchmark, and small sites are the regime the setting targets. Section 4.1 and the caption of Table 2 now give test sizes, and Section 4.2 names the round-2 transfer regressions you point out.

W2 and Q2, LendingClub. You are right to suspect this entry: Last_Payment_Amount is observed after origination. Appendix C.7 lists the features, Section 4.1 notes the leak, and the paper no longer draws on this result.

W2 and Q3, scorer versus LoRA. You are right that the uploaded scorer is itself an adapter trained on private feedback. The claim concerns what crosses the boundary under a stated policy, and the paper gives no privacy guarantee. Section 2 now makes this explicit and requires membership and property inference testing of the scorers before deployment.

W3, writing. Thank you for these. "Limited-shot" is now "few-round" throughout, and Section 3.1 has a worked example. Generators start from Qwen3-4B-Instruct and the predictor backbone is Qwen3-4B, as Appendix C.1 now states. Appendix C.2 now follows the order of Algorithm 1: upload the scorer, pool, then refine the generator. Figure 2 (Appendix H) now labels the two rounds, and its caption describes the UMAP projection and the reward coloring.

Thank you for the response. I accept the LendingClub clarification in W2 and the local references added in Table 3, and I have raised my Soundness score. Some points remain unresolved:

For W1(a), you state that the single-node rows are zero-shot. But Section 4.1 still says all methods use the same LoRA configuration, and Appendix C.7 still says they use the same local optimization steps. So Table 2 compares zero-shot inference with 1,500 steps of LoRA training

For W2, Travel is a single-client stream, but Table 5 lists three scorers. Which scorers refined the Travel generator? Second, you use the fixed-generator row of Table 3 to separate scorer-weighted training from generator refinement. Per Appendix H, the round-1 predictor is also trained on the initial generator. The Table 3 caption names only the scorer-head initialization as a difference. Yet the two differ substantially on Cleveland and Hungarian, by more than the round-2 gains. On Hungarian, the full method is also below the fixed generator

Some points in W1(b) and W2 are still not addressed, for example class-weighted baselines and a shuffled or random scorer control

For above reasons, I keep my overall rating

Thank you for the follow-up and for raising the Soundness score.

The question the paper addresses is whether an LLM predictor can be adapted for a client without a gradient step on a real record, when only a scorer may leave the site. Table 3 answers it under one protocol for every row: three seeds, held-out checkpoint selection, and a single evaluation on the test split. Trained on synthetic tables, the predictor is within seed variance of the same predictor trained on real records on Cleveland, Hungarian and German, and it exceeds that predictor on Travel, where real-data training collapses to the majority class. Parameter averaging collapses to a near-constant predictor on healthcare and is near chance on finance. This is the main evidence for the setting.

We will provide further evidence and clarifications on the points you raise in the final version.

Reviewer_1gmcOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes Concordia for adapting LLMs to tabular prediction in cross-silo settings where institutions can export neither data nor model updates. Each client fine-tunes a predictor with LoRA on synthetic tables from its own generator, trains a small scorer to reweight synthetic samples, and uploads only the scorer to the other clients. And then, each client uses the pooled scores as a reward to refine its own generator with GRPO. Experiments on three finance datasets and four Heart Disease sites report MCC gains over the Qwen3-4B backbones.

Critical strengths

The paper studies an unusual setting in which institutions can export neither data nor model updates, and can communicate only once or twice. Up to my knowledge, this setting has not been studied before, and sharing a validation-derived scorer instead of model updates is a distinctive way to approach it.

Critical weaknesses

W1. The validation set becomes a training signal. The scorer is trained from validation MCC changes, the generator is optimized against the scorer, and the decision threshold is also selected on the same validation split. With validation sets of only 17–41 rows in healthcare (Table 1), the gains may reflect fitting to the validation set rather than better synthetic data.

W2. Baselines cannot isolate where the gain comes from. The only feasible baselines are the Qwen 3 4B. Since the proposed method additionally learns from validation feedback, outperforming them is trivially expected and says little about the method itself. Please add ablations that remove each component (synthetic training without the scorer, scorer weighting with a fixed generator, refinement without pooling), and compare against local training on the client's own real data and static synthesizers (e.g., CTGAN, GReaT, EPIC).

Questions for authors

Q1. The setting forbids exporting task LoRA updates, yet permits exporting the scorer, which is itself a LoRA adapter trained on private validation data. Is there a real regulation or deployment that uses these kinds of regulations? Have the authors compared the information leakage of the them?

4 comments
AuthorsOpenReview ↗

Response to Reviewer 1gmc

Thank you for recognizing that this setting has not been studied before and that sharing a validation-derived scorer instead of model updates is a distinctive approach. The concerns below are about evidence and scope within that setting.

W1, validation as training signal. You are right that validation feedback drives learning and that these sets are small. Private validation utility is the signal by design: under this boundary it is the only feedback a client can act on. The test split plays no part in it. In Table 3, checkpoints are selected without the test split, which is evaluated once. One of the three uses you list comes from an error in our Appendix C.1. LLM predictions take the label with the highest log-likelihood, with no threshold selection. We have corrected that sentence, and Section 5 lists validation size as a limitation.

W2, baselines. You are right that beating the Qwen backbones alone says little. Table 3 in the main text now gives the local references and the fixed-generator row, which isolates scorer-weighted synthetic training from generator refinement. Appendix A now discusses static synthesizers such as CTGAN (Xu et al., 2019), GReaT (Borisov et al., 2023) and EPIC (Kim et al., 2024).

Q1, deployment and leakage. You are right that the scorer is also trained on private feedback, so the paper should not present it as inherently safe. The motivation for few rounds is operational and documented. In distributed research networks such as OHDSI and PCORnet, each round requires cross-institution coordination, synchronization, secure transfer and human steps, under data use agreements and IRB protocols (Hripcsak et al., 2015; Fleurence et al., 2014; Chen et al., 2026). Iterative distributed methods such as GLORE (Wu et al., 2012) were followed for this reason by communication-efficient and one-shot methods (Jordan et al., 2019; Duan et al., 2019; Luo et al., 2022; Wu et al., 2025). The export rule is a deployment policy, and the paper states that it gives no formal privacy guarantee. Section 2 now cites these networks, presents the boundary as a deployment policy, and requires leakage testing of the scorers before deployment.

Thank you again for your review. We would appreciate it if you could let us know whether our response addresses your concerns.

Reviewer_1gmcOpenReview ↗

Response to the review

Thank you for your response. I read the revised manuscript carefully, but parts of the explanation of Table 3 remain unclear to me. As I understand it, LendingClub is the only one of the five datasets on which the predictor trained on real data outperforms the same predictor trained on synthetic data. This suggests that training the predictor on real data offers very limited benefit in this comparison. I therefore plan to keep my rating at Weak Reject.

Thank you for reading the revision. You are right that training the predictor on real records brings limited benefit in Table 3. This is the reality a regulated client faces, and it is the point we make. The healthcare sites have 93 to 231 training rows, and on Travel, with a positive rate near 0.01, real-data training collapses to the majority class. Training on its own records gains a client little and exposes those records to memorization. Synthetic training reaches the same level without a gradient step on a real record, and it exceeds real training where the imbalance is extreme.

Because no single site holds enough records, clients need to collaborate. Our setting is strict: raw records, validation sets, synthetic tables, generators and LoRA updates all stay on premise. Standard aggregation has to relax this boundary, and even then it fails in this regime, since FedLoRA collapses to a near-constant predictor on healthcare. Concordia lets clients collaborate while exporting only a scorer.

Reviewer_FeeUOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper aims to target the regulated cross-silo settings. In these scenarios, the raw data, validation sets, synthetic tables, generators, and LoRA updates both cannot leave the client, and only one to two communications between the client and server are allowed. The paper proposed a method that each client can generate synthetic tables with a local generator. It then fine-tunes a LoRA predictor with per-sample weights. The weights come from a utility scorer that is updated at every 15 steps based on the variations in private-validation MCC. Then the clients only upload the scorer (i.e., a rank-8 LoRA with a scalar head). The server will then pools scorers within each of task and then redistributes them. Each client uses group-rank-normalized pooled scores as the GRPO reward score to refine the generator.

Critical strengths
  • Within-group rank normalization removes scale differences across scorers.
  • The Task-local pooling avoids the ill-posed cross-task aggregation.
  • The GRPO update is regularized with a KL penalty toward the round-start generator.
  • The communication-cost calculation is concrete, and reproducibility details (hyperparameters, model revision hash) are fairly complete.
Critical weaknesses
  • All three finance tasks are singleton task streams, so each scorer pool contains a single scorer. The most striking results (Travel, LendingClub) therefore have nothing to do with cross-client pooling, and labeling the finance rows of Table 2 “after N scorer-pooling rounds” is misleading. The only multi-client setting is the three Heart Disease sites, and there is no comparison of local-scorer-only versus pooled-scorer refinement there. As a result, there is no evidence that pooling helps.

  • Baselines are largely missing, and the claim that “the three levels are jointly necessary” has no ablation support. The only feasible baselines are single-node Qwen3-4B and Qwen3-4B-Instruct, and the paper does not say whether these are zero-shot or fine-tuned on something.

  • The Travel gain may largely reflect more positives in the synthetic data, yet the label distribution of the synthetic data is never reported for any round. Several closely related works are not discussed. EPIC (NeurIPS 2024) uses class-balanced LLM prompting for imbalanced tabular synthesis and is a natural on-premise baseline. CLLM (ICML 2024) curates LLM-generated tables via learning dynamics. SunGen (ICLR 2023) reweights synthetic samples via bilevel optimization, close to the scorer weighting. Generative Teaching Networks (ICML 2020) update a generator through the learner’s meta-gradient, the hypergradient Concordia approximates. TabLLM (AISTATS 2023), whose recipe the predictor follows, is not cited.

  • The paper excludes GBDTs, oversampling, CTGAN, and similar methods because they “act on a single client’s own data and provide no signal that can cross the export boundary.” However, the finance experiments are themselves single-client. Moreover, every client holds a real training set D_i, and training on it locally violates nothing in the protocol. The key question for readers is how Concordia compares with simply training XGBoost or a LoRA predictor on local real data.

Questions for authors
  • What exactly is the single-node baseline: zero-shot, or fine-tuned, and if so on which data? Why are some standard deviations exactly 0?
  • Which 21 features are used for LendingClub?
  • What is the positive rate of the synthetic data in each round, and how is the class chosen for the “in-class exemplars”?
  • On Travel, what fraction of scorer refreshes receive a non-zero δ?
  • With the ablations listed in Weakness 2, how much does each component contribute?
  • Can you provide full FedLoRA results and its configuration?
4 comments
AuthorsOpenReview ↗

Response to Reviewer FeeU (1/2)

Thank you for recognizing the within-group rank normalization, task-local pooling, the KL-anchored GRPO update, and the concrete communication accounting. These components make a scorer-only exchange workable, and the points below concern how we scoped and reported around them.

Finance rows and pooling. You are right that singleton streams contain one scorer, so they say nothing about pooling. They test the local refinement loop under the same export boundary, as the abstract states, and healthcare is the multi-client setting. Table 2 now labels the finance rows as local refinement rounds of single-client streams.

Evidence for pooling. You are right that the paper has no local-versus-pooled comparison. The contribution is a protocol in which the scorer is the only exported artifact, and pooling is the exchange that protocol permits. Section 1 now describes pooling in these terms, and Section 5 lists the comparison as future work.

Baselines and "jointly necessary". You are right that the wording went beyond the ablations shown. Table 3 adds the missing references, including the fixed-generator row, and Section 1 now describes the role of each level in place of "jointly necessary".

Why real local data is not the target comparison. You are right that local training violates no export rule, and our exclusion sentence did not give the actual reason. LLMs can retain and regenerate their training records (Carlini et al., 2021; 2023). In medicine, Li et al. (2026) report that 4-bit LoRA fine-tuning on 10,000 inpatient records regenerated 3,192 PHI instances, and that de-identification may not suffice. LoRA reduces this risk without removing it (Bossy et al., 2026; Wang and Li, 2025), and memorization grows with LoRA rank (Savine, 2025). Synthetic text can also leak (Meeus et al., 2025), so the design reduces exposure without guaranteeing privacy. Tabular learners on real data do not adapt an LLM. Among LLM predictors, the comparison that matters is synthetic against real training of the same predictor. In Table 3, synthetic training is within seed variance of real training on Cleveland, Hungarian and German and exceeds it on Travel. Section 2 now gives this rationale, and Table 3 is in the main text.

Related work. You are right that EPIC (Kim et al., 2024), CLLM (Seedat et al., 2024), SunGen (Gao et al., 2023), Generative Teaching Networks (Such et al., 2020) and TabLLM (Hegselmann et al., 2023) belong in the discussion. None of them exports only a validation-trained scorer under a one- or two-round boundary. Appendix A now discusses each and positions Concordia against them.

AuthorsOpenReview ↗

Response to Reviewer FeeU (2/2)

Q1. Single-node rows are zero-shot, with no fine-tuning, and use no chat template. With the template, the model predicts one class on all four healthcare sites. Exactly 0.00 ± 0.00 in the method rows marks single-class prediction. Each single-node entry comes from one zero-shot run, and its ± is a bootstrap standard error. The two VA entries with zero spread were reporting errors. The correct values are 21.00 ± 8.86 (Qwen3-4B) and -9.64 ± 6.73 (Qwen3-4B-Instruct). Table 2 now has the corrected values, and Section 4.1 gives the evaluation mode and the meaning of ±.

Q2. The 21 features are Installment, Loan_Purpose, Loan_Application_Type, Interest_Rate, Last_Payment_Amount, Loan_Amount, Revolving_Balance, Delinquency_In_2_years, Inquiries_In_6_Months, Mortgage_Accounts, Grade, Open_Accounts, Revolving_Utilization_Rate, Total_Accounts, Fico_Range_Low, Fico_Range_High, Address_State, Employment_Length, Home_Ownership, Verification_Status, Annual_Income. Appendix C.7 now lists them.

Q3. You are right that this should be reported. Prompts are built per label in equal numbers, and the five exemplars are drawn from the pool of the same label, so synthetic tables are close to class-balanced by construction. Positive rate in round 0 / round 1 tables: Travel .50 / .50, German .50 / .50, LendingClub .51 / .51, Switzerland .50 / .50, Hungarian .56 / .20, Cleveland .72 / .29. Class-balanced generation is part of the method, and real records cannot offer it. At Travel's real rate near .01, the same LoRA predictor trained on real records collapses to the majority class (0.00 in Table 3), while synthetic training reaches 7.99. The Travel rate stays at .50 in both rounds, so refinement changes which records are generated while the label mix stays fixed. Appendix C.4 now reports these rates.

Q4. You are right that this statistic matters at a positive rate near .01, where a small batch can leave validation MCC unchanged. We did not retain per-refresh values for the reported runs, so we cannot give the fraction. Section 5 now lists the sparsity of this signal as a limitation.

Q5. The fixed-generator row in Table 3 gives the contribution of scorer-weighted synthetic training without generator refinement.

Q6. FedLoRA uniformly merges the round-1 LoRA adapters of all healthcare clients for each seed and evaluates the merged model on each site, over 3 seeds. On finance, where each task has one client, it merges the adapters of the three tasks in the same way. Results are in Table 3, and Section 4.1 and Appendix C.7 give the configuration.

Thanks for the detailed rebuttal which addressed many of my concerns. I have raised my score.

Thank you for reading our responses and for raising your score.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Shared results for all reviewers

We have uploaded a revised PDF. Changes made during the discussion are shown in blue. Section, table and appendix numbers below refer to the revision.

The table below gives the local references requested by all reviewers, now Table 3 in Section 4.2. MCC ×100, own-site test, mean ± sd over 3 seeds. Tabular models use validation-tuned thresholds, and LLM rows take the label with the highest log-likelihood.

MethodClevelandHungarianSwitzerlandGermanLendingClubTravel
XGBoost, real data75.92 ± 1.7162.95 ± 4.510.00 ± 0.0044.35 ± 2.4560.80 ± 0.5216.57 ± 1.16
Logistic regression, real data80.51 ± 0.0056.57 ± 0.000.00 ± 0.0028.97 ± 0.0058.70 ± 0.0012.92 ± 0.00
LoRA, real data68.06 ± 12.2051.44 ± 6.77-4.10 ± 7.1130.16 ± 1.4893.69 ± 0.440.00 ± 0.00
LoRA, synthetic, fixed generator with scorer63.50 ± 4.3057.08 ± 10.770.00 ± 0.0027.72 ± 9.8885.82 ± 4.327.99 ± 1.88
FedLoRA0.00 ± 0.000.00 ± 0.000.00 ± 0.009.02 ± 23.978.27 ± 14.32-1.16 ± 3.20

These baselines use the same test splits and metric as Table 2. Checkpoints are selected on a held-out selection split, and the test split is evaluated once. The fixed-generator row uses a scorer-head initialization that differs from the submitted configuration. Real-data LoRA trains the same predictor on each client's real training records and selects on its validation split. On finance each task has a single client, so FedLoRA uniformly merges the German, LendingClub and Travel adapters of each seed. LendingClub includes a post-origination payment amount (see hVuo W2).

The table answers the question the setting poses: how well can an LLM predictor do without a gradient step on a real record? Trained on synthetic tables, it is within seed variance of the same predictor trained on real records on Cleveland, Hungarian and German. On Travel it reaches 7.99, while the real-data predictor collapses to the majority class at a positive rate near .01. It is lower only on LendingClub. Parameter averaging collapses to a near-constant predictor. FedLoRA moves the Cleveland label margin from about -14 to between -1 and -3 but compresses its spread, so all records fall on one side. Merged across finance tasks, it is unstable and near chance on all three tasks, and LendingClub falls from 85.82 to 8.27. Section 4.2 discusses these results in a new paragraph.

AuthorsOpenReview ↗

References cited in the responses

Memorization and privacy

  • Carlini et al. (2021). Extracting Training Data from Large Language Models. USENIX Security Symposium, 2633–2650.
  • Carlini et al. (2023). Quantifying Memorization Across Neural Language Models. ICLR.
  • Li et al. (2026). Memorization in large language models in medicine prevalence characteristics and implications. Nature Communications 17, 7729.
  • Bossy et al. (2026). Mitigating Unintended Memorization with LoRA in Federated Learning for LLMs. TMLR.
  • Wang and Li (2025). Leaner Training, Lower Leakage: Revisiting Memorization in LLM Fine-Tuning with LoRA. arXiv:2506.20856.
  • Savine (2025). Memorization in Fine-Tuned Large Language Models. arXiv:2507.21009.
  • Meeus et al. (2025). The Canary's Echo: Auditing Privacy Risks of LLM-Generated Synthetic Text. ICML, PMLR 267, 43557–43580.

Multi-site networks and few-round methods

  • Chen et al. (2026). PDA (Privacy-Preserving Distributed Algorithms) in action: ten principles for high-quality multi-site clinical evidence generation. JAMIA 33(9), 1636–1648.
  • Hripcsak et al. (2015). Observational Health Data Sciences and Informatics (OHDSI): Opportunities for Observational Researchers. Studies in Health Technology and Informatics 216, 574–578.
  • Fleurence et al. (2014). Launching PCORnet, a national patient-centered clinical research network. JAMIA 21(4), 578–582.
  • Wu et al. (2012). Grid Binary LOgistic REgression (GLORE): building shared models without sharing data. JAMIA 19(5), 758–764.
  • Jordan et al. (2019). Communication-Efficient Distributed Statistical Inference. JASA 114(526), 668–681.
  • Duan et al. (2019). ODAL: A one-shot distributed algorithm to perform logistic regressions on electronic health records data from multiple clinical sites. Pacific Symposium on Biocomputing 24, 30–41.
  • Luo et al. (2022). DLMM as a lossless one-shot algorithm for collaborative multi-site distributed linear mixed models. Nature Communications 13, 1678.
  • Wu et al. (2025). COLA-GLM: collaborative one-shot and lossless algorithms of generalized linear models for decentralized observational healthcare data. npj Digital Medicine 8, 442.

Synthetic tabular data and related methods

  • Kim et al. (2024). EPIC: Effective Prompting for Imbalanced-Class Data Synthesis in Tabular Data Classification via Large Language Models. NeurIPS 37, 31504–31542.
  • Seedat et al. (2024). Curated LLM: Synergy of LLMs and Data Curation for tabular augmentation in low-data regimes. ICML, PMLR 235, 44060–44092.
  • Gao et al. (2023). Self-Guided Noise-Free Data Generation for Efficient Zero-Shot Learning. ICLR.
  • Such et al. (2020). Generative Teaching Networks: Accelerating Neural Architecture Search by Learning to Generate Synthetic Training Data. ICML, PMLR 119, 9206–9216.
  • Hegselmann et al. (2023). TabLLM: Few-shot Classification of Tabular Data with Large Language Models. AISTATS, PMLR 206, 5549–5581.
  • Borisov et al. (2023). Language Models are Realistic Tabular Data Generators. ICLR.
  • Xu et al. (2019). Modeling Tabular data using Conditional GAN. NeurIPS 32.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.