- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2(+1)
Read review
Paper summary
Concordia studies LLM based tabular prediction across institutions that cannot share data, synthetic tables, generators or LoRA updates, and that can communicate only once or twice. Each client trains a LoRA predictor on synthetic rows from a local LLM generator, learns a small LoRA scorer from changes in validation MCC to weight these rows, and uploads only the scorer. The server pools the scorers within a task and sends them back, and each client then refines its generator with GRPO. On three finance datasets, each treated as a single client, and on the Heart Disease sites, the paper reports MCC gains over the Qwen backbones after one and two rounds.
Critical strengths
S1: The paper looks at a realistic and under-studied setting, regulated cross-silo deployment where even model updates cannot leave the institution and only one or two rounds are possible. It also states clearly what can and cannot be shared.
S2: Using private validation utility to guide the synthetic data generator, instead of only making the data look realistic, is a reasonable direction, and it links sample reweighting with GRPO based refinement of the generator.
S3: The authors are open about several limitations. The finance results are single client self-improvement, there is no formal privacy guarantee, and transfer to the VA site is uneven. Appendices C, E and F give fairly complete hyperparameters, prompts and payload sizes.
Critical weaknesses
W1: The baselines do not support the main claims. (a) The local adaptation procedure for the single-node baselines is not clearly specified, despite Section 4.1 and Appendix C.7 stating that methods use the same LoRA configuration and local optimization budget. (b) Section 4.1 excludes local supervised learners, imbalance aware training and static synthesizers because they "provide no signal that can cross the export boundary". These methods do not need to cross the boundary, since every client already holds its own data. In the finance experiments there is no communication between clients at all, so local XGBoost or CatBoost with class weights, LoRA on the real local data and LoRA on static synthetic data are all allowed and are the most relevant comparisons. The next sentence then says "They are included only to test whether relaxing the communication boundary to standard update aggregation is sufficient", so it is unclear whether these methods were run at all. (c) FedLoRA is said to collapse to near zero MCC, but no numbers are reported.
W2: The evidence for the method is weak. First, no ablation supports the claim that the three levels are jointly necessary. The paper does not report synthetic LoRA without the scorer, scorer weighting without GRPO, GRPO with a local scorer compared with the pooled one, or a random or shuffled scorer. The scorer signal is also very weak, since a single change of validation MCC over 15 steps is credited to the average score of the last batch of 4 samples, and the healthcare validation sets have only 17 to 41 rows. Second, the results are statistically fragile. The healthcare test sets have 13 to 31 rows. Switzerland has a positive rate of 0.94 in its training split, so its 13 test rows probably contain only one or two negatives, and standard deviations reach about 40 MCC points with 3 seeds. The same small validation set is used to train the scorer, to define the GRPO reward and to choose the MCC threshold. Several results get worse in round 2, for example the Hungarian model on VA drops from 72.76 to 34.3 and the Switzerland model on VA from 42.01 to -9.64. An MCC of 90.22 on LendingClub is unusually high for charged off prediction and may point to leakage from features recorded after loan origination, and the feature list is not given. Finally, the privacy argument is weak. Section 2 forbids sending "parameter-efficient task-model updates such as LoRA or adapter weights", yet the uploaded scorer is itself a LoRA adapter (rank 8 on q and v, about 2.95M parameters, Appendix C.5 and F) trained on private validation feedback, and no membership or property inference test is reported. The federated part is only tested in healthcare with three small clients, and the strongest result, on Travel, does not involve federation.
W3: The writing is hard to follow and contains several inconsistencies. (a) The text uses many coined terms, such as tri-level limited-shot, scorer pooling, compliance-constrained and reduced-exposure channel, and repeats the list of artifacts that stay on premise many times. Section 3 would be clearer with a concrete example. (b) Section 4.1 says the generators start from Qwen3-4B-Instruct, but Appendix C.1 says Qwen3-4B is the backbone for both the predictor and the generator. (c) Algorithm 1 refines the generator after receiving the pooled scorers, while Appendix C.2 refines it before uploading the scorer. (d) The generator audit uses two hand picked pairs (Tables 3 and 4), and Figure 2 has no axes, no round labels and no description of the embedding method, so it cannot support the claim that refinement changes the support of the synthetic data.
Questions for authors
Q1 (see W1): Are the single node baselines zero-shot? Please add local GBDT with class weights on the real data, LoRA on the real data with the same threshold tuning, LoRA on static synthetic data, and the FedLoRA numbers. If Concordia does not beat the first two in finance, the case for training only on synthetic data needs to be reconsidered.
Q2 (see W2 ): Please report the component ablations listed in W2, including a shuffled or random scorer. Could the authors also use cross validation or larger held out splits, more seeds, and separate data for threshold selection and scorer training? Please list the LendingClub features and explain the 90.22 MCC.
Q3 (see W2 ): Can the authors report membership and property inference attacks on the uploaded scorers, compared with a task LoRA of similar size, to support treating the scorer differently from the LoRA updates that the threat model forbids?
Q4 (see W3 ): Please clarify which checkpoint the generator starts from and in which order generator refinement and scorer upload happen. I also suggest adding a small worked example in Section 3, and redrawing Figure 2 with axes, round labels and a description of the embedding method, together with a quantitative comparison of the synthetic data before and after refinement instead of two hand picked pairs.
3 comments
Thank you for the response. I accept the LendingClub clarification in W2 and the local references added in Table 3, and I have raised my Soundness score. Some points remain unresolved:
For W1(a), you state that the single-node rows are zero-shot. But Section 4.1 still says all methods use the same LoRA configuration, and Appendix C.7 still says they use the same local optimization steps. So Table 2 compares zero-shot inference with 1,500 steps of LoRA training
For W2, Travel is a single-client stream, but Table 5 lists three scorers. Which scorers refined the Travel generator? Second, you use the fixed-generator row of Table 3 to separate scorer-weighted training from generator refinement. Per Appendix H, the round-1 predictor is also trained on the initial generator. The Table 3 caption names only the scorer-head initialization as a difference. Yet the two differ substantially on Cleveland and Hungarian, by more than the round-2 gains. On Hungarian, the full method is also below the fixed generator
Some points in W1(b) and W2 are still not addressed, for example class-weighted baselines and a shuffled or random scorer control
For above reasons, I keep my overall rating
Thank you for the follow-up and for raising the Soundness score.
The question the paper addresses is whether an LLM predictor can be adapted for a client without a gradient step on a real record, when only a scorer may leave the site. Table 3 answers it under one protocol for every row: three seeds, held-out checkpoint selection, and a single evaluation on the test split. Trained on synthetic tables, the predictor is within seed variance of the same predictor trained on real records on Cleveland, Hungarian and German, and it exceeds that predictor on Travel, where real-data training collapses to the majority class. Parameter averaging collapses to a near-constant predictor on healthcare and is near chance on finance. This is the main evidence for the setting.
We will provide further evidence and clarifications on the points you raise in the final version.
Response to Reviewer hVuo
Thank you for recognizing the realism of this under-studied regulated setting, the direction of guiding generation by private validation utility rather than realism alone, and our openness about limitations. We keep that openness below.
W1, baselines. You are right that these methods need not cross the boundary and that the sentence did not say why we excluded them. Single-node rows are zero-shot (see FeeU Q1). The setting adapts an LLM without gradient updates on real records, given documented memorization under LoRA (Li et al., 2026; Bossy et al., 2026). The relevant comparison is the same LLM predictor trained on synthetic versus real records, and Table 3 reports it on all six tasks together with the tabular learners and FedLoRA. Section 2 now states the reason.
W2, ablations and signal strength. You are right that the evidence on individual components is thinner than the claim. The fixed-generator row in Table 3 isolates scorer-weighted synthetic training. Section 1 now describes the levels by role, and the claims stay with what the tables show.
W2, statistical fragility. You are right that healthcare test sets have 13–31 rows, and Switzerland has very few negatives. These sites form the public multi-site clinical benchmark, and small sites are the regime the setting targets. Section 4.1 and the caption of Table 2 now give test sizes, and Section 4.2 names the round-2 transfer regressions you point out.
W2 and Q2, LendingClub. You are right to suspect this entry:
Last_Payment_Amountis observed after origination. Appendix C.7 lists the features, Section 4.1 notes the leak, and the paper no longer draws on this result.W2 and Q3, scorer versus LoRA. You are right that the uploaded scorer is itself an adapter trained on private feedback. The claim concerns what crosses the boundary under a stated policy, and the paper gives no privacy guarantee. Section 2 now makes this explicit and requires membership and property inference testing of the scorers before deployment.
W3, writing. Thank you for these. "Limited-shot" is now "few-round" throughout, and Section 3.1 has a worked example. Generators start from Qwen3-4B-Instruct and the predictor backbone is Qwen3-4B, as Appendix C.1 now states. Appendix C.2 now follows the order of Algorithm 1: upload the scorer, pool, then refine the generator. Figure 2 (Appendix H) now labels the two rounds, and its caption describes the UMAP projection and the reward coloring.