- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The authors propose to explicitly learn continuous SFT data by aligning the gradients that it induces with those from human-readable SFT data. They apply this approach to a variety of pretrained models and SFT datasets, and find that finetuning on their continuous SFT data matches or performs very slightly better than finetuning on the original human-readable SFT data.
Critical strengths
- The setting of learning continuous representations and training on them is generally interesting.
- The empirical evaluation is fairly comprehensive in terms of the number of models and SFT datasets it considers, although the gains are small.
Critical weaknesses
- The novelty is limited. The claim that continuous inputs can effectively finetune an LLM is well-established in prior dataset distillation works, such as the cited GRADMM paper. So from my perspective, this work provides novelty if it gives an approach which we expect could actually perform better than directly training on the human-readable SFT dataset. The authors do report very mild gains over training on human-readable SFT data. However, this result simply doesn't make sense given the objective: the update direction that minimizes their objective is simply the gradient on the original text. So if the synthetic examples successfully optimize their objective, they should learn to imitate the direction that the original dataset already provides directly. Therefore I have a hard time understanding or believing that this approach works any better than direct SFT on human-readable text.
- Clarity is poor due to heavy use of AI writing. The entire paper reads as heavily AI-written, in a manner that hurts readability. As an example, there are several sentences that are clear hallmarks of AI writing, but human authors would likely never produce, e.g., L107: "the residual expansion analysis motivates this objective but does not guarantee the outcome of finite-step fine-tuning." Which expansion analysis (it has not been mentioned up to this point)? And when would we ever expect theory to guarantee the outcome of multiple steps of finetuning? Similar at L112, where preliminaries/notation that are needed to understand the main equations in the main body are deferred to the Appendix: "The notation, basic setup, and standing assumptions used in this section are given in Appendix A". Another example at L125: "This is an existence result for the expanded model class; it does not assert that a particular dataset or optimizer will find the improving direction." And so on. The whole paper needs significant rewriting to clean things like this up. I see these phrases as going far beyond "polishing" the writing of the human authors, as they disclosed---rather it seems like GPT-6 generated the majority of the words in the paper.
- There is no motivation provided for why we would intrinsically care about training on representations that are not human-readable. In fact, this seems to be throwing away a highly desirable property for essentially no gain.
Questions for authors
The paper is heavily AI-written and hard to read in its current form. Could the authors take a few passes and remove sentences like the above, and smooth out the writing significantly? This may be too significant of an undertaking for a rebuttal period.
3 comments
Rebuttal by Authors
Weakness 2
English is not our native language, and we used GPT-6 to assist with the English translation and language polishing of parts of the manuscript. We take responsibility for all content and wording in the paper and take the reading difficulties you identified seriously. In response, we have revised the wording throughout the manuscript to help readers understand the method and its theoretical basis.
Weakness 3
We recognize the value of natural text for human inspection and data management. Our paper investigates whether human readability is also necessary for effective adaptation of pretrained models. Therefore, even when synthetic inputs achieve performance comparable to real text, examining whether training utility can be retained after changing the input form has research value.
First, understanding what pretrained large language models use during subsequent learning. Natural text contains both human-readable forms of expression and training signals that models can use. We investigate whether these learning signals can still support effective fine-tuning when the inputs no longer retain fluent natural-language forms. We analyze this question through the parameter updates induced by the inputs and their effects on the reference loss. This helps us understand the relationship between training utility and the surface form of language and explore what forms of information pretrained models can use to continue learning.
Second, exploring fine-tuning approaches for settings involving confidential data. When original texts cannot be provided directly to the training party, one possible approach is for the data holder to optimize synthetic training representations using reference data and then provide these representations for subsequent fine-tuning, so that the training process does not require directly sharing the original reference texts. Our paper provides a method for transforming reference learning signals into continuous training inputs, which can serve as a basis for this direction. Practical applications would also require dedicated privacy protection mechanisms and leakage evaluations.
Third, exploring the utility of expanding small datasets. When real samples are limited, the learning signals provided by a small reference dataset can be used to optimize continuous inputs from different initializations and construct additional training instances. Existing data can therefore be used both for direct fine-tuning and to guide the generation of other training representations. Our method provides a concrete optimization criterion for this purpose: adjusting inputs according to the effects of the updates they induce, and using downstream tasks to assess whether the expanded data can improve adaptation. For experimental evidence on data expansion, please see our response to Reviewer pDgy’s Q1. We report five-run GSM8K experiments that expand 5, 10, 20, or 50 real examples into 100 continuous training inputs, without additional reference data. Under the same number of downstream LoRA steps, DASA achieves mean accuracy gains of 0.11–0.22 percentage points over repeatedly training on the original examples.
Thanks to the authors for their response. I appreciate the effort put into the new theory and experiments, but I remain unconvinced about the paper's overall contribution.
First, while the new theory establishes that optimizing continuous inputs can improve the one-step reference loss relative to the initialization texts, this is a pretty limited guarantee. I don't see this as a meaningful explanation for why the method could result in notably better performance than directly finetuning on natural language SFT data.
Second, I am still unconvinced by the motivations for converting natural language SFT data into continuous representations. The applications mentioned, like privacy-preserving SFT or data augmentation are interesting, but they are speculative, and so I would expect the authors to demonstrate these applications more completely given the notable con of losing interpretability when moving to continuous representations.
Lastly, though the empirical improvements over standard SFT are fairly consistent, they're quite small in magnitude. Given the existing literature demonstrating that continuous synthetic training data works, I don't think these gains are a notable empirical contribution.
Altogether, I appreciate the authors' efforts to address my concerns, but I don't feel that the rebuttal has changed my opinion of the paper, and therefore will maintain my score.
Rebuttal by Authors
Weakness 1
We study continuous synthetic inputs for adaptation on knowledge, mathematical reasoning, code generation, and commonsense tasks across two model families and six models up to 32B parameters. In the evaluated settings, DASA improves downstream performance over the GRADMM implementation used in our paper and achieves a 3.6–4.9× synthesis speedup under matched settings, with similar peak memory usage.
Another contribution of our paper is to show that when the activation changes induced by synthetic inputs align with the loss descent direction indicated by the reference activation gradients, even continuous inputs that are difficult for humans to interpret can achieve downstream performance comparable to that of real-text fine-tuning on the evaluated tasks.
DASA optimizes inputs using the first-order effect of their induced LoRA updates on the reference loss. We compute a candidate update from the current synthetic inputs, evaluate it using reference activation gradients, and backpropagate this evaluation to the inputs. The resulting continuous inputs are then used for fine-tuning. Our contributions are this input-optimization method, its analysis under LoRA, and the systematic evaluation of its adaptation utility.
Appendix C provides a mean–covariance decomposition of the first-order risk change. Theorem 1 bounds the covariance term. Theorem 2 establishes when continuous input optimization yields a lower reference loss than training on the initialization texts after one update, with the same input count and length, LoRA initialization, and parameter step size.
Conditions and mathematical explanation. Let J(Z) be the input objective and Z0 the initialization embeddings. Theorem 2 assumes:
For a sufficiently small input step ρ>0, setting Zs=Z0−ρ∇ZJ(Z0) gives:
ΔJ:=J(Z0)−J(Zs)=ρ∥∇ZJ(Z0)∥22+o(ρ)>0.
Thus, the synthetic inputs induce a smaller directional derivative of the reference loss. Let vs=v(Zs) and vr=v(Z0). Under A3, when both update paths remain in U:
Lref(fηvs)−Lref(fηvr)≤−ηΔJ+2βη2(∥vs∥22+∥vr∥22).
Consequently, the synthetic inputs yield a strictly lower reference loss whenever:
0<η<ηmax:=β(∥vs∥22+∥vr∥22)2ΔJ.
The reason is explicit: input optimization creates a first-order advantage ηΔJ that exceeds the second-order remainder bound. The initialization texts need not induce the best update for the reference objective; optimizing their continuous representations can improve that update and lower the actual loss after one adaptation step.
Experimental verification. We conducted five independent runs on Llama-3.2-1B using FP64. From the same LoRA initialization, we separately applied the updates induced by the initialization texts and synthetic inputs, then recomputed the reference losses through forward passes. Let G=Lr−Ls be the measured loss advantage. With a guaranteed local smoothness upper bound β, Theorem 2 gives:
ηΔJG≥1−ηmaxη.
Table R1. Theoretical lower bounds and measured normalized loss advantages (mean ± SD).
Table R2. Objective improvement, smoothness bound, and actual loss advantage.
At η/ηmax=0.5 and 0.9, which satisfy the strict improvement condition, the measured normalized advantages exceed the theoretical lower bounds and are close to the first-order prediction of 1. At η=2×10−4, the predicted mean advantage is 1.650×10−7, close to the measured 1.649×10−7. These results connect the input-objective improvement to the observed reference-loss advantage after actual updates.