acceptodds
Under review as a conference paper at ICLR 2027

Output-Derived Supervision for Controllable Synthetic Text

Abstract

Attribute-conditioned text generators can produce fluent text that does not express the requested attributes. We ask whether a generator's own outputs can provide useful feedback for improving attribute control after access to source records ends. We generate text from requested attribute tuples and treat the requests as noisy labels for the resulting outputs. These pairs train a lightweight body-only verifier without new human or language-model annotations. The frozen verifier then guides reinforcement fine-tuning or best-of-eight candidate selection. On Enron, reinforcement fine-tuning improves automatic attribute recovery in all six seeds, with a mean gain of percentage points, and two independent language-model judges confirm the improvement on three evaluated seeds. An Amazon transfer study shows a -point independently judged gain. The improvements are attribute-dependent and reduce distributional fidelity. In a matched comparison, reinforcement fine-tuning achieves higher judged recovery than reranking, while reranking better preserves fidelity. These results show that requested attributes paired with generated outputs provide useful but imperfect control supervision without additional source access.

Primary area:
foundation or frontier models, including LLMs
Keywords:
controllable text generation, synthetic data generation, differential privacy, reinforcement fine-tuning, learning from noisy labels, inference cost, enterprise text data, LLM-as-a-judge evaluation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 2 reviews, above 27% of the others.
Reviewer_65mrOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper asks whether a generator’s own outputs can improve attribute control after source access ends, using requested attribute tuples as noisy labels for generated bodies. It trains a lightweight body-only verifier on these request–body pairs and freezes it to guide reinforcement fine-tuning or best-of-eight reranking.

Critical strengths

The paper attempts to address the attribute control problem from an output-verifier perspective, without requiring human annotation or LLM annotation.

Critical weaknesses
  1. There is some gap between the description of the framework in Figure 1 and the method section; for example, the variable G0G_0 does not appear in the method. I suggest the authors revise the paper so that the method and the framework figure correspond more clearly.

  2. I have concerns about the implementation of the verifier. In line 169, the authors state that TF-IDF is used, but the specific implementation details are not clearly presented.

  3. I think the place where the output-verifier is most likely to work is that it uses clusters to smooth the noisy tags, together with a TF-IDF-based verifier. However, I remain concerned about the upper limit of these rather simple approaches: are they really sufficient to solve problems in general scenarios?

  4. I think the experimental validation is insufficient. The authors appear to have conducted only a very limited baseline comparison, and the results do not look particularly strong. This raises doubts about the design of the method.

Questions for authors

See Weaknesses

6 comments
AuthorsOpenReview ↗

Response to Reviewer 65mr

We thank the reviewer. We will clarify the three points raised below, and we added the requested baselines.

Q1. There is some gap between the description of the framework in Figure 1 and the method section; for example, the variable G0 does not appear in the method. I suggest the authors revise the paper so that the method and the framework figure correspond more clearly.

A1. Thank you. G₀ is the adapted generator, L4 in the primary experiment, not the frozen pretrained model. It supplies the verifier-fitting bodies, initializes RFT, and is the KL reference. We will define it at first use in Section 3 and use the same notation in the method and Figure 1.

Q2. I have concerns about the implementation of the verifier. In line 169, the authors state that TF-IDF is used, but the specific implementation details are not clearly presented.

A2. We will state the full configuration in Section 3.4. It is currently only in Appendix G.4. The verifier reads only the body (its first 600 characters). One TF-IDF vectorizer with at most 5000 unigram and bigram features is shared across attributes, followed by one class-balanced logistic-regression head per attribute (scikit-learn, L2 regularization with C = 1.0, lbfgs solver, at most 300 iterations). It is fit once on generated bodies paired with their requested tags and then frozen, and the reward is the fraction of requested attributes whose predicted value matches the request (Eq. 1). In the new label-source study, each seed uses 4000 fitting bodies and 992 diagnostic bodies.

Q3. I think the place where the output-verifier is most likely to work is that it uses clusters to smooth the noisy tags, together with a TF-IDF-based verifier. However, I remain concerned about the upper limit of these rather simple approaches: are they really sufficient to solve problems in general scenarios?

A3. Clusters only decide which attribute tuples are requested. They do not smooth or correct the verifier's labels. The information flow is:

  1. public label vocabulary → public clusters
  2. clusters and noised statistics → requested tuple y
  3. G₀ conditioned on y → body x
  4. pairs (x, y) → body-only verifier V(x)
  5. agreement between V(x) and y → reward

The verifier never receives the cluster or the requested tuple as input. We will draw this flow in Figure 1.

On the upper limit, the method is not tied to TF-IDF: any body-only classifier fit on the same pairs can supply the reward in Eq. 1. We agree that a lexical verifier has a ceiling: gains concentrate on attributes with lexical signatures, such as topic and formality, and urgency does not improve (0.385 → 0.363 in the new full-budget runs). Our judge-label ablation changes the labels but keeps the lexical verifier, so a stronger verifier remains future work.

Q4. I think the experimental validation is insufficient. The authors appear to have conducted only a very limited baseline comparison, and the results do not look particularly strong. This raises doubts about the design of the method.

A4. We added the controllable-generation baselines this point asks for: engineered and plan-then-write prompts on three models and a judge-labelled verifier (response to PF7S, Table R2), plus full-budget runs with a new selector (response to the Area Chair, Table R1). Under an independent judge, requested-tag RFT improves recovery over the adapted generator by +0.076 [+0.047, +0.102]. Strong prompting reaches higher raw adherence (0.623 for 27B plan-then-write versus 0.521), but with much worse domain fidelity, so we present the results as an adherence-fidelity trade-off rather than a ranking. We have not run matched comparisons with CTCL, DP-RFT, or guided decoding, and we will say so. DP-RFT's reward consults the private corpus at every update, which our setting rules out, and CTCL is an upstream generator design to which our control stage could be applied.

We hope these responses and the new results address your concerns.

I thank the authors for their response. However, because the authors’ rebuttal comes late, I can only provide feedback near the end of the rebuttal period. In the rebuttal, the authors do provide many detailed explanations and additional experiments, and I acknowledge that some issues are addressed. Nevertheless, I still have two concerns. First, regarding the verifier, I feel that, due to noise, I still cannot fully accept this approach. Second, regarding the experiments, as I mention in Weakness 4, Tables 1 and 4 give me the impression that only recovery improves, while the other metrics do not look good. Therefore, I retain my score. That said, I acknowledge that my understanding may have some deviations, so if other reviewers or the chairs strongly support this paper, I would remain open to it.

Thank you for your response, Reviewer 65mr. We understand and sincerely apologize for the delay. Our reply took longer than anticipated for two reasons: we received the third review late, and running the additional experiments required extra time. We look forward to any further comments you may have.

Reviewer_PF7SOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

In this work, the authors study how to improve attribute control in DP synthetic text generation tasks, without further access to the private source data. The authors train a lightweight verifier on the generator’s outputs, labeled with the attributes with which the generator was prompted. This verifier is then used in either best of N reranking or RL fine-tuning. On Enron emails and Amazon reviews, this modestly improves attribute adherence.

Critical strengths

Generating useful synthetic text from private data without further privacy cost is a practically important goal.

Critical weaknesses
  • Limited novelty. Attribute control seems to be largely solved by prompting alone: the frozen, prompted base already reaches 0.429 recovery versus 0.436 for L4 (Section 5.4). The contribution reduces to a small (+3.6 point) gain from a TF-IDF reward, which makes the novelty incremental.
  • Missing low-hanging baselines. The paper is missing simple, privacy-free baselines such as CoT or plan-then-write prompting (planning each attribute before generating), carefully engineered prompts, and a somewhat larger model prompted with the attribute requirements. None of these touches private data, and they may match or exceed the reported recovery gains.
  • Unjustified reliance on noisy labels. The verifier is trained on requested tags that the authors acknowledge are noisy. An LLM judge labeling synthetic text never touches private data and can provide stronger supervision than verifiers trained on noisy labels. The paper does not justify avoiding LLM-derived labels, or the LLM itself, as the reward.
  • Limited models and scales. Nearly all evidence comes from a limited set of models (in terms of scale). There is no study across model sizes, so it is unclear whether the gains persist, shrink, or vanish as the base model gets stronger.
  • Circular supervision. The verifier is trained solely on the generator's own outputs paired with its own requests, so no new information about how attributes are expressed enters the loop. The method can at best sharpen behaviors the generator already exhibits.
  • Weak presentation. The paper is dense and difficult to follow, and heavy hedging throughout makes the main takeaways and contributions hard to identify. A clearer statement of the central claim and a more streamlined narrative across sections would substantially improve readability.
Questions for authors

Please see weaknesses above and clarify any misunderstanding I may have had.

2 comments
AuthorsOpenReview ↗

Response to Reviewer PF7S (1/2)

We thank the reviewer. The review led us to add the prompting and judge-label comparisons below, which sharpen how we position the paper.

Q1. Limited novelty. Attribute control seems to be largely solved by prompting alone: the frozen, prompted base already reaches 0.429 recovery versus 0.436 for L4 (Section 5.4). The contribution reduces to a small (+3.6 point) gain from a TF-IDF reward, which makes the novelty incremental.

A1. We agree that prompting is strong, and our new comparison (Table R2) quantifies both its strength and its cost. With plan-then-write prompts, the frozen 4B model reaches 0.568 recovery and a frozen 27B model 0.623, both above requested-tag RFT (0.521). These prompts, however, move text away from the domain: frozen-4B plan-then-write has MMD² 0.083 and coverage 0.21, against 0.017 and 0.62 for RFT. On the adapted model itself, plan-then-write raises recovery to 0.504 but cuts coverage from 0.60 to 0.22, whereas RFT reaches 0.521 with coverage 0.62.

Our contribution is therefore an operating point, not maximal adherence. Annotation-free feedback raises recovery over the adapted generator by +0.076 [+0.047, +0.102] under an independent judge while keeping most of its domain fidelity (coverage 0.62 versus 0.60, and MMD² 0.017 versus 0.007, against 0.054 to 0.083 for prompting). The automatic +3.6 points understate the judged effect (+0.084 under Qwen3.6-27B in the submission). The novelty is the supervision source: the generator's own request-output pairs train a verifier that feeds back to the same generator after source access has ended, with no new annotations. We will state this operating point explicitly in the paper.

Q2. Missing low-hanging baselines. The paper is missing simple, privacy-free baselines such as CoT or plan-then-write prompting (planning each attribute before generating), carefully engineered prompts, and a somewhat larger model prompted with the attribute requirements. None of these touches private data, and they may match or exceed the reported recovery gains.

A2. We added them: engineered and plan-then-write prompts on the adapted 4B model, the frozen 4B model, and a frozen 27B model (Qwen3.6-27B), evaluated on the same 1000 requests per seed by an independent judge (Phi-4).

Table R2. Prompting and verifier-label comparison.

Generator and controlRecoveryMMD²CoverageFailed (%)
Adapted 4B (L4), plain0.4460.00680.5970.0
Adapted 4B, engineered0.4890.05630.2620.2
Adapted 4B, plan-then-write0.5040.07290.2225.4
Frozen 4B, plain0.4830.05350.1990.0
Frozen 4B, engineered0.5490.07960.2180.0
Frozen 4B, plan-then-write0.5680.08260.2100.3
Frozen 27B, plain0.5870.06900.2000.1
Frozen 27B, engineered0.6210.06850.2060.3
Frozen 27B, plan-then-write0.6230.08040.1962.0
Adapted 4B, RFT with requested tags0.5210.01720.6180.0
Adapted 4B, RFT with judge labels0.5460.01840.4910.0

Three seeds, the same 1000 requests per arm and seed (33,000 outputs), and a Phi-4 judge. Failed generations score zero. Recovery averages eight attributes. Temperature 0.9, top-p 0.95, 120 new tokens. MMD² and coverage use one fixed real reference and one encoder. Higher recovery and coverage, and lower MMD², are better.

Prompting raises adherence on every model, and the 27B model reaches the highest adherence, but every prompted variant has three to five times RFT's MMD² and coverage of only 0.20 to 0.26. We report this as a trade-off, not a ranking. Plan-then-write also spends 96 extra planning tokens per output, so it uses more inference compute than RFT, and our prompts were tested rather than exhaustively optimized. Prompting and RFT are complementary, since the verifier scores bodies regardless of the prompt. Testing RFT on top of plan-then-write is a natural next step.

AuthorsOpenReview ↗

Response to Reviewer PF7S (2/2)

Q3. Unjustified reliance on noisy labels. The verifier is trained on requested tags that the authors acknowledge are noisy. An LLM judge labeling synthetic text never touches private data and can provide stronger supervision than verifiers trained on noisy labels. The paper does not justify avoiding LLM-derived labels, or the LLM itself, as the reward.

A3. We ran this ablation. Holding the synthetic bodies, verifier architecture, initialization, and RFT recipe fixed, we replaced requested tags with Qwen3.6-27B labels of the same bodies (4000 fitting bodies per seed) and evaluated with Phi-4, a different model family from the labeller. Judge labels improve recovery over requested tags by +0.025 [+0.013, +0.035], at lower coverage (0.49 versus 0.62, Table R2). So fresh labels help adherence, and keeping the labeller and the evaluator separate avoids circular evaluation. We will say both. Our claim is that useful control does not require fresh labels. Requested-tag RFT already obtains three quarters of the judge-labelled gain over the adapted generator (+0.076 of +0.100), which matters when labelling generated text is costly or not permitted. We did not test an LLM as the online reward at every RFT step, which would mean judging 12,800 rollouts per seed.

Q4. Limited models and scales. Nearly all evidence comes from a limited set of models (in terms of scale). There is no study across model sizes, so it is unclear whether the gains persist, shrink, or vanish as the base model gets stronger.

A4. We agree that scale remains open. RFT improves recovery over its initialization in both model families we tested, Qwen3.5-4B and SmolLM3-3B, though SmolLM3 is a short-budget screen with automatic evaluation only. The new 27B model is a prompted baseline (Q2), not RFT at 27B, so a controlled scaling study remains future work, and we will list it as such.

Q5. Circular supervision. The verifier is trained solely on the generator's own outputs paired with its own requests, so no new information about how attributes are expressed enters the loop. The method can at best sharpen behaviors the generator already exhibits.

A5. This is correct, and by design: the control stage cannot add information about the source data. Our claim is that this sharpening is useful. The generator already expresses the request in some samples. The verifier, fit on thousands of request-body pairs, rewards the realizations that accompany each request, and RFT shifts probability toward them. Independent judges confirm the effect, and a new within-request test shows that the verifier's pick beats a random candidate by +0.023 [+0.006, +0.039] (response to the Area Chair, Q8). Fresh labels add more (Q3), but sharpening alone yields three quarters of the judged gain. The sharpening view of Huang et al. (ICLR 2025) is relevant context.

Q6. Weak presentation. The paper is dense and difficult to follow, and heavy hedging throughout makes the main takeaways and contributions hard to identify. A clearer statement of the central claim and a more streamlined narrative across sections would substantially improve readability.

A6. We will improve the presentation. We will open with a one-paragraph statement of the claim, as in the general response, collect the privacy and provenance qualifications in one scope block instead of repeating them across sections, and place the control and fidelity comparisons, including the unfavorable prompting results, together.

Q7. Please see weaknesses above and clarify any misunderstanding I may have had.

A7. The main clarification is Q1: prompting can reach higher adherence than RFT, but only with a large loss in domain fidelity, and our claim is about that operating point rather than maximal adherence. The other concerns are addressed by new experiments (Q2 and Q3), the mechanism (Q5), and the presentation changes (Q6). We hope these responses address your concerns.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.