acceptodds
Under review as a conference paper at ICLR 2027

Learning from Late Rewards: Asynchronous Data Synthesis with Realized Learning Utility

Abstract

Synthetic data is increasingly central to improving large language models, yet its value is commonly estimated from signals available before learning takes place. We argue that the true value of synthetic data is instead learner-relative and outcome-defined: it should be measured by the downstream improvement that training on the data actually induces. We call this quantity Realized Learning Utility. Grounding synthesis in realized utility, however, fundamentally turns data synthesis into a late-reward optimization problem: synchronous optimization preserves faithful feedback but blocks synthesis, whereas naive asynchronous optimization exposes the entire synthesis update to stale feedback from an evolving learner. We introduce LateSyn, a Predict-Early, Correct-Late framework that estimates future realized utility at generation time, applies a reliability-adaptive early update, and later corrects the original synthesis trajectory using the residual revealed by the actual learner outcome. A learning-utility memory further warm-starts prediction from previously resolved outcomes. Theoretically, LateSyn preserves the realized-utility target while reducing the synthesis signal exposed to delay from the full advantage to only its unresolved residual. Experiments across diverse agentic environments show that LateSyn produces substantially healthier learner trajectories in regimes where delayed feedback becomes harmful.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Data Synthesis, Multi-Agent LLMs, Reinforcement Learning

Reviews

Among:
Shape:
1234this paper, Bottom 25%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.75 over 4 reviews, above 25% of the others.
Reviewer_pfb2OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes LateSyn and studies how to optimize a synthetic data generator using actual training improvement from the data as the reward. The authors note that this reward comes only after learner training and evaluation, and becomes outdated under asynchronous execution. LateSyn first uses a critic to predict future data utility and update the generator immediately, then applies residual correction after true results return. The authors theoretically show that delay affects only the residual, and report better results than the baselines on three agent environments: Search-QA, ALFWorld, and WebShop.

Critical strengths

The problem the paper focuses on has practical relevance, in that the reward becomes stale under asynchronous execution when the learner's actual training improvement is used as the reward, and the proposed predict-then-correct method is also simple and easy to integrate into existing workflows.

Critical weaknesses

The paper's core empirical claim lacks supporting evidence. The authors attribute the gains to reducing the signal exposed to delay, but on Search-QA the delayed residual ratio is about 0.99, indicating that the mechanism has almost no effect, yet the improvement there is the largest. In Figure 4, λ is zero before step 10, so LateSyn and the λ=0 control should follow identical code paths and their curves should coincide, but the curves have already separated from step 2, and the gap cannot be attributed to the method. Synchronous LU that directly uses the true learning outcome as the reward performs worse than the proxy baseline on two of the three environments, and the paper's starting point is also not supported by its own data. In addition, the experiments use only one 4B model and two seeds, so the main conclusions are difficult to establish.

Questions for authors
  1. What is the essential difference in problem setting or method between this paper and learning-to-teach, teacher–student curriculum, and data valuation, where rewarding the data generator with the learner's actual progress has already been studied?
  2. Does the paper's premise that actual learning performance is more suitable as a reward than proxy signals hold, even though synchronous LU is worse than the proxy baseline on two environments and the correlation between LU and held-out improvement is only about 0.25?
  3. Why is such a large degree of asynchrony needed in practical settings, given that one-step delay performs better than the synchronous case in Figure 3, the high-delay problem appears only at M=8, and the paper reports no benefit in time or throughput?
Reviewer_CTnJOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies the research question "what makes a synthetic example valuable". The authors argue that the value of synthetic data is learner-relative and outcome-defined, and hence should be measured by the improvement that actually results from training on the data (which the authors call Realized Learning Utility). Using this quantity as a reward for the synthesizer turns data synthesis into a delayed problem: a synchronous loop gives feedback but blocks generation, while a naive asynchronous loop assigns stale rewards to an outdated synthesizer policy and an evolved learner. The authors propose LateSyn, an asynchronous data synthesis framework. At generation time, a critic estimates the future realized learning utility of each synthesized batch.

The realized learning utility (LU) is the difference in performance on a validation utility probe before and after one learner update on the new data. However, in practice, LU is unavailable when the synthesis decision is made. Therefore, the authors use a lightweight critic to estimate it. The realized credit is then decomposed into two components: predict early and correct late. In the "predict early" step, a fraction of the predicted utility, controlled by a trust coefficient lambda estimated online from past prediction outcome pairs, is immediately applied to the synthesizer policy that generated the batch. Once the data is consumed and the real LU is available, LateSyn applies a "correct late" update with the correction. Altogether, this lets generation proceed asynchronously while the gradient error scales with the residual rather than the full advantage.

The experiments use Qwen3-4B as both the synthesizer and the learner on three agentic domains: Search-QA, ALFWorld, and WebShop. The baselines include the base learner, static synthesis, synchronous LU optimization, and naive asynchronous LU optimization. LateSyn outperforms the baselines. However, on Search-QA, both synchronous and naive asynchronous LU optimization score below the base learner. Similarly, LU is only weakly correlated with held-out improvement, which suggests that the claim that LU is a good training signal is not fully supported. I would also recommend that the authors experiment with another model family (e.g., Llama or Gemma) and replicate the experiments. Moreover, I do not see a comparison with methods similar to those mentioned in L1030-1035. Finally, one of the main arguments for LateSyn is its asynchrony and the throughput it enables. However, I do not see any comparison along that dimension.

Overall, the paper tackles an interesting problem, and the Predict-Early, Correct-Late decomposition is a simple way to optimize a data synthesizer against delayed, outcome-defined feedback. However, I lean toward rejection because the empirical evidence does not yet support the main claims: realized LU is not convincingly shown to be a reliable training signal, the evaluation is limited to a single model family without comparisons to similar learner-aware synthesis methods, and the throughput benefit that motivates asynchrony is never measured.

Critical strengths
  • Well-motivated problem formulation
  • Simple and principled mechanism
Critical weaknesses
  • Limited support for realized LU as a training signal

  • Insufficient empirical evaluation

Questions for authors

See summary

Reviewer_SfHQOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper identifies the aysnc shift between synthetic data evaluation and learner states inherent in conventional proxy-based methods. To address the resulting feedback delay, it proposes LateSyn, an anticipatory prediction and delayed correction framework for data synthesis, evaluation, and training. Specifically, it introduces Realized Learning Utility to quantify the value of a data batch and performs delayed compensation based on actual training gains. Extensive experiments demonstrate the substantial performance improvements achieved by the proposed method.

Critical strengths
  1. The paper targets the issue of feedback latency between data synthesis and downstream learner training and conducts corresponding experiments to show the delays indeedly induce performance degradation. It introduces Realized Learning Utility to measure the data value.
  2. The design of the synthesizer and utility critic is well-conceived, which is iteratively refined based on feedback and updated synchronously. The overall framework is complemented by detailed theoretical analysis in the paper.
  3. LateSyn achieves consistent and notable performance over controlled synthesis baselines across three distinct agentic benchmarks over controlled synthesis baselines. The multi-dimensional experiments further validate its async latentcy resistance and transferiability.
Critical weaknesses
  1. The core claim is tackled by the black-box updates of the data synthesizer. While the synthesizer and critic are expected to capture the update dhynamics of the learner, experiments have not yet fully validated the reliability under changing training phases and distribution shifts in addtion to demonstration for the their capabilities in mitigating latency and reflecting progress. Provide additional empirical evidence concerning these update dynamics would make the claim more well-grounded.
  2. The current results show the effectiveness of LateSyn against naive asynchronous baselines, but do not yet compare it with baselines including utility prediction such as CoEvolve.
  3. The framework involves memory, adaptive λ\lambda and delayed residual correction, providing an ablation study to isolate the contribution of each individual component would offer a more complete analysis.
Questions for authors
  1. Existing learner-aware proxy evaluators and LateSyn both use information available at generation time, while LateSyn use LU as supervise signals, which also eill induce latency. Could clarify which aspects of learner-state staleness are addressed, beyond the effects of synthesizer-policy staleness?
  2. Could provide cost analysis for the co-training framework?
  3. The introduction intriguingly mentions that the value of synthetic data changes as the learner's state evolves. Given the relatively simple Ridge Regression predictor employed, to what extent does the current generation context truly capture the complex interaction between data and learner state?
Reviewer_17k2OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates asynchronous synthetic data generation, where the utility of generated data can only be observed after the downstream learner has been updated. The authors introduce Realized Learning Utility, which measures the value of synthetic data through the change in downstream learner performance, and propose LateSyn, a Predict-Early, Correct-Late framework for handling delayed learning feedback. The proposed method employs a utility critic to predict future learning utility and applies a reliability-adaptive early update to the synthesizer. Once the actual learning utility becomes available, a delayed residual correction is performed through off-policy optimization. A learning-utility memory is also introduced to improve prediction. The theoretical analysis shows that, under certain assumptions, the proposed decomposition reduces the portion of the synthesis update exposed to delayed policy drift. Experiments on Search-QA, ALFWorld, and WebShop demonstrate that LateSyn achieves better downstream performance than the evaluated baselines, particularly under substantial feedback delays.

Critical strengths
  1. A well-motivated and practically relevant problem. The paper examines an important challenge in synthetic data generation: the actual utility of generated data can only be assessed after the downstream learner has been updated. By formulating this as a delayed-feedback optimization problem and distinguishing between synthesizer-policy staleness and learner-state staleness, the authors provide a useful perspective on the interaction between data synthesis and learning. I believe this problem is relevant to the development of adaptive synthetic data generation methods.

  2. A practical framework with promising empirical results. The proposed Predict-Early, Correct-Late approach offers an intuitive way to handle delayed learning feedback while retaining the realized learning outcome for subsequent correction. Experiments show improvements over the evaluated baselines across Search-QA, ALFWorld, and WebShop, particularly under substantial feedback delays. These results suggest that the proposed method has practical potential, although the underlying mechanisms and statistical reliability require further clarification.

Critical weaknesses
  1. The statistics for the experimental results were not well explained or supported. Table 1 reports a Search-QA F1 score of 45.7 for LateSyn, while the two runs shown in Figure 4 achieve final scores of 37.7 and 43.6. The relationship between these results is not yet clear, especially regarding the choice of seeds, the choice of checkpoints, and the aggregation method. In addition, the main comparisons do not include uncertainty estimates, and the evidence for improved training stability relies heavily on two paired seeds. I would like to see more trials using multiple seeds, with clear reporting of means and standard deviations, and comparisons across different feedback delays. These results are crucial for validating the robustness and reproducibility of the reported improvements, which constitute a central contribution of this paper.

  2. The theoretical analysis is limited, and the experimental evidence for the proposed mechanism is insufficient. Theorem 1 provides a per-decision error decomposition and a local bound on delayed policy drift, but the practical algorithm, which uses PPO clipping, is not fully covered by the theoretical analysis. Additionally, the experiments report a reduction in the scalar advantage residual, rather than directly measuring the actual gradient mismatch. For example, LateSyn improves Search-QA performance by 16.3 F1 points over Naive Async-LU, while reducing the RMS of the delayed advantage by only about 0.6%. While these results are not necessarily contradictory, the underlying mechanism still lacks strong validation. I would like to see more experiments measuring actual gradient mismatch, policy drift, and the frequency and impact of PPO clipping. An ablation study comparing adaptive and fixed nonzero reliability coefficients, with and without Learning-Utility Memory, and with and without delayed residual correction, would also help explain the contributions of each component. These experiments are important to determine whether the observed performance gains can be linked to the proposed mechanism, and whether the theoretical analysis explains the behavior of the practical algorithm.

Questions for authors
  1. Interpretation and validity of the main test results. Table 1 shows a Search-QA F1 score of 45.7 for LateSyn, while Figure 4 shows the final results of 37.7 and 43.6 for the two seeds. Can the authors explain the relationship between these results, especially with respect to seed selection, checkpoint selection, and aggregation methods? Can the authors also provide the average performance and standard deviations for additional independent seeds, as well as comparisons across different feedback delays?

  2. Evaluation of the proposed mechanism. LateSyn improves Search-QA performance by 16.3 F1 points over Naive Async-LU, but the reported reduction in the RMS of the delayed advantage is only about 0.6%. Can the authors explain this difference and provide appropriate measures of actual gradient mismatch and policy drift? These results would help determine whether the proposed reduction in delayed-policy mismatch contributes to the observed improvements.

  3. Connection between theory and practical implementation. Theorem 1 analyzes updates where the importance weights are not clipped, but the practical algorithm uses PPO clipping. How often is clipping activated during training, and how does it affect the actual gradient updates? Can the authors provide experimental evidence showing that the proposed reduction in delayed-policy mismatch remains effective when clipping is active?

  4. Contribution of each component. Can the authors provide additional ablation experiments comparing adaptive reliability coefficients with fixed nonzero coefficients, with and without Learning-Utility Memory, and with and without delayed residual correction? Are the optimizer update counts matched across these variants? These comparisons would help explain the contributions of each component to the reported performance improvements.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.