acceptodds
Under review as a conference paper at ICLR 2027

TUCO: Curating Simulation Demonstrations for Sim-to-Real Robot Policy Co-Training

Abstract

Simulation demonstrations can supplement scarce real-world data for robot policy co-training. However, the value of using data curation to actively select these demonstrations for sim-to-real co-training remains underexplored. Existing curation methods also lack a unified criterion for measuring trajectory-level utility and set-level coverage from closed-loop target behavior. To address these gaps, we present the first systematic study of data curation for sim-to-real robot policy co-training and propose **T**rajectory-level **U**tility and set-level **C**overage **O**ptimization (**TUCO**). TUCO uses influence functions to trace how each source demonstration affects target-domain scoring rollouts. Our key insight is that these effects can be decomposed into an overall contribution to target return and variation across rollouts, providing a common closed-loop basis for measuring trajectory utility and set coverage. We further propose a performance-aligned subset optimizer that combines these measures in a unified curation objective to reduce redundancy and select complementary demonstrations. Extensive experiments on RoboMimic and OmniReset establish the value of active simulation data curation for sim-to-real policy co-training and show that TUCO achieves state-of-the-art performance across single-simulator, sim-to-sim, and sim-to-real settings.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Data Curation, Sim-to-Real, Robot Learning, Influence Function

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_QAkwOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper applies influence function approach to data curation in a multiple-domain setting: choosing source-domain data (e.g., simulation data) to complement target-domain dataset (e.g., real robot) for sim-and-real co-training.

The key finding is that existing influence function approach maximizes a single scalar objective associated with the overall improvement on the target-domain utility, which loses information about “how” each source-domain data point affects the utility of different target domain data points (influence pattern). The paper proposes a new utility function for data curation that reflects both overall improvement and the influence pattern. Maximizing the new utility function helps with selecting source-domain data with broader coverage of the influence pattern.

Critical strengths
  1. The work introduces a useful insight for data curation that the coverage of the selected dataset can influence the final co-training performance sometimes significantly. This coverage can be omitted by existing approach.
Critical weaknesses
  1. Presentation: The clarity of the presentation can be significantly improved in general, which currently makes it hard to give review feedback on the actual contributions made in this work.
    1. For example, starting from the second paragraph of the Introduction, the presentation becomes jargon-heavy with a lot of abstract concepts of existing and proposed methods without a clearly streamlined core message and clear explanation of what the jargons are. I recommend to move large part of the detailed explanation to Related Work and extract a few most important, focused messages and intuition to be presented in the Introduction. This also avoids a lot of repetition between the two sections.
    2. Mathematical clarity: There are also other relevant small things. For example, at line 162, the behavior cloning loss is not properly defined; at the next line, the return is not defined either. I encourage the authors to take greater care when writing mathematical expressions and take a pass to fix such places.
    3. Contribution relative to existing work: the paper makes it really hard to distill the actual contribution made in this work relative to existing work. For example: section 4.2 is in its essence taken from CUPID with the main novelty being applying the influence profile approach in CUPID to the multi-domain setting. However, everything is presented as “Approach” in this work.
  2. It is slightly confusing to call “source” and “target” domains in the single-simulator setting in the context of sim-and-real co-training. It would be good to clarify this setting as a “sanity check”.
  3. Figure 3: in large part of the settings, the error bars in this figure are heavily overlapping between methods, which suggest the number of random seeds being insufficient to support statistical significance. Hence, the claimed improvement by the proposed approach is not supported here.
  4. The implicit function theorem used to construct the influence function requires the Hessian matrix to be invertible. It is not tested and discussed anywhere whether and when this assumption is satisfied in practice.
Questions for authors

N/A

6 comments
AuthorsOpenReview ↗

Response to Reviewer QAkw - Part 1

We thank the reviewer for your careful reading and insightful feedback. Below, we clarify the mathematical definitions and distinguish our methodological contributions from the attribution foundation provided by prior work.

W1: Clarity and organization of the presentation

R1: As you suggested, we have meticulously rewritten the abstract, Introduction, Preliminary, and Methodology to make the core idea and contributions easier to follow. Thank you for identifying how the abstract terminology obscured our message.

  1. Clearer concepts and motivation in the Introduction. We now explain trajectory-level utility as the expected contribution of an individual demonstration to target-policy performance, and set-level coverage as the diversity of the selected dataset. We summarize existing approaches through the demonstrations they seek to select and explain why individual usefulness and collective diversity should be considered together. We have reduced technical detail in this discussion to keep the Introduction focused on the motivation and key idea.
  2. A more direct explanation of our contributions. We distinguish our methodological contribution, combining utility and coverage through influence decomposition and subset optimization, from our study of simulation data curation for sim-to-real co-training.
  3. Intuition alongside the mathematics. We have simplified the Preliminary and problem formulation, separated the main equations from the prose, and explained the quantities and their roles. Sections 4.2 and 4.3 now explain how influence attribution contains both individual learning value and information about relationships among demonstrations. Section 4.4 motivates how the selection criterion uses this information to reduce redundancy and promote complementarity.

W2: Incomplete definitions of the behavior-cloning loss and return

R2: We have clarified both definitions in the revised paper. Thank you for highlighting these omissions. The imitation loss measures how well the policy reproduces demonstrated actions, while the return measures whether a complete target-domain execution succeeds.

Behavior-cloning loss. We now define ℓ(ot,at∣θ)\ell(o_t,a_t\mid\theta) as the supervised imitation loss for predicting the demonstrated action ata_t from observation oto_t. For a demonstration ξ=(ot,at)t=1Tξ\xi=\\{(o_t,a_t)\\}_{t=1}^{T_\xi} of length TξT_\xi, the trajectory loss used in influence attribution is

=\sum_{t=1}^{T_\xi}\ell(o_t,a_t\mid\theta).$$ This definition explicitly connects the demonstration-level influence calculation to the policy's imitation training loss. **Target-domain return.** We now explicitly define the return of a target-domain rollout $\tau$ as $$R_{\mathrm t}(\tau):= \begin{cases} +1, & \text{if the target task is completed successfully},\\\\ -1, & \text{otherwise}. \end{cases}$$ The curation objective uses its expectation, $J_{\mathrm t}(\theta)=\mathbb E_{\tau\sim p_{\mathrm t}(\cdot\mid\pi_\theta)}[R_{\mathrm t}(\tau)]$, where $p_{\mathrm t}$ is the target rollout distribution. Under this definition, expected return equals twice the task success probability minus one, so maximizing it is equivalent to maximizing task success probability.
AuthorsOpenReview ↗

Response to Reviewer QAkw - Part 2

W3: Methodological contributions relative to CUPID

R3: Thank you for highlighting the need to distinguish this contribution more clearly from CUPID.

First, we provide a new mathematical perspective that reveals diversity information left unused by CUPID's ranking rule. In Section 4.2, we express each demonstration's influence across target-policy executions as a profile. Our analysis shows that these profiles describe both individual learning value and differences in how demonstrations support target behavior. CUPID selects demonstrations using an aggregate utility score. These individual scores do not capture the relationships needed to assess redundancy and coverage within a selected dataset.

Second, we propose an influence-function-based decomposition to extract utility and coverage information. In Section 4.3, we separate each profile into its component along the return vector and an orthogonal component. The former determines trajectory-level utility, while the latter retains differences across target executions and provides a representation for measuring set-level coverage. This decomposition allows us to assess individual usefulness and collective diversity through the same closed-loop target behavior.

Third, we propose a performance-aligned subset optimizer that uses both measures during selection. In Section 4.4, we restrict coverage-driven selection to candidates with positive utility and sufficient alignment with target return. We then balance individual utility with the additional coverage each candidate provides. We update coverage gains as the selected set grows, so selection accounts for demonstrations already retained. This favors useful and complementary demonstrations while reducing redundancy.

Our experimental contribution is a systematic evaluation across single-simulator, sim-to-sim, and sim-to-real settings. To our knowledge, we provide the first systematic study establishing the value of simulation data curation for sim-to-real policy co-training. This contribution concerns both the new evaluation setting and the evidence that active curation can improve real-world performance.

W4: Source and target terminology in the single-simulator setting

R4: The single-simulator experiment serves as a sanity check of curation without domain shift. Thank you for this helpful interpretation and for highlighting the need to clarify our terminology.

We use “source” for the domain supplying candidate demonstrations and “target” for the domain in which the policy is evaluated. In the single-simulator setting, these domains share the same environment. The terms distinguish their roles in curation without implying a domain gap. We retain them to provide a consistent formulation across single-simulator, sim-to-sim, and sim-to-real settings.

Following your suggestion, we have explicitly stated in Section 4.1 that the single-simulator and sim-to-sim settings serve as sanity tests for our sim-to-real study. They evaluate curation without domain shift and under a controlled simulator shift, respectively.

Revisions: Section 4.1, clarification of the three source-to-target settings and the role of single-simulator and sim-to-sim experiments as sanity tests.

AuthorsOpenReview ↗

Response to Reviewer QAkw - Part 3

W5: Overlapping error bars and evidence across training seeds R5: Following your insightful advice, we have conducted five additional random seeds for the sim-to-sim experiments and combined them with the original three, yielding eight seeds per configuration. Thank you for encouraging us to strengthen the evaluation.

The expanded results show that TUCO maintains its strong performance across tasks and selection budgets while exhibiting the lowest aerage cross-seed standard deviation among the compared curation methods. Error-bar overlap is reduced in the updated comparison. These results provide stronger evidence for TUCO's overall performance and consistency across training runs.

The remaining variability is consistent with a recognized challenge in robot data curation: the scores used to select training data can themselves depend on stochastic learning and noisy influence estimation. DemInf reports substantial cross-seed instability for parametric mutual-information estimators (Hejna et al., 2025). CUPID discusses ensembling attribution scores across independently trained models to account for learning non-determinism (Agia et al., 2025), while DataMIL uses clustering to reduce noise in sample-level influence estimates (Dass et al., 2026).

During this rebuttal, we have updated Figure 3 and the corresponding experimental analysis with the eight-seed results and clarified the statistical convention in the caption.

Revisions: Figure 3, eight-seed results and caption; Section 5.1, updated Sim-to-Sim Co-Training analysis.

W6: Hessian invertibility and practical influence estimation

R6: Thank you for raising this point. Our implementation follows the practical influence estimator used by CUPID, and this approximation is already specified in Appendix B.2, Equations (11)–(12). The nonsingular-Hessian assumption in Appendix B.1 supports the exact influence-function derivation. The implementation uses a projected-gradient curvature matrix for efficient estimation.

Specifically, we follow CUPID (Agia et al., 2025) in using the TRAK-based approximation (Park et al., 2023). As defined in Appendix B.2, the matrix used for scoring is

=\widetilde{\mathbf G}^{\top}\widetilde{\mathbf G}+\eta\mathbf I_d,$$ where the rows of $\widetilde{\mathbf G}$ are projected source gradients. This is the same matrix construction and inversion used in CUPID's released TRAK implementation (`get_x_xtx_inv`). Our zero-damping setting also matches CUPID's provided experiment script (`lambda_reg=0.0`). In this setting, invertibility requires the projected gradient matrix to have full column rank. Positive damping makes the matrix positive definite. Thus, the relevant practical condition concerns the projected Gram matrix specified in Appendix B.2, rather than the full neural-network Hessian. **TUCO builds its utility–coverage decomposition and subset selection on these established influence estimates.** Appendix B.3 further states that the decomposition identities hold for any fixed influence profiles and a nonzero return vector. Our implementation therefore follows the estimation procedure used in prior work, while our contribution concerns how these estimates guide set-level curation. **References:** - Hejna, J., et al. (2025). [Robot Data Curation with Mutual Information Estimators](https://arxiv.org/abs/2502.08623). *Robotics: Science and Systems (RSS).* - Agia, C., et al. (2025). [CUPID: Curating Data your Robot Loves with Influence Functions](https://arxiv.org/abs/2506.19121). *Conference on Robot Learning (CoRL).* - Dass, S., et al. (2026). [DataMIL: Selecting Data for Robot Imitation Learning with Datamodels](https://proceedings.iclr.cc/paper_files/paper/2026/hash/033d9e8dbbbc0ad90e59222cf1db0fc2-Abstract-Conference.html). *International Conference on Learning Representations (ICLR).*
Reviewer_heH2OpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies how to select useful simulation demonstrations for training robot policies with limited real-world data. The authors propose TUCO, which uses influence functions to estimate how each simulation demonstration affects the policy in the target environment. The method considers both how useful each demonstration is and whether it adds new information to the selected data. Experiments on RoboMimic and OmniReset cover single-simulator, sim-to-sim, and sim-to-real settings.

Critical strengths

The real CupCake result stands out: 100% success, versus 20% with the full simulation pool and 12.5% with random selection. TUCO and random selection both use 600 simulation demonstrations and the same ten real rollouts, with the sampling ratio held fixed. This gives concrete evidence that choosing the simulation data matters beyond choosing how much to use.

I also like that coverage is measured through effects on target rollouts, rather than differences in the source data that may have little relevance to the target task.

Critical weaknesses
  • CUPID already accounts for much of the improvement over full-pool training. TUCO adds 2.1 pp on average in sim-to-sim; on real Peg, the difference is 3 additional successes out of 40 trials. CupCake shows a larger gain over CUPID, from 80% to 100%, but the reported significance tests compare TUCO only with full-pool training and random selection. The evidence for data selection in general is stronger than the evidence for TUCO’s particular selection rule.

  • The 10 successful real rollouts define both utility and coverage, yet the paper does not report repeating selection with a different set. I am uncertain how much the results depend on these particular examples, especially if they cover only a narrow range of successful behaviors.

Questions for authors

Please see the negatives above. In addition, the coverage term treats a residual vector and its negative as the same span (Appendix B.3). Could this cause the selector to treat two demonstrations as redundant even when they mainly support different successful rollouts?

3 comments
AuthorsOpenReview ↗

Response to Reviewer heH2

The authors want to thank the reviewer for the constructive feedback. Your comments help us distinguish the evidence for data curation from the additional benefits of TUCO and clarify the interpretation of our coverage measure. We respond point by point below.

W1: Evidence for TUCO beyond the benefits of data curation

R1: We have added a direct comparison with CUPID on a new real-world pick-and-place task. Thank you for encouraging us to strengthen the evidence for TUCO's selection rule. This experiment specifically compares the two curation methods, complementing the original comparisons against full-pool training and random selection.

We partition 30 real demonstrations into three disjoint groups of 10. For each group, CUPID and TUCO each select 50% of the same simulation candidate pool. We then train each policy using its selected simulation subset together with all 10 real demonstrations from that group. Thus, both methods use the same source-data budget and the same real demonstrations within each comparison. We evaluate each policy over 20 physical trials.

MethodGroup 1Group 2Group 3Overall
CUPID10.0%10.0%10.0%10.0%
TUCO35.0%50.0%25.0%36.7%

TUCO outperforms CUPID in all three groups, with gains of 15.0 to 40.0 percentage points.

Revisions: Extended Experiments appendix, “Additional Real-World Evaluation with Different Demonstration Sets,” new task setup, CUPID comparison table, and results analysis.

W2: Sensitivity to the choice of real scoring rollouts

R2: We have repeated curation and co-training using three disjoint sets of real demonstrations. Thank you for pointing out the importance of testing whether our findings depend on a particular set of successful executions. In the additional experiment described in R1, each group provides a different set of 10 real demonstrations for curation and subsequent co-training. CUPID and TUCO receive the same group and select the same fraction of simulation data in each comparison.

TUCO's success rates are 35.0%, 50.0%, and 25.0% across the three groups, compared with 10.0% for CUPID in every group. The absolute performance varies with the real demonstration set, but TUCO retains its advantage in all three comparisons. This supports the consistency of its benefit across the tested sets rather than only one particular collection of real executions. The revised appendix reports the results for each group as well as the overall success rate.

Revisions: Extended Experiments appendix, “Additional Real-World Evaluation with Different Demonstration Sets,” three-group evaluation protocol and comparison across real demonstration sets.

Q1: Can opposite residuals hide complementarity between demonstrations?

A1: Thank you for this thoughtful question. Opposite residuals do not cancel in our objective or prevent both demonstrations from being selected. TUCO measures coverage of independent influence directions and combines this measure with each demonstration's utility. The same-span property therefore produces diminishing coverage gains, rather than a rule that excludes one of the demonstrations.

Importantly, opposite residuals need not imply conflicting contributions to target performance. For two successful rollouts, the profiles (3,1)⊤(3,1)^\top and (1,3)⊤(1,3)^\top both have utility q=4q=4 and positive influence on both executions. Their residuals, (1,−1)⊤(1,-1)^\top and (−1,1)⊤(-1,1)^\top, describe which execution receives more support relative to their common utility component. The negative residual entry denotes below-average support, not a negative original influence.

The coverage objective accumulates positive-semidefinite outer products. Thus, for normalized residuals v\mathbf v and −v-\mathbf v, their contributions reinforce the same direction rather than cancel:

=(w_i+w_j)\mathbf v\mathbf v^\top.$$ For example, if the current set contains only demonstration $i$, adding $j$ gives $$\Delta_j(\\{i\\}) =\log\\!\left(1+\frac{\rho w_j}{1+\rho w_i}\right)>0,$$ where $w_i,w_j>0$ within the feasible pool. Demonstration $j$ also retains its utility term in the selection rule. Both demonstrations can therefore be selected when their combined utility and marginal coverage make them preferable to the remaining candidates. The sign invariance has a specific interpretation: reversing an existing direction adds support along that direction but does not introduce an independent direction of variation. Accordingly, the coverage term does not give an additional reward solely for the sign reversal, even when the demonstrations emphasize different successful rollouts. This is consistent with our directional-coverage objective. It does not equate the demonstrations' full learning effects or remove their individual utility.

Thank the authors for the detailed answer. They have effectively tackled the majority of my concerns.

AuthorsOpenReview ↗

Appreciation for the Reviewer heH2’s Feedback and Further Consideration of the Evaluation

Thank you very much for your thoughtful feedback. We are genuinely encouraged that our responses have addressed most of your concerns! Your suggestions were particularly valuable in guiding the extra experiments, which strengthened the evidence for our conclusions and helped us present the findings more clearly.

Given that most of your concerns have now been addressed, we would be sincerely grateful if you could reconsider the overall score in light of the revisions and additional experimental evidence. We would also be happy to address any remaining questions. Thank you again for the time and care you have devoted to helping us improve this work.

Reviewer_T51mOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes ways to measure the effectiveness of simulation trajectories in improving policy performance in a different domain (real). It also proposes an optimizer to combine these measures to select the best demonstrations.

The paper builds on the CUPID paper (https://arxiv.org/abs/2506.19121). The contributions are: a couple of measures, an optimizer, and a demonstration that the approach improves sim-to-real.

Critical strengths
  • Overall, the paper is mostly well-written.
  • The approach seems reasonable, borrowing heavily from CUPID.
  • Results on sim-to-real improvement (Tab. 1) are convincing.
Critical weaknesses
  • The paper requires quite a bit of effort to understand. For example, the two primary measures being proposed are "trajectory-level utility" and "set-level coverage". These are so important that they form the acronym of the method, TUCO. Yet the definition of these two terms is nowhere to be found in the paper, when one would expect them to be prominently displayed up front.
  • Similarly, there is quite a bit of math. Most of it checks out just fine, but some of it is not clearly explained to provide intuition.
  • Figures 2 and 3 which form some of the main experiments of the paper are almost impossible to decipher.
Questions for authors
  • Please define "trajectory-level utility" and "set-level coverage". Explain why they're important, and why you chose them rather than something else.
  • L057 raises question about two terms that haven't been defined.
  • Sec 3 (L137-147) should cite CUPID.
  • L144 What is ∇2 θL? If the gradient is a vector, what does it mean to square it?
  • What is the reader supposed to glean from Figure 2?
  • What is the reader supposed to glean from Figure 3? Why are some algorithms only displayed on some graphs? CUPID, for example, is only shown for Utility, not Quality nor Diversity. Can you increase the font sizes to make it more readable?
  • What is Figure 4 about?
  • The abstract is confusing. The 3rd sentence says "existing methods lack a unified criterion for TLU and SLC", but as far as I can tell, this paper proposes TLU and SLC for the first time, so why not phrase it that way? The next sentence says the paper is the first to study data curation, but isn't that what CUPID does? Why claim your paper is the first if it's not? The next sentence says that TLU and SLC are "effects" -- in what sense are they effects? I thought they were metrics/measures. What does it mean to be "decomposed into an overall contribution"? Decomposition means to split one thing into multiple things, but the phrase suggests that multiple things are put into a single overall thing -- which one is it? "providing a common closed-loop basis" -- what does "basis" mean here? "for measuring TU and SC" -- isn't this circular reasoning: Our key insight is that we can combine TLU and SLC to measure TLU and SLC? In the next sentence, "optimizer that combines these measures": optimizers don't combine things, they optimize things that are combined. "reduce redundancy and select complementary" - these make intuitive sense, but I couldn't find anywhere later in the paper where they are explicitly tied back to the optimization objective.
  • L211-224 appears to be a central part of the work, b/c it presents the math underlying utility and coverage. Please explain these concepts more clearly. I cannot follow the paragraph, which begins by mentioning "CUPID’s linear subset objective", but the CUPID paper doesn't contain these words, nor does it mathematically define "utility".
  • Please explain Eq. (1) in a way that conveys intuition behind the symbols. It's not obvious to me.

I am borderline on this paper. I would be happy to increase my score if the authors can explain their work more clearly, and/or if my understanding can be shown to be wrong.

9 comments
AuthorsOpenReview ↗

Response to Reviewer T51m - Part 1

We sincerely thank the reviewer for the careful reading and constructive feedback. We have revised the manuscript to address the presentation concerns you raised and provide detailed responses below. We kindly invite you to review the revised paper alongside these explanations.

W1: Unclear definitions of utility and coverage

R1: We have added explicit definitions of trajectory-level utility and set-level coverage earlier in the introduction and clarified why both are needed for data curation.

W2: Insufficient intuition behind the mathematics

R2: We have revised the mathematical presentation to explain the notation and connect the equations to their underlying intuition.

W3: Poor readability of Figures 2 and 3

R3: We have improved the readability of Figures 2 and 3 by clarifying the captions and labeling. We also explain that the rows in Figure 3 group baseline methods by selection criterion.

Q1: What do utility and coverage mean, and why use both?

A1: Trajectory-level utility measures how useful an individual demonstration is for learning, expressed as its expected contribution to target-policy performance. A higher utility score suggests a greater expected learning benefit. Utility-based methods use these scores to select demonstrations that are individually valuable for the target task.

However, these individual utility scores do not describe the inner relationships among demonstrations. For example, two demonstrations with equal utility may help the policy in different situations, demonstrations with different scores may teach essentially the same behavior but with different degrees of usefulness. As a result, selecting demonstrations only by their individual utility scores can therefore keep redundant examples and overlook useful combinations.

Set-level coverage measures how diverse a selected dataset is, seeking a collection of demonstrations that spans varied behaviors with limited repetition. It accounts for relationships among demonstrations and whether they provide complementary information. In TUCO, this diversity concerns how demonstrations support closed-loop target performance.

We use both because individual utility and coverage to address complementary needs in data curation. Several high-utility demonstrations may provide similar learning benefits and therefore be redundant. Likewise, a diverse dataset can include demonstrations with little learning value for the target task. Combining utility and coverage allows selection to consider both the learning value of individual demonstrations and the diversity of the selected dataset. The intuition is to retain useful demonstrations while reducing redundancy and promoting complementarity.

Revisions: Abstract, brief definitions of trajectory-level utility and set-level coverage; Introduction, second paragraph, intuitive definitions and motivation for combining utility with coverage.

Q2: Undefined terms at L057.

A2: Please see A1 for detailed explanations of these two concepts.

Q3: Missing CUPID citation in Section 3.

A3: Thank you for the reminder. As you suggested, we have added a citation to CUPID and fix similar problems.

Q4: What does the Hessian notation at L144 mean?

A4: The superscript 22 denotes second-order differentiation. For parameters θ∈Rdθ\theta\in\mathbb R^{d_\theta}, ∇θ2L\nabla_\theta^2\mathcal L is the dθ×dθd_\theta\times d_\theta Hessian matrix, whose entries are

=\frac{\partial^2\mathcal{L}(\theta\mid\mathcal{D})}{\partial\theta_j\\,\partial\theta_k}, \qquad j,k\in\\{1,\ldots,d_\theta\\}.$$ Intuitively, the gradient describes how the loss changes with the parameters, while the Hessian describes how that gradient changes. The notation therefore represents the local curvature of the loss, not squaring a vector. Here, $\mathbf H_0$ is the Hessian evaluated at the base parameters $\theta_0$. Thank you for flagging this ambiguity. Following your feedback, we have explicitly explained the Hessian's dimensions and second-derivative meaning in Section 3 and added intuition for its role in the influence formula to make the notation easier to follow. **Revisions:** Section 3, the definition of $\mathbf H_0$ following the influence-function equation and the accompanying intuitive explanation.
AuthorsOpenReview ↗

Response to Reviewer T51m - Part 2

Q5: What does Figure 2 show?

A5: We designed TUCO to support data curation across single-simulator, sim-to-sim, and sim-to-real settings. We perform experiments one by one. Figure 2 establishes its effectiveness in the controlled single-simulator setting, where we can evaluate curation without the additional effects of domain shift.

The figure covers two complementary scenarios: filtering an existing training dataset and selecting demonstrations to augment a small fixed dataset, following CUPID. The horizontal axis indicates the fraction removed or selected respectively, and the vertical axis reports policy success rate.

The main takeaway is that TUCO performs competitively across both scenarios and different data budgets. The filtering results also show that retaining fewer demonstrations can preserve or improve performance relative to full-data training. This establishes that useful training data need not coincide with all available data and provides a controlled starting point for the cross-domain studies.

Revisions: Figure 2 caption; Section 5.1, Single-Simulator results and analysis.

Q6: Figure 3: takeaways, method grouping, and readability.

A6: Quality, Utility, and Diversity identify families of curation methods, following the Introduction. All panels use the same evaluation metric, target-domain policy success rate. CUPID appears in the Utility row because it ranks demonstrations by their estimated contribution to policy performance. We group methods to reduce visual overlap and repeat TUCO and the reference baselines so readers can compare it with each family.

Figure 3 shows that active curation remains beneficial when source data and target evaluation come from different simulators. Curated subsets improve over full-pool co-training, and TUCO outperforms random selection at every tested budget. TUCO achieves the highest mean success rate in 16 of 18 task and budget combinations. These findings show that choosing suitable demonstrations matters beyond simply reducing the source-data volume.

Thank you for highlighting the readability issue. Following your advice, we have clarified the grouping in the caption and relabeled the rows as “TUCO vs. Quality-based methods,” “TUCO vs. Utility-based methods,” and “TUCO vs. Diversity-based methods.” We have also increased the font sizes to make the comparisons easier to read.

Revisions: Figure 3, row labels, font sizes, and caption; Section 5.1, Sim-to-Sim Co-Training analysis updated for eight seeds.

Q7: What does Figure 4 explain?

A7: Figure 4 explains why we constrain coverage-based selection to demonstrations whose estimated effects align with target success. A demonstration can affect different target executions in different ways without providing much overall benefit. Such a demonstration may appear valuable for diversity even when its contribution to target performance is weak.

The figure shows where source demonstrations lie along this performance-alignment axis. Its key message is that the cone constraint targets candidates whose variation across executions is large relative to their estimated performance contribution. Although these candidates constitute a small portion of the pool, they can influence a coverage-driven selection disproportionately. Figure 4 therefore provides a geometric explanation for the constraint, while the ablation in Table 2 supplies the empirical evidence for its benefit.

Q8: Clarifying the abstract's claims, terminology, and logic.

A8: Thank you for the detailed feedback. Following your advice, we have rewritten the abstract to clarify the concepts, scope of our contribution, and key idea. We address each point below.

8.1. Are trajectory-level utility and set-level coverage concepts introduced for the first time in this paper?

No, we do not claim that the general concepts of individual utility and set coverage originate in our work. They correspond to two existing perspectives on data curation that emphasize individual usefulness and set diversity respectively. We use utility and coverage to summarize the central ideas of these perspectives. Our methodological contribution is to characterize both aspects within the same representation of target-domain closed-loop behavior and use them for data curation. By “lack a unified criterion,” we intended to say that these aspects are typically treated separately in existing methods.

Revisions: Abstract, definitions of utility and coverage and the statement of TUCO's contribution.

AuthorsOpenReview ↗

Response to Reviewer T51m - Part 3

8.2. Since CUPID already studies data curation, what does the first-study claim refer to?

Our first-study claim specifically concerns simulation data curation for sim-to-real robot policy co-training. Many work already studies robot data curation, and we recognize them as important foundations for our work. Previous robot data curation studies have primarily considered selection within a simulator or selection of costly real-world demonstrations. To our knowledge, our work provides the first systematic study of simulation data curation for sim-to-real co-training and demonstrates its value and potential for improving real-world performance. During this rebuttal, we have revised the abstract to make the scope of the first-study claim explicit.

Revisions: Abstract, first-study claim restricted to simulation data curation for sim-to-real robot policy co-training.

8.3. Does “these effects” refer to utility and coverage?

No. It referred to each demonstration's estimated effects on target-policy executions, from which we derive utility and coverage. Utility and coverage are measures of this information. Your comment helped us identify the unclear reference, and we have removed “these effects” from the abstract and stated directly what information influence functions provide.

Revisions: Abstract, replacement of “these effects” with an explicit description of the information used to measure utility and coverage.

8.4. Does “decomposed into an overall contribution” mean splitting one representation or combining several effects?

In the original version of the paper, we split one influence representation into two components: its “overall contribution to target return,” which reflects individual learning value, and its “variation across rollouts.” Each demonstration has a collection of estimated effects on different target rollouts. The component aligned with the target-return direction captures its overall performance contribution and characterizes utility. The orthogonal component retains differences across rollouts beyond that contribution and provides the representation for measuring set coverage. “Overall contribution” names only the first component, and the full description also includes the second. Following your feedback, we have simplified this explanation in the abstract and explicitly identified both components and their roles in Section 4.3.

Revisions: Abstract, simplified explanation of the key insight; Section 4.3, decomposition equation and interpretation of its two components.

8.5. What does “basis” mean in “a common closed-loop basis”?

It means that both measures use the same information about how demonstrations affect target-policy executions. Thank you for pointing out that “basis” was too abstract here. We have replaced the phrase with a direct explanation that influence functions provide information about individual learning value and the diversity of contributions to closed-loop target performance.

Revisions: Abstract, replacement of “a common closed-loop basis” with a concrete explanation of the shared influence information.

8.6. Is the reasoning circular?

No. We start from influence estimates, then derive utility and coverage from that information. Our insight is that influence attribution contains both individual learning value and differences among demonstrations. Following your advice, we have rewritten the key-insight sentence to make this starting point explicit and avoid suggesting that the measures define themselves.

Revisions: Abstract, key-insight sentence identifying influence functions as the source of information for both measures.

8.7. What does the optimizer combine or optimize?

The selection criterion combines utility and marginal coverage, and the optimizer selects demonstrations according to this criterion. Following your suggestion, the revised abstract describes the optimizer as balancing utility and coverage during selection. Section 4.4 gives the criterion and iterative update.

Revisions: Abstract, description of the optimizer; Section 4.4, explanation of the selection criterion and iterative update.

Comments on the paper

3 comments
AuthorsOpenReview ↗

Summary of the rebuttal and the major changes of revised manuscript

Dear reviewers,

We would like to express our heartfelt gratitude for your invaluable time and constructive feedback on our paper. We appreciate all reviewers give us positive feedback with insightful advice that greatly help us improve the quality of our work.

We appreciate the reviewers' recognition of the value of our findings and the motivation behind our approach:

  • Reviewer T51m: “Results on sim-to-real improvement (Tab. 1) are convincing.”
  • Reviewer heH2: “I also like that coverage is measured through effects on target rollouts, rather than differences in the source data that may have little relevance to the target task.”
  • Reviewer QAkw: “The work introduces a useful insight for data curation that the coverage of the selected dataset can influence the final co-training performance sometimes significantly.”

In response to the concerns raised, we have diligently addressed all the issues by revising the manuscript and conducting additional simulation and real-world experiments. A revised manuscript has been submitted. The revisions are highlighted in violet. Please allow us to summarize the significant alterations made in both the rebuttal and the revised manuscript:

  • Writing Revision: Clarified the core concepts, intuition, and contributions relative to prior work (Abstract, Introduction, and Methodology).
  • Mathematical Clarity: Clarified notation and definitions and explained the intuition behind the main equations (Sections 3 and 4).
  • Figure Presentation: Revised the captions to clarify experimental protocols and baseline groupings (Figures 2 and 3).
  • Expanded Simulation Evaluation: Added five random seeds, bringing the sim-to-sim comparison to eight seeds per configuration (Figure 3).
  • Additional Real-World Evaluation: Compared TUCO with CUPID on a new physical task using three distinct real demonstration sets (Extended Experiments in appendix).

We provide point-by-point responses below and kindly invite the reviewers to consider the revised manuscript and additional evidence. Thank you again for helping us improve the clarity and evaluation of this work.

Best Regards,

Authors of Submission 19082

AuthorsOpenReview ↗

Follow-up on rebuttal and additional evidence

Dear ICLR 2027 Reviewers, AC, SAC, and PC,

We would like to express our sincere gratitude to all the reviewers for their valuable and insightful feedback. We have carefully considered all suggestions and updated our submissions accordingly.

However, we have not yet received follow-up responses from the reviewers. To briefly highlight the additional evidence included in our revision:

  1. Direct comparison with CUPID on a new real-world task: Under matched simulation-data budgets and real demonstrations, TUCO achieves 36.7% overall success, compared with 10.0% for CUPID.

  2. Evaluation with three disjoint real demonstration sets: We repeated curation and co-training using three different sets of 10 real demonstrations. TUCO achieves 35%, 50%, and 25% success, compared with 10% for CUPID in each group, with 20 physical evaluation trials per policy.

  3. Expanded sim-to-sim evaluation from three to eight training seeds per configuration: We updated Figure 3 and the corresponding analysis to better assess performance and variability across training runs.

As the discussion period approaches its end, we would greatly appreciate your feedback on whether our responses and additional evidence have addressed your concerns.

If there are any remaining issues or additional points you would like us to clarify, please do not hesitate to let us know. We would be more than happy to address any remaining issues to improve our work.

Thank you again for your time, effort, and thoughtful consideration of our work. We sincerely appreciate your valuable contributions throughout the review process.

Best Regards,

Authors of Submission 19082

AuthorsOpenReview ↗

Summary of Revisions and Additional Evidence Addressing Reviewer Concerns

Dear Area Chair, Program Chairs, and Reviewers,

We sincerely thank you again for your valuable time, careful evaluation and constructive feedback. Following your insightful suggestions, we have meticulously made the following revisions and additions to address the main concerns.

  • Presentation clarity (Reviewers T51m and QAkw). We refined the abstract, introduction, and methodology to make the motivation, core concepts, and intuition clearer. The core method and technical contributions remain unchanged.

  • Mathematical clarity (Reviewers T51m and QAkw). We clarified the notation, sources, and intuitive meaning of the key equations in Sections 3 and 4, with explicit connections to the method illustration.

  • Figure interpretation (Reviewer T51m). We clarified the captions of Figures 2 and 3 and explained in our follow-up response how Figure 4 illustrates the cone constraint and complements the ablation results.

  • Novelty beyond CUPID (Reviewers heH2 and QAkw). We clarified how TUCO’s influence decomposition and subset optimizer jointly use utility and coverage, extending beyond independent utility ranking.

  • Opposite residuals (Reviewer heH2). We explained why opposite residuals do not cancel in the coverage objective and why both demonstrations remain eligible for selection.

  • Experimental variability (Reviewer QAkw). We added five random seeds, bringing the sim-to-sim evaluation to eight seeds per configuration. The updated results retain TUCO’s strong performance and show the lowest average cross-seed standard deviation among the compared curation methods.

  • Direct comparison and robustness across real demonstration sets (Reviewer heH2). We added a physical Ketchup pick-and-place task using three disjoint real demonstration sets. TUCO outperforms CUPID in every group under matched budgets, achieving 36.7% overall success versus 10.0%. Details are included in the Extended Experiments appendix.

  • Hessian invertibility (Reviewer QAkw). We clarified the practical curvature approximation and its theoretical basis, and supplemented this explanation with numerical tests supporting the invertibility and inversion accuracy of the implemented matrix in the tested configuration.

We are genuinely grateful to the reviewers, Area Chair, and Program Chairs for their time and constructive guidance. We sincerely hope these revisions and additional findings address the concerns raised.

Best regards,
Authors of Submission 19082

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.