acceptodds
Withdrawn from ICLR 2027

EV-Cloth: Seeing Cloth Deformation Between RGB Frames

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Tracking deforming cloth requires estimating the 3D motion of persistent material points, not merely reconstructing a surface in each image. Between RGB frames, however, visual observers must rely on dynamics predictions: increasing the prediction rate does not provide additional evidence about the actual deformation. We introduce EV-Cloth, an event-assisted observer that uses asynchronous visual measurements to correct cloth states during these observation gaps. Its core module, Action-conditioned, Asynchronous, Association-aware Event correction (AE), augments a frozen action-conditioned graph dynamics prior. It associates events with the predicted cloth surface, pools their evidence onto persistent material nodes, and estimates a residual correction to the nominal dynamics update. Corrected positions and displacement history are written back into the recurrent state, informing subsequent prediction and event association, while RGB frames periodically refine geometry. On 24 held-out simulated trajectories with approximately 30-Hz RGB observations and a matched 100-Hz state-update schedule, EV-Cloth reduces inter-frame material tracking error by 36.0% relative to an RGB-only observer and improves 22 of 24 trajectories. Controlled studies examine observed event content, material association, and recurrent state correction. The association-by-recurrence interaction persists across three event representations, while timing controls show that reducing recurrent writeback frequency degrades accuracy even when elapsed event evidence is accumulated. These results support event-assisted recurrent correction for persistent cloth-state estimation between RGB observations.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
event-based vision, cloth state estimation, deformable object perception, asynchronous sensing, 3D state estimation, deformable object manipulation

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_C1hrOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces EV-Cloth and its core module, A³E, for estimating the persistent state of deforming cloth between RGB frames. Events are associated with the predicted cloth surface and persistent material points; the resulting evidence is aggregated at material nodes, and a graph network predicts a residual correction to an action-conditioned dynamics prior. Corrected positions and displacement history are written back into the recurrent state, while subsequent RGB frames refine the geometry. On 24 held-out simulated trajectories, with approximately 30-Hz RGB observations and a 100-Hz state-update schedule, the method reduces material tracking error from 1.637 mm for the RGB-only observer to 1.047 mm, improving 22 of 24 trajectories. The real-world evaluation is limited in scale and uses image-plane checker correspondences rather than independently measured full 3D ground truth.

Critical strengths
  1. The paper studies the meaningful problem of estimating cloth state continuously between RGB frames, and integrates event association, material-node feature updates, recurrent state correction, and later RGB-based geometric refinement into a coherent loop.
  2. In the matched simulation setting, EV-Cloth reduces material tracking error by approximately 36% relative to the RGB-only baseline and improves 22 of 24 trajectories. The paper also includes controlled analyses that provide evidence for the roles of event information and recurrent updates in this setting.
Critical weaknesses
  1. Limited evidence for generalization and real-world performance. The simulation evaluation centers on 24 held-out trajectories, but the training and test sets share the same pool of cloth meshes. The experiments therefore do not establish generalization to unseen cloth geometries or physical scales, and the action-shift experiments do not directly evaluate held-out grasp locations. The real-world evaluation is also limited in scope and lacks independently measured full 3D material-state ground truth. These limitations make it difficult to determine whether the method generalizes beyond the specific simulated setting.
  2. Limited methodological novelty and insufficient comparison with alternative fusion strategies. Geometric association, feature concatenation, and graph-based residual prediction are individually established techniques. The main contribution appears to lie in their task-specific integration with recurrent state writeback, rather than in a substantially new association or fusion mechanism. Although the paper includes internal ablations, it lacks comparisons with alternative RGB-plus-event fusion or association methods under the same task and evaluation protocol. Thus, the experiments demonstrate the effectiveness of the proposed system relative to an RGB-only observer, but do not sufficiently establish the distinctiveness of the proposed design. In addition, Figure 2 appears to show addition between Hj[i]H_j[i] and ziz_i, whereas Equation 11 specifies concatenation, creating an ambiguity in the method description. Moreover, Figure 2 raises concerns beyond visual presentation.
  3. The practical value for downstream manipulation remains unverified. The paper motivates persistent cloth-state estimation partly through its potential utility for downstream action planning, but it does not specify and evaluate a concrete downstream manipulation task. No task-level metrics, such as manipulation success rate or improvement in execution quality, are reported. Consequently, the practical benefit of the improved intermediate state estimates remains a plausible motivation rather than a demonstrated result.
Questions for authors
  1. In Figure 2, are Hj[i]H_j[i] and ziz_i added or concatenated? Equation 11 specifies concatenation, while the figure appears to show addition. Please clarify the actual implementation and make the figure and equation consistent.
  2. Do the held-out simulation trajectories include cloth sizes or shapes, colors or patterns, grasp locations, and action conditions not seen during training? Please describe the train/test split and provide any available cross-condition generalization results. This affects the scope of the paper’s claims.
  3. Are there quantitative real-world results based on independently measured 3D ground truth? Are the checker references image-plane correspondences only? Are results available across multiple cloth configurations, grasp locations, or trajectories?
  4. What specific downstream task is intended by the discussion of downstream action planning, and has EV-Cloth been shown to improve its quantitative performance? If this has not been evaluated, should the downstream benefit be presented as a potential application rather than a demonstrated result?
  5. Can the authors compare against alternative fusion or association methods using the same RGB-plus-event input? How do the existing internal controls distinguish the contribution of event information from the contribution of the specific proposed architecture?
  6. Please clarify the intended interpretation of the centered residual R(H,Z)−R(H,0)R(H,Z)-R(H,0). This difference centers the response at zero event descriptor for a fixed HH, but because RR is nonlinear, it does not generally make the correction independent of interactions between HH and ZZ. Is the paper claiming baseline centering only, or a stronger independence from HH?
Reviewer_h39aOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes EV-Cloth, a method that uses the A3E module to associate asynchronous event information with 3D cloth material nodes and recursively correct the states of a frozen graph-based dynamics model, enabling cloth tracking between RGB frames. On 24 simulated test trajectories, the method reduces material tracking error by 36.0% compared to an RGB-only baseline. The authors conduct ablation studies to evaluate the contributions of event information, material association, and recursive updates.

Critical strengths

1.Practically meaningful problem. The paper addresses the lack of visual observations between RGB frames by introducing event cameras to support continuous 3D cloth state estimation. The research motivation is clear and relevant.

2.Well-motivated method design. A3E establishes associations between events and persistent material nodes through the predicted surface, and feeds the corrections back into the dynamics state, forming a closed-loop estimation framework.

3.Significant simulation results. Under the same RGB observations and update frequency, the method reduces tracking error by 36.0% and achieves improvements on 22 out of 24 trajectories. Multiple ablation studies further support the effectiveness of the core design choices.

Critical weaknesses
  1. Insufficient attribution of improvements to actual event information. Table 1 shows that removing the observed event content still preserves approximately 77% of the overall error reduction. Moreover, empty event packets can still produce nonzero corrections. Therefore, it is difficult to clearly distinguish the contribution of actual event information from that of additional learned dynamics corrections.
  2. Confounding factors in validating the interaction between association and recursion. The Output-only variant in Table 2 is implemented by disabling state feedback at test time in a model trained with recursive updates, introducing a train–test mismatch. Furthermore, Appendix G shows that blocking the feedback from corrected states to subsequent event associations has almost no effect on performance, weakening the evidence for the necessity of the proposed bidirectional closed-loop mechanism.
  3. Dependence of the core event association mechanism on accurate predicted geometry. A3E assigns events through projections of the predicted surface, but inaccurate geometry may lead to incorrect associations that propagate through recursive updates. The experiments use ground-truth 3D meshes for initialization and lack systematic evaluations under geometric prediction errors and camera calibration perturbations. Therefore, the reliability of this mechanism under inaccurate state estimates remains unverified.
Questions for authors

See Critical Weaknesses.

Reviewer_wJY5OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes a method for improving cloth tracking quality from videos between RGB frames, using additional observations captured by event cameras.

Assuming a cloth dynamic prior, known material graph, and ground-truth initial state, the metod uses a "A3E" module to reduce tracking error of material nodes on the cloth between RGB frames.

Critical strengths

The paper provides a practical method that leverage event camera's signals to improve cloth tracking from videos. The pipeline design makes sense to me, and the reported experiment results show that the method is able to reduce tracking error between RGB frames.

Critical weaknesses

Scope and real-world feasibility. The scope of this paper is focused on event-based perception for cloth. It also has strong assumptions on the input, including priviledged information like known material graph and ground-truth initialization state. Additionally, the action condition is limited to a format of one three-dimentional action, one picked-node index, and one grasp indicator, which does not generalize to general manipulation actions and contact modes. These would limit the method's application potential in the real world on general cloth manipulation recordings.

For real-world experiments, the paper provides only qualitative results in the supplementary. I understand that quantitative result is hard to obtain due to missing priviledged information in the real world, but I also found the qualitative results in Appendix H fairly hard to interpret. Particularly in Figure S4(b) "RGB only", why dpes the orange point (RGB-only prediction?) drift so far away from the cloth zone in the RGB image?

Reproducibility. The paper does not seem to provide critical details about the frozen action-conditioned dynamics prior (Sec. 4.1) (also see "Questions for Authors" below). This information is important, especially for reproducibility, because all other modules are built upon this prior.

Presentation. The paper also contains other undefined terms (see questions below) that confuse the reader. I am suspecting that some of them are AI-generated terms from a certain context (codebase?), which lost their meanings out of the context.

Questions for authors
  1. Does the method generalize to cloth shapes other than single-layer rectangular towels? I understand that the paper does not claim that its trained model generalizes to OOD meshes (l. 464; l. 726), but I wonder whether the method generalizes to other input meshes, like garments?

  2. Does the method apply to all cloth materials? Appendix A only provides step size and substep count, without specifying the physical parameters for the cloth such as elasticity and bending stiffness.

  3. In Sec 4.1, what exactly is the "frozen action-conditioned dynamics prior"? What model architecture does it use, and what data is it trained on? Sec. 4.1 and Appendix only provides the model interface, checkpoint name ("S0" and "A2RGNS100Hz") and data format, without citing or describing the exact model or dataset used (if I didn't miss anything).

  4. The paper contains many other undefined or uncited terms, like "test24", "DEV", "egistered-parent-disjoint", "the pinned synthetic-cloth-data codebase", etc.

  5. The appendix claims that there is are supplementary materials including a video (l. 1300) and a CSV file (l. 1321) but I did not find downloadable link on openreview. Was it a bug or is the supplemenraty not uploaded?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.