EV-Cloth: Seeing Cloth Deformation Between RGB Frames

Tracking deforming cloth requires estimating the 3D motion of persistent material points, not merely reconstructing a surface in each image. Between RGB frames, however, visual observers must rely on dynamics predictions: increasing the prediction rate does not provide additional evidence about the actual deformation. We introduce EV-Cloth, an event-assisted observer that uses asynchronous visual measurements to correct cloth states during these observation gaps. Its core module, Action-conditioned, Asynchronous, Association-aware Event correction (A${}^3$E), augments a frozen action-conditioned graph dynamics prior. It associates events with the predicted cloth surface, pools their evidence onto persistent material nodes, and estimates a residual correction to the nominal dynamics update. Corrected positions and displacement history are written back into the recurrent state, informing subsequent prediction and event association, while RGB frames periodically refine geometry. On 24 held-out simulated trajectories with approximately 30-Hz RGB observations and a matched 100-Hz state-update schedule, EV-Cloth reduces inter-frame material tracking error by 36.0% relative to an RGB-only observer and improves 22 of 24 trajectories. Controlled studies examine observed event content, material association, and recurrent state correction. The association-by-recurrence interaction persists across three event representations, while timing controls show that reducing recurrent writeback frequency degrades accuracy even when elapsed event evidence is accumulated. These results support event-assisted recurrent correction for persistent cloth-state estimation between RGB observations.