Trust, but Verify Temporal Memory: Procedure-Consistency Firewall for Surgical Video Segmentation

Temporal memory provides valuable context for online surgical video segmentatio, but most existing methods assume that the previous states remain relavant to the current scene. We find that this assumption can break under procedural context shifts. When the query memory is obtained from an incompatible procedural context, harmful temporal information can corrupt current predictions and recursively damage future states. Estimating memory utility with a generic indicator of does not fully resolve this problem, as such utility gate tends to be overly conservative and rejects useful history. Motivated by these observations, we propose a **Procedure-Consistency Memory Firewall** and a **Trustworthy Memory Writer**. The Procedure-Consistency Memory Firewall is a annotation-free, inference-time mechanism that compares the current frame procedure distribution with the provenance distribution attached to temporal memory. The Trustworthy Memory Writer determines whether the resulting state is sufficiently reliable to update trusted memory, limiting dirty context from propagating forward. Across a controlled suite of temporal interventions, our firewall selectively targets provenance mismatch rather than broadly suppressing temporal context. Our method clean performance under clean conditions while improving robustness under core stress by 6.31% and worst-case Dice by 32.77%, with the largest gain under cross-procedure mismatch. These results suggest that robust temporal reasoning depends not only on remembering the past, but also on verifying whether the memory is suitable before reuse and propagation.