
Video Diffusion Models for Temporal Modeling of 2D Medical Imaging Tasks ↗
Longitudinal medical imaging tracks how disease progresses over time, but forecasting future scans is difficult because clinically meaningful changes are spatially complex and often irregular in timing. This project developed a text-conditioned video diffusion model for temporal modeling of 2D medical imaging tasks: first pre-training on individual frames, then extending the architecture into a spatio-temporal U-Net that generates full image sequences, with masking to support padded, irregularly-spaced longitudinal inputs.
The framework was validated on a synthetic dataset built from a known linear time-invariant system before being applied to real clinical data, including ADNI brain MRI, longitudinal skin lesion images, and chest X-rays. MRI experiments produced realistic-looking sequences but also revealed the limits of a 2D slice-based model applied to inherently 3D anatomy, which motivated a shift toward lower-resolution 2D modalities that better match the model’s assumptions. On skin and chest X-ray data, the model was benchmarked against a ControlNet baseline using pixel-space error and CLIP/BiomedCLIP embedding similarity: history-guided sampling achieved the lowest pixel error on skin images and the strongest embedding similarity overall, while causal sampling performed best in pixel space for chest X-rays. The result is an end-to-end pipeline for temporal modeling of 2D medical imaging with video diffusion models.
