
Video Diffusion Models for Temporal Modeling of 2D Medical Imaging Tasks ↗
Read moreShow less
Longitudinal medical imaging tracks how disease progresses over time, but forecasting future scans is difficult because clinically meaningful changes are spatially complex and often irregular in timing. This project developed a text-conditioned video diffusion model for temporal modeling of 2D medical imaging tasks: first pre-training on individual frames, then extending the architecture into a spatio-temporal U-Net that generates full image sequences, with masking to support padded, irregularly-spaced longitudinal inputs.
The framework was validated on a synthetic dataset built from a known linear time-invariant system before being applied to real clinical data, including ADNI brain MRI, longitudinal skin lesion images, and chest X-rays. MRI experiments produced realistic-looking sequences but also revealed the limits of a 2D slice-based model applied to inherently 3D anatomy, which motivated a shift toward lower-resolution 2D modalities that better match the model’s assumptions. On skin and chest X-ray data, the model was benchmarked against a ControlNet baseline using pixel-space error and CLIP/BiomedCLIP embedding similarity: history-guided sampling achieved the lowest pixel error on skin images and the strongest embedding similarity overall, while causal sampling performed best in pixel space for chest X-rays. The result is an end-to-end pipeline for temporal modeling of 2D medical imaging with video diffusion models.
A text-conditioned video diffusion model that forecasts how medical scans change over time, built on a spatio-temporal U-Net and validated on brain MRI, skin lesions, and chest X-rays.
