Skip to main content

Fast Coherent Video Style Transfer via Flow Errors Reduction.

Wang, L., Yang, X. and Zhang, J., 2024. Fast Coherent Video Style Transfer via Flow Errors Reduction. Applied Sciences, 14 (6), 2630.

Full text available as:

applsci-14-02630.pdf - Published Version
Available under License Creative Commons Attribution.


DOI: 10.3390/app14062630


For video style transfer, naively applying still image techniques to process a video frame-by-frame independently often causes flickering artefacts. Some works adopt optical flow into the design of temporal constraint loss to secure temporal consistency. However, these works still suffer from incoherence (including ghosting artefacts) where large motions or occlusions occur, as optical flow fails to detect the boundaries of objects accurately. To address this problem, we propose a novel framework which consists of the following two stages: (1) creating new initialization images from proposed mask techniques, which are able to significantly reduce the flow errors; (2) process these initialized images iteratively with proposed losses to obtain stylized videos which are free of artefacts, which also increases the speed from over 3 min per frame to less than 2 s per frame for the gradient-based optimization methods. To be specific, we propose a multi-scale mask fusion scheme to reduce untraceable flow errors, and obtain an incremental mask to reduce ghosting artefacts. In addition, a multi-frame mask fusion scheme is designed to reduce traceable flow errors. In our proposed losses, the Sharpness Losses are used to deal with the potential image blurriness artefacts over long-range frames, and the Coherent Losses are performed to restrict the temporal consistency at both the multi-frame RGB level and Feature level. Overall, our approach produces stable video stylization outputs even in large motion or occlusion scenarios. The experiments demonstrate that the proposed method outperforms the state-of-the-art video style transfer methods qualitatively and quantitatively on the MPI Sintel dataset.

Item Type:Article
Uncontrolled Keywords:style transfer; video stylization; video stabilization; deep networks
Group:Faculty of Media & Communication
ID Code:39724
Deposited By: Symplectic RT2
Deposited On:22 Apr 2024 10:13
Last Modified:22 Apr 2024 10:13


Downloads per month over past year

More statistics for this item...
Repository Staff Only -