Video retargeting: Video saliency and optical flow based hybrid approach

As smart phones, tablets and similar computing devices become an integral part of our lives, we increasingly watch various types of streaming visual data on the display of those devices, especially as cloud video services become ubiquitous. One challenge is to present videos on diverse devices with an acceptable quality. Video retargeting is the key technology in video adaptation of cloud based video streaming. The most important challenge of video retargeting is to retain the shape of important objects, while ensuring temporal smoothness and coherence. We propose in this paper a new approach that adopts to the content of the video. We describe a cropping video retargeting method that ensures temporal coherence while enforcing spatial constraints by a saliency method. The average motion dynamics is calculated for each frame with optical flow and merged with the information of the user attention model for a given video. The resulting information is used to estimate a cropping window size. The output is a video that preserves important actions and the important parts of the scene. The results are promising in respect to overcoming the temporal and spatial challenges of video retargeting.