Earlier this yr, researchers from two universities and Google printed a brand new AI-powered approach they developed known as “Depth-Aware Video Frame Interpolation” or DAIN, and it’s merely mind-blowing. The tech can interpolate a 30fps video all the best way to 120fps and even 480fps with virtually no seen artifacts.
The crew behind this breakthrough was led by Wenbo Bao from Shanghai Jiao Tong College, and included pc scientists from the College of California Merced, and Google. Collectively, they used the ability of deep convoluted neural networks to considerably enhance the standard and functionality of video body interpolation, to the purpose the place you’d be hard-pressed to identify any artifacts.
You’ll be able to see the know-how at work within the cease movement video up prime, which has been up-framed from 15fps to 60fps with none seen artifacts by any means.
For a extra excessive instance, take a look at the video under. The unique footage (left) is simply 30fps. Utilizing DAIN, it’s been remodeled to 120fps (center) and even 480fps (proper), taking regular footage and creating super-slow movement shot utilizing nothing greater than AI to create the intervening frames de novo.
The strategy works by utilizing a “depth-aware movement projection layer,” which creates and considers a depth map and “optical movement layer” for the video because it decides find out how to create the intervening frames. This permits the algorithm to extra precisely predict the movement of assorted objects primarily based on the place they sit within the body, and accounts for occlusions extra precisely as effectively.
The outcome, because the researchers put it, “performs favorably in opposition to state-of-the-art body interpolation strategies on all kinds of datasets.”
Right here’s yet one more pattern video posted by Bao himself, the place DAIN is in comparison with different state-of-the-art body interpolation strategies when changing a video from 12fps, to 24fps, to 48fps. This video features a mixture of digital camera movement, a fast paced object, and slower transferring objects as effectively:
Positive, when you watch carefully sufficient you might even see the occasional artifact or spot an imperfection within the interpolation, however they’re shockingly uncommon, even when the body charge is being tripled, quadrupled, or extra.
Admittedly, this paper was printed on the very starting of 2020, and we’ve really already shared samples that took benefit of this method so as to add frames to traditional footage—see here, here, and here. However we’ve by no means dived into the approach itself or proven the outcomes which might be doable once you actually crank this as much as create super-slow movement video.
Try the samples above to be taught extra about precisely how this technique works and see the outcomes for your self. And if you wish to dive even deeper, you’ll be able to learn the total analysis paper or obtain the newest DAIN construct and take a look at it out from this link.
(by way of Reddit)