- Google trials instant video background replacement on YouTube.
- Replaces the famous Hollywood ‘green screen’.
- Technology has been flagged for a smartphone rollout.
Pixel 2 users could be about to get a killer video app that replaces their backgrounds instantly.
For decades, film studios have relied on “chroma keying” to place human characters in otherworldy situations without them having to leave the studio. Most people know it as the “green screen”.
But video segmentation — separating the foreground scene from the background scene — is unwieldy, because you need a giant green screen; or time-consuming, if you choose instead to rotoscope every frame.
You can read all the technical details of how Google has brought the technology to mobile phones on its blog here. For now, the preliminary results are promising:
Not perfect, but already available to early (beta) users of YouTube’s stories feature.
Google says it developed the technique to work instantly, specifically in a way “that is suitable for mobile phones”.
It pulled data from “tens of thousands” of images that captured foreground and background poses, “hair, glasses, neck, skin, lips” and background scenes to achieve a 98% accuracy Intersection Over Union (IOU) rate.
Google says that accuracy stands with videos running at 100+ frames per second (FPS) on iPhone 7 and 40+ FPS on Pixel 2.
What the Pixel 2 does have, right now, is Google Augmented Reality service, which allows you to do things like this:
Google AR on Pixel is insane pic.twitter.com/VmzwBNxllc
— Harry Tucker (@harrytuckerr) December 13, 2017
Google says its “immediate goal” is to roll out its video segmentation tech to YouTube users, but “plan to integrate it into Google’s broader Augmented Reality services”.
If the search giant wanted a killer app to launch the Pixel 3 around October, sending out videos from anywhere in the galaxy would be a great place to start.