In 2010, Adobe introduced one of Photoshop’s first true ‘smart’ features: a tool called Content-Aware Fill that could intelligently remove and replace objects in a scene. Today, Adobe is now bringing a version of that tool to its compositing software, After Effects, that does essentially the same thing but on moving video clips, which is far more challenging.
One of the many reasons Adobe’s photography and video software has remained so dominant over the years is the company’s heavy investment in research and development, and the fact that many of the seemingly magical automated tools born from its R&D efforts actually make their way into its software. Admittedly, when it was initially rolled out in Photoshop CS5, Content-Aware Fill rarely worked as effortlessly or flawlessly as it appeared to in the company’s demos. But over time, and several versions of Photoshop, the tool was refined and improved to the point where, more often than not, its automated corrections became impossible to detect.
Adobe is now bringing Content-Aware Fill to After Effects, its digital compositing software that can be used for everything from creating animated graphics, to integrating CG animations with live footage, to replacing green screens. But intelligently erasing and replacing a portion of a static image is one thing; doing it on a video, across multiple frames of changing content, is an entirely other challenge.
Adobe says it’s made possible by not only the improvements made to the original Photoshop version of the tool over the years, but also by Adobe’s Sensei platform; the company’s AI and machine learning framework introduced a few years ago that now powers many of the company’s apps. As with Photoshop CS5's Content-Aware Fill tool, After Effects Creative Cloud handles all of the image processing on the local machine, so you’ll want to throw as much CPU at it as you can. But as Sensei continues to learn and improve its smarts, future updates of AE will get better and better at patching holes in footage.
Removing an object from a video won’t be as straightforward a process as it is for a photo. Users will have to create masks that define the region of a video that’s to be erased, either using hand-animated or painted masks, or After Effects’ automated tracking tools. It will be even more challenging for the software, as regions of a frame it uses to fill in holes after an object is removed might not actually still be in frame a few seconds later. So for optimal results, AE also allows a user to point the software to specific areas of a frame that will be best suited as filler for holes, or users can create reference still frames in Photoshop, that AE will use to guide its own edits.
Content-Aware Fill tool for video will be available for After Effects CC 2019 users starting today after a software update, and Adobe suggests it will be especially useful for jobs like removing rigging or boom mics from shots, erasing unwanted watermarks, or deleting a person from a clip altogether. Don’t expect flawless results right away, but over time this will undoubtedly become a powerful, and potentially controversial, tool. If you decide to run for office in 2020, there’s probably a few karaoke videos you wouldn’t mind disappearing from.
Correction, 2:34 p.m. EST/EDT: This story has been corrected to reflect that Adobe After Effects CC does not upload any footage or handle any of the image processing in the cloud; it’s all done on the local machine.