Dr. Erica Walker has a diverse background in visual communications which includes feature film production, web design & development, print buying, marketing, and graphic design for print. As a faculty member in the Department of Graphic Communications in the College of Business at Clemson University, Walker teaches courses in Photography, Video, Web Development, and Entrepreneurship. Walker is interested in many areas of research including software application, curriculum development, the entrepreneurial mindset, and artificial intelligence applications in color management for brands.

ColorNet: Use case of Artificial Intelligence in Sports Broadcasting

Dr. Erica Walker, Clemson University

Artificial Intelligence (AI) is being integrated into many areas of modern life from predictive texting to driverless vehicles. One industry that has seen significant growth in the application of AI is digital imaging. Arsenal, an AI-based camera assistant, raised nearly four million dollars on Kickstarter with the promise to take the best possible pictures by determining the optimal camera settings in any given situation (Tabora, 2020; This “AI Camera Assistant” Has Raised Over $3.7 Million on Kickstarter, 2020). The breakthrough algorithm, DeOldify that launched in 2018 and has since been used to add color and restore thousands of historical family photographs on MyHertiage.com (Chai Time Data Science, Premiered Feb 2, 2020; MyHeritage In Color™, World’s Most Advanced Technology for Colorizing Black and White Photos -MyHeritage, n.d.). For the past several years, Adobe has released new upgrades in their imageediting software (Photoshop for still images and Premiere and After Effects for video editing) that integrate AI tools. Examples include automating previously difficult tasks such as sky replacement, smart stock footage searches, frame-by-frame subject tracking in video, and smart selection tools (Adobe, n.d., Adobe Named One of Fast Company’s “Most Innovative Companies” for AI, n.d.).

Building on this rapid development of AI-based digital imaging tools, ColorNet is a patented AI software developed at Clemson University that performs targeted color correction to live video feeds in real-time. This AI-based color correction is unique because of several factors: 1. Targeted adjustment affects a specified color without negatively impacting other colors in the frame, 2. Real-time processing during a live video feed, and 3. The ability to continually adjust colors despite changing weather and lighting conditions. Initially, ColorNet was trained to recognize and correct a single specified brand color, but recent work has shifted the focus so that ColorNet can segment and assign regions of the video frame to allow for multiple independent color adjustments. This new approach allows for more flexibility from sport to sport and across different branded colors to complete multi-color, segmented adjustments in real-time.

Though successful, the original ColorNet approach lacked flexibility. It required a new, manually corrected dataset for each additional brand color that needed correction. In addition, ColorNet worked as a “black box” preventing video producers from controlling the outputs based on their expertise. The new ColorNet2 approach explicitly overcomes these limitations. Rather than specializing in one or two brand colors, like the original ColorNet, ColorNet2

extracts regions of the input video that are similar to an input reference brand color provided by the user. To correct a new brand color, the user no longer provides a new manually corrected dataset, but simply changes the input reference color. By selecting several reference colors, the user can target multiple brand colors at the same time. The ability of ColorNet2 to extract regions enables explicit controls over the colors within the selected regions. This gives the video production team much finer control over the presentation of brand colors in real-time.

Like ColorNet, ColorNet2 is based upon a machine learning algorithm. As input, ColorNet2 takes an image that needs to be color corrected along with a reference color used to control which brand colors are targeted. As output, ColorNet2 produces a “mask” which represents the pixel locations in the input image that correspond to targeted brand colors that need to be corrected. In order to train ColorNet2 to produce high-quality correction masks, the team has collected a novel dataset consisting of frames sampled from the video of sporting events paired with manually-produced segmentation masks (Figure 1).

Figure 1. A video frame from a Clemson football game (left) along with manual segmentation of Clemson orange (middle) and Clemson purple (right).

With this recent development, the team has developed a data augmentation process that allows the model to handle a wide variety of brand colors without manually producing a dataset for each color. This augmentation process synthesizes replicas of the collected data but where the native brand colors are transformed into a different color (for example, Clemson orange could appear blue, Figure 2). This allows the team to train ColorNet2 to handle a wide variety of brand colors even though the manually collected dataset only contains a small number of distinct brand colors.

Figure 2. An example of the ColorNet2 data augmentation. Clemson orange has been transformed into Duke blue.

Preservation of brand color integrity in live video broadcasting is an unsolved problem that is complicated by the varied imaging and environmental conditions as well as the need to make adjustments to local regions within the image without adjusting the overall tones throughout the image. The original ColorNet demonstrated how AI can be leveraged to address both of these issues in live video contexts, and now, ColorNet2 extends the applicability to a much wider array of brand colors while giving more control to video producers.

References

Adobe. (n.d.). Retrieved November 12, 2020, from https://www.fastcompany.com/company/adobe

Adobe Named One of Fast Company’s “Most Innovative Companies” for AI. (n.d.). Retrieved November 12, 2020, from https://blog.adobe.com/en/2018/02/20/adobe-named-one-fast-companys-innovative-companies-ai.html

Chai Time Data Science. (Premiered Feb 2, 2020). DeOldify | Fast.ai & NoGAN | Machine Learning & Software Engineering | Interview with Jason Antic. Youtube. https://www.youtube.com/watch?v=A5Cq8SWudts

MyHeritage In Color™, world’s most advanced technology for colorizing black and white photos -MyHeritage. (n.d.). Retrieved November 20, 2020, from https://www.myheritage.com/incolor

Tabora, V. (2020, January 27). The Dawn Of Intelligent Camera Assistants. High-Definition Pro. https://medium.com/hd-pro/the-dawn-of-intelligent-camera-assistants-b86cb789d412

This “AI Camera Assistant” Has Raised Over $3.7 Million on Kickstarter. (2020, September 24). https://petapixel.com/2020/09/24/this-ai-powered-camera-assistant-has-raised-3-7-million-on-kickstarter/